From 841a3ea77e7b0a1c10aa0e0727bb2595e22f07fa Mon Sep 17 00:00:00 2001 From: Stephen Belanger Date: Tue, 26 Jun 2018 18:36:43 -0700 Subject: [PATCH 01/11] docs: add performance tuning doc --- docs/performance.asciidoc | 86 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 86 insertions(+) create mode 100644 docs/performance.asciidoc diff --git a/docs/performance.asciidoc b/docs/performance.asciidoc new file mode 100644 index 0000000000..dd8c1136f7 --- /dev/null +++ b/docs/performance.asciidoc @@ -0,0 +1,86 @@ +[[performance]] + +ifdef::env-github[] +NOTE: For the best reading experience, +please view this documentation at https://www.elastic.co/guide/en/apm/agent/nodejs/current/performance.html[elastic.co] +endif::[] + +[float] +[[performance-tuning]] +== Performance Tuning + +There are many options available to tune agent performance. +Which option to adjust depends on whether you are optimizing for speed, memory usage, bandwidth or storage. + +[float] +[[performance-sampling]] +=== Sampling + +The first knob to reach for when tuning the performance of the agent is <>. +Adjusting the sampling rate controls what percent of requests are traced. +By default, the sample rate is set at `1.0`, meaning _all_ requests are traced. + +The sample rate will impact all four performance categories, so simply turning down the sample rate is an easy way to improve performance. + +[source,js] +---- +var apm = require('elastic-apm-node').start({ + transactionSampleRate: 0.2 +}) +---- + +[float] +[[performance-stack-traces]] +=== Stack Traces + +Stack traces can be a significant contributor to memory usage. There are several settings to adjust how they are used. + +[float] +[[performance-span-stack-traces]] +==== Span Stack Traces + +In a complex application, a request may produce many spans. +Capturing a stack trace for every span can result in significant memory usage. +To disable capturing stack traces for spans, set <> to `false`. + +This will mainly impact memory usage, but may also have a noticeable impact on speed too. +The cpu time required to capture and convert a stack frame to something JavaScript can understand is not insignificant. + +[float] +[[performance-source-lines]] +=== Source Lines + +If you want to keep span stack traces enabled for context, the next thing to try is adjusting how many source lines are reported for each stack trace. +When a stack trace is captured, the agent will also capture several lines of source code around each stack frame location in the stack trace. + +The are four different settings to control this behaviour: +- <> +- <> +- <> +- <> + +Source line settings are divided into app frames representing your app code and library frames representing the code of your dependencies. +App and library categories are both split into error and span groups. +Spans, by default, do not capture source lines. +Errors, by default, will capture five lines of code around each stack frame. + +Source lines are cached in-process. +In memory-constrained environments, the source line cache may use more memory than desired. +Turning the limits down will help prevent excessive memory use + +[float] +[[performance-stack-frame-limit]] +==== Stack Frame Limit + +The <> controls how many stack frames should be captured when producing an `Error` instance of any kind. + +This will mainly impact memory usage, but may also have a noticeable impact on speed too. +The cpu time required to capture and convert a stack frame to something JavaScript can understand is not insignificant. + +[float] +[[performance-error-log-stack-traces]] +==== Error Log Stack Traces + +Most stack traces recorded by the agent will point to where the error was instantiated, not where it was identified and reported to the agent with <>. +For this reason, the agent also has a feature to enable capturing an additional stack trace pointing to the place an error was reported to the agent. +By default, it will only capture the stack trace to the reporting point when <> is called with a string message. From a0fafe63765ed4c4a9b724937361c02e94776c74 Mon Sep 17 00:00:00 2001 From: Thomas Watson Date: Wed, 4 Jul 2018 11:56:52 +0200 Subject: [PATCH 02/11] docs: add link to performance page from index --- docs/index.asciidoc | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/index.asciidoc b/docs/index.asciidoc index dd6d29db6b..7dc5524231 100644 --- a/docs/index.asciidoc +++ b/docs/index.asciidoc @@ -28,6 +28,8 @@ include::./custom-transactions.asciidoc[] include::./custom-spans.asciidoc[] +include::./performance.asciidoc[] + include::./source-maps.asciidoc[] include::./compatibility.asciidoc[] From 70c35ff7507b8c18f547f56491be2cae7639cf3a Mon Sep 17 00:00:00 2001 From: Thomas Watson Date: Wed, 4 Jul 2018 11:57:17 +0200 Subject: [PATCH 03/11] docs: fix performance page header --- docs/performance.asciidoc | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/performance.asciidoc b/docs/performance.asciidoc index dd8c1136f7..4bf5d161c5 100644 --- a/docs/performance.asciidoc +++ b/docs/performance.asciidoc @@ -1,12 +1,10 @@ -[[performance]] +[[performance-tuning]] ifdef::env-github[] NOTE: For the best reading experience, please view this documentation at https://www.elastic.co/guide/en/apm/agent/nodejs/current/performance.html[elastic.co] endif::[] -[float] -[[performance-tuning]] == Performance Tuning There are many options available to tune agent performance. From d637a48ac41c84db5295787359f7a46aedb8c5e6 Mon Sep 17 00:00:00 2001 From: Thomas Watson Date: Wed, 4 Jul 2018 11:57:55 +0200 Subject: [PATCH 04/11] docs: add linebreaks after all punctuation --- docs/performance.asciidoc | 53 +++++++++++++++++++++++++++------------ 1 file changed, 37 insertions(+), 16 deletions(-) diff --git a/docs/performance.asciidoc b/docs/performance.asciidoc index 4bf5d161c5..5845daa40b 100644 --- a/docs/performance.asciidoc +++ b/docs/performance.asciidoc @@ -8,7 +8,9 @@ endif::[] == Performance Tuning There are many options available to tune agent performance. -Which option to adjust depends on whether you are optimizing for speed, memory usage, bandwidth or storage. +Which option to adjust depends on whether you are optimizing for speed, +memory usage, +bandwidth or storage. [float] [[performance-sampling]] @@ -16,9 +18,12 @@ Which option to adjust depends on whether you are optimizing for speed, memory u The first knob to reach for when tuning the performance of the agent is <>. Adjusting the sampling rate controls what percent of requests are traced. -By default, the sample rate is set at `1.0`, meaning _all_ requests are traced. +By default, +the sample rate is set at `1.0`, +meaning _all_ requests are traced. -The sample rate will impact all four performance categories, so simply turning down the sample rate is an easy way to improve performance. +The sample rate will impact all four performance categories, +so simply turning down the sample rate is an easy way to improve performance. [source,js] ---- @@ -31,27 +36,34 @@ var apm = require('elastic-apm-node').start({ [[performance-stack-traces]] === Stack Traces -Stack traces can be a significant contributor to memory usage. There are several settings to adjust how they are used. +Stack traces can be a significant contributor to memory usage. +There are several settings to adjust how they are used. [float] [[performance-span-stack-traces]] ==== Span Stack Traces -In a complex application, a request may produce many spans. +In a complex application, +a request may produce many spans. Capturing a stack trace for every span can result in significant memory usage. -To disable capturing stack traces for spans, set <> to `false`. +To disable capturing stack traces for spans, +set <> to `false`. -This will mainly impact memory usage, but may also have a noticeable impact on speed too. +This will mainly impact memory usage, +but may also have a noticeable impact on speed too. The cpu time required to capture and convert a stack frame to something JavaScript can understand is not insignificant. [float] [[performance-source-lines]] === Source Lines -If you want to keep span stack traces enabled for context, the next thing to try is adjusting how many source lines are reported for each stack trace. -When a stack trace is captured, the agent will also capture several lines of source code around each stack frame location in the stack trace. +If you want to keep span stack traces enabled for context, +the next thing to try is adjusting how many source lines are reported for each stack trace. +When a stack trace is captured, +the agent will also capture several lines of source code around each stack frame location in the stack trace. The are four different settings to control this behaviour: + - <> - <> - <> @@ -59,11 +71,16 @@ The are four different settings to control this behaviour: Source line settings are divided into app frames representing your app code and library frames representing the code of your dependencies. App and library categories are both split into error and span groups. -Spans, by default, do not capture source lines. -Errors, by default, will capture five lines of code around each stack frame. +Spans, +by default, +do not capture source lines. +Errors, +by default, +will capture five lines of code around each stack frame. Source lines are cached in-process. -In memory-constrained environments, the source line cache may use more memory than desired. +In memory-constrained environments, +the source line cache may use more memory than desired. Turning the limits down will help prevent excessive memory use [float] @@ -72,13 +89,17 @@ Turning the limits down will help prevent excessive memory use The <> controls how many stack frames should be captured when producing an `Error` instance of any kind. -This will mainly impact memory usage, but may also have a noticeable impact on speed too. +This will mainly impact memory usage, +but may also have a noticeable impact on speed too. The cpu time required to capture and convert a stack frame to something JavaScript can understand is not insignificant. [float] [[performance-error-log-stack-traces]] ==== Error Log Stack Traces -Most stack traces recorded by the agent will point to where the error was instantiated, not where it was identified and reported to the agent with <>. -For this reason, the agent also has a feature to enable capturing an additional stack trace pointing to the place an error was reported to the agent. -By default, it will only capture the stack trace to the reporting point when <> is called with a string message. +Most stack traces recorded by the agent will point to where the error was instantiated, +not where it was identified and reported to the agent with <>. +For this reason, +the agent also has a feature to enable capturing an additional stack trace pointing to the place an error was reported to the agent. +By default, +it will only capture the stack trace to the reporting point when <> is called with a string message. From ef9823b193ce7e60eae40bc0f59cc51699a11c3b Mon Sep 17 00:00:00 2001 From: Thomas Watson Date: Wed, 4 Jul 2018 11:58:15 +0200 Subject: [PATCH 05/11] docs: improve code example in performance page --- docs/performance.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/performance.asciidoc b/docs/performance.asciidoc index 5845daa40b..020cc6976b 100644 --- a/docs/performance.asciidoc +++ b/docs/performance.asciidoc @@ -27,7 +27,7 @@ so simply turning down the sample rate is an easy way to improve performance. [source,js] ---- -var apm = require('elastic-apm-node').start({ +require('elastic-apm-node').start({ transactionSampleRate: 0.2 }) ---- From bc624a364d3666233ffbb416f8f247ae1e6de55a Mon Sep 17 00:00:00 2001 From: Thomas Watson Date: Wed, 4 Jul 2018 12:01:35 +0200 Subject: [PATCH 06/11] docs: add extra comma --- docs/performance.asciidoc | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/performance.asciidoc b/docs/performance.asciidoc index 020cc6976b..676ed0a4de 100644 --- a/docs/performance.asciidoc +++ b/docs/performance.asciidoc @@ -10,7 +10,8 @@ endif::[] There are many options available to tune agent performance. Which option to adjust depends on whether you are optimizing for speed, memory usage, -bandwidth or storage. +bandwidth, +or storage. [float] [[performance-sampling]] From 73986a9df87211b15d1c3a5641ea001558708a22 Mon Sep 17 00:00:00 2001 From: Stephen Belanger Date: Thu, 5 Jul 2018 12:36:04 -0700 Subject: [PATCH 07/11] docs: expand performance doc --- docs/performance.asciidoc | 49 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 48 insertions(+), 1 deletion(-) diff --git a/docs/performance.asciidoc b/docs/performance.asciidoc index 676ed0a4de..dc5085e0b3 100644 --- a/docs/performance.asciidoc +++ b/docs/performance.asciidoc @@ -33,6 +33,41 @@ require('elastic-apm-node').start({ }) ---- +[float] +[[performance-transaction-queue]] +=== Transaction Queue + +The agent uses a queue to send to the apm server in batches. +This queue uses both a time interval and a size limit to balance memory usage against socket descriptor saturation. + +[float] +[[performance-max-queue-size]] +==== Max Queue Size + +The <> controls the maximum number of request transactions that may remain in the queue before they must be sent. + +Lowering this will reduce memory consumption, +however it will increase the number of requests made to the apm server. + +[float] +[[performance-flush-interval]] +==== Flush Interval + +To prevent items from staying in the queue for a long time during low activity, +a flush interval is used to ensure the queue empties if anything in it is too old. + +Lowering the flush interval will ensure transactions are sent to the apm server faster, +but may also result in increased HTTP traffic and cpu usage. + +[float] +[[performance-server-timeout]] +=== Server Timeout + +In the event that the connection to the apm server is slow or unstable, +a server timeout can be set to ensure connections don't stay open too long. +If the server timeout is too high, +it may result in too many socket descriptors being held open. + [float] [[performance-stack-traces]] === Stack Traces @@ -56,7 +91,7 @@ The cpu time required to capture and convert a stack frame to something JavaScri [float] [[performance-source-lines]] -=== Source Lines +===== Source Lines If you want to keep span stack traces enabled for context, the next thing to try is adjusting how many source lines are reported for each stack trace. @@ -104,3 +139,15 @@ For this reason, the agent also has a feature to enable capturing an additional stack trace pointing to the place an error was reported to the agent. By default, it will only capture the stack trace to the reporting point when <> is called with a string message. + +[float] +[[performance-transaction-max-spans]] +=== Spans + +The <> setting limits the number of spans which may be recorded within a single transaction before remaining spans are dropped. + +Spans may include many things such as a stack trace and context data. +Limiting the number of spans that may be recorded will reduce memory usage. + +Reducing max spans could result in loss of useful data about what occured within a request, +if it is set too low. From 71be1b3beda28fe4b72b5be06df738a71d761271 Mon Sep 17 00:00:00 2001 From: Stephen Belanger Date: Thu, 5 Jul 2018 12:37:26 -0700 Subject: [PATCH 08/11] docs: add performance tuning link to readme --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index f082d0078b..afef4a604b 100644 --- a/README.md +++ b/README.md @@ -29,6 +29,7 @@ please post them on the [Discuss forum](https://discuss.elastic.co/c/apm). - [API Reference](https://www.elastic.co/guide/en/apm/agent/nodejs/current/api.html) - [Custom Transactions](https://www.elastic.co/guide/en/apm/agent/nodejs/current/custom-transactions.html) - [Custom Spans](https://www.elastic.co/guide/en/apm/agent/nodejs/current/custom-spans.html) +- [Performance Tuning](https://www.elastic.co/guide/en/apm/agent/nodejs/current/performance-tuning.html) - [Source Map Support](https://www.elastic.co/guide/en/apm/agent/nodejs/current/source-maps.html) - [Compatibility Overview](https://www.elastic.co/guide/en/apm/agent/nodejs/current/compatibility.html) - [Upgrading](https://www.elastic.co/guide/en/apm/agent/nodejs/current/upgrading.html) From 5e96fbd0b7dcca1a118827a4287a12d44098e90c Mon Sep 17 00:00:00 2001 From: Stephen Belanger Date: Fri, 6 Jul 2018 12:58:53 -0700 Subject: [PATCH 09/11] docs: fix up some linking --- docs/index.asciidoc | 2 +- ...e.asciidoc => performance-tuning.asciidoc} | 42 +++++++++++-------- 2 files changed, 25 insertions(+), 19 deletions(-) rename docs/{performance.asciidoc => performance-tuning.asciidoc} (79%) diff --git a/docs/index.asciidoc b/docs/index.asciidoc index 7dc5524231..c5e2233bec 100644 --- a/docs/index.asciidoc +++ b/docs/index.asciidoc @@ -28,7 +28,7 @@ include::./custom-transactions.asciidoc[] include::./custom-spans.asciidoc[] -include::./performance.asciidoc[] +include::./performance-tuning.asciidoc[] include::./source-maps.asciidoc[] diff --git a/docs/performance.asciidoc b/docs/performance-tuning.asciidoc similarity index 79% rename from docs/performance.asciidoc rename to docs/performance-tuning.asciidoc index dc5085e0b3..8a8f7038fc 100644 --- a/docs/performance.asciidoc +++ b/docs/performance-tuning.asciidoc @@ -2,7 +2,7 @@ ifdef::env-github[] NOTE: For the best reading experience, -please view this documentation at https://www.elastic.co/guide/en/apm/agent/nodejs/current/performance.html[elastic.co] +please view this documentation at https://www.elastic.co/guide/en/apm/agent/nodejs/current/performance-tuning.html[elastic.co] endif::[] == Performance Tuning @@ -37,34 +37,37 @@ require('elastic-apm-node').start({ [[performance-transaction-queue]] === Transaction Queue -The agent uses a queue to send to the apm server in batches. -This queue uses both a time interval and a size limit to balance memory usage against socket descriptor saturation. - -[float] -[[performance-max-queue-size]] -==== Max Queue Size - -The <> controls the maximum number of request transactions that may remain in the queue before they must be sent. - -Lowering this will reduce memory consumption, -however it will increase the number of requests made to the apm server. +The agent buffers the collected data using an in-memory queue before sending it to the APM Server. +The queue is flushed either after a specific <> or when it reaches <> - +whichever comes first. +Lowering these defaults can reduce memory usage, +but will increase the number of requests to the APM Server. [float] [[performance-flush-interval]] ==== Flush Interval To prevent items from staying in the queue for a long time during low activity, -a flush interval is used to ensure the queue empties if anything in it is too old. +the <> setting is used to ensure the queue empties if anything in it is too old. -Lowering the flush interval will ensure transactions are sent to the apm server faster, +Lowering the flush interval will ensure transactions are sent to the APM Server faster, but may also result in increased HTTP traffic and cpu usage. +[float] +[[performance-max-queue-size]] +==== Max Queue Size + +The <> controls the maximum number of transactions that may remain in the queue before they must be sent. + +Lowering this will reduce memory consumption, +however it will increase the number of requests made to the APM Server. + [float] [[performance-server-timeout]] -=== Server Timeout +=== APM Server Timeout -In the event that the connection to the apm server is slow or unstable, -a server timeout can be set to ensure connections don't stay open too long. +In the event that the connection to the APM Server is slow or unstable, +the <> setting can be set to ensure connections don't stay open too long. If the server timeout is too high, it may result in too many socket descriptors being held open. @@ -136,10 +139,13 @@ The cpu time required to capture and convert a stack frame to something JavaScri Most stack traces recorded by the agent will point to where the error was instantiated, not where it was identified and reported to the agent with <>. For this reason, -the agent also has a feature to enable capturing an additional stack trace pointing to the place an error was reported to the agent. +the agent also has the <> setting to enable capturing an additional stack trace pointing to the place an error was reported to the agent. By default, it will only capture the stack trace to the reporting point when <> is called with a string message. +Setting this to `always` will increase memory and bandwidth usage, +so it helps to consider how frequently the app may capture errors. + [float] [[performance-transaction-max-spans]] === Spans From 88ce13cc83b32e3cc8bfd161b2ee7e776ff439df Mon Sep 17 00:00:00 2001 From: Stephen Belanger Date: Fri, 6 Jul 2018 14:04:24 -0700 Subject: [PATCH 10/11] docs: explain example --- docs/performance-tuning.asciidoc | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/performance-tuning.asciidoc b/docs/performance-tuning.asciidoc index 8a8f7038fc..d7c79bbf86 100644 --- a/docs/performance-tuning.asciidoc +++ b/docs/performance-tuning.asciidoc @@ -26,6 +26,8 @@ meaning _all_ requests are traced. The sample rate will impact all four performance categories, so simply turning down the sample rate is an easy way to improve performance. +Example setting the sample rate to 20%: + [source,js] ---- require('elastic-apm-node').start({ From 841f278540209e54d7bed7ddbd943021f6d8868c Mon Sep 17 00:00:00 2001 From: Stephen Belanger Date: Fri, 6 Jul 2018 14:09:39 -0700 Subject: [PATCH 11/11] docs: i accidentally an entire period --- docs/performance-tuning.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/performance-tuning.asciidoc b/docs/performance-tuning.asciidoc index d7c79bbf86..7556e72776 100644 --- a/docs/performance-tuning.asciidoc +++ b/docs/performance-tuning.asciidoc @@ -122,7 +122,7 @@ will capture five lines of code around each stack frame. Source lines are cached in-process. In memory-constrained environments, the source line cache may use more memory than desired. -Turning the limits down will help prevent excessive memory use +Turning the limits down will help prevent excessive memory use. [float] [[performance-stack-frame-limit]]