From 27a2064481dcd59e2c3c12bca42cba6111428e9c Mon Sep 17 00:00:00 2001 From: simitt Date: Thu, 5 Jul 2018 16:07:35 +0200 Subject: [PATCH 1/4] Restructure docs for RUM agent and ES related parts. * Remove the auto generated fields listing, and explain how users can fetch the APM template instead. * Add a separate RUM section and split up sourcemap docs according to the existing structure. * Introduce a new part how to explore data in ES and move everything related. * This leads to only having a general overview of Event Types left on top level, for easier high level understanding of APM Events. --- .gitignore | 1 + docs/configuration-rum.asciidoc | 14 + docs/configuring.asciidoc | 4 +- docs/error-api.asciidoc | 5 +- docs/error-indices.asciidoc | 38 + docs/errors.asciidoc | 36 +- docs/event-types.asciidoc | 7 +- docs/exploring-es-data.asciidoc | 63 ++ docs/fields.asciidoc | 1287 ----------------------------- docs/frontend.asciidoc | 17 - docs/healthcheck-indices.asciidoc | 6 + docs/index.asciidoc | 8 +- docs/intake-api.asciidoc | 2 + docs/rum.asciidoc | 55 ++ docs/sourcemap-api.asciidoc | 49 ++ docs/sourcemap-indices.asciidoc | 16 + docs/sourcemaps.asciidoc | 94 --- docs/span-indices.asciidoc | 38 + docs/spans.asciidoc | 43 - docs/transaction-api.asciidoc | 5 +- docs/transaction-indices.asciidoc | 37 + docs/transactions.asciidoc | 39 +- 22 files changed, 351 insertions(+), 1513 deletions(-) create mode 100644 docs/configuration-rum.asciidoc create mode 100644 docs/error-indices.asciidoc create mode 100644 docs/exploring-es-data.asciidoc delete mode 100644 docs/fields.asciidoc delete mode 100644 docs/frontend.asciidoc create mode 100644 docs/healthcheck-indices.asciidoc create mode 100644 docs/rum.asciidoc create mode 100644 docs/sourcemap-api.asciidoc create mode 100644 docs/sourcemap-indices.asciidoc delete mode 100644 docs/sourcemaps.asciidoc create mode 100644 docs/span-indices.asciidoc create mode 100644 docs/transaction-indices.asciidoc diff --git a/.gitignore b/.gitignore index ea1ad714592..ca0fb83f5b7 100644 --- a/.gitignore +++ b/.gitignore @@ -16,6 +16,7 @@ /_meta/kibana/*/index-pattern /_meta/kibana.generated/* /fields.yml +/docs/fields.asciidoc /apm-server.template-es.json html_docs diff --git a/docs/configuration-rum.asciidoc b/docs/configuration-rum.asciidoc new file mode 100644 index 00000000000..2136c788983 --- /dev/null +++ b/docs/configuration-rum.asciidoc @@ -0,0 +1,14 @@ +[[configuration-rum]] +== Configuring Real User Monitoring (RUM) Support + +This section describes configuration settings related to real user monitoring. + +[[rum-enable]] +[float] +=== Enable RUM Support +To try out real user montioring, set the `apm-server.frontend.enabled` to `true`. +See https://github.com/elastic/apm-server/blob/{doc-branch}/apm-server.yml[`apm-server.yml`] for configuration options. + +Read more about specific features related to <> +and how to set up the +{apm-rum-ref}/index.html[JavaScript RUM Agent]. diff --git a/docs/configuring.asciidoc b/docs/configuring.asciidoc index 86f0d0ffa58..36e7ce2fd32 100644 --- a/docs/configuring.asciidoc +++ b/docs/configuring.asciidoc @@ -13,7 +13,7 @@ The following topics describe how to configure APM Server: * <> * <> * <> -* <> +* <> * <> * <> -- @@ -32,7 +32,7 @@ include::./copied-from-beats/shared-kibana-config.asciidoc[] include::./copied-from-beats/dashboardsconfig.asciidoc[] -include::./frontend.asciidoc[] +include::./configuration-rum.asciidoc[] :standalone: include::./copied-from-beats/shared-env-vars.asciidoc[] diff --git a/docs/error-api.asciidoc b/docs/error-api.asciidoc index 70d59f42557..c071e83faa0 100644 --- a/docs/error-api.asciidoc +++ b/docs/error-api.asciidoc @@ -20,8 +20,9 @@ To send an error record you need to send a `HTTP POST` request to the APM Server http(s)://{hostname}:{port}/v1/errors ------------------------------------------------------------ -To send a record for a frontend error (see <>), -you need to send a `HTTP POST` request to the APM Server `frontend errors` endpoint: +To send a record for an error monitored by the +{apm-rum-ref}/index.html[JavaScript RUM Agent] +you need to send a `HTTP POST` request to the APM Server `client-side errors` endpoint: [source,bash] ------------------------------------------------------------ diff --git a/docs/error-indices.asciidoc b/docs/error-indices.asciidoc new file mode 100644 index 00000000000..31dcb148954 --- /dev/null +++ b/docs/error-indices.asciidoc @@ -0,0 +1,38 @@ +[[error-indices]] +== Error Indices + +Errors are stored in separate indices of the format `apm-[version]-error-[date]`. +Read more about the general purpose of <> documents. + +Some of the key attributes of an error are described in more detail: + +[[errors-context]] +include::./context.asciidoc[] + +[[errors-error]] +[float] +=== Error +An error event contains at least +information about the original `exception` that occured +or information about a `log` that was created when the exception occured. + +Both the captured `exception` and the captured `log` of an error can contain `stack trace` information, +helpful for debugging an error. + +The `culprit` of an error gives some information about the origin of the error. + +Errors also contain a `grouping_key` which can be used to group together identical or similar errors. + +An error can be mapped to the <> during which it happened, +via the `transaction.id`, indexed together with the error event. + +[[error-example]] +[float] +=== Example document + +Example of an error document indexed in Elasticsearch: + +[source,json] +---- +include::./data/elasticsearch/error.json[] +---- diff --git a/docs/errors.asciidoc b/docs/errors.asciidoc index 76c6d66bdfb..e63ff373c2c 100644 --- a/docs/errors.asciidoc +++ b/docs/errors.asciidoc @@ -3,35 +3,11 @@ An error record represents one error event, captured by Elastic APM agents within one service. It is identified by a unique ID. +Errors contain information about the `exception` that has been captured +and some meta information about the setting in which the error occured. -Some of the key attributes of an error are described in more detail: +Errors can either relate to monitored <> +or be monitored stand alone, +depending on how your APM agents are set up. -[[errors-context]] -include::./context.asciidoc[] - -[[errors-error]] -[float] -=== Error -An error event contains at least -information about the original `exception` that occured -or information about a `log` that was created when the exception occured. - -Both the captured `exception` and the captured `log` of an error can contain `stack trace` information, -helpful for debugging an error. - -The `culprit` of an error gives some information about the origin of the error. - -Errors also contain a `grouping_key` which can be used to group together identical or similar errors. - -An error can be mapped to the <> during which it happened, -via the `transaction.id`, indexed together with the error event. - - -[[error-example]] -[float] -=== Example document - -[source,json] ----- -include::./data/elasticsearch/error.json[] ----- +Read more about how an <> is indexed in Elasticsearch and which information erorr documents hold. diff --git a/docs/event-types.asciidoc b/docs/event-types.asciidoc index fb4dc843819..129435c2824 100644 --- a/docs/event-types.asciidoc +++ b/docs/event-types.asciidoc @@ -1,17 +1,14 @@ [[event-types]] = Event Types + [partintro] -- - -Elastic APM agents capture information about `transaction`, `span` and `error` events and send the information to the APM Server. +Elastic APM agents capture information about `transaction` and `error` events and send the information to the APM Server. The actual available information might be optimized per agent. * <> -* <> * <> - -- include::./transactions.asciidoc[] -include::./spans.asciidoc[] include::./errors.asciidoc[] diff --git a/docs/exploring-es-data.asciidoc b/docs/exploring-es-data.asciidoc new file mode 100644 index 00000000000..80f21db169c --- /dev/null +++ b/docs/exploring-es-data.asciidoc @@ -0,0 +1,63 @@ +[[exploring-es-data]] += Exploring data in Elasticsearch + +[partintro] +-- +By default Elastic APM data are stored in different index types. + +Indices storing monitored data are built in the format of `apm-[version]-[type]-[date]` by default. + +This format makes it easy to query for all Elastic APM data at once by using `apm-*`. + +It also allows for querying documents that have been collected with a specific APM Server version, e.g. `apm-{version}*`. + +If you are only interested in e.g. error documents you can use the type in your query `apm-*-error-*`. + +Adding the date is useful when you want to query for data in a specific time range, +or <>. + +Read more about the specific indices: + +* <> +* <> +* <> +* <> +* <> + +You can explore the raw data stored in Elasticsearch by querying the indices. + +For getting an overview of existing indices you can run: +["source","sh"] +------------------------------------------------------------ +GET _cat/indices +------------------------------------------------------------ +// CONSOLE + + +If you want to see which _settings_ and _mappings_ are applied to the Elastic APM indices you can fetch the index +templates. +First send a query to see which templates exist: + +["source","sh"] +------------------------------------------------------------ +GET _cat/templates +------------------------------------------------------------ +// CONSOLE + +Then you can retrieve the specific template you are interested in by sending: + +["source","sh"] +------------------------------------------------------------ +GET /_template/your-template-name +------------------------------------------------------------ +// CONSOLE + +You can read more about {ref}/indices-templates.html[Index Templates] and how they are used. +-- + + +include::./transaction-indices.asciidoc[] +include::./span-indices.asciidoc[] +include::./error-indices.asciidoc[] +include::./sourcemap-indices.asciidoc[] +include::./healthcheck-indices.asciidoc[] diff --git a/docs/fields.asciidoc b/docs/fields.asciidoc deleted file mode 100644 index 0f31df317f9..00000000000 --- a/docs/fields.asciidoc +++ /dev/null @@ -1,1287 +0,0 @@ - -//// -This file is generated! See _meta/fields.yml and scripts/generate_field_docs.py -//// - -[[exported-fields]] -= Exported fields - -[partintro] - --- -This document describes the fields that are exported by apm-server. They are -grouped in the following categories: - -* <> -* <> -* <> -* <> -* <> -* <> -* <> -* <> -* <> -* <> -* <> - --- -[[exported-fields-apm]] -== General APM fields fields - -None - - -*`listening`*:: -+ --- -type: keyword - -Address the server is listening on. - - --- - -*`processor.name`*:: -+ --- -type: keyword - -Processor name. - --- - -*`processor.event`*:: -+ --- -type: keyword - -Processor event. - --- - -[float] -== context fields - -Any arbitrary contextual information regarding the event, captured by the agent, optionally provided by the user. - - - -*`context.tags`*:: -+ --- -type: object - -A flat mapping of user-defined tags with string values. - - --- - - -*`context.user.username`*:: -+ --- -type: keyword - -The username of the logged in user. - - --- - -*`context.user.id`*:: -+ --- -type: keyword - -Identifier of the logged in user. - - --- - -*`context.user.email`*:: -+ --- -type: keyword - -Email of the logged in user. - - --- - -*`context.user.ip`*:: -+ --- -type: ip - -IP of the user where the event is recorded, typically a web browser. This is obtained from the X-Forwarded-For header, of which the first entry is the IP of the original client. This value however might not be necessarily trusted, as it can be forged by a malicious user. - - --- - -*`context.user.user-agent`*:: -+ --- -type: text - -Software agent acting in behalf of a user, eg. a web browser / OS combination. - - --- - - -[float] -== url fields - -A complete Url, with scheme, host and path. - - - -*`context.request.url.raw`*:: -+ --- -type: keyword - -The raw, unparsed URL of the request, e.g https://example.com:443/search?q=elasticsearch#top. - - --- - -*`context.request.url.protocol`*:: -+ --- -type: keyword - -The protocol of the request, e.g. "https:". - - --- - -*`context.request.url.full`*:: -+ --- -type: keyword - -The full, possibly agent-assembled URL of the request, e.g https://example.com:443/search?q=elasticsearch#top. - - --- - -*`context.request.url.hostname`*:: -+ --- -type: keyword - -The hostname of the request, e.g. "example.com". - - --- - -*`context.request.url.port`*:: -+ --- -type: keyword - -The port of the request, e.g. 443. - - --- - -*`context.request.url.pathname`*:: -+ --- -type: keyword - -The path of the request, e.g. "/search". - - --- - -*`context.request.url.search`*:: -+ --- -type: keyword - -The search describes the query string of the request, e.g. "q=elasticsearch". - - --- - -*`context.request.url.hash`*:: -+ --- -type: keyword - -The hash of the request URL, e.g. "top". - - --- - -*`context.request.http_version`*:: -+ --- -type: keyword - -The http version of the request leading to this event. - - --- - -*`context.request.method`*:: -+ --- -type: keyword - -The http method of the request leading to this event. - - --- - - -*`context.response.status_code`*:: -+ --- -type: long - -The http status code of the response, eg. '200'. - - --- - -*`context.response.finished`*:: -+ --- -type: boolean - -A boolean indicating whether the response was finished or not. - - --- - -[float] -== system fields - -Optional system fields. - - - -*`context.system.hostname`*:: -+ --- -type: keyword - -The hostname of the system the event was recorded on. - - --- - -*`context.system.architecture`*:: -+ --- -type: keyword - -The architecture of the system the event was recorded on. - - --- - -*`context.system.platform`*:: -+ --- -type: keyword - -The platform of the system the event was recorded on. - - --- - -*`context.system.ip`*:: -+ --- -type: ip - -IP of the host that records the event. - - --- - -[float] -== process fields - -Information pertaining to the running process where the data was collected - - - -*`context.process.pid`*:: -+ --- -type: long - -Numeric process ID of the service process. - - --- - -*`context.process.ppid`*:: -+ --- -type: long - -Numeric ID of the service's parent process. - - --- - -*`context.process.title`*:: -+ --- -type: keyword - -Service process title. - - --- - -[float] -== service fields - -Service fields. - - - -*`context.service.name`*:: -+ --- -type: keyword - -format: url - -Immutable unique name of the service emitting this event. - - --- - -*`context.service.version`*:: -+ --- -type: keyword - -Version of the service emitting this event. - - --- - -*`context.service.environment`*:: -+ --- -type: keyword - -Service environment. - - --- - - -*`context.service.language.name`*:: -+ --- -type: keyword - -Name of the programming language used. - - --- - -*`context.service.language.version`*:: -+ --- -type: keyword - -Version of the programming language used. - - --- - - -*`context.service.runtime.name`*:: -+ --- -type: keyword - -Name of the runtime used. - - --- - -*`context.service.runtime.version`*:: -+ --- -type: keyword - -Version of the runtime used. - - --- - - -*`context.service.framework.name`*:: -+ --- -type: keyword - -Name of the framework used. - - --- - -*`context.service.framework.version`*:: -+ --- -type: keyword - -Version of the framework used. - - --- - - -*`context.service.agent.name`*:: -+ --- -type: keyword - -Name of the agent used. - - --- - -*`context.service.agent.version`*:: -+ --- -type: keyword - -Version of the agent used. - - --- - - -*`transaction.id`*:: -+ --- -type: keyword - -format: url - -A UUID4 transaction ID. - - --- - -[[exported-fields-apm-error]] -== APM Error fields - -Error-specific data for APM - - -*`view errors`*:: -+ --- -type: keyword - -format: url - --- - -*`error id icon`*:: -+ --- -type: keyword - -format: url - --- - -[float] -== error fields - -Data captured by an agent representing an event occurring in a monitored service. - - - -*`error.id`*:: -+ --- -type: keyword - -A UUID for the error. - - --- - -*`error.culprit`*:: -+ --- -type: text - -Function call which was the primary perpetrator of this event. - --- - -*`error.grouping_key`*:: -+ --- -type: keyword - -format: url - -GroupingKey of the logged error for use in grouping. - - --- - -[float] -== exception fields - -Information about the originally thrown error. - - - -*`error.exception.code`*:: -+ --- -type: keyword - -The error code set when the error happened, e.g. database error code. - --- - -*`error.exception.message`*:: -+ --- -type: text - -The original error message. - --- - -*`error.exception.module`*:: -+ --- -type: keyword - -The module namespace of the original error. - --- - -*`error.exception.type`*:: -+ --- -type: keyword - --- - -*`error.exception.handled`*:: -+ --- -type: boolean - -Indicator whether the error was caught somewhere in the code or not. - --- - -[float] -== log fields - -Additional information added by logging the error. - - - -*`error.log.level`*:: -+ --- -type: keyword - -The severity of the record. - --- - -*`error.log.logger_name`*:: -+ --- -type: keyword - -The name of the logger instance used. - --- - -*`error.log.message`*:: -+ --- -type: text - -The additionally logged error message. - --- - -*`error.log.param_message`*:: -+ --- -type: keyword - -A parametrized message. E.g. 'Could not connect to %s'. The property message is still required, and should be equal to the param_message, but with placeholders replaced. In some situations the param_message is used to group errors together. - - --- - -[[exported-fields-apm-metric]] -== APM Metric fields - -APM Metrics for correlation with traces and logs - - -*`metric`*:: -+ --- -type: object - -Sampled metrics collected from the agent - - --- - -[[exported-fields-apm-sourcemap]] -== APM Sourcemap fields - -Sourcemap files enriched with metadata - - - -[float] -== service fields - -Service fields. - - - -*`sourcemap.service.name`*:: -+ --- -type: keyword - -The name of the service this sourcemap belongs to. - - --- - -*`sourcemap.service.version`*:: -+ --- -type: keyword - -Service version. - - --- - -*`sourcemap.bundle_filepath`*:: -+ --- -type: keyword - -Location of the sourcemap relative to the file requesting it. - - --- - -[[exported-fields-apm-span]] -== APM Span fields - -Span-specific data for APM. - - -*`view spans`*:: -+ --- -format: url - --- - - -*`span.id`*:: -+ --- -type: long - -A locally unique ID of the span. - - --- - -*`span.name`*:: -+ --- -type: keyword - -Generic designation of a span in the scope of a transaction. - - --- - -*`span.type`*:: -+ --- -type: keyword - -Keyword of specific relevance in the service's domain (eg: 'db.postgresql.query', 'template.erb', 'cache', etc). - - --- - -[float] -== start fields - -None - - -*`span.start.us`*:: -+ --- -type: long - -Offset relative to the transaction's timestamp identifying the start of the span, in microseconds. - - --- - -[float] -== duration fields - -None - - -*`span.duration.us`*:: -+ --- -type: long - -format: duration - -Duration of the span, in microseconds. - - --- - -*`span.parent`*:: -+ --- -type: long - -The locally unique ID of the parent of the span. - - --- - -[[exported-fields-apm-transaction]] -== APM Transaction fields - -Transaction-specific data for APM - - - -*`transaction.name`*:: -+ --- -type: text - -Generic designation of a transaction in the scope of a single service (eg. 'GET /users/:id'). - - -*`transaction.name.keyword`*:: -+ --- -type: keyword - --- - --- - -*`transaction.type`*:: -+ --- -type: keyword - -Keyword of specific relevance in the service's domain (eg. 'request', 'backgroundjob', etc) - - --- - -[float] -== duration fields - -None - - -*`transaction.duration.us`*:: -+ --- -type: long - -format: duration - -Total duration of this transaction, in microseconds. - - --- - -*`transaction.result`*:: -+ --- -type: keyword - -The result of the transaction. HTTP status code for HTTP-related transactions. - - --- - -*`transaction.marks`*:: -+ --- -type: object - -A user-defined mapping of groups of marks in milliseconds. - - --- - -*`transaction.marks.navigationTiming`*:: -+ --- -type: object - --- - -*`transaction.sampled`*:: -+ --- -type: boolean - -Transactions that are 'sampled' will include all available information. Transactions that are not sampled will not have spans or context. Defaults to true. - - --- - - - -*`transaction.span_count.dropped.total`*:: -+ --- -type: long - -The total amount of dropped spans for this transaction. - --- - -[[exported-fields-beat]] -== Beat fields - -Contains common beat fields available in all event types. - - - -*`beat.name`*:: -+ --- -The name of the Beat sending the log messages. If the Beat name is set in the configuration file, then that value is used. If it is not set, the hostname is used. To set the Beat name, use the `name` option in the configuration file. - - --- - -*`beat.hostname`*:: -+ --- -The hostname as returned by the operating system on which the Beat is running. - - --- - -*`beat.timezone`*:: -+ --- -The timezone as returned by the operating system on which the Beat is running. - - --- - -*`beat.version`*:: -+ --- -The version of the beat that generated this event. - - --- - -*`@timestamp`*:: -+ --- -type: date - -example: August 26th 2016, 12:35:53.332 - -format: date - -required: True - -The timestamp when the event log record was generated. - - --- - -*`tags`*:: -+ --- -Arbitrary tags that can be set per Beat and per transaction type. - - --- - -*`fields`*:: -+ --- -type: object - -Contains user configurable fields. - - --- - -[float] -== error fields - -Error fields containing additional info in case of errors. - - - -*`error.message`*:: -+ --- -type: text - -Error message. - - --- - -*`error.code`*:: -+ --- -type: long - -Error code. - - --- - -*`error.type`*:: -+ --- -type: keyword - -Error type. - - --- - -[[exported-fields-cloud]] -== Cloud provider metadata fields - -Metadata from cloud providers added by the add_cloud_metadata processor. - - - -*`meta.cloud.provider`*:: -+ --- -example: ec2 - -Name of the cloud provider. Possible values are ec2, gce, or digitalocean. - - --- - -*`meta.cloud.instance_id`*:: -+ --- -Instance ID of the host machine. - - --- - -*`meta.cloud.instance_name`*:: -+ --- -Instance name of the host machine. - - --- - -*`meta.cloud.machine_type`*:: -+ --- -example: t2.medium - -Machine type of the host machine. - - --- - -*`meta.cloud.availability_zone`*:: -+ --- -example: us-east-1c - -Availability zone in which this host is running. - - --- - -*`meta.cloud.project_id`*:: -+ --- -example: project-x - -Name of the project in Google Cloud. - - --- - -*`meta.cloud.region`*:: -+ --- -Region in which this host is running. - - --- - -[[exported-fields-docker-processor]] -== Docker fields - -Docker stats collected from Docker. - - - - -*`docker.container.id`*:: -+ --- -type: keyword - -Unique container id. - - --- - -*`docker.container.image`*:: -+ --- -type: keyword - -Name of the image the container was built on. - - --- - -*`docker.container.name`*:: -+ --- -type: keyword - -Container name. - - --- - -*`docker.container.labels`*:: -+ --- -type: object - -Image labels. - - --- - -[[exported-fields-host-processor]] -== Host fields - -Info collected for the host machine. - - - - -*`host.name`*:: -+ --- -type: keyword - -Hostname. - - --- - -*`host.id`*:: -+ --- -type: keyword - -Unique host id. - - --- - -*`host.architecture`*:: -+ --- -type: keyword - -Host architecture (e.g. x86_64, arm, ppc, mips). - - --- - -*`host.os.platform`*:: -+ --- -type: keyword - -OS platform (e.g. centos, ubuntu, windows). - - --- - -*`host.os.version`*:: -+ --- -type: keyword - -OS version. - - --- - -*`host.os.family`*:: -+ --- -type: keyword - -OS family (e.g. redhat, debian, freebsd, windows). - - --- - -*`host.ip`*:: -+ --- -type: ip - -List of IP-addresses. - - --- - -*`host.mac`*:: -+ --- -type: keyword - -List of hardware-addresses, usually MAC-addresses. - - --- - -[[exported-fields-kubernetes-processor]] -== Kubernetes fields - -Kubernetes metadata added by the kubernetes processor - - - - -*`kubernetes.pod.name`*:: -+ --- -type: keyword - -Kubernetes pod name - - --- - -*`kubernetes.pod.uid`*:: -+ --- -type: keyword - -Kubernetes pod uid - - --- - -*`kubernetes.namespace`*:: -+ --- -type: keyword - -Kubernetes namespace - - --- - -*`kubernetes.node.name`*:: -+ --- -type: keyword - -Kubernetes node name - - --- - -*`kubernetes.labels`*:: -+ --- -type: object - -Kubernetes labels map - - --- - -*`kubernetes.annotations`*:: -+ --- -type: object - -Kubernetes annotations map - - --- - -*`kubernetes.container.name`*:: -+ --- -type: keyword - -Kubernetes container name - - --- - -*`kubernetes.container.image`*:: -+ --- -type: keyword - -Kubernetes container image - - --- - diff --git a/docs/frontend.asciidoc b/docs/frontend.asciidoc deleted file mode 100644 index 74c204e1835..00000000000 --- a/docs/frontend.asciidoc +++ /dev/null @@ -1,17 +0,0 @@ -[[frontend]] -== Frontend support - -This section describes features specifically designed to support frontend application monitoring -and a guide on how to enable frontend support. - -[[frontend-enable]] -[float] -=== Enable Frontend Support -To try out frontend support, set the `apm-server.frontend.enabled` to `true`. -See https://github.com/elastic/apm-server/blob/{doc-branch}/apm-server.yml[`apm-server.yml`] for configuration options. - -Read more about frontend specific features: - -* <> - -include::./sourcemaps.asciidoc[] diff --git a/docs/healthcheck-indices.asciidoc b/docs/healthcheck-indices.asciidoc new file mode 100644 index 00000000000..406cf4c2f7d --- /dev/null +++ b/docs/healthcheck-indices.asciidoc @@ -0,0 +1,6 @@ +[[healthcheck-indices]] +== Healthcheck Indices + +Healthcheck indices by default are in the format of `apm-version-date`. +When starting an APM Server a healtcheck document is sent to Elasticsearch, +ensuring the configuration is set up properly. diff --git a/docs/index.asciidoc b/docs/index.asciidoc index 26f608a56cb..4404315ee42 100644 --- a/docs/index.asciidoc +++ b/docs/index.asciidoc @@ -34,15 +34,17 @@ include::./setting-up-and-running.asciidoc[] include::./configuring.asciidoc[] +include::./event-types.asciidoc[] + +include::./rum.asciidoc[] + include::./data_ingestion.asciidoc[] include::./storage_management.asciidoc[] include::./intake-api.asciidoc[] -include::./event-types.asciidoc[] - -include::./fields.asciidoc[] +include::./exploring-es-data.asciidoc[] include::./troubleshooting.asciidoc[] diff --git a/docs/intake-api.asciidoc b/docs/intake-api.asciidoc index a76acca5866..af8f3fd4bf3 100644 --- a/docs/intake-api.asciidoc +++ b/docs/intake-api.asciidoc @@ -6,6 +6,8 @@ The APM Server exposes API Endpoints for * <> * <> +* <> -- include::./transaction-api.asciidoc[] include::./error-api.asciidoc[] +include::./sourcemap-api.asciidoc[] diff --git a/docs/rum.asciidoc b/docs/rum.asciidoc new file mode 100644 index 00000000000..3787b876aa7 --- /dev/null +++ b/docs/rum.asciidoc @@ -0,0 +1,55 @@ +[[rum]] += Real User Monitoring (RUM) +[partintro] +-- +By supporting real user monitoring (RUM). +Elastic APM allows to collect performance data about your frontend JavaScript applications +by setting up the {apm-rum-ref}/index.html[JavaScript RUM Agent]. + +You can make use of <> when using Elastic RUM. +-- + +[[sourcemaps]] +== Source Maps +It is common practice to minify client-side JavaScript code for several reasons, e.g. performance gain. +This can make debugging very difficult, as it is hard to read the minified files. +Source mapping can help for debugging minfied JavaScript files, +by mapping code from the minified files to the original source code. + +APM Server provides a <> +which accepts source maps complying to the +https://docs.google.com/document/d/1U1RGAehQwRypUTovF1KRlpiOFze0b-_2gc6fAH0KY0k[Source map revision 3 proposal]. + +Uploaded source maps are used to map _stack trace_ information from recorded transaction and error documents +to the original source code files for easier debugging. + +[[sourcemap-apply]] +[float] +=== How source maps are applied + +When source maps have been <> and <> is enabled, +source mapping is automatically applied to the _stack trace frames_ of all errors and transactions +recorded with the +{apm-rum-ref}/index.html[JavaScript RUM Agent]. + +The server tries to find an uploaded source map for every `stack trace frame` of the record. +The following information is used to find the previously uploaded source map entry: + +* the record's `service.name` is matched against the source map's `service_name` +* the record's `service.version` is matched against the source map's `service_version` +* the stack trace frame's `abs_path` is matched against the source map's `bundle_filepath` + +If multiple source maps with the same meta information are found, +the source map with the latest upload timestamp is used. + +In case a matching source map is found and the source map can be applied to the `stack trace frame`, +the frame's information is updated with the mapped information before the record is indexed. +The following information is changed to reflect the original source code file, when source mapping is applied: + +* `filename` +* `function` +* `line number` +* `column number` +* `abs path`: is https://golang.org/pkg/path/#Clean[cleaned] to be the shortest path name equivalent to the given path name + +See how an <> looks like in Elasticsearch. diff --git a/docs/sourcemap-api.asciidoc b/docs/sourcemap-api.asciidoc new file mode 100644 index 00000000000..c4b6320b5b1 --- /dev/null +++ b/docs/sourcemap-api.asciidoc @@ -0,0 +1,49 @@ +[[sourcemap-api]] +== Sourcemap Upload API + +The APM Server exposes an API Endpoint to upload source maps, +helpful if you are using <>. + +[[sourcemap-endpoint]] +[float] +=== Upload Endpoint +To upload a source map you need to send a `HTTP POST` request +with `Content-Type` set to `multipart/form-data` to the APM Server source maps endpoint: + +[source,bash] +------------------------------------------------------------ +http(s)://{hostname}:{port}/v1/client-side/sourcemaps +------------------------------------------------------------ + +[[sourcemap-request-fields]] +[float] +==== Request Fields +The request consists of some meta information and the actual source map. +The meta information must contain the following fields: + +* `service_name` +* `service_version` +* `bundle_filepath`: needs to be the absolute path of the final bundle as it is used in the web application + +The meta information is used to identify a `source map` when source mapping is applied. + +The actual source map must be attached to the request as a `file upload` +and it must match the specification for +https://docs.google.com/document/d/1U1RGAehQwRypUTovF1KRlpiOFze0b-_2gc6fAH0KY0k[Source map revision 3 proposal]. + + +[[sourcemap-api-examples]] +[float] +==== Example + +Send an example source map to the APM Server: + +["source","sh",subs="attributes"] +--------------------------------------------------------------------------- +curl -X POST http://127.0.0.1:8200/v1/client-side/sourcemaps \ + -F service_name="test-service" \ + -F service_version="1.0" \ + -F bundle_filepath="http://localhost/static/js/bundle.js" \ + -F sourcemap=@bundle.js.map +--------------------------------------------------------------------------- + diff --git a/docs/sourcemap-indices.asciidoc b/docs/sourcemap-indices.asciidoc new file mode 100644 index 00000000000..48c20b8dafc --- /dev/null +++ b/docs/sourcemap-indices.asciidoc @@ -0,0 +1,16 @@ +[[sourcemap-indices]] +== Source Map Indices +Source maps are also stored in a separate index, +defined by the Elastic APM Server version and the date. + + +[[sourcemap-example]] +[float] +==== Example document + +Example of an acceptable source map file: + +[source,json] +---- +include::./data/intake-api/generated/sourcemap/bundle.js.map[] +---- diff --git a/docs/sourcemaps.asciidoc b/docs/sourcemaps.asciidoc deleted file mode 100644 index f787dfc893f..00000000000 --- a/docs/sourcemaps.asciidoc +++ /dev/null @@ -1,94 +0,0 @@ -[[sourcemap]] -=== Source Maps -It is common practice to minify client-side JavaScript code for several reasons, e.g. performance gain. -This can make debugging very difficult, as it is hard to read the minified files. -Source mapping can help for debugging minfied JavaScript files, -by mapping code from the minified files to the original source code. - -APM Server has a source map <> -which accepts source maps complying to the -https://docs.google.com/document/d/1U1RGAehQwRypUTovF1KRlpiOFze0b-_2gc6fAH0KY0k[Source map revision 3 proposal]. - -Uploaded source maps are used to map `stack trace` information from recorded `transaction` and `error` documents -to the original source code files for easier debugging. - -[[sourcemap-apply]] -[float] -==== How source maps are applied - -When source maps have been <> and <>, -source mapping is automatically applied to the `stack trace frames` of all `errors` and `transactions` -recorded with the https://github.com/elastic/apm-agent-js-base[JavaScript frontend agent]. - -The server tries to find an uploaded source map for every `stack trace frame` of the record. -The following information is used to find the previously uploaded source map entry: - -* the record's `service.name` is matched against the source map's `service_name` -* the record's `service.version` is matched against the source map's `service_version` -* the stack trace frame's `abs_path` is matched against the source map's `bundle_filepath` - -If multiple source maps with the same meta information are found, -the source map with the latest upload timestamp is used. - -In case a matching source map is found and the source map can be applied to the `stack trace frame`, -the frame's information is updated with the mapped information before the record is indexed. -The following information is changed to reflect the original source code file, when source mapping is applied: - -* `filename` -* `function` -* `line number` -* `column number` -* `abs path`: is https://golang.org/pkg/path/#Clean[cleaned] to be the shortest path name equivalent to the given path name - -[[sourcemap-endpoint]] -[float] -==== Upload Endpoint -To upload a source map you need to send a `HTTP POST` request -with `Content-Type` set to `multipart/form-data` to the APM Server source maps endpoint: - -[source,bash] ------------------------------------------------------------- -http(s)://{hostname}:{port}/v1/client-side/sourcemaps ------------------------------------------------------------- - -[[sourcemap-request-fields]] -[float] -===== Request Fields -The request consists of some meta information and the actual source map. -The meta information must contain the following fields: - -* `service_name` -* `service_version` -* `bundle_filepath`: needs to be the absolute path of the final bundle as it is used in the web application - -The meta information is used to identify a `source map` when source mapping is applied. - -The actual source map must be attached to the request as a `file upload` -and it must match the specification for -https://docs.google.com/document/d/1U1RGAehQwRypUTovF1KRlpiOFze0b-_2gc6fAH0KY0k[Source map revision 3 proposal]. - - -[[sourcemap-api-examples]] -[float] -==== Example - -Send an example source map to the APM Server: - -["source","sh",subs="attributes"] ---------------------------------------------------------------------------- -curl -X POST http://127.0.0.1:8200/v1/client-side/sourcemaps \ - -F service_name="test-service" \ - -F service_version="1.0" \ - -F bundle_filepath="http://localhost/static/js/bundle.js" \ - -F sourcemap=@bundle.js.map ---------------------------------------------------------------------------- - -[[sourcemap-json-example]] -[float] -===== Source map json file -Example of an acceptable source map file: - -[source,json] ----- -include::./data/intake-api/generated/sourcemap/bundle.js.map[] ----- diff --git a/docs/span-indices.asciidoc b/docs/span-indices.asciidoc new file mode 100644 index 00000000000..0b2b1084f61 --- /dev/null +++ b/docs/span-indices.asciidoc @@ -0,0 +1,38 @@ +[[span-indices]] +== Span Indices + +Spans are monitored as parts of <>. +They are stored in their separate indices `apm-[version]-span-[date]` though. + +Spans at least collect information about when the code path execution started, +the duration, and the type of the code path execution. + +This means, if for example a database query happens within a recorded transaction, +a span representing this database query will be created. +In such a case the name of the span will contain information about the query itself, +and the type will hold information about the database type. + +Some of the key attributes of a span are described in more detail: + +[[span-context]] +[float] +=== Context +In case a database query was captured, the span's context contains information about this database access. +The context also contains information about the service the agent is monitoring. + +[[span-span]] +[float] +=== Span +The information available within the span group includes the duration of the recorded span, +a unique ID within a transaction and an automatically retrieved name. +Additionally a span can contain `stack trace` information. + +[[span-example]] +[float] +=== Example document + +[source,json] +---- +include::./data/elasticsearch/span.json[] +---- + diff --git a/docs/spans.asciidoc b/docs/spans.asciidoc index 7e719f4c39b..e69de29bb2d 100644 --- a/docs/spans.asciidoc +++ b/docs/spans.asciidoc @@ -1,43 +0,0 @@ -[[spans]] -== Spans - -A span contains information about a specific code path, -executed as part of a <>. -Every span is identified by an unique ID per transaction. - -Elastic APM agents automatically instrument a variety of libraries, -but also support custom instrumentation for code paths. - -Every code path that is captured by an agent creates a span. -Spans at least collect information about when the code path execution started, -the duration, and the type of the code path execution. - -This means, if for example a database query happens within a recorded transaction, -a span representing this database query will be created. -In such a case the name of the span will contain information about the query itself, -and the type will hold information about the database type. - -Some of the key attributes of a span are described in more detail: - -[[span-context]] -[float] -=== Context -In case a database query was captured, the span's context contains information about this database access. -The context also contains information about the service the agent is monitoring. - -[[span-span]] -[float] -=== Span -The information available within the span group includes the duration of the recorded span, -a unique ID within a transaction and an automatically retrieved name. -Additionally a span can contain `stack trace` information. - -[[span-example]] -[float] -=== Example document - -[source,json] ----- -include::./data/elasticsearch/span.json[] ----- - diff --git a/docs/transaction-api.asciidoc b/docs/transaction-api.asciidoc index 9b950a16f7d..f3676695665 100644 --- a/docs/transaction-api.asciidoc +++ b/docs/transaction-api.asciidoc @@ -20,8 +20,9 @@ To send a transaction record you need to send a `HTTP POST` request to the APM S http(s)://{hostname}:{port}/v1/transactions ------------------------------------------------------------ -To send a record for a frontend transaction (see <>), -you need to send a `HTTP POST` request to the APM Server `frontend transactions` endpoint: +To send a record for a transaction monitored by the +{apm-rum-ref}/index.html[JavaScript RUM Agent] +you need to send a `HTTP POST` request to the APM Server `client-side transactions` endpoint: [source,bash] ------------------------------------------------------------ diff --git a/docs/transaction-indices.asciidoc b/docs/transaction-indices.asciidoc new file mode 100644 index 00000000000..fe6998d0da2 --- /dev/null +++ b/docs/transaction-indices.asciidoc @@ -0,0 +1,37 @@ +[[transaction-indices]] +== Transaction Indices + +Transactions are by default stored to indices of the format `apm-[version]-transaction-[date]`. +Read more about the general purpose of a <>. + +A transaction is identified by a unique ID and holds some meta information about the event. +It at least holds information about when the event started and the duration until it completed, +as well as information about the event type. + +Some of the key attributes of a transaction are described in more detail: + +[[transactions-context]] +include::./context.asciidoc[] + +[[transaction-transaction]] +[float] + +=== Transaction +The information available within the transaction group includes the duration of the transaction, +a unique id, the type and an automatically retrieved name, +as well as an indication whether or not the transaction was handled successfully. +The transaction can also contain: + +* sampled: if a transaction was sampled, the spans of the transaction were captured and are available as seperate documents +* span_count.dropped: how many spans have not been captured, according to configuration on the agent side +* marks: captures the timing in milliseconds of a significant event during the lifetime of a transaction, set by the user or the agent + +[[transaction-example]] +[float] +=== Example Document +See an example document of an indexed transaction: +[source,json] +---- +include::./data/elasticsearch/transaction.json[] +---- + diff --git a/docs/transactions.asciidoc b/docs/transactions.asciidoc index 2679da4a4ff..89fcf96794a 100644 --- a/docs/transactions.asciidoc +++ b/docs/transactions.asciidoc @@ -3,38 +3,21 @@ A transaction represents one event, captured by an Elastic APM agent within one service. It is identified by a unique ID. - -Within one transaction several <> can be captured, which have a `transaction.id` as a referrer to the transaction they belong to. - -A transaction at least holds information about when the event started and the duration until it completed, -as well as information about the kind of the event. A transaction can for example be a single HTTP request or an asynchrounous background job within one service. -Some of the key attributes of a transaction are described in more detail: - -[[transactions-context]] -include::./context.asciidoc[] - -[[transaction-transaction]] -[float] - -=== Transaction -The information available within the transaction group includes the duration of the transaction, -a unique id, the type and an automatically retrieved name, -as well as an indication whether or not the transaction was handled successfully. -The transaction can also contain: +Several spans can be related to a transaction. +The relationship is indicated by a `transaction.id`. -* sampled: if a transaction was sampled, the <> of the transaction were captured and are available as seperate documents -* span_count.dropped: how many spans have not been captured, according to configuration on the agent side -* marks: captures the timing in milliseconds of a significant event during the lifetime of a transaction, set by the user or the agent +A span contains information about a specific code path, +executed as part of a transaction. +Every span is identified by an unique ID per transaction. +Elastic APM agents automatically instrument a variety of libraries, +but also support custom instrumentation for code paths. -[[transaction-example]] -[float] -=== Example document +Every code path that is captured by an agent creates a span. -[source,json] ----- -include::./data/elasticsearch/transaction.json[] ----- +Transactions and spans are stored in separated indices by default. +Read more about their representation in <>. +See how <> and <> documents are indexed in Elasticsearch. From 67f418427587ac5d30dd3b3146f4c2e84e60193c Mon Sep 17 00:00:00 2001 From: simitt Date: Fri, 6 Jul 2018 15:58:58 +0200 Subject: [PATCH 2/4] changes according to PR comments. --- docs/exploring-es-data.asciidoc | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/docs/exploring-es-data.asciidoc b/docs/exploring-es-data.asciidoc index 80f21db169c..4e2e02e2ffc 100644 --- a/docs/exploring-es-data.asciidoc +++ b/docs/exploring-es-data.asciidoc @@ -3,9 +3,8 @@ [partintro] -- -By default Elastic APM data are stored in different index types. - -Indices storing monitored data are built in the format of `apm-[version]-[type]-[date]` by default. +By default Elastic APM data are stored in different indices, +in the format of `apm-[version]-[type]-[date]`. This format makes it easy to query for all Elastic APM data at once by using `apm-*`. @@ -29,7 +28,7 @@ You can explore the raw data stored in Elasticsearch by querying the indices. For getting an overview of existing indices you can run: ["source","sh"] ------------------------------------------------------------ -GET _cat/indices +GET _cat/indices/apm* ------------------------------------------------------------ // CONSOLE @@ -40,7 +39,7 @@ First send a query to see which templates exist: ["source","sh"] ------------------------------------------------------------ -GET _cat/templates +GET _cat/templates/apm* ------------------------------------------------------------ // CONSOLE From 7de27b19d62d71890e694ea3d5b5b2cc0afe5b2f Mon Sep 17 00:00:00 2001 From: simitt Date: Mon, 9 Jul 2018 11:59:09 +0200 Subject: [PATCH 3/4] Refactor and add PR comments. --- docs/context.asciidoc | 4 --- docs/error-indices.asciidoc | 21 --------------- docs/errors.asciidoc | 22 +++++++++++----- docs/exploring-es-data.asciidoc | 44 +++++++++++++++++++------------ docs/span-indices.asciidoc | 22 ---------------- docs/spans.asciidoc | 0 docs/storage_management.asciidoc | 4 +-- docs/transaction-indices.asciidoc | 22 ---------------- docs/transactions.asciidoc | 39 ++++++++++++++++++++++----- 9 files changed, 76 insertions(+), 102 deletions(-) delete mode 100644 docs/spans.asciidoc diff --git a/docs/context.asciidoc b/docs/context.asciidoc index 2c0bb05f212..ba8bf3c7a86 100644 --- a/docs/context.asciidoc +++ b/docs/context.asciidoc @@ -1,5 +1,3 @@ -[float] -=== Context An event's context bundles information regarding the environment in which it is recorded. It describes the `service` in which the event is captured, the `system` in which the monitored service is running and the event's `process` information. @@ -12,5 +10,3 @@ In case a http request is captured, it contains information about the `url`, `co The agents provide some configuration options with which the users can also capture customized information. The non-indexed information is captured within a `custom` object, while the searchable information is stored within `tags`. - -The context information is defined for <> and <> events. diff --git a/docs/error-indices.asciidoc b/docs/error-indices.asciidoc index 31dcb148954..ede65191406 100644 --- a/docs/error-indices.asciidoc +++ b/docs/error-indices.asciidoc @@ -4,27 +4,6 @@ Errors are stored in separate indices of the format `apm-[version]-error-[date]`. Read more about the general purpose of <> documents. -Some of the key attributes of an error are described in more detail: - -[[errors-context]] -include::./context.asciidoc[] - -[[errors-error]] -[float] -=== Error -An error event contains at least -information about the original `exception` that occured -or information about a `log` that was created when the exception occured. - -Both the captured `exception` and the captured `log` of an error can contain `stack trace` information, -helpful for debugging an error. - -The `culprit` of an error gives some information about the origin of the error. - -Errors also contain a `grouping_key` which can be used to group together identical or similar errors. - -An error can be mapped to the <> during which it happened, -via the `transaction.id`, indexed together with the error event. [[error-example]] [float] diff --git a/docs/errors.asciidoc b/docs/errors.asciidoc index e63ff373c2c..3716caba6d2 100644 --- a/docs/errors.asciidoc +++ b/docs/errors.asciidoc @@ -1,13 +1,21 @@ [[errors]] == Errors -An error record represents one error event, captured by Elastic APM agents within one service. +An error record represents one error event, +captured by Elastic APM agents within one service. It is identified by a unique ID. -Errors contain information about the `exception` that has been captured -and some meta information about the setting in which the error occured. +An error event contains at least +information about the original `exception` that occured +or information about a `log` that was created when the exception occured. -Errors can either relate to monitored <> -or be monitored stand alone, -depending on how your APM agents are set up. +Both the captured `exception` and the captured `log` of an error can contain `stack trace` information, +helpful for debugging an error. -Read more about how an <> is indexed in Elasticsearch and which information erorr documents hold. +The `culprit` of an error gives some information about the origin of the error. + +An error can be mapped to the <> during which it happened, +via the `transaction.id`. + +include::./context.asciidoc[] + +Read more about how an <> is indexed in Elasticsearch. diff --git a/docs/exploring-es-data.asciidoc b/docs/exploring-es-data.asciidoc index 4e2e02e2ffc..375e51dc189 100644 --- a/docs/exploring-es-data.asciidoc +++ b/docs/exploring-es-data.asciidoc @@ -4,18 +4,17 @@ [partintro] -- By default Elastic APM data are stored in different indices, -in the format of `apm-[version]-[type]-[date]`. +in the format of +`apm-%{[beat.version]}-{type}-%{+yyyy.MM.dd}`. -This format makes it easy to query for all Elastic APM data at once by using `apm-*`. - -It also allows for querying documents that have been collected with a specific APM Server version, e.g. `apm-{version}*`. - -If you are only interested in e.g. error documents you can use the type in your query `apm-*-error-*`. - -Adding the date is useful when you want to query for data in a specific time range, -or <>. +For getting an overview of existing indices you can run: +["source","sh"] +------------------------------------------------------------ +GET _cat/indices/apm* +------------------------------------------------------------ +// CONSOLE -Read more about the specific indices: +Following indices are created by default: * <> * <> @@ -23,19 +22,31 @@ Read more about the specific indices: * <> * <> -You can explore the raw data stored in Elasticsearch by querying the indices. -For getting an overview of existing indices you can run: +For querying all APM data: ["source","sh"] ------------------------------------------------------------ -GET _cat/indices/apm* +GET apm*/_search ------------------------------------------------------------ // CONSOLE +Querying documents that have been collected with a specific APM Server version: +["source","sh",subs="attributes"] +------------------------------------------------------------ +GET apm-{version}-*/_search +------------------------------------------------------------ +// CONSOLE -If you want to see which _settings_ and _mappings_ are applied to the Elastic APM indices you can fetch the index -templates. -First send a query to see which templates exist: +If you are only interested in specific document types, e.g. error documents you can use the type in your query: +["source","sh",subs="attributes"] +------------------------------------------------------------ +GET apm-*error-*/_search +------------------------------------------------------------ +// CONSOLE + +If you are interested in the _settings_ and _mappings_ applied to the Elastic APM indices, +you can fetch the index templates. +First run a query for figuring out which templates exist: ["source","sh"] ------------------------------------------------------------ @@ -44,7 +55,6 @@ GET _cat/templates/apm* // CONSOLE Then you can retrieve the specific template you are interested in by sending: - ["source","sh"] ------------------------------------------------------------ GET /_template/your-template-name diff --git a/docs/span-indices.asciidoc b/docs/span-indices.asciidoc index 0b2b1084f61..e8fcca53f6b 100644 --- a/docs/span-indices.asciidoc +++ b/docs/span-indices.asciidoc @@ -4,28 +4,6 @@ Spans are monitored as parts of <>. They are stored in their separate indices `apm-[version]-span-[date]` though. -Spans at least collect information about when the code path execution started, -the duration, and the type of the code path execution. - -This means, if for example a database query happens within a recorded transaction, -a span representing this database query will be created. -In such a case the name of the span will contain information about the query itself, -and the type will hold information about the database type. - -Some of the key attributes of a span are described in more detail: - -[[span-context]] -[float] -=== Context -In case a database query was captured, the span's context contains information about this database access. -The context also contains information about the service the agent is monitoring. - -[[span-span]] -[float] -=== Span -The information available within the span group includes the duration of the recorded span, -a unique ID within a transaction and an automatically retrieved name. -Additionally a span can contain `stack trace` information. [[span-example]] [float] diff --git a/docs/spans.asciidoc b/docs/spans.asciidoc deleted file mode 100644 index e69de29bb2d..00000000000 diff --git a/docs/storage_management.asciidoc b/docs/storage_management.asciidoc index 84c01888f09..1da9bd27632 100644 --- a/docs/storage_management.asciidoc +++ b/docs/storage_management.asciidoc @@ -62,7 +62,7 @@ To do this you can use a tool like {curator-ref-current}[Curator] and set up a c By default APM indices have the pattern `apm-%{[beat.version]}-{type}-%{+yyyy.MM.dd}`. With the curator command line interface you can, for instance, see all your existing indices: -["source","sh"] +["source","sh",subs="attributes"] ------------------------------------------------------------ curator_cli --host localhost show_indices --filter_list '[\{"filtertype":"pattern","kind":"prefix","value":"apm-"\}]' @@ -80,7 +80,7 @@ apm-{stack-version}-transaction-{sample_date_2} And then delete any span indices older than 1 day: -["source","sh"] +["source","sh",subs="attributes"] ------------------------------------------------------------ curator_cli --host localhost delete_indices --filter_list '[\{"filtertype":"pattern","kind":"prefix","value":"apm-{stack-version}-span-"\}, \{"filtertype":"age","source":"name","timestring":"%Y.%m.%d","unit":"days","unit_count":1,"direction":"older"\}]' diff --git a/docs/transaction-indices.asciidoc b/docs/transaction-indices.asciidoc index fe6998d0da2..d187ab0958d 100644 --- a/docs/transaction-indices.asciidoc +++ b/docs/transaction-indices.asciidoc @@ -4,28 +4,6 @@ Transactions are by default stored to indices of the format `apm-[version]-transaction-[date]`. Read more about the general purpose of a <>. -A transaction is identified by a unique ID and holds some meta information about the event. -It at least holds information about when the event started and the duration until it completed, -as well as information about the event type. - -Some of the key attributes of a transaction are described in more detail: - -[[transactions-context]] -include::./context.asciidoc[] - -[[transaction-transaction]] -[float] - -=== Transaction -The information available within the transaction group includes the duration of the transaction, -a unique id, the type and an automatically retrieved name, -as well as an indication whether or not the transaction was handled successfully. -The transaction can also contain: - -* sampled: if a transaction was sampled, the spans of the transaction were captured and are available as seperate documents -* span_count.dropped: how many spans have not been captured, according to configuration on the agent side -* marks: captures the timing in milliseconds of a significant event during the lifetime of a transaction, set by the user or the agent - [[transaction-example]] [float] === Example Document diff --git a/docs/transactions.asciidoc b/docs/transactions.asciidoc index 89fcf96794a..6cc2965dede 100644 --- a/docs/transactions.asciidoc +++ b/docs/transactions.asciidoc @@ -2,22 +2,47 @@ == Transactions A transaction represents one event, captured by an Elastic APM agent within one service. -It is identified by a unique ID. A transaction can for example be a single HTTP request or an asynchrounous background job within one service. +The transaction shows the duration of the event, +a unique id, the type and an automatically retrieved name, +as well as an indication whether or not the transaction was handled successfully. + + +If a transaction was sampled, +the <> of the transaction were captured and are available as seperate documents. +There is a configuration option on how many spans should be captured per transaction. + +A transaction can also contain information regarding `marks`. +Marks capture the timing in milliseconds of a significant event during the lifetime of a transaction, +set by the user or the agent + +[[transactions-context]] +include::./context.asciidoc[] + +[[transaction-spans]] +[float] +=== Spans Several spans can be related to a transaction. The relationship is indicated by a `transaction.id`. A span contains information about a specific code path, executed as part of a transaction. -Every span is identified by an unique ID per transaction. - Elastic APM agents automatically instrument a variety of libraries, but also support custom instrumentation for code paths. - Every code path that is captured by an agent creates a span. -Transactions and spans are stored in separated indices by default. -Read more about their representation in <>. +Every span is identified by an unique ID per transaction. +Spans at least collect information about when the code path execution started, +the duration, and the type of the code path execution. -See how <> and <> documents are indexed in Elasticsearch. +This means, if for example a database query happens within a recorded transaction, +a span representing this database query will be created. +In such a case the name of the span will contain information about the query itself, +and the type will hold information about the database type. + +A span can also contain `stack trace` information. + +Transactions and spans are stored in separated indices by default. +Read more about their representation in Elasticsearch in <> and +<>. From cbc4c2670934b6a82f04d337524c329e678b7155 Mon Sep 17 00:00:00 2001 From: simitt Date: Mon, 9 Jul 2018 16:57:28 +0200 Subject: [PATCH 4/4] Add fields again after discussions. --- .gitignore | 1 - docs/exploring-es-data.asciidoc | 3 +- docs/fields.asciidoc | 1287 +++++++++++++++++++++++++++++++ docs/index.asciidoc | 2 + 4 files changed, 1291 insertions(+), 2 deletions(-) create mode 100644 docs/fields.asciidoc diff --git a/.gitignore b/.gitignore index ca0fb83f5b7..ea1ad714592 100644 --- a/.gitignore +++ b/.gitignore @@ -16,7 +16,6 @@ /_meta/kibana/*/index-pattern /_meta/kibana.generated/* /fields.yml -/docs/fields.asciidoc /apm-server.template-es.json html_docs diff --git a/docs/exploring-es-data.asciidoc b/docs/exploring-es-data.asciidoc index 375e51dc189..5ef08e4adfa 100644 --- a/docs/exploring-es-data.asciidoc +++ b/docs/exploring-es-data.asciidoc @@ -14,8 +14,9 @@ GET _cat/indices/apm* ------------------------------------------------------------ // CONSOLE -Following indices are created by default: +Default APM `template` and `indices`: +//* <> * <> * <> * <> diff --git a/docs/fields.asciidoc b/docs/fields.asciidoc new file mode 100644 index 00000000000..0f31df317f9 --- /dev/null +++ b/docs/fields.asciidoc @@ -0,0 +1,1287 @@ + +//// +This file is generated! See _meta/fields.yml and scripts/generate_field_docs.py +//// + +[[exported-fields]] += Exported fields + +[partintro] + +-- +This document describes the fields that are exported by apm-server. They are +grouped in the following categories: + +* <> +* <> +* <> +* <> +* <> +* <> +* <> +* <> +* <> +* <> +* <> + +-- +[[exported-fields-apm]] +== General APM fields fields + +None + + +*`listening`*:: ++ +-- +type: keyword + +Address the server is listening on. + + +-- + +*`processor.name`*:: ++ +-- +type: keyword + +Processor name. + +-- + +*`processor.event`*:: ++ +-- +type: keyword + +Processor event. + +-- + +[float] +== context fields + +Any arbitrary contextual information regarding the event, captured by the agent, optionally provided by the user. + + + +*`context.tags`*:: ++ +-- +type: object + +A flat mapping of user-defined tags with string values. + + +-- + + +*`context.user.username`*:: ++ +-- +type: keyword + +The username of the logged in user. + + +-- + +*`context.user.id`*:: ++ +-- +type: keyword + +Identifier of the logged in user. + + +-- + +*`context.user.email`*:: ++ +-- +type: keyword + +Email of the logged in user. + + +-- + +*`context.user.ip`*:: ++ +-- +type: ip + +IP of the user where the event is recorded, typically a web browser. This is obtained from the X-Forwarded-For header, of which the first entry is the IP of the original client. This value however might not be necessarily trusted, as it can be forged by a malicious user. + + +-- + +*`context.user.user-agent`*:: ++ +-- +type: text + +Software agent acting in behalf of a user, eg. a web browser / OS combination. + + +-- + + +[float] +== url fields + +A complete Url, with scheme, host and path. + + + +*`context.request.url.raw`*:: ++ +-- +type: keyword + +The raw, unparsed URL of the request, e.g https://example.com:443/search?q=elasticsearch#top. + + +-- + +*`context.request.url.protocol`*:: ++ +-- +type: keyword + +The protocol of the request, e.g. "https:". + + +-- + +*`context.request.url.full`*:: ++ +-- +type: keyword + +The full, possibly agent-assembled URL of the request, e.g https://example.com:443/search?q=elasticsearch#top. + + +-- + +*`context.request.url.hostname`*:: ++ +-- +type: keyword + +The hostname of the request, e.g. "example.com". + + +-- + +*`context.request.url.port`*:: ++ +-- +type: keyword + +The port of the request, e.g. 443. + + +-- + +*`context.request.url.pathname`*:: ++ +-- +type: keyword + +The path of the request, e.g. "/search". + + +-- + +*`context.request.url.search`*:: ++ +-- +type: keyword + +The search describes the query string of the request, e.g. "q=elasticsearch". + + +-- + +*`context.request.url.hash`*:: ++ +-- +type: keyword + +The hash of the request URL, e.g. "top". + + +-- + +*`context.request.http_version`*:: ++ +-- +type: keyword + +The http version of the request leading to this event. + + +-- + +*`context.request.method`*:: ++ +-- +type: keyword + +The http method of the request leading to this event. + + +-- + + +*`context.response.status_code`*:: ++ +-- +type: long + +The http status code of the response, eg. '200'. + + +-- + +*`context.response.finished`*:: ++ +-- +type: boolean + +A boolean indicating whether the response was finished or not. + + +-- + +[float] +== system fields + +Optional system fields. + + + +*`context.system.hostname`*:: ++ +-- +type: keyword + +The hostname of the system the event was recorded on. + + +-- + +*`context.system.architecture`*:: ++ +-- +type: keyword + +The architecture of the system the event was recorded on. + + +-- + +*`context.system.platform`*:: ++ +-- +type: keyword + +The platform of the system the event was recorded on. + + +-- + +*`context.system.ip`*:: ++ +-- +type: ip + +IP of the host that records the event. + + +-- + +[float] +== process fields + +Information pertaining to the running process where the data was collected + + + +*`context.process.pid`*:: ++ +-- +type: long + +Numeric process ID of the service process. + + +-- + +*`context.process.ppid`*:: ++ +-- +type: long + +Numeric ID of the service's parent process. + + +-- + +*`context.process.title`*:: ++ +-- +type: keyword + +Service process title. + + +-- + +[float] +== service fields + +Service fields. + + + +*`context.service.name`*:: ++ +-- +type: keyword + +format: url + +Immutable unique name of the service emitting this event. + + +-- + +*`context.service.version`*:: ++ +-- +type: keyword + +Version of the service emitting this event. + + +-- + +*`context.service.environment`*:: ++ +-- +type: keyword + +Service environment. + + +-- + + +*`context.service.language.name`*:: ++ +-- +type: keyword + +Name of the programming language used. + + +-- + +*`context.service.language.version`*:: ++ +-- +type: keyword + +Version of the programming language used. + + +-- + + +*`context.service.runtime.name`*:: ++ +-- +type: keyword + +Name of the runtime used. + + +-- + +*`context.service.runtime.version`*:: ++ +-- +type: keyword + +Version of the runtime used. + + +-- + + +*`context.service.framework.name`*:: ++ +-- +type: keyword + +Name of the framework used. + + +-- + +*`context.service.framework.version`*:: ++ +-- +type: keyword + +Version of the framework used. + + +-- + + +*`context.service.agent.name`*:: ++ +-- +type: keyword + +Name of the agent used. + + +-- + +*`context.service.agent.version`*:: ++ +-- +type: keyword + +Version of the agent used. + + +-- + + +*`transaction.id`*:: ++ +-- +type: keyword + +format: url + +A UUID4 transaction ID. + + +-- + +[[exported-fields-apm-error]] +== APM Error fields + +Error-specific data for APM + + +*`view errors`*:: ++ +-- +type: keyword + +format: url + +-- + +*`error id icon`*:: ++ +-- +type: keyword + +format: url + +-- + +[float] +== error fields + +Data captured by an agent representing an event occurring in a monitored service. + + + +*`error.id`*:: ++ +-- +type: keyword + +A UUID for the error. + + +-- + +*`error.culprit`*:: ++ +-- +type: text + +Function call which was the primary perpetrator of this event. + +-- + +*`error.grouping_key`*:: ++ +-- +type: keyword + +format: url + +GroupingKey of the logged error for use in grouping. + + +-- + +[float] +== exception fields + +Information about the originally thrown error. + + + +*`error.exception.code`*:: ++ +-- +type: keyword + +The error code set when the error happened, e.g. database error code. + +-- + +*`error.exception.message`*:: ++ +-- +type: text + +The original error message. + +-- + +*`error.exception.module`*:: ++ +-- +type: keyword + +The module namespace of the original error. + +-- + +*`error.exception.type`*:: ++ +-- +type: keyword + +-- + +*`error.exception.handled`*:: ++ +-- +type: boolean + +Indicator whether the error was caught somewhere in the code or not. + +-- + +[float] +== log fields + +Additional information added by logging the error. + + + +*`error.log.level`*:: ++ +-- +type: keyword + +The severity of the record. + +-- + +*`error.log.logger_name`*:: ++ +-- +type: keyword + +The name of the logger instance used. + +-- + +*`error.log.message`*:: ++ +-- +type: text + +The additionally logged error message. + +-- + +*`error.log.param_message`*:: ++ +-- +type: keyword + +A parametrized message. E.g. 'Could not connect to %s'. The property message is still required, and should be equal to the param_message, but with placeholders replaced. In some situations the param_message is used to group errors together. + + +-- + +[[exported-fields-apm-metric]] +== APM Metric fields + +APM Metrics for correlation with traces and logs + + +*`metric`*:: ++ +-- +type: object + +Sampled metrics collected from the agent + + +-- + +[[exported-fields-apm-sourcemap]] +== APM Sourcemap fields + +Sourcemap files enriched with metadata + + + +[float] +== service fields + +Service fields. + + + +*`sourcemap.service.name`*:: ++ +-- +type: keyword + +The name of the service this sourcemap belongs to. + + +-- + +*`sourcemap.service.version`*:: ++ +-- +type: keyword + +Service version. + + +-- + +*`sourcemap.bundle_filepath`*:: ++ +-- +type: keyword + +Location of the sourcemap relative to the file requesting it. + + +-- + +[[exported-fields-apm-span]] +== APM Span fields + +Span-specific data for APM. + + +*`view spans`*:: ++ +-- +format: url + +-- + + +*`span.id`*:: ++ +-- +type: long + +A locally unique ID of the span. + + +-- + +*`span.name`*:: ++ +-- +type: keyword + +Generic designation of a span in the scope of a transaction. + + +-- + +*`span.type`*:: ++ +-- +type: keyword + +Keyword of specific relevance in the service's domain (eg: 'db.postgresql.query', 'template.erb', 'cache', etc). + + +-- + +[float] +== start fields + +None + + +*`span.start.us`*:: ++ +-- +type: long + +Offset relative to the transaction's timestamp identifying the start of the span, in microseconds. + + +-- + +[float] +== duration fields + +None + + +*`span.duration.us`*:: ++ +-- +type: long + +format: duration + +Duration of the span, in microseconds. + + +-- + +*`span.parent`*:: ++ +-- +type: long + +The locally unique ID of the parent of the span. + + +-- + +[[exported-fields-apm-transaction]] +== APM Transaction fields + +Transaction-specific data for APM + + + +*`transaction.name`*:: ++ +-- +type: text + +Generic designation of a transaction in the scope of a single service (eg. 'GET /users/:id'). + + +*`transaction.name.keyword`*:: ++ +-- +type: keyword + +-- + +-- + +*`transaction.type`*:: ++ +-- +type: keyword + +Keyword of specific relevance in the service's domain (eg. 'request', 'backgroundjob', etc) + + +-- + +[float] +== duration fields + +None + + +*`transaction.duration.us`*:: ++ +-- +type: long + +format: duration + +Total duration of this transaction, in microseconds. + + +-- + +*`transaction.result`*:: ++ +-- +type: keyword + +The result of the transaction. HTTP status code for HTTP-related transactions. + + +-- + +*`transaction.marks`*:: ++ +-- +type: object + +A user-defined mapping of groups of marks in milliseconds. + + +-- + +*`transaction.marks.navigationTiming`*:: ++ +-- +type: object + +-- + +*`transaction.sampled`*:: ++ +-- +type: boolean + +Transactions that are 'sampled' will include all available information. Transactions that are not sampled will not have spans or context. Defaults to true. + + +-- + + + +*`transaction.span_count.dropped.total`*:: ++ +-- +type: long + +The total amount of dropped spans for this transaction. + +-- + +[[exported-fields-beat]] +== Beat fields + +Contains common beat fields available in all event types. + + + +*`beat.name`*:: ++ +-- +The name of the Beat sending the log messages. If the Beat name is set in the configuration file, then that value is used. If it is not set, the hostname is used. To set the Beat name, use the `name` option in the configuration file. + + +-- + +*`beat.hostname`*:: ++ +-- +The hostname as returned by the operating system on which the Beat is running. + + +-- + +*`beat.timezone`*:: ++ +-- +The timezone as returned by the operating system on which the Beat is running. + + +-- + +*`beat.version`*:: ++ +-- +The version of the beat that generated this event. + + +-- + +*`@timestamp`*:: ++ +-- +type: date + +example: August 26th 2016, 12:35:53.332 + +format: date + +required: True + +The timestamp when the event log record was generated. + + +-- + +*`tags`*:: ++ +-- +Arbitrary tags that can be set per Beat and per transaction type. + + +-- + +*`fields`*:: ++ +-- +type: object + +Contains user configurable fields. + + +-- + +[float] +== error fields + +Error fields containing additional info in case of errors. + + + +*`error.message`*:: ++ +-- +type: text + +Error message. + + +-- + +*`error.code`*:: ++ +-- +type: long + +Error code. + + +-- + +*`error.type`*:: ++ +-- +type: keyword + +Error type. + + +-- + +[[exported-fields-cloud]] +== Cloud provider metadata fields + +Metadata from cloud providers added by the add_cloud_metadata processor. + + + +*`meta.cloud.provider`*:: ++ +-- +example: ec2 + +Name of the cloud provider. Possible values are ec2, gce, or digitalocean. + + +-- + +*`meta.cloud.instance_id`*:: ++ +-- +Instance ID of the host machine. + + +-- + +*`meta.cloud.instance_name`*:: ++ +-- +Instance name of the host machine. + + +-- + +*`meta.cloud.machine_type`*:: ++ +-- +example: t2.medium + +Machine type of the host machine. + + +-- + +*`meta.cloud.availability_zone`*:: ++ +-- +example: us-east-1c + +Availability zone in which this host is running. + + +-- + +*`meta.cloud.project_id`*:: ++ +-- +example: project-x + +Name of the project in Google Cloud. + + +-- + +*`meta.cloud.region`*:: ++ +-- +Region in which this host is running. + + +-- + +[[exported-fields-docker-processor]] +== Docker fields + +Docker stats collected from Docker. + + + + +*`docker.container.id`*:: ++ +-- +type: keyword + +Unique container id. + + +-- + +*`docker.container.image`*:: ++ +-- +type: keyword + +Name of the image the container was built on. + + +-- + +*`docker.container.name`*:: ++ +-- +type: keyword + +Container name. + + +-- + +*`docker.container.labels`*:: ++ +-- +type: object + +Image labels. + + +-- + +[[exported-fields-host-processor]] +== Host fields + +Info collected for the host machine. + + + + +*`host.name`*:: ++ +-- +type: keyword + +Hostname. + + +-- + +*`host.id`*:: ++ +-- +type: keyword + +Unique host id. + + +-- + +*`host.architecture`*:: ++ +-- +type: keyword + +Host architecture (e.g. x86_64, arm, ppc, mips). + + +-- + +*`host.os.platform`*:: ++ +-- +type: keyword + +OS platform (e.g. centos, ubuntu, windows). + + +-- + +*`host.os.version`*:: ++ +-- +type: keyword + +OS version. + + +-- + +*`host.os.family`*:: ++ +-- +type: keyword + +OS family (e.g. redhat, debian, freebsd, windows). + + +-- + +*`host.ip`*:: ++ +-- +type: ip + +List of IP-addresses. + + +-- + +*`host.mac`*:: ++ +-- +type: keyword + +List of hardware-addresses, usually MAC-addresses. + + +-- + +[[exported-fields-kubernetes-processor]] +== Kubernetes fields + +Kubernetes metadata added by the kubernetes processor + + + + +*`kubernetes.pod.name`*:: ++ +-- +type: keyword + +Kubernetes pod name + + +-- + +*`kubernetes.pod.uid`*:: ++ +-- +type: keyword + +Kubernetes pod uid + + +-- + +*`kubernetes.namespace`*:: ++ +-- +type: keyword + +Kubernetes namespace + + +-- + +*`kubernetes.node.name`*:: ++ +-- +type: keyword + +Kubernetes node name + + +-- + +*`kubernetes.labels`*:: ++ +-- +type: object + +Kubernetes labels map + + +-- + +*`kubernetes.annotations`*:: ++ +-- +type: object + +Kubernetes annotations map + + +-- + +*`kubernetes.container.name`*:: ++ +-- +type: keyword + +Kubernetes container name + + +-- + +*`kubernetes.container.image`*:: ++ +-- +type: keyword + +Kubernetes container image + + +-- + diff --git a/docs/index.asciidoc b/docs/index.asciidoc index 4404315ee42..8def6aad86f 100644 --- a/docs/index.asciidoc +++ b/docs/index.asciidoc @@ -46,6 +46,8 @@ include::./intake-api.asciidoc[] include::./exploring-es-data.asciidoc[] +include::./fields.asciidoc[] + include::./troubleshooting.asciidoc[] include::./release-notes.asciidoc[]