Skip to content

Docs: WIP V2/6.5 additions and changes #1530

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 13 commits into from
Nov 12, 2018
Merged

Docs: WIP V2/6.5 additions and changes #1530

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
c17bb5e
docs: rework apm overview
bmorelli25 Nov 6, 2018
39c7472
docs: move events and add distributed tracing
bmorelli25 Nov 6, 2018
c632c8c
docs: add more v2 and change layout
bmorelli25 Nov 7, 2018
d79b392
docs: more v2 additions
bmorelli25 Nov 7, 2018
5e99ca5
Merge branch 'docs-v2-updates' into 1484-doc-intake-api
bmorelli25 Nov 7, 2018
eb5a74b
docs: rework apm overview
bmorelli25 Nov 6, 2018
592f15e
docs: more v2
bmorelli25 Nov 7, 2018
8312f9a
docs: review improvements and fix compat
bmorelli25 Nov 8, 2018
1fed7ab
docs: v2 revisions and fixes
bmorelli25 Nov 8, 2018
c212c33
Update example files for official docs. (#2)
simitt Nov 9, 2018
a54db05
docs: update structure and config
bmorelli25 Nov 9, 2018
97420d3
docs: add redirects file
bmorelli25 Nov 9, 2018
dd62bce
docs: fix agent server compat
bmorelli25 Nov 12, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions docs/common-problems.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,8 @@ Particularly, if you are using Docker, ensure to bind to the right interface (fo



If you see requests coming trough the APM Server but they are not accepted (response code other than `202`), consider
the response code to narrow down the possible causes (see sections bellow).
If you see requests coming through the APM Server but they are not accepted (response code other than `202`), consider
the response code to narrow down the possible causes (see sections below).

Another reason for data not showing up is that the agent is not auto-instrumenting something you were expecting, check
the {apm-agents-ref}/index.html[agent documentation] for details on what is automatically instrumented.
Expand All @@ -46,7 +46,7 @@ the {apm-agents-ref}/index.html[agent documentation] for details on what is auto
[float]
=== HTTP 400: Data decoding error / Data validation error

The most likely cause for this is that you are using non compatible versions of agent and APM Server.
The most likely cause for this is that you are using incompatible versions of agent and APM Server.
For instance, APM Server 6.2.0 changed the Intake API spec and requires a minimum version of each agent.

[[unauthorized]]
Expand Down Expand Up @@ -103,7 +103,7 @@ You have a few options to solve this problem:
=== HTTP 503: Request timed out waiting to be processed

This happens when APM Server exceeds the maximum number of requests that it can process concurrently.
This limit is determined by the `apm-server.concurrent_requests` configuration parameter.
This limit is determined by the `apm-server.concurrent_requests` configuration parameter deprecated[6.5].

To alleviate this problem,
you can try to:
Expand Down Expand Up @@ -157,20 +157,20 @@ To resolve this problem, try one of these solutions:
* Create a DNS entry for the hostname mapping it to the server's IP.
* Create an entry in `/etc/hosts` for the hostname. Or on Windows add an entry to
`C:\Windows\System32\drivers\etc\hosts`.
* Re-create the server certificate and add a SubjectAltName (SAN) for the IP address of the server. This make the
* Re-create the server certificate and add a SubjectAltName (SAN) for the IP address of the server. This makes the
server's certificate valid for both the hostname and the IP address.

[float]
[[getsockopt-no-route-to-host]]
===== getsockopt: no route to host

This is not a SSL problem. It's a networking problem. Make sure the two hosts can communicate.
This is not an SSL problem. It's a networking problem. Make sure the two hosts can communicate.

[float]
[[getsockopt-connection-refused]]
===== getsockopt: connection refused

This is not a SSL problem. Make sure that Logstash is running and that there is no firewall blocking the traffic.
This is not an SSL problem. Make sure that Logstash is running and that there is no firewall blocking the traffic.

[float]
[[target-machine-refused-connection]]
Expand Down
14 changes: 10 additions & 4 deletions docs/configuration-process.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,10 +28,16 @@ Defaults to 'localhost:8200'.

Copy link
Contributor

@simitt simitt Nov 9, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I realized there are a couple of config options missing in the Configuring APM Server chapter. Some of them were newly introduced, some not. I suggest to check with the apm-server.yml for missing config options (e.g.: apm-server.rum.event_rate and apm-server.rum.source_mapping.elasticsearch should be added to Set up Real User Monitoring (RUM) support.

For max_event_size you explicitly added that it was introduced with 6.5. On one hand it is nice to see newly introduced options, on the other hand it means we have to remove this again for 6.6 or add the info when an option was introduced everywhere.

For everything related to tuning and config options we need some strategy to point out what to tweak and which options to use when agents talk Intake v1 vs. agents talking intake v2, e.g. if somebody uses a ruby agent with intake v1 and a nodejs agent with intake v2, then both - the deprecated and the newly introduced config options - should be set properly.
Not sure how much you want to put into this PR, feel free to create a new issue and defer this.

Copy link
Member Author

@bmorelli25 bmorelli25 Nov 9, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like apm-server.rum.source_mapping.elasticsearch was already added, I've now added in event_rate as well.

configuration-process.asciidoc and configuration-rum.asciidoc have been updated. Changed the order of configuration options to better mirror apm-server.yml, and added in notes about v1/v2 to hopefully make things clearer.

The tuning is a bit more complex, so I've opened #1537

[[max_unzipped_size]]
[float]
==== `max_unzipped_size`
==== `max_unzipped_size` deprecated[6.5]
Maximum permitted size of an unzipped request accepted by the server to be processed (in Bytes).
Defaults to 31457280 Bytes (30 MB).

[[max_event_size]]
[float]
==== `max_event_size added[6.5]
Maximum permitted size of an event accepted by the server to be processed (in Bytes).
Defaults to 307200 Bytes.

[[max_header_size]]
[float]
==== `max_header_size`
Expand All @@ -40,7 +46,7 @@ Defaults to 1048576 Bytes (1 MB).

[[max_request_queue_time]]
[float]
==== `max_request_queue_time`
==== `max_request_queue_time` deprecated[6.5]
Maximum duration a request will be queued before being read.
Defaults to 2 seconds.

Expand All @@ -64,7 +70,7 @@ Defaults to 5 seconds.

[[concurrent_requests]]
[float]
==== `concurrent_request`
==== `concurrent_request` deprecated[6.5]
Maximum number of requests the server can process concurrently.
Read more about how to tune data ingestion by <<adjust-concurrent-requests, adjusting concurrent_requests>>.
Default value is 5.
Expand All @@ -87,7 +93,7 @@ Disabled by default.
Authorization token for sending data to the APM server.
If a token is set, the agents must send it in the following format:
Authorization: Bearer <secret-token>.
The token is not used for RUM endpoints. By default no authorization token is set.
The token is not used for RUM endpoints. By default, no authorization token is set.

It is recommended to use an authorization token in combination with SSL enabled.
Read more about <<securing-apm-server, Securing APM Server>> and the <<secret-token, secret token>>.
Expand Down
20 changes: 10 additions & 10 deletions docs/configuration-rum.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,25 +34,25 @@ Default value is set to 10.

[float]
==== `allow_origins`
Comma separated list of permitted origins for RUM supprt.
Comma separated list of permitted origins for RUM support.
User-agents send an Origin header that will be validated against this list.
This is done automatically by modern browsers as part of the https://www.w3.org/TR/cors/[CORS specification].
An origin is made of a protocol scheme, host and port, without the url path.
An origin is made of a protocol scheme, host and port, without the URL path.
Default value is set to `['*']`, which allows everything.

[float]
==== `library_pattern`
Regexp to be matched against a stacktrace frame's `file_name` and `abs_path` attributes.
If the regexp matches, the stacktrace frame is considered to be a library frame.
When source mapping is applied the `error.culprit` is set to reflect the _function_ and the _filename_
of the first not library frame.
RegExp to be matched against a stacktrace frame's `file_name` and `abs_path` attributes.
If the RegExp matches, the stacktrace frame is considered to be a library frame.
When source mapping is applied, the `error.culprit` is set to reflect the _function_ and the _filename_
of the first non library frame.
This aims to provide an entry point for identifying issues.
Default value is `"node_modules|bower_components|~"`.

[float]
==== `exclude_from_grouping`
Regexp to be matched against a stacktrace frame's `file_name`.
If the regexp matches, the stacktrace frame is excluded from being used for calculating error groups.
RegExp to be matched against a stacktrace frame's `file_name`.
If the RegExp matches, the stacktrace frame is excluded from being used for calculating error groups.
The default pattern excludes stacktrace frames that have a filename starting with `/webpack`.

[[rum-sourcemap-cache]]
Expand All @@ -69,10 +69,10 @@ Default value is 5 minutes.
==== `source_mapping.elasticsearch`
Configure the Elasticsearch source map retrieval location, taking the same options as <<elasticsearch-output,output.elasticsearch>>.
This must be set when using an output other than Elasticsearch, and that output is writing to Elasticsearch.
Othewise leave this section empty.
Otherwise leave this section empty.

[float]
==== `source_mapping.index_pattern`
Source maps are stored in a seperate index `apm-%{[beat.version]}-sourcemap` by default.
Source maps are stored in a separate index `apm-%{[beat.version]}-sourcemap` by default.
If changed, a matching index pattern needs to be specified here.

2 changes: 1 addition & 1 deletion docs/configuring-ingest.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ Automatic pipeline registration requires `output.elasticsearch` to be enabled an
as well as having the required Elasticsearch plugins installed.

APM Server default pipelines require you to have the Ingest user agent plugin installed.
If pipelines are enabeld but required plugins are missing,
If pipelines are enabled but required plugins are missing,
the APM Server will not be able to properly connect to the Elasticsearch output.
Find the default pipeline configuration at `ingest/pipeline/definition.json` of your APM Server
installation's home directory.
Expand Down
2 changes: 1 addition & 1 deletion docs/configuring-output-after.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,4 +24,4 @@ fields: {project: "myproject", instance-id: "574734885120952459"}
------------------------------------------------------------------------------

To store the custom fields as top-level fields, set the `fields_under_root` option to true.
This is not recommended as backwards compatibility can not be ensured as new fields are added to APM documents.
This is not recommended as when new fields are added to APM documents backward compatibility cannot be ensured.
2 changes: 1 addition & 1 deletion docs/data-ingestion.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ Increasing `queue.mem.events` can significantly affect APM Server memory usage.
[float]
=== Adjust concurrent requests
Copy link
Contributor

@simitt simitt Nov 9, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for not being more clear in my last review - adjusting concurrent request is only an option if you use agents talking Intake v1. I'd therefore add a deprecated warning to this whole section. But we also need to find a way to make it visible which adjustments help for agents using v1 and which for agents using v2 (see my comment above).

Copy link
Member Author

@bmorelli25 bmorelli25 Nov 9, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the tag for now, opened #1537 for later

APM Server has a limit to how many requests can be processed concurrently.
This limit is determined by the `apm-server.concurrent_requests` setting.
This limit is determined by the `apm-server.concurrent_requests` setting deprecated[6.5].
Increasing this value will improve throughput, but it can significantly affect APM Server memory usage.

[[add-apm-server-instances]]
Expand Down
139 changes: 9 additions & 130 deletions docs/error-api.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,148 +1,27 @@
[[error-api]]
== Error API
=== Error API

The following section contains information about:

* <<error-endpoint>>
* <<error-schema-definition>>
* <<error-api-examples>>

[[error-endpoint]]
[float]
=== Endpoint

Send a `HTTP POST` request to the APM Server `errors` endpoint:
[source,bash]
------------------------------------------------------------
http(s)://{hostname}:{port}/v1/errors
------------------------------------------------------------

For <<rum, RUM>> send a `HTTP POST` request to the `rum errors` endpoint instead:

[source,bash]
------------------------------------------------------------
http(s)://{hostname}:{port}/v1/rum/errors
------------------------------------------------------------
* <<error-schema>>

[[error-schema-definition]]
[float]
=== Schema Definition

The APM Server uses JSON Schema for validating requests. The specification for errors is defined bellow:
* <<error-payload-schema>>
* <<error-service-schema>>
* <<error-system-schema>>
* <<error-context-schema>>
* <<error-stacktraceframe-schema>>
* <<error-request-schema>>
* <<error-user-schema>>


[[error-payload-schema]]
[float]
==== Payload

[source,json]
----
include::./spec/errors/v1_error.json[]
----

[[error-service-schema]]
[float]
==== Service

[source,json]
----
include::./spec/service.json[]
----

[[error-system-schema]]
[float]
==== System

[source,json]
----
include::./spec/system.json[]
----

[[error-context-schema]]
[float]
==== Context
==== Schema Definition

[source,json]
----
include::./spec/context.json[]
----
The APM Server uses JSON Schema for validating requests. The specification for errors is defined below:

[[error-stacktraceframe-schema]]
[float]
==== Stacktrace Frame
[[error-schema]]
===== Error

[source,json]
----
include::./spec/stacktrace_frame.json[]
include::./spec/errors/v2_error.json[]
----

[[error-request-schema]]
[float]
==== Request

[source,json]
----
include::./spec/request.json[]
----

[[error-user-schema]]
[float]
==== User

[source,json]
----
include::./spec/user.json[]
----

[[error-api-examples]]
[float]
=== Examples

Request example:

["source","sh",subs="attributes"]
------------------------------------------------------------
curl http://localhost:8200/v1/errors \
--header "Content-Type: application/json" \
--data @docs/data/intake-api/generated/error/payload.json
------------------------------------------------------------

Example error requests:

* <<payload-with-error>>
* <<payload-with-minimal-exception>>
* <<payload-with-minimal-log>>

[[payload-with-error]]
[float]
==== Payload with an Error

[source,json]
----
include::./data/intake-api/generated/error/payload.json[]
----

[[payload-with-minimal-exception]]
[float]
==== Payload with an Error with minimal Exception Information

[source,json]
----
include::./data/intake-api/generated/error/minimal_payload_exception.json[]
----

[[payload-with-minimal-log]]
[float]
==== Payload with an Error with minimal Log Information

[source,json]
----
include::./data/intake-api/generated/error/minimal_payload_log.json[]
----
include::./spec/errors/common_error.json[]
----
21 changes: 0 additions & 21 deletions docs/errors.asciidoc
View file
Open in desktop

This file was deleted.

13 changes: 0 additions & 13 deletions docs/event-types.asciidoc
View file
Open in desktop

This file was deleted.

Loading