Docs: WIP V2/6.5 additions and changes

Makefile

@@ -46,16 +46,14 @@ go-generate:
 
 .PHONY: create-docs
 create-docs:
-	@mkdir -p docs/data/intake-api/generated/{error,transaction,metricset,sourcemap}
-	@cp testdata/error/payload.json docs/data/intake-api/generated/error/
-	@cp testdata/error/rum.json docs/data/intake-api/generated/error/
-	@cp testdata/error/minimal_payload_exception.json docs/data/intake-api/generated/error/
-	@cp testdata/error/minimal_payload_log.json docs/data/intake-api/generated/error/
-	@cp testdata/metricset/payload.json docs/data/intake-api/generated/metricset/
-	@cp testdata/transaction/payload.json docs/data/intake-api/generated/transaction/
-	@cp testdata/transaction/minimal_payload.json docs/data/intake-api/generated/transaction/
-	@cp testdata/transaction/minimal_span.json docs/data/intake-api/generated/transaction/
+	@mkdir -p docs/data/intake-api/generated/sourcemap
+	@cp testdata/intake-v2/events.ndjson docs/data/intake-api/generated/
 	@cp testdata/sourcemap/bundle.js.map docs/data/intake-api/generated/sourcemap/
+	@mkdir -p docs/data/elasticsearch/generated/
+	@cp processor/stream/approved-es-documents/testV2IntakeIntegrationErrors.approved.json docs/data/elasticsearch/generated/errors.json
+	@cp processor/stream/approved-es-documents/testV2IntakeIntegrationTransactions.approved.json docs/data/elasticsearch/generated/transactions.json
+	@cp processor/stream/approved-es-documents/testV2IntakeIntegrationSpans.approved.json docs/data/elasticsearch/generated/spans.json
+	@cp processor/stream/approved-es-documents/testV2IntakeIntegrationMetricsets.approved.json docs/data/elasticsearch/generated/metricsets.json
 
 # Start manual testing environment with agents
 start-env:

docs/common-problems.asciidoc

@@ -36,8 +36,8 @@ Particularly, if you are using Docker, ensure to bind to the right interface (fo
 
 
 
-If you see requests coming trough the APM Server but they are not accepted (response code other than `202`), consider
-the response code to narrow down the possible causes (see sections bellow).
+If you see requests coming through the APM Server but they are not accepted (response code other than `202`), consider
+the response code to narrow down the possible causes (see sections below).
 
 Another reason for data not showing up is that the agent is not auto-instrumenting something you were expecting, check
 the {apm-agents-ref}/index.html[agent documentation] for details on what is automatically instrumented.
@@ -46,7 +46,7 @@ the {apm-agents-ref}/index.html[agent documentation] for details on what is auto
 [float]
 === HTTP 400: Data decoding error / Data validation error
 
-The most likely cause for this is that you are using non compatible versions of agent and APM Server.
+The most likely cause for this is that you are using incompatible versions of agent and APM Server.
 For instance, APM Server 6.2.0 changed the Intake API spec and requires a minimum version of each agent.
 
 [[unauthorized]]
@@ -103,7 +103,7 @@ You have a few options to solve this problem:
 === HTTP 503: Request timed out waiting to be processed
 
 This happens when APM Server exceeds the maximum number of requests that it can process concurrently.
-This limit is determined by the `apm-server.concurrent_requests` configuration parameter.
+This limit is determined by the `apm-server.concurrent_requests` configuration parameter deprecated[6.5].
 
 To alleviate this problem,
 you can try to:
@@ -157,20 +157,20 @@ To resolve this problem, try one of these solutions:
 * Create a DNS entry for the hostname mapping it to the server's IP.
 * Create an entry in `/etc/hosts` for the hostname. Or on Windows add an entry to
 `C:\Windows\System32\drivers\etc\hosts`.
-* Re-create the server certificate and add a SubjectAltName (SAN) for the IP address of the server. This make the
+* Re-create the server certificate and add a SubjectAltName (SAN) for the IP address of the server. This makes the
 server's certificate valid for both the hostname and the IP address.
 
 [float]
 [[getsockopt-no-route-to-host]]
 ===== getsockopt: no route to host
 
-This is not a SSL problem. It's a networking problem. Make sure the two hosts can communicate.
+This is not an SSL problem. It's a networking problem. Make sure the two hosts can communicate.
 
 [float]
 [[getsockopt-connection-refused]]
 ===== getsockopt: connection refused
 
-This is not a SSL problem. Make sure that Logstash is running and that there is no firewall blocking the traffic.
+This is not an SSL problem. Make sure that Logstash is running and that there is no firewall blocking the traffic.
 
 [float]
 [[target-machine-refused-connection]]

docs/configuration-process.asciidoc

@@ -17,7 +17,8 @@ max_procs: 4
 ----
 
 [float]
-=== Configuration options `apm-server.*`
+[[configuration-apm-server]]
+=== Configuration options: `apm-server.*`
 
 [[host]]
 [float]
@@ -26,24 +27,12 @@ Defines the host and port the server is listening on.
 Use "unix:/path/to.sock" to listen on a unix domain socket.
 Defaults to 'localhost:8200'.
 
-[[max_unzipped_size]]
-[float]
-==== `max_unzipped_size`
-Maximum permitted size of an unzipped request accepted by the server to be processed (in Bytes).
-Defaults to 31457280 Bytes (30 MB).
-
 [[max_header_size]]
 [float]
 ==== `max_header_size`
 Maximum permitted size of a request's header accepted by the server to be processed (in Bytes).
 Defaults to 1048576 Bytes (1 MB).
 
-[[max_request_queue_time]]
-[float]
-==== `max_request_queue_time`
-Maximum duration a request will be queued before being read.
-Defaults to 2 seconds.
-
 [[read_timeout]]
 [float]
 ==== `read_timeout`
@@ -62,32 +51,64 @@ Defaults to 30 seconds.
 Maximum duration in seconds before releasing resources when shutting down the server.
 Defaults to 5 seconds.
 
+[float]
+[[configuration-v1-api]]
+=== Configuration options: v1 Intake API deprecated[6.5]
+
+Version 6.5 of the APM Server introduced a new intake API. You can learn more about this change in the <<intake-api-changes-65, intake API changes>> documentation. 
+
+If you do not upgrade your APM Agent, you'll continue to use the deprecated v1 intake API endpoint, and thus, the following configuration options.
+
+[[max_unzipped_size]]
+[float]
+==== `max_unzipped_size` deprecated[6.5]
+Maximum permitted size of an unzipped request accepted by the server to be processed (in Bytes).
+Defaults to 31457280 Bytes (30 MB).
+
+[[max_request_queue_time]]
+[float]
+==== `max_request_queue_time` deprecated[6.5]
+Maximum duration a request will be queued before being read.
+Defaults to 2 seconds.
+
 [[concurrent_requests]]
 [float]
-==== `concurrent_request`
+==== `concurrent_request` deprecated[6.5]
 Maximum number of requests the server can process concurrently.
 Read more about how to tune data ingestion by <<adjust-concurrent-requests, adjusting concurrent_requests>>.
 Default value is 5.
 
+[float]
+[[configuration-v2-api]]
+=== Configuration options: v2 Intake API
+
+Version 6.5 of the APM Server introduced a new intake API. You can learn more about this change in the <<intake-api-changes-65, intake API changes>> documentation. 
+
+The following configuration options only affect users that have upgraded their agent to take advantage of the new v2 intake API. 
+
+[[max_event_size]]
+[float]
+==== `max_event_size`
+Maximum permitted size of an event accepted by the server to be processed (in Bytes).
+Defaults to 307200 Bytes.
+
+[float]
+[[configuration-other]]
+=== Configuration options: general
+
 [[max_connections]]
 [float]
 ==== `max_connections`
 Maximum number of TCP connections to accept simultaneously.
 Default value is 0, which means _unlimited_.
 
-[[instrumentation.enabled]]
-[float]
-==== `instrumentation.enabled`
-Enables self instrumentation of the APM Server itself.
-Disabled by default.
-
 [[config-secret-token]]
 [float]
 ==== `secret_token`
 Authorization token for sending data to the APM server.
 If a token is set, the agents must send it in the following format:
 Authorization: Bearer <secret-token>.
-The token is not used for RUM endpoints. By default no authorization token is set.
+The token is not used for RUM endpoints. By default, no authorization token is set.
 
 It is recommended to use an authorization token in combination with SSL enabled.
 Read more about <<securing-apm-server, Securing APM Server>> and the <<secret-token, secret token>>.
@@ -111,6 +132,12 @@ Disabled by default.
 Configure the url to expose expvar.
 Defaults to `debug/vars`.
 
+[[instrumentation.enabled]]
+[float]
+==== `instrumentation.enabled`
+Enables self instrumentation of the APM Server itself.
+Disabled by default.
+
 [[metrics.enabled]]
 [float]
 ==== `metrics`
@@ -129,9 +156,9 @@ Defaults to false.
 Overwrites existing pipeline definitions in Elasticsearch. Defaults to true.
 
 [float]
-=== Configuration options `queue.mem.*`
+=== Configuration options: `queue.mem.*`
 Data is buffered in a memory queue before it is published to the configured output.
-`queue.mem.*` settings modify the queue behaviour.
+`queue.mem.*` settings modify the queue behavior.
 
 [[mem.events]]
 [float]
@@ -155,7 +182,7 @@ if the number of events stored in the queue is < _min_flush_events_.
 Default value is 1 second.
 
 [float]
-=== Configuration options `max_procs`
+=== Configuration options: `max_procs`
 
 [[max_procs]]
 [float]

docs/configuration-rum.asciidoc

@@ -17,7 +17,7 @@ apm-server.rum.source_mapping.index_pattern: "apm-*-sourcemap*"
 ----
 
 [float]
-=== Configuration options
+=== Configuration
 
 [[rum-enable]]
 [float]
@@ -26,35 +26,74 @@ For enabling RUM support, set the `apm-server.rum.enabled` to `true`.
 By default this is disabled.
 
 [float]
-==== `rate_limit`
+[[configuration-rum-v1]]
+=== Configuration options: v1 RUM endpoint deprecated[6.5]
+
+Version 6.5 of the APM Server introduced a new intake API. You can learn more about this change in the <<intake-api-changes-65, intake API changes>> documentation. 
+
+If you do not upgrade your APM Agent, you'll continue to use the deprecated v1 RUM intake API endpoint, and thus, the following configuration options.
+
+[float]
+==== `rate_limit` deprecated[6.5]
 Rate limit per second and IP address for requests sent to the RUM endpoint.
 If the rate limit is hit, the APM Server will return an HTTP status code `429`.
 The rate limit cannot be disabled. Ensure to have it set to a number suiting your requirements.
 Default value is set to 10.
 
+[float]
+[[configuration-rum-v2]]
+=== Configuration options: v2 RUM endpoint
+
+Version 6.5 of the APM Server introduced a new intake API. You can learn more about this change in the <<intake-api-changes-65, intake API changes>> documentation. 
+
+The following configuration options only affect users that have upgraded their agent to take advantage of the new v2 intake API. 
+
+[float]
+==== `event_rate.limit`
+Defines the maximum amount of events allowed to be sent to the APM Server v2 RUM endpoint per ip per second.
+Defaults to 300.
+
+[float]
+==== `event_rate.lru_size`
+An LRU cache is used to keep a rate limit per IP for the most recently seen IPs.
+This setting defines the number of unique IPs that can be tracked in the cache.
+Sites with many concurrent clients should consider increasing this limit.
+Defaults to 1000.
+
+[float]
+[[configuration-rum-general]]
+=== Configuration options: general
+
 [float]
 ==== `allow_origins`
-Comma separated list of permitted origins for RUM supprt.
+Comma separated list of permitted origins for RUM support.
 User-agents send an Origin header that will be validated against this list.
 This is done automatically by modern browsers as part of the https://www.w3.org/TR/cors/[CORS specification].
-An origin is made of a protocol scheme, host and port, without the url path.
+An origin is made of a protocol scheme, host and port, without the URL path.
 Default value is set to `['*']`, which allows everything.
 
 [float]
 ==== `library_pattern`
-Regexp to be matched against a stacktrace frame's `file_name` and `abs_path` attributes.
-If the regexp matches, the stacktrace frame is considered to be a library frame.
-When source mapping is applied the `error.culprit` is set to reflect the _function_ and the _filename_
-of the first not library frame.
+RegExp to be matched against a stacktrace frame's `file_name` and `abs_path` attributes.
+If the RegExp matches, the stacktrace frame is considered to be a library frame.
+When source mapping is applied, the `error.culprit` is set to reflect the _function_ and the _filename_
+of the first non library frame.
 This aims to provide an entry point for identifying issues.
 Default value is `"node_modules|bower_components|~"`.
 
 [float]
 ==== `exclude_from_grouping`
-Regexp to be matched against a stacktrace frame's `file_name`.
-If the regexp matches, the stacktrace frame is excluded from being used for calculating error groups.
+RegExp to be matched against a stacktrace frame's `file_name`.
+If the RegExp matches, the stacktrace frame is excluded from being used for calculating error groups.
 The default pattern excludes stacktrace frames that have a filename starting with `/webpack`.
 
+[[config-sourcemapping-elasticsearch]]
+[float]
+==== `source_mapping.elasticsearch`
+Configure the Elasticsearch source map retrieval location, taking the same options as <<elasticsearch-output,output.elasticsearch>>.
+This must be set when using an output other than Elasticsearch, and that output is writing to Elasticsearch.
+Otherwise leave this section empty.
+
 [[rum-sourcemap-cache]]
 [float]
 ==== `source_mapping.cache.expiration`
@@ -64,15 +103,7 @@ Source maps are fetched from Elasticsearch and then kept in an in-memory cache f
 Values configured without a time unit are treated as seconds.
 Default value is 5 minutes.
 
-[[config-sourcemapping-elasticsearch]]
-[float]
-==== `source_mapping.elasticsearch`
-Configure the Elasticsearch source map retrieval location, taking the same options as <<elasticsearch-output,output.elasticsearch>>.
-This must be set when using an output other than Elasticsearch, and that output is writing to Elasticsearch.
-Othewise leave this section empty.
-
 [float]
 ==== `source_mapping.index_pattern`
-Source maps are stored in a seperate index `apm-%{[beat.version]}-sourcemap` by default.
-If changed, a matching index pattern needs to be specified here.
-
+Source maps are stored in a separate index `apm-%{[beat.version]}-sourcemap` by default.
+If changed, a matching index pattern needs to be specified here.

docs/configuring-ingest.asciidoc

@@ -26,7 +26,7 @@ Automatic pipeline registration requires `output.elasticsearch` to be enabled an
 as well as having the required Elasticsearch plugins installed.
 
 APM Server default pipelines require you to have the Ingest user agent plugin installed.
-If pipelines are enabeld but required plugins are missing, 
+If pipelines are enabled but required plugins are missing, 
 the APM Server will not be able to properly connect to the Elasticsearch output.
 Find the default pipeline configuration at `ingest/pipeline/definition.json` of your APM Server
 installation's home directory.

docs/configuring-output-after.asciidoc

@@ -24,4 +24,4 @@ fields: {project: "myproject", instance-id: "574734885120952459"}
 ------------------------------------------------------------------------------
 
 To store the custom fields as top-level fields, set the `fields_under_root` option to true.
-This is not recommended as backwards compatibility can not be ensured as new fields are added to APM documents.
+This is not recommended as when new fields are added to APM documents backward compatibility cannot be ensured.

docs/data-ingestion.asciidoc

@@ -48,7 +48,7 @@ Increasing `queue.mem.events` can significantly affect APM Server memory usage.
 
 [[adjust-concurrent-requests]]
 [float]
-=== Adjust concurrent requests
+=== Adjust concurrent requests deprecated[6.5]
 APM Server has a limit to how many requests can be processed concurrently.
 This limit is determined by the `apm-server.concurrent_requests` setting.
 Increasing this value will improve throughput, but it can significantly affect APM Server memory usage.