Add http to span context properties

[hmdhk]

For certain spans the frontend agent sends the raw url under the span context with the following structure:

{
  "name": "...",
  "context": {
    "http": {
      "url": {
        "raw": "http://example.com"
      }
    }
  }
}

We need to add this to the span context, however the raw url can be longer than our default 1024 string length limit. There are already cases that exceed this limit so we should not limit this field.

Would be great to hear your thoughts on this, @elastic/apm-server , @elastic/apm-agent-devs

cc @roncohen , @alvarolobato

Update: Decided to use the following structure:

{
  "name": "...",
  "context": {
    "http": {
      "url": "http://example.com"
    }
  }
}

[simitt]

FYI: We are limiting fields when they are indexed as an Elasticsearch keyword, because the default value for ignore_above is set to 1024 in beats.

[simitt]

Afaik this is hardcoded in beats atm.

[hmdhk]

Thanks @simitt, I don't think http.url.raw should be a keyword, so as you said, we don't need to limit it's length. But we still need to define this field in the spec as string field.

[simitt]

Are you generally referring to what is currently nested as context.request.url.raw or do you want to introduce a new context.http.url.raw. If you want to introduce a new field here, could you please motivate why.

Are you suggesting to index http.url.raw as a text instead of a keyword datatype or are you suggesting to not index it at all (not make it query nor searchable)?

[hmdhk]

@simitt, This is the span context not the transaction context which has context.request.url.raw. I'm proposing to add context.http.url.raw to the span context. The reason for this is that in order to keep the span name short, we need to truncate the url that was requested, but it is still useful to have the raw url. Furthermore we will add more fields to context.http.url object, for example the parsed query string or the url fragment.

context.http.url.raw does not need to be searchable but it needs to show up in the discover and also be included in the Kibana query results so that the UI can show this field.

[simitt]

Thanks for the clarification. If we don't need to index the field, I don't see an issue with not specifying a string length limit.

cc @elastic/apm-ui

[hmdhk]

I should also clarify that even though I am taking the url structure from the Transaction.context.request.url these http requests are different. Transaction.context.request.url is for the incoming request to the server, but Span.context.http.url (my proposal) is the outgoing request that browser makes to the server.

[felixbarny]

I guess this would also apply for outgoing http requests. @beniwohli where are you currently setting the url of external http calls?

[alvarolobato]

@elastic/apm-server are we ok with this proposal? Should we work more on this?

[simitt]

If the agents can align on this structure and define all attributes they'd like to have introduced I don't see an issue in implementing this.

[felixbarny]

@hamid have you thought about just having the same structure as in the transaction i.e. Span.context.request? That would bring spans and transactions closer together instead of making them diverge more.

Especially in the context of distributed tracing, it makes sense to have a more unified view of transactions and spans.

I'm in line with you that http would actually be nicer that requests but this is a different issue which is already being discussed in #794.

[alvarolobato]

@elastic/apm-agent-devs @roncohen please lets try make a decision about this, it has been waiting for some time.