Add indices defaults in code

[roncohen]

Index default configuration is currently only existing in apm-server.yml. For people upgrading, this is problematic. Since we now have a way to override beats configuration from APM Server code, we should align the defaults so that even without an apm-server.yml, the right defaults are used for indices.

[roncohen]

marked it with v6.5 to show our hope that it will ship in 6.5, but not a blocker

[graphaelli]

moved the other defaults to code in #1494, default indices are proving much trickier. I think there's actually bug in here, working on a minimal case.

[graphaelli]

After some discussion and go-ucfg schooling we've decided it's not a bug. Unfortunately, there's also no current solution for providing a default list of indices that can be overridden by a config file / command line options. In the strangest case, this is what happens right now:

code wants:

output.elasticsearch:
  hosts: ["localhost:9200"]
  indices:
    - index: "apm-%{[beat.version]}-sourcemap-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "sourcemap"

user config says:

output.elasticsearch:
  hosts: ["localhost:9200"]
  indices:
    - index: "apm-%{[beat.version]}-foo-%{+yyyy.MM.dd}"

actual config becomes:

output:
  elasticsearch:
    hosts:
    - localhost:9200
    indices:
    - index: apm-%{[beat.version]}-foo-%{+yyyy.MM.dd}
      when:
        contains:
          processor:
            event: sourcemap

[roncohen]

Alright. We can’t just discard the code default if the user has anything under “output.elasticsearch.indices” their config file?

…

On Tue, 30 Oct 2018 at 13.34, Gil Raphaelli ***@***.***> wrote: After some discussion and go-ucfg schooling we've decided it's not a bug. Unfortunately, there's also no current solution for providing a default list of indices that can be overridden by a config file / command line options. In he strangest case, this is what happens right now: code wants: output.elasticsearch: hosts: ["localhost:9200"] indices: - index: "apm-%{[beat.version]}-sourcemap-%{+yyyy.MM.dd}" when.contains: processor.event: "sourcemap" user config says: output.elasticsearch: hosts: ["localhost:9200"] indices: - index: "apm-%{[beat.version]}-foo-%{+yyyy.MM.dd}" actual config becomes: output: elasticsearch: hosts: - localhost:9200 indices: - index: apm-%{[beat.version]}-foo-%{+yyyy.MM.dd} when: contains: processor: event: sourcemap — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1480 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAC6ApR2-3appX84Es7ZDRSUrJ9eB4Gks5uqEdBgaJpZM4X7xLI> .

[graphaelli]

Not simply, no. There are other cases in beats where arrays cause issues like this and we're brainstorming a nice general way to fix this.