Expose external refreshes through the stats API

[clandry94]

An implementation from this issue #36712.

Right now, the stats API only provides refresh metrics regarding internal refreshes. This isn't very useful and somewhat misleading for cluster administrators since internal refreshes are not indicative of documents being available for search..

In this PR I added a new metric for collecting external refreshes as they occur and exposing them through the stats API. Now, calling an endpoint for stats will yield external refresh metrics as well.

GET /test/_stats/refresh

{
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "_all" : {
    "primaries" : {
      "refresh" : {
        "total" : 7,
        "total_time_in_millis" : 157,
        "external_total" : 5,
        "external_total_time_in_millis" : 157,
        "listeners" : 0
      }
    },
    "total" : {
      "refresh" : {
        "total" : 7,
        "total_time_in_millis" : 157,
        "external_total" : 5,
        "external_total_time_in_millis" : 157,
        "listeners" : 0
      }
    }
  },
  "indices" : {
    "test" : {
      "uuid" : "4UxNKfIjSCGvGrBEU1l7tg",
      "primaries" : {
        "refresh" : {
          "total" : 7,
          "total_time_in_millis" : 157,
          "external_total" : 5,
          "external_total_time_in_millis" : 157,
          "listeners" : 0
        }
      },
      "total" : {
        "refresh" : {
          "total" : 7,
          "total_time_in_millis" : 157,
          "external_total" : 5,
          "external_total_time_in_millis" : 157,
          "listeners" : 0
        }
      }
    }
  }
}

Also, how is the naming? I'm not sure external_total and external_total_time_in_millis is the best way to express this, but it gets the point across. Also, should total and total_time_in_millis be changed to internal_total and so on?

cc @s1monw

[elasticmachine]

Pinging @elastic/es-distributed

[s1monw]

LGTM left some nit-picks. I am sorry for the delay on this. This looks pretty good, thank you!

[s1monw]

extra newline

[s1monw]

extra newline

[s1monw]

@elasticmachine ok to test

[s1monw]

@clandry94 can you merge your branch up to current master please

[s1monw]

@clandry94 there are some test failures, would you mind taking a look?

[clandry94]

Sure, I'll resolve them

[s1monw]

@clandry94 you don't have to forcepush for all changes it's easier to review if you don't. you can still just do a git merge master if you wanna catch up. we will do a squash and merge once we are ready here

[clandry94]

@s1monw in the elasticsearch-ci/2 IndexStatsMonitoringDocTests.testToXContent is failing, but for reasons that seem unrelated to the changes I made. Specifically,
expected:
{"query_total":18,"query_time_in_millis":19}
and actual:
{"query_total":19,"query_time_in_millis":20}

The actual values seem to be consistent over multiple tests, so should the expected values be changed to match that?

[s1monw]

Sorry for the late reply.

The actual values seem to be consistent over multiple tests, so should the expected values be changed to match that?

I think this change is the reason for this off-by-one: https://github.com/elastic/elasticsearch/pull/38643/files#diff-07c9dc27cfd5ff0ef6a21088ddf8b8d6R327

[s1monw]

you also need to merge with master since we updated to a new and incompatible lucene snapshot.

[dnhatn]

@clandry94 I tried to get this ready, but I did not have your permission. Could you please merge the master branch to your branch and apply this patch 38643.txt? We need to handle BWC. Thank you!

[clandry94]

Hey @dnhatn, merged master and applied your patch. Thanks for the help 😁

[dnhatn]

run elasticsearch-ci/packaging-sample

[dnhatn]

Hey @clandry94, can you please merge master into your branch? BWC was disabled in the revision that you merged. Thank you!

[dnhatn]

Thanks @clandry94 for working on this.