DateIndexNameProcessorTests.testTemplatedFields is repeatedly failing

[jasontedor]

This testing is failing intake builds and PR builds. I am going to mute it. The reproduction and failure is:

20:14:36    2> REPRODUCE WITH: ./gradlew :modules:ingest-common:unitTest -Dtests.seed=64E04C0DAD455367 -Dtests.class=org.elasticsearch.ingest.common.DateIndexNameProcessorTests -Dtests.method="testTemplatedFields" -Dtests.security.manager=true -Dtests.locale=hi -Dtests.timezone=America/Swift_Current -Dcompiler.java=11 -Druntime.java=8
20:14:36  Suite: org.elasticsearch.ingest.common.DateIndexNameProcessorTests
20:14:36    1> [2019-01-30T19:14:35,691][INFO ][o.e.i.c.DateIndexNameProcessorTests] [testTAI64N] before test
20:14:36    1> [2019-01-30T19:14:35,902][INFO ][o.e.i.c.DateIndexNameProcessorTests] [testTAI64N] after test
20:14:36    1> [2019-01-30T19:14:35,911][INFO ][o.e.i.c.DateIndexNameProcessorTests] [testUnix] before test
20:14:36    1> [2019-01-30T19:14:35,914][INFO ][o.e.i.c.DateIndexNameProcessorTests] [testUnix] after test
20:14:36    1> [2019-01-30T19:14:35,955][INFO ][o.e.i.c.DateIndexNameProcessorTests] [testTemplatedFields] before test
20:14:36    1> [2019-01-30T19:14:35,960][INFO ][o.e.i.c.DateIndexNameProcessorTests] [testTemplatedFields] after test
20:14:36  ERROR   0.09s J0 | DateIndexNameProcessorTests.testTemplatedFields <<< FAILURES!
20:14:36     > Throwable #1: java.lang.IllegalArgumentException: Illegal pattern component: XX
20:14:36    2> NOTE: leaving temporary files on disk at: /var/lib/jenkins/workspace/elastic+elasticsearch+pull-request-1/modules/ingest-common/build/testrun/unitTest/J0/temp/org.elasticsearch.ingest.common.DateIndexNameProcessorTests_64E04C0DAD455367-001
20:14:36    2> NOTE: test params are: codec=Asserting(Lucene80): {}, docValues:{}, maxPointsInLeafNode=1266, maxMBSortInHeap=7.28252914267392, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@52481cbc), locale=hi, timezone=America/Swift_Current
20:14:36    2> NOTE: Linux 4.15.0-1026-gcp amd64/Oracle Corporation 1.8.0_202 (64-bit)/cpus=16,threads=1,free=454933992,total=514850816
20:14:36    2> NOTE: All tests run in this JVM: [AppendProcessorFactoryTests, SortProcessorFactoryTests, FailProcessorTests, UppercaseProcessorFactoryTests, RemoveProcessorFactoryTests, JoinProcessorFactoryTests, DateIndexNameProcessorTests]
20:14:36     > 	at __randomizedtesting.SeedInfo.seed([64E04C0DAD455367:B203A55D45D331BD]:0)
20:14:36     > 	at org.joda.time.format.DateTimeFormat.parsePatternTo(DateTimeFormat.java:566)
20:14:36     > 	at org.joda.time.format.DateTimeFormat.createFormatterForPattern(DateTimeFormat.java:687)
20:14:36     > 	at org.joda.time.format.DateTimeFormat.forPattern(DateTimeFormat.java:177)
20:14:36     > 	at org.elasticsearch.ingest.common.DateIndexNameProcessor.execute(DateIndexNameProcessor.java:93)
20:14:36     > 	at org.elasticsearch.ingest.common.DateIndexNameProcessorTests.testTemplatedFields(DateIndexNameProcessorTests.java:95)
20:14:36     > 	at java.lang.Thread.run(Thread.java:748)
20:14:36    1> [2019-01-30T19:14:36,024][INFO ][o.e.i.c.DateIndexNameProcessorTests] [testUnixMs] before test
20:14:36    1> [2019-01-30T19:14:36,028][INFO ][o.e.i.c.DateIndexNameProcessorTests] [testUnixMs] after test
20:14:36    1> [2019-01-30T19:14:36,042][INFO ][o.e.i.c.DateIndexNameProcessorTests] [testJavaPattern] before test
20:14:36    1> [2019-01-30T19:14:36,099][INFO ][o.e.i.c.DateIndexNameProcessorTests] [testJavaPattern] after test
20:14:36  Completed [37/50] on J0 in 0.61s, 5 tests, 1 error <<< FAILURES!

[elasticmachine]

Pinging @elastic/es-core-infra

[jasontedor]

Muted in 89bffc2

[droberts195]

Sorry I broke this.

Although the test failure is fixed now, this has revealed that there is a place in the date index name processor that is not using the Joda BWC layer in 6.7 and not using Java 8 time formats in 7.0. It is

elasticsearch/modules/ingest-common/src/main/java/org/elasticsearch/ingest/common/DateIndexNameProcessor.java

Line 93 in cde126d

DateTimeFormatter formatter = DateTimeFormat.forPattern(indexNameFormat);

This is perplexing, because when I wrote "However, while making this change I found that changing the default for date processing in ingest also affects index naming with the date index name processor" in #37935 I must have observed a place where the date index name processor was being affected by the changes to org.elasticsearch.ingest.common.DateFormat in #37407. So I suspect that the date index name processor is currently in an inconsistent state where one aspect of it uses the Joda BWC layer or Java time formatting and another aspect of it still uses Joda time directly.

So I think to adhere to the principle that 6.7 will output deprecation warnings for uses of Joda time formats that won't work in 7.0 org.elasticsearch.ingest.common.DateIndexNameProcessor needs changing for 6.7 and then again in 7.0 to only use Java timestamp formats.

[spinscale]

I'll take it and work on that one today