[ML] Failures in ML tests in UpgradeClusterClientYamlTestSuiteIT

[imotov]

Failure: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+g1gc/5918

This seems to be caused by the error in C library, so it is most likely different form #30456

  2> REPRODUCE WITH: ./gradlew :x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestRunner -Dtests.seed=2400057C67EBE734 -Dtests.class=org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT -Dtests.method="test {p0=upgraded_cluster/30_ml_jobs_crud/Test job with no model memory limit has established model memory after reopening}" -Dtests.security.manager=true -Dtests.jvm.argline="-XX:-UseConcMarkSweepGC -XX:+UseG1GC" -Dtests.locale=en-IE -Dtests.timezone=EST5EDT -Dtests.rest.suite=upgraded_cluster
FAILURE 1.08s | UpgradeClusterClientYamlTestSuiteIT.test {p0=upgraded_cluster/30_ml_jobs_crud/Test job with no model memory limit has established model memory after reopening} <<< FAILURES!
   > Throwable #1: java.lang.AssertionError: Failure at [upgraded_cluster/30_ml_jobs_crud:88]: expected [2xx] status code but api [xpack.ml.close_job] returned [500 Internal Server Error] [{"error":{"root_cause":[{"type":"exception","reason":"Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n","stack_trace":"ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n]\n\tat org.elasticsearch.xpack.core.ml.utils.ExceptionsHelper.serverError(ExceptionsHelper.java:34)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.NativeAutodetectProcess.close(NativeAutodetectProcess.java:221)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectCommunicator.lambda$close$2(AutodetectCommunicator.java:157)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectProcessManager$AutodetectWorkerExecutorService.start(AutodetectProcessManager.java:740)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.lang.Thread.run(Thread.java:844)\n"}],"type":"exception","reason":"Exception closing autodetect process","caused_by":{"type":"exception","reason":"java.util.concurrent.ExecutionException: ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n]","caused_by":{"type":"execution_exception","reason":"execution_exception: ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n]","caused_by":{"type":"exception","reason":"Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n","stack_trace":"ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n]\n\tat org.elasticsearch.xpack.core.ml.utils.ExceptionsHelper.serverError(ExceptionsHelper.java:34)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.NativeAutodetectProcess.close(NativeAutodetectProcess.java:221)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectCommunicator.lambda$close$2(AutodetectCommunicator.java:157)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectProcessManager$AutodetectWorkerExecutorService.start(AutodetectProcessManager.java:740)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.lang.Thread.run(Thread.java:844)\n"},"stack_trace":"NotSerializableExceptionWrapper[execution_exception: ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n]]; nested: ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n];\n\tat java.util.concurrent.FutureTask.report(FutureTask.java:122)\n\tat java.util.concurrent.FutureTask.get(FutureTask.java:191)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectCommunicator.close(AutodetectCommunicator.java:170)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectProcessManager.closeJob(AutodetectProcessManager.java:571)\n\tat org.elasticsearch.xpack.ml.action.TransportOpenJobAction$JobTask.closeJob(TransportOpenJobAction.java:731)\n\tat org.elasticsearch.xpack.ml.action.TransportCloseJobAction$1.doRun(TransportCloseJobAction.java:270)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:724)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.lang.Thread.run(Thread.java:844)\nCaused by: ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n]\n\tat org.elasticsearch.xpack.core.ml.utils.ExceptionsHelper.serverError(ExceptionsHelper.java:34)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.NativeAutodetectProcess.close(NativeAutodetectProcess.java:221)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectCommunicator.lambda$close$2(AutodetectCommunicator.java:157)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectProcessManager$AutodetectWorkerExecutorService.start(AutodetectProcessManager.java:740)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625)\n\t... 3 more\n"},"stack_trace":"ElasticsearchException[java.util.concurrent.ExecutionException: ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n]]; nested: NotSerializableExceptionWrapper[execution_exception: ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n]]; nested: ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n];\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectCommunicator.close(AutodetectCommunicator.java:179)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectProcessManager.closeJob(AutodetectProcessManager.java:571)\n\tat org.elasticsearch.xpack.ml.action.TransportOpenJobAction$JobTask.closeJob(TransportOpenJobAction.java:731)\n\tat org.elasticsearch.xpack.ml.action.TransportCloseJobAction$1.doRun(TransportCloseJobAction.java:270)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:724)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.lang.Thread.run(Thread.java:844)\nCaused by: NotSerializableExceptionWrapper[execution_exception: ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n]]; nested: ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n];\n\tat java.util.concurrent.FutureTask.report(FutureTask.java:122)\n\tat java.util.concurrent.FutureTask.get(FutureTask.java:191)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectCommunicator.close(AutodetectCommunicator.java:170)\n\t... 8 more\nCaused by: ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n]\n\tat org.elasticsearch.xpack.core.ml.utils.ExceptionsHelper.serverError(ExceptionsHelper.java:34)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.NativeAutodetectProcess.close(NativeAutodetectProcess.java:221)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectCommunicator.lambda$close$2(AutodetectCommunicator.java:157)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectProcessManager$AutodetectWorkerExecutorService.start(AutodetectProcessManager.java:740)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625)\n\t... 3 more\n"},"stack_trace":"ElasticsearchException[Exception closing autodetect process]; nested: ElasticsearchException[java.util.concurrent.ExecutionException: ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n]]; nested: NotSerializableExceptionWrapper[execution_exception: ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n]]; nested: ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n];\n\tat org.elasticsearch.xpack.core.ml.utils.ExceptionsHelper.serverError(ExceptionsHelper.java:38)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectProcessManager.closeJob(AutodetectProcessManager.java:580)\n\tat org.elasticsearch.xpack.ml.action.TransportOpenJobAction$JobTask.closeJob(TransportOpenJobAction.java:731)\n\tat org.elasticsearch.xpack.ml.action.TransportCloseJobAction$1.doRun(TransportCloseJobAction.java:270)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:724)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.lang.Thread.run(Thread.java:844)\nCaused by: ElasticsearchException[java.util.concurrent.ExecutionException: ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n]]; nested: NotSerializableExceptionWrapper[execution_exception: ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n]]; nested: ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n];\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectCommunicator.close(AutodetectCommunicator.java:179)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectProcessManager.closeJob(AutodetectProcessManager.java:571)\n\t... 7 more\nCaused by: NotSerializableExceptionWrapper[execution_exception: ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n]]; nested: ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n];\n\tat java.util.concurrent.FutureTask.report(FutureTask.java:122)\n\tat java.util.concurrent.FutureTask.get(FutureTask.java:191)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectCommunicator.close(AutodetectCommunicator.java:170)\n\t... 8 more\nCaused by: ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n]\n\tat org.elasticsearch.xpack.core.ml.utils.ExceptionsHelper.serverError(ExceptionsHelper.java:34)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.NativeAutodetectProcess.close(NativeAutodetectProcess.java:221)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectCommunicator.lambda$close$2(AutodetectCommunicator.java:157)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectProcessManager$AutodetectWorkerExecutorService.start(AutodetectProcessManager.java:740)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625)\n\t... 3 more\n"},"status":500}]
   > 	at __randomizedtesting.SeedInfo.seed([2400057C67EBE734:AC543AA6C9178ACC]:0)
   > 	at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:365)
   > 	at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:342)
   > 	at java.lang.Thread.run(Thread.java:748)
   > Caused by: java.lang.AssertionError: expected [2xx] status code but api [xpack.ml.close_job] returned [500 Internal Server Error] [{"error":{"root_cause":[{"type":"exception","reason":"Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n","stack_trace":"ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n]\n\tat org.elasticsearch.xpack.core.ml.utils.ExceptionsHelper.serverError(ExceptionsHelper.java:34)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.NativeAutodetectProcess.close(NativeAutodetectProcess.java:221)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectCommunicator.lambda$close$2(AutodetectCommunicator.java:157)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectProcessManager$AutodetectWorkerExecutorService.start(AutodetectProcessManager.java:740)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.lang.Thread.run(Thread.java:844)\n"}],"type":"exception","reason":"Exception closing autodetect process","caused_by":{"type":"exception","reason":"java.util.concurrent.ExecutionException: ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n]","caused_by":{"type":"execution_exception","reason":"execution_exception: ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n]","caused_by":{"type":"exception","reason":"Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n","stack_trace":"ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n]\n\tat org.elasticsearch.xpack.core.ml.utils.ExceptionsHelper.serverError(ExceptionsHelper.java:34)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.NativeAutodetectProcess.close(NativeAutodetectProcess.java:221)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectCommunicator.lambda$close$2(AutodetectCommunicator.java:157)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectProcessManager$AutodetectWorkerExecutorService.start(AutodetectProcessManager.java:740)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.lang.Thread.run(Thread.java:844)\n"},"stack_trace":"NotSerializableExceptionWrapper[execution_exception: ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n]]; nested: ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n];\n\tat java.util.concurrent.FutureTask.report(FutureTask.java:122)\n\tat java.util.concurrent.FutureTask.get(FutureTask.java:191)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectCommunicator.close(AutodetectCommunicator.java:170)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectProcessManager.closeJob(AutodetectProcessManager.java:571)\n\tat org.elasticsearch.xpack.ml.action.TransportOpenJobAction$JobTask.closeJob(TransportOpenJobAction.java:731)\n\tat org.elasticsearch.xpack.ml.action.TransportCloseJobAction$1.doRun(TransportCloseJobAction.java:270)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:724)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.lang.Thread.run(Thread.java:844)\nCaused by: ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n]\n\tat org.elasticsearch.xpack.core.ml.utils.ExceptionsHelper.serverError(ExceptionsHelper.java:34)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.NativeAutodetectProcess.close(NativeAutodetectProcess.java:221)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectCommunicator.lambda$close$2(AutodetectCommunicator.java:157)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectProcessManager$AutodetectWorkerExecutorService.start(AutodetectProcessManager.java:740)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625)\n\t... 3 more\n"},"stack_trace":"ElasticsearchException[java.util.concurrent.ExecutionException: ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n]]; nested: NotSerializableExceptionWrapper[execution_exception: ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n]]; nested: ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n];\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectCommunicator.close(AutodetectCommunicator.java:179)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectProcessManager.closeJob(AutodetectProcessManager.java:571)\n\tat org.elasticsearch.xpack.ml.action.TransportOpenJobAction$JobTask.closeJob(TransportOpenJobAction.java:731)\n\tat org.elasticsearch.xpack.ml.action.TransportCloseJobAction$1.doRun(TransportCloseJobAction.java:270)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:724)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.lang.Thread.run(Thread.java:844)\nCaused by: NotSerializableExceptionWrapper[execution_exception: ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n]]; nested: ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n];\n\tat java.util.concurrent.FutureTask.report(FutureTask.java:122)\n\tat java.util.concurrent.FutureTask.get(FutureTask.java:191)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectCommunicator.close(AutodetectCommunicator.java:170)\n\t... 8 more\nCaused by: ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n]\n\tat org.elasticsearch.xpack.core.ml.utils.ExceptionsHelper.serverError(ExceptionsHelper.java:34)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.NativeAutodetectProcess.close(NativeAutodetectProcess.java:221)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectCommunicator.lambda$close$2(AutodetectCommunicator.java:157)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectProcessManager$AutodetectWorkerExecutorService.start(AutodetectProcessManager.java:740)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625)\n\t... 3 more\n"},"stack_trace":"ElasticsearchException[Exception closing autodetect process]; nested: ElasticsearchException[java.util.concurrent.ExecutionException: ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n]]; nested: NotSerializableExceptionWrapper[execution_exception: ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n]]; nested: ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n];\n\tat org.elasticsearch.xpack.core.ml.utils.ExceptionsHelper.serverError(ExceptionsHelper.java:38)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectProcessManager.closeJob(AutodetectProcessManager.java:580)\n\tat org.elasticsearch.xpack.ml.action.TransportOpenJobAction$JobTask.closeJob(TransportOpenJobAction.java:731)\n\tat org.elasticsearch.xpack.ml.action.TransportCloseJobAction$1.doRun(TransportCloseJobAction.java:270)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:724)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.lang.Thread.run(Thread.java:844)\nCaused by: ElasticsearchException[java.util.concurrent.ExecutionException: ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n]]; nested: NotSerializableExceptionWrapper[execution_exception: ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n]]; nested: ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n];\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectCommunicator.close(AutodetectCommunicator.java:179)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectProcessManager.closeJob(AutodetectProcessManager.java:571)\n\t... 7 more\nCaused by: NotSerializableExceptionWrapper[execution_exception: ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n]]; nested: ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n];\n\tat java.util.concurrent.FutureTask.report(FutureTask.java:122)\n\tat java.util.concurrent.FutureTask.get(FutureTask.java:191)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectCommunicator.close(AutodetectCommunicator.java:170)\n\t... 8 more\nCaused by: ElasticsearchException[Fatal error: 'si_signo 11, si_code: 1, si_errno: 0, address: 0x7f692bfcd56c, library: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/cluster/v6.4.0-SNAPSHOT#mixedClusterTestCluster node0/elasticsearch-7.0.0-alpha1-SNAPSHOT/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f692bd6c000, normalized address: 0x26156c', version: 7.0.0-alpha1-SNAPSHOT (build 71dc485b6fd7da)\n]\n\tat org.elasticsearch.xpack.core.ml.utils.ExceptionsHelper.serverError(ExceptionsHelper.java:34)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.NativeAutodetectProcess.close(NativeAutodetectProcess.java:221)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectCommunicator.lambda$close$2(AutodetectCommunicator.java:157)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625)\n\tat org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectProcessManager$AutodetectWorkerExecutorService.start(AutodetectProcessManager.java:740)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625)\n\t... 3 more\n"},"status":500}]
   > 	at org.elasticsearch.test.rest.yaml.section.DoSection.execute(DoSection.java:241)
   > 	at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:358)
   > 	... 38 more
  1> [2018-05-30T16:47:41,100][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] [test {p0=upgraded_cluster/30_ml_jobs_crud/Test open old jobs}]: before test
  1> [2018-05-30T16:47:42,726][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] Stash dump on test failure [{
  1>   "stash" : {
  1>     "body" : {
  1>       "count" : 1,
  1>       "jobs" : [
  1>         {
  1>           "job_id" : "mixed-cluster-job",
  1>           "data_counts" : {
  1>             "job_id" : "mixed-cluster-job",
  1>             "processed_record_count" : 2,
  1>             "processed_field_count" : 4,
  1>             "input_bytes" : 178,
  1>             "input_field_count" : 6,
  1>             "invalid_date_count" : 0,
  1>             "missing_field_count" : 0,
  1>             "out_of_order_timestamp_count" : 0,
  1>             "empty_bucket_count" : 0,
  1>             "sparse_bucket_count" : 0,
  1>             "bucket_count" : 1,
  1>             "earliest_record_timestamp" : 1403481600000,
  1>             "latest_record_timestamp" : 1403481700000,
  1>             "last_data_time" : 1527713202621,
  1>             "input_record_count" : 2
  1>           },
  1>           "model_size_stats" : {
  1>             "job_id" : "mixed-cluster-job",
  1>             "result_type" : "model_size_stats",
  1>             "model_bytes" : 125204,
  1>             "total_by_field_count" : 4,
  1>             "total_over_field_count" : 0,
  1>             "total_partition_field_count" : 2,
  1>             "bucket_allocation_failures_count" : 0,
  1>             "memory_status" : "ok",
  1>             "log_time" : 1527713202000,
  1>             "timestamp" : 1403481600000
  1>           },
  1>           "state" : "opened",
  1>           "assignment_explanation" : ""
  1>         }
  1>       ]
  1>     }
  1>   }
  1> }]
  1> [2018-05-30T16:47:42,766][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] There are still tasks running after this test that might break subsequent tests [xpack/ml/job[c]].
  1> [2018-05-30T16:47:42,767][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] [test {p0=upgraded_cluster/30_ml_jobs_crud/Test open old jobs}]: after test
  2> REPRODUCE WITH: ./gradlew :x-pack:qa:rolling-upgrade:with-system-key:v6.4.0-SNAPSHOT#upgradedClusterTestRunner -Dtests.seed=2400057C67EBE734 -Dtests.class=org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT -Dtests.method="test {p0=upgraded_cluster/30_ml_jobs_crud/Test open old jobs}" -Dtests.security.manager=true -Dtests.jvm.argline="-XX:-UseConcMarkSweepGC -XX:+UseG1GC" -Dtests.locale=en-IE -Dtests.timezone=EST5EDT -Dtests.rest.suite=upgraded_cluster
FAILURE 1.68s | UpgradeClusterClientYamlTestSuiteIT.test {p0=upgraded_cluster/30_ml_jobs_crud/Test open old jobs} <<< FAILURES!
   > Throwable #1: java.lang.AssertionError: Failure at [upgraded_cluster/30_ml_jobs_crud:35]: field [jobs.0.node] doesn't have a true value
   > Expected: not null
   >      but: was null
   > 	at __randomizedtesting.SeedInfo.seed([2400057C67EBE734:AC543AA6C9178ACC]:0)
   > 	at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:365)
   > 	at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:342)
   > 	at java.lang.Thread.run(Thread.java:748)
   > Caused by: java.lang.AssertionError: field [jobs.0.node] doesn't have a true value
   > Expected: not null
   >      but: was null
   > 	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 	at org.elasticsearch.test.rest.yaml.section.IsTrueAssertion.doAssert(IsTrueAssertion.java:55)
   > 	at org.elasticsearch.test.rest.yaml.section.Assertion.execute(Assertion.java:76)
   > 	at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:358)
   > 	... 38 more
  1> [2018-05-30T16:47:42,777][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] [test {p0=upgraded_cluster/30_ml_jobs_crud/Test get job with function shortcut should expand}]: before test
  1> [2018-05-30T16:47:43,185][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] There are still tasks running after this test that might break subsequent tests [xpack/ml/job[c]].
  1> [2018-05-30T16:47:43,185][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] [test {p0=upgraded_cluster/30_ml_jobs_crud/Test get job with function shortcut should expand}]: after test
  1> [2018-05-30T16:47:43,208][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] [test {p0=upgraded_cluster/10_basic/Index data and search on the upgraded cluster}]: before test
  1> [2018-05-30T16:47:44,132][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] There are still tasks running after this test that might break subsequent tests [xpack/ml/job[c]].
  1> [2018-05-30T16:47:44,133][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] [test {p0=upgraded_cluster/10_basic/Index data and search on the upgraded cluster}]: after test
  2> NOTE: leaving temporary files on disk at: /var/lib/jenkins/workspace/elastic+elasticsearch+master+g1gc/x-pack/qa/rolling-upgrade/with-system-key/build/testrun/v6.4.0-SNAPSHOT#upgradedClusterTestRunner/J0/temp/org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT_2400057C67EBE734-001
  2> NOTE: test params are: codec=Lucene70, sim=RandomSimilarity(queryNorm=false): {}, locale=en-IE, timezone=EST5EDT
  2> NOTE: Linux 3.16.0-4-amd64 amd64/Oracle Corporation 1.8.0_172 (64-bit)/cpus=4,threads=1,free=435702720,total=536870912
  2> NOTE: All tests run in this JVM: [UpgradeClusterClientYamlTestSuiteIT]
Completed [1/3] in 22.18s, 9 tests, 2 failures <<< FAILURES!

[elasticmachine]

Pinging @elastic/ml-core

[hub-cap]

This hit me on a pull request as well. Did we mute it?

[imotov]

No. Sorry, was trying to figure out the best way to mute them and got side tracked. Will mute them in a bit.

[hub-cap]

p0=upgraded_cluster/30_ml_jobs_crud/Test open old jobs was the test that failed on the PR build.

[droberts195]

@tveasey this failure almost certainly means that the changes introduced in elastic/ml-cpp#108 mean that a 7.0.0-alpha1 autodetect process cannot load the model state saved by a 6.4.0 autodetect process. This problem started happening yesterday and elastic/ml-cpp#108 was the only C++ change yesterday that touched the maths library.

[droberts195]

I muted the test of the problematic job in the mixed cluster in 0ff2c60. It was previously muted in the upgraded cluster, but can fail in either.

[tveasey]

This issue was caused by elastic/ml-cpp#108 being present on master and not 6.4 (although it was targeted at 6.4, I hadn't backported). I'm preparing a version which fixes the preexisting bug, which causes these test failures in a different way. In particular, this doesn't rely on changing the state we write and so still works even with the buggy version of 6.4 writing out state.