roachtest: jepsen/sets/majority-ring-start-kill-2 failed

[cockroach-teamcity]

roachtest.jepsen/sets/majority-ring-start-kill-2 failed with artifacts on release-25.3 @ b6086c251bd146161763c378ad9a3a3610f8380b:

(jepsen.go:442).runJepsen: timed out
test artifacts and logs in: /artifacts/jepsen/sets/majority-ring-start-kill-2/cpu_arch=s390x/run_1

Parameters:

• arch=s390x
• cloud=ibm
• coverageBuild=false
• cpu=4
• encrypted=false
• metamorphicBufferedSender=true
• metamorphicLeases=default
• runtimeAssertionsBuild=false
• ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for ibm clusters

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-53431

[srosenberg]

It appears to be an SSH flake which caused the jepsen test runner to hang. From invoke.log,

WARN [2025-08-12 09:23:45,275] jepsen node 10.249.1.228 - jepsen.control Encountered error with conn [:control "10.249.1.228"]; reopening

This correlates with journalctl on n3,

Aug 12 09:23:45 teamcity-20328972-1754973075-145-n6cpu4-0003 sshd[37143]: Disconnecting user ubuntu 10.249.1.225 port 40920: data packet referred to nonexistent channel 0
Aug 12 09:23:45 teamcity-20328972-1754973075-145-n6cpu4-0003 sshd[37286]: ssh_dispatch_run_fatal: Connection from 10.249.1.225 port 54864: message authentication code incorrect [preauth]

Immediately after, we can see that reset-clock [1] was successfully executed on every worker node with the exception of n3 (10.249.1.228),

INFO [2025-08-12 09:23:45,955] jepsen node 10.249.1.226 - jepsen.cockroach.auto 10.249.1.226 Cockroach installed
INFO [2025-08-12 09:23:45,985] jepsen node 10.249.1.229 - jepsen.cockroach.auto 10.249.1.229 Cockroach installed
INFO [2025-08-12 09:23:45,993] jepsen node 10.249.1.227 - jepsen.cockroach.auto 10.249.1.227 Cockroach installed
INFO [2025-08-12 09:23:46,059] jepsen node 10.249.1.230 - jepsen.cockroach.auto 10.249.1.230 Cockroach installed
INFO [2025-08-12 09:23:52,767] jepsen node 10.249.1.226 - jepsen.cockroach.time 10.249.1.226 clock reset: 12 Aug 09:23:52 ntpdate[34995]: step time server 216.232.132.102 offset -0.000050 sec
INFO [2025-08-12 09:23:52,796] jepsen node 10.249.1.229 - jepsen.cockroach.time 10.249.1.229 clock reset: 12 Aug 09:23:52 ntpdate[38840]: step time server 216.232.132.102 offset -0.000230 sec
INFO [2025-08-12 09:23:52,805] jepsen node 10.249.1.227 - jepsen.cockroach.time 10.249.1.227 clock reset: 12 Aug 09:23:52 ntpdate[36067]: step time server 216.232.132.102 offset -0.000280 sec
INFO [2025-08-12 09:23:52,870] jepsen node 10.249.1.230 - jepsen.cockroach.time 10.249.1.230 clock reset: 12 Aug 09:23:52 ntpdate[37470]: step time server 149.56.19.163 offset +0.000049 sec

The thread dump indicates that the test runner is stuck waiting for each Future to finish (via jepsen.control/on-nodes [2]),

"jepsen test runner" #1 prio=5 os_prio=0 cpu=1773.80ms elapsed=2400.40s tid=0x000003ff9801b1f0 nid=0x2c890 waiting on condition  [0x000003ff9f8fc000]
   java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park(java.base@17.0.16/Native Method)
        - parking to wait for  <0x00000007131ab508> (a java.util.concurrent.FutureTask)
        at java.util.concurrent.locks.LockSupport.park(java.base@17.0.16/LockSupport.java:211)
        at java.util.concurrent.FutureTask.awaitDone(java.base@17.0.16/FutureTask.java:447)
        at java.util.concurrent.FutureTask.get(java.base@17.0.16/FutureTask.java:190)
        at clojure.core$deref_future.invokeStatic(core.clj:2208)
        at clojure.core$future_call$reify__6962.deref(core.clj:6688)
        at clojure.core$deref.invokeStatic(core.clj:2228)
        at clojure.core$deref.invoke(core.clj:2214)
        at clojure.core$map$fn__4785.invoke(core.clj:2644)
        at clojure.lang.LazySeq.sval(LazySeq.java:40)
        - eliminated <0x00000007131ab9a8> (a clojure.lang.LazySeq)
        at clojure.lang.LazySeq.seq(LazySeq.java:49)
        - locked <0x00000007131ab9a8> (a clojure.lang.LazySeq)
        at clojure.lang.RT.seq(RT.java:521)
        at clojure.core$seq__4357.invokeStatic(core.clj:137)
        at clojure.core.protocols$seq_reduce.invokeStatic(protocols.clj:24)
        at clojure.core.protocols$fn__6738.invokeStatic(protocols.clj:75)
        at clojure.core.protocols$fn__6738.invoke(protocols.clj:75)
        at clojure.core.protocols$fn__6684$G__6679__6697.invoke(protocols.clj:13)
        at clojure.core$reduce.invokeStatic(core.clj:6545)
        at clojure.core$into.invokeStatic(core.clj:6610)
        at clojure.core$into.invoke(core.clj:6604)
        at jepsen.control$on_nodes.invokeStatic(control.clj:373)
        at jepsen.control$on_nodes.invoke(control.clj:357)
        at jepsen.control$on_nodes.invokeStatic(control.clj:362)
        at jepsen.control$on_nodes.invoke(control.clj:357)
        at jepsen.db$cycle_BANG_$fn__2210.invoke(db.clj:45)

while the remaining worker nodes are waiting on CyclicBarrier (via jepsen.synchronize [3]), e.g.,

"jepsen node 10.249.1.230" #17 prio=5 os_prio=0 cpu=203.89ms elapsed=2398.46s tid=0x000003ff98a4ab70 nid=0x2c8aa waiting on condition  [0x000003ff790fa000]
   java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park(java.base@17.0.16/Native Method)
        - parking to wait for  <0x00000007087d29e0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(java.base@17.0.16/LockSupport.java:341)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@17.0.16/AbstractQueuedSynchronizer.java:506)
        at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@17.0.16/ForkJoinPool.java:3465)
        at java.util.concurrent.ForkJoinPool.managedBlock(java.base@17.0.16/ForkJoinPool.java:3436)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@17.0.16/AbstractQueuedSynchronizer.java:1630)
        at java.util.concurrent.CyclicBarrier.dowait(java.base@17.0.16/CyclicBarrier.java:236)
        at java.util.concurrent.CyclicBarrier.await(java.base@17.0.16/CyclicBarrier.java:364)
        at jepsen.core$synchronize.invokeStatic(core.clj:44)
        at jepsen.core$synchronize.invoke(core.clj:39)
        at jepsen.cockroach$db$reify__903.setup_BANG_(cockroach.clj:59)
        at jepsen.db$fn__2152$G__2131__2156.invoke(db.clj:8)
        at jepsen.db$fn__2152$G__2130__2161.invoke(db.clj:8)
        at clojure.core$partial$fn__4759.invoke(core.clj:2516)
        at jepsen.control$on_nodes$fn__2116.invoke(control.clj:372)

[1] https://github.com/cockroachdb/jepsen/blob/4beba6c2d63587ab9ffaa24580f73b5674e65e55/cockroachdb/src/jepsen/cockroach.clj#L57-L58
[2] https://github.com/cockroachdb/jepsen/blob/4beba6c2d63587ab9ffaa24580f73b5674e65e55/jepsen/src/jepsen/control.clj#L357-L373
[3] https://github.com/cockroachdb/jepsen/blob/4beba6c2d63587ab9ffaa24580f73b5674e65e55/jepsen/src/jepsen/core.clj#L40-L53

[srosenberg]

Yep, I think we've been running into this deadlock [1] 😱, which hasn't been backported into our tc-nightly-main branch.

@herkolategan @golgeek mind doing a quick sanity check by looking at another open jepsen issue. My bet is you'll find the same recurring pattern.

[1] cockroachdb/jepsen@dfd82f4

[herkolategan]

Yes, seems we have a few with the same problem.

[srosenberg]

For tracking: #152140