LinearAlgebra/addmul all-NAN batman?

[vtjnash]

https://buildkite.com/julialang/julia-master/builds/39891#0191dc33-022c-4259-91a9-73052a416b15

The global RNG seed was 0x86c87ef1d9841135b01925d2cddb12f3.

2024-09-10 09:58:46 EDT	LinearAlgebra/addmul                             (4) |        started at 2024-09-10T06:58:46.809
2024-09-10 09:58:46 EDT	      From worker 2:	┌ Info: Pkg test output is being logged to file
2024-09-10 09:58:52 EDT	      From worker 2:	└   logfile = "/private/var/tmp/agent-tempdirs/default-honeycrisp-R17H3W25T9.0/tmp/jl_hY6Npb/Pkg.log"
2024-09-10 09:58:52 EDT	LinearAlgebra/addmul                             (4) |         failed at 2024-09-10T07:02:19.349
2024-09-10 10:02:19 EDT	Test Failed at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-R17H3W25T9.0/build/default-honeycrisp-R17H3W25T9-0/julialang/julia-master/julia-03bcfabc49/share/julia/stdlib/v1.12/LinearAlgebra/test/addmul.jl:217
2024-09-10 10:02:19 EDT	  Expression: ≈(collect(returned_mat), β * Cc, rtol = rtol)
2024-09-10 10:02:19 EDT	   Evaluated: BigFloat[NaN NaN NaN; NaN NaN NaN; NaN NaN NaN] ≈ BigFloat[-0.0 -0.0 0.0; 0.0 -0.0 0.0; 0.0 -0.0 0.0] (rtol=1.490116119384765625e-08)
2024-09-10 10:02:19 EDT	
2024-09-10 10:02:19 EDT	Test Failed at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-R17H3W25T9.0/build/default-honeycrisp-R17H3W25T9-0/julialang/julia-master/julia-03bcfabc49/share/julia/stdlib/v1.12/LinearAlgebra/test/addmul.jl:217
2024-09-10 10:02:19 EDT	  Expression: ≈(collect(returned_mat), β * Cc, rtol = rtol)
2024-09-10 10:02:19 EDT	   Evaluated: BigFloat[NaN NaN NaN; NaN NaN NaN; NaN NaN NaN] ≈ BigFloat[-0.0 0.0 0.0; -0.0 -0.0 0.0; -0.0 0.0 -0.0] (rtol=1.490116119384765625e-08)
2024-09-10 10:02:19 EDT	
2024-09-10 10:02:19 EDT	Test Failed at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-R17H3W25T9.0/build/default-honeycrisp-R17H3W25T9-0/julialang/julia-master/julia-03bcfabc49/share/julia/stdlib/v1.12/LinearAlgebra/test/addmul.jl:217
2024-09-10 10:02:19 EDT	  Expression: ≈(collect(returned_mat), β * Cc, rtol = rtol)
2024-09-10 10:02:19 EDT	   Evaluated: BigFloat[NaN NaN NaN; NaN NaN NaN; NaN NaN NaN] ≈ BigFloat[0.2468775747606804121793463971704163831753556924030321117799588633012763949636792 -0.2944033898256054340326811335427821244620981450965170537467435404876958671138709 -0.0; 0.1105704801122580387262135695449126682700732513004014353054797249222007682334379 0.01646088265496153652586773609943219368760753659892790074862834517064116846007948 -0.1226737091866446431679522666013409844046948042256403986634792635523893977733678; 0.0 -0.01750725476545199720173290779507530887563182922943404251344367666661308580167145 0.002899617618022406755159186450545626774260204146964936920699178391673880850198941] (rtol=1.490116119384765625e-08)
2024-09-10 10:02:19 EDT	
2024-09-10 10:02:19 EDT	Test Failed at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-R17H3W25T9.0/build/default-honeycrisp-R17H3W25T9-0/julialang/julia-master/julia-03bcfabc49/share/julia/stdlib/v1.12/LinearAlgebra/test/addmul.jl:217
2024-09-10 10:02:19 EDT	  Expression: ≈(collect(returned_mat), β * Cc, rtol = rtol)
2024-09-10 10:02:19 EDT	   Evaluated: BigFloat[NaN NaN NaN; NaN NaN NaN; NaN NaN NaN] ≈ BigFloat[-0.0 0.0 0.0; -0.0 -0.0 0.0; -0.0 0.0 -0.0] (rtol=1.490116119384765625e-08)
2024-09-10 10:02:19 EDT	
2024-09-10 10:02:19 EDT	Test Failed at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-R17H3W25T9.0/build/default-honeycrisp-R17H3W25T9-0/julialang/julia-master/julia-03bcfabc49/share/julia/stdlib/v1.12/LinearAlgebra/test/addmul.jl:217
2024-09-10 10:02:19 EDT	  Expression: ≈(collect(returned_mat), β * Cc, rtol = rtol)
2024-09-10 10:02:19 EDT	   Evaluated: BigFloat[NaN NaN NaN; NaN NaN NaN; NaN NaN NaN] ≈ BigFloat[-0.0 0.0 0.0; -0.0 -0.0 0.0; -0.0 0.0 -0.0] (rtol=1.490116119384765625e-08)
2024-09-10 10:02:19 EDT	
2024-09-10 10:02:19 EDT	Test Failed at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-R17H3W25T9.0/build/default-honeycrisp-R17H3W25T9-0/julialang/julia-master/julia-03bcfabc49/share/julia/stdlib/v1.12/LinearAlgebra/test/addmul.jl:217
2024-09-10 10:02:19 EDT	  Expression: ≈(collect(returned_mat), β * Cc, rtol = rtol)
2024-09-10 10:02:19 EDT	   Evaluated: BigFloat[NaN NaN NaN; NaN NaN NaN; NaN NaN NaN] ≈ BigFloat[-1.113088120070599243882381288576113639043096494006868279418693504917436953860488 1.327366068145556101898014989487982975685379414509607144676060601351519352419928 0.0; -0.4985251818143617240339265206227892132114873224174599783961182972343018082739263 -0.07421659479146842765250359575712698909471069923687955952914250997961079225895276 0.5530945792586307315223928613717379923951757535568742754172703868231764090523939; -0.0 0.07893433542258561184688271709265617550732497951056374907217905477008548786876929 -0.01307340258221861982298668123734899736966454416226622751775273109154686113075883] (rtol=1.490116119384765625e-08)
2024-09-10 10:02:19 EDT	
2024-09-10 10:02:19 EDT	Test Failed at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-R17H3W25T9.0/build/default-honeycrisp-R17H3W25T9-0/julialang/julia-master/julia-03bcfabc49/share/julia/stdlib/v1.12/LinearAlgebra/test/addmul.jl:217
2024-09-10 10:02:19 EDT	  Expression: ≈(collect(returned_mat), β * Cc, rtol = rtol)
2024-09-10 10:02:19 EDT	   Evaluated: BigFloat[NaN NaN NaN; NaN NaN NaN; NaN NaN NaN] ≈ BigFloat[-0.0 0.0 0.0; -0.0 -0.0 0.0; -0.0 0.0 -0.0] (rtol=1.490116119384765625e-08)
2024-09-10 10:02:19 EDT	
2024-09-10 10:02:19 EDT	Test Failed at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-R17H3W25T9.0/build/default-honeycrisp-R17H3W25T9-0/julialang/julia-master/julia-03bcfabc49/share/julia/stdlib/v1.12/LinearAlgebra/test/addmul.jl:217
2024-09-10 10:02:19 EDT	  Expression: ≈(collect(returned_mat), β * Cc, rtol = rtol)
2024-09-10 10:02:19 EDT	   Evaluated: BigFloat[NaN NaN NaN; NaN NaN NaN; NaN NaN NaN] ≈ BigFloat[-0.0 0.0 0.0; -0.0 -0.0 0.0; -0.0 0.0 -0.0] (rtol=1.490116119384765625e-08)
2024-09-10 10:02:19 EDT	
2024-09-10 10:02:19 EDT	Test Failed at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-R17H3W25T9.0/build/default-honeycrisp-R17H3W25T9-0/julialang/julia-master/julia-03bcfabc49/share/julia/stdlib/v1.12/LinearAlgebra/test/addmul.jl:217
2024-09-10 10:02:19 EDT	  Expression: ≈(collect(returned_mat), β * Cc, rtol = rtol)
2024-09-10 10:02:19 EDT	   Evaluated: BigFloat[NaN NaN NaN; NaN NaN NaN; NaN NaN NaN] ≈ BigFloat[-1.007554231665407050220161121354512645541517175546129657276848514305054408732153 1.201516101747902929721657384828846022140123550526431434315544382173673871359584 0.0; -0.4512591118993957371598474800272126494025785938803991888462954271059635825740176 -0.06717998583723808515505069221320737671182511222226632237746894271729015265232513 0.5006546865381021468214915041445218650931317827578371899983806882678231514976733; -0.0 0.07145042898641162678941307286874432769679450534382355872511671656660318304969854 -0.01183388974912818793693487458767585561447531789184800193336755352446658688589767] (rtol=1.490116119384765625e-08)
2024-09-10 10:02:19 EDT	
2024-09-10 10:02:19 EDT	Test Failed at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-R17H3W25T9.0/build/default-honeycrisp-R17H3W25T9-0/julialang/julia-master/julia-03bcfabc49/share/julia/stdlib/v1.12/LinearAlgebra/test/addmul.jl:217
2024-09-10 10:02:19 EDT	  Expression: ≈(collect(returned_mat), β * Cc, rtol = rtol)
2024-09-10 10:02:19 EDT	   Evaluated: BigFloat[NaN NaN NaN; NaN NaN NaN; NaN NaN NaN] ≈ BigFloat[0.0 0.0 0.0; -0.0 -0.0 0.0; 0.0 -0.0 0.0] (rtol=1.490116119384765625e-08)
2024-09-10 10:02:19 EDT	
2024-09-10 10:02:19 EDT	Test Failed at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-R17H3W25T9.0/build/default-honeycrisp-R17H3W25T9-0/julialang/julia-master/julia-03bcfabc49/share/julia/stdlib/v1.12/LinearAlgebra/test/addmul.jl:217
2024-09-10 10:02:19 EDT	  Expression: ≈(collect(returned_mat), β * Cc, rtol = rtol)
2024-09-10 10:02:19 EDT	   Evaluated: BigFloat[NaN NaN NaN; NaN NaN NaN; NaN NaN NaN] ≈ BigFloat[-0.0 0.0 0.0; -0.0 0.0 -0.0; -0.0 -0.0 0.0] (rtol=1.490116119384765625e-08)
2024-09-10 10:02:19 EDT	
2024-09-10 10:02:19 EDT	Test Failed at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-R17H3W25T9.0/build/default-honeycrisp-R17H3W25T9-0/julialang/julia-master/julia-03bcfabc49/share/julia/stdlib/v1.12/LinearAlgebra/test/addmul.jl:217
2024-09-10 10:02:19 EDT	  Expression: ≈(collect(returned_mat), β * Cc, rtol = rtol)
2024-09-10 10:02:19 EDT	   Evaluated: BigFloat[NaN NaN NaN; NaN NaN NaN; NaN NaN NaN] ≈ BigFloat[0.6313289309426461994253539008909743887760256155186406643885547139427583118731491 -0.0 -0.0; 0.0 -0.4742224132978760569619307627870916304696846405181849111614192891772656679449938 0.7113336199468141547644161908287268079560758484219563935608806118371383320209134; 0.0 0.1015172606274782901005221336413898955240526705909182497827159116429718288543427 -0.08459771718956523597364177414715079941574815152204195591736728652987320487000034] (rtol=1.490116119384765625e-08)
2024-09-10 10:02:19 EDT	
2024-09-10 10:02:19 EDT	Test Failed at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-R17H3W25T9.0/build/default-honeycrisp-R17H3W25T9-0/julialang/julia-master/julia-03bcfabc49/share/julia/stdlib/v1.12/LinearAlgebra/test/addmul.jl:217
2024-09-10 10:02:19 EDT	  Expression: ≈(collect(returned_mat), β * Cc, rtol = rtol)
2024-09-10 10:02:19 EDT	   Evaluated: BigFloat[NaN NaN NaN; NaN NaN NaN; NaN NaN NaN] ≈ BigFloat[-0.0 0.0 0.0; -0.0 0.0 -0.0; -0.0 -0.0 0.0] (rtol=1.490116119384765625e-08)
2024-09-10 10:02:19 EDT	
2024-09-10 10:02:19 EDT	Test Failed at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-R17H3W25T9.0/build/default-honeycrisp-R17H3W25T9-0/julialang/julia-master/julia-03bcfabc49/share/julia/stdlib/v1.12/LinearAlgebra/test/addmul.jl:217
2024-09-10 10:02:19 EDT	  Expression: ≈(collect(returned_mat), β * Cc, rtol = rtol)
2024-09-10 10:02:19 EDT	   Evaluated: BigFloat[NaN NaN NaN; NaN NaN NaN; NaN NaN NaN] ≈ BigFloat[-0.0 0.0 0.0; -0.0 0.0 -0.0; -0.0 -0.0 0.0] (rtol=1.490116119384765625e-08)
2024-09-10 10:02:19 EDT	
2024-09-10 10:02:19 EDT	Test Failed at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-R17H3W25T9.0/build/default-honeycrisp-R17H3W25T9-0/julialang/julia-master/julia-03bcfabc49/share/julia/stdlib/v1.12/LinearAlgebra/test/addmul.jl:217
2024-09-10 10:02:19 EDT	  Expression: ≈(collect(returned_mat), β * Cc, rtol = rtol)
2024-09-10 10:02:19 EDT	   Evaluated: BigFloat[NaN NaN NaN; NaN NaN NaN; NaN NaN NaN] ≈ BigFloat[-2.84645024389251445875345570840184645368065042915015304646604533608811316014453 0.0 0.0; -0.0 2.138109688677746990415719194676375558023152075792217637066519164007472042499103 -3.207164533016620798171044895238445668609438779980923135186029505894200752891265; -0.0 -0.4577072538730793327403025549822756476192873039994018773958891653161340506983379 0.3814227115608994179046299538832395120515135311417851745310534077731956514867306] (rtol=1.490116119384765625e-08)
2024-09-10 10:02:19 EDT	
2024-09-10 10:02:19 EDT	Test Failed at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-R17H3W25T9.0/build/default-honeycrisp-R17H3W25T9-0/julialang/julia-master/julia-03bcfabc49/share/julia/stdlib/v1.12/LinearAlgebra/test/addmul.jl:217
2024-09-10 10:02:19 EDT	  Expression: ≈(collect(returned_mat), β * Cc, rtol = rtol)
2024-09-10 10:02:19 EDT	   Evaluated: BigFloat[NaN NaN NaN; NaN NaN NaN; NaN NaN NaN] ≈ BigFloat[-0.0 0.0 0.0; -0.0 0.0 -0.0; -0.0 -0.0 0.0] (rtol=1.490116119384765625e-08)
2024-09-10 10:02:19 EDT	
2024-09-10 10:02:19 EDT	Test Failed at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-R17H3W25T9.0/build/default-honeycrisp-R17H3W25T9-0/julialang/julia-master/julia-03bcfabc49/share/julia/stdlib/v1.12/LinearAlgebra/test/addmul.jl:217
2024-09-10 10:02:19 EDT	  Expression: ≈(collect(returned_mat), β * Cc, rtol = rtol)
2024-09-10 10:02:19 EDT	   Evaluated: BigFloat[NaN NaN NaN; NaN NaN NaN; NaN NaN NaN] ≈ BigFloat[-0.0 0.0 0.0; -0.0 0.0 -0.0; -0.0 -0.0 0.0] (rtol=1.490116119384765625e-08)
2024-09-10 10:02:19 EDT	
2024-09-10 10:02:19 EDT	Test Failed at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-R17H3W25T9.0/build/default-honeycrisp-R17H3W25T9-0/julialang/julia-master/julia-03bcfabc49/share/julia/stdlib/v1.12/LinearAlgebra/test/addmul.jl:217
2024-09-10 10:02:19 EDT	  Expression: ≈(collect(returned_mat), β * Cc, rtol = rtol)
2024-09-10 10:02:19 EDT	   Evaluated: BigFloat[NaN NaN NaN; NaN NaN NaN; NaN NaN NaN] ≈ BigFloat[-2.576573172191460272496978786761521099958325768979042052410957058249792605285215 0.0 0.0; -0.0 1.935391660145858756365593670084214505481215450048670570950510255331884803662584 -2.903087490218788417462668710368561458108727148374035142952801846346018334298358; -0.0 -0.4143112051851957548043292823690831771948847202910582995936218906491879846689642 0.3452593376543297720940845515373826726718286024674628091174318702619740598237634] (rtol=1.490116119384765625e-08)

[jishnub]

Oof this is the same failure that was seen in JuliaLang/julia#55450 (comment). I thought that this had been resolved mysteriously. Perhaps we should revert the PR for now

[giordano]

This is also failing on aarch64-linux: https://buildkite.com/julialang/julia-master/builds/39896#0191dce5-b3fb-4144-aa96-bc1ab80453c8. I guess it's an aarch64 thing?

[Zentrik]

Managed to hit this on a PGO+LTO build on x86, https://buildkite.com/julialang/julia-buildkite/builds/1763#0191e0bb-4870-45e4-85aa-72a3cc0af159

[Zentrik]

Also hit in the standard Linux i686 build, https://buildkite.com/julialang/julia-buildkite/builds/1763#0191e0bb-3e46-4b6b-8dc1-a1faed97b14a.

[giordano]

Windows x86_64, just to cover all families of operating systems: https://buildkite.com/julialang/julia-master/builds/39918#0191e1c8-2f2a-4da4-b1d0-4d5e3c2f4a3e

[JeffBezanson]

Why is this intermittent? Is something accessing uninitialized memory?

[giordano]

If anyone wants to investigate this, I could reproduce the issue (on 467ab852bb7) by running the tests multiple times with

for _ in 1:20; Base.runtests("LinearAlgebra/addmul"); end

Make 20 arbitrary large for higher chances to catch it. Each run of the whole test set shouldn't last long, a couple of minutes on the server I used.

However I don't have an RR trace, nor time to look into this.

[vtjnash]

You can set the random seed based on the CI failure, so this should fail every time (I tested):

../julia runtests.jl --seed=0x86c87ef1d9841135b01925d2cddb12f3 LinearAlgebra/addmul

[giordano]

And of course once you catch it you can reproduce it by using the same seed. For me

Base.runtests("LinearAlgebra/addmul"; seed=0x8fd036c919ca6b04d09997a08163bff1)

Always reproduces the test failure, not sure how good the seed it cross platform though. Edit: ah, apparently the seed it pretty good cross-platform, I can also reproduce with Base.runtests("LinearAlgebra/addmul"; seed=0x86c87ef1d9841135b01925d2cddb12f3) as suggested above!