ccl/backupccl: TestBackupRestoreJobTagAndLabel failed

[cockroach-teamcity]

ccl/backupccl.TestBackupRestoreJobTagAndLabel failed on master @ cedd409f2115655efe2c529f6052f3056673b277:

=== RUN   TestBackupRestoreJobTagAndLabel
initialized metamorphic constant "export_request_verbose_tracing" with value true
I231102 08:58:45.497112 1 (gostd) rand.go:250  [-] 1  random seed: -1746867873555533214
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestBackupRestoreJobTagAndLabel2368049754
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:152: automatically injected virtual cluster under test; see comment at top of test_server_shim.go for details.
    backup_test.go:293: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/ccl/backupccl/backup_test.go:293
        	Error:      	Should be true
        	Test:       	TestBackupRestoreJobTagAndLabel
    testutils.go:289: no Invalid Descriptors
    panic.go:523: -- test log scope end --
test logs left over in: outputs.zip/logTestBackupRestoreJobTagAndLabel2368049754
--- FAIL: TestBackupRestoreJobTagAndLabel (32.54s)

Parameters: attempt=1 , run=3 , shard=20

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-33091

[herkolategan]

I'm a little suspicious that the introduction of a shared process test tenant caused this failure #112822, although I haven't been able to reproduce it locally under stress with a shared process test tenant explicitly enabled, it also failed on a backport https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_BazelEssentialCi/12469000.
Added an issue to have better logging for what process mode the default test virtual cluster is running as: #113668

[cockroach-teamcity]

ccl/backupccl.TestBackupRestoreJobTagAndLabel failed on master @ 694861a16c8d72a52ac059ef82cf2763ca4406b0:

=== RUN   TestBackupRestoreJobTagAndLabel
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestBackupRestoreJobTagAndLabel3453187597
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:152: automatically injected virtual cluster under test; see comment at top of test_server_shim.go for details.
    backup_test.go:293: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/ccl/backupccl/backup_test.go:293
        	Error:      	Should be true
        	Test:       	TestBackupRestoreJobTagAndLabel
    testutils.go:289: no Invalid Descriptors
    panic.go:523: -- test log scope end --
test logs left over in: outputs.zip/logTestBackupRestoreJobTagAndLabel3453187597
--- FAIL: TestBackupRestoreJobTagAndLabel (18.16s)

Parameters: attempt=1 , run=15 , shard=20

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

[herkolategan]

PR #113679 should be merged shortly and give an indication under what virtual cluster process mode this test fails predominantly.

[cockroach-teamcity]

ccl/backupccl.TestBackupRestoreJobTagAndLabel failed on master @ 90a1f850ccfc9a123c9dcaba317296151ae32a8c:

=== RUN   TestBackupRestoreJobTagAndLabel
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestBackupRestoreJobTagAndLabel3630953856
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:165: automatically injected a shared process virtual cluster under test; see comment at top of test_server_shim.go for details.
    backup_test.go:293: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/ccl/backupccl/backup_test.go:293
        	Error:      	Should be true
        	Test:       	TestBackupRestoreJobTagAndLabel
    testutils.go:289: no Invalid Descriptors
    panic.go:523: -- test log scope end --
test logs left over in: outputs.zip/logTestBackupRestoreJobTagAndLabel3630953856
--- FAIL: TestBackupRestoreJobTagAndLabel (21.09s)

Parameters: attempt=1 , run=22 , shard=20

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

[cockroach-teamcity]

ccl/backupccl.TestBackupRestoreJobTagAndLabel failed on master @ d131065cc4ebb176c027892cac5cdb96cd69c214:

=== RUN   TestBackupRestoreJobTagAndLabel
I231106 08:56:13.211727 1 (gostd) rand.go:250  [-] 1  random seed: -2401888641699808069
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestBackupRestoreJobTagAndLabel2120868728
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:165: automatically injected a shared process virtual cluster under test; see comment at top of test_server_shim.go for details.
    backup_test.go:293: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/ccl/backupccl/backup_test.go:293
        	Error:      	Should be true
        	Test:       	TestBackupRestoreJobTagAndLabel
    testutils.go:289: no Invalid Descriptors
    panic.go:523: -- test log scope end --
test logs left over in: outputs.zip/logTestBackupRestoreJobTagAndLabel2120868728
--- FAIL: TestBackupRestoreJobTagAndLabel (18.98s)

Parameters: attempt=1 , run=24 , shard=20

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

[stevendanna]

New logs confirm that this appears to be a shared process issue.