Skip to content

skip mkl_fused_batch_norm_op_test test for TensorFlow 2.5.0 (broken on AMD EPYC systems) #14151

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Oct 21, 2021
Merged

skip mkl_fused_batch_norm_op_test test for TensorFlow 2.5.0 (broken on AMD EPYC systems) #14151

merged 1 commit into from
Oct 21, 2021

Conversation

@boegel
Copy link
Member

@boegel boegel commented Oct 14, 2021 •
edited
Loading

(created using eb --new-pr)

cfr. easybuilders/easybuild-easyblocks#2583

@boegel
Copy link
Member Author

boegel commented Oct 14, 2021

@boegelbot please test @ generoso
CORE_CNT=16

@boegel boegel added this to the next release (4.5.0?) milestone Oct 14, 2021
@boegelbot
Copy link
Collaborator

boegelbot commented Oct 14, 2021

@boegel: Request for testing this PR well received on login1

PR test command 'EB_PR=14151 EB_ARGS= /opt/software/slurm/bin/sbatch --job-name test_PR_14151 --ntasks="16" ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 7082

Test results coming soon (I hope)...

- notification for comment with ID 943324537 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegel boegel added the bug fix label Oct 14, 2021
@boegel boegel mentioned this pull request Oct 14, 2021
Merged
@boegel
Copy link
Member Author

boegel commented Oct 14, 2021

Test report by @boegel
SUCCESS
Build succeeded for 3 out of 3 (3 easyconfigs in total)
node3302.joltik.os - Linux centos linux 7.9.2009, x86_64, Intel(R) Xeon(R) Gold 6242 CPU @ 2.80GHz (cascadelake), 4 x NVIDIA NVIDIA Tesla V100-SXM2-32GB, 465.19.01, Python 3.6.8
See https://gist.github.com/931bf7abaa198132a1b9226a5910f243 for a full test report.

@boegel
Copy link
Member Author

boegel commented Oct 14, 2021

Test report by @boegel
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#2583
SUCCESS
Build succeeded for 3 out of 3 (3 easyconfigs in total)
node3520.doduo.os - Linux RHEL 8.2, x86_64, AMD EPYC 7552 48-Core Processor (zen2), Python 3.6.8
See https://gist.github.com/faa1b5eb89cfe4670b8d5ed765bb10c1 for a full test report.

@boegelbot
Copy link
Collaborator

boegelbot commented Oct 14, 2021

Test report by @boegelbot
SUCCESS
Build succeeded for 3 out of 3 (3 easyconfigs in total)
cnx1 - Linux rocky linux 8.4, x86_64, Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz (haswell), Python 3.6.8
See https://gist.github.com/15d456deeb3a5eb0630c154c3e85c862 for a full test report.

@boegel
Copy link
Member Author

boegel commented Oct 15, 2021 •
edited
Loading

Test report by @boegel
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#2583
FAILED
Build succeeded for 2 out of 3 (3 easyconfigs in total)
node3900.accelgor.os - Linux RHEL 8.4, x86_64, AMD EPYC 7413 24-Core Processor (zen2), 1 x NVIDIA NVIDIA A100-SXM-80GB, 470.57.02, Python 3.6.8
See https://gist.github.com/9ce26f920962baee41aecfb6a1ce4580 for a full test report.

edit: test for `` failed because 8.0 CUDA compute capability (for A100) is not supported yet by CUDA 10.1.243, so good to ignore

nvcc fatal   : Unsupported gpu architecture 'compute_80'

OK to ignore, not related to changes in this PR or in easybuilders/easybuild-easyblocks#2583...

@boegel
Copy link
Member Author

boegel commented Oct 15, 2021

Test report by @boegel
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#2583
SUCCESS
Build succeeded for 3 out of 3 (3 easyconfigs in total)
node3309.joltik.os - Linux centos linux 7.9.2009, x86_64, Intel(R) Xeon(R) Gold 6242 CPU @ 2.80GHz (cascadelake), 1 x NVIDIA NVIDIA Tesla V100-SXM2-32GB, 465.19.01, Python 3.6.8
See https://gist.github.com/9d06455e3306adf24baf66c5ffc51d13 for a full test report.

@boegel
Copy link
Member Author

boegel commented Oct 15, 2021

Test report by @boegel
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#2583
SUCCESS
Build succeeded for 12 out of 12 (3 easyconfigs in total)
node2618.swalot.os - Linux centos linux 7.9.2009, x86_64, Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz (haswell), Python 3.6.8
See https://gist.github.com/12ffee2cd34bdb30842eef1d87863fd5 for a full test report.

@lexming
Copy link
Contributor

lexming commented Oct 15, 2021

Test report by @lexming
FAILED
Build succeeded for 83 out of 88 (3 easyconfigs in total)
node402.hydra.os - Linux centos linux 7.9.2009, x86_64, AMD EPYC 7282 16-Core Processor, 1 x NVIDIA A100-PCIE-40GB, 460.73.01, Python 2.7.5
See https://gist.github.com/e416c200107d35eb1a674e377771ca7a for a full test report.

@boegel boegel mentioned this pull request Oct 15, 2021
Merged
@boegel
Copy link
Member Author

boegel commented Oct 15, 2021

@lexming Installation of Java/1.8 failed due to missing source file...

@lexming
Copy link
Contributor

lexming commented Oct 17, 2021

Test report by @lexming
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#2583
FAILED
Build succeeded for 5 out of 7 (3 easyconfigs in total)
node401.hydra.os - Linux centos linux 7.9.2009, x86_64, AMD EPYC 7282 16-Core Processor, 1 x NVIDIA A100-PCIE-40GB, 460.73.01, Python 2.7.5
See https://gist.github.com/cf7c5b8c951170cde1c79a640498e934 for a full test report.

@boegel
Copy link
Member Author

boegel commented Oct 17, 2021

@lexming Now test failed due to missing cuDNN sources :)

@boegel
Copy link
Member Author

boegel commented Oct 20, 2021

@lexming Ping on getting this merged?

@lexming
Copy link
Contributor

lexming commented Oct 21, 2021

Test report by @lexming
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#2583
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
node401.hydra.os - Linux centos linux 7.9.2009, x86_64, AMD EPYC 7282 16-Core Processor, 1 x NVIDIA A100-PCIE-40GB, 460.73.01, Python 2.7.5
See https://gist.github.com/2b72da0a9f4ea29ef64dc7062379a3b2 for a full test report.

lexming
lexming approved these changes Oct 21, 2021
View reviewed changes
Copy link
Contributor

@lexming lexming left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll drop my tests for 2019b, setting up that toolchain in our EPYC needs more work.

LGTM

@lexming
Copy link
Contributor

lexming commented Oct 21, 2021

Going in, thanks @boegel!

@lexming lexming merged commit 911a707 into easybuilders:develop Oct 21, 2021
@lexming
Copy link
Contributor

lexming commented Oct 21, 2021

Test report by @lexming
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#2583
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
node401.hydra.os - Linux centos linux 7.9.2009, x86_64, AMD EPYC 7282 16-Core Processor, 1 x NVIDIA A100-PCIE-40GB, 460.73.01, Python 2.7.5
See https://gist.github.com/b5d1b0168f98feb4a0ad0dc9c095f2bc for a full test report.

@boegel boegel deleted the 20211014143054_new_pr_TensorFlow250 branch October 22, 2021 06:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lexming lexming lexming approved these changes

Assignees

No one assigned

Labels

bug fix

Projects

None yet

Milestone

4.5.0

Development

Successfully merging this pull request may close these issues.

3 participants

@boegel @boegelbot @lexming