don't use --config=mkl for TensorFlow 2.4+ #2583

Flamefire · 2021-09-24T14:43:28Z

(created using eb --new-pr)

fixes #2577, fixes easybuilders/easybuild-easyconfigs#14120

Flamefire · 2021-09-24T15:55:31Z

Test report by @Flamefire

Overview of tested easyconfigs (in order)

SUCCESS TensorFlow-2.4.1-foss-2020b.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
taurusi8028 - Linux centos linux 7.9.2009, x86_64, AMD EPYC 7352 24-Core Processor (zen2), Python 2.7.5
See https://gist.github.com/7695dcc6671f0e18cba427cd975be612 for a full test report.

Flamefire · 2021-09-24T16:14:54Z

Test report by @Flamefire

Overview of tested easyconfigs (in order)

FAIL (build issue) TensorFlow-2.5.0-fosscuda-2020b.eb (partial log available at https://gist.github.com/28739e94fd8a23a182d901b804213a66)

Build succeeded for 0 out of 1 (1 easyconfigs in total)
taurusi8029 - Linux centos linux 7.9.2009, x86_64, AMD EPYC 7352 24-Core Processor (zen2), Python 2.7.5
See https://gist.github.com/116468f218bd966c105b57c10100b2b5 for a full test report.

Flamefire · 2021-09-24T16:24:33Z

Test report by @Flamefire

Overview of tested easyconfigs (in order)

FAIL (build issue) TensorFlow-2.6.0-foss-2021a-CUDA-11.3.1.eb (partial log available at https://gist.github.com/9e9079f5bf8323999ead071516d555d9)

Build succeeded for 0 out of 1 (1 easyconfigs in total)
taurusi8012 - Linux centos linux 7.9.2009, x86_64, AMD EPYC 7352 24-Core Processor (zen2), Python 2.7.5
See https://gist.github.com/d1f6d771007592bad70ffb7f04174cf7 for a full test report.

Flamefire · 2021-09-24T18:03:29Z

Test report by @Flamefire

Overview of tested easyconfigs (in order)

SUCCESS TensorFlow-2.4.1-foss-2020b.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
taurusa7 - Linux centos linux 7.7.1908, x86_64, Intel(R) Xeon(R) CPU E5-2603 v4 @ 1.70GHz (broadwell), Python 2.7.5
See https://gist.github.com/7cb754a88b326bdc7bf9ca9fa278efd2 for a full test report.

Flamefire · 2021-09-24T23:32:32Z

Test report by @Flamefire

Overview of tested easyconfigs (in order)

SUCCESS TensorFlow-2.6.0-foss-2021a-CUDA-11.3.1.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
taurusa7 - Linux centos linux 7.7.1908, x86_64, Intel(R) Xeon(R) CPU E5-2603 v4 @ 1.70GHz (broadwell), Python 2.7.5
See https://gist.github.com/57cb680dc34cf746f429ed91ee3185a7 for a full test report.

Flamefire · 2021-09-25T04:34:12Z

Test report by @Flamefire

Overview of tested easyconfigs (in order)

SUCCESS h5py-3.1.0-fosscuda-2020b.eb
SUCCESS TensorFlow-2.5.0-fosscuda-2020b.eb

Build succeeded for 2 out of 2 (1 easyconfigs in total)
taurusa7 - Linux centos linux 7.7.1908, x86_64, Intel(R) Xeon(R) CPU E5-2603 v4 @ 1.70GHz (broadwell), Python 2.7.5
See https://gist.github.com/b0791e3a9b5d45149e3e50472c3f9e68 for a full test report.

Flamefire · 2021-09-25T08:42:15Z

Test report by @Flamefire

Overview of tested easyconfigs (in order)

SUCCESS UnZip-6.0-GCCcore-8.3.0.eb
SUCCESS TensorFlow-2.4.1-fosscuda-2019b-Python-3.7.4.eb

Build succeeded for 2 out of 2 (1 easyconfigs in total)
taurusa7 - Linux centos linux 7.7.1908, x86_64, Intel(R) Xeon(R) CPU E5-2603 v4 @ 1.70GHz (broadwell), Python 2.7.5
See https://gist.github.com/161ab61dee8f5bf2204c7bedce4551b3 for a full test report.

easybuild/easyblocks/t/tensorflow.py

Flamefire · 2021-09-29T11:47:28Z

More or less blocked by tensorflow/tensorflow#52151

boegel · 2021-10-13T21:12:52Z

I consider this change important enough to not let it be blocked by the failing mkl_fused_batch_norm_op_test, we can (temporarily) add that as an ignored test, so we can go ahead and merge this PR.

Especially since this apparently fixes two performance issues: the threading oversubscription for CPU-only TensorFlow reported in #2577, but also tf.matmul preferring MKL on CPU over GPU issue reported in easybuilders/easybuild-easyconfigs#14120...

boegel · 2021-10-14T10:18:32Z

Test report by @boegel

Overview of tested easyconfigs (in order)

SUCCESS TensorFlow-2.5.0-foss-2020b.eb
SUCCESS TensorFlow-2.6.0-foss-2021a.eb
SUCCESS TensorFlow-2.4.1-foss-2020b.eb

Build succeeded for 3 out of 3 (3 easyconfigs in total)
node3139.skitty.os - Linux centos linux 7.9.2009, x86_64, Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz (skylake_avx512), Python 3.6.8
See https://gist.github.com/294c57503da360ddded7dfc4d42c75d4 for a full test report.

boegel · 2021-10-14T11:22:44Z

Test report by @boegel

Overview of tested easyconfigs (in order)

FAIL (build issue) TensorFlow-2.5.0-foss-2020b.eb (partial log available at https://gist.github.com/9924c02c66171de09050070270b15dd3)
FAIL (build issue) TensorFlow-2.6.0-foss-2021a.eb (partial log available at https://gist.github.com/fb1d1f2ea4150497d7775a0bc1eae5fe)
SUCCESS TensorFlow-2.4.1-foss-2020b.eb

Build succeeded for 1 out of 3 (3 easyconfigs in total)
node3520.doduo.os - Linux RHEL 8.2, x86_64, AMD EPYC 7552 48-Core Processor (zen2), Python 3.6.8
See https://gist.github.com/3469ae5c80fe0b2c05f376841e47dee0 for a full test report.

boegel · 2021-10-14T11:58:37Z

Failing tests on systems with AMD CPUs due to tensorflow/tensorflow#52151, I'll open a PR to filter out those that broken test so we can proceed here...

akesandgren · 2021-10-14T14:29:06Z

I get these two failing for TensorFlow-2.6.0-foss-2021a-CUDA-11.3.1.eb --include-easyblocks-from-pr 2583
on our broadwell with K80, twice in a row now.
//tensorflow/core/kernels:fused_batch_norm_ex_op_test_gpu
//tensorflow/core/kernels/mlir_generated:gpu_unary_ops_test_gpu

Trying without this PR...

And my original production build of it on that system combo did not fail and that was without this PR.

Weird, I get that problem even without this PR now.

boegel · 2021-10-14T17:29:06Z

@akesandgren I'd say let's open an issue on that?

akesandgren · 2021-10-14T17:40:14Z

I will once I've figured out a bit more on what's happening... I think my container env wasn't fully correct for TF-CUDA when I initially built it, but now it is. It was previously lacking nvidia-smi and thus decided to skip the GPU tests. Something I only discovered today...

And at least one of the failing tests may be due to too little memory on the K80, or that multiple tests are running at the same time and stealing memory from each other...

boegel · 2021-10-15T10:31:25Z

Test report by @boegel

Overview of tested easyconfigs (in order)

SUCCESS TensorFlow-2.4.1-foss-2020b.eb
SUCCESS TensorFlow-2.4.1-fosscuda-2019b-Python-3.7.4.eb
SUCCESS TensorFlow-2.4.1-fosscuda-2020b.eb

Build succeeded for 3 out of 3 (3 easyconfigs in total)
node3309.joltik.os - Linux centos linux 7.9.2009, x86_64, Intel(R) Xeon(R) Gold 6242 CPU @ 2.80GHz (cascadelake), 1 x NVIDIA NVIDIA Tesla V100-SXM2-32GB, 465.19.01, Python 3.6.8
See https://gist.github.com/cabb68c4f78eda268517b8f6709c5fb2 for a full test report.

boegel · 2021-10-15T17:47:11Z

Test report by @boegel

Overview of tested easyconfigs (in order)

SUCCESS TensorFlow-2.4.1-foss-2020b.eb
SUCCESS TensorFlow-2.4.1-fosscuda-2019b-Python-3.7.4.eb
SUCCESS TensorFlow-2.4.1-fosscuda-2020b.eb

Build succeeded for 3 out of 3 (3 easyconfigs in total)
node3309.joltik.os - Linux centos linux 7.9.2009, x86_64, Intel(R) Xeon(R) Gold 6242 CPU @ 2.80GHz (cascadelake), 1 x NVIDIA NVIDIA Tesla V100-SXM2-32GB, 465.19.01, Python 3.6.8
See https://gist.github.com/02ba89107f021cc2ce39e6c8a56ffe43 for a full test report.

boegel · 2021-10-15T18:59:39Z

@akesandgren Any additional updates?

I would really like to move forward with this (but we also need easybuilders/easybuild-easyconfigs#14151 and easybuilders/easybuild-easyconfigs#14153 merged, first ideally).

akesandgren · 2021-10-15T19:27:05Z

Yeah, most likely caused by K80's having too little memory. Works fine on our V100's so I'll just drop those two tests in our hooks when on K80's

akesandgren · 2021-10-18T06:15:08Z

So in case I was unclear, the problem only appears on K80 and appears regardless of this PRs existance.
Move forward...

boegel · 2021-10-26T19:03:39Z

With both easybuilders/easybuild-easyconfigs#14151 and easybuilders/easybuild-easyconfigs#14153 merged, I don't see any reason to hold this back any further...

…ions < 2.4.0

boegel · 2021-10-27T21:25:07Z

Test report by @boegel

Overview of tested easyconfigs (in order)

SUCCESS TensorFlow-2.2.3-foss-2020b.eb
SUCCESS TensorFlow-2.6.0-foss-2021a.eb

Build succeeded for 2 out of 2 (2 easyconfigs in total)
node2635.swalot.os - Linux centos linux 7.9.2009, x86_64, Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz (haswell), Python 3.6.8
See https://gist.github.com/45f2ef7fc6dacfd098afaea2543f1cbe for a full test report.

Micket · 2021-10-27T21:40:57Z

Is the subject still accurate here? Because we should do this for all 2.x versions in the CUDA builds.

boegel · 2021-10-27T22:32:41Z

Test report by @boegel

Overview of tested easyconfigs (in order)

SUCCESS TensorFlow-2.4.1-foss-2020b.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
node3151.skitty.os - Linux centos linux 7.9.2009, x86_64, Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz (skylake_avx512), Python 3.6.8
See https://gist.github.com/96cde227f8a5f06b42f6c1097bb0c98f for a full test report.

Don't use --config=mkl for TF 2.4+

738b703

Add author

ba9b449

boegel added change performance labels Sep 29, 2021

boegel added this to the next release (4.5.0?) milestone Sep 29, 2021

boegel changed the title ~~Don't use --config=mkl for TF 2.4+~~ don't use --config=mkl for TensorFlow 2.4+ Sep 29, 2021

boegel requested changes Sep 29, 2021

View reviewed changes

easybuild/easyblocks/t/tensorflow.py Outdated Show resolved Hide resolved

akesandgren mentioned this pull request Oct 13, 2021

TensorFlow tf.matmul ends up using CPU backend for 32bit floats easybuilders/easybuild-easyconfigs#14120

Closed

boegel added 2 commits October 27, 2021 20:44

clarify comment w.r.t. auto-enabling --config=mkl for TensorFlow vers…

572197b

…ions < 2.4.0

restructure handling of with_mkl_dnn logic in TensorFlow easyblock

517129b

boegel force-pushed the 20210924164325_new_pr_tensorflow branch from 11264eb to 517129b Compare October 27, 2021 18:57

boegel approved these changes Oct 27, 2021

View reviewed changes

boegel merged commit 2b5134b into easybuilders:develop Oct 27, 2021

Flamefire deleted the 20210924164325_new_pr_tensorflow branch October 29, 2021 09:02

don't use --config=mkl for TensorFlow 2.4+ #2583

don't use --config=mkl for TensorFlow 2.4+ #2583

Uh oh!

Conversation

Flamefire commented Sep 24, 2021 • edited by boegel Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Flamefire commented Sep 24, 2021

Overview of tested easyconfigs (in order)

Uh oh!

Flamefire commented Sep 24, 2021

Overview of tested easyconfigs (in order)

Uh oh!

Flamefire commented Sep 24, 2021

Overview of tested easyconfigs (in order)

Uh oh!

Flamefire commented Sep 24, 2021

Overview of tested easyconfigs (in order)

Uh oh!

Flamefire commented Sep 24, 2021

Overview of tested easyconfigs (in order)

Uh oh!

Flamefire commented Sep 25, 2021

Overview of tested easyconfigs (in order)

Uh oh!

Flamefire commented Sep 25, 2021

Overview of tested easyconfigs (in order)

Uh oh!

Uh oh!

Flamefire commented Sep 29, 2021

Uh oh!

boegel commented Oct 13, 2021

Uh oh!

boegel commented Oct 14, 2021

Overview of tested easyconfigs (in order)

Uh oh!

boegel commented Oct 14, 2021

Overview of tested easyconfigs (in order)

Uh oh!

boegel commented Oct 14, 2021

Uh oh!

akesandgren commented Oct 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

boegel commented Oct 14, 2021

Uh oh!

akesandgren commented Oct 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

boegel commented Oct 15, 2021

Overview of tested easyconfigs (in order)

Uh oh!

boegel commented Oct 15, 2021

Overview of tested easyconfigs (in order)

Uh oh!

boegel commented Oct 15, 2021

Uh oh!

akesandgren commented Oct 15, 2021

Uh oh!

akesandgren commented Oct 18, 2021

Uh oh!

boegel commented Oct 26, 2021

Uh oh!

boegel commented Oct 27, 2021

Overview of tested easyconfigs (in order)

Uh oh!

Micket commented Oct 27, 2021

Uh oh!

boegel commented Oct 27, 2021

Overview of tested easyconfigs (in order)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Flamefire commented Sep 24, 2021 •

edited by boegel

Loading

akesandgren commented Oct 14, 2021 •

edited

Loading

akesandgren commented Oct 14, 2021 •

edited

Loading