[BACKEND] Add barrier after assert op to avoid race condition #5035
Conversation
|
After this commit (and @peterbell10's #5037, which turns debug=True back on for the test_side_effectful_reduction test), test_side_effectful_reduction and test_side_effectful_scan are hanging for me |
hmm I wonder if we end up with barrier in control flow? @peterbell10 would this op be lowered within control flow? |
|
ah, it makes sense that they are as we are effectively doing a sync inside an if statement lol |
yes, #4811 introduces predicated asserts |
| // an op that may trap if the assert condition is true. Since the tensor in | ||
| // those two operations may have different layout we need to make sure all | ||
| // the threads are done executing the assert before going to the next op. | ||
| barrier(); |
There was a problem hiding this comment.
Can we only insert the barrier for RankedTensor conditionals. That still fixes the issue, avoids an unnecessary barrier for scalar conds and fixes the hang with reductions all in one go.
There was a problem hiding this comment.
ah yeah sounds like an easy enough solution
…-lang#5035) Add a barrier to avoid a race condition in case an assert is followed by an op that may trap if the assert condition is true. Since the tensor in those two operations may have different layout we need to make sure all the threads are done executing the assert before going to the next op.
…-lang#5035) Add a barrier to avoid a race condition in case an assert is followed by an op that may trap if the assert condition is true. Since the tensor in those two operations may have different layout we need to make sure all the threads are done executing the assert before going to the next op.
4 participants
Add a barrier to avoid a race condition in case an assert is followed by an op that may trap if the assert condition is true. Since the tensor in those two operations may have different layout we need to make sure all the threads are done executing the assert before going to the next op.