IMA error in nightlies

[msaroufim]

This passed our regression tests #373 but then when we merged into main caused all sorts of IMA issues

https://github.com/pytorch/ao/actions/runs/9528682196/job/26266630042#step:12:7176

Testing to see if I can repro here #377 which so far I can't xD. Meaning this an upstream failure from torch since a revert also has no impact #378

[msaroufim]

I could no longer repro this