ptr_map currently has a separate codepath for Miri, which is always dicy since it means the codepath used 'for real' is not actually checked by Miri. (FWIW that codepath would work fine in Miri these days, but it would lead to warnings about int2ptr casts.) So it might make sense to track removing that temporary hack when it is no longer needed.
Judging from rust-lang/rust#96152, once Rust uses LLVM 15, the code used for Miri should also produce good runtime code, hopefully making the hack no longer needed.