Enable adaptive mesh support on libMesh tallies

[nuclearkevin]

Description

This PR allows for the use of adaptive meshes (which have an adaptive mesh refinement hierarchy) in unstructured mesh tallies which use a LibMesh. Adaptive meshes (unlike libMesh meshes which set allow_renumbering = false) do not guarantee that active elements (which we want to tally on) are contiguous in memory relative to their DoF ids, which results in errors when OpenMC computes bin indices as bin are defined over [0, num_active_elem]. This results in scrambled tallies for libMesh meshes which have an adaptivity hierarchy. An additional issue is that the libMesh::EquationSystems object added by OpenMC to enable LibMesh exodus output reacts poorly to adaptive meshes, throwing errors when the mesh is refined/coarsened or dumped to exodus after an OpenMC solve.

To fix the first issue, an indirection layer is added which maps between element DoF ids and bin indices. This is generated when LibMesh is initialized based on a constant adaptivity flag (adaptive_). The indirection layer is used in get_bin_from_element(...) and get_element_from_bin(...) only when adaptive_ = true. This should result in no performance changes for existing users that use libMesh-based tallies without adaptivity, though it does result in a small (but noticeable) decrease in tally performance on adaptive libMesh meshes compared to creating a deep copy of the mesh which only contains active elements (due to cache misses).

The second issue is resolved by constructing and setting up the libMesh::EquationSystems class in LibMesh::add_score(...) if it hasn't been initialized.

I've tested these changes in Cardinal to make sure adaptive mesh tallies work, and the results generated are equivalent to the previous approach we took. Regression tests for libMesh tallies also pass on my machine, so OpenMC users should be unaffected by this fix for adaptive meshes.

Closes #3182

Checklist

• I have performed a self-review of my own code
• I have run clang-format (version 15) on any C++ source files (if applicable)
• I have followed the style guidelines for Python source files (if applicable)
• I have made corresponding changes to the documentation (if applicable)
• I have added tests that prove my fix is effective or that my feature works (if applicable)

[pshriwise]

Thanks @nuclearkevin!! I'm glad it didn't take too much to make this happen. Just a few minor thoughts/comments from me here.

Also, is there an easy way to test this? Not a blocking issue for me here, but it might be a nice follow-on PR at some point.

[paulromano]

A few small comments/questions. Thanks @nuclearkevin!

[nuclearkevin]

Also, is there an easy way to test this? Not a blocking issue for me here, but it might be a nice follow-on PR at some point.

@pshriwise The easiest way to test these changes outside of Cardinal would be to write a C++ test where a mesh is loaded, refined once uniformly (using the adaptivity system) and then used for tallying. I don't know if there's an easy way to test this in the Python API since the LibMesh class isn't setup to perform mesh restarts (which is how you load a previously adapted mesh) with a given Exodus mesh filename.

[paulromano]

Thanks for the updates!

[pshriwise]

Thanks @nuclearkevin! Looks good to me!

[pshriwise]

Also, is there an easy way to test this? Not a blocking issue for me here, but it might be a nice follow-on PR at some point.

@pshriwise The easiest way to test these changes outside of Cardinal would be to write a C++ test where a mesh is loaded, refined once uniformly (using the adaptivity system) and then used for tallying. I don't know if there's an easy way to test this in the Python API since the LibMesh class isn't setup to perform mesh restarts (which is how you load a previously adapted mesh) with a given Exodus mesh filename.

That's about what I figured. Testing is probably best left to the AMR repo you're working on for now. Thanks!