This is especially true in sliced [distinct] facets where you have n time periods x m slices. And to make it worse, the BytesRef objects in the keys don't get reused, each one is a different object.
We could invert the collector logic and collect the slice labels first, and then the time periods. Or have a separate lookup table of labels, each of which is just indexed by an int. Like ordinals, but across the whole query.