I really thought we had this written down at some point as using a cbor big int, but i can't seem to find it.
In any case, We should use cbor big ints here (its an extension). It costs us one extra byte for the tag, but means other CBOR parsers can more easily understand our datastructures.