Xbitinfo analyses datasets based on their bitwise real information content and applies lossy compression accordingly. Being based on xarray it integrates seamlessly into common research workflows. Additional convienient functions help users to visualize the bitwise information content and to make informed decisions on the real information threshold that is subsequently used as the preserved precision during the compression.
Xbitinfo works in four steps:
- Analyse the bitwise information content of a dataset
- Decide on a threshold of real information to preserve (e.g. 99%)
- Reduce the precision of the dataset accordingly (bitrounding)
- Apply lossless compression (e.g. zlib, blosc, zstd) and store the dataset
To fullfill these steps, Xbitinfo relies on:
xarrayfor handling multi-dimensional arrays and file formats (e.g. netcdf, zarr, hdf5, grib)daskfor scaling to large datasetsBitInformation.jl(optional) for computing the bitwise information content based on the original Julia implementation. Continuous integration tests ensure however that the python-implementation shipped with xbitinfo result in identical results.numcodecsfor a wide-range of lossless compression algorithms
Overall, the package presents a pipeline to compress (climate) datasets based on the real information content.
Xbitinfo is packaged and distributed both via PyPI and conda-forge and can be installed via pip or conda respectively.
Depending on whether one wants to use the Julia implementation of the bitinformation algorithm (BitInformation.jl) or the native python implementation shipped with xbitinfo,
one might choose one installation option over the other.
pip install xbitinfo
or
conda install -c conda-forge xbitinfo-python
conda install -c conda-forge xbitinfo
or
pip install xbitinfo # julia needs to be installed manually
import xarray as xr
import xbitinfo as xb
example_dataset = 'eraint_uvz'
ds = xr.tutorial.load_dataset(example_dataset)
# Step 1: analyze bitwise information content
bitinfo = xb.get_bitinformation(ds, dim="longitude")
# Step 2: decide on a threshold of real information to preserve (e.g. 99%)
keepbits = xb.get_keepbits(bitinfo, inflevel=0.99) # get number of mantissa bits to keep for 99% real information
# Step 3: reduce the precision of the dataset accordingly (bitrounding)
ds_bitrounded = xb.xr_bitround(ds, keepbits) # bitrounding keeping only keepbits mantissa bits
# Step 4: apply lossless compression (e.g. zlib, blosc, zstd) and store the dataset
ds_bitrounded.to_compressed_netcdf(outpath)Klöwer, M., Razinger, M., Dominguez, J. J., Düben, P. D., & Palmer, T. N. (2021). Compressing atmospheric data into its real information content. Nature Computational Science, 1(11), 713–724. doi: 10/gnm4jj