Fory binary format - is it worthwhile compressing for transferring over the wire

[pjfanning]

I'm looking at create a Fory based serializer for Apache Pekko. Pekko supports comms between Pekko nodes running as a cluster. It already has Jackson support as well as using Protobuf for some internal classes. The serializers used are pluggable so a Fory based one could be an useful option.
The Jackson based serialized for Pekko supports compression (eg LZ4) of the JSON and CBOR formats.
Would it be useful to also support compression for Fory's data format?

[chaokunyang]

Fory use varint for innt32/64 compression, and has a highly-optimized format for struct meta compression. If reference tracking enableld, the shared refrence will be serialized only once like dict encoding. But for other cases, fory doesn't apply compression. Whether compression help will depend on the data distribution. If the object being serialized has some repeated pattern which are not captured by fory, the compression like LZ4 will help.

I think the such high-level compression can be a separate feature, and can be applied the all serialized binary in pekko, whether it is serialized by fory or not

[pjfanning]

Thanks @chaokunyang. We have the compression code in Pekko so I don't need to move it to Fory.
I'll allow users of my serializer to enable the LZ4 compression but it will be optional and I'll probably default to having it turned off.

[chaokunyang]

Great, default to having it turned off is better. Actually if the data is not persisted, bigger size won't be an issue in most cases. And fury binary size is smaller than JSON text protocol.

furyMediaContentBytes size 336
furyStructBytes size 90
jacksonMediaContentBytes size 664
jacksonStructBytes size 278

You can dive into https://github.com/chaokunyang/fury-benchmarks?tab=readme-ov-file#fury-vs-jackson for more details

FYI, We are releasing fory v0.11.2, I think this is a stable version to be integrated into pekko