Forest CAR roadmap

[aatifsyed]

• Uncompressed CARs can be used as a blockstore. feat: CAR-backed Blockstore #3085
  Forest scans a CAR file, building an index. It then scrolls to file offsets to read IPLD data.
• Manyframe compressed CARs can be used as a blockstore. feat: use zstd compressed cars as blockstores #3149
  Forest can produce manyframe-compressed CAR files with forest snapshot compress. They are backwards compatible with lotus.
  Forest scans manyframe-compressed CAR files, building an index. It scrolls to file offsets to read compressed zstd frames. It scrolls to offsets in zstd frames to read IPLD data.
• Custom car format with pre-indexing and compression. feat: read and write support for forest.car.zst files #3213
  Forest can produce .forest-car.car files with forest snapshot compress. They are backwards compatible with lotus.
  Forest loads an index from .forest.car files. It scrolls to fille offsets to read compressed zstd frames. It scrolls to offsets in zstd frames to read IPLD data.
• Remove manyframe compressed CAR support (superseded by forest-car.car format)
• All applicable cli frontends (can) use custom car format
• All applicable cli frontends (can) export custom car format
• Multiple CARs can be used as a blockstore
• CAR backed blockstores are multithreaded

  Blockstore::new(||File::open("<name>.forest-car.car"))

• CAR backed blockstores have intelligent inter-thread zstd frame caching.
  Might not even need this
• CARv2 support
• zstd_seekable support unlocks CARv2-compatible compression support

[lemmih]

Closed as complete. CARv2 support and zstd_seekable support may be opened as separate issues later on.