Add support for 7z/LZMA1/LZMA2 to System.IO.Compression

[joshfree]

http://7-zip.org/sdk.html

[GSPP]

LZMA is pretty much the state of the art general purpose compression algorithm. It pushes the Pareto Frontier outwards in quite a large range.

If there is interest in adding more algorithms there would be room for a super fast one that compresses in the range 10-300MB/s. Such algorithms exist and can be found using any search engine.

[ericstj]

I've already ported the C# impl for internal use in CLI: https://github.com/dotnet/cli/tree/rel/1.0.0/src/Microsoft.DotNet.Archive/LZMA

The work to do to pull this in is the following:

1. Better API design to follow wrapping stream design used by DeflateStream
2. Perf

For perf we could do a similar thing to what was done for ZLIB where we just pick up the native impl and add that to CLRCompression.

@GSPP if you have requests for other algos please file a separate issue.

[ianhays]

Something we should consider before we start working on adding support for Tar/Deflate/7z/LZMA/LZMA2/LZ4/BZ2/etc is how we want to design the APIs for the classes to keep them relatively similar without cluttering up Sys.IO.Compression. Options there off the top of my head:

• Our current design is to give each compression method its own class (DeflateStream/GZipStream). The most consistent option would be to continue this trend for new algorithms e.g. LZMAStream/ZLibStream/BZ2Stream
• Have a CompressionStream that takes an optional CompressionType enum (Deflate/Zlib/LZMA/etc) as a constructor param
• Some blend of the above two.

I imagine we'll want to go with the first option so we can provide fine-grained control of the algorithm variables, but its worth considering nonetheless.

[ericstj]

First option is what I was thinking. I think we can look at higher-level simple to use archive APIs that might allow for selection of the compression algorithm for an archive where the caller doesn't necessarily need to know about the specific algo type or params but we'd also want to support fine-grained composition of these things to allow for maximum flexibility/tuning. Also for reading APIs should be able to "sniff" a data stream to determine what it is (if the format allows for that deterministically) without forcing the user to tell us via type instantiation.

[ianhays]

I think we can look at higher-level simple to use archive APIs that might allow for selection of the compression algorithm for an archive where the caller doesn't necessarily need to know about the specific algo type or params

I think this would be the most happy compromise between full functionality and ease of use. I'm picturing a bunch of algorithm-specific streams (DeflateStream/GZipStream/LZMAStream) as well as one high-level stream (CompressionStream) that has simple write/read functionality using the defaults of the chosen compression type e.g.

    public enum Compressiontype
    {
        Deflate = 0,
        GZip = 1,
        ZLib = 2,
        LZMA = 3
    }
    public partial class CompressionStream : System.IO.Stream
    {
        public CompressionStream(System.IO.Stream stream, System.IO.Compression.CompressionMode mode) { } // default Deflate for CompressionMode.Compress. CompressionMode.Decompress attempts to detect type from header and defaults to Deflate if it can not be determined.
        public CompressionStream(System.IO.Stream stream, System.IO.Compression.CompressionMode mode, System.IO.Compression.CompressionType type) { } // no auto-header detection for Decompression.
        public CompressionStream(System.IO.Stream stream, System.IO.Compression.CompressionMode mode, bool leaveOpen) { }
        public CompressionStream(System.IO.Stream stream, System.IO.Compression.CompressionMode mode, System.IO.Compression.CompressionType type, bool leaveOpen) { }
        ...
    }

Also for reading APIs should be able to "sniff" a data stream to determine what it is (if the format allows for that deterministically) without forcing the user to tell us via type instantiation.

That would be ideal. The algorithm-specific APIs (e.g. DeflateStream) should require the given stream adhere to the format, but the higher-level API should attempt to detect the header type and default (probably to Deflate) it it can't be determined.

[ericstj]

Seems reasonable so long as it doesn't hurt perf (eg: cause an additional buffer copy). I'd say that's a separate feature, but definitely one worth looking at.

[ianhays]

I'd say that's a separate feature

Agreed. I opened https://github.com/dotnet/corefx/issues/9709 to track the discussion on that.

[GSPP]

I think the enum approach suffers from non-extensibility. Wouldn't it be better to use one derived type for each algorithm? Algorithms have different parameters and sometimes different capabilities. For example, Zip has the capability to add named entries (files). Many compression libraries offer a ZipStream that behaves like a normal stream plus the ability to define files.

Proposal: Make CompressionStream an abstract base class that anyone including user code can derive from. That way there is a standardized infrastructure for the compression mode and leaving the stream open. Also, maybe we can have standardized compression levels 1-9 that each derived algorithm interprets and uses to set it's settings.

Also, I do not see in what way a class-based approach (as opposed to an enum-based approach) would be inferior(?). Seems equal or better in every regard (apart from format detection).

I advise against format detection when decompressing. The format should be statically known and usually is. Some formats are headerless which breaks the scheme.

[ericstj]

I think the enum approach suffers from non-extensibility

Good point.

I don't see how CompressionStream really solves this. We already have the Stream abstraction and all the things we're talking about controlling are on construction which isn't part of the abstraction. The only things we might expose on the stream are getters for construction parameters but I don't know if that provides enough value to justify another type in the heirarchy.

class-based approach (as opposed to an enum-based approach) would be inferior(?)

I don't think we disagree. @ianhays was talking about some convenience construction pattern that would use the public Stream derived types behind the scenes. One option for the construction pattern would be enums, but as you mention these don't version well. Perhaps we have a type CompressionParameters that has levels like you suggest and compression Stream implementations can translate those to appropriate algo-specific params.

Let's use https://github.com/dotnet/corefx/issues/9709 to track coming up with a convenience construction pattern and use this issue for describing the new streams we intend to add.

[ianhays]

Back to the LZMA discussion:

Not sure how we're going to want to do this since the LZMA SDK includes implementations in C as well as C#. Some options and related thoughts:

• Include LZMA C code in clrcompression and add a stream wrapper in Sys.IO.Compression like we did for ZLib
  • Isn't NetStandard.Library friendly (https://github.com/dotnet/corefx/issues/6602)
  • Probably won't work on UWP since it imports a ton of stuff from kernel32 that I doubt is supported.
  • Probably faster than managed, but perf testing that will require a lot of setup to get both methods stood up
  • Keeps our code nicely separate from LZMA code and makes upgrading easy
  • Works for LZMA/LZMA2/XZ/7z
• Add the LZMA C# code to Sys.IO.Compression
  • Platform agnostic
  • Works only for LZMA (though could probably make it work for the others without too much trouble - would need to investigate further
  • We would become custodians of this code as it intermingles with our other compression code.
  • Easiest, fastest way to get LZMA support into Sys.IO.Compression
• Build the C# LZMA as a separate assembly and have Sys.IO.Compression reference it.
  • Same as above, but keeps the LZMA sdk separate from our junk to simplify upgrading/updating in the future.

[GSPP]

Ideally, it would be a native code solution where possible with a managed fallback. How much slower is the C# version? My guess would be 2-3x. C compilers excel at this kind of code.

[ianhays]

Alright, I've got native and managed versions working in Sys.IO.Compression on Windows so we can get some early perf results. Properties are set to the defaults for both.

As with ZLib, the lzma native implementation is significantly faster. Wall-clock time for 10 compress/decompress of the files in the Canterbury Corpus:

CompressionMode	Managed/Native	FileName	Elapsed Time	Output File Size
Compress	Managed	alice29.txt	00:00:02.5441260	48452
Compress	Native	alice29.txt	00:00:01.5311828	48466
Decompress	Managed	alice29.txt	00:00:00.1464692	152089
Decompress	Native	alice29.txt	00:00:00.1598902	152089
Compress	Managed	asyoulik.txt	00:00:01.9258908	44498
Compress	Native	asyoulik.txt	00:00:01.4281719	44493
Decompress	Managed	asyoulik.txt	00:00:00.1296718	125179
Decompress	Native	asyoulik.txt	00:00:00.2165359	125179
Compress	Managed	cp.html	00:00:00.4323960	7640
Compress	Native	cp.html	00:00:01.0320704	7632
Decompress	Managed	cp.html	00:00:00.0369613	24603
Decompress	Native	cp.html	00:00:00.1209998	24603
Compress	Managed	fields.c	00:00:00.2833704	2990
Compress	Native	fields.c	00:00:00.9942155	2995
Decompress	Managed	fields.c	00:00:00.0223107	11581
Decompress	Native	fields.c	00:00:00.0944675	11581
Compress	Managed	grammar.lsp	00:00:00.1832102	1242
Compress	Native	grammar.lsp	00:00:00.9598283	1242
Decompress	Managed	grammar.lsp	00:00:00.0161157	3721
Decompress	Native	grammar.lsp	00:00:00.0733605	3721
Compress	Managed	kennedy.xls	00:00:46.0431617	50422
Compress	Native	kennedy.xls	00:00:08.7667059	51396
Decompress	Managed	kennedy.xls	00:00:00.5156042	1029744
Decompress	Native	kennedy.xls	00:00:00.2552746	1029744
Compress	Managed	lcet10.txt	00:00:07.8654577	119556
Compress	Native	lcet10.txt	00:00:02.6410334	119527
Decompress	Managed	lcet10.txt	00:00:00.3143749	426754
Decompress	Native	lcet10.txt	00:00:00.2462124	426754
Compress	Managed	plrabn12.txt	00:00:08.1932977	165353
Compress	Native	plrabn12.txt	00:00:02.7910135	165319
Decompress	Managed	plrabn12.txt	00:00:00.4050662	481861
Decompress	Native	plrabn12.txt	00:00:00.2855161	481861
Compress	Managed	ptt5	00:00:04.0817633	43788
Compress	Native	ptt5	00:00:01.9063002	43503
Decompress	Managed	ptt5	00:00:00.1783090	513216
Decompress	Native	ptt5	00:00:00.1627669	513216
Compress	Managed	sum	00:00:00.6918768	9427
Compress	Native	sum	00:00:01.1139660	9430
Decompress	Managed	sum	00:00:00.0472818	38240
Decompress	Native	sum	00:00:00.1067857	38240
Compress	Managed	xargs.1	00:00:00.1904065	1761
Compress	Native	xargs.1	00:00:00.9939127	1760
Decompress	Managed	xargs.1	00:00:00.0208210	4227
Decompress	Native	xargs.1	00:00:00.0815763	4227

Note that the native lzma here is doing File IO itself, so it is going to be a little bit faster than what the end-product would be since we'd want to do the IO in C# so LZMA could follow the Stream pattern.

[GSPP]

I wonder why native and managed have so different results. kennedy.xls is smaller with managed. kennedy.xls has a crazy managed compression time. Weird.