Add a Forward-only API for System.IO.Compression

[adamhathcock]

This was started in https://github.com/dotnet/corefx/issues/9657

There seems to be a growing desire/need for a forward-only API that accesses compressed file formats (e.g. zip, gzip, tar, etc.) in a streaming manner. This means very large files as well as streams like network streams can be read and decompressed on the fly. Basically, the API reads from Stream objects and never seeks on it. This is how the Reader/Writer API from SharpCompress works.

Here's a sample from the unit tests:

using (Stream stream = new ForwardOnlyStream(File.OpenRead(path)))
using (IReader reader = ReaderFactory.Open(stream))
{
    while (reader.MoveToNextEntry())
    {
        if (!reader.Entry.IsDirectory)
        {
            reader.WriteEntryToDirectory(test.SCRATCH_FILES_PATH, new ExtractionOptions()
            {
                ExtractFullPath = true,
                Overwrite = true
            });
        }
    }
}

public interface IReader : IDisposable
{
    event EventHandler<ReaderExtractionEventArgs<IEntry>> EntryExtractionProgress;

    event EventHandler<CompressedBytesReadEventArgs> CompressedBytesRead;
    event EventHandler<FilePartExtractionBeginEventArgs> FilePartExtractionBegin;

    ArchiveType ArchiveType { get; }

    IEntry Entry { get; }

    /// <summary>
    /// Decompresses the current entry to the stream.  This cannot be called twice for the current entry.
    /// </summary>
    /// <param name="writableStream"></param>
    void WriteEntryTo(Stream writableStream);

    bool Cancelled { get; }
    void Cancel();

    /// <summary>
    /// Moves to the next entry by reading more data from the underlying stream.  This skips if data has not been read.
    /// </summary>
    /// <returns></returns>
    bool MoveToNextEntry();

    /// <summary>
    /// Opens the current entry as a stream that will decompress as it is read.
    /// Read the entire stream or use SkipEntry on EntryStream.
    /// </summary>
    EntryStream OpenEntryStream();
}

WriteEntryToDirectory is an extension method that provides some shortcuts for dealing with file operations but what it really does is just grab the internal stream and decompresses. The actual entry method is just IReader.WriteEntryTo(Stream); If the entry isn't decompressed then the internal stream is just moved forward and not decompressed if possible (some formats require decompression since compressed length can be unknown)

The Writer API works similarly.

There is also a generic API from ReaderFactory or WriterFactory that doesn't require knowledge of the format beforehand. There is also a similar ArchiveFactory that is writeable (that uses WriterFactory internally to save) that could also be used for the current ZipArchive API and beyond.

As the author of SharpCompress, I'd like to push a lot of the ideas into core library but having native access to the compression streams (like the internal zlib) would be a great performance benefit for me. I haven't ever written any compression algorithm implementations myself so I'm sure my managed implementations need a lot of love.

I would start by creating ZipReader and ZipWriter (as well as starting the generic API) using a lot of the internal code already in the core library to prove out the API. This kind of relates to https://github.com/dotnet/corefx/issues/14853 but forward-only access is something most libraries don't account for so I'm not sure. Other easy additions would be Reader/Writer support for GZip, BZip2 and LZip (with an LZMA compressor).

Tar support linked with the previous single file formats would be a larger addition. SharpCompress deals with tar.gz, tar.bz2 and tar.lz auto-magically by detecting first the compressed format then a tar file inside. The above API works the same.

Thoughts?

Summoning a few people from the other issue
cc: @ianhays @karelz @qmfrederik

[karelz]

@ianhays what is the next step? You did +1 above, does it mean you like the idea? Do we need API proposal? Or is it ready for API review? (Disclaimer: I just skimmed through it)

[adamhathcock]

I can put together a new API based on what I've done. It would be very similar to SharpCompress.

[svick]

1. Is a new writer API necessary? AFAIK, ZipArchive already supports writing to non-seekable streams on .Net Core (https://github.com/dotnet/corefx/issues/11497).
2. For reading, I think the API should return an IEnumerable<T>. That way, the user can use a foreach and they don't have to manually call something like MoveNext in a while loop. (Some LINQ methods also become a possibility.)

[adamhathcock]

1. You're right, this probably isn't strictly necessary. It would be the inverse of a proposed Reader API and it would have a generic writing implementation while the current one is specific to Zip and could probably not apply in a generic case. If a generic API isn't desired then this could possibly be dropped.
2. I've gone back and forth if I wanted to do it that way. The reason why I haven't is because people could be tempted to do ToList() on the reader which would definitely not work and defeat the point of the reader API when they should have used the archive API.

[svick]

Then maybe the API should return a type that can be used in a foreach, but does not implement IEnumerable<T>. Though that could be confusing to someone who sees the API for the first time.

[ianhays]

If a generic API isn't desired then this could possibly be dropped.

I've gone back-and-forth about this myself. I opened https://github.com/dotnet/corefx/issues/9709 to track my thoughts long-term on a way to centralize our compression algorithms somehow. It's clear that as we continue adding more compression types to corefx it's going to become trickier and tricker to integrate them smoothly and expose them to devs of all skill levels without being overwhelming or overly simplistic.

I've gone back and forth if I wanted to do it that way. The reason why I haven't is because people could be tempted to do ToList() on the reader which would definitely not work and defeat the point of the reader API when they should have used the archive API.

I think in this case increased usability is more valuable than protecting devs from themselves. You could implement a code analysis rule specifically to look for instances where devs do a ToList() and suggest they avoid it.

[ianhays]

https://github.com/dotnet/corefx/issues/9237 is also somewhat related, though less so.

[ianhays]

@ianhays what is the next step? You did +1 above, does it mean you like the idea? Do we need API proposal? Or is it ready for API review? (Disclaimer: I just skimmed through it)

I do like the idea. My main concern is that I really want to do our compression expansion right. There are a ton of questions around design and usability that I want answered well before we take anything like this issue, dotnet/corefx#9237, or dotnet/corefx#9709. Adding something like LZMA is easy because we can model it after DeflateStream/GZipStream with a similar API, but this issue (and the others linked above) is far more complex in the long-term implications of its design and API expansion.

[karelz]

OK, what is the next step? (you coming back with larger design?) What is rough ETA?

[ianhays]

OK, what is the next step? (you coming back with larger design?)

Continue discussing design choices in this issue, dotnet/corefx#9709, and dotnet/corefx#9237. Wait for other people to be interested and comment/vote on these threads with their opinion.

What is rough ETA?

Whenever we reach quorum and have a sizable mass of users requesting the feature. It's not something that we're hurting for right now or that a lot of customers are asking for, so I don't feel comfortable assigning an ETA for it.

[adamhathcock]

Here's a rough Reader API based on what I've done with SharpCompress and making the Reader itself be IEnumerable might not quite be correct.

//The file format of the Archive.  Can be confusing as GZip is a single file format as well as a compression type (using Tar as archive type).
public enum ArchiveType
{
    Rar,
    Zip,
    Tar,
    SevenZip,
    GZip
}

//The compression type used on the Archive/Entry.  GZip,BZip2,LZip,Xz are single file formats as well as a compression type.
public enum CompressionType
{
    None,
    GZip,
    BZip2,
    PPMd,
    Deflate,
    Rar,
    LZMA,
    BCJ,
    BCJ2,
    LZip,
    Xz,
    Unknown
}

//possibly not needed
public class ReaderOptions
{
    //Password protected archives (out of scope?)
    public string Password { get; set; }

    //Take ownership of the stream
    public bool CloseStream { get; set; }
}

public static class ReaderFactory
{
    public static IReader Open(Stream stream, ReaderOptions options = null);
#if NETSTANDARD1_3
    public static IReader Open(FileInfo file, ReaderOptions options = null);
    public static IReader Open(string filePath, ReaderOptions options = null);
#endif
}

public interface IReader : IEnumerable<IReaderEntry>
{
    IEnumerator<IReaderEntry> GetEnumerator();

    ArchiveType ArchiveType { get; }

    IReaderEntry CurrentEntry { get; }

    void Dispose();
}

public interface IReaderEntry
{
    Stream OpenRead();
}

public interface IEntry
{
    //This is on entry because some archive types (like ZIP) can have compression types different per entry
    CompressionType CompressionType { get; }
    string Key { get; }

    //everything below here is optional depending on archive type
    DateTime? ArchivedTime { get; }
    
    long? CompressedSize { get; }
    long? Crc { get; }
    long? Size { get; }

    DateTime? CreatedTime { get; }
    
    bool? IsDirectory { get; }
    bool? IsEncrypted { get; }
    bool? IsSplit { get; }
    
    DateTime? LastAccessedTime { get; }
    DateTime? LastModifiedTime { get; }
}

public static class IReaderExtensions
{
#if NETSTANDARD1_3
    public static void WriteEntryTo(this IReaderEntry reader, string filePath);
    /// <summary>
    /// Extract all remaining unread entries to specific directory, retaining filename
    /// </summary>
    public static void WriteAllToDirectory(this IReader reader, string destinationDirectory,
                                            ExtractionOptions options = null);

    /// <summary>
    /// Extract to specific directory, retaining filename
    /// </summary>
    public static void WriteEntryToDirectory(this IReaderEntry reader, string destinationDirectory,
                                                ExtractionOptions options = null);

    /// <summary>
    /// Extract to specific file
    /// </summary>
    public static void WriteEntryToFile(this IReaderEntry reader, string destinationFileName,
                                        ExtractionOptions options = null);
#endif
}
[Flags]
public enum ExtractionOptions
{
    None,
    /// <summary>
    /// overwrite target if it exists
    /// </summary>
    Overwrite,

    /// <summary>
    /// extract with internal directory structure
    /// </summary>
    ExtractFullPath,

    /// <summary>
    /// preserve file time
    /// </summary>
    PreserveFileTime,

    /// <summary>
    /// preserve windows file attributes
    /// </summary>
    PreserveAttributes,
}

Sample usage

public static void Main() 
{
    using(var reader = ReaderFactory.Open("C:\\test.zip"))
    {
        foreach(var entry in reader)
        {
            if (!entry.IsDirectory)
            {
                entry.WriteEntryToDirectory("D:\\temp", ExtractionOptions.ExtractFullPath);
            }
        }
    }

    using(var reader = ReaderFactory.Open("C:\\test.zip"))
    {
        reader.WriteAllToDirectory("D:\\temp2", ExtractionOptions.ExtractFullPath);
    }

//no explicit using statement for IReader
    foreach(var entry in ReaderFactory.Open("C:\\test.zip")))
    {
        if (entry.Key.StartsWith("x"))
        {
            using(var data = entry.OpenRead()) 
            {
                data.CopyTo(new MemoryStream());
            }
        }
    }
}

Can expand out to the show the generic Archive API too.

[svick]

@ianhays

My main concern is that I really want to do our compression expansion right. There are a ton of questions around design and usability that I want answered well before we take anything like this issue, dotnet/corefx#9237, or dotnet/corefx#9709.

Would it then make sense to design (and implement) an initial version of the whole compression expansion in corefxlab? That way, all the related APIs can be designed together and easily inform each other.

[adamhathcock]

Putting it wholesale together as a real thing in corefxlab makes sense. Evolving the API might be too slow.

[ianhays]

Would it then make sense to design (and implement) an initial version of the whole compression expansion in corefxlab? That way, all the related APIs can be designed together and easily inform each other.

That is probably the best idea. We can design the large structure in corefxlab, then move small pieces to corefx one-by-one with their own justification so that we can actually get them through API review.

Here's a rough Reader API based on what I've done with SharpCompress and making the Reader itself be IEnumerable might not quite be correct.

I like that too. Your rough API is pretty similar to what I had in mind.

A couple suggestions:

• ReaderOptions should include an option to specify the ArchiveType explicitly.
• ArchiveType.GZip should be called TarGZip or GZipTar to avoid confusion.
• ExtractionOptions should have a PreservePermissions entry.
• Should there be some relation between IReaderEntry and IEntry? Where is the latter used?
• Instead of having IReader.WriteEntryTo and friends, I think those should be extension methods on the IReaderEntry (or IEntry) itself.
• All of the instances of Reader/IReader should be replaced with ArchiveReader or IArchiveReader (or something like that).

On a more general note, could you explain why you split reading and writing to their own interfaces/classes? Having a single IArchive seems more intuitive and usable despite opening up some notsupported operations e.g. attempting to write to an archive that is being read.

[ianhays]

public interface IEntry

It should be possible to combine this with the concept of a compression algorithm interface a la https://github.com/dotnet/corefx/issues/9709.