Idea: switch to libzip for ZIP archive implementation

[grendello]

By way of introduction, I work on the Xamarin/Mono team at Microsoft (Xamarin.Android and, from time to time, a foray to the Mono BCL/runtime land) and we've been struggling with various issues related to ZIP archive support over the past few years.

The main problem with all (as far as we can tell) managed implementations is that they implement only a subset of what's supported by the ZIP format and ignore parts that are important on non-Windows systems (especially all flavors of Unix). The biggest issue is lack of support for the extended Unix fields (described in the PKWARE appnote as well as in this document. Support for those fields is important because they make difference between special files and regular files as stored in the archive (that includes file permission bits on Unix). Another issue (less common but still occasionally relevant) is lack of support for symbolic links in managed libraries. Yet another problem is an occasional issue with even opening some ZIP archives (especially created with older versions of ZIP and friends) to the point that we had to resort to using the external unzip program to open some archives.

For the record, the minimum set of extended fields that would need to be implemented is: 0x000d, 0x5855, 0x6375, 0x7855, 0x7875, 0x5455 and symbolic links (which are stored as special files in the ZIP archive)

To address those problems we decided to implement our own support for the ZIP archive and after looking around we chose libzip as the base for our code. The library implements support for ZIP archive format while deferring (de)compression to the venerable zlib library (the same one corefx uses). libzip is cross-platform, works very well on both Unix and Windows systems, is (IMO) very well written and maintained, and uses a permissive license. It does not implement support for interpreting of the extended fields, merely supports writing/reading them as blobs of binary data. For that reason we created a managed wrapper around libzip which not only provides and object-oriented implementation of the libzip APIs but also implements support for the most important extended fields, as listed above. We've been using both libzip and libzipsharp in Xamarin.Android for a while now and found them to work without any issues.

I think it would be great if corefx implemented full, cross-platform, support for ZIP archives and while it could be done in-house I think supporting (by using and contributing to) libzip is would be the best way to go, since many projects would benefit from a single, solid implementation of the ZIP archive format.

[ianhays]

The biggest issue is lack of support for the extended Unix fields (described in the PKWARE appnote as well as in this document. Support for those fields is important because they make difference between special files and regular files as stored in the archive (that includes file permission bits on Unix).

The main thing stopping me from adding support for these is time. There are just so many different uses of the fields and so many values, it's difficult to add them in a friendly API and I haven't had the time to nail down a design for them. Exposing the binary data may be the easiest pill to swallow API-wise, but I also like your managed wrapper. It's quite a bit of new surface area, but I strongly agree we need to expose the fields somehow.

This issue would be properly resolved if we had extended field support and this one would be if we had password support.

Yet another problem is an occasional issue with even opening some ZIP archives (especially created with older versions of ZIP and friends) to the point that we had to resort to using the external unzip program to open some archives.

Yeah, we don't have great support for zips created a long time ago or with zip programs from a long time ago. We only just recently added support for zips containing entries compressed with Deflate64.

I think it would be great if corefx implemented full, cross-platform, support for ZIP archives and while it could be done in-house I think supporting (by using and contributing to) libzip is would be the best way to go, since many projects would benefit from a single, solid implementation of the ZIP archive format.

I'd be very interested in deferring the zip work to a library like we did with zlib. We generally prefer using existing open-source implementations over creating new ones, so if the license works out then it's feasible to use libzip or whatever. I would mostly be concerned with compatibility with the existing ZipArchive API and functionality: we would want no/little net behavioral change in what is already implemented.

I think a good rough plan for this would be:

• Ensure the license of the target library works with dotnet
• Implement the existing ZipArchive API using the target library and make sure all of our unit tests pass for it. Do some perf testing and compare with the ZipArchive we have right now. Submit the changes as a PR against CoreFX and include the testing data alongside reasoning for the changes.
• Submit a separate API proposal to add support for extended fields, symlinks, passwords, etc.
• Implement the new API using the target library and submit as a PR against CoreFX.

[grendello]

I think our managed wrapper solves the issue of exposing the extended fields in quite an elegant matter (see here and here). The thing is that ZIP implementations for different operating systems usually just ignore the extended fields their target system doesn't support. In our case, living in the manged land, this is not necessary and, I think, unwelcome - there should be a single managed library if possible with runtime selection of the OS-specific code. This way user can easily dynamically cast the OS-specific classes from their abstract bases.

Regarding old ZIPs support. We had issues with some Android SDK zips a few years back, they were created with newer versions of ZIP and yet posed problems for the managed libraries. Alas, I can't recall which archives they were :( libzip is very backwards compatible, I threw at it archives created 20 years ago on a selection of systems and it worked in all the cases.

libzip license is BSD, I don't think it will be any problem with it. The existing ZipArchive API won't be a problem either since libzip is very flexible in what you can do with it. As an example of its power, take a look at this code. libzip supports different archive data "sources" - they can be files, in-memory blobs etc. The source is basically a callback which handles various requests sent to it by libzip. The code I linked to is an implementation of System.IO.Stream source. If you take a look at the set of libzip native APIs you'll notice that it's very rich and flexible. I don't think .NET ZipArchive would require any behavioral changes whatsoever and it could nicely be extended to support the extended fields and/or other operations not currently supported (encryption, Zip64)

Regarding your proposed plan:

• I'll leave the license question to people versed in licenses :)
• I can implement the current ZipArchive API using libzip/libzipsharp as a POC, when I have some time on my hands
• please take a look at our managed wrapper design - I think it's a good start and it already supports what we need. I'd appreciate any comments (or even PRs! :D) regarding the code
• sounds good!

Thanks! :)

[ianhays]

cc: @stephentoub

[carlossanlop]

Triage:
This would involve heavy work, compatibility problems, not enough value to overcome those challenges.