System.IO.Packaging.Open throws FileFormatException saying File contains corrupted data when the size of the package is more than 4GB and targeting .NET Framework

[SeenaAugusty]

Description

Our DacFx code uses System.IO.Packaging for packing the schema and when it is packaged with more than 4GB, it throw FileFormat Exception saying File contains corrupted data when it is used in a console app targeting .NET Framework but the same package loads fine in .NET Core

File contains corrupted data.

Program Location:

at MS.Internal.IO.Zip.ZipIOLocalFileBlock.Validate(String fileName, ZipIOCentralDirectoryBlock centralDir, ZipIOCentralDirectoryFileHeader centralDirFileHeader)
at MS.Internal.IO.Zip.ZipIOLocalFileBlock.ParseRecord(BinaryReader reader, String fileName, Int64 position, ZipIOCentralDirectoryBlock centralDir, ZipIOCentralDirectoryFileHeader centralDirFileHeader)
at MS.Internal.IO.Zip.ZipIOLocalFileBlock.SeekableLoad(ZipIOBlockManager blockManager, String fileName)
at MS.Internal.IO.Zip.ZipIOBlockManager.LoadLocalFileBlock(String zipFileName)
at MS.Internal.IO.Zip.ZipArchive.GetFile(String zipFileName)
at MS.Internal.IO.Zip.ZipArchive.GetFiles()
at System.IO.Packaging.ZipPackage.ContentTypeHelper..ctor(ZipArchive zipArchive, IgnoredItemHelper ignoredItemHelper)
at System.IO.Packaging.ZipPackage..ctor(String path, FileMode mode, FileAccess access, FileShare share, Boolean streaming)
at System.IO.Packaging.Package.Open(String path, FileMode packageMode, FileAccess packageAccess, FileShare packageShare, Boolean streaming)
at Microsoft.Data.Tools.Schema.Sql.Build.SqlPackage.Artifact.InitializefromFile(FileInfo fileInfo, FileMode mode, FileAccess access)

Reproduction Steps

1. Create a console app targeting .NET core and package a file of size greater than 4 or 6GB using System.IO.Packaging library
2. Create another console app targeting .NET framework and try to open the package created in step 1 using System.IO.Packaging library, it would throw File Format Exception.

Expected behavior

It should open the package in the console app targeting .Net Framework

Actual behavior

It throws File Format exception saying File contains corrupted data when it works fine in the console app targeting .NET core

Regression?

No response

Known Workarounds

Open the package in the console app targeting .NET Core

Configuration

No response

Other information

It has been reported here in dotnet/wpf#2676 and because of this issue, we have an issue opened in DacFx microsoft/DacFx#363

Tagging subscribers to this area: @dotnet/area-system-io-compression
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

Our DacFx code uses System.IO.Packaging for packing the schema and when it is packaged with more than 4GB, it throw FileFormat Exception saying File contains corrupted data when it is used in a console app targeting .NET Framework but the same package loads fine in .NET Core

File contains corrupted data.

Program Location:

at MS.Internal.IO.Zip.ZipIOLocalFileBlock.Validate(String fileName, ZipIOCentralDirectoryBlock centralDir, ZipIOCentralDirectoryFileHeader centralDirFileHeader)
at MS.Internal.IO.Zip.ZipIOLocalFileBlock.ParseRecord(BinaryReader reader, String fileName, Int64 position, ZipIOCentralDirectoryBlock centralDir, ZipIOCentralDirectoryFileHeader centralDirFileHeader)
at MS.Internal.IO.Zip.ZipIOLocalFileBlock.SeekableLoad(ZipIOBlockManager blockManager, String fileName)
at MS.Internal.IO.Zip.ZipIOBlockManager.LoadLocalFileBlock(String zipFileName)
at MS.Internal.IO.Zip.ZipArchive.GetFile(String zipFileName)
at MS.Internal.IO.Zip.ZipArchive.GetFiles()
at System.IO.Packaging.ZipPackage.ContentTypeHelper..ctor(ZipArchive zipArchive, IgnoredItemHelper ignoredItemHelper)
at System.IO.Packaging.ZipPackage..ctor(String path, FileMode mode, FileAccess access, FileShare share, Boolean streaming)
at System.IO.Packaging.Package.Open(String path, FileMode packageMode, FileAccess packageAccess, FileShare packageShare, Boolean streaming)
at Microsoft.Data.Tools.Schema.Sql.Build.SqlPackage.Artifact.InitializefromFile(FileInfo fileInfo, FileMode mode, FileAccess access)

Reproduction Steps

1. Create a console app targeting .NET core and package a file of size greater than 4 or 6GB using System.IO.Packaging library
2. Create another console app targeting .NET framework and try to open the package created in step 1 using System.IO.Packaging library, it would throw File Format Exception.

Expected behavior

It should open the package in the console app targeting .Net Framework

Actual behavior

It throws File Format exception saying File contains corrupted data when it works fine in the console app targeting .NET core

Regression?

No response

Known Workarounds

Open the package in the console app targeting .NET Core

Configuration

No response

Other information

It has been reported here in dotnet/wpf#2676 and because of this issue, we have an issue opened in DacFx microsoft/DacFx#363

Author:	SeenaAugusty
Assignees:	-
Labels:	area-System.IO.Compression
Milestone:	-

[zijchen]

We can repro this with version 5.0.0 but it's possible the issue was in prior versions too

[karakasa]

Exception occurs due to the versionNeeded is different in the local header (20, Deflate) compared to that in central directory (45, ZIP64).

The underlying reason is that all small files after large files (>4GB) requires ZIP64 header because their LH offsets are >4GB. ZipArchiveEntry.WriteCentralDirectoryFileHeader() catches that so CD is correct, however LHs are unaware of the offset issue when created in ZipArchiveEntry.WriteCrcAndSizesInLocalHeader() so the version in LH is 20.

It's unclear in the PKWare spec whether the discrepancy is allowed, but the latest .NET doesn't validate that as the Framework used to do. 7-Zip set both ZIP64 flags in LH and CD for file offsetting >4GB.

If the discrepancy is allowed, the bug can only be fixed from the Framework side because the issue is within WindowsBase.dll.

If it's not allowed, the current ZIP LH writing impl in .NET is wrong. I would propose a fix as in #94970.

[pwoosam]

@karakasa, My understanding is that the central directory headers must match the local file headers.
Reference to APPNOTE-6.3.10.TXT on pkware.cachefly.net

The central directory header is intended to provide a quick lookup of all zip file metadata without needing to scan the entire file. It should contain identical data to the local file header. Mismatched headers may indicate corruption and is invalid.

As an example, multiple compression methods in the spec also specify the expectation that the compression method will match in both the central directory headers and local file headers.

[SeenaAugusty]

@karakasa I was wondering if this fixed in net8?

[carlossanlop]

@SeenaAugusty not yet but we will backport it to 8.0 once the fix is merged.

[llali]

@carlossanlop do you have any ETA on when this fix will be ported to 8.0? in which sdk version?