Sorting MFT entry: 0 $DATA attribute list entries based on file reference fails to open volume

[kgermanov]

On one of volume:

bash$ ./fsntfsinfo ./dst_mft
fsntfsinfo 20240501

Unable to open: ./dst_mft.
libfsntfs_file_system_read_mft: unable to read attribute list data MFT entry: 1764965-5.
libfsntfs_internal_volume_open_read: unable to read MFT (MFT entry: 0).
libfsntfs_volume_open_file_io_handle: unable to read from file IO handle.
info_handle_open_input: unable to open input volume.

Trace logs: ntfs.log

[kgermanov]

Looks like was read empty records:

bash# dd if=dst_mft count=1024 bs=1 skip=513774875648 | hexdump -C
00000000  46 49 4c 45 30 00 03 00  2d 3c 0e fd 57 01 00 00  |FILE0...-<..W...|
00000010  03 00 02 00 38 00 [00 00]  38 02 00 00 00 04 00 00  |....8...8.......|
00000020  00 00 00 00 00 00 00 00  04 00 00 00 9d 45 1c 00  |.............E..|
00000030  01 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000040  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 01 00  |................|
00000200  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000003f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 01 00  |................|
00000400

With this patch:

diff --git a/libfsntfs/libfsntfs_file_system.c b/libfsntfs/libfsntfs_file_system.c
index ca912a5..5f03f56 100644
--- a/libfsntfs/libfsntfs_file_system.c
+++ b/libfsntfs/libfsntfs_file_system.c
@@ -588,7 +588,7 @@ int libfsntfs_file_system_read_mft(
                             file_system->mft->mft_entry_vector,
                             file_system->mft->mft_entry_cache,
                             file_reference,
-                            error ) != 1 )
+                                error ) == -1 )
                        {
                                libcerror_error_set(
                                 error,

It is openable:

bash$ ./fsntfsinfo ./dst_mft
fsntfsinfo 20240501

New Technology File System information:

Volume information:
        Name                            :
        Version                         : 3.1
        Serial number                   : b02e60332e5ff0b8
        Bytes per sector                : 512
        Cluster block size              : 4096
        MFT entry size                  : 1024
        Index entry size                : 4096
        Flags                           : 0x0000

But there is problem on open mft:

bash$ ./fsntfsinfo -E 0 ./dst_mft
fsntfsinfo 20240501

Error reading MFT entry: 0
libfsntfs_mft_attribute_get_data_extents_array: invalid attribute data VCN offset value out of bounds.
libfsntfs_file_entry_initialize: unable to create extents array.
libfsntfs_volume_get_file_entry_by_index: unable to create file entry with MFT entry: 0.

Unable to print MFT entry: 0.

There is trace logs:
ntfs.log.zip

[kgermanov]

After comparing with ntfs-3g was founded, that there is unordered file reference in attribute list:

[kgermanov]

After replace insert to append it is openable(even without first patch), fsntfsmount works too:

bash$ ./fsntfsinfo  ./dst_mft
fsntfsinfo 20240501

New Technology File System information:

Volume information:
        Name                            :
        Version                         : 3.1
        Serial number                   : b02e60332e5ff0b8
        Bytes per sector                : 512
        Cluster block size              : 4096
        MFT entry size                  : 1024
        Index entry size                : 4096
        Flags                           : 0x0000

bash$ ./fsntfsinfo -E 0 ./dst_mft
fsntfsinfo 20240501

MFT entry: 0 information:
        Is allocated                    : true
        File reference                  : 0-1
        Base record file reference      : Not set (0)
        Journal sequence number         : 1883348917785
        Number of attributes            : 20

Attribute: 1
        Attribute type                  : $STANDARD_INFORMATION (0x00000010)
        Creation time                   : Jan 01, 1970 00:00:00.000000000 UTC
        Modification time               : Jan 01, 1970 00:00:00.000000000 UTC
        Access time                     : Jan 01, 1970 00:00:00.000000000 UTC
        Entry modification time         : Jan 01, 1970 00:00:00.000000000 UTC
        Owner identifier                : 0
        Security descriptor identifier  : 256
        Update sequence number          : 0
        File attribute flags            : 0x00000006
                Is hidden (FILE_ATTRIBUTE_HIDDEN)
                Is system (FILE_ATTRIBUTE_SYSTEM)

Attribute: 2
        Attribute type                  : $ATTRIBUTE_LIST (0x00000020)
        Data VCN range                  : 0 - 63
        Data size                       : 608 bytes
        Number of entries               : 19
        Entry: 0                        : $STANDARD_INFORMATION (0x00000010) in file reference: 0-1
        Entry: 1                        : $FILE_NAME (0x00000030) in file reference: 0-1
        Entry: 2                        : $DATA (0x00000080) in file reference: 0-1
        Entry: 3                        : $DATA (0x00000080) in file reference: 15-15
        Entry: 4                        : $DATA (0x00000080) in file reference: 16-1
        Entry: 5                        : $DATA (0x00000080) in file reference: 17-1
        Entry: 6                        : $DATA (0x00000080) in file reference: 18-1
        Entry: 7                        : $DATA (0x00000080) in file reference: 19-1
        Entry: 8                        : $DATA (0x00000080) in file reference: 20-1
        Entry: 9                        : $DATA (0x00000080) in file reference: 21-1
        Entry: 10                       : $DATA (0x00000080) in file reference: 22-1
        Entry: 11                       : $DATA (0x00000080) in file reference: 1483293-220
        Entry: 12                       : $DATA (0x00000080) in file reference: 1512604-189
        Entry: 13                       : $DATA (0x00000080) in file reference: 1764965-5
        Entry: 14                       : $DATA (0x00000080) in file reference: 1487494-246
        Entry: 15                       : $DATA (0x00000080) in file reference: 1517799-58
        Entry: 16                       : $DATA (0x00000080) in file reference: 1509523-239
        Entry: 17                       : $DATA (0x00000080) in file reference: 712401-11
        Entry: 18                       : $BITMAP (0x000000b0) in file reference: 0-1

Attribute: 3
        Attribute type                  : $FILE_NAME (0x00000030)
        Parent file reference           : 5-5
        Creation time                   : Jan 01, 1970 00:00:00.000000000 UTC
        Modification time               : Jan 01, 1970 00:00:00.000000000 UTC
        Access time                     : Jan 01, 1970 00:00:00.000000000 UTC
        Entry modification time         : Jan 01, 1970 00:00:00.000000000 UTC
        File attribute flags            : 0x00000006
                Is hidden (FILE_ATTRIBUTE_HIDDEN)
                Is system (FILE_ATTRIBUTE_SYSTEM)
        Name space                      : DOS and Windows (3)
        Name                            : $MFT

Attribute: 4
        Attribute type                  : $DATA (0x00000080)
        Data VCN range                  : 0 - 3
        Data size                       : 1923612672 bytes
        Data flags                      : 0x0000

Attribute: 5
        Attribute type                  : $BITMAP (0x000000b0)
        Data VCN range                  : 0 - 58
        Data size                       : 237576 bytes

Attribute: 6
        Attribute type                  : $DATA (0x00000080)
        Data VCN range                  : 4 - 370958
        Data flags                      : 0x0000

Attribute: 7
        Attribute type                  : $DATA (0x00000080)
        Data VCN range                  : 370959 - 419967
        Data flags                      : 0x0000

Attribute: 8
        Attribute type                  : $DATA (0x00000080)
        Data VCN range                  : 419968 - 423569
        Data flags                      : 0x0000

Attribute: 9
        Attribute type                  : $DATA (0x00000080)
        Data VCN range                  : 423570 - 428151
        Data flags                      : 0x0000

Attribute: 10
        Attribute type                  : $DATA (0x00000080)
        Data VCN range                  : 428152 - 432551
        Data flags                      : 0x0000

Attribute: 11
        Attribute type                  : $DATA (0x00000080)
        Data VCN range                  : 432552 - 435249
        Data flags                      : 0x0000

Attribute: 12
        Attribute type                  : $DATA (0x00000080)
        Data VCN range                  : 435250 - 436987
        Data flags                      : 0x0000

Attribute: 13
        Attribute type                  : $DATA (0x00000080)
        Data VCN range                  : 436988 - 439241
        Data flags                      : 0x0000

Attribute: 14
        Attribute type                  : $DATA (0x00000080)
        Data VCN range                  : 439242 - 441792
        Data flags                      : 0x0000

Attribute: 15
        Attribute type                  : $DATA (0x00000080)
        Data VCN range                  : 441793 - 442497
        Data flags                      : 0x0000

Attribute: 16
        Attribute type                  : $DATA (0x00000080)
        Data VCN range                  : 442498 - 444429
        Data flags                      : 0x0000

Attribute: 17
        Attribute type                  : $DATA (0x00000080)
        Data VCN range                  : 444430 - 446259
        Data flags                      : 0x0000

Attribute: 18
        Attribute type                  : $DATA (0x00000080)
        Data VCN range                  : 446260 - 448777
        Data flags                      : 0x0000

Attribute: 19
        Attribute type                  : $DATA (0x00000080)
        Data VCN range                  : 448778 - 461207
        Data flags                      : 0x0000

Attribute: 20
        Attribute type                  : $DATA (0x00000080)
        Data VCN range                  : 461208 - 469631
        Data flags                      : 0x0000

PR was created

[joachimmetz]

Thanks for the detailed report. Am I correct that the format edge (or corruption?) case is:

• MFT entry 0 contains an attribute list
• One of the attribute list entry refers to MFT entry 1764965-5
• MFT entry 1764965-5 is empty

[kgermanov]

@joachimmetz No.

1. MFT entry 0 contains an attribute list as another entry
2. MFT entry 1764965-5 is not empty.
   Problem in that how combined data of attribute due full list: it is collected due filerefernce list. But if we sort this list - than corresponded data chunk will meshed.
   So root cause in that this list can be not sorted and we should not reorder it.

[joachimmetz]

So this issue is that sorting on file reference causes the order of the VCNs to change. However MFT entry: 0 with an attribute list will need to read the MFT attributes corresponding to an attribute list in-order of the file reference to correctly grow the $MFT. What seems to be missing is processing the data runs in order of VCN.