UnicodeDecodeError: 'ascii' codec can't decode byte 0xfc in position 79: ordinal not in range(128) (decodeRfc2047)

[DavidZha1994]

In order to get your bug addressed in a timely manner, or at all 😃, please fill out the below bug report. Please try to make it as easy as possible for us to understand what is going on. We may close out any bugs or issues without warning that are not complete or coherent.

In the bug template below anything is [square brackets] should be filled out or removed if the item doesn't apply.

Should you encounter an error that has not already been reported, please do the following when reporting it:
Bug Metadata

• Version of extract_msg: [0.45.0]
• Your python version: Python [3.8]
• How did you launch extract_msg?
  • My command line or
  • I used the extract_msg package

Describe the bug
A clear and concise description of what the bug is.
Exception has occurred: UnicodeDecodeError
'ascii' codec can't decode byte 0xfc in position 79: ordinal not in range(128)
File "/Users/yu/Workspace/Digital Poststelle/Dataset/getElement.py", line 72, in getFiles
msg = Message(msg_data)
File "/Users/yu/Workspace/Digital Poststelle/Dataset/getElement.py", line 228, in
msg = getFiles(token, files_id)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xfc in position 79: ordinal not in range(128)
[ If applicable ]
**What code did you use or can we use to reproduce this error?

def getFiles(token, fileId):
    payload = {}
    headers = {
        'Accept': 'text/plain',
        'Authorization': f'Bearer {token}'
    }
    
    url = f"{base_url}/definitions/{definitionId}/files/{fileId}"
    response = requests.request("GET", url, headers=headers, data=payload)

    if response.status_code == 200:
        msg_data = BytesIO(response.content)
        msg = Message(msg_data)
        return msg
    return

Is there a message.msg file you want to share to help us reproduce this?

• Uploaded message (drag and drop on this window)
• Emailed message as an attachment to admins: [Enter Subject Line Here]

Traceback

[Put your traceback here]

Screenshots
[Insert any screenshots or debug pictures here]

Additional context
[Add any other context about the problem here.]

[TheElementalOfDestruction]

Either part of your traceback is missing or it didn't happen in the extract_msg code as no part of that traceback mentions the files. The last line mentioned is the call to the function, which shouldn't be throwing a decode error from the look of it. Can you double check?

Edit: Looking closer at it, the lines even seem to be in reverse order, but I'm still not seeing a mention of the line from extract_msg that is throwing the error

[DavidZha1994]

Whether I get the msg files via request or I download them locally and use extract_msg read them directly , I get almost the same error
'ascii' codec can't decode byte 0xfc in position 79: ordinal not in range(128)
'ascii' codec can't decode byte 0xf6 in position 178: ordinal not in range(128)

I've downloaded almost 5000 msg files, less than 10 have a similar problem.

By the way these emails are in German.

[TheElementalOfDestruction]

I need the traceback so I can figure out what is actually causing the error. I have no idea where in extract_msg the error is coming from due to the partial traceback

After having further read the traceback, it looks like it happens sometime while opening the file, which could be a rather large number of possible causes to try to narrow down. Most of them come down to the codec specified by the msg file being ascii when it's not.

[TheElementalOfDestruction]

Hello again, this is a reminder to please ensure that you send a copy of the full traceback that includes what part of extract_msg raised that exception. The exception message is too generic for me to locate it's source without a complete traceback. All I can tell from the provided code and traceback is that the issue happened at some point while opening the message, and that the issue was non-ascii data being parsed as ascii.

[DavidZha1994]

failed msg.zip
Here are those msgs that i can't read by msg-extractor

[TheElementalOfDestruction]

Okay, using this I managed to track down the source of the issue, and it comes from the function in extract_msg that is designed to handle data in header fields that is not ASCII and would be represented in a special way. The function is designed to take data that is exactly ASCII, and any part of the process that does not have a specified encoding SHOULD be ASCII, assuming the input data is actually an ASCII string.

Turns out, these files end up giving it non ASCII strings, which is why an exception is raised. At first I was thinking that the data in these messages was just corrupted, but it turns out that is not the case. Instead, the header parsing is acting in ways I did not predict, and is actually doing that decoding ahead of time on parts of the string, but not all parts of the string. Since it doesn't do it on all parts of the string, it still needs to be properly parsed using that function, but because it parses it at all, the function fails.

So I need to go and track down why Python's email module is partially doing the conversion and figure out how to stop it, since that is the behavior that was not intended in this case. Thanks for bringing this to my attention.

[TheElementalOfDestruction]

Alright, I've got some idea of what causes it, and it's going to be complicated to do a proper fix from the look of it. It seems like the header data not properly following RFC 2047 section 5 is the source of the weirdness, but you can sort of patch the issue by change line 217 in extract_msg/utils.py. Simple replace ascii with raw-unicode-escape. This should suppress the issue and you will only notice weirdness if you access the header directly.

You can also wait for the more complete fix in version 0.46.0, but I'm not sure when that will be completed.

[DavidZha1994]

Thanks for the quick solution!

I appreciate your explanation of the problem and the workaround you provided. It's good to know you're working on a more permanent fix for version 0.46.0.

[TheElementalOfDestruction]

Alright, for 0.46.0 I've switched the policy to use Compat32 instead of the default policy, that way it just skips all of the partial parsing that default was doing. This should resolve all of the issues, but I've left in the patch to make the decode function properly accept non-ASCII data, so any non-ASCII data it receives will just be spit back out without modification.

Not sure when 0.46.0 will actually be released since I'm working on finishing a bunch of things before hand, but it will at least be more reliable. Also, good news is that even if there is still an issue at some point, it should only happen when those particular bits of data are actually accessed, since 0.46.0 removes a lot of the initialization so it's not all happening when the file is opened. This should make tracebacks much easier to read through.