Illegal multibyte sequence

[wuodar]

Bug Metadata

• Version of extract_msg: 0.54.0
• Your python version: Python 3.11.6
• How did you launch extract_msg?
  • My command line or
  • I used the extract_msg package

Describe the bug
For some .msg files, I'm getting UnicodeDecodeError: 'XXX' codec can't decode byte (...): illegal multibyte sequence
The example codecs that fails are

• windows-950
• shift_jis
• charmap
• gb2312

Traceback

  File "src/doc_parser/loaders.py", line 183, in get_msg_content
    with extract_msg.openMsg(path) as msg:
         ^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/extract_msg/open_msg.py", line 124, in openMsg
    return Message(path, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/extract_msg/msg_classes/message_base.py", line 83, in __init__
    super().__init__(path, **kwargs)
  File ".venv/lib/python3.11/site-packages/extract_msg/msg_classes/msg.py", line 221, in __init__
    self.attachments
  File "/Users/kacperwlodarczyk/.local/share/uv/python/cpython-3.11.6-macos-aarch64-none/lib/python3.11/functools.py", line 1001, in __get__
    val = self.func(instance)
          ^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/extract_msg/msg_classes/msg.py", line 862, in attachments
    attachments.append(self.initAttachmentFunc(self, attachmentDir))
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/extract_msg/attachments/__init__.py", line 108, in initStandardAttachment
    return EmbeddedMsgAttachment(msg, dir_, propStore)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/extract_msg/attachments/emb_msg_att.py", line 38, in __init__
    self.__data = openMsg(self.msg.path, prefix = self.__prefix, parentMsg = self.msg, treePath = self.treePath, **self.msg.kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/extract_msg/open_msg.py", line 90, in openMsg
    msg = MSGFile(path, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/extract_msg/msg_classes/msg.py", line 206, in __init__
    filename = self.getStringStream(prefixl[:-1] + ['__substg1.0_3001'], prefix = False)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/extract_msg/msg_classes/msg.py", line 738, in getStringStream
    return None if tmp is None else tmp.decode(self.stringEncoding)
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'shift_jis' codec can't decode byte 0xfc in position 74: illegal multibyte sequence

[TheElementalOfDestruction]

It's possible you may want to try adding overrideEncoding='chardet' to the options for openMsg and see if that changes anything. If it does, then it's likely the encoding for those files is being incorrectly reported. Otherwise, the encoding for some properties is likely corrupted or it may be that the filename property is encoded different for some absurd reason

[wuodar]

overriding to chardet actually helped, at least for two messages that were causing the error, thanks! before, I tried some other encodings but I was wrong

[TheElementalOfDestruction]

For the ones still failing, is it failing in the same place for a different place?

[wuodar]

I mean, I could only test on 2 files as I can't access rest of them (we delete them after ttl in our system)

[TheElementalOfDestruction]

I'm going to close this issue for now. If you get a file that continues to fail despite using chardet, please see if you can grab a traceback to see if it is failing in the same place.

[TheElementalOfDestruction]

Wait a second, sorry I misread the line before, I think I might know what is happening on this one. The call for filename (where it is failing) accesses a stream from a containing MSG file which may have a different encoding. It actually needs to request the stream through the parent. Let me fix that, and it should fix the issue entirely if the files aren't corrupted

Sorry about that!

[TheElementalOfDestruction]

Version 0.54.1 was released which should hopefully fix this issue on any more files you have, assuming it isn't because of misreporting or corruption.

[bllp12]

I have same problem, when sender name have chinese word, it will return error "Failed to parse .msg: 'gb2312' codec can't decode byte 0xab in position 19: illegal multibyte sequence" ,if i use msg = extract_msg.Message(tmp_path, overrideEncoding='chardet') ,it‘s not work.

'Traceback (most recent call last):\n File , line 260, in extract_text_from_file\n sender = msg.sender or ""\n ^^^^^^^^^^\n File "D:\App\Python3.11.0\Lib\functools.py", line 1001, in get\n val = self.func(instance)\n ^^^^^^^^^^^^^^^^^^^\n File "D:\App\Python3.11.0\Lib\site-packages\extract_msg\msg_classes\message_base.py", line 1394, in sender\n return decodeRfc2047(headerResult)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "D:\App\Python3.11.0\Lib\site-packages\extract_msg\utils.py", line 224, in decodeRfc2047\n return ''.join(\n ^^^^^^^^\n File "D:\App\Python3.11.0\Lib\site-packages\extract_msg\utils.py", line 225, in \n x[0].decode(x[1] or 'raw-unicode-escape') if isinstance(x[0], bytes) else x[0]\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nUnicodeDecodeError: 'gb2312' codec can't decode byte 0xab in position 19: illegal multibyte sequence\n\nDuring handling of the above excepti...