Set fallback encoding for message strings

[pudo]

My proposal is to return a fallback encoding when a message does not specify any encoding; and to log a warning alongside. Here is the stack trace we're seeing:

File "/ingestors/ingestors/manager.py", line 120, in ingest_entity
    self.ingest(file_path, entity)
  File "/ingestors/ingestors/manager.py", line 133, in ingest
    self.delegate(ingestor_class, file_path, entity)
  File "/ingestors/ingestors/manager.py", line 147, in delegate
    ingestor_class(self).ingest(file_path, entity)
  File "/ingestors/ingestors/email/outlookmsg.py", line 37, in ingest
    msg = FieldMessage(file_path)
  File "/usr/local/lib/python3.7/dist-packages/extract_msg/message.py", line 87, in __init__
    self.header
  File "/usr/local/lib/python3.7/dist-packages/extract_msg/message.py", line 212, in header
    headerText = self._getStringStream('__substg1.0_007D')
  File "/usr/local/lib/python3.7/dist-packages/extract_msg/message.py", line 166, in _getStringStream
    return None if tmp is None else tmp.decode(self.stringEncoding)
  File "/usr/local/lib/python3.7/dist-packages/extract_msg/message.py", line 292, in stringEncoding
    raise Exception('Encoding property not found')
Exception: Encoding property not found

Unfortunately, I cannot share the files causing this (they are subject to an active journalistic investigation). But it is reasonable to assume that they might not originate from Microsoft Outlook (instead, e.g., an eDiscovery tool), or have been tampered with.

[TheElementalOfDestruction]

If the msg file does not come from Microsoft outlook, then an error like this is the fault of whatever program it does come from. That msg file violates msg specifications and you should report it to them. This is not a problem with our software.

If I have misunderstood what you are saying, please correct me.

[pudo]

Respectfully, I think this would much lessen the value of this package. Like any file format, .msg is a thing that exists in the wild, not just in a spec book.

Can you explain to me what - to you - the purpose of this package is? I thought that attempting to use it for data forensics would be very well in scope. In that case a world-as-it-exists rather than world-as-it-should-be approach seems necessary.

[TheElementalOfDestruction]

@pudo this is a project which I took the time to carefully learn the msg standard (which is an utterly stupid standard).

While it is terrible, it DOES do a few things right, and some things are mandatory in it rather than optional. The specification of an encoding when it is not Unicode is one of these things. While we could fallback to an encoding, there is no guarantee that it would work. You see, msg files first divide encodings in two ways: Unicode and any other encoding.

Let me give you a counter example. Let's say you have a program that writes zip files, but only your program can read it because you did something wrong. Should the other programs be forced to accommodate for your mistakes? Because that is what you are asking me to do. There should not be an error if the msg files are made correctly.

I implore you: please report this to the makers of whatever program created those files.

[pudo]

@TheElementalOfCreation I do not know who made the file. They are material from an organised crime group that would likely want to harm me if they knew I have their files. So fixing it upstream would not be healthy.

[TheElementalOfDestruction]

I mean those who wrote the program that made the file. I suspect it is likely marked in the file somehow.

Anyways, it's late right now and I'm going to bed. I'll take a look at trying to do this in the dev versions (intended for bad msg files) of the classes at the very least (if it doesn't already exist) in the morning.

[pudo]

Thanks!

[TheElementalOfDestruction]

Re-opening until this has been properly resolved by a code update.

[TheElementalOfDestruction]

I've added the ability to manually override the encoding of a message in the dev_classes. You can use the dev Message class by using from extract_msg.dev_classes import Message and then set the encoding using msg_instance.stringEncoding = encoding.

Please tell me if you have any problems