Better C++ runtime error for non-UTF-8 content

[iterumllc]

Passing a file with non-UTF-8 to a C++-runtime-compiled Antlr 4 parser typically yields an error like:

libc++abi.dylib: terminating with uncaught exception of type std::range_error: wstring_convert: from_bytes error

In this situation a user would greatly benefit from knowing 1) that their file has non-UTF-8 content (nothing in the error message indicates this) and 2) The (preceding-UTF-8-relative) line number and offset of the first non-UTF-8 bytes.

[mike-lischke]

@parrt This is a valid and good FR.

[parrt]

Hi @mike-lischke is there a PR associated with this request?

[iterumllc]

When I filed this I hadn't looked at the code in any detail but I have a little time right now and decided to take a look.

It looks to me like this problem may be a configuration issue. The error I get is from utf32_to_utf8 which is calling wstring_convert. The latter should only be called if USE_UTF8_INSTEAD_OF_CODECVT is not defined, and utfcpp may have a better error message.

This is all confusing to me because I had a number of build problems related to utfcpp and I just verified that the package is pulled into runtime/Cpp/runtime/thirdparty/utfcpp. But looking closer it appears that USE_UTF8_INSTEAD_OF_CODECVT is not defined for the build of the parser/generator itself.

Just defining it on its own leads to include path problems. So I'm going to try to fix that temporarily and see if utf8's copy method throws a more useful exception. If it does I'll either try to throw a draft PR together or put more information in this issue.

[iterumllc]

Compiling with USE_UTF8_INSTEAD_OF_CODECVT with the supplied ExternalAntlr4Cpp.cmake isn't very easy but the result with the error is:

terminate called after throwing an instance of 'utf8::invalid_utf8'
  what():  Invalid UTF-8

Which is at least an improvement.

The "Introductory Sample" from the main utfcpp page includes the use of getline() and utf8::find_invalid() to do the rest, which would be an easy enough fallback to add in StringUtils.h after catching the exception. I would be willing to do a PR for that if someone else can sort out the utfcpp build issues -- it's not clear why the package is pulled down, built, but not used and I don't think I'm the right person to answer that.

[mike-lischke]

@parrt I don't know of any PR for this issue.

[jcking]

@mike-lischke The new standalone UTF-8 handling in #3398 could be easily updated to keep track of the byte offset (code units), code point offset and include it in the error message.

[iterumllc]

If somebody else doesn't first I may take a look once #3398 is merged.

[mike-lischke]

@iterumllc That would be great if you could add the error handling you want and create a new PR! In the meantime we can close this issue here, right?

[iterumllc]

I pulled over and built a copy of the Antlr 4 repository matching 43fb4c2 . Then I built a project against that same revision of the Cpp runtime. Everything compiles OK but when I run I get an exception thrown by std::call_once() called at line 31 of ATNDeserializationOptions.cpp. So I can't currently see the utf8-specific error.

That line was changed towards the end of October. I'm on Arch Linux compiling with gcc 10.3.0. This project's use of Antlr 4 is not trivial but pretty vanilla. I'm willing to look into this more but probably not solo. @jcking any adivce?

(BTW @parrt may be amused to know that the project is makeotfexe in Adobe's AFDKO and the port to Antlr 4 was from PCCTS 1.33.)

[jcking]

Its probably because pthread is not linked to whatever you are trying to run. You may need to pass -pthread when compiling and linking.