Unify CodePoints and CodeUnits parsers

[hdgarrood]

Arising out of discussion in #46.

Right now, we have parsers in the modules Text.Parsing.StringParser.CodePoints and Text.Parsing.StringParser.CodeUnits, which use the same Parser data type, except that the former treats the integer pos field in the parser state as the number of code points we have consumed in the string being parsed, and the latter treats the pos field as the number of code units. This can cause problems if they are mixed:

> runParser (Tuple <$> CP.string "🐱" <*> CP.anyChar) "🐱hi"
(Right (Tuple "🐱" 'h'))

> runParser (Tuple <$> CP.string "🐱" <*> CU.anyChar) "🐱hi"
(Right (Tuple "🐱" '�'))

Addtionally, storing an index into code points is not really justifiable from a performance perspective, since indexing into a string using code points is an O(n) operation, where n is the index; it requires looking at every code point in the string up to the given index.

If we compare the APIs exported by the Text.Parsing.StringParser.Code{Units,Points} modules, they are basically the same; in particular, the CodePoints parsers still use Char almost everywhere, which limits their utility quite severely. As far as I can tell, the only difference between the CodePoints and CodeUnits parsers (now that #46 has been merged) is that the CodePoints ones will fail rather than splitting up surrogate pairs.

I think the ideal solution would be to do the following:

• Say that the pos field in the parser state always counts code units
• Unify Text.Parsing.StringParser.CodePoints and Text.Parsing.StringParser.CodeUnits into just one module; effectively, get rid of the former and move most/all of the contents of the latter back to Text.Parsing.StringParser.
• Clearly demarcate parsers which have the ability to split up surrogate pairs, like anyChar. This could be done with doc-comments or we could move them into their own module Text.Parsing.StringParser.CodeUnits.
• Provide CodePoint-based alternatives to any of the parsers which are currently based on Char, so that it is possible to do everything you might want to do without having to resort to using parsers like anyChar which can split up surrogate pairs.

[hdgarrood]

I suppose another option is to continue in the tradition of the CodePoints parsers in that we architect the Parser data type such that it's not possible to split up a surrogate pair, but we come up with a smarter way of representing the internal parser state so that we don't have to sacrifice so much performance. A lazily generated array of code points perhaps?

I think the wider problem, that is, efficiently iterating over strings without making use of the underlying encoding, is something which probably ought to be solved further up in the ecosystem, preferably in the core libraries.

[github-actions]

This issue is stale because it has been open for 60 days with no activity. Remove the stale label or comment to keep this issue open. Otherwise, this issue will be closed in 14 days.

[thomashoneyman]

Thanks for reopening!

[thomashoneyman]

We may not be able to get this in for 0.14 due to a backlog of other work, so I'm going to remove the label.

[JordanMartinez]

Since this repo is in a weird state, would it be better to revert #59?