Some characters may cause internal errors

[brandon-gong]

The character 𢡊 produces a lot of strange behavior with Pyret. For example, the program

#|𢡊|#

causes an "internal error", while the program

string-length("𢡊"))

(note mismatched parens) is valid syntax and outputs 2.

I'm not sure what other characters cause this issue. Interestingly Javascript says "𢡊".length is 1, but calling string-to-code-point("𢡊")) in the CPO repl returns a [list: 55370, 56394]. So it may be some issue with this offset that causes Pyret to maybe not parse the closing tag of that block comment properly, or causes Pyret to require that extra closing parentheses in the second snippet.

This bug can also be reproduced in the local version of Pyret that I got from npm (0.0.23).

File bug.arr:

#|𢡊|#
check: string-length("a") is 1 end

Output of pyret bug.arr:

[client]  2021-11-02T23:34:01.796Z Starting up server...
Compiling worklist...TypeError: Cannot read property 'name' of undefined
TypeError: Cannot read property 'name' of undefined
    at Tokenizer.next (/usr/local/lib/node_modules/pyret-npm/pyret-lang/build/phaseA/pyret.jarr:43310:34)
    at Grammar.parseRNGLR (/usr/local/lib/node_modules/pyret-npm/pyret-lang/build/phaseA/pyret.jarr:41260:51)
    at Grammar.parse (/usr/local/lib/node_modules/pyret-npm/pyret-lang/build/phaseA/pyret.jarr:40399:21)
    at parseDataRaw (/usr/local/lib/node_modules/pyret-npm/pyret-lang/build/phaseA/pyret.jarr:196890:30)
    at PFunction.parsePyret [as app] (/usr/local/lib/node_modules/pyret-npm/pyret-lang/build/phaseA/pyret.jarr:196943:20)
    at PMethod._94b3bc6bdfa78eb8ad9ae94f78841a12a681f8905e1c3c4f3d0198b676cf76f5__7 [as full_meth] (eval at <anonymous> (/usr/local/lib/node_modules/pyret-npm/pyret-lang/build/phaseA/pyret.jarr:152568:32), <anonymous>:392:49)
    at Object.maybeMethodCall0 (/usr/local/lib/node_modules/pyret-npm/pyret-lang/build/phaseA/pyret.jarr:147977:22)
    at PMethod._94b3bc6bdfa78eb8ad9ae94f78841a12a681f8905e1c3c4f3d0198b676cf76f5__23 [as full_meth] (eval at <anonymous> (/usr/local/lib/node_modules/pyret-npm/pyret-lang/build/phaseA/pyret.jarr:152568:32), <anonymous>:455:14)
    at Object.maybeMethodCall0 (/usr/local/lib/node_modules/pyret-npm/pyret-lang/build/phaseA/pyret.jarr:147977:22)
    at _64bb1f6807b984d0bb622f545c77a8fde3f260bf9c8638942becaf8890f40912__132 (eval at <anonymous> (/usr/local/lib/node_modules/pyret-npm/pyret-lang/build/phaseA/pyret.jarr:152568:32), <anonymous>:2934:14)
    at PObject.variant [as $app_fields] (eval at <anonymous> (/usr/local/lib/node_modules/pyret-npm/pyret-lang/build/phaseA/pyret.jarr:152568:32), <anonymous>:336:8)
    at PFunction._64bb1f6807b984d0bb622f545c77a8fde3f260bf9c8638942becaf8890f40912__121 [as app] (eval at <anonymous> (/usr/local/lib/node_modules/pyret-npm/pyret-lang/build/phaseA/pyret.jarr:152568:32), <anonymous>:3112:22)
    at PFunction._64bb1f6807b984d0bb622f545c77a8fde3f260bf9c8638942becaf8890f40912__120 [as app] (eval at <anonymous> (/usr/local/lib/node_modules/pyret-npm/pyret-lang/build/phaseA/pyret.jarr:152568:32), <anonymous>:3146:27)
    at PFunction._64bb1f6807b984d0bb622f545c77a8fde3f260bf9c8638942becaf8890f40912__116 [as app] (eval at <anonymous> (/usr/local/lib/node_modules/pyret-npm/pyret-lang/build/phaseA/pyret.jarr:152568:32), <anonymous>:2638:23)
    at PFunction._9507b364704724a507ae0b85544fd159e615700c86c9200d4a9fa9f67811169a__321 [as app] (eval at <anonymous> (/usr/local/lib/node_modules/pyret-npm/pyret-lang/build/phaseA/pyret.jarr:152568:32), <anonymous>:4671:54)
    at PFunction._9507b364704724a507ae0b85544fd159e615700c86c9200d4a9fa9f67811169a__410 [as app] (eval at <anonymous> (/usr/local/lib/node_modules/pyret-npm/pyret-lang/build/phaseA/pyret.jarr:152568:32), <anonymous>:5565:35)
    at PFunction._3e2cbea101d6bedc02098bed697b8aa0b5fbf66363158ec1f14427fd55dc0848__0 [as app] (eval at <anonymous> (/usr/local/lib/node_modules/pyret-npm/pyret-lang/build/phaseA/pyret.jarr:152568:32), <anonymous>:485:62)
    at PFunction._3e2cbea101d6bedc02098bed697b8aa0b5fbf66363158ec1f14427fd55dc0848__88 [as app] (eval at <anonymous> (/usr/local/lib/node_modules/pyret-npm/pyret-lang/build/phaseA/pyret.jarr:152568:32), <anonymous>:1088:29)
    at thisRuntime.run.sync (/usr/local/lib/node_modules/pyret-npm/pyret-lang/build/phaseA/pyret.jarr:151056:24)
    at ActivationRecord.fun (/usr/local/lib/node_modules/pyret-npm/pyret-lang/build/phaseA/pyret.jarr:150646:18)
    at iter (/usr/local/lib/node_modules/pyret-npm/pyret-lang/build/phaseA/pyret.jarr:150764:28)
    at Object.run (/usr/local/lib/node_modules/pyret-npm/pyret-lang/build/phaseA/pyret.jarr:150863:7)
    at Pause.resumer (/usr/local/lib/node_modules/pyret-npm/pyret-lang/build/phaseA/pyret.jarr:151055:21)
    at iter (/usr/local/lib/node_modules/pyret-npm/pyret-lang/build/phaseA/pyret.jarr:150790:41)
    at Object.run (/usr/local/lib/node_modules/pyret-npm/pyret-lang/build/phaseA/pyret.jarr:150863:7)
    at Object.runThunk (/usr/local/lib/node_modules/pyret-npm/pyret-lang/build/phaseA/pyret.jarr:151034:26)
    at tryQueue (/usr/local/lib/node_modules/pyret-npm/pyret-lang/build/phaseA/pyret.jarr:233157:23)
    at WebSocket.<anonymous> (/usr/local/lib/node_modules/pyret-npm/pyret-lang/build/phaseA/pyret.jarr:233200:15)
    at WebSocket.emit (events.js:198:13)
    at Receiver.receiverOnMessage (/usr/local/lib/node_modules/pyret-npm/node_modules/ws/lib/websocket.js:720:20)
    at Receiver.emit (events.js:198:13)
    at Receiver.dataMessage (/usr/local/lib/node_modules/pyret-npm/node_modules/ws/lib/receiver.js:414:14)
    at Receiver.getData (/usr/local/lib/node_modules/pyret-npm/node_modules/ws/lib/receiver.js:346:17)
    at Receiver.startLoop (/usr/local/lib/node_modules/pyret-npm/node_modules/ws/lib/receiver.js:133:22)
    at Receiver._write (/usr/local/lib/node_modules/pyret-npm/node_modules/ws/lib/receiver.js:69:10)
    at doWrite (_stream_writable.js:415:12)
    at writeOrBuffer (_stream_writable.js:399:5)
    at Receiver.Writable.write (_stream_writable.js:299:11)
    at Socket.socketOnData (/usr/local/lib/node_modules/pyret-npm/node_modules/ws/lib/websocket.js:795:35)
    at Socket.emit (events.js:198:13)
    at addChunk (_stream_readable.js:288:12)
    at readableAddChunk (_stream_readable.js:269:11)
    at Socket.Readable.push (_stream_readable.js:224:10)
    at Pipe.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)

[brandon-gong]

If this is not a pressing issue that needs to be resolved immediately, could I maybe try to take a crack at it? I'm probably underqualified right now but I would love to contribute to Pyret one day (if not now, I'll come back once I've got 0190 and 1730 under my belt), and this issue doesn't seem like it requires too much understanding of the internals.