promote self-hosted C tokenizer and parser to a general-purpose C tokenizer and parser in the standard library

[andrewrk]

Currently Zig has the file src-self-hosted/c_tokenizer.zig which is used to tokenize macros for translate-c purposes. It also has some functions in src-self-hosted/translate_c.zig which do AST parsing of C tokens:

• parseCPrimaryExpr
• parseCExpr
• parseCSuffixOpExpr
• etc

Zig has set a (in my opinion successful) precedent in exposing its own tokenizer, parser, and AST in the standard library, in these files:

• std/lib/zig/ast.zig
• std/lib/zig/parse.zig
• std/lib/zig/render.zig
• std/lib/zig/tokenizer.zig

Now it is time to do the same for C:

• Decouple C tokenization and AST building from translate-c
• Expose C tokenization and AST building (and rendering?) in std/lib/c/* in the same way as zig.
• Complete the implementation of C tokenization and AST building. Make it compatible with the specification as well as GCC extensions, and make it robust. Someone should be able to write a C compiler in zig comfortably by using the standard library C tokenizer and parser.
• translate-c should use this generic and robust C tokenizer and parser API, rather than its current, only-what-we-need-to-make-it-work implementation.

cc @Vexu

[Vexu]

I'm assuming we also want a preprocessor? Are we trying to eventually replace clang in translate-c completely?

[andrewrk]

That could be a direction to go with this, although I wouldn't call it "planned" yet. The main idea here is to make the API abstraction layer of the C tokenization and parsing clear and understandable. That will have the benefit of making it easier to test - there are plenty of pre-existing test cases for parsing C code, for example. So this will result in a more robust translate-c feature.

[andrewrk]

Vexu landed #4317 in 0cc2489. This started the efforts toward implementing this issue. What remains until this can be closed:

• set up testing for the parser to make it more contributor friendly
• integration with translate-c