From dac1b32143c64ea2234d9d1ff12d488602e19aa1 Mon Sep 17 00:00:00 2001 From: Kai Jellinghaus Date: Mon, 8 Aug 2022 18:20:56 +0200 Subject: [PATCH 1/7] Update Docs --- .../for-contributors/generators/README.md | 8 ++-- .../for-contributors/generators/emitter.md | 3 -- .../generators/emitter/README.md | 5 ++ .../generators/emitter/visitor.md | 2 + .../for-contributors/generators/overloader.md | 3 -- .../for-contributors/generators/scraper.md | 38 ++------------- .../generators/shared-infrastructure.md | 46 ------------------- .../generators/symbol-layer/README.md | 9 ++++ .../generators/symbol-layer/symbol-visitor.md | 12 +++++ .../generators/symbol-layer/type-store.md | 9 ++++ .../for-contributors/generators/testing.md | 2 +- .../generators/type-resolution.md | 5 ++ 12 files changed, 50 insertions(+), 92 deletions(-) delete mode 100644 documentation/for-contributors/generators/emitter.md create mode 100644 documentation/for-contributors/generators/emitter/README.md delete mode 100644 documentation/for-contributors/generators/overloader.md delete mode 100644 documentation/for-contributors/generators/shared-infrastructure.md create mode 100644 documentation/for-contributors/generators/symbol-layer/README.md create mode 100644 documentation/for-contributors/generators/symbol-layer/symbol-visitor.md create mode 100644 documentation/for-contributors/generators/symbol-layer/type-store.md create mode 100644 documentation/for-contributors/generators/type-resolution.md diff --git a/documentation/for-contributors/generators/README.md b/documentation/for-contributors/generators/README.md index 710984539b..8975112a6d 100644 --- a/documentation/for-contributors/generators/README.md +++ b/documentation/for-contributors/generators/README.md @@ -5,9 +5,9 @@ SilkTouch is the Silk.NET project's C# native interoperability code generation s It does this by splitting the generation process into multiple phases, all anchored around the concept of a "shared infrastructure" - abstractions for SilkTouch generators which are agnostic of where they're running. Currently the only places SilkTouch can run is in a Roslyn Source Generator and using a `dotnet tool` Command Line Interface, but the idea is so long as no incompatibilities are introduced by each individual phase (such as being locked to .NET 6 instead of .NET Standard 2.0) the generator should be able to run in whatever form factor is desirable - the Shared Infrastructure will provide everything a generator could need for consuming a JSON configuration with C# input code and producing additional generated C# code as a result. Learn more about each individual cornerstone of the SilkTouch Stack: -- [Shared Infrastructure](shared-infrastructure.md) -- [Emitter](emitter.md) -- [Overloader](overloader.md) -- [Scraper](scraper.md) + +- [Symbol Layer](./symbol-layer/README.md) +- [Emitter](emitter.md) +- [Scraper](scraper.md) TODO: Add a link to the original proposal once the proposals folder has been reorganised according to the Software Development Plan. diff --git a/documentation/for-contributors/generators/emitter.md b/documentation/for-contributors/generators/emitter.md deleted file mode 100644 index f46d2b527d..0000000000 --- a/documentation/for-contributors/generators/emitter.md +++ /dev/null @@ -1,3 +0,0 @@ -# Emitter - -The Emitter. \ No newline at end of file diff --git a/documentation/for-contributors/generators/emitter/README.md b/documentation/for-contributors/generators/emitter/README.md new file mode 100644 index 0000000000..b71e350048 --- /dev/null +++ b/documentation/for-contributors/generators/emitter/README.md @@ -0,0 +1,5 @@ +# Emitter + +The emitter is responsible for creating Roslyn C# Code Symbols from our symbol layer. + +See [here](./about%20formatting.md) for how formatting works, and [here](./visitor.md) for details on the visitor implementation. diff --git a/documentation/for-contributors/generators/emitter/visitor.md b/documentation/for-contributors/generators/emitter/visitor.md index 39f7b62004..83e6fba60d 100644 --- a/documentation/for-contributors/generators/emitter/visitor.md +++ b/documentation/for-contributors/generators/emitter/visitor.md @@ -1,5 +1,7 @@ # Visitor +See [here](../symbol-layer/symbol-visitor.md) for details on how the base symbol visitor works. + To Transform our internal symbol layer to C# code a visitor (`CSharpEmitter.Visitor`) is used. ## State diff --git a/documentation/for-contributors/generators/overloader.md b/documentation/for-contributors/generators/overloader.md deleted file mode 100644 index 427bb135ad..0000000000 --- a/documentation/for-contributors/generators/overloader.md +++ /dev/null @@ -1,3 +0,0 @@ -# Overloader - -The Overloader. \ No newline at end of file diff --git a/documentation/for-contributors/generators/scraper.md b/documentation/for-contributors/generators/scraper.md index 6eb51ac3c1..29ec4dc81f 100644 --- a/documentation/for-contributors/generators/scraper.md +++ b/documentation/for-contributors/generators/scraper.md @@ -1,39 +1,7 @@ # Scraper -The SilkTouch Scraper is responsible for walking through the abstract syntax tree (AST) of a C/C++ header and all the referenced headers therein to generate C# bindings to as much of this as possible where the configuration indicates this is desirable. Only certain headers will have bindings generated for them, which can be specified by providing the paths to the headers to "traverse" in the JSON configuration. +The scraper calls into clang to generate XML and can scrape this XML for Symbols of the [Symbol Layer](./symbol-layer/README.md) -## ClangSharp +## Scraping XML -To walk through the C/C++ AST, ClangSharp's P/Invoke Generator is used, which is a proven tool for generating C# bindings to C/C++ APIs published by Microsoft. It does this by using the clang compiler to walk through the AST. - -ClangSharp's P/Invoke Generator is capable of producing C# bindings on its own, however these are raw bindings that use DllImport and don't aim to lift the bindings to be more C#-friendly. However, as part of Scraper development work was done to add an "XML output mode" to ClangSharp which instead of producing pure C# code, it outputs XML which roughly represents what the C# code would've looked like. - -## The Subagent - -ClangSharp uses libclang, which has funny rules over multithreading. To maintain a degree of parallelism during generation, the invocation of ClangSharp was moved to be done in a separate process. The original intention was to invoke the `ClangSharpPInvokeGenerator` tool as shipped publicly, but unfortunately `dotnet tool`s cannot be referenced by other libraries or `dotnet tool`s. As such, we created our own shim which mimics the behaviour of and duplicates a lot of code from the `ClangSharpPInvokeGenerator` tool, but within the same CLI executable. - -### The `ISubagent` abstraction - -`ISubagent` is responsible for launching the ClangSharp tool with certain parameters, inputs, and configuration. This is an abstraction such that the main Scraper assembly remains easily portable to another form factor should we enable the Scraper to run in another form factor in the future, as well as to prevent the library assembly knowing too much about the assembly in which its executing. - -The only implementation of this interface is the `ClangSharpSubagent` class in the `SilkTouch` assembly (CLI implementation of the Shared Infrastructure) which gathers information on the currently executing assembly, and relaunches it with `clangsharp` as the first argument and a JSON-encoded record as the second. - -### The `silktouch clangsharp` command - -This is the command that invokes ClangSharp as described in the previous section. This deserializes the JSON-encoded record and passes the data held therein to the `PInvokeGenerator` class within ClangSharp. This will generate the XML dump and then quit, possibly with some ClangSharp-generated diagnostics and a non-zero error code if those diagnostics are severe. The implementation of this is in the `ClangSharpHandoff` class. - -## XML Mods - -Once an initial XML dump has been generated by The Subagent, the XML dump will be parsed by the `Silk.NET.SilkTouch.ClangSharp.Xml` library and produce a tree of records. This is done in the `ScraperGenerator` ahead of the XML mods stage. - -The purpose of the XML mods stage is to walk this tree of records, and modify it depending on the project configuration (for example changing all names to be PascalCase, or adding custom attributes to cause the Overloader to generate certain overloads). Each mod must be specifically enabled, and has free range over the XML. As such, they execute in the order in which they were specified in the project configuration. - -Each XML mod has access to the `ScraperJobConfiguration.ModOptions` and the full XML tree. It does not have any access to resources beyond that, such as the SilkTouch Context. - -TODO: Write words here about each XML mod once XML mods have been implemented - -## Transformation - -After the XML mods have been applied, the transformation stage is responsible for converting the bindings represented by the XML tree back into C# code that can be output via the SilkTouch Context (and added to the project compilation). Unlike ClangSharp, this stage will output bindings which are compatible with/invoke the Emitter and Overloader instead of using DllImport directly. - -TODO: Write more words here once this stage is implemented. +To scrape XML there is a visitor that visits each XML node and outputs several symbols per node. diff --git a/documentation/for-contributors/generators/shared-infrastructure.md b/documentation/for-contributors/generators/shared-infrastructure.md deleted file mode 100644 index fe2d3327d8..0000000000 --- a/documentation/for-contributors/generators/shared-infrastructure.md +++ /dev/null @@ -1,46 +0,0 @@ -# Shared Infrastructure - -The Shared Infrastructure contains common code and abstractions for bootstrapping and running generators within the SilkTouch Stack. It aims to ensure that SilkTouch generators are easily portable, and have a unified interface over multiple generation mediums such that no extra code is required in individual generators to accomodate new form factors. - -## "Form Factor" - -A "Form Factor", referred to throughout the documentation, is simply a distinct type of application which invokes the SilkTouch Stack via the Shared Infrastructure (i.e. "it's a place where SilkTouch runs"). Distinct types such as Roslyn Source Generators and Command Line Interfaces, which are the two first-party form factors supported today. - -## Configuration - -All JSON configuration structures are represented by C# records in the Silk.NET.SilkTouch.Configuration namespace (in the Configuration folder of Silk.NET.SilkTouch.Common). - -### Project Configuration - -Each project has its own JSON configuration file which defines what each generator does for that particular project. This file should be added by the user as an `AdditionalFiles` file, with the name `silktouch.json` unless the `silktouch_config_file` `.editorconfig` option in the source generator format factor or the `config` option in the CLI form factor are used to override this with a new file name or file path (either absolute or relative). - -Each generator has its own specific configuration structure, which is outlined in XML documentation in the `Config.cs` file as well as the markdown documents for each individual generator. - -The project configuration also provides a (relative or absolute) path to a file containing the global configuration. - -### Global Configuration - -The global configuration contains variables which aren't specific to a particular project or generation process/job. Stuff like the License Header, which doesn't need to be duplicated across every single project configuration. - -### CLI Configuration -The CLI has 3 ways to be configured: -- Command line parameters, possible formats are "key1=value1 --key2=value2 /key3=value3 --key4 value4 /key5 value5" -- A JSON file, with the path taken either from the config option `config` or by default silktouch.json (in the working directory) -- Environment Variables, prefixed with SilkTouch_ (to configure hierarchical data, use two underscores like `set SilkTouch_Parent__Root=value`) -The CLI uses the Microsoft.Extensions.Configuration package for this, for more information on how to use the above options see [here](https://docs.microsoft.com/en-us/aspnet/core/fundamentals/configuration/) (note that we do not use user secrets) - -## Generation - -### SilkTouch "Context" - -The SilkTouch "Context" (represented by the `SilkTouchContext` class in the `Silk.NET.SilkTouch.Generation` namespace, or Generation folder of Silk.MET.SilkTouch.Common) encapsulates all inputs and outputs passed into and retrieved from a generator running atop the Shared Infrastructure. It exists to isolate each individual generator from one another, and provide one unified data structure containing everything a generator might reasonably need to access about a project. - -Some SilkTouch Contexts have access to outputs generated by other generators in the `AdditionalSyntaxTrees` property. To ease context consumption, there is a `AllSyntaxTrees` property which will contain both the Roslyn `Compilation`'s syntax trees as well as the newly-added syntax trees. Currently this is only the Emitter and Overloader if the Scraper has ran before them, though it is intended that the Shared Infrastructure will be expanded to have a full-blown dependency resolution mechanism to allow more generator modularity and dynamic definition of a generator's soft or hard dependencies. - -### The Generator Object - -The Generator Object (represented by the `SilkTouchGenerator` class) is responsible for the consumption of basic project information, and morphing/storing this information in a way that allows for Contexts to be created from the generator object. After a context is created, it will be consumed by a generator, and then once again "ingested" by the generator object. This generator object forwards the results to bindable events, which are subscribed to by the form factor. - -The intention of this is to reduce duplicated code between the two form factors we maintain, and provide an interface to ease the bootstrapping of SilkTouch code generation - this class significantly reduces the amount of work required to implement another form factor. - -Currently there are no methods on `SilkTouchGenerator` which can create a `SilkTouchContext` directly, nor are there APIs which nicely facilitate the passing of generated outputs to a stage depending on them. Both of which are things we'd like to address in the future. \ No newline at end of file diff --git a/documentation/for-contributors/generators/symbol-layer/README.md b/documentation/for-contributors/generators/symbol-layer/README.md new file mode 100644 index 0000000000..b4984e159e --- /dev/null +++ b/documentation/for-contributors/generators/symbol-layer/README.md @@ -0,0 +1,9 @@ +# Symbol Layer + +The symbol layer is a set of types in the Silk.NET.SilkTouch.Symbols. +They all inherit from the central Silk.NET.SilkTouch.Symbols.Symbol type. + +The symbol layer is intended as a way to represent data passed between components of SilkTouch +The symbol layer is entirely immutable, with the only exception being the [Type Store](./type-store.md). + +The way to update & iterate symbols is using a [Symbol Visitor](./symbol-visitor.md). diff --git a/documentation/for-contributors/generators/symbol-layer/symbol-visitor.md b/documentation/for-contributors/generators/symbol-layer/symbol-visitor.md new file mode 100644 index 0000000000..d4e0b12651 --- /dev/null +++ b/documentation/for-contributors/generators/symbol-layer/symbol-visitor.md @@ -0,0 +1,12 @@ +# Symbol Visitor + +A symbol visitor visits each symbol and then calls the appropriate methods to recursively visit all parts. + +## Updating Symbols + +Each method returns the same type it gets as parameter, so prefer to override the most specific method possible. +For example, if you wish to rename all `TypeSymbol`s, override `VisitTypeSymbol`. But if you want to add a field to every struct, override `VisitStructSymbol`. If you want to rewrite the type of a symbol, for example generate a class for every struct, again, override the most specific method that is compatible with both (in this case `VisitTypeSymbol`) and do type checks as necessary. + +## Managing State + +While the base `SymbolVisitor` is stateless, derived types are free to introduce (mutable) state. This allow propagating state other then the new symbol up the tree. diff --git a/documentation/for-contributors/generators/symbol-layer/type-store.md b/documentation/for-contributors/generators/symbol-layer/type-store.md new file mode 100644 index 0000000000..ae6bc0ec3d --- /dev/null +++ b/documentation/for-contributors/generators/symbol-layer/type-store.md @@ -0,0 +1,9 @@ +# Type Store + +The type store is a simple helper to resolve type identifier to the latest instance. +It's available from inside every [Symbol Visitor](./symbol-visitor.md). + +## Updating + +The type store needs updating when changing a type. Usually [Symbol Visitors](./symbol-visitor) will do this automatically after visiting a type (`VisitType`), but inheritors overriding this should do this manually using +`Store(TypeSymbol)`. diff --git a/documentation/for-contributors/generators/testing.md b/documentation/for-contributors/generators/testing.md index 740e0b2c7c..c4c86f36c4 100644 --- a/documentation/for-contributors/generators/testing.md +++ b/documentation/for-contributors/generators/testing.md @@ -25,6 +25,6 @@ Tests should be written covering XML -> Symbol conversion. No C(++) -> XML tests Tests should be written covering Symbol -> C# conversion. -### Integration +## Integration Tests Integration Tests should exist to cover most if not all areas of the generator. These are simply a sanity check & samples of what the generator can do. Unit Tests are still required. diff --git a/documentation/for-contributors/generators/type-resolution.md b/documentation/for-contributors/generators/type-resolution.md new file mode 100644 index 0000000000..8964b68628 --- /dev/null +++ b/documentation/for-contributors/generators/type-resolution.md @@ -0,0 +1,5 @@ +# Type Resolution + +The resolution code lives in Silk.NET.SilkTouch.TypeResolution and is concerned with resolving `UnresolvedTypeReference` to actual `TypeReference`s. + +We do not document the individual parts here, please check the XML docs of each of the classes. From dee37a4d919c3ec991c5f930457d83db615bd3ac Mon Sep 17 00:00:00 2001 From: Kai Jellinghaus Date: Mon, 8 Aug 2022 18:35:13 +0200 Subject: [PATCH 2/7] Template Symbol docs --- .../generators/symbol-layer/README.md | 2 ++ .../generators/symbol-layer/symbols/README.md | 29 +++++++++++++++++++ .../symbol-layer/symbols/template.md | 22 ++++++++++++++ 3 files changed, 53 insertions(+) create mode 100644 documentation/for-contributors/generators/symbol-layer/symbols/README.md create mode 100644 documentation/for-contributors/generators/symbol-layer/symbols/template.md diff --git a/documentation/for-contributors/generators/symbol-layer/README.md b/documentation/for-contributors/generators/symbol-layer/README.md index b4984e159e..59ede01741 100644 --- a/documentation/for-contributors/generators/symbol-layer/README.md +++ b/documentation/for-contributors/generators/symbol-layer/README.md @@ -7,3 +7,5 @@ The symbol layer is intended as a way to represent data passed between component The symbol layer is entirely immutable, with the only exception being the [Type Store](./type-store.md). The way to update & iterate symbols is using a [Symbol Visitor](./symbol-visitor.md). + +See [symbols](./symbols/README.md) for a list of all symbols, their visitor method, test details, etc. diff --git a/documentation/for-contributors/generators/symbol-layer/symbols/README.md b/documentation/for-contributors/generators/symbol-layer/symbols/README.md new file mode 100644 index 0000000000..e274d661fd --- /dev/null +++ b/documentation/for-contributors/generators/symbol-layer/symbols/README.md @@ -0,0 +1,29 @@ +# Symbols + +This document / folder is for tracking all the available symbols, and documenting what creation of one entails. + +## List + +| Name | Symbol Layer File | Symbol Layer Tests | Emitter Tests | +| ---- | ----------------- | ------------------ | ------------- | + +## How to create a symbol + +Checklist: + +- [ ] Write a short description of what this symbol does in the above list. +- [ ] Add the symbol to Silk.NET.SilkTouch.Symbols +- [ ] Add a partial of SymbolVisitor to below the definition, and add visiting method in there. +- [ ] Call the new visiting method either from + - The root Visit(Symbol) method, this should rarely be needed + - The parent's visit method +- [ ] Add Tests to Silk.NET.SilkTouch.Symbols.Tests for + - [ ] The type being visited with it's own visiting method + - [ ] The type being visited with it's parent (not needed if this is a new root type, with no parent) + - [ ] Each part being visited correctly. Only shallowly check this, don't check parts of parts. +- [ ] Handle symbol in Silk.NET.SilkTouch.Emitter +- [ ] Add Tests to Silk.NET.SilkTouch.Emitter.Tests + - [ ] Transforming a sample Symbol matching a specific string + - [ ] Other tests, this is highly dependend on how complex the C# output looks like + - [ ] More is better! +- [ ] You're done. Feel free to use the symbol wherever needed. Prefer to add usages in a separate PR. diff --git a/documentation/for-contributors/generators/symbol-layer/symbols/template.md b/documentation/for-contributors/generators/symbol-layer/symbols/template.md new file mode 100644 index 0000000000..7898e04741 --- /dev/null +++ b/documentation/for-contributors/generators/symbol-layer/symbols/template.md @@ -0,0 +1,22 @@ +(This is a template for other symbol docs) + +# My Symbol + +Checklist: + +- [ ] Write a short description of what this symbol does here. +- [ ] Add the symbol to Silk.NET.SilkTouch.Symbols +- [ ] Add a partial of SymbolVisitor to below the definition, and add visiting method in there. +- [ ] Call the new visiting method either from + - The root Visit(Symbol) method, this should rarely be needed + - The parent's visit method +- [ ] Add Tests to Silk.NET.SilkTouch.Symbols.Tests for + - [ ] The type being visited with it's own visiting method + - [ ] The type being visited with it's parent (not needed if this is a new root type, with no parent) + - [ ] Each part being visited correctly. Only shallowly check this, don't check parts of parts. +- [ ] Handle symbol in Silk.NET.SilkTouch.Emitter +- [ ] Add Tests to Silk.NET.SilkTouch.Emitter.Tests + - [ ] Transforming a sample Symbol matching a specific string + - [ ] Other tests, this is highly dependend on how complex the C# output looks like + - [ ] More is better! +- [ ] You're done. Feel free to use the symbol wherever needed. Prefer to add usages in a separate PR. From c47b14fbd2596f129b67ff88de8e0636438d280d Mon Sep 17 00:00:00 2001 From: Kai Jellinghaus Date: Mon, 8 Aug 2022 18:46:50 +0200 Subject: [PATCH 3/7] Add List of Symbols --- .../generators/symbol-layer/symbols/README.md | 23 +++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/documentation/for-contributors/generators/symbol-layer/symbols/README.md b/documentation/for-contributors/generators/symbol-layer/symbols/README.md index e274d661fd..4527ba7f80 100644 --- a/documentation/for-contributors/generators/symbol-layer/symbols/README.md +++ b/documentation/for-contributors/generators/symbol-layer/symbols/README.md @@ -4,8 +4,27 @@ This document / folder is for tracking all the available symbols, and documentin ## List -| Name | Symbol Layer File | Symbol Layer Tests | Emitter Tests | -| ---- | ----------------- | ------------------ | ------------- | +(Order alphabetically please!) + +Parent Symbols (Unlisted, abstract): +| Name | +| ---- | +| MemberSymbol | +| Symbol | +| TypeReference | +| TypeSymbol | +| + +| Name | Symbol Layer File | Symbol Layer Tests | Emitter Tests | +| ----------------------- | ----------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | +| ExternalTypeReference | [here](../../../../../src/generators/Silk.NET.SilkTouch.Symbols/ExternalTypeReference.cs) | [here](../../../../../tests/Silk.NET.SilkTouch.Symbols.Tests/SymbolVisitorTests/ExternalTypeReferenceTests.cs) | TODO!!! | +| FieldSymbol | [here](../../../../../src/generators/Silk.NET.SilkTouch.Symbols/FieldSymbol.cs) | [here](../../../../../tests/Silk.NET.SilkTouch.Symbols.Tests/SymbolVisitorTests/FieldTests.cs) | [here](../../../../../tests/Silk.NET.SilkTouch.Emitter.Tests/EmitterFieldTests.cs) | +| IdentifierSymbol | [here](../../../../../src/generators/Silk.NET.SilkTouch.Symbols/IdentifierSymbol.cs) | [here](../../../../../tests/Silk.NET.SilkTouch.Symbols.Tests/SymbolVisitorTests/IdentifierTests.cs) | TODO!!! | +| InternalTypeReference | [here](../../../../../src/generators/Silk.NET.SilkTouch.Symbols/InternalTypeReference.cs) | TODO!!! | TODO!!! | +| NamespaceSymbol | [here](../../../../../src/generators/Silk.NET.SilkTouch.Symbols/NamespaceSymbol.cs) | [here](../../../../../tests/Silk.NET.SilkTouch.Symbols.Tests/SymbolVisitorTests/NamespaceTests.cs) | [here](../../../../../tests/Silk.NET.SilkTouch.Emitter.Tests/EmitterNamespaceTests.cs) | +| PointerTypeReference | [here](../../../../../src/generators/Silk.NET.SilkTouch.Symbols/PointerTypeReference.cs) | [here](../../../../../tests/Silk.NET.SilkTouch.Symbols.Tests/SymbolVisitorTests/PointerTypeReferenceTests.cs) | TODO!!! | +| StructSymbol | [here](../../../../../src/generators/Silk.NET.SilkTouch.Symbols/StructSymbol.cs) | [here](../../../../../tests/Silk.NET.SilkTouch.Symbols.Tests/SymbolVisitorTests/StructTests.cs) | [here](../../../../../tests/Silk.NET.SilkTouch.Emitter.Tests/EmitterStructTests.cs) | +| UnresolvedTypeReference | [here](src/generators/Silk.NET.SilkTouch.Symbols/UnresolvedTypeReference.cs) | [here](tests/Silk.NET.SilkTouch.Symbols.Tests/SymbolVisitorTests/UnresolvedTypeReferenceTests.cs) | - | ## How to create a symbol From 5fb99265ea3fb782c6dc8ade7614efb5ff597fab Mon Sep 17 00:00:00 2001 From: Kai Jellinghaus Date: Mon, 8 Aug 2022 18:53:41 +0200 Subject: [PATCH 4/7] Pass 2 of docs --- .../for-contributors/generators/README.md | 2 +- .../generators/emitter/README.md | 2 ++ .../for-contributors/generators/scraper.md | 6 +++++ .../generators/symbol-layer/README.md | 6 ++--- .../symbol-layer/symbols/template.md | 22 ------------------- .../for-contributors/generators/testing.md | 2 ++ .../generators/type-resolution.md | 2 +- 7 files changed, 14 insertions(+), 28 deletions(-) delete mode 100644 documentation/for-contributors/generators/symbol-layer/symbols/template.md diff --git a/documentation/for-contributors/generators/README.md b/documentation/for-contributors/generators/README.md index 8975112a6d..c5b04edb01 100644 --- a/documentation/for-contributors/generators/README.md +++ b/documentation/for-contributors/generators/README.md @@ -2,7 +2,7 @@ SilkTouch is the Silk.NET project's C# native interoperability code generation stack. It contains useful generators which together generate the majority of the library's codebase. -It does this by splitting the generation process into multiple phases, all anchored around the concept of a "shared infrastructure" - abstractions for SilkTouch generators which are agnostic of where they're running. Currently the only places SilkTouch can run is in a Roslyn Source Generator and using a `dotnet tool` Command Line Interface, but the idea is so long as no incompatibilities are introduced by each individual phase (such as being locked to .NET 6 instead of .NET Standard 2.0) the generator should be able to run in whatever form factor is desirable - the Shared Infrastructure will provide everything a generator could need for consuming a JSON configuration with C# input code and producing additional generated C# code as a result. +It does this by centering all parts around a [common symbol layer](./symbol-layer/README.md), which allows other parts to be used in any configuration deemed useful, parts don't have a way other then the symbol layer to exchange data, enforcing interoperability. Learn more about each individual cornerstone of the SilkTouch Stack: diff --git a/documentation/for-contributors/generators/emitter/README.md b/documentation/for-contributors/generators/emitter/README.md index b71e350048..fe733a1755 100644 --- a/documentation/for-contributors/generators/emitter/README.md +++ b/documentation/for-contributors/generators/emitter/README.md @@ -3,3 +3,5 @@ The emitter is responsible for creating Roslyn C# Code Symbols from our symbol layer. See [here](./about%20formatting.md) for how formatting works, and [here](./visitor.md) for details on the visitor implementation. + +In general the emitter tries it's best to be very basic doing only the job of 1:1 translating shared symbol layer into C# roslyn symbols. diff --git a/documentation/for-contributors/generators/scraper.md b/documentation/for-contributors/generators/scraper.md index 29ec4dc81f..283862ebaa 100644 --- a/documentation/for-contributors/generators/scraper.md +++ b/documentation/for-contributors/generators/scraper.md @@ -5,3 +5,9 @@ The scraper calls into clang to generate XML and can scrape this XML for Symbols ## Scraping XML To scrape XML there is a visitor that visits each XML node and outputs several symbols per node. +This is what happens in `XmlVisitor.cs` + +## Generating XML using clang + +This is what happens in `ClangScraper.cs`, which is configured using `ClangScraperConfiguration.cs` +Check XML docs for details on what methods do. diff --git a/documentation/for-contributors/generators/symbol-layer/README.md b/documentation/for-contributors/generators/symbol-layer/README.md index 59ede01741..b44cda79cd 100644 --- a/documentation/for-contributors/generators/symbol-layer/README.md +++ b/documentation/for-contributors/generators/symbol-layer/README.md @@ -1,10 +1,8 @@ # Symbol Layer -The symbol layer is a set of types in the Silk.NET.SilkTouch.Symbols. -They all inherit from the central Silk.NET.SilkTouch.Symbols.Symbol type. +The symbol layer is a set of types in the Silk.NET.SilkTouch.Symbols, all inheriting from the central Silk.NET.SilkTouch.Symbols.Symbol type. -The symbol layer is intended as a way to represent data passed between components of SilkTouch -The symbol layer is entirely immutable, with the only exception being the [Type Store](./type-store.md). +The symbol layer is intended as a way to represent data passed between components of SilkTouch and is entirely immutable, with the only exception being the [Type Store](./type-store.md). The way to update & iterate symbols is using a [Symbol Visitor](./symbol-visitor.md). diff --git a/documentation/for-contributors/generators/symbol-layer/symbols/template.md b/documentation/for-contributors/generators/symbol-layer/symbols/template.md deleted file mode 100644 index 7898e04741..0000000000 --- a/documentation/for-contributors/generators/symbol-layer/symbols/template.md +++ /dev/null @@ -1,22 +0,0 @@ -(This is a template for other symbol docs) - -# My Symbol - -Checklist: - -- [ ] Write a short description of what this symbol does here. -- [ ] Add the symbol to Silk.NET.SilkTouch.Symbols -- [ ] Add a partial of SymbolVisitor to below the definition, and add visiting method in there. -- [ ] Call the new visiting method either from - - The root Visit(Symbol) method, this should rarely be needed - - The parent's visit method -- [ ] Add Tests to Silk.NET.SilkTouch.Symbols.Tests for - - [ ] The type being visited with it's own visiting method - - [ ] The type being visited with it's parent (not needed if this is a new root type, with no parent) - - [ ] Each part being visited correctly. Only shallowly check this, don't check parts of parts. -- [ ] Handle symbol in Silk.NET.SilkTouch.Emitter -- [ ] Add Tests to Silk.NET.SilkTouch.Emitter.Tests - - [ ] Transforming a sample Symbol matching a specific string - - [ ] Other tests, this is highly dependend on how complex the C# output looks like - - [ ] More is better! -- [ ] You're done. Feel free to use the symbol wherever needed. Prefer to add usages in a separate PR. diff --git a/documentation/for-contributors/generators/testing.md b/documentation/for-contributors/generators/testing.md index c4c86f36c4..7c8fa690aa 100644 --- a/documentation/for-contributors/generators/testing.md +++ b/documentation/for-contributors/generators/testing.md @@ -4,6 +4,8 @@ SilkTouch Generators should be extensively Tested. ## Unit Tests +See also [symbol creation instructions](./symbol-layer/symbols/README.md) for details on symbol unit tests. + ### Traits All Unit & Integration tests should be flagged with the appropriate Category (Emitter, Scraper, Symbols, Integration) using `Trait("Category", "MyCategory")`. diff --git a/documentation/for-contributors/generators/type-resolution.md b/documentation/for-contributors/generators/type-resolution.md index 8964b68628..6f8cbd3840 100644 --- a/documentation/for-contributors/generators/type-resolution.md +++ b/documentation/for-contributors/generators/type-resolution.md @@ -2,4 +2,4 @@ The resolution code lives in Silk.NET.SilkTouch.TypeResolution and is concerned with resolving `UnresolvedTypeReference` to actual `TypeReference`s. -We do not document the individual parts here, please check the XML docs of each of the classes. +As the individual parts are only loosely connected and aren't generally dependent on each other, we do not document the individual parts here, please check the XML docs of each of the classes. From f27d88f48f744201d26cb4498de8a6394fb4af24 Mon Sep 17 00:00:00 2001 From: Kai Jellinghaus Date: Mon, 8 Aug 2022 18:55:11 +0200 Subject: [PATCH 5/7] Add info on SymbolVisitor --- .../generators/symbol-layer/symbols/README.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/documentation/for-contributors/generators/symbol-layer/symbols/README.md b/documentation/for-contributors/generators/symbol-layer/symbols/README.md index 4527ba7f80..bb5607c993 100644 --- a/documentation/for-contributors/generators/symbol-layer/symbols/README.md +++ b/documentation/for-contributors/generators/symbol-layer/symbols/README.md @@ -2,6 +2,10 @@ This document / folder is for tracking all the available symbols, and documenting what creation of one entails. +## Relation to SymbolVisitor + +In general each symbol listed below should have a corresponding method `VisitMySymbol` the only exception being `UnresolvedTypeReference` as only few visitors should ever interact with it. + ## List (Order alphabetically please!) @@ -13,7 +17,6 @@ Parent Symbols (Unlisted, abstract): | Symbol | | TypeReference | | TypeSymbol | -| | Name | Symbol Layer File | Symbol Layer Tests | Emitter Tests | | ----------------------- | ----------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | From 0cbf5359a2112bd5e72c1486a53b43a76c835e13 Mon Sep 17 00:00:00 2001 From: Kai Jellinghaus Date: Tue, 9 Aug 2022 11:46:01 +0200 Subject: [PATCH 6/7] Add details on Type Store --- .../generators/symbol-layer/symbol-visitor.md | 7 +++++++ .../for-contributors/generators/symbol-layer/type-store.md | 6 ++++++ 2 files changed, 13 insertions(+) diff --git a/documentation/for-contributors/generators/symbol-layer/symbol-visitor.md b/documentation/for-contributors/generators/symbol-layer/symbol-visitor.md index d4e0b12651..a1c3b97d58 100644 --- a/documentation/for-contributors/generators/symbol-layer/symbol-visitor.md +++ b/documentation/for-contributors/generators/symbol-layer/symbol-visitor.md @@ -1,12 +1,19 @@ # Symbol Visitor A symbol visitor visits each symbol and then calls the appropriate methods to recursively visit all parts. +Each symbol visitor has access to a [type store](./type-store.md). It has to be provided in the constructor of every symbol visitor. +It's generally adviced to simply bubble up this constructor parameter to users of your type. ## Updating Symbols Each method returns the same type it gets as parameter, so prefer to override the most specific method possible. For example, if you wish to rename all `TypeSymbol`s, override `VisitTypeSymbol`. But if you want to add a field to every struct, override `VisitStructSymbol`. If you want to rewrite the type of a symbol, for example generate a class for every struct, again, override the most specific method that is compatible with both (in this case `VisitTypeSymbol`) and do type checks as necessary. +### Type IDs + +See [type store](./type-store.md) for details on what type Ids are. +**Do not change type IDs when visiting symbols. This breaks all references to that type.** + ## Managing State While the base `SymbolVisitor` is stateless, derived types are free to introduce (mutable) state. This allow propagating state other then the new symbol up the tree. diff --git a/documentation/for-contributors/generators/symbol-layer/type-store.md b/documentation/for-contributors/generators/symbol-layer/type-store.md index ae6bc0ec3d..1f63fc658b 100644 --- a/documentation/for-contributors/generators/symbol-layer/type-store.md +++ b/documentation/for-contributors/generators/symbol-layer/type-store.md @@ -3,6 +3,12 @@ The type store is a simple helper to resolve type identifier to the latest instance. It's available from inside every [Symbol Visitor](./symbol-visitor.md). +## Type IDs + +`TypeId`s are a small `struct` intended to help reference types across trees. +This is useful because all symbols are immutable, and this makes it impossible to have loops in the symbol tree. +To allow loops, the type store is mutable and resolves each type id to the latest immutable version. + ## Updating The type store needs updating when changing a type. Usually [Symbol Visitors](./symbol-visitor) will do this automatically after visiting a type (`VisitType`), but inheritors overriding this should do this manually using From c61ddf66342bdb471cbfd457f6d219536c50939d Mon Sep 17 00:00:00 2001 From: Kai Jellinghaus Date: Tue, 9 Aug 2022 12:20:39 +0200 Subject: [PATCH 7/7] Add info on type references --- .../generators/symbol-layer/README.md | 2 ++ .../generators/symbol-layer/symbols/README.md | 12 ++++++------ .../generators/symbol-layer/type-references.md | 14 ++++++++++++++ 3 files changed, 22 insertions(+), 6 deletions(-) create mode 100644 documentation/for-contributors/generators/symbol-layer/type-references.md diff --git a/documentation/for-contributors/generators/symbol-layer/README.md b/documentation/for-contributors/generators/symbol-layer/README.md index b44cda79cd..028353bce1 100644 --- a/documentation/for-contributors/generators/symbol-layer/README.md +++ b/documentation/for-contributors/generators/symbol-layer/README.md @@ -7,3 +7,5 @@ The symbol layer is intended as a way to represent data passed between component The way to update & iterate symbols is using a [Symbol Visitor](./symbol-visitor.md). See [symbols](./symbols/README.md) for a list of all symbols, their visitor method, test details, etc. + +See [type references](./type-references.md) for an explanation of type references. diff --git a/documentation/for-contributors/generators/symbol-layer/symbols/README.md b/documentation/for-contributors/generators/symbol-layer/symbols/README.md index bb5607c993..bfd848e009 100644 --- a/documentation/for-contributors/generators/symbol-layer/symbols/README.md +++ b/documentation/for-contributors/generators/symbol-layer/symbols/README.md @@ -11,12 +11,12 @@ In general each symbol listed below should have a corresponding method `VisitMyS (Order alphabetically please!) Parent Symbols (Unlisted, abstract): -| Name | -| ---- | -| MemberSymbol | -| Symbol | -| TypeReference | -| TypeSymbol | +| Name | See Also | +| ---- | -------- | +| MemberSymbol | | +| Symbol | | +| TypeReference | [here](../type-references.md) | +| TypeSymbol | | | Name | Symbol Layer File | Symbol Layer Tests | Emitter Tests | | ----------------------- | ----------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | diff --git a/documentation/for-contributors/generators/symbol-layer/type-references.md b/documentation/for-contributors/generators/symbol-layer/type-references.md new file mode 100644 index 0000000000..a17f3c482d --- /dev/null +++ b/documentation/for-contributors/generators/symbol-layer/type-references.md @@ -0,0 +1,14 @@ +# Type References + +Type references are a special kind of symbol used to reference types. + +There are several kinds of type references. +During scraping (see [scraper](../scraper.md)) `UnresolvedTypeReference`s are created. +These should be resolved right after, in a process called [type resolution](../type-resolution.md). + +Type resolution leads to a variety of more specialized type references, with the two base ones being + +- `InternalTypeReference`, referencing another newly defined type somewhere in the tree. (Note that it's generally not adviced to define types this way, they should already occur somewhere higher up in the tree) +- `ExternalTypeReference`, referencing some external type, usually well known types from the standard library + +There are several other type reference types, but they are wrappers around other references, for example `PointerTypeReference`.