Add Layering support for OpenAPI documents

[MaggieKimani1]

Feature description:
Currently, DevX API has been encountering performance bottlenecks within the API's capacity as a result of growth of Graph resources and the size of the OpenAPI file over time. In a bid to optimize performance and memory usage while adding support for different sovereign clouds, a new proposal targeting layering of OpenAPI documents was born.
Using this layering approach should also help optimize external reference resolution.

Possible solution:
Using a layering approach when dealing with different versions of OpenAPI documents e.g. v1-fairfax.yaml and beta-fairfax.yaml
To layer two OpenAPI documents, you can merge the content of both documents into a single OpenAPI document.

Example:

1. First document (petStore1.yaml)

openapi: 3.0.0
info:
  title: Document 1
  version: 1.0.0
paths:
  /pets:
    get:
      summary: Get all pets
      responses:
        200:
          description: Success

2. Second document (petStore2.yaml)

openapi: 3.0.0
info:
  title: Document 2
  version: 1.0.0
paths:
  /pets/{id}:
    get:
      summary: Get a pet's details
      responses:
        200:
          description: Success

3. Combined document

openapi: 3.0.0
info:
  title: Combined Document
  version: 1.0.0
paths:
  /pets:
    get:
      summary: Get all pets
      responses:
        200:
          description: Success
  /pets/{id}:
    get:
      summary: Get a pet's details
      responses:
        200:
          description: Success

Additional Context:
There are two possible approaches we're considering:

1. Using a collection that dedupes objects and shares memory between document object model graphs. We will however need to handle conflicts that may occur when both documents define the same elements (e.g., the same paths or properties) by removing the duplicate elements appearing in both documents. E.g.:

// merge 2 documents
petStore1.Attach(petStore2);

// Get DOM of the merged documents
var mergedInfo = petStore1.Info;

2. Adding a layer property to the object models which in memory will be represented as a collection e.g. a dictionary of type <string, OpenAPIElement> which can help us index into a particular document and its properties e.g.

// Get DOM of a specific layer
var petStore1Info = petStore1.Layer["petStore1"].Info;
var petStore2Info = petStore1.Layer["petStore2"].Info;

Cc: @baywet, @darrelmiller

[baywet]

Thanks for starting the conversation here Maggie. I think you've captured some of the scenarios and I'd like to go through those to make sure we're not forgetting anything before we design a solution.

One question I had was "do we need a layer to be able to remove things from a layer below? if so what should be the experience?"

Another thing Darrel brought up is that this layering mechanism might be able to also cover external needs, or we might at least be able to mutualize implementation layers. If we do so, what happens to the references? do we "remove" the document information as we internalize the references to make library users' lives easier? That'd put us at risk of having collisions (two documents might have the same component name), or do we use some kind of runtime convention to deduplicate them? At the risk of confusing the library user and/or not being idem-potent on read/write operations?

What should happen when schemas (inline or components) conflict between layers? (e.g. User has propertyA of type string in layer 1, but it's of type int in 2, or different set of properties? especially if it's used in both layers and the path using it in layer 1 doesn't exist in layer 2)

Do we want to be able to write to different layers? Even from the ether (we built the layers in memory, not parsing a document).

Do we want to be able to identify which layer is any information coming from?

[darrelmiller]

Based on recent conversations we are going to prototype an effort to do "de-duping" of objects in DevX API before we trying implementing this in OpenAPI.NET.