Handling Large OpenAPI Definitions

[ggpwnkthx]

I'm trying to work with some rather large OpenAPI files (specifically this one Microsoft Graph OpenAPI). Since the majority of that file is unnecessary to load at runtime (especially when I may only need one endpoint for a given process/instance), I wrote some code that essentially indexes the components and paths into a zip file as JSON data, as well as some code to generate an OpenAPI file based the desired paths and any $refs.

This works well as it only takes about a minute to generate the zip file and less than a second to compile only the necessary assets. That said, sometimes I'm still left with definitions that can be 100k-200k lines long (in YAML format). The method that seems to take the longest to process is _init_schema_types

aiopenapi3/aiopenapi3/openapi.py

Line 467 in 0a0789c

def _init_schema_types(self):

I was going to dive into this and see what I could improve on, but figured I'd touch base to see if you had any thoughts.

[commonism]

Hi,

dealing with large description documents?
Been there already.

But 56MB is … interesting.

First - convert to json, test loading speed with orjson, pycsimdjson, ijson
import … as json in loader.py should do.
pycsimdjson should be great.
Even if you stick to stdlib json, it's much faster than yaml

Second - as you already do, limit models to the models you need.

    class SchemaSelector(Init):
        """
        remove the schemas we do not need models for
        """

        def __init__(self, *schemas):
            super().__init__()
            self.schemas = schemas

        def schema(self, ctx: "Init.Context") -> "Init.Context":
            ctx.schema = {k: ctx.schema[k] for k in (set(self.schemas) & set(ctx.schema.keys()))}
            return ctx

The logic will resolve required/related/linked models - as your own logic does.

remove pathes you do not need

    class RemovePaths(Document):
        def parsed(self, ctx: "Document.Context") -> "Document.Context":
            """
            emtpy the paths - not needed
            """
            ctx.document["paths"] = {}
            return ctx

src

Last, do it "once" - once you are satisfied with the result, use cache_store/cache_load.
It cuts off the parsing of the yaml/json, and reduces the init phase to model class creation (_init_schema_types).

You can profile easily.

aiopenapi3 --profile-file /tmp/ms-graph.profile validate https://raw.githubusercontent.com/microsoftgraph/msgraph-metadata/master/openapi/beta/openapi.yaml

flameprof /tmp/msgraph-metadata-openapi.profile > ~/Downloads/msgraph.svg
firefox ~/Downloads/msgraph.svg

Currently does not load …

aiopenapi3.errors.OperationParameterValidationError: ("/applications(appId='{appId}')", 'delete', 'applications.application.DeleteApplicationByAppId', 'Parameter name not found in path: uniqueName')

      - name: uniqueName
        in: path
        description: Alternate key of application
        required: true
        schema:
          type: string
          nullable: true

required, nullable pathparameter which does not exist - I consider this invalid.
you can use plugins to yank such things, but better file bugs.
Always file bugs for inconsistencies, they do not know and can't fix.

aiopenapi3.errors.OperationParameterValidationError: ("/applications(uniqueName='{uniqueName}')", 'patch', 'applications.application.UpdateApplicationByUniqueName', 'Parameter name not found in path: appId')

Now it's taking me about 6 GB RAM and I'll submit this before my machine dies oom … .

_init_schema_types() creates the model classes from the json schemas - time which needs to be spent.
I know startup times for large description documents ain't fun.
Generators should have a benefit here.

[commonism]

[502220.895540] Out of memory: Killed process 3752484 (aiopenapi3) total-vm:7776692kB, anon-rss:7554512kB, file-rss:768kB, shmem-rss:0kB, UID:1000 pgtables:15016kB oom_score_adj:0

Would have to run on a machine more capable than my notebook.

Forgot to mention .clone()
Useful if you need multiple parallel clients with some larger description documents.

and could have linked cache_*

[commonism]

docker run --rm -v $PWD:/spec redocly/cli stats msgraph-metadata-openapi.json

Document: msgraph-metadata-openapi.json stats:

🚗 References: 7728 
📦 External Documents: 6772 
📈 Schemas: 6851 
👉 Parameters: 1022 
🔗 Links: 0 
➡️  Path Items: 14168 
👷 Operations: 22125 
🔖 Tags: 864 

msgraph-metadata-openapi.json: stats processed in 6152ms

Interesting.

[ggpwnkthx]

Yeah, the MS Graph API is pretty extensive and not perfect.

In this situation, ujson benchmarked the fastest for (de)serialization, but I haven't tried pycsimdjson. The best way I've figured so far to handle that many operations and schemas is to wrap them in a ZIP/TAR file as a cheap, convenient, way to partition (by nested key path) and byte index the original definition (with the added benefit of compression). I considered using an SQLite database that handled each OpenAPI object schema but that became complicated very fast, and I didn't see much benefit to that since deserializing (at least with ujson) was pretty quick.

I'm currently using aiopenapi3 with wrapper classes to handle instantiating (as needed) a pretty large library of API clients where authentication is handled via env vars and key-vaults. This way our dev team can just do things like:

from libs.openapi.clients import MetaAPI, MicrosoftGraphAPI

MA = MetaAPI(paths=["/me/adaccounts", "/adaccounts/insights"])
_, adaccounts, _ = MA.createRequest(("/me/adaccounts", "get")).request((parameters=...)

MGA = MicrosoftGraphAPI(paths=["/me", "/teams/{team-id}/channels/{channel-id}/*"])
_, message, _ = MGA .createRequest(("/teams/{team-id}/channels/{channel-id}/", "post")).request((data=...)

Something I should be using more is the Plugin functionality you've already built out. I'll probably dive into that tomorrow for generating and cleaning up the definitions.

I also have some ideas for possible performance improvements with the _iterate_schemas method, but I need to test them tomorrow.

Thank you, as always, for your insights!

[commonism]

aiopenapi3 --profile --profile-file test.prof validate msgraph-metadata-openapi.json
OK
         2249363382 function calls (1582516201 primitive calls) in 1623.525 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.002    0.002 1624.492 1624.492 /usr/local/lib/python3.11/site-packages/aiopenapi3/cli.py:258(cmd_validate)
        1    0.042    0.042 1624.490 1624.490 /usr/local/lib/python3.11/site-packages/aiopenapi3/openapi.py:107(load_file)
        1    0.000    0.000 1624.393 1624.393 /usr/local/lib/python3.11/site-packages/aiopenapi3/openapi.py:134(loads)
        1    0.006    0.006 1623.642 1623.642 /usr/local/lib/python3.11/site-packages/aiopenapi3/openapi.py:183(__init__)
        1    0.554    0.554 1564.094 1564.094 /usr/local/lib/python3.11/site-packages/aiopenapi3/openapi.py:463(_init_schema_types)
   108824  266.172    0.002 1389.933    0.013 /usr/local/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py:423(complete_model_class)
    89160    0.682    0.000 1281.292    0.014 /usr/local/lib/python3.11/site-packages/pydantic/main.py:423(model_rebuild)
…

model_rebuild is pydantic - without involving the pydantic crew this has to be considered static.

I pushed mypy branch for you to work with.
Currently mypy runs take ~30 minutes, it is not mypy clean and I have little hope in ever getting there, but having more type annotations is nice.

The custom schema selection could/has to be be improved to match yours - work on a PathItem basis and include only the required schemas for the operations chosen. I think about skipping the components.schemas lookup in _init_schema_types_collect in certain conditions - maybe the self._root.paths got changed (by len() or id()) _init_operationindex, we'll see …
Forgot to mention Init.paths as possible plugin interface.

Great to see you are putting the library to good use.
What is your experiences with other libraries/generators with wrt. to overall usability your special requirements?

[ggpwnkthx]

Cloned the mypy branch, and I'm running in to this. Not an issue with the branch, but with pydantic.

[ggpwnkthx]

This works:

[commonism]

The ref to pydantic@github is a relict from the v2 migration.
I failed to manually regenerate the requirements after setting the dependency in the pyproject.toml.
The automation took care of it in #133.
You can avoid the requirements.txt by using pip to install using the dependencies from the pyproject.toml - I usually use "pip install -e .".

[ggpwnkthx]

Is there a functional purpose for using a ChainMap in this method?

aiopenapi3/aiopenapi3/openapi.py

Lines 399 to 459 in 9947b29

	@staticmethod
	def _iterate_schemas(schemas: Dict[int, "SchemaType"], next: Set[int], processed: Set[int]):
	"""
	recursively collect all schemas related to the starting set
	"""
	if not next:
	return processed
	processed.update(next)
	def not_in_processed(z: Tuple[int, SchemaBase]) -> TypeGuard[Tuple[int, "SchemaType"]]:
	return z[0] not in processed
	def is_instance_schema_ref(
	x: Union["SchemaType", "ReferenceType"]
	) -> TypeGuard[Union["SchemaType", "ReferenceType"]]:
	return isinstance(x, (ReferenceBase, SchemaBase))
	new: Dict[int, "SchemaType"] = cast(
	Dict[int, "SchemaType"],
	collections.ChainMap(
	*[
	dict(
	filter(
	not_in_processed,
	map(
	lambda y: (id(y._target), y._target) if isinstance(y, ReferenceBase) else (id(y), y),
	filter(
	is_instance_schema_ref,
	getattr(schemas[i], "oneOf", []) # Swagger compat
	+ getattr(schemas[i], "anyOf", []) # Swagger compat
	+ schemas[i].allOf
	+ list(schemas[i].properties.values())
	+ (
	[schemas[i].items]
	if schemas[i].type == "array"
	and schemas[i].items
	and not isinstance(schemas[i].items, list)
	else []
	)
	+ (
	schemas[i].items
	if schemas[i].type == "array"
	and schemas[i].items
	and isinstance(schemas[i].items, list)
	else []
	),
	), # type: ignore[operator]
	),
	)
	) # type: ignore[arg-type]
	for i in next
	]
	),
	)
	sets = set(new.keys())
	schemas.update(new)
	processed.update(sets)
	return OpenAPI._iterate_schemas(schemas, sets, processed)

I did some testing and iterating without recursion, using just dict comprehension, and avoiding filtering/mapping looks to be significantly more performant with large schemas:

    @staticmethod
    def _get_combined_attributes(schema):
        """Combine attributes from the schema."""
        return (
            getattr(schema, "oneOf", [])  # Swagger compat
            + getattr(schema, "anyOf", [])  # Swagger compat
            + schema.allOf
            + list(schema.properties.values())
            + (
                [schema.items]
                if schema.type == "array"
                and schema.items
                and not isinstance(schema, list)
                else []
            )
            + (
                schema.items
                if schema.type == "array" and schema.items and isinstance(schema, list)
                else []
            )
        )
        
    @classmethod
    def _process_schema_attributes(cls, schema, processed):
        """Process attributes of a schema and filter out the processed ones."""
        combined_attributes = cls._get_combined_attributes(schema)
        return {
            id(item._target) 
            if isinstance(item, ReferenceBase) 
            else id(item): item._target 
            if isinstance(item, ReferenceBase) 
            else item
            for item in combined_attributes 
            if isinstance(item, (ReferenceBase, SchemaBase)) 
            and id(item) not in processed
        }
        
    @classmethod
    def _iterate_schemas(cls, schemas: dict, next_set: set, processed: set):
        """Iteratively collect all schemas related to the starting set.""" 
        while next_set:
            processed.update(next_set)
            # Using dictionary comprehension to combine the results, 
            # which seems more efficient than ChainMap
            new = {
                k: v 
                for i in next_set
                for k, v in cls._process_schema_attributes(schemas[i], processed).items()
            }
            next_set = set(new.keys()) - processed
            schemas.update(new)
            processed.update(next_set)
        return processed

For my tests, the todo set contained 33,181 items, but my suggested version of the _iterate_schemas method completed in ~0.15s vs ~21.1s for the current method.

[commonism]

simdjson is fast but looses when all fields get accessed to orjson which is faster than ujson.
https://martin-thoma.com/json-encoding-decoding/
https://gist.github.com/kgriffs/e34364896ae2f329aa8016ec6f71fdce
https://github.com/tktech/json_benchmark

As startup reduction is way easier and has the benefit to allow removing all not required data from memory … I gave it a shot.

This is an easy way to test it …

aiopenapi3 --tracemalloc -v validate tests/data/petstorev3-openapi.yaml 
…  0:00:00.338639 (processing time)
… 19 #operations
… 19 #operations (with operationId)
… 0 tests/data/petstorev3-openapi.yaml: 8
… 8 schemas total
OK
Top 3 lines
#1: /home/c01mako/venv/openapi3/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py:183: 350.0 KiB
    return f(schema.copy(), self._walk)
#2: /home/c01mako/venv/openapi3/lib/python3.10/site-packages/pydantic/main.py:504: 321.0 KiB
    return cls.__pydantic_validator__.validate_python(
#3: /home/c01mako/venv/openapi3/lib/python3.10/site-packages/pydantic/_internal/_model_construction.py:486: 130.1 KiB
    cls.__pydantic_validator__ = SchemaValidator(simplified_core_schema, core_config)
494 other: 1084.5 KiB
Total allocated size: 1885.6 KiB

using tests/reduced_test.py with

class PetStoreReduced(Reduced):
    def __init__(self):
        super().__init__({"/user/{username}":None})

aiopenapi3 --tracemalloc -v -P tests/reduced_test.py:PetStoreReduced validate tests/data/petstorev3-openapi.yaml 
…  0:00:00.198641 (processing time)
… 3 #operations
… 3 #operations (with operationId)
… 0 tests/data/petstorev3-openapi.yaml: 1
… 1 schemas total
OK
Top 3 lines
#1: /home/c01mako/venv/openapi3/lib/python3.10/site-packages/pydantic/main.py:504: 90.0 KiB
    return cls.__pydantic_validator__.validate_python(
#2: /usr/lib/python3.10/abc.py:123: 69.0 KiB
    return _abc_subclasscheck(cls, subclass)
#3: /home/c01mako/venv/openapi3/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py:183: 64.3 KiB
    return f(schema.copy(), self._walk)
471 other: 400.0 KiB
Total allocated size: 623.3 KiB

the ms-graph api reduced to one operation

aiopenapi3 -v --tracemalloc -P tests/reduced_test.py:MSGraphReduced validate tests/data/ms-graph-openapi.json 
5 only_required=True
5
…  0:00:10.610795 (processing time)
… 1 #operations
… 1 #operations (with operationId)
… 0 tests/data/ms-graph-openapi.json: 1
… 1 schemas total
OK
Top 3 lines
#1: /home/c01mako/venv/openapi3/lib/python3.10/site-packages/pydantic/main.py:504: 67794.1 KiB
    return cls.__pydantic_validator__.validate_python(
#2: /usr/lib/python3.10/json/decoder.py:353: 13246.0 KiB
    obj, end = self.scan_once(s, idx)
#3: /home/c01mako/venv/openapi3/lib/python3.10/site-packages/pydantic/main.py:165: 4155.9 KiB
    __pydantic_self__.__pydantic_validator__.validate_python(data, self_instance=__pydantic_self__)
384 other: 13237.2 KiB
Total allocated size: 98433.2 KiB

real	0m13,957s
user	0m13,643s
sys	0m0,313s

and tracemalloc is quite slow, without …

aiopenapi3 -v -P tests/reduced_test.py:MSGraphReduced validate tests/data/ms-graph-openapi.json 
5 only_required=True
5
…  0:00:04.615062 (processing time)
… 1 #operations
… 1 #operations (with operationId)
… 0 tests/data/ms-graph-openapi.json: 1
… 1 schemas total
OK

real	0m6,087s
user	0m5,878s
sys	0m0,209s

It's most likely the same as you do to reduce description documents but embedded.

[ggpwnkthx]

I refactored my reduction workflow to use orjson alone. This only takes about a second and the majority of that is orjson loading the full spec:

import orjson

def extract_references(data, root=None):
    """Extracts references from the data using a generator."""
    stack = [(data, root or data)]
    while stack:
        data, root = stack.pop()
        if isinstance(data, dict):
            for key, value in data.items():
                if key == "$ref" or (isinstance(value, str) and value.startswith("#/")):
                    yield value
                else:
                    stack.append((value, root))
        elif isinstance(data, list):
            for item in data:
                stack.append((item, root))


def update_references(source, destination, target=None):
    """Updates references in the destination based on the source."""
    has_changes = False

    for ref in extract_references(target or destination):
        if ref.startswith("#"):
            keys = ref[2:].split("/")
            source_cursor = source
            destination_cursor = destination

            for key in keys[:-1]:
                if key not in destination_cursor:
                    destination_cursor[key] = {}
                destination_cursor = destination_cursor[key]
                source_cursor = source_cursor.get(key, {})

            last_key = keys[-1]
            if last_key not in destination_cursor:
                has_changes = True
                destination_cursor[last_key] = source_cursor.get(last_key, {})

    return has_changes


# Load the full specification
with open("msgraph_openapi.json") as msg:
    full_specification = orjson.loads(msg.read())

# Create a new specification excluding certain keys
new_specification = {k: v for k, v in full_specification.items() if k not in ["components", "paths", "tags"]}
new_specification["paths"] = {}

# Add specific paths to the new specification
for path in ["/me/sendMail", "/me/mailFolders"]:
    if path in full_specification.get("paths", {}):
        new_specification["paths"][path] = full_specification["paths"][path]

# Update references in the new specification
while update_references(full_specification, new_specification):
    pass

# Consider depth limit
# for _ in range(max_depth):
#     if not update_references(full_specification, new_specification):
#         break

The resulting JSON is over 5MB (without any formatting). Certainly more manageable than 56MB, but front-loading all the OpenAPI schemas as Pydantic models still takes a like 6-8min.

I have to jump into some other projects this week, but I'll look into the lazy-loading idea we talked about next week.

[commonism]

For your example with 2 paths on msgraph - 5s total.
https://github.com/commonism/aiopenapi3/pull/new/reduced

[ggpwnkthx]

Had a chance this afternoon to test the reduced branch out. I like the direction it's going in, but it doesn't pass QC because the necessary response schemas get deleted. I think the issue specifically with the official OpenAPI document for the Microsoft Graph API is that there's a reference in the allOf list for nearly every response schema:
{ "$ref": "#/components/schemas/microsoft.graph.entity" }

That particular reference uses a discriminator that contains, presumably, every @odata.type, and it also seems to be self-referential. For example:

...
    "schemas": {
      "microsoft.graph.mailFolder": {
        "allOf": [
          { "$ref": "#/components/schemas/microsoft.graph.entity" },
          {
            "title": "mailFolder",
            "required": ["@odata.type"],
            "type": "object",
            "properties": {
              "childFolderCount": {
                "maximum": 2147483647,
                "minimum": -2147483648,
                "type": "integer",
                "description": "The number of immediate child mailFolders in the current mailFolder.",
                "format": "int32",
                "nullable": true
              }, ...
            }, ...
          ], ...
        }, ...
      },
      ... ,
      "microsoft.graph.entity": {
        "title": "entity",
        "required": ["@odata.type"],
        "type": "object",
        "properties": {
          "id": {
            "type": "string",
            "description": "The unique identifier for an entity. Read-only."
          },
          "@odata.type": { "type": "string" }
        },
        "discriminator": {
          "propertyName": "@odata.type",
          "mapping": {
            ... ,
            "#microsoft.graph.mailFolder": "#/components/schemas/microsoft.graph.mailFolder",
            ... ,
          }
        },
      },
      ... ,
    }, 
...

Seems to me the best course of action would be to piggyback off the Reduced plugin to strip these unnecessary elements out.

[commonism]

I reproduced this in a unit test and fixed it - missing dependency tracking for discriminator mapping schemas.
Actually I'd expect this would require a oneOf to be valid OpenAPI3, not just having the discriminator mapping defined.

[ggpwnkthx]

Curious of your thoughts on my changes here: ggpwnkthx@645f1af

Sub 4 second but I'm culling things from the ctx.document in the parsed phase.

[commonism]

The ref tracking is impressive - I like it.
It's rough sanding. Error prone as it does not have typing information and does not work with validated data, but I think it can not remove too much. Great addition to the fine sanding in the initialization later on.
I consider having this possibility in a plugin great, I can admire this without having to think about maintenance.

The list comprehensions aren't easy to read, black obfuscates them even more, but I see the speed benefit.

The response type correction - I'd expect removing components.schemas."microsoft.graph.entity".discriminator would do as well.

    def parsed(self, ctx: "Document.Context") -> "Document.Context":
        super().parsed(ctx)
        del ctx.document["components"]["schemas"]["microsoft.graph.entity"]["discriminator"]
        return ctx

The entity object can not be instantiated anyway, it's discriminator is shared with it's discriminated unions.
It's basically used as a container for the common "odata.type" property.
Adding odata.type to each of it's discriminated unions/allOf variants, should allow removing entity from components.schemas, though I do not expect it to be worth the time spent compared to yanking the entity discriminator.

3.3s is impressive
"Pydantic parsing" the full description document takes more time & memory.
The ondemand-model-creation we discussed has to be inferior to this approach in every aspect, more time spent & resources used for initialization and extra time for each operation used iteratively.

[commonism]

One safe but slow optimization and one aggressive but you don't know variant would be fine for me - therefore I currently prefer to have both - Reduce and Cull - in extra.

[ggpwnkthx]

Is parsed the right place to perform the culling process?

If so, that plugin method doesn't ever get called if the OpenAPI object is handled via the class constructor - only if a Loader is used.
If not, the Plugin.initialized method doesn't get called until after the OpenAPI._init_operationindex and OpenAPI._init_schema_types calls in the constructor, which doesn't make it a reasonable place to do the culling process either.

I presume, at least in the vast majority of cases, the document has to be loaded from somewhere. So, it'd be rare that an OpenAPI object ever needs to be initialized directly by the constructor, and if it is needed, it's reasonable to require that the definition is in its final form.

Which is to say, I think the answer to my question is most likely, yes - parsed is the correct place, with the acknowledgement that it's only called with a loader method and not by the constructor. Unless you feel like it makes sense to have some kind of "pre-initialization" plugin method that happens in the constructor after the self._init_plugins(plugins) call?

[commonism]

I consider parsed to be the right place - I know it does not play nice with OpenAPI.init.
Been here before - parsed() was called twice, in Loader.parse and somewhere in OpenAPI.* back then.
To feed a prepared JSON datastructure - I'd go with OpenAPI.load_obj() or something to avoid effects.

[ggpwnkthx]

Just pushed to my fork an update that handles things with class inheritance. I'm not 100% on their names, though.

ggpwnkthx@19c7d78

[commonism]

cherry picked - master...reduced
Dropped the LazyLoader, based Cull on Reduce, some words on it.

Looking good to you?

[ggpwnkthx]

I put two comments in there on the documentation. Otherwise, I think it's great!

[commonism]

merged reduced in #162

thanks

[ggpwnkthx]

Thank you for your patience, cooperation, and insights!

Is there anything needed from my fork? If not, I'll delete it and close out this discussion.

[commonism]

It's been a pleasure.
I've merged everything required, you can go ahead with your cleanup.

I'll reach out if I feel the need to cull multi file large description documents.