REST code generation for 2024-07 prerelease

[jhamon]

Problem

We need to generate code off of the openapi specs stored in our apis repo. We'll need to massage the output a bit in order to avoid having a lot of confusing duplication of generated exception classes and other utils.

Solution

• I create a new codegen directory to act as the home for everything codegen related. Inside this directory there are several things:
  • Git submodule of our template files (slightly tweaked from the open source generator, but in a separate repo so they can be shared with plugins)
  • Git submodule of our apis repo
  • A build-oas.sh script which does everything to generate the code off the spec, adjust import paths, and place things into the pinecone module package structure.

The script does the following:

• Pulls in the latest from the apis repo
• Builds the api repo to produce versioned spec files
• For each of data plane and control plane, generates a python package.
• Performs surgery to extract the duplicated files across these two packages into a shared package.
• Use sed to adjust import paths to reflect the modified code structure.

The surgery to extract shared code is more than just a cosmetic change, as without it you can end up with a lot of confusion in the areas of configuration and error handling due to there being multiple objects that despite being identical in name and functionality remain distinct from the perspective of language utils such as isinstance, .__class__ and except.

When the script is done running, the generated outputs have this structure:

pinecone/core
└── openapi
    ├── control
    │   ├── api
    │   │   ├── inference_api.py
    │   │   └── manage_indexes_api.py
    │   └── model
    │       ├── __init__.py
    │       ├── create_index_request.py
    │       ├── index_model.py
    │       └── ...etc
    ├── data
    │   ├── api
    │   │   └── data_plane_api.py
    │   └── model
    │       ├── __init__.py
    │       ├── upsert_request.py
    │       ├── vector.py
    │       └── ...etc
    └── shared
        ├── api_client.py
        ├── configuration.py
        ├── exceptions.py
        ├── model_utils.py
        └── rest.py

What actually changed in here?

It seems the main substantive change is in how the spec info is expected to be passed when creating an index. So I had to write some additional tests and make some modifications to the create_index method.

Also, inference-related stuff is currently generated. But for now, all the wrapper implementations for that functionality are only available by installing a separate plugin, pinecone-plugin-inference.

Type of Change

• Breaking change (fix or feature that would cause existing functionality to not work as expected)

Test Plan

To run the script, run make generate-oas. Before pushing, I ran tests locally with make test-unit and PINECONE_API_KEY='key' make test-integration. This helped me catch a lot of small issues related to the sed rewrite of import paths.

Want to see tests passing even though I changed quite a bit about how the generated code is structured.

[jhamon]

A decent amount of change was needed in this file due to changes in generated code related to index spec.

[haruska]

I didn't thoroughly review all the code changes for the generated code but overall I really like this approach to code-gen for the python client.

[aulorbe]

Looks awesome!

[austin-denoble]

Really nice work on this Jen, the approach feels clear and easy to follow especially given everything going on under the hood. 🚢

[austin-denoble]

Would it be useful to have something similar for making sure the codegen/python-oas-templates submodule is also up to date? I doubt it would have as much churn as the APIs repo, but just in case.

[austin-denoble]

Nice cleanup! 👍

[austin-denoble]

Also cool, very clever piece of scripting here, nice work.