Skip to content

Commit

Permalink
Add codegen script and update generated code for 2024-07 (#140)
Browse files Browse the repository at this point in the history
## Problem

Need to update generated code for the upcoming 2024-07 API release.

## Solution

This PR closely follows the approach used in the python and typescript
repositories.

- I create a new `codegen` directory to act as the home for everything
codegen related. Inside this directory there are several things:
  - Git submodule of our apis repo
- A `build-oas.sh` script which does everything to generate the code off
the spec

The script does the following:
- Pulls in the latest from the apis repo
- Builds the api repo to produce versioned spec files
- Generates java code off the spec file.
- Copies files into their respective places in `src/`

Run the script like this, passing api version number as a positional
argument:

```sh
./codegen/build-oas.sh 2024-07
```

## Non-generated changes

The updated spec produced slightly different generated code than in the
past. In order for the project to compile, I had to make some
adjustments:

- Options for metric type are now part of the `IndexModel` class instead
of a standalone metric object, requiring a small refactor along the
lines of `IndexMetric.COSINE.toString()` to
`IndexModel.MetricEnum.COSINE.toString()` and similar for other metric
types.
- The name and organization of create index spec objects has changed.
Now you use:
  - `IndexSpec` instead of `CreateInexRequestSpec`.
  - `PodSpec` instead of `CreateIndexRequestSpecPod`
- `PodSpecMetadataConfig` instead of
`CreateIndexRequestSpecPodMetadataConfig`

## Type of Change

- [x] None of the above: Update generated core, adjust usage for new
generated classes

## Test Plan

Code builds and tests pass
  • Loading branch information
jhamon authored Jul 10, 2024
1 parent 8034dab commit 3411d21
Show file tree
Hide file tree
Showing 62 changed files with 2,505 additions and 1,027 deletions.
4 changes: 3 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -79,4 +79,6 @@ nb-configuration.xml
##############################
## OS X
##############################
.DS_Store
.DS_Store

gen
3 changes: 3 additions & 0 deletions .gitmodules
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[submodule "codegen/apis"]
path = codegen/apis
url = [email protected]:pinecone-io/apis.git
5 changes: 2 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,7 @@ having to obtain the *pinecone-client* dependencies separately.
### Initializing the client

Before you can use the Pinecone Java SDK, you must sign up for a Pinecone account and find your API key in the Pinecone
console
dashboard at [https://app.pinecone.io](https://app.pinecone.io).
console dashboard at [https://app.pinecone.io](https://app.pinecone.io).

#### Using apiKey

Expand Down Expand Up @@ -75,7 +74,7 @@ public class InitializeClientExample {

OkHttpClient httpClient = builder.build();

Pinecone pinecone = new Pinecone.Builder("PINECONE_API_KEY").build();
Pinecone pinecone = new Pinecone.Builder("PINECONE_API_KEY").withOkHttpClient(httpClient).build();
}
}
```
Expand Down
1 change: 1 addition & 0 deletions codegen/apis
Submodule apis added at fbd9d8
87 changes: 87 additions & 0 deletions codegen/build-oas.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
#!/bin/bash

set -eux -o pipefail

version=$1 # e.g. 2024-07
modules=("control")

destination="src/main/java/org/openapitools"
build_dir="gen"

update_apis_repo() {
echo "Updating apis repo"
pushd codegen/apis
git fetch
git checkout main
git pull
just build
popd
}

verify_spec_version() {
local version=$1
echo "Verifying spec version $version exists in apis repo"
if [ -z "$version" ]; then
echo "Version is required"
exit 1
fi

verify_directory_exists "codegen/apis/_build/${version}"
}

verify_file_exists() {
local filename=$1
if [ ! -f "$filename" ]; then
echo "File does not exist at $filename"
exit 1
fi
}

verify_directory_exists() {
local directory=$1
if [ ! -d "$directory" ]; then
echo "Directory does not exist at $directory"
exit 1
fi
}

generate_client() {
local module_name=$1

oas_file="codegen/apis/_build/${version}/${module_name}_${version}.oas.yaml"

verify_file_exists $oas_file

# Cleanup previous build files
echo "Cleaning up previous build files"
rm -rf "${build_dir}"

# Generate client module
docker run --rm -v $(pwd):/workspace openapitools/openapi-generator-cli:v7.0.1 generate \
--input-spec "/workspace/$oas_file" \
--generator-name java \
--additional-properties=dateLibrary='java8',disallowAdditionalPropertiesIfNotPresent=false \
--output "/workspace/${build_dir}"

# Copy the generated module to the correct location
rm -rf "${destination}/${module_name}"
mkdir -p "${destination}/${module_name}"

path_to_copy="${build_dir}/src/main/java/org/openapitools/client"
cp -r $path_to_copy "${destination}/${module_name}"

# Adjust package names in generated file
find "${destination}/${module_name}" -name "*.java" | while IFS= read -r file; do
sed -i '' "s/org\.openapitools\.client/org\.openapitools\.${module_name}\.client/g" "$file"
done
}

update_apis_repo
verify_spec_version $version

rm -rf "${destination}"
mkdir -p "${destination}"

for module in "${modules[@]}"; do
generate_client $module
done
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
import io.pinecone.clients.Pinecone;
import io.pinecone.exceptions.PineconeException;
import io.pinecone.proto.DescribeIndexStatsResponse;
import org.openapitools.client.model.*;
import org.openapitools.control.client.model.*;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

Expand Down Expand Up @@ -50,8 +50,8 @@ public class TestResourcesManager {
? "us-east4-gcp"
: System.getenv("PINECONE_ENVIRONMENT");
private static final String metric = System.getenv("METRIC") == null
? IndexMetric.DOTPRODUCT.toString()
: IndexMetric.valueOf(System.getenv("METRIC")).toString();
? IndexModel.MetricEnum.DOTPRODUCT.toString()
: IndexModel.MetricEnum.valueOf(System.getenv("METRIC")).toString();
private static final String cloud = System.getenv("CLOUD") == null
? ServerlessSpec.CloudEnum.AWS.toString()
: System.getenv("CLOUD");
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
import io.pinecone.clients.Pinecone;
import io.pinecone.proto.DescribeIndexStatsResponse;
import io.pinecone.proto.NamespaceSummary;
import org.openapitools.client.model.*;
import org.openapitools.control.client.model.*;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Disabled;
import org.junit.jupiter.api.Test;
import org.openapitools.client.model.*;
import org.openapitools.control.client.model.*;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

Expand Down Expand Up @@ -117,11 +117,11 @@ public void testIndexFromCollectionHappyPath() throws InterruptedException {
public void testCreateIndexFromCollectionWithDiffMetric() throws InterruptedException {
// Use a different metric than the source index
String[] metrics = {
IndexMetric.COSINE.toString(),
IndexMetric.EUCLIDEAN.toString(),
IndexMetric.DOTPRODUCT.toString()
IndexModel.MetricEnum.COSINE.toString(),
IndexModel.MetricEnum.EUCLIDEAN.toString(),
IndexModel.MetricEnum.DOTPRODUCT.toString()
};
String targetMetric = IndexMetric.COSINE.toString();
String targetMetric = IndexModel.MetricEnum.COSINE.toString();
for (String metric : metrics) {
if (!metric.equals(sourceIndexMetric)) {
targetMetric = metric;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import org.openapitools.client.model.IndexModel;
import org.openapitools.client.model.IndexModelStatus;
import org.openapitools.client.model.PodSpec;
import org.openapitools.control.client.model.IndexModel;
import org.openapitools.control.client.model.IndexModelStatus;
import org.openapitools.control.client.model.PodSpec;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
import io.pinecone.helpers.TestResourcesManager;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import org.openapitools.client.model.*;
import org.openapitools.control.client.model.*;

import static org.junit.jupiter.api.Assertions.*;

Expand Down Expand Up @@ -36,7 +36,7 @@ public void describeAndListIndex() {
assertNotNull(indexModel);
assertEquals(indexDimension, indexModel.getDimension());
assertEquals(indexName, indexModel.getName());
assertEquals(IndexMetric.DOTPRODUCT, indexModel.getMetric());
assertEquals(IndexModel.MetricEnum.DOTPRODUCT, indexModel.getMetric());
assertNotNull(indexModel.getSpec().getPod());
assertEquals(indexPodType, indexModel.getSpec().getPod().getPodType());

Expand All @@ -63,7 +63,7 @@ public void createPodsIndexWithMinimumRequiredParams() {
assertEquals(podType, podsIndex.getSpec().getPod().getPodType());

// Confirm defaults are put in by the backend when not supplied by the user
assertEquals(IndexMetric.COSINE, podsIndex.getMetric());
assertEquals(IndexModel.MetricEnum.COSINE, podsIndex.getMetric());
assertEquals(1, podsIndex.getSpec().getPod().getPods());
assertEquals(1, podsIndex.getSpec().getPod().getReplicas());
assertEquals(1, podsIndex.getSpec().getPod().getShards());
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
import io.pinecone.helpers.TestResourcesManager;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import org.openapitools.client.model.*;
import org.openapitools.control.client.model.*;

import java.util.Arrays;

Expand Down Expand Up @@ -36,7 +36,7 @@ public void describeAndListIndex() {
assertNotNull(indexModel);
assertEquals(dimension, indexModel.getDimension());
assertEquals(indexName, indexModel.getName());
assertEquals(IndexMetric.DOTPRODUCT, indexModel.getMetric());
assertEquals(IndexModel.MetricEnum.DOTPRODUCT, indexModel.getMetric());
assertNotNull(indexModel.getSpec().getServerless());

// List the index
Expand Down
Loading

0 comments on commit 3411d21

Please sign in to comment.