-
Notifications
You must be signed in to change notification settings - Fork 13
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add collections operations to PineconeControlPlaneClient
with integration tests
#65
Merged
austin-denoble
merged 22 commits into
main
from
adenoble/collection-operations-integration-tests
Feb 21, 2024
Merged
Add collections operations to PineconeControlPlaneClient
with integration tests
#65
austin-denoble
merged 22 commits into
main
from
adenoble/collection-operations-integration-tests
Feb 21, 2024
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ion tests for collections, add new helpers for new collections tests
…things properly, use waitUntilReady properly in createCollection helper
austin-denoble
changed the title
Add collections operations to
Add collections operations to Feb 14, 2024
PineconeControlPlaneClient
and collections integration testsPineconeControlPlaneClient
with integration tests
…waitUntilIndexReady
…ain, max-parallel again
…where we create one
austin-denoble
commented
Feb 15, 2024
austin-denoble
commented
Feb 15, 2024
austin-denoble
commented
Feb 15, 2024
austin-denoble
commented
Feb 15, 2024
austin-denoble
commented
Feb 15, 2024
austin-denoble
commented
Feb 15, 2024
austin-denoble
commented
Feb 15, 2024
austin-denoble
commented
Feb 15, 2024
src/integration/java/io/pinecone/integration/controlPlane/pod/ConfigureIndexTest.java
Show resolved
Hide resolved
austin-denoble
commented
Feb 15, 2024
src/integration/java/io/pinecone/integration/controlPlane/pod/ConfigureIndexTest.java
Show resolved
Hide resolved
austin-denoble
commented
Feb 15, 2024
...tion/java/io/pinecone/integration/controlPlane/pod/CreateDescribeListAndDeleteIndexTest.java
Show resolved
Hide resolved
austin-denoble
commented
Feb 15, 2024
src/integration/java/io/pinecone/integration/dataplane/PineconeClientLiveIntegTest.java
Show resolved
Hide resolved
austin-denoble
commented
Feb 15, 2024
austin-denoble
commented
Feb 15, 2024
austin-denoble
commented
Feb 15, 2024
rohanshah18
reviewed
Feb 15, 2024
rohanshah18
reviewed
Feb 15, 2024
src/integration/java/io/pinecone/integration/controlPlane/pod/CollectionErrorTest.java
Outdated
Show resolved
Hide resolved
rohanshah18
approved these changes
Feb 15, 2024
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great work!! I have left some comments but one area especially I'd like to think about is the hard-coded time outs added for collection tests in indexManager's waitUntilReady methods.
rohanshah18
reviewed
Feb 16, 2024
rohanshah18
reviewed
Feb 16, 2024
src/integration/java/io/pinecone/integration/controlPlane/pod/ConfigureIndexTest.java
Show resolved
Hide resolved
rohanshah18
reviewed
Feb 16, 2024
… in assertWithRetry to actuall throw on failure
austin-denoble
deleted the
adenoble/collection-operations-integration-tests
branch
February 21, 2024 01:35
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
The Java SDK is currently missing collections functionality. Now that we've added generated code for the OpenAPI spec, we can hook up collections operations in
PineconeControlPlaneClient
.We'd also like to add integration tests to cover collections.
Solution
PineconeControlPlaneClient
to include collections operations for create, list, delete, and describe.CollectionTest
andCollectionErrorTest
.IndexManager
for creating and connecting to an index byindexName
, creating a collection and waiting for it to be ready, and polling until an index is ready.BuildUpsertRequest
for generating vectors by dimension.Bonuses:
build.gradle
fortest
and theintegrationTest
task. This was primarily to help me debug things in CI and log a bit more info to the console. We can tweak as needed, but I think having something like this will be really helpful as opposed to the generic output.max-parallel: 1
in theintegration-test
job in thepr.yml
workflow. Because of the wayfindIndexWithDimensionAndType
works we cannot run integration tests in parallel without flakiness.AssertRetry
. We were incrementingdelay
at the class level which is a static variable, meaning if you re-used the function in one place it would continue increasing the delay for all subsequent calls based on thebackOff
.Type of Change
Test Plan
Run integration tests to make sure
CollectionTest
andCollectionErrorTest
pass as expected. Integration tests should also be passing in CI barring any flakiness.