[Inputs Needed] Add New VectorDB PGVector

[rahulbansal16]

I am working on the issue.

Here is the current progress.

1. Added the vecs SDK to query the PGVector.
2. The functions are working as expected.
3. I have tested them locally via the commented code in the python file.

Pending Items

1. Need inputs should we use vecs (It's new SDK from supabase) or use the ORM sqlalchemy.
2. Add test for the code.
3. Make the upsert method for the add_text async.
4. The flow for creating indexs on the vector table.

[arpagon]

Hello @rahulbansal16,

I think it's a good idea to use 'vecs', but keeping this in mind. supabase/vecs#6

Are you in the Vocode Open Source Discord community?

[rahulbansal16]

Hey @arpagon I have joined the discord.

They have the goal to make vecs async in the near future.
so, I will wrap the call in a thread for a now.

And will remove the thread when vecs is done with the async support.

Let me know if you have a different approach.

[rahulbansal16]

I have updated the code and wrapped the insert and query method in the asyncio threads.
Tested them locally and able to query and insert the data into the PGVector in Supabase.

Since it's postgres. It's recommended to create the indexes before querrying.
I am not exposing the functionality to create indexes via the API.
It's better to use the GUI for that.

[rahulbansal16]

I did not find any test for other vector db.
Do I have to write test for the pg_vector?

[arpagon]

While other vector db tests might be missing, writing tests for pg_vector is crucial for confidence, stability, and adoption. It benefits both your project and the community. Thorough tests build trust, encouraging wider use and contributions. Don't hesitate to ask for help getting started – the community is here to support you!

[rahulbansal16]

@arpagon I will add them, I am occupied with other work but will be able to contribute on weekend.

[rjheeta]

Hello @rahulbansal16 I'm interested in using this patch. Is it in a stable enough state that I can use it?

[rahulbansal16]

@rjheeta I have not written test for it but I was able to test the functionality and it was working.

[rjheeta]

Perfect, thank you. I will give this a shot.

@arpagon At a high level, what tests are you looking for in order to merge & support this change? Maybe I can try to help with writing some of these?