-
Notifications
You must be signed in to change notification settings - Fork 114
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Warnings while creating cosine based index #106
Comments
Could you run the command below to get your index's information.
|
I tried to reproduce your problem with the datasets I have, but I could not. Since the problem might depend on datasets, could you provide your dataset, if possible. |
Hey @masajiro — Thanks for the command, it details out the index meta which is quite helpful. This is the output for an index created with above-mentioned warnings.
The dataset had empty vectors which may or may not be the reason for warnings. I created another index with a clean 1 Mn vectors & it didn't give any warnings this time. Here's the command output for it.
Also, want to mention that the optimization guide helped me a lot to achieve desired accuracy & performance with the ONNG index. Thanks a lot for putting it together. |
The dataset is 6.6 Mn, I'll try to reproduce the issue with a minimal dataset & share it with you. Let me get back on this by Monday. |
Did you solve this issue? |
While building NGT index using the cosine distance metric, I see lot many warnings like below.
Created the index using this command where I don't specify any
-S
param (default is 40)I feel this suspicious as there are differences compared to another index built with L2 (Euclidean) distance metric using the same input vectors.
The warning seems to be originating from here due to which I think the cosine based index is not properly built hence the impact on accuracy. Any thoughts on this or it's expected?
The text was updated successfully, but these errors were encountered: