Live index update or retrain

[NajiAboo]

I am doing research on haystack. I am planning to do similarly using fiass document store.
Here I am little confused with the retraining the model. What will be best way to do retraining?
Do we need to drop existing embedding and have to create new?please help me on it.

[vblagoje]

Hi Mohamed, would you please elaborate more on what you are trying to achieve. If I understand you correctly, you want to do a document search using FAISSDocumentStore, but I don't understand the retraining part of your question.

[NajiAboo]

Thanks for the response. Once I have created the document store and retriever How I will update it for new documents?. I see that there is a "write_document" method to insert the new document. But to update embeddings, we need to call update_embeddings. This is taking so long.
I am sure that there will be some simple method so that I can update new document and its embeddings easily

[vblagoje]

Unfortunately, after inserting new documents, you need to update the embeddings - otherwise, the retriever doesn't have vector representations of the newly added documents. Two simple and quick options you can try here:

• If possible, get a faster GPU for FAISSDocumentStore
• Try another document store? PineconeDocumentStore would be able to update embeddings quicker if you assign to it more pods.

[NajiAboo]

Thanks for the support. In between, I have read somewhere about adding embedding while we insert the document. got the below code.
It says when we add a document create the embedding and add it to the document. But its not working as mentioned. Do you think this will help? If so can you share with me some sample? I am really new to this framework.

from haystack import Document
dicts = [{"text": "some text"}, ...]
docs = [Document.from_dict(d) for d in dicts]

get the embedding for a batch of docs

batch_docs = docs[:32]
batch_emb = retriever.embed_passages(batch_docs)

attach embeddings to the docs

for emb, doc in zip(batch_emb, batch_docs):
doc.embedding = emb
...

later: write everything to the doc store

doc_store.write_documents(all_docs)

[vblagoje]

Hey, no problem, why don't you start with something as simple as:

document_store = FAISSDocumentStore(faiss_index_factory_str="Flat")
document_store.write_documents([{'content': t} for t in corpus])            


retriever = TheRetrieverYouWantToUse(document_store=document_store)
document_store.update_embeddings(retriever)

[NajiAboo]

Sure , Thanks