Local chat issue : On startup I have "Error allocating memory ErrorOutOfDeviceMemory" in the log

[yarel0]

Hello all,
I am trying to use Khoj with Obsidian.
Indexation and search work fine but I have an issue with local chat.

And indeed something goes wrong on startup, it seems that Language Model is not loaded correctly :

[23:52:12] INFO 🌘 Starting Khoj main.py:78
[23:52:13] INFO 💬 Setting up conversation processor configure.py:147
Error allocating memory ErrorOutOfDeviceMemory
error loading model: Error allocating vulkan memory.
llama_load_model_from_file: failed to load model
LLAMA ERROR: failed to load model from C:\\Users\\guill\\.cache\\gpt4all\mistral-7b-instruct-v0.1.Q4_0.gguf
[23:52:20] INFO 🔍 📜 Setting up text search model indexer.py:172
...

Configuration : Windows 11 - RAM 16GB - Intel i7
It should be ok for local chat.
I have re-downloaded the mistral file in case of a corrupted file but it doesn't works better.

Any clue on what's going wrong?

Regards,

[debanjum]

Hey @yarel0, thanks for sharing details of the issue you're hitting.

The ErrorOutOfDeviceMemory error is because there isn't enough GPU memory (VRAM) to load the Mistral chat model. See chat docs for system requirements

To load the memory into RAM, try start Khoj with khoj --disable-chat-on-gpu to see if chat works? This will load the mistral chat model onto RAM. Note: In this case, chat will be slower than if the chat model was loaded onto the GPU

[yarel0]

Hello.
Correct.
It is slow but it works thanks to this option.
Thanks, great job!