How to use with local models like llama3.1 70b running on ollama

[Gimel12]

Hi,

I am trying to install it but I want to be able to run it with ollama running as the backend for my LLM on my machine.

How can we do that?

[Gimel12]

I found that there is too many steps for running with local LLms and still not working.

This is what i did:

1- Install gpt
pip install gptme

2- Install litellm
pip install litellm

3- Follow the steps on the documentation

ollama pull mistral
ollama serve
litellm --model ollama/mistral
export OPENAI_API_BASE="http://localhost:8000"

4- Try to launch gptme on terminal
gptme --model local/ollama/llama3.2:1b

And i keep getting errors:

File "/Users/rubenfernandez/.pyenv/versions/3.12.7/lib/python3.12/site-packages/openai/_base_client.py", line 1014, in _request
return self._retry_request(
^^^^^^^^^^^^^^^^^^^^
File "/Users/rubenfernandez/.pyenv/versions/3.12.7/lib/python3.12/site-packages/openai/_base_client.py", line 1092, in _retry_request
return self._request(
^^^^^^^^^^^^^^
File "/Users/rubenfernandez/.pyenv/versions/3.12.7/lib/python3.12/site-packages/openai/_base_client.py", line 1014, in _request
return self._retry_request(
^^^^^^^^^^^^^^^^^^^^
File "/Users/rubenfernandez/.pyenv/versions/3.12.7/lib/python3.12/site-packages/openai/_base_client.py", line 1092, in _retry_request
return self._request(
^^^^^^^^^^^^^^
File "/Users/rubenfernandez/.pyenv/versions/3.12.7/lib/python3.12/site-packages/openai/_base_client.py", line 1024, in _request
raise APIConnectionError(request=request) from err
openai.APIConnectionError: Connection error.

[ProfiGMan]

The exact same issue here. Looks like the support for local/ollama is not implemented.

[ErikBjare]

Looks like some default ports got changed somewhere since I last tried this.

Here are exact steps I just ran to get it working:

# install litellm with proxy
pipx install litellm[proxy]

MODEL=llama3.2:1b
ollama pull $MODEL
ollama serve
litellm --model ollama/$MODEL
OPENAI_API_BASE="http://127.0.0.1:4000" gptme 'hello' -m local/ollama/$MODEL

I will update the documentation and code to make this easier.