Nix wrapper for running LLMs behind an OpenAI-compatible API proxy.
Important
This project is not designed to be used on NixOS, we do not use patchelf since that caused issues with the CUDA runtime.
Instead, it is intended to be used with the nix package manager on other Linux distributions like Ubuntu.
You may use it together with nix-ld on NixOS.
CUDA_VISIBLE_DEVICES=0 OLLAMA_HOST=0.0.0.0:6060 nix run github:recap-utr/nixllm#ollama -- serve
# then, in another terminal pull the models before performing API requests
OLLAMA_HOST=0.0.0.0:6060 nix run github:recap-utr/nixllm#ollama -- pull MODEL_NAME