Add ollama support #313

keturiosakys · 2024-10-09T03:41:31Z

A cursory test of the Llama 3.1 model shows that it has become quite capable of supporting some of our request generation use cases. This makes for a good case to add support for using the Fiberplane Studio with Ollama.

To test:

Set up ollama locally, make sure you have llama 3.1 (8B) running: ollama run llama3.1
Rebuild types (pnpm build:types from the root)
Go to settings, and set your model provider to ollama

api/src/lib/ai/ollama.ts

brettimus · 2024-10-09T13:49:08Z

api/src/lib/ai/prompts.ts

+ * NOTE - I had to stop instructing the AI to create very long data in this prompt.
+ *        It would end up repeating 9999999 ad infinitum and break JSON responses.
+ */
+export const LLAMA_3_8B_QA_PARAMETER_GENERATION_SYSTEM_PROMPT = cleanPrompt(`


NOTE!!! I have not updated the "hostile" prompt at all yet, it may run into issues

brettimus · 2024-10-09T13:50:56Z

api/src/lib/ai/prompts.ts

@@ -302,6 +308,176 @@ Use the tool "make_request". Always respond in valid JSON.
 ***Don't make your responses too long, otherwise we cannot parse your JSON response.***
 `);

+export const LLAMA_3_8B_FRIENDLY_PARAMETER_GENERATION_SYSTEM_PROMPT =


this prompt has been modified from the OG system prompt we usually send to anthropic and openai

i found that reducing the scope of what we're trying to do worked better. so, we ONLY tell it to deal with json bodies, not worry about text + form data + etc

api/src/routes/inference/inference.ts

packages/types/src/settings.ts

studio/src/pages/SettingsPage/AISettingsForm.tsx

…n be both reused

…cluding a sample prompt

pkg-pr-new · 2024-10-21T22:17:58Z

Open in Stackblitz

pnpm add https://pkg.pr.new/fiberplane/fpx/@fiberplane/studio@313

pnpm add https://pkg.pr.new/fiberplane/fpx/@fiberplane/hono-otel@313

commit: 9dab0b4

mies · 2024-10-21T23:04:09Z

we should show a nice message when you've picked a model in ollama that has not been pulled

api/package.json

keturiosakys and others added 12 commits October 8, 2024 20:36

initial soup of commits to add ollama support

e07c382

formatting

240ba6a

Add mechanism to invoke ollama-specific prompts

6e3fbd8

Add some tweaks to ollama prompting and response parsing

41e6969

Remove excessive logging

4cabaf0

Reinforce schema in prompt

eab12e2

Merge branch 'main' into ollama

01f7984

Add handler context - 3.1b has a big context window

2e75e6c

Do not use static anlaysis when ai provider is ollama

9a26c9b

Remove console.log

b5caf4c

Make ollamaBaseUrl optional

ae3f615

Remove default value from the ollamaBaseUrl type

269e4ea