README.md

openplayground

An LLM playground.

Forked from https://github.com/nat/openplayground, I've added more default models and some cosmetic updates. Feel free to submit PR's.

Features

• Use any model from OpenAI, Anthropic, Cohere, Forefront, HuggingFace, Aleph Alpha, Replicate, Banana and llama.cpp.
• Full playground UI, including history, parameter tuning, keyboard shortcuts, and logprops.
• Compare models side-by-side with the same prompt, individually tune model parameters, and retry with different parameters.
• Automatically detects local models in your HuggingFace cache, and lets you install new ones.
• Works OK on your phone.
• Probably won't kill everyone.

How to run for development

# set up python env
python -m venv venv
source venv/bin/activate

# install
make install

# in one terminal run the frontend
make frontend

# in another terminal
make backend
# or
make backend-dev

# clean config (after making changes to model.json)
make clean

Contributions welcome. Some ideas:

• Add a token counter #1
• Ability to Delete local Hugging Face models from the cache #2
• Display progress-bar when downloading Hugging Face models #3
• Add a cost counter compare page #4
• Measure and display time to first token #5
• Chatgpt plugin support #6

Adding models to openplayground

Models and providers have three types in openplayground:

• Searchable
• Local inference
• API

You can add models in server/models.json with the following schema:

Local inference

For models running locally on your device you can add them to openplayground like the following (a minimal example):

"llama": {
    "api_key" : false,
    "models" : {
        "llama-70b": {
            "parameters": {
                "temperature": {
                    "value": 0.5,
                    "range": [
                        0.1,
                        1.0
                    ]
                },
            }
        }
    }
}

Keep in mind you will need to add a generation method for your model in server/app.py. Take a look at local_text_generation() as an example.

API Provider Inference

This is for model providers like OpenAI, cohere, forefront, and more. You can connect them easily into openplayground (a minimal example):

"cohere": {
    "api_key" : true,
    "models" : {
        "xlarge": {
            "parameters": {
                "temperature": {
                    "value": 0.5,
                    "range": [
                        0.1,
                        1.0
                    ]
                },
            }
        }
    }
}

You will need to add a generation method for your model in server/app.py. Take a look at openai_text_generation() or cohere_text_generation() as an example.

Searchable models

We use this for Huggingface Remote Inference models, the search endpoint is useful for scaling to N models in the settings page.

"provider_name": {
    "api_key": true,
    "search": {
        "endpoint": "ENDPOINT_URL"
    },
    "parameters": {
        "parameter": {
            "value": 1.0,
            "range": [
                0.1,
                1.0
            ]
        },
    }
}

Credits

Instigated by Nat Friedman. Initial implementation by Zain Huda as a repl.it bounty. Many features and extensive refactoring by Alex Lourenco.