Custom API Integration for Enhanced Model Control

[SilkePilon]

Hi everyone!

I have an idea suggestion for adding support that allows users to define their own API instead of relying solely on OpenAI or Hugging Face models. This feature would enable users to utilize a custom API as a model, which would enable them to split the code into two parts. The first part would handle the guidance code, while the second part would be responsible for running the model on a more powerful PC.

as an example:

import guidance

# set the default language model to an custom API.
guidance.llm = guidance.llms.Custom("https://api.example.com/myownapi", json_token="generated_text")

[slundberg]

In fact we plan to create a Guidance API so that you can start a guidance server on one machine and then use guidance like a client to run inference against that server. The key thing about that setup is that then you can execute any local library commands on the remote server without any round trip network latency (since you can ship the guidance program to the server and then ship the executed (or partially executed) program back to the client)