Enable messages api

[radames]

Address #574: The idea is to utilize the endpoint as the base client, requiring users to specify the model name.

• I had to replace model with endpointUrl because model is a required parameter for all APIs, including TGI.
• Types are still incomplete, but choices?: Choice[]; is included in the StreamResponse when using messageAPI, maybe we can have a way to differentiate the input.
• For custom streams endpoints, options are not needed and often throw a backend error, so it's preferable to exclude them on streamRequest

Considering the requirement for the model parameter, we might not need TaskWithNoAccessTokenNoModel WDYT?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[julien-c]

be sure to tag @SBrandeis and @Wauplin for a review when this is ready to review (given we'd use the generated types)

[radames]

interesting, point @julien-c, reading @Wauplin implementation here huggingface/huggingface_hub#2094 it's very complete! , in that regard, it makes sense to also support similar API with the js client in another PR, where we can use @xenova jinja + hub , to build the chat_template in case the user wants to use chat completion with a simple text generation model

[Wauplin]

interesting, point @julien-c, reading @Wauplin implementation here huggingface/huggingface_hub#2094 it's very complete! , in that regard, it makes sense to also support similar API with the js client in another PR, where we can use @xenova jinja + hub , to build the chat_template in case the user wants to use chat completion with a simple text generation model

@radames I agree focus should be put on handling the /v1/chat/completions route (i.e. server-side template rendering). The solution with jinja+hub is nice but in the end, only useful for a very few models AFAIK. Typically the microsoft/DialoGPT-*** family because all major chat-based LLMs are already supported and served with TGI. If I had to start over the PR in huggingface_hub I wouldn't start with the client-side rendering 😬

EDIT: especially the part where I try to handle Inference Endpoints URLs for which I don't have the model_id / chat_template. It's quite complex logic -hence the figure- for very little impact IMO, retrospectively 😕

[Wauplin]

I do think it'd be best to separate completely chatCompletion from the textGeneration method. Both are generating text but their API is very different. In the new spec-ed types we differenciated them (see chat completion here, and text generation here).

In particular parameters/options are not sent on the same level. For all HF tasks, we have a parameters key that is a mapping with all parameters. For chat completion, we wanted to mimic OpenAI's API which sets all the options at the root level. Also, output types are completely different and we don't benefit from combining them IMO.

Also, chat completion URL is not the same for models served on serverless Inference API. Usually, the url is https://api-inference.huggingface.co/models/{model_id}. For chat-completion, it's https://api-inference.huggingface.co/models/{model_id}/v1/chat/completions. In huggingface_hub, this is handled here. This rule also applies for TGI-served models where / serves text-generation API while /v1/chat/completions serves chat-completion API.

following @Wauplin's review

[julien-c]

thanks for the awesome review + help + feedback from the python implem, @Wauplin

[radames]

Hi @Wauplin, thank you for the feedback. I think separating chatCompletion and textGeneration is a great idea!
Also, @coyotte508, is it the plan to remove all types from the inference package and unify by using types from tasks? it's already possible to do this
import type { TextGenerationInput, TextGenerationStreamOutput } from "@huggingface/tasks";

[julien-c]

Also, @coyotte508, is it the plan to remove all types from the inference package and unify by using types from tasks?

indeed

[coyotte508]

Yes but we need to handle #584 first (and also using the types for validation would be great)

[radames]

shall we wait then to split this into chatCompletion and textGeneration ?

[coyotte508]

yes it should definitely be split.

Maybe make a separate PR / start from scratch? Maybe we should pass model along only for chatCompletion as well? (still we can move from model to endpointUrl)

[gary149]

Any news on this? that is quite needed imo

[radames]

Any news on this? that is quite needed imo

hi @gary149 I'll do a new PR today, this one will be throw away in favor of a split creating a new chatCompletion method that will be cleaner than using it all with textGeneration , as well as matching the python api

[radames]

close in favor of #645