feat: Add chat completion mutation

[anticorrelator]

resolves #4774

Adds a playgroundChatCompletion mutation that returns a FinishedChatCompletion and accepts the same payload as the chatCompletion subscription

For simplicity, for now it simply reuses the streaming interfaces we've used, collates the results and returns them all at the end

[Parker-Stafford]

can we get this returned as an error so it's easier to display in the front end, the FinishedChatCompletion type has an errors field right?

[axiomofjoy]

BadRequest is a custom graphql error, so it should be displayable in the frontend

[axiomofjoy]

Relay has an onError hook https://relay.dev/docs/api-reference/use-mutation/#return-value

[Parker-Stafford]

yeah it just requires a whole lot of parsing to get the real error from the relay error

[axiomofjoy]

i see. probably a broader discussion if we want to add error types on mutations / query. we've avoided that pattern for the sake of simplicity so far.

[Parker-Stafford]

don't we have that on the streaming version of this though? an error member of the union, i guess that is technically not mutation or query (subscription) but was thinking it would set the precedent here

[Parker-Stafford]

but yes don't need to use it here just saying it can be helpful for handling known errors more gracefully for us, I know it's been shared before but this is the inspiration https://sachee.medium.com/200-ok-error-handling-in-graphql-7ec869aec9bc
and
https://productionreadygraphql.com/2020-08-01-guide-to-graphql-errors

[axiomofjoy]

don't we have that on the streaming version of this though? an error member of the union, i guess that is technically not mutation or query (subscription) but was thinking it would set the precedent here

the error types in the subscription were unavoidable because we need to continue yielding payloads (FinishedChatCompletion in particular) even after an error has occurred. We don't currently have the same pattern on mutations and queries afaik

[Parker-Stafford]

ahh okay, i didn't know that was why those were added, we can punt it for now, we need to parse errors from relay anyways for various other pages so will make a ticket to get that working

[Parker-Stafford]

nit: I know lots of other things are named as playground in here, any reason they would not be usable for generating a chat completion outside of the playground? would probably prefer to name it something like generate_chat_completion unless there is a reason we think this won't work outside of hte playground

[anticorrelator]

there are a ton of playground-specific type definitions and I think our codebase has a lot of generic names that aren't actually reusable

[Parker-Stafford]

Yeah that makes a ton of sense was just curious but feel free to keep it!

[axiomofjoy]

I think this should be chat_completion since it is part of the GraphQL API. A consumer of the GraphQL API could use this resolver to do a chat completion regardless of whether they are in playground.

[Parker-Stafford]

oh interesting is this just called span because it's named that on line 73, feels a bit weird to have the chat completion attached to a span

[axiomofjoy]

this is the way the streaming_llm_span context manager works atm. it's changed in my open pr

[axiomofjoy]

Suggested change

	chunks = []
	async for chunk in llm_client.chat_completion_create(
	messages=messages, tools=input.tools or [], **invocation_parameters
	):
	span.add_response_chunk(chunk)
	chunks.append(chunk)
	async for chunk in llm_client.chat_completion_create(
	messages=messages, tools=input.tools or [], **invocation_parameters
	):
	span.add_response_chunk(chunk)

[Parker-Stafford]

I think this return type needs to be something like

@strawberry.type
class ChatCompletionText:
    content: str
    span: Span


@strawberry.type
class ChatCompletionError:
    message: str
    span: Span # maybe don't need this or definitely optional


@strawberry.type
class ChatCompletionToolCall:
    id: str
    function: FunctionCall # this is just json i think
    span: Span


ChatCompletionMutationPayload: TypeAlias = Annotated[
    Union[ChatCompletionText, ChatCompletionToolCall, ChatCompletionError],
    strawberry.union("ChatCompletionMutationPayload"),
]

not just the span, if i'm understanding correctly

[axiomofjoy]

I agree

[Parker-Stafford]

Any reason to not mke this a union, also do tool_calls need to be Optional?

[Parker-Stafford]

if this is not apart of a union but an optional portion of the result type, it doesn't need the sapn since the span will be on the top level result