AnthropicChatGenerator StreamingChunk "meta" is empty

[wursterje]

I am looking for some meta information, similar to the OpenAIChatGenerator, to capture the stop event. Unfortunately, the meta field for the AnthropicChatGenerator is empty.

I added some print statements for debugging to AnthropicChatGenerator (line 221 and 227):

  # if streaming is enabled, the response is a Stream[MessageStreamEvent]
        if isinstance(response, Stream):
            chunks: List[StreamingChunk] = []
            stream_event, delta, start_event = None, None, None
            for stream_event in response:
                print(stream_event)
                if isinstance(stream_event, MessageStartEvent):
                    # capture start message to count input tokens
                    start_event = stream_event
                if isinstance(stream_event, ContentBlockDeltaEvent):
                    chunk_delta: StreamingChunk = self._build_chunk(stream_event.delta)
                    print(chunk_delta)
                    chunks.append(chunk_delta)

And here is the output where you can see that the meta field is empty:

RawContentBlockDeltaEvent(delta=TextDelta(text='d of test', type='text_delta'), index=0, type='content_block_delta')
StreamingChunk(content='d of test', meta={})
RawContentBlockDeltaEvent(delta=TextDelta(text=" you're", type='text_delta'), index=0, type='content_block_delta')
StreamingChunk(content=" you're", meta={})
RawContentBlockDeltaEvent(delta=TextDelta(text=' referring to? Once', type='text_delta'), index=0, type='content_block_delta')
StreamingChunk(content=' referring to? Once', meta={})
RawContentBlockDeltaEvent(delta=TextDelta(text=' you give', type='text_delta'), index=0, type='content_block_delta')
StreamingChunk(content=' you give', meta={})
RawContentBlockDeltaEvent(delta=TextDelta(text=' me more context or', type='text_delta'), index=0, type='content_block_delta')
StreamingChunk(content=' me more context or', meta={})
RawContentBlockDeltaEvent(delta=TextDelta(text=' a specific question, I', type='text_delta'), index=0, type='content_block_delta')
StreamingChunk(content=' a specific question, I', meta={})
RawContentBlockDeltaEvent(delta=TextDelta(text="'ll do my best to", type='text_delta'), index=0, type='content_block_delta')
StreamingChunk(content="'ll do my best to", meta={})
RawContentBlockDeltaEvent(delta=TextDelta(text=' assist you', type='text_delta'), index=0, type='content_block_delta')
StreamingChunk(content=' assist you', meta={})
RawContentBlockDeltaEvent(delta=TextDelta(text='.', type='text_delta'), index=0, type='content_block_delta')
StreamingChunk(content='.', meta={})
RawContentBlockStopEvent(index=0, type='content_block_stop')
RawMessageDeltaEvent(delta=Delta(stop_reason='end_turn', stop_sequence=None), type='message_delta', usage=MessageDeltaUsage(output_tokens=93))
RawMessageStopEvent(type='message_stop')

Version: anthropic-haystack 1.1.0

[Amnah199]

Hi @wursterje,
I reviewed the mentioned issue and it appears that you're trying to capture meta for RawContentBlockDeltaEvent, which isn't provided in the streaming responses from the Anthropic API.

In Anthropic's streaming API, responses include a series of structured events, such as RawMessageStartEvent at the start and RawMessageDeltaEvent throughout, which provide updates like stop_reason and stop_sequence. These events should contain the meta you are looking for, such as stop_sequence.

Reference: https://docs.anthropic.com/en/api/messages-streaming

[wursterje]

@Amnah199
I'm using OpenAIChatGenerator, MistralChatGenerator, and AnthropicChatGenerator, passing a streaming_callback. I expect the behavior to be the same for all three, but it is not for AnthropicChatGenerator.

This is my code for the streaming_callback, and I need to get the stop reason there for Anthropic as well. It would be very easy if I received the reason in the meta field, as I do with the other generators.

def add(self, document):
    self.response += document.content
    self.on_new_chunk(self.response)

    if ("done" in document.meta and document.meta["done"]) or (
        "finish_reason" in document.meta and document.meta["finish_reason"] == "stop"
    ):
        self.finish_event.set()