[Langchain-AWS-67] Refactor ChatBedrock._combine_llm_outputs() to mat… #68
Conversation
…ch other Langchain model output # Changes * Replaced `usage` with `token_usage` * Replaced `model_id` with `model_name` These change make the Runnable output of Bedrock hosted models consistent with other models supported by Langchain, such as OpenAI chat models.
| final_usage[token_type] += token_count | ||
| final_output.update(output) | ||
| final_output["usage"] = final_usage |
There was a problem hiding this comment.
this is a breaking change.
what we probably want to do is use the new AIMessage.token_usage attribute which is meant to be the standardized way to record token usage: langchain-ai/langchain#21944
There was a problem hiding this comment.
+1, the standard Langchain is moving toward is the usage_metadata attribute on AIMessage, which is just a dict with keys "input_tokens", "output_tokens", "total_tokens". If you're able to access the usage data within the _generate method, you can construct this dict and then populate usage_metadata on the AIMessage it generates. Note that we'd need to bump langchain-core >= 0.2.2 in the dependencies.
|
Believe usage_metadata is now correctly populated on output messages. Please let me know if more development is needed here. |
…ch other Langchain model output
Changes
usagewithtoken_usagemodel_idwithmodel_nameThese change make the Runnable output of Bedrock hosted models consistent with other models supported by Langchain, such as OpenAI chat models.