Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix "test_tool_message_histories_list_content" integration tests for Mistral provider #235

Open
3coins opened this issue Oct 10, 2024 · 0 comments
Labels
bedrock bug Something isn't working
Milestone

Comments

@3coins
Copy link
Collaborator

3coins commented Oct 10, 2024

The standard integration test "test_tool_message_histories_list_content" is failing for ChatBedrockConverse for Mistral models.

Error log

======================================================================= FAILURES =======================================================================
_________________________________________ TestBedrockMistralStandard.test_tool_message_histories_list_content __________________________________________

self = <tests.integration_tests.chat_models.test_bedrock_converse.TestBedrockMistralStandard object at 0x1055067a0>
model = ChatBedrockConverse(disable_streaming='tool_calling', client=<botocore.client.BedrockRuntime object at 0x105537cd0>, m...equences=[], temperature=0.0, region_name='us-west-2', provider='mistral', supports_tool_choice_values=('auto', 'any'))

    @pytest.mark.xfail(reason="Human messages following AI messages not supported.")
    def test_tool_message_histories_list_content(self, model: BaseChatModel) -> None:
>       super().test_tool_message_histories_list_content(model)

tests/integration_tests/chat_models/test_bedrock_converse.py:48: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/Users/pijain/Library/Caches/pypoetry/virtualenvs/langchain-aws-eH7P7gjZ-py3.10/lib/python3.10/site-packages/langchain_standard_tests/integration_tests/chat_models.py:431: in test_tool_message_histories_list_content
    result_list_content = model_with_tools.invoke(messages_list_content)
/Users/pijain/Library/Caches/pypoetry/virtualenvs/langchain-aws-eH7P7gjZ-py3.10/lib/python3.10/site-packages/langchain_core/runnables/base.py:5343: in invoke
    return self.bound.invoke(
/Users/pijain/Library/Caches/pypoetry/virtualenvs/langchain-aws-eH7P7gjZ-py3.10/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py:284: in invoke
    self.generate_prompt(
/Users/pijain/Library/Caches/pypoetry/virtualenvs/langchain-aws-eH7P7gjZ-py3.10/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py:784: in generate_prompt
    return self.generate(prompt_messages, stop=stop, callbacks=callbacks, **kwargs)
/Users/pijain/Library/Caches/pypoetry/virtualenvs/langchain-aws-eH7P7gjZ-py3.10/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py:641: in generate
    raise e
/Users/pijain/Library/Caches/pypoetry/virtualenvs/langchain-aws-eH7P7gjZ-py3.10/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py:631: in generate
    self._generate_with_cache(
/Users/pijain/Library/Caches/pypoetry/virtualenvs/langchain-aws-eH7P7gjZ-py3.10/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py:853: in _generate_with_cache
    result = self._generate(
langchain_aws/chat_models/bedrock_converse.py:492: in _generate
    response = self.client.converse(
/Users/pijain/Library/Caches/pypoetry/virtualenvs/langchain-aws-eH7P7gjZ-py3.10/lib/python3.10/site-packages/botocore/client.py:569: in _api_call
    return self._make_api_call(operation_name, kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <botocore.client.BedrockRuntime object at 0x105537cd0>, operation_name = 'Converse'
api_params = {'inferenceConfig': {'maxTokens': 100, 'stopSequences': [], 'temperature': 0.0}, 'messages': [{'content': [{'text': 'W...'success', 'toolUseId': 'abc123'}}], 'role': 'user'}], 'modelId': 'mistral.mistral-large-2402-v1:0', 'system': [], ...}

    def _make_api_call(self, operation_name, api_params):
        operation_model = self._service_model.operation_model(operation_name)
        service_name = self._service_model.service_name
        history_recorder.record(
            'API_CALL',
            {
                'service': service_name,
                'operation': operation_name,
                'params': api_params,
            },
        )
        if operation_model.deprecated:
            logger.debug(
                'Warning: %s.%s() is deprecated', service_name, operation_name
            )
        request_context = {
            'client_region': self.meta.region_name,
            'client_config': self.meta.config,
            'has_streaming_input': operation_model.has_streaming_input,
            'auth_type': operation_model.resolved_auth_type,
            'unsigned_payload': operation_model.unsigned_payload,
        }
    
        api_params = self._emit_api_params(
            api_params=api_params,
            operation_model=operation_model,
            context=request_context,
        )
        (
            endpoint_url,
            additional_headers,
            properties,
        ) = self._resolve_endpoint_ruleset(
            operation_model, api_params, request_context
        )
        if properties:
            # Pass arbitrary endpoint info with the Request
            # for use during construction.
            request_context['endpoint_properties'] = properties
        request_dict = self._convert_to_request_dict(
            api_params=api_params,
            operation_model=operation_model,
            endpoint_url=endpoint_url,
            context=request_context,
            headers=additional_headers,
        )
        resolve_checksum_context(request_dict, operation_model, api_params)
    
        service_id = self._service_model.service_id.hyphenize()
        handler, event_response = self.meta.events.emit_until_response(
            f'before-call.{service_id}.{operation_name}',
            model=operation_model,
            params=request_dict,
            request_signer=self._request_signer,
            context=request_context,
        )
    
        if event_response is not None:
            http, parsed_response = event_response
        else:
            maybe_compress_request(
                self.meta.config, request_dict, operation_model
            )
            apply_request_checksum(request_dict)
            http, parsed_response = self._make_request(
                operation_model, request_dict, request_context
            )
    
        self.meta.events.emit(
            f'after-call.{service_id}.{operation_name}',
            http_response=http,
            parsed=parsed_response,
            model=operation_model,
            context=request_context,
        )
    
        if http.status_code >= 300:
            error_info = parsed_response.get("Error", {})
            error_code = error_info.get("QueryErrorCode") or error_info.get(
                "Code"
            )
            error_class = self.exceptions.from_code(error_code)
>           raise error_class(parsed_response, operation_name)
E           botocore.errorfactory.ValidationException: An error occurred (ValidationException) when calling the Converse operation: messages.1.content: Conversation blocks and tool use blocks cannot be provided in the same turn.

/Users/pijain/Library/Caches/pypoetry/virtualenvs/langchain-aws-eH7P7gjZ-py3.10/lib/python3.10/site-packages/botocore/client.py:1023: ValidationException
@3coins 3coins changed the title Fix ChatBedrockConverse to support "test_tool_message_histories_list_content" integration tests Fix "test_tool_message_histories_list_content" integration tests for Mistral provider Oct 10, 2024
@3coins 3coins added bug Something isn't working bedrock labels Oct 11, 2024
@3coins 3coins added this to the v0.2.5 milestone Oct 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bedrock bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant