Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Bedrock multimodal model interface can't work with neural search #3261

Open
ylwu-amzn opened this issue Dec 6, 2024 · 1 comment · May be fixed by #3289
Open

[BUG] Bedrock multimodal model interface can't work with neural search #3261

ylwu-amzn opened this issue Dec 6, 2024 · 1 comment · May be fixed by #3289
Assignees
Labels
bug Something isn't working

Comments

@ylwu-amzn
Copy link
Collaborator

What is the bug?
Model interface for Bedrock multi-modal model can't work with neural search

How can one reproduce the bug?
Steps to reproduce the behavior:

  1. Create connector and model following this blueprint
  2. Create ingest pipeline and try to ingest doc following this doc , then can see such error when ingest doc
PUT /my-nlp-index/_doc/1
{
 "image_description": "Orange table"
}

error

{
  "error": {
    "root_cause": [
      {
        "type": "status_exception",
        "reason": "Error validating input schema: Validation failed: [$: required property 'parameters' not found] for instance: {\"algorithm\":\"REMOTE\",\"text_docs\":[\"Orange table\"],\"return_bytes\":false,\"return_number\":true,\"target_response\":[\"sentence_embedding\"]} with schema: {\n    \"type\": \"object\",\n    \"properties\": {\n        \"parameters\": {\n            \"type\": \"object\",\n            \"properties\": {\n                \"inputText\": {\n                    \"type\": \"string\"\n                },\n                \"inputImage\": {\n                    \"type\": \"string\"\n                }\n            }\n        }\n    },\n    \"required\": [\n        \"parameters\"\n    ]\n}"
      }
    ],
    "type": "status_exception",
    "reason": "Error validating input schema: Validation failed: [$: required property 'parameters' not found] for instance: {\"algorithm\":\"REMOTE\",\"text_docs\":[\"Orange table\"],\"return_bytes\":false,\"return_number\":true,\"target_response\":[\"sentence_embedding\"]} with schema: {\n    \"type\": \"object\",\n    \"properties\": {\n        \"parameters\": {\n            \"type\": \"object\",\n            \"properties\": {\n                \"inputText\": {\n                    \"type\": \"string\"\n                },\n                \"inputImage\": {\n                    \"type\": \"string\"\n                }\n            }\n        }\n    },\n    \"required\": [\n        \"parameters\"\n    ]\n}"
  },
  "status": 400
}

What is the expected behavior?
This is a break change. We should not break BWC by introducing model interface.

What is your host/environment?

  • OS: Linux
  • Version 2.17.1
  • Plugins

Do you have any screenshots?
If applicable, add screenshots to help explain your problem.

Do you have any additional context?
Add any other context about the problem.

@maltelorbach
Copy link

Same behaviour in 2.18.0.

Digging into the code, I think the issue is the algorithm type used by the text_image_embedding processor. It uses a hardcoded value FunctionName.TEXT_EMBEDDING when instantiating the MLInput here:
https://github.com/opensearch-project/neural-search/blob/7feacd67b3c7694ff4a1c1c2b430f2447a1ed4ab/src/main/java/org/opensearch/neuralsearch/ml/MLCommonsClientAccessor.java#L293

but for Bedrock (and possibly others), the algorithm type should be FunctionName.REMOTE such that the inputs are parsed from parameters:

case REMOTE:
RemoteInferenceInputDataSet remoteInferenceInputDataSet = (RemoteInferenceInputDataSet) this.inputDataset;
Map<String, String> parameters = remoteInferenceInputDataSet.getParameters();
builder.field(PARAMETERS_FIELD, parameters);

@dhrubo-os dhrubo-os moved this to In Progress in ml-commons projects Dec 17, 2024
@b4sjoo b4sjoo linked a pull request Dec 18, 2024 that will close this issue
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: In Progress
Development

Successfully merging a pull request may close this issue.

4 participants