We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
What is the bug? Model interface for Bedrock multi-modal model can't work with neural search
How can one reproduce the bug? Steps to reproduce the behavior:
PUT /my-nlp-index/_doc/1 { "image_description": "Orange table" }
error
{ "error": { "root_cause": [ { "type": "status_exception", "reason": "Error validating input schema: Validation failed: [$: required property 'parameters' not found] for instance: {\"algorithm\":\"REMOTE\",\"text_docs\":[\"Orange table\"],\"return_bytes\":false,\"return_number\":true,\"target_response\":[\"sentence_embedding\"]} with schema: {\n \"type\": \"object\",\n \"properties\": {\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"inputText\": {\n \"type\": \"string\"\n },\n \"inputImage\": {\n \"type\": \"string\"\n }\n }\n }\n },\n \"required\": [\n \"parameters\"\n ]\n}" } ], "type": "status_exception", "reason": "Error validating input schema: Validation failed: [$: required property 'parameters' not found] for instance: {\"algorithm\":\"REMOTE\",\"text_docs\":[\"Orange table\"],\"return_bytes\":false,\"return_number\":true,\"target_response\":[\"sentence_embedding\"]} with schema: {\n \"type\": \"object\",\n \"properties\": {\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"inputText\": {\n \"type\": \"string\"\n },\n \"inputImage\": {\n \"type\": \"string\"\n }\n }\n }\n },\n \"required\": [\n \"parameters\"\n ]\n}" }, "status": 400 }
What is the expected behavior? This is a break change. We should not break BWC by introducing model interface.
What is your host/environment?
Do you have any screenshots? If applicable, add screenshots to help explain your problem.
Do you have any additional context? Add any other context about the problem.
The text was updated successfully, but these errors were encountered:
Same behaviour in 2.18.0.
Digging into the code, I think the issue is the algorithm type used by the text_image_embedding processor. It uses a hardcoded value FunctionName.TEXT_EMBEDDING when instantiating the MLInput here: https://github.com/opensearch-project/neural-search/blob/7feacd67b3c7694ff4a1c1c2b430f2447a1ed4ab/src/main/java/org/opensearch/neuralsearch/ml/MLCommonsClientAccessor.java#L293
text_image_embedding
FunctionName.TEXT_EMBEDDING
MLInput
but for Bedrock (and possibly others), the algorithm type should be FunctionName.REMOTE such that the inputs are parsed from parameters:
FunctionName.REMOTE
parameters
ml-commons/common/src/main/java/org/opensearch/ml/common/input/MLInput.java
Lines 203 to 206 in c2a40c1
Sorry, something went wrong.
b4sjoo
Successfully merging a pull request may close this issue.
What is the bug?
Model interface for Bedrock multi-modal model can't work with neural search
How can one reproduce the bug?
Steps to reproduce the behavior:
error
What is the expected behavior?
This is a break change. We should not break BWC by introducing model interface.
What is your host/environment?
Do you have any screenshots?
If applicable, add screenshots to help explain your problem.
Do you have any additional context?
Add any other context about the problem.
The text was updated successfully, but these errors were encountered: