Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve image search UX #431

Open
ohltyler opened this issue Oct 17, 2024 · 2 comments
Open

Improve image search UX #431

ohltyler opened this issue Oct 17, 2024 · 2 comments
Labels
feature A new feature for the plugin v2.19.0

Comments

@ohltyler
Copy link
Member

Images must be in binary or base64-encoded formats to be understood by popular MM embedding models. As such, we should add helper UI around producing such formats, based on raw images users can drag into the UI. And/or, provide a URL to some image.

The rough flow can be:

  1. Users opens a portal for selecting an image from local filesystem, dragging directly, or providing a url
  2. User selects what format they want to convert it to (can be auto-populated based on the associated model, ideally)
  3. UI handles conversion and injecting into the sample document (if ingest) or query (if search)

Furthermore, URL support is targeted to be an optional input by 2.19 - see opensearch-project/ml-commons#3054. As such, we can add URL with no further conversion as an option once that is available.

@ohltyler ohltyler added feature A new feature for the plugin v2.19.0 labels Oct 17, 2024
@brianf-aws
Copy link

Hi Tyler currently looking into a feature that targets your use case

Im trying to think the big picture what are some use cases a user would do once they have this requirement. For example searching an image by text "a orange cat". But what else are we aiming for?

@ohltyler
Copy link
Member Author

This issue is around allowing users on the UI to search with an image, such as to find related images, as one example. This may / may not also include text.

This issue is just around the UI supporting searching with an image, and converting it to the supported formats. The open ML commons issue is just expanding on the formats/types that can be used as input for an image (a url).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature A new feature for the plugin v2.19.0
Projects
None yet
Development

No branches or pull requests

2 participants