MultimodalQnA Image and Audio Support Phase 1 #1071

mhbuehler · 2024-11-05T00:52:13Z

Description

This PR adds the following new features as specified in "Phase 1" of this RFC. The related PR in GenAIComps is opea-project/GenAIComps#852 and it will need to be merged before this one in GenAIExamples.

Data prep and ingestion enhancements:

Accept image only
Accept image and text
Accept speech audio only

Other enhancements:

Allow the user to choose the embedding model and LVM when starting the services

Note that the planned query enhancement "Accept speech audio only" has been moved to Phase 2 and a PR for that phase will be submitted for the next release.

Issues

MultimodalQnA Image & Audio Support RFC

Type of change

List the type of change like below. Please delete options that are not relevant.

New feature (non-breaking change which adds new functionality)
Breaking change (fix or feature that would break existing design and interface)
Others (enhancement, documentation, validation, etc.)

Dependencies

No new dependencies

Tests

Updated the individual microservice's test scripts, the GenAIExamples' MultimodalQnA test scripts, and did manual testing of the UI and documented curl commands.

Signed-off-by: Melanie Buehler <[email protected]>

* added entrypoint override Signed-off-by: okhleif-IL <[email protected]> * fixed entrypoint container Signed-off-by: okhleif-IL <[email protected]> * removed workdir Signed-off-by: okhleif-IL <[email protected]> * reverted set env vars changes Signed-off-by: okhleif-IL <[email protected]> * added entrypoint for embedding model Signed-off-by: okhleif-IL <[email protected]> * added llava model to readme Signed-off-by: okhleif-IL <[email protected]> * removed stray \n Signed-off-by: okhleif-IL <[email protected]> * added entrypoint override in bridgetower hpu Signed-off-by: okhleif-IL <[email protected]> --------- Signed-off-by: okhleif-IL <[email protected]>

Signed-off-by: Melanie Buehler <[email protected]>

…e/mm-rag-enhanced

* changed variable names Signed-off-by: okhleif-IL <[email protected]> * changed gaudi env vars Signed-off-by: okhleif-IL <[email protected]> * changed vars + docs Signed-off-by: okhleif-IL <[email protected]> * verified + updated readme Signed-off-by: okhleif-IL <[email protected]> * updated readme Signed-off-by: okhleif-IL <[email protected]> * updated tests Signed-off-by: okhleif-IL <[email protected]> --------- Signed-off-by: okhleif-IL <[email protected]>

Signed-off-by: okhleif-IL <[email protected]>

Signed-off-by: Melanie Buehler <[email protected]>

…estion Image ingestion improvements

…e/mm-rag-enhanced

Signed-off-by: dmsuehir <[email protected]>

Update LVM model for Xeon

Signed-off-by: Melanie Buehler <[email protected]>

Signed-off-by: dmsuehir <[email protected]>

…e/mm-rag-enhanced

Signed-off-by: dmsuehir <[email protected]>

Signed-off-by: Melanie Buehler <[email protected]>

Image+Text Upload

Signed-off-by: Melanie Buehler <[email protected]>

UI Audio Upload

Signed-off-by: Melanie Buehler <[email protected]>

UI fixes

for more information, see https://pre-commit.ci

Signed-off-by: okhleif-IL <[email protected]>

ashahba

Thanks @mhbuehler @dmsuehir and @okhleif-IL for this well put together PR!

LGTM!

Signed-off-by: Melanie Buehler <[email protected]>

Updated main README and supported_examples with MMQA details

for more information, see https://pre-commit.ci

Fix for lvm-dependent tests and clear textbox

Signed-off-by: okhleif-IL <[email protected]>

ashahba · 2024-11-06T18:17:31Z

MultimodalQnA/tests/test_compose_on_gaudi.sh

 export video_fn="WeAreGoingOnBullrun.mp4"
+export caption_fn="apple.txt"

 function build_docker_images() {
    cd $WORKPATH/docker_image_build


Please update the line 23 so that you are cloning the right repo/branch during tests and while this PR opea-project/GenAIComps#852 still being merged.

git clone https://github.com/opea-project/GenAIComps.git should be replaced with:

git clone --single-branch --branch="melanie/mm-rag-enhanced" https://github.com/mhbuehler/GenAIComps.git

ashahba · 2024-11-06T18:19:49Z

MultimodalQnA/tests/test_compose_on_xeon.sh

 export video_fn="WeAreGoingOnBullrun.mp4"
+export caption_fn="apple.txt"

 function build_docker_images() {
    cd $WORKPATH/docker_image_build


Please update the line 23 so that you are cloning the right repo/branch during tests and while this PR opea-project/GenAIComps#852 still being merged.

git clone https://github.com/opea-project/GenAIComps.git should be replaced with:

git clone --single-branch --branch="melanie/mm-rag-enhanced" https://github.com/mhbuehler/GenAIComps.git

Signed-off-by: Melanie Buehler <[email protected]>

Fixed typos for MMQA in Docker Image List doc

for more information, see https://pre-commit.ci

Clone specific branch of GenAIComps for tests

ashahba

Putting this PR on hold until opea-project/GenAIComps#852 is merged.

ashahba · 2024-11-06T21:44:43Z

MultimodalQnA/tests/test_compose_on_gaudi.sh

+    # TODO: Revert to the clone of opea-project "main" after the merge of https://github.com/opea-project/GenAIComps/pull/852
+    # git clone https://github.com/opea-project/GenAIComps.git && cd GenAIComps && git checkout "${opea_branch:-"main"}" && cd ../
+    git clone --single-branch --branch="melanie/mm-rag-enhanced" https://github.com/mhbuehler/GenAIComps.git


As said in the comments this will be reverted prior to merged.

ashahba · 2024-11-06T21:45:07Z

MultimodalQnA/tests/test_compose_on_xeon.sh

+    # TODO: Revert to the clone of opea-project "main" after the merge of https://github.com/opea-project/GenAIComps/pull/852
+    # git clone https://github.com/opea-project/GenAIComps.git && cd GenAIComps && git checkout "${opea_branch:-"main"}" && cd ../
+    git clone --single-branch --branch="melanie/mm-rag-enhanced" https://github.com/mhbuehler/GenAIComps.git


As said in the comments this will be reverted prior to merged.

Signed-off-by: dmsuehir <[email protected]>

for more information, see https://pre-commit.ci

Signed-off-by: dmsuehir <[email protected]>

mhbuehler and others added 30 commits October 14, 2024 16:28

Initial implementation of image ingestion

97f4bc0

Signed-off-by: Melanie Buehler <[email protected]>

Use same endpoint for image and video ingestion

56bd8c3

Signed-off-by: Melanie Buehler <[email protected]>

Update tests and docs

b334dc2

Signed-off-by: Melanie Buehler <[email protected]>

Merge branch 'main' of github.com:mhbuehler/GenAIExamples into melani…

13c752a

…e/mm-rag-enhanced

added LVM_MODEL_ID var to test file (#4)

271117e

Signed-off-by: okhleif-IL <[email protected]>

Updates tests per feedback

c4e0259

Signed-off-by: Melanie Buehler <[email protected]>

Merge pull request #2 from mhbuehler/melanie/combined_image_video_ing…

a541140

…estion Image ingestion improvements

Merge branch 'main' of github.com:mhbuehler/GenAIExamples into melani…

30c311d

…e/mm-rag-enhanced

Update LVM model for Xeon

e83fc44

Signed-off-by: dmsuehir <[email protected]>

Merge pull request #5 from mhbuehler/dina/lvm_model

4b8a5ad

Update LVM model for Xeon

Initial setup for ingest_with_text

a6d826c

Signed-off-by: Melanie Buehler <[email protected]>

Write and send custom caption file

547a139

Signed-off-by: Melanie Buehler <[email protected]>

Update docs and tests

7cfc343

Signed-off-by: Melanie Buehler <[email protected]>

MMQnA doc updates for audio ingestion (#7)

69dbdfc

Signed-off-by: dmsuehir <[email protected]>

Merge branch 'main' of github.com:mhbuehler/GenAIExamples into melani…

834c668

…e/mm-rag-enhanced

Fix UI request for follow up queries with no image (#8)

a8f8dc9

Signed-off-by: dmsuehir <[email protected]>

Updated for review suggestions

718d02e

Signed-off-by: Melanie Buehler <[email protected]>

Merge branch 'melanie/mm-rag-enhanced' into melanie/images_and_text

d05cfb3

Add audio upload functionality to UI

72591c1

Signed-off-by: Melanie Buehler <[email protected]>

Merge pull request #6 from mhbuehler/melanie/images_and_text

431e41b

Image+Text Upload

Merge branch 'melanie/mm-rag-enhanced' into melanie/audio_ingest_ui

d535aa7

Minor refactor, improve display text, and suppress PDF tab

39f43fc

Signed-off-by: Melanie Buehler <[email protected]>

Merge pull request #9 from mhbuehler/melanie/audio_ingest_ui

afc3c8a

UI Audio Upload

Small fixes

ddd5dfb

Signed-off-by: Melanie Buehler <[email protected]>

Improve appearance

426c739

Signed-off-by: Melanie Buehler <[email protected]>

Improve upload errors and revert multimodal query box

cfa1c8c

Signed-off-by: Melanie Buehler <[email protected]>

Small text edit as suggested

cdec83f

Signed-off-by: Melanie Buehler <[email protected]>

Merge pull request #11 from mhbuehler/melanie/mm-fixes

48baceb

UI fixes

[pre-commit.ci] auto fixes from pre-commit.com hooks

3a23e5b

for more information, see https://pre-commit.ci

mhbuehler mentioned this pull request Nov 5, 2024

MultimodalQnA Image and Audio Support Phase 1 opea-project/GenAIComps#852

Open

3 tasks

ashahba requested review from ashahba and tileintel November 5, 2024 18:06

ashahba added this to the v1.1 milestone Nov 5, 2024

ashahba added the r1.1 label Nov 5, 2024

ashahba and others added 7 commits November 5, 2024 10:49

Merge branch 'main' into melanie/mm-rag-enhanced

65a8afd

updated readmes with MMQA info

f95b946

Signed-off-by: okhleif-IL <[email protected]>

removed stray char

c4d5138

Signed-off-by: okhleif-IL <[email protected]>

Fixed header

a1350c5

Signed-off-by: okhleif-IL <[email protected]>

addressed review comments

c7aadd2

Signed-off-by: okhleif-IL <[email protected]>

removed video

7288faa

Signed-off-by: okhleif-IL <[email protected]>

Merge branch 'main' into melanie/mm-rag-enhanced

e108ee9

ashahba approved these changes Nov 6, 2024

View reviewed changes

mhbuehler and others added 5 commits November 6, 2024 09:00

Reorder new lvm-dependent tests and fix clear textbox

9fdd6fe

Signed-off-by: Melanie Buehler <[email protected]>

Merge pull request #13 from mhbuehler/omar/mmqa-docs

ee387a2

Updated main README and supported_examples with MMQA details

[pre-commit.ci] auto fixes from pre-commit.com hooks

aafcfe1

for more information, see https://pre-commit.ci

Merge pull request #14 from mhbuehler/melanie/fix_tests

359b6f8

Fix for lvm-dependent tests and clear textbox

fixed multimodalqna typos

9d3ed45

Signed-off-by: okhleif-IL <[email protected]>

ashahba reviewed Nov 6, 2024

View reviewed changes

mhbuehler and others added 4 commits November 6, 2024 10:30

Point git clone at specific branch of GenAIComps

54cff40

Signed-off-by: Melanie Buehler <[email protected]>

Merge pull request #15 from mhbuehler/omar/dockimg_doc

fd9fd84

Fixed typos for MMQA in Docker Image List doc

[pre-commit.ci] auto fixes from pre-commit.com hooks

d88513a

for more information, see https://pre-commit.ci

Merge pull request #16 from mhbuehler/melanie/clone_specific_branch

24438e5

Clone specific branch of GenAIComps for tests

ashahba requested changes Nov 6, 2024

View reviewed changes

dmsuehir and others added 4 commits November 6, 2024 14:34

Fix xeon test to use llava-hf/llava-1.5-7b-hf (#17)

df2511b

Signed-off-by: dmsuehir <[email protected]>

Update MMQnA xeon test to wait for LVM to be ready (#18)

6631601

Signed-off-by: dmsuehir <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

59acc77

for more information, see https://pre-commit.ci

Test update: Increase wait time and add more messages (#19)

9d4cb5f

Signed-off-by: dmsuehir <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MultimodalQnA Image and Audio Support Phase 1 #1071

MultimodalQnA Image and Audio Support Phase 1 #1071

mhbuehler commented Nov 5, 2024 •

edited

Loading

ashahba left a comment •

edited

Loading

ashahba Nov 6, 2024

ashahba Nov 6, 2024

ashahba left a comment

ashahba Nov 6, 2024

ashahba Nov 6, 2024

MultimodalQnA Image and Audio Support Phase 1 #1071

Are you sure you want to change the base?

MultimodalQnA Image and Audio Support Phase 1 #1071

Conversation

mhbuehler commented Nov 5, 2024 • edited Loading

Description

Issues

Type of change

Dependencies

Tests

ashahba left a comment • edited Loading

Choose a reason for hiding this comment

ashahba Nov 6, 2024

Choose a reason for hiding this comment

ashahba Nov 6, 2024

Choose a reason for hiding this comment

ashahba left a comment

Choose a reason for hiding this comment

ashahba Nov 6, 2024

Choose a reason for hiding this comment

ashahba Nov 6, 2024

Choose a reason for hiding this comment

mhbuehler commented Nov 5, 2024 •

edited

Loading

ashahba left a comment •

edited

Loading