-
Notifications
You must be signed in to change notification settings - Fork 855
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Vision] Support Phi-3.5-vision, the first VLM in WebLLM #563
Merged
Commits on Sep 18, 2024
-
Support image input in chatCompletionRequest
- Enable content being an array, which can have image_url - Introduce ModelType.VLM so that only VLM can handle non-string message content - Thus pass in loadedModelType to postInitCheck, hence add loadedModelIdToModelType in Engine - Change unit tests correspondingly
Configuration menu - View commit details
-
Copy full SHA for fad3df9 - Browse repository at this point
Copy the full SHA fad3df9View commit details
Commits on Sep 20, 2024
-
Support formatting image input in Conversation.getPromptArray
- getPromptArray element can now be an array itself - Conversation.messages message can either be a string or an array of content parts - Update compareConversationObject - Next step is to update llm_chat.ts getInputTokens
Configuration menu - View commit details
-
Copy full SHA for 19c9991 - Browse repository at this point
Copy the full SHA 19c9991View commit details -
Configuration menu - View commit details
-
Copy full SHA for b398412 - Browse repository at this point
Copy the full SHA b398412View commit details
Commits on Sep 22, 2024
-
Separate embed from forward, add getChunkedPrefillInputData
- Implement getInputData to replace getInputTokens - Instead of returning a list of tokens, return a list of mixture of number[] and imageUrl - Implement getChunkedPrefillInputData that transforms output of getInputData into chunks, thoroughly tested - Replace forward with embedAndForward, which takes in a chunk, embeds it, and forward - Within embedAndForward, we first embed all components in the chunk - Note chunking is taken care of in getChunkedPrefillInputData, so embedAndForward has data length less than prefill chunk size - TODOs: implement getImageEmbeddings, concatenation of embeddings, E2E test with image input - Tested: - simple chat, logit processor, get_started, multi-round example - prefill multimple messages - 32 prefill chunk size, prefill 300 tokens
Configuration menu - View commit details
-
Copy full SHA for db4bb25 - Browse repository at this point
Copy the full SHA db4bb25View commit details -
Implement getImageEmbedding, make phi3_5-vision work E2E
- Impelment getImageEmbedding that loads into ImageData and call embed_image kernel - Use TVM global function concatEmbeddings to combine text and image embeddings - Imlpement helper function that loads ImageData from url, either http or base64
Configuration menu - View commit details
-
Copy full SHA for fd57f7c - Browse repository at this point
Copy the full SHA fd57f7cView commit details
Commits on Sep 23, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 576a5c7 - Browse repository at this point
Copy the full SHA 576a5c7View commit details -
Configuration menu - View commit details
-
Copy full SHA for 453f9c9 - Browse repository at this point
Copy the full SHA 453f9c9View commit details
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.