Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[0.6.0] Release Candidate #481

Merged
merged 721 commits into from
Sep 3, 2024
Merged

[0.6.0] Release Candidate #481

merged 721 commits into from
Sep 3, 2024

Conversation

AlpinDale
Copy link
Member

@AlpinDale AlpinDale commented May 25, 2024 •
edited
Loading

Target all PRs to this branch.

Warning

Highly experimental branch. A lot of things may not work.

Currently, the following are dysfunctional in this branch:

@AlpinDale AlpinDale mentioned this pull request Aug 5, 2024
Open
AlpinDale and others added 29 commits August 22, 2024 14:11
@AlpinDale
chore: skip the driver worker
23408b9
@AlpinDale
chore: bump lmfe version to 0.10.3
34fc26c
@AlpinDale
chore: some more marlin cleanups
7e9d4f3
@AlpinDale
chore: deprecation warning for beam search
bf15e1b
@AlpinDale
feat: support FP8 for DeepSeekV2 MoE
1efd0f8
@AlpinDale
feat: add fuyu vision model and persimmon language model support
e13a669
@AlpinDale
fix: turn off cutlass scaled_mm for ada lovelace cards
e7e847c
@AlpinDale
chore: allow quantizing all layers of deepseek-v2
b82c397
@AlpinDale
fix: build with pylimited api in the docker file
156a249
@AlpinDale
OpenAI API Refactor (#591)
cf381a0 
* feat: massive api server refactoring

* fix: tokenizer endpoint issues

* fix: BatchResponseData body should be optional
@AlpinDale
chore: simplify pipeline parallel code in llama
497bf64
@AlpinDale
fix: convert image to RGB by default
d6bf4bc
@AlpinDale
fix: allow getting the chat template from a url
96d5b8c
@AlpinDale
chore: avoid loading the unused layers and init the VLM up to the req…
e26a4ac 
…uired feature space
@AlpinDale
chore: enable bias w/ FP8 layers in CUTLASS kernels
b5d23ab
@AlpinDale
chore: upgrade flashinfer to 0.0.9
f83bbc6
@AlpinDale
feat: add custom triton cache manager
c8d398a
@AlpinDale
chore: add CustomAP interface to UnquantizedFusedMoEMethod
4bbf664
@AlpinDale
chore: handle aborted requests for jamba
e76bbe7
@AlpinDale
fix: minor fix for prompt adapter config
90b2f79
@AlpinDale
feat: chat completions tokenization endpoint (#592)
8432cae
@AlpinDale
feat: optimize throughput to 1.4x by using numpy for token padding
ebf8a53
@AlpinDale
feat: MoE support with Pallas GMM kernel for TPUs
e1475fb
@AlpinDale
chore: log spec decoding metrics
fc38c74
@AlpinDale
chore: separate kv_scale into k_scale and v_scale
9d7beaa
@AlpinDale
feat: Asymmetric Tensor Parallel (#594)
5289c14 
* add utils for getting the partition offset and size for current tp rank

* disable asymmetric TP for quants and lora, handle GQA allocation

* the actual splitting work in the linear layers

* padding size for the vocab/lm_head should be optional

* cache engine and spec decode model runner (kwargs only)

* pass the tp_rank to model runners

* llama support
@AlpinDale
update dockerfile
6dd6408
@AlpinDale
let's not build these for now
321a089
@AlpinDale
Revert "update dockerfile"
b71a865 
This reverts commit 6dd6408.
AlpinDale and others added 28 commits September 2, 2024 03:18
@AlpinDale
chore: simplify output processing with shortcut for non-parallel samp…
29f0478 
…ling and non-beam search usecase (#616)
@AlpinDale
refactor: minicpmv and port Idefix2VisionTransformer
9a50e3b
@AlpinDale
refactor: factor out code for running uvicorn again
040e5af
@AlpinDale
feat: port SiglipVisionModel from transformers
c3ee71a
@AlpinDale
chore: add proper logging for spec decoding verification
edffcec
@AlpinDale
fix: support flashinfer for draft model runner
6c2e24d
@AlpinDale
fix: use ipv4 localhost form for zmq bind
e8008f2
@AlpinDale
fix: use args.trust_remote_code
bd210a6
@AlpinDale
chore: update cutlass to 3.5.1
9d98f29
@AlpinDale
fix: specify device when loading lora and embedding tensors
2a349ca
@AlpinDale
feat: non-blocking transfer in prepare_input
6c1eab6
@AlpinDale
feat: re-add GGUF (#600)
0e6c400 
* refactor gguf kernels

* fix: incorrect filename for vecdotq header

* finish up the re-impl

* add requirements
@AlpinDale
minor CI fixes
e63be8e
@AlpinDale
ci: a few more ignores
dc00aa7
@AlpinDale
ci: remove clang-format
6b1f965
@AlpinDale
ci: take one of fixing lint issues
4d4e767
@AlpinDale
ci: codespell fixes
f18eeaf
@AlpinDale
ci: remove yapf
616de67
@AlpinDale
ci: remove yapf from the formatting script
c4933b1
@AlpinDale
ci: remove isort
2424207
@AlpinDale
chore: minor cleanups
2b85ffb
@AlpinDale
fix: allow loading GGUF model without .gguf extension
2894676
@AlpinDale
fix: cpu offloading with gptq
208cd54
@AlpinDale
docs: finalize User & Developer Documentation for Release Candidate (#…
57968a5 
…618)

* add getting started page

* add debugging tips

* add openai docs

* add distributed guide

* add production metrics and model support matrix

* add guide for adding new models

* huge update

* add vlm usage docs
@AlpinDale
ci: add action for deploying docs
cc7a636
@AlpinDale
docs: fix typos
638784c
@AlpinDale
chore: refactor wheel build script
75122b2
@AlpinDale
bump version to 0.6.0
c1c37c7
@AlpinDale AlpinDale changed the title [0.5.4] Release Candidate [0.6.0] Release Candidate Sep 3, 2024
@AlpinDale AlpinDale merged commit f1d0b77 into main Sep 3, 2024
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@AlpinDale @ahme-dev @ewof