Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

if LoRAX is based on punica kernels will it be able to support LoRA Adapters for Mistral NeMO 12B? #549

Open
tensimixt opened this issue Jul 20, 2024 · 2 comments

Comments

@tensimixt
Copy link

Feature request

if LoRAX is based on punica kernels will it be able to support LoRA Adapters for Mistral NeMO 12B? which has a vocab size > 130k.
Currently Vllm for example doesn't support vocab_size > 128512 when enable_lora=True

I think if huggingface and LoRAX are based on punica kernels they will also have this limitation or this limitation does not exist for TGI and LoRAX?

Thank you!

Motivation

be able to run inference with Mistral NeMO + LoRA Adapter (in a multi-lora world)

Your contribution

Checked various deployment providers and found the limitation

@Nero10578
Copy link

Feature request

if LoRAX is based on punica kernels will it be able to support LoRA Adapters for Mistral NeMO 12B? which has a vocab size > 130k. Currently Vllm for example doesn't support vocab_size > 128512 when enable_lora=True

I think if huggingface and LoRAX are based on punica kernels they will also have this limitation or this limitation does not exist for TGI and LoRAX?

Thank you!

Motivation

be able to run inference with Mistral NeMO + LoRA Adapter (in a multi-lora world)

Your contribution

Checked various deployment providers and found the limitation

did you figure out if Mistral Nemo 12B works with lora adapters with lorax? It does not work with VLLM or Aphrodite still and I am looking for alternatives.

@preduct0r
Copy link

Feature request

if LoRAX is based on punica kernels will it be able to support LoRA Adapters for Mistral NeMO 12B? which has a vocab size > 130k. Currently Vllm for example doesn't support vocab_size > 128512 when enable_lora=True
I think if huggingface and LoRAX are based on punica kernels they will also have this limitation or this limitation does not exist for TGI and LoRAX?
Thank you!

Motivation

be able to run inference with Mistral NeMO + LoRA Adapter (in a multi-lora world)

Your contribution

Checked various deployment providers and found the limitation

did you figure out if Mistral Nemo 12B works with lora adapters with lorax? It does not work with VLLM or Aphrodite still and I am looking for alternatives.

Did you find alternatives to vllm? I still strugle with this problem of serving mistral_nemo with lora

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants