-
Notifications
You must be signed in to change notification settings - Fork 141
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Int8DynActInt4WeightQATQuantizer doesn't support qwen series #1080
Comments
i read the code
add a judgment condition
skip the linear layer where bias is True |
cc @andrewor14 can you take a look |
Hi @elfisworking, yes the easy fix would be to skip the replacement when bias is False. Would you like to submit a fix for this? If not I can do it too. Probably the longer term fix would be to actually support the |
@andrewor14 ok, i will submit a fix. |
i use
Int8DynActInt4WeightQATQuantizer
to quantize qwen2-1.5B model. But after prepare function, i find that bias is set to False.This is my Code
The output is
we can see that after prepare function,
(q_proj): Linear(in_features=1536, out_features=1536, bias=True)
has been(q_proj): Int8DynActInt4WeightQATLinear(in_features=1536, out_features=1536, bias=False)
From torchao code, we can see In function
bias is set to False.
So has any Solution about this problem ?
The text was updated successfully, but these errors were encountered: