Quantization issue tracker

[SalmanMohammadi]

There's a few different issues cropping up with our quantization offerings. Keeping track of them here

Tasks

Beta Give feedback

1. Implement quantized model inference for generate_v2 #1814
   enhancement high-priority inference +3
   

Relevant issues

• qwen2 is not supported by QAT #1818
• Guard core version when compiling quantized models #1798
• Quantized model inference with our generate recipe is abysmally slow (see Fix quantization with generate #1784)

cc @andrewor14 who might be able to shed some light here : )