Assisted decoding with faster-whisper

[ostegm]

I'm wondering if anyone has experience trying to combine the benefits of assisted decoding with the benefits of quantization we get in faster-whisper? Ive done some experimenting with the hugging face implementation and it does give nice latency improvements, but even with assisted decoding I find its still slower than using faster-whisper. It would be quite nice to have both combined.

Wondering if there's a good approach to doing this already? I looked into exporting a checkpoint from hugging face with assisted decoding included, but it seems the assisted decoding is a decoding strategy, so not part of the actual model that gets exported, so therefore it wont be included when we run ctranslate2.

[phineas-pta]

from the article it's for AutoModelForCausalLM

not sure if whisper pipeline is supported

[ostegm]

Their demo just shows that model - but the assistant_model arg is available on any transformer model.generate() call. For example:

wav_file = "sample.wav"
audio, _ = librosa.load(wav_file, sr=16000, dtype='float32')
processor = WhisperProcessor.from_pretrained("openai/whisper-large-v2")
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-large-v2").to(device)
assistant_model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny").to(device)
input_features = processor(audio, sampling_rate=16000, return_tensors="pt").input_features.to(device)
predicted_ids = model.generate(input_features, assistant_model=assistant_model)
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)

[phineas-pta]

u can pass a lot of things into generate(), no error doesnt mean it's supported, it can be a placeholder or simply ignored

[ostegm]

I'm not sure if you read the blog post, but the blog post includes benchmarks for whisper. Here's the code which he used for the blog post.

If you still aren't convinced, run the code yourself with and without an assistant model.

[phineas-pta]

sr my bad i didnt see the benchmark

[phineas-pta]

ok so it's something more deep down in c++ code in ctranslate2, u could open a feature request there but not sure if guillaume has time to implement it since he just got new position