Improve asr (whisper) with speaker diarization result

[yinruiqing]

I combined whisper and pyannote.audio in pyannote-whisper to get "who speaks when and what" from an audio file. The strategy is very simple: choose the major speaker in a sentence. The performance may degrade a lot for fast speech turns. Can we find a more elegant way to combine these tools? For example, do diarization first and then feed to whisper (this may also cause some problems in the overlap part).

[hbredin]

I tried this here.

One way to improve it would be to merge adjacent speech turns of the same speaker so that Whisper can benefit from more context.

This does not solve the overlapping speech issue, however.