How to Process Byte Data with faster_whisper

[Hu-nie]

Hello, I'm reaching out for assistance.

The code below is designed to enable client and server communication over WebSocket to process data as bytes. While it works correctly when saving the data as a WAV file, it does not return accurate results when processing the byte data directly.

Thank you in advance for your help.

#client.py
import asyncio
import websockets
import pyaudio


CHANNELS = 1
SAMPLE_RATE = 16000
CHUNK_SIZE = 1024
FORMAT = pyaudio.paInt16

async def send_audio():
    uri = "ws://0.0.0.0:8080"
    async with websockets.connect(uri) as websocket:
        audio = pyaudio.PyAudio()
        stream = audio.open(
            format=FORMAT,
            channels=CHANNELS,
            rate=SAMPLE_RATE,
            input=True,
            frames_per_buffer=CHUNK_SIZE,
        )

        try:
            print("Streaming audio for 5 seconds...")
            end_time = asyncio.get_event_loop().time() + 2
            while True:
                if asyncio.get_event_loop().time() >= end_time:
                    break
                data = stream.read(CHUNK_SIZE)
                await websocket.send(data)  
        finally:
            print("Stopped streaming. Sending END signal to server.")
            await websocket.send("END")  
            stream.stop_stream()
            stream.close()
            audio.terminate()

asyncio.run(send_audio())

# server.py

import asyncio
import websockets
import numpy as np
import time 
from faster_whisper import WhisperModel
model = WhisperModel("large-v3", device="cuda", compute_type="float16")


async def handle_audio(websocket, path):
    print("Client connected")
    audio_buffer = []  
    try:
        async for message in websocket:

            if isinstance(message, str) and message == "END":
                print("End of audio stream.")
                
                start_time = time.time()
                audio_data = np.frombuffer(b''.join(audio_buffer), dtype=np.float32)

                segments, _ = model.transcribe(audio_data, language='ko', task='transcribe')
                for segment in segments:
                    print("[%.2fs -> %.2fs] %s" % (segment.start, segment.end, segment.text))
                end_time = time.time()
                print("Processing time:", end_time - start_time, "seconds")
                
                break
            elif isinstance(message, bytes):
                print(message[:10] , f'{len(message)}byte')  
                print("--------------------------------")

                audio_buffer.append(message)
    except websockets.exceptions.ConnectionClosed:
        print("Client disconnected")

# WebSocket server start
start_server = websockets.serve(handle_audio, "0.0.0.0", 8080)

asyncio.get_event_loop().run_until_complete(start_server)
print("Server started, waiting for connections...")
asyncio.get_event_loop().run_forever()

[MahmoudAshraf97]

the transcribe function expects either a string that represents the path to an audio file, or a byte array or a numpy array that represents the waveform sampled at 16khz, you need to revise how you open the audio and how you receive it, I haven't run your code but it seems that you are not resampling it correctly when opening and not using the correct dtype when receiving, as long as the model outputs are correct when using a WAV file, then the error is caused by the way you handle the stream

[Hu-nie]

@MahmoudAshraf97

I would really appreciate it if you could let me know about the correct resampling method and the appropriate data type (dtype) to use. If you don't have specific guidance, any resources or references where I can find this information would also be greatly appreciated. Thank you in advance for your help!

[MahmoudAshraf97]

Check #1077

[Hu-nie]

Thank you, I have resolved the issue!