- Website: https://ga642381.github.io/SpeechPrompt/
- Paper Link: https://arxiv.org/abs/2303.00733
- Pipeline Charts: https://github.com/ga642381/SpeechPrompt-v2/blob/main/docs/pipeline.png
- Datasets Doc: https://github.com/ga642381/SpeechPrompt-v2/blob/main/docs/dataset.md
- Sampling Rate for Downstream Task:
- When performing prompting on the downstream task, ensure that the sampling rate of the audios is 16kHz.
- Modification: There is a recent commit to force librosa to load the audio in 16kHz.
- Pre-trained Model Loading:
- Make sure the pre-trained model is loaded correctly for reasonable results with prompting.
- Observation: When loading the pre-trained model correctly, the training epoch for prompts should start at epoch 46, not epoch 1. This is because the pre-trained GSLM is already trained for 45 epochs.
There are 4 files you will be having:
- HuBERT model: encoding speech
- K-means model: quantizing the speech representations into discrete units
- dictionary file: defining the unit space for the unit language model.
- unit Language Model (uLM): performing generative language modeling on the disrete units
These models can be automatically downloaded when running preprocessing pipeline.
-
There are 4 steps in the data preprocess (Speech2unit) pipline. The main task here is to perform speech-to-units and collating the task labels
- generate manifest
- quantize
- reduce_quantized
- create_lm_dataset
-
We save intermediate data in each step so that we can do further analysis on the data that we are interested in. Also, you can better understand how it works by checking each intermediate data.
- Download the dataset
- Modify the dataset config ([downstream]/config.yaml)
- Modify the global config (preprocess/config.yaml)
- Run Preporcess/runner.py
- option 1
# You can run --action all to run through all the 4 stages: python runner.py --model GSLM --downstream SCR_google_speech_commands --action all
- option 2
# Or you can run through these 4 stages sequentially by the following command: python runner.py --model GSLM --downstream SCR_google_speech_commands --action generate_manifest python runner.py --model GSLM --downstream SCR_google_speech_commands --action quantize python runner.py --model GSLM --downstream SCR_google_speech_commands --action reduce_quantized python runner.py --model GSLM --downstream SCR_google_speech_commands --action create_lm_dataset
- There are 2 steps in Verbalizer, which maps the task labels into language model's vocabulary.
- run verbalizer.py
- example:
python verbalizer.py --downstream SCR_google_speech_commands --action all --method freq
This step converts the verbalized data to binary files that will be used for fairseq training.
- run fairseq_preprocess.py
- example:
python fairseq_preprocess.py --downstream SCR_google_speech_commands --vb_method freq
- During training, 2 kinds of checkpoints will be saved
- base_model
- prompt
- run train.py
- example:
python train.py \ --downstream SCR_google_speech_commands \ --vb_method freq \ --exp_name SCR_google_speech_commands_plen.5 \ --prompt_length 5 \ --deep_prompt
- Load base_model and prompts to perform sampling
- run sample.py
- example:
python sample.py \ --exp_name SCR_google_speech_commands_plen.5 \ --downstream SCR_google_speech_commands \ --vb_method freq