tools/StreamingTDSModelConverter.cpp?

[Adportas]

Hi,
Where is that file -> tools/StreamingTDSModelConverter.cpp
I need it to transform the model so I can transcribe ... but I can't find it in the github repository of w2l
Thank's in advance
Daniel

[abhinavkulkarni]

@Adportas: The inference code has been moved to v0.2 branch. You can find StreamingTDSModelConverter.cpp under tools directory.

[Adportas]

@Adportas: The inference code has been moved to v0.2 branch. You can find StreamingTDSModelConverter.cpp under tools directory.

Hi @abhinavkulkarni
Thank's for the reference.
Now i got v0.2 branch with the command: git clone -b v0.2 https://github.com/facebookresearch/wav2letter
And following the "General Build Instructions": https://github.com/facebookresearch/wav2letter/wiki/General-building-instructions
I set this flag W2L_BUILD_TOOLS to ON in CMakeLists.txt in order to obtain the program StreamingTDSModelConverter

The build of StreamingTDSModelConverter fail with this error:

/home/empresa/wav2letter_v0.2/tools/StreamingTDSModelConverter.cpp:24:10: fatal error: inference/module/feature/feature.h: File or directory does not exist (No existe el archivo o el directorio)
#include "inference/module/feature/feature.h"
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The path is really bad because the inference folder is duplicated: /home/empresa/wav2letter_v0.2/inference/inference/module/feature/feature.h

I tried various ways to fix the path's errors, in the end the one that worked best was adding this:
include_directories(/home/empresa/wav2letter_v0.2/src /home/empresa/wav2letter_v0.2/inference/)
To that file: /home/empresa/wav2letter_v0.2/tools/CMakeLists.txt

Then I get this error from a truly non-existent file

In file included from /home/empresa/wav2letter_v0.2/inference/inference/module/nn/backend/fbgemm/fbgemm.h:11:0,
from /home/empresa/wav2letter_v0.2/inference/inference/module/nn/nn.h:23,
from /home/empresa/wav2letter_v0.2/tools/StreamingTDSModelConverter.cpp:26:
/home/empresa/wav2letter_v0.2/inference/inference/module/nn/backend/fbgemm/Conv1dFbGemm.h:15:10: fatal error: fbgemm/FbgemmFP16.h: No existe el archivo o el directorio
#include <fbgemm/FbgemmFP16.h>
^~~~~~~~~~~~~~~~~~~~~

It will be that I must use some special configuration option somewhere?

Thank's in advance
Daniel

[abhinavkulkarni]

@Adportas: The following are the CMake options for my build:

-DW2L_BUILD_TESTS=OFF -DW2L_INFERENCE_BUILD_TESTS=OFF -DW2L_BUILD_TOOLS=ON -DW2L_BUILD_INFERENCE=ON -DW2L_INFERENCE_BACKEND=fbgemm

Also, make sure you have the right version of Flashlight (when in doubt either use the Docker container or the instructions in the Dockerfile and replicate them on your machine). There's something around checking out a specific branch of Flashlight and installing contrib packages - just follow instructions in the Dockerfile, if you are having trouble.

Yes, the inference folder is duplicated and you shouldn't have to de-duplicate it.

The repository and the branch v0.2 work as is if built correctly.

[Adportas]

@Adportas: The following are the CMake options for my build:

-DW2L_BUILD_TESTS=OFF -DW2L_INFERENCE_BUILD_TESTS=OFF -DW2L_BUILD_TOOLS=ON -DW2L_BUILD_INFERENCE=ON -DW2L_INFERENCE_BACKEND=fbgemm

Also, make sure you have the right version of Flashlight (when in doubt either use the Docker container or the instructions in the Dockerfile and replicate them on your machine). There's something around checking out a specific branch of Flashlight and installing contrib packages - just follow instructions in the Dockerfile, if you are having trouble.

Yes, the inference folder is duplicated and you shouldn't have to de-duplicate it.

The repository and the branch v0.2 work as is if built correctly.

Hi @abhinavkulkarni
Thank's for the parameters and Docker reference.
Docker is new to me, it took me a bit to understand it and choose the right image of the 3 available, finally the CPU type image allow me to generate the executable, but when using it I got some errors:

I1124 13:37:44.484884 16905 StreamingTDSModelConverter.cpp:192] Number of classes (network): 45
Skipping View module: V -1 1 NFEAT 0
terminate called after throwing an instance of 'std::invalid_argument'
what(): stoi
*** Aborted at 1606225064 (unix time) try "date -d @1606225064" if you are using GNU date ***
PC: @ 0x7f1b97733e97 gsignal
*** SIGABRT (@0x4209) received by PID 16905 (TID 0x7f1ba2b68840) from PID 16905; stack trace: ***
@ 0x7f1b98a8a890 (unknown)
@ 0x7f1b97733e97 gsignal
@ 0x7f1b97735801 abort

Which seem to be referenced in that previous post with broken usefull links inside unfortunately.

streaming_tds_model_converter error #646

Summary -> "We support only TDS based modules for streaming inference at the moment"

# 8 Temporal Convolutions followed by 2 Linear layers with ReLU activations.
# First Conv layer has stride of 2. All Conv layers perform "SAME" padding.
V -1 1 NFEAT 0
C2 NFEAT 256 8 1 2 1 -1 -1
R
C2 256 256 8 1 1 1 -1 -1
R
C2 256 256 8 1 1 1 -1 -1
R
C2 256 256 8 1 1 1 -1 -1
R
C2 256 256 8 1 1 1 -1 -1
R
C2 256 256 8 1 1 1 -1 -1
R
C2 256 256 8 1 1 1 -1 -1
R
C2 256 256 8 1 1 1 -1 -1
R
RO 2 0 3 1
L 256 512
R
L 512 NLABEL

I guess it would not be tds and I have to change it to that mode, I will try with this model I hope it works

V -1 NFEAT 1 0
C2 1 10 21 1 2 1 -1 -1
R
DO 0.2
LN 3
TDS 10 21 80 0.2
TDS 10 21 80 0.2
C2 10 14 21 1 2 1 -1 -1
R
DO 0.2
LN 3
TDS 14 21 80 0.2
TDS 14 21 80 0.2
TDS 14 21 80 0.2
C2 14 18 21 1 2 1 -1 -1
R
DO 0.2
LN 3
TDS 18 21 80 0.2
TDS 18 21 80 0.2
TDS 18 21 80 0.2
TDS 18 21 80 0.2
TDS 18 21 80 0.2
TDS 18 21 80 0.2
V 0 1440 1 0
RO 1 0 3 2
L 1440 1024

Finally i will leave the steps i use to compile and try the streaming_tds_model_converter in case they help:

Install Docker
sudo docker run --rm -itd --ipc=host --name w2l wav2letter/wav2letter:cpu-latest
sudo docker exec -it w2l bash
export LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2018.5.274/linux/mkl/lib/intel64:$LD_IBRARY_PAT
export MKLROOT=/opt/intel/mkl
export KENLM_ROOT_DIR=/root/kenlm
cmake -DW2L_BUILD_TESTS=OFF -DW2L_INFERENCE_BUILD_TESTS=OFF -DW2L_BUILD_TOOLS=ON -DW2L_BUILD_INFERENCE=ON -DW2L_INFERENCE_BACKEND=fbgemm ..
...
[100%] Building CXX object tools/CMakeFiles/streaming_tds_model_converter.dir/__/inference/inference/module/ModuleProcessingState.cpp.o
[100%] Linking CXX executable streaming_tds_model_converter
[100%] Built target streaming_tds_model_converter

./streaming_tds_model_converter -am Daniel/001_model_last.bin --outdir Daniel/Resultado

NOTE: I had to recreate some paths and copy referenced files of the model in the docker image with combination of "mkdir -p" and "sudo docker cp"

[abhinavkulkarni]

@Adportas: Not sure if you were able to run the TDS converter, but here's a sample run from my machine:

$ /home/w2luser/Projects/wav2letter/cmake-build-debug-fbgemm/tools/streaming_tds_model_converter --am /data/podcaster/model/wav2letter/am_500ms_future_context_dev_other/am_500ms_future_context_dev_other.bin --outdir /home/w2luser/models --flagsfile /home/w2luser/Projects/wav2letter/recipes/models/streaming_convnets/librispeech/train_am_500ms_future_context.cfg --logtostderr=1
I1203 21:45:45.544514 35115 StreamingTDSModelConverter.cpp:152] [Network] Reading acoustic model from /data/podcaster/model/wav2letter/am_500ms_future_context_dev_other/am_500ms_future_context_dev_other.bin
I1203 21:45:46.513530 35115 StreamingTDSModelConverter.cpp:157] [Network] Sequential [input -> (0) -> (1) -> (2) -> (3) -> (4) -> (5) -> (6) -> (7) -> (8) -> (9) -> (10) -> (11) -> (12) -> (13) -> (14) -> (15) -> (16) -> (17) -> (18) -> (19) -> (20) -> (21) -> (22) -> (23) -> (24) -> (25) -> (26) -> (27) -> (28) -> (29) -> (30) -> (31) -> (32) -> (33) -> (34) -> (35) -> (36) -> (37) -> (38) -> (39) -> output]
	(0): View (-1 80 1 0)
	(1): SpecAugment ( W: 80, F: 27, mF: 2, T: 100, p: 1, mT: 2 )
	(2): Padding (0, { (5, 3), (0, 0), (0, 0), (0, 0), })
	(3): Conv2D (1->15, 10x1, 2,1, 0,0, 1, 1) (with bias)
	(4): ReLU
	(5): Dropout (0.100000)
	(6): LayerNorm ( axis : { 1 2 } , size : -1)
	(7): Time-Depth Separable Block (9, 80, 15) [1200 -> 1200 -> 1200]
	(8): Time-Depth Separable Block (9, 80, 15) [1200 -> 1200 -> 1200]
	(9): Padding (0, { (7, 1), (0, 0), (0, 0), (0, 0), })
	(10): Conv2D (15->19, 10x1, 2,1, 0,0, 1, 1) (with bias)
	(11): ReLU
	(12): Dropout (0.100000)
	(13): LayerNorm ( axis : { 1 2 } , size : -1)
	(14): Time-Depth Separable Block (9, 80, 19) [1520 -> 1520 -> 1520]
	(15): Time-Depth Separable Block (9, 80, 19) [1520 -> 1520 -> 1520]
	(16): Time-Depth Separable Block (9, 80, 19) [1520 -> 1520 -> 1520]
	(17): Padding (0, { (9, 1), (0, 0), (0, 0), (0, 0), })
	(18): Conv2D (19->23, 12x1, 2,1, 0,0, 1, 1) (with bias)
	(19): ReLU
	(20): Dropout (0.100000)
	(21): LayerNorm ( axis : { 1 2 } , size : -1)
	(22): Time-Depth Separable Block (11, 80, 23) [1840 -> 1840 -> 1840]
	(23): Time-Depth Separable Block (11, 80, 23) [1840 -> 1840 -> 1840]
	(24): Time-Depth Separable Block (11, 80, 23) [1840 -> 1840 -> 1840]
	(25): Time-Depth Separable Block (11, 80, 23) [1840 -> 1840 -> 1840]
	(26): Padding (0, { (10, 0), (0, 0), (0, 0), (0, 0), })
	(27): Conv2D (23->27, 11x1, 1,1, 0,0, 1, 1) (with bias)
	(28): ReLU
	(29): Dropout (0.100000)
	(30): LayerNorm ( axis : { 1 2 } , size : -1)
	(31): Time-Depth Separable Block (11, 80, 27) [2160 -> 2160 -> 2160]
	(32): Time-Depth Separable Block (11, 80, 27) [2160 -> 2160 -> 2160]
	(33): Time-Depth Separable Block (11, 80, 27) [2160 -> 2160 -> 2160]
	(34): Time-Depth Separable Block (11, 80, 27) [2160 -> 2160 -> 2160]
	(35): Time-Depth Separable Block (11, 80, 27) [2160 -> 2160 -> 2160]
	(36): Reorder (2,1,0,3)
	(37): View (2160 -1 1 0)
	(38): Linear (2160->9998) (with bias)
	(39): View (9998 0 -1 1)
I1203 21:45:46.513664 35115 StreamingTDSModelConverter.cpp:158] [Criterion] ConnectionistTemporalClassificationCriterion
I1203 21:45:46.513669 35115 StreamingTDSModelConverter.cpp:159] [Network] Number of params: 115111823
I1203 21:45:46.513698 35115 StreamingTDSModelConverter.cpp:165] [Network] Updating flags from config file: /data/podcaster/model/wav2letter/am_500ms_future_context_dev_other/am_500ms_future_context_dev_other.bin
I1203 21:45:46.514137 35115 StreamingTDSModelConverter.cpp:174] Gflags after parsing 
--flagfile=; --fromenv=; --tryfromenv=; --undefok=; --tab_completion_columns=80; --tab_completion_word=; --help=false; --helpfull=false; --helpmatch=; --helpon=; --helppackage=false; --helpshort=false; --helpxml=false; --version=false; --adambeta1=0.90000000000000002; --adambeta2=0.999; --am=/data/podcaster/model/wav2letter/am_500ms_future_context_dev_other/am_500ms_future_context_dev_other.bin; --am_decoder_tr_dropout=0; --am_decoder_tr_layerdrop=0; --am_decoder_tr_layers=1; --arch=am_500ms_future_context.arch; --archdir=/home/w2luser/Projects/wav2letter/recipes/models/streaming_convnets/librispeech; --attention=content; --attentionthreshold=0; --attnWindow=no; --attnconvchannel=0; --attnconvkernel=0; --attndim=0; --batchsize=8; --beamsize=2500; --beamsizetoken=250000; --beamthreshold=25; --blobdata=false; --channels=1; --criterion=ctc; --critoptim=sgd; --datadir=; --dataorder=input; --decoderattnround=1; --decoderdropout=0; --decoderrnnlayer=1; --decodertype=wrd; --devwin=0; --emission_dir=; --emission_queue_size=3000; --enable_distributed=true; --encoderdim=0; --eosscore=0; --eostoken=false; --everstoredb=false; --fftcachesize=1; --filterbanks=80; --flagsfile=/home/w2luser/Projects/wav2letter/recipes/models/streaming_convnets/librispeech/train_am_500ms_future_context.cfg; --framesizems=25; --framestridems=10; --gamma=1; --gumbeltemperature=1; --input=flac; --inputbinsize=100; --inputfeeding=false; --isbeamdump=false; --iter=1000000; --itersave=false; --labelsmooth=0; --leftWindowSize=50; --lexicon=/home/w2luser/w2l/am/librispeech-train+dev-unigram-10000-nbest10.lexicon; --linlr=-1; --linlrcrit=-1; --linseg=0; --lm=; --lm_memory=5000; --lm_vocab=; --lmtype=kenlm; --lmweight=0; --localnrmlleftctx=300; --localnrmlrightctx=0; --logadd=false; --lr=0.40000000000000002; --lr_decay=9223372036854775807; --lr_decay_step=9223372036854775807; --lrcosine=false; --lrcrit=0; --max_devices_per_node=8; --maxdecoderoutputlen=200; --maxgradnorm=0.5; --maxisz=33000; --maxload=-1; --maxrate=10; --maxsil=50; --maxtsz=9223372036854775807; --maxword=-1; --melfloor=1; --memstepsize=10485760; --mfcc=false; --mfcccoeffs=13; --mfsc=true; --minisz=200; --minrate=3; --minsil=0; --mintsz=2; --momentum=0; --netoptim=sgd; --noresample=false; --nthread=6; --nthread_decoder=1; --nthread_decoder_am_forward=1; --numattnhead=8; --onorm=target; --optimepsilon=1e-08; --optimrho=0.90000000000000002; --outputbinsize=5; --pctteacherforcing=100; --pcttraineval=1; --pow=false; --pretrainWindow=0; --replabel=0; --reportiters=1000; --rightWindowSize=50; --rndv_filepath=/checkpoint/vineelkpratap/experiments/speech/inference_tds//inference_paper_500ms_do0.1_lr0.4_G32_archtds_k10s_d8_p100m_do0.1_saug_mln_500ms.arch_bch8/rndvz.21621542; --rundir=[...]; --runname=inference_2019; --samplerate=16000; --sampletarget=0; --samplingstrategy=rand; --saug_fmaskf=27; --saug_fmaskn=2; --saug_start_update=-1; --saug_tmaskn=2; --saug_tmaskp=1; --saug_tmaskt=100; --sclite=; --seed=0; --show=false; --showletters=false; --silscore=0; --smearing=none; --smoothingtemperature=1; --softwoffset=10; --softwrate=5; --softwstd=5; --sqnorm=true; --stepsize=1000000; --surround=; --tag=; --target=tkn; --test=; --tokens=librispeech-train-all-unigram-10000.tokens; --tokensdir=/home/w2luser/w2l/am; --train=/home/w2luser/w2l/lists/train-clean-100.lst,/home/w2luser/w2l/lists/train-clean-360.lst,/home/w2luser/w2l/lists/train-other-500.lst,[DATA_DST_librilight]/lists/librilight.lst; --trainWithWindow=false; --transdiag=0; --unkscore=-inf; --use_memcache=false; --uselexicon=true; --usewordpiece=true; --valid=dev-clean:/home/w2luser/w2l/lists/dev-clean.lst,dev-other:/home/w2luser/w2l/lists/dev-other.lst; --validbatchsize=-1; --warmup=1; --weightdecay=0; --wordscore=0; --wordseparator=_; --world_rank=0; --world_size=32; --outdir=/home/w2luser/models; --alsologtoemail=; --alsologtostderr=false; --colorlogtostderr=false; --drop_log_memory=true; --log_backtrace_at=; --log_dir=; --log_link=; --log_prefix=true; --logbuflevel=0; --logbufsecs=30; --logemaillevel=999; --logfile_mode=436; --logmailer=/bin/mail; --logtostderr=true; --max_log_size=1800; --minloglevel=0; --stderrthreshold=2; --stop_logging_if_full_disk=false; --symbolize_stacktrace=true; --v=0; --vmodule=; 
I1203 21:45:46.533406 35115 StreamingTDSModelConverter.cpp:192] Number of classes (network): 9998
Skipping View module: V -1 NFEAT 1 0
Skipping SpecAugment module: SAUG 80 27 2 100 1.0 2
Skipping Dropout module: DO 0.1
Skipping Dropout module: DO 0.1
Skipping Dropout module: DO 0.1
Skipping Dropout module: DO 0.1
Skipping Reorder module: RO 2 1 0 3
Skipping View module: V 2160 -1 1 0
Skipping View module: V NLABEL 0 -1 1
I1203 21:45:54.853623 35115 StreamingTDSModelConverter.cpp:289] Serializing acoustic model to '/home/w2luser/models/acoustic_model.bin'
I1203 21:46:00.994504 35115 StreamingTDSModelConverter.cpp:301] Writing tokens file to '/home/w2luser/models/tokens.txt'
I1203 21:46:01.005759 35115 StreamingTDSModelConverter.cpp:328] Serializing feature extraction model to '/home/w2luser/models/feature_extractor.bin'
I1203 21:46:01.710419 35115 StreamingTDSModelConverter.cpp:344] verifying serialization ...
I1203 21:46:03.194089 35115 StreamingTDSModelConverter.cpp:372] Done !

Process finished with exit code 0

[xiaosdawn]

Hi, I have tried streaming_tds_model_converter.
Here are my steps:

1. use the Docker container to build flashlight

   docker build -f .docker/Dockerfile-CUDA -t fl:81c4d8d .
   nvidia-docker run -t -i -v /data/w2l:/app/w2l-share --name=fl-81c4d8d fl:81c4d8d /bin/bash
   

2. clone wav2letter in the container above
3. edit /root/wav2letter/recipes/CMakeLists.txt, add line:

   add_subdirectory(${CMAKE_CURRENT_LIST_DIR}/streaming_convnets/tools)
   

   edit /root/wav2letter/recipes/streaming_convnets/tools/CMakeLists.txt, add line:

   include_directories(/root/wav2letter/recipes/streaming_convnets/inference/)
   

4. cd wav2letter && mkdir build && cd build && cmake .. -DW2L_BUILD_TOOLS=ON
   But, I got errors the same as @Adportas :

In file included from /root/wav2letter/recipes/streaming_convnets/inference/inference/module/nn/backend/fbgemm/fbgemm.h:10:0,
                 from /root/wav2letter/recipes/streaming_convnets/inference/inference/module/nn/nn.h:22,
                 from /root/wav2letter/recipes/streaming_convnets/tools/StreamingTDSModelConverter.cpp:29:
/root/wav2letter/recipes/streaming_convnets/inference/inference/module/nn/backend/fbgemm/Conv1dFbGemm.h:14:10: fatal error: fbgemm/FbgemmFP16.h: No such file or directory
 #include <fbgemm/FbgemmFP16.h>
          ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
recipes/streaming_convnets/tools/CMakeFiles/streaming_tds_model_converter.dir/build.make:62: recipe for target 'recipes/streaming_convnets/tools/CMakeFiles/streaming_tds_model_converter.dir/StreamingTDSModelConverter.cpp.o' failed
make[2]: *** [recipes/streaming_convnets/tools/CMakeFiles/streaming_tds_model_converter.dir/StreamingTDSModelConverter.cpp.o] Error 1
CMakeFiles/Makefile2:219: recipe for target 'recipes/streaming_convnets/tools/CMakeFiles/streaming_tds_model_converter.dir/all' failed
make[1]: *** [recipes/streaming_convnets/tools/CMakeFiles/streaming_tds_model_converter.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2

Then, I change the cmake parameters @abhinavkulkarni suggested.

cmake .. -DW2L_BUILD_TESTS=OFF -DW2L_INFERENCE_BUILD_TESTS=OFF -DW2L_BUILD_TOOLS=ON -DW2L_BUILD_INFERENCE=ON -DW2L_INFERENCE_BACKEND=fbgemm

But, the same error comes up again.

In addtion, I tried wav2letter v0.2

1. clone in the container from step1
2. cd wav2letter-v0.2 && mkdir build && cd build && cmake .. -DW2L_BUILD_TOOLS=ON
   I got errors as:

-- Found SndFile: /usr/local/include
CMake Error at /usr/local/share/flashlight/cmake/FindSndFile.cmake:65 (get_target_property):
  get_target_property() called with non-existent target "Vorbis::vorbis".
Call Stack (most recent call first):
  /usr/local/share/cmake-3.10/Modules/CMakeFindDependencyMacro.cmake:48 (find_package)
  /usr/local/share/flashlight/cmake/flashlightConfig.cmake:57 (find_dependency)
  CMakeLists.txt:99 (find_package)


CMake Error at /usr/local/share/flashlight/cmake/FindSndFile.cmake:66 (get_target_property):
  get_target_property() called with non-existent target "Vorbis::vorbisenc".
Call Stack (most recent call first):
  /usr/local/share/cmake-3.10/Modules/CMakeFindDependencyMacro.cmake:48 (find_package)
  /usr/local/share/flashlight/cmake/flashlightConfig.cmake:57 (find_dependency)
  CMakeLists.txt:99 (find_package)


CMake Error at /usr/local/share/flashlight/cmake/FindSndFile.cmake:68 (get_target_property):
  get_target_property() called with non-existent target "Ogg::ogg".
Call Stack (most recent call first):
  /usr/local/share/cmake-3.10/Modules/CMakeFindDependencyMacro.cmake:48 (find_package)
  /usr/local/share/flashlight/cmake/flashlightConfig.cmake:57 (find_dependency)
  CMakeLists.txt:99 (find_package)


-- Found libsndfile: (lib: /usr/local/lib/libsndfile.so include: /usr/local/include
-- The CUDA compiler identification is NVIDIA 10.0.130
-- Check for working CUDA compiler: /usr/local/cuda/bin/nvcc
-- Check for working CUDA compiler: /usr/local/cuda/bin/nvcc -- works
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- flashlight found (include:  lib: flashlight::flashlight )
CMake Error at CMakeLists.txt:103 (message):
  flashlight must be build in distributed mode for wav2letter++


-- Configuring incomplete, errors occurred!
See also "/root/wav2letter-0.2/build/CMakeFiles/CMakeOutput.log".

Maybe I have missed something. Any help will be appreciate. Thanks.

[abhinavkulkarni]

@xiaosdawn: Have you checked if any of the pre-built Docker images streaming_tds_model_converter binary built? The CPU/CUDA ones for training should have that pre-built (or you may be able to build the binary by switching on a particular CMake option).

[xiaosdawn]

I didn't find any pre-built Docker images streaming_tds_model_converter binary built? Could you give me some advice or the link for getting the specific image? Thanks very much.

[abhinavkulkarni]

@xiaosdawn: You may want to check pre-built Docker images detailed here: https://github.com/facebookresearch/wav2letter/wiki/Building-Running-with-Docker

The ones used for training should have the binary built.

[xiaosdawn]

Ok, thanks.

[xiaosdawn]

Thank you @abhinavkulkarni .
I use wav2letter/wav2letter:cuda-latest image to build a container.
Then

cd /root/wav2letter/build
cmake .. -DW2L_BUILD_TOOLS=ON && make -j8

So the streaming_tds_model_converter binary is built at build/tools/streaming_tds_model_converter.

Thanks again.

[xiaosdawn]

Hi, sorry to have bothered you again.
After I built the streaming_tds_model_converter binary, I start to convert my well-trained ConvNet model. But I got errors:

E0115 07:58:02.565189  4068 Serial.h:77] Error while loading "/app/w2l-share/AM_rundir/am_convnet_inference_2019/001_model_dev-clean.bin": Trying to load an unregistered polymorphic type (fl::SpecAugment).
Make sure your type is registered with CEREAL_REGISTER_TYPE and that the archive you are using was included (and registered with CEREAL_REGISTER_ARCHIVE) prior to calling CEREAL_REGISTER_TYPE.
If your type is already registered and you still see this error, you may need to use CEREAL_REGISTER_DYNAMIC_INIT.
terminate called after throwing an instance of 'cereal::Exception'
  what():  Trying to load an unregistered polymorphic type (fl::SpecAugment).
Make sure your type is registered with CEREAL_REGISTER_TYPE and that the archive you are using was included (and registered with CEREAL_REGISTER_ARCHIVE) prior to calling CEREAL_REGISTER_TYPE.
If your type is already registered and you still see this error, you may need to use CEREAL_REGISTER_DYNAMIC_INIT.
*** Aborted at 1610697482 (unix time) try "date -d @1610697482" if you are using GNU date ***
PC: @     0x7fa535b3be97 gsignal
*** SIGABRT (@0xfe4) received by PID 4068 (TID 0x7fa57b00b3c0) from PID 4068; stack trace: ***
    @     0x7fa53d839890 (unknown)
    @     0x7fa535b3be97 gsignal
    @     0x7fa535b3d801 abort
    @     0x7fa536530957 (unknown)
    @     0x7fa536536ab6 (unknown)
    @     0x7fa536536af1 std::terminate()
    @     0x7fa536536d79 __cxa_rethrow
    @     0x55ca685f92bb main
    @     0x7fa535b1eb97 __libc_start_main
    @     0x55ca686678ca _start
Aborted (core dumped)

Hope you show me some advice continuely. Any suggestion will be appreciate.

Here is my train.cfg details:

--runname=am_convnet_inference_2019
--rundir=/app/w2l-share/AM_rundir
--archdir=/app/w2l-share/configs
--arch=am_arch/am_500ms_future_context.arch
--tokensdir=/app/w2l-share/Lexicon-and-Tokens/Download
--tokens=librispeech-train-all-unigram-10000.tokens
--lexicon=/app/w2l-share/Lexicon-and-Tokens/Download/librispeech-train+dev-unigram-10000-nbest10.lexicon
--train=/app/w2l-share/lists/train-clean-100.lst,/app/w2l-share/lists/train-clean-360.lst,/app/w2l-share/lists/train-other-500.lst
--valid=dev-clean:/app/w2l-share/lists/dev-clean.lst,dev-other:/app/w2l-share/lists/dev-other.lst
--criterion=ctc
--batchsize=8
--lr=0.4
--momentum=0.0
--maxgradnorm=0.5
--reportiters=1000
--nthread=6
--mfsc=true
--usewordpiece=true
--wordseparator=_
--filterbanks=80
--minisz=200
--mintsz=2
--maxisz=33000
--enable_distributed=true
--pcttraineval=1
--minloglevel=0
--logtostderr
--onorm=target
--sqnorm
--localnrmlleftctx=300
--lr_decay=10000

and the arch details as:

V -1 NFEAT 1 0
SAUG 80 27 2 100 1.0 2
PD 0 5 3
C2 1 15 10 1 2 1 0 0
R
DO 0.1
LN 1 2
TDS 15 9 80 0.1 0 1 0
TDS 15 9 80 0.1 0 1 0
PD 0 7 1
C2 15 19 10 1 2 1 0 0
R
DO 0.1
LN 1 2
TDS 19 9 80 0.1 0 1 0
TDS 19 9 80 0.1 0 1 0
TDS 19 9 80 0.1 0 1 0
PD 0 9 1
C2 19 23 12 1 2 1 0 0
R
DO 0.1
LN 1 2
TDS 23 11 80 0.1 0 1 0
TDS 23 11 80 0.1 0 1 0
TDS 23 11 80 0.1 0 1 0
TDS 23 11 80 0.1 0 0 0
PD 0 10 0
C2 23 27 11 1 1 1 0 0
R
DO 0.1
LN 1 2
TDS 27 11 80 0.1 0 0 0
TDS 27 11 80 0.1 0 0 0
TDS 27 11 80 0.1 0 0 0
TDS 27 11 80 0.1 0 0 0
TDS 27 11 80 0.1 0 0 0
RO 2 1 0 3
V 2160 -1 1 0
L 2160 NLABEL
V NLABEL 0 -1 1

[tlikhomanenko]

cc @vineelpratap could you navigate here with latest wav2letter/flashlight?

Otherwise @xiaosdawn you needd to use docker built at time of branch v0.2 before we did refactor of the whole code base and merged flashlight and wav2letter.

[xiaosdawn]

Thanks for the continued help.
I used docker built at time of branch v0.2:

git clone -b v0.2 https://github.com/facebookresearch/wav2letter.git
cd wav2letter
docker build -f Dockerfile-CUDA -t w2l_v0.2:b1d1f89 .
nvidia-docker run -t -i -v /data/w2l:/app/w2l-share --name=w2l_v0.2 w2l_v0.2:b1d1f89 /bin/bash

Then, in the container:

cd /root/wav2letter/build
cmake .. -DW2L_BUILD_TOOLS=ON && make -j10
./build/tools/streaming_tds_model_converter --am=/app/w2l-share/AM_rundir/am_convnet_inference_2019/003_model_dev-clean.bin --outdir=/app/w2l-share/AM_rundir/am_convnet_inference_2019/003_model_dev-clean.bin.serialized

But, I got the same errors as above:

E0118 07:04:13.625599  3589 Serial.h:77] Error while loading "/app/w2l-share/AM_rundir/am_convnet_inference_2019/003_model_dev-clean.bin": Trying to load an unregistered polymorphic type (fl::SpecAugment).
Make sure your type is registered with CEREAL_REGISTER_TYPE and that the archive you are using was included (and registered with CEREAL_REGISTER_ARCHIVE) prior to calling CEREAL_REGISTER_TYPE.
If your type is already registered and you still see this error, you may need to use CEREAL_REGISTER_DYNAMIC_INIT.
terminate called after throwing an instance of 'cereal::Exception'
  what():  Trying to load an unregistered polymorphic type (fl::SpecAugment).
Make sure your type is registered with CEREAL_REGISTER_TYPE and that the archive you are using was included (and registered with CEREAL_REGISTER_ARCHIVE) prior to calling CEREAL_REGISTER_TYPE.
If your type is already registered and you still see this error, you may need to use CEREAL_REGISTER_DYNAMIC_INIT.
*** Aborted at 1610953453 (unix time) try "date -d @1610953453" if you are using GNU date ***
PC: @     0x7f3d3f413e97 gsignal
*** SIGABRT (@0xe05) received by PID 3589 (TID 0x7f3d848e33c0) from PID 3589; stack trace: ***
    @     0x7f3d47111890 (unknown)
    @     0x7f3d3f413e97 gsignal
    @     0x7f3d3f415801 abort
    @     0x7f3d3fe08957 (unknown)
    @     0x7f3d3fe0eab6 (unknown)
    @     0x7f3d3fe0eaf1 std::terminate()
    @     0x7f3d3fe0ed79 __cxa_rethrow
    @     0x55e2dec8d2bb main
    @     0x7f3d3f3f6b97 __libc_start_main
    @     0x55e2decfb8ca _start
Aborted (core dumped)

[tlikhomanenko]

Inside Dockerfile it is used latest master of flashlight. You need to fix Dockerfile to use flashlight with v0.2 branch too.

[xiaosdawn]

Inside Dockerfile, I checked flashlight with v0.2 branch:

cd /root/flashlight/
git checkout v0.2
cd build && cmake .. -DCMAKE_BUILD_TYPE=Release -DFLASHLIGHT_BACKEND=CUDA
&& make -j10 && make install

cd /root/wav2letter/build
cmake .. -DCMAKE_BUILD_TYPE=Release -DW2L_LIBRARIES_USE_CUDA=ON -DW2L_BUILD_INFERENCE=ON -DW2L_BUILD_TOOLS=ON
&& make -j10

But, the error is the same as above:

E0120 01:40:46.838344  1131 Serial.h:77] Error while loading "/app/w2l-share/AM_rundir/am_convnet_inference_2019/003_model_dev-clean.bin": Trying to load an unregistered polymorphic type (fl::SpecAugment).
Make sure your type is registered with CEREAL_REGISTER_TYPE and that the archive you are using was included (and registered with CEREAL_REGISTER_ARCHIVE) prior to calling CEREAL_REGISTER_TYPE.
If your type is already registered and you still see this error, you may need to use CEREAL_REGISTER_DYNAMIC_INIT.
terminate called after throwing an instance of 'cereal::Exception'
  what():  Trying to load an unregistered polymorphic type (fl::SpecAugment).
Make sure your type is registered with CEREAL_REGISTER_TYPE and that the archive you are using was included (and registered with CEREAL_REGISTER_ARCHIVE) prior to calling CEREAL_REGISTER_TYPE.
If your type is already registered and you still see this error, you may need to use CEREAL_REGISTER_DYNAMIC_INIT.
*** Aborted at 1611106846 (unix time) try "date -d @1611106846" if you are using GNU date ***
PC: @     0x7f37c1ee2e97 gsignal
*** SIGABRT (@0x46b) received by PID 1131 (TID 0x7f38073b23c0) from PID 1131; stack trace: ***
    @     0x7f37c9be0890 (unknown)
    @     0x7f37c1ee2e97 gsignal
    @     0x7f37c1ee4801 abort
    @     0x7f37c28d7957 (unknown)
    @     0x7f37c28ddab6 (unknown)
    @     0x7f37c28ddaf1 std::terminate()
    @     0x7f37c28ddd79 __cxa_rethrow
    @     0x558de5bb62bb main
    @     0x7f37c1ec5b97 __libc_start_main
    @     0x558de5c248ca _start
Aborted (core dumped)

Is there a specific flashlight version to train the convnet model? I trained the convnet model using 36b242e. Maybe it is too old?

[tlikhomanenko]

Maybe I misunderstood but what I see now:

1. you are not using released model, you retrained the model
2. you retrained model with the version of flashlight where w2l code is already merged there
3. In this case you need to use tools which we have now in flashlight https://github.com/facebookresearch/flashlight/tree/master/flashlight/app/asr/tools
4. for inference it is still WIP for make it build/work with cmake but you can give a try, as we moved inference code here https://github.com/facebookresearch/wav2letter/tree/master/recipes/streaming_convnets/inference cc @vineelpratap on the usage of inference code with latest stuff.

[xiaosdawn]

All yes. Thanks again. I'll try more.

Maybe I misunderstood but what I see now:
you are not using released model, you retrained the model
you retrained model with the version of flashlight where w2l code is already merged there
In this case you need to use tools which we have now in flashlight https://github.com/facebookresearch/flashlight/tree/master/flashlight/app/asr/tools

[vineelpratap]

Hi,
To make it work, you can manually change the namespace of SpecAugment class from fl::SpecAugment to w2l::SpecAugment and build the binary again... Please let me know if you see any issues...

We are working to fix this in - #930 so that you don't need to do any hacks like above...