You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
(et_xnnpack) /content/HI/executorch# python -m extension.llm.tokenizer.tokenizer -t /root/.llama/checkpoints/Llama3.2-1B/tokenizer.model -o tokenizer.bin
Traceback (most recent call last):
File "/usr/local/envs/et_xnnpack/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/local/envs/et_xnnpack/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/content/HI/executorch/extension/llm/tokenizer/tokenizer.py", line 140, in <module>
t = Tokenizer(args.tokenizer_model)
File "/content/HI/executorch/extension/llm/tokenizer/tokenizer.py", line 26, in __init__
self.sp_model = SentencePieceProcessor(model_file=model_path)
File "/usr/local/envs/et_xnnpack/lib/python3.10/site-packages/sentencepiece/__init__.py", line 468, in Init
self.Load(model_file=model_file, model_proto=model_proto)
File "/usr/local/envs/et_xnnpack/lib/python3.10/site-packages/sentencepiece/__init__.py", line 961, in Load
return self.LoadFromFile(model_file)
File "/usr/local/envs/et_xnnpack/lib/python3.10/site-packages/sentencepiece/__init__.py", line 316, in LoadFromFile
return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
RuntimeError: Internal: could not parse ModelProto from /root/.llama/checkpoints/Llama3.2-1B/tokenizer.model
The text was updated successfully, but these errors were encountered:
a8nova
changed the title
Error when converting tokenizer.model to bin
Error when converting llama3.2-1B tokenizer.model to bin
Oct 5, 2024
a8nova
changed the title
Error when converting llama3.2-1B tokenizer.model to bin
Error when converting llama3.2-1B tokenizer.model to tokenizer.bin
Oct 5, 2024
🐛 Describe the bug
When following instructions from https://github.com/pytorch/executorch/blob/main/examples/demo-apps/apple_ios/LLaMA/docs/delegates/xnnpack_README.md and when trying to convert llama3.2-1B tokenizer to bin you will see below error:
The text was updated successfully, but these errors were encountered: