Set tokenizers version to <0.20.2 (#460)

0.20.2 had some changes that break lm_eval API
HabanaAI · Nov 5, 2024 · 1033c3e · 1033c3e
1 parent 653e56c
commit 1033c3e
Showing 1 changed file with 2 additions and 1 deletion.
diff --git a/.jenkins/requirements-test-hpu.txt b/.jenkins/requirements-test-hpu.txt
@@ -1,2 +1,3 @@
 lm_eval
-pytest
+pytest
+tokenizers<0.20.2