Sylvain Gugger 21b3922e35 Authorize last version of tokenizer (#9799) ... * Authorize last version of tokenizer * Update version table * Fix conversion of spm tokenizers and fix some hub links * Bump tokenizers version to 0.10.1rc1 * Add script to check tokenizers conversion with XNLI * Add some more mask_token lstrip support * Must modify mask_token in slow tokenizers too * Keep using the old method for Pegasus * add missing import Co-authored-by: Anthony MOI <m.anthony.moi@gmail.com>		2021-02-04 14:18:33 -05:00
..
fsmt	Copyright (#8970)	2020-12-07 18:36:34 -05:00
pegasus	Copyright (#8970)	2020-12-07 18:36:34 -05:00
tatoeba	update tatoeba workflow (#9051)	2020-12-11 20:29:15 +05:30
check_tokenizers.py	Authorize last version of tokenizer (#9799)	2021-02-04 14:18:33 -05:00