Duygu Altinok 52e22cbf67 Fix for slow the bug tokenizer adding spaces to single id decodes (#32564) ... * _decode signature change and quick return * added bunch of decoding tests * signature match and return * added tests for decoding * merged decoding test * more tests for special tokens * cosmetics * fixed param * ruffed the file * refinement for single special tokens * added test for single special tokens * slight change to test name Co-authored-by: Ita Zaporozhets <31893021+itazap@users.noreply.github.com> * minor change test name for skip tokens Co-authored-by: Ita Zaporozhets <31893021+itazap@users.noreply.github.com> * killed already defined var Co-authored-by: Ita Zaporozhets <31893021+itazap@users.noreply.github.com> * minor update with vars Co-authored-by: Ita Zaporozhets <31893021+itazap@users.noreply.github.com> * killed already defined var once more Co-authored-by: Ita Zaporozhets <31893021+itazap@users.noreply.github.com> --------- Co-authored-by: Ita Zaporozhets <31893021+itazap@users.noreply.github.com>		2024-09-18 12:32:02 +02:00
..
__init__.py
test_tokenization_fast.py	Skip tests properly (#31308)	2024-06-26 21:59:08 +01:00
test_tokenization_utils.py	Fix for slow the bug tokenizer adding spaces to single id decodes (#32564)	2024-09-18 12:32:02 +02:00