Arthur f9cc333805 [ PreTrainedTokenizerFast] Keep properties from fast tokenizer (#25053) ... * draft solution * use `setdefault` * nits * add tests and fix truncation issue * fix test * test passes locally * quality * updates * update tsets		2023-07-25 18:45:01 +02:00
..
__init__.py
test_tokenization_fast.py	[ PreTrainedTokenizerFast] Keep properties from fast tokenizer (#25053)	2023-07-25 18:45:01 +02:00
test_tokenization_utils.py	Update quality tooling for formatting (#21480)	2023-02-06 18:10:56 -05:00