saymrwulf/transformers
1
0
Fork 0
mirror of https://github.com/saymrwulf/transformers.git synced 2026-05-14 20:58:08 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
transformers/tests/tokenization
History
Ita Zaporozhets 3d7c2f9dea
#32184 save total_vocab_size (#32240) 
* save total_vocab_size = vocab_size + user added tokens to speed up operation

* updating length when added_tokens_decoder is set

* add test len(tokenizer)
2024-08-05 09:22:48 +02:00
..
__init__.py
test_tokenization_fast.py Skip tests properly (#31308) 2024-06-26 21:59:08 +01:00
test_tokenization_utils.py #32184 save total_vocab_size (#32240) 2024-08-05 09:22:48 +02:00