Raushan Turganbay 187439c3fa VLM: special multimodal Tokenizer (#34461) ... * kinda works * update * add tests * update * use special tokens in processors * typo * fix copies * fix * fix moshi after rebase * update * fix tests * update * Update docs/source/en/main_classes/tokenizer.md Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * update docs * test for load time adding tokens * fix some more tests which are now fetched better * one more fix --------- Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>		2024-11-04 16:37:51 +01:00
..
__init__.py
test_tokenization_fast.py	Skip tests properly (#31308)	2024-06-26 21:59:08 +01:00
test_tokenization_utils.py	VLM: special multimodal Tokenizer (#34461)	2024-11-04 16:37:51 +01:00