Tom Aarsen 32e0db8a69 [tokenizers] Ensure that add_prefix_space is propagated to backend_tokenizer.pre_tokenizer (#35593) ... * Ensure that add_prefix_space is propagated to backend_tokenizer.pre_tokenizer in PreTrainedTokenizerFast, rather than relying on subclasses to take care of this. * Simplify setting self.add_prefix_space, ensure pre_tok exists * Wrap in try-except to catch 'Custom PreTokenizer cannot be serialized' 862d1a346a/bindings/python/src/pre_tokenizers.rs (L672) produces the Exception. They're triggered by the roformer tests, as the RoFormerTokenizerFast uses a custom PreTokenizer. * Propagate add_prefix_space in T5TokenizerFast to superclass		2025-01-09 17:46:50 +01:00
..
agents	Change is_soundfile_availble to is_soundfile_available (#35030)	2025-01-03 14:37:42 +01:00
benchmark
bettertransformer
deepspeed	Trainer - deprecate tokenizer for processing_class (#32385)	2024-10-02 14:08:46 +01:00
extended
fixtures
fsdp	FSDP grad accum fix (#34645)	2024-11-15 22:28:06 +01:00
generation	fix: Qwen2-VL generate with inputs_embeds (#35466)	2025-01-08 16:36:03 +01:00
models	Fix flaky SwitchTransformersModelTest::test_training_gradient (#35587)	2025-01-09 15:36:22 +01:00
optimization
peft_integration	added logic for deleting adapters once loaded (#34650)	2025-01-06 18:36:40 +00:00
pipelines	Pipeline: simple API for assisted generation (#34504)	2025-01-08 17:08:02 +00:00
quantization	Add Gemma2 GGUF support (#34002)	2025-01-03 14:50:07 +01:00
repo_utils	Fix modular edge case + modular sorting order (#35562)	2025-01-09 17:17:52 +01:00
sagemaker	Trainer - deprecate tokenizer for processing_class (#32385)	2024-10-02 14:08:46 +01:00
tokenization	tokenizer train from iterator without pre_tokenizers (#35396)	2025-01-09 15:34:43 +01:00
tp	Simplify Tensor Parallel implementation with PyTorch TP (#34184)	2024-11-18 19:51:49 +01:00
trainer	Fix all output_dir in test_trainer.py to use tmp_dir (#35266)	2025-01-08 19:44:39 +01:00
utils	More model refactoring! (#35359)	2025-01-09 11:09:09 +01:00
__init__.py
test_backbone_common.py
test_configuration_common.py	Load sub-configs from composite configs (#34410)	2024-11-05 11:34:01 +01:00
test_feature_extraction_common.py
test_image_processing_common.py	Fix Qwen2VL processor to handle odd number of frames (#35431)	2025-01-08 13:49:00 +01:00
test_image_transforms.py
test_modeling_common.py	Fix flaky SwitchTransformersModelTest::test_training_gradient (#35587)	2025-01-09 15:36:22 +01:00
test_modeling_flax_common.py	🚨All attention refactor🚨 (#35235)	2024-12-18 16:53:39 +01:00
test_modeling_tf_common.py	🚨All attention refactor🚨 (#35235)	2024-12-18 16:53:39 +01:00
test_pipeline_mixin.py	Add image text to text pipeline (#34170)	2024-10-31 15:48:11 -04:00
test_processing_common.py	VLMs: major clean up 🧼 (#34502)	2025-01-08 10:35:23 +01:00
test_sequence_feature_extraction_common.py
test_tokenization_common.py	[tokenizers] Ensure that add_prefix_space is propagated to backend_tokenizer.pre_tokenizer (#35593)	2025-01-09 17:46:50 +01:00