mirror of
https://github.com/saymrwulf/transformers.git
synced 2026-05-14 20:58:08 +00:00
deba7655e6
* seems like `split_special_tokens` is used here * split special token * add new line at end of file * moving split special token test to common tests * added assertions * test * fixup * add co-author * passing rest of args to gptsan_japanese, fixing tests * removing direct comparison of fast and slow models * adding test support for UDOP and LayoutXLM * ruff fix * readd check if slow tokenizer * modify test to handle bos tokens * removing commented function * trigger build * applying review feedback - updated docstrings, var names, and simplified tests * ruff fixes * Update tests/test_tokenization_common.py Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * applying feedback, comments * shutil temp directory fix --------- Co-authored-by: Arthur Zucker <arthur.zucker@gmail.com> Co-authored-by: Ita Zaporozhets <itazaporozhets@Itas-MBP.localdomain> Co-authored-by: itazap <itazap@users.noreply.github.com> Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> Co-authored-by: Ita Zaporozhets <itazaporozhets@Itas-MacBook-Pro.local> |
||
|---|---|---|
| .. | ||
| agents | ||
| benchmark | ||
| bettertransformer | ||
| deepspeed | ||
| extended | ||
| fixtures | ||
| fsdp | ||
| generation | ||
| models | ||
| optimization | ||
| peft_integration | ||
| pipelines | ||
| quantization | ||
| repo_utils | ||
| sagemaker | ||
| tokenization | ||
| trainer | ||
| utils | ||
| __init__.py | ||
| test_backbone_common.py | ||
| test_cache_utils.py | ||
| test_configuration_common.py | ||
| test_configuration_utils.py | ||
| test_feature_extraction_common.py | ||
| test_feature_extraction_utils.py | ||
| test_image_processing_common.py | ||
| test_image_processing_utils.py | ||
| test_image_transforms.py | ||
| test_modeling_common.py | ||
| test_modeling_flax_common.py | ||
| test_modeling_flax_utils.py | ||
| test_modeling_tf_common.py | ||
| test_modeling_tf_utils.py | ||
| test_modeling_utils.py | ||
| test_pipeline_mixin.py | ||
| test_processing_common.py | ||
| test_sequence_feature_extraction_common.py | ||
| test_tokenization_common.py | ||
| test_tokenization_utils.py | ||