saymrwulf/transformers
1
0
Fork 0
mirror of https://github.com/saymrwulf/transformers.git synced 2026-05-14 20:58:08 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
transformers/tests/models
History
Ita Zaporozhets deba7655e6
Add split special tokens (#30772) 
* seems like `split_special_tokens` is used here

* split special token

* add new line at end of file

* moving split special token test to common tests

* added assertions

* test

* fixup

* add co-author

* passing rest of args to gptsan_japanese, fixing tests

* removing direct comparison of fast and slow models

* adding test support for UDOP and LayoutXLM

* ruff fix

* readd check if slow tokenizer

* modify test to handle bos tokens

* removing commented function

* trigger build

* applying review feedback - updated docstrings, var names, and simplified tests

* ruff fixes

* Update tests/test_tokenization_common.py

Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

* applying feedback, comments

* shutil temp directory fix

---------

Co-authored-by: Arthur Zucker <arthur.zucker@gmail.com>
Co-authored-by: Ita Zaporozhets <itazaporozhets@Itas-MBP.localdomain>
Co-authored-by: itazap <itazap@users.noreply.github.com>
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
Co-authored-by: Ita Zaporozhets <itazaporozhets@Itas-MacBook-Pro.local>
2024-05-24 08:38:58 -07:00
..
albert
align update ruff version (#30932) 2024-05-22 06:40:15 +02:00
altclip update ruff version (#30932) 2024-05-22 06:40:15 +02:00
audio_spectrogram_transformer update ruff version (#30932) 2024-05-22 06:40:15 +02:00
auto
autoformer update ruff version (#30932) 2024-05-22 06:40:15 +02:00
bark update ruff version (#30932) 2024-05-22 06:40:15 +02:00
bart update ruff version (#30932) 2024-05-22 06:40:15 +02:00
barthez
bartpho
beit update ruff version (#30932) 2024-05-22 06:40:15 +02:00
bert Fix accelerate failing tests (#30836) 2024-05-23 17:18:58 +02:00
bert_generation
bert_japanese
bertweet
big_bird update ruff version (#30932) 2024-05-22 06:40:15 +02:00
bigbird_pegasus update ruff version (#30932) 2024-05-22 06:40:15 +02:00
biogpt update ruff version (#30932) 2024-05-22 06:40:15 +02:00
bit update ruff version (#30932) 2024-05-22 06:40:15 +02:00
blenderbot update ruff version (#30932) 2024-05-22 06:40:15 +02:00
blenderbot_small update ruff version (#30932) 2024-05-22 06:40:15 +02:00
blip update ruff version (#30932) 2024-05-22 06:40:15 +02:00
blip_2 update ruff version (#30932) 2024-05-22 06:40:15 +02:00
bloom
bridgetower update ruff version (#30932) 2024-05-22 06:40:15 +02:00
bros update ruff version (#30932) 2024-05-22 06:40:15 +02:00
byt5 update ruff version (#30932) 2024-05-22 06:40:15 +02:00
camembert
canine update ruff version (#30932) 2024-05-22 06:40:15 +02:00
chinese_clip update ruff version (#30932) 2024-05-22 06:40:15 +02:00
clap update ruff version (#30932) 2024-05-22 06:40:15 +02:00
clip update ruff version (#30932) 2024-05-22 06:40:15 +02:00
clipseg update ruff version (#30932) 2024-05-22 06:40:15 +02:00
clvp update ruff version (#30932) 2024-05-22 06:40:15 +02:00
code_llama
codegen
cohere update ruff version (#30932) 2024-05-22 06:40:15 +02:00
conditional_detr update ruff version (#30932) 2024-05-22 06:40:15 +02:00
convbert update ruff version (#30932) 2024-05-22 06:40:15 +02:00
convnext update ruff version (#30932) 2024-05-22 06:40:15 +02:00
convnextv2 update ruff version (#30932) 2024-05-22 06:40:15 +02:00
cpm
cpmant update ruff version (#30932) 2024-05-22 06:40:15 +02:00
ctrl
cvt update ruff version (#30932) 2024-05-22 06:40:15 +02:00
data2vec update ruff version (#30932) 2024-05-22 06:40:15 +02:00
dbrx Fix accelerate failing tests (#30836) 2024-05-23 17:18:58 +02:00
deberta
deberta_v2
decision_transformer update ruff version (#30932) 2024-05-22 06:40:15 +02:00
deformable_detr update ruff version (#30932) 2024-05-22 06:40:15 +02:00
deit update ruff version (#30932) 2024-05-22 06:40:15 +02:00
depth_anything update ruff version (#30932) 2024-05-22 06:40:15 +02:00
deta update ruff version (#30932) 2024-05-22 06:40:15 +02:00
detr update ruff version (#30932) 2024-05-22 06:40:15 +02:00
dinat update ruff version (#30932) 2024-05-22 06:40:15 +02:00
dinov2 update ruff version (#30932) 2024-05-22 06:40:15 +02:00
distilbert
dit
donut update ruff version (#30932) 2024-05-22 06:40:15 +02:00
dpr
dpt update ruff version (#30932) 2024-05-22 06:40:15 +02:00
efficientformer update ruff version (#30932) 2024-05-22 06:40:15 +02:00
efficientnet update ruff version (#30932) 2024-05-22 06:40:15 +02:00
electra update ruff version (#30932) 2024-05-22 06:40:15 +02:00
encodec update ruff version (#30932) 2024-05-22 06:40:15 +02:00
encoder_decoder [tests] add torch.use_deterministic_algorithms for XPU (#30774) 2024-05-23 16:53:07 +01:00
ernie
ernie_m update ruff version (#30932) 2024-05-22 06:40:15 +02:00
esm update ruff version (#30932) 2024-05-22 06:40:15 +02:00
falcon update ruff version (#30932) 2024-05-22 06:40:15 +02:00
fastspeech2_conformer update ruff version (#30932) 2024-05-22 06:40:15 +02:00
flaubert
flava update ruff version (#30932) 2024-05-22 06:40:15 +02:00
fnet update ruff version (#30932) 2024-05-22 06:40:15 +02:00
focalnet update ruff version (#30932) 2024-05-22 06:40:15 +02:00
fsmt Encoder-decoder models: move embedding scale to nn.Module (#30410) 2024-05-01 12:33:00 +05:00
funnel update ruff version (#30932) 2024-05-22 06:40:15 +02:00
fuyu update ruff version (#30932) 2024-05-22 06:40:15 +02:00
gemma update ruff version (#30932) 2024-05-22 06:40:15 +02:00
git
glpn update ruff version (#30932) 2024-05-22 06:40:15 +02:00
gpt2 update ruff version (#30932) 2024-05-22 06:40:15 +02:00
gpt_bigcode
gpt_neo update ruff version (#30932) 2024-05-22 06:40:15 +02:00
gpt_neox update ruff version (#30932) 2024-05-22 06:40:15 +02:00
gpt_neox_japanese update ruff version (#30932) 2024-05-22 06:40:15 +02:00
gpt_sw3
gptj
gptsan_japanese Fix accelerate failing tests (#30836) 2024-05-23 17:18:58 +02:00
graphormer update ruff version (#30932) 2024-05-22 06:40:15 +02:00
grounding_dino update ruff version (#30932) 2024-05-22 06:40:15 +02:00
groupvit update ruff version (#30932) 2024-05-22 06:40:15 +02:00
herbert
hubert update ruff version (#30932) 2024-05-22 06:40:15 +02:00
ibert Encoder-decoder models: move embedding scale to nn.Module (#30410) 2024-05-01 12:33:00 +05:00
idefics update ruff version (#30932) 2024-05-22 06:40:15 +02:00
idefics2 Encoder-decoder models: move embedding scale to nn.Module (#30410) 2024-05-01 12:33:00 +05:00
imagegpt Encoder-decoder models: move embedding scale to nn.Module (#30410) 2024-05-01 12:33:00 +05:00
informer update ruff version (#30932) 2024-05-22 06:40:15 +02:00
instructblip update ruff version (#30932) 2024-05-22 06:40:15 +02:00
jamba update ruff version (#30932) 2024-05-22 06:40:15 +02:00
jetmoe Add JetMoE model (#30005) 2024-05-14 16:32:01 +02:00
jukebox
kosmos2 update ruff version (#30932) 2024-05-22 06:40:15 +02:00
layoutlm update ruff version (#30932) 2024-05-22 06:40:15 +02:00
layoutlmv2 update ruff version (#30932) 2024-05-22 06:40:15 +02:00
layoutlmv3 update ruff version (#30932) 2024-05-22 06:40:15 +02:00
layoutxlm Add split special tokens (#30772) 2024-05-24 08:38:58 -07:00
led update ruff version (#30932) 2024-05-22 06:40:15 +02:00
levit update ruff version (#30932) 2024-05-22 06:40:15 +02:00
lilt
llama add prefix space ignored in llama #29625 (#30964) 2024-05-24 01:03:00 -07:00
llava Support arbitrary processor (#30875) 2024-05-17 16:51:31 +02:00
llava_next LLaVa-Next: Update docs with batched inference (#30857) 2024-05-20 13:45:56 +05:00
longformer update ruff version (#30932) 2024-05-22 06:40:15 +02:00
longt5
luke update ruff version (#30932) 2024-05-22 06:40:15 +02:00
lxmert update ruff version (#30932) 2024-05-22 06:40:15 +02:00
m2m_100 update ruff version (#30932) 2024-05-22 06:40:15 +02:00
mamba
marian update ruff version (#30932) 2024-05-22 06:40:15 +02:00
markuplm update ruff version (#30932) 2024-05-22 06:40:15 +02:00
mask2former update ruff version (#30932) 2024-05-22 06:40:15 +02:00
maskformer update ruff version (#30932) 2024-05-22 06:40:15 +02:00
mbart update ruff version (#30932) 2024-05-22 06:40:15 +02:00
mbart50
mega
megatron_bert update ruff version (#30932) 2024-05-22 06:40:15 +02:00
megatron_gpt2
mgp_str update ruff version (#30932) 2024-05-22 06:40:15 +02:00
mistral [Port] TensorFlow implementation of Mistral (#29708) 2024-05-23 17:48:49 +01:00
mixtral update ruff version (#30932) 2024-05-22 06:40:15 +02:00
mluke
mobilebert update ruff version (#30932) 2024-05-22 06:40:15 +02:00
mobilenet_v1 update ruff version (#30932) 2024-05-22 06:40:15 +02:00
mobilenet_v2 update ruff version (#30932) 2024-05-22 06:40:15 +02:00
mobilevit update ruff version (#30932) 2024-05-22 06:40:15 +02:00
mobilevitv2 update ruff version (#30932) 2024-05-22 06:40:15 +02:00
mpnet update ruff version (#30932) 2024-05-22 06:40:15 +02:00
mpt Update 4 MptIntegrationTests expected outputs (#30989) 2024-05-23 18:27:54 +02:00
mra update ruff version (#30932) 2024-05-22 06:40:15 +02:00
mt5 Fix accelerate failing tests (#30836) 2024-05-23 17:18:58 +02:00
musicgen update ruff version (#30932) 2024-05-22 06:40:15 +02:00
musicgen_melody update ruff version (#30932) 2024-05-22 06:40:15 +02:00
mvp update ruff version (#30932) 2024-05-22 06:40:15 +02:00
nat update ruff version (#30932) 2024-05-22 06:40:15 +02:00
nezha
nllb Remove deprecated properties in tokenization_nllb.py and tokenization_nllb_fast.py (#29834) 2024-05-23 18:53:26 +02:00
nllb_moe update ruff version (#30932) 2024-05-22 06:40:15 +02:00
nougat
nystromformer update ruff version (#30932) 2024-05-22 06:40:15 +02:00
olmo update ruff version (#30932) 2024-05-22 06:40:15 +02:00
oneformer update ruff version (#30932) 2024-05-22 06:40:15 +02:00
openai
opt update ruff version (#30932) 2024-05-22 06:40:15 +02:00
owlv2 update ruff version (#30932) 2024-05-22 06:40:15 +02:00
owlvit update ruff version (#30932) 2024-05-22 06:40:15 +02:00
paligemma Paligemma causal attention mask (#30967) 2024-05-22 19:37:15 +02:00
patchtsmixer update ruff version (#30932) 2024-05-22 06:40:15 +02:00
patchtst update ruff version (#30932) 2024-05-22 06:40:15 +02:00
pegasus update ruff version (#30932) 2024-05-22 06:40:15 +02:00
pegasus_x update ruff version (#30932) 2024-05-22 06:40:15 +02:00
perceiver Perceiver interpolate position embedding (#30979) 2024-05-24 11:13:58 +01:00
persimmon update ruff version (#30932) 2024-05-22 06:40:15 +02:00
phi update ruff version (#30932) 2024-05-22 06:40:15 +02:00
phi3 update ruff version (#30932) 2024-05-22 06:40:15 +02:00
phobert
pix2struct update ruff version (#30932) 2024-05-22 06:40:15 +02:00
plbart update ruff version (#30932) 2024-05-22 06:40:15 +02:00
poolformer update ruff version (#30932) 2024-05-22 06:40:15 +02:00
pop2piano update ruff version (#30932) 2024-05-22 06:40:15 +02:00
prophetnet update ruff version (#30932) 2024-05-22 06:40:15 +02:00
pvt update ruff version (#30932) 2024-05-22 06:40:15 +02:00
pvt_v2
qdqbert update ruff version (#30932) 2024-05-22 06:40:15 +02:00
qwen2 update ruff version (#30932) 2024-05-22 06:40:15 +02:00
qwen2_moe update ruff version (#30932) 2024-05-22 06:40:15 +02:00
rag
realm update ruff version (#30932) 2024-05-22 06:40:15 +02:00
recurrent_gemma update ruff version (#30932) 2024-05-22 06:40:15 +02:00
reformer
regnet update ruff version (#30932) 2024-05-22 06:40:15 +02:00
rembert update ruff version (#30932) 2024-05-22 06:40:15 +02:00
resnet update ruff version (#30932) 2024-05-22 06:40:15 +02:00
roberta
roberta_prelayernorm
roc_bert update ruff version (#30932) 2024-05-22 06:40:15 +02:00
roformer update ruff version (#30932) 2024-05-22 06:40:15 +02:00
rwkv
sam update ruff version (#30932) 2024-05-22 06:40:15 +02:00
seamless_m4t Remove deprecated properties in tokenization_nllb.py and tokenization_nllb_fast.py (#29834) 2024-05-23 18:53:26 +02:00
seamless_m4t_v2 update ruff version (#30932) 2024-05-22 06:40:15 +02:00
segformer update ruff version (#30932) 2024-05-22 06:40:15 +02:00
seggpt update ruff version (#30932) 2024-05-22 06:40:15 +02:00
sew update ruff version (#30932) 2024-05-22 06:40:15 +02:00
sew_d update ruff version (#30932) 2024-05-22 06:40:15 +02:00
siglip Fix accelerate failing tests (#30836) 2024-05-23 17:18:58 +02:00
speech_encoder_decoder [tests] add torch.use_deterministic_algorithms for XPU (#30774) 2024-05-23 16:53:07 +01:00
speech_to_text update ruff version (#30932) 2024-05-22 06:40:15 +02:00
speech_to_text_2 update ruff version (#30932) 2024-05-22 06:40:15 +02:00
speecht5 [tests] add torch.use_deterministic_algorithms for XPU (#30774) 2024-05-23 16:53:07 +01:00
splinter update ruff version (#30932) 2024-05-22 06:40:15 +02:00
squeezebert
stablelm update ruff version (#30932) 2024-05-22 06:40:15 +02:00
starcoder2 update ruff version (#30932) 2024-05-22 06:40:15 +02:00
superpoint Removal of deprecated maps (#30576) 2024-05-09 14:15:56 +02:00
swiftformer update ruff version (#30932) 2024-05-22 06:40:15 +02:00
swin update ruff version (#30932) 2024-05-22 06:40:15 +02:00
swin2sr update ruff version (#30932) 2024-05-22 06:40:15 +02:00
swinv2 update ruff version (#30932) 2024-05-22 06:40:15 +02:00
switch_transformers Fix accelerate failing tests (#30836) 2024-05-23 17:18:58 +02:00
t5 Fix accelerate failing tests (#30836) 2024-05-23 17:18:58 +02:00
table_transformer update ruff version (#30932) 2024-05-22 06:40:15 +02:00
tapas update ruff version (#30932) 2024-05-22 06:40:15 +02:00
time_series_transformer update ruff version (#30932) 2024-05-22 06:40:15 +02:00
timesformer update ruff version (#30932) 2024-05-22 06:40:15 +02:00
timm_backbone 🚨 out_indices always a list (#30941) 2024-05-22 15:23:04 +01:00
trocr update ruff version (#30932) 2024-05-22 06:40:15 +02:00
tvlt update ruff version (#30932) 2024-05-22 06:40:15 +02:00
tvp update ruff version (#30932) 2024-05-22 06:40:15 +02:00
udop Add split special tokens (#30772) 2024-05-24 08:38:58 -07:00
umt5 Fix accelerate failing tests (#30836) 2024-05-23 17:18:58 +02:00
unispeech update ruff version (#30932) 2024-05-22 06:40:15 +02:00
unispeech_sat update ruff version (#30932) 2024-05-22 06:40:15 +02:00
univnet
upernet update ruff version (#30932) 2024-05-22 06:40:15 +02:00
video_llava update ruff version (#30932) 2024-05-22 06:40:15 +02:00
videomae update ruff version (#30932) 2024-05-22 06:40:15 +02:00
vilt update ruff version (#30932) 2024-05-22 06:40:15 +02:00
vipllava update ruff version (#30932) 2024-05-22 06:40:15 +02:00
vision_encoder_decoder update ruff version (#30932) 2024-05-22 06:40:15 +02:00
vision_text_dual_encoder update ruff version (#30932) 2024-05-22 06:40:15 +02:00
visual_bert update ruff version (#30932) 2024-05-22 06:40:15 +02:00
vit update ruff version (#30932) 2024-05-22 06:40:15 +02:00
vit_hybrid update ruff version (#30932) 2024-05-22 06:40:15 +02:00
vit_mae added interpolation for vitmae model in pytorch as well as tf. (#30732) 2024-05-24 16:20:09 +01:00
vit_msn update ruff version (#30932) 2024-05-22 06:40:15 +02:00
vitdet update ruff version (#30932) 2024-05-22 06:40:15 +02:00
vitmatte update ruff version (#30932) 2024-05-22 06:40:15 +02:00
vits update ruff version (#30932) 2024-05-22 06:40:15 +02:00
vivit update ruff version (#30932) 2024-05-22 06:40:15 +02:00
wav2vec2 update ruff version (#30932) 2024-05-22 06:40:15 +02:00
wav2vec2_bert update ruff version (#30932) 2024-05-22 06:40:15 +02:00
wav2vec2_conformer update ruff version (#30932) 2024-05-22 06:40:15 +02:00
wav2vec2_phoneme update ruff version (#30932) 2024-05-22 06:40:15 +02:00
wav2vec2_with_lm Add AutoFeatureExtractor support to Wav2Vec2ProcessorWithLM (#28706) 2024-05-20 13:40:42 +02:00
wavlm update ruff version (#30932) 2024-05-22 06:40:15 +02:00
whisper update ruff version (#30932) 2024-05-22 06:40:15 +02:00
x_clip update ruff version (#30932) 2024-05-22 06:40:15 +02:00
xglm
xlm
xlm_prophetnet
xlm_roberta
xlm_roberta_xl
xlnet update ruff version (#30932) 2024-05-22 06:40:15 +02:00
xmod
yolos update ruff version (#30932) 2024-05-22 06:40:15 +02:00
yoso update ruff version (#30932) 2024-05-22 06:40:15 +02:00
__init__.py