saymrwulf/transformers
1
0
Fork 0
mirror of https://github.com/saymrwulf/transformers.git synced 2026-05-14 20:58:08 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
transformers/docs/source/model_doc
History
Sidd Karamcheti 3a8de58c51
Add Mistral GPT-2 Stability Tweaks (#13573) 
* Add layer-wise scaling

* Add reorder & upcasting argument

* Add OpenAI GPT-2 weight initialization scheme

* start `layer_idx` count at zero for consistency

* disentangle attn and reordered and upscaled attn function

* rename `scale_attn_by_layer` to `scale_attn_by_layer_id`

* make autocast from amp compatible with pytorch<1.6

* fix docstring

* style fixes

* Add fixes from PR feedback, style tweaks

* Fix doc whitespace

* Reformat

* First pass scale_attn_by_layer_idx and reorder_and_upcast_attn tests

* Rename scale_attn_by_layer_idx, add tip

* Remove extra newline

* add test for weight initialization

* update code format

* add assert check weights are fp32

* remove assert

* Fix incorrect merge

* Fix shape mismatch in baddbmm

* Add generation test for Mistral flags

Co-authored-by: leandro <leandro.vonwerra@spoud.io>
Co-authored-by: Keshav Santhanam <keshav2@stanford.edu>
Co-authored-by: J38 <jebolton@stanford.edu>
2021-10-04 07:37:09 -04:00
..
albert.rst albert flax (#13294) 2021-08-30 17:29:27 +02:00
auto.rst Add Speech AutoModels (#13655) 2021-09-21 08:50:33 +02:00
bart.rst FlaxBart (#11537) 2021-06-14 15:16:08 +05:30
barthez.rst Examples reorg (#11350) 2021-04-21 11:11:20 -04:00
beit.rst beit-flax (#13515) 2021-09-21 13:34:19 +02:00
bert.rst [Flax] Correct flax docs (#12782) 2021-08-04 16:31:23 +02:00
bert_japanese.rst Honor contributors to models (#11329) 2021-04-21 09:47:27 -04:00
bertgeneration.rst Honor contributors to models (#11329) 2021-04-21 09:47:27 -04:00
bertweet.rst Honor contributors to models (#11329) 2021-04-21 09:47:27 -04:00
bigbird.rst Flax Big Bird (#11967) 2021-06-14 20:01:03 +01:00
bigbird_pegasus.rst Add BigBirdPegasus (#10991) 2021-05-07 09:27:43 +02:00
blenderbot.rst Honor contributors to models (#11329) 2021-04-21 09:47:27 -04:00
blenderbot_small.rst Add BlenderBot small tokenizer to the init (#13367) 2021-09-22 19:00:47 -04:00
bort.rst Honor contributors to models (#11329) 2021-04-21 09:47:27 -04:00
byt5.rst Improve T5 docs (#13240) 2021-09-01 15:05:40 +02:00
camembert.rst Honor contributors to models (#11329) 2021-04-21 09:47:27 -04:00
canine.rst Wrong model is used in example, should be character instead of subword model (#12676) 2021-07-13 08:40:27 -04:00
clip.rst add and fix examples (#12810) 2021-07-20 09:28:50 -04:00
convbert.rst Honor contributors to models (#11329) 2021-04-21 09:47:27 -04:00
cpm.rst Honor contributors to models (#11329) 2021-04-21 09:47:27 -04:00
ctrl.rst Honor contributors to models (#11329) 2021-04-21 09:47:27 -04:00
deberta.rst Deberta tf (#12972) 2021-08-12 05:01:26 -04:00
deberta_v2.rst Deberta_v2 tf (#13120) 2021-08-31 06:32:47 -04:00
deit.rst Honor contributors to models (#11329) 2021-04-21 09:47:27 -04:00
detr.rst Improve detr (#12147) 2021-06-17 10:37:54 -04:00
dialogpt.rst ADD BORT (#9813) 2021-01-27 21:25:11 +03:00
distilbert.rst Fix typo distilbert doc (#13643) 2021-09-20 15:10:33 -04:00
dpr.rst [DPR] Correct init (#13796) 2021-09-30 18:55:20 +02:00
electra.rst [Flax] Add Electra models (#11426) 2021-05-04 20:56:09 +02:00
encoderdecoder.rst Make Flax GPT2 working with cross attention (#13008) 2021-08-23 17:57:29 +02:00
flaubert.rst Honor contributors to models (#11329) 2021-04-21 09:47:27 -04:00
fnet.rst Add FNet (#13045) 2021-09-20 13:24:30 +02:00
fsmt.rst Honor contributors to models (#11329) 2021-04-21 09:47:27 -04:00
funnel.rst Honor contributors to models (#11329) 2021-04-21 09:47:27 -04:00
gpt.rst Honor contributors to models (#11329) 2021-04-21 09:47:27 -04:00
gpt2.rst Add Mistral GPT-2 Stability Tweaks (#13573) 2021-10-04 07:37:09 -04:00
gpt_neo.rst FlaxGPTNeo (#12493) 2021-07-06 18:55:18 +05:30
gptj.rst [docs/gpt-j] fix typo (#13851) 2021-10-04 12:30:50 +02:00
herbert.rst Honor contributors to models (#11329) 2021-04-21 09:47:27 -04:00
hubert.rst Add Wav2Vec2 & Hubert ForSequenceClassification (#13153) 2021-08-27 20:52:51 +03:00
ibert.rst Honor contributors to models (#11329) 2021-04-21 09:47:27 -04:00
layoutlm.rst Honor contributors to models (#11329) 2021-04-21 09:47:27 -04:00
layoutlmv2.rst Add LayoutLMv2 + LayoutXLM (#12604) 2021-08-30 12:35:42 +02:00
layoutxlm.rst Add tokenizer docs (#13373) 2021-09-02 09:46:05 +02:00
led.rst Make gradient_checkpointing a training argument (#13657) 2021-09-22 07:51:38 -04:00
longformer.rst Honor contributors to models (#11329) 2021-04-21 09:47:27 -04:00
luke.rst Add LUKE (#11223) 2021-05-03 09:07:29 -04:00
lxmert.rst Honor contributors to models (#11329) 2021-04-21 09:47:27 -04:00
m2m_100.rst replace tgt_lang by tgt_text (#13061) 2021-08-09 22:47:05 +05:30
marian.rst Rely on huggingface_hub for common tools (#13100) 2021-08-12 14:59:02 +02:00
mbart.rst fix example (#13387) 2021-09-02 11:32:18 +02:00
megatron_bert.rst Honor contributors to models (#11329) 2021-04-21 09:47:27 -04:00
megatron_gpt2.rst Honor contributors to models (#11329) 2021-04-21 09:47:27 -04:00
mobilebert.rst Honor contributors to models (#11329) 2021-04-21 09:47:27 -04:00
mpnet.rst MPNet copyright files (#9015) 2020-12-10 09:29:38 -05:00
mt5.rst Fix mT5 documentation (#13639) 2021-09-20 07:53:31 -04:00
pegasus.rst [Flax] Addition of FlaxPegasus (#13420) 2021-09-14 17:15:19 +02:00
phobert.rst Honor contributors to models (#11329) 2021-04-21 09:47:27 -04:00
prophetnet.rst Copyright (#8970) 2020-12-07 18:36:34 -05:00
rag.rst Honor contributors to models (#11329) 2021-04-21 09:47:27 -04:00
reformer.rst Honor contributors to models (#11329) 2021-04-21 09:47:27 -04:00
rembert.rst Add RemBERT model code to huggingface (#10692) 2021-07-24 11:31:42 -04:00
retribert.rst Examples reorg (#11350) 2021-04-21 11:11:20 -04:00
roberta.rst [FlaxRoberta] Add FlaxRobertaModels & adapt run_mlm_flax.py (#11470) 2021-05-04 19:57:59 +02:00
roformer.rst [RoFormer] Fix some issues (#12397) 2021-07-06 03:31:57 -04:00
speech_to_text.rst fix: typo spelling grammar (#13212) 2021-08-30 08:09:14 -04:00
speech_to_text_2.rst Add SpeechEncoderDecoder & Speech2Text2 (#13186) 2021-09-01 13:33:31 +02:00
speechencoderdecoder.rst Add SpeechEncoderDecoder & Speech2Text2 (#13186) 2021-09-01 13:33:31 +02:00
splinter.rst Add splinter (#12955) 2021-08-17 08:29:01 -04:00
squeezebert.rst Honor contributors to models (#11329) 2021-04-21 09:47:27 -04:00
t5.rst Improve T5 docs (#13240) 2021-09-01 15:05:40 +02:00
t5v1.1.rst Improve T5 docs (#13240) 2021-09-01 15:05:40 +02:00
tapas.rst Honor contributors to models (#11329) 2021-04-21 09:47:27 -04:00
transformerxl.rst Honor contributors to models (#11329) 2021-04-21 09:47:27 -04:00
visual_bert.rst Fix VisualBERT docs (#13106) 2021-08-13 11:44:04 +05:30
vit.rst Add DINO conversion script (#13265) 2021-08-26 17:25:20 +02:00
wav2vec2.rst Add Wav2Vec2 & Hubert ForSequenceClassification (#13153) 2021-08-27 20:52:51 +03:00
xlm.rst Honor contributors to models (#11329) 2021-04-21 09:47:27 -04:00
xlmprophetnet.rst Copyright (#8970) 2020-12-07 18:36:34 -05:00
xlmroberta.rst Honor contributors to models (#11329) 2021-04-21 09:47:27 -04:00
xlnet.rst Examples reorg (#11350) 2021-04-21 11:11:20 -04:00
xlsr_wav2vec2.rst [XLSR-Wav2Vec2] Add multi-lingual Wav2Vec2 models (#10648) 2021-03-11 17:44:18 +03:00