Nicolas Patry c9837a0d27 Conversion from slow to fast for BPE spm vocabs contained an error. (#10120) ... * Conversion from slow to fast for BPE spm vocabs contained an error. - There is only 1 test currently (tokenizers + slow) that used the modified path and it's reformer, which does not contain any ids modification so the bug was silent for now. - The real issue is that vocab variable was overloaded by SentencePieceExtractor, leading to Slow specific vocab oddities to be completely ignored - The bug was reported here https://github.com/huggingface/transformers/issues/9518 - Ran the complete tokenization test suite with slow without error (`RUN_SLOW=1 pytest -sv tests/test_tokenization_*`) * Remove rebase error. * Adding the fixture.		2021-02-13 08:24:53 -05:00
..
tests_samples	New run_seq2seq script (#9605)	2021-01-19 15:22:17 -05:00
dummy-config.json
empty.txt
input.txt
sample_text.txt
sample_text_no_unicode.txt	[Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659)	2020-10-18 20:51:24 +02:00
spiece.model
test_sentencepiece.model
test_sentencepiece_bpe.model	Conversion from slow to fast for BPE spm vocabs contained an error. (#10120)	2021-02-13 08:24:53 -05:00
test_sentencepiece_no_bos.model	[pegasus] Faster tokenizer tests (#7672)	2020-10-09 11:10:32 -04:00