saymrwulf/transformers
1
0
Fork 0
mirror of https://github.com/saymrwulf/transformers.git synced 2026-05-14 20:58:08 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

add max_length to showcase the use of truncation (#8975)

Browse source
This commit is contained in:
Navjot 2020-12-07 15:35:39 -08:00 committed by GitHub
parent 62d30e0583
commit c108d0b5a4
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
1 changed files with 2 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
docs/source/quicktour.rst
View file

@ -182,6 +182,7 @@ and get tensors back. You can specify all of that to the tokenizer:
... ["We are very happy to show you the 🤗 Transformers library.", "We hope you don't hate it."], ... ["We are very happy to show you the 🤗 Transformers library.", "We hope you don't hate it."],
... padding=True, ... padding=True,
... truncation=True, ... truncation=True,
... max_length=512,
... return_tensors="pt" ... return_tensors="pt"
... ) ... )
>>> ## TENSORFLOW CODE >>> ## TENSORFLOW CODE
@ -189,6 +190,7 @@ and get tensors back. You can specify all of that to the tokenizer:
... ["We are very happy to show you the 🤗 Transformers library.", "We hope you don't hate it."], ... ["We are very happy to show you the 🤗 Transformers library.", "We hope you don't hate it."],
... padding=True, ... padding=True,
... truncation=True, ... truncation=True,
... max_length=512,
... return_tensors="tf" ... return_tensors="tf"
... ) ... )