saymrwulf/transformers
1
0
Fork 0
mirror of https://github.com/saymrwulf/transformers.git synced 2026-05-15 21:01:19 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

fix gpt-2 unk token test

Browse source
This commit is contained in:
thomwolf 2019-07-12 11:46:57 +02:00
parent 762ded9b1c
commit 699bc7e86e
2 changed files with 4 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
docs/README.md
View file

@ -57,4 +57,4 @@ It should build the static app that will be available under `/docs/_build/html`
## Adding a new element to the tree (toc-tree)
Accepted files are reStructuredText (.rst) and Markdown (.md). Create a file with its extension and put it
in the source directory. You can then link it to the toc-tree by putting the filename without the extension.
in the source directory. You can then link it to the toc-tree by putting the filename without the extension.

4
pytorch_transformers/tokenization_gpt2.py
View file

@ -177,7 +177,9 @@ class GPT2Tokenizer(PreTrainedTokenizer):
def _convert_token_to_id(self, token):
""" Converts a token (str/unicode) in an id using the vocab. """
return self.encoder.get(token)
if token in self.encoder:
return self.encoder.get(token)
return self.encoder.get(self.unk_token)
def _convert_id_to_token(self, index):
"""Converts an index (integer) in a token (string/unicode) using the vocab."""