saymrwulf/onnxruntime
1
0
Fork 0
mirror of https://github.com/saymrwulf/onnxruntime.git synced 2026-07-13 18:08:13 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Add SentencePieceTokenizer and GPT2Tokenizer examples for Extensions Python converter API docs (#17708)

Browse source 
Co-authored-by: Sayan Shaw <sayanshaw@microsoft.com>
This commit is contained in:
Sayan Shaw 2023-10-05 15:37:09 -07:00 committed by GitHub
parent 1ad850fd72
commit fb7a2e08fd
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
1 changed files with 8 additions and 3 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

11
docs/extensions/index.md
View file

@ -99,11 +99,16 @@ If the pre processing operator is a HuggingFace tokenizer, you can also easily g
```python
import onnxruntime as _ort
from transformers import AutoTokenizer
from transformers import AutoTokenizer, GPT2Tokenizer
from onnxruntime_extensions import OrtPyFunction, gen_processing_models
tokenizer = AutoTokenizer.from_pretrained('distilbert-base-uncased')
model = OrtPyFunction(gen_processing_models(tokenizer, pre_kwargs={})[0])
# SentencePieceTokenizer
spm_hf_tokenizer = AutoTokenizer.from_pretrained("t5-base", model_max_length=512)
spm_onnx_model = OrtPyFunction(gen_processing_models(spm_hf_tokenizer, pre_kwargs={})[0])
# GPT2Tokenizer
gpt2_hf_tokenizer = GPT2Tokenizer.from_pretrained("Xenova/gpt-4", use_fast=False)
gpt2_onnx_model = OrtPyFunction(gen_processing_models(gpt2_hf_tokenizer, pre_kwargs={})[0])
```
For more information, you can check the API using the following: