mirror of
https://github.com/saymrwulf/onnxruntime.git
synced 2026-07-13 18:08:13 +00:00
Added initial ORT Extensions docs for onnxruntime website (#16723)
## Preview staged site [here](https://sayanshaw24.github.io/onnxruntime/docs/extensions/). **TODO**: In the future, make PR to add/update descriptions for the following new/deprecated ops: **NLP Ops** - BlingFireSentenceBreaker - BpeTokenizer **String Ops** - StringSplit - StringUpper - StringLower - StringECMARegexSplitWithOffsets - StringRaggedTensorToDense - StringMapping **Math Ops** - Inverse - NegPos - SegmentExtraction - SegmentSum **Tensor Ops** - RaggedTensorToSparse - RaggedTensorToDense --------- Co-authored-by: Sayan Shaw <sayanshaw@microsoft.com> Co-authored-by: Nat Kershaw (MSFT) <nakersha@microsoft.com>
This commit is contained in:
Sayan Shaw
GitHub
parent
43da400aa6
commit
cdba245a6e
8 changed files with 311 additions and 4 deletions
2
.github/workflows/check-website-links.yml
vendored
2
.github/workflows/check-website-links.yml
vendored
|
|
@ -23,4 +23,4 @@ jobs:
|
|||
run: bundle exec jekyll build --drafts
|
||||
- name: Check for broken links
|
||||
run: |
|
||||
bundle exec htmlproofer --assume_extension --checks_to_ignore ImageCheck,ScriptCheck --only_4xx --http_status_ignore 429,403 --allow_hash_href --url_ignore "https://onnxruntime.ai/docs/reference/api/c-api.html,https://www.onnxruntime.ai/docs/reference/execution-providers/TensorRT-ExecutionProvider.html#c-api-example,https://www.onnxruntime.ai/docs/resources/graph-optimizations.html,onnxruntime/capi/onnxruntime_pybind11_state.html,https://github.com/microsoft/onnx-converters-private/" --log-level :info ./_site
|
||||
bundle exec htmlproofer --assume_extension --checks_to_ignore ImageCheck,ScriptCheck --only_4xx --http_status_ignore 429,403 --allow_hash_href --url_ignore "https://onnxruntime.ai/docs/reference/api/c-api.html,https://www.onnxruntime.ai/docs/reference/execution-providers/TensorRT-ExecutionProvider.html#c-api-example,https://www.onnxruntime.ai/docs/resources/graph-optimizations.html,onnxruntime/capi/onnxruntime_pybind11_state.html,https://github.com/microsoft/onnx-converters-private/issues/new/choose" --log-level :info ./_site
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
---
|
||||
title: Ecosystem
|
||||
description: See examples of how ONNX Runtime working end to end within the Azure AI and ML landscape and ecosystem
|
||||
nav_order: 8
|
||||
nav_order: 9
|
||||
redirect_from: /docs/tutorials/ecosystem
|
||||
---
|
||||
# ORT Ecosystem
|
||||
|
|
|
|||
82
docs/extensions/add-op.md
Normal file
82
docs/extensions/add-op.md
Normal file
|
|
@ -0,0 +1,82 @@
|
|||
---
|
||||
title: Add Operators
|
||||
description: Instructions to add a new custom operator
|
||||
parent: Extensions
|
||||
nav_order: 2
|
||||
---
|
||||
|
||||
# Creating custom operators using Python functions
|
||||
|
||||
Custom operators are a powerful feature in ONNX Runtime that allows users to extend the functionality of the runtime by implementing their own operators to perform specific operations not available in the standard ONNX operator set.
|
||||
|
||||
In this document, we will introduce how to create a custom operator using Python functions and integrate it into ONNX Runtime for inference.
|
||||
|
||||
|
||||
## Step 1: Define the Python function for the custom operator
|
||||
Start by defining the Python function that will serve as the implementation for your custom operator. Ensure that the function is compatible with the input and output tensor shapes you expect for your custom operator.
|
||||
the Python decorator @onnx_op will convert the function to be a custom operator implementation. The following is example we create a function for a tokenizer
|
||||
|
||||
```Python
|
||||
@onnx_op(op_type="GPT2Tokenizer",
|
||||
inputs=[PyCustomOpDef.dt_string],
|
||||
outputs=[PyCustomOpDef.dt_int64, PyCustomOpDef.dt_int64],
|
||||
attrs={"padding_length": PyCustomOpDef.dt_int64})
|
||||
def bpe_tokenizer(s, **kwargs):
|
||||
padding_length = kwargs["padding_length"]
|
||||
input_ids, attention_mask = cls.tokenizer.tokenizer_sentence([s[0]], padding_length)
|
||||
return input_ids, attention_mask
|
||||
```
|
||||
Because ONNXRuntimme needs the custom operator schema on loading a model, please specify them by onnx_op arguments. Also 'attrs' is needed if there are attributes for the ONNX node, which can be dict that mapping from its name to its type, or be a list if all types are string only.
|
||||
|
||||
## Step 2: Create an ONNX model with the custom operator
|
||||
Now that the custom operator is registered with ONNX Runtime, you can create an ONNX model that utilizes it. You can either modify an existing ONNX model to include the custom operator or create a new one from scratch.
|
||||
|
||||
To create a new ONNX model with the custom operator, you can use the ONNX Python API. Here is an example: [test_pyops.py](https://github.com/microsoft/onnxruntime-extensions/blob/main/test/test_pyops.py)
|
||||
|
||||
# Create a Custom Operator from Scratch in C++
|
||||
|
||||
Before implementing a custom operator, you need an ONNX model with one or more ORT custom operators, created by ONNX converters, such as [ONNX-Script](https://github.com/microsoft/onnx-script), [ONNX model API](https://onnx.ai/onnx/api/helper.html), etc.
|
||||
|
||||
|
||||
## 1. Quick verification with PythonOp (optional)
|
||||
|
||||
Before you actually develop a custom operator for your use case, if you want to quickly verify the ONNX model with Python, you can wrap the custom operator with Python functions as described above.
|
||||
|
||||
```python
|
||||
import numpy
|
||||
from onnxruntime_extensions import PyOp, onnx_op
|
||||
|
||||
# Implement the CustomOp by decorating a function with onnx_op
|
||||
@onnx_op(op_type="Inverse", inputs=[PyOp.dt_float])
|
||||
def inverse(x):
|
||||
# the user custom op implementation here:
|
||||
return numpy.linalg.inv(x)
|
||||
|
||||
# Run the model with this custom op
|
||||
# model_func = PyOrtFunction(model_path)
|
||||
# outputs = model_func(inputs)
|
||||
# ...
|
||||
```
|
||||
|
||||
## 2. Generate the C++ template code of the Custom operator from the ONNX Model (optional)
|
||||
python -m onnxruntime-extensions.cmd --cpp-gen <model_path> <repository_dir>`
|
||||
If you are familiar with the ONNX model detail, you create the custom operator C++ classes directly.
|
||||
|
||||
|
||||
## 3. Implement the CustomOp Kernel Compute method in the generated C++ files.
|
||||
the custom operator kernel C++ code example can be found [operators](https://github.com/microsoft/onnxruntime-extensions/tree/main/operators) folder, like [gaussian_blur](https://github.com/microsoft/onnxruntime-extensions/blob/main/operators/cv2/imgproc/gaussian_blur.hpp). All C++ APIs that can be used in the kernel implementation are listed below
|
||||
|
||||
* [ONNXRuntime Custom API docs](https://onnxruntime.ai/docs/api/c/struct_ort_custom_op.html)
|
||||
* the third libraries API docs integrated in ONNXRuntime Extensions the can be used in C++ code
|
||||
- OpenCV API docs https://docs.opencv.org/4.x/
|
||||
- Google SentencePiece Library docs https://github.com/google/sentencepiece/blob/master/doc/api.md
|
||||
- dlib(matrix and ML library) C++ API docs http://dlib.net/algorithms.html
|
||||
- BlingFire Library https://github.com/microsoft/BlingFire
|
||||
- Google RE2 Library https://github.com/google/re2/wiki/CplusplusAPI
|
||||
- JSON library https://json.nlohmann.me/api/basic_json/
|
||||
|
||||
## 3. Build and Test
|
||||
- The unit tests can be implemented as Python or C++, check [test](https://github.com/microsoft/onnxruntime-extensions/tree/main/test) folder for more examples
|
||||
- Check [build-package](./build.md) on how to build the different language package to be used for production.
|
||||
|
||||
Please check the [contribution](./index.md#contributing) to see if it is possible to contribute the custom operator to onnxruntime-extensions.
|
||||
90
docs/extensions/build.md
Normal file
90
docs/extensions/build.md
Normal file
|
|
@ -0,0 +1,90 @@
|
|||
---
|
||||
title: Build
|
||||
description: Instructions for building and developing ORT Extensions.
|
||||
parent: Extensions
|
||||
nav_order: 3
|
||||
---
|
||||
# Build from Source
|
||||
|
||||
This project supports Python and can be built from source easily, or a simple cmake build without Python dependency.
|
||||
## Python package
|
||||
The package contains all custom operators and some Python scripts to manipulate the ONNX models.
|
||||
- Install Visual Studio with C++ development tools on Windows, or gcc(>8.0) for Linux or xcode for macOS, and cmake on the unix-like platform. (**hints**: in Windows platform, if cmake bundled in Visual Studio was used, please specify the set _VSDEVCMD=%ProgramFiles(x86)%\Microsoft Visual Studio\<VERSION_YEAR>\<Edition>\Common7\Tools\VsDevCmd.bat_)
|
||||
- If running on Windows, ensure that long file names are enabled, both for the [operating system](https://docs.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=cmd) and for git: `git config --system core.longpaths true`
|
||||
- Prepare Python env and install the pip packages in the requirements.txt.
|
||||
- `pip install .` to build and install the package.<br/> OR `pip install -e .` to install the package in the development mode, which is more friendly for the developer since the Python code change will take effect without having to copy the files to a different location in the disk.(**hints**: debug=1 in setup.cfg wil make C++ code be debuggable in a Python process.)
|
||||
|
||||
Test:
|
||||
- 'pip install -r requirements-dev.txt' to install pip packages for development.
|
||||
- run `pytest test` in the project root directory.
|
||||
|
||||
For a complete list of verified build configurations see [here](./build.md#dependencies)
|
||||
|
||||
## Java package
|
||||
|
||||
Run `bash ./build.sh -DOCOS_BUILD_JAVA=ON` to build jar package in out/<OS>/Release folder
|
||||
|
||||
## Android package
|
||||
- pre-requisites: [Android Studio](https://developer.android.com/studio)
|
||||
|
||||
Use `./tools/android/build_aar.py` to build an Android AAR package.
|
||||
|
||||
## iOS package
|
||||
Use `./tools/ios/build_xcframework.py` to build an iOS xcframework package.
|
||||
|
||||
## Web-Assembly
|
||||
ONNXRuntime-Extensions will be built as a static library and linked with ONNXRuntime due to the lack of a good dynamic linking mechanism in WASM. Here are two additional arguments [–-use_extensions and --extensions_overridden_path](https://github.com/microsoft/onnxruntime/blob/860ba8820b72d13a61f0d08b915cd433b738ffdc/tools/ci_build/build.py#L416) on building onnxruntime to include ONNXRuntime-Extensions footprint in the ONNXRuntime package.
|
||||
|
||||
## The C++ shared library
|
||||
for any other cases, please run `build.bat` or `bash ./build.sh` to build the library. By default, the DLL or the library will be generated in the directory `out/<OS>/<FLAVOR>`. There is a unit test to help verify the build.
|
||||
|
||||
|
||||
**VC Runtime static linkage**
|
||||
If you want to build the binary with VC Runtime static linkage, please add a parameter _-DCMAKE_MSVC_RUNTIME_LIBRARY="MultiThreaded$<$<CONFIG:Debug>:Debug>"_ on running build.bat
|
||||
|
||||
## Copyright guidance
|
||||
check this link https://docs.opensource.microsoft.com/releasing/general-guidance/copyright-headers/ for source file copyright header.
|
||||
|
||||
|
||||
|
||||
# Build ONNX Runtime with onnxruntime-extensions for Java package
|
||||
|
||||
*The following step are demonstrated for Windows Platform only, the others like Linux and MacOS can be done similarly.*
|
||||
|
||||
> Android build was supported as well; check [here](https://onnxruntime.ai/docs/build/android.html#cross-compiling-on-windows) for arguments to build AAR package.
|
||||
|
||||
## Tools required
|
||||
1. install visual studio 2022 (with cmake, git, desktop C++)
|
||||
2. install miniconda to have Python support (for onnxruntime build)
|
||||
3. OpenJDK: https://docs.microsoft.com/en-us/java/openjdk/download
|
||||
(OpenJDK 11.0.15 LTS)
|
||||
4. Gradle: https://gradle.org/releases/
|
||||
(v6.9.2)
|
||||
|
||||
## Commands
|
||||
Launch **Developer PowerShell for VS 2022** in Windows Tereminal
|
||||
```
|
||||
. $home\miniconda3\shell\condabin\conda-hook.ps1
|
||||
conda activate base
|
||||
|
||||
$env:JAVA_HOME="C:\Program Files\Microsoft\jdk-11.0.15.10-hotspot"
|
||||
# clone ONNXRuntime
|
||||
git clone -b rel-1.12.0 https://github.com/microsoft/onnxruntime.git onnxruntime
|
||||
|
||||
# clone onnxruntime-extensions
|
||||
git clone https://github.com/microsoft/onnxruntime-extensions.git onnxruntime_extensions
|
||||
|
||||
# build JAR package in this folder
|
||||
mkdir ortall.build
|
||||
cd ortall.build
|
||||
python ..\onnxruntime\tools\ci_build\build.py --config Release --cmake_generator "Visual Studio 17 2022" --build_java --build_dir . --use_extensions --extensions_overridden_path "..\onnxruntime-extensions"
|
||||
```
|
||||
|
||||
|
||||
## Dependencies
|
||||
|
||||
The matrix below lists the versions of individual dependencies of onnxruntime-extensions. These are the configurations that are routinely and extensively verified by our CI.
|
||||
|
||||
Python | 3.8 | 3.9 | 3.10 | 3.11 |
|
||||
---|---|---|---|---
|
||||
Onnxruntime |1.12.1 (Aug 4, 2022) |1.13.1(Oct 24, 2022) |1.14.1 (Mar 2, 2023) |1.15.0 (May 24, 2023) |
|
||||
135
docs/extensions/index.md
Normal file
135
docs/extensions/index.md
Normal file
|
|
@ -0,0 +1,135 @@
|
|||
---
|
||||
title: Extensions
|
||||
has_children: true
|
||||
nav_order: 7
|
||||
---
|
||||
|
||||
# ONNXRuntime-Extensions
|
||||
|
||||
[](https://dev.azure.com/onnxruntime/onnxruntime/_build/latest?definitionId=209&branchName=main)
|
||||
|
||||
## What is ONNXRuntime-Extensions?
|
||||
|
||||
ONNXRuntime-Extensions is a library that extends the capability of the ONNX models and inference with ONNX Runtime, via the ONNX Runtime custom operator interface. It includes a set of Custom Operators to support common model pre and post-processing for audio, vision, text, and language models. As with ONNX Runtime, Extensions also supports multiple languages and platforms (Python on Windows/Linux/macOS, Android and iOS mobile platforms and Web-Assembly for web.
|
||||
|
||||
The basic workflow is to add the custom operators to an ONNX model and then to perform inference on the enhanced model with ONNX Runtime and ONNXRuntime-Extensions packages.
|
||||
|
||||
|
||||
<img src="../../images/combine-ai-extensions-img.png" alt="Pre and post-processing custom operators for vision, text, and NLP models" width="100%"/>
|
||||
<sub>This image was created using <a href="https://github.com/sayanshaw24/combine" target="_blank">Combine.AI</a>, which is powered by Bing Chat, Bing Image Creator, and EdgeGPT.</sub>
|
||||
|
||||
## Quickstart
|
||||
|
||||
### **Python installation**
|
||||
```bash
|
||||
pip install onnxruntime-extensions
|
||||
```
|
||||
|
||||
#### **Nightly Build**
|
||||
|
||||
##### on Windows
|
||||
```cmd
|
||||
pip install --index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT-Nightly/pypi/simple/ onnxruntime-extensions
|
||||
```
|
||||
Please ensure that you have met the prerequisites of onnxruntime-extensions (e.g., onnx and onnxruntime) in your Python environment.
|
||||
#### <strong>on Linux/macOS</strong>
|
||||
Please make sure the compiler toolkit like gcc(later than g++ 8.0) or clang are installed before the following command
|
||||
```bash
|
||||
python -m pip install git+https://github.com/microsoft/onnxruntime-extensions.git
|
||||
```
|
||||
|
||||
### **NuGet installation (with .NET CLI)**
|
||||
```bash
|
||||
dotnet add package Microsoft.ML.OnnxRuntime.Extensions --version 0.8.1-alpha
|
||||
```
|
||||
|
||||
## Add pre and post-processing to the model
|
||||
There are multiple ways to get the ONNX processing graph:
|
||||
- [Use the pre-processing pipeline API if the model and its pre-processing is supported by the pipeline API](https://github.com/microsoft/onnxruntime-extensions/blob/main/onnxruntime_extensions/tools/pre_post_processing/pre_post_processor.py)
|
||||
- [Export to ONNX from a PyTorch model](https://github.com/microsoft/onnxruntime-extensions/blob/main/tutorials/superresolution_e2e.py#L69)
|
||||
- [Create an ONNX model with a model graph that includes your custom op node](https://github.com/microsoft/onnxruntime-extensions/blob/main/onnxruntime_extensions/_ortapi2.py#L50)
|
||||
- [Compose the pre-processing with an ONNX model using ONNX APIs if you already have the pre processing in an ONNX graph](https://onnx.ai/onnx/api/compose.html)
|
||||
|
||||
If the pre processing operator is a HuggingFace tokenizer, you can also easily get the ONNX processing graph by converting from Huggingface transformer data processing classes such as in the following example:
|
||||
```python
|
||||
import onnxruntime as _ort
|
||||
from transformers import AutoTokenizer
|
||||
from onnxruntime_extensions import OrtPyFunction, gen_processing_models
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained('distilbert-base-uncased')
|
||||
model = OrtPyFunction(gen_processing_models(tokenizer, pre_kwargs={})[0])
|
||||
```
|
||||
|
||||
For more information, you can check the API using the following:
|
||||
```python
|
||||
help(onnxruntime_extensions.gen_processing_models)
|
||||
```
|
||||
|
||||
### What if I cannot find the custom operator I am looking for?
|
||||
Find the custom operators we currently support [here](https://github.com/microsoft/onnxruntime-extensions/tree/main/operators). If you do not find the custom operator you are looking for, you can add a new custom operator to ONNX Runtime Extensions like [this](./add-op.md). Note that if you do add a new operator, you will have to [build from source](./build.md).
|
||||
|
||||
## Inference with ONNX Runtime and Extensions
|
||||
|
||||
### Python
|
||||
There are individual packages for the following languages, please install it for the build.
|
||||
```python
|
||||
import onnxruntime as _ort
|
||||
from onnxruntime_extensions import get_library_path as _lib_path
|
||||
|
||||
so = _ort.SessionOptions()
|
||||
so.register_custom_ops_library(_lib_path())
|
||||
|
||||
# Run the ONNXRuntime Session as per ONNXRuntime docs suggestions.
|
||||
sess = _ort.InferenceSession(model, so)
|
||||
sess.run (...)
|
||||
```
|
||||
### C++
|
||||
|
||||
```c++
|
||||
// The line loads the customop library into ONNXRuntime engine to load the ONNX model with the custom op
|
||||
Ort::ThrowOnError(Ort::GetApi().RegisterCustomOpsLibrary((OrtSessionOptions*)session_options, custom_op_library_filename, &handle));
|
||||
|
||||
// The regular ONNXRuntime invoking to run the model.
|
||||
Ort::Session session(env, model_uri, session_options);
|
||||
RunSession(session, inputs, outputs);
|
||||
```
|
||||
### Java
|
||||
```java
|
||||
var env = OrtEnvironment.getEnvironment();
|
||||
var sess_opt = new OrtSession.SessionOptions();
|
||||
|
||||
/* Register the custom ops from onnxruntime-extensions */
|
||||
sess_opt.registerCustomOpLibrary(OrtxPackage.getLibraryPath());
|
||||
```
|
||||
|
||||
### C#
|
||||
```java
|
||||
SessionOptions options = new SessionOptions();
|
||||
options.RegisterOrtExtensions();
|
||||
session = new InferenceSession(model, options);
|
||||
```
|
||||
|
||||
## Tutorials
|
||||
|
||||
Check out some end to end tutorials with our custom operators:
|
||||
- NLP: [An end-to-end BERT tutorial](https://github.com/microsoft/onnxruntime-extensions/blob/main/tutorials/bert_e2e.py)
|
||||
- Audio: [Using audio encoding and decoding for Whisper](https://github.com/microsoft/onnxruntime-extensions/blob/main/tutorials/whisper_e2e.py)
|
||||
- Vision: [The YOLO model with our DrawBoundingBoxes operator](https://github.com/microsoft/onnxruntime-extensions/blob/main/tutorials/yolo_e2e.py)
|
||||
|
||||
## Contributing
|
||||
|
||||
This project welcomes contributions and suggestions. Most contributions require you to agree to a
|
||||
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
|
||||
the rights to use your contribution. For details, visit https://cla.microsoft.com.
|
||||
|
||||
When you submit a pull request, a CLA-bot will automatically determine whether you need to provide
|
||||
a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions
|
||||
provided by the bot. You will only need to do this once across all repos using our CLA.
|
||||
|
||||
This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
|
||||
For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or
|
||||
contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.
|
||||
|
||||
## License
|
||||
|
||||
[MIT License](https://github.com/microsoft/onnxruntime-extensions/blob/main/LICENSE)
|
||||
|
|
@ -1,7 +1,7 @@
|
|||
---
|
||||
title: Performance
|
||||
has_children: true
|
||||
nav_order: 7
|
||||
nav_order: 8
|
||||
---
|
||||
|
||||
# ONNX Runtime Performance
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
---
|
||||
title: Reference
|
||||
has_children: true
|
||||
nav_order: 9
|
||||
nav_order: 10
|
||||
redirect_from: /docs/resources
|
||||
---
|
||||
|
||||
|
|
|
|||
BIN
images/combine-ai-extensions-img.png
Normal file
BIN
images/combine-ai-extensions-img.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 1.1 MiB |
Loading…
Reference in a new issue