saymrwulf/onnxruntime
1
0
Fork 0
mirror of https://github.com/saymrwulf/onnxruntime.git synced 2026-05-20 21:40:57 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
onnxruntime/js/README.md
Edward Chen 454f77cd94
Update kernel matching logic: decouple from op schemas and remove kernel def hashes (#12791) 
# Motivation
Currently, ORT minimal builds use kernel def hashes to map from nodes to
kernels to execute when loading the model. As the kernel def hashes must
be known ahead of time, this works for statically registered kernels.
This works well for the CPU EP.
For this approach to work, the kernel def hashes must also be known at
ORT format model conversion time, which means the EP with statically
registered kernels must also be enabled then. This is not an issue for
the always-available CPU EP. However, we do not want to require that any
EP which statically registers kernels is always available too.
Consequently, we explore another approach to match nodes to kernels that
does not rely on kernel def hashes. An added benefit of this is the
possibility of moving away from kernel def hashes completely, which
would eliminate the maintenance burden of keeping the hashes stable.

# Approach
In a full build, ORT uses some information from the ONNX op schema to
match a node to a kernel. We want to avoid including the ONNX op schema
in a minimal build to reduce binary size. Essentially, we take the
necessary information from the ONNX op schema and make it available in a
minimal build.
We decouple the ONNX op schema from the kernel matching logic. The
kernel matching logic instead relies on per-op information which can
either be obtained from the ONNX op schema or another source.
This per-op information must be available in a minimal build when there
are no ONNX op schemas. We put it in the ORT format model.
Existing uses of kernel def hashes to look up kernels are replaced
with the updated kernel matching logic. We no longer store
kernel def hashes in the ORT format model’s session state and runtime
optimization representations. We no longer keep the logic to
generate and ensure stability of kernel def hashes.
2022-09-20 14:24:59 -07:00

446 lines
18 KiB
Markdown
Raw Blame History

# ONNX Runtime JavaScript API
This directory contains multiple NPM projects:
- [onnxruntime-common](#onnxruntime-common)
- [onnxruntime-node](#onnxruntime-node)
- [onnxruntime-web](#onnxruntime-web)
- [onnxruntime-react-native](#onnxruntime-react-native)
## Development
This folder contains a `.vscode` folder for Visual Studio Code workspace configs. Using VSCode to open this folder
will allow code-formatting and linting features on typescript and C/C++ source code inside this folder. Following files
are used for code-formatting and linting features for developers:
- .vscode/\*\*
- package.json
- packages-lock.json
- .eslintrc.js
- .clang-format
Please follow the steps described below to setup development environment.
### Prerequisites
- Node.js (16.0+): https://nodejs.org/ - (Optional) Use nvm ([Windows](https://github.com/coreybutler/nvm-windows) / [Mac/Linux](https://github.com/creationix/nvm)) to install Node.js
- Python (2.7 or 3.6+): https://www.python.org/downloads/
- python should be added to the PATH environment variable
- Visual Studio Code: https://code.visualstudio.com/
- **required** extension: [ESLint](https://marketplace.visualstudio.com/items?itemName=dbaeumer.vscode-eslint)
- **required** extension: [Clang-Format](https://marketplace.visualstudio.com/items?itemName=xaver.clang-format)
- **required** extension: [Debugger for Chrome](https://marketplace.visualstudio.com/items?itemName=msjsdiag.debugger-for-chrome)
- Chrome or Edge Browser
### Setup TypeScript development environment
In `<ORT_ROOT>/js`, run:
```
npm ci
```
This will install Clang-format and ESLint for code-formatting and linting features. This is a one-time setup unless a `git clean` is performed or folder `<ORT_ROOT>/js/node_modules` is removed manually.
### Using VSCode:
Use VSCode to open folder `<ORT_ROOT>/js`.
Make sure to open the correct folder to allow VSCode to load workspace configuration. Otherwise typescript and code formatter may not work as expected.
To populate typescript type declarations, in each project folder, run `npm ci`.
### Run code formatter and linter manually
In `<ORT_ROOT>/js`, use `npm run lint` to run ESLint , and use `npm run format` to run clang-format.
## onnxruntime-common
> language: typescript
> dependency:
> folder: <ORT_ROOT>/js/common
This project is designed to include all "common" code, which are pure javascript that can run in both Node.js and browsers.
### Requirements
Node.js v12+ (recommended v14+)
### Build
Use following command in folder `<ORT_ROOT>/js/common` to install NPM packages, build typescript files and generate bundles:
```
npm ci
```
### Distribution
It should be able to consumed by both from projects that uses NPM packages (through a Node.js folder structure of `node_modules` folder that generated by `npm install onnxruntime-common`) and from a CDN service that serves a `.min.js` bundle file.
### Features
Following features are included in `onnxruntime-common`:
- `InferenceSession` interfaces
- `Tensor`/`OnnxValue` interfaces, implementation and a set of utility functions
- `Backend` interfaces and a set of functions for backend registration
### Generate API reference document
Use following command in folder `<ORT_ROOT>/js/common` to generate API reference document:
```
npx typedoc
```
Document will be generated in folder `<ORT_ROOT>/js/common/docs`.
## onnxruntime-node
> language: typescript/C++
> dependency: onnxruntime-common, ONNXRuntime.dll
> folder: <ORT_ROOT>/js/node
This project is designed to be used as a NPM package to enable Node.js users to consume ONNX Runtime via Node.js binding, in Node.js or any Node.js compatible environment.
### Requirements
Node.js v12+ (recommended v14+)
### Build
#### Build ONNX Runtime and Node.js binding
Follow [instructions](https://www.onnxruntime.ai/docs/how-to/build.html#apis-and-language-bindings) for building ONNX Runtime Node.js binding
#### Build Node.js binding only
Use following command in folder `<ORT_ROOT>/js/node` to install NPM packages and build typescript files:
```
npm ci
```
This will download the latest pre-built ONNX Runtime binaries for the current platform.
### Distribution
It should be able to consumed by from projects that uses NPM packages (through a Node.js folder structure of `node_modules` folder that generated by `npm install onnxruntime-node`).
## onnxruntime-web
> language: typescript
> dependency: onnxruntime-common, ONNXRuntime WebAssembly
> folder: <ORT_ROOT>/js/web
This project is a library for running ONNX models on browsers. It is the successor of [ONNX.js](https://github.com/Microsoft/onnxjs).
### Build
1. Install NPM packages
1. in `<ORT_ROOT>/js/`, run `npm ci`.
2. in `<ORT_ROOT>/js/common/`, run `npm ci`.
3. in `<ORT_ROOT>/js/web/`, run `npm ci`.
2. Prepare ONNX Runtime WebAssembly artifacts.
You can either use the prebuilt artifacts or build it by yourself.
- Setup by script.
In `<ORT_ROOT>/js/web/`, run `npm run pull:wasm` to pull WebAssembly artifacts for latest main branch from CI pipeline.
- Download artifacts from pipeline manually.
you can download prebuilt WebAssembly artifacts from [Windows WebAssembly CI Pipeline](https://dev.azure.com/onnxruntime/onnxruntime/_build?definitionId=161&_a=summary). Select a build, download artifacts "Release_ort-wasm" and "Release_ort-wasm-threaded" and unzip. See instructions below to put files into destination folders.
- Build WebAssembly artifacts.
1. Build ONNX Runtime WebAssembly
~~Follow [instructions](https://www.onnxruntime.ai/docs/how-to/build.html#apis-and-language-bindings) for building ONNX Runtime WebAssembly. (TODO: document is not ready. we are working on it. Please see steps described as below.)~~
in `<ORT_ROOT>/`, run one of the following commands to build WebAssembly:
```sh
# In windows, use 'build' to replace './build.sh'
# The following command build debug.
./build.sh --build_wasm
# The following command build debug with debug info.
./build.sh --build_wasm --skip_tests --enable_wasm_debug_info
# The following command build release.
./build.sh --config Release --build_wasm --skip_tests --disable_wasm_exception_catching --disable_rtti
```
To build with multi-thread support, append flag `--enable_wasm_threads` to the command. To build with SIMD support, append flag `--enable_wasm_simd` to the command. Make sure to build both single-thread and multi-thread with and without SIMD before next step.
2. Copy following files from build output folder to `<ORT_ROOT>/js/web/dist/`:
- ort-wasm.wasm
- ort-wasm-threaded.wasm (build with flag '--enable_wasm_threads')
- ort-wasm-simd.wasm (build with flag '--enable_wasm_simd')
- ort-wasm-simd-threaded.wasm (build with flags '--enable_wasm_threads --enable_wasm_simd')
3. Copy following files from build output folder to `<ORT_ROOT>/js/web/lib/wasm/binding/`:
- ort-wasm.js
- ort-wasm-threaded.js (build with flag '--enable_wasm_threads')
- ort-wasm-threaded.worker.js (build with flag '--enable_wasm_threads')
3. Use following command in folder `<ORT_ROOT>/js/web` to build:
```
npm run build
```
### Test
We use command `npm test` (test runner) and `npm run test:e2e` (E2E test) for tests in ONNXRuntime Web.
#### test runner
In folder `<ORT_ROOT>/js/web`,
- Run `npm test -- --help` for a full CLI instruction.
- Run `npm test -- <your-args> --debug` to run one or more test cases.
There are multiple levels of tests for ONNXRuntime Web:
- unit test: tests for individual components written in TypeScript. Launch unit test by:
```
npm test -- unittest
```
- model test: run a single model. The model folder should contains one .onnx model file and one or more folders for test cases, each folder contains several input*\*.pb and output*\*.pb as test data. Launch model test by:
```
npm test -- model <model_folder>
```
- op test: test a single operator. An op test is described in a `.jsonc` file which specify the operator type, its attributes and one or more test case(s), each includes a list of expected input tensor(s) and output tensor(s). The `.jsonc` file is located at `<ORT_ROOT>/js/web/test/data/ops`. Launch op test by:
```
npm test -- op <file_name>
```
- suite test: suite test includes unit test, a list of model tests and op tests. Launch suite test by:
```
npm test
```
#### E2E test
E2E test is for testing end-to-end package consuming. In this test, NPM packages for `onnxruntime-common` and `onnxruntime-web` are generated and a clean folder is used for installing packages. Then a simple mocha test is performed to make sure package can be consumed correctly.
To launch E2E test:
```
npm run test:e2e
```
### Debugging
#### Debugging TypeScript on Desktop/Chrome
To debug the code from test-runner on Chrome:
- Launch `npm test -- <your_args> --debug`. It opens an instance of Chrome browser.
- In the open Chrome browser, click the `DEBUG` button on the top-right of the page.
- In VSCode, click [side bar]->Run and Debug->select [Attach to Chrome]->click [Start Debugging] to attach.
- put breakpoints in source code, and Refresh the page to reload.
#### Debugging TypeScript on iOS/Safari
To debug on an Apple iOS device, please refer to the following steps:
- install [
RemoteDebug iOS WebKit Adapter](https://github.com/RemoteDebug/remotedebug-ios-webkit-adapter) by following its instructions.
- launch the adapter in commandline: `remotedebug_ios_webkit_adapter --port=9000`.
- in VSCode, select debug configuration `Remote Browser via Webkit Adaptor`.
- follow the steps above to debug.
#### Debugging TypeScript on Android/Chrome
To debug on an Android device, please refer to the following steps:
- Install [Android SDK Platform Tools](https://developer.android.com/studio/releases/platform-tools) and make sure `adb` is ready to use.
- Follow instructions in [Remote Debugging on Android](https://developer.chrome.com/devtools/docs/remote-debugging-legacy) to launch `adb`. Make sure to use port 9000 so that the existing debug configuration works.
- in VSCode, select debug configuration `Remote Browser via Webkit Adaptor`.
- follow the steps above to debug.
#### Debugging C/C++ for ONNX Runtime WebAssembly
To debug C/C++ code for ONNX Runtime WebAssembly, you need to build ONNX Runtime with debug info (see [Build](#Build-2)).
Currently debugging C/C++ code in WebAssembly is not supported in VSCode yet. Please follow [this instruction](https://developer.chrome.com/blog/wasm-debugging-2020/) to debug in browser devtool using extension [C/C++ DevTools Support (DWARF)](https://chrome.google.com/webstore/detail/cc%20%20-devtools-support-dwa/pdcpmagijalfljmkmjngeonclgbbannb).
### Generating Document
This section describes how to generate the latest document for ONNX Runtime Web.
The document contains information about operators WebGL backend supports. It should align with the operator resolve rules in code and spec definition from ONNX.
In folder `<ORT_ROOT>/js/web`, use command `npm run build:doc` to generate the latest documents.
### Distribution
It should be able to consumed by both from projects that uses NPM packages (through a Node.js folder structure of `node_modules` folder that generated by `npm install onnxruntime-web`) and from a CDN service that serves a `ort.min.js` file and one or multiple `.wasm` file(s).
#### Reduced WebAssembly artifacts
By default, the WebAssembly artifacts from onnxruntime-web package allows use of both standard ONNX models (.onnx) and ORT format models (.ort). There is an option to use a minimal build of ONNX Runtime to reduce the binary size, which only supports ORT format models. See also [ORT format model](https://onnxruntime.ai/docs/tutorials/mobile/overview.html) for more information.
#### Reduced JavaScript bundle file fize
By default, the main bundle file `ort.min.js` of ONNX Runtime Web contains all features. However, its size is over 500kB and for some scenarios we want a smaller sized bundle file, if we don't use all the features. The following table lists all available bundles with their support status of features.
|bundle file name|file size|file size (gzipped)|WebGL|WASM-core|WASM-proxy|WASM-threads|ES5 backward compatibility|
|-|-|-|-|------|-----|---|-|
|ort.es5.min.js|594.15KB|134.25KB|O|O|O|O|O|
|ort.min.js|526.02KB|125.07KB|O|O|O|O|X|
|ort.webgl.min.js|385.25KB|83.83KB|O|X|X|X|X|
|ort.wasm.min.js|148.56|44KB|X|O|O|O|X|
|ort.wasm-core.min.js|40.56KB|12.74KB|X|O|X|X|X|
#### Build ONNX Runtime as a WebAssembly static library
When `--build_wasm_static_lib` is given instead of `--build_wasm`, it builds a WebAssembly static library of ONNX Runtime and creates a `libonnxruntime_webassembly.a` file at a build output directory. Developers who have their own C/C++ project and build it as WebAssembly with ONNX Runtime, this build option would be useful. This static library is not published by a pipeline, so a manual build is required if necessary.
## onnxruntime-react-native
> language: typescript, java, objective-c
> dependency: onnxruntime-common
> folder: <ORT_ROOT>/js/react_native
This project provides an ONNX Runtime React Native JavaScript library to run ONNX models on React Native Android and iOS app.
### Requirements
- Yarn
- Android SDK and NDK, which can be installed via Android Studio or sdkmanager command line tool
- A Mac computer with the latest macOS
- [Xcode](https://developer.apple.com/xcode/)
- [CMake](https://cmake.org/download/)
- [Python 3](https://www.python.org/downloads/mac-osx/)
### Models with ORT format
By default, ONNX Runtime React Native leverages ONNX Runtime Mobile package with ORT format. Follow the [instruciton](https://onnxruntime.ai/docs/tutorials/mobile/model-conversion.html) to covert ONNX model to ORT format.
### Build
1. Install NPM packages for ONNX Runtime common JavaScript library and required React Native JavaScript libraries
- in `<ORT_ROOT>/js/`, run `npm ci`.
- in `<ORT_ROOT>/js/common/`, run `npm ci`.
- in `<ORT_ROOT>/js/react_native/`, run `yarn`.
2. Build Android ONNX Runtime package
1. To use a published Android ONNX Runtime Mobile package from Maven, go to step 5.
2. Set up an Android build environment referring to [instruction](https://www.onnxruntime.ai/docs/how-to/build/android-ios.html#android)
3. In `<ORT_ROOT>`, run this python script to build ONNX Runtime Android archive file. In windows, this requires an admin account to build. To build a model specific package with reduced size, refer to [instruction](https://www.onnxruntime.ai/docs/how-to/build/reduced.html#build-ort-with-reduced-size).
```sh
python tools/ci_build/github/android/build_aar_package.py tools/ci_build/github/android/default_mobile_aar_build_settings.json --config MinSizeRel --android_sdk_path <ANDROID_SDK_PATH> --android_ndk_path <ANDROID_NDK_PATH> --build_dir <BUILD_DIRECTORY> --include_ops_by_config tools/ci_build/github/android/mobile_package.required_operators.config
```
4. Copy `<BUILD_DIRECTORY>/aar_out/MinSizeRel/com/microsoft/onnxruntime/onnxruntime-mobile/<version>/onnxruntime-mobile-<version>.aar` into `<ORT_ROOT>/js/react_native/android/libs` directory.
5. To verify, open Android Emulator and run this command from `<ORT_ROOT>/js/react_native/android`
```sh
./gradlew connectedDebugAndroidTest
```
3. Build iOS ONNX Runtime package
1. To use a published c/c++ ONNX Runtime Mobile package from CocoaPods, skip all steps below.
2. Set up iOS build environment referring to [instruction](https://www.onnxruntime.ai/docs/how-to/build/android-ios.html#ios).
3. Build a fat ONNX Runtime Mobile Framework for iOS and iOS simulator from `<ORT_ROOT>` using this command,
```sh
python tools/ci_build/github/apple/build_ios_framework.py tools/ci_build/github/apple/default_mobile_ios_framework_build_settings.json --config MinSizeRel --include_ops_by_config tools/ci_build/github/android/mobile_package.required_operators.config
```
It creates `Headers`, `LICENSE`, and `onnxruntime.xcframework` in `build/iOS_framework/framework_out` directory. From `framework_out` directory, create an archive file named `onnxruntime-mobile-c.zip` as follows and copy to `<ORT_ROOT>/js/react_native/local_pods` directory.
```sh
zip -r onnxruntime-mobile-c.zip .
```
4. To verify, open iOS Simulator and run this command from `<ORT_ROOT>/js/react_native/ios`. Change a destination to specify a running iOS Simulator.
```sh
pod install
xcodebuild test -workspace OnnxruntimeModule.xcworkspace -scheme OnnxruntimeModuleTest -destination 'platform=iOS Simulator,OS=latest,name=iPhone 13'
```
4. Test Android and iOS apps. In Windows, open Android Emulator first.
`debug.keystore` must be generated ahead for Android example.
```sh
keytool -genkey -v -keystore <ORT_ROOT>/js/react_native/e2e/android/debug.keystore -alias androiddebugkey -storepass android -keypass android -keyalg RSA -keysize 2048 -validity 999999 -dname "CN=Android Debug,O=Android,C=US"
```
From `<ORT_ROOT>/js/react_native,
```sh
yarn bootstrap
```
When testing with a custom built ONNX Runtime Android package, copy `<BUILD_DIRECTORY>/aar_out/MinSizeRel/com/microsoft/onnxruntime/onnxruntime-mobile/<version>/onnxruntime-mobile-<version>.aar` into `<ORT_ROOT>/js/react_native/e2e/android/app/libs` directory. Using a custom built ONNX Runtime iOS package, copy `onnxruntime-mobile-c.zip` into `<ORT_ROOT>/js/react_native/local_pods` directory if it's not already done.
From `<ORT_ROOT>/js/react_native/e2e/android`, run e2e Android tests as follows,
```sh
./gradlew :app:connectedDebugAndroidTest
```
From `<ORT_ROOT>/js/react_native/e2e/ios`, run e2e iOS tests as follows,
```sh
xcrun xcodebuild test -workspace OnnxruntimeModuleExample.xcworkspace -scheme OnnxruntimeModuleExample -destination 'platform=iOS Simulator,OS=latest,name=iPhone 13'
```
***`yarn bootstrap` changes `packages.json` and `yarn.lock` files. Once testing is done, restore changes to avoid unwanted commit.***
5. Run Android and iOS apps.
```sh
yarn e2e android
yarn e2e ios
```
### NPM Packaging
1. Update a version using `npm verison <version>` from `<ORT_ROOT>/js/react_native` folder. If it's for a dev, use `npm version <version>-dev.<subversion>`
2. Run `npm pack` and verify NPM package contents
3. Run `npm publish <tgz> --dry-run` to see how it's going to be published
4. Run `npm publish <tgz>` to publish to npmjs. If it's for a dev, add flag `--tag dev`.
### Distribution
It should be able to consumed by React Native projects that uses Yarn packages through `yarn add onnxruntime-react-native`.
Reference in a new issue View git blame Copy permalink