mirror of
https://github.com/saymrwulf/onnxruntime.git
synced 2026-07-17 18:40:28 +00:00
[doc] update Web tutorial (#19849)
### Description This PR re-arranges "tutorial" documentation of ORT web on onnxruntime.ai Current doc: https://onnxruntime.ai/docs/ Change preview: https://fs-eire.github.io/onnxruntime/docs/ ### Changes - Added page: - [`Using WebGPU`](https://fs-eire.github.io/onnxruntime/docs/tutorials/web/ep-webgpu.html) - [`Working with Large Models`](https://fs-eire.github.io/onnxruntime/docs/tutorials/web/large-models.html) - [`Performance Diagnosis`](https://fs-eire.github.io/onnxruntime/docs/tutorials/web/performance-diagnosis.html) - [`Deploying ONNX Runtime Web`](https://fs-eire.github.io/onnxruntime/docs/tutorials/web/deploy.html) - [`Troubleshooting`](https://fs-eire.github.io/onnxruntime/docs/tutorials/web/trouble-shooting.html) (a placeholder for now)
This commit is contained in:
Yulong Wang
GitHub
parent
f3efd8d265
commit
73173cc2c2
9 changed files with 914 additions and 3 deletions
|
|
@ -3,7 +3,7 @@ title: Build a web app with ONNX Runtime
|
|||
description: Considerations and options for building a web application with ONNX Runtime
|
||||
parent: Web
|
||||
grand_parent: Tutorials
|
||||
nav_order: 3
|
||||
nav_order: 1
|
||||
redirect_from: /reference/build-web-app
|
||||
|
||||
---
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ description: Classify images in a NextJS web application built from a GitHub tem
|
|||
parent: Web
|
||||
grand_parent: Tutorials
|
||||
has_children: false
|
||||
nav_order: 1
|
||||
nav_order: 8
|
||||
---
|
||||
|
||||
|
||||
|
|
|
|||
129
docs/tutorials/web/deploy.md
Normal file
129
docs/tutorials/web/deploy.md
Normal file
|
|
@ -0,0 +1,129 @@
|
|||
---
|
||||
title: Deploying ONNX Runtime Web
|
||||
description: Deploying ONNX Runtime Web
|
||||
parent: Web
|
||||
grand_parent: Tutorials
|
||||
has_children: false
|
||||
nav_order: 6
|
||||
---
|
||||
|
||||
{::options toc_levels="2..4" /}
|
||||
|
||||
# Deploying ONNX Runtime Web
|
||||
{: .no_toc }
|
||||
|
||||
This document provides some guidance on how to deploy ONNX Runtime Web in a production environment.
|
||||
|
||||
## Contents
|
||||
{: .no_toc}
|
||||
|
||||
* TOC
|
||||
{:toc}
|
||||
|
||||
## Assets
|
||||
|
||||
When deploying ONNX Runtime Web in a production environment, the following assets are required:
|
||||
|
||||
- **JavaScript code bundle**: The JavaScript code bundle that contains the application code and maybe the ONNX Runtime Web JavaScript code as well, depending on the how the application is built.
|
||||
|
||||
- **WebAssembly binaries**: The WebAssembly binary file(s) of ONNX Runtime Web library.
|
||||
|
||||
- **Model file(s)**: The ONNX model file(s) that you want to run in the browser.
|
||||
|
||||
### JavaScript code bundle
|
||||
|
||||
The JavaScript code bundle is usually a minified JavaScript file that contains the application code, generated by a bundler such as Webpack, Rollup or ESBuild. Depending on the bundler's configuration, the ONNX Runtime Web JavaScript code may be included in the bundle or not (if specified as an external dependency).
|
||||
|
||||
#### Conditional Importing
|
||||
|
||||
To reduce the size of the JavaScript code bundle, you can use [Conditional Importing](https://github.com/microsoft/onnxruntime-inference-examples/tree/main/js/importing_onnxruntime-web#conditional-importing) to import only the necessary parts of ONNX Runtime Web library. For example, you can import `onnxruntime-web/wasm` if you only uses the WebAssembly execution provider, which can reduce the size of the JavaScript code bundle.
|
||||
|
||||
#### Inlined worker
|
||||
|
||||
The ONNX Runtime Web JavaScript code include 3 inlined source code:
|
||||
1. the web worker for proxy feature
|
||||
2. the web worker for WebAssembly multi-threading feature
|
||||
3. the WebAssembly entry generated by `function.toString()` required by (2) for multi-threading feature
|
||||
|
||||
The use of inlined worker helps to keep ONNX Runtime Web to a single JavaScript file, which is easier to deploy and use. However, it may not work in some environments, such as Content Security Policy (CSP) restricted environments. See [Security considerations](#security-considerations) for more details.
|
||||
|
||||
### WebAssembly binaries
|
||||
|
||||
The standard ONNX Runtime Web library includes the following WebAssembly binary files:
|
||||
|
||||
|
||||
| File | SIMD | Multi-threading | JSEP | Training |
|
||||
|-----------|-------------|--|---|---|
|
||||
| `ort-wasm.wasm` | ❌ | ❌ | ❌ | ❌ |
|
||||
| `ort-wasm-simd.wasm` | ✔️ | ❌ | ❌ | ❌ |
|
||||
| `ort-wasm-threaded.wasm` | ❌ | ✔️ | ❌ | ❌ |
|
||||
| `ort-wasm-simd-threaded.wasm` | ✔️ | ✔️ | ❌ | ❌ |
|
||||
| `ort-wasm-simd.jsep.wasm` | ✔️ | ❌ | ✔️ | ❌ |
|
||||
| `ort-wasm-simd-threaded.jsep.wasm` | ✔️ | ✔️ | ✔️ | ❌ |
|
||||
| `ort-training-wasm-simd.wasm` | ✔️ | ❌ | ❌ | ✔️ |
|
||||
|
||||
|
||||
The columns indicate whether the feature is supported by the WebAssembly artifact.
|
||||
|
||||
- SIMD: whether the Single Instruction, Multiple Data (SIMD) feature is supported.
|
||||
- Multi-threading: whether the WebAssembly multi-threading feature is supported.
|
||||
- JSEP: whether the JavaScript Execution Provider (JSEP) feature is enabled. This feature powers the WebGPU and WebNN execution providers.
|
||||
- Training: whether the training feature is enabled.
|
||||
|
||||
When deploying ONNX Runtime Web in a production environment, you should consider which WebAssembly binary file(s) to include in the application. By default, ONNX Runtime Web JavaScript code will check the environment and load the appropriate WebAssembly binary file(s) automatically. This means you should include all combinations of WebAssembly binary file(s) in the deployment for the best compatibility.
|
||||
|
||||
However, when your application code imports ONNX Runtime Web with WebGPU or WebNN support, you can just include the 2 WebAssembly binary file(s) for JSEP. Furthermore, if you set the `ort.env.wasm.numThreads` to 1, you can just include file `ort-wasm-simd.jsep.wasm` in your deploy.
|
||||
|
||||
#### Ensure the WebAssembly binary file(s) are correctly served
|
||||
|
||||
You should ensure that the WebAssembly binary file(s) are correctly served on the server. If you didn't copy the necessary WebAssembly binary file(s) when building the application, or if the WebAssembly binary file(s) are not in the expected path, ONNX Runtime Web will fail to initialize.
|
||||
|
||||
#### Override WebAssembly file path
|
||||
|
||||
ONNX Runtime Web tries to locate the WebAssembly binary file(s) by using the relative path of the JavaScript code bundle. If the WebAssembly binary file(s) are not located in the same directory as the JavaScript code bundle, you can override the file path by setting the value of `ort.env.wasm.wasmPaths`.
|
||||
|
||||
You can also set the `ort.env.wasm.wasmPaths` to an absolute URL to a public CDN, like jsdelivr or unpkg, if you are using a release version of ONNX Runtime Web:
|
||||
|
||||
```js
|
||||
// Set the WebAssembly binary file path to jsdelivr CDN for latest dev version
|
||||
ort.env.wasm.wasmPaths = 'https://cdn.jsdelivr.net/npm/onnxruntime-web@dev/dist/';
|
||||
|
||||
// Set the WebAssembly binary file path to unpkg CDN for latest dev version
|
||||
ort.env.wasm.wasmPaths = 'https://unpkg.com/onnxruntime-web@dev/dist/';
|
||||
```
|
||||
|
||||
See [API reference: env.wasm.wasmPaths](https://onnxruntime.ai/docs/api/js/interfaces/Env.WebAssemblyFlags.html#wasmPaths) for more details.
|
||||
|
||||
### Model file(s)
|
||||
|
||||
If your ONNX model file(s) are large and they need some time to download, you can consider to use IndexedDB to cache the model file(s) to avoid loading the model every time the page is refreshed.
|
||||
|
||||
If the model contains external data, you need to pass the external data information to ONNX Runtime Web. See [External Data](./large-models.md#external-data) for more details.
|
||||
|
||||
## File size considerations
|
||||
|
||||
The size of the artifacts is an important factor to consider when deploying ONNX Runtime Web in a production environment. Reducing the file size can improve the load time of the application and reduce the memory consumption on the client's device.
|
||||
|
||||
To reduce the deployment size, you can consider the following options:
|
||||
- Use [Conditional Importing](https://github.com/microsoft/onnxruntime-inference-examples/tree/main/js/importing_onnxruntime-web#conditional-importing) to import only the necessary parts of ONNX Runtime Web library.
|
||||
- Serve only necessary WebAssembly binaries, or use the `ort.env.wasm.wasmPaths` to set the WebAssembly binary file path to a public CDN.
|
||||
|
||||
If you want ultimate control over the size of the artifacts, you can also perform a custom build of ONNX Runtime Web.
|
||||
|
||||
### Custom build
|
||||
|
||||
By using a custom build of ONNX Runtime Web, you can build ONNX Runtime Web with only the kernels that required by your model, which can significantly reduce the size of the WebAssembly binary file(s). The steps are however more complex and require some knowledge of the ONNX Runtime Web build system.
|
||||
|
||||
The content of this part is under construction.
|
||||
|
||||
## Security considerations
|
||||
|
||||
### Secure Context
|
||||
|
||||
WebGPU is accessible only to secure contexts. In short, a page loaded using HTTPS or using HTTP from localhost/127.0.0.1 is considered secure context.
|
||||
|
||||
See [Secure Context](https://developer.mozilla.org/en-US/docs/Web/Security/Secure_Contexts) and [WebGPU: Troubleshooting tips and fixes](https://developer.chrome.com/docs/web-platform/webgpu/troubleshooting-tips) for more details.
|
||||
|
||||
### Content Security Policy (CSP) restricted environments
|
||||
|
||||
Currently, ONNX Runtime Web uses inline web workers to enable the proxy feature and WebAssembly multi-threading feature. This means in a CSP restricted environment, the features mentioned above may not work. We are working on a solution to make it work in a CSP restricted environment.
|
||||
263
docs/tutorials/web/env-flags-and-session-options.md
Normal file
263
docs/tutorials/web/env-flags-and-session-options.md
Normal file
|
|
@ -0,0 +1,263 @@
|
|||
---
|
||||
title: The 'env' Flags and Session Options
|
||||
description: The 'env' Flags and Session Options
|
||||
parent: Web
|
||||
grand_parent: Tutorials
|
||||
has_children: false
|
||||
nav_order: 2
|
||||
---
|
||||
{::options toc_levels="2..4" /}
|
||||
|
||||
# The 'env' Flags and Session Options
|
||||
{: .no_toc }
|
||||
|
||||
This document explains how to configure ONNX Runtime Web, using the following methods:
|
||||
|
||||
- [The 'env' flags](#the-environment-flags-env)
|
||||
- [Session options](#session-options)
|
||||
|
||||
The biggest difference between the two is that the 'env' flags are global settings that affect the entire ONNX Runtime Web environment, while session options are settings that are specific to a single inference session.
|
||||
|
||||
## Contents
|
||||
{: .no_toc}
|
||||
|
||||
* TOC
|
||||
{:toc}
|
||||
|
||||
|
||||
## The environment flags ('env')
|
||||
|
||||
### Summary
|
||||
|
||||
The environment flags are a set of global flags that can be used to configure the behavior of ONNX Runtime Web. They are accessible via the `ort.env` object:
|
||||
|
||||
```js
|
||||
import * as ort from 'onnxruntime-web';
|
||||
|
||||
// get the 'env' object
|
||||
const env = ort.env;
|
||||
```
|
||||
|
||||
These flags are usually required to be set before any inference session is created.
|
||||
|
||||
|
||||
For more information, see [API reference: Interface Env](https://onnxruntime.ai/docs/api/js/interfaces/Env-1.html).
|
||||
|
||||
### `env.debug`
|
||||
|
||||
The `env.debug` flag is used to enable/disable the debug mode. When enabled, ONNX Runtime Web will do extra checks and logging to help diagnose issues. It is disabled by default.
|
||||
|
||||
```js
|
||||
// enable the debug mode
|
||||
ort.env.debug = true;
|
||||
```
|
||||
|
||||
For more information, see [API reference: env.debug](https://onnxruntime.ai/docs/api/js/interfaces/Env-1.html#debug).
|
||||
|
||||
### `env.logLevel`
|
||||
|
||||
The `env.logLevel` flag is used to set the log level. It can be set to one of `"error" | "verbose" | "info" | "warning" | "fatal"`. The default value is `"warning"`.
|
||||
|
||||
```js
|
||||
// set the log level to 'verbose'
|
||||
ort.env.logLevel = 'verbose';
|
||||
```
|
||||
|
||||
For more information, see [API reference: env.logLevel](https://onnxruntime.ai/docs/api/js/interfaces/Env-1.html#logLevel).
|
||||
|
||||
### `env.wasm`
|
||||
|
||||
The `env.wasm` object contains flags that are used to configure the behavior of the WebAssembly instance.
|
||||
|
||||
For more information, see [API reference: Interface WebAssemblyFlags](https://onnxruntime.ai/docs/api/js/interfaces/Env.WebAssemblyFlags.html).
|
||||
|
||||
#### `env.wasm.numThreads`
|
||||
|
||||
The `env.wasm.numThreads` flag is used to set the number of threads that ONNX Runtime Web will use for model inference. This value includes the main thread.
|
||||
|
||||
The default value is `0`, which means it will be determined by ONNX Runtime Web based on the environment. In browsers, it will be set to half of `navigator.hardwareConcurrency` or `4`, whichever is smaller.
|
||||
|
||||
Setting it to `1` will force disable multi-threading. Otherwize, ONNX Runtime Web will perform a check for whether the environment supports multi-threading. Only when the browser supports WebAssembly multi-threading and `crossOriginIsolated` mode is enabled, multi-threading will be enabled. See [Cross Origin Isolation Guide](https://web.dev/cross-origin-isolation-guide/) for more info.
|
||||
|
||||
When multi-threading is enabled, ONNX Runtime Web will load the multi-threaded WebAssembly binary file. The corresponding file name will include `-threaded`.
|
||||
|
||||
```js
|
||||
// Disable multi-threading
|
||||
ort.env.wasm.numThreads = 1;
|
||||
```
|
||||
|
||||
For more information, see [API reference: env.wasm.numThreads](https://onnxruntime.ai/docs/api/js/interfaces/Env.WebAssemblyFlags.html#numThreads).
|
||||
|
||||
#### `env.wasm.simd`
|
||||
|
||||
The `env.wasm.simd` flag is used to enable/disable the SIMD (Single Instruction, Multiple Data) feature. It is enabled by default.
|
||||
|
||||
When SIMD is enabled, ONNX Runtime Web will perform a check for whether the environment supports SIMD. If the environment supports SIMD, ONNX Runtime Web will load the SIMD WebAssembly binary file. The corresponding file name will include `-simd`.
|
||||
|
||||
It is not recommended to set this flag to `false` unless you are sure that the environment does not support SIMD.
|
||||
|
||||
For more information, see [API reference: env.wasm.simd](https://onnxruntime.ai/docs/api/js/interfaces/Env.WebAssemblyFlags.html#simd).
|
||||
|
||||
#### `env.wasm.proxy`
|
||||
|
||||
The `env.wasm.proxy` flag is used to enable/disable the proxy worker feature. It is disabled by default.
|
||||
|
||||
When the proxy worker is enabled, ONNX Runtime Web will offload the heavy computation to a separate Web Worker. Using the proxy worker can improve the responsiveness of the UI to improve the user experience, because the computation will not block the main thread.
|
||||
|
||||
```js
|
||||
// Enable proxy worker
|
||||
ort.env.wasm.proxy = true;
|
||||
```
|
||||
|
||||
However, there are some limitations when using the proxy worker:
|
||||
- The proxy worker cannot work with WebGPU EP. This is because a GPU buffer is not transferable. If you want to use WebGPU EP in a Web Worker, you can use `importScripts()` to import the ONNX Runtime Web library in the Web Worker.
|
||||
- The proxy worker cannot work in a Content Security Policy (CSP) restricted environment. This is because the proxy worker uses `Blob` to create a Web Worker, and the CSP may block the creation of the Web Worker.
|
||||
|
||||
For more information, see [API reference: env.wasm.proxy](https://onnxruntime.ai/docs/api/js/interfaces/Env.WebAssemblyFlags.html#proxy).
|
||||
|
||||
#### `env.wasm.wasmPaths`
|
||||
|
||||
The `env.wasm.wasmPaths` flag is used to override the WebAssembly binary file path. It can be used in 2 ways:
|
||||
|
||||
- Set `env.wasm.wasmPaths` to a string as a path prefix.
|
||||
```js
|
||||
// Set the WebAssembly binary file path to jsdelivr CDN for a specific release version
|
||||
ort.env.wasm.wasmPaths = 'https://cdn.jsdelivr.net/npm/onnxruntime-web@1.17.3/dist/';
|
||||
```
|
||||
|
||||
- Set `env.wasm.wasmPaths` to an object with keys as the WebAssembly binary file name and values as the path to the WebAssembly binary file.
|
||||
```js
|
||||
// Set separate WebAssembly binary file paths
|
||||
ort.env.wasm.wasmPaths = {
|
||||
'ort-wasm-simd.jsep.wasm': 'https://example.com/path/to/ort-wasm-simd.jsep.wasm'
|
||||
'ort-wasm-simd-threaded.jsep.wasm': 'https://example.com/path/to/ort-wasm-simd-threaded.jsep.wasm',
|
||||
};
|
||||
```
|
||||
|
||||
This flag is useful when the WebAssembly binary file(s) are not located in the same directory as the JavaScript code bundle. It is also useful when you want to use a public CDN to serve the WebAssembly binary file(s).
|
||||
|
||||
NOTE: Please make sure the the JavaScript code bundle and the WebAssembly binary file(s) are from the same build. Otherwise, ONNX Runtime Web will fail to initialize due to a mismatch of the minimized function names between the JavaScript code bundle and the WebAssembly binary file(s). This means you cannot use this feature to load the WebAssembly binary file(s) from a different version.
|
||||
|
||||
For more information, see [API reference: env.wasm.wasmPaths](https://onnxruntime.ai/docs/api/js/interfaces/Env.WebAssemblyFlags.html#wasmPaths).
|
||||
|
||||
### `env.webgpu`
|
||||
|
||||
The `env.webgpu` object contains flags that are used to configure the behavior of the WebGPU EP.
|
||||
|
||||
For more information, see [API reference: Interface WebGpuFlags](https://onnxruntime.ai/docs/api/js/interfaces/Env.WebGpuFlags.html).
|
||||
|
||||
#### `env.webgpu.device` and `env.webgpu.adapter`
|
||||
|
||||
These 2 flags are used to get the WebGPU device and adapter after a WebGPU inference session is created.
|
||||
|
||||
The `env.webgpu.adapter` flag can also be used to set the adapter that will be used by the WebGPU EP before the first WebGPU inference session is created. It is useful when you want to use a specific adapter.
|
||||
|
||||
For more information, see [API reference: env.webgpu.device](https://onnxruntime.ai/docs/api/js/interfaces/Env.WebGpuFlags.html#device) and [API reference: env.webgpu.adapter](https://onnxruntime.ai/docs/api/js/interfaces/Env.WebGpuFlags.html#adapter).
|
||||
|
||||
#### `env.webgpu.powerPreference` and `env.webgpu.forceFallbackAdapter`
|
||||
|
||||
These 2 flags are used to set the power preference and force fallback adapter for the WebGPU EP. They will be used when the WebGPU EP is initialized without any pre-configured adapter is set via `env.webgpu.adapter`.
|
||||
|
||||
For more information, see [API reference: env.webgpu.powerPreference](https://onnxruntime.ai/docs/api/js/interfaces/Env.WebGpuFlags.html#powerPreference) and [API reference: env.webgpu.forceFallbackAdapter](https://onnxruntime.ai/docs/api/js/interfaces/Env.WebGpuFlags.html#forceFallbackAdapter).
|
||||
|
||||
#### `env.webgpu.profiling`
|
||||
|
||||
The `env.webgpu.profiling` flag is used to enable WebGPU profiling.
|
||||
|
||||
Please see [WebGPU Profiling](./performance-diagnosis.md#webgpu-profiling) for more details.
|
||||
|
||||
For more information, see [API reference: env.webgpu.profiling](https://onnxruntime.ai/docs/api/js/interfaces/Env.WebGpuFlags.html#profiling).
|
||||
|
||||
## Session options
|
||||
|
||||
### Summary
|
||||
|
||||
Session options are used to configure the behavior of a single inference session. They are passed to the `InferenceSession.create()` method.
|
||||
|
||||
For more information, see [API reference: Interface InferenceSession.SessionOptions](https://onnxruntime.ai/docs/api/js/interfaces/InferenceSession.SessionOptions.html).
|
||||
|
||||
### `executionProviders`
|
||||
|
||||
The `executionProviders` option is used to specify a list of execution providers that will be used by the inference session.
|
||||
|
||||
The following execution providers are available in ONNX Runtime Web:
|
||||
|
||||
- `'wasm'`: The default CPU execution provider.
|
||||
- `'webgpu'`: The WebGPU execution provider. See [WebGPU EP](./ep-webgpu.md) for more details.
|
||||
- `'webnn'`: The WebNN execution provider.
|
||||
- `'webgl'`: The WebGL execution provider.
|
||||
|
||||
```js
|
||||
const mySession = await ort.InferenceSession.create(modelUrl, {
|
||||
...,
|
||||
// specify the EP list
|
||||
executionProviders: ['webgpu', 'wasm']
|
||||
});
|
||||
```
|
||||
|
||||
For more information, see [API reference: executionProviders](https://onnxruntime.ai/docs/api/js/interfaces/InferenceSession.SessionOptions.html#executionProviders).
|
||||
|
||||
### `externalData`
|
||||
|
||||
The `externalData` option is used to pass the external data information to ONNX Runtime Web. When a model's weights are stored in external data files, you need to pass the external data information to ONNX Runtime Web. See [External Data](./large-models.md#external-data) for more details.
|
||||
|
||||
For more information, see [API reference: externalData](https://onnxruntime.ai/docs/api/js/interfaces/InferenceSession.SessionOptions.html#externalData).
|
||||
|
||||
### `freeDimensionOverrides`
|
||||
|
||||
The `freeDimensionOverrides` option is used to override the free dimensions of the model.
|
||||
|
||||
ONNX models may have some dimensions as free dimensions, which means that the model can accept inputs of any size in that dimension. For example, an image model may define its input shape as `[batch, 3, height, width]`, which means that the model can accept any numbers of images of any size, as long as the number of channels is 3. However, if your application always uses images of a specific size, you can override the free dimensions to a specific size, which can be helpful to optimize the performance of the model. For example, if your web app always use a single image of 224x224, you can override the free dimensions to `[1, 3, 224, 224]` by specifying the following config in your session options:
|
||||
|
||||
```js
|
||||
const mySessionOptions = {
|
||||
...,
|
||||
freeDimensionOverrides: {
|
||||
batch: 1,
|
||||
height: 224,
|
||||
width: 224
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
For more information, see [API reference: freeDimensionOverrides](https://onnxruntime.ai/docs/api/js/interfaces/InferenceSession.SessionOptions.html#freeDimensionOverrides).
|
||||
|
||||
### `enableGraphCapture`
|
||||
|
||||
The `enableGraphCapture` option is used to enable graph capture feature. Currently, this feature is only available for WebGPU EP.
|
||||
|
||||
If ONNX Runtime determines that a model has static shapes, and all its computing kernels are running on the registered EP, it can capture the kernel executions in the first run and replay them in the following runs. This can lead to better performance when CPU sometimes is the bottleneck to prepare for the commands.
|
||||
|
||||
```js
|
||||
const mySessionOptions = {
|
||||
...,
|
||||
enableGraphCapture: true
|
||||
};
|
||||
```
|
||||
|
||||
Not all models are suitable for graph capture. Some models with dynamic input shapes can use this feature together with [free dimension override](#freedimensionoverrides). Some models just don't work with this feature. You can try it out and see if it works for your model. If it doesn't work, the model initialization will fail, and you can disable this feature for this model.
|
||||
|
||||
See [API reference: enableGraphCapture](https://onnxruntime.ai/docs/api/js/interfaces/InferenceSession.SessionOptions.html#enablegraphcapture) for more details.
|
||||
|
||||
|
||||
### `optimizedModelFilePath`
|
||||
|
||||
The `optimizedModelFilePath` option is used to specify the file path of the optimized model. In browsers, the value of this option is ignored. Instead, the a new tab is opened with the content of the optimized model as a blob, allowing the user to download and save the optimized model.
|
||||
|
||||
```js
|
||||
const mySessionOptions = {
|
||||
...,
|
||||
// specify this option to allow downloading the optimized model
|
||||
optimizedModelFilePath: 'optimized_model.onnx'
|
||||
};
|
||||
```
|
||||
|
||||
NOTE: This feature is not available by default. It requires to rebuild ONNX Runtime Web with the `--cmake_extra_defines onnxruntime_ENABLE_WEBASSEMBLY_OUTPUT_OPTIMIZED_MODEL=1` command line option.
|
||||
|
||||
For more information, see [API reference: optimizedModelFilePath](https://onnxruntime.ai/docs/api/js/interfaces/InferenceSession.SessionOptions.html#optimizedModelFilePath).
|
||||
|
||||
### `preferredOutputLocation`
|
||||
|
||||
The `preferredOutputLocation` option is used to specify the preferred location of the output data. It can be used to keep the output data on GPU for further processing. See [Keep tensor data on GPU (IO binding)](./ep-webgpu.md#keep-tensor-data-on-gpu-io-binding) for more details.
|
||||
|
||||
For more information, see [API reference: preferredOutputLocation](https://onnxruntime.ai/docs/api/js/interfaces/InferenceSession.SessionOptions.html#preferredOutputLocation).
|
||||
178
docs/tutorials/web/ep-webgpu.md
Normal file
178
docs/tutorials/web/ep-webgpu.md
Normal file
|
|
@ -0,0 +1,178 @@
|
|||
---
|
||||
title: Using WebGPU
|
||||
description: Using WebGPU
|
||||
parent: Web
|
||||
grand_parent: Tutorials
|
||||
has_children: false
|
||||
nav_order: 3
|
||||
---
|
||||
{::options toc_levels="2..4" /}
|
||||
|
||||
# Using the WebGPU Execution Provider
|
||||
{: .no_toc }
|
||||
|
||||
This document explains how to use the WebGPU execution provider in ONNX Runtime.
|
||||
|
||||
## Contents
|
||||
{: .no_toc}
|
||||
|
||||
* TOC
|
||||
{:toc}
|
||||
|
||||
|
||||
## Basics
|
||||
|
||||
### What is WebGPU? Should I use it?
|
||||
|
||||
WebGPU is a new web standard for general purpose GPU compute and graphics. It is designed to be a low-level API, based on D3D12, Vulkan and Metal, and is designed to be used in the browser. It is designed to be more efficient and performant than WebGL, and is designed to be used for machine learning, graphics, and other compute tasks.
|
||||
|
||||
WebGPU is available out-of-box in latest versions of Chrome and Edge on Windows, macOS, Android and ChromeOS. It is also available in Firefox behind a flag and Safari Technology Preview. Check [WebGPU status](https://webgpu.io/status/) for the latest information.
|
||||
|
||||
If you are using ONNX Runtime Web for inferencing very lightweight models in you web application, and you want to have a small binary size, you can keep using the default WebAssembly (WASM) execution provider. If you want to run more compute intensive models, or you want to take advantage of the GPU in the client's device, you can use the WebGPU execution provider.
|
||||
|
||||
### How to use WebGPU EP in ONNX Runtime Web
|
||||
|
||||
This section assumes you have already set up your web application with ONNX Runtime Web. If you haven't, you can follow the [Get Started](../../get-started/with-javascript/web.md) for some basic info.
|
||||
|
||||
To use WebGPU EP, you just need to make 2 small changes:
|
||||
1. Update your import statement:
|
||||
|
||||
- For HTML script tag, change `ort.min.js` to `ort.webgpu.min.js`:
|
||||
```html
|
||||
<script src="https://example.com/path/ort.webgpu.min.js"></script>
|
||||
```
|
||||
- For JavaScript import statement, change `onnxruntime-web` to `onnxruntime-web/webgpu`:
|
||||
```js
|
||||
import * as ort from 'onnxruntime-web/webgpu';
|
||||
```
|
||||
|
||||
See [Conditional Importing](https://github.com/microsoft/onnxruntime-inference-examples/tree/main/js/importing_onnxruntime-web#conditional-importing) for details.
|
||||
|
||||
2. Specify 'webgpu' EP explicitly in session options:
|
||||
```js
|
||||
const session = await ort.InferenceSession.create(modelPath, { ..., executionProviders: ['webgpu'] });
|
||||
```
|
||||
|
||||
You might also consider installing the latest nightly build version of ONNX Runtime Web (onnxruntime-web@dev) to benefit from the latest features and improvments.
|
||||
|
||||
## WebGPU EP features
|
||||
|
||||
ONNX Runtime Web offers the following features which may be helpful to use with WebGPU EP:
|
||||
|
||||
### Graph Capture
|
||||
|
||||
You can try the graph capture feature if your model has static shapes and all its computing kernels are running on WebGPU EP. This feature may potentially improve the performance of your model.
|
||||
|
||||
See [Graph Capture](./env-flags-and-session-options.md#enablegraphcapture) for more details.
|
||||
|
||||
### Using `ort.env.webgpu` flags
|
||||
|
||||
See [`env.webgpu`](./env-flags-and-session-options.md#envwebgpu) for more details.
|
||||
|
||||
## Keep tensor data on GPU (IO binding)
|
||||
|
||||
By default, a model's inputs and outputs are tensors that hold data in CPU memory. When you run a session with WebGPU EP, the data is copied to GPU memory, and the results are copied back to CPU memory. If you get your input data from a GPU-based source, or you want to keep the output data on GPU for further processing, you can use IO binding to keep the data on GPU. This will be especially helpful when running transformer based models, which usually runs a single model multiple times with previous output as the next input.
|
||||
|
||||
For model input, if your input data is a WebGPU storage buffer, you can [create a GPU tensor and use it as input tensor](#create-input-tensor-from-a-gpu-buffer).
|
||||
|
||||
For model output, there are 2 ways to use the IO binding feature:
|
||||
- [Use pre-allocated GPU tensors](#use-pre-allocated-gpu-tensors)
|
||||
- [Specify the output data location](#specify-the-output-data-location)
|
||||
|
||||
Please also check the following topics:
|
||||
- [Zero-sized tensors](#zero-sized-tensors)
|
||||
- [GPU tensor life cycle management](#gpu-tensor-life-cycle-management)
|
||||
|
||||
### Create input tensor from a GPU buffer
|
||||
|
||||
If your input data is a WebGPU storage buffer, you can create a GPU tensor and use it as input tensor:
|
||||
|
||||
```js
|
||||
const inputTensor = ort.Tensor.fromGpuBuffer(inputGpuBuffer, {
|
||||
dataType: 'float32',
|
||||
dims: [1, 3, 224, 224]
|
||||
});
|
||||
|
||||
```
|
||||
|
||||
Use this tensor as model inputs(feeds) so that the input data will be kept on GPU.
|
||||
|
||||
### Use pre-allocated GPU tensors
|
||||
|
||||
If you know the output shape in advance, you can create a GPU tensor and use it as output tensor:
|
||||
|
||||
```js
|
||||
|
||||
// Create a pre-allocated buffer and the corresponding tensor. Assuming that the output shape is [10, 1000].
|
||||
const bufferSize = (10 * 1000) /* number of elements */ * 4 /* bytes per element */;
|
||||
const device = ort.env.webgpu.device;
|
||||
const myPreAllocatedBuffer = device.createBuffer({
|
||||
usage: GPUBufferUsage.COPY_SRC | GPUBufferUsage.COPY_DST | GPUBufferUsage.STORAGE,
|
||||
size: Math.ceil(bufferSize / 16) * 16 /* align to 16 bytes */
|
||||
});
|
||||
|
||||
const myPreAllocatedOutputTensor = ort.Tensor.fromGpuBuffer(myPreAllocatedBuffer, {
|
||||
dataType: 'float32',
|
||||
dims: [10, 1000]
|
||||
});
|
||||
|
||||
// ...
|
||||
|
||||
// Run the session with fetches
|
||||
const feeds = { 'input_0': myInputTensor };
|
||||
const fetches = { 'output_0': myPreAllocatedOutputTensor };
|
||||
const results = await mySession.run(feeds, fetches);
|
||||
|
||||
```
|
||||
|
||||
By specifying the output tensor in the fetches, ONNX Runtime Web will use the pre-allocated buffer as the output buffer. If there is a shape mismatch, the `run()` call will fail.
|
||||
|
||||
### Specify the output data location
|
||||
|
||||
If you don't want to use pre-allocated GPU tensors for outputs, you can also specify the output data location in the session options:
|
||||
|
||||
```js
|
||||
const mySessionOptions1 = {
|
||||
...,
|
||||
// keep all output data on GPU
|
||||
preferredOutputLocation: 'gpu-buffer'
|
||||
};
|
||||
|
||||
const mySessionOptions2 = {
|
||||
...,
|
||||
// alternatively, you can specify the output location for each output tensor
|
||||
preferredOutputLocation: {
|
||||
'output_0': 'cpu', // keep output_0 on CPU. This is the default behavior.
|
||||
'output_1': 'gpu-buffer' // keep output_1 on GPU buffer
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
By specifying the config `preferredOutputLocation`, ONNX Runtime Web will keep the output data on the specified device.
|
||||
|
||||
See [API reference: preferredOutputLocation](https://onnxruntime.ai/docs/api/js/interfaces/InferenceSession.SessionOptions.html#preferredOutputLocation) for more details.
|
||||
|
||||
## Notes
|
||||
|
||||
### Zero-sized tensors
|
||||
|
||||
If a tensor's shape contains 1 or more dimensions with size 0, the tensor is considered as a zero-sized tensor. Zero-sized tensors do not have any data, so the data location is not applied. ONNX Runtime Web always treats zero-sized tensors as CPU tensors. To create a zero-sized tensor, you can use the following code:
|
||||
|
||||
```js
|
||||
const zeroSizedTensor = new ort.Tensor('float32', [], [3, 256, 0, 64]);
|
||||
```
|
||||
|
||||
### GPU tensor life cycle management
|
||||
|
||||
It is important to understand how the underlying GPU buffer is managed so that you can avoid memory leaks and improve buffer usage efficiency.
|
||||
|
||||
A GPU tensor is created either by user code or by ONNX Runtime Web as model's output.
|
||||
- When it is created by user code, it is always created with an existing GPU buffer using `Tensor.fromGpuBuffer()`. In this case, the tensor does not "own" the GPU buffer.
|
||||
|
||||
- It is user's responsibility to make sure the underlying buffer is valid during the inference, and call `buffer.destroy()` to dispose the buffer when it is no longer needed.
|
||||
- Avoid calling `tensor.getData()` and `tensor.dispose()`. Use the GPU buffer directly.
|
||||
- Using a GPU tensor with a destroyed GPU buffer will cause the session run to fail.
|
||||
- When it is created by ONNX Runtime Web as model's output (not a pre-allocated GPU tensor), the tensor "owns" the buffer.
|
||||
- You don't need to worry about the case that the buffer is destroyed before the tensor is used.
|
||||
- Call `tensor.getData()` to download the data from the GPU buffer to CPU and get the data as a typed array.
|
||||
- Call `tensor.dispose()` explicitly to destroy the underlying GPU buffer when it is no longer needed.
|
||||
|
|
@ -4,7 +4,7 @@ description: Custom Excel Functions for BERT Tasks in JavaScript
|
|||
parent: Web
|
||||
grand_parent: Tutorials
|
||||
has_children: false
|
||||
nav_order: 2
|
||||
nav_order: 9
|
||||
---
|
||||
|
||||
# ONNX Runtime Custom Excel Functions for BERT NLP Tasks in JavaScript
|
||||
|
|
|
|||
115
docs/tutorials/web/large-models.md
Normal file
115
docs/tutorials/web/large-models.md
Normal file
|
|
@ -0,0 +1,115 @@
|
|||
---
|
||||
title: Working with Large Models
|
||||
description: Working with Large Models in ONNX Runtime Web
|
||||
parent: Web
|
||||
grand_parent: Tutorials
|
||||
has_children: false
|
||||
nav_order: 4
|
||||
---
|
||||
{::options toc_levels="2..4" /}
|
||||
|
||||
# Working with Large Models
|
||||
{: .no_toc }
|
||||
|
||||
The size of ONNX models can vary greatly depending on the complexity of the model and the number of parameters. They can be as small as a few KBs or as large as several GBs. While ONNX Runtime Web is designed to run all models in the browser, there are some considerations to keep in mind when working with large models.
|
||||
|
||||
## Contents
|
||||
{: .no_toc}
|
||||
|
||||
* TOC
|
||||
{:toc}
|
||||
|
||||
|
||||
## Platform restrictions
|
||||
|
||||
There are some platform restrictions that you should be aware of when working with large models in the browser:
|
||||
|
||||
### Maximum size of ArrayBuffer
|
||||
|
||||
Although there is no hard limit on the size of an ArrayBuffer in JavaScript, each browser has its own limitations. For example, the maximum size of an ArrayBuffer in Chrome is 0x7fe00000 bytes (around 2GB). Be careful when using the `fetch` API to load large models, as it may fail if you call `response.arrayBuffer()` on a large file.
|
||||
|
||||
ONNX Runtime Web creates array buffers > 2GB by using [`new WebAssembly.Memory()`](https://developer.mozilla.org/en-US/docs/WebAssembly/JavaScript_interface/Memory/Memory) to bypass this limitation. However, an ArrayBuffer instance created by `new WebAssembly.Memory()` is not transferable, so it cannot work with the [Proxy](https://onnxruntime.ai/docs/api/js/interfaces/Env.WebAssemblyFlags.html#proxy) feature.
|
||||
|
||||
### Protobuf file size limit
|
||||
|
||||
The ONNX model is serialized in the protobuf format. The maximum size of a protobuf file is 2GB. If an ONNX model is larger than 2GB, it's usually generated with external data. See [External Data](#external-data) for more details.
|
||||
|
||||
### WebAssembly memory limit
|
||||
|
||||
WebAssembly has a memory limit of 4GB. This is the maximum amount of memory that a WebAssembly module can access because of the 32-bit addressing. Currently, there is no way for ONNX Runtime Web to run models larger than 4GB. We may support it in the future either by using WASM64 or by using direct GPU weight loading.
|
||||
|
||||
## Cache the model
|
||||
|
||||
To avoid loading the model every time the page is refreshed, you can cache the model by using the Cache API or Origin private file system. This way, the model can be loaded from the cache instead of being fetched from the server every time.
|
||||
|
||||
See [Cache API](https://developer.mozilla.org/en-US/docs/Web/API/Cache) and [Origin private file system](https://developer.mozilla.org/en-US/docs/Web/API/File_System_API/Origin_private_file_system) for more details.
|
||||
|
||||
## External Data
|
||||
|
||||
When you work with an large ONNX model, it is usually generated with external data. Because of the protobuf file size limit, ONNX models larger than 2GB have to work with external data. The external data is one or more separate file(s) and it's usually generated by an ONNX exporter. The external data is usually put in the same directory as the ONNX model file.
|
||||
|
||||
ONNX Runtime supports loading models with external data. This is done automatically in C/C++/Python APIs without any extra steps because ONNX Runtime for these language bindings can access the file system. However, in the browser, the JavaScript code cannot access the file system directly. Therefore, you need one more step to pass the external data information to ONNX Runtime Web.
|
||||
|
||||
### How external data works
|
||||
|
||||
Before we dive into the details, let's first understand how external data works in ONNX Runtime. This information is important because otherwise the steps may look confusing.
|
||||
|
||||
An ONNX model is technically a protobuf file. The protobuf file contains the model graph and the weights. The ONNX spec allows the weights to be stored either in the protobuf file or in external data files. When the weights are stored in the protobuf file, it is fully included in the protobuf file. When the weights are stored in external data files, the protobuf file contains the following information of that specific weight:
|
||||
- "location" (a string) that specifies the relative file path of the external data file
|
||||
- "offset" (an integer) that specifies the byte offset in the external data file where the weight starts
|
||||
- "length" (an integer) that specifies the length of the weight in bytes
|
||||
|
||||
The "location" is usually determined by the ONNX exporter. For example, an exporter may output the model file as `model_a.onnx`, and the external data file as `model_a.data` in the same directory. Some weights in the model are stored in the `model_a.data` file, so the "location" of these weights is set to `./model_a.data`. This information is stored in file `model_a.onnx`.
|
||||
|
||||
This explains why in native platforms it is important to make sure you **never rename the external data file**. If you rename the external data file, the "location" in the protobuf file will mismatch the actual file name, and ONNX Runtime will fail to load the model.
|
||||
|
||||
For ONNX Runtime Web, you always need to pass the external data information to ONNX Runtime Web. It's important to understand that the "location" defined in the protobuf file is a different concept from the actual external file path. These 2 different concepts will be represented as "path" and "data" in the JavaScript code in the following section.
|
||||
|
||||
### Load the model with external data in ONNX Runtime Web
|
||||
|
||||
We use an example to illustrate how to load an ONNX model with external data in the browser. Suppose we have an ONNX model `model_a.onnx` and an external data file `model_a.data`. The following code shows how to load the model with external data in the browser:
|
||||
|
||||
```js
|
||||
const modelUrl = 'https://example.com/path/model_a.onnx';
|
||||
const externalDataUrl = 'https://example.com/path/model_a.data';
|
||||
|
||||
const mySession = await ort.InferenceSession.create(modelUrl, {
|
||||
...,
|
||||
externalData: [
|
||||
{
|
||||
path: './model_a.data',
|
||||
data: externalDataUrl
|
||||
}
|
||||
]
|
||||
});
|
||||
```
|
||||
|
||||
In the code above, we pass the external data information to the `InferenceSession.create()` method. The `externalData` is an array of objects, each object representing one external data file. The object has two properties:
|
||||
- `path` (a string) that should match the weights' "location" info in the protobuf file
|
||||
- `data` (a string) that specifies the external data file. It can be a URL, a Blob or a Uint8Array.
|
||||
|
||||
When you store the model and external data in the IndexedDB, you can load the model with external data from the IndexedDB. The following code shows how to load the model with external data from the IndexedDB:
|
||||
|
||||
```js
|
||||
// assume loadFromIndexedDB() is a function implemented by your app that loads the data from the IndexedDB
|
||||
|
||||
// Load the model and external data from the IndexedDB
|
||||
const modelBlob = await loadFromIndexedDB('model_a.onnx');
|
||||
const externalDataBlob = await loadFromIndexedDB('model_a.data');
|
||||
|
||||
const mySession = await ort.InferenceSession.create(modelBlob, {
|
||||
...,
|
||||
externalData: [
|
||||
{
|
||||
path: './model_a.data',
|
||||
data: externalDataBlob
|
||||
}
|
||||
]
|
||||
});
|
||||
```
|
||||
|
||||
See [ONNX External Data](https://onnx.ai/onnx/repo-docs/ExternalData.html) for more details.
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
This section is under construction.
|
||||
203
docs/tutorials/web/performance-diagnosis.md
Normal file
203
docs/tutorials/web/performance-diagnosis.md
Normal file
|
|
@ -0,0 +1,203 @@
|
|||
---
|
||||
title: Performance Diagnosis
|
||||
description: Performance Diagnosis
|
||||
parent: Web
|
||||
grand_parent: Tutorials
|
||||
has_children: false
|
||||
nav_order: 5
|
||||
---
|
||||
{::options toc_levels="2..4" /}
|
||||
|
||||
# Performance Diagnosis
|
||||
{: .no_toc }
|
||||
|
||||
ONNX Runtime Web is designed to be fast and efficient, but there are a number of factors that can affect the performance of your application. This document provides some guidance on how to diagnose performance issues in ONNX Runtime Web.
|
||||
|
||||
Before you start, make sure that ONNX Runtime Web successfully loads and runs your model. If you encounter any issues, see the [troubleshooting guide](./trouble-shooting.md) for help first.
|
||||
|
||||
|
||||
## Contents
|
||||
{: .no_toc}
|
||||
|
||||
* TOC
|
||||
{:toc}
|
||||
|
||||
|
||||
## General performance tips
|
||||
|
||||
Here are some general tips to improve the performance of your application:
|
||||
|
||||
### Use the right model
|
||||
|
||||
Choose a model that is appropriate for web scenario. A model that is too large or too complex may not run efficiently on less powerful hardware. Usually, the "tiny" or "small" versions of models are more commonly used in web applications. This does not mean that you cannot use larger models, but you should be aware of the potential hit on user experience due to longer load times and slower inference.
|
||||
|
||||
### Use the right execution provider
|
||||
|
||||
Choose the right execution provider for your scenario.
|
||||
|
||||
- **WebAssembly (wasm)**: This is the default CPU execution provider for ONNX Runtime Web. Use it for very small models or environments where GPU is not available.
|
||||
|
||||
- **WebGPU (webgpu)**: This is the default GPU execution provider. Use it when the device has a decent GPU which supports WebGPU.
|
||||
|
||||
- **WebNN (webnn)**: This is the option which offers potential near-native performance on the web. It is currently not supported by default in browsers, but you can enable WebNN feature manually in browser's settings.
|
||||
|
||||
- **WebGL (webgl)**: This execution provider is designed to run models using GPU on older devices that do not support WebGPU.
|
||||
|
||||
### Use the diagnostic features
|
||||
|
||||
Use the [diagnostic features](#diagnostic-features) to get detailed information about the execution of the model. This can be helpful to understand the performance characteristics of the model and to identify potential problems or bottlenecks.
|
||||
|
||||
## CPU tips
|
||||
|
||||
If you are using the WebAssembly (wasm) execution provider, you can use the following tips to improve the performance of your application:
|
||||
|
||||
### Enable multi-threading
|
||||
|
||||
Always enable multi-threading if the environment supports it. Multi-threading can significantly improve the performance of your application by utilizing multiple CPU cores.
|
||||
|
||||
This feature is enabled by default in ONNX Runtime Web, however it only works when `crossOriginIsolated` mode is enabled. See [https://web.dev/cross-origin-isolation-guide/](https://web.dev/cross-origin-isolation-guide/) for more info.
|
||||
|
||||
You can also use flag `ort.env.wasm.numThreads` to set the number of threads to be used.
|
||||
|
||||
```js
|
||||
// Set the number of threads to 4
|
||||
ort.env.wasm.numThreads = 4;
|
||||
|
||||
// Disable multi-threading
|
||||
ort.env.wasm.numThreads = 1;
|
||||
|
||||
// Let ONNX Runtime Web decide the number of threads to use
|
||||
ort.env.wasm.numThreads = 0;
|
||||
```
|
||||
|
||||
See [API reference: env.wasm.numThreads](https://onnxruntime.ai/docs/api/js/interfaces/Env.WebAssemblyFlags.html#numThreads) for more details.
|
||||
|
||||
### Enable SIMD
|
||||
|
||||
Always enable SIMD if it's supported. SIMD (Single Instruction, Multiple Data) is a set of instructions that perform the same operation on multiple data points simultaneously. This can significantly improve the performance of your application.
|
||||
|
||||
This feature is enabled by default in ONNX Runtime Web, unless you explicitly disable it by setting `ort.env.wasm.simd = false`.
|
||||
|
||||
See [API reference: env.wasm.simd](https://onnxruntime.ai/docs/api/js/interfaces/Env.WebAssemblyFlags.html#simd) for more details.
|
||||
|
||||
### Prefer uint8 quantized models
|
||||
|
||||
If you are using a quantized model, prefer uint8 quantized models. Avoid float16 models if possible, as float16 is not natively supported by CPU and it is going to be slow.
|
||||
|
||||
### Enable Proxy Worker
|
||||
|
||||
Proxy worker is a feature that allows ONNX Runtime Web to offload the heavy computation to a separate Web Worker. Using the proxy worker cannot improve the performance of the model, but it can improve the responsiveness of the UI to improve the user experience.
|
||||
|
||||
If you didn't import ONNX Runtime Web in a Web Worker, and the model takes a while to inference, it is recommended to enable the proxy worker.
|
||||
|
||||
```js
|
||||
// Enable proxy worker
|
||||
ort.env.wasm.proxy = true;
|
||||
```
|
||||
|
||||
See [API reference: env.wasm.proxy](https://onnxruntime.ai/docs/api/js/interfaces/Env.WebAssemblyFlags.html#proxy) for more details.
|
||||
|
||||
## WebGPU tips
|
||||
|
||||
If you are using the WebGPU execution provider, you can use the following tips to improve the performance of your application:
|
||||
|
||||
### Try Using graph capture
|
||||
|
||||
See [Graph capture](env-flags-and-session-options.md#enablegraphcapture) for feature introduction.
|
||||
|
||||
|
||||
if your model has static shapes, and all its computing kernels are running on the WebGPU EP, you can try to enable the graph capture feature, unless you need to feed input data with dynamic shape (eg. transformer based decoder model). Even with static shape input, this feature does not always work for all models. You can try it out and see if it works for your model. If it doesn't work, the model initialization will fail, and you can disable this feature for this model.
|
||||
|
||||
### Try using free dimension override
|
||||
|
||||
See [Free dimension override](env-flags-and-session-options.md#freedimensionoverrides) for feature introduction.
|
||||
|
||||
Using free dimension override does not necessarily improve the performance. It's quite model by model. You can try it out and see if it works for your model. If you see performance degradation or larger memory usage, you may disable this feature.
|
||||
|
||||
### Try keep tensor data on GPU
|
||||
|
||||
See [Keep tensor data on GPU (IO binding)](ep-webgpu.md#keep-tensor-data-on-gpu-io-binding) for feature introduction.
|
||||
|
||||
Keeping tensor data on GPU can avoid unnecessary data transfer between CPU and GPU, which can improve the performance. Try to find out the best way to use this feature for your model.
|
||||
|
||||
Please be careful of the [GPU tensor life cycle management](ep-webgpu.md#gpu-tensor-life-cycle-management) when using this feature.
|
||||
|
||||
## Diagnostic features
|
||||
|
||||
### Profiling
|
||||
|
||||
You can enable profiling to get detailed information about the execution of the model. This can be helpful to understand the performance characteristics of the model and to identify potential bottlenecks.
|
||||
|
||||
#### CPU profiling
|
||||
|
||||
To enable CPU profiling:
|
||||
|
||||
- step.1: Specify the `enableProfiling` option in the session options:
|
||||
```js
|
||||
const mySessionOptions = {
|
||||
...,
|
||||
enableProfiling: true
|
||||
};
|
||||
```
|
||||
|
||||
By specifying this option, ONNX Runtime Web will collect CPU profiling data for each run.
|
||||
|
||||
- step.2: Get the profiling data after the inference:
|
||||
```js
|
||||
mySession.endProfiling();
|
||||
```
|
||||
|
||||
After calling `endProfiling()`, the profiling data will be outputted to the console.
|
||||
|
||||
See [In Code Performance Profiling](../../performance/tune-performance/profiling-tools.md#in-code-performance-profiling) for how to use the profiling data.
|
||||
|
||||
#### WebGPU profiling
|
||||
|
||||
To enable WebGPU profiling:
|
||||
|
||||
- set `ort.env.webgpu.profiling = { mode: 'default' }` to enable WebGPU profiling. GPU Profiling data will be outputted to the console with prefix `[profiling]`.
|
||||
- alternatively, you can set `ort.env.webgpu.profiling` with a function to handle the profiling data:
|
||||
```js
|
||||
ort.env.webgpu.profiling = {
|
||||
mode: 'default',
|
||||
ondata: (data) => {
|
||||
// handle the profiling data
|
||||
}
|
||||
};
|
||||
```
|
||||
See [API reference: env.webgpu.profiling](https://onnxruntime.ai/docs/api/js/interfaces/Env.WebGpuFlags.html#profiling) for more details.
|
||||
|
||||
### Trace
|
||||
|
||||
You can enable trace by specifying the following flag:
|
||||
```js
|
||||
ort.env.trace = true;
|
||||
```
|
||||
|
||||
This feature uses `console.timeStamp` to log the trace data. You can use the browser's performance tool to analyze the trace data.
|
||||
|
||||
See [API reference: env.trace](https://onnxruntime.ai/docs/api/js/interfaces/Env-1.html#trace) for more details.
|
||||
|
||||
### Log level 'verbose'
|
||||
|
||||
You can set the log level to 'verbose' to get more detailed logs:
|
||||
```js
|
||||
ort.env.logLevel = 'verbose';
|
||||
```
|
||||
|
||||
See [API reference: env.logLevel](https://onnxruntime.ai/docs/api/js/interfaces/Env-1.html#logLevel) for more details.
|
||||
|
||||
### Enable debug mode
|
||||
|
||||
You can enable the debug mode by specifying the following flag:
|
||||
```js
|
||||
ort.env.debug = true;
|
||||
```
|
||||
|
||||
In debug mode, ONNX Runtime Web will log detailed information about the execution of the model, and also apply some additional checks. Usually you need to use `verbose` log level to see the debug logs.
|
||||
|
||||
See [API reference: env.debug](https://onnxruntime.ai/docs/api/js/interfaces/Env-1.html#debug) for more details.
|
||||
|
||||
## Analyze the profiling data
|
||||
|
||||
This part is under construction.
|
||||
23
docs/tutorials/web/trouble-shooting.md
Normal file
23
docs/tutorials/web/trouble-shooting.md
Normal file
|
|
@ -0,0 +1,23 @@
|
|||
---
|
||||
title: Troubleshooting
|
||||
description: Troubleshooting
|
||||
parent: Web
|
||||
grand_parent: Tutorials
|
||||
has_children: false
|
||||
nav_order: 7
|
||||
---
|
||||
|
||||
{::options toc_levels="2..4" /}
|
||||
|
||||
# Troubleshooting
|
||||
{: .no_toc }
|
||||
|
||||
This document provides some guidance on how to troubleshoot common issues in ONNX Runtime Web.
|
||||
|
||||
## Contents
|
||||
{: .no_toc}
|
||||
|
||||
* TOC
|
||||
{:toc}
|
||||
|
||||
The content of this document is under construction. Please check back later.
|
||||
Loading…
Reference in a new issue