mirror of
https://github.com/saymrwulf/onnxruntime.git
synced 2026-06-21 02:18:09 +00:00
### Description This PR make numbers of optimizations to onnxruntime-web's module export and deployment. See each section below for more details. #### Preview > [onnxruntime-web@1.19.0-esmtest.20240513-a16cd2bd21](https://www.npmjs.com/package/onnxruntime-web/v/1.19.0-esmtest.20240513-a16cd2bd21) > ~~onnxruntime-web@1.19.0-esmtest.20240430-c7edbcc63d~~ > ~~onnxruntime-web@1.18.0-esmtest.20240428-624c681c83~~ > ~~onnxruntime-web@1.18.0-esmtest.20240411-1abb64e894~~ <details> <summary><h4>Breaking changes</h4></summary> There is no code change required, but there are a few differences regarding **code import**, **flags**, **bundler config** and **deployment steps**. #### Importing: Import table is changed. See following for details. <details> <summary><h5>Current import table:</h5></summary> | Target Name | Path for "import" or "require" | WebGL | JSEP | wasm | Proxy | Training | |------|-----|-----|-----|-----|-----|-----| | `ort` (default) | `onnxruntime-web` | ✔️ | ❌ | ✔️ | ✔️ | ❌ | | `ort.all` | `onnxruntime-web/experimental` | ✔️ | ✔️ | ✔️ | ✔️ | ❌ | | `ort.node` | `onnxruntime-web` | ❌ | ❌ | ✔️ | ❌ | ❌ | | `ort.training` | `onnxruntime-web/training` | ❌ | ❌ | ✔️ | ✔️<sup>\[1]</sup> | ✔️ | | `ort.wasm` | `onnxruntime-web/wasm` | ❌ | ❌ | ✔️ | ✔️ | ❌ | | `ort.wasm-core` | `onnxruntime-web/wasm-core` | ❌ | ❌ | ✔️ | ❌ | ❌ | | `ort.webgl` | `onnxruntime-web/webgl` | ✔️ | ❌ | ❌ | ✔️<sup>\[2]</sup> | ❌ | | `ort.webgpu` | `onnxruntime-web/webgpu` | ❌ | ✔️ | ✔️ | ✔️ | ❌ | * [1] didn't test. may not actually work. * [2] not working. this is a mistake in build config. </details> <details> <summary><h5>Proposed update:</h5></summary> | Target Name | Path for "import" or "require" | WebGL | JSEP | wasm | Proxy | Training | |------|-----|-----|-----|-----|-----|-----| | `ort` (default) | `onnxruntime-web` | ✔️ | ❌ | ✔️ | ✔️ | ❌ | | `ort.all` | ~~`onnxruntime-web/experimental`~~<br/>`onnxruntime-web/all` | ✔️ | ✔️ | ✔️ | ✔️ | ❌ | | `ort.node` | `onnxruntime-web` | ❌ | ❌ | ✔️ | ❌ | ❌ | | `ort.training` | `onnxruntime-web/training` | ❌ | ❌ | ✔️ | ✔️ | ✔️ | | `ort.wasm` | `onnxruntime-web/wasm` | ❌ | ❌ | ✔️ | ✔️ | ❌ | | ~~`ort.wasm-core`~~ | ~~`onnxruntime-web/wasm-core`~~ | ~~❌~~ | ~~❌~~ | ~~✔️~~ | ~~❌~~ | ~~❌~~ | | `ort.webgl` | `onnxruntime-web/webgl` | ✔️ | ❌ | ❌ | ~~✔️~~ ❌ | ❌ | | `ort.webgpu` | `onnxruntime-web/webgpu` | ❌ | ✔️ | ✔️ | ✔️ | ❌ | </details> #### Flags: The following flags are deprecated: - `env.wasm.simd` (boolean): will be ignored. SIMD is always enabled in build. The following flags changed their type: - `env.wasm.wasmPaths`: When using this flag as a string ( for the URL prefix ), nothing is changed. When using this flag as an object ( for per-file path override ), the type changed: ```diff - export interface Old_WasmFilePaths{ - 'ort-wasm.wasm'?: string; - 'ort-wasm-threaded.wasm'?: string; - 'ort-wasm-simd.wasm'?: string; - 'ort-training-wasm-simd.wasm'?: string; - 'ort-wasm-simd-threaded.wasm'?: string; - }; + export interface New_WasmFilePaths { + /** + * Specify the override path for the main .wasm file. + * + * This path should be an absolute path. + * + * If not modified, the filename of the .wasm file is: + * - `ort-wasm-simd-threaded.wasm` for default build + * - `ort-wasm-simd-threaded.jsep.wasm` for JSEP build (with WebGPU and WebNN) + * - `ort-training-wasm-simd-threaded.wasm` for training build + */ + wasm?: URL|string; + /** + * Specify the override path for the main .mjs file. + * + * This path should be an absolute path. + * + * If not modified, the filename of the .mjs file is: + * - `ort-wasm-simd-threaded.mjs` for default build + * - `ort-wasm-simd-threaded.jsep.mjs` for JSEP build (with WebGPU and WebNN) + * - `ort-training-wasm-simd-threaded.mjs` for training build + */ + mjs?: URL|string; + } ``` #### Bundler compatibility: Config changes are need for bundlers. See usage example in /js/web/test/e2e/ for Webpack, parcel and rollup. #### Deployment: - if consuming from a CDN, there is no breaking change. - if consuming from a local server, need to copy all `ort-*.wasm` and `ort-*.mjs` files (totally 6 files) in the dist folder. (previously only need to copy `ort-*.wasm` files.) </details> <details> <summary><h4>Problems</h4></summary> There are a few problems with the current module export and deployment: - Script URL cannot be correctly inferred when imported as ESM. - Workers are forcefully encoded using Blob URL, which makes onnxruntime-web not working in CSP environment and Node.js, when using proxy or multi-threading feature. - Generated JS code (by Emscripten) is encoded using `function.toString()`, which is unstable and error-prone. - When running with a different Emscripten build, always need the build step. Making it difficult to swap artifacts in deveopment/debug. </details> <details> <summary><h4>Goals</h4></summary> - Full ESM support - Support variances of ways to import. Including: - import from HTML's `<script>` tag (IIFE format, exporting to global variable `ort`) ```html <script src="https://example.com/cdn-path-to-onnxruntime-web/dist/ort.min.js"></script> ``` - import from source code inside `<script type="module">` tag (ESM) ```html <script type="module"> import * as ort from "https://example.com/cdn-path-to-onnxruntime-web/dist/ort.min.mjs"; // using 'ort' </script> ``` - import in a CommonJS project (CJS format, resolve from package.json "exports" field) ```js // myProject/main.js const ort = require('onnxruntime-web'); ``` - import in an ESM project (ESM format, resolve from package.json "exports" field) ```js // myProject/main.js (or main.mjs) import * as ort from 'onnxruntime-web'; ``` - Support popular bundlers when importing onnxruntime-web into a CJS/ESM project. - webpack (esm requires extra post-process step) - rollup - parcel (esm requires extra post-process step) - More bundlers **TBD** - Multi-threading support for Node.js NOTE: keeping single JavaScript file (the all-in-one bundle) is no longer a goal. This is because technically there is a conflict with the other requirements. </details> <details> <summary><h4>Important Design Decisions</h4></summary> - Drop support of single JavaScript output. - The current onnxruntime-web distribution uses a single JavaScript file to include all code. While there are a few benefits, it also creates problems as mentioned above. Since ESM is being used more and more widely, and browsers are making more restricted security checks and requirement, the old Blob based solution is going to be replaced. - To achieve the requirement, specifically, the CSP environment support, we have to offer a non Blob based solution. Therefore, we have to distribute multiple files and drop the single file solution. - Do not run parser/postprocess on Emscripten generated JavaScript. - Emscripten is evolving quickly so we should only depends on what's in its documentation instead of a certain implementation details. (for example, currently we patch on its code to deal with a special variable `_scriptDir`) - Keep the generated files as-is also helps to: - reduce the size of ort.min.js - make it easier to replace build artifacts when in development/debug - Drop support for non-SIMD and non-MultiThread. This helps to reduce the number of artifacts in distribution. - (fixed-sized) SIMD is supported in any mainstream JS environment. - Multi-thread as WebAssembly feature is supported in any mainstream JS environment. In some environment the feature is guarded with cross origin policy, but it can still work if not trying to create any worker. - Use ESM output for Emscripten generated JavaScript. - There are 2 ways to dynamically import classic (umd) modules and neither of them are recommended: - dynamically creating a <script> tag. This changes the HTML structure and have quite a lot of compatibility issue - use `fetch()` and `eval()`. However `eval` is strongly suggested to be avoid because there is a great perf hit. - importing ESM is super easy - just use the `import()` call. Considering ESM is widely supported in modern browsers and Node.js this is the better option. - Add Blob based solution as a fallback for cross-origin workers. - There are still wide use case of importing onnxruntime-web from CDN. In this usage, make it able create worker by using `fetch()`+`Blob` to create a same-origin Blob URL. </details> <details> <summary><h4>Distribution File Manifest</h4></summary> The distribution folder contains the following files: - WebAssembly artifacts. These files are the result of compiling the ONNX Runtime C++ code to WebAssembly by Emscripten. | File Name | Build Flags | |------|-----| | ort-wasm-simd-threaded.mjs <br/> ort-wasm-simd-threaded.wasm | `--enable_wasm_simd` <br/> `--enable_wasm_threads` | | ort-training-wasm-simd-threaded.mjs <br/> ort-training-wasm-simd-threaded.wasm | `--enable_training_apis` <br/> `--enable_wasm_simd` <br/> `--enable_wasm_threads` | | ort-wasm-simd-threaded.jsep.mjs <br/> ort-wasm-simd-threaded.jsep.wasm | `--enable_wasm_simd` <br/> `--enable_wasm_threads` <br/> `--use_jsep` <br/> `--use_webnn` | - onnxruntime-web JavaScript artifacts. These files are generated by ESBuild as the entry point for onnxruntime-web. There are multiple build targets for different use cases: | Target Name | Path for "import" or "require" | Description | |------|-----|-----| | `ort` | `onnxruntime-web` | The default target. | | `ort.all` | `onnxruntime-web/all` | The target including webgl. | | `ort.node` | `onnxruntime-web` | The default target for Node.js. | | `ort.training` | `onnxruntime-web/training` | The target including training APIs | | `ort.wasm` | `onnxruntime-web/wasm` | The target including only WebAssembly (CPU) EP | | `ort.webgl` | `onnxruntime-web/webgl` | The target including only WebGL EP | For each target, there are multiple files generated: | File Name | Description | |------|-----| | [target].js | The entry point for the target. IIFE and CommonJS format. | | [target].mjs | The entry point for the target. ESM format. | | [target].min.js <br/> [target].min.js.map | The entry point for the target. Minimized with sourcemap. IIFE and CommonJS format. | | [target].min.mjs <br/> [target].min.mjs.map | The entry point for the target. Minimized with sourcemap. ESM format. | | [target].proxy.mjs | (if appliable) The proxy ESM module for the target. | | [target].proxy.min.mjs <br/> [target].proxy.min.mjs.map | (if appliable) The proxy ESM module for the target. Minimized with sourcemap. | </details> <details> <summary><h4>Dynamic Import Explained</h4></summary> - Local Served | No Proxy: ``` [Bundle or ort.min.js] | + import()--> [ort-wasm-simd-threaded.mjs] | + WebAssembly.instantiateStreaming()--> [ort-wasm-simd-threaded.wasm] | + new Worker()--> [ort-wasm-simd-threaded.mjs (worker)] | + WebAssembly.instantiateStreaming()--> [ort-wasm-simd-threaded.wasm] ``` - Local Served | Proxy: ``` [Bundle or ort.min.js] | + import()--> [ort.proxy.min.mjs] | + new Worker()--> [ort.proxy.min.mjs (worker)] | + import()--> [ort-wasm-simd-threaded.mjs] | + WebAssembly.instantiateStreaming()--> [ort-wasm-simd-threaded.wasm] | + new Worker()--> [ort-wasm-simd-threaded.mjs (worker)] | + WebAssembly.instantiateStreaming()--> [ort-wasm-simd-threaded.wasm] ``` - Cross Origin | No Proxy: ``` [Bundle or ort.min.js] | + fetch('ort-wasm-simd-threaded.mjs') | + URL.createObjectURL(res.blob()) | + import()--> [blob:... (ort-wasm-simd-threaded)] | + WebAssembly.instantiateStreaming()--> [ort-wasm-simd-threaded.wasm] | + new Worker()--> [blob:... (ort-wasm-simd-threaded) (worker)] | + WebAssembly.instantiateStreaming()--> [ort-wasm-simd-threaded.wasm] ``` - Cross Origin | Proxy ``` [Bundle or ort.min.js] | + fetch('ort.proxy.min.mjs') | + URL.createObjectURL(res.blob()) | + import()--> [blob:... (ort.proxy)] | + new Worker()--> [blob:... (ort.proxy) (worker)] | + fetch('ort-wasm-simd-threaded.mjs') | + URL.createObjectURL(res.blob()) | + import()--> [blob:... (ort-wasm-simd-threaded)] | + WebAssembly.instantiateStreaming()--> [ort-wasm-simd-threaded.wasm] | + new Worker()--> [blob:... (ort-wasm-simd-threaded) (worker)] | + WebAssembly.instantiateStreaming()--> [ort-wasm-simd-threaded.wasm] ``` </details>
156 lines
6.4 KiB
TypeScript
156 lines
6.4 KiB
TypeScript
// Copyright (c) Microsoft Corporation. All rights reserved.
|
|
// Licensed under the MIT License.
|
|
|
|
import type {OrtWasmModule} from './wasm-types';
|
|
import {isNode} from './wasm-utils-env';
|
|
|
|
/**
|
|
* The classic script source URL. This is not always available in non ESModule environments.
|
|
*
|
|
* In Node.js, this is undefined.
|
|
*/
|
|
export const scriptSrc =
|
|
// if Nodejs, return undefined
|
|
isNode ? undefined :
|
|
// if It's ESM, use import.meta.url
|
|
BUILD_DEFS.ESM_IMPORT_META_URL ??
|
|
// use `document.currentScript.src` if available
|
|
(typeof document !== 'undefined' ? (document.currentScript as HTMLScriptElement)?.src :
|
|
// use `self.location.href` if available
|
|
(typeof self !== 'undefined' ? self.location?.href : undefined));
|
|
|
|
/**
|
|
* The origin of the current location.
|
|
*
|
|
* In Node.js, this is undefined.
|
|
*/
|
|
const origin = isNode || typeof location === 'undefined' ? undefined : location.origin;
|
|
|
|
/**
|
|
* Check if the given filename with prefix is from the same origin.
|
|
*/
|
|
const isSameOrigin = (filename: string, prefixOverride?: string) => {
|
|
try {
|
|
const baseUrl = prefixOverride ?? scriptSrc;
|
|
const url = baseUrl ? new URL(filename, baseUrl) : new URL(filename);
|
|
return url.origin === origin;
|
|
} catch {
|
|
return false;
|
|
}
|
|
};
|
|
|
|
/**
|
|
* Normalize the inputs to an absolute URL with the given prefix override. If failed, return undefined.
|
|
*/
|
|
const normalizeUrl = (filename: string, prefixOverride?: string) => {
|
|
const baseUrl = prefixOverride ?? scriptSrc;
|
|
try {
|
|
const url = baseUrl ? new URL(filename, baseUrl) : new URL(filename);
|
|
return url.href;
|
|
} catch {
|
|
return undefined;
|
|
}
|
|
};
|
|
|
|
/**
|
|
* Create a fallback URL if an absolute URL cannot be created by the normalizeUrl function.
|
|
*/
|
|
const fallbackUrl = (filename: string, prefixOverride?: string) => `${prefixOverride ?? './'}${filename}`;
|
|
|
|
/**
|
|
* This helper function is used to preload a module from a URL.
|
|
*
|
|
* If the origin of the worker URL is different from the current origin, the worker cannot be loaded directly.
|
|
* See discussions in https://github.com/webpack-contrib/worker-loader/issues/154
|
|
*
|
|
* In this case, we will fetch the worker URL and create a new Blob URL with the same origin as a workaround.
|
|
*
|
|
* @param absoluteUrl - The absolute URL to preload.
|
|
*
|
|
* @returns - A promise that resolves to a new Blob URL
|
|
*/
|
|
const preload = async(absoluteUrl: string): Promise<string> => {
|
|
const response = await fetch(absoluteUrl, {credentials: 'same-origin'});
|
|
const blob = await response.blob();
|
|
return URL.createObjectURL(blob);
|
|
};
|
|
|
|
/**
|
|
* This helper function is used to dynamically import a module from a URL.
|
|
*
|
|
* The build script has special handling for this function to ensure that the URL is not bundled into the final output.
|
|
*
|
|
* @param url - The URL to import.
|
|
*
|
|
* @returns - A promise that resolves to the default export of the module.
|
|
*/
|
|
const dynamicImportDefault = async<T>(url: string): Promise<T> => (await import(/* webpackIgnore: true */ url)).default;
|
|
|
|
/**
|
|
* The proxy worker factory imported from the proxy worker module.
|
|
*
|
|
* This is only available when the WebAssembly proxy is not disabled.
|
|
*/
|
|
const createProxyWorker: ((urlOverride?: string) => Worker)|undefined =
|
|
// eslint-disable-next-line @typescript-eslint/no-require-imports, @typescript-eslint/no-var-requires
|
|
BUILD_DEFS.DISABLE_WASM_PROXY ? undefined : require('./proxy-worker/main').default;
|
|
|
|
/**
|
|
* Import the proxy worker.
|
|
*
|
|
* This function will perform the following steps:
|
|
* 1. If a preload is needed, it will preload the module and return the object URL.
|
|
* 2. Use the proxy worker factory to create the proxy worker.
|
|
*
|
|
* @returns - A promise that resolves to a tuple of 2 elements:
|
|
* - The object URL of the preloaded module, or undefined if no preload is needed.
|
|
* - The proxy worker.
|
|
*/
|
|
export const importProxyWorker = async(): Promise<[undefined | string, Worker]> => {
|
|
if (!scriptSrc) {
|
|
throw new Error('Failed to load proxy worker: cannot determine the script source URL.');
|
|
}
|
|
|
|
// If the script source is from the same origin, we can use the embedded proxy module directly.
|
|
if (isSameOrigin(scriptSrc)) {
|
|
return [undefined, createProxyWorker!()];
|
|
}
|
|
|
|
// Otherwise, need to preload
|
|
const url = await preload(scriptSrc);
|
|
return [url, createProxyWorker!(url)];
|
|
};
|
|
|
|
/**
|
|
* Import the WebAssembly module.
|
|
*
|
|
* This function will perform the following steps:
|
|
* 1. If a preload is needed, it will preload the module and return the object URL.
|
|
* 2. Otherwise, it will perform a dynamic import of the module.
|
|
*
|
|
* @returns - A promise that resolves to a tuple of 2 elements:
|
|
* - The object URL of the preloaded module, or undefined if no preload is needed.
|
|
* - The default export of the module, which is a factory function to create the WebAssembly module.
|
|
*/
|
|
export const importWasmModule = async(
|
|
urlOverride: string|undefined, prefixOverride: string|undefined,
|
|
isMultiThreaded: boolean): Promise<[undefined | string, EmscriptenModuleFactory<OrtWasmModule>]> => {
|
|
const wasmModuleFilename = !BUILD_DEFS.DISABLE_TRAINING ? 'ort-training-wasm-simd-threaded.mjs' :
|
|
!BUILD_DEFS.DISABLE_JSEP ? 'ort-wasm-simd-threaded.jsep.mjs' :
|
|
'ort-wasm-simd-threaded.mjs';
|
|
const wasmModuleUrl = urlOverride ?? normalizeUrl(wasmModuleFilename, prefixOverride);
|
|
// need to preload if all of the following conditions are met:
|
|
// 1. not in Node.js.
|
|
// - Node.js does not have the same origin policy for creating workers.
|
|
// 2. multi-threaded is enabled.
|
|
// - If multi-threaded is disabled, no worker will be created. So we don't need to preload the module.
|
|
// 3. the absolute URL is available.
|
|
// - If the absolute URL is failed to be created, the origin cannot be determined. In this case, we will not
|
|
// preload the module.
|
|
// 4. the worker URL is not from the same origin.
|
|
// - If the worker URL is from the same origin, we can create the worker directly.
|
|
const needPreload = !isNode && isMultiThreaded && wasmModuleUrl && !isSameOrigin(wasmModuleUrl, prefixOverride);
|
|
const url =
|
|
needPreload ? (await preload(wasmModuleUrl)) : (wasmModuleUrl ?? fallbackUrl(wasmModuleFilename, prefixOverride));
|
|
return [needPreload ? url : undefined, await dynamicImportDefault<EmscriptenModuleFactory<OrtWasmModule>>(url)];
|
|
};
|