### Description
This PR is a preview of cherry-picks for ort-web to `rel-1.17.3` based
on `rel-1.17.2`.
<details>
<summary>Changes of ort-web to cherry-pick</summary>
The following commits are from main branch.
`o` stands for pick, and `x` stands for skip.
```
o 2e0a388c36 [js/webgpu] Add HardSigmoid support (#19215)
o d226e40856 [js/webgpu] set query type in onRunStart (#19202)
o 61610ff986 [js/webgpu] Add FusedConv clip test case (#18900)
o a33b5bd1fa [JS/WebGPU] Added Uniforms to SkipLayerNorm. (#18788)
o 591f90c0b9 [js/webgpu] Fix issue of timestamp query (#19258)
o 7252c6e747 [WebNN EP] Support WebNN async API with Asyncify (#19145)
o 5b06505073 [js/webgpu] Fix Tanh explosion (#19201)
o 656ca66186 [js/webgpu] Support uniforms for conv, conv transpose, conv grouped (#18753)
o a3f0e2422b [js/webgpu] Support f16 uniform (#19098)
o 9e69606360 fix f16 for attention, enable slice and flatten for more types (#19262)
o 624b4e2063 [js/webgpu] Remove enableShapesUniforms (#19279)
o 90883a366a [js/webgpu] Add hardSigmoid activation for fusedConv (#19233)
o 85cef0af8c [js/webgpu] Support capture and replay for jsep (#18989)
o d73131cf0f [js/webgpu] Use DataType as uniform cpu type (#19281)
o dd1f6ccc45 [js/webgpu] resolve codescan alert (#19343)
o 3a2ab1963a [js/webgpu] Refactor createTensorShapeVariables (#18883)
o efc17e79de [js/webgpu] Fix the undefined push error (#19366)
x 50806a7dd5 [js/web] support external data in npm test (#19377)
o ccbe264a39 [js/webgpu] Add LeakyRelu activation for fusedConv (#19369)
o 5ff27ef02a [js/webgpu] support customop FastGelu (#19392)
x 03be65e064 [js/web] fix types exports in package.json (#19458)
o 06269a3952 [js/webgpu] allow uint8 tensors for webgpu (#19545)
o dfeda9019c [JS/WebGPU] Add MatMulNBits (#19446)
o 1b48054e1b [js/webgpu] Create Split indices helpers by rank, not by shape (#19554)
o 3fe2c137ee [js] small fix to workaround formatter (#19400)
x 70567a4b3a [js/web] use ApiTensor insteadof onnxjs Tensor in TensorResultValidator (#19358)
o 6e04e36e3f [js/common] upgrade tsc in common from 4.9.5 to 5.2.2 (#19317)
o 58f4921686 [js] changes to allow Float16Array if any polyfill is available (#19305)
o 57d6819212 [js/web] Fix fused-conv is not included in npm test (#19581)
o ebd220b073 Misspelling in README.md (#19433)
o 38c3432393 Bump ip from 1.1.8 to 1.1.9 in /js/react_native (#19582)
o fe82fccf1a [js/webgpu] Fix Conv2DTransposeMatMul f16 compilation failure (#19596)
o 76a2a487a1 Bump ip from 1.1.8 to 1.1.9 in /js/react_native/e2e (#19583)
o 29b1106033 [node] Switch to setImmediate to avoid starving the Node.js event loop (#19610)
o ae3d73c981 [JS/WebGPU] Fix Split and Where to handle corner cases. (#19613)
o aec2389ad0 [js/webgpu] allows a ProgramInfo's RunData to use zero sized output (#19614)
o bb43a0f133 [js/webgpu] minor fixes to make tinyllama work (#19564)
o 0edb035808 [js/web] fix suite test list for zero sized tensor (#19638)
o 3cb81cdde2 [js/common] move 'env.wasm.trace' to 'env.trace' (#19617)
o e30618d055 [js/webgpu] use Headless for webgpu test by default (#19702)
o f06164ef8b [js/web] transfer input buffer back to caller thread (#19677)
x a788514027 [js/web] dump debug logs for karma for diagnose purpose (#19785)
o 24b72d2613 [JS/WebGPU] Preserve zero size input tensor dims. (#19737)
o 4538d31a8b [js/webgpu] expose a few properties in WebGPU API (#19857)
o 53de2d8cb0 [js/webgpu] Enable GroupedConvVectorize path (#19791)
o ed250b88c3 [JS/WebGPU] Optimize MatMulNBits (#19852)
x e771a763c3 [js/test] align web test runner flags with ort.env (#19790)
o 79e50aeef3 [js/web] rewrite backend resolve to allow multiple EPs (#19735)
o acb0df2280Fix#19931 broken Get Started link of "ONNX Runtime JavaScript API" page (#19932)
o b29849a287 [js/common] fix typedoc warnings (#19933)
o afdab62f53 Bump follow-redirects from 1.15.4 to 1.15.6 in /js/web (#19949)
o 28ad6c3955 Bump follow-redirects from 1.15.4 to 1.15.6 in /js/node (#19951)
o 7e0d424934 accumulate in fp32 for Reduce* (#19868)
o 4c6a6a37f7 [js/webgpu] Fix NAN caused by un-initialized buffer in instance-norm (#19387)
o 01c7aaf6aa [js/webgpu] allow setting env.webgpu.adapter (#19940)
o c45cff60cf [js/webgpu] fix maxpool / fp16 (#19981)
```
</details>
<details>
<summary>Cherry-pick commandlines</summary>
```sh
git cherry-pick 2e0a388c36
git cherry-pick d226e40856
git cherry-pick 61610ff986
git cherry-pick a33b5bd1fa
git cherry-pick 591f90c0b9
git cherry-pick 7252c6e747
git cherry-pick 5b06505073
git cherry-pick 656ca66186
git cherry-pick a3f0e2422b
git cherry-pick 9e69606360
git cherry-pick 624b4e2063
git cherry-pick 90883a366a
git cherry-pick 85cef0af8c #<<<<< Note: conflicts
git cherry-pick d73131cf0f
git cherry-pick dd1f6ccc45
git cherry-pick 3a2ab1963a
git cherry-pick efc17e79de
git cherry-pick ccbe264a39
git cherry-pick 5ff27ef02a
git cherry-pick 06269a3952
git cherry-pick dfeda9019c
git cherry-pick 1b48054e1b
git cherry-pick 3fe2c137ee
git cherry-pick 6e04e36e3f
git cherry-pick 58f4921686
git cherry-pick 57d6819212
git cherry-pick ebd220b073
git cherry-pick 38c3432393
git cherry-pick fe82fccf1a
git cherry-pick 76a2a487a1
git cherry-pick 29b1106033
git cherry-pick ae3d73c981
git cherry-pick aec2389ad0
git cherry-pick bb43a0f133
git cherry-pick 0edb035808
git cherry-pick 3cb81cdde2
git cherry-pick e30618d055
git cherry-pick f06164ef8b
git cherry-pick 24b72d2613
git cherry-pick 4538d31a8b
git cherry-pick 53de2d8cb0
git cherry-pick ed250b88c3
git cherry-pick 79e50aeef3
git cherry-pick acb0df2280
git cherry-pick b29849a287
git cherry-pick afdab62f53
git cherry-pick 28ad6c3955
git cherry-pick 7e0d424934
git cherry-pick 4c6a6a37f7
git cherry-pick 01c7aaf6aa
git cherry-pick c45cff60cf
```
</details>
<details>
<summary>Cherry-pick conflicts</summary>
- 85cef0af8c#18989
this change is for enabling graph capture feature for JSEP, and it is
done after ROCM EP enabled graph capture feature. However, the ROCM EP
graph capture feature is not cherry-picked in rel-1.17.2.
</details>
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Jiajia Qin <jiajia.qin@intel.com>
Co-authored-by: Xu Xing <xing.xu@intel.com>
Co-authored-by: satyajandhyala <satya.k.jandhyala@gmail.com>
Co-authored-by: Yang Gu <yang.gu@intel.com>
Co-authored-by: Wanming Lin <wanming.lin@intel.com>
Co-authored-by: Jiajie Hu <jiajie.hu@intel.com>
Co-authored-by: Guenther Schmuelling <guschmue@microsoft.com>
Co-authored-by: Matttttt <18152455+martholomew@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Segev Finer <segev208@gmail.com>
Co-authored-by: Belem Zhang <belem.zhang@intel.com>
### Description
This PR contains a few changes in /js/common/ to support a coming PR for
a full implementation of webgpu IO binding.
- allows pass-through if value is already a Tensor instance in return
value of `handler.run()` called by `InferenceSession.run()`
(inference-session-impl.ts). Specifically, onnxruntime-node and
onnxruntime-react-native uses native bindings to generate a Tensor-like
object so we need to create a real Tensor instance here; for
onnxruntime-web the return value is already a Tensor instance.
- adds new types for GPU buffer supported types: `'float32'|'int32'` ->
`'float32'|'float16'|'int32'|'int64'|'uint32'|'bool'`
- exposes types `GpuBufferDataTypes` together with `CpuPinnedDataTypes`
and `TextureDataTypes` as exported
### Description
clean up JSDoc for onnxruntime-common:
- replace "@internal" to "@ignore" as JSDoc do not use "@internal".
Using "@ignore" will let the content not show on the generated doc.
[//]: # (## Work In Progress. Feedbacks are welcome!)
### Description
This PR adds a few properties, methods and factories to Tensor type to
support IO-binding feature. This will allow user to create tensor from
GPU/CPU bound data without a force transferring of data between CPU and
GPU.
This change is a way to resolve#15312
### Change Summary
1. Add properties to `Tensor` type:
a. `location`: indicating where the data is sitting. valid values are
`cpu`, `cpu-pinned`, `texture`, `gpu-buffer`.
b. `texture`: sit side to `data`, a readonly property of `WebGLTexture`
type. available only when `location === 'texture'`
c. `gpuBuffer`: sit side to `data`, a readonly property of `GPUBuffer`
type. available only when `location === 'gpu-buffer'`
2. Add methods to `Tensor` type (usually dealing with inference
outputs):
- async function `getData()` allows user to download data from GPU to
CPU manually.
- function `dispose()` allows user to release GPU resources manually.
3. Add factories for creating `Tensor` instances:
a. `fromTexture()` to create a WebGL texture bound tensor data
b. `fromGpuBuffer()` to create a WebGPUBuffer bound tensor data
c. `fromPinnedBuffer()` to create a tensor using a CPU pinned buffer
### Examples:
create tensors from texture and pass to inference session as inputs
```js
// when create session, specify we prefer 'image_output:0' to be stored on GPU as texture
const session = await InferenceSession.create('./my_model.onnx', {
executionProviders: [ 'webgl' ],
preferredOutputLocation: { 'image_output:0': 'texture' }
});
...
const myImageTexture = getTexture(); // user's function to get a texture
const myFeeds = { input0: Tensor.fromTexture(myImageTexture, { width: 224, height: 224 }) }; // shape [1, 224, 224, 4], RGBA format.
const results = await session.run(myFeeds);
const myOutputTexture = results['image_output:0'].texture;
```
### Description
allow creating (u)int64 tensors from either a number array or a bigint
array.
before:
```js
// TypeScript think is good, but actually does not work
// runtime error: Uncaught TypeError: Cannot convert 1 to a BigInt
const myTensor1 = new Tensor('int64', [1, 2, 3, 4], [2, 2]);
// runtime good, but TypeScript thinks myTensor2 is a string tensor
const myTensor2 = new Tensor('int64', [1n, 2n, 3n, 4n], [2, 2]);
```
after:
```js
// both work at runtime and TypeScript populates the correct types
const myTensor1 = new Tensor('int64', [1, 2, 3, 4], [2, 2]);
const myTensor2 = new Tensor('int64', [1n, 2n, 3n, 4n], [2, 2]);
```
### Description
<!-- Describe your changes. -->
refactor tensor type in onnxruntime-common.
### Motivation and Context
There major motivation is that I am doing a local change to address the
API part of #15312. And I am doing a refactoring of onnxruntime-common
anyway (#15772).
The `tensor.ts` and `tensor-impl.ts` are too large, so I split contents
into multiple files to make the type declarations clearer.
The original target of this change is for API only ( ie. do not refactor
any implementation.). However, there are a few type/implementation
inconsistencies so I also made minimal changes to fix them.
### Changes
- extract `TensorUtils` for non-template interfaces
- extract `TensorFactory` for all overloads of `Tensor.fromImage()`
- refactor options type that used for `Tensor.fromImage()`
- fix JSDoc comments to make option descriptions consistent with actual
type declarations
- fix an inconsistency for `options.format` and `options.bitmapFormat`;
change all `bitmapFormat` to `format`
- extract `ConversionUtils` for `tensor.toDataURL()` and
`tensor.toImageData()`
- put implementations into multiple files from `tensor-impl.ts`
- fix a bug that cause unittest fail. put comments for future fix.
We implemented a number of new ops and data types to support running
segment anything model on Chromium WebNN DML backend (POC) in a forked
branch https://github.com/honry/onnxruntime/tree/stable-diffusion
In this PR, we migrate the changes in the forked branch to main branch,
includes:
- 22 new ops
- New tensor data types: bool, int32, uint32, uint64, int64, float16 (As
JavaScript hasn't shipped Float16Array, we use Uint16Array as a
workaound)
- Handle empty input tensors and duplicated outputs
- Fixed some nits
### Description
Enabled the use of per channel Bias and Mean normalization when converting an image <--> tensor.
Added a few bug fixes and updates to the relevant E2E tests.
---------
Co-authored-by: shalvamist <shalva.mist@microsoft.com>
### Description
This change delays the execution of checking whether bigint is available
in the context. This allows polyfill for
`BigInt64Array`/`BigUint64Array` (if there is any)
### Description
Data processing capabilities to ORT Web.
This PR will focus augmenting raw data to and from Tensors.
### Motivation and Context
Enabling different app building use cases to leverage ORT in a more
natural form.
Currently, the user needs to process the data and call Tensor
constructors - these util functions will provide a direct path to
generating ORT tensors.
Co-authored-by: shalvamist <shalva.mist@microsoft.com>
* working on re-organizing js code for ortweb
* remove dup files
* move folder
* fix common references
* fix common es5
* add webpack to common
* split interfact/impl
* use cjs for node
* add npmignore for common
* update sourcemap config for common
* update node
* adjust folder/path in CI and build
* update folder
* nit: readme
* add bundle for dev
* correct nodejs paths
* enable ORT_API_MANUAL_INIT
* set name for umd library
* correct name for commonjs export
* add priority into registerBackend()
* fix npm ci pwd
* update eslintrc
* revise code
* revert package-lock lockfileVersion 2->1
* update prebuild
* resolve comments
* update document
* revise eslint config
* update eslint for typescript rules
* revert changes by mistake in backend.ts
* add env
* resolve comments
