saymrwulf/onnxruntime
1
0
Fork 0
mirror of https://github.com/saymrwulf/onnxruntime.git synced 2026-07-09 17:28:58 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
8290 commits 2856 branches 57 tags 1.3 GiB
Commit graph

8290 commits

This branch
This branch
All branches
Author SHA1 Message Date
pengwa
448e989df8
Op slicing upstream refactor (#14832) 
### Slice op upstream refactor

A refactor work for https://github.com/microsoft/onnxruntime/pull/13672.

### Motivation and Context

There is a similar optimization opportunity for other operator
upstreaming, to reduce compute flops. So refactor the existing code base
for making it easier to support other ops.

The changes in this PR are mainly about renaming and moving. 
- Move common logic (from compute_optimizer.h/cc) into
upstream_transformer_base.h/cc and shared_utils.h/cc.
- For upstream common logic, they are moved into
upstream_transformer_base.h/cc
   - For shared utilities, they are moved to shared_utils.h/cc.
- After the move, compute_optimizer.h/cc mainly for upstreaming gather
implementation (inheriting upstream_transformer_base.h/cc). Ideally it
should be renamed, but for easier review this time, I keep its name.
 2023-03-13 08:19:32 +08:00
Yi-Hong Lyu
cce9e0eaad
Add float32 hardsigmoid tests (#14948) 2023-03-12 10:56:29 -07:00
G. Ramalingam
930e009567
[WIP] Update call to GetFunction (#14949) 
### Description

OpSchema::GetFunction() changed in ONNX to support
opset-version-dependent function-body. Update the call to GetFunction
appropriately.

### Motivation and Context

Motivated by https://github.com/microsoft/onnxruntime/issues/14810

---------

Signed-off-by: Ganesan Ramalingam <grama@microsoft.com>
 2023-03-11 07:04:17 -08:00
Yi Zhang
ca315b9148
Use ADO cache to cache docker image instead of ACR (#14496) 
### Description
Now, we only enable image cache in pipeline cache for Linux Aten
Pipeline.
It'll be enabled in other Linux pipelines gradually.

### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->

Fixed
[AB#13143](https://aiinfra.visualstudio.com/6a833879-cd9b-44a4-a9de-adc2d818f13c/_workitems/edit/13143)


### Verification
1. No Image Cache in Pipeline

https://dev.azure.com/onnxruntime/onnxruntime/_build/results?buildId=904531&view=results
2. Use Cached Image in Pipeline

https://dev.azure.com/onnxruntime/onnxruntime/_build/results?buildId=904533&view=results
 2023-03-11 10:32:02 +08:00
Vincent Wang
7950189920
[CUDA] Optimize Perf for AtomicAdd of Half Type (#14992) 2023-03-11 08:52:01 +08:00
Changming Sun
a8ad0edbeb
BUG FIX: the if...else in telemetry-steps.yml does not really work (#14972) 
### Description
BUG FIX: the if...else in telemetry-steps.yml does not really work. It
always says "Telemetry is disabled." even through the pipeline doesn't
have the pipeline variable.

### Motivation and Context
For example, recently I setup a new pipeline in
https://dev.azure.com/onnxruntime/onnxruntime/_build without setting the
ADO variable, but the powershell code still thinks that we have enabled
telemetry.

See:

https://dev.azure.com/onnxruntime/onnxruntime/_build/results?buildId=910107&view=results

The reason it didn't work because when the pipeline
variable("TELEMETRYGUID") doesn't exist,  the occurrence of
 "$(TELEMETRYGUID)" would be not replace to anything. It will remain as
it is.
 2023-03-10 15:39:07 -08:00
Adrian Lizarraga
d8ddd25272
Add InstanceNormalization operator to QNN EP (#14867) 
### Description

QNN EP:
- Adds the
[InstanceNormalization](https://onnx.ai/onnx/operators/onnx__InstanceNormalization.html)
operator to QNN EP.
- Fixes graph composition bug when Transpose node is the last node in a
graph.
- Adds check for input shape when GetCapability is called (before and
after layout transformation)
- Should add similar checks for other layout sensitive ops (conv, pool,
...) in a separate PR
- Adds initial QNN op tests for QDQ conv and QDQ  InstanceNormalization
  - Should add tests for other ops in a separate PR

Optimizer:
- Makes InstanceNormalization a layout sensitive operator.
- Adds a custom QDQ group selector for InstanceNormalization.

Quantization tool:
- Adds QDQ support for InstanceNormalization operator.
- Adds python unit test for InstanceNormalization quantization.

### Motivation and Context
Needed to support stable diffusion models with QNN.

---------

Co-authored-by: Hector Li <hecli@microsoft.com>
 2023-03-10 14:42:41 -08:00
Ryan Hill
a5c436e148
Fix prefast warnings (#14975) 
### Description

In transpose.cc:
Arithmetic overflow: Using operator '-' on a 4 byte value and then
casting the result to a 8 byte value. Cast the value to the wider type
before calling operator '-' to avoid overflow (io.2).

In cuda_provider_factory.h:
The type 'struct onnxruntime::CUDA_Provider' with a virtual function
needs either public virtual or protected non-virtual destructor (c.35).
 2023-03-10 14:31:55 -08:00
Dmitri Smirnov
0d7855ea5a
Re-work global objects dependancies in pybind layer. (#14941) 
### Description
Re-work handling of static objects in pybind.
Make sure we ref-count Environment from Sessions.

The following has been done:

- Make global objects function static. This ensures that the objects are
constructed on demand. The first object constructed is destructed last.
This is platform independent.
- Make global objects ownership shared as suggested by pybind since they
are not surfaced at Python level, and they cannot be referred to by
dependent python objects. Verified that all python objects are GCed
before globals are destroyed. This takes care of inference session
dependency on environment and its default logger and this is also
platform independent.
- Utilize pybind atexit mechanism to clear execution providers and
unload CUDA libraries (as suggested by
https://github.com/microsoft/onnxruntime/pull/14903) . Since this is
registered for module exit, it takes place before any other global are
destroyed and clears shared objects state or even unloads the libraries.
This should also work in a platform independent way.

### Motivation and Context

- Global object destruction order is managed manually and that becomes
source of trouble. We want to make it deterministic and platform
independent.
- Frequent hangs in Python layer due to the static object's destruction
order. Some of the Python session objects are being garbage collected
after main exits and they require ORT environment to be alive. (Use
after free)
 2023-03-10 13:55:31 -08:00
Adrian Lizarraga
e2febe87f6
[QNN EP] Update QNN SDK to 2.8 (#14978) 
### Description
- Add QNN 2.8 SDK
- Make QNN SDK version a pipeline template parameter for QNN pipelines.

### Motivation and Context
Updates to latest QNN SDK version, and allows testing different QNN SDK
versions without modifying yaml files.
 2023-03-10 13:21:19 -08:00
Edward Chen
bd142bfb04
Gradle clean up (#14973) 
- Use java/gradlew directly in .github/workflows/publish-java-apidocs.yml.
- Remove use of deleted step from tools/ci_build/github/azure-pipelines/android-arm64-v8a-QNN-crosscompile-ci-pipeline.yml.
- Remove Gradle installations and PATH updates from Dockerfiles and scripts. Now Gradle wrapper is used so a system Gradle installation is not needed.
 2023-03-10 10:50:32 -08:00
Baiju Meswani
748758c135
Address issue with uninitialized variable (#14988) 2023-03-10 09:24:04 -08:00
Maximilian Müller
ad4db12699
TensorRT EP - timing cache (#14767) 
### Description

This will enable a user to use a TensorRT timing cache based on #10297
to accelerate build times on a device with the same compute capability.
This will work across models as it simply store kernel runtimes for
specific configurations. Those files are usually very small (only a few
MB) which makes them very easy to ship with an application to accelerate
the build time on the user end.

### Motivation and Context
Especially for workstation use cases TRT build times can be a roadblock.
With a few model from ONNX model zoo i evaluated speedups when a timing
cache is present.
`./build/onnxruntime_perf_test -e tensorrt -I -t 5 -i
"trt_timing_cache_enable|true" <onnx_path>`

|Model | no Cache | with Cache|
| ------------- | ------------- | ------------- |
|efficientnet-lite4-11 | 34.6 s | 7.7 s|
|yolov4 | 108.62 s | 9.4 s|

To capture this is had to modify the onnxruntime_perf_test. The time is
sometimes not captured within "Session creation time cost:" which is why
i introduced "First inference time cost:".

---------

Co-authored-by: Chi Lo <Chi.Lo@microsoft.com>
 2023-03-10 09:02:27 -08:00
Yi Zhang
acbb7ad453
enable cache in orttraining-mac-ci (#14979) 
### Description
enable compilation cache  in orttraining-mac-ci

### Motivation and Context
The workflow duration can be reduced to 12 minutes from about 100
minutes at best.

https://dev.azure.com/onnxruntime/onnxruntime/_build/results?buildId=911536&view=results
 2023-03-10 07:34:25 +08:00
Yulong Wang
1187d4ade6
[wasm] extend build timeout for static lib (#14952) 
### Description
extend build timeout for web assembly static lib.
 2023-03-09 15:03:34 -08:00
Preetha Veeramalai
79d47c1530
Enable sorting of initializers (#14631) 
Add intializers to model proto in sorted order.



### Motivation and Context
Onnxruntime OpenVino Execution Provider interacts with Openvino API by
passing onnx serialised model proto.
Current flow is that onnx serialised model proto will be passed into
Read_model() API of OpenVino that creates an OpenVino execution network
thats passed to compile_model() API.

As part of optimizations we have combined the API's (Read_model and
Compile_model) into single compile_model() API that directly accepts
serialized onnx model proto. A hash function will be computed on this
serialized input for internal Openvino optimizations. This requires the
model_proto to be deterministic during each inference requests.

With the current flow, the [initializers are added to
model_proto](c1ff4b468d/onnxruntime/core/graph/graph_proto_serializer.cc (L48))
from an [unordered_map data
structure](8ed3dfe063/onnxruntime/core/providers/shared_library/provider_interfaces.h (L93))
that brings in random ordering of these initializers for inference runs.


The proposed solution is to add these initializers by iterating through
a sorted[ vector consisting of the initializer
names](2c7146cef8/onnxruntime/core/graph/graph_proto_serializer.cc (L49)).
 2023-03-09 12:12:46 -08:00
Jian Chen
b4fe98ac2e
Update to MacOS-12 (#14924) 
### Description
<!-- Describe your changes. -->


Update to MacOS-12
### Motivation and Context

Fixed
[AB#13233](https://aiinfra.visualstudio.com/6a833879-cd9b-44a4-a9de-adc2d818f13c/_workitems/edit/13233)
 2023-03-09 10:18:14 -08:00
cloudhan
51b67fa15c
Make ROCm Attention biased+masked and biased+nomask scaling logic consistent (#14976) 
The biased+masked and biased+nomask have different scaling logic in current ROCm implementation

Currently,

biased + masked:  (QK'+ bias) * scale + convert(mask)
biased + nomask:   QK' * scale + bias

which is not correct. What we want is

  QK' * scale [+ bias]

That is, bias should not be scaled.

This effectively follows
https://github.com/microsoft/onnxruntime/pull/14517/files?w=1#diff-e4768ce15a73499f584f9cd7d71adcb1ff2ed8d68ad7e496723a4775cbc35e33
 2023-03-09 23:37:50 +08:00
mindest
f83923d5df
fix rocBLAS extensions API issue; add batched- and strided_batched- cases (#14883) 
### Description
For rocBLAS extensions API:
* fix `alpha`/`beta` dtype mismatch in `rocblas_gemm_ex()`, which should
be the same as `compute_type`.
* add support for `BatchedGemm` and `StridedBatchedGemm` cases.
 2023-03-09 23:23:35 +08:00
mindest
bf2cc808a1
[ROCm] SkipLayerNorm: add more configs for block size; loosen constraints (#14900) 
### Description
* add more configs for `threads_per_block` in SkipLayerNorm, also in
kernel explorer.
* loosen constraints for hidden_size, so that `SkipLayerNormSmallOp` can
be selected for larger hidden sizes.
* add flag for optional output in kernel_explorer


### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
 2023-03-09 22:27:01 +08:00
Yi Zhang
d55ae490e1
detach patch manylinux from get_docker_image (#14958) 
### Description
Make patch manylinux one single step.


### Motivation and Context
If we want to use hash of docker-related files as the cache key, the
files should keep consistent before and after docker build.
And changes in generated build_scripts should trigger rebuilding the
image as well.
 2023-03-09 15:40:58 +08:00
zhijiang
80e25ad6ac
fix cg issue (#14372) 
### Description
tensorboard depends on rsa>=3.1.4, while rsa 4.5 has vuln issue, so pin
it to higher version as suggested

Fixed
[AB#7352](https://aiinfra.visualstudio.com/6a833879-cd9b-44a4-a9de-adc2d818f13c/_workitems/edit/7352)



### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
 2023-03-09 15:28:11 +08:00
Yulong Wang
3c4efd2e77
[js/common] allows polyfill for bigint (#14921) 
### Description
This change delays the execution of checking whether bigint is available
in the context. This allows polyfill for
`BigInt64Array`/`BigUint64Array` (if there is any)
 2023-03-08 15:29:04 -08:00
Yulong Wang
8844474083
[js] remove 'npm bin' (#14943) 
### Description
'npm bin' is deprecated in latest version. use 'npx' instead. 

This PR resolves #14934
 2023-03-08 15:03:27 -08:00
Ye Wang
d8d96f0788
Fix a build issue (#14944) 
### Description
<!-- Describe your changes. -->



### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->

https://github.com/microsoft/onnxruntime/issues/14940
 2023-03-08 13:05:49 -08:00
Edward Chen
c46c7ccba5
Update Gradle version (#14862) 
- Update Gradle version used in most places from 6.8.3 to 8.0.1. Update Android Gradle Plugin version where applicable.
  Not updated in this change: React Native Android projects (under `js/react_native/`). That can be done later along with updating the React Native projects.

- Add Gradle wrapper in `java/` to make it easier to consistently use a specific Gradle version.
 2023-03-08 12:22:06 -08:00
Changming Sun
d9436407b6
Use safe allocator for JNI code (#13999) 
### Description
Use a customized allocarray function to replace the original malloc
calls to avoid integer overflow.

### Motivation and Context
Fix Prefast warnings. 

Fixed
[AB#8990](https://aiinfra.visualstudio.com/6a833879-cd9b-44a4-a9de-adc2d818f13c/_workitems/edit/8990)
Fixed
[AB#8991](https://aiinfra.visualstudio.com/6a833879-cd9b-44a4-a9de-adc2d818f13c/_workitems/edit/8991)
Fixed
[AB#9016](https://aiinfra.visualstudio.com/6a833879-cd9b-44a4-a9de-adc2d818f13c/_workitems/edit/9016)
 2023-03-08 11:40:55 -08:00
Adam Pocock
47f00b5d49
[Java] Initial on device training support (#14027) 
contributor: @Craigacp
 2023-03-08 10:01:08 -08:00
Ashwini Khade
f14ab63c19
fix prefast warnings (#14931) 
### Description
Fixes prefast warnings

Fixed
[AB#11328](https://aiinfra.visualstudio.com/6a833879-cd9b-44a4-a9de-adc2d818f13c/_workitems/edit/11328)
Fixed
[AB#11329](https://aiinfra.visualstudio.com/6a833879-cd9b-44a4-a9de-adc2d818f13c/_workitems/edit/11329)
 2023-03-08 09:49:15 -08:00
Hariharan Seshadri
112a4d215a
[CUDA] Support decoding multihead self-attention implementation (#14848) 2023-03-08 09:17:54 -08:00
Kyushick Lee
c696392f0c
Support external output tensors for DORT (#14516) 
### Description
<!-- Describe your changes. -->
Support externally-managed output tensors (torch Tensors) for dort. 
Add `preallocate_output` option to OrtBackend to rely on
externally-managed output tensors for dort.


### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
DORT currently allocates and returns output ortvalues and convert them
to torch Tensors. The conversion based on dlpack does not support torch
Tensors for custom Aten backends, and it is not yet possible to transfer
the ownership from ortvalue to external handle (torch Tensor).

To avoid this issue, the PR change provides an option
(`preallocate_output`) to allocate output tensors externally in pytorch,
which creates torch Tensor for an Aten backend, and let dort take
pointers from torch Tensors to construct output ortvalues instead of
allocating them inside InferenceSession.
 2023-03-07 21:32:23 -08:00
edgchen1
2ef25a2200 Update CODEOWNERS file. 2023-03-07 17:56:37 -08:00
edgchen1
5b3f79a11a Add gradle wrapper validation workflow. 2023-03-07 17:56:37 -08:00
Ashwini Khade
f71ac9859e
Update acpt image in the training pipeline (#14855) 
### Description
Current pipeline refers to an old image which is causing test failures.
Updating the image to the latest one.



### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
Fixes pipeline failure:
https://dev.azure.com/onnxruntime/onnxruntime/_build?definitionId=198
- If it fixes an open issue, please link to the issue here. -->
 2023-03-07 14:10:32 -08:00
pengwa
5d8ce817cb
Fix simplified layer norm fusion for training (#14866) 
### Fix simplified layer norm fusion for training

Co-author with @prathikr.

Fix bug identified by @prathikr.
https://github.com/microsoft/onnxruntime/issues/14822.

Running T5 model enabling deepspeed, we see simplified layer norm is not
fused because the device check did not pass

b7fde84341/onnxruntime/core/optimizer/layer_norm_fusion.cc (L568).
Since during pretraining optimization pass, there is no device
placement, so the device check not fulfilled is expected.

On the other hand, the device check is still valid to avoid simplified
layer norm fusion works correctly for CPU runs. As a mitigation, added a
flag to indicate whether the fusion is triggered by pre-training
optimization or not. There is a risk though, when we run ORTModule
training with CPU EP, but I feel the risk can be much reduced if we
check CUDA/ROCM is enabled for the build.

```
CUDA_VISIBLE_DEVICES=0 python examples/onnxruntime/training/summarization/run_summarization.py --model_name_or_path t5-small --do_train --dataset_name cnn_dailymail --dataset_config "3.0.0" --source_prefix "summarize: " --predict_with_generate --overwrite_output_dir --output_dir /bert_ort/pengwa/output --fp16 --max_steps 1 --logging_steps 1 --deepspeed aml_ds_config_zero_1.json
```

### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
 2023-03-07 13:59:20 -08:00
Patrice Vignola
65f1f840f6
[DML EP] Fix Attention regression caused by removing transposes (#14908) 
By removing the transposes and using strides instead, the metacommands
are not able to be reached anymore since it's not using NCHW layout.
 2023-03-07 11:17:28 -08:00
Xavier Dupré
6b604521a6
Fix tree implementation when left, right node have lower index (#14839) 
### Description
Previous implementation did not support left or right node of a node to
have an index lower than the node itself. This condition would forbid
the tree to enter an infinite loop. Lightgbm does not follow that rule.
The changes do not change the algorithm but remove the test enforcing
that condition.



### Motivation and Context
It fixes a regression introduced by #14670.
 2023-03-07 19:47:12 +01:00
Hitesh Shah
66101c02a2 Implement AllToAll collective op 2023-03-07 10:17:07 -08:00
Adam Pocock
150043f74f
Adds a Java accessor for GetVersionString (#14876) 
### Description
Java part of #14873.
 2023-03-07 09:46:56 -08:00
Xavier Dupré
5930e7e22f
Introduce RemovableAttributes (#14868) 
### Description
TreeEnsemble* kernels fully copies all the parameters from the onnx
graph. Even if they are no longer needed or unused (hitrates), they
remain in memory. For big models >= 200 trees, max_depth > 10, the model
usually weights more than 10 Mb. This change offers a kernel the
possibility to remove all unneeded attributes after they were used to
create the session. Attributes are deleted after the model was possibly
saved, at the of the session creation.

The current design is to be debatted:
* it stored the list of removable attributes in class
`onnxruntime::Node`,
* the node is marked as `const` everytime this implementation needs to
register the name of a removable attribute or to remove them.

The current implementation is just a POC as it needs to cast
`onnxruntime::Node*` into `const onnxruntime::Node*`.

Should we keep the list of removable attributes in `onnxruntime::Node`?

### Motivation and Context
Motivation is mostly to reduce memory consumption.

---------

Signed-off-by: xadupre <xadupre@microsoft.com>
 2023-03-07 12:37:12 +01:00
JiCheng
be1416d032
prefast C26451 (#14933) 
Fixed
[AB#13290](https://aiinfra.visualstudio.com/6a833879-cd9b-44a4-a9de-adc2d818f13c/_workitems/edit/13290)
 2023-03-07 15:16:50 +08:00
Changming Sun
3e08a67dd6
Add Linux ARM64 CI pipeline (#14904) 2023-03-06 21:47:10 -08:00
Adrian Lizarraga
d45b47945c
Linux QNN Pipeline: fix build error reporting (#14922) 
### Description
Split up the ORT build step in the Linux QNN CI Pipeline.


### Motivation and Context
Build errors were not being immediately reported at the end of the build
step. The build step currently concatenates multiple shell commands, and
the return code for the last (mkdir) was being reported. This PR ensures
that the return code of the `python build.py ...` command is reported
for the build step.
 2023-03-06 17:49:35 -08:00
Sheil Kumar
f88b97ede2
Cast to int32_t->size_t to avoid prefast overflow warning (#14902) 
Cast to int32_t->size_t to avoid prefast overflow warning
 2023-03-05 06:21:46 -08:00
Tianlei Wu
6c8538f086
Fix prefast warning (#14895) 
Fix a prefast warning: `The const variable 'spatial_dim_start' can be
computed at compile-time. Consider using constexpr (con.5).`
 2023-03-03 12:54:28 -08:00
Changming Sun
c1155b70c5
Remove 37 and 50 from CUDA compute archs (#14874) 
### Description
To reduce CUDA package's size a little bit. 37 is for Tesla K80. Azure's
NC-series uses it, but in most cases CUDA can dynamic generate device
code .
 2023-03-03 12:24:21 -08:00
George Wu
289f7dbcdd
enable pybind for qnn ep (#14897) 
enable python bindings for QNN EP.
tested on Windows Dev Kit 2023 (ARM64) with python 3.11 (ARM64) from 
https://www.python.org/ftp/python/3.11.1/python-3.11.1-arm64.exe
 2023-03-03 07:26:53 -08:00
pengwa
f6c81d8aca
Introduce padding inspector in ORTModule (#14652) 
### Introduce padding inspector in ORTModule

In some Transformer-based LLM training recipes, high data sparsity is
observed due to 1). token padding (to max sequence length), 2). labels
contains many ignore_index for calculate loss.

This PR introduces a switch to enable data sparsity inspection, which 
1). in short term, can inform training users to use techniques like
dynamic batching to amortize the issue.
2). in medium and longer term, also helps us (training team) to have
better understanding what our training customers' models looks like from
perspective of data sparsity (and potentially motivate us to improve
with runtime).

Here is an example of different data sparsity with same training model
arch, same training input, but with different user models.

**Low Embed Density, High Label Density Case - Sentence Classification**
`
python -m torch.distributed.launch --nproc_per_node=4
examples/onnxruntime/training/text-classification/run_glue.py
--model_name_or_path roberta-large-openai-detector --task_name mnli
--do_train --do_eval --max_seq_length 128 --per_device_train_batch_size
32 --learning_rate 2e-5 --num_train_epochs 3 --overwrite_output_dir
--output_dir ./outputs/ --per_device_eval_batch_size 32 --seed 1137
--fp16 True --ignore_mismatched_sizes True --optim adamw_ort_fused
`
```
>>>Valid token/label density (e.g. valid/total) in passing 10 steps:
        | STEP       | INPUT TYPE |  INPUT NAME     | PAD IDX    | DENSITY    | VALID TOKENS    | TOTAL TOKENS    | VALID TOKENS/BATCH |
        | 60         | EMBED      | input_ids       | 1          | 35.21    % | 1442            | 4096            | [50, 81, 35, 11, 29, 36, 66, 19, 40, 22, 21, 42, 17, 37, 40, 41, 26, 58, 38, 54, 41, 73, 48, 57, 50, 51, 49, 85, 48, 36, 79, 62] |
        | 61         | LABEL      | labels          | -100       | 100.00   % | 32              | 32              | N/A             |
        | 62         | EMBED      | input_ids       | 1          | 30.00    % | 1229            | 4096            | [36, 73, 13, 47, 27, 33, 53, 25, 51, 28, 36, 42, 42, 32, 39, 52, 27, 13, 31, 66, 42, 45, 52, 45, 58, 42, 37, 66, 12, 18, 29, 17] |
        | 63         | LABEL      | labels          | -100       | 100.00   % | 32              | 32              | N/A             |
        | 64         | EMBED      | input_ids       | 1          | 26.73    % | 1095            | 4096            | [37, 28, 20, 53, 16, 20, 44, 52, 27, 28, 16, 19, 16, 24, 63, 31, 24, 42, 33, 41, 44, 60, 44, 67, 54, 30, 20, 19, 33, 23, 24, 43] |
        | 65         | LABEL      | labels          | -100       | 100.00   % | 32              | 32              | N/A             |
        | 66         | EMBED      | input_ids       | 1          | 30.03    % | 1230            | 4096            | [22, 46, 36, 41, 46, 43, 26, 50, 60, 16, 24, 42, 56, 35, 35, 59, 29, 39, 34, 20, 66, 23, 47, 53, 19, 35, 44, 23, 34, 81, 21, 25] |
        | 67         | LABEL      | labels          | -100       | 100.00   % | 32              | 32              | N/A             |
        | 68         | EMBED      | input_ids       | 1          | 31.62    % | 1295            | 4096            | [75, 36, 48, 20, 38, 21, 49, 54, 38, 41, 26, 28, 80, 45, 48, 16, 22, 41, 34, 28, 37, 16, 74, 63, 62, 34, 22, 45, 23, 27, 37, 67] |
        | 69         | LABEL      | labels          | -100       | 100.00   % | 32              | 32              | N/A             |
<<<
```

**High Embed Density, Low Label Density Case - masked language model** 
`
python -m torch.distributed.launch --nproc_per_node=4
examples/onnxruntime/training/language-modeling/run_mlm.py
--model_name_or_path bert-base-uncased --dataset_name wikitext
--dataset_config_name wikitext-2-raw-v1 --num_train_epochs 10
--per_device_train_batch_size 8 --per_device_eval_batch_size 8
--do_train --do_eval --overwrite_output_dir --output_dir ./outputs/
--seed 1137 --fp16 --report_to none --optim adamw_ort_fused
`
```
>>>Valid token/label density (e.g. valid/total) in passing 10 steps:
        | STEP       | INPUT TYPE |  INPUT NAME     | PAD IDX    | DENSITY    | VALID TOKENS    | TOTAL TOKENS    | VALID TOKENS/BATCH |
        | 710        | EMBED      | input_ids       | 0          | 100.00   % | 4096            | 4096            | [512, 512, 512, 512, 512, 512, 512, 512] |
        | 711        | LABEL      | labels          | -100       | 13.77    % | 564             | 4096            | N/A             |
        | 712        | EMBED      | input_ids       | 0          | 100.00   % | 4096            | 4096            | [512, 512, 512, 512, 512, 512, 512, 512] |
        | 713        | LABEL      | labels          | -100       | 14.48    % | 593             | 4096            | N/A             |
        | 714        | EMBED      | input_ids       | 0          | 100.00   % | 4096            | 4096            | [512, 512, 512, 512, 512, 512, 512, 512] |
        | 715        | LABEL      | labels          | -100       | 14.18    % | 581             | 4096            | N/A             |
        | 716        | EMBED      | input_ids       | 0          | 100.00   % | 4096            | 4096            | [512, 512, 512, 512, 512, 512, 512, 512] |
        | 717        | LABEL      | labels          | -100       | 14.53    % | 595             | 4096            | N/A             |
        | 718        | EMBED      | input_ids       | 0          | 100.00   % | 4096            | 4096            | [512, 512, 512, 512, 512, 512, 512, 512] |
        | 719        | LABEL      | labels          | -100       | 15.31    % | 627             | 4096            | N/A             |
<<<
```

#### Next Step

Let's see how we leverage the data sparsity for improvement.
Optimizations on the way around compute optimizer wave 2:
> Loss compute flops reduction.
> Flatten/Unflatten embedding tokens to save compute flops.
 2023-03-03 18:36:08 +08:00
Yi Zhang
8c454a76e0
Check Mac silicon package name (#14898) 
### Description
1. add comments 
2. check Mac silicon package name 

### Motivation and Context
There isn't Mac silicon Agent in ADO.
We couldn't add smoking test to test the wheel can be installed.
But We can check whether the package name is correct to avoid the
mistake in 1.14 release.

Test run

https://dev.azure.com/aiinfra/Lotus/_build/results?buildId=283100&view=logs&j=fe710151-df7c-5aa4-0cea-cf5331faa499&t=3182cefe-2612-53c6-4445-e5b3e0c4ac57
 2023-03-03 18:27:54 +08:00
cloudhan
a997bb46b6
Refactor rocm attention (#14688) 
Extract QKV projection and attention computation into pipelines (composed from gemms and kernel launch). 

This will allow us to introduce ck flash attention in next PR
 2023-03-03 12:16:11 +08:00