* add centos tests to linux cpu ci pipeline
* Disable failing test
* use centos6 instead of centos7
* change back to centos7
* add dotnet runtime dependency
* fix dotnet runtime dependencies
* install dotnet sdk instead of runtimes
* add more dotnet dependencies
* temporary skip failing test
* ix lib path
* reenable failing test
Add support of GPT2 model optimization:
* Match subgraph of Gelu Approximation (using Tanh).
* Fuse LayerNormalization if SkipLayerNormalization is not ready.
* Output model even if embedding layer is not fused.
* Improve Reshape Fusion to improve coverage.
* Refine constant input checking, and output fused op counter.
Update script according to latest op improvements:
* Fusion of Add Bias and Gelu.
* Fuse SkipLayerNormalization and Add Bias.
Other:
* Add ReduceSum for mask as intermediate step.
* Refactor verbose setting.
* Constant folding bug fix/improvements
- Handle constant folding for node that is assigned to a non cpu EP
- Check for errors in optimizer execution frame setup
- Improve CUDA partitioning to look for initializers in parent graphs
- Add unit test
Fixes#2474
* [NupharEP] Add parallel schedule to JIT function name
Update Nuphar docker to use Python 3.6 and ubuntu 18.04
* Update notebook
* Avoid JIT cache file name conflict
* [NupharEP] Enable parallel schedule
* Update TVM with the fix to TVM threadpool to use OpenMP if possible
* Add parallel schedule when trying to vectorize
With this change, BERT squad perf on a 4-core (8 HT) CPU goes from 187ms to 150ms
* Address CR, docs and cmake update
* Doc fix
* Fix mkl
* Fix TVM windows build when using mklml
- Add --skip_tests option to build.py based on github feedback
- Add debug output at end of run_subprocess so it's clearer when the output is from a different process running
- Add check for scipy as it's required by gen_test_models.py for the onnx tests
- Use log.warning instead of warnings.warn for consistency. We use the logger almost everywhere and somewhat randomly used warnings.warn in two places.
- Add check for 'wheel' dependency not being found in setup.py and handle more gracefully
- Fix invalid input name in Keras tests
* Cuda Clip() for op set 11.
* make min_val and max_value input CPU memory directly.
* Remove original cu file useless "#pragma once"
* merge duplicate logic into one class.
Enable conv/conv_transpose and existing pooling for opset 11 in cuda execution provider.
They are of spec dilates/strides change related cuda pooling ops for op set 11.
* Optimize CPU Transpose for one axis moving either inwards or outwards. We have optimizations for NCHW <-> NHWC in CUDA but not CPU. This provides a more generic optimization to the CPU implementation.
Tested performance in both directions with data sizes of 8, 16, 32 and 64 bits, size of axis being moved of 3, 16 and 32, and number of elements to move of 100x100, 300x300 and 1000x1000.
Across all tests the average improvement even with the overhead of python was 2.5x. No cases were slower. Some were 6x faster.
Binary size increase in RelWithDebInfo build is ~5K.
NOTE: See PR comments for details of performance comparison with Eigen. Eigen is slightly faster but increases binary size by 55K just for support of rank 4 input. Binary size would be further increased to support different ranks.
Add Attention Fusion Transformer to fuse multi-head self attention subgraph to one node for optimizing Bert model inference performance.
It supports BERT model exported from PyTorch. It fuses about 20 nodes into one Attention node, and could significantly improve the inference speed of BERT model.
Support symbolic (first dimension for batch size) in input shape.
* add SAS token to download internal test data for nuget pipeline
* update azure endpoint
* fix keyvault download step
* fix variable declaration for secret group
* fix indentation
* fix yaml syntax for variables
* fix setting secrets for script
* fix env synctax
* Fix macos pipeline
* attempt to add secrets to windows download data
* fix mac and win data download
* fix windows data download
* update test data set url and location
* Symbolic shape inference improvements:
- add a mode to guess unknown ops' output rank
- add support for GatherND
- add support for If
- fix a bug in get_int_values when then tensor rank > 1D, by treating it as no sympy data
- add symbol to literal merge when ONNX silently merges dims
- fix a bug in Concat when input dim is 0
- fix a bug in ConstantOfShape that computed dim is not updated
- add support for dynamic shape in ConstantOfShape
- fix a bug in Loop output shape that loop iterator dim is not inserted at dim 0
- add support for dynamic padding in Pad
- add support for dynamic shape in Reshape
- add support for Resize with opset > 10, by treating output dims as dynamic
- fix a bug in Slice when starts/ends are dynamic
- restrict input model to opset 7 and above
- make output model optional to avoid disk write when testing
Run model tests for symbolic shape inference
Reduce 2GB docker image size of nuphar
- Added C-API test for loading custom op shared lib.
- Made some changes in C++ api header and C-api implementation to get it working.
- Added C# API and corresponding test for loading custom op shared library.
* Correctly handle implicit inputs for fused nodes
Previously, nuphar's partitioning function didn't include
node's implicit inputs into the inputs list of MetaDef, and hence
a crash was triggered in the onnx graph checker.
This commit fixed the issue. Furthermore, it also fixed a related
issue where we didn't add implicit inputs into
graph_inputs_excluding_initializers_ in Graph::SetGraphInputsOutputs.
the issue was that graph_inputs_including_initializers_ populated by
SetInputs (e.g. called by FunctionImpl::FunctionImpl) may contain
implicit inputs which were not of any node's initializers in the graph.
Because they were not part of any initializers, these implicit inputs
couldn't be visited by going through all nodes' inputs.
Consequently, they would *not* be added into graph_inputs_excluding_initializers_.
We fixed the issue by first copying the populated graph_inputs_including_initializers_
into graph_inputs_excluding_initalizers_, which then had both initializers and
non-initializers as its initial content. Later, we erase initializers from the
list. In this way, we can ensure all implicit inputs to remain in
graph_inputs_excluding_initializers_.
* refined comments and fixed duplicates
Address CR by revisiting comments in terms of implicit inputs
Also fixed an issue by skipping duplicates while copying inputs
from graph_inputs_including_initializers_.
* address CR
explain why we need to collect nodes' implicit inputs
* don't rely on pointer values for iterating std::set
Previously, openvino relied on iterating a set of NodeArg pointers
to construct inputs and outputs for a fused graph. It could cause
non-determinism. The reason was that although iterating std::set by
itself is stable, pointer values of NodeArgs may vary. Consequently,
we could end up visiting the set's elements in different orders for
different runs for the same test, which resulted in constructing
inputs (and outputs) with different orders to the fused graph.
For example, for the same test, we may have inputs [A, B] in some
runs but inputs[B, A] in others.
Let's use std::string as the key type to avoid such nondeterminism.
This commit also added implicit inputs into meta->inputs while returning
the capability from the openvino provider.
* Fixed another latent issue in openvino's GetCapability function
The issue was that we couldn't simply erase fused_inputs and fused_outputs
while iterating the nodes. For example, an output NodeArg may have multiple
uses, and it's wrong if we erase it from fused_outputs when we encounter only
one of its uses as input.
* Add Scan CUDA op.
Uses CPU implementation for logic.
Added some device specific functors for handling when data needs to be manipulated on a different device.
Added ability to override the materialization logic in the OrtValue slicer so DML can plugin their handling.
