onnxruntime/winml/api/Windows.AI.MachineLearning.idl

768 lines
41 KiB
Text
Raw Normal View History

Merge windowsai (winml layering) into master (#2956) * Initial Commit * Merged PR 3985217: add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc (#2346) add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc and violating our OS layering requirements. We linked against onecoreuap_apiset.lib in VB so we will continue doing this, but I am still unsure why not to link against onecore instead since that is where we ship. However, since Sheil is the owner of this code we will wait to discuss with him before changing anything. * Initial changes for layering * more snipping to get core into ort * update build instructions to include --build_shared_lib (#2358) * update build instructions to include --build_shared_lib * fix line breaks * Task 23998197: add winml_lib_core into onnnxruntime.dll (#2368) * Task 23998197: add winml_lib_core into onnnxruntime.dll * PR feedback build break on perf_test * return proper error when the model path isn't found (#2391) * LearningModelSession is cleaned up to use the adapter, and parts of b… (#2382) this is a big PR. we are going to move it up to layer_dev , which is still a L3 so we are still safe to do work there agile. we are going to move this into the L3 so that ryan can start doing intergration testing. we will pause for a full code review and integration test result prior to going into the L2. >>>> raw comments from previous commits >>> * LearningModelSession is cleaned up to use the adapter, and parts of binding are. * moved everything in the winmladapter made it all nano-com using, WRL to construct objects in the ORT side. base interfaces for everythign for winml to call cleaned up a bunch of winml to use the base interfaces. * more pieces * GetData across the abi. * renamed some namepsace cleaned up OrtValue cleaned up Tensor cleaned up custom ops. everything *but* learnignmodel should be clean * make sure it's building. winml.dll is still a monolith. * model moved over. everything builds clean. step ! * weak ref comment * Layer dev paulm (#2408) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * Layer dev paulm (#2414) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * Layer dev paulm (#2423) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * Layer dev paulm (#2424) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * Layer dev paulm (#2425) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Layer dev paulm (#2426) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * fixed map and sequence when passing stl types across the ABI . found a leak in nvidia driver, but skipped it. all winmlapitests pass now * Moved SessionOptions over to the abi * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * Register WinML TraceLogging provider on Onnxruntime.dll (#2455) * Register WinML TraceLogging provider on Onnxruntime.dll * Add ifdef to make sure trace logging provider has telemetry option when LAYERING_DONE * No need for ifdef for TraceLoggingOptionMicrosoftTelemetry * PR feedback * Move etw registration into lotus environment constructor and deresgister in lotus environment destructor * Brianma/cpuwinml (#2466) * allow building winml cpu without dml. * Brianma/breaks (#2469) * fix some more breaks * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * move dml checks out of winml and into the adapter * better error handling * Brianma/fi (#2470) * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * fix bad merge * Checking in a staging checkpoint point so that Ryan can work with me in parrallel * build break. * Brianma/testfails (#2473) * add missing ir version to dictvectorizer-string.onnx * add missing ir version to relu.onnx * add missing ir version to zipmap*onnx * add IR version to manually generated models * remove an unnecessary ifdef dml * Brianma/windowsai fi (#2475) * update dockerfiles/README (#2336) * Make elementwise op run 4 items per thread (#2335) Description: Describe your changes. Make elementwise op run 4 items per thread unroll for loop to leverage ILP remove unnessary N==0 check inside elementwise GPU kernel Motivation and Context Why is this change required? What problem does it solve? It can improve the performance of GPU elementwise ops. ~2% performance gain on popular NLP bert model. If it fixes an open issue, please link to the issue here. * Add CUDA GatherElements kernel (#2310) * Updates * Update test * Update * Updates * nits * PR feedback * Update * Update * PR feedback * PR comments * Update * Fix build * Fix build * Nits * Fix * Layer Normalization Fusion (#2319) basic layer normalization transform * Add FastGelu Cuda Op for Gelu and Add bias fusion (#2293) * Add FastGelu cuda op * Add AddBiasGelu for experiment * Revert "Add AddBiasGelu for experiment" This reverts commit 5c1ee019858c657e6bb75887265cb85675626e5b. * Add bias * Add unit tests * update comment * update script * fix build error * update coding style * update for CR feedback Enable half2 optimization only when cuda arch >= 7.0 * move _Tanh to common.cuh * implement CPU contrib OP Attention (#2333) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanUnusedInitializers. (#2320) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanupUnusedInitializers. This means initializers that have been replaced during graph optimizations are not left in the GraphProto when we save an optimized model. * Handle edge case where a model has an unused initializer with matching graph input by also removing the graph input. * Use non-const iterators in std::find_if calls to make centos build happy. * Nuget pipeline changes (#2305) 1. refactor the pipeline, remove some duplicated code 2. Move Windows_py_GPU_Wheels job to Win-GPU-CUDA10. We'll deprecated the "Win-GPU" pool 3. Delete cpu-nocontribops-esrp-pipeline.yml and cpu-nocontribops-pipeline.yml 4. In Linux nuget jobs, run "make install" before creating the package. So that extra RPAH info will be removed * Cuda Reverse Sequence Op, maping types of same size using same template function. (#2281) * Set ElementType to String type of node metadata, instead of byte[] (#2348) * Set ElementType to String type of node metadata, instead of byte[] * Fix spacing * Introduce PrimitiveType into a Type System along with an integer constant (#2307) Improve perf by avoiding GetType<T>() calls. Introduce MLTypeCallDispatcher to switch on Input Type. Add Tensor IsType<T>() fast method. * Fix/test dim value of 0 handling in a couple of places (#2337) * Update the CUDA Where implementation broadcasting logic to handle a dim with value of 0. Add unit test Also add unit test for unary op with dim value of 0 * Exclude ngraph from Where test with 0 dim. * Openvino EP R3.1 onnxrt server (#2357) * onnxrt server with OVEP * onnxrt server with OVEP * Update Dockerfile.server.openvino * onnxrt server OVEP fix reviews * onnxrt server OVEP fix reviews * Implement cuda nonzero op. (#2056) Implement cuda nonzero op. * Direct use python numpy array's memory if already contiguous. (#2355) * Direct use python numpy array's memory if already contiguous. This could greatly improve performance for session with large input, like big image 1920x1080 fastrcnn, 30~40% speed up could be achieved. * Add test case enforce contiguous/non-contiguos numpy array as inputs. * Add helper to create output to minimize binary size. (#2365) Add ConstEigenTensorMap typedef so we don't unnecessarily const_cast the const input Tensor. * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS (#2369) * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS * update * Add Tracelogging for profiling (#1639) Enabled only if onnxruntime_ENABLE_INSTRUMENT is ON * test bidaf with nuphar for avx target (#2370) increase nuphar test coverage a bit * Fix a bug in TLS refcount that may destabilized CUDA CI (#2374) * update output size calculation for resize (#2366) * change how output size is calculated for resize op * add tests for ver 10 resize * Extend OneHot CPU kernel to support more types (#2311) * Extend OneHot CPU kernel to support input int64_t, depth int32_t, output float * Skip BERT before the test data fix is picked up * Fix bug with Slice. Need to pass in flattened input dimensions so the initial offset into the input is calculated correctly. (#2372) * Add opset 11 version of Split to CUDA ops (#2376) Organize the CUDA ops definitions so all the opset 10 and 11 parts are together (same setup used for CPU ops) * Layer Norm Fusion Fix (#2379) * layer norm fusion fix * Add input shape check in code and unit tests * Fuse Add + Gelu (#2360) Implement the transformer to fuse add + gelu Implement the accurate kernel * Skip layer norm transform (#2350) * skip layer normalization transformer * Another try to stabilize CUDA CI (#2383) The root cause seems to be failure in CUDA dealloc when tear down. cudaFree return code was ignored before, so should the debug check. * fix BUILD.md typo (#2375) build.py: error: argument --config: invalid choice: 'RelWithDebugInfo' (choose from 'Debug', 'MinSizeRel', 'Release', 'RelWithDebInfo') * Fixed compilation with ngraph (#2388) * Fix reuse logic in allocation planner. (#2393) * Fix reuse logic in allocation planner. * PR comments * Add helpful comments * Don't allow reuse across string tensors. * [NupharEP] Multiple optimizations (#2380) Fuse transpose into MatMul Implement Pow and constant scalar simplification Vectorize ReduceMean Improve symbolic shape inference Minor updates for better debugging in fused function name * Avoid using the default logger in the graph lib and optimizers (#2361) 1. Use the session logger if it is available. 2. Don't disable warning 4100 globally. We should fix the warnings instead of disabling it. * Change CUDA implementation of Transpose to support all fixed size tensor types (#2387) * Change CUDA implementation of Transpose to not use a typed kernel so we can support more types with minimum binary size. Add support for 8, 16, 32 and 64 bit types. Add unit tests. Add method so the implementation can be called directly (will be used by CUDA Scan very soon). * Disable TensorRT for MLFloat16 and int8 unit tests. * Address PR comment and add support for calling cublas implementation if type is mlfloat16. * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. (#2398) * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. * [NupharEP] force some low/zero cost ops to be inlined (#2409) * fix cross compile bug (#2415) * Minor optimization: if a node has already been placed, there's no need to find a kernel for it. (#2417) * Add Reshape Fusion (#2395) * Add reshape fusion * Add some comments * update comments * update comment format * update according to feedback * update for recent logger change * fix build error * (1) Support both input and output edges in find path in graphutils (2) Add a test case of only one constant initializer of Concat input. (3) Refactor ReshapeFusion class to allow add more subgraph fusion in the future. * fix error * (1) loose constraint on initializer: non constant is allowed for reshape fusion. (2) Change versions type to vector. (3) Add logging. (4) Return false when multiple output edges matched in FindPath. Add comments. * only allow one direction (input or output) in FindPath * [NupharEP] Update notebook and docker image (#2416) Add BERT squad in Nuphar tutorial Enhance speed comparsion readability * Fix the issue in matmul_add_fusion (#2407) Fix the issue in matmul_add_fusion If Muatmul + Add has shape [K] * [K, N], reset it to [1, K] * [K, N] will make the output shape to [1, N] will also requires a reshape on the output. Fix: just remove the shape reset to not fuse it. Add a negative test case for matmul+add fusion * feat(treeregressor): Update TreeEnsembleRegressor for type support (#2389) Updates the `TreeEnsembleRegressor` to allow for `double`, `float`, `int64`, and `int32` inputs to match the upstream specification. Signed-off-by: Nick Groszewski <nicholas.groszewski@capitalone.com> * onnxrt server documentation update (#2396) * Added support for Pad-2 operator in OpenVINO-EP (#2405) * Add CUDA If operator. (#2377) * Add CUDA If operator. Uses CPU operator for implementation. By adding a CUDA version the inputs/outputs (with the exception of the 'cond' input) stay on GPU, and no other logic is required to avoid a copy to CPU across the control flow node. * Improved documentation for onnxruntime::utils::SwapByteOrderCopy(), added precondition check. * Fix the type constraints on CUDA If operator to exclude strings. (#2431) * add Im2col<uint8_t> (#2438) * Adjust codegen vectorization width from target (#2439) * Adjust codegen vectorization width from target * Add CUDA Scan operator. (#2403) * Add Scan CUDA op. Uses CPU implementation for logic. Added some device specific functors for handling when data needs to be manipulated on a different device. Added ability to override the materialization logic in the OrtValue slicer so DML can plugin their handling. * Fix Windows GPU C API packaging pipeline failure (#2440) Fix Windows GPU C API packaging pipeline failure (#2440) * Correctly handle implicit inputs for fused nodes (#2390) * Correctly handle implicit inputs for fused nodes Previously, nuphar's partitioning function didn't include node's implicit inputs into the inputs list of MetaDef, and hence a crash was triggered in the onnx graph checker. This commit fixed the issue. Furthermore, it also fixed a related issue where we didn't add implicit inputs into graph_inputs_excluding_initializers_ in Graph::SetGraphInputsOutputs. the issue was that graph_inputs_including_initializers_ populated by SetInputs (e.g. called by FunctionImpl::FunctionImpl) may contain implicit inputs which were not of any node's initializers in the graph. Because they were not part of any initializers, these implicit inputs couldn't be visited by going through all nodes' inputs. Consequently, they would *not* be added into graph_inputs_excluding_initializers_. We fixed the issue by first copying the populated graph_inputs_including_initializers_ into graph_inputs_excluding_initalizers_, which then had both initializers and non-initializers as its initial content. Later, we erase initializers from the list. In this way, we can ensure all implicit inputs to remain in graph_inputs_excluding_initializers_. * refined comments and fixed duplicates Address CR by revisiting comments in terms of implicit inputs Also fixed an issue by skipping duplicates while copying inputs from graph_inputs_including_initializers_. * address CR explain why we need to collect nodes' implicit inputs * don't rely on pointer values for iterating std::set Previously, openvino relied on iterating a set of NodeArg pointers to construct inputs and outputs for a fused graph. It could cause non-determinism. The reason was that although iterating std::set by itself is stable, pointer values of NodeArgs may vary. Consequently, we could end up visiting the set's elements in different orders for different runs for the same test, which resulted in constructing inputs (and outputs) with different orders to the fused graph. For example, for the same test, we may have inputs [A, B] in some runs but inputs[B, A] in others. Let's use std::string as the key type to avoid such nondeterminism. This commit also added implicit inputs into meta->inputs while returning the capability from the openvino provider. * Fixed another latent issue in openvino's GetCapability function The issue was that we couldn't simply erase fused_inputs and fused_outputs while iterating the nodes. For example, an output NodeArg may have multiple uses, and it's wrong if we erase it from fused_outputs when we encounter only one of its uses as input. * Remove DeviceAllocatorRegistry class (#2451) Remove DeviceAllocatorRegistry class * CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library. * Parallel Gelu with ParallelFor (#2399) Parallel Gelu to get better performance for Gelu * Clean up build.py (#2446) * Pull the latest image before running docker build * Fuse SkipLayerNorm with Bias (#2453) Fuse SkipLayerNorm with Bias * Allow more than one invocation of CreateEnv in the same process. (#2467) * Allow more than one invocation of CreateEnv in the same process. * Fix centos build * Symbolic shape inference improvements: (#2460) * Symbolic shape inference improvements: - add a mode to guess unknown ops' output rank - add support for GatherND - add support for If - fix a bug in get_int_values when then tensor rank > 1D, by treating it as no sympy data - add symbol to literal merge when ONNX silently merges dims - fix a bug in Concat when input dim is 0 - fix a bug in ConstantOfShape that computed dim is not updated - add support for dynamic shape in ConstantOfShape - fix a bug in Loop output shape that loop iterator dim is not inserted at dim 0 - add support for dynamic padding in Pad - add support for dynamic shape in Reshape - add support for Resize with opset > 10, by treating output dims as dynamic - fix a bug in Slice when starts/ends are dynamic - restrict input model to opset 7 and above - make output model optional to avoid disk write when testing Run model tests for symbolic shape inference Reduce 2GB docker image size of nuphar * add additional test data set for nuget pipeline (#2448) * add SAS token to download internal test data for nuget pipeline * update azure endpoint * fix keyvault download step * fix variable declaration for secret group * fix indentation * fix yaml syntax for variables * fix setting secrets for script * fix env synctax * Fix macos pipeline * attempt to add secrets to windows download data * fix mac and win data download * fix windows data download * update test data set url and location * Revert "Brianma/windowsai fi (#2475)" This reverts commit 5780b864a15513fda4eadbfc2b5345fefe70b5ec. * Add scenario tests (#2457) * Add scenario tests * Remove TODO from model license * Add winml_api test dependency * fix model load test. fi from master changed the constructor (#2483) * make api tests all pass (#2486) * fix bad merge * fix bad model merge * Layer dev paulm (#2492) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * Rename ambiguous header (#2489) * fix one more missing IR version model (#2500) * add missing IR version to 4 more models used by scenario tests (#2501) * Add CLI parameters to test runner, build WinML in ARM and x86 CI (#2479) * Support test parameters through CLI arguments * Add WinML do Windows x86/ARM CI builds * Code style fixes * Update googletest Remove GPUTEST macros everywhere now that GTEST_SKIP is supported * Refactor main.cpp * Build scenario tests without DML * Link scenario tests to DML when it's enabled (#2502) * Layer dev release pipeline (#2488) Adds winml binaries to existing cpu nuget package, and creates new gpu dml nuget package with winml binaries and DML EP. * Layer dev paulm (#2506) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * Remove usage of IOBinding in WinML and use C_API Run method (#2504) * remove usage of iobinding * Change data structure to use vector of Ort::Values * Polish bind input / output * Use C APIrun method * Update providers on evaluate getresults * Remove run and IObinding interface from WinMLAdapter * Remove use of IObinding * bind unbound outputs code moved to learningmodelbinding * clean up unneeded istensor adapter function * Fix comment * Check if session is closed before binding and clearing * PR feedback * Layer dev paulm (#2507) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * Make tests dependend on winml_dll (#2509) * add dml binaries to DirectML package and be more explicit about condition variables (#2520) * re-enable warnings for winml builds and fix the warnings that were hiding (#2526) * turn devmode back on for winml builds * fix some warnings. include protobuf in a way that disables some warnings * undo protobufhelpers changes and just ignore 4100 errors in pb code * attempt to isolate protobufhelpers errors * add template specialization for getting tensor proto data * Layer dev paulm (#2533) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * Spawn child process to run DeviceLostRecovery scenario test (#2530) * Spawn child process to run DeviceLostRecovery scenario test * Layer dev paulm (#2536) ori said yes * add missing namespace to winml_trace_logging_provider in lotusenvironment.h (#2542) * Handle exception thrown from all apis in WinMLAdapter (#2539) * various changes to unblock windowsai ADO build * Fix custom ops scenario tests (#2562) * Do not shutdown protobuf after ort environment gets destroyed. Lazy load lotus environment first time it is needed * comment typo * pr comment about calling phoenix singleton * Make lotus_environment static in winmladapter * Layer dev paulm (#2567) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * making windowsAI azure dev ops work. * code review comments. * revert changes * Cmake and preprocessor fixes that where uncovered by building on agents without DML available via SDK * Layer dev dml delayload (#2580) * Brianma/cpu (#2583) * don't include dml stuff in cpu builds * tests that link the image lib also need the telemetry lib now * Throw Winml_err_invalid_binding if binding gpu resource on cpu device (#2589) * Throw Winml_err_invalid_binding if binding gpu resource on cpu device * PR comments. No need to query executionprovider for is gpu device * User/xianz/ortthrow (#2596) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * User/xianz/ortthrow (#2599) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * add status error message * Remove uwp onsuspending winrt call because logruntimeperf is getting removed (#2630) * User/xianz/dedup telemetry (#2631) * investigate duplication of telemetry in winml and ort * remove winml telemetry events * telemetry executionProviderEvent * remove unneccessary file and refactor code little bit * Revert back TelemetryEvent, which send up ETW event. * merge changes from layer_dev to windowsai (#2638) * Remove underscore from googletest names (#2616) * Fix leaking memory allocator Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24278761 and https://microsoft.visualstudio.com/OS/_workitems/edit/24330198 * Explicitly initialize Ort::Value with nullptr * Cache WinML adapter * bad merge * define private version of dxcore enum that is added in 19H1 SDK. (#2654) * add comment for explaning private definition of dxcore d3d feature level ennum value. (#2672) * do not package directml.pdb for redist packages. (#2676) * Fix leaking operator registry (#2645) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24354916 * User/orilevari/windowsai master merge (#2674) merge resolutions included pulling in telemetry logic that was merged to master and not windowsai and dereferencing InferenceSession::sessionstate now that it is a unique pointer * Delete Ort Allocator in LearningModelBinding (#2653) * Delete OrtAllocator in LearningModelBinding * PR comments to make Ort::Allocator a smart pointer * Small comment change * PR feedback to clean up code * PR feedback on move semantics * Clean up std::move * Fix memory leaks (#2679) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24356109, https://microsoft.visualstudio.com/OS/_workitems/edit/24388361 and https://microsoft.visualstudio.com/OS/_workitems/edit/24388596 * various changes to properly organize and skip GPU tests. For now for No DML builds we will not run GPU tests at all. In the future we should adapt the tests to expect the appropiate errors. (#2695) * Windowsai without fi (#2701) * Disable Attention fusion tests when DISABLE_CONTRIB_OPS is defined (#2529) * Setup java ci (#2528) * Add provision in ORT for session options to be parsed when available via model file (#2449) * Initial commit * Fix gitmodules * Nits * Nits * Updates * Update * More changes * Updates * Update * Some updates * More changes * Update * Update * Merge * Update * Updates * More changes * Update * Fix nits * Updates * Fix warning * Fix build * Add comment * PR feedback * PR feedback * Updates * Updates * Update * More changes * Fix build break * Comment test for now * Updates * Updates * PR feedback * Updates * Nits * Add tests * Fix build * Fix build * Fix build * Fix build break * Fix build * Nits * PR feedback * More change * Expose GetSessionOptions in pybind logic and add unit test for python * Fix build * PR feedback * PR feedback * Revert "Disable thread pool creation when enabled OpenMP (#2485)" (#2535) This reverts commit 7c7d5a149c9ed52eec67304bae5c4b132166a8a1. * Add dynamic shape support in TensorRT execution provider (#2450) * remove onnx-tensorrt submodule * add new onnx-tensorrt submodule (experiment) for trt6 * update engine build for trt6 * update compile and compute for tensorrt6.0 * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * switch to onnx-tensorrt master for TensorRT6' * Update tensorrt_execution_provider.cc * Handle dynamic batch size and add memcpy in TensorRT EP * update test cases * Update tensorrt_execution_provider.cc * update onnx-tensorrt submodule * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.ubuntu_tensorrt * Update run_dockerbuild.sh * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update concat_op_test.cc * Update tensorrt_execution_provider.cc * Upgrade TensorRT to version 6.0.1.5 * Update onnxruntime_providers.cmake * Update CMakeLists.txt * Update reduction_ops_test.cc * Update install_ubuntu.sh * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.tensorrt * Update BUILD.md * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update onnxruntime_providers.cmake * Update install_ubuntu.sh * Update install_ubuntu.sh * Update gemm_test.cc * Update gather_op_test.cc * Update CMakeLists.txt * Removed submodule * update onnx-tensorrt submodule * update header file * Removed submodule * add submodule onnx-tensorrt kevin's branch shape-test' * add debugging code * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * merge master * Removed submodule * update onnx-tensorrt submodule * add more changes for dynamic shapes * Update tensorrt_execution_provider.cc * update for dynamic shape * update dynamic shape processing * fix logger issue * remove submodule onnx-tensorrt * add submodule onnx-tensorrt * add env variable min_subgraph_size * remove redundency * update document * use onnxruntime::make_unique * fix multi-run issue * remove some tests to save CI build time * Add dynamic shape test * Update TensorRT-ExecutionProvider.md * Add example of running Faster R-CNN model on TensorRT EP * Add more details on env variables * update environment variables * Update tensorrt_basic_test.cc * Update model tests * Update tensor_op_test.cc * remove --use_full_protobuf * Update build.py * User/xianz/telemetry (#2458) * enabme telemetry * enable telemetry * set enable telemetry as default * for debugging * remove log and set disable telemetry as default back * delete private file while testing * resolve comment: mainly add license header, rename macro and update docs * rewording in privacy.md * Fix integer overflow in cuda NonMaxSuppression implementation (#2540) * add test case that should pass but fail * fix nms * extract int_max_output_boxes_per_class * Introduce container type runtime checks and other improvements (#2522) Rework TensorSeq in a manner consistent with Tensor and SparseTensor in terms of type system setup. Reduce templating. Introduce helpers to ensure the same data type. Make OrtValue __dtor not virtual. Introduce ContainerChecker * Fix C API tests for centos and mac (#2544) * change c++14 to c++11 * add ld lib path for centos * enable csharp tests on macos * fix C API test on MacOS + fix manylinux dotnet install * fix manylinux dotnet install * fix lib link * Add back executable bit to build.py * Fix a bug handling negative begin pad values in Pad op (#2550) * Fix bug in Pad op * Update * DNNL CMAKE update (#2548) * Fix android build (#2558) * Update win-x86-ci.yml (#2557) Fix build pipeline break * Re-enable Windows C# tests (#2564) * disable onnx_test_runner -x invocations for dnnl (#2568) * Allow sequence length to be symbolic (#2559) * setup java ci mac (#2570) * make layernorm fusion to support opset 11 (#2545) * Fix a warning found in the latest VS release * Add more check on SkipLayerNorm and BiasGelu fusion (#2574) * Fix file not found error during docker build. (#2569) * Add ConvTranspose1D (#2578) * Ryanunderhill/packagename test (#2582) * [Nuphar EP] fixes for some object detection models (#2581) Update notebook tutorial with multi-threaded int8 GEMM from #2517 * EmbedLayerNormalization Fusion Improvement (#2553) Embedding layer norm fusion improvements - add more checks * Update version (#2584) * Temporarily exclude vgg19 test from Python backend test 1. temporarily exclude vgg19 test which comsumes too much memory, run out of memory on Upsquared device. Single test pass for vgg19, need furture investigation (#2588) 2. Update docker file to decrease the docker image size * Update docs for Android NNAPI EP (#2586) * Fix lto bug for protobuf and ubuntu * add path to build dir before test run (#2590) * Add missig env variables for mac pipeline test (#2595) * Fixed an issue in updating realized dims (#2597) when we update realized dims for scan's output, the sliced axis also needs to be inclusive, i.e. we should check with "dim >= insert_inclusive_axis", because the offset in the symbols are based on Scan sugraph. Otherwise, we would end up with shape mismatch later. * Java API for onnxruntime (#2215) * Add support for opset 11 in reshape fusion (#2592) Support opset verion 11 in reshape fusion * Rename automl python tools folder to featurizer_ops. (#2593) * Support opset 11 subgraph of Squad model in Embed Layer Normalization (#2605) Support opset 11 Squad model that is exported from PyTorch nightly. The embed layer uses Range op which is missed in the transformer. * symbolic shape inference: fix warnings in GPT-2 model (#2608) And revise nuphar perf test on BERT squad * Dump subgraph ID and fused graph ID (#2607) * Dump subgraph ID and fused graph ID Dump subgraph ID and fused graph ID for better debugging * Remove local static fused_count added a field global_fused_count_ to NupharExecutionProvider class * EmbedLayerNormalization Fusion For Dynamic Squad Model Opset 10 (#2613) Support subgraph of SQuAD model exported from pytorch with dynamic input axes * Allow providers to be set for InferenceSession at construction (#2606) * Remove unnecessary parameter in some places in GatherElements implementation (#2612) * Remove unnecessary parameter in some places * Update * Update * Make sure fenced tensor could not reuse other tensor. (#2561) Fix random error caused by this. * Improve Embed Layer Norm Fusion for SQuAD with static input shape (#2621) * fix float16 comparison in initializer (#2629) * epsilon attribute for layernormalization fusion (#2639) * removed unnecessary batch file and fix path (#2640) * Add shape inference to ConvTransposeWithDynamicPads schema (#2632) * Improve cuda expand() opeator's performance. (#2624) * Cuda pad optimize when no padding is needed. (#2625) * Shortcut cuda Pad() when no padding is needed. * Optimize cuda scatter() on 2D compatible. (#2628) * Optimize cuda scatter() on 2D compatible. * Add some comments. * fix build error for ARM (#2648) * Improve performance of resize() in Nearest mode (#2626) Special treatment for 2D, check same size as input image. And in 2d kernel, template use_expolation. * Fix memory exception in Layer Norm Fusion (#2644) * Windows CI changes(#2650) * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Revert "Windowsai without fi (#2701)" This reverts commit 285d4c85ff5c4e265f963208170304ef3461e684. * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Deref unique pointer for session_state * send shutdown event when dll is unloaded and EvaluationStop, SessionC… (#2704) * send shutdown event when dll is unloaded and EvaluationStop, SessionCreationStart Events. * Add EvalutationStart Event * add comment * use correct type for for loop (#2755) * ARM CI (#2759) * Set ARM agent pool * Set CMake generator to VS 2019 in ARM * Use system-wide CMake instead of custom version Our custom version is too old for VS 2019 * Use DML and build shared lib in ARM CI * Restore nuget packages in ARM CI * Disable DML * Refactor ARM debug/release builds * Use system packaged Python version * Remove hardcoded Python path * Downgrade Python to 3.7 for build * Remove explicit CMake path * Fix invalid JSON in cgmanifest.json (#2760) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * Remove usage of WHOLEARCHIVE in WinML CMake and add WinMLAdapterFactory (#2726) * Remove usage of WHOLEARCHIVE in WinMLAdapter CMake and add WinMLAdapterFactory * PR feedback, no need for dll(export) since using def file * PR comments * Small comment in gen_def.py * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * Move winml_provider_factory.h to proper location (#2801) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Filter CPU case for IsFloat16Supported (#2802) * Merge fixes * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * Fix warnings that cause build to fail * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * Make DML include path global (#2882) * Make DML include path global * Add generated cppwinrt headers to winml_lib_common * Integrate changes to WindowsAI to make ADO Build (#2886) * Revert "CMake cross-generator fixes (#2790)" This reverts commit dbe7d97fa1ab155f1309bced87199527e8f35bd2. * add additional suppress warning in onnx_proto * ignore /wd4996 warning * DML execution provider fixes * Revert "Revert "CMake cross-generator fixes (#2790)"" This reverts commit 1ae7b4bcbc02edc881ad28685da98e095dfceb17. * Update func signature of custom op function overloads * common devicehelpers fixes * Add pch.h for winml_lib_common * re-add winml_lib_common_dir/inc to include path for winml_adapter * User/orilevari/dml redist shared folder (#2890) * move dml nuget package directory up one level to make it shared between build flavors * Merge conflict fix * Revert "Merge conflict fix" This reverts commit 142fa72cf9ce4344ad717b50b7ea2b8582aadc7c. * Revert "Merge remote-tracking branch 'origin/master' into windowsai" This reverts commit 6e2126d46e5e5f564d65da37dd4f70c93dd81165, reversing changes made to b3f5583dc9249834b947c8ea905f6a98060d5bd6. * Make winml_test_common free of test macros (#2902) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Don't generate files outside the build root (#2914) * Don't generate files outside the build root * Add onnxruntime_EXTERNAL_DEPENDENCIES to WinML * Add DML depedency on RESTORE_PACKAGES * User/orilevari/fix yaml merge bugs (#2918) * Add winml test source parameter into cmake function (#2919) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Add arguments to unittest cmake functions * remove comment * Revert "Revert "Merge remote-tracking branch 'origin/master' into windowsai"" This reverts commit ade5abe72a4234fdbc3623093c61c02c6b0bdc26. * Fix breaks from merge with ORT master * Brianma/linux (#2917) * don't include windows.h in cross-plat header * add default case for switch statement * signed/unsigned mismatch fix Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> * User/sheilk/winml adapter c api (#2891) * Create winml adapter c api * fix build * make it build * move adapter into onnxruntime core/session * entry point not exported * minor changes * make model metadata work * make tests pass * implement all the model reflection apis on the adapter c abi * update the new ort interface to create a lotus ennvironment with a logging sink * start adding ort env * move all winml code into adapter folder/lib to isolate it * ensure a single logging manager at a time * start refactoring session * refactor session creation interface * add cpu and dml session option methods to adapter * finish session init * stub out interfaces in ort lib to perform similar mechanics of iinference session * enable profiling, and enable schema override * update session register graph transformers * turn back on custom registry for custom ops * Add sync api * add last c api stubs * should build... but all feature values are broken since this is in flight to moving all implementation details into ivalue * remove ep adapter header * Implement DML execution provider functions from adapter (#2846) * Implement DML execution provider functions from adapter * Use functions in OnnxruntimeEngine.cpp * make map/sequence type_infos freeable, and start implementing ivalue * make it build again * implement value methods * implement remaining methods * remove com adapter abi * check dml session * cache the allocator on ivalue * check if resource is cpu/gpu when access its mutable data * update tensor * mismatched parentheses * fix tensor base and binding obj * it evaluates tensors! sometimes... * minor fixes * enable gpu evals * wrapper all existing winml adapter apis with API_IMPL to try catch (#2854) * update winml... tensor strings are broken, need to template tensorbase to do different things for strings * make tensor strings work with 2 copies in/2 copies out * Fix tensor string and allocator bug * make maps work again... needs some fixes still * Make it build! * enable map inputs * map outputs * unbound outputs for sequences and maps * User/xianz/merge windowsai (#2883) * Packaging pipeline changes for VS 2019 (#2711) * Tiny fix to codegen * Simplify cache implementation and avoid static variables that may carry over between models * Extend DML kernels (#2641) * Additional DML operators * Check unsupported attributes and inputs * Address PR comments * Add kernel capability function used for partitioning, and re-enable stride-based int64 support based on value range * Fix test failures * Build fix * PR comments * Update Nuphar tutorial notebook (#2721) 1. Reflect int8 GEMV improvements for multi-threading from #2696 2. Add notes on multi-threading control using OpenMP 3. Add samples of running multi-isa AOT, and show int8 GEMM differences between AVX and AVX2 4. Add rnn_benchmark example to resolve #1993 * Add schema for new Qops (#2611) * Add schema for new Qops * adding shape inference + qlinearaveragepool * plus review comments * plus review comments * updates per review comments * plus review comments * [server] Add supposed for model_name and model_version as cli parameter (#2708) * remove 64bit warning message from python validation. (#2727) * MLAS: ARM64 build fix (#2734) fix bad usage of vreinterpret to cast vector element types * Fix broken python docs links (#2740) * Fix build on Mac OS (#2731) mac os ld doesn't support --while-archive, correct option is -all_load * fix ngraph wheel (#2737) * fix ngraph wheel 1.1.0 onnxruntime_ngraph wheel doesn't work * remove libdnnl.so in nGraph Libs * make it easy to compare * Split onnxruntime server to a separated folder (#2744) * Fix build for Python 3.8 (#2747) * Fix build for Python 3.8 * Update protobuf to 3.11.2 (#1928) Update protobuf to 3.11.2 (#1928) * Change default optimization level to All (from Basic) (#2745) * change default optimization level to All (from Basic) * fix test * fix c# test * Update numpy to 1.18 (#2758) * Update numpy to 1.18 * Pipeline changes for python 3.8 (#2753) 1. Pipeline changes for python 3.8 2. Fix a regression in setup.py which was just introduced in the previous commit. Please notice, we still haven't made python 3.8 + Windows + CUDA work. * Add basic stacktrace output for posix debug builds. (#2749) * [NupharEP] fix a race condition when multiple sessions running different models concurrently (#2772) * Revert "Change default optimization level to All (from Basic) (#2745)" This reverts commit 56bb503c2f26474b6613bcb2a198691a11dcef38. * Fix typo in error message (#2736) * Rename MKL-DNN to DNNL to fix broken link (#2730) * Fix nightly build version number issue * Pass BUILD_BUILDNUMBER to linux docker * Disable featurizers in python packages * Import more featurizers (#2781) Make kernels non-template. Add input constraint for learnt data. Add min_max_scalar_transformer, robust_scalar_transformer, inputation_marker_transfomer, label_encoder_transformer, missing_dummies_transformer along with tests. Advance Featurizers library commit. * Implement a more stable softmax (#2715) * Implement a more stable SoftMax e^x is represented as infinity if x is large enough, like 100.f. Infinity divided by Infinity is a NAN. Thus, softmax gets a NAN if one or more item are large enough. A math transform as below is leveraged to get a stable softmax: e^xi/(e^x1 + ...e^xn) = e^(xi - max) / (e^(x1 - max) + ... + e^(xn - max)) And for convenience, force max to 0.f if all xi are negative * Contributing: Fix a typo (#2784) * ACL EP GEMM improvements (#2780) When it is posible we use a fully connected layer instead of the gemm implementation. This will let the library use the best implementation based on the input data. * ACL EP convolution improvements (#2774) Added the optimized implementation for depthwise convolution for both ACL v19.02 and ACL 19.05. Also the pointwise convolution seems to be more optimal in the CPU implementation so we opted for that instead. * Add script for release Nuget validation (#2719) * Initial commit * Nits * Disable a test temporarily * Change working directory * Test * Add download python step * Test update * More changes * Fix space issue * Fix * Verify nuget signing * Fix * Spaces * PR feedback * Nit * Fix * Fix * Remove temporary changes * add uint8 support to where op (#2792) * Improve bert optimization script: (#2712) (1) Move input int64=>int32 conversion to embed layer fusion. (2) Output epsilon attribute for LayerNormalization fusion. * add session creation time cost. (#2798) * ML.NET team needs featurizers within a package (#2789) Add auto ml featurizers to Windows, MacOS as well as to GPU packaging-pipelines. * Initialize max of softmax with lowest of float (#2786) * MLAS: update SGEMM threading parameters (#2808) * add interface to copy batch tensors. (#2807) * add interface to copy batch tensors. * onnxruntime * speed up Windows TRT CI (#2811) * don't run cuda tests if building with tensorrt * remove unnecessary build options for win trt ci * refactor win gpu tensorrt ci yml * --numpy_version=1.17 * update * update * azcopy and cuda path * Update test data (#2356) * Add timeseries imputer transformer featurizer kernel (#2813) Make kernels non-template. Add input constraint for learnt data. Fixup tests. Add two more featurizers along with tests. Tests fail. min_max_scalar_transformer robust_scalar_transformer Fix tests serialized stream by prepending version bytes. Add inputation_marker_transfomer and the test. Fix up float/double type designations. Added label_encoder_transformer along with a test. string_throw case is broken at the momement. Fix labelencodertransfomer_test.cc string_throw case Rename maxabsscalertransformer_test.cc Add MissingDummiesTransformer along with the test. Update manifest. Add TimeSeriesImputerTransformer definition, implementation and tests * Fix memory leak in TRT (#2815) * fix memory leak issue * revert EP_FAIL on enueueV2 * Add manifest missing comma * Run static code analyzer on most of our code (#2817) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * update quantization doc (#2783) * update documentation for quantization script * plus some spell corrections * Filter CPU case for IsFloat16Supported (#2802) * update default optimization level + fix gemm_activation fusion (#2791) * update defualt optimization level + fix gemm_activation fusion * fix typo * add unit test and incorporate review comments * fix test comment * Fix dnnl wheel package name (#2823) * Append '-dnnl' to whl package name when --use_dnnl * Update build.py * Update Ubuntu & TensorRT version in README (#2820) Dockerfile.tensorrt is using nvcr.io/nvidia/tensorrt:19.09-py3 as base Image, update Ubuntu and TensorRT version according to https://docs.nvidia.com/deeplearning/sdk/tensorrt-container-release-notes/rel_19-09.html#rel_19-09 * Merge fixes * Add OneHotEncoder and HashOneHotEncoder kernels. (#2830) Add defs and imlementation for OneHotEncoders, adjuist date_time_transformer kernel and test. Add OneHotEncoder kernel test. Add HashOneHotVectorizerTransformer unit test. This does not link due to multiple definitions of functions that are included into header from a CPP file. * Upgrade gtest to the latest version (#2827) WinML would like to update the googletest submodule. They want some newer features (namely GTEST_SKIP to skip tests programmatically and be able to skip entire fixtures easily) and would need to update the submodule version. However, because the new version of code hit a bug in gcc, even though the bug is already fixed in the latest gcc but we're using gcc 4.8.x and it won't get patched for the bug, so we have to do a compromise, change our code a little bit to make it work. The gcc bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=51213 * Add support for int64_t for topk CPU. Fixes github issue #2806. (#2833) * Ignore allocator type in ExecutionProviders allocator map. Make default initialization of OrtMemoryInfo more clearly invalid. (#2768) * Remove allocator type from the key comparison in ExecutionProviders. Remove usage of DummyArena as it's no longer necessary. * Fix x86 tests where arena allocator is disabled. Make initialization of OrtMemoryInfo clearer by adding Invalid enum value. * Make OrtValueNameIdxMap::MaxIdx more intuitive. * Convert ExternalProject Featurizers into git submodule (#2834) Add git submodule for Featurizer library. Update cmake to build for git submodule. * add domain check for nodes + update documentation (#2831) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Fix Linux CUDA nuget packaging pipeline break * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * update optimization doc for BERT related fusions (#2819) * Add bert related transformers to doc * Add execution provider and comment for bert optimizations * Add comment about accuracy impact of approximation * Fix warnings that cause build to fail * MLAS: enable threading for quantized GEMMs (#2844) * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * remove old winmladapter cpp Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * move sequence implementation into ort lib... still commented out... need to turn back on... * begin sequence implementation * make maps and sequences work * fix broken tests * remove dead code * misc cleanup * CR feedback * User/xianz/winml adapter c api (#2869) * wrapper all existing winml adapter apis with API_IMPL to try catch * Return HR or Throw for WinML adapter APIs if failed * undo macro wrapper for two places * Wrap error macros around ort apis, too. * address CR feedback #2 * add more api throw/return macros * Revert changes no longer needed * revert changes to cxx api * format winml lib.ort and winml adapter * remove static pheonix singleton Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * missing use_dml check in winml_adapter_session (#2930) * --use_dnnl flag was mangled in merge (#2931) * use dml macro not wrapping custom registry code (#2934) * Disable LNK4199 winml_dll to enable cuda builds (#2936) * Disable LNK4199 in winml_dll * linkler->linker * LearningModelSessionAPITestGpu.CreateSessionWithCastToFloat16InModel should return DXGI_ERROR_UNSUPPORTED when FP16 not supported (#2937) * Disable LNK4199 in winml_dll * linkler->linker * Need to return DXGI_ERROR_UNSUPPORTED when Model does not support fp16 * Publish build symbols (#2939) * Publish build symbols * Don't upload PDBs for .exe files * Make x86 build (#2943) * fix last remaining size_t/int64_t warnings->errors (#2948) * TensorString, Sequences and Maps use the first allocator, but should use the cpu default allocator. (#2952) * fix tensor string allcoator * clean up default allocator usage for strings in winml lib/api.ort Co-authored-by: Ryan Lai <ryalai96@gmail.com> * Handle tensor shape of zero (#2954) Co-authored-by: Ryan Lai <ryalai96@gmail.com> * CR feedback (#2970) * CR feedback * fix weird formatting on privacy readme * Add 'All rights reserved.' everywhere * readd all rights reserved to winml_provider_factory.h * remove extra space in comment * remove extra whitespace * fixes post master merge * remove winml from nuget gpu pipeline * set IR VERSION on generated_model in rnn_benchmark (#2972) * Fix slice conformance failures (#2908) Co-authored-by: Adrian Tsai <adtsai@microsoft.com> Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Paul McDaniel <paul_mcdaniel@hotmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Dwayne Robinson <fdwr@hotmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com>
2020-02-05 01:12:19 +00:00
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.
//! \file Windows.AI.MachineLearning.idl
import "Windows.Foundation.idl";
import "windows.graphics.idl";
import "windows.media.idl";
#ifndef WINDOWSAI_RAZZLE_BUILD
// Pull in definition for DualApiPartitionAttribute, because the WinML IDL
// does not build in the OS Repo, and needs to access internal definitions for
// various custom attirbute definitions.
import "dualapipartitionattribute.idl";
import "windows.graphics.directx.direct3d11.idl";
import "windows.graphics.imaging.idl";
import "windows.storage.idl";
#endif
#include <sdkddkver.h>
#ifdef BUILD_INBOX
#define ROOT_NS Windows
#define INBOX_ONLY(x) x
#define OTB_ONLY(x)
#else
#define INBOX_ONLY(x)
#define OTB_ONLY(x) x
#endif
#ifndef ROOT_NS
#define ROOT_NS Microsoft
#endif
namespace ROOT_NS.AI.MachineLearning
Merge windowsai (winml layering) into master (#2956) * Initial Commit * Merged PR 3985217: add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc (#2346) add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc and violating our OS layering requirements. We linked against onecoreuap_apiset.lib in VB so we will continue doing this, but I am still unsure why not to link against onecore instead since that is where we ship. However, since Sheil is the owner of this code we will wait to discuss with him before changing anything. * Initial changes for layering * more snipping to get core into ort * update build instructions to include --build_shared_lib (#2358) * update build instructions to include --build_shared_lib * fix line breaks * Task 23998197: add winml_lib_core into onnnxruntime.dll (#2368) * Task 23998197: add winml_lib_core into onnnxruntime.dll * PR feedback build break on perf_test * return proper error when the model path isn't found (#2391) * LearningModelSession is cleaned up to use the adapter, and parts of b… (#2382) this is a big PR. we are going to move it up to layer_dev , which is still a L3 so we are still safe to do work there agile. we are going to move this into the L3 so that ryan can start doing intergration testing. we will pause for a full code review and integration test result prior to going into the L2. >>>> raw comments from previous commits >>> * LearningModelSession is cleaned up to use the adapter, and parts of binding are. * moved everything in the winmladapter made it all nano-com using, WRL to construct objects in the ORT side. base interfaces for everythign for winml to call cleaned up a bunch of winml to use the base interfaces. * more pieces * GetData across the abi. * renamed some namepsace cleaned up OrtValue cleaned up Tensor cleaned up custom ops. everything *but* learnignmodel should be clean * make sure it's building. winml.dll is still a monolith. * model moved over. everything builds clean. step ! * weak ref comment * Layer dev paulm (#2408) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * Layer dev paulm (#2414) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * Layer dev paulm (#2423) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * Layer dev paulm (#2424) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * Layer dev paulm (#2425) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Layer dev paulm (#2426) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * fixed map and sequence when passing stl types across the ABI . found a leak in nvidia driver, but skipped it. all winmlapitests pass now * Moved SessionOptions over to the abi * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * Register WinML TraceLogging provider on Onnxruntime.dll (#2455) * Register WinML TraceLogging provider on Onnxruntime.dll * Add ifdef to make sure trace logging provider has telemetry option when LAYERING_DONE * No need for ifdef for TraceLoggingOptionMicrosoftTelemetry * PR feedback * Move etw registration into lotus environment constructor and deresgister in lotus environment destructor * Brianma/cpuwinml (#2466) * allow building winml cpu without dml. * Brianma/breaks (#2469) * fix some more breaks * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * move dml checks out of winml and into the adapter * better error handling * Brianma/fi (#2470) * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * fix bad merge * Checking in a staging checkpoint point so that Ryan can work with me in parrallel * build break. * Brianma/testfails (#2473) * add missing ir version to dictvectorizer-string.onnx * add missing ir version to relu.onnx * add missing ir version to zipmap*onnx * add IR version to manually generated models * remove an unnecessary ifdef dml * Brianma/windowsai fi (#2475) * update dockerfiles/README (#2336) * Make elementwise op run 4 items per thread (#2335) Description: Describe your changes. Make elementwise op run 4 items per thread unroll for loop to leverage ILP remove unnessary N==0 check inside elementwise GPU kernel Motivation and Context Why is this change required? What problem does it solve? It can improve the performance of GPU elementwise ops. ~2% performance gain on popular NLP bert model. If it fixes an open issue, please link to the issue here. * Add CUDA GatherElements kernel (#2310) * Updates * Update test * Update * Updates * nits * PR feedback * Update * Update * PR feedback * PR comments * Update * Fix build * Fix build * Nits * Fix * Layer Normalization Fusion (#2319) basic layer normalization transform * Add FastGelu Cuda Op for Gelu and Add bias fusion (#2293) * Add FastGelu cuda op * Add AddBiasGelu for experiment * Revert "Add AddBiasGelu for experiment" This reverts commit 5c1ee019858c657e6bb75887265cb85675626e5b. * Add bias * Add unit tests * update comment * update script * fix build error * update coding style * update for CR feedback Enable half2 optimization only when cuda arch >= 7.0 * move _Tanh to common.cuh * implement CPU contrib OP Attention (#2333) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanUnusedInitializers. (#2320) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanupUnusedInitializers. This means initializers that have been replaced during graph optimizations are not left in the GraphProto when we save an optimized model. * Handle edge case where a model has an unused initializer with matching graph input by also removing the graph input. * Use non-const iterators in std::find_if calls to make centos build happy. * Nuget pipeline changes (#2305) 1. refactor the pipeline, remove some duplicated code 2. Move Windows_py_GPU_Wheels job to Win-GPU-CUDA10. We'll deprecated the "Win-GPU" pool 3. Delete cpu-nocontribops-esrp-pipeline.yml and cpu-nocontribops-pipeline.yml 4. In Linux nuget jobs, run "make install" before creating the package. So that extra RPAH info will be removed * Cuda Reverse Sequence Op, maping types of same size using same template function. (#2281) * Set ElementType to String type of node metadata, instead of byte[] (#2348) * Set ElementType to String type of node metadata, instead of byte[] * Fix spacing * Introduce PrimitiveType into a Type System along with an integer constant (#2307) Improve perf by avoiding GetType<T>() calls. Introduce MLTypeCallDispatcher to switch on Input Type. Add Tensor IsType<T>() fast method. * Fix/test dim value of 0 handling in a couple of places (#2337) * Update the CUDA Where implementation broadcasting logic to handle a dim with value of 0. Add unit test Also add unit test for unary op with dim value of 0 * Exclude ngraph from Where test with 0 dim. * Openvino EP R3.1 onnxrt server (#2357) * onnxrt server with OVEP * onnxrt server with OVEP * Update Dockerfile.server.openvino * onnxrt server OVEP fix reviews * onnxrt server OVEP fix reviews * Implement cuda nonzero op. (#2056) Implement cuda nonzero op. * Direct use python numpy array's memory if already contiguous. (#2355) * Direct use python numpy array's memory if already contiguous. This could greatly improve performance for session with large input, like big image 1920x1080 fastrcnn, 30~40% speed up could be achieved. * Add test case enforce contiguous/non-contiguos numpy array as inputs. * Add helper to create output to minimize binary size. (#2365) Add ConstEigenTensorMap typedef so we don't unnecessarily const_cast the const input Tensor. * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS (#2369) * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS * update * Add Tracelogging for profiling (#1639) Enabled only if onnxruntime_ENABLE_INSTRUMENT is ON * test bidaf with nuphar for avx target (#2370) increase nuphar test coverage a bit * Fix a bug in TLS refcount that may destabilized CUDA CI (#2374) * update output size calculation for resize (#2366) * change how output size is calculated for resize op * add tests for ver 10 resize * Extend OneHot CPU kernel to support more types (#2311) * Extend OneHot CPU kernel to support input int64_t, depth int32_t, output float * Skip BERT before the test data fix is picked up * Fix bug with Slice. Need to pass in flattened input dimensions so the initial offset into the input is calculated correctly. (#2372) * Add opset 11 version of Split to CUDA ops (#2376) Organize the CUDA ops definitions so all the opset 10 and 11 parts are together (same setup used for CPU ops) * Layer Norm Fusion Fix (#2379) * layer norm fusion fix * Add input shape check in code and unit tests * Fuse Add + Gelu (#2360) Implement the transformer to fuse add + gelu Implement the accurate kernel * Skip layer norm transform (#2350) * skip layer normalization transformer * Another try to stabilize CUDA CI (#2383) The root cause seems to be failure in CUDA dealloc when tear down. cudaFree return code was ignored before, so should the debug check. * fix BUILD.md typo (#2375) build.py: error: argument --config: invalid choice: 'RelWithDebugInfo' (choose from 'Debug', 'MinSizeRel', 'Release', 'RelWithDebInfo') * Fixed compilation with ngraph (#2388) * Fix reuse logic in allocation planner. (#2393) * Fix reuse logic in allocation planner. * PR comments * Add helpful comments * Don't allow reuse across string tensors. * [NupharEP] Multiple optimizations (#2380) Fuse transpose into MatMul Implement Pow and constant scalar simplification Vectorize ReduceMean Improve symbolic shape inference Minor updates for better debugging in fused function name * Avoid using the default logger in the graph lib and optimizers (#2361) 1. Use the session logger if it is available. 2. Don't disable warning 4100 globally. We should fix the warnings instead of disabling it. * Change CUDA implementation of Transpose to support all fixed size tensor types (#2387) * Change CUDA implementation of Transpose to not use a typed kernel so we can support more types with minimum binary size. Add support for 8, 16, 32 and 64 bit types. Add unit tests. Add method so the implementation can be called directly (will be used by CUDA Scan very soon). * Disable TensorRT for MLFloat16 and int8 unit tests. * Address PR comment and add support for calling cublas implementation if type is mlfloat16. * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. (#2398) * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. * [NupharEP] force some low/zero cost ops to be inlined (#2409) * fix cross compile bug (#2415) * Minor optimization: if a node has already been placed, there's no need to find a kernel for it. (#2417) * Add Reshape Fusion (#2395) * Add reshape fusion * Add some comments * update comments * update comment format * update according to feedback * update for recent logger change * fix build error * (1) Support both input and output edges in find path in graphutils (2) Add a test case of only one constant initializer of Concat input. (3) Refactor ReshapeFusion class to allow add more subgraph fusion in the future. * fix error * (1) loose constraint on initializer: non constant is allowed for reshape fusion. (2) Change versions type to vector. (3) Add logging. (4) Return false when multiple output edges matched in FindPath. Add comments. * only allow one direction (input or output) in FindPath * [NupharEP] Update notebook and docker image (#2416) Add BERT squad in Nuphar tutorial Enhance speed comparsion readability * Fix the issue in matmul_add_fusion (#2407) Fix the issue in matmul_add_fusion If Muatmul + Add has shape [K] * [K, N], reset it to [1, K] * [K, N] will make the output shape to [1, N] will also requires a reshape on the output. Fix: just remove the shape reset to not fuse it. Add a negative test case for matmul+add fusion * feat(treeregressor): Update TreeEnsembleRegressor for type support (#2389) Updates the `TreeEnsembleRegressor` to allow for `double`, `float`, `int64`, and `int32` inputs to match the upstream specification. Signed-off-by: Nick Groszewski <nicholas.groszewski@capitalone.com> * onnxrt server documentation update (#2396) * Added support for Pad-2 operator in OpenVINO-EP (#2405) * Add CUDA If operator. (#2377) * Add CUDA If operator. Uses CPU operator for implementation. By adding a CUDA version the inputs/outputs (with the exception of the 'cond' input) stay on GPU, and no other logic is required to avoid a copy to CPU across the control flow node. * Improved documentation for onnxruntime::utils::SwapByteOrderCopy(), added precondition check. * Fix the type constraints on CUDA If operator to exclude strings. (#2431) * add Im2col<uint8_t> (#2438) * Adjust codegen vectorization width from target (#2439) * Adjust codegen vectorization width from target * Add CUDA Scan operator. (#2403) * Add Scan CUDA op. Uses CPU implementation for logic. Added some device specific functors for handling when data needs to be manipulated on a different device. Added ability to override the materialization logic in the OrtValue slicer so DML can plugin their handling. * Fix Windows GPU C API packaging pipeline failure (#2440) Fix Windows GPU C API packaging pipeline failure (#2440) * Correctly handle implicit inputs for fused nodes (#2390) * Correctly handle implicit inputs for fused nodes Previously, nuphar's partitioning function didn't include node's implicit inputs into the inputs list of MetaDef, and hence a crash was triggered in the onnx graph checker. This commit fixed the issue. Furthermore, it also fixed a related issue where we didn't add implicit inputs into graph_inputs_excluding_initializers_ in Graph::SetGraphInputsOutputs. the issue was that graph_inputs_including_initializers_ populated by SetInputs (e.g. called by FunctionImpl::FunctionImpl) may contain implicit inputs which were not of any node's initializers in the graph. Because they were not part of any initializers, these implicit inputs couldn't be visited by going through all nodes' inputs. Consequently, they would *not* be added into graph_inputs_excluding_initializers_. We fixed the issue by first copying the populated graph_inputs_including_initializers_ into graph_inputs_excluding_initalizers_, which then had both initializers and non-initializers as its initial content. Later, we erase initializers from the list. In this way, we can ensure all implicit inputs to remain in graph_inputs_excluding_initializers_. * refined comments and fixed duplicates Address CR by revisiting comments in terms of implicit inputs Also fixed an issue by skipping duplicates while copying inputs from graph_inputs_including_initializers_. * address CR explain why we need to collect nodes' implicit inputs * don't rely on pointer values for iterating std::set Previously, openvino relied on iterating a set of NodeArg pointers to construct inputs and outputs for a fused graph. It could cause non-determinism. The reason was that although iterating std::set by itself is stable, pointer values of NodeArgs may vary. Consequently, we could end up visiting the set's elements in different orders for different runs for the same test, which resulted in constructing inputs (and outputs) with different orders to the fused graph. For example, for the same test, we may have inputs [A, B] in some runs but inputs[B, A] in others. Let's use std::string as the key type to avoid such nondeterminism. This commit also added implicit inputs into meta->inputs while returning the capability from the openvino provider. * Fixed another latent issue in openvino's GetCapability function The issue was that we couldn't simply erase fused_inputs and fused_outputs while iterating the nodes. For example, an output NodeArg may have multiple uses, and it's wrong if we erase it from fused_outputs when we encounter only one of its uses as input. * Remove DeviceAllocatorRegistry class (#2451) Remove DeviceAllocatorRegistry class * CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library. * Parallel Gelu with ParallelFor (#2399) Parallel Gelu to get better performance for Gelu * Clean up build.py (#2446) * Pull the latest image before running docker build * Fuse SkipLayerNorm with Bias (#2453) Fuse SkipLayerNorm with Bias * Allow more than one invocation of CreateEnv in the same process. (#2467) * Allow more than one invocation of CreateEnv in the same process. * Fix centos build * Symbolic shape inference improvements: (#2460) * Symbolic shape inference improvements: - add a mode to guess unknown ops' output rank - add support for GatherND - add support for If - fix a bug in get_int_values when then tensor rank > 1D, by treating it as no sympy data - add symbol to literal merge when ONNX silently merges dims - fix a bug in Concat when input dim is 0 - fix a bug in ConstantOfShape that computed dim is not updated - add support for dynamic shape in ConstantOfShape - fix a bug in Loop output shape that loop iterator dim is not inserted at dim 0 - add support for dynamic padding in Pad - add support for dynamic shape in Reshape - add support for Resize with opset > 10, by treating output dims as dynamic - fix a bug in Slice when starts/ends are dynamic - restrict input model to opset 7 and above - make output model optional to avoid disk write when testing Run model tests for symbolic shape inference Reduce 2GB docker image size of nuphar * add additional test data set for nuget pipeline (#2448) * add SAS token to download internal test data for nuget pipeline * update azure endpoint * fix keyvault download step * fix variable declaration for secret group * fix indentation * fix yaml syntax for variables * fix setting secrets for script * fix env synctax * Fix macos pipeline * attempt to add secrets to windows download data * fix mac and win data download * fix windows data download * update test data set url and location * Revert "Brianma/windowsai fi (#2475)" This reverts commit 5780b864a15513fda4eadbfc2b5345fefe70b5ec. * Add scenario tests (#2457) * Add scenario tests * Remove TODO from model license * Add winml_api test dependency * fix model load test. fi from master changed the constructor (#2483) * make api tests all pass (#2486) * fix bad merge * fix bad model merge * Layer dev paulm (#2492) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * Rename ambiguous header (#2489) * fix one more missing IR version model (#2500) * add missing IR version to 4 more models used by scenario tests (#2501) * Add CLI parameters to test runner, build WinML in ARM and x86 CI (#2479) * Support test parameters through CLI arguments * Add WinML do Windows x86/ARM CI builds * Code style fixes * Update googletest Remove GPUTEST macros everywhere now that GTEST_SKIP is supported * Refactor main.cpp * Build scenario tests without DML * Link scenario tests to DML when it's enabled (#2502) * Layer dev release pipeline (#2488) Adds winml binaries to existing cpu nuget package, and creates new gpu dml nuget package with winml binaries and DML EP. * Layer dev paulm (#2506) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * Remove usage of IOBinding in WinML and use C_API Run method (#2504) * remove usage of iobinding * Change data structure to use vector of Ort::Values * Polish bind input / output * Use C APIrun method * Update providers on evaluate getresults * Remove run and IObinding interface from WinMLAdapter * Remove use of IObinding * bind unbound outputs code moved to learningmodelbinding * clean up unneeded istensor adapter function * Fix comment * Check if session is closed before binding and clearing * PR feedback * Layer dev paulm (#2507) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * Make tests dependend on winml_dll (#2509) * add dml binaries to DirectML package and be more explicit about condition variables (#2520) * re-enable warnings for winml builds and fix the warnings that were hiding (#2526) * turn devmode back on for winml builds * fix some warnings. include protobuf in a way that disables some warnings * undo protobufhelpers changes and just ignore 4100 errors in pb code * attempt to isolate protobufhelpers errors * add template specialization for getting tensor proto data * Layer dev paulm (#2533) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * Spawn child process to run DeviceLostRecovery scenario test (#2530) * Spawn child process to run DeviceLostRecovery scenario test * Layer dev paulm (#2536) ori said yes * add missing namespace to winml_trace_logging_provider in lotusenvironment.h (#2542) * Handle exception thrown from all apis in WinMLAdapter (#2539) * various changes to unblock windowsai ADO build * Fix custom ops scenario tests (#2562) * Do not shutdown protobuf after ort environment gets destroyed. Lazy load lotus environment first time it is needed * comment typo * pr comment about calling phoenix singleton * Make lotus_environment static in winmladapter * Layer dev paulm (#2567) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * making windowsAI azure dev ops work. * code review comments. * revert changes * Cmake and preprocessor fixes that where uncovered by building on agents without DML available via SDK * Layer dev dml delayload (#2580) * Brianma/cpu (#2583) * don't include dml stuff in cpu builds * tests that link the image lib also need the telemetry lib now * Throw Winml_err_invalid_binding if binding gpu resource on cpu device (#2589) * Throw Winml_err_invalid_binding if binding gpu resource on cpu device * PR comments. No need to query executionprovider for is gpu device * User/xianz/ortthrow (#2596) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * User/xianz/ortthrow (#2599) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * add status error message * Remove uwp onsuspending winrt call because logruntimeperf is getting removed (#2630) * User/xianz/dedup telemetry (#2631) * investigate duplication of telemetry in winml and ort * remove winml telemetry events * telemetry executionProviderEvent * remove unneccessary file and refactor code little bit * Revert back TelemetryEvent, which send up ETW event. * merge changes from layer_dev to windowsai (#2638) * Remove underscore from googletest names (#2616) * Fix leaking memory allocator Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24278761 and https://microsoft.visualstudio.com/OS/_workitems/edit/24330198 * Explicitly initialize Ort::Value with nullptr * Cache WinML adapter * bad merge * define private version of dxcore enum that is added in 19H1 SDK. (#2654) * add comment for explaning private definition of dxcore d3d feature level ennum value. (#2672) * do not package directml.pdb for redist packages. (#2676) * Fix leaking operator registry (#2645) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24354916 * User/orilevari/windowsai master merge (#2674) merge resolutions included pulling in telemetry logic that was merged to master and not windowsai and dereferencing InferenceSession::sessionstate now that it is a unique pointer * Delete Ort Allocator in LearningModelBinding (#2653) * Delete OrtAllocator in LearningModelBinding * PR comments to make Ort::Allocator a smart pointer * Small comment change * PR feedback to clean up code * PR feedback on move semantics * Clean up std::move * Fix memory leaks (#2679) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24356109, https://microsoft.visualstudio.com/OS/_workitems/edit/24388361 and https://microsoft.visualstudio.com/OS/_workitems/edit/24388596 * various changes to properly organize and skip GPU tests. For now for No DML builds we will not run GPU tests at all. In the future we should adapt the tests to expect the appropiate errors. (#2695) * Windowsai without fi (#2701) * Disable Attention fusion tests when DISABLE_CONTRIB_OPS is defined (#2529) * Setup java ci (#2528) * Add provision in ORT for session options to be parsed when available via model file (#2449) * Initial commit * Fix gitmodules * Nits * Nits * Updates * Update * More changes * Updates * Update * Some updates * More changes * Update * Update * Merge * Update * Updates * More changes * Update * Fix nits * Updates * Fix warning * Fix build * Add comment * PR feedback * PR feedback * Updates * Updates * Update * More changes * Fix build break * Comment test for now * Updates * Updates * PR feedback * Updates * Nits * Add tests * Fix build * Fix build * Fix build * Fix build break * Fix build * Nits * PR feedback * More change * Expose GetSessionOptions in pybind logic and add unit test for python * Fix build * PR feedback * PR feedback * Revert "Disable thread pool creation when enabled OpenMP (#2485)" (#2535) This reverts commit 7c7d5a149c9ed52eec67304bae5c4b132166a8a1. * Add dynamic shape support in TensorRT execution provider (#2450) * remove onnx-tensorrt submodule * add new onnx-tensorrt submodule (experiment) for trt6 * update engine build for trt6 * update compile and compute for tensorrt6.0 * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * switch to onnx-tensorrt master for TensorRT6' * Update tensorrt_execution_provider.cc * Handle dynamic batch size and add memcpy in TensorRT EP * update test cases * Update tensorrt_execution_provider.cc * update onnx-tensorrt submodule * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.ubuntu_tensorrt * Update run_dockerbuild.sh * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update concat_op_test.cc * Update tensorrt_execution_provider.cc * Upgrade TensorRT to version 6.0.1.5 * Update onnxruntime_providers.cmake * Update CMakeLists.txt * Update reduction_ops_test.cc * Update install_ubuntu.sh * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.tensorrt * Update BUILD.md * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update onnxruntime_providers.cmake * Update install_ubuntu.sh * Update install_ubuntu.sh * Update gemm_test.cc * Update gather_op_test.cc * Update CMakeLists.txt * Removed submodule * update onnx-tensorrt submodule * update header file * Removed submodule * add submodule onnx-tensorrt kevin's branch shape-test' * add debugging code * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * merge master * Removed submodule * update onnx-tensorrt submodule * add more changes for dynamic shapes * Update tensorrt_execution_provider.cc * update for dynamic shape * update dynamic shape processing * fix logger issue * remove submodule onnx-tensorrt * add submodule onnx-tensorrt * add env variable min_subgraph_size * remove redundency * update document * use onnxruntime::make_unique * fix multi-run issue * remove some tests to save CI build time * Add dynamic shape test * Update TensorRT-ExecutionProvider.md * Add example of running Faster R-CNN model on TensorRT EP * Add more details on env variables * update environment variables * Update tensorrt_basic_test.cc * Update model tests * Update tensor_op_test.cc * remove --use_full_protobuf * Update build.py * User/xianz/telemetry (#2458) * enabme telemetry * enable telemetry * set enable telemetry as default * for debugging * remove log and set disable telemetry as default back * delete private file while testing * resolve comment: mainly add license header, rename macro and update docs * rewording in privacy.md * Fix integer overflow in cuda NonMaxSuppression implementation (#2540) * add test case that should pass but fail * fix nms * extract int_max_output_boxes_per_class * Introduce container type runtime checks and other improvements (#2522) Rework TensorSeq in a manner consistent with Tensor and SparseTensor in terms of type system setup. Reduce templating. Introduce helpers to ensure the same data type. Make OrtValue __dtor not virtual. Introduce ContainerChecker * Fix C API tests for centos and mac (#2544) * change c++14 to c++11 * add ld lib path for centos * enable csharp tests on macos * fix C API test on MacOS + fix manylinux dotnet install * fix manylinux dotnet install * fix lib link * Add back executable bit to build.py * Fix a bug handling negative begin pad values in Pad op (#2550) * Fix bug in Pad op * Update * DNNL CMAKE update (#2548) * Fix android build (#2558) * Update win-x86-ci.yml (#2557) Fix build pipeline break * Re-enable Windows C# tests (#2564) * disable onnx_test_runner -x invocations for dnnl (#2568) * Allow sequence length to be symbolic (#2559) * setup java ci mac (#2570) * make layernorm fusion to support opset 11 (#2545) * Fix a warning found in the latest VS release * Add more check on SkipLayerNorm and BiasGelu fusion (#2574) * Fix file not found error during docker build. (#2569) * Add ConvTranspose1D (#2578) * Ryanunderhill/packagename test (#2582) * [Nuphar EP] fixes for some object detection models (#2581) Update notebook tutorial with multi-threaded int8 GEMM from #2517 * EmbedLayerNormalization Fusion Improvement (#2553) Embedding layer norm fusion improvements - add more checks * Update version (#2584) * Temporarily exclude vgg19 test from Python backend test 1. temporarily exclude vgg19 test which comsumes too much memory, run out of memory on Upsquared device. Single test pass for vgg19, need furture investigation (#2588) 2. Update docker file to decrease the docker image size * Update docs for Android NNAPI EP (#2586) * Fix lto bug for protobuf and ubuntu * add path to build dir before test run (#2590) * Add missig env variables for mac pipeline test (#2595) * Fixed an issue in updating realized dims (#2597) when we update realized dims for scan's output, the sliced axis also needs to be inclusive, i.e. we should check with "dim >= insert_inclusive_axis", because the offset in the symbols are based on Scan sugraph. Otherwise, we would end up with shape mismatch later. * Java API for onnxruntime (#2215) * Add support for opset 11 in reshape fusion (#2592) Support opset verion 11 in reshape fusion * Rename automl python tools folder to featurizer_ops. (#2593) * Support opset 11 subgraph of Squad model in Embed Layer Normalization (#2605) Support opset 11 Squad model that is exported from PyTorch nightly. The embed layer uses Range op which is missed in the transformer. * symbolic shape inference: fix warnings in GPT-2 model (#2608) And revise nuphar perf test on BERT squad * Dump subgraph ID and fused graph ID (#2607) * Dump subgraph ID and fused graph ID Dump subgraph ID and fused graph ID for better debugging * Remove local static fused_count added a field global_fused_count_ to NupharExecutionProvider class * EmbedLayerNormalization Fusion For Dynamic Squad Model Opset 10 (#2613) Support subgraph of SQuAD model exported from pytorch with dynamic input axes * Allow providers to be set for InferenceSession at construction (#2606) * Remove unnecessary parameter in some places in GatherElements implementation (#2612) * Remove unnecessary parameter in some places * Update * Update * Make sure fenced tensor could not reuse other tensor. (#2561) Fix random error caused by this. * Improve Embed Layer Norm Fusion for SQuAD with static input shape (#2621) * fix float16 comparison in initializer (#2629) * epsilon attribute for layernormalization fusion (#2639) * removed unnecessary batch file and fix path (#2640) * Add shape inference to ConvTransposeWithDynamicPads schema (#2632) * Improve cuda expand() opeator's performance. (#2624) * Cuda pad optimize when no padding is needed. (#2625) * Shortcut cuda Pad() when no padding is needed. * Optimize cuda scatter() on 2D compatible. (#2628) * Optimize cuda scatter() on 2D compatible. * Add some comments. * fix build error for ARM (#2648) * Improve performance of resize() in Nearest mode (#2626) Special treatment for 2D, check same size as input image. And in 2d kernel, template use_expolation. * Fix memory exception in Layer Norm Fusion (#2644) * Windows CI changes(#2650) * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Revert "Windowsai without fi (#2701)" This reverts commit 285d4c85ff5c4e265f963208170304ef3461e684. * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Deref unique pointer for session_state * send shutdown event when dll is unloaded and EvaluationStop, SessionC… (#2704) * send shutdown event when dll is unloaded and EvaluationStop, SessionCreationStart Events. * Add EvalutationStart Event * add comment * use correct type for for loop (#2755) * ARM CI (#2759) * Set ARM agent pool * Set CMake generator to VS 2019 in ARM * Use system-wide CMake instead of custom version Our custom version is too old for VS 2019 * Use DML and build shared lib in ARM CI * Restore nuget packages in ARM CI * Disable DML * Refactor ARM debug/release builds * Use system packaged Python version * Remove hardcoded Python path * Downgrade Python to 3.7 for build * Remove explicit CMake path * Fix invalid JSON in cgmanifest.json (#2760) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * Remove usage of WHOLEARCHIVE in WinML CMake and add WinMLAdapterFactory (#2726) * Remove usage of WHOLEARCHIVE in WinMLAdapter CMake and add WinMLAdapterFactory * PR feedback, no need for dll(export) since using def file * PR comments * Small comment in gen_def.py * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * Move winml_provider_factory.h to proper location (#2801) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Filter CPU case for IsFloat16Supported (#2802) * Merge fixes * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * Fix warnings that cause build to fail * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * Make DML include path global (#2882) * Make DML include path global * Add generated cppwinrt headers to winml_lib_common * Integrate changes to WindowsAI to make ADO Build (#2886) * Revert "CMake cross-generator fixes (#2790)" This reverts commit dbe7d97fa1ab155f1309bced87199527e8f35bd2. * add additional suppress warning in onnx_proto * ignore /wd4996 warning * DML execution provider fixes * Revert "Revert "CMake cross-generator fixes (#2790)"" This reverts commit 1ae7b4bcbc02edc881ad28685da98e095dfceb17. * Update func signature of custom op function overloads * common devicehelpers fixes * Add pch.h for winml_lib_common * re-add winml_lib_common_dir/inc to include path for winml_adapter * User/orilevari/dml redist shared folder (#2890) * move dml nuget package directory up one level to make it shared between build flavors * Merge conflict fix * Revert "Merge conflict fix" This reverts commit 142fa72cf9ce4344ad717b50b7ea2b8582aadc7c. * Revert "Merge remote-tracking branch 'origin/master' into windowsai" This reverts commit 6e2126d46e5e5f564d65da37dd4f70c93dd81165, reversing changes made to b3f5583dc9249834b947c8ea905f6a98060d5bd6. * Make winml_test_common free of test macros (#2902) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Don't generate files outside the build root (#2914) * Don't generate files outside the build root * Add onnxruntime_EXTERNAL_DEPENDENCIES to WinML * Add DML depedency on RESTORE_PACKAGES * User/orilevari/fix yaml merge bugs (#2918) * Add winml test source parameter into cmake function (#2919) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Add arguments to unittest cmake functions * remove comment * Revert "Revert "Merge remote-tracking branch 'origin/master' into windowsai"" This reverts commit ade5abe72a4234fdbc3623093c61c02c6b0bdc26. * Fix breaks from merge with ORT master * Brianma/linux (#2917) * don't include windows.h in cross-plat header * add default case for switch statement * signed/unsigned mismatch fix Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> * User/sheilk/winml adapter c api (#2891) * Create winml adapter c api * fix build * make it build * move adapter into onnxruntime core/session * entry point not exported * minor changes * make model metadata work * make tests pass * implement all the model reflection apis on the adapter c abi * update the new ort interface to create a lotus ennvironment with a logging sink * start adding ort env * move all winml code into adapter folder/lib to isolate it * ensure a single logging manager at a time * start refactoring session * refactor session creation interface * add cpu and dml session option methods to adapter * finish session init * stub out interfaces in ort lib to perform similar mechanics of iinference session * enable profiling, and enable schema override * update session register graph transformers * turn back on custom registry for custom ops * Add sync api * add last c api stubs * should build... but all feature values are broken since this is in flight to moving all implementation details into ivalue * remove ep adapter header * Implement DML execution provider functions from adapter (#2846) * Implement DML execution provider functions from adapter * Use functions in OnnxruntimeEngine.cpp * make map/sequence type_infos freeable, and start implementing ivalue * make it build again * implement value methods * implement remaining methods * remove com adapter abi * check dml session * cache the allocator on ivalue * check if resource is cpu/gpu when access its mutable data * update tensor * mismatched parentheses * fix tensor base and binding obj * it evaluates tensors! sometimes... * minor fixes * enable gpu evals * wrapper all existing winml adapter apis with API_IMPL to try catch (#2854) * update winml... tensor strings are broken, need to template tensorbase to do different things for strings * make tensor strings work with 2 copies in/2 copies out * Fix tensor string and allocator bug * make maps work again... needs some fixes still * Make it build! * enable map inputs * map outputs * unbound outputs for sequences and maps * User/xianz/merge windowsai (#2883) * Packaging pipeline changes for VS 2019 (#2711) * Tiny fix to codegen * Simplify cache implementation and avoid static variables that may carry over between models * Extend DML kernels (#2641) * Additional DML operators * Check unsupported attributes and inputs * Address PR comments * Add kernel capability function used for partitioning, and re-enable stride-based int64 support based on value range * Fix test failures * Build fix * PR comments * Update Nuphar tutorial notebook (#2721) 1. Reflect int8 GEMV improvements for multi-threading from #2696 2. Add notes on multi-threading control using OpenMP 3. Add samples of running multi-isa AOT, and show int8 GEMM differences between AVX and AVX2 4. Add rnn_benchmark example to resolve #1993 * Add schema for new Qops (#2611) * Add schema for new Qops * adding shape inference + qlinearaveragepool * plus review comments * plus review comments * updates per review comments * plus review comments * [server] Add supposed for model_name and model_version as cli parameter (#2708) * remove 64bit warning message from python validation. (#2727) * MLAS: ARM64 build fix (#2734) fix bad usage of vreinterpret to cast vector element types * Fix broken python docs links (#2740) * Fix build on Mac OS (#2731) mac os ld doesn't support --while-archive, correct option is -all_load * fix ngraph wheel (#2737) * fix ngraph wheel 1.1.0 onnxruntime_ngraph wheel doesn't work * remove libdnnl.so in nGraph Libs * make it easy to compare * Split onnxruntime server to a separated folder (#2744) * Fix build for Python 3.8 (#2747) * Fix build for Python 3.8 * Update protobuf to 3.11.2 (#1928) Update protobuf to 3.11.2 (#1928) * Change default optimization level to All (from Basic) (#2745) * change default optimization level to All (from Basic) * fix test * fix c# test * Update numpy to 1.18 (#2758) * Update numpy to 1.18 * Pipeline changes for python 3.8 (#2753) 1. Pipeline changes for python 3.8 2. Fix a regression in setup.py which was just introduced in the previous commit. Please notice, we still haven't made python 3.8 + Windows + CUDA work. * Add basic stacktrace output for posix debug builds. (#2749) * [NupharEP] fix a race condition when multiple sessions running different models concurrently (#2772) * Revert "Change default optimization level to All (from Basic) (#2745)" This reverts commit 56bb503c2f26474b6613bcb2a198691a11dcef38. * Fix typo in error message (#2736) * Rename MKL-DNN to DNNL to fix broken link (#2730) * Fix nightly build version number issue * Pass BUILD_BUILDNUMBER to linux docker * Disable featurizers in python packages * Import more featurizers (#2781) Make kernels non-template. Add input constraint for learnt data. Add min_max_scalar_transformer, robust_scalar_transformer, inputation_marker_transfomer, label_encoder_transformer, missing_dummies_transformer along with tests. Advance Featurizers library commit. * Implement a more stable softmax (#2715) * Implement a more stable SoftMax e^x is represented as infinity if x is large enough, like 100.f. Infinity divided by Infinity is a NAN. Thus, softmax gets a NAN if one or more item are large enough. A math transform as below is leveraged to get a stable softmax: e^xi/(e^x1 + ...e^xn) = e^(xi - max) / (e^(x1 - max) + ... + e^(xn - max)) And for convenience, force max to 0.f if all xi are negative * Contributing: Fix a typo (#2784) * ACL EP GEMM improvements (#2780) When it is posible we use a fully connected layer instead of the gemm implementation. This will let the library use the best implementation based on the input data. * ACL EP convolution improvements (#2774) Added the optimized implementation for depthwise convolution for both ACL v19.02 and ACL 19.05. Also the pointwise convolution seems to be more optimal in the CPU implementation so we opted for that instead. * Add script for release Nuget validation (#2719) * Initial commit * Nits * Disable a test temporarily * Change working directory * Test * Add download python step * Test update * More changes * Fix space issue * Fix * Verify nuget signing * Fix * Spaces * PR feedback * Nit * Fix * Fix * Remove temporary changes * add uint8 support to where op (#2792) * Improve bert optimization script: (#2712) (1) Move input int64=>int32 conversion to embed layer fusion. (2) Output epsilon attribute for LayerNormalization fusion. * add session creation time cost. (#2798) * ML.NET team needs featurizers within a package (#2789) Add auto ml featurizers to Windows, MacOS as well as to GPU packaging-pipelines. * Initialize max of softmax with lowest of float (#2786) * MLAS: update SGEMM threading parameters (#2808) * add interface to copy batch tensors. (#2807) * add interface to copy batch tensors. * onnxruntime * speed up Windows TRT CI (#2811) * don't run cuda tests if building with tensorrt * remove unnecessary build options for win trt ci * refactor win gpu tensorrt ci yml * --numpy_version=1.17 * update * update * azcopy and cuda path * Update test data (#2356) * Add timeseries imputer transformer featurizer kernel (#2813) Make kernels non-template. Add input constraint for learnt data. Fixup tests. Add two more featurizers along with tests. Tests fail. min_max_scalar_transformer robust_scalar_transformer Fix tests serialized stream by prepending version bytes. Add inputation_marker_transfomer and the test. Fix up float/double type designations. Added label_encoder_transformer along with a test. string_throw case is broken at the momement. Fix labelencodertransfomer_test.cc string_throw case Rename maxabsscalertransformer_test.cc Add MissingDummiesTransformer along with the test. Update manifest. Add TimeSeriesImputerTransformer definition, implementation and tests * Fix memory leak in TRT (#2815) * fix memory leak issue * revert EP_FAIL on enueueV2 * Add manifest missing comma * Run static code analyzer on most of our code (#2817) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * update quantization doc (#2783) * update documentation for quantization script * plus some spell corrections * Filter CPU case for IsFloat16Supported (#2802) * update default optimization level + fix gemm_activation fusion (#2791) * update defualt optimization level + fix gemm_activation fusion * fix typo * add unit test and incorporate review comments * fix test comment * Fix dnnl wheel package name (#2823) * Append '-dnnl' to whl package name when --use_dnnl * Update build.py * Update Ubuntu & TensorRT version in README (#2820) Dockerfile.tensorrt is using nvcr.io/nvidia/tensorrt:19.09-py3 as base Image, update Ubuntu and TensorRT version according to https://docs.nvidia.com/deeplearning/sdk/tensorrt-container-release-notes/rel_19-09.html#rel_19-09 * Merge fixes * Add OneHotEncoder and HashOneHotEncoder kernels. (#2830) Add defs and imlementation for OneHotEncoders, adjuist date_time_transformer kernel and test. Add OneHotEncoder kernel test. Add HashOneHotVectorizerTransformer unit test. This does not link due to multiple definitions of functions that are included into header from a CPP file. * Upgrade gtest to the latest version (#2827) WinML would like to update the googletest submodule. They want some newer features (namely GTEST_SKIP to skip tests programmatically and be able to skip entire fixtures easily) and would need to update the submodule version. However, because the new version of code hit a bug in gcc, even though the bug is already fixed in the latest gcc but we're using gcc 4.8.x and it won't get patched for the bug, so we have to do a compromise, change our code a little bit to make it work. The gcc bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=51213 * Add support for int64_t for topk CPU. Fixes github issue #2806. (#2833) * Ignore allocator type in ExecutionProviders allocator map. Make default initialization of OrtMemoryInfo more clearly invalid. (#2768) * Remove allocator type from the key comparison in ExecutionProviders. Remove usage of DummyArena as it's no longer necessary. * Fix x86 tests where arena allocator is disabled. Make initialization of OrtMemoryInfo clearer by adding Invalid enum value. * Make OrtValueNameIdxMap::MaxIdx more intuitive. * Convert ExternalProject Featurizers into git submodule (#2834) Add git submodule for Featurizer library. Update cmake to build for git submodule. * add domain check for nodes + update documentation (#2831) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Fix Linux CUDA nuget packaging pipeline break * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * update optimization doc for BERT related fusions (#2819) * Add bert related transformers to doc * Add execution provider and comment for bert optimizations * Add comment about accuracy impact of approximation * Fix warnings that cause build to fail * MLAS: enable threading for quantized GEMMs (#2844) * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * remove old winmladapter cpp Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * move sequence implementation into ort lib... still commented out... need to turn back on... * begin sequence implementation * make maps and sequences work * fix broken tests * remove dead code * misc cleanup * CR feedback * User/xianz/winml adapter c api (#2869) * wrapper all existing winml adapter apis with API_IMPL to try catch * Return HR or Throw for WinML adapter APIs if failed * undo macro wrapper for two places * Wrap error macros around ort apis, too. * address CR feedback #2 * add more api throw/return macros * Revert changes no longer needed * revert changes to cxx api * format winml lib.ort and winml adapter * remove static pheonix singleton Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * missing use_dml check in winml_adapter_session (#2930) * --use_dnnl flag was mangled in merge (#2931) * use dml macro not wrapping custom registry code (#2934) * Disable LNK4199 winml_dll to enable cuda builds (#2936) * Disable LNK4199 in winml_dll * linkler->linker * LearningModelSessionAPITestGpu.CreateSessionWithCastToFloat16InModel should return DXGI_ERROR_UNSUPPORTED when FP16 not supported (#2937) * Disable LNK4199 in winml_dll * linkler->linker * Need to return DXGI_ERROR_UNSUPPORTED when Model does not support fp16 * Publish build symbols (#2939) * Publish build symbols * Don't upload PDBs for .exe files * Make x86 build (#2943) * fix last remaining size_t/int64_t warnings->errors (#2948) * TensorString, Sequences and Maps use the first allocator, but should use the cpu default allocator. (#2952) * fix tensor string allcoator * clean up default allocator usage for strings in winml lib/api.ort Co-authored-by: Ryan Lai <ryalai96@gmail.com> * Handle tensor shape of zero (#2954) Co-authored-by: Ryan Lai <ryalai96@gmail.com> * CR feedback (#2970) * CR feedback * fix weird formatting on privacy readme * Add 'All rights reserved.' everywhere * readd all rights reserved to winml_provider_factory.h * remove extra space in comment * remove extra whitespace * fixes post master merge * remove winml from nuget gpu pipeline * set IR VERSION on generated_model in rnn_benchmark (#2972) * Fix slice conformance failures (#2908) Co-authored-by: Adrian Tsai <adtsai@microsoft.com> Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Paul McDaniel <paul_mcdaniel@hotmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Dwayne Robinson <fdwr@hotmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com>
2020-02-05 01:12:19 +00:00
{
Sync ORTModule branch with master and fix tests (#6526) * Deprecate Python global configuration functions [Part 1] (#5923) Enable options to be set via execution provider (EP)-specific options and log deprecation warning from current global configuration functions. * remove dnnl_dll_path from post build copy (#6142) * Model Fusion For Bart (#6105) Fusion fix for Bart models * Unify IExecutionProvider and IExecutionProviderFactory interfaces (#6108) * Remove Provider_IExecutionProvider and make the internal IExecutionProvider usable by shared providers * Change Provider_IExecutionProviderFactory to be the core version. * Enable running the mnist_training sample without cuda (#6085) Signed-off-by: George Nash <george.nash@intel.com> * nnapi add min max support (#6117) * Fix CUDA test hang: (#6138) - Make condition check in `CUDAAllocatorTest` to ensure CUDA device is present. * Fix TensorRT kernel conflict issue for subgraphs of control flow operators (#6115) * add static subgraph kernel index * change kernel naming to avoid conflicts * Add gradient registration for Abs. (#6139) * Partition initial optimizer state for Zero-1 (#6093) * Initial changes * Working changes * Working changes * Cleanup * fix windows CI * Review comments * review comments * Fix edge case in BFCArena where allocation failures could lead to an infinite loop. (#6145) #4656 * Revert "work around of the build break in mac (#6069)" (#6150) This reverts commit 3cae28699bed5de1fcaadb219fa69bae0fc3cee8. * Fix clean_docker_image_cache.py detection of image pushes. (#6151) Fix clean_docker_image_cache.py detection of image pushes. They were being ignored because the expected HTTP status code was wrong. For pushes, it's 201 instead of 200. * MLAS: add NEON version of int8 depthwise convolution (#6152) * Using a map of of ops to stages as input of partition function. (#5940) * New partition algorithm running before AD * Convert cut_group_info into device map. Work in progress -- works for bert-tiny with pp=2 * Removing code for partition of bwd graphs * Remove old code * Adding some verification code * Handle Shared Initializer * Renaming rank with stage * Added first unit test * new test * redundant check * undo change in bert * Moved cut-based partition to testing utils file Co-authored-by: xzhu1900 Co-authored-by: wschin * New conversion function and tests * minor * remove test that is not needed2 * improve GetDeviceAssignment and PR comments * minor changes * PR comments * improving documentation and variable naming * add documentation * Variable naming and docs * more doc improvements * more doc improvements * missing static cast * Fix test file for windows * Fix test file for windows * Fix test file for windows * stage id is not the same as rank id * PR comments * PR comments * More comments * More comments * Minor fix to satisfy c++14 (#6162) * Deprecating Horovod and refactored Adasum computations (#5468) deprecated horovod submodule refactored adasum logic to be ort-native added tests for native kernel and e2e tests * Update TensorRT-ExecutionProvider.md (#6161) * Bugfix for topk cuda kernel (#6164) * fix the issue that std::numeric_limits cannot handle half type * adding a test Co-authored-by: Du Li <duli@OrtTrainingDev4.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net> * Revert "Fuse MatMulIntegerToFloat only when scales are scalar (#6008)" (#6169) This reverts commit f2dcba7afe0d42ebdaaef0c6cdf913a1156c9e98. * Remove ignored build warnings for pybind on Mac (#6165) * save_checkpoint, load_checkpoint and aggregate_checkpoints (#6136) * save_checkpoint and load_checkpoint implementations * checkpoint aggregation logic * unit tests for save_checkpoint, load_checkpoint and aggregate_checkpoints * Don't try to bind unused inputs in the Training frontend (#6166) * Update documentation for contributing a PR and add deprecation notices for PyOp and ORT server. (#6172) * aggregate model states only for the case when mixed precision was true (#6176) * [NNAPI EP] Enable per-channel quantization for QlinearConv (#6155) * Enable qlinearconv per-channel quantization * Fix the android CI test failure * Add Android Version Check for Per-Channel Quant * Address PR comments * Fix some minor issues * Add verification of per-channel zero points * Make the error tolerance configurable * Fix typo in BERT pretraining script (#6175) A misplaced `}` meant that the `'enable_adasum'` option was interpreted incorrectly, causing the test to fail. * Update get_docker_image.py to enable use without image cache container registry. (#6177) Update get_docker_image.py to enable use without image cache container registry. * Helper for compiling EP to generate deterministic unique ids for use in MetaDef names (#6156) * Create a helper for generating unique ids that can be used by an EP that creates compiled nodes and needs ids to be deterministic for a model when used in multiple sessions. Added to IExecutionProvider as this can potentially be used by all compiling EPs and is more robust than a simplistic counter (although EP implementer is free to choose either approach). * Restructure the helper so it can be called across the EP bridge. Add ability to call id generation helper from EP bridge - convert DNNL EP to use helper to validate Address issue where a new Model may be loaded into the same address as a previous one. - hash the bytes in the Graph instance (1728 bytes currently) to use as the key to the full hash for the model Add lock around id generation to ensure no issues if multiple sessions partitions graphs at exactly the same time. - Extremely unlikely but would be hard to debug and the locking cost is not an issue as it's only incurred during graph partitioning and not execution. * Backend APIs for checkpointing (#5803) * Add backend API GetOptimizerState and GetModelState * add GetPartitionInfoMap * Android coverage dashboard (#6163) * Write the report to a file. * Post code coverage to the Dashboard database. * Add usage details of unified MCR container image (#6182) Going forward, a single unifed docker image will be published in MCR. The hardware accelerator target choice will have to be made in the application using OpenVINO EP's runtime config options. * improve perf for softmax (#6128) * improve perf for both gathergrad and softmax * revert the change in gathergrad and will be done in another PR. * address comments from code review. * Tune fast Gelu to use exp(x) instead of tanh(x) on Rocm platform (#6174) * tune fast gelu to use exp(x) instead of tanh(x) on rocm * update to use expression 2/(1+exp(-2x))-1 for stability * Add Status.csv to EP Perf Tool (#6167) * merge master, keep postprocess status commit * download float16.py everytime * removing hardcoded values * Lochi/quantization tool for trt (#6103) * Initial implementation of generating calibration dynamic range table * Initialize validation support for Quantization * Initialize validation support for Quantization (cont.) * Improve validation support for Quantization * Improve validation support for Quantization * Rewrite/Refine for calibration and validation * Rewrite/Refine for calibration and validation (cont.) * Refine code * Refine code * Add data reader for BERT * Add flatbuffers to serialize calibration table * Refine code and add BERT evaluation * Refine the code * minor modification * Add preprocess/postprocess of vision team yolov3 and refine the code * Update annotation * Make bbox cooridates more accurate * Fix bug * Add support of batch processing * Batch processing for model zoo yolov3 * Add batch inference for evaluation * Refine the code * Add README * Add comments * Refine the code for PR * Remove batch support checking in data_reader and refine the code * Refine the code for PR * Refine the code for PR review Co-authored-by: Olivia Jain <oljain@microsoft.com> * Implement ScatterND for CUDA EP (#6184) * Condition fix in Resize operator (#6193) * Clean up checkpoint tests to use the new checkpoint functions (#6188) * add deprecation warning for old checkpoint functions * update all the distributed checkpoint tests to use new checkpoint functions * Implement comparing outputs that are sequence of maps of strings to floats (#6180) * Implement conversion from ortvalue to Itensor for string tensors and comparing sequence of maps of strings to floats * PR comments * Dockerfile to build onnxruntime with ROCm 4.0 * Add ability to skip GPU tests based on GPU adapter name (#6198) * Implement conversion from ortvalue to Itensor for string tensors and comparing sequence of maps of strings to floats * PR comments * Add ability to skip gpu tests according to adapter description * spacing * spacing * spacing * Openvino ep 2021.2 (#6196) * Enabling fasterrcnn variant and vehicle detector * changes for 2021_2 branch * yolov3_pytorch commit * fixed braces in basic_backend.cc * ci information added * faster rcnn variant and vehicle detector changes were made in 2021.1 and not in 2021.2 * some changes to support unit tests * disable some tests which are failing * fix myriad tests for vehicle detector * Did some cleanup *cleaned up comments *Disabled Add_Broadcast_0x1 and Add_Broadcast_1x0 tests on MYRIAD_FP16 backend due to a bug *cleaned up capability_2021_2.cc file *Removed extra conditions which were added for some validation in backend_utils Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com> * yolov3 pytorch workaround to ensure that the output names are matched * gemmoptest fixed on myriad * Fixed MYRIADX CPP Test Failures *Expand,GatherND,Range,Round op's are only supported in model *where op with float input data types are not supported and fixed *Scatter and ScatterElements op's with negative axis are fixed *Reshape op with 0 dim value are not supported and fixed *Disabled InstanceNorm_2 test on MYRIADX Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com> * make changes to yolov3 pytorch * Fixed python unit tests *Fixed failing python tests on vpu, GPU and CPU Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com> * Fixes POW op failures on GPU_FP16 Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com> * Clean up capability_2021_2.cc Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com> * Updated docx for MultiThreading option *Added extra info on setting the num_of_threads option using the API and it's actual usage Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com> * fixed slice and removed extra prints * Disabled failing python tests Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com> * Minor changes added in capabilty_2021_2 Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com> * made changes to slice to avoid failures * Disabling FP16 support for GPU_FP32 ->Inferencing an FP16 model on GPU_FP32 leads to accuracy mismatches. so, we would rather use GPU_FP16 to infer an FP16 model on GPU Device Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com> * Updated docx for Inferencing a FP16 Model Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com> * fix for mask rcnn * Script for installing openvino from source * Updated with openvino 2021.2 online installation * code comment fixes fixed accuracy mismatch for div * Update OpenvinoEP-ExecutionProvider.md updated for 2021.2 branch * Update README.md updated dockerfile documentation * Update BUILD.md build.md update documentation * permissiong change of install_openvino.sh * made changes to align with microsoft onnxruntime changes * Updated with ov 2021.2.200 Co-authored-by: suryasidd <surya.siddharth.pemmaraju@intel.com> Co-authored-by: sfatimar <sahar.fatima@intel/com> Co-authored-by: MaajidKhan <n.maajidkhan@gmail.com> Co-authored-by: mohdansx <mohdx.ansari@intel.com> * Fix a memory leak in test_inference.cc (#6201) * Fix a memory leak in test_inference.cc * Use TArray in AMD element-wise kernels, rather than manually copying memory to device. * Remove most ROCm-specific element-wise code and reuse CUDA element-wise code. * Minor change to improve performance for operator Pad. (#5537) * small improvment for pad * Support double for operators Log, Reciprocal, Sum (CPU) (#6032) * Support double for operators Log, Reciprocal, Sum * remove tesdt erf_double * Support double for operators Where, LpNormalisation (#6034) * Support double for operators Relu, Tanh, Sigmoid (#6221) * Fix ImportError in build.py (#6231) There is a possible ImportError where build.py can import the wrong 'util' package if there are others present in `sys.path` already * Removed executor todo that looks dead. (#6234) * Remove MKLML/openblas/jemalloc build config (#6212) * Remove python 3.5 * Update the readme file * Upgrade build.py to assert for python 3.6+ Upgrade build.py to assert for python 3.6+ as python 3.5 cannot build anymore todays master. * Support MLFloat16 type in Pow opset-12 CUDA kernel (#6233) * MLAS: handle MlasGemm(M/N/K==0) cases (#6238) * Support double for operator TopK + fix one bug in TopK implementation for GPU for double (#6220) * Support double for operator TopK * add static classes for topk/double * fix cast issue in topk * Support double for operator Gemm + fix bug in gemm implementation for cuda, rocm when sizeof(type) != sizeof(float) (#6223) * Support double for operator Gemm * fix type size while copying data in gemm operator for GPU * fix type in gemm implementation for rocm * Support double for operator ReduceMean, ReduceLogSumExp (#6217) * Support double for operators ReduceMean, ReduceLogSumExp * Support double for operator ArgMin (#6222) * Support double for operator ArgMin * add test specifically for double * add new test on pai-excluded-tests.txt * Update BUILD.md * Update manylinux docker image to the latest (#6242) * Fix allocator issue for TensorRT IOBinding (#6240) * Fix issue: https://github.com/microsoft/onnxruntime/issues/6094 Root cause: we didn't expose the OrtMemoryInfo for TRT, so it will cause issue if user want use IObinding for Tensorrt. Short term fix, add the OrtMemoryInfo for TRT. Long term should unify the allocator for CUDA and TRT * Tune BiasGeluGradDx kernel in approximation mode to avoid tanh(...) on Rocm (#6239) * bias gelu grad use exp(...) instead * update cuda to rocm * missing semicolon * comment * remove dockerfile * missing factor of two * Refactor EP Perf Tool (#6202) * merge master, keep postprocess status commit * download float16.py everytime * using variables to reference eps * adding ACL EP to ep perf tool * accuracy with absolute tolerance configurable * add acl to dict + remove commented line * Documentation for distributed CI tests pipeline (#6140) * Remove a debug log in provider_test_utils.cc (#6200) * Add the Concat Slice Elimination transform, fix constant_folding transform (#5457) * Add concat slice transform + test * Cosmetic improvements in concat slice transform * Remove unrelated file, fix comment, fix constant folding bug * Add test onnx graph * fix windows build * Review comments * review comment Co-authored-by: Aishwarya <aibhanda@OrtTrainingDev4.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net> * Add MakeStringLite which uses current locale, update some MakeString call sites to use it instead. (#6252) * Add MakeStringLite which uses current locale, update macros to use that to generate messages. * Convert calls to MakeStringLite(). * Liqun/speech model loop to scan (#6070) Provide a tool to convert Loop to Scan for Nuphar performance Fix Nuphar CI pipeline failures. Co-authored-by: liqun <liqun@OrtTrainingDev4.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net> * model parallel refinement (#6244) * Megatron Transformation as a seperate step * remove useless header * clang formating * Re-Structure megatron transformer for subsquent changes * fix comments * Allow querying a GraphProto's doc_string as part of ModelMetadata (#6248) * Fix Linux/Mac error message on input type mismatch (#6256) * add bfloat16 to gathergrad type constrains (#6267) Co-authored-by: Cheng Tang <chenta@microsoft.com> * Fix VS 2017 build break (#6276) * Deprecate Python global configuration functions [Part 2] (#6171) Update Python API to allow more flexibility for setting providers and provider options. The providers argument (InferenceSession/TrainingSession constructors, InferenceSession.set_providers()) now also accepts a tuple of (name, options dict). Fix get_available_providers() API (and the corresponding function in the C API) to return the providers in default priority order. Now it can be used as a starting point for the providers argument and maintain the default priority order. Convert some usages of the deprecated global configuration functions to use EP-specific options instead. Update some EP-specific option parsing to fail on unknown options. Other clean up. * Add script to preprocess python documentation before publishing (#6129) * add script to preprocessing python documentation before publishing * rename past to past_key_values for GPT-2 (#6269) rename past to past_key_values for transformers 4.* * Rename MakeString and ParseString functions. (#6272) Rename MakeString to MakeStringWithClassicLocale, MakeStringLite to MakeString, *ParseString to *ParseStringWithClassicLocale. Add missing pass-through versions of MakeStringWithClassicLocale for string types. * Increase timeout for Linux GPU CUDA11 build. (#6280) * Add helper to compare model with different precision (#6270) * add parity_check_helper.py * add real example * remove lines * Fix Min/Max CPU kernels for float16 type (#6205) * fix data_ptr assertion error for past_sequence_length=0 in GPT-2 (#6284) fix io binding crash for past_sequence_length=0 * A list of changes in transformers tool (#6224) * longformer fp16 e2e * add fp16/fp32 parity check helper file * excludes nodes with subgraph in profiling * use onnxconverter_common to do fp32->fp16 * add version check for onnxconverter_common * remove helper file * add pkg installation on notebooks and script * Workaround for static_cast<double>(half) * Add workaround to remove ROCm-specific binary-elementwise files. * Update nuget build (#6297) 1. Update the ProtoSrc path. The old one is not used anymore. 2. Regenerate OnnxMl.cs 3. Delete some unused code in tools/ci_build/build.py 4. Avoid set intra_op_param.thread_pool_size in ModelTests in OpenMP build. 5. Fix a typo in the C API pipeline. * Enable ONNX backend test of SequenceProto input/output (#6043) * assert sequence tensor and remove skips * update testdata json * use ONNX 1.8 in cgmanifest.json * use previous commit to workaround * update ONNX commit ID in docker * skip test_maxpool_2d_dilations test for now * update function name * add --sequence_lengths option (#6285) * more dtype for Equal CUDA kernel (#6288) Co-authored-by: Vincent Wang <weicwang@microsoft.com> * Force reinstall onnx python package on Windows (#6309) * update transformers required package versions (#6315) * Remove abs in LpPool (#6303) * Support 1D input for Conv + Mul/Add fusion optimizer with test (#6295) * Support 1D input (N C H) for Conv + Mul/Add fusion optimizer with test cases and test models. * Add longformer to python package (#6314) * add longformer to python package * move test related script and data to a new folder * Avoid false sharing on thread pool data structures (#6298) Description: This change adds alignment and padding to avoid false sharing on fields in the thread pool. It also adds a new microbenchmark to profile thread-pool performance over short loops. Motivation and Context MobileNet on a 2*12-core system showed a performance gap between the ORT thread pool and OpenMP. One cause appeared to be false sharing on fields in the thread pool: ThreadPoolParallelSection::tasks_finished (which the main thread spins on waiting for workers to complete a loop), and the RunQueue::front_ and back_ fields (used respectively by the worker thread and the main thread). The additional micro-benchmark BM_ThreadPoolSimpleParallelFor tests performance of loops of different sizes at different thread counts. The results below are on a machine with 2*14-core processors (E5-2690 v4) running with 1, 14, 15, and 28 threads. For each test, the microbenchmark has N threads run a loop with N iterations; hence a perfect result is for the time taken to be constant as additional threads are added (although we will also see power management effects helping at very low thread counts). The loop durations (100000, 10000, 1000) correspond roughly to 200us, 20us, and 2us on this machine. Before change: BM_ThreadPoolSimpleParallelFor/1/1/100000/real_time 17153 us 17154 us 32 BM_ThreadPoolSimpleParallelFor/14/14/100000/real_time 22553 us 22553 us 30 BM_ThreadPoolSimpleParallelFor/15/15/100000/real_time 21521 us 21521 us 29 BM_ThreadPoolSimpleParallelFor/28/28/100000/real_time 24111 us 24111 us 24 BM_ThreadPoolSimpleParallelFor/1/1/10000/real_time 1719 us 1719 us 407 BM_ThreadPoolSimpleParallelFor/14/14/10000/real_time 3409 us 3409 us 200 BM_ThreadPoolSimpleParallelFor/15/15/10000/real_time 3541 us 3541 us 201 BM_ThreadPoolSimpleParallelFor/28/28/10000/real_time 4576 us 4576 us 151 BM_ThreadPoolSimpleParallelFor/1/1/1000/real_time 174 us 174 us 4017 BM_ThreadPoolSimpleParallelFor/14/14/1000/real_time 1586 us 1586 us 402 BM_ThreadPoolSimpleParallelFor/15/15/1000/real_time 1586 us 1586 us 397 BM_ThreadPoolSimpleParallelFor/28/28/1000/real_time 2864 us 2864 us 232 After change: BM_ThreadPoolSimpleParallelFor/1/1/100000/real_time 17160 us 17160 us 33 BM_ThreadPoolSimpleParallelFor/14/14/100000/real_time 20989 us 20989 us 31 BM_ThreadPoolSimpleParallelFor/15/15/100000/real_time 22286 us 22286 us 31 BM_ThreadPoolSimpleParallelFor/28/28/100000/real_time 24631 us 24631 us 25 BM_ThreadPoolSimpleParallelFor/1/1/10000/real_time 1718 us 1718 us 407 BM_ThreadPoolSimpleParallelFor/14/14/10000/real_time 2868 us 2868 us 242 BM_ThreadPoolSimpleParallelFor/15/15/10000/real_time 2907 us 2907 us 240 BM_ThreadPoolSimpleParallelFor/28/28/10000/real_time 3872 us 3872 us 186 BM_ThreadPoolSimpleParallelFor/1/1/1000/real_time 175 us 175 us 3938 BM_ThreadPoolSimpleParallelFor/14/14/1000/real_time 933 us 933 us 659 BM_ThreadPoolSimpleParallelFor/15/15/1000/real_time 912 us 912 us 591 BM_ThreadPoolSimpleParallelFor/28/28/1000/real_time 1976 us 1976 us 317 * fix opset imports for function body (#6287) * fix function opsets * add tests and update onnx * changes per review comments * add comments * plus updates * build fix * Remove false positive prefast warning from threadpool (#6324) * Java: add Semmle to Java publishing pipelines (#6326) Add Semmle to Java API pipeline Add security results publishing and add Java GPU. * Quantization support for split operator with its NHWC support (#6107) * Make split working for quantization. * NHWC transformer support for split operator * Refactor some according to Feedback. Will add test cases soon. * Fix build error on windows. * Add test case for split op on uint8_t support * Add nhwc_transformer_test for split uint8_t support * Some change according to PR feedbacks. * Liqun/enable pipeline parallel test (#6331) enable pipeline parallel test Co-authored-by: liqun <liqun@OrtTrainingDev4.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net> * Use onnxruntime_USE_FULL_PROTOBUF=OFF for the cuda execution provider (#6340) This removes a special case of the cuda EP. * MLAS: add fallback implementation for quantized GEMM (#6335) Add a non-vectorized version of the kernel used for the quantized version of MlasGemm. * Delete float16.py (#6336) No longer needed. Also doesn't pass policheck. * Enable add + softmax fusion for Rocm platform (#6259) * add bias softmax; tests appear to pass * check fusion occurs for rocm as well * check for rocm provider compatible as well * build for cpu scenario as well * try again; broader cope * proper scope on kGpuExecutionProvider * been editing wrong file * remove commented #include lines * try again due to mac os ci error * try again * test fusion both cuda and rocm to avoid mac ci error * add external data support to tensor proto utils (#6257) * update unpack tensor utilities to support loading external data * more updates * fix test * fix nuphar build * minor build fix * add tests * fix Android CI * fix warning * fix DML build failure and some warnings * more updates * more updates * plus few updates * plus some refactoring * changes per review * plus some change * remove temp code * plus updates to safeint usage * build fix * fix for safeint * changed wording. (#6337) * Remove OpSchema dummy definition. Only needed for Function now, and we can just exclude the method in Function (#6321) * remove gemmlowp submodule (#6341) * [NNAPI] Add pow support (#6310) * Add support for running Android emulator from build.py on Windows. (#6317) * fix the pipeline failure (#6346) * Train BERT Using BFloat16 on A100 (#6090) * traing bert using bf16 * Adam support bf16 * bugfix * add fusedmatmul support * fix after merge from master. * bugfix * bugfix after merge from master * fast reduction for bf16. * resolve comments * fix win build * bugfix * change header file. Co-authored-by: Vincent Wang <weicwang@microsoft.com> * Fix DerefNullPtr issues raised by SDLNativeRules. (#6348) * update quantize to support basic optimization and e2e example for image classification (#6313) update the resnet50-v1 to standard one from onnx zoo. add an example for mobilenet run basic optimization before quantization fix a bug in Clip * Enable graph save for orttrainer (#6333) * Enable graph save for orttrainer * Fix CI * Update orttraining/orttraining/python/training/orttrainer_options.py * Update orttraining/orttraining/python/training/orttrainer_options.py * Update orttraining/orttraining/python/training/orttrainer_options.py * Update orttraining/orttraining/python/training/orttrainer_options.py * Update orttraining/orttraining/python/training/orttrainer_options.py Co-authored-by: Thiago Crepaldi <thiago.crepaldi@microsoft.com> * Add PREfast to python packaging pipeline (#6343) * Add PREfast to python packaging pipeline * fix longformer benchmark io_binding output_buffers (#6345) * fix longformer benchmark io_binding output_buffers * format * import benchmark_helper from parent directory. * Use readelf for minimal build binary size checks. (#6338) * Use readelf for minimal build binary size checks. The on-disk size grows in 4KB chunks which makes it hard to see how much growth an individual checkin causes. Only downside is that the sum of the sections is larger than the on-disk size (assumably things get packed smaller on disk and some of the section alignment constraints can be ignored) * Remove unused function * Java: Set C language warnings to W4 and adjust JNI code (#6347) Set /W3 for C language and fix up JNI warnings. * Pipeline Parallel Experimental Python API (#5815) * Add create session to WinML telemetry to track WinML Usage (#6356) * Fix one more SDL warning (#6359) * fix -Wdangling-gsl (#6357) * Add python example of TensorRT INT8 inference on ResNet model (#6255) * add trt int8 example on resnet model * Update e2e_tensorrt_resnet_example.py * remove keras dependency and update class names * move ImageNetDataReader and ImageClassificationEvaluator to tensorrt resnet example * simplify e2e_tensorrt_resnet_example.py * Update preprocessing.py * merge tensorrt_calibrate * Update calibrate.py * Update calibrate.py * generalize calibrate * Update calibrate.py * fix issues * fix formating * remove augment_all * This added telemetry isn't needed (#6363) * Wezuo/memory analysis (#5658) * merged alloc_plan * pass compilation * Start running, incorrect allocation memory info * add in comments * fix a bug of recording pattern too early. * debugging lifetime * fix lifetime * passed mnist * in process of visualization * Add code to generate chrome trace for allocations. * in process of collecting fragmentation * before rebuild * passed mnist * passed bert tiny * fix the inplace reuse * fix the exception of weight in pinned memory * add guards to ensure the tensor is in AllocPlan * add customized profiling * debugging * debugging * fix the reuse of differnt location type * add rank * add the rank * add fragmentation * add time_step_trace * Add summary for each execution step (total bytes, used/free bytes). * add top k * change type of top k parameter * remove prints * change heap to set{ * add the name pattern * add the useage for pattern * add partition * change to static class * add custom group * remove const * update memory_info * in process of adding it as runtime config * change the memory profiling to be an argument * add some comments * add checks to recored meomry_info in traaining session * set the "local rank setting" to correct argument. * addressing comments * format adjustment * formatting * remove alloc_interval * update memory_info.cc to skip session when there is no tensor for a particular memory type * fix memory_info multiple iteration seg-fault * consolidate mainz changes * fixed some minor errors * guard by ORT_MINIMAL_BUILD * add ORT_MEMORY_PROFILE flag * added compiler flag to turn on/off memory profiling related code * clean up the code regarding comments * add comments * revoke the onnx version * clean up the code to match master * clean up the code to match master * clean up the code to match master Co-authored-by: Jesse Benson <benson.jesse@gmail.com> Co-authored-by: Wei Zuo <wezuo@OrtTrainingDev3.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net> Co-authored-by: wezuo <wezuo@az-eus-v100-32gb-5-worker-mgtbby.eastus.cloudapp.azure.com> Co-authored-by: wezuo <wezuo@az-eus-v100-32gb-5-worker-yclzsf.eastus.cloudapp.azure.com> * Support MLFloat16 in CumSum Cuda op for Opset 14 (#6355) * Add CumSum-14 for Cuda * fix convert_common version retrival (#6382) * Refine auto_pad based pad computation in ConvTranspose (#6305) * Fix SDL warning (#6390) * Add max_norm for gradient clipping. (#6289) * add max_norm as user option for gradient clipping * add adam and lamb test cases for clip norm * add frontend tests * Add the custom op project information (#6334) * Dont use default string marshalling in C# (#6219) * Fix Windows x86 compiler warnings in the optimizers project (#6377) * [Perf] Optimize Tile CPU and CUDA kernels for a corner case (#6376) * Unblock Android CI code coverage failure (#6393) * fix build on cuda11 (#6394) Co-authored-by: Vincent Wang <weicwang@microsoft.com> * Load the model path correctly (#6369) * Fix some compile warnings (#6316) * OpenVino docker file changes to bypass privileged mode Description: Builds and installs libusb without UDEV support, which is used for communicating with the VPU device. Motivation and Context This enables the resulting docker container to be run without '--privileged' and '--network host' options which may not be suitable in deployment environments. * Megatron checkpointing (#6293) * Add bart fairseq run script * Add frontend change to enable megatron * Initial changes for checkpointing * Megatron optim state loading, checkpoint aggregation, frontend distributed tests for H, D+H * Add load_checkpoint changes * Fix CI * Cleanup * Fix CI * review comments * review comments * review comments: * Fix generate_submodule_cgmanifest.py Windows issues. (#6404) * Continue memory planning when unknown shape tensor is encountered. (#6413) * Reintroduce experimental api changes and fix remote build break (#6385) Co-authored-by: Ori Levari <orlevari@microsoft.com> * Add support for custom ops to minimal build. (#6228) * Add support for custom ops to minimal build. Cost is only ~8KB so including in base minimal build. * enable pipeline to run quantization tests (#6416) * enable pipeline to run quantization tests setup test pipeline for quantization * Minor cmake change (#6431) * Liqun/liqun/enable pipeline parallel test2 (#6399) * enable data and pipeline parallism test Co-authored-by: liqun <liqun@OrtTrainingDev4.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net> * Farewell TrainableDropout (#5793) * Deprecate TrainableDropout kernel. * Update bert_toy_postprocessed.onnx to opset 12. * Add more dropout tests. * Fix BiasDropout kernel. Co-authored-by: Ubuntu <OrtTrainingDev3@OrtTrainingDev3.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net> Co-authored-by: Sherlock Huang <bahuang@OrtTrainingDev3.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net> Co-authored-by: Sergii Dymchenko <sedymche@microsoft.com> * fix null dereference warning (#6437) * Expose graph ModelPath to TensorRT shared library (#6353) * Update graph_viewer.cc * Update tensorrt_execution_provider.cc * Update graph_viewer.h * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update provider_api.h * Update provider_bridge_ort.cc * Update provider_interfaces.h * Update provider_interfaces.h * expose GraphViewer ModelPath API to TRT shared lib * add modelpath to compile * update * add model_path to onnx tensorrt parser * use GenerateMetaDefId to generate unique TRT kernel name * use GenerateMetaDefId to generate unique TRT engine name * fix issue * Update tensorrt_execution_provider.cc * remove GetVecHash * Update tensorrt_execution_provider.h * convert wchar_t to char for tensorrt parser * update tensorrt parser to include latest changes * fix issues * Update tensorrt_execution_provider.cc * merge trt parser latest change * add PROVIDER_DISALLOW_ALL(Path) * add tool for generating test data for longformer (#6415) * only build experimental api in redist (#6465) Co-authored-by: Sheil Kumar <sheilk@microsoft.com> * Add an option to save the training graph after optimization (#6410) * expose optimized_model_filepath in SessionOptions as `debug.graph_save_paths.model_with_training_graph_after_optimization_path` in `ORTTrainerOptions` * Share allocator between CUDA EP & TRT EP. (#6332) * Share allocator between CUDA EP & TRT EP. limitation: 1. Does not cover the per-thread allocator created by CUDA EP, still need to figure out the way to remove it 2. Need to have more identifiers to make it able to share CPU allocator across all EPs * fix max norm clipping test in python packaging pipeline test (#6468) * fix python packaging pipeline * make clip norm test compatabile with both V100 and M60 GPUs * Initial version of CoreML EP (#6392) * Bug 31463811: Servicing: Redist (Nuget) conflicts with Microsoft.AI.MachineLearning starting 21H1+ (#6460) * update load library code to have the fullly qualified path * make it work for syswow32 * git Revert "make it work for syswow32" This reverts commit b9f594341b7cf07241b18d0c376af905edcabae3. Co-authored-by: Sheil Kumar <sheilk@microsoft.com> * dequantize 1st input of lstm back if it is quantized (#6444) * [java] Adds support for OrtEnvironment thread pools (#6406) * Updates for Gradle 7. * Adding support for OrtThreadingOptions into the Java API. * Fixing a typo in the JNI code. * Adding a test for the environment's thread pool. * Fix cuda test, add comment to failure. * Updating build.gradle * fix SDL native rule warning #6246 (#6461) * fix SDL rule (#6464) * use tickcount64 (#6447) Co-authored-by: Ori Levari <orlevari@microsoft.com> * Update pypi package metadata (#6354) * Update setup file data * add missing comma * remove python 3.5 * fix typo bracket * Delete nuget extra configs (#6477) * Op kernel type reduction infrastructure. (#6466) Add infrastructure to support type reduction in Op kernel implementations. Update Cast and IsInf CPU kernels to use it. * Fixing a leak in OnnxSequences with String keys or values. (#6473) * Increase the distributes tests pipeline timeout to 120 minutes (#6479) * [CoreML EP] Add CI for CoreML EP (macOS) and add coreml_flags for EP options (#6481) * Add macos coreml CI and coreml_flags * Move save debuggubg model to use environment var * Move pipeline off from macos CI template * Fix an issue building using unix make, add parallel to build script * Fixed build break for shared_lib and cmpile warning * Fix a compile warning * test * Revert the accidental push from another branch This reverts commit 472029ba25d50f9508474c9eeceb3454cead7877. * Add ability to track per operator types in reduced build config. (#6428) * Add ability to generate configuration that includes required types for individual operators, to allow build size reduction based on that. - Add python bindings for ORT format models - Add script to update bindings and help info - Add parsing of ORT format models - Add ability to enable type reduction to config generation - Update build.py to only allow operator/type reduction via config - simpler to require config to be generated first - can't mix a type aware (ORT format model only) and non-type aware config as that may result in insufficient types being enabled - Add script to create reduced build config - Update CIs * merge e2e with distributed pipeline (#6443) merge e2e with distributed pipeline * Fix test breaks in Windows ingestion pipeline (#6476) * fix various build breaks with Windows build * fix runtime errors loading libraries from system32 * add build_inbox check to winml_test_common * use raw string * cleanup * fix dll load Co-authored-by: Sheil Kumar <sheilk@microsoft.com> * Speed up the Mac CI runs (#6483) * expose learningmodelpixelrange property (#5877) * Fix of support api version bug for [de]quantize (#6492) * SDL fixes: add proper casts/format specifiers (#6446) * SDL annotation fixes (#6448) Co-authored-by: Ori Levari <orlevari@microsoft.com> * [OpenVINO-EP] Remove support for OpenVINO 2020.2 (#6493) * Removed OpenVINO 2020.2 support * Updated documentation and build.py * Removed unnecessary libraries from setup.py * Support pad operator in quantization and quantized nhwc transformer. Fix Pad operator bug. (#6325) Support pad operator in quantization tool. Support pad operator in quantized nhwc transformer. Fix pad() operator bug when pad input's inner(right) most axis value is zero for Edge and Reflect mode, it copied wrong value to the cells to be padded. Note the Constant mode will not trigger this bug, as Edge/Reflect need copy value from the already copied array while Constant mode only fill specified value. Add more test cases to cover pad() operator bug fixed here. Fix quantization tools uint8/int8 value overflow issue when quantize weights in python. * Improve work distribution for Expand operator, and sharded LoopCounter configuration (#6454) Description: This PR makes two changes identified while looking at a PGAN model. First, it uses ThreadPool::TryParallelFor for the main parallel loops in the Expand operator. This lets the thread pool decide on the granularity at which to distribute work (unlike TrySimpleParallelFor). Profiling showed high costs when running "simple" loops with 4M iterations each of which copied only 4 bytes. Second, it updates the sharded loop counter in the thread pool so that the number of shards is capped by the number of threads. This helps make the performance of any other high-contention "simple" loops more robust at low thread counts by letting each thread work on its own "home" shard for longer. Motivation and Context Profiling showed a PGAN model taking 2x+ longer with the non-OpenMP build. The root cause was that the OpenMP build uses simple static scheduling of loop iterations, while the non-OpenMP build uses dynamic scheduling. The combination of large numbers of tiny iterations is less significant with static scheduling --- although still desirable to avoid, given that each iteration incurs a std::function invocation. * Update document of transformer optimization (#6487) * nuphar test to avoid test data download to improve passing rate (#6467) nuphar test to avoid test data download to improve passing rate * Fuse cuda conv with activation (#6351) * optimize cuda conv by fused activation * remove needless print out * exclude test from cpu * handle status error from cudnn 8.x * add reference to base class * add hipify * [CoreML EP] Add support for some activations/Transpose, move some shared helpers from NNAPI to shared space (#6498) * Init change * Move some helper from nnapi ep to shared * Add transpose support * Fix trt ci build break * Refine transformers profiler output (#6502) * output nodes in the original order; grouped by node name * add document for profiler * Update to match new test setup. (#6496) * Update to match new test setup. * Add Gemm(7) manually for now. Will fix properly on Monday. It's used by mnist.ort as that is created by optimizing mnist.onnx to level 1 causing 2 nodes to be replaced by a Gemm and the op to be missing from the required list as that is created using the original onnx model. * Enable dense sequence optimized version of Pytorch exported BERT-L on AMD GPU (#6504) * Permit dense seq optimization on BERT-L pytorch export by enabling ReduceSumTraining, Equal, and NonZero on AMD * enable Equal tests * enable fast_matrix_reduction test case * Optimize GatherGrad for AMD GPU (#6381) * optimize gathergrad * address comments Co-authored-by: Weixing Zhang <wezhan@microsoft.com> * add explicit barriers for buffer overread and overrwrite (#6484) Co-authored-by: Ori Levari <orlevari@microsoft.com> * fix sdl bugs for uninitialized variables and returns (#6450) Co-authored-by: Ori Levari <orlevari@microsoft.com> * handle hr error conditions (#6449) Co-authored-by: Ori Levari <orlevari@microsoft.com> * Dnnl training (#6045) * Add ReluGrad and ConvGrad ops for the dnnl provider * the mnist sample is updated to add the --use_dnnl option that will cause the sample to use the dnnl execution provider for nodes that exist in dnnl provider. * Added the ability to find forward ops. Dnnl backward gradient ops require the forward primitive description and workspace from the forward operation. * Enable specifying the execution provider for Gradient Checker Tests * Prevent memory leak when running dnnl_provider in training mode Prevent creating a SubgraphPrimitivePool when the code is built with the ENABLE_TRAINING build flag. Instead create a SubgraphPrimitive directly. The SubgraphPrimitivePool was causing a pool of SubgraphPrimitives to be stashed in a map for reuse. Due to the way the Training Loop uses threads the pool of SubgraphPrimitives were not being reuse instead a new pool of SubgraphPrimitives being created each run. The old pool was not instantly freed. This behavior could be a language error when using thread_local memory. Signed-off-by: George Nash <george.nash@intel.com> * Added fixes to maxpoolgrad and memory leak. Maxpoolgrad will now pass all unit tests. With the conv and convgrad disabled for dnnl, mnist is able to train till 95% Signed-off-by: Chethan Palangotu Keshava <chethan.palangotu.keshava@intel.com> * Fixed misc issues when testing training code with dnnl provider * fix conv_grad dnnl tests with dilation to run dnnl execution provider * update mnist training sample to accept convolution type models convolution models require the input shape to be {1, 28, 28} instead of the flat {728} image that is used for the gemm models this will enable models that require the different shape by adding `--model_type conv` to the command line when running the mnist sample. (while testing a workaround was used see #4762) * Disable weight caching in dnnl conv operator when using training When training we can not use cached weights because the weight will be updated each run. This re-enables dnnl Conv and ConvGrad Ops. The weight caching was the source of the error from Conv when training. * Fix issues found when building grad ops on Linux * The dnnl_convgrad code was over using the scope operator causing a compilation problem. * The dnnl_maxpoolgrad code had a logic error that is was comparing with the source description when it should have been comparing with the destination despription. * Update BUILD.md so it shows DNNL for training * Updated the table of contents. Since the same providers are listed twice. Once for Infrance and again for Training an HTML anchor was added to distinguish the second header from the first for the TOC. * Fix build failure when not using --enable-training build option * reorganize the gradient operators so they are grouped together * Fix issues found when running onnx_backend_test_series.py * Pooling code only supports 2 outputs when built with --enable-training * Address code review feedback * class member variables end in underscore_ * use dst instead of dist to match pattern use elsewhere in DNNL code. * Remove workaround that was introduced to handle problems running convolution based training models. See issue #4762 Signed-off-by: George Nash <george.nash@intel.com> * Isolate training code and code cleanup * Do not build if dnnl_gpu_runtime if enable_training is set training code does not support dnnl_gpu_runtime yet. * Isolated Training code inside ifdefs so that they wont affect project if built without training enabled * Inadvertant changes in whitespace were removed to make code review simpler * Undid some code reordering that was not needed * comments added to closing #endif statments to simplify reading complex ifdefs * Modified the GetPrimitiveDesc functions to return shared_ptr instead of raw pointer. This matches what was done in Pool code and is safer memory code. Signed-off-by: George Nash <george.nash@intel.com> * Address code review issues - whitespace changes caused by running clang-format on the code - Several spelling errors fixed - Removed/changed some ifdefs to improve readability - other misc. changes in responce to code review. Signed-off-by: George Nash <george.nash@intel.com> * Code changes to address code review - Simplify iteration code using `auto` keyword - remove C style cast that was not needed - remove instance variable that was not needed [relugrad.h] - added the execution providers to `ComputeGradientErrorInternal()` and `ComputeTheoreticalJacobianTranspose()` instead of using a pointer to an instance varaible [gradient_checker.h/.cc] Signed-off-by: George Nash <george.nash@intel.com> * Combined the default gradient ops test and dnnl gradient ops test for ConvGrad and MaxPoolGrad into one function with the help of a helper function. This will reduce repeated code. Signed-off-by: Palangotu Keshava, Chethan's avatarChethan Palangotu Keshava <chethan.palangotu.keshava@intel.com> * Replaced the stack used by convgrad to vector so that the vector(used as stack) can be easily cleared everytime the graph is created. This will prevent memory leak from convolution kernels being pushed constantly onto the stack. Signed-off-by: chethan.palangotu.keshava@intel.com * Code clean up and formating updates - Removed empty else statment - updated indentation of code that was causing double curly brackets to look unususal - Changed check for NumDimensions to Size in Relu and ReluGrad error checking code. - isolated training code Signed-off-by: George Nash <george.nash@intel.com> * Restore inadvertantly removed ConvGrad tests When combining the DNNL and CPU version of the ConvGrad tests two test were inadvertantly excluded. This adds back the Conv3d and Conv3d with strides test cases. Signed-off-by: George Nash <george.nash@intel.com> * Add validation to ConvGrad This validates the dimensions of the ConvGrad match the passed in Convolution forward primitive description. The current code for DNNL ConvGrad makes the assumption that the ConvGrad nodes will be visited in the reverse order from the corresponding Conv nodes The added validation will return an error if this assumption is not true. Signed-off-by: George Nash <george.nash@intel.com> * Do not create new execution providers in provider_test_utils This removes the code that generated new execution providers in the OpTester::Run function. This was added because the std::move was leaving the `entry` value empty so subsequent calls would cause a segfault. Problem is this potentially changed the execution_provider because it would create the default provider dropping any custom arguments. When the now removed code was originally added the std::move was causing crashes when the GradientChecker unit tests were run. However, it is no longer causing problems even with the code removed. Signed-off-by: George Nash <george.nash@intel.com> * Change the forward conv stack to a forward conv map This changes how the forward conv kernel is mapped to the bwd ConvGrad kernel the problematic stack is no longer used. The convolution stack made the assumption that the corresponding ConvGrad operator would be visited in reverse order of the forward Conv operators. This was always problematic and was unlikely to work for inception models. Important changes: - The weight_name is added to the ConvGrad dnnl_node making it possible to use the weight_name as a lookup key to find the Conv forward Kernel - the `std::vector fwd_conv_stack_` has been replaced with a `std::map fwd_conv_kernel_map_` - Although it is not needed lock_guards were added when writing to and reading from the fwd_conv_kernel_map_ as well as the fwd_kernel_map_. These should always be accessed by a single thread when preparing the dnnl subgraphs so the guard should not be needed but its added just in case. - Updated the comments ConvGrad.h code to no longer mention the stack. The error check is not removed. It will be good to verify there are no errors as we continue to test against more models. Signed-off-by: George Nash <george.nash@intel.com> Co-authored-by: Chethan Palangotu Keshava <chethan.palangotu.keshava@intel.com> Co-authored-by: unknown <63478620+jeyblu@users.noreply.github.com> * Lochi/refactor yolov3 quantization (#6290) * Refactor the code and move data reader, preprocessing, evaluation to E2E_example_mode * Refactor the code. Move data reader, preprocessing, evaluation to model specific example under E2E_example_mode * refactor code * Move yolov3 example to specific folder and add additional pre/post processing * Print a warning message for using newer c_api header on old binary (#6507) * Fix issues with ArmNN build setup (#6495) * ArmNN build fixes * Update BUILD.md to document that the ACL paths must be specified to build ArmNN * Fix CUDA build error. We don't setup the link libraries correctly/consistently so improve that. * Fix Windows CI builds by updating test scripts to work with numpy 1.20. (#6518) * Update onnxruntime_test_python.py to work with numpy 1.20. Some aliases are deprecated in favor of the built-in python types. See https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations np.array with bytes for entries and dtype of np.void no longer automatically pads. Change a test to adjust for that. * Fix another test script * Fix ORTModule branch for orttraining-* pipelines * Update pytorch nightly version dependency Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Cecilia Liu <ziyue.liu7@gmail.com> Co-authored-by: Ryan Hill <38674843+RyanUnderhill@users.noreply.github.com> Co-authored-by: George Nash <george.nash@intel.com> Co-authored-by: Guoyu Wang <62914304+gwang-msft@users.noreply.github.com> Co-authored-by: Yateng Hong <toothache9010@gmail.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Derek Murray <Derek.Murray@microsoft.com> Co-authored-by: ashbhandare <ash.bhandare@gmail.com> Co-authored-by: Scott McKay <skottmckay@gmail.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Juliana Franco <jufranc@microsoft.com> Co-authored-by: Pranav Sharma <prs@microsoft.com> Co-authored-by: Tixxx <tix@microsoft.com> Co-authored-by: Jay Rodge <jayrodge@live.com> Co-authored-by: Du Li <duli1@microsoft.com> Co-authored-by: Du Li <duli@OrtTrainingDev4.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: baijumeswani <bmeswani@microsoft.com> Co-authored-by: Sergii Dymchenko <sedymche@microsoft.com> Co-authored-by: jingyanwangms <47403504+jingyanwangms@users.noreply.github.com> Co-authored-by: satyajandhyala <satya.k.jandhyala@gmail.com> Co-authored-by: S. Manohar Karlapalem <manohar.karlapalem@intel.com> Co-authored-by: Weixing Zhang <weixingzhang@users.noreply.github.com> Co-authored-by: Suffian Khan <sukha@microsoft.com> Co-authored-by: Olivia Jain <oljain@microsoft.com> Co-authored-by: Chi Lo <54722500+chilo-ms@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Ryan Lai <rylai@microsoft.com> Co-authored-by: Jesse Benson <jesseb@microsoft.com> Co-authored-by: sfatimar <64512376+sfatimar@users.noreply.github.com> Co-authored-by: suryasidd <surya.siddharth.pemmaraju@intel.com> Co-authored-by: sfatimar <sahar.fatima@intel/com> Co-authored-by: MaajidKhan <n.maajidkhan@gmail.com> Co-authored-by: mohdansx <mohdx.ansari@intel.com> Co-authored-by: Xavier Dupré <xadupre@users.noreply.github.com> Co-authored-by: Michael Goin <mgoin@vols.utk.edu> Co-authored-by: Michael Giba <michaelgiba@gmail.com> Co-authored-by: William Tambellini <wtambellini@sdl.com> Co-authored-by: Hector Li <hecli@microsoft.com> Co-authored-by: Aishwarya <aibhanda@OrtTrainingDev4.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net> Co-authored-by: liqunfu <liqfu@microsoft.com> Co-authored-by: liqun <liqun@OrtTrainingDev4.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net> Co-authored-by: pengwa <pengwa@microsoft.com> Co-authored-by: Tang, Cheng <souptc@gmail.com> Co-authored-by: Cheng Tang <chenta@microsoft.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ye Wang <52801275+wangyems@users.noreply.github.com> Co-authored-by: Chun-Wei Chen <jacky82226@gmail.com> Co-authored-by: Vincent Wang <wangwchpku@outlook.com> Co-authored-by: Vincent Wang <weicwang@microsoft.com> Co-authored-by: Luyao Ren <375833274@qq.com> Co-authored-by: Zhang Lei <zhang.huanning@hotmail.com> Co-authored-by: Tim Harris <tiharr@microsoft.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Alberto Magni <49027342+alberto-magni@users.noreply.github.com> Co-authored-by: Wei-Sheng Chin <wschin@outlook.com> Co-authored-by: wezuo <49965641+wezuo@users.noreply.github.com> Co-authored-by: Jesse Benson <benson.jesse@gmail.com> Co-authored-by: Wei Zuo <wezuo@OrtTrainingDev3.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net> Co-authored-by: wezuo <wezuo@az-eus-v100-32gb-5-worker-mgtbby.eastus.cloudapp.azure.com> Co-authored-by: wezuo <wezuo@az-eus-v100-32gb-5-worker-yclzsf.eastus.cloudapp.azure.com> Co-authored-by: Wenbing Li <10278425+wenbingl@users.noreply.github.com> Co-authored-by: Martin Man <supermt@gmail.com> Co-authored-by: M. Zeeshan Siddiqui <mzs@microsoft.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Ori Levari <orlevari@microsoft.com> Co-authored-by: Ubuntu <OrtTrainingDev3@OrtTrainingDev3.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net> Co-authored-by: Sherlock Huang <bahuang@OrtTrainingDev3.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net> Co-authored-by: Sheil Kumar <smk2007@gmail.com> Co-authored-by: Sheil Kumar <sheilk@microsoft.com> Co-authored-by: Ryota Tomioka <ryoto@microsoft.com> Co-authored-by: Adam Pocock <adam.pocock@oracle.com> Co-authored-by: Yulong Wang <f.s@qq.com> Co-authored-by: Faith Xu <faxu@microsoft.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: suryasidd <48925384+suryasidd@users.noreply.github.com> Co-authored-by: RandySheriffH <48490400+RandySheriffH@users.noreply.github.com> Co-authored-by: Weixing Zhang <wezhan@microsoft.com> Co-authored-by: Chethan Palangotu Keshava <chethan.palangotu.keshava@intel.com> Co-authored-by: unknown <63478620+jeyblu@users.noreply.github.com>
2021-02-02 16:59:56 +00:00
[contractversion(5)]
Merge windowsai (winml layering) into master (#2956) * Initial Commit * Merged PR 3985217: add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc (#2346) add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc and violating our OS layering requirements. We linked against onecoreuap_apiset.lib in VB so we will continue doing this, but I am still unsure why not to link against onecore instead since that is where we ship. However, since Sheil is the owner of this code we will wait to discuss with him before changing anything. * Initial changes for layering * more snipping to get core into ort * update build instructions to include --build_shared_lib (#2358) * update build instructions to include --build_shared_lib * fix line breaks * Task 23998197: add winml_lib_core into onnnxruntime.dll (#2368) * Task 23998197: add winml_lib_core into onnnxruntime.dll * PR feedback build break on perf_test * return proper error when the model path isn't found (#2391) * LearningModelSession is cleaned up to use the adapter, and parts of b… (#2382) this is a big PR. we are going to move it up to layer_dev , which is still a L3 so we are still safe to do work there agile. we are going to move this into the L3 so that ryan can start doing intergration testing. we will pause for a full code review and integration test result prior to going into the L2. >>>> raw comments from previous commits >>> * LearningModelSession is cleaned up to use the adapter, and parts of binding are. * moved everything in the winmladapter made it all nano-com using, WRL to construct objects in the ORT side. base interfaces for everythign for winml to call cleaned up a bunch of winml to use the base interfaces. * more pieces * GetData across the abi. * renamed some namepsace cleaned up OrtValue cleaned up Tensor cleaned up custom ops. everything *but* learnignmodel should be clean * make sure it's building. winml.dll is still a monolith. * model moved over. everything builds clean. step ! * weak ref comment * Layer dev paulm (#2408) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * Layer dev paulm (#2414) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * Layer dev paulm (#2423) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * Layer dev paulm (#2424) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * Layer dev paulm (#2425) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Layer dev paulm (#2426) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * fixed map and sequence when passing stl types across the ABI . found a leak in nvidia driver, but skipped it. all winmlapitests pass now * Moved SessionOptions over to the abi * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * Register WinML TraceLogging provider on Onnxruntime.dll (#2455) * Register WinML TraceLogging provider on Onnxruntime.dll * Add ifdef to make sure trace logging provider has telemetry option when LAYERING_DONE * No need for ifdef for TraceLoggingOptionMicrosoftTelemetry * PR feedback * Move etw registration into lotus environment constructor and deresgister in lotus environment destructor * Brianma/cpuwinml (#2466) * allow building winml cpu without dml. * Brianma/breaks (#2469) * fix some more breaks * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * move dml checks out of winml and into the adapter * better error handling * Brianma/fi (#2470) * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * fix bad merge * Checking in a staging checkpoint point so that Ryan can work with me in parrallel * build break. * Brianma/testfails (#2473) * add missing ir version to dictvectorizer-string.onnx * add missing ir version to relu.onnx * add missing ir version to zipmap*onnx * add IR version to manually generated models * remove an unnecessary ifdef dml * Brianma/windowsai fi (#2475) * update dockerfiles/README (#2336) * Make elementwise op run 4 items per thread (#2335) Description: Describe your changes. Make elementwise op run 4 items per thread unroll for loop to leverage ILP remove unnessary N==0 check inside elementwise GPU kernel Motivation and Context Why is this change required? What problem does it solve? It can improve the performance of GPU elementwise ops. ~2% performance gain on popular NLP bert model. If it fixes an open issue, please link to the issue here. * Add CUDA GatherElements kernel (#2310) * Updates * Update test * Update * Updates * nits * PR feedback * Update * Update * PR feedback * PR comments * Update * Fix build * Fix build * Nits * Fix * Layer Normalization Fusion (#2319) basic layer normalization transform * Add FastGelu Cuda Op for Gelu and Add bias fusion (#2293) * Add FastGelu cuda op * Add AddBiasGelu for experiment * Revert "Add AddBiasGelu for experiment" This reverts commit 5c1ee019858c657e6bb75887265cb85675626e5b. * Add bias * Add unit tests * update comment * update script * fix build error * update coding style * update for CR feedback Enable half2 optimization only when cuda arch >= 7.0 * move _Tanh to common.cuh * implement CPU contrib OP Attention (#2333) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanUnusedInitializers. (#2320) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanupUnusedInitializers. This means initializers that have been replaced during graph optimizations are not left in the GraphProto when we save an optimized model. * Handle edge case where a model has an unused initializer with matching graph input by also removing the graph input. * Use non-const iterators in std::find_if calls to make centos build happy. * Nuget pipeline changes (#2305) 1. refactor the pipeline, remove some duplicated code 2. Move Windows_py_GPU_Wheels job to Win-GPU-CUDA10. We'll deprecated the "Win-GPU" pool 3. Delete cpu-nocontribops-esrp-pipeline.yml and cpu-nocontribops-pipeline.yml 4. In Linux nuget jobs, run "make install" before creating the package. So that extra RPAH info will be removed * Cuda Reverse Sequence Op, maping types of same size using same template function. (#2281) * Set ElementType to String type of node metadata, instead of byte[] (#2348) * Set ElementType to String type of node metadata, instead of byte[] * Fix spacing * Introduce PrimitiveType into a Type System along with an integer constant (#2307) Improve perf by avoiding GetType<T>() calls. Introduce MLTypeCallDispatcher to switch on Input Type. Add Tensor IsType<T>() fast method. * Fix/test dim value of 0 handling in a couple of places (#2337) * Update the CUDA Where implementation broadcasting logic to handle a dim with value of 0. Add unit test Also add unit test for unary op with dim value of 0 * Exclude ngraph from Where test with 0 dim. * Openvino EP R3.1 onnxrt server (#2357) * onnxrt server with OVEP * onnxrt server with OVEP * Update Dockerfile.server.openvino * onnxrt server OVEP fix reviews * onnxrt server OVEP fix reviews * Implement cuda nonzero op. (#2056) Implement cuda nonzero op. * Direct use python numpy array's memory if already contiguous. (#2355) * Direct use python numpy array's memory if already contiguous. This could greatly improve performance for session with large input, like big image 1920x1080 fastrcnn, 30~40% speed up could be achieved. * Add test case enforce contiguous/non-contiguos numpy array as inputs. * Add helper to create output to minimize binary size. (#2365) Add ConstEigenTensorMap typedef so we don't unnecessarily const_cast the const input Tensor. * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS (#2369) * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS * update * Add Tracelogging for profiling (#1639) Enabled only if onnxruntime_ENABLE_INSTRUMENT is ON * test bidaf with nuphar for avx target (#2370) increase nuphar test coverage a bit * Fix a bug in TLS refcount that may destabilized CUDA CI (#2374) * update output size calculation for resize (#2366) * change how output size is calculated for resize op * add tests for ver 10 resize * Extend OneHot CPU kernel to support more types (#2311) * Extend OneHot CPU kernel to support input int64_t, depth int32_t, output float * Skip BERT before the test data fix is picked up * Fix bug with Slice. Need to pass in flattened input dimensions so the initial offset into the input is calculated correctly. (#2372) * Add opset 11 version of Split to CUDA ops (#2376) Organize the CUDA ops definitions so all the opset 10 and 11 parts are together (same setup used for CPU ops) * Layer Norm Fusion Fix (#2379) * layer norm fusion fix * Add input shape check in code and unit tests * Fuse Add + Gelu (#2360) Implement the transformer to fuse add + gelu Implement the accurate kernel * Skip layer norm transform (#2350) * skip layer normalization transformer * Another try to stabilize CUDA CI (#2383) The root cause seems to be failure in CUDA dealloc when tear down. cudaFree return code was ignored before, so should the debug check. * fix BUILD.md typo (#2375) build.py: error: argument --config: invalid choice: 'RelWithDebugInfo' (choose from 'Debug', 'MinSizeRel', 'Release', 'RelWithDebInfo') * Fixed compilation with ngraph (#2388) * Fix reuse logic in allocation planner. (#2393) * Fix reuse logic in allocation planner. * PR comments * Add helpful comments * Don't allow reuse across string tensors. * [NupharEP] Multiple optimizations (#2380) Fuse transpose into MatMul Implement Pow and constant scalar simplification Vectorize ReduceMean Improve symbolic shape inference Minor updates for better debugging in fused function name * Avoid using the default logger in the graph lib and optimizers (#2361) 1. Use the session logger if it is available. 2. Don't disable warning 4100 globally. We should fix the warnings instead of disabling it. * Change CUDA implementation of Transpose to support all fixed size tensor types (#2387) * Change CUDA implementation of Transpose to not use a typed kernel so we can support more types with minimum binary size. Add support for 8, 16, 32 and 64 bit types. Add unit tests. Add method so the implementation can be called directly (will be used by CUDA Scan very soon). * Disable TensorRT for MLFloat16 and int8 unit tests. * Address PR comment and add support for calling cublas implementation if type is mlfloat16. * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. (#2398) * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. * [NupharEP] force some low/zero cost ops to be inlined (#2409) * fix cross compile bug (#2415) * Minor optimization: if a node has already been placed, there's no need to find a kernel for it. (#2417) * Add Reshape Fusion (#2395) * Add reshape fusion * Add some comments * update comments * update comment format * update according to feedback * update for recent logger change * fix build error * (1) Support both input and output edges in find path in graphutils (2) Add a test case of only one constant initializer of Concat input. (3) Refactor ReshapeFusion class to allow add more subgraph fusion in the future. * fix error * (1) loose constraint on initializer: non constant is allowed for reshape fusion. (2) Change versions type to vector. (3) Add logging. (4) Return false when multiple output edges matched in FindPath. Add comments. * only allow one direction (input or output) in FindPath * [NupharEP] Update notebook and docker image (#2416) Add BERT squad in Nuphar tutorial Enhance speed comparsion readability * Fix the issue in matmul_add_fusion (#2407) Fix the issue in matmul_add_fusion If Muatmul + Add has shape [K] * [K, N], reset it to [1, K] * [K, N] will make the output shape to [1, N] will also requires a reshape on the output. Fix: just remove the shape reset to not fuse it. Add a negative test case for matmul+add fusion * feat(treeregressor): Update TreeEnsembleRegressor for type support (#2389) Updates the `TreeEnsembleRegressor` to allow for `double`, `float`, `int64`, and `int32` inputs to match the upstream specification. Signed-off-by: Nick Groszewski <nicholas.groszewski@capitalone.com> * onnxrt server documentation update (#2396) * Added support for Pad-2 operator in OpenVINO-EP (#2405) * Add CUDA If operator. (#2377) * Add CUDA If operator. Uses CPU operator for implementation. By adding a CUDA version the inputs/outputs (with the exception of the 'cond' input) stay on GPU, and no other logic is required to avoid a copy to CPU across the control flow node. * Improved documentation for onnxruntime::utils::SwapByteOrderCopy(), added precondition check. * Fix the type constraints on CUDA If operator to exclude strings. (#2431) * add Im2col<uint8_t> (#2438) * Adjust codegen vectorization width from target (#2439) * Adjust codegen vectorization width from target * Add CUDA Scan operator. (#2403) * Add Scan CUDA op. Uses CPU implementation for logic. Added some device specific functors for handling when data needs to be manipulated on a different device. Added ability to override the materialization logic in the OrtValue slicer so DML can plugin their handling. * Fix Windows GPU C API packaging pipeline failure (#2440) Fix Windows GPU C API packaging pipeline failure (#2440) * Correctly handle implicit inputs for fused nodes (#2390) * Correctly handle implicit inputs for fused nodes Previously, nuphar's partitioning function didn't include node's implicit inputs into the inputs list of MetaDef, and hence a crash was triggered in the onnx graph checker. This commit fixed the issue. Furthermore, it also fixed a related issue where we didn't add implicit inputs into graph_inputs_excluding_initializers_ in Graph::SetGraphInputsOutputs. the issue was that graph_inputs_including_initializers_ populated by SetInputs (e.g. called by FunctionImpl::FunctionImpl) may contain implicit inputs which were not of any node's initializers in the graph. Because they were not part of any initializers, these implicit inputs couldn't be visited by going through all nodes' inputs. Consequently, they would *not* be added into graph_inputs_excluding_initializers_. We fixed the issue by first copying the populated graph_inputs_including_initializers_ into graph_inputs_excluding_initalizers_, which then had both initializers and non-initializers as its initial content. Later, we erase initializers from the list. In this way, we can ensure all implicit inputs to remain in graph_inputs_excluding_initializers_. * refined comments and fixed duplicates Address CR by revisiting comments in terms of implicit inputs Also fixed an issue by skipping duplicates while copying inputs from graph_inputs_including_initializers_. * address CR explain why we need to collect nodes' implicit inputs * don't rely on pointer values for iterating std::set Previously, openvino relied on iterating a set of NodeArg pointers to construct inputs and outputs for a fused graph. It could cause non-determinism. The reason was that although iterating std::set by itself is stable, pointer values of NodeArgs may vary. Consequently, we could end up visiting the set's elements in different orders for different runs for the same test, which resulted in constructing inputs (and outputs) with different orders to the fused graph. For example, for the same test, we may have inputs [A, B] in some runs but inputs[B, A] in others. Let's use std::string as the key type to avoid such nondeterminism. This commit also added implicit inputs into meta->inputs while returning the capability from the openvino provider. * Fixed another latent issue in openvino's GetCapability function The issue was that we couldn't simply erase fused_inputs and fused_outputs while iterating the nodes. For example, an output NodeArg may have multiple uses, and it's wrong if we erase it from fused_outputs when we encounter only one of its uses as input. * Remove DeviceAllocatorRegistry class (#2451) Remove DeviceAllocatorRegistry class * CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library. * Parallel Gelu with ParallelFor (#2399) Parallel Gelu to get better performance for Gelu * Clean up build.py (#2446) * Pull the latest image before running docker build * Fuse SkipLayerNorm with Bias (#2453) Fuse SkipLayerNorm with Bias * Allow more than one invocation of CreateEnv in the same process. (#2467) * Allow more than one invocation of CreateEnv in the same process. * Fix centos build * Symbolic shape inference improvements: (#2460) * Symbolic shape inference improvements: - add a mode to guess unknown ops' output rank - add support for GatherND - add support for If - fix a bug in get_int_values when then tensor rank > 1D, by treating it as no sympy data - add symbol to literal merge when ONNX silently merges dims - fix a bug in Concat when input dim is 0 - fix a bug in ConstantOfShape that computed dim is not updated - add support for dynamic shape in ConstantOfShape - fix a bug in Loop output shape that loop iterator dim is not inserted at dim 0 - add support for dynamic padding in Pad - add support for dynamic shape in Reshape - add support for Resize with opset > 10, by treating output dims as dynamic - fix a bug in Slice when starts/ends are dynamic - restrict input model to opset 7 and above - make output model optional to avoid disk write when testing Run model tests for symbolic shape inference Reduce 2GB docker image size of nuphar * add additional test data set for nuget pipeline (#2448) * add SAS token to download internal test data for nuget pipeline * update azure endpoint * fix keyvault download step * fix variable declaration for secret group * fix indentation * fix yaml syntax for variables * fix setting secrets for script * fix env synctax * Fix macos pipeline * attempt to add secrets to windows download data * fix mac and win data download * fix windows data download * update test data set url and location * Revert "Brianma/windowsai fi (#2475)" This reverts commit 5780b864a15513fda4eadbfc2b5345fefe70b5ec. * Add scenario tests (#2457) * Add scenario tests * Remove TODO from model license * Add winml_api test dependency * fix model load test. fi from master changed the constructor (#2483) * make api tests all pass (#2486) * fix bad merge * fix bad model merge * Layer dev paulm (#2492) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * Rename ambiguous header (#2489) * fix one more missing IR version model (#2500) * add missing IR version to 4 more models used by scenario tests (#2501) * Add CLI parameters to test runner, build WinML in ARM and x86 CI (#2479) * Support test parameters through CLI arguments * Add WinML do Windows x86/ARM CI builds * Code style fixes * Update googletest Remove GPUTEST macros everywhere now that GTEST_SKIP is supported * Refactor main.cpp * Build scenario tests without DML * Link scenario tests to DML when it's enabled (#2502) * Layer dev release pipeline (#2488) Adds winml binaries to existing cpu nuget package, and creates new gpu dml nuget package with winml binaries and DML EP. * Layer dev paulm (#2506) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * Remove usage of IOBinding in WinML and use C_API Run method (#2504) * remove usage of iobinding * Change data structure to use vector of Ort::Values * Polish bind input / output * Use C APIrun method * Update providers on evaluate getresults * Remove run and IObinding interface from WinMLAdapter * Remove use of IObinding * bind unbound outputs code moved to learningmodelbinding * clean up unneeded istensor adapter function * Fix comment * Check if session is closed before binding and clearing * PR feedback * Layer dev paulm (#2507) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * Make tests dependend on winml_dll (#2509) * add dml binaries to DirectML package and be more explicit about condition variables (#2520) * re-enable warnings for winml builds and fix the warnings that were hiding (#2526) * turn devmode back on for winml builds * fix some warnings. include protobuf in a way that disables some warnings * undo protobufhelpers changes and just ignore 4100 errors in pb code * attempt to isolate protobufhelpers errors * add template specialization for getting tensor proto data * Layer dev paulm (#2533) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * Spawn child process to run DeviceLostRecovery scenario test (#2530) * Spawn child process to run DeviceLostRecovery scenario test * Layer dev paulm (#2536) ori said yes * add missing namespace to winml_trace_logging_provider in lotusenvironment.h (#2542) * Handle exception thrown from all apis in WinMLAdapter (#2539) * various changes to unblock windowsai ADO build * Fix custom ops scenario tests (#2562) * Do not shutdown protobuf after ort environment gets destroyed. Lazy load lotus environment first time it is needed * comment typo * pr comment about calling phoenix singleton * Make lotus_environment static in winmladapter * Layer dev paulm (#2567) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * making windowsAI azure dev ops work. * code review comments. * revert changes * Cmake and preprocessor fixes that where uncovered by building on agents without DML available via SDK * Layer dev dml delayload (#2580) * Brianma/cpu (#2583) * don't include dml stuff in cpu builds * tests that link the image lib also need the telemetry lib now * Throw Winml_err_invalid_binding if binding gpu resource on cpu device (#2589) * Throw Winml_err_invalid_binding if binding gpu resource on cpu device * PR comments. No need to query executionprovider for is gpu device * User/xianz/ortthrow (#2596) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * User/xianz/ortthrow (#2599) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * add status error message * Remove uwp onsuspending winrt call because logruntimeperf is getting removed (#2630) * User/xianz/dedup telemetry (#2631) * investigate duplication of telemetry in winml and ort * remove winml telemetry events * telemetry executionProviderEvent * remove unneccessary file and refactor code little bit * Revert back TelemetryEvent, which send up ETW event. * merge changes from layer_dev to windowsai (#2638) * Remove underscore from googletest names (#2616) * Fix leaking memory allocator Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24278761 and https://microsoft.visualstudio.com/OS/_workitems/edit/24330198 * Explicitly initialize Ort::Value with nullptr * Cache WinML adapter * bad merge * define private version of dxcore enum that is added in 19H1 SDK. (#2654) * add comment for explaning private definition of dxcore d3d feature level ennum value. (#2672) * do not package directml.pdb for redist packages. (#2676) * Fix leaking operator registry (#2645) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24354916 * User/orilevari/windowsai master merge (#2674) merge resolutions included pulling in telemetry logic that was merged to master and not windowsai and dereferencing InferenceSession::sessionstate now that it is a unique pointer * Delete Ort Allocator in LearningModelBinding (#2653) * Delete OrtAllocator in LearningModelBinding * PR comments to make Ort::Allocator a smart pointer * Small comment change * PR feedback to clean up code * PR feedback on move semantics * Clean up std::move * Fix memory leaks (#2679) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24356109, https://microsoft.visualstudio.com/OS/_workitems/edit/24388361 and https://microsoft.visualstudio.com/OS/_workitems/edit/24388596 * various changes to properly organize and skip GPU tests. For now for No DML builds we will not run GPU tests at all. In the future we should adapt the tests to expect the appropiate errors. (#2695) * Windowsai without fi (#2701) * Disable Attention fusion tests when DISABLE_CONTRIB_OPS is defined (#2529) * Setup java ci (#2528) * Add provision in ORT for session options to be parsed when available via model file (#2449) * Initial commit * Fix gitmodules * Nits * Nits * Updates * Update * More changes * Updates * Update * Some updates * More changes * Update * Update * Merge * Update * Updates * More changes * Update * Fix nits * Updates * Fix warning * Fix build * Add comment * PR feedback * PR feedback * Updates * Updates * Update * More changes * Fix build break * Comment test for now * Updates * Updates * PR feedback * Updates * Nits * Add tests * Fix build * Fix build * Fix build * Fix build break * Fix build * Nits * PR feedback * More change * Expose GetSessionOptions in pybind logic and add unit test for python * Fix build * PR feedback * PR feedback * Revert "Disable thread pool creation when enabled OpenMP (#2485)" (#2535) This reverts commit 7c7d5a149c9ed52eec67304bae5c4b132166a8a1. * Add dynamic shape support in TensorRT execution provider (#2450) * remove onnx-tensorrt submodule * add new onnx-tensorrt submodule (experiment) for trt6 * update engine build for trt6 * update compile and compute for tensorrt6.0 * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * switch to onnx-tensorrt master for TensorRT6' * Update tensorrt_execution_provider.cc * Handle dynamic batch size and add memcpy in TensorRT EP * update test cases * Update tensorrt_execution_provider.cc * update onnx-tensorrt submodule * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.ubuntu_tensorrt * Update run_dockerbuild.sh * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update concat_op_test.cc * Update tensorrt_execution_provider.cc * Upgrade TensorRT to version 6.0.1.5 * Update onnxruntime_providers.cmake * Update CMakeLists.txt * Update reduction_ops_test.cc * Update install_ubuntu.sh * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.tensorrt * Update BUILD.md * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update onnxruntime_providers.cmake * Update install_ubuntu.sh * Update install_ubuntu.sh * Update gemm_test.cc * Update gather_op_test.cc * Update CMakeLists.txt * Removed submodule * update onnx-tensorrt submodule * update header file * Removed submodule * add submodule onnx-tensorrt kevin's branch shape-test' * add debugging code * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * merge master * Removed submodule * update onnx-tensorrt submodule * add more changes for dynamic shapes * Update tensorrt_execution_provider.cc * update for dynamic shape * update dynamic shape processing * fix logger issue * remove submodule onnx-tensorrt * add submodule onnx-tensorrt * add env variable min_subgraph_size * remove redundency * update document * use onnxruntime::make_unique * fix multi-run issue * remove some tests to save CI build time * Add dynamic shape test * Update TensorRT-ExecutionProvider.md * Add example of running Faster R-CNN model on TensorRT EP * Add more details on env variables * update environment variables * Update tensorrt_basic_test.cc * Update model tests * Update tensor_op_test.cc * remove --use_full_protobuf * Update build.py * User/xianz/telemetry (#2458) * enabme telemetry * enable telemetry * set enable telemetry as default * for debugging * remove log and set disable telemetry as default back * delete private file while testing * resolve comment: mainly add license header, rename macro and update docs * rewording in privacy.md * Fix integer overflow in cuda NonMaxSuppression implementation (#2540) * add test case that should pass but fail * fix nms * extract int_max_output_boxes_per_class * Introduce container type runtime checks and other improvements (#2522) Rework TensorSeq in a manner consistent with Tensor and SparseTensor in terms of type system setup. Reduce templating. Introduce helpers to ensure the same data type. Make OrtValue __dtor not virtual. Introduce ContainerChecker * Fix C API tests for centos and mac (#2544) * change c++14 to c++11 * add ld lib path for centos * enable csharp tests on macos * fix C API test on MacOS + fix manylinux dotnet install * fix manylinux dotnet install * fix lib link * Add back executable bit to build.py * Fix a bug handling negative begin pad values in Pad op (#2550) * Fix bug in Pad op * Update * DNNL CMAKE update (#2548) * Fix android build (#2558) * Update win-x86-ci.yml (#2557) Fix build pipeline break * Re-enable Windows C# tests (#2564) * disable onnx_test_runner -x invocations for dnnl (#2568) * Allow sequence length to be symbolic (#2559) * setup java ci mac (#2570) * make layernorm fusion to support opset 11 (#2545) * Fix a warning found in the latest VS release * Add more check on SkipLayerNorm and BiasGelu fusion (#2574) * Fix file not found error during docker build. (#2569) * Add ConvTranspose1D (#2578) * Ryanunderhill/packagename test (#2582) * [Nuphar EP] fixes for some object detection models (#2581) Update notebook tutorial with multi-threaded int8 GEMM from #2517 * EmbedLayerNormalization Fusion Improvement (#2553) Embedding layer norm fusion improvements - add more checks * Update version (#2584) * Temporarily exclude vgg19 test from Python backend test 1. temporarily exclude vgg19 test which comsumes too much memory, run out of memory on Upsquared device. Single test pass for vgg19, need furture investigation (#2588) 2. Update docker file to decrease the docker image size * Update docs for Android NNAPI EP (#2586) * Fix lto bug for protobuf and ubuntu * add path to build dir before test run (#2590) * Add missig env variables for mac pipeline test (#2595) * Fixed an issue in updating realized dims (#2597) when we update realized dims for scan's output, the sliced axis also needs to be inclusive, i.e. we should check with "dim >= insert_inclusive_axis", because the offset in the symbols are based on Scan sugraph. Otherwise, we would end up with shape mismatch later. * Java API for onnxruntime (#2215) * Add support for opset 11 in reshape fusion (#2592) Support opset verion 11 in reshape fusion * Rename automl python tools folder to featurizer_ops. (#2593) * Support opset 11 subgraph of Squad model in Embed Layer Normalization (#2605) Support opset 11 Squad model that is exported from PyTorch nightly. The embed layer uses Range op which is missed in the transformer. * symbolic shape inference: fix warnings in GPT-2 model (#2608) And revise nuphar perf test on BERT squad * Dump subgraph ID and fused graph ID (#2607) * Dump subgraph ID and fused graph ID Dump subgraph ID and fused graph ID for better debugging * Remove local static fused_count added a field global_fused_count_ to NupharExecutionProvider class * EmbedLayerNormalization Fusion For Dynamic Squad Model Opset 10 (#2613) Support subgraph of SQuAD model exported from pytorch with dynamic input axes * Allow providers to be set for InferenceSession at construction (#2606) * Remove unnecessary parameter in some places in GatherElements implementation (#2612) * Remove unnecessary parameter in some places * Update * Update * Make sure fenced tensor could not reuse other tensor. (#2561) Fix random error caused by this. * Improve Embed Layer Norm Fusion for SQuAD with static input shape (#2621) * fix float16 comparison in initializer (#2629) * epsilon attribute for layernormalization fusion (#2639) * removed unnecessary batch file and fix path (#2640) * Add shape inference to ConvTransposeWithDynamicPads schema (#2632) * Improve cuda expand() opeator's performance. (#2624) * Cuda pad optimize when no padding is needed. (#2625) * Shortcut cuda Pad() when no padding is needed. * Optimize cuda scatter() on 2D compatible. (#2628) * Optimize cuda scatter() on 2D compatible. * Add some comments. * fix build error for ARM (#2648) * Improve performance of resize() in Nearest mode (#2626) Special treatment for 2D, check same size as input image. And in 2d kernel, template use_expolation. * Fix memory exception in Layer Norm Fusion (#2644) * Windows CI changes(#2650) * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Revert "Windowsai without fi (#2701)" This reverts commit 285d4c85ff5c4e265f963208170304ef3461e684. * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Deref unique pointer for session_state * send shutdown event when dll is unloaded and EvaluationStop, SessionC… (#2704) * send shutdown event when dll is unloaded and EvaluationStop, SessionCreationStart Events. * Add EvalutationStart Event * add comment * use correct type for for loop (#2755) * ARM CI (#2759) * Set ARM agent pool * Set CMake generator to VS 2019 in ARM * Use system-wide CMake instead of custom version Our custom version is too old for VS 2019 * Use DML and build shared lib in ARM CI * Restore nuget packages in ARM CI * Disable DML * Refactor ARM debug/release builds * Use system packaged Python version * Remove hardcoded Python path * Downgrade Python to 3.7 for build * Remove explicit CMake path * Fix invalid JSON in cgmanifest.json (#2760) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * Remove usage of WHOLEARCHIVE in WinML CMake and add WinMLAdapterFactory (#2726) * Remove usage of WHOLEARCHIVE in WinMLAdapter CMake and add WinMLAdapterFactory * PR feedback, no need for dll(export) since using def file * PR comments * Small comment in gen_def.py * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * Move winml_provider_factory.h to proper location (#2801) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Filter CPU case for IsFloat16Supported (#2802) * Merge fixes * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * Fix warnings that cause build to fail * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * Make DML include path global (#2882) * Make DML include path global * Add generated cppwinrt headers to winml_lib_common * Integrate changes to WindowsAI to make ADO Build (#2886) * Revert "CMake cross-generator fixes (#2790)" This reverts commit dbe7d97fa1ab155f1309bced87199527e8f35bd2. * add additional suppress warning in onnx_proto * ignore /wd4996 warning * DML execution provider fixes * Revert "Revert "CMake cross-generator fixes (#2790)"" This reverts commit 1ae7b4bcbc02edc881ad28685da98e095dfceb17. * Update func signature of custom op function overloads * common devicehelpers fixes * Add pch.h for winml_lib_common * re-add winml_lib_common_dir/inc to include path for winml_adapter * User/orilevari/dml redist shared folder (#2890) * move dml nuget package directory up one level to make it shared between build flavors * Merge conflict fix * Revert "Merge conflict fix" This reverts commit 142fa72cf9ce4344ad717b50b7ea2b8582aadc7c. * Revert "Merge remote-tracking branch 'origin/master' into windowsai" This reverts commit 6e2126d46e5e5f564d65da37dd4f70c93dd81165, reversing changes made to b3f5583dc9249834b947c8ea905f6a98060d5bd6. * Make winml_test_common free of test macros (#2902) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Don't generate files outside the build root (#2914) * Don't generate files outside the build root * Add onnxruntime_EXTERNAL_DEPENDENCIES to WinML * Add DML depedency on RESTORE_PACKAGES * User/orilevari/fix yaml merge bugs (#2918) * Add winml test source parameter into cmake function (#2919) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Add arguments to unittest cmake functions * remove comment * Revert "Revert "Merge remote-tracking branch 'origin/master' into windowsai"" This reverts commit ade5abe72a4234fdbc3623093c61c02c6b0bdc26. * Fix breaks from merge with ORT master * Brianma/linux (#2917) * don't include windows.h in cross-plat header * add default case for switch statement * signed/unsigned mismatch fix Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> * User/sheilk/winml adapter c api (#2891) * Create winml adapter c api * fix build * make it build * move adapter into onnxruntime core/session * entry point not exported * minor changes * make model metadata work * make tests pass * implement all the model reflection apis on the adapter c abi * update the new ort interface to create a lotus ennvironment with a logging sink * start adding ort env * move all winml code into adapter folder/lib to isolate it * ensure a single logging manager at a time * start refactoring session * refactor session creation interface * add cpu and dml session option methods to adapter * finish session init * stub out interfaces in ort lib to perform similar mechanics of iinference session * enable profiling, and enable schema override * update session register graph transformers * turn back on custom registry for custom ops * Add sync api * add last c api stubs * should build... but all feature values are broken since this is in flight to moving all implementation details into ivalue * remove ep adapter header * Implement DML execution provider functions from adapter (#2846) * Implement DML execution provider functions from adapter * Use functions in OnnxruntimeEngine.cpp * make map/sequence type_infos freeable, and start implementing ivalue * make it build again * implement value methods * implement remaining methods * remove com adapter abi * check dml session * cache the allocator on ivalue * check if resource is cpu/gpu when access its mutable data * update tensor * mismatched parentheses * fix tensor base and binding obj * it evaluates tensors! sometimes... * minor fixes * enable gpu evals * wrapper all existing winml adapter apis with API_IMPL to try catch (#2854) * update winml... tensor strings are broken, need to template tensorbase to do different things for strings * make tensor strings work with 2 copies in/2 copies out * Fix tensor string and allocator bug * make maps work again... needs some fixes still * Make it build! * enable map inputs * map outputs * unbound outputs for sequences and maps * User/xianz/merge windowsai (#2883) * Packaging pipeline changes for VS 2019 (#2711) * Tiny fix to codegen * Simplify cache implementation and avoid static variables that may carry over between models * Extend DML kernels (#2641) * Additional DML operators * Check unsupported attributes and inputs * Address PR comments * Add kernel capability function used for partitioning, and re-enable stride-based int64 support based on value range * Fix test failures * Build fix * PR comments * Update Nuphar tutorial notebook (#2721) 1. Reflect int8 GEMV improvements for multi-threading from #2696 2. Add notes on multi-threading control using OpenMP 3. Add samples of running multi-isa AOT, and show int8 GEMM differences between AVX and AVX2 4. Add rnn_benchmark example to resolve #1993 * Add schema for new Qops (#2611) * Add schema for new Qops * adding shape inference + qlinearaveragepool * plus review comments * plus review comments * updates per review comments * plus review comments * [server] Add supposed for model_name and model_version as cli parameter (#2708) * remove 64bit warning message from python validation. (#2727) * MLAS: ARM64 build fix (#2734) fix bad usage of vreinterpret to cast vector element types * Fix broken python docs links (#2740) * Fix build on Mac OS (#2731) mac os ld doesn't support --while-archive, correct option is -all_load * fix ngraph wheel (#2737) * fix ngraph wheel 1.1.0 onnxruntime_ngraph wheel doesn't work * remove libdnnl.so in nGraph Libs * make it easy to compare * Split onnxruntime server to a separated folder (#2744) * Fix build for Python 3.8 (#2747) * Fix build for Python 3.8 * Update protobuf to 3.11.2 (#1928) Update protobuf to 3.11.2 (#1928) * Change default optimization level to All (from Basic) (#2745) * change default optimization level to All (from Basic) * fix test * fix c# test * Update numpy to 1.18 (#2758) * Update numpy to 1.18 * Pipeline changes for python 3.8 (#2753) 1. Pipeline changes for python 3.8 2. Fix a regression in setup.py which was just introduced in the previous commit. Please notice, we still haven't made python 3.8 + Windows + CUDA work. * Add basic stacktrace output for posix debug builds. (#2749) * [NupharEP] fix a race condition when multiple sessions running different models concurrently (#2772) * Revert "Change default optimization level to All (from Basic) (#2745)" This reverts commit 56bb503c2f26474b6613bcb2a198691a11dcef38. * Fix typo in error message (#2736) * Rename MKL-DNN to DNNL to fix broken link (#2730) * Fix nightly build version number issue * Pass BUILD_BUILDNUMBER to linux docker * Disable featurizers in python packages * Import more featurizers (#2781) Make kernels non-template. Add input constraint for learnt data. Add min_max_scalar_transformer, robust_scalar_transformer, inputation_marker_transfomer, label_encoder_transformer, missing_dummies_transformer along with tests. Advance Featurizers library commit. * Implement a more stable softmax (#2715) * Implement a more stable SoftMax e^x is represented as infinity if x is large enough, like 100.f. Infinity divided by Infinity is a NAN. Thus, softmax gets a NAN if one or more item are large enough. A math transform as below is leveraged to get a stable softmax: e^xi/(e^x1 + ...e^xn) = e^(xi - max) / (e^(x1 - max) + ... + e^(xn - max)) And for convenience, force max to 0.f if all xi are negative * Contributing: Fix a typo (#2784) * ACL EP GEMM improvements (#2780) When it is posible we use a fully connected layer instead of the gemm implementation. This will let the library use the best implementation based on the input data. * ACL EP convolution improvements (#2774) Added the optimized implementation for depthwise convolution for both ACL v19.02 and ACL 19.05. Also the pointwise convolution seems to be more optimal in the CPU implementation so we opted for that instead. * Add script for release Nuget validation (#2719) * Initial commit * Nits * Disable a test temporarily * Change working directory * Test * Add download python step * Test update * More changes * Fix space issue * Fix * Verify nuget signing * Fix * Spaces * PR feedback * Nit * Fix * Fix * Remove temporary changes * add uint8 support to where op (#2792) * Improve bert optimization script: (#2712) (1) Move input int64=>int32 conversion to embed layer fusion. (2) Output epsilon attribute for LayerNormalization fusion. * add session creation time cost. (#2798) * ML.NET team needs featurizers within a package (#2789) Add auto ml featurizers to Windows, MacOS as well as to GPU packaging-pipelines. * Initialize max of softmax with lowest of float (#2786) * MLAS: update SGEMM threading parameters (#2808) * add interface to copy batch tensors. (#2807) * add interface to copy batch tensors. * onnxruntime * speed up Windows TRT CI (#2811) * don't run cuda tests if building with tensorrt * remove unnecessary build options for win trt ci * refactor win gpu tensorrt ci yml * --numpy_version=1.17 * update * update * azcopy and cuda path * Update test data (#2356) * Add timeseries imputer transformer featurizer kernel (#2813) Make kernels non-template. Add input constraint for learnt data. Fixup tests. Add two more featurizers along with tests. Tests fail. min_max_scalar_transformer robust_scalar_transformer Fix tests serialized stream by prepending version bytes. Add inputation_marker_transfomer and the test. Fix up float/double type designations. Added label_encoder_transformer along with a test. string_throw case is broken at the momement. Fix labelencodertransfomer_test.cc string_throw case Rename maxabsscalertransformer_test.cc Add MissingDummiesTransformer along with the test. Update manifest. Add TimeSeriesImputerTransformer definition, implementation and tests * Fix memory leak in TRT (#2815) * fix memory leak issue * revert EP_FAIL on enueueV2 * Add manifest missing comma * Run static code analyzer on most of our code (#2817) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * update quantization doc (#2783) * update documentation for quantization script * plus some spell corrections * Filter CPU case for IsFloat16Supported (#2802) * update default optimization level + fix gemm_activation fusion (#2791) * update defualt optimization level + fix gemm_activation fusion * fix typo * add unit test and incorporate review comments * fix test comment * Fix dnnl wheel package name (#2823) * Append '-dnnl' to whl package name when --use_dnnl * Update build.py * Update Ubuntu & TensorRT version in README (#2820) Dockerfile.tensorrt is using nvcr.io/nvidia/tensorrt:19.09-py3 as base Image, update Ubuntu and TensorRT version according to https://docs.nvidia.com/deeplearning/sdk/tensorrt-container-release-notes/rel_19-09.html#rel_19-09 * Merge fixes * Add OneHotEncoder and HashOneHotEncoder kernels. (#2830) Add defs and imlementation for OneHotEncoders, adjuist date_time_transformer kernel and test. Add OneHotEncoder kernel test. Add HashOneHotVectorizerTransformer unit test. This does not link due to multiple definitions of functions that are included into header from a CPP file. * Upgrade gtest to the latest version (#2827) WinML would like to update the googletest submodule. They want some newer features (namely GTEST_SKIP to skip tests programmatically and be able to skip entire fixtures easily) and would need to update the submodule version. However, because the new version of code hit a bug in gcc, even though the bug is already fixed in the latest gcc but we're using gcc 4.8.x and it won't get patched for the bug, so we have to do a compromise, change our code a little bit to make it work. The gcc bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=51213 * Add support for int64_t for topk CPU. Fixes github issue #2806. (#2833) * Ignore allocator type in ExecutionProviders allocator map. Make default initialization of OrtMemoryInfo more clearly invalid. (#2768) * Remove allocator type from the key comparison in ExecutionProviders. Remove usage of DummyArena as it's no longer necessary. * Fix x86 tests where arena allocator is disabled. Make initialization of OrtMemoryInfo clearer by adding Invalid enum value. * Make OrtValueNameIdxMap::MaxIdx more intuitive. * Convert ExternalProject Featurizers into git submodule (#2834) Add git submodule for Featurizer library. Update cmake to build for git submodule. * add domain check for nodes + update documentation (#2831) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Fix Linux CUDA nuget packaging pipeline break * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * update optimization doc for BERT related fusions (#2819) * Add bert related transformers to doc * Add execution provider and comment for bert optimizations * Add comment about accuracy impact of approximation * Fix warnings that cause build to fail * MLAS: enable threading for quantized GEMMs (#2844) * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * remove old winmladapter cpp Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * move sequence implementation into ort lib... still commented out... need to turn back on... * begin sequence implementation * make maps and sequences work * fix broken tests * remove dead code * misc cleanup * CR feedback * User/xianz/winml adapter c api (#2869) * wrapper all existing winml adapter apis with API_IMPL to try catch * Return HR or Throw for WinML adapter APIs if failed * undo macro wrapper for two places * Wrap error macros around ort apis, too. * address CR feedback #2 * add more api throw/return macros * Revert changes no longer needed * revert changes to cxx api * format winml lib.ort and winml adapter * remove static pheonix singleton Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * missing use_dml check in winml_adapter_session (#2930) * --use_dnnl flag was mangled in merge (#2931) * use dml macro not wrapping custom registry code (#2934) * Disable LNK4199 winml_dll to enable cuda builds (#2936) * Disable LNK4199 in winml_dll * linkler->linker * LearningModelSessionAPITestGpu.CreateSessionWithCastToFloat16InModel should return DXGI_ERROR_UNSUPPORTED when FP16 not supported (#2937) * Disable LNK4199 in winml_dll * linkler->linker * Need to return DXGI_ERROR_UNSUPPORTED when Model does not support fp16 * Publish build symbols (#2939) * Publish build symbols * Don't upload PDBs for .exe files * Make x86 build (#2943) * fix last remaining size_t/int64_t warnings->errors (#2948) * TensorString, Sequences and Maps use the first allocator, but should use the cpu default allocator. (#2952) * fix tensor string allcoator * clean up default allocator usage for strings in winml lib/api.ort Co-authored-by: Ryan Lai <ryalai96@gmail.com> * Handle tensor shape of zero (#2954) Co-authored-by: Ryan Lai <ryalai96@gmail.com> * CR feedback (#2970) * CR feedback * fix weird formatting on privacy readme * Add 'All rights reserved.' everywhere * readd all rights reserved to winml_provider_factory.h * remove extra space in comment * remove extra whitespace * fixes post master merge * remove winml from nuget gpu pipeline * set IR VERSION on generated_model in rnn_benchmark (#2972) * Fix slice conformance failures (#2908) Co-authored-by: Adrian Tsai <adtsai@microsoft.com> Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Paul McDaniel <paul_mcdaniel@hotmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Dwayne Robinson <fdwr@hotmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com>
2020-02-05 01:12:19 +00:00
apicontract MachineLearningContract{};
//! Forward declarations
runtimeclass LearningModelBinding;
//! \enum LearningModelFeatureKind
//! \brief Defines the list of input and output feature types for a machine learning model.
//! Each of these maps to a corresponding FeatureDescriptor that you can use to learn more
//! about how to pass the feature into and out of the the model.
[contract(MachineLearningContract, 1)]
enum LearningModelFeatureKind
{
//! The feature is a tensor, use TensorFeatureDescriptor
Tensor = 0,
//! The feature is a sequence, use SequenceFeatureDescriptor
Sequence,
//! The feature is a map, use MapFeatureDescriptor
Map,
//! The feature is an image, use ImageFeatureDescriptor
Image
};
//! \brief Describes the common properties that all features have.
INBOX_ONLY([uuid(bc08cf7c-6ed0-4004-97ba-b9a2eecd2b4f)])
Merge windowsai (winml layering) into master (#2956) * Initial Commit * Merged PR 3985217: add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc (#2346) add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc and violating our OS layering requirements. We linked against onecoreuap_apiset.lib in VB so we will continue doing this, but I am still unsure why not to link against onecore instead since that is where we ship. However, since Sheil is the owner of this code we will wait to discuss with him before changing anything. * Initial changes for layering * more snipping to get core into ort * update build instructions to include --build_shared_lib (#2358) * update build instructions to include --build_shared_lib * fix line breaks * Task 23998197: add winml_lib_core into onnnxruntime.dll (#2368) * Task 23998197: add winml_lib_core into onnnxruntime.dll * PR feedback build break on perf_test * return proper error when the model path isn't found (#2391) * LearningModelSession is cleaned up to use the adapter, and parts of b… (#2382) this is a big PR. we are going to move it up to layer_dev , which is still a L3 so we are still safe to do work there agile. we are going to move this into the L3 so that ryan can start doing intergration testing. we will pause for a full code review and integration test result prior to going into the L2. >>>> raw comments from previous commits >>> * LearningModelSession is cleaned up to use the adapter, and parts of binding are. * moved everything in the winmladapter made it all nano-com using, WRL to construct objects in the ORT side. base interfaces for everythign for winml to call cleaned up a bunch of winml to use the base interfaces. * more pieces * GetData across the abi. * renamed some namepsace cleaned up OrtValue cleaned up Tensor cleaned up custom ops. everything *but* learnignmodel should be clean * make sure it's building. winml.dll is still a monolith. * model moved over. everything builds clean. step ! * weak ref comment * Layer dev paulm (#2408) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * Layer dev paulm (#2414) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * Layer dev paulm (#2423) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * Layer dev paulm (#2424) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * Layer dev paulm (#2425) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Layer dev paulm (#2426) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * fixed map and sequence when passing stl types across the ABI . found a leak in nvidia driver, but skipped it. all winmlapitests pass now * Moved SessionOptions over to the abi * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * Register WinML TraceLogging provider on Onnxruntime.dll (#2455) * Register WinML TraceLogging provider on Onnxruntime.dll * Add ifdef to make sure trace logging provider has telemetry option when LAYERING_DONE * No need for ifdef for TraceLoggingOptionMicrosoftTelemetry * PR feedback * Move etw registration into lotus environment constructor and deresgister in lotus environment destructor * Brianma/cpuwinml (#2466) * allow building winml cpu without dml. * Brianma/breaks (#2469) * fix some more breaks * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * move dml checks out of winml and into the adapter * better error handling * Brianma/fi (#2470) * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * fix bad merge * Checking in a staging checkpoint point so that Ryan can work with me in parrallel * build break. * Brianma/testfails (#2473) * add missing ir version to dictvectorizer-string.onnx * add missing ir version to relu.onnx * add missing ir version to zipmap*onnx * add IR version to manually generated models * remove an unnecessary ifdef dml * Brianma/windowsai fi (#2475) * update dockerfiles/README (#2336) * Make elementwise op run 4 items per thread (#2335) Description: Describe your changes. Make elementwise op run 4 items per thread unroll for loop to leverage ILP remove unnessary N==0 check inside elementwise GPU kernel Motivation and Context Why is this change required? What problem does it solve? It can improve the performance of GPU elementwise ops. ~2% performance gain on popular NLP bert model. If it fixes an open issue, please link to the issue here. * Add CUDA GatherElements kernel (#2310) * Updates * Update test * Update * Updates * nits * PR feedback * Update * Update * PR feedback * PR comments * Update * Fix build * Fix build * Nits * Fix * Layer Normalization Fusion (#2319) basic layer normalization transform * Add FastGelu Cuda Op for Gelu and Add bias fusion (#2293) * Add FastGelu cuda op * Add AddBiasGelu for experiment * Revert "Add AddBiasGelu for experiment" This reverts commit 5c1ee019858c657e6bb75887265cb85675626e5b. * Add bias * Add unit tests * update comment * update script * fix build error * update coding style * update for CR feedback Enable half2 optimization only when cuda arch >= 7.0 * move _Tanh to common.cuh * implement CPU contrib OP Attention (#2333) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanUnusedInitializers. (#2320) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanupUnusedInitializers. This means initializers that have been replaced during graph optimizations are not left in the GraphProto when we save an optimized model. * Handle edge case where a model has an unused initializer with matching graph input by also removing the graph input. * Use non-const iterators in std::find_if calls to make centos build happy. * Nuget pipeline changes (#2305) 1. refactor the pipeline, remove some duplicated code 2. Move Windows_py_GPU_Wheels job to Win-GPU-CUDA10. We'll deprecated the "Win-GPU" pool 3. Delete cpu-nocontribops-esrp-pipeline.yml and cpu-nocontribops-pipeline.yml 4. In Linux nuget jobs, run "make install" before creating the package. So that extra RPAH info will be removed * Cuda Reverse Sequence Op, maping types of same size using same template function. (#2281) * Set ElementType to String type of node metadata, instead of byte[] (#2348) * Set ElementType to String type of node metadata, instead of byte[] * Fix spacing * Introduce PrimitiveType into a Type System along with an integer constant (#2307) Improve perf by avoiding GetType<T>() calls. Introduce MLTypeCallDispatcher to switch on Input Type. Add Tensor IsType<T>() fast method. * Fix/test dim value of 0 handling in a couple of places (#2337) * Update the CUDA Where implementation broadcasting logic to handle a dim with value of 0. Add unit test Also add unit test for unary op with dim value of 0 * Exclude ngraph from Where test with 0 dim. * Openvino EP R3.1 onnxrt server (#2357) * onnxrt server with OVEP * onnxrt server with OVEP * Update Dockerfile.server.openvino * onnxrt server OVEP fix reviews * onnxrt server OVEP fix reviews * Implement cuda nonzero op. (#2056) Implement cuda nonzero op. * Direct use python numpy array's memory if already contiguous. (#2355) * Direct use python numpy array's memory if already contiguous. This could greatly improve performance for session with large input, like big image 1920x1080 fastrcnn, 30~40% speed up could be achieved. * Add test case enforce contiguous/non-contiguos numpy array as inputs. * Add helper to create output to minimize binary size. (#2365) Add ConstEigenTensorMap typedef so we don't unnecessarily const_cast the const input Tensor. * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS (#2369) * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS * update * Add Tracelogging for profiling (#1639) Enabled only if onnxruntime_ENABLE_INSTRUMENT is ON * test bidaf with nuphar for avx target (#2370) increase nuphar test coverage a bit * Fix a bug in TLS refcount that may destabilized CUDA CI (#2374) * update output size calculation for resize (#2366) * change how output size is calculated for resize op * add tests for ver 10 resize * Extend OneHot CPU kernel to support more types (#2311) * Extend OneHot CPU kernel to support input int64_t, depth int32_t, output float * Skip BERT before the test data fix is picked up * Fix bug with Slice. Need to pass in flattened input dimensions so the initial offset into the input is calculated correctly. (#2372) * Add opset 11 version of Split to CUDA ops (#2376) Organize the CUDA ops definitions so all the opset 10 and 11 parts are together (same setup used for CPU ops) * Layer Norm Fusion Fix (#2379) * layer norm fusion fix * Add input shape check in code and unit tests * Fuse Add + Gelu (#2360) Implement the transformer to fuse add + gelu Implement the accurate kernel * Skip layer norm transform (#2350) * skip layer normalization transformer * Another try to stabilize CUDA CI (#2383) The root cause seems to be failure in CUDA dealloc when tear down. cudaFree return code was ignored before, so should the debug check. * fix BUILD.md typo (#2375) build.py: error: argument --config: invalid choice: 'RelWithDebugInfo' (choose from 'Debug', 'MinSizeRel', 'Release', 'RelWithDebInfo') * Fixed compilation with ngraph (#2388) * Fix reuse logic in allocation planner. (#2393) * Fix reuse logic in allocation planner. * PR comments * Add helpful comments * Don't allow reuse across string tensors. * [NupharEP] Multiple optimizations (#2380) Fuse transpose into MatMul Implement Pow and constant scalar simplification Vectorize ReduceMean Improve symbolic shape inference Minor updates for better debugging in fused function name * Avoid using the default logger in the graph lib and optimizers (#2361) 1. Use the session logger if it is available. 2. Don't disable warning 4100 globally. We should fix the warnings instead of disabling it. * Change CUDA implementation of Transpose to support all fixed size tensor types (#2387) * Change CUDA implementation of Transpose to not use a typed kernel so we can support more types with minimum binary size. Add support for 8, 16, 32 and 64 bit types. Add unit tests. Add method so the implementation can be called directly (will be used by CUDA Scan very soon). * Disable TensorRT for MLFloat16 and int8 unit tests. * Address PR comment and add support for calling cublas implementation if type is mlfloat16. * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. (#2398) * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. * [NupharEP] force some low/zero cost ops to be inlined (#2409) * fix cross compile bug (#2415) * Minor optimization: if a node has already been placed, there's no need to find a kernel for it. (#2417) * Add Reshape Fusion (#2395) * Add reshape fusion * Add some comments * update comments * update comment format * update according to feedback * update for recent logger change * fix build error * (1) Support both input and output edges in find path in graphutils (2) Add a test case of only one constant initializer of Concat input. (3) Refactor ReshapeFusion class to allow add more subgraph fusion in the future. * fix error * (1) loose constraint on initializer: non constant is allowed for reshape fusion. (2) Change versions type to vector. (3) Add logging. (4) Return false when multiple output edges matched in FindPath. Add comments. * only allow one direction (input or output) in FindPath * [NupharEP] Update notebook and docker image (#2416) Add BERT squad in Nuphar tutorial Enhance speed comparsion readability * Fix the issue in matmul_add_fusion (#2407) Fix the issue in matmul_add_fusion If Muatmul + Add has shape [K] * [K, N], reset it to [1, K] * [K, N] will make the output shape to [1, N] will also requires a reshape on the output. Fix: just remove the shape reset to not fuse it. Add a negative test case for matmul+add fusion * feat(treeregressor): Update TreeEnsembleRegressor for type support (#2389) Updates the `TreeEnsembleRegressor` to allow for `double`, `float`, `int64`, and `int32` inputs to match the upstream specification. Signed-off-by: Nick Groszewski <nicholas.groszewski@capitalone.com> * onnxrt server documentation update (#2396) * Added support for Pad-2 operator in OpenVINO-EP (#2405) * Add CUDA If operator. (#2377) * Add CUDA If operator. Uses CPU operator for implementation. By adding a CUDA version the inputs/outputs (with the exception of the 'cond' input) stay on GPU, and no other logic is required to avoid a copy to CPU across the control flow node. * Improved documentation for onnxruntime::utils::SwapByteOrderCopy(), added precondition check. * Fix the type constraints on CUDA If operator to exclude strings. (#2431) * add Im2col<uint8_t> (#2438) * Adjust codegen vectorization width from target (#2439) * Adjust codegen vectorization width from target * Add CUDA Scan operator. (#2403) * Add Scan CUDA op. Uses CPU implementation for logic. Added some device specific functors for handling when data needs to be manipulated on a different device. Added ability to override the materialization logic in the OrtValue slicer so DML can plugin their handling. * Fix Windows GPU C API packaging pipeline failure (#2440) Fix Windows GPU C API packaging pipeline failure (#2440) * Correctly handle implicit inputs for fused nodes (#2390) * Correctly handle implicit inputs for fused nodes Previously, nuphar's partitioning function didn't include node's implicit inputs into the inputs list of MetaDef, and hence a crash was triggered in the onnx graph checker. This commit fixed the issue. Furthermore, it also fixed a related issue where we didn't add implicit inputs into graph_inputs_excluding_initializers_ in Graph::SetGraphInputsOutputs. the issue was that graph_inputs_including_initializers_ populated by SetInputs (e.g. called by FunctionImpl::FunctionImpl) may contain implicit inputs which were not of any node's initializers in the graph. Because they were not part of any initializers, these implicit inputs couldn't be visited by going through all nodes' inputs. Consequently, they would *not* be added into graph_inputs_excluding_initializers_. We fixed the issue by first copying the populated graph_inputs_including_initializers_ into graph_inputs_excluding_initalizers_, which then had both initializers and non-initializers as its initial content. Later, we erase initializers from the list. In this way, we can ensure all implicit inputs to remain in graph_inputs_excluding_initializers_. * refined comments and fixed duplicates Address CR by revisiting comments in terms of implicit inputs Also fixed an issue by skipping duplicates while copying inputs from graph_inputs_including_initializers_. * address CR explain why we need to collect nodes' implicit inputs * don't rely on pointer values for iterating std::set Previously, openvino relied on iterating a set of NodeArg pointers to construct inputs and outputs for a fused graph. It could cause non-determinism. The reason was that although iterating std::set by itself is stable, pointer values of NodeArgs may vary. Consequently, we could end up visiting the set's elements in different orders for different runs for the same test, which resulted in constructing inputs (and outputs) with different orders to the fused graph. For example, for the same test, we may have inputs [A, B] in some runs but inputs[B, A] in others. Let's use std::string as the key type to avoid such nondeterminism. This commit also added implicit inputs into meta->inputs while returning the capability from the openvino provider. * Fixed another latent issue in openvino's GetCapability function The issue was that we couldn't simply erase fused_inputs and fused_outputs while iterating the nodes. For example, an output NodeArg may have multiple uses, and it's wrong if we erase it from fused_outputs when we encounter only one of its uses as input. * Remove DeviceAllocatorRegistry class (#2451) Remove DeviceAllocatorRegistry class * CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library. * Parallel Gelu with ParallelFor (#2399) Parallel Gelu to get better performance for Gelu * Clean up build.py (#2446) * Pull the latest image before running docker build * Fuse SkipLayerNorm with Bias (#2453) Fuse SkipLayerNorm with Bias * Allow more than one invocation of CreateEnv in the same process. (#2467) * Allow more than one invocation of CreateEnv in the same process. * Fix centos build * Symbolic shape inference improvements: (#2460) * Symbolic shape inference improvements: - add a mode to guess unknown ops' output rank - add support for GatherND - add support for If - fix a bug in get_int_values when then tensor rank > 1D, by treating it as no sympy data - add symbol to literal merge when ONNX silently merges dims - fix a bug in Concat when input dim is 0 - fix a bug in ConstantOfShape that computed dim is not updated - add support for dynamic shape in ConstantOfShape - fix a bug in Loop output shape that loop iterator dim is not inserted at dim 0 - add support for dynamic padding in Pad - add support for dynamic shape in Reshape - add support for Resize with opset > 10, by treating output dims as dynamic - fix a bug in Slice when starts/ends are dynamic - restrict input model to opset 7 and above - make output model optional to avoid disk write when testing Run model tests for symbolic shape inference Reduce 2GB docker image size of nuphar * add additional test data set for nuget pipeline (#2448) * add SAS token to download internal test data for nuget pipeline * update azure endpoint * fix keyvault download step * fix variable declaration for secret group * fix indentation * fix yaml syntax for variables * fix setting secrets for script * fix env synctax * Fix macos pipeline * attempt to add secrets to windows download data * fix mac and win data download * fix windows data download * update test data set url and location * Revert "Brianma/windowsai fi (#2475)" This reverts commit 5780b864a15513fda4eadbfc2b5345fefe70b5ec. * Add scenario tests (#2457) * Add scenario tests * Remove TODO from model license * Add winml_api test dependency * fix model load test. fi from master changed the constructor (#2483) * make api tests all pass (#2486) * fix bad merge * fix bad model merge * Layer dev paulm (#2492) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * Rename ambiguous header (#2489) * fix one more missing IR version model (#2500) * add missing IR version to 4 more models used by scenario tests (#2501) * Add CLI parameters to test runner, build WinML in ARM and x86 CI (#2479) * Support test parameters through CLI arguments * Add WinML do Windows x86/ARM CI builds * Code style fixes * Update googletest Remove GPUTEST macros everywhere now that GTEST_SKIP is supported * Refactor main.cpp * Build scenario tests without DML * Link scenario tests to DML when it's enabled (#2502) * Layer dev release pipeline (#2488) Adds winml binaries to existing cpu nuget package, and creates new gpu dml nuget package with winml binaries and DML EP. * Layer dev paulm (#2506) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * Remove usage of IOBinding in WinML and use C_API Run method (#2504) * remove usage of iobinding * Change data structure to use vector of Ort::Values * Polish bind input / output * Use C APIrun method * Update providers on evaluate getresults * Remove run and IObinding interface from WinMLAdapter * Remove use of IObinding * bind unbound outputs code moved to learningmodelbinding * clean up unneeded istensor adapter function * Fix comment * Check if session is closed before binding and clearing * PR feedback * Layer dev paulm (#2507) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * Make tests dependend on winml_dll (#2509) * add dml binaries to DirectML package and be more explicit about condition variables (#2520) * re-enable warnings for winml builds and fix the warnings that were hiding (#2526) * turn devmode back on for winml builds * fix some warnings. include protobuf in a way that disables some warnings * undo protobufhelpers changes and just ignore 4100 errors in pb code * attempt to isolate protobufhelpers errors * add template specialization for getting tensor proto data * Layer dev paulm (#2533) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * Spawn child process to run DeviceLostRecovery scenario test (#2530) * Spawn child process to run DeviceLostRecovery scenario test * Layer dev paulm (#2536) ori said yes * add missing namespace to winml_trace_logging_provider in lotusenvironment.h (#2542) * Handle exception thrown from all apis in WinMLAdapter (#2539) * various changes to unblock windowsai ADO build * Fix custom ops scenario tests (#2562) * Do not shutdown protobuf after ort environment gets destroyed. Lazy load lotus environment first time it is needed * comment typo * pr comment about calling phoenix singleton * Make lotus_environment static in winmladapter * Layer dev paulm (#2567) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * making windowsAI azure dev ops work. * code review comments. * revert changes * Cmake and preprocessor fixes that where uncovered by building on agents without DML available via SDK * Layer dev dml delayload (#2580) * Brianma/cpu (#2583) * don't include dml stuff in cpu builds * tests that link the image lib also need the telemetry lib now * Throw Winml_err_invalid_binding if binding gpu resource on cpu device (#2589) * Throw Winml_err_invalid_binding if binding gpu resource on cpu device * PR comments. No need to query executionprovider for is gpu device * User/xianz/ortthrow (#2596) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * User/xianz/ortthrow (#2599) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * add status error message * Remove uwp onsuspending winrt call because logruntimeperf is getting removed (#2630) * User/xianz/dedup telemetry (#2631) * investigate duplication of telemetry in winml and ort * remove winml telemetry events * telemetry executionProviderEvent * remove unneccessary file and refactor code little bit * Revert back TelemetryEvent, which send up ETW event. * merge changes from layer_dev to windowsai (#2638) * Remove underscore from googletest names (#2616) * Fix leaking memory allocator Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24278761 and https://microsoft.visualstudio.com/OS/_workitems/edit/24330198 * Explicitly initialize Ort::Value with nullptr * Cache WinML adapter * bad merge * define private version of dxcore enum that is added in 19H1 SDK. (#2654) * add comment for explaning private definition of dxcore d3d feature level ennum value. (#2672) * do not package directml.pdb for redist packages. (#2676) * Fix leaking operator registry (#2645) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24354916 * User/orilevari/windowsai master merge (#2674) merge resolutions included pulling in telemetry logic that was merged to master and not windowsai and dereferencing InferenceSession::sessionstate now that it is a unique pointer * Delete Ort Allocator in LearningModelBinding (#2653) * Delete OrtAllocator in LearningModelBinding * PR comments to make Ort::Allocator a smart pointer * Small comment change * PR feedback to clean up code * PR feedback on move semantics * Clean up std::move * Fix memory leaks (#2679) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24356109, https://microsoft.visualstudio.com/OS/_workitems/edit/24388361 and https://microsoft.visualstudio.com/OS/_workitems/edit/24388596 * various changes to properly organize and skip GPU tests. For now for No DML builds we will not run GPU tests at all. In the future we should adapt the tests to expect the appropiate errors. (#2695) * Windowsai without fi (#2701) * Disable Attention fusion tests when DISABLE_CONTRIB_OPS is defined (#2529) * Setup java ci (#2528) * Add provision in ORT for session options to be parsed when available via model file (#2449) * Initial commit * Fix gitmodules * Nits * Nits * Updates * Update * More changes * Updates * Update * Some updates * More changes * Update * Update * Merge * Update * Updates * More changes * Update * Fix nits * Updates * Fix warning * Fix build * Add comment * PR feedback * PR feedback * Updates * Updates * Update * More changes * Fix build break * Comment test for now * Updates * Updates * PR feedback * Updates * Nits * Add tests * Fix build * Fix build * Fix build * Fix build break * Fix build * Nits * PR feedback * More change * Expose GetSessionOptions in pybind logic and add unit test for python * Fix build * PR feedback * PR feedback * Revert "Disable thread pool creation when enabled OpenMP (#2485)" (#2535) This reverts commit 7c7d5a149c9ed52eec67304bae5c4b132166a8a1. * Add dynamic shape support in TensorRT execution provider (#2450) * remove onnx-tensorrt submodule * add new onnx-tensorrt submodule (experiment) for trt6 * update engine build for trt6 * update compile and compute for tensorrt6.0 * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * switch to onnx-tensorrt master for TensorRT6' * Update tensorrt_execution_provider.cc * Handle dynamic batch size and add memcpy in TensorRT EP * update test cases * Update tensorrt_execution_provider.cc * update onnx-tensorrt submodule * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.ubuntu_tensorrt * Update run_dockerbuild.sh * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update concat_op_test.cc * Update tensorrt_execution_provider.cc * Upgrade TensorRT to version 6.0.1.5 * Update onnxruntime_providers.cmake * Update CMakeLists.txt * Update reduction_ops_test.cc * Update install_ubuntu.sh * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.tensorrt * Update BUILD.md * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update onnxruntime_providers.cmake * Update install_ubuntu.sh * Update install_ubuntu.sh * Update gemm_test.cc * Update gather_op_test.cc * Update CMakeLists.txt * Removed submodule * update onnx-tensorrt submodule * update header file * Removed submodule * add submodule onnx-tensorrt kevin's branch shape-test' * add debugging code * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * merge master * Removed submodule * update onnx-tensorrt submodule * add more changes for dynamic shapes * Update tensorrt_execution_provider.cc * update for dynamic shape * update dynamic shape processing * fix logger issue * remove submodule onnx-tensorrt * add submodule onnx-tensorrt * add env variable min_subgraph_size * remove redundency * update document * use onnxruntime::make_unique * fix multi-run issue * remove some tests to save CI build time * Add dynamic shape test * Update TensorRT-ExecutionProvider.md * Add example of running Faster R-CNN model on TensorRT EP * Add more details on env variables * update environment variables * Update tensorrt_basic_test.cc * Update model tests * Update tensor_op_test.cc * remove --use_full_protobuf * Update build.py * User/xianz/telemetry (#2458) * enabme telemetry * enable telemetry * set enable telemetry as default * for debugging * remove log and set disable telemetry as default back * delete private file while testing * resolve comment: mainly add license header, rename macro and update docs * rewording in privacy.md * Fix integer overflow in cuda NonMaxSuppression implementation (#2540) * add test case that should pass but fail * fix nms * extract int_max_output_boxes_per_class * Introduce container type runtime checks and other improvements (#2522) Rework TensorSeq in a manner consistent with Tensor and SparseTensor in terms of type system setup. Reduce templating. Introduce helpers to ensure the same data type. Make OrtValue __dtor not virtual. Introduce ContainerChecker * Fix C API tests for centos and mac (#2544) * change c++14 to c++11 * add ld lib path for centos * enable csharp tests on macos * fix C API test on MacOS + fix manylinux dotnet install * fix manylinux dotnet install * fix lib link * Add back executable bit to build.py * Fix a bug handling negative begin pad values in Pad op (#2550) * Fix bug in Pad op * Update * DNNL CMAKE update (#2548) * Fix android build (#2558) * Update win-x86-ci.yml (#2557) Fix build pipeline break * Re-enable Windows C# tests (#2564) * disable onnx_test_runner -x invocations for dnnl (#2568) * Allow sequence length to be symbolic (#2559) * setup java ci mac (#2570) * make layernorm fusion to support opset 11 (#2545) * Fix a warning found in the latest VS release * Add more check on SkipLayerNorm and BiasGelu fusion (#2574) * Fix file not found error during docker build. (#2569) * Add ConvTranspose1D (#2578) * Ryanunderhill/packagename test (#2582) * [Nuphar EP] fixes for some object detection models (#2581) Update notebook tutorial with multi-threaded int8 GEMM from #2517 * EmbedLayerNormalization Fusion Improvement (#2553) Embedding layer norm fusion improvements - add more checks * Update version (#2584) * Temporarily exclude vgg19 test from Python backend test 1. temporarily exclude vgg19 test which comsumes too much memory, run out of memory on Upsquared device. Single test pass for vgg19, need furture investigation (#2588) 2. Update docker file to decrease the docker image size * Update docs for Android NNAPI EP (#2586) * Fix lto bug for protobuf and ubuntu * add path to build dir before test run (#2590) * Add missig env variables for mac pipeline test (#2595) * Fixed an issue in updating realized dims (#2597) when we update realized dims for scan's output, the sliced axis also needs to be inclusive, i.e. we should check with "dim >= insert_inclusive_axis", because the offset in the symbols are based on Scan sugraph. Otherwise, we would end up with shape mismatch later. * Java API for onnxruntime (#2215) * Add support for opset 11 in reshape fusion (#2592) Support opset verion 11 in reshape fusion * Rename automl python tools folder to featurizer_ops. (#2593) * Support opset 11 subgraph of Squad model in Embed Layer Normalization (#2605) Support opset 11 Squad model that is exported from PyTorch nightly. The embed layer uses Range op which is missed in the transformer. * symbolic shape inference: fix warnings in GPT-2 model (#2608) And revise nuphar perf test on BERT squad * Dump subgraph ID and fused graph ID (#2607) * Dump subgraph ID and fused graph ID Dump subgraph ID and fused graph ID for better debugging * Remove local static fused_count added a field global_fused_count_ to NupharExecutionProvider class * EmbedLayerNormalization Fusion For Dynamic Squad Model Opset 10 (#2613) Support subgraph of SQuAD model exported from pytorch with dynamic input axes * Allow providers to be set for InferenceSession at construction (#2606) * Remove unnecessary parameter in some places in GatherElements implementation (#2612) * Remove unnecessary parameter in some places * Update * Update * Make sure fenced tensor could not reuse other tensor. (#2561) Fix random error caused by this. * Improve Embed Layer Norm Fusion for SQuAD with static input shape (#2621) * fix float16 comparison in initializer (#2629) * epsilon attribute for layernormalization fusion (#2639) * removed unnecessary batch file and fix path (#2640) * Add shape inference to ConvTransposeWithDynamicPads schema (#2632) * Improve cuda expand() opeator's performance. (#2624) * Cuda pad optimize when no padding is needed. (#2625) * Shortcut cuda Pad() when no padding is needed. * Optimize cuda scatter() on 2D compatible. (#2628) * Optimize cuda scatter() on 2D compatible. * Add some comments. * fix build error for ARM (#2648) * Improve performance of resize() in Nearest mode (#2626) Special treatment for 2D, check same size as input image. And in 2d kernel, template use_expolation. * Fix memory exception in Layer Norm Fusion (#2644) * Windows CI changes(#2650) * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Revert "Windowsai without fi (#2701)" This reverts commit 285d4c85ff5c4e265f963208170304ef3461e684. * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Deref unique pointer for session_state * send shutdown event when dll is unloaded and EvaluationStop, SessionC… (#2704) * send shutdown event when dll is unloaded and EvaluationStop, SessionCreationStart Events. * Add EvalutationStart Event * add comment * use correct type for for loop (#2755) * ARM CI (#2759) * Set ARM agent pool * Set CMake generator to VS 2019 in ARM * Use system-wide CMake instead of custom version Our custom version is too old for VS 2019 * Use DML and build shared lib in ARM CI * Restore nuget packages in ARM CI * Disable DML * Refactor ARM debug/release builds * Use system packaged Python version * Remove hardcoded Python path * Downgrade Python to 3.7 for build * Remove explicit CMake path * Fix invalid JSON in cgmanifest.json (#2760) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * Remove usage of WHOLEARCHIVE in WinML CMake and add WinMLAdapterFactory (#2726) * Remove usage of WHOLEARCHIVE in WinMLAdapter CMake and add WinMLAdapterFactory * PR feedback, no need for dll(export) since using def file * PR comments * Small comment in gen_def.py * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * Move winml_provider_factory.h to proper location (#2801) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Filter CPU case for IsFloat16Supported (#2802) * Merge fixes * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * Fix warnings that cause build to fail * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * Make DML include path global (#2882) * Make DML include path global * Add generated cppwinrt headers to winml_lib_common * Integrate changes to WindowsAI to make ADO Build (#2886) * Revert "CMake cross-generator fixes (#2790)" This reverts commit dbe7d97fa1ab155f1309bced87199527e8f35bd2. * add additional suppress warning in onnx_proto * ignore /wd4996 warning * DML execution provider fixes * Revert "Revert "CMake cross-generator fixes (#2790)"" This reverts commit 1ae7b4bcbc02edc881ad28685da98e095dfceb17. * Update func signature of custom op function overloads * common devicehelpers fixes * Add pch.h for winml_lib_common * re-add winml_lib_common_dir/inc to include path for winml_adapter * User/orilevari/dml redist shared folder (#2890) * move dml nuget package directory up one level to make it shared between build flavors * Merge conflict fix * Revert "Merge conflict fix" This reverts commit 142fa72cf9ce4344ad717b50b7ea2b8582aadc7c. * Revert "Merge remote-tracking branch 'origin/master' into windowsai" This reverts commit 6e2126d46e5e5f564d65da37dd4f70c93dd81165, reversing changes made to b3f5583dc9249834b947c8ea905f6a98060d5bd6. * Make winml_test_common free of test macros (#2902) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Don't generate files outside the build root (#2914) * Don't generate files outside the build root * Add onnxruntime_EXTERNAL_DEPENDENCIES to WinML * Add DML depedency on RESTORE_PACKAGES * User/orilevari/fix yaml merge bugs (#2918) * Add winml test source parameter into cmake function (#2919) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Add arguments to unittest cmake functions * remove comment * Revert "Revert "Merge remote-tracking branch 'origin/master' into windowsai"" This reverts commit ade5abe72a4234fdbc3623093c61c02c6b0bdc26. * Fix breaks from merge with ORT master * Brianma/linux (#2917) * don't include windows.h in cross-plat header * add default case for switch statement * signed/unsigned mismatch fix Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> * User/sheilk/winml adapter c api (#2891) * Create winml adapter c api * fix build * make it build * move adapter into onnxruntime core/session * entry point not exported * minor changes * make model metadata work * make tests pass * implement all the model reflection apis on the adapter c abi * update the new ort interface to create a lotus ennvironment with a logging sink * start adding ort env * move all winml code into adapter folder/lib to isolate it * ensure a single logging manager at a time * start refactoring session * refactor session creation interface * add cpu and dml session option methods to adapter * finish session init * stub out interfaces in ort lib to perform similar mechanics of iinference session * enable profiling, and enable schema override * update session register graph transformers * turn back on custom registry for custom ops * Add sync api * add last c api stubs * should build... but all feature values are broken since this is in flight to moving all implementation details into ivalue * remove ep adapter header * Implement DML execution provider functions from adapter (#2846) * Implement DML execution provider functions from adapter * Use functions in OnnxruntimeEngine.cpp * make map/sequence type_infos freeable, and start implementing ivalue * make it build again * implement value methods * implement remaining methods * remove com adapter abi * check dml session * cache the allocator on ivalue * check if resource is cpu/gpu when access its mutable data * update tensor * mismatched parentheses * fix tensor base and binding obj * it evaluates tensors! sometimes... * minor fixes * enable gpu evals * wrapper all existing winml adapter apis with API_IMPL to try catch (#2854) * update winml... tensor strings are broken, need to template tensorbase to do different things for strings * make tensor strings work with 2 copies in/2 copies out * Fix tensor string and allocator bug * make maps work again... needs some fixes still * Make it build! * enable map inputs * map outputs * unbound outputs for sequences and maps * User/xianz/merge windowsai (#2883) * Packaging pipeline changes for VS 2019 (#2711) * Tiny fix to codegen * Simplify cache implementation and avoid static variables that may carry over between models * Extend DML kernels (#2641) * Additional DML operators * Check unsupported attributes and inputs * Address PR comments * Add kernel capability function used for partitioning, and re-enable stride-based int64 support based on value range * Fix test failures * Build fix * PR comments * Update Nuphar tutorial notebook (#2721) 1. Reflect int8 GEMV improvements for multi-threading from #2696 2. Add notes on multi-threading control using OpenMP 3. Add samples of running multi-isa AOT, and show int8 GEMM differences between AVX and AVX2 4. Add rnn_benchmark example to resolve #1993 * Add schema for new Qops (#2611) * Add schema for new Qops * adding shape inference + qlinearaveragepool * plus review comments * plus review comments * updates per review comments * plus review comments * [server] Add supposed for model_name and model_version as cli parameter (#2708) * remove 64bit warning message from python validation. (#2727) * MLAS: ARM64 build fix (#2734) fix bad usage of vreinterpret to cast vector element types * Fix broken python docs links (#2740) * Fix build on Mac OS (#2731) mac os ld doesn't support --while-archive, correct option is -all_load * fix ngraph wheel (#2737) * fix ngraph wheel 1.1.0 onnxruntime_ngraph wheel doesn't work * remove libdnnl.so in nGraph Libs * make it easy to compare * Split onnxruntime server to a separated folder (#2744) * Fix build for Python 3.8 (#2747) * Fix build for Python 3.8 * Update protobuf to 3.11.2 (#1928) Update protobuf to 3.11.2 (#1928) * Change default optimization level to All (from Basic) (#2745) * change default optimization level to All (from Basic) * fix test * fix c# test * Update numpy to 1.18 (#2758) * Update numpy to 1.18 * Pipeline changes for python 3.8 (#2753) 1. Pipeline changes for python 3.8 2. Fix a regression in setup.py which was just introduced in the previous commit. Please notice, we still haven't made python 3.8 + Windows + CUDA work. * Add basic stacktrace output for posix debug builds. (#2749) * [NupharEP] fix a race condition when multiple sessions running different models concurrently (#2772) * Revert "Change default optimization level to All (from Basic) (#2745)" This reverts commit 56bb503c2f26474b6613bcb2a198691a11dcef38. * Fix typo in error message (#2736) * Rename MKL-DNN to DNNL to fix broken link (#2730) * Fix nightly build version number issue * Pass BUILD_BUILDNUMBER to linux docker * Disable featurizers in python packages * Import more featurizers (#2781) Make kernels non-template. Add input constraint for learnt data. Add min_max_scalar_transformer, robust_scalar_transformer, inputation_marker_transfomer, label_encoder_transformer, missing_dummies_transformer along with tests. Advance Featurizers library commit. * Implement a more stable softmax (#2715) * Implement a more stable SoftMax e^x is represented as infinity if x is large enough, like 100.f. Infinity divided by Infinity is a NAN. Thus, softmax gets a NAN if one or more item are large enough. A math transform as below is leveraged to get a stable softmax: e^xi/(e^x1 + ...e^xn) = e^(xi - max) / (e^(x1 - max) + ... + e^(xn - max)) And for convenience, force max to 0.f if all xi are negative * Contributing: Fix a typo (#2784) * ACL EP GEMM improvements (#2780) When it is posible we use a fully connected layer instead of the gemm implementation. This will let the library use the best implementation based on the input data. * ACL EP convolution improvements (#2774) Added the optimized implementation for depthwise convolution for both ACL v19.02 and ACL 19.05. Also the pointwise convolution seems to be more optimal in the CPU implementation so we opted for that instead. * Add script for release Nuget validation (#2719) * Initial commit * Nits * Disable a test temporarily * Change working directory * Test * Add download python step * Test update * More changes * Fix space issue * Fix * Verify nuget signing * Fix * Spaces * PR feedback * Nit * Fix * Fix * Remove temporary changes * add uint8 support to where op (#2792) * Improve bert optimization script: (#2712) (1) Move input int64=>int32 conversion to embed layer fusion. (2) Output epsilon attribute for LayerNormalization fusion. * add session creation time cost. (#2798) * ML.NET team needs featurizers within a package (#2789) Add auto ml featurizers to Windows, MacOS as well as to GPU packaging-pipelines. * Initialize max of softmax with lowest of float (#2786) * MLAS: update SGEMM threading parameters (#2808) * add interface to copy batch tensors. (#2807) * add interface to copy batch tensors. * onnxruntime * speed up Windows TRT CI (#2811) * don't run cuda tests if building with tensorrt * remove unnecessary build options for win trt ci * refactor win gpu tensorrt ci yml * --numpy_version=1.17 * update * update * azcopy and cuda path * Update test data (#2356) * Add timeseries imputer transformer featurizer kernel (#2813) Make kernels non-template. Add input constraint for learnt data. Fixup tests. Add two more featurizers along with tests. Tests fail. min_max_scalar_transformer robust_scalar_transformer Fix tests serialized stream by prepending version bytes. Add inputation_marker_transfomer and the test. Fix up float/double type designations. Added label_encoder_transformer along with a test. string_throw case is broken at the momement. Fix labelencodertransfomer_test.cc string_throw case Rename maxabsscalertransformer_test.cc Add MissingDummiesTransformer along with the test. Update manifest. Add TimeSeriesImputerTransformer definition, implementation and tests * Fix memory leak in TRT (#2815) * fix memory leak issue * revert EP_FAIL on enueueV2 * Add manifest missing comma * Run static code analyzer on most of our code (#2817) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * update quantization doc (#2783) * update documentation for quantization script * plus some spell corrections * Filter CPU case for IsFloat16Supported (#2802) * update default optimization level + fix gemm_activation fusion (#2791) * update defualt optimization level + fix gemm_activation fusion * fix typo * add unit test and incorporate review comments * fix test comment * Fix dnnl wheel package name (#2823) * Append '-dnnl' to whl package name when --use_dnnl * Update build.py * Update Ubuntu & TensorRT version in README (#2820) Dockerfile.tensorrt is using nvcr.io/nvidia/tensorrt:19.09-py3 as base Image, update Ubuntu and TensorRT version according to https://docs.nvidia.com/deeplearning/sdk/tensorrt-container-release-notes/rel_19-09.html#rel_19-09 * Merge fixes * Add OneHotEncoder and HashOneHotEncoder kernels. (#2830) Add defs and imlementation for OneHotEncoders, adjuist date_time_transformer kernel and test. Add OneHotEncoder kernel test. Add HashOneHotVectorizerTransformer unit test. This does not link due to multiple definitions of functions that are included into header from a CPP file. * Upgrade gtest to the latest version (#2827) WinML would like to update the googletest submodule. They want some newer features (namely GTEST_SKIP to skip tests programmatically and be able to skip entire fixtures easily) and would need to update the submodule version. However, because the new version of code hit a bug in gcc, even though the bug is already fixed in the latest gcc but we're using gcc 4.8.x and it won't get patched for the bug, so we have to do a compromise, change our code a little bit to make it work. The gcc bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=51213 * Add support for int64_t for topk CPU. Fixes github issue #2806. (#2833) * Ignore allocator type in ExecutionProviders allocator map. Make default initialization of OrtMemoryInfo more clearly invalid. (#2768) * Remove allocator type from the key comparison in ExecutionProviders. Remove usage of DummyArena as it's no longer necessary. * Fix x86 tests where arena allocator is disabled. Make initialization of OrtMemoryInfo clearer by adding Invalid enum value. * Make OrtValueNameIdxMap::MaxIdx more intuitive. * Convert ExternalProject Featurizers into git submodule (#2834) Add git submodule for Featurizer library. Update cmake to build for git submodule. * add domain check for nodes + update documentation (#2831) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Fix Linux CUDA nuget packaging pipeline break * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * update optimization doc for BERT related fusions (#2819) * Add bert related transformers to doc * Add execution provider and comment for bert optimizations * Add comment about accuracy impact of approximation * Fix warnings that cause build to fail * MLAS: enable threading for quantized GEMMs (#2844) * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * remove old winmladapter cpp Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * move sequence implementation into ort lib... still commented out... need to turn back on... * begin sequence implementation * make maps and sequences work * fix broken tests * remove dead code * misc cleanup * CR feedback * User/xianz/winml adapter c api (#2869) * wrapper all existing winml adapter apis with API_IMPL to try catch * Return HR or Throw for WinML adapter APIs if failed * undo macro wrapper for two places * Wrap error macros around ort apis, too. * address CR feedback #2 * add more api throw/return macros * Revert changes no longer needed * revert changes to cxx api * format winml lib.ort and winml adapter * remove static pheonix singleton Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * missing use_dml check in winml_adapter_session (#2930) * --use_dnnl flag was mangled in merge (#2931) * use dml macro not wrapping custom registry code (#2934) * Disable LNK4199 winml_dll to enable cuda builds (#2936) * Disable LNK4199 in winml_dll * linkler->linker * LearningModelSessionAPITestGpu.CreateSessionWithCastToFloat16InModel should return DXGI_ERROR_UNSUPPORTED when FP16 not supported (#2937) * Disable LNK4199 in winml_dll * linkler->linker * Need to return DXGI_ERROR_UNSUPPORTED when Model does not support fp16 * Publish build symbols (#2939) * Publish build symbols * Don't upload PDBs for .exe files * Make x86 build (#2943) * fix last remaining size_t/int64_t warnings->errors (#2948) * TensorString, Sequences and Maps use the first allocator, but should use the cpu default allocator. (#2952) * fix tensor string allcoator * clean up default allocator usage for strings in winml lib/api.ort Co-authored-by: Ryan Lai <ryalai96@gmail.com> * Handle tensor shape of zero (#2954) Co-authored-by: Ryan Lai <ryalai96@gmail.com> * CR feedback (#2970) * CR feedback * fix weird formatting on privacy readme * Add 'All rights reserved.' everywhere * readd all rights reserved to winml_provider_factory.h * remove extra space in comment * remove extra whitespace * fixes post master merge * remove winml from nuget gpu pipeline * set IR VERSION on generated_model in rnn_benchmark (#2972) * Fix slice conformance failures (#2908) Co-authored-by: Adrian Tsai <adtsai@microsoft.com> Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Paul McDaniel <paul_mcdaniel@hotmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Dwayne Robinson <fdwr@hotmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com>
2020-02-05 01:12:19 +00:00
[contract(MachineLearningContract, 1)]
interface ILearningModelFeatureDescriptor
{
//! \brief The name you use to bind values to this feature.
//! This property is required and will always be there. All features have a name as
//! primary key for the model. Usually as a single word. You use this name when
//! enumerating the features of the model and then later when binding a value to one
//! those feature using a LearningModelBinding. It will be unique across all features.
String Name{ get; };
//! \brief A description of what this feature is used for in the model
//! This property is optional. If provided by an author model it will be a description
//! of what the feature is for the model.
String Description{ get; };
//! \brief The kind of feature - use this to know which derived class to use.
LearningModelFeatureKind Kind{ get; };
//! \brief If true, you must bind a value to this feature before calling Evalaute().
Boolean IsRequired{ get; };
}
INBOX_ONLY([uuid(2a222e5d-afb1-47ed-bfad-b5b3a459ec04)])
OTB_ONLY([uuid(ae066239-6b19-4509-be3e-502ba40203b3)])
Merge windowsai (winml layering) into master (#2956) * Initial Commit * Merged PR 3985217: add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc (#2346) add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc and violating our OS layering requirements. We linked against onecoreuap_apiset.lib in VB so we will continue doing this, but I am still unsure why not to link against onecore instead since that is where we ship. However, since Sheil is the owner of this code we will wait to discuss with him before changing anything. * Initial changes for layering * more snipping to get core into ort * update build instructions to include --build_shared_lib (#2358) * update build instructions to include --build_shared_lib * fix line breaks * Task 23998197: add winml_lib_core into onnnxruntime.dll (#2368) * Task 23998197: add winml_lib_core into onnnxruntime.dll * PR feedback build break on perf_test * return proper error when the model path isn't found (#2391) * LearningModelSession is cleaned up to use the adapter, and parts of b… (#2382) this is a big PR. we are going to move it up to layer_dev , which is still a L3 so we are still safe to do work there agile. we are going to move this into the L3 so that ryan can start doing intergration testing. we will pause for a full code review and integration test result prior to going into the L2. >>>> raw comments from previous commits >>> * LearningModelSession is cleaned up to use the adapter, and parts of binding are. * moved everything in the winmladapter made it all nano-com using, WRL to construct objects in the ORT side. base interfaces for everythign for winml to call cleaned up a bunch of winml to use the base interfaces. * more pieces * GetData across the abi. * renamed some namepsace cleaned up OrtValue cleaned up Tensor cleaned up custom ops. everything *but* learnignmodel should be clean * make sure it's building. winml.dll is still a monolith. * model moved over. everything builds clean. step ! * weak ref comment * Layer dev paulm (#2408) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * Layer dev paulm (#2414) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * Layer dev paulm (#2423) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * Layer dev paulm (#2424) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * Layer dev paulm (#2425) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Layer dev paulm (#2426) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * fixed map and sequence when passing stl types across the ABI . found a leak in nvidia driver, but skipped it. all winmlapitests pass now * Moved SessionOptions over to the abi * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * Register WinML TraceLogging provider on Onnxruntime.dll (#2455) * Register WinML TraceLogging provider on Onnxruntime.dll * Add ifdef to make sure trace logging provider has telemetry option when LAYERING_DONE * No need for ifdef for TraceLoggingOptionMicrosoftTelemetry * PR feedback * Move etw registration into lotus environment constructor and deresgister in lotus environment destructor * Brianma/cpuwinml (#2466) * allow building winml cpu without dml. * Brianma/breaks (#2469) * fix some more breaks * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * move dml checks out of winml and into the adapter * better error handling * Brianma/fi (#2470) * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * fix bad merge * Checking in a staging checkpoint point so that Ryan can work with me in parrallel * build break. * Brianma/testfails (#2473) * add missing ir version to dictvectorizer-string.onnx * add missing ir version to relu.onnx * add missing ir version to zipmap*onnx * add IR version to manually generated models * remove an unnecessary ifdef dml * Brianma/windowsai fi (#2475) * update dockerfiles/README (#2336) * Make elementwise op run 4 items per thread (#2335) Description: Describe your changes. Make elementwise op run 4 items per thread unroll for loop to leverage ILP remove unnessary N==0 check inside elementwise GPU kernel Motivation and Context Why is this change required? What problem does it solve? It can improve the performance of GPU elementwise ops. ~2% performance gain on popular NLP bert model. If it fixes an open issue, please link to the issue here. * Add CUDA GatherElements kernel (#2310) * Updates * Update test * Update * Updates * nits * PR feedback * Update * Update * PR feedback * PR comments * Update * Fix build * Fix build * Nits * Fix * Layer Normalization Fusion (#2319) basic layer normalization transform * Add FastGelu Cuda Op for Gelu and Add bias fusion (#2293) * Add FastGelu cuda op * Add AddBiasGelu for experiment * Revert "Add AddBiasGelu for experiment" This reverts commit 5c1ee019858c657e6bb75887265cb85675626e5b. * Add bias * Add unit tests * update comment * update script * fix build error * update coding style * update for CR feedback Enable half2 optimization only when cuda arch >= 7.0 * move _Tanh to common.cuh * implement CPU contrib OP Attention (#2333) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanUnusedInitializers. (#2320) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanupUnusedInitializers. This means initializers that have been replaced during graph optimizations are not left in the GraphProto when we save an optimized model. * Handle edge case where a model has an unused initializer with matching graph input by also removing the graph input. * Use non-const iterators in std::find_if calls to make centos build happy. * Nuget pipeline changes (#2305) 1. refactor the pipeline, remove some duplicated code 2. Move Windows_py_GPU_Wheels job to Win-GPU-CUDA10. We'll deprecated the "Win-GPU" pool 3. Delete cpu-nocontribops-esrp-pipeline.yml and cpu-nocontribops-pipeline.yml 4. In Linux nuget jobs, run "make install" before creating the package. So that extra RPAH info will be removed * Cuda Reverse Sequence Op, maping types of same size using same template function. (#2281) * Set ElementType to String type of node metadata, instead of byte[] (#2348) * Set ElementType to String type of node metadata, instead of byte[] * Fix spacing * Introduce PrimitiveType into a Type System along with an integer constant (#2307) Improve perf by avoiding GetType<T>() calls. Introduce MLTypeCallDispatcher to switch on Input Type. Add Tensor IsType<T>() fast method. * Fix/test dim value of 0 handling in a couple of places (#2337) * Update the CUDA Where implementation broadcasting logic to handle a dim with value of 0. Add unit test Also add unit test for unary op with dim value of 0 * Exclude ngraph from Where test with 0 dim. * Openvino EP R3.1 onnxrt server (#2357) * onnxrt server with OVEP * onnxrt server with OVEP * Update Dockerfile.server.openvino * onnxrt server OVEP fix reviews * onnxrt server OVEP fix reviews * Implement cuda nonzero op. (#2056) Implement cuda nonzero op. * Direct use python numpy array's memory if already contiguous. (#2355) * Direct use python numpy array's memory if already contiguous. This could greatly improve performance for session with large input, like big image 1920x1080 fastrcnn, 30~40% speed up could be achieved. * Add test case enforce contiguous/non-contiguos numpy array as inputs. * Add helper to create output to minimize binary size. (#2365) Add ConstEigenTensorMap typedef so we don't unnecessarily const_cast the const input Tensor. * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS (#2369) * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS * update * Add Tracelogging for profiling (#1639) Enabled only if onnxruntime_ENABLE_INSTRUMENT is ON * test bidaf with nuphar for avx target (#2370) increase nuphar test coverage a bit * Fix a bug in TLS refcount that may destabilized CUDA CI (#2374) * update output size calculation for resize (#2366) * change how output size is calculated for resize op * add tests for ver 10 resize * Extend OneHot CPU kernel to support more types (#2311) * Extend OneHot CPU kernel to support input int64_t, depth int32_t, output float * Skip BERT before the test data fix is picked up * Fix bug with Slice. Need to pass in flattened input dimensions so the initial offset into the input is calculated correctly. (#2372) * Add opset 11 version of Split to CUDA ops (#2376) Organize the CUDA ops definitions so all the opset 10 and 11 parts are together (same setup used for CPU ops) * Layer Norm Fusion Fix (#2379) * layer norm fusion fix * Add input shape check in code and unit tests * Fuse Add + Gelu (#2360) Implement the transformer to fuse add + gelu Implement the accurate kernel * Skip layer norm transform (#2350) * skip layer normalization transformer * Another try to stabilize CUDA CI (#2383) The root cause seems to be failure in CUDA dealloc when tear down. cudaFree return code was ignored before, so should the debug check. * fix BUILD.md typo (#2375) build.py: error: argument --config: invalid choice: 'RelWithDebugInfo' (choose from 'Debug', 'MinSizeRel', 'Release', 'RelWithDebInfo') * Fixed compilation with ngraph (#2388) * Fix reuse logic in allocation planner. (#2393) * Fix reuse logic in allocation planner. * PR comments * Add helpful comments * Don't allow reuse across string tensors. * [NupharEP] Multiple optimizations (#2380) Fuse transpose into MatMul Implement Pow and constant scalar simplification Vectorize ReduceMean Improve symbolic shape inference Minor updates for better debugging in fused function name * Avoid using the default logger in the graph lib and optimizers (#2361) 1. Use the session logger if it is available. 2. Don't disable warning 4100 globally. We should fix the warnings instead of disabling it. * Change CUDA implementation of Transpose to support all fixed size tensor types (#2387) * Change CUDA implementation of Transpose to not use a typed kernel so we can support more types with minimum binary size. Add support for 8, 16, 32 and 64 bit types. Add unit tests. Add method so the implementation can be called directly (will be used by CUDA Scan very soon). * Disable TensorRT for MLFloat16 and int8 unit tests. * Address PR comment and add support for calling cublas implementation if type is mlfloat16. * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. (#2398) * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. * [NupharEP] force some low/zero cost ops to be inlined (#2409) * fix cross compile bug (#2415) * Minor optimization: if a node has already been placed, there's no need to find a kernel for it. (#2417) * Add Reshape Fusion (#2395) * Add reshape fusion * Add some comments * update comments * update comment format * update according to feedback * update for recent logger change * fix build error * (1) Support both input and output edges in find path in graphutils (2) Add a test case of only one constant initializer of Concat input. (3) Refactor ReshapeFusion class to allow add more subgraph fusion in the future. * fix error * (1) loose constraint on initializer: non constant is allowed for reshape fusion. (2) Change versions type to vector. (3) Add logging. (4) Return false when multiple output edges matched in FindPath. Add comments. * only allow one direction (input or output) in FindPath * [NupharEP] Update notebook and docker image (#2416) Add BERT squad in Nuphar tutorial Enhance speed comparsion readability * Fix the issue in matmul_add_fusion (#2407) Fix the issue in matmul_add_fusion If Muatmul + Add has shape [K] * [K, N], reset it to [1, K] * [K, N] will make the output shape to [1, N] will also requires a reshape on the output. Fix: just remove the shape reset to not fuse it. Add a negative test case for matmul+add fusion * feat(treeregressor): Update TreeEnsembleRegressor for type support (#2389) Updates the `TreeEnsembleRegressor` to allow for `double`, `float`, `int64`, and `int32` inputs to match the upstream specification. Signed-off-by: Nick Groszewski <nicholas.groszewski@capitalone.com> * onnxrt server documentation update (#2396) * Added support for Pad-2 operator in OpenVINO-EP (#2405) * Add CUDA If operator. (#2377) * Add CUDA If operator. Uses CPU operator for implementation. By adding a CUDA version the inputs/outputs (with the exception of the 'cond' input) stay on GPU, and no other logic is required to avoid a copy to CPU across the control flow node. * Improved documentation for onnxruntime::utils::SwapByteOrderCopy(), added precondition check. * Fix the type constraints on CUDA If operator to exclude strings. (#2431) * add Im2col<uint8_t> (#2438) * Adjust codegen vectorization width from target (#2439) * Adjust codegen vectorization width from target * Add CUDA Scan operator. (#2403) * Add Scan CUDA op. Uses CPU implementation for logic. Added some device specific functors for handling when data needs to be manipulated on a different device. Added ability to override the materialization logic in the OrtValue slicer so DML can plugin their handling. * Fix Windows GPU C API packaging pipeline failure (#2440) Fix Windows GPU C API packaging pipeline failure (#2440) * Correctly handle implicit inputs for fused nodes (#2390) * Correctly handle implicit inputs for fused nodes Previously, nuphar's partitioning function didn't include node's implicit inputs into the inputs list of MetaDef, and hence a crash was triggered in the onnx graph checker. This commit fixed the issue. Furthermore, it also fixed a related issue where we didn't add implicit inputs into graph_inputs_excluding_initializers_ in Graph::SetGraphInputsOutputs. the issue was that graph_inputs_including_initializers_ populated by SetInputs (e.g. called by FunctionImpl::FunctionImpl) may contain implicit inputs which were not of any node's initializers in the graph. Because they were not part of any initializers, these implicit inputs couldn't be visited by going through all nodes' inputs. Consequently, they would *not* be added into graph_inputs_excluding_initializers_. We fixed the issue by first copying the populated graph_inputs_including_initializers_ into graph_inputs_excluding_initalizers_, which then had both initializers and non-initializers as its initial content. Later, we erase initializers from the list. In this way, we can ensure all implicit inputs to remain in graph_inputs_excluding_initializers_. * refined comments and fixed duplicates Address CR by revisiting comments in terms of implicit inputs Also fixed an issue by skipping duplicates while copying inputs from graph_inputs_including_initializers_. * address CR explain why we need to collect nodes' implicit inputs * don't rely on pointer values for iterating std::set Previously, openvino relied on iterating a set of NodeArg pointers to construct inputs and outputs for a fused graph. It could cause non-determinism. The reason was that although iterating std::set by itself is stable, pointer values of NodeArgs may vary. Consequently, we could end up visiting the set's elements in different orders for different runs for the same test, which resulted in constructing inputs (and outputs) with different orders to the fused graph. For example, for the same test, we may have inputs [A, B] in some runs but inputs[B, A] in others. Let's use std::string as the key type to avoid such nondeterminism. This commit also added implicit inputs into meta->inputs while returning the capability from the openvino provider. * Fixed another latent issue in openvino's GetCapability function The issue was that we couldn't simply erase fused_inputs and fused_outputs while iterating the nodes. For example, an output NodeArg may have multiple uses, and it's wrong if we erase it from fused_outputs when we encounter only one of its uses as input. * Remove DeviceAllocatorRegistry class (#2451) Remove DeviceAllocatorRegistry class * CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library. * Parallel Gelu with ParallelFor (#2399) Parallel Gelu to get better performance for Gelu * Clean up build.py (#2446) * Pull the latest image before running docker build * Fuse SkipLayerNorm with Bias (#2453) Fuse SkipLayerNorm with Bias * Allow more than one invocation of CreateEnv in the same process. (#2467) * Allow more than one invocation of CreateEnv in the same process. * Fix centos build * Symbolic shape inference improvements: (#2460) * Symbolic shape inference improvements: - add a mode to guess unknown ops' output rank - add support for GatherND - add support for If - fix a bug in get_int_values when then tensor rank > 1D, by treating it as no sympy data - add symbol to literal merge when ONNX silently merges dims - fix a bug in Concat when input dim is 0 - fix a bug in ConstantOfShape that computed dim is not updated - add support for dynamic shape in ConstantOfShape - fix a bug in Loop output shape that loop iterator dim is not inserted at dim 0 - add support for dynamic padding in Pad - add support for dynamic shape in Reshape - add support for Resize with opset > 10, by treating output dims as dynamic - fix a bug in Slice when starts/ends are dynamic - restrict input model to opset 7 and above - make output model optional to avoid disk write when testing Run model tests for symbolic shape inference Reduce 2GB docker image size of nuphar * add additional test data set for nuget pipeline (#2448) * add SAS token to download internal test data for nuget pipeline * update azure endpoint * fix keyvault download step * fix variable declaration for secret group * fix indentation * fix yaml syntax for variables * fix setting secrets for script * fix env synctax * Fix macos pipeline * attempt to add secrets to windows download data * fix mac and win data download * fix windows data download * update test data set url and location * Revert "Brianma/windowsai fi (#2475)" This reverts commit 5780b864a15513fda4eadbfc2b5345fefe70b5ec. * Add scenario tests (#2457) * Add scenario tests * Remove TODO from model license * Add winml_api test dependency * fix model load test. fi from master changed the constructor (#2483) * make api tests all pass (#2486) * fix bad merge * fix bad model merge * Layer dev paulm (#2492) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * Rename ambiguous header (#2489) * fix one more missing IR version model (#2500) * add missing IR version to 4 more models used by scenario tests (#2501) * Add CLI parameters to test runner, build WinML in ARM and x86 CI (#2479) * Support test parameters through CLI arguments * Add WinML do Windows x86/ARM CI builds * Code style fixes * Update googletest Remove GPUTEST macros everywhere now that GTEST_SKIP is supported * Refactor main.cpp * Build scenario tests without DML * Link scenario tests to DML when it's enabled (#2502) * Layer dev release pipeline (#2488) Adds winml binaries to existing cpu nuget package, and creates new gpu dml nuget package with winml binaries and DML EP. * Layer dev paulm (#2506) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * Remove usage of IOBinding in WinML and use C_API Run method (#2504) * remove usage of iobinding * Change data structure to use vector of Ort::Values * Polish bind input / output * Use C APIrun method * Update providers on evaluate getresults * Remove run and IObinding interface from WinMLAdapter * Remove use of IObinding * bind unbound outputs code moved to learningmodelbinding * clean up unneeded istensor adapter function * Fix comment * Check if session is closed before binding and clearing * PR feedback * Layer dev paulm (#2507) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * Make tests dependend on winml_dll (#2509) * add dml binaries to DirectML package and be more explicit about condition variables (#2520) * re-enable warnings for winml builds and fix the warnings that were hiding (#2526) * turn devmode back on for winml builds * fix some warnings. include protobuf in a way that disables some warnings * undo protobufhelpers changes and just ignore 4100 errors in pb code * attempt to isolate protobufhelpers errors * add template specialization for getting tensor proto data * Layer dev paulm (#2533) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * Spawn child process to run DeviceLostRecovery scenario test (#2530) * Spawn child process to run DeviceLostRecovery scenario test * Layer dev paulm (#2536) ori said yes * add missing namespace to winml_trace_logging_provider in lotusenvironment.h (#2542) * Handle exception thrown from all apis in WinMLAdapter (#2539) * various changes to unblock windowsai ADO build * Fix custom ops scenario tests (#2562) * Do not shutdown protobuf after ort environment gets destroyed. Lazy load lotus environment first time it is needed * comment typo * pr comment about calling phoenix singleton * Make lotus_environment static in winmladapter * Layer dev paulm (#2567) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * making windowsAI azure dev ops work. * code review comments. * revert changes * Cmake and preprocessor fixes that where uncovered by building on agents without DML available via SDK * Layer dev dml delayload (#2580) * Brianma/cpu (#2583) * don't include dml stuff in cpu builds * tests that link the image lib also need the telemetry lib now * Throw Winml_err_invalid_binding if binding gpu resource on cpu device (#2589) * Throw Winml_err_invalid_binding if binding gpu resource on cpu device * PR comments. No need to query executionprovider for is gpu device * User/xianz/ortthrow (#2596) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * User/xianz/ortthrow (#2599) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * add status error message * Remove uwp onsuspending winrt call because logruntimeperf is getting removed (#2630) * User/xianz/dedup telemetry (#2631) * investigate duplication of telemetry in winml and ort * remove winml telemetry events * telemetry executionProviderEvent * remove unneccessary file and refactor code little bit * Revert back TelemetryEvent, which send up ETW event. * merge changes from layer_dev to windowsai (#2638) * Remove underscore from googletest names (#2616) * Fix leaking memory allocator Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24278761 and https://microsoft.visualstudio.com/OS/_workitems/edit/24330198 * Explicitly initialize Ort::Value with nullptr * Cache WinML adapter * bad merge * define private version of dxcore enum that is added in 19H1 SDK. (#2654) * add comment for explaning private definition of dxcore d3d feature level ennum value. (#2672) * do not package directml.pdb for redist packages. (#2676) * Fix leaking operator registry (#2645) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24354916 * User/orilevari/windowsai master merge (#2674) merge resolutions included pulling in telemetry logic that was merged to master and not windowsai and dereferencing InferenceSession::sessionstate now that it is a unique pointer * Delete Ort Allocator in LearningModelBinding (#2653) * Delete OrtAllocator in LearningModelBinding * PR comments to make Ort::Allocator a smart pointer * Small comment change * PR feedback to clean up code * PR feedback on move semantics * Clean up std::move * Fix memory leaks (#2679) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24356109, https://microsoft.visualstudio.com/OS/_workitems/edit/24388361 and https://microsoft.visualstudio.com/OS/_workitems/edit/24388596 * various changes to properly organize and skip GPU tests. For now for No DML builds we will not run GPU tests at all. In the future we should adapt the tests to expect the appropiate errors. (#2695) * Windowsai without fi (#2701) * Disable Attention fusion tests when DISABLE_CONTRIB_OPS is defined (#2529) * Setup java ci (#2528) * Add provision in ORT for session options to be parsed when available via model file (#2449) * Initial commit * Fix gitmodules * Nits * Nits * Updates * Update * More changes * Updates * Update * Some updates * More changes * Update * Update * Merge * Update * Updates * More changes * Update * Fix nits * Updates * Fix warning * Fix build * Add comment * PR feedback * PR feedback * Updates * Updates * Update * More changes * Fix build break * Comment test for now * Updates * Updates * PR feedback * Updates * Nits * Add tests * Fix build * Fix build * Fix build * Fix build break * Fix build * Nits * PR feedback * More change * Expose GetSessionOptions in pybind logic and add unit test for python * Fix build * PR feedback * PR feedback * Revert "Disable thread pool creation when enabled OpenMP (#2485)" (#2535) This reverts commit 7c7d5a149c9ed52eec67304bae5c4b132166a8a1. * Add dynamic shape support in TensorRT execution provider (#2450) * remove onnx-tensorrt submodule * add new onnx-tensorrt submodule (experiment) for trt6 * update engine build for trt6 * update compile and compute for tensorrt6.0 * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * switch to onnx-tensorrt master for TensorRT6' * Update tensorrt_execution_provider.cc * Handle dynamic batch size and add memcpy in TensorRT EP * update test cases * Update tensorrt_execution_provider.cc * update onnx-tensorrt submodule * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.ubuntu_tensorrt * Update run_dockerbuild.sh * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update concat_op_test.cc * Update tensorrt_execution_provider.cc * Upgrade TensorRT to version 6.0.1.5 * Update onnxruntime_providers.cmake * Update CMakeLists.txt * Update reduction_ops_test.cc * Update install_ubuntu.sh * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.tensorrt * Update BUILD.md * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update onnxruntime_providers.cmake * Update install_ubuntu.sh * Update install_ubuntu.sh * Update gemm_test.cc * Update gather_op_test.cc * Update CMakeLists.txt * Removed submodule * update onnx-tensorrt submodule * update header file * Removed submodule * add submodule onnx-tensorrt kevin's branch shape-test' * add debugging code * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * merge master * Removed submodule * update onnx-tensorrt submodule * add more changes for dynamic shapes * Update tensorrt_execution_provider.cc * update for dynamic shape * update dynamic shape processing * fix logger issue * remove submodule onnx-tensorrt * add submodule onnx-tensorrt * add env variable min_subgraph_size * remove redundency * update document * use onnxruntime::make_unique * fix multi-run issue * remove some tests to save CI build time * Add dynamic shape test * Update TensorRT-ExecutionProvider.md * Add example of running Faster R-CNN model on TensorRT EP * Add more details on env variables * update environment variables * Update tensorrt_basic_test.cc * Update model tests * Update tensor_op_test.cc * remove --use_full_protobuf * Update build.py * User/xianz/telemetry (#2458) * enabme telemetry * enable telemetry * set enable telemetry as default * for debugging * remove log and set disable telemetry as default back * delete private file while testing * resolve comment: mainly add license header, rename macro and update docs * rewording in privacy.md * Fix integer overflow in cuda NonMaxSuppression implementation (#2540) * add test case that should pass but fail * fix nms * extract int_max_output_boxes_per_class * Introduce container type runtime checks and other improvements (#2522) Rework TensorSeq in a manner consistent with Tensor and SparseTensor in terms of type system setup. Reduce templating. Introduce helpers to ensure the same data type. Make OrtValue __dtor not virtual. Introduce ContainerChecker * Fix C API tests for centos and mac (#2544) * change c++14 to c++11 * add ld lib path for centos * enable csharp tests on macos * fix C API test on MacOS + fix manylinux dotnet install * fix manylinux dotnet install * fix lib link * Add back executable bit to build.py * Fix a bug handling negative begin pad values in Pad op (#2550) * Fix bug in Pad op * Update * DNNL CMAKE update (#2548) * Fix android build (#2558) * Update win-x86-ci.yml (#2557) Fix build pipeline break * Re-enable Windows C# tests (#2564) * disable onnx_test_runner -x invocations for dnnl (#2568) * Allow sequence length to be symbolic (#2559) * setup java ci mac (#2570) * make layernorm fusion to support opset 11 (#2545) * Fix a warning found in the latest VS release * Add more check on SkipLayerNorm and BiasGelu fusion (#2574) * Fix file not found error during docker build. (#2569) * Add ConvTranspose1D (#2578) * Ryanunderhill/packagename test (#2582) * [Nuphar EP] fixes for some object detection models (#2581) Update notebook tutorial with multi-threaded int8 GEMM from #2517 * EmbedLayerNormalization Fusion Improvement (#2553) Embedding layer norm fusion improvements - add more checks * Update version (#2584) * Temporarily exclude vgg19 test from Python backend test 1. temporarily exclude vgg19 test which comsumes too much memory, run out of memory on Upsquared device. Single test pass for vgg19, need furture investigation (#2588) 2. Update docker file to decrease the docker image size * Update docs for Android NNAPI EP (#2586) * Fix lto bug for protobuf and ubuntu * add path to build dir before test run (#2590) * Add missig env variables for mac pipeline test (#2595) * Fixed an issue in updating realized dims (#2597) when we update realized dims for scan's output, the sliced axis also needs to be inclusive, i.e. we should check with "dim >= insert_inclusive_axis", because the offset in the symbols are based on Scan sugraph. Otherwise, we would end up with shape mismatch later. * Java API for onnxruntime (#2215) * Add support for opset 11 in reshape fusion (#2592) Support opset verion 11 in reshape fusion * Rename automl python tools folder to featurizer_ops. (#2593) * Support opset 11 subgraph of Squad model in Embed Layer Normalization (#2605) Support opset 11 Squad model that is exported from PyTorch nightly. The embed layer uses Range op which is missed in the transformer. * symbolic shape inference: fix warnings in GPT-2 model (#2608) And revise nuphar perf test on BERT squad * Dump subgraph ID and fused graph ID (#2607) * Dump subgraph ID and fused graph ID Dump subgraph ID and fused graph ID for better debugging * Remove local static fused_count added a field global_fused_count_ to NupharExecutionProvider class * EmbedLayerNormalization Fusion For Dynamic Squad Model Opset 10 (#2613) Support subgraph of SQuAD model exported from pytorch with dynamic input axes * Allow providers to be set for InferenceSession at construction (#2606) * Remove unnecessary parameter in some places in GatherElements implementation (#2612) * Remove unnecessary parameter in some places * Update * Update * Make sure fenced tensor could not reuse other tensor. (#2561) Fix random error caused by this. * Improve Embed Layer Norm Fusion for SQuAD with static input shape (#2621) * fix float16 comparison in initializer (#2629) * epsilon attribute for layernormalization fusion (#2639) * removed unnecessary batch file and fix path (#2640) * Add shape inference to ConvTransposeWithDynamicPads schema (#2632) * Improve cuda expand() opeator's performance. (#2624) * Cuda pad optimize when no padding is needed. (#2625) * Shortcut cuda Pad() when no padding is needed. * Optimize cuda scatter() on 2D compatible. (#2628) * Optimize cuda scatter() on 2D compatible. * Add some comments. * fix build error for ARM (#2648) * Improve performance of resize() in Nearest mode (#2626) Special treatment for 2D, check same size as input image. And in 2d kernel, template use_expolation. * Fix memory exception in Layer Norm Fusion (#2644) * Windows CI changes(#2650) * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Revert "Windowsai without fi (#2701)" This reverts commit 285d4c85ff5c4e265f963208170304ef3461e684. * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Deref unique pointer for session_state * send shutdown event when dll is unloaded and EvaluationStop, SessionC… (#2704) * send shutdown event when dll is unloaded and EvaluationStop, SessionCreationStart Events. * Add EvalutationStart Event * add comment * use correct type for for loop (#2755) * ARM CI (#2759) * Set ARM agent pool * Set CMake generator to VS 2019 in ARM * Use system-wide CMake instead of custom version Our custom version is too old for VS 2019 * Use DML and build shared lib in ARM CI * Restore nuget packages in ARM CI * Disable DML * Refactor ARM debug/release builds * Use system packaged Python version * Remove hardcoded Python path * Downgrade Python to 3.7 for build * Remove explicit CMake path * Fix invalid JSON in cgmanifest.json (#2760) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * Remove usage of WHOLEARCHIVE in WinML CMake and add WinMLAdapterFactory (#2726) * Remove usage of WHOLEARCHIVE in WinMLAdapter CMake and add WinMLAdapterFactory * PR feedback, no need for dll(export) since using def file * PR comments * Small comment in gen_def.py * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * Move winml_provider_factory.h to proper location (#2801) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Filter CPU case for IsFloat16Supported (#2802) * Merge fixes * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * Fix warnings that cause build to fail * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * Make DML include path global (#2882) * Make DML include path global * Add generated cppwinrt headers to winml_lib_common * Integrate changes to WindowsAI to make ADO Build (#2886) * Revert "CMake cross-generator fixes (#2790)" This reverts commit dbe7d97fa1ab155f1309bced87199527e8f35bd2. * add additional suppress warning in onnx_proto * ignore /wd4996 warning * DML execution provider fixes * Revert "Revert "CMake cross-generator fixes (#2790)"" This reverts commit 1ae7b4bcbc02edc881ad28685da98e095dfceb17. * Update func signature of custom op function overloads * common devicehelpers fixes * Add pch.h for winml_lib_common * re-add winml_lib_common_dir/inc to include path for winml_adapter * User/orilevari/dml redist shared folder (#2890) * move dml nuget package directory up one level to make it shared between build flavors * Merge conflict fix * Revert "Merge conflict fix" This reverts commit 142fa72cf9ce4344ad717b50b7ea2b8582aadc7c. * Revert "Merge remote-tracking branch 'origin/master' into windowsai" This reverts commit 6e2126d46e5e5f564d65da37dd4f70c93dd81165, reversing changes made to b3f5583dc9249834b947c8ea905f6a98060d5bd6. * Make winml_test_common free of test macros (#2902) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Don't generate files outside the build root (#2914) * Don't generate files outside the build root * Add onnxruntime_EXTERNAL_DEPENDENCIES to WinML * Add DML depedency on RESTORE_PACKAGES * User/orilevari/fix yaml merge bugs (#2918) * Add winml test source parameter into cmake function (#2919) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Add arguments to unittest cmake functions * remove comment * Revert "Revert "Merge remote-tracking branch 'origin/master' into windowsai"" This reverts commit ade5abe72a4234fdbc3623093c61c02c6b0bdc26. * Fix breaks from merge with ORT master * Brianma/linux (#2917) * don't include windows.h in cross-plat header * add default case for switch statement * signed/unsigned mismatch fix Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> * User/sheilk/winml adapter c api (#2891) * Create winml adapter c api * fix build * make it build * move adapter into onnxruntime core/session * entry point not exported * minor changes * make model metadata work * make tests pass * implement all the model reflection apis on the adapter c abi * update the new ort interface to create a lotus ennvironment with a logging sink * start adding ort env * move all winml code into adapter folder/lib to isolate it * ensure a single logging manager at a time * start refactoring session * refactor session creation interface * add cpu and dml session option methods to adapter * finish session init * stub out interfaces in ort lib to perform similar mechanics of iinference session * enable profiling, and enable schema override * update session register graph transformers * turn back on custom registry for custom ops * Add sync api * add last c api stubs * should build... but all feature values are broken since this is in flight to moving all implementation details into ivalue * remove ep adapter header * Implement DML execution provider functions from adapter (#2846) * Implement DML execution provider functions from adapter * Use functions in OnnxruntimeEngine.cpp * make map/sequence type_infos freeable, and start implementing ivalue * make it build again * implement value methods * implement remaining methods * remove com adapter abi * check dml session * cache the allocator on ivalue * check if resource is cpu/gpu when access its mutable data * update tensor * mismatched parentheses * fix tensor base and binding obj * it evaluates tensors! sometimes... * minor fixes * enable gpu evals * wrapper all existing winml adapter apis with API_IMPL to try catch (#2854) * update winml... tensor strings are broken, need to template tensorbase to do different things for strings * make tensor strings work with 2 copies in/2 copies out * Fix tensor string and allocator bug * make maps work again... needs some fixes still * Make it build! * enable map inputs * map outputs * unbound outputs for sequences and maps * User/xianz/merge windowsai (#2883) * Packaging pipeline changes for VS 2019 (#2711) * Tiny fix to codegen * Simplify cache implementation and avoid static variables that may carry over between models * Extend DML kernels (#2641) * Additional DML operators * Check unsupported attributes and inputs * Address PR comments * Add kernel capability function used for partitioning, and re-enable stride-based int64 support based on value range * Fix test failures * Build fix * PR comments * Update Nuphar tutorial notebook (#2721) 1. Reflect int8 GEMV improvements for multi-threading from #2696 2. Add notes on multi-threading control using OpenMP 3. Add samples of running multi-isa AOT, and show int8 GEMM differences between AVX and AVX2 4. Add rnn_benchmark example to resolve #1993 * Add schema for new Qops (#2611) * Add schema for new Qops * adding shape inference + qlinearaveragepool * plus review comments * plus review comments * updates per review comments * plus review comments * [server] Add supposed for model_name and model_version as cli parameter (#2708) * remove 64bit warning message from python validation. (#2727) * MLAS: ARM64 build fix (#2734) fix bad usage of vreinterpret to cast vector element types * Fix broken python docs links (#2740) * Fix build on Mac OS (#2731) mac os ld doesn't support --while-archive, correct option is -all_load * fix ngraph wheel (#2737) * fix ngraph wheel 1.1.0 onnxruntime_ngraph wheel doesn't work * remove libdnnl.so in nGraph Libs * make it easy to compare * Split onnxruntime server to a separated folder (#2744) * Fix build for Python 3.8 (#2747) * Fix build for Python 3.8 * Update protobuf to 3.11.2 (#1928) Update protobuf to 3.11.2 (#1928) * Change default optimization level to All (from Basic) (#2745) * change default optimization level to All (from Basic) * fix test * fix c# test * Update numpy to 1.18 (#2758) * Update numpy to 1.18 * Pipeline changes for python 3.8 (#2753) 1. Pipeline changes for python 3.8 2. Fix a regression in setup.py which was just introduced in the previous commit. Please notice, we still haven't made python 3.8 + Windows + CUDA work. * Add basic stacktrace output for posix debug builds. (#2749) * [NupharEP] fix a race condition when multiple sessions running different models concurrently (#2772) * Revert "Change default optimization level to All (from Basic) (#2745)" This reverts commit 56bb503c2f26474b6613bcb2a198691a11dcef38. * Fix typo in error message (#2736) * Rename MKL-DNN to DNNL to fix broken link (#2730) * Fix nightly build version number issue * Pass BUILD_BUILDNUMBER to linux docker * Disable featurizers in python packages * Import more featurizers (#2781) Make kernels non-template. Add input constraint for learnt data. Add min_max_scalar_transformer, robust_scalar_transformer, inputation_marker_transfomer, label_encoder_transformer, missing_dummies_transformer along with tests. Advance Featurizers library commit. * Implement a more stable softmax (#2715) * Implement a more stable SoftMax e^x is represented as infinity if x is large enough, like 100.f. Infinity divided by Infinity is a NAN. Thus, softmax gets a NAN if one or more item are large enough. A math transform as below is leveraged to get a stable softmax: e^xi/(e^x1 + ...e^xn) = e^(xi - max) / (e^(x1 - max) + ... + e^(xn - max)) And for convenience, force max to 0.f if all xi are negative * Contributing: Fix a typo (#2784) * ACL EP GEMM improvements (#2780) When it is posible we use a fully connected layer instead of the gemm implementation. This will let the library use the best implementation based on the input data. * ACL EP convolution improvements (#2774) Added the optimized implementation for depthwise convolution for both ACL v19.02 and ACL 19.05. Also the pointwise convolution seems to be more optimal in the CPU implementation so we opted for that instead. * Add script for release Nuget validation (#2719) * Initial commit * Nits * Disable a test temporarily * Change working directory * Test * Add download python step * Test update * More changes * Fix space issue * Fix * Verify nuget signing * Fix * Spaces * PR feedback * Nit * Fix * Fix * Remove temporary changes * add uint8 support to where op (#2792) * Improve bert optimization script: (#2712) (1) Move input int64=>int32 conversion to embed layer fusion. (2) Output epsilon attribute for LayerNormalization fusion. * add session creation time cost. (#2798) * ML.NET team needs featurizers within a package (#2789) Add auto ml featurizers to Windows, MacOS as well as to GPU packaging-pipelines. * Initialize max of softmax with lowest of float (#2786) * MLAS: update SGEMM threading parameters (#2808) * add interface to copy batch tensors. (#2807) * add interface to copy batch tensors. * onnxruntime * speed up Windows TRT CI (#2811) * don't run cuda tests if building with tensorrt * remove unnecessary build options for win trt ci * refactor win gpu tensorrt ci yml * --numpy_version=1.17 * update * update * azcopy and cuda path * Update test data (#2356) * Add timeseries imputer transformer featurizer kernel (#2813) Make kernels non-template. Add input constraint for learnt data. Fixup tests. Add two more featurizers along with tests. Tests fail. min_max_scalar_transformer robust_scalar_transformer Fix tests serialized stream by prepending version bytes. Add inputation_marker_transfomer and the test. Fix up float/double type designations. Added label_encoder_transformer along with a test. string_throw case is broken at the momement. Fix labelencodertransfomer_test.cc string_throw case Rename maxabsscalertransformer_test.cc Add MissingDummiesTransformer along with the test. Update manifest. Add TimeSeriesImputerTransformer definition, implementation and tests * Fix memory leak in TRT (#2815) * fix memory leak issue * revert EP_FAIL on enueueV2 * Add manifest missing comma * Run static code analyzer on most of our code (#2817) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * update quantization doc (#2783) * update documentation for quantization script * plus some spell corrections * Filter CPU case for IsFloat16Supported (#2802) * update default optimization level + fix gemm_activation fusion (#2791) * update defualt optimization level + fix gemm_activation fusion * fix typo * add unit test and incorporate review comments * fix test comment * Fix dnnl wheel package name (#2823) * Append '-dnnl' to whl package name when --use_dnnl * Update build.py * Update Ubuntu & TensorRT version in README (#2820) Dockerfile.tensorrt is using nvcr.io/nvidia/tensorrt:19.09-py3 as base Image, update Ubuntu and TensorRT version according to https://docs.nvidia.com/deeplearning/sdk/tensorrt-container-release-notes/rel_19-09.html#rel_19-09 * Merge fixes * Add OneHotEncoder and HashOneHotEncoder kernels. (#2830) Add defs and imlementation for OneHotEncoders, adjuist date_time_transformer kernel and test. Add OneHotEncoder kernel test. Add HashOneHotVectorizerTransformer unit test. This does not link due to multiple definitions of functions that are included into header from a CPP file. * Upgrade gtest to the latest version (#2827) WinML would like to update the googletest submodule. They want some newer features (namely GTEST_SKIP to skip tests programmatically and be able to skip entire fixtures easily) and would need to update the submodule version. However, because the new version of code hit a bug in gcc, even though the bug is already fixed in the latest gcc but we're using gcc 4.8.x and it won't get patched for the bug, so we have to do a compromise, change our code a little bit to make it work. The gcc bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=51213 * Add support for int64_t for topk CPU. Fixes github issue #2806. (#2833) * Ignore allocator type in ExecutionProviders allocator map. Make default initialization of OrtMemoryInfo more clearly invalid. (#2768) * Remove allocator type from the key comparison in ExecutionProviders. Remove usage of DummyArena as it's no longer necessary. * Fix x86 tests where arena allocator is disabled. Make initialization of OrtMemoryInfo clearer by adding Invalid enum value. * Make OrtValueNameIdxMap::MaxIdx more intuitive. * Convert ExternalProject Featurizers into git submodule (#2834) Add git submodule for Featurizer library. Update cmake to build for git submodule. * add domain check for nodes + update documentation (#2831) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Fix Linux CUDA nuget packaging pipeline break * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * update optimization doc for BERT related fusions (#2819) * Add bert related transformers to doc * Add execution provider and comment for bert optimizations * Add comment about accuracy impact of approximation * Fix warnings that cause build to fail * MLAS: enable threading for quantized GEMMs (#2844) * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * remove old winmladapter cpp Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * move sequence implementation into ort lib... still commented out... need to turn back on... * begin sequence implementation * make maps and sequences work * fix broken tests * remove dead code * misc cleanup * CR feedback * User/xianz/winml adapter c api (#2869) * wrapper all existing winml adapter apis with API_IMPL to try catch * Return HR or Throw for WinML adapter APIs if failed * undo macro wrapper for two places * Wrap error macros around ort apis, too. * address CR feedback #2 * add more api throw/return macros * Revert changes no longer needed * revert changes to cxx api * format winml lib.ort and winml adapter * remove static pheonix singleton Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * missing use_dml check in winml_adapter_session (#2930) * --use_dnnl flag was mangled in merge (#2931) * use dml macro not wrapping custom registry code (#2934) * Disable LNK4199 winml_dll to enable cuda builds (#2936) * Disable LNK4199 in winml_dll * linkler->linker * LearningModelSessionAPITestGpu.CreateSessionWithCastToFloat16InModel should return DXGI_ERROR_UNSUPPORTED when FP16 not supported (#2937) * Disable LNK4199 in winml_dll * linkler->linker * Need to return DXGI_ERROR_UNSUPPORTED when Model does not support fp16 * Publish build symbols (#2939) * Publish build symbols * Don't upload PDBs for .exe files * Make x86 build (#2943) * fix last remaining size_t/int64_t warnings->errors (#2948) * TensorString, Sequences and Maps use the first allocator, but should use the cpu default allocator. (#2952) * fix tensor string allcoator * clean up default allocator usage for strings in winml lib/api.ort Co-authored-by: Ryan Lai <ryalai96@gmail.com> * Handle tensor shape of zero (#2954) Co-authored-by: Ryan Lai <ryalai96@gmail.com> * CR feedback (#2970) * CR feedback * fix weird formatting on privacy readme * Add 'All rights reserved.' everywhere * readd all rights reserved to winml_provider_factory.h * remove extra space in comment * remove extra whitespace * fixes post master merge * remove winml from nuget gpu pipeline * set IR VERSION on generated_model in rnn_benchmark (#2972) * Fix slice conformance failures (#2908) Co-authored-by: Adrian Tsai <adtsai@microsoft.com> Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Paul McDaniel <paul_mcdaniel@hotmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Dwayne Robinson <fdwr@hotmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com>
2020-02-05 01:12:19 +00:00
[contract(MachineLearningContract, 1)]
interface ILearningModelOperatorProvider : IInspectable
{
}
//! \interface LearningModel
//! \brief Represents a trained machine learning model.
//! \details This is the main object you use to interact with Windows Machine Learning. You use
//! it to load, bind, and evaluate trained ONNX models. To load the model you use
//! one of the Load constructors. You can then enumerate the InputFeatures and
//! OutputFeatures. To bind and evaluate you create a LearningModelSession.
[contract(MachineLearningContract, 1)]
INBOX_ONLY([static_name("Windows.AI.MachineLearning.ILearningModelStatics", e3b977e8-6952-4e47-8ef4-1f7f07897c6d)])
INBOX_ONLY([interface_name("Windows.AI.MachineLearning.ILearningModel", 5b8e4920-489f-4e86-9128-265a327b78fa)])
Merge windowsai (winml layering) into master (#2956) * Initial Commit * Merged PR 3985217: add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc (#2346) add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc and violating our OS layering requirements. We linked against onecoreuap_apiset.lib in VB so we will continue doing this, but I am still unsure why not to link against onecore instead since that is where we ship. However, since Sheil is the owner of this code we will wait to discuss with him before changing anything. * Initial changes for layering * more snipping to get core into ort * update build instructions to include --build_shared_lib (#2358) * update build instructions to include --build_shared_lib * fix line breaks * Task 23998197: add winml_lib_core into onnnxruntime.dll (#2368) * Task 23998197: add winml_lib_core into onnnxruntime.dll * PR feedback build break on perf_test * return proper error when the model path isn't found (#2391) * LearningModelSession is cleaned up to use the adapter, and parts of b… (#2382) this is a big PR. we are going to move it up to layer_dev , which is still a L3 so we are still safe to do work there agile. we are going to move this into the L3 so that ryan can start doing intergration testing. we will pause for a full code review and integration test result prior to going into the L2. >>>> raw comments from previous commits >>> * LearningModelSession is cleaned up to use the adapter, and parts of binding are. * moved everything in the winmladapter made it all nano-com using, WRL to construct objects in the ORT side. base interfaces for everythign for winml to call cleaned up a bunch of winml to use the base interfaces. * more pieces * GetData across the abi. * renamed some namepsace cleaned up OrtValue cleaned up Tensor cleaned up custom ops. everything *but* learnignmodel should be clean * make sure it's building. winml.dll is still a monolith. * model moved over. everything builds clean. step ! * weak ref comment * Layer dev paulm (#2408) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * Layer dev paulm (#2414) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * Layer dev paulm (#2423) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * Layer dev paulm (#2424) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * Layer dev paulm (#2425) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Layer dev paulm (#2426) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * fixed map and sequence when passing stl types across the ABI . found a leak in nvidia driver, but skipped it. all winmlapitests pass now * Moved SessionOptions over to the abi * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * Register WinML TraceLogging provider on Onnxruntime.dll (#2455) * Register WinML TraceLogging provider on Onnxruntime.dll * Add ifdef to make sure trace logging provider has telemetry option when LAYERING_DONE * No need for ifdef for TraceLoggingOptionMicrosoftTelemetry * PR feedback * Move etw registration into lotus environment constructor and deresgister in lotus environment destructor * Brianma/cpuwinml (#2466) * allow building winml cpu without dml. * Brianma/breaks (#2469) * fix some more breaks * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * move dml checks out of winml and into the adapter * better error handling * Brianma/fi (#2470) * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * fix bad merge * Checking in a staging checkpoint point so that Ryan can work with me in parrallel * build break. * Brianma/testfails (#2473) * add missing ir version to dictvectorizer-string.onnx * add missing ir version to relu.onnx * add missing ir version to zipmap*onnx * add IR version to manually generated models * remove an unnecessary ifdef dml * Brianma/windowsai fi (#2475) * update dockerfiles/README (#2336) * Make elementwise op run 4 items per thread (#2335) Description: Describe your changes. Make elementwise op run 4 items per thread unroll for loop to leverage ILP remove unnessary N==0 check inside elementwise GPU kernel Motivation and Context Why is this change required? What problem does it solve? It can improve the performance of GPU elementwise ops. ~2% performance gain on popular NLP bert model. If it fixes an open issue, please link to the issue here. * Add CUDA GatherElements kernel (#2310) * Updates * Update test * Update * Updates * nits * PR feedback * Update * Update * PR feedback * PR comments * Update * Fix build * Fix build * Nits * Fix * Layer Normalization Fusion (#2319) basic layer normalization transform * Add FastGelu Cuda Op for Gelu and Add bias fusion (#2293) * Add FastGelu cuda op * Add AddBiasGelu for experiment * Revert "Add AddBiasGelu for experiment" This reverts commit 5c1ee019858c657e6bb75887265cb85675626e5b. * Add bias * Add unit tests * update comment * update script * fix build error * update coding style * update for CR feedback Enable half2 optimization only when cuda arch >= 7.0 * move _Tanh to common.cuh * implement CPU contrib OP Attention (#2333) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanUnusedInitializers. (#2320) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanupUnusedInitializers. This means initializers that have been replaced during graph optimizations are not left in the GraphProto when we save an optimized model. * Handle edge case where a model has an unused initializer with matching graph input by also removing the graph input. * Use non-const iterators in std::find_if calls to make centos build happy. * Nuget pipeline changes (#2305) 1. refactor the pipeline, remove some duplicated code 2. Move Windows_py_GPU_Wheels job to Win-GPU-CUDA10. We'll deprecated the "Win-GPU" pool 3. Delete cpu-nocontribops-esrp-pipeline.yml and cpu-nocontribops-pipeline.yml 4. In Linux nuget jobs, run "make install" before creating the package. So that extra RPAH info will be removed * Cuda Reverse Sequence Op, maping types of same size using same template function. (#2281) * Set ElementType to String type of node metadata, instead of byte[] (#2348) * Set ElementType to String type of node metadata, instead of byte[] * Fix spacing * Introduce PrimitiveType into a Type System along with an integer constant (#2307) Improve perf by avoiding GetType<T>() calls. Introduce MLTypeCallDispatcher to switch on Input Type. Add Tensor IsType<T>() fast method. * Fix/test dim value of 0 handling in a couple of places (#2337) * Update the CUDA Where implementation broadcasting logic to handle a dim with value of 0. Add unit test Also add unit test for unary op with dim value of 0 * Exclude ngraph from Where test with 0 dim. * Openvino EP R3.1 onnxrt server (#2357) * onnxrt server with OVEP * onnxrt server with OVEP * Update Dockerfile.server.openvino * onnxrt server OVEP fix reviews * onnxrt server OVEP fix reviews * Implement cuda nonzero op. (#2056) Implement cuda nonzero op. * Direct use python numpy array's memory if already contiguous. (#2355) * Direct use python numpy array's memory if already contiguous. This could greatly improve performance for session with large input, like big image 1920x1080 fastrcnn, 30~40% speed up could be achieved. * Add test case enforce contiguous/non-contiguos numpy array as inputs. * Add helper to create output to minimize binary size. (#2365) Add ConstEigenTensorMap typedef so we don't unnecessarily const_cast the const input Tensor. * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS (#2369) * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS * update * Add Tracelogging for profiling (#1639) Enabled only if onnxruntime_ENABLE_INSTRUMENT is ON * test bidaf with nuphar for avx target (#2370) increase nuphar test coverage a bit * Fix a bug in TLS refcount that may destabilized CUDA CI (#2374) * update output size calculation for resize (#2366) * change how output size is calculated for resize op * add tests for ver 10 resize * Extend OneHot CPU kernel to support more types (#2311) * Extend OneHot CPU kernel to support input int64_t, depth int32_t, output float * Skip BERT before the test data fix is picked up * Fix bug with Slice. Need to pass in flattened input dimensions so the initial offset into the input is calculated correctly. (#2372) * Add opset 11 version of Split to CUDA ops (#2376) Organize the CUDA ops definitions so all the opset 10 and 11 parts are together (same setup used for CPU ops) * Layer Norm Fusion Fix (#2379) * layer norm fusion fix * Add input shape check in code and unit tests * Fuse Add + Gelu (#2360) Implement the transformer to fuse add + gelu Implement the accurate kernel * Skip layer norm transform (#2350) * skip layer normalization transformer * Another try to stabilize CUDA CI (#2383) The root cause seems to be failure in CUDA dealloc when tear down. cudaFree return code was ignored before, so should the debug check. * fix BUILD.md typo (#2375) build.py: error: argument --config: invalid choice: 'RelWithDebugInfo' (choose from 'Debug', 'MinSizeRel', 'Release', 'RelWithDebInfo') * Fixed compilation with ngraph (#2388) * Fix reuse logic in allocation planner. (#2393) * Fix reuse logic in allocation planner. * PR comments * Add helpful comments * Don't allow reuse across string tensors. * [NupharEP] Multiple optimizations (#2380) Fuse transpose into MatMul Implement Pow and constant scalar simplification Vectorize ReduceMean Improve symbolic shape inference Minor updates for better debugging in fused function name * Avoid using the default logger in the graph lib and optimizers (#2361) 1. Use the session logger if it is available. 2. Don't disable warning 4100 globally. We should fix the warnings instead of disabling it. * Change CUDA implementation of Transpose to support all fixed size tensor types (#2387) * Change CUDA implementation of Transpose to not use a typed kernel so we can support more types with minimum binary size. Add support for 8, 16, 32 and 64 bit types. Add unit tests. Add method so the implementation can be called directly (will be used by CUDA Scan very soon). * Disable TensorRT for MLFloat16 and int8 unit tests. * Address PR comment and add support for calling cublas implementation if type is mlfloat16. * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. (#2398) * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. * [NupharEP] force some low/zero cost ops to be inlined (#2409) * fix cross compile bug (#2415) * Minor optimization: if a node has already been placed, there's no need to find a kernel for it. (#2417) * Add Reshape Fusion (#2395) * Add reshape fusion * Add some comments * update comments * update comment format * update according to feedback * update for recent logger change * fix build error * (1) Support both input and output edges in find path in graphutils (2) Add a test case of only one constant initializer of Concat input. (3) Refactor ReshapeFusion class to allow add more subgraph fusion in the future. * fix error * (1) loose constraint on initializer: non constant is allowed for reshape fusion. (2) Change versions type to vector. (3) Add logging. (4) Return false when multiple output edges matched in FindPath. Add comments. * only allow one direction (input or output) in FindPath * [NupharEP] Update notebook and docker image (#2416) Add BERT squad in Nuphar tutorial Enhance speed comparsion readability * Fix the issue in matmul_add_fusion (#2407) Fix the issue in matmul_add_fusion If Muatmul + Add has shape [K] * [K, N], reset it to [1, K] * [K, N] will make the output shape to [1, N] will also requires a reshape on the output. Fix: just remove the shape reset to not fuse it. Add a negative test case for matmul+add fusion * feat(treeregressor): Update TreeEnsembleRegressor for type support (#2389) Updates the `TreeEnsembleRegressor` to allow for `double`, `float`, `int64`, and `int32` inputs to match the upstream specification. Signed-off-by: Nick Groszewski <nicholas.groszewski@capitalone.com> * onnxrt server documentation update (#2396) * Added support for Pad-2 operator in OpenVINO-EP (#2405) * Add CUDA If operator. (#2377) * Add CUDA If operator. Uses CPU operator for implementation. By adding a CUDA version the inputs/outputs (with the exception of the 'cond' input) stay on GPU, and no other logic is required to avoid a copy to CPU across the control flow node. * Improved documentation for onnxruntime::utils::SwapByteOrderCopy(), added precondition check. * Fix the type constraints on CUDA If operator to exclude strings. (#2431) * add Im2col<uint8_t> (#2438) * Adjust codegen vectorization width from target (#2439) * Adjust codegen vectorization width from target * Add CUDA Scan operator. (#2403) * Add Scan CUDA op. Uses CPU implementation for logic. Added some device specific functors for handling when data needs to be manipulated on a different device. Added ability to override the materialization logic in the OrtValue slicer so DML can plugin their handling. * Fix Windows GPU C API packaging pipeline failure (#2440) Fix Windows GPU C API packaging pipeline failure (#2440) * Correctly handle implicit inputs for fused nodes (#2390) * Correctly handle implicit inputs for fused nodes Previously, nuphar's partitioning function didn't include node's implicit inputs into the inputs list of MetaDef, and hence a crash was triggered in the onnx graph checker. This commit fixed the issue. Furthermore, it also fixed a related issue where we didn't add implicit inputs into graph_inputs_excluding_initializers_ in Graph::SetGraphInputsOutputs. the issue was that graph_inputs_including_initializers_ populated by SetInputs (e.g. called by FunctionImpl::FunctionImpl) may contain implicit inputs which were not of any node's initializers in the graph. Because they were not part of any initializers, these implicit inputs couldn't be visited by going through all nodes' inputs. Consequently, they would *not* be added into graph_inputs_excluding_initializers_. We fixed the issue by first copying the populated graph_inputs_including_initializers_ into graph_inputs_excluding_initalizers_, which then had both initializers and non-initializers as its initial content. Later, we erase initializers from the list. In this way, we can ensure all implicit inputs to remain in graph_inputs_excluding_initializers_. * refined comments and fixed duplicates Address CR by revisiting comments in terms of implicit inputs Also fixed an issue by skipping duplicates while copying inputs from graph_inputs_including_initializers_. * address CR explain why we need to collect nodes' implicit inputs * don't rely on pointer values for iterating std::set Previously, openvino relied on iterating a set of NodeArg pointers to construct inputs and outputs for a fused graph. It could cause non-determinism. The reason was that although iterating std::set by itself is stable, pointer values of NodeArgs may vary. Consequently, we could end up visiting the set's elements in different orders for different runs for the same test, which resulted in constructing inputs (and outputs) with different orders to the fused graph. For example, for the same test, we may have inputs [A, B] in some runs but inputs[B, A] in others. Let's use std::string as the key type to avoid such nondeterminism. This commit also added implicit inputs into meta->inputs while returning the capability from the openvino provider. * Fixed another latent issue in openvino's GetCapability function The issue was that we couldn't simply erase fused_inputs and fused_outputs while iterating the nodes. For example, an output NodeArg may have multiple uses, and it's wrong if we erase it from fused_outputs when we encounter only one of its uses as input. * Remove DeviceAllocatorRegistry class (#2451) Remove DeviceAllocatorRegistry class * CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library. * Parallel Gelu with ParallelFor (#2399) Parallel Gelu to get better performance for Gelu * Clean up build.py (#2446) * Pull the latest image before running docker build * Fuse SkipLayerNorm with Bias (#2453) Fuse SkipLayerNorm with Bias * Allow more than one invocation of CreateEnv in the same process. (#2467) * Allow more than one invocation of CreateEnv in the same process. * Fix centos build * Symbolic shape inference improvements: (#2460) * Symbolic shape inference improvements: - add a mode to guess unknown ops' output rank - add support for GatherND - add support for If - fix a bug in get_int_values when then tensor rank > 1D, by treating it as no sympy data - add symbol to literal merge when ONNX silently merges dims - fix a bug in Concat when input dim is 0 - fix a bug in ConstantOfShape that computed dim is not updated - add support for dynamic shape in ConstantOfShape - fix a bug in Loop output shape that loop iterator dim is not inserted at dim 0 - add support for dynamic padding in Pad - add support for dynamic shape in Reshape - add support for Resize with opset > 10, by treating output dims as dynamic - fix a bug in Slice when starts/ends are dynamic - restrict input model to opset 7 and above - make output model optional to avoid disk write when testing Run model tests for symbolic shape inference Reduce 2GB docker image size of nuphar * add additional test data set for nuget pipeline (#2448) * add SAS token to download internal test data for nuget pipeline * update azure endpoint * fix keyvault download step * fix variable declaration for secret group * fix indentation * fix yaml syntax for variables * fix setting secrets for script * fix env synctax * Fix macos pipeline * attempt to add secrets to windows download data * fix mac and win data download * fix windows data download * update test data set url and location * Revert "Brianma/windowsai fi (#2475)" This reverts commit 5780b864a15513fda4eadbfc2b5345fefe70b5ec. * Add scenario tests (#2457) * Add scenario tests * Remove TODO from model license * Add winml_api test dependency * fix model load test. fi from master changed the constructor (#2483) * make api tests all pass (#2486) * fix bad merge * fix bad model merge * Layer dev paulm (#2492) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * Rename ambiguous header (#2489) * fix one more missing IR version model (#2500) * add missing IR version to 4 more models used by scenario tests (#2501) * Add CLI parameters to test runner, build WinML in ARM and x86 CI (#2479) * Support test parameters through CLI arguments * Add WinML do Windows x86/ARM CI builds * Code style fixes * Update googletest Remove GPUTEST macros everywhere now that GTEST_SKIP is supported * Refactor main.cpp * Build scenario tests without DML * Link scenario tests to DML when it's enabled (#2502) * Layer dev release pipeline (#2488) Adds winml binaries to existing cpu nuget package, and creates new gpu dml nuget package with winml binaries and DML EP. * Layer dev paulm (#2506) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * Remove usage of IOBinding in WinML and use C_API Run method (#2504) * remove usage of iobinding * Change data structure to use vector of Ort::Values * Polish bind input / output * Use C APIrun method * Update providers on evaluate getresults * Remove run and IObinding interface from WinMLAdapter * Remove use of IObinding * bind unbound outputs code moved to learningmodelbinding * clean up unneeded istensor adapter function * Fix comment * Check if session is closed before binding and clearing * PR feedback * Layer dev paulm (#2507) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * Make tests dependend on winml_dll (#2509) * add dml binaries to DirectML package and be more explicit about condition variables (#2520) * re-enable warnings for winml builds and fix the warnings that were hiding (#2526) * turn devmode back on for winml builds * fix some warnings. include protobuf in a way that disables some warnings * undo protobufhelpers changes and just ignore 4100 errors in pb code * attempt to isolate protobufhelpers errors * add template specialization for getting tensor proto data * Layer dev paulm (#2533) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * Spawn child process to run DeviceLostRecovery scenario test (#2530) * Spawn child process to run DeviceLostRecovery scenario test * Layer dev paulm (#2536) ori said yes * add missing namespace to winml_trace_logging_provider in lotusenvironment.h (#2542) * Handle exception thrown from all apis in WinMLAdapter (#2539) * various changes to unblock windowsai ADO build * Fix custom ops scenario tests (#2562) * Do not shutdown protobuf after ort environment gets destroyed. Lazy load lotus environment first time it is needed * comment typo * pr comment about calling phoenix singleton * Make lotus_environment static in winmladapter * Layer dev paulm (#2567) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * making windowsAI azure dev ops work. * code review comments. * revert changes * Cmake and preprocessor fixes that where uncovered by building on agents without DML available via SDK * Layer dev dml delayload (#2580) * Brianma/cpu (#2583) * don't include dml stuff in cpu builds * tests that link the image lib also need the telemetry lib now * Throw Winml_err_invalid_binding if binding gpu resource on cpu device (#2589) * Throw Winml_err_invalid_binding if binding gpu resource on cpu device * PR comments. No need to query executionprovider for is gpu device * User/xianz/ortthrow (#2596) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * User/xianz/ortthrow (#2599) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * add status error message * Remove uwp onsuspending winrt call because logruntimeperf is getting removed (#2630) * User/xianz/dedup telemetry (#2631) * investigate duplication of telemetry in winml and ort * remove winml telemetry events * telemetry executionProviderEvent * remove unneccessary file and refactor code little bit * Revert back TelemetryEvent, which send up ETW event. * merge changes from layer_dev to windowsai (#2638) * Remove underscore from googletest names (#2616) * Fix leaking memory allocator Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24278761 and https://microsoft.visualstudio.com/OS/_workitems/edit/24330198 * Explicitly initialize Ort::Value with nullptr * Cache WinML adapter * bad merge * define private version of dxcore enum that is added in 19H1 SDK. (#2654) * add comment for explaning private definition of dxcore d3d feature level ennum value. (#2672) * do not package directml.pdb for redist packages. (#2676) * Fix leaking operator registry (#2645) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24354916 * User/orilevari/windowsai master merge (#2674) merge resolutions included pulling in telemetry logic that was merged to master and not windowsai and dereferencing InferenceSession::sessionstate now that it is a unique pointer * Delete Ort Allocator in LearningModelBinding (#2653) * Delete OrtAllocator in LearningModelBinding * PR comments to make Ort::Allocator a smart pointer * Small comment change * PR feedback to clean up code * PR feedback on move semantics * Clean up std::move * Fix memory leaks (#2679) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24356109, https://microsoft.visualstudio.com/OS/_workitems/edit/24388361 and https://microsoft.visualstudio.com/OS/_workitems/edit/24388596 * various changes to properly organize and skip GPU tests. For now for No DML builds we will not run GPU tests at all. In the future we should adapt the tests to expect the appropiate errors. (#2695) * Windowsai without fi (#2701) * Disable Attention fusion tests when DISABLE_CONTRIB_OPS is defined (#2529) * Setup java ci (#2528) * Add provision in ORT for session options to be parsed when available via model file (#2449) * Initial commit * Fix gitmodules * Nits * Nits * Updates * Update * More changes * Updates * Update * Some updates * More changes * Update * Update * Merge * Update * Updates * More changes * Update * Fix nits * Updates * Fix warning * Fix build * Add comment * PR feedback * PR feedback * Updates * Updates * Update * More changes * Fix build break * Comment test for now * Updates * Updates * PR feedback * Updates * Nits * Add tests * Fix build * Fix build * Fix build * Fix build break * Fix build * Nits * PR feedback * More change * Expose GetSessionOptions in pybind logic and add unit test for python * Fix build * PR feedback * PR feedback * Revert "Disable thread pool creation when enabled OpenMP (#2485)" (#2535) This reverts commit 7c7d5a149c9ed52eec67304bae5c4b132166a8a1. * Add dynamic shape support in TensorRT execution provider (#2450) * remove onnx-tensorrt submodule * add new onnx-tensorrt submodule (experiment) for trt6 * update engine build for trt6 * update compile and compute for tensorrt6.0 * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * switch to onnx-tensorrt master for TensorRT6' * Update tensorrt_execution_provider.cc * Handle dynamic batch size and add memcpy in TensorRT EP * update test cases * Update tensorrt_execution_provider.cc * update onnx-tensorrt submodule * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.ubuntu_tensorrt * Update run_dockerbuild.sh * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update concat_op_test.cc * Update tensorrt_execution_provider.cc * Upgrade TensorRT to version 6.0.1.5 * Update onnxruntime_providers.cmake * Update CMakeLists.txt * Update reduction_ops_test.cc * Update install_ubuntu.sh * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.tensorrt * Update BUILD.md * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update onnxruntime_providers.cmake * Update install_ubuntu.sh * Update install_ubuntu.sh * Update gemm_test.cc * Update gather_op_test.cc * Update CMakeLists.txt * Removed submodule * update onnx-tensorrt submodule * update header file * Removed submodule * add submodule onnx-tensorrt kevin's branch shape-test' * add debugging code * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * merge master * Removed submodule * update onnx-tensorrt submodule * add more changes for dynamic shapes * Update tensorrt_execution_provider.cc * update for dynamic shape * update dynamic shape processing * fix logger issue * remove submodule onnx-tensorrt * add submodule onnx-tensorrt * add env variable min_subgraph_size * remove redundency * update document * use onnxruntime::make_unique * fix multi-run issue * remove some tests to save CI build time * Add dynamic shape test * Update TensorRT-ExecutionProvider.md * Add example of running Faster R-CNN model on TensorRT EP * Add more details on env variables * update environment variables * Update tensorrt_basic_test.cc * Update model tests * Update tensor_op_test.cc * remove --use_full_protobuf * Update build.py * User/xianz/telemetry (#2458) * enabme telemetry * enable telemetry * set enable telemetry as default * for debugging * remove log and set disable telemetry as default back * delete private file while testing * resolve comment: mainly add license header, rename macro and update docs * rewording in privacy.md * Fix integer overflow in cuda NonMaxSuppression implementation (#2540) * add test case that should pass but fail * fix nms * extract int_max_output_boxes_per_class * Introduce container type runtime checks and other improvements (#2522) Rework TensorSeq in a manner consistent with Tensor and SparseTensor in terms of type system setup. Reduce templating. Introduce helpers to ensure the same data type. Make OrtValue __dtor not virtual. Introduce ContainerChecker * Fix C API tests for centos and mac (#2544) * change c++14 to c++11 * add ld lib path for centos * enable csharp tests on macos * fix C API test on MacOS + fix manylinux dotnet install * fix manylinux dotnet install * fix lib link * Add back executable bit to build.py * Fix a bug handling negative begin pad values in Pad op (#2550) * Fix bug in Pad op * Update * DNNL CMAKE update (#2548) * Fix android build (#2558) * Update win-x86-ci.yml (#2557) Fix build pipeline break * Re-enable Windows C# tests (#2564) * disable onnx_test_runner -x invocations for dnnl (#2568) * Allow sequence length to be symbolic (#2559) * setup java ci mac (#2570) * make layernorm fusion to support opset 11 (#2545) * Fix a warning found in the latest VS release * Add more check on SkipLayerNorm and BiasGelu fusion (#2574) * Fix file not found error during docker build. (#2569) * Add ConvTranspose1D (#2578) * Ryanunderhill/packagename test (#2582) * [Nuphar EP] fixes for some object detection models (#2581) Update notebook tutorial with multi-threaded int8 GEMM from #2517 * EmbedLayerNormalization Fusion Improvement (#2553) Embedding layer norm fusion improvements - add more checks * Update version (#2584) * Temporarily exclude vgg19 test from Python backend test 1. temporarily exclude vgg19 test which comsumes too much memory, run out of memory on Upsquared device. Single test pass for vgg19, need furture investigation (#2588) 2. Update docker file to decrease the docker image size * Update docs for Android NNAPI EP (#2586) * Fix lto bug for protobuf and ubuntu * add path to build dir before test run (#2590) * Add missig env variables for mac pipeline test (#2595) * Fixed an issue in updating realized dims (#2597) when we update realized dims for scan's output, the sliced axis also needs to be inclusive, i.e. we should check with "dim >= insert_inclusive_axis", because the offset in the symbols are based on Scan sugraph. Otherwise, we would end up with shape mismatch later. * Java API for onnxruntime (#2215) * Add support for opset 11 in reshape fusion (#2592) Support opset verion 11 in reshape fusion * Rename automl python tools folder to featurizer_ops. (#2593) * Support opset 11 subgraph of Squad model in Embed Layer Normalization (#2605) Support opset 11 Squad model that is exported from PyTorch nightly. The embed layer uses Range op which is missed in the transformer. * symbolic shape inference: fix warnings in GPT-2 model (#2608) And revise nuphar perf test on BERT squad * Dump subgraph ID and fused graph ID (#2607) * Dump subgraph ID and fused graph ID Dump subgraph ID and fused graph ID for better debugging * Remove local static fused_count added a field global_fused_count_ to NupharExecutionProvider class * EmbedLayerNormalization Fusion For Dynamic Squad Model Opset 10 (#2613) Support subgraph of SQuAD model exported from pytorch with dynamic input axes * Allow providers to be set for InferenceSession at construction (#2606) * Remove unnecessary parameter in some places in GatherElements implementation (#2612) * Remove unnecessary parameter in some places * Update * Update * Make sure fenced tensor could not reuse other tensor. (#2561) Fix random error caused by this. * Improve Embed Layer Norm Fusion for SQuAD with static input shape (#2621) * fix float16 comparison in initializer (#2629) * epsilon attribute for layernormalization fusion (#2639) * removed unnecessary batch file and fix path (#2640) * Add shape inference to ConvTransposeWithDynamicPads schema (#2632) * Improve cuda expand() opeator's performance. (#2624) * Cuda pad optimize when no padding is needed. (#2625) * Shortcut cuda Pad() when no padding is needed. * Optimize cuda scatter() on 2D compatible. (#2628) * Optimize cuda scatter() on 2D compatible. * Add some comments. * fix build error for ARM (#2648) * Improve performance of resize() in Nearest mode (#2626) Special treatment for 2D, check same size as input image. And in 2d kernel, template use_expolation. * Fix memory exception in Layer Norm Fusion (#2644) * Windows CI changes(#2650) * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Revert "Windowsai without fi (#2701)" This reverts commit 285d4c85ff5c4e265f963208170304ef3461e684. * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Deref unique pointer for session_state * send shutdown event when dll is unloaded and EvaluationStop, SessionC… (#2704) * send shutdown event when dll is unloaded and EvaluationStop, SessionCreationStart Events. * Add EvalutationStart Event * add comment * use correct type for for loop (#2755) * ARM CI (#2759) * Set ARM agent pool * Set CMake generator to VS 2019 in ARM * Use system-wide CMake instead of custom version Our custom version is too old for VS 2019 * Use DML and build shared lib in ARM CI * Restore nuget packages in ARM CI * Disable DML * Refactor ARM debug/release builds * Use system packaged Python version * Remove hardcoded Python path * Downgrade Python to 3.7 for build * Remove explicit CMake path * Fix invalid JSON in cgmanifest.json (#2760) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * Remove usage of WHOLEARCHIVE in WinML CMake and add WinMLAdapterFactory (#2726) * Remove usage of WHOLEARCHIVE in WinMLAdapter CMake and add WinMLAdapterFactory * PR feedback, no need for dll(export) since using def file * PR comments * Small comment in gen_def.py * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * Move winml_provider_factory.h to proper location (#2801) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Filter CPU case for IsFloat16Supported (#2802) * Merge fixes * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * Fix warnings that cause build to fail * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * Make DML include path global (#2882) * Make DML include path global * Add generated cppwinrt headers to winml_lib_common * Integrate changes to WindowsAI to make ADO Build (#2886) * Revert "CMake cross-generator fixes (#2790)" This reverts commit dbe7d97fa1ab155f1309bced87199527e8f35bd2. * add additional suppress warning in onnx_proto * ignore /wd4996 warning * DML execution provider fixes * Revert "Revert "CMake cross-generator fixes (#2790)"" This reverts commit 1ae7b4bcbc02edc881ad28685da98e095dfceb17. * Update func signature of custom op function overloads * common devicehelpers fixes * Add pch.h for winml_lib_common * re-add winml_lib_common_dir/inc to include path for winml_adapter * User/orilevari/dml redist shared folder (#2890) * move dml nuget package directory up one level to make it shared between build flavors * Merge conflict fix * Revert "Merge conflict fix" This reverts commit 142fa72cf9ce4344ad717b50b7ea2b8582aadc7c. * Revert "Merge remote-tracking branch 'origin/master' into windowsai" This reverts commit 6e2126d46e5e5f564d65da37dd4f70c93dd81165, reversing changes made to b3f5583dc9249834b947c8ea905f6a98060d5bd6. * Make winml_test_common free of test macros (#2902) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Don't generate files outside the build root (#2914) * Don't generate files outside the build root * Add onnxruntime_EXTERNAL_DEPENDENCIES to WinML * Add DML depedency on RESTORE_PACKAGES * User/orilevari/fix yaml merge bugs (#2918) * Add winml test source parameter into cmake function (#2919) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Add arguments to unittest cmake functions * remove comment * Revert "Revert "Merge remote-tracking branch 'origin/master' into windowsai"" This reverts commit ade5abe72a4234fdbc3623093c61c02c6b0bdc26. * Fix breaks from merge with ORT master * Brianma/linux (#2917) * don't include windows.h in cross-plat header * add default case for switch statement * signed/unsigned mismatch fix Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> * User/sheilk/winml adapter c api (#2891) * Create winml adapter c api * fix build * make it build * move adapter into onnxruntime core/session * entry point not exported * minor changes * make model metadata work * make tests pass * implement all the model reflection apis on the adapter c abi * update the new ort interface to create a lotus ennvironment with a logging sink * start adding ort env * move all winml code into adapter folder/lib to isolate it * ensure a single logging manager at a time * start refactoring session * refactor session creation interface * add cpu and dml session option methods to adapter * finish session init * stub out interfaces in ort lib to perform similar mechanics of iinference session * enable profiling, and enable schema override * update session register graph transformers * turn back on custom registry for custom ops * Add sync api * add last c api stubs * should build... but all feature values are broken since this is in flight to moving all implementation details into ivalue * remove ep adapter header * Implement DML execution provider functions from adapter (#2846) * Implement DML execution provider functions from adapter * Use functions in OnnxruntimeEngine.cpp * make map/sequence type_infos freeable, and start implementing ivalue * make it build again * implement value methods * implement remaining methods * remove com adapter abi * check dml session * cache the allocator on ivalue * check if resource is cpu/gpu when access its mutable data * update tensor * mismatched parentheses * fix tensor base and binding obj * it evaluates tensors! sometimes... * minor fixes * enable gpu evals * wrapper all existing winml adapter apis with API_IMPL to try catch (#2854) * update winml... tensor strings are broken, need to template tensorbase to do different things for strings * make tensor strings work with 2 copies in/2 copies out * Fix tensor string and allocator bug * make maps work again... needs some fixes still * Make it build! * enable map inputs * map outputs * unbound outputs for sequences and maps * User/xianz/merge windowsai (#2883) * Packaging pipeline changes for VS 2019 (#2711) * Tiny fix to codegen * Simplify cache implementation and avoid static variables that may carry over between models * Extend DML kernels (#2641) * Additional DML operators * Check unsupported attributes and inputs * Address PR comments * Add kernel capability function used for partitioning, and re-enable stride-based int64 support based on value range * Fix test failures * Build fix * PR comments * Update Nuphar tutorial notebook (#2721) 1. Reflect int8 GEMV improvements for multi-threading from #2696 2. Add notes on multi-threading control using OpenMP 3. Add samples of running multi-isa AOT, and show int8 GEMM differences between AVX and AVX2 4. Add rnn_benchmark example to resolve #1993 * Add schema for new Qops (#2611) * Add schema for new Qops * adding shape inference + qlinearaveragepool * plus review comments * plus review comments * updates per review comments * plus review comments * [server] Add supposed for model_name and model_version as cli parameter (#2708) * remove 64bit warning message from python validation. (#2727) * MLAS: ARM64 build fix (#2734) fix bad usage of vreinterpret to cast vector element types * Fix broken python docs links (#2740) * Fix build on Mac OS (#2731) mac os ld doesn't support --while-archive, correct option is -all_load * fix ngraph wheel (#2737) * fix ngraph wheel 1.1.0 onnxruntime_ngraph wheel doesn't work * remove libdnnl.so in nGraph Libs * make it easy to compare * Split onnxruntime server to a separated folder (#2744) * Fix build for Python 3.8 (#2747) * Fix build for Python 3.8 * Update protobuf to 3.11.2 (#1928) Update protobuf to 3.11.2 (#1928) * Change default optimization level to All (from Basic) (#2745) * change default optimization level to All (from Basic) * fix test * fix c# test * Update numpy to 1.18 (#2758) * Update numpy to 1.18 * Pipeline changes for python 3.8 (#2753) 1. Pipeline changes for python 3.8 2. Fix a regression in setup.py which was just introduced in the previous commit. Please notice, we still haven't made python 3.8 + Windows + CUDA work. * Add basic stacktrace output for posix debug builds. (#2749) * [NupharEP] fix a race condition when multiple sessions running different models concurrently (#2772) * Revert "Change default optimization level to All (from Basic) (#2745)" This reverts commit 56bb503c2f26474b6613bcb2a198691a11dcef38. * Fix typo in error message (#2736) * Rename MKL-DNN to DNNL to fix broken link (#2730) * Fix nightly build version number issue * Pass BUILD_BUILDNUMBER to linux docker * Disable featurizers in python packages * Import more featurizers (#2781) Make kernels non-template. Add input constraint for learnt data. Add min_max_scalar_transformer, robust_scalar_transformer, inputation_marker_transfomer, label_encoder_transformer, missing_dummies_transformer along with tests. Advance Featurizers library commit. * Implement a more stable softmax (#2715) * Implement a more stable SoftMax e^x is represented as infinity if x is large enough, like 100.f. Infinity divided by Infinity is a NAN. Thus, softmax gets a NAN if one or more item are large enough. A math transform as below is leveraged to get a stable softmax: e^xi/(e^x1 + ...e^xn) = e^(xi - max) / (e^(x1 - max) + ... + e^(xn - max)) And for convenience, force max to 0.f if all xi are negative * Contributing: Fix a typo (#2784) * ACL EP GEMM improvements (#2780) When it is posible we use a fully connected layer instead of the gemm implementation. This will let the library use the best implementation based on the input data. * ACL EP convolution improvements (#2774) Added the optimized implementation for depthwise convolution for both ACL v19.02 and ACL 19.05. Also the pointwise convolution seems to be more optimal in the CPU implementation so we opted for that instead. * Add script for release Nuget validation (#2719) * Initial commit * Nits * Disable a test temporarily * Change working directory * Test * Add download python step * Test update * More changes * Fix space issue * Fix * Verify nuget signing * Fix * Spaces * PR feedback * Nit * Fix * Fix * Remove temporary changes * add uint8 support to where op (#2792) * Improve bert optimization script: (#2712) (1) Move input int64=>int32 conversion to embed layer fusion. (2) Output epsilon attribute for LayerNormalization fusion. * add session creation time cost. (#2798) * ML.NET team needs featurizers within a package (#2789) Add auto ml featurizers to Windows, MacOS as well as to GPU packaging-pipelines. * Initialize max of softmax with lowest of float (#2786) * MLAS: update SGEMM threading parameters (#2808) * add interface to copy batch tensors. (#2807) * add interface to copy batch tensors. * onnxruntime * speed up Windows TRT CI (#2811) * don't run cuda tests if building with tensorrt * remove unnecessary build options for win trt ci * refactor win gpu tensorrt ci yml * --numpy_version=1.17 * update * update * azcopy and cuda path * Update test data (#2356) * Add timeseries imputer transformer featurizer kernel (#2813) Make kernels non-template. Add input constraint for learnt data. Fixup tests. Add two more featurizers along with tests. Tests fail. min_max_scalar_transformer robust_scalar_transformer Fix tests serialized stream by prepending version bytes. Add inputation_marker_transfomer and the test. Fix up float/double type designations. Added label_encoder_transformer along with a test. string_throw case is broken at the momement. Fix labelencodertransfomer_test.cc string_throw case Rename maxabsscalertransformer_test.cc Add MissingDummiesTransformer along with the test. Update manifest. Add TimeSeriesImputerTransformer definition, implementation and tests * Fix memory leak in TRT (#2815) * fix memory leak issue * revert EP_FAIL on enueueV2 * Add manifest missing comma * Run static code analyzer on most of our code (#2817) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * update quantization doc (#2783) * update documentation for quantization script * plus some spell corrections * Filter CPU case for IsFloat16Supported (#2802) * update default optimization level + fix gemm_activation fusion (#2791) * update defualt optimization level + fix gemm_activation fusion * fix typo * add unit test and incorporate review comments * fix test comment * Fix dnnl wheel package name (#2823) * Append '-dnnl' to whl package name when --use_dnnl * Update build.py * Update Ubuntu & TensorRT version in README (#2820) Dockerfile.tensorrt is using nvcr.io/nvidia/tensorrt:19.09-py3 as base Image, update Ubuntu and TensorRT version according to https://docs.nvidia.com/deeplearning/sdk/tensorrt-container-release-notes/rel_19-09.html#rel_19-09 * Merge fixes * Add OneHotEncoder and HashOneHotEncoder kernels. (#2830) Add defs and imlementation for OneHotEncoders, adjuist date_time_transformer kernel and test. Add OneHotEncoder kernel test. Add HashOneHotVectorizerTransformer unit test. This does not link due to multiple definitions of functions that are included into header from a CPP file. * Upgrade gtest to the latest version (#2827) WinML would like to update the googletest submodule. They want some newer features (namely GTEST_SKIP to skip tests programmatically and be able to skip entire fixtures easily) and would need to update the submodule version. However, because the new version of code hit a bug in gcc, even though the bug is already fixed in the latest gcc but we're using gcc 4.8.x and it won't get patched for the bug, so we have to do a compromise, change our code a little bit to make it work. The gcc bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=51213 * Add support for int64_t for topk CPU. Fixes github issue #2806. (#2833) * Ignore allocator type in ExecutionProviders allocator map. Make default initialization of OrtMemoryInfo more clearly invalid. (#2768) * Remove allocator type from the key comparison in ExecutionProviders. Remove usage of DummyArena as it's no longer necessary. * Fix x86 tests where arena allocator is disabled. Make initialization of OrtMemoryInfo clearer by adding Invalid enum value. * Make OrtValueNameIdxMap::MaxIdx more intuitive. * Convert ExternalProject Featurizers into git submodule (#2834) Add git submodule for Featurizer library. Update cmake to build for git submodule. * add domain check for nodes + update documentation (#2831) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Fix Linux CUDA nuget packaging pipeline break * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * update optimization doc for BERT related fusions (#2819) * Add bert related transformers to doc * Add execution provider and comment for bert optimizations * Add comment about accuracy impact of approximation * Fix warnings that cause build to fail * MLAS: enable threading for quantized GEMMs (#2844) * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * remove old winmladapter cpp Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * move sequence implementation into ort lib... still commented out... need to turn back on... * begin sequence implementation * make maps and sequences work * fix broken tests * remove dead code * misc cleanup * CR feedback * User/xianz/winml adapter c api (#2869) * wrapper all existing winml adapter apis with API_IMPL to try catch * Return HR or Throw for WinML adapter APIs if failed * undo macro wrapper for two places * Wrap error macros around ort apis, too. * address CR feedback #2 * add more api throw/return macros * Revert changes no longer needed * revert changes to cxx api * format winml lib.ort and winml adapter * remove static pheonix singleton Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * missing use_dml check in winml_adapter_session (#2930) * --use_dnnl flag was mangled in merge (#2931) * use dml macro not wrapping custom registry code (#2934) * Disable LNK4199 winml_dll to enable cuda builds (#2936) * Disable LNK4199 in winml_dll * linkler->linker * LearningModelSessionAPITestGpu.CreateSessionWithCastToFloat16InModel should return DXGI_ERROR_UNSUPPORTED when FP16 not supported (#2937) * Disable LNK4199 in winml_dll * linkler->linker * Need to return DXGI_ERROR_UNSUPPORTED when Model does not support fp16 * Publish build symbols (#2939) * Publish build symbols * Don't upload PDBs for .exe files * Make x86 build (#2943) * fix last remaining size_t/int64_t warnings->errors (#2948) * TensorString, Sequences and Maps use the first allocator, but should use the cpu default allocator. (#2952) * fix tensor string allcoator * clean up default allocator usage for strings in winml lib/api.ort Co-authored-by: Ryan Lai <ryalai96@gmail.com> * Handle tensor shape of zero (#2954) Co-authored-by: Ryan Lai <ryalai96@gmail.com> * CR feedback (#2970) * CR feedback * fix weird formatting on privacy readme * Add 'All rights reserved.' everywhere * readd all rights reserved to winml_provider_factory.h * remove extra space in comment * remove extra whitespace * fixes post master merge * remove winml from nuget gpu pipeline * set IR VERSION on generated_model in rnn_benchmark (#2972) * Fix slice conformance failures (#2908) Co-authored-by: Adrian Tsai <adtsai@microsoft.com> Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Paul McDaniel <paul_mcdaniel@hotmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Dwayne Robinson <fdwr@hotmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com>
2020-02-05 01:12:19 +00:00
[threading(both)]
[marshaling_behavior(agile)]
[dualapipartition(1)]
runtimeclass LearningModel : Windows.Foundation.IClosable
{
//! Loads an ONNX model from a StorageFile asynchronously.
[remote_async]
static Windows.Foundation.IAsyncOperation<LearningModel> LoadFromStorageFileAsync(Windows.Storage.IStorageFile modelFile);
//! Loads an ONNX model from a stream asynchronously.
[remote_async]
static Windows.Foundation.IAsyncOperation<LearningModel> LoadFromStreamAsync(Windows.Storage.Streams.IRandomAccessStreamReference modelStream);
//! Loads an ONNX model from a file on disk.
static LearningModel LoadFromFilePath(String filePath);
//! Loads an ONNX model from a stream.
static LearningModel LoadFromStream(Windows.Storage.Streams.IRandomAccessStreamReference modelStream);
//! Loads an ONNX model from a StorageFile asynchronously.
[remote_async]
[method_name("LoadFromStorageFileWithOperatorProviderAsync")] static Windows.Foundation.IAsyncOperation<LearningModel> LoadFromStorageFileAsync(Windows.Storage.IStorageFile modelFile, ILearningModelOperatorProvider operatorProvider);
//! Loads an ONNX model from a stream asynchronously.
[remote_async]
[method_name("LoadFromStreamWithOperatorProviderAsync")] static Windows.Foundation.IAsyncOperation<LearningModel> LoadFromStreamAsync(Windows.Storage.Streams.IRandomAccessStreamReference modelStream, ILearningModelOperatorProvider operatorProvider);
//! Loads an ONNX model from a file on disk.
[method_name("LoadFromFilePathWithOperatorProvider")] static LearningModel LoadFromFilePath(String filePath, ILearningModelOperatorProvider operatorProvider);
//! Loads an ONNX model from a stream.
[method_name("LoadFromStreamWithOperatorProvider")] static LearningModel LoadFromStream(Windows.Storage.Streams.IRandomAccessStreamReference modelStream, ILearningModelOperatorProvider operatorProvider);
//! The name of the model author.
String Author{ get; };
//! The name of the model.
String Name{ get; };
//! The namespace of the imported model operator set. All models implicitly import the default ONNX operator set.
String Domain{ get; };
//! A description of the model.
String Description{ get; };
//! The ONNX version assumed by the model.
Int64 Version{ get; };
//! The raw ONNX model provided <string,string> metadata.
Windows.Foundation.Collections.IMapView <String, String> Metadata{ get; };
//! All of the input features.
Windows.Foundation.Collections.IVectorView<ILearningModelFeatureDescriptor > InputFeatures{ get; };
//! All of the output features.
Windows.Foundation.Collections.IVectorView<ILearningModelFeatureDescriptor > OutputFeatures{ get; };
}
//! \enum LearningModelDeviceKind
//! \brief Defines the list of devices that can evaluate a machine learning model.
[contract(MachineLearningContract, 1)]
enum LearningModelDeviceKind
{
//! Let the system decide which device to use.
Default = 0,
//! Use the CPU to evaluate the model.
Cpu,
//! Use a GPU or other DirectX device to evaluate the model.
DirectX,
//! Use the system policy defined device for high performance.
DirectXHighPerformance,
//! Use the system policy defined device for minimum power.
DirectXMinPower
};
//! \class LearningModelDevice
//! \brief Create an instance specific to which device you want to evaluate the machine learning model on.
//! \namespace Windows.AI.MachineLearning
Merge windowsai (winml layering) into master (#2956) * Initial Commit * Merged PR 3985217: add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc (#2346) add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc and violating our OS layering requirements. We linked against onecoreuap_apiset.lib in VB so we will continue doing this, but I am still unsure why not to link against onecore instead since that is where we ship. However, since Sheil is the owner of this code we will wait to discuss with him before changing anything. * Initial changes for layering * more snipping to get core into ort * update build instructions to include --build_shared_lib (#2358) * update build instructions to include --build_shared_lib * fix line breaks * Task 23998197: add winml_lib_core into onnnxruntime.dll (#2368) * Task 23998197: add winml_lib_core into onnnxruntime.dll * PR feedback build break on perf_test * return proper error when the model path isn't found (#2391) * LearningModelSession is cleaned up to use the adapter, and parts of b… (#2382) this is a big PR. we are going to move it up to layer_dev , which is still a L3 so we are still safe to do work there agile. we are going to move this into the L3 so that ryan can start doing intergration testing. we will pause for a full code review and integration test result prior to going into the L2. >>>> raw comments from previous commits >>> * LearningModelSession is cleaned up to use the adapter, and parts of binding are. * moved everything in the winmladapter made it all nano-com using, WRL to construct objects in the ORT side. base interfaces for everythign for winml to call cleaned up a bunch of winml to use the base interfaces. * more pieces * GetData across the abi. * renamed some namepsace cleaned up OrtValue cleaned up Tensor cleaned up custom ops. everything *but* learnignmodel should be clean * make sure it's building. winml.dll is still a monolith. * model moved over. everything builds clean. step ! * weak ref comment * Layer dev paulm (#2408) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * Layer dev paulm (#2414) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * Layer dev paulm (#2423) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * Layer dev paulm (#2424) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * Layer dev paulm (#2425) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Layer dev paulm (#2426) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * fixed map and sequence when passing stl types across the ABI . found a leak in nvidia driver, but skipped it. all winmlapitests pass now * Moved SessionOptions over to the abi * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * Register WinML TraceLogging provider on Onnxruntime.dll (#2455) * Register WinML TraceLogging provider on Onnxruntime.dll * Add ifdef to make sure trace logging provider has telemetry option when LAYERING_DONE * No need for ifdef for TraceLoggingOptionMicrosoftTelemetry * PR feedback * Move etw registration into lotus environment constructor and deresgister in lotus environment destructor * Brianma/cpuwinml (#2466) * allow building winml cpu without dml. * Brianma/breaks (#2469) * fix some more breaks * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * move dml checks out of winml and into the adapter * better error handling * Brianma/fi (#2470) * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * fix bad merge * Checking in a staging checkpoint point so that Ryan can work with me in parrallel * build break. * Brianma/testfails (#2473) * add missing ir version to dictvectorizer-string.onnx * add missing ir version to relu.onnx * add missing ir version to zipmap*onnx * add IR version to manually generated models * remove an unnecessary ifdef dml * Brianma/windowsai fi (#2475) * update dockerfiles/README (#2336) * Make elementwise op run 4 items per thread (#2335) Description: Describe your changes. Make elementwise op run 4 items per thread unroll for loop to leverage ILP remove unnessary N==0 check inside elementwise GPU kernel Motivation and Context Why is this change required? What problem does it solve? It can improve the performance of GPU elementwise ops. ~2% performance gain on popular NLP bert model. If it fixes an open issue, please link to the issue here. * Add CUDA GatherElements kernel (#2310) * Updates * Update test * Update * Updates * nits * PR feedback * Update * Update * PR feedback * PR comments * Update * Fix build * Fix build * Nits * Fix * Layer Normalization Fusion (#2319) basic layer normalization transform * Add FastGelu Cuda Op for Gelu and Add bias fusion (#2293) * Add FastGelu cuda op * Add AddBiasGelu for experiment * Revert "Add AddBiasGelu for experiment" This reverts commit 5c1ee019858c657e6bb75887265cb85675626e5b. * Add bias * Add unit tests * update comment * update script * fix build error * update coding style * update for CR feedback Enable half2 optimization only when cuda arch >= 7.0 * move _Tanh to common.cuh * implement CPU contrib OP Attention (#2333) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanUnusedInitializers. (#2320) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanupUnusedInitializers. This means initializers that have been replaced during graph optimizations are not left in the GraphProto when we save an optimized model. * Handle edge case where a model has an unused initializer with matching graph input by also removing the graph input. * Use non-const iterators in std::find_if calls to make centos build happy. * Nuget pipeline changes (#2305) 1. refactor the pipeline, remove some duplicated code 2. Move Windows_py_GPU_Wheels job to Win-GPU-CUDA10. We'll deprecated the "Win-GPU" pool 3. Delete cpu-nocontribops-esrp-pipeline.yml and cpu-nocontribops-pipeline.yml 4. In Linux nuget jobs, run "make install" before creating the package. So that extra RPAH info will be removed * Cuda Reverse Sequence Op, maping types of same size using same template function. (#2281) * Set ElementType to String type of node metadata, instead of byte[] (#2348) * Set ElementType to String type of node metadata, instead of byte[] * Fix spacing * Introduce PrimitiveType into a Type System along with an integer constant (#2307) Improve perf by avoiding GetType<T>() calls. Introduce MLTypeCallDispatcher to switch on Input Type. Add Tensor IsType<T>() fast method. * Fix/test dim value of 0 handling in a couple of places (#2337) * Update the CUDA Where implementation broadcasting logic to handle a dim with value of 0. Add unit test Also add unit test for unary op with dim value of 0 * Exclude ngraph from Where test with 0 dim. * Openvino EP R3.1 onnxrt server (#2357) * onnxrt server with OVEP * onnxrt server with OVEP * Update Dockerfile.server.openvino * onnxrt server OVEP fix reviews * onnxrt server OVEP fix reviews * Implement cuda nonzero op. (#2056) Implement cuda nonzero op. * Direct use python numpy array's memory if already contiguous. (#2355) * Direct use python numpy array's memory if already contiguous. This could greatly improve performance for session with large input, like big image 1920x1080 fastrcnn, 30~40% speed up could be achieved. * Add test case enforce contiguous/non-contiguos numpy array as inputs. * Add helper to create output to minimize binary size. (#2365) Add ConstEigenTensorMap typedef so we don't unnecessarily const_cast the const input Tensor. * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS (#2369) * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS * update * Add Tracelogging for profiling (#1639) Enabled only if onnxruntime_ENABLE_INSTRUMENT is ON * test bidaf with nuphar for avx target (#2370) increase nuphar test coverage a bit * Fix a bug in TLS refcount that may destabilized CUDA CI (#2374) * update output size calculation for resize (#2366) * change how output size is calculated for resize op * add tests for ver 10 resize * Extend OneHot CPU kernel to support more types (#2311) * Extend OneHot CPU kernel to support input int64_t, depth int32_t, output float * Skip BERT before the test data fix is picked up * Fix bug with Slice. Need to pass in flattened input dimensions so the initial offset into the input is calculated correctly. (#2372) * Add opset 11 version of Split to CUDA ops (#2376) Organize the CUDA ops definitions so all the opset 10 and 11 parts are together (same setup used for CPU ops) * Layer Norm Fusion Fix (#2379) * layer norm fusion fix * Add input shape check in code and unit tests * Fuse Add + Gelu (#2360) Implement the transformer to fuse add + gelu Implement the accurate kernel * Skip layer norm transform (#2350) * skip layer normalization transformer * Another try to stabilize CUDA CI (#2383) The root cause seems to be failure in CUDA dealloc when tear down. cudaFree return code was ignored before, so should the debug check. * fix BUILD.md typo (#2375) build.py: error: argument --config: invalid choice: 'RelWithDebugInfo' (choose from 'Debug', 'MinSizeRel', 'Release', 'RelWithDebInfo') * Fixed compilation with ngraph (#2388) * Fix reuse logic in allocation planner. (#2393) * Fix reuse logic in allocation planner. * PR comments * Add helpful comments * Don't allow reuse across string tensors. * [NupharEP] Multiple optimizations (#2380) Fuse transpose into MatMul Implement Pow and constant scalar simplification Vectorize ReduceMean Improve symbolic shape inference Minor updates for better debugging in fused function name * Avoid using the default logger in the graph lib and optimizers (#2361) 1. Use the session logger if it is available. 2. Don't disable warning 4100 globally. We should fix the warnings instead of disabling it. * Change CUDA implementation of Transpose to support all fixed size tensor types (#2387) * Change CUDA implementation of Transpose to not use a typed kernel so we can support more types with minimum binary size. Add support for 8, 16, 32 and 64 bit types. Add unit tests. Add method so the implementation can be called directly (will be used by CUDA Scan very soon). * Disable TensorRT for MLFloat16 and int8 unit tests. * Address PR comment and add support for calling cublas implementation if type is mlfloat16. * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. (#2398) * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. * [NupharEP] force some low/zero cost ops to be inlined (#2409) * fix cross compile bug (#2415) * Minor optimization: if a node has already been placed, there's no need to find a kernel for it. (#2417) * Add Reshape Fusion (#2395) * Add reshape fusion * Add some comments * update comments * update comment format * update according to feedback * update for recent logger change * fix build error * (1) Support both input and output edges in find path in graphutils (2) Add a test case of only one constant initializer of Concat input. (3) Refactor ReshapeFusion class to allow add more subgraph fusion in the future. * fix error * (1) loose constraint on initializer: non constant is allowed for reshape fusion. (2) Change versions type to vector. (3) Add logging. (4) Return false when multiple output edges matched in FindPath. Add comments. * only allow one direction (input or output) in FindPath * [NupharEP] Update notebook and docker image (#2416) Add BERT squad in Nuphar tutorial Enhance speed comparsion readability * Fix the issue in matmul_add_fusion (#2407) Fix the issue in matmul_add_fusion If Muatmul + Add has shape [K] * [K, N], reset it to [1, K] * [K, N] will make the output shape to [1, N] will also requires a reshape on the output. Fix: just remove the shape reset to not fuse it. Add a negative test case for matmul+add fusion * feat(treeregressor): Update TreeEnsembleRegressor for type support (#2389) Updates the `TreeEnsembleRegressor` to allow for `double`, `float`, `int64`, and `int32` inputs to match the upstream specification. Signed-off-by: Nick Groszewski <nicholas.groszewski@capitalone.com> * onnxrt server documentation update (#2396) * Added support for Pad-2 operator in OpenVINO-EP (#2405) * Add CUDA If operator. (#2377) * Add CUDA If operator. Uses CPU operator for implementation. By adding a CUDA version the inputs/outputs (with the exception of the 'cond' input) stay on GPU, and no other logic is required to avoid a copy to CPU across the control flow node. * Improved documentation for onnxruntime::utils::SwapByteOrderCopy(), added precondition check. * Fix the type constraints on CUDA If operator to exclude strings. (#2431) * add Im2col<uint8_t> (#2438) * Adjust codegen vectorization width from target (#2439) * Adjust codegen vectorization width from target * Add CUDA Scan operator. (#2403) * Add Scan CUDA op. Uses CPU implementation for logic. Added some device specific functors for handling when data needs to be manipulated on a different device. Added ability to override the materialization logic in the OrtValue slicer so DML can plugin their handling. * Fix Windows GPU C API packaging pipeline failure (#2440) Fix Windows GPU C API packaging pipeline failure (#2440) * Correctly handle implicit inputs for fused nodes (#2390) * Correctly handle implicit inputs for fused nodes Previously, nuphar's partitioning function didn't include node's implicit inputs into the inputs list of MetaDef, and hence a crash was triggered in the onnx graph checker. This commit fixed the issue. Furthermore, it also fixed a related issue where we didn't add implicit inputs into graph_inputs_excluding_initializers_ in Graph::SetGraphInputsOutputs. the issue was that graph_inputs_including_initializers_ populated by SetInputs (e.g. called by FunctionImpl::FunctionImpl) may contain implicit inputs which were not of any node's initializers in the graph. Because they were not part of any initializers, these implicit inputs couldn't be visited by going through all nodes' inputs. Consequently, they would *not* be added into graph_inputs_excluding_initializers_. We fixed the issue by first copying the populated graph_inputs_including_initializers_ into graph_inputs_excluding_initalizers_, which then had both initializers and non-initializers as its initial content. Later, we erase initializers from the list. In this way, we can ensure all implicit inputs to remain in graph_inputs_excluding_initializers_. * refined comments and fixed duplicates Address CR by revisiting comments in terms of implicit inputs Also fixed an issue by skipping duplicates while copying inputs from graph_inputs_including_initializers_. * address CR explain why we need to collect nodes' implicit inputs * don't rely on pointer values for iterating std::set Previously, openvino relied on iterating a set of NodeArg pointers to construct inputs and outputs for a fused graph. It could cause non-determinism. The reason was that although iterating std::set by itself is stable, pointer values of NodeArgs may vary. Consequently, we could end up visiting the set's elements in different orders for different runs for the same test, which resulted in constructing inputs (and outputs) with different orders to the fused graph. For example, for the same test, we may have inputs [A, B] in some runs but inputs[B, A] in others. Let's use std::string as the key type to avoid such nondeterminism. This commit also added implicit inputs into meta->inputs while returning the capability from the openvino provider. * Fixed another latent issue in openvino's GetCapability function The issue was that we couldn't simply erase fused_inputs and fused_outputs while iterating the nodes. For example, an output NodeArg may have multiple uses, and it's wrong if we erase it from fused_outputs when we encounter only one of its uses as input. * Remove DeviceAllocatorRegistry class (#2451) Remove DeviceAllocatorRegistry class * CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library. * Parallel Gelu with ParallelFor (#2399) Parallel Gelu to get better performance for Gelu * Clean up build.py (#2446) * Pull the latest image before running docker build * Fuse SkipLayerNorm with Bias (#2453) Fuse SkipLayerNorm with Bias * Allow more than one invocation of CreateEnv in the same process. (#2467) * Allow more than one invocation of CreateEnv in the same process. * Fix centos build * Symbolic shape inference improvements: (#2460) * Symbolic shape inference improvements: - add a mode to guess unknown ops' output rank - add support for GatherND - add support for If - fix a bug in get_int_values when then tensor rank > 1D, by treating it as no sympy data - add symbol to literal merge when ONNX silently merges dims - fix a bug in Concat when input dim is 0 - fix a bug in ConstantOfShape that computed dim is not updated - add support for dynamic shape in ConstantOfShape - fix a bug in Loop output shape that loop iterator dim is not inserted at dim 0 - add support for dynamic padding in Pad - add support for dynamic shape in Reshape - add support for Resize with opset > 10, by treating output dims as dynamic - fix a bug in Slice when starts/ends are dynamic - restrict input model to opset 7 and above - make output model optional to avoid disk write when testing Run model tests for symbolic shape inference Reduce 2GB docker image size of nuphar * add additional test data set for nuget pipeline (#2448) * add SAS token to download internal test data for nuget pipeline * update azure endpoint * fix keyvault download step * fix variable declaration for secret group * fix indentation * fix yaml syntax for variables * fix setting secrets for script * fix env synctax * Fix macos pipeline * attempt to add secrets to windows download data * fix mac and win data download * fix windows data download * update test data set url and location * Revert "Brianma/windowsai fi (#2475)" This reverts commit 5780b864a15513fda4eadbfc2b5345fefe70b5ec. * Add scenario tests (#2457) * Add scenario tests * Remove TODO from model license * Add winml_api test dependency * fix model load test. fi from master changed the constructor (#2483) * make api tests all pass (#2486) * fix bad merge * fix bad model merge * Layer dev paulm (#2492) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * Rename ambiguous header (#2489) * fix one more missing IR version model (#2500) * add missing IR version to 4 more models used by scenario tests (#2501) * Add CLI parameters to test runner, build WinML in ARM and x86 CI (#2479) * Support test parameters through CLI arguments * Add WinML do Windows x86/ARM CI builds * Code style fixes * Update googletest Remove GPUTEST macros everywhere now that GTEST_SKIP is supported * Refactor main.cpp * Build scenario tests without DML * Link scenario tests to DML when it's enabled (#2502) * Layer dev release pipeline (#2488) Adds winml binaries to existing cpu nuget package, and creates new gpu dml nuget package with winml binaries and DML EP. * Layer dev paulm (#2506) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * Remove usage of IOBinding in WinML and use C_API Run method (#2504) * remove usage of iobinding * Change data structure to use vector of Ort::Values * Polish bind input / output * Use C APIrun method * Update providers on evaluate getresults * Remove run and IObinding interface from WinMLAdapter * Remove use of IObinding * bind unbound outputs code moved to learningmodelbinding * clean up unneeded istensor adapter function * Fix comment * Check if session is closed before binding and clearing * PR feedback * Layer dev paulm (#2507) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * Make tests dependend on winml_dll (#2509) * add dml binaries to DirectML package and be more explicit about condition variables (#2520) * re-enable warnings for winml builds and fix the warnings that were hiding (#2526) * turn devmode back on for winml builds * fix some warnings. include protobuf in a way that disables some warnings * undo protobufhelpers changes and just ignore 4100 errors in pb code * attempt to isolate protobufhelpers errors * add template specialization for getting tensor proto data * Layer dev paulm (#2533) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * Spawn child process to run DeviceLostRecovery scenario test (#2530) * Spawn child process to run DeviceLostRecovery scenario test * Layer dev paulm (#2536) ori said yes * add missing namespace to winml_trace_logging_provider in lotusenvironment.h (#2542) * Handle exception thrown from all apis in WinMLAdapter (#2539) * various changes to unblock windowsai ADO build * Fix custom ops scenario tests (#2562) * Do not shutdown protobuf after ort environment gets destroyed. Lazy load lotus environment first time it is needed * comment typo * pr comment about calling phoenix singleton * Make lotus_environment static in winmladapter * Layer dev paulm (#2567) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * making windowsAI azure dev ops work. * code review comments. * revert changes * Cmake and preprocessor fixes that where uncovered by building on agents without DML available via SDK * Layer dev dml delayload (#2580) * Brianma/cpu (#2583) * don't include dml stuff in cpu builds * tests that link the image lib also need the telemetry lib now * Throw Winml_err_invalid_binding if binding gpu resource on cpu device (#2589) * Throw Winml_err_invalid_binding if binding gpu resource on cpu device * PR comments. No need to query executionprovider for is gpu device * User/xianz/ortthrow (#2596) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * User/xianz/ortthrow (#2599) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * add status error message * Remove uwp onsuspending winrt call because logruntimeperf is getting removed (#2630) * User/xianz/dedup telemetry (#2631) * investigate duplication of telemetry in winml and ort * remove winml telemetry events * telemetry executionProviderEvent * remove unneccessary file and refactor code little bit * Revert back TelemetryEvent, which send up ETW event. * merge changes from layer_dev to windowsai (#2638) * Remove underscore from googletest names (#2616) * Fix leaking memory allocator Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24278761 and https://microsoft.visualstudio.com/OS/_workitems/edit/24330198 * Explicitly initialize Ort::Value with nullptr * Cache WinML adapter * bad merge * define private version of dxcore enum that is added in 19H1 SDK. (#2654) * add comment for explaning private definition of dxcore d3d feature level ennum value. (#2672) * do not package directml.pdb for redist packages. (#2676) * Fix leaking operator registry (#2645) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24354916 * User/orilevari/windowsai master merge (#2674) merge resolutions included pulling in telemetry logic that was merged to master and not windowsai and dereferencing InferenceSession::sessionstate now that it is a unique pointer * Delete Ort Allocator in LearningModelBinding (#2653) * Delete OrtAllocator in LearningModelBinding * PR comments to make Ort::Allocator a smart pointer * Small comment change * PR feedback to clean up code * PR feedback on move semantics * Clean up std::move * Fix memory leaks (#2679) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24356109, https://microsoft.visualstudio.com/OS/_workitems/edit/24388361 and https://microsoft.visualstudio.com/OS/_workitems/edit/24388596 * various changes to properly organize and skip GPU tests. For now for No DML builds we will not run GPU tests at all. In the future we should adapt the tests to expect the appropiate errors. (#2695) * Windowsai without fi (#2701) * Disable Attention fusion tests when DISABLE_CONTRIB_OPS is defined (#2529) * Setup java ci (#2528) * Add provision in ORT for session options to be parsed when available via model file (#2449) * Initial commit * Fix gitmodules * Nits * Nits * Updates * Update * More changes * Updates * Update * Some updates * More changes * Update * Update * Merge * Update * Updates * More changes * Update * Fix nits * Updates * Fix warning * Fix build * Add comment * PR feedback * PR feedback * Updates * Updates * Update * More changes * Fix build break * Comment test for now * Updates * Updates * PR feedback * Updates * Nits * Add tests * Fix build * Fix build * Fix build * Fix build break * Fix build * Nits * PR feedback * More change * Expose GetSessionOptions in pybind logic and add unit test for python * Fix build * PR feedback * PR feedback * Revert "Disable thread pool creation when enabled OpenMP (#2485)" (#2535) This reverts commit 7c7d5a149c9ed52eec67304bae5c4b132166a8a1. * Add dynamic shape support in TensorRT execution provider (#2450) * remove onnx-tensorrt submodule * add new onnx-tensorrt submodule (experiment) for trt6 * update engine build for trt6 * update compile and compute for tensorrt6.0 * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * switch to onnx-tensorrt master for TensorRT6' * Update tensorrt_execution_provider.cc * Handle dynamic batch size and add memcpy in TensorRT EP * update test cases * Update tensorrt_execution_provider.cc * update onnx-tensorrt submodule * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.ubuntu_tensorrt * Update run_dockerbuild.sh * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update concat_op_test.cc * Update tensorrt_execution_provider.cc * Upgrade TensorRT to version 6.0.1.5 * Update onnxruntime_providers.cmake * Update CMakeLists.txt * Update reduction_ops_test.cc * Update install_ubuntu.sh * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.tensorrt * Update BUILD.md * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update onnxruntime_providers.cmake * Update install_ubuntu.sh * Update install_ubuntu.sh * Update gemm_test.cc * Update gather_op_test.cc * Update CMakeLists.txt * Removed submodule * update onnx-tensorrt submodule * update header file * Removed submodule * add submodule onnx-tensorrt kevin's branch shape-test' * add debugging code * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * merge master * Removed submodule * update onnx-tensorrt submodule * add more changes for dynamic shapes * Update tensorrt_execution_provider.cc * update for dynamic shape * update dynamic shape processing * fix logger issue * remove submodule onnx-tensorrt * add submodule onnx-tensorrt * add env variable min_subgraph_size * remove redundency * update document * use onnxruntime::make_unique * fix multi-run issue * remove some tests to save CI build time * Add dynamic shape test * Update TensorRT-ExecutionProvider.md * Add example of running Faster R-CNN model on TensorRT EP * Add more details on env variables * update environment variables * Update tensorrt_basic_test.cc * Update model tests * Update tensor_op_test.cc * remove --use_full_protobuf * Update build.py * User/xianz/telemetry (#2458) * enabme telemetry * enable telemetry * set enable telemetry as default * for debugging * remove log and set disable telemetry as default back * delete private file while testing * resolve comment: mainly add license header, rename macro and update docs * rewording in privacy.md * Fix integer overflow in cuda NonMaxSuppression implementation (#2540) * add test case that should pass but fail * fix nms * extract int_max_output_boxes_per_class * Introduce container type runtime checks and other improvements (#2522) Rework TensorSeq in a manner consistent with Tensor and SparseTensor in terms of type system setup. Reduce templating. Introduce helpers to ensure the same data type. Make OrtValue __dtor not virtual. Introduce ContainerChecker * Fix C API tests for centos and mac (#2544) * change c++14 to c++11 * add ld lib path for centos * enable csharp tests on macos * fix C API test on MacOS + fix manylinux dotnet install * fix manylinux dotnet install * fix lib link * Add back executable bit to build.py * Fix a bug handling negative begin pad values in Pad op (#2550) * Fix bug in Pad op * Update * DNNL CMAKE update (#2548) * Fix android build (#2558) * Update win-x86-ci.yml (#2557) Fix build pipeline break * Re-enable Windows C# tests (#2564) * disable onnx_test_runner -x invocations for dnnl (#2568) * Allow sequence length to be symbolic (#2559) * setup java ci mac (#2570) * make layernorm fusion to support opset 11 (#2545) * Fix a warning found in the latest VS release * Add more check on SkipLayerNorm and BiasGelu fusion (#2574) * Fix file not found error during docker build. (#2569) * Add ConvTranspose1D (#2578) * Ryanunderhill/packagename test (#2582) * [Nuphar EP] fixes for some object detection models (#2581) Update notebook tutorial with multi-threaded int8 GEMM from #2517 * EmbedLayerNormalization Fusion Improvement (#2553) Embedding layer norm fusion improvements - add more checks * Update version (#2584) * Temporarily exclude vgg19 test from Python backend test 1. temporarily exclude vgg19 test which comsumes too much memory, run out of memory on Upsquared device. Single test pass for vgg19, need furture investigation (#2588) 2. Update docker file to decrease the docker image size * Update docs for Android NNAPI EP (#2586) * Fix lto bug for protobuf and ubuntu * add path to build dir before test run (#2590) * Add missig env variables for mac pipeline test (#2595) * Fixed an issue in updating realized dims (#2597) when we update realized dims for scan's output, the sliced axis also needs to be inclusive, i.e. we should check with "dim >= insert_inclusive_axis", because the offset in the symbols are based on Scan sugraph. Otherwise, we would end up with shape mismatch later. * Java API for onnxruntime (#2215) * Add support for opset 11 in reshape fusion (#2592) Support opset verion 11 in reshape fusion * Rename automl python tools folder to featurizer_ops. (#2593) * Support opset 11 subgraph of Squad model in Embed Layer Normalization (#2605) Support opset 11 Squad model that is exported from PyTorch nightly. The embed layer uses Range op which is missed in the transformer. * symbolic shape inference: fix warnings in GPT-2 model (#2608) And revise nuphar perf test on BERT squad * Dump subgraph ID and fused graph ID (#2607) * Dump subgraph ID and fused graph ID Dump subgraph ID and fused graph ID for better debugging * Remove local static fused_count added a field global_fused_count_ to NupharExecutionProvider class * EmbedLayerNormalization Fusion For Dynamic Squad Model Opset 10 (#2613) Support subgraph of SQuAD model exported from pytorch with dynamic input axes * Allow providers to be set for InferenceSession at construction (#2606) * Remove unnecessary parameter in some places in GatherElements implementation (#2612) * Remove unnecessary parameter in some places * Update * Update * Make sure fenced tensor could not reuse other tensor. (#2561) Fix random error caused by this. * Improve Embed Layer Norm Fusion for SQuAD with static input shape (#2621) * fix float16 comparison in initializer (#2629) * epsilon attribute for layernormalization fusion (#2639) * removed unnecessary batch file and fix path (#2640) * Add shape inference to ConvTransposeWithDynamicPads schema (#2632) * Improve cuda expand() opeator's performance. (#2624) * Cuda pad optimize when no padding is needed. (#2625) * Shortcut cuda Pad() when no padding is needed. * Optimize cuda scatter() on 2D compatible. (#2628) * Optimize cuda scatter() on 2D compatible. * Add some comments. * fix build error for ARM (#2648) * Improve performance of resize() in Nearest mode (#2626) Special treatment for 2D, check same size as input image. And in 2d kernel, template use_expolation. * Fix memory exception in Layer Norm Fusion (#2644) * Windows CI changes(#2650) * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Revert "Windowsai without fi (#2701)" This reverts commit 285d4c85ff5c4e265f963208170304ef3461e684. * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Deref unique pointer for session_state * send shutdown event when dll is unloaded and EvaluationStop, SessionC… (#2704) * send shutdown event when dll is unloaded and EvaluationStop, SessionCreationStart Events. * Add EvalutationStart Event * add comment * use correct type for for loop (#2755) * ARM CI (#2759) * Set ARM agent pool * Set CMake generator to VS 2019 in ARM * Use system-wide CMake instead of custom version Our custom version is too old for VS 2019 * Use DML and build shared lib in ARM CI * Restore nuget packages in ARM CI * Disable DML * Refactor ARM debug/release builds * Use system packaged Python version * Remove hardcoded Python path * Downgrade Python to 3.7 for build * Remove explicit CMake path * Fix invalid JSON in cgmanifest.json (#2760) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * Remove usage of WHOLEARCHIVE in WinML CMake and add WinMLAdapterFactory (#2726) * Remove usage of WHOLEARCHIVE in WinMLAdapter CMake and add WinMLAdapterFactory * PR feedback, no need for dll(export) since using def file * PR comments * Small comment in gen_def.py * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * Move winml_provider_factory.h to proper location (#2801) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Filter CPU case for IsFloat16Supported (#2802) * Merge fixes * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * Fix warnings that cause build to fail * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * Make DML include path global (#2882) * Make DML include path global * Add generated cppwinrt headers to winml_lib_common * Integrate changes to WindowsAI to make ADO Build (#2886) * Revert "CMake cross-generator fixes (#2790)" This reverts commit dbe7d97fa1ab155f1309bced87199527e8f35bd2. * add additional suppress warning in onnx_proto * ignore /wd4996 warning * DML execution provider fixes * Revert "Revert "CMake cross-generator fixes (#2790)"" This reverts commit 1ae7b4bcbc02edc881ad28685da98e095dfceb17. * Update func signature of custom op function overloads * common devicehelpers fixes * Add pch.h for winml_lib_common * re-add winml_lib_common_dir/inc to include path for winml_adapter * User/orilevari/dml redist shared folder (#2890) * move dml nuget package directory up one level to make it shared between build flavors * Merge conflict fix * Revert "Merge conflict fix" This reverts commit 142fa72cf9ce4344ad717b50b7ea2b8582aadc7c. * Revert "Merge remote-tracking branch 'origin/master' into windowsai" This reverts commit 6e2126d46e5e5f564d65da37dd4f70c93dd81165, reversing changes made to b3f5583dc9249834b947c8ea905f6a98060d5bd6. * Make winml_test_common free of test macros (#2902) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Don't generate files outside the build root (#2914) * Don't generate files outside the build root * Add onnxruntime_EXTERNAL_DEPENDENCIES to WinML * Add DML depedency on RESTORE_PACKAGES * User/orilevari/fix yaml merge bugs (#2918) * Add winml test source parameter into cmake function (#2919) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Add arguments to unittest cmake functions * remove comment * Revert "Revert "Merge remote-tracking branch 'origin/master' into windowsai"" This reverts commit ade5abe72a4234fdbc3623093c61c02c6b0bdc26. * Fix breaks from merge with ORT master * Brianma/linux (#2917) * don't include windows.h in cross-plat header * add default case for switch statement * signed/unsigned mismatch fix Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> * User/sheilk/winml adapter c api (#2891) * Create winml adapter c api * fix build * make it build * move adapter into onnxruntime core/session * entry point not exported * minor changes * make model metadata work * make tests pass * implement all the model reflection apis on the adapter c abi * update the new ort interface to create a lotus ennvironment with a logging sink * start adding ort env * move all winml code into adapter folder/lib to isolate it * ensure a single logging manager at a time * start refactoring session * refactor session creation interface * add cpu and dml session option methods to adapter * finish session init * stub out interfaces in ort lib to perform similar mechanics of iinference session * enable profiling, and enable schema override * update session register graph transformers * turn back on custom registry for custom ops * Add sync api * add last c api stubs * should build... but all feature values are broken since this is in flight to moving all implementation details into ivalue * remove ep adapter header * Implement DML execution provider functions from adapter (#2846) * Implement DML execution provider functions from adapter * Use functions in OnnxruntimeEngine.cpp * make map/sequence type_infos freeable, and start implementing ivalue * make it build again * implement value methods * implement remaining methods * remove com adapter abi * check dml session * cache the allocator on ivalue * check if resource is cpu/gpu when access its mutable data * update tensor * mismatched parentheses * fix tensor base and binding obj * it evaluates tensors! sometimes... * minor fixes * enable gpu evals * wrapper all existing winml adapter apis with API_IMPL to try catch (#2854) * update winml... tensor strings are broken, need to template tensorbase to do different things for strings * make tensor strings work with 2 copies in/2 copies out * Fix tensor string and allocator bug * make maps work again... needs some fixes still * Make it build! * enable map inputs * map outputs * unbound outputs for sequences and maps * User/xianz/merge windowsai (#2883) * Packaging pipeline changes for VS 2019 (#2711) * Tiny fix to codegen * Simplify cache implementation and avoid static variables that may carry over between models * Extend DML kernels (#2641) * Additional DML operators * Check unsupported attributes and inputs * Address PR comments * Add kernel capability function used for partitioning, and re-enable stride-based int64 support based on value range * Fix test failures * Build fix * PR comments * Update Nuphar tutorial notebook (#2721) 1. Reflect int8 GEMV improvements for multi-threading from #2696 2. Add notes on multi-threading control using OpenMP 3. Add samples of running multi-isa AOT, and show int8 GEMM differences between AVX and AVX2 4. Add rnn_benchmark example to resolve #1993 * Add schema for new Qops (#2611) * Add schema for new Qops * adding shape inference + qlinearaveragepool * plus review comments * plus review comments * updates per review comments * plus review comments * [server] Add supposed for model_name and model_version as cli parameter (#2708) * remove 64bit warning message from python validation. (#2727) * MLAS: ARM64 build fix (#2734) fix bad usage of vreinterpret to cast vector element types * Fix broken python docs links (#2740) * Fix build on Mac OS (#2731) mac os ld doesn't support --while-archive, correct option is -all_load * fix ngraph wheel (#2737) * fix ngraph wheel 1.1.0 onnxruntime_ngraph wheel doesn't work * remove libdnnl.so in nGraph Libs * make it easy to compare * Split onnxruntime server to a separated folder (#2744) * Fix build for Python 3.8 (#2747) * Fix build for Python 3.8 * Update protobuf to 3.11.2 (#1928) Update protobuf to 3.11.2 (#1928) * Change default optimization level to All (from Basic) (#2745) * change default optimization level to All (from Basic) * fix test * fix c# test * Update numpy to 1.18 (#2758) * Update numpy to 1.18 * Pipeline changes for python 3.8 (#2753) 1. Pipeline changes for python 3.8 2. Fix a regression in setup.py which was just introduced in the previous commit. Please notice, we still haven't made python 3.8 + Windows + CUDA work. * Add basic stacktrace output for posix debug builds. (#2749) * [NupharEP] fix a race condition when multiple sessions running different models concurrently (#2772) * Revert "Change default optimization level to All (from Basic) (#2745)" This reverts commit 56bb503c2f26474b6613bcb2a198691a11dcef38. * Fix typo in error message (#2736) * Rename MKL-DNN to DNNL to fix broken link (#2730) * Fix nightly build version number issue * Pass BUILD_BUILDNUMBER to linux docker * Disable featurizers in python packages * Import more featurizers (#2781) Make kernels non-template. Add input constraint for learnt data. Add min_max_scalar_transformer, robust_scalar_transformer, inputation_marker_transfomer, label_encoder_transformer, missing_dummies_transformer along with tests. Advance Featurizers library commit. * Implement a more stable softmax (#2715) * Implement a more stable SoftMax e^x is represented as infinity if x is large enough, like 100.f. Infinity divided by Infinity is a NAN. Thus, softmax gets a NAN if one or more item are large enough. A math transform as below is leveraged to get a stable softmax: e^xi/(e^x1 + ...e^xn) = e^(xi - max) / (e^(x1 - max) + ... + e^(xn - max)) And for convenience, force max to 0.f if all xi are negative * Contributing: Fix a typo (#2784) * ACL EP GEMM improvements (#2780) When it is posible we use a fully connected layer instead of the gemm implementation. This will let the library use the best implementation based on the input data. * ACL EP convolution improvements (#2774) Added the optimized implementation for depthwise convolution for both ACL v19.02 and ACL 19.05. Also the pointwise convolution seems to be more optimal in the CPU implementation so we opted for that instead. * Add script for release Nuget validation (#2719) * Initial commit * Nits * Disable a test temporarily * Change working directory * Test * Add download python step * Test update * More changes * Fix space issue * Fix * Verify nuget signing * Fix * Spaces * PR feedback * Nit * Fix * Fix * Remove temporary changes * add uint8 support to where op (#2792) * Improve bert optimization script: (#2712) (1) Move input int64=>int32 conversion to embed layer fusion. (2) Output epsilon attribute for LayerNormalization fusion. * add session creation time cost. (#2798) * ML.NET team needs featurizers within a package (#2789) Add auto ml featurizers to Windows, MacOS as well as to GPU packaging-pipelines. * Initialize max of softmax with lowest of float (#2786) * MLAS: update SGEMM threading parameters (#2808) * add interface to copy batch tensors. (#2807) * add interface to copy batch tensors. * onnxruntime * speed up Windows TRT CI (#2811) * don't run cuda tests if building with tensorrt * remove unnecessary build options for win trt ci * refactor win gpu tensorrt ci yml * --numpy_version=1.17 * update * update * azcopy and cuda path * Update test data (#2356) * Add timeseries imputer transformer featurizer kernel (#2813) Make kernels non-template. Add input constraint for learnt data. Fixup tests. Add two more featurizers along with tests. Tests fail. min_max_scalar_transformer robust_scalar_transformer Fix tests serialized stream by prepending version bytes. Add inputation_marker_transfomer and the test. Fix up float/double type designations. Added label_encoder_transformer along with a test. string_throw case is broken at the momement. Fix labelencodertransfomer_test.cc string_throw case Rename maxabsscalertransformer_test.cc Add MissingDummiesTransformer along with the test. Update manifest. Add TimeSeriesImputerTransformer definition, implementation and tests * Fix memory leak in TRT (#2815) * fix memory leak issue * revert EP_FAIL on enueueV2 * Add manifest missing comma * Run static code analyzer on most of our code (#2817) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * update quantization doc (#2783) * update documentation for quantization script * plus some spell corrections * Filter CPU case for IsFloat16Supported (#2802) * update default optimization level + fix gemm_activation fusion (#2791) * update defualt optimization level + fix gemm_activation fusion * fix typo * add unit test and incorporate review comments * fix test comment * Fix dnnl wheel package name (#2823) * Append '-dnnl' to whl package name when --use_dnnl * Update build.py * Update Ubuntu & TensorRT version in README (#2820) Dockerfile.tensorrt is using nvcr.io/nvidia/tensorrt:19.09-py3 as base Image, update Ubuntu and TensorRT version according to https://docs.nvidia.com/deeplearning/sdk/tensorrt-container-release-notes/rel_19-09.html#rel_19-09 * Merge fixes * Add OneHotEncoder and HashOneHotEncoder kernels. (#2830) Add defs and imlementation for OneHotEncoders, adjuist date_time_transformer kernel and test. Add OneHotEncoder kernel test. Add HashOneHotVectorizerTransformer unit test. This does not link due to multiple definitions of functions that are included into header from a CPP file. * Upgrade gtest to the latest version (#2827) WinML would like to update the googletest submodule. They want some newer features (namely GTEST_SKIP to skip tests programmatically and be able to skip entire fixtures easily) and would need to update the submodule version. However, because the new version of code hit a bug in gcc, even though the bug is already fixed in the latest gcc but we're using gcc 4.8.x and it won't get patched for the bug, so we have to do a compromise, change our code a little bit to make it work. The gcc bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=51213 * Add support for int64_t for topk CPU. Fixes github issue #2806. (#2833) * Ignore allocator type in ExecutionProviders allocator map. Make default initialization of OrtMemoryInfo more clearly invalid. (#2768) * Remove allocator type from the key comparison in ExecutionProviders. Remove usage of DummyArena as it's no longer necessary. * Fix x86 tests where arena allocator is disabled. Make initialization of OrtMemoryInfo clearer by adding Invalid enum value. * Make OrtValueNameIdxMap::MaxIdx more intuitive. * Convert ExternalProject Featurizers into git submodule (#2834) Add git submodule for Featurizer library. Update cmake to build for git submodule. * add domain check for nodes + update documentation (#2831) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Fix Linux CUDA nuget packaging pipeline break * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * update optimization doc for BERT related fusions (#2819) * Add bert related transformers to doc * Add execution provider and comment for bert optimizations * Add comment about accuracy impact of approximation * Fix warnings that cause build to fail * MLAS: enable threading for quantized GEMMs (#2844) * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * remove old winmladapter cpp Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * move sequence implementation into ort lib... still commented out... need to turn back on... * begin sequence implementation * make maps and sequences work * fix broken tests * remove dead code * misc cleanup * CR feedback * User/xianz/winml adapter c api (#2869) * wrapper all existing winml adapter apis with API_IMPL to try catch * Return HR or Throw for WinML adapter APIs if failed * undo macro wrapper for two places * Wrap error macros around ort apis, too. * address CR feedback #2 * add more api throw/return macros * Revert changes no longer needed * revert changes to cxx api * format winml lib.ort and winml adapter * remove static pheonix singleton Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * missing use_dml check in winml_adapter_session (#2930) * --use_dnnl flag was mangled in merge (#2931) * use dml macro not wrapping custom registry code (#2934) * Disable LNK4199 winml_dll to enable cuda builds (#2936) * Disable LNK4199 in winml_dll * linkler->linker * LearningModelSessionAPITestGpu.CreateSessionWithCastToFloat16InModel should return DXGI_ERROR_UNSUPPORTED when FP16 not supported (#2937) * Disable LNK4199 in winml_dll * linkler->linker * Need to return DXGI_ERROR_UNSUPPORTED when Model does not support fp16 * Publish build symbols (#2939) * Publish build symbols * Don't upload PDBs for .exe files * Make x86 build (#2943) * fix last remaining size_t/int64_t warnings->errors (#2948) * TensorString, Sequences and Maps use the first allocator, but should use the cpu default allocator. (#2952) * fix tensor string allcoator * clean up default allocator usage for strings in winml lib/api.ort Co-authored-by: Ryan Lai <ryalai96@gmail.com> * Handle tensor shape of zero (#2954) Co-authored-by: Ryan Lai <ryalai96@gmail.com> * CR feedback (#2970) * CR feedback * fix weird formatting on privacy readme * Add 'All rights reserved.' everywhere * readd all rights reserved to winml_provider_factory.h * remove extra space in comment * remove extra whitespace * fixes post master merge * remove winml from nuget gpu pipeline * set IR VERSION on generated_model in rnn_benchmark (#2972) * Fix slice conformance failures (#2908) Co-authored-by: Adrian Tsai <adtsai@microsoft.com> Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Paul McDaniel <paul_mcdaniel@hotmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Dwayne Robinson <fdwr@hotmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com>
2020-02-05 01:12:19 +00:00
[contract(MachineLearningContract, 1)]
INBOX_ONLY([constructor_name("Windows.AI.MachineLearning.ILearningModelDeviceFactory", 9cffd74d-b1e5-4f20-80ad-0a56690db06b)])
INBOX_ONLY([static_name("Windows.AI.MachineLearning.ILearningModelDeviceStatics", 49f32107-a8bf-42bb-92c7-10b12dc5d21f)])
INBOX_ONLY([interface_name("Windows.AI.MachineLearning.ILearningModelDevice", f5c2c8fe-3f56-4a8c-ac5f-fdb92d8b8252)])
Merge windowsai (winml layering) into master (#2956) * Initial Commit * Merged PR 3985217: add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc (#2346) add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc and violating our OS layering requirements. We linked against onecoreuap_apiset.lib in VB so we will continue doing this, but I am still unsure why not to link against onecore instead since that is where we ship. However, since Sheil is the owner of this code we will wait to discuss with him before changing anything. * Initial changes for layering * more snipping to get core into ort * update build instructions to include --build_shared_lib (#2358) * update build instructions to include --build_shared_lib * fix line breaks * Task 23998197: add winml_lib_core into onnnxruntime.dll (#2368) * Task 23998197: add winml_lib_core into onnnxruntime.dll * PR feedback build break on perf_test * return proper error when the model path isn't found (#2391) * LearningModelSession is cleaned up to use the adapter, and parts of b… (#2382) this is a big PR. we are going to move it up to layer_dev , which is still a L3 so we are still safe to do work there agile. we are going to move this into the L3 so that ryan can start doing intergration testing. we will pause for a full code review and integration test result prior to going into the L2. >>>> raw comments from previous commits >>> * LearningModelSession is cleaned up to use the adapter, and parts of binding are. * moved everything in the winmladapter made it all nano-com using, WRL to construct objects in the ORT side. base interfaces for everythign for winml to call cleaned up a bunch of winml to use the base interfaces. * more pieces * GetData across the abi. * renamed some namepsace cleaned up OrtValue cleaned up Tensor cleaned up custom ops. everything *but* learnignmodel should be clean * make sure it's building. winml.dll is still a monolith. * model moved over. everything builds clean. step ! * weak ref comment * Layer dev paulm (#2408) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * Layer dev paulm (#2414) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * Layer dev paulm (#2423) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * Layer dev paulm (#2424) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * Layer dev paulm (#2425) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Layer dev paulm (#2426) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * fixed map and sequence when passing stl types across the ABI . found a leak in nvidia driver, but skipped it. all winmlapitests pass now * Moved SessionOptions over to the abi * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * Register WinML TraceLogging provider on Onnxruntime.dll (#2455) * Register WinML TraceLogging provider on Onnxruntime.dll * Add ifdef to make sure trace logging provider has telemetry option when LAYERING_DONE * No need for ifdef for TraceLoggingOptionMicrosoftTelemetry * PR feedback * Move etw registration into lotus environment constructor and deresgister in lotus environment destructor * Brianma/cpuwinml (#2466) * allow building winml cpu without dml. * Brianma/breaks (#2469) * fix some more breaks * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * move dml checks out of winml and into the adapter * better error handling * Brianma/fi (#2470) * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * fix bad merge * Checking in a staging checkpoint point so that Ryan can work with me in parrallel * build break. * Brianma/testfails (#2473) * add missing ir version to dictvectorizer-string.onnx * add missing ir version to relu.onnx * add missing ir version to zipmap*onnx * add IR version to manually generated models * remove an unnecessary ifdef dml * Brianma/windowsai fi (#2475) * update dockerfiles/README (#2336) * Make elementwise op run 4 items per thread (#2335) Description: Describe your changes. Make elementwise op run 4 items per thread unroll for loop to leverage ILP remove unnessary N==0 check inside elementwise GPU kernel Motivation and Context Why is this change required? What problem does it solve? It can improve the performance of GPU elementwise ops. ~2% performance gain on popular NLP bert model. If it fixes an open issue, please link to the issue here. * Add CUDA GatherElements kernel (#2310) * Updates * Update test * Update * Updates * nits * PR feedback * Update * Update * PR feedback * PR comments * Update * Fix build * Fix build * Nits * Fix * Layer Normalization Fusion (#2319) basic layer normalization transform * Add FastGelu Cuda Op for Gelu and Add bias fusion (#2293) * Add FastGelu cuda op * Add AddBiasGelu for experiment * Revert "Add AddBiasGelu for experiment" This reverts commit 5c1ee019858c657e6bb75887265cb85675626e5b. * Add bias * Add unit tests * update comment * update script * fix build error * update coding style * update for CR feedback Enable half2 optimization only when cuda arch >= 7.0 * move _Tanh to common.cuh * implement CPU contrib OP Attention (#2333) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanUnusedInitializers. (#2320) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanupUnusedInitializers. This means initializers that have been replaced during graph optimizations are not left in the GraphProto when we save an optimized model. * Handle edge case where a model has an unused initializer with matching graph input by also removing the graph input. * Use non-const iterators in std::find_if calls to make centos build happy. * Nuget pipeline changes (#2305) 1. refactor the pipeline, remove some duplicated code 2. Move Windows_py_GPU_Wheels job to Win-GPU-CUDA10. We'll deprecated the "Win-GPU" pool 3. Delete cpu-nocontribops-esrp-pipeline.yml and cpu-nocontribops-pipeline.yml 4. In Linux nuget jobs, run "make install" before creating the package. So that extra RPAH info will be removed * Cuda Reverse Sequence Op, maping types of same size using same template function. (#2281) * Set ElementType to String type of node metadata, instead of byte[] (#2348) * Set ElementType to String type of node metadata, instead of byte[] * Fix spacing * Introduce PrimitiveType into a Type System along with an integer constant (#2307) Improve perf by avoiding GetType<T>() calls. Introduce MLTypeCallDispatcher to switch on Input Type. Add Tensor IsType<T>() fast method. * Fix/test dim value of 0 handling in a couple of places (#2337) * Update the CUDA Where implementation broadcasting logic to handle a dim with value of 0. Add unit test Also add unit test for unary op with dim value of 0 * Exclude ngraph from Where test with 0 dim. * Openvino EP R3.1 onnxrt server (#2357) * onnxrt server with OVEP * onnxrt server with OVEP * Update Dockerfile.server.openvino * onnxrt server OVEP fix reviews * onnxrt server OVEP fix reviews * Implement cuda nonzero op. (#2056) Implement cuda nonzero op. * Direct use python numpy array's memory if already contiguous. (#2355) * Direct use python numpy array's memory if already contiguous. This could greatly improve performance for session with large input, like big image 1920x1080 fastrcnn, 30~40% speed up could be achieved. * Add test case enforce contiguous/non-contiguos numpy array as inputs. * Add helper to create output to minimize binary size. (#2365) Add ConstEigenTensorMap typedef so we don't unnecessarily const_cast the const input Tensor. * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS (#2369) * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS * update * Add Tracelogging for profiling (#1639) Enabled only if onnxruntime_ENABLE_INSTRUMENT is ON * test bidaf with nuphar for avx target (#2370) increase nuphar test coverage a bit * Fix a bug in TLS refcount that may destabilized CUDA CI (#2374) * update output size calculation for resize (#2366) * change how output size is calculated for resize op * add tests for ver 10 resize * Extend OneHot CPU kernel to support more types (#2311) * Extend OneHot CPU kernel to support input int64_t, depth int32_t, output float * Skip BERT before the test data fix is picked up * Fix bug with Slice. Need to pass in flattened input dimensions so the initial offset into the input is calculated correctly. (#2372) * Add opset 11 version of Split to CUDA ops (#2376) Organize the CUDA ops definitions so all the opset 10 and 11 parts are together (same setup used for CPU ops) * Layer Norm Fusion Fix (#2379) * layer norm fusion fix * Add input shape check in code and unit tests * Fuse Add + Gelu (#2360) Implement the transformer to fuse add + gelu Implement the accurate kernel * Skip layer norm transform (#2350) * skip layer normalization transformer * Another try to stabilize CUDA CI (#2383) The root cause seems to be failure in CUDA dealloc when tear down. cudaFree return code was ignored before, so should the debug check. * fix BUILD.md typo (#2375) build.py: error: argument --config: invalid choice: 'RelWithDebugInfo' (choose from 'Debug', 'MinSizeRel', 'Release', 'RelWithDebInfo') * Fixed compilation with ngraph (#2388) * Fix reuse logic in allocation planner. (#2393) * Fix reuse logic in allocation planner. * PR comments * Add helpful comments * Don't allow reuse across string tensors. * [NupharEP] Multiple optimizations (#2380) Fuse transpose into MatMul Implement Pow and constant scalar simplification Vectorize ReduceMean Improve symbolic shape inference Minor updates for better debugging in fused function name * Avoid using the default logger in the graph lib and optimizers (#2361) 1. Use the session logger if it is available. 2. Don't disable warning 4100 globally. We should fix the warnings instead of disabling it. * Change CUDA implementation of Transpose to support all fixed size tensor types (#2387) * Change CUDA implementation of Transpose to not use a typed kernel so we can support more types with minimum binary size. Add support for 8, 16, 32 and 64 bit types. Add unit tests. Add method so the implementation can be called directly (will be used by CUDA Scan very soon). * Disable TensorRT for MLFloat16 and int8 unit tests. * Address PR comment and add support for calling cublas implementation if type is mlfloat16. * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. (#2398) * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. * [NupharEP] force some low/zero cost ops to be inlined (#2409) * fix cross compile bug (#2415) * Minor optimization: if a node has already been placed, there's no need to find a kernel for it. (#2417) * Add Reshape Fusion (#2395) * Add reshape fusion * Add some comments * update comments * update comment format * update according to feedback * update for recent logger change * fix build error * (1) Support both input and output edges in find path in graphutils (2) Add a test case of only one constant initializer of Concat input. (3) Refactor ReshapeFusion class to allow add more subgraph fusion in the future. * fix error * (1) loose constraint on initializer: non constant is allowed for reshape fusion. (2) Change versions type to vector. (3) Add logging. (4) Return false when multiple output edges matched in FindPath. Add comments. * only allow one direction (input or output) in FindPath * [NupharEP] Update notebook and docker image (#2416) Add BERT squad in Nuphar tutorial Enhance speed comparsion readability * Fix the issue in matmul_add_fusion (#2407) Fix the issue in matmul_add_fusion If Muatmul + Add has shape [K] * [K, N], reset it to [1, K] * [K, N] will make the output shape to [1, N] will also requires a reshape on the output. Fix: just remove the shape reset to not fuse it. Add a negative test case for matmul+add fusion * feat(treeregressor): Update TreeEnsembleRegressor for type support (#2389) Updates the `TreeEnsembleRegressor` to allow for `double`, `float`, `int64`, and `int32` inputs to match the upstream specification. Signed-off-by: Nick Groszewski <nicholas.groszewski@capitalone.com> * onnxrt server documentation update (#2396) * Added support for Pad-2 operator in OpenVINO-EP (#2405) * Add CUDA If operator. (#2377) * Add CUDA If operator. Uses CPU operator for implementation. By adding a CUDA version the inputs/outputs (with the exception of the 'cond' input) stay on GPU, and no other logic is required to avoid a copy to CPU across the control flow node. * Improved documentation for onnxruntime::utils::SwapByteOrderCopy(), added precondition check. * Fix the type constraints on CUDA If operator to exclude strings. (#2431) * add Im2col<uint8_t> (#2438) * Adjust codegen vectorization width from target (#2439) * Adjust codegen vectorization width from target * Add CUDA Scan operator. (#2403) * Add Scan CUDA op. Uses CPU implementation for logic. Added some device specific functors for handling when data needs to be manipulated on a different device. Added ability to override the materialization logic in the OrtValue slicer so DML can plugin their handling. * Fix Windows GPU C API packaging pipeline failure (#2440) Fix Windows GPU C API packaging pipeline failure (#2440) * Correctly handle implicit inputs for fused nodes (#2390) * Correctly handle implicit inputs for fused nodes Previously, nuphar's partitioning function didn't include node's implicit inputs into the inputs list of MetaDef, and hence a crash was triggered in the onnx graph checker. This commit fixed the issue. Furthermore, it also fixed a related issue where we didn't add implicit inputs into graph_inputs_excluding_initializers_ in Graph::SetGraphInputsOutputs. the issue was that graph_inputs_including_initializers_ populated by SetInputs (e.g. called by FunctionImpl::FunctionImpl) may contain implicit inputs which were not of any node's initializers in the graph. Because they were not part of any initializers, these implicit inputs couldn't be visited by going through all nodes' inputs. Consequently, they would *not* be added into graph_inputs_excluding_initializers_. We fixed the issue by first copying the populated graph_inputs_including_initializers_ into graph_inputs_excluding_initalizers_, which then had both initializers and non-initializers as its initial content. Later, we erase initializers from the list. In this way, we can ensure all implicit inputs to remain in graph_inputs_excluding_initializers_. * refined comments and fixed duplicates Address CR by revisiting comments in terms of implicit inputs Also fixed an issue by skipping duplicates while copying inputs from graph_inputs_including_initializers_. * address CR explain why we need to collect nodes' implicit inputs * don't rely on pointer values for iterating std::set Previously, openvino relied on iterating a set of NodeArg pointers to construct inputs and outputs for a fused graph. It could cause non-determinism. The reason was that although iterating std::set by itself is stable, pointer values of NodeArgs may vary. Consequently, we could end up visiting the set's elements in different orders for different runs for the same test, which resulted in constructing inputs (and outputs) with different orders to the fused graph. For example, for the same test, we may have inputs [A, B] in some runs but inputs[B, A] in others. Let's use std::string as the key type to avoid such nondeterminism. This commit also added implicit inputs into meta->inputs while returning the capability from the openvino provider. * Fixed another latent issue in openvino's GetCapability function The issue was that we couldn't simply erase fused_inputs and fused_outputs while iterating the nodes. For example, an output NodeArg may have multiple uses, and it's wrong if we erase it from fused_outputs when we encounter only one of its uses as input. * Remove DeviceAllocatorRegistry class (#2451) Remove DeviceAllocatorRegistry class * CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library. * Parallel Gelu with ParallelFor (#2399) Parallel Gelu to get better performance for Gelu * Clean up build.py (#2446) * Pull the latest image before running docker build * Fuse SkipLayerNorm with Bias (#2453) Fuse SkipLayerNorm with Bias * Allow more than one invocation of CreateEnv in the same process. (#2467) * Allow more than one invocation of CreateEnv in the same process. * Fix centos build * Symbolic shape inference improvements: (#2460) * Symbolic shape inference improvements: - add a mode to guess unknown ops' output rank - add support for GatherND - add support for If - fix a bug in get_int_values when then tensor rank > 1D, by treating it as no sympy data - add symbol to literal merge when ONNX silently merges dims - fix a bug in Concat when input dim is 0 - fix a bug in ConstantOfShape that computed dim is not updated - add support for dynamic shape in ConstantOfShape - fix a bug in Loop output shape that loop iterator dim is not inserted at dim 0 - add support for dynamic padding in Pad - add support for dynamic shape in Reshape - add support for Resize with opset > 10, by treating output dims as dynamic - fix a bug in Slice when starts/ends are dynamic - restrict input model to opset 7 and above - make output model optional to avoid disk write when testing Run model tests for symbolic shape inference Reduce 2GB docker image size of nuphar * add additional test data set for nuget pipeline (#2448) * add SAS token to download internal test data for nuget pipeline * update azure endpoint * fix keyvault download step * fix variable declaration for secret group * fix indentation * fix yaml syntax for variables * fix setting secrets for script * fix env synctax * Fix macos pipeline * attempt to add secrets to windows download data * fix mac and win data download * fix windows data download * update test data set url and location * Revert "Brianma/windowsai fi (#2475)" This reverts commit 5780b864a15513fda4eadbfc2b5345fefe70b5ec. * Add scenario tests (#2457) * Add scenario tests * Remove TODO from model license * Add winml_api test dependency * fix model load test. fi from master changed the constructor (#2483) * make api tests all pass (#2486) * fix bad merge * fix bad model merge * Layer dev paulm (#2492) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * Rename ambiguous header (#2489) * fix one more missing IR version model (#2500) * add missing IR version to 4 more models used by scenario tests (#2501) * Add CLI parameters to test runner, build WinML in ARM and x86 CI (#2479) * Support test parameters through CLI arguments * Add WinML do Windows x86/ARM CI builds * Code style fixes * Update googletest Remove GPUTEST macros everywhere now that GTEST_SKIP is supported * Refactor main.cpp * Build scenario tests without DML * Link scenario tests to DML when it's enabled (#2502) * Layer dev release pipeline (#2488) Adds winml binaries to existing cpu nuget package, and creates new gpu dml nuget package with winml binaries and DML EP. * Layer dev paulm (#2506) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * Remove usage of IOBinding in WinML and use C_API Run method (#2504) * remove usage of iobinding * Change data structure to use vector of Ort::Values * Polish bind input / output * Use C APIrun method * Update providers on evaluate getresults * Remove run and IObinding interface from WinMLAdapter * Remove use of IObinding * bind unbound outputs code moved to learningmodelbinding * clean up unneeded istensor adapter function * Fix comment * Check if session is closed before binding and clearing * PR feedback * Layer dev paulm (#2507) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * Make tests dependend on winml_dll (#2509) * add dml binaries to DirectML package and be more explicit about condition variables (#2520) * re-enable warnings for winml builds and fix the warnings that were hiding (#2526) * turn devmode back on for winml builds * fix some warnings. include protobuf in a way that disables some warnings * undo protobufhelpers changes and just ignore 4100 errors in pb code * attempt to isolate protobufhelpers errors * add template specialization for getting tensor proto data * Layer dev paulm (#2533) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * Spawn child process to run DeviceLostRecovery scenario test (#2530) * Spawn child process to run DeviceLostRecovery scenario test * Layer dev paulm (#2536) ori said yes * add missing namespace to winml_trace_logging_provider in lotusenvironment.h (#2542) * Handle exception thrown from all apis in WinMLAdapter (#2539) * various changes to unblock windowsai ADO build * Fix custom ops scenario tests (#2562) * Do not shutdown protobuf after ort environment gets destroyed. Lazy load lotus environment first time it is needed * comment typo * pr comment about calling phoenix singleton * Make lotus_environment static in winmladapter * Layer dev paulm (#2567) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * making windowsAI azure dev ops work. * code review comments. * revert changes * Cmake and preprocessor fixes that where uncovered by building on agents without DML available via SDK * Layer dev dml delayload (#2580) * Brianma/cpu (#2583) * don't include dml stuff in cpu builds * tests that link the image lib also need the telemetry lib now * Throw Winml_err_invalid_binding if binding gpu resource on cpu device (#2589) * Throw Winml_err_invalid_binding if binding gpu resource on cpu device * PR comments. No need to query executionprovider for is gpu device * User/xianz/ortthrow (#2596) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * User/xianz/ortthrow (#2599) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * add status error message * Remove uwp onsuspending winrt call because logruntimeperf is getting removed (#2630) * User/xianz/dedup telemetry (#2631) * investigate duplication of telemetry in winml and ort * remove winml telemetry events * telemetry executionProviderEvent * remove unneccessary file and refactor code little bit * Revert back TelemetryEvent, which send up ETW event. * merge changes from layer_dev to windowsai (#2638) * Remove underscore from googletest names (#2616) * Fix leaking memory allocator Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24278761 and https://microsoft.visualstudio.com/OS/_workitems/edit/24330198 * Explicitly initialize Ort::Value with nullptr * Cache WinML adapter * bad merge * define private version of dxcore enum that is added in 19H1 SDK. (#2654) * add comment for explaning private definition of dxcore d3d feature level ennum value. (#2672) * do not package directml.pdb for redist packages. (#2676) * Fix leaking operator registry (#2645) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24354916 * User/orilevari/windowsai master merge (#2674) merge resolutions included pulling in telemetry logic that was merged to master and not windowsai and dereferencing InferenceSession::sessionstate now that it is a unique pointer * Delete Ort Allocator in LearningModelBinding (#2653) * Delete OrtAllocator in LearningModelBinding * PR comments to make Ort::Allocator a smart pointer * Small comment change * PR feedback to clean up code * PR feedback on move semantics * Clean up std::move * Fix memory leaks (#2679) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24356109, https://microsoft.visualstudio.com/OS/_workitems/edit/24388361 and https://microsoft.visualstudio.com/OS/_workitems/edit/24388596 * various changes to properly organize and skip GPU tests. For now for No DML builds we will not run GPU tests at all. In the future we should adapt the tests to expect the appropiate errors. (#2695) * Windowsai without fi (#2701) * Disable Attention fusion tests when DISABLE_CONTRIB_OPS is defined (#2529) * Setup java ci (#2528) * Add provision in ORT for session options to be parsed when available via model file (#2449) * Initial commit * Fix gitmodules * Nits * Nits * Updates * Update * More changes * Updates * Update * Some updates * More changes * Update * Update * Merge * Update * Updates * More changes * Update * Fix nits * Updates * Fix warning * Fix build * Add comment * PR feedback * PR feedback * Updates * Updates * Update * More changes * Fix build break * Comment test for now * Updates * Updates * PR feedback * Updates * Nits * Add tests * Fix build * Fix build * Fix build * Fix build break * Fix build * Nits * PR feedback * More change * Expose GetSessionOptions in pybind logic and add unit test for python * Fix build * PR feedback * PR feedback * Revert "Disable thread pool creation when enabled OpenMP (#2485)" (#2535) This reverts commit 7c7d5a149c9ed52eec67304bae5c4b132166a8a1. * Add dynamic shape support in TensorRT execution provider (#2450) * remove onnx-tensorrt submodule * add new onnx-tensorrt submodule (experiment) for trt6 * update engine build for trt6 * update compile and compute for tensorrt6.0 * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * switch to onnx-tensorrt master for TensorRT6' * Update tensorrt_execution_provider.cc * Handle dynamic batch size and add memcpy in TensorRT EP * update test cases * Update tensorrt_execution_provider.cc * update onnx-tensorrt submodule * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.ubuntu_tensorrt * Update run_dockerbuild.sh * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update concat_op_test.cc * Update tensorrt_execution_provider.cc * Upgrade TensorRT to version 6.0.1.5 * Update onnxruntime_providers.cmake * Update CMakeLists.txt * Update reduction_ops_test.cc * Update install_ubuntu.sh * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.tensorrt * Update BUILD.md * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update onnxruntime_providers.cmake * Update install_ubuntu.sh * Update install_ubuntu.sh * Update gemm_test.cc * Update gather_op_test.cc * Update CMakeLists.txt * Removed submodule * update onnx-tensorrt submodule * update header file * Removed submodule * add submodule onnx-tensorrt kevin's branch shape-test' * add debugging code * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * merge master * Removed submodule * update onnx-tensorrt submodule * add more changes for dynamic shapes * Update tensorrt_execution_provider.cc * update for dynamic shape * update dynamic shape processing * fix logger issue * remove submodule onnx-tensorrt * add submodule onnx-tensorrt * add env variable min_subgraph_size * remove redundency * update document * use onnxruntime::make_unique * fix multi-run issue * remove some tests to save CI build time * Add dynamic shape test * Update TensorRT-ExecutionProvider.md * Add example of running Faster R-CNN model on TensorRT EP * Add more details on env variables * update environment variables * Update tensorrt_basic_test.cc * Update model tests * Update tensor_op_test.cc * remove --use_full_protobuf * Update build.py * User/xianz/telemetry (#2458) * enabme telemetry * enable telemetry * set enable telemetry as default * for debugging * remove log and set disable telemetry as default back * delete private file while testing * resolve comment: mainly add license header, rename macro and update docs * rewording in privacy.md * Fix integer overflow in cuda NonMaxSuppression implementation (#2540) * add test case that should pass but fail * fix nms * extract int_max_output_boxes_per_class * Introduce container type runtime checks and other improvements (#2522) Rework TensorSeq in a manner consistent with Tensor and SparseTensor in terms of type system setup. Reduce templating. Introduce helpers to ensure the same data type. Make OrtValue __dtor not virtual. Introduce ContainerChecker * Fix C API tests for centos and mac (#2544) * change c++14 to c++11 * add ld lib path for centos * enable csharp tests on macos * fix C API test on MacOS + fix manylinux dotnet install * fix manylinux dotnet install * fix lib link * Add back executable bit to build.py * Fix a bug handling negative begin pad values in Pad op (#2550) * Fix bug in Pad op * Update * DNNL CMAKE update (#2548) * Fix android build (#2558) * Update win-x86-ci.yml (#2557) Fix build pipeline break * Re-enable Windows C# tests (#2564) * disable onnx_test_runner -x invocations for dnnl (#2568) * Allow sequence length to be symbolic (#2559) * setup java ci mac (#2570) * make layernorm fusion to support opset 11 (#2545) * Fix a warning found in the latest VS release * Add more check on SkipLayerNorm and BiasGelu fusion (#2574) * Fix file not found error during docker build. (#2569) * Add ConvTranspose1D (#2578) * Ryanunderhill/packagename test (#2582) * [Nuphar EP] fixes for some object detection models (#2581) Update notebook tutorial with multi-threaded int8 GEMM from #2517 * EmbedLayerNormalization Fusion Improvement (#2553) Embedding layer norm fusion improvements - add more checks * Update version (#2584) * Temporarily exclude vgg19 test from Python backend test 1. temporarily exclude vgg19 test which comsumes too much memory, run out of memory on Upsquared device. Single test pass for vgg19, need furture investigation (#2588) 2. Update docker file to decrease the docker image size * Update docs for Android NNAPI EP (#2586) * Fix lto bug for protobuf and ubuntu * add path to build dir before test run (#2590) * Add missig env variables for mac pipeline test (#2595) * Fixed an issue in updating realized dims (#2597) when we update realized dims for scan's output, the sliced axis also needs to be inclusive, i.e. we should check with "dim >= insert_inclusive_axis", because the offset in the symbols are based on Scan sugraph. Otherwise, we would end up with shape mismatch later. * Java API for onnxruntime (#2215) * Add support for opset 11 in reshape fusion (#2592) Support opset verion 11 in reshape fusion * Rename automl python tools folder to featurizer_ops. (#2593) * Support opset 11 subgraph of Squad model in Embed Layer Normalization (#2605) Support opset 11 Squad model that is exported from PyTorch nightly. The embed layer uses Range op which is missed in the transformer. * symbolic shape inference: fix warnings in GPT-2 model (#2608) And revise nuphar perf test on BERT squad * Dump subgraph ID and fused graph ID (#2607) * Dump subgraph ID and fused graph ID Dump subgraph ID and fused graph ID for better debugging * Remove local static fused_count added a field global_fused_count_ to NupharExecutionProvider class * EmbedLayerNormalization Fusion For Dynamic Squad Model Opset 10 (#2613) Support subgraph of SQuAD model exported from pytorch with dynamic input axes * Allow providers to be set for InferenceSession at construction (#2606) * Remove unnecessary parameter in some places in GatherElements implementation (#2612) * Remove unnecessary parameter in some places * Update * Update * Make sure fenced tensor could not reuse other tensor. (#2561) Fix random error caused by this. * Improve Embed Layer Norm Fusion for SQuAD with static input shape (#2621) * fix float16 comparison in initializer (#2629) * epsilon attribute for layernormalization fusion (#2639) * removed unnecessary batch file and fix path (#2640) * Add shape inference to ConvTransposeWithDynamicPads schema (#2632) * Improve cuda expand() opeator's performance. (#2624) * Cuda pad optimize when no padding is needed. (#2625) * Shortcut cuda Pad() when no padding is needed. * Optimize cuda scatter() on 2D compatible. (#2628) * Optimize cuda scatter() on 2D compatible. * Add some comments. * fix build error for ARM (#2648) * Improve performance of resize() in Nearest mode (#2626) Special treatment for 2D, check same size as input image. And in 2d kernel, template use_expolation. * Fix memory exception in Layer Norm Fusion (#2644) * Windows CI changes(#2650) * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Revert "Windowsai without fi (#2701)" This reverts commit 285d4c85ff5c4e265f963208170304ef3461e684. * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Deref unique pointer for session_state * send shutdown event when dll is unloaded and EvaluationStop, SessionC… (#2704) * send shutdown event when dll is unloaded and EvaluationStop, SessionCreationStart Events. * Add EvalutationStart Event * add comment * use correct type for for loop (#2755) * ARM CI (#2759) * Set ARM agent pool * Set CMake generator to VS 2019 in ARM * Use system-wide CMake instead of custom version Our custom version is too old for VS 2019 * Use DML and build shared lib in ARM CI * Restore nuget packages in ARM CI * Disable DML * Refactor ARM debug/release builds * Use system packaged Python version * Remove hardcoded Python path * Downgrade Python to 3.7 for build * Remove explicit CMake path * Fix invalid JSON in cgmanifest.json (#2760) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * Remove usage of WHOLEARCHIVE in WinML CMake and add WinMLAdapterFactory (#2726) * Remove usage of WHOLEARCHIVE in WinMLAdapter CMake and add WinMLAdapterFactory * PR feedback, no need for dll(export) since using def file * PR comments * Small comment in gen_def.py * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * Move winml_provider_factory.h to proper location (#2801) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Filter CPU case for IsFloat16Supported (#2802) * Merge fixes * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * Fix warnings that cause build to fail * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * Make DML include path global (#2882) * Make DML include path global * Add generated cppwinrt headers to winml_lib_common * Integrate changes to WindowsAI to make ADO Build (#2886) * Revert "CMake cross-generator fixes (#2790)" This reverts commit dbe7d97fa1ab155f1309bced87199527e8f35bd2. * add additional suppress warning in onnx_proto * ignore /wd4996 warning * DML execution provider fixes * Revert "Revert "CMake cross-generator fixes (#2790)"" This reverts commit 1ae7b4bcbc02edc881ad28685da98e095dfceb17. * Update func signature of custom op function overloads * common devicehelpers fixes * Add pch.h for winml_lib_common * re-add winml_lib_common_dir/inc to include path for winml_adapter * User/orilevari/dml redist shared folder (#2890) * move dml nuget package directory up one level to make it shared between build flavors * Merge conflict fix * Revert "Merge conflict fix" This reverts commit 142fa72cf9ce4344ad717b50b7ea2b8582aadc7c. * Revert "Merge remote-tracking branch 'origin/master' into windowsai" This reverts commit 6e2126d46e5e5f564d65da37dd4f70c93dd81165, reversing changes made to b3f5583dc9249834b947c8ea905f6a98060d5bd6. * Make winml_test_common free of test macros (#2902) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Don't generate files outside the build root (#2914) * Don't generate files outside the build root * Add onnxruntime_EXTERNAL_DEPENDENCIES to WinML * Add DML depedency on RESTORE_PACKAGES * User/orilevari/fix yaml merge bugs (#2918) * Add winml test source parameter into cmake function (#2919) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Add arguments to unittest cmake functions * remove comment * Revert "Revert "Merge remote-tracking branch 'origin/master' into windowsai"" This reverts commit ade5abe72a4234fdbc3623093c61c02c6b0bdc26. * Fix breaks from merge with ORT master * Brianma/linux (#2917) * don't include windows.h in cross-plat header * add default case for switch statement * signed/unsigned mismatch fix Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> * User/sheilk/winml adapter c api (#2891) * Create winml adapter c api * fix build * make it build * move adapter into onnxruntime core/session * entry point not exported * minor changes * make model metadata work * make tests pass * implement all the model reflection apis on the adapter c abi * update the new ort interface to create a lotus ennvironment with a logging sink * start adding ort env * move all winml code into adapter folder/lib to isolate it * ensure a single logging manager at a time * start refactoring session * refactor session creation interface * add cpu and dml session option methods to adapter * finish session init * stub out interfaces in ort lib to perform similar mechanics of iinference session * enable profiling, and enable schema override * update session register graph transformers * turn back on custom registry for custom ops * Add sync api * add last c api stubs * should build... but all feature values are broken since this is in flight to moving all implementation details into ivalue * remove ep adapter header * Implement DML execution provider functions from adapter (#2846) * Implement DML execution provider functions from adapter * Use functions in OnnxruntimeEngine.cpp * make map/sequence type_infos freeable, and start implementing ivalue * make it build again * implement value methods * implement remaining methods * remove com adapter abi * check dml session * cache the allocator on ivalue * check if resource is cpu/gpu when access its mutable data * update tensor * mismatched parentheses * fix tensor base and binding obj * it evaluates tensors! sometimes... * minor fixes * enable gpu evals * wrapper all existing winml adapter apis with API_IMPL to try catch (#2854) * update winml... tensor strings are broken, need to template tensorbase to do different things for strings * make tensor strings work with 2 copies in/2 copies out * Fix tensor string and allocator bug * make maps work again... needs some fixes still * Make it build! * enable map inputs * map outputs * unbound outputs for sequences and maps * User/xianz/merge windowsai (#2883) * Packaging pipeline changes for VS 2019 (#2711) * Tiny fix to codegen * Simplify cache implementation and avoid static variables that may carry over between models * Extend DML kernels (#2641) * Additional DML operators * Check unsupported attributes and inputs * Address PR comments * Add kernel capability function used for partitioning, and re-enable stride-based int64 support based on value range * Fix test failures * Build fix * PR comments * Update Nuphar tutorial notebook (#2721) 1. Reflect int8 GEMV improvements for multi-threading from #2696 2. Add notes on multi-threading control using OpenMP 3. Add samples of running multi-isa AOT, and show int8 GEMM differences between AVX and AVX2 4. Add rnn_benchmark example to resolve #1993 * Add schema for new Qops (#2611) * Add schema for new Qops * adding shape inference + qlinearaveragepool * plus review comments * plus review comments * updates per review comments * plus review comments * [server] Add supposed for model_name and model_version as cli parameter (#2708) * remove 64bit warning message from python validation. (#2727) * MLAS: ARM64 build fix (#2734) fix bad usage of vreinterpret to cast vector element types * Fix broken python docs links (#2740) * Fix build on Mac OS (#2731) mac os ld doesn't support --while-archive, correct option is -all_load * fix ngraph wheel (#2737) * fix ngraph wheel 1.1.0 onnxruntime_ngraph wheel doesn't work * remove libdnnl.so in nGraph Libs * make it easy to compare * Split onnxruntime server to a separated folder (#2744) * Fix build for Python 3.8 (#2747) * Fix build for Python 3.8 * Update protobuf to 3.11.2 (#1928) Update protobuf to 3.11.2 (#1928) * Change default optimization level to All (from Basic) (#2745) * change default optimization level to All (from Basic) * fix test * fix c# test * Update numpy to 1.18 (#2758) * Update numpy to 1.18 * Pipeline changes for python 3.8 (#2753) 1. Pipeline changes for python 3.8 2. Fix a regression in setup.py which was just introduced in the previous commit. Please notice, we still haven't made python 3.8 + Windows + CUDA work. * Add basic stacktrace output for posix debug builds. (#2749) * [NupharEP] fix a race condition when multiple sessions running different models concurrently (#2772) * Revert "Change default optimization level to All (from Basic) (#2745)" This reverts commit 56bb503c2f26474b6613bcb2a198691a11dcef38. * Fix typo in error message (#2736) * Rename MKL-DNN to DNNL to fix broken link (#2730) * Fix nightly build version number issue * Pass BUILD_BUILDNUMBER to linux docker * Disable featurizers in python packages * Import more featurizers (#2781) Make kernels non-template. Add input constraint for learnt data. Add min_max_scalar_transformer, robust_scalar_transformer, inputation_marker_transfomer, label_encoder_transformer, missing_dummies_transformer along with tests. Advance Featurizers library commit. * Implement a more stable softmax (#2715) * Implement a more stable SoftMax e^x is represented as infinity if x is large enough, like 100.f. Infinity divided by Infinity is a NAN. Thus, softmax gets a NAN if one or more item are large enough. A math transform as below is leveraged to get a stable softmax: e^xi/(e^x1 + ...e^xn) = e^(xi - max) / (e^(x1 - max) + ... + e^(xn - max)) And for convenience, force max to 0.f if all xi are negative * Contributing: Fix a typo (#2784) * ACL EP GEMM improvements (#2780) When it is posible we use a fully connected layer instead of the gemm implementation. This will let the library use the best implementation based on the input data. * ACL EP convolution improvements (#2774) Added the optimized implementation for depthwise convolution for both ACL v19.02 and ACL 19.05. Also the pointwise convolution seems to be more optimal in the CPU implementation so we opted for that instead. * Add script for release Nuget validation (#2719) * Initial commit * Nits * Disable a test temporarily * Change working directory * Test * Add download python step * Test update * More changes * Fix space issue * Fix * Verify nuget signing * Fix * Spaces * PR feedback * Nit * Fix * Fix * Remove temporary changes * add uint8 support to where op (#2792) * Improve bert optimization script: (#2712) (1) Move input int64=>int32 conversion to embed layer fusion. (2) Output epsilon attribute for LayerNormalization fusion. * add session creation time cost. (#2798) * ML.NET team needs featurizers within a package (#2789) Add auto ml featurizers to Windows, MacOS as well as to GPU packaging-pipelines. * Initialize max of softmax with lowest of float (#2786) * MLAS: update SGEMM threading parameters (#2808) * add interface to copy batch tensors. (#2807) * add interface to copy batch tensors. * onnxruntime * speed up Windows TRT CI (#2811) * don't run cuda tests if building with tensorrt * remove unnecessary build options for win trt ci * refactor win gpu tensorrt ci yml * --numpy_version=1.17 * update * update * azcopy and cuda path * Update test data (#2356) * Add timeseries imputer transformer featurizer kernel (#2813) Make kernels non-template. Add input constraint for learnt data. Fixup tests. Add two more featurizers along with tests. Tests fail. min_max_scalar_transformer robust_scalar_transformer Fix tests serialized stream by prepending version bytes. Add inputation_marker_transfomer and the test. Fix up float/double type designations. Added label_encoder_transformer along with a test. string_throw case is broken at the momement. Fix labelencodertransfomer_test.cc string_throw case Rename maxabsscalertransformer_test.cc Add MissingDummiesTransformer along with the test. Update manifest. Add TimeSeriesImputerTransformer definition, implementation and tests * Fix memory leak in TRT (#2815) * fix memory leak issue * revert EP_FAIL on enueueV2 * Add manifest missing comma * Run static code analyzer on most of our code (#2817) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * update quantization doc (#2783) * update documentation for quantization script * plus some spell corrections * Filter CPU case for IsFloat16Supported (#2802) * update default optimization level + fix gemm_activation fusion (#2791) * update defualt optimization level + fix gemm_activation fusion * fix typo * add unit test and incorporate review comments * fix test comment * Fix dnnl wheel package name (#2823) * Append '-dnnl' to whl package name when --use_dnnl * Update build.py * Update Ubuntu & TensorRT version in README (#2820) Dockerfile.tensorrt is using nvcr.io/nvidia/tensorrt:19.09-py3 as base Image, update Ubuntu and TensorRT version according to https://docs.nvidia.com/deeplearning/sdk/tensorrt-container-release-notes/rel_19-09.html#rel_19-09 * Merge fixes * Add OneHotEncoder and HashOneHotEncoder kernels. (#2830) Add defs and imlementation for OneHotEncoders, adjuist date_time_transformer kernel and test. Add OneHotEncoder kernel test. Add HashOneHotVectorizerTransformer unit test. This does not link due to multiple definitions of functions that are included into header from a CPP file. * Upgrade gtest to the latest version (#2827) WinML would like to update the googletest submodule. They want some newer features (namely GTEST_SKIP to skip tests programmatically and be able to skip entire fixtures easily) and would need to update the submodule version. However, because the new version of code hit a bug in gcc, even though the bug is already fixed in the latest gcc but we're using gcc 4.8.x and it won't get patched for the bug, so we have to do a compromise, change our code a little bit to make it work. The gcc bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=51213 * Add support for int64_t for topk CPU. Fixes github issue #2806. (#2833) * Ignore allocator type in ExecutionProviders allocator map. Make default initialization of OrtMemoryInfo more clearly invalid. (#2768) * Remove allocator type from the key comparison in ExecutionProviders. Remove usage of DummyArena as it's no longer necessary. * Fix x86 tests where arena allocator is disabled. Make initialization of OrtMemoryInfo clearer by adding Invalid enum value. * Make OrtValueNameIdxMap::MaxIdx more intuitive. * Convert ExternalProject Featurizers into git submodule (#2834) Add git submodule for Featurizer library. Update cmake to build for git submodule. * add domain check for nodes + update documentation (#2831) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Fix Linux CUDA nuget packaging pipeline break * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * update optimization doc for BERT related fusions (#2819) * Add bert related transformers to doc * Add execution provider and comment for bert optimizations * Add comment about accuracy impact of approximation * Fix warnings that cause build to fail * MLAS: enable threading for quantized GEMMs (#2844) * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * remove old winmladapter cpp Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * move sequence implementation into ort lib... still commented out... need to turn back on... * begin sequence implementation * make maps and sequences work * fix broken tests * remove dead code * misc cleanup * CR feedback * User/xianz/winml adapter c api (#2869) * wrapper all existing winml adapter apis with API_IMPL to try catch * Return HR or Throw for WinML adapter APIs if failed * undo macro wrapper for two places * Wrap error macros around ort apis, too. * address CR feedback #2 * add more api throw/return macros * Revert changes no longer needed * revert changes to cxx api * format winml lib.ort and winml adapter * remove static pheonix singleton Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * missing use_dml check in winml_adapter_session (#2930) * --use_dnnl flag was mangled in merge (#2931) * use dml macro not wrapping custom registry code (#2934) * Disable LNK4199 winml_dll to enable cuda builds (#2936) * Disable LNK4199 in winml_dll * linkler->linker * LearningModelSessionAPITestGpu.CreateSessionWithCastToFloat16InModel should return DXGI_ERROR_UNSUPPORTED when FP16 not supported (#2937) * Disable LNK4199 in winml_dll * linkler->linker * Need to return DXGI_ERROR_UNSUPPORTED when Model does not support fp16 * Publish build symbols (#2939) * Publish build symbols * Don't upload PDBs for .exe files * Make x86 build (#2943) * fix last remaining size_t/int64_t warnings->errors (#2948) * TensorString, Sequences and Maps use the first allocator, but should use the cpu default allocator. (#2952) * fix tensor string allcoator * clean up default allocator usage for strings in winml lib/api.ort Co-authored-by: Ryan Lai <ryalai96@gmail.com> * Handle tensor shape of zero (#2954) Co-authored-by: Ryan Lai <ryalai96@gmail.com> * CR feedback (#2970) * CR feedback * fix weird formatting on privacy readme * Add 'All rights reserved.' everywhere * readd all rights reserved to winml_provider_factory.h * remove extra space in comment * remove extra whitespace * fixes post master merge * remove winml from nuget gpu pipeline * set IR VERSION on generated_model in rnn_benchmark (#2972) * Fix slice conformance failures (#2908) Co-authored-by: Adrian Tsai <adtsai@microsoft.com> Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Paul McDaniel <paul_mcdaniel@hotmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Dwayne Robinson <fdwr@hotmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com>
2020-02-05 01:12:19 +00:00
[threading(both)]
[marshaling_behavior(agile)]
[dualapipartition(1)]
runtimeclass LearningModelDevice
{
//! Create a LearningModelDevice from the specified IDirect3DDevice.
//! During evaluation, the specified IDirect3DDevice will be used to create resources and queue work during execution.
static LearningModelDevice CreateFromDirect3D11Device(Windows.Graphics.DirectX.Direct3D11.IDirect3DDevice device);
//! Create a LearningModelDevice from the specified list of devices enumerated in LearningModelDeviceKind.
[method_name("Create")] LearningModelDevice(LearningModelDeviceKind deviceKind);
// BUGBUG: this needs to be Windows.Graphics.DisplayAdapterId which is only there in the RS4 winmd
//! Returns the unique identifier for the chosen adapter for model
Windows.Graphics.DisplayAdapterId AdapterId{ get; };
//! Returns the chosen IDirect3DDevice for model evaluation.
Windows.Graphics.DirectX.Direct3D11.IDirect3DDevice Direct3D11Device{ get; };
}
[contract(MachineLearningContract, 1)]
INBOX_ONLY([interface_name("Windows.AI.MachineLearning.ILearningModelEvaluationResult", b2f9bfcd-960e-49c0-8593-eb190ae3eee2)])
Merge windowsai (winml layering) into master (#2956) * Initial Commit * Merged PR 3985217: add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc (#2346) add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc and violating our OS layering requirements. We linked against onecoreuap_apiset.lib in VB so we will continue doing this, but I am still unsure why not to link against onecore instead since that is where we ship. However, since Sheil is the owner of this code we will wait to discuss with him before changing anything. * Initial changes for layering * more snipping to get core into ort * update build instructions to include --build_shared_lib (#2358) * update build instructions to include --build_shared_lib * fix line breaks * Task 23998197: add winml_lib_core into onnnxruntime.dll (#2368) * Task 23998197: add winml_lib_core into onnnxruntime.dll * PR feedback build break on perf_test * return proper error when the model path isn't found (#2391) * LearningModelSession is cleaned up to use the adapter, and parts of b… (#2382) this is a big PR. we are going to move it up to layer_dev , which is still a L3 so we are still safe to do work there agile. we are going to move this into the L3 so that ryan can start doing intergration testing. we will pause for a full code review and integration test result prior to going into the L2. >>>> raw comments from previous commits >>> * LearningModelSession is cleaned up to use the adapter, and parts of binding are. * moved everything in the winmladapter made it all nano-com using, WRL to construct objects in the ORT side. base interfaces for everythign for winml to call cleaned up a bunch of winml to use the base interfaces. * more pieces * GetData across the abi. * renamed some namepsace cleaned up OrtValue cleaned up Tensor cleaned up custom ops. everything *but* learnignmodel should be clean * make sure it's building. winml.dll is still a monolith. * model moved over. everything builds clean. step ! * weak ref comment * Layer dev paulm (#2408) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * Layer dev paulm (#2414) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * Layer dev paulm (#2423) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * Layer dev paulm (#2424) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * Layer dev paulm (#2425) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Layer dev paulm (#2426) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * fixed map and sequence when passing stl types across the ABI . found a leak in nvidia driver, but skipped it. all winmlapitests pass now * Moved SessionOptions over to the abi * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * Register WinML TraceLogging provider on Onnxruntime.dll (#2455) * Register WinML TraceLogging provider on Onnxruntime.dll * Add ifdef to make sure trace logging provider has telemetry option when LAYERING_DONE * No need for ifdef for TraceLoggingOptionMicrosoftTelemetry * PR feedback * Move etw registration into lotus environment constructor and deresgister in lotus environment destructor * Brianma/cpuwinml (#2466) * allow building winml cpu without dml. * Brianma/breaks (#2469) * fix some more breaks * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * move dml checks out of winml and into the adapter * better error handling * Brianma/fi (#2470) * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * fix bad merge * Checking in a staging checkpoint point so that Ryan can work with me in parrallel * build break. * Brianma/testfails (#2473) * add missing ir version to dictvectorizer-string.onnx * add missing ir version to relu.onnx * add missing ir version to zipmap*onnx * add IR version to manually generated models * remove an unnecessary ifdef dml * Brianma/windowsai fi (#2475) * update dockerfiles/README (#2336) * Make elementwise op run 4 items per thread (#2335) Description: Describe your changes. Make elementwise op run 4 items per thread unroll for loop to leverage ILP remove unnessary N==0 check inside elementwise GPU kernel Motivation and Context Why is this change required? What problem does it solve? It can improve the performance of GPU elementwise ops. ~2% performance gain on popular NLP bert model. If it fixes an open issue, please link to the issue here. * Add CUDA GatherElements kernel (#2310) * Updates * Update test * Update * Updates * nits * PR feedback * Update * Update * PR feedback * PR comments * Update * Fix build * Fix build * Nits * Fix * Layer Normalization Fusion (#2319) basic layer normalization transform * Add FastGelu Cuda Op for Gelu and Add bias fusion (#2293) * Add FastGelu cuda op * Add AddBiasGelu for experiment * Revert "Add AddBiasGelu for experiment" This reverts commit 5c1ee019858c657e6bb75887265cb85675626e5b. * Add bias * Add unit tests * update comment * update script * fix build error * update coding style * update for CR feedback Enable half2 optimization only when cuda arch >= 7.0 * move _Tanh to common.cuh * implement CPU contrib OP Attention (#2333) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanUnusedInitializers. (#2320) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanupUnusedInitializers. This means initializers that have been replaced during graph optimizations are not left in the GraphProto when we save an optimized model. * Handle edge case where a model has an unused initializer with matching graph input by also removing the graph input. * Use non-const iterators in std::find_if calls to make centos build happy. * Nuget pipeline changes (#2305) 1. refactor the pipeline, remove some duplicated code 2. Move Windows_py_GPU_Wheels job to Win-GPU-CUDA10. We'll deprecated the "Win-GPU" pool 3. Delete cpu-nocontribops-esrp-pipeline.yml and cpu-nocontribops-pipeline.yml 4. In Linux nuget jobs, run "make install" before creating the package. So that extra RPAH info will be removed * Cuda Reverse Sequence Op, maping types of same size using same template function. (#2281) * Set ElementType to String type of node metadata, instead of byte[] (#2348) * Set ElementType to String type of node metadata, instead of byte[] * Fix spacing * Introduce PrimitiveType into a Type System along with an integer constant (#2307) Improve perf by avoiding GetType<T>() calls. Introduce MLTypeCallDispatcher to switch on Input Type. Add Tensor IsType<T>() fast method. * Fix/test dim value of 0 handling in a couple of places (#2337) * Update the CUDA Where implementation broadcasting logic to handle a dim with value of 0. Add unit test Also add unit test for unary op with dim value of 0 * Exclude ngraph from Where test with 0 dim. * Openvino EP R3.1 onnxrt server (#2357) * onnxrt server with OVEP * onnxrt server with OVEP * Update Dockerfile.server.openvino * onnxrt server OVEP fix reviews * onnxrt server OVEP fix reviews * Implement cuda nonzero op. (#2056) Implement cuda nonzero op. * Direct use python numpy array's memory if already contiguous. (#2355) * Direct use python numpy array's memory if already contiguous. This could greatly improve performance for session with large input, like big image 1920x1080 fastrcnn, 30~40% speed up could be achieved. * Add test case enforce contiguous/non-contiguos numpy array as inputs. * Add helper to create output to minimize binary size. (#2365) Add ConstEigenTensorMap typedef so we don't unnecessarily const_cast the const input Tensor. * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS (#2369) * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS * update * Add Tracelogging for profiling (#1639) Enabled only if onnxruntime_ENABLE_INSTRUMENT is ON * test bidaf with nuphar for avx target (#2370) increase nuphar test coverage a bit * Fix a bug in TLS refcount that may destabilized CUDA CI (#2374) * update output size calculation for resize (#2366) * change how output size is calculated for resize op * add tests for ver 10 resize * Extend OneHot CPU kernel to support more types (#2311) * Extend OneHot CPU kernel to support input int64_t, depth int32_t, output float * Skip BERT before the test data fix is picked up * Fix bug with Slice. Need to pass in flattened input dimensions so the initial offset into the input is calculated correctly. (#2372) * Add opset 11 version of Split to CUDA ops (#2376) Organize the CUDA ops definitions so all the opset 10 and 11 parts are together (same setup used for CPU ops) * Layer Norm Fusion Fix (#2379) * layer norm fusion fix * Add input shape check in code and unit tests * Fuse Add + Gelu (#2360) Implement the transformer to fuse add + gelu Implement the accurate kernel * Skip layer norm transform (#2350) * skip layer normalization transformer * Another try to stabilize CUDA CI (#2383) The root cause seems to be failure in CUDA dealloc when tear down. cudaFree return code was ignored before, so should the debug check. * fix BUILD.md typo (#2375) build.py: error: argument --config: invalid choice: 'RelWithDebugInfo' (choose from 'Debug', 'MinSizeRel', 'Release', 'RelWithDebInfo') * Fixed compilation with ngraph (#2388) * Fix reuse logic in allocation planner. (#2393) * Fix reuse logic in allocation planner. * PR comments * Add helpful comments * Don't allow reuse across string tensors. * [NupharEP] Multiple optimizations (#2380) Fuse transpose into MatMul Implement Pow and constant scalar simplification Vectorize ReduceMean Improve symbolic shape inference Minor updates for better debugging in fused function name * Avoid using the default logger in the graph lib and optimizers (#2361) 1. Use the session logger if it is available. 2. Don't disable warning 4100 globally. We should fix the warnings instead of disabling it. * Change CUDA implementation of Transpose to support all fixed size tensor types (#2387) * Change CUDA implementation of Transpose to not use a typed kernel so we can support more types with minimum binary size. Add support for 8, 16, 32 and 64 bit types. Add unit tests. Add method so the implementation can be called directly (will be used by CUDA Scan very soon). * Disable TensorRT for MLFloat16 and int8 unit tests. * Address PR comment and add support for calling cublas implementation if type is mlfloat16. * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. (#2398) * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. * [NupharEP] force some low/zero cost ops to be inlined (#2409) * fix cross compile bug (#2415) * Minor optimization: if a node has already been placed, there's no need to find a kernel for it. (#2417) * Add Reshape Fusion (#2395) * Add reshape fusion * Add some comments * update comments * update comment format * update according to feedback * update for recent logger change * fix build error * (1) Support both input and output edges in find path in graphutils (2) Add a test case of only one constant initializer of Concat input. (3) Refactor ReshapeFusion class to allow add more subgraph fusion in the future. * fix error * (1) loose constraint on initializer: non constant is allowed for reshape fusion. (2) Change versions type to vector. (3) Add logging. (4) Return false when multiple output edges matched in FindPath. Add comments. * only allow one direction (input or output) in FindPath * [NupharEP] Update notebook and docker image (#2416) Add BERT squad in Nuphar tutorial Enhance speed comparsion readability * Fix the issue in matmul_add_fusion (#2407) Fix the issue in matmul_add_fusion If Muatmul + Add has shape [K] * [K, N], reset it to [1, K] * [K, N] will make the output shape to [1, N] will also requires a reshape on the output. Fix: just remove the shape reset to not fuse it. Add a negative test case for matmul+add fusion * feat(treeregressor): Update TreeEnsembleRegressor for type support (#2389) Updates the `TreeEnsembleRegressor` to allow for `double`, `float`, `int64`, and `int32` inputs to match the upstream specification. Signed-off-by: Nick Groszewski <nicholas.groszewski@capitalone.com> * onnxrt server documentation update (#2396) * Added support for Pad-2 operator in OpenVINO-EP (#2405) * Add CUDA If operator. (#2377) * Add CUDA If operator. Uses CPU operator for implementation. By adding a CUDA version the inputs/outputs (with the exception of the 'cond' input) stay on GPU, and no other logic is required to avoid a copy to CPU across the control flow node. * Improved documentation for onnxruntime::utils::SwapByteOrderCopy(), added precondition check. * Fix the type constraints on CUDA If operator to exclude strings. (#2431) * add Im2col<uint8_t> (#2438) * Adjust codegen vectorization width from target (#2439) * Adjust codegen vectorization width from target * Add CUDA Scan operator. (#2403) * Add Scan CUDA op. Uses CPU implementation for logic. Added some device specific functors for handling when data needs to be manipulated on a different device. Added ability to override the materialization logic in the OrtValue slicer so DML can plugin their handling. * Fix Windows GPU C API packaging pipeline failure (#2440) Fix Windows GPU C API packaging pipeline failure (#2440) * Correctly handle implicit inputs for fused nodes (#2390) * Correctly handle implicit inputs for fused nodes Previously, nuphar's partitioning function didn't include node's implicit inputs into the inputs list of MetaDef, and hence a crash was triggered in the onnx graph checker. This commit fixed the issue. Furthermore, it also fixed a related issue where we didn't add implicit inputs into graph_inputs_excluding_initializers_ in Graph::SetGraphInputsOutputs. the issue was that graph_inputs_including_initializers_ populated by SetInputs (e.g. called by FunctionImpl::FunctionImpl) may contain implicit inputs which were not of any node's initializers in the graph. Because they were not part of any initializers, these implicit inputs couldn't be visited by going through all nodes' inputs. Consequently, they would *not* be added into graph_inputs_excluding_initializers_. We fixed the issue by first copying the populated graph_inputs_including_initializers_ into graph_inputs_excluding_initalizers_, which then had both initializers and non-initializers as its initial content. Later, we erase initializers from the list. In this way, we can ensure all implicit inputs to remain in graph_inputs_excluding_initializers_. * refined comments and fixed duplicates Address CR by revisiting comments in terms of implicit inputs Also fixed an issue by skipping duplicates while copying inputs from graph_inputs_including_initializers_. * address CR explain why we need to collect nodes' implicit inputs * don't rely on pointer values for iterating std::set Previously, openvino relied on iterating a set of NodeArg pointers to construct inputs and outputs for a fused graph. It could cause non-determinism. The reason was that although iterating std::set by itself is stable, pointer values of NodeArgs may vary. Consequently, we could end up visiting the set's elements in different orders for different runs for the same test, which resulted in constructing inputs (and outputs) with different orders to the fused graph. For example, for the same test, we may have inputs [A, B] in some runs but inputs[B, A] in others. Let's use std::string as the key type to avoid such nondeterminism. This commit also added implicit inputs into meta->inputs while returning the capability from the openvino provider. * Fixed another latent issue in openvino's GetCapability function The issue was that we couldn't simply erase fused_inputs and fused_outputs while iterating the nodes. For example, an output NodeArg may have multiple uses, and it's wrong if we erase it from fused_outputs when we encounter only one of its uses as input. * Remove DeviceAllocatorRegistry class (#2451) Remove DeviceAllocatorRegistry class * CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library. * Parallel Gelu with ParallelFor (#2399) Parallel Gelu to get better performance for Gelu * Clean up build.py (#2446) * Pull the latest image before running docker build * Fuse SkipLayerNorm with Bias (#2453) Fuse SkipLayerNorm with Bias * Allow more than one invocation of CreateEnv in the same process. (#2467) * Allow more than one invocation of CreateEnv in the same process. * Fix centos build * Symbolic shape inference improvements: (#2460) * Symbolic shape inference improvements: - add a mode to guess unknown ops' output rank - add support for GatherND - add support for If - fix a bug in get_int_values when then tensor rank > 1D, by treating it as no sympy data - add symbol to literal merge when ONNX silently merges dims - fix a bug in Concat when input dim is 0 - fix a bug in ConstantOfShape that computed dim is not updated - add support for dynamic shape in ConstantOfShape - fix a bug in Loop output shape that loop iterator dim is not inserted at dim 0 - add support for dynamic padding in Pad - add support for dynamic shape in Reshape - add support for Resize with opset > 10, by treating output dims as dynamic - fix a bug in Slice when starts/ends are dynamic - restrict input model to opset 7 and above - make output model optional to avoid disk write when testing Run model tests for symbolic shape inference Reduce 2GB docker image size of nuphar * add additional test data set for nuget pipeline (#2448) * add SAS token to download internal test data for nuget pipeline * update azure endpoint * fix keyvault download step * fix variable declaration for secret group * fix indentation * fix yaml syntax for variables * fix setting secrets for script * fix env synctax * Fix macos pipeline * attempt to add secrets to windows download data * fix mac and win data download * fix windows data download * update test data set url and location * Revert "Brianma/windowsai fi (#2475)" This reverts commit 5780b864a15513fda4eadbfc2b5345fefe70b5ec. * Add scenario tests (#2457) * Add scenario tests * Remove TODO from model license * Add winml_api test dependency * fix model load test. fi from master changed the constructor (#2483) * make api tests all pass (#2486) * fix bad merge * fix bad model merge * Layer dev paulm (#2492) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * Rename ambiguous header (#2489) * fix one more missing IR version model (#2500) * add missing IR version to 4 more models used by scenario tests (#2501) * Add CLI parameters to test runner, build WinML in ARM and x86 CI (#2479) * Support test parameters through CLI arguments * Add WinML do Windows x86/ARM CI builds * Code style fixes * Update googletest Remove GPUTEST macros everywhere now that GTEST_SKIP is supported * Refactor main.cpp * Build scenario tests without DML * Link scenario tests to DML when it's enabled (#2502) * Layer dev release pipeline (#2488) Adds winml binaries to existing cpu nuget package, and creates new gpu dml nuget package with winml binaries and DML EP. * Layer dev paulm (#2506) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * Remove usage of IOBinding in WinML and use C_API Run method (#2504) * remove usage of iobinding * Change data structure to use vector of Ort::Values * Polish bind input / output * Use C APIrun method * Update providers on evaluate getresults * Remove run and IObinding interface from WinMLAdapter * Remove use of IObinding * bind unbound outputs code moved to learningmodelbinding * clean up unneeded istensor adapter function * Fix comment * Check if session is closed before binding and clearing * PR feedback * Layer dev paulm (#2507) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * Make tests dependend on winml_dll (#2509) * add dml binaries to DirectML package and be more explicit about condition variables (#2520) * re-enable warnings for winml builds and fix the warnings that were hiding (#2526) * turn devmode back on for winml builds * fix some warnings. include protobuf in a way that disables some warnings * undo protobufhelpers changes and just ignore 4100 errors in pb code * attempt to isolate protobufhelpers errors * add template specialization for getting tensor proto data * Layer dev paulm (#2533) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * Spawn child process to run DeviceLostRecovery scenario test (#2530) * Spawn child process to run DeviceLostRecovery scenario test * Layer dev paulm (#2536) ori said yes * add missing namespace to winml_trace_logging_provider in lotusenvironment.h (#2542) * Handle exception thrown from all apis in WinMLAdapter (#2539) * various changes to unblock windowsai ADO build * Fix custom ops scenario tests (#2562) * Do not shutdown protobuf after ort environment gets destroyed. Lazy load lotus environment first time it is needed * comment typo * pr comment about calling phoenix singleton * Make lotus_environment static in winmladapter * Layer dev paulm (#2567) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * making windowsAI azure dev ops work. * code review comments. * revert changes * Cmake and preprocessor fixes that where uncovered by building on agents without DML available via SDK * Layer dev dml delayload (#2580) * Brianma/cpu (#2583) * don't include dml stuff in cpu builds * tests that link the image lib also need the telemetry lib now * Throw Winml_err_invalid_binding if binding gpu resource on cpu device (#2589) * Throw Winml_err_invalid_binding if binding gpu resource on cpu device * PR comments. No need to query executionprovider for is gpu device * User/xianz/ortthrow (#2596) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * User/xianz/ortthrow (#2599) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * add status error message * Remove uwp onsuspending winrt call because logruntimeperf is getting removed (#2630) * User/xianz/dedup telemetry (#2631) * investigate duplication of telemetry in winml and ort * remove winml telemetry events * telemetry executionProviderEvent * remove unneccessary file and refactor code little bit * Revert back TelemetryEvent, which send up ETW event. * merge changes from layer_dev to windowsai (#2638) * Remove underscore from googletest names (#2616) * Fix leaking memory allocator Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24278761 and https://microsoft.visualstudio.com/OS/_workitems/edit/24330198 * Explicitly initialize Ort::Value with nullptr * Cache WinML adapter * bad merge * define private version of dxcore enum that is added in 19H1 SDK. (#2654) * add comment for explaning private definition of dxcore d3d feature level ennum value. (#2672) * do not package directml.pdb for redist packages. (#2676) * Fix leaking operator registry (#2645) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24354916 * User/orilevari/windowsai master merge (#2674) merge resolutions included pulling in telemetry logic that was merged to master and not windowsai and dereferencing InferenceSession::sessionstate now that it is a unique pointer * Delete Ort Allocator in LearningModelBinding (#2653) * Delete OrtAllocator in LearningModelBinding * PR comments to make Ort::Allocator a smart pointer * Small comment change * PR feedback to clean up code * PR feedback on move semantics * Clean up std::move * Fix memory leaks (#2679) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24356109, https://microsoft.visualstudio.com/OS/_workitems/edit/24388361 and https://microsoft.visualstudio.com/OS/_workitems/edit/24388596 * various changes to properly organize and skip GPU tests. For now for No DML builds we will not run GPU tests at all. In the future we should adapt the tests to expect the appropiate errors. (#2695) * Windowsai without fi (#2701) * Disable Attention fusion tests when DISABLE_CONTRIB_OPS is defined (#2529) * Setup java ci (#2528) * Add provision in ORT for session options to be parsed when available via model file (#2449) * Initial commit * Fix gitmodules * Nits * Nits * Updates * Update * More changes * Updates * Update * Some updates * More changes * Update * Update * Merge * Update * Updates * More changes * Update * Fix nits * Updates * Fix warning * Fix build * Add comment * PR feedback * PR feedback * Updates * Updates * Update * More changes * Fix build break * Comment test for now * Updates * Updates * PR feedback * Updates * Nits * Add tests * Fix build * Fix build * Fix build * Fix build break * Fix build * Nits * PR feedback * More change * Expose GetSessionOptions in pybind logic and add unit test for python * Fix build * PR feedback * PR feedback * Revert "Disable thread pool creation when enabled OpenMP (#2485)" (#2535) This reverts commit 7c7d5a149c9ed52eec67304bae5c4b132166a8a1. * Add dynamic shape support in TensorRT execution provider (#2450) * remove onnx-tensorrt submodule * add new onnx-tensorrt submodule (experiment) for trt6 * update engine build for trt6 * update compile and compute for tensorrt6.0 * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * switch to onnx-tensorrt master for TensorRT6' * Update tensorrt_execution_provider.cc * Handle dynamic batch size and add memcpy in TensorRT EP * update test cases * Update tensorrt_execution_provider.cc * update onnx-tensorrt submodule * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.ubuntu_tensorrt * Update run_dockerbuild.sh * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update concat_op_test.cc * Update tensorrt_execution_provider.cc * Upgrade TensorRT to version 6.0.1.5 * Update onnxruntime_providers.cmake * Update CMakeLists.txt * Update reduction_ops_test.cc * Update install_ubuntu.sh * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.tensorrt * Update BUILD.md * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update onnxruntime_providers.cmake * Update install_ubuntu.sh * Update install_ubuntu.sh * Update gemm_test.cc * Update gather_op_test.cc * Update CMakeLists.txt * Removed submodule * update onnx-tensorrt submodule * update header file * Removed submodule * add submodule onnx-tensorrt kevin's branch shape-test' * add debugging code * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * merge master * Removed submodule * update onnx-tensorrt submodule * add more changes for dynamic shapes * Update tensorrt_execution_provider.cc * update for dynamic shape * update dynamic shape processing * fix logger issue * remove submodule onnx-tensorrt * add submodule onnx-tensorrt * add env variable min_subgraph_size * remove redundency * update document * use onnxruntime::make_unique * fix multi-run issue * remove some tests to save CI build time * Add dynamic shape test * Update TensorRT-ExecutionProvider.md * Add example of running Faster R-CNN model on TensorRT EP * Add more details on env variables * update environment variables * Update tensorrt_basic_test.cc * Update model tests * Update tensor_op_test.cc * remove --use_full_protobuf * Update build.py * User/xianz/telemetry (#2458) * enabme telemetry * enable telemetry * set enable telemetry as default * for debugging * remove log and set disable telemetry as default back * delete private file while testing * resolve comment: mainly add license header, rename macro and update docs * rewording in privacy.md * Fix integer overflow in cuda NonMaxSuppression implementation (#2540) * add test case that should pass but fail * fix nms * extract int_max_output_boxes_per_class * Introduce container type runtime checks and other improvements (#2522) Rework TensorSeq in a manner consistent with Tensor and SparseTensor in terms of type system setup. Reduce templating. Introduce helpers to ensure the same data type. Make OrtValue __dtor not virtual. Introduce ContainerChecker * Fix C API tests for centos and mac (#2544) * change c++14 to c++11 * add ld lib path for centos * enable csharp tests on macos * fix C API test on MacOS + fix manylinux dotnet install * fix manylinux dotnet install * fix lib link * Add back executable bit to build.py * Fix a bug handling negative begin pad values in Pad op (#2550) * Fix bug in Pad op * Update * DNNL CMAKE update (#2548) * Fix android build (#2558) * Update win-x86-ci.yml (#2557) Fix build pipeline break * Re-enable Windows C# tests (#2564) * disable onnx_test_runner -x invocations for dnnl (#2568) * Allow sequence length to be symbolic (#2559) * setup java ci mac (#2570) * make layernorm fusion to support opset 11 (#2545) * Fix a warning found in the latest VS release * Add more check on SkipLayerNorm and BiasGelu fusion (#2574) * Fix file not found error during docker build. (#2569) * Add ConvTranspose1D (#2578) * Ryanunderhill/packagename test (#2582) * [Nuphar EP] fixes for some object detection models (#2581) Update notebook tutorial with multi-threaded int8 GEMM from #2517 * EmbedLayerNormalization Fusion Improvement (#2553) Embedding layer norm fusion improvements - add more checks * Update version (#2584) * Temporarily exclude vgg19 test from Python backend test 1. temporarily exclude vgg19 test which comsumes too much memory, run out of memory on Upsquared device. Single test pass for vgg19, need furture investigation (#2588) 2. Update docker file to decrease the docker image size * Update docs for Android NNAPI EP (#2586) * Fix lto bug for protobuf and ubuntu * add path to build dir before test run (#2590) * Add missig env variables for mac pipeline test (#2595) * Fixed an issue in updating realized dims (#2597) when we update realized dims for scan's output, the sliced axis also needs to be inclusive, i.e. we should check with "dim >= insert_inclusive_axis", because the offset in the symbols are based on Scan sugraph. Otherwise, we would end up with shape mismatch later. * Java API for onnxruntime (#2215) * Add support for opset 11 in reshape fusion (#2592) Support opset verion 11 in reshape fusion * Rename automl python tools folder to featurizer_ops. (#2593) * Support opset 11 subgraph of Squad model in Embed Layer Normalization (#2605) Support opset 11 Squad model that is exported from PyTorch nightly. The embed layer uses Range op which is missed in the transformer. * symbolic shape inference: fix warnings in GPT-2 model (#2608) And revise nuphar perf test on BERT squad * Dump subgraph ID and fused graph ID (#2607) * Dump subgraph ID and fused graph ID Dump subgraph ID and fused graph ID for better debugging * Remove local static fused_count added a field global_fused_count_ to NupharExecutionProvider class * EmbedLayerNormalization Fusion For Dynamic Squad Model Opset 10 (#2613) Support subgraph of SQuAD model exported from pytorch with dynamic input axes * Allow providers to be set for InferenceSession at construction (#2606) * Remove unnecessary parameter in some places in GatherElements implementation (#2612) * Remove unnecessary parameter in some places * Update * Update * Make sure fenced tensor could not reuse other tensor. (#2561) Fix random error caused by this. * Improve Embed Layer Norm Fusion for SQuAD with static input shape (#2621) * fix float16 comparison in initializer (#2629) * epsilon attribute for layernormalization fusion (#2639) * removed unnecessary batch file and fix path (#2640) * Add shape inference to ConvTransposeWithDynamicPads schema (#2632) * Improve cuda expand() opeator's performance. (#2624) * Cuda pad optimize when no padding is needed. (#2625) * Shortcut cuda Pad() when no padding is needed. * Optimize cuda scatter() on 2D compatible. (#2628) * Optimize cuda scatter() on 2D compatible. * Add some comments. * fix build error for ARM (#2648) * Improve performance of resize() in Nearest mode (#2626) Special treatment for 2D, check same size as input image. And in 2d kernel, template use_expolation. * Fix memory exception in Layer Norm Fusion (#2644) * Windows CI changes(#2650) * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Revert "Windowsai without fi (#2701)" This reverts commit 285d4c85ff5c4e265f963208170304ef3461e684. * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Deref unique pointer for session_state * send shutdown event when dll is unloaded and EvaluationStop, SessionC… (#2704) * send shutdown event when dll is unloaded and EvaluationStop, SessionCreationStart Events. * Add EvalutationStart Event * add comment * use correct type for for loop (#2755) * ARM CI (#2759) * Set ARM agent pool * Set CMake generator to VS 2019 in ARM * Use system-wide CMake instead of custom version Our custom version is too old for VS 2019 * Use DML and build shared lib in ARM CI * Restore nuget packages in ARM CI * Disable DML * Refactor ARM debug/release builds * Use system packaged Python version * Remove hardcoded Python path * Downgrade Python to 3.7 for build * Remove explicit CMake path * Fix invalid JSON in cgmanifest.json (#2760) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * Remove usage of WHOLEARCHIVE in WinML CMake and add WinMLAdapterFactory (#2726) * Remove usage of WHOLEARCHIVE in WinMLAdapter CMake and add WinMLAdapterFactory * PR feedback, no need for dll(export) since using def file * PR comments * Small comment in gen_def.py * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * Move winml_provider_factory.h to proper location (#2801) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Filter CPU case for IsFloat16Supported (#2802) * Merge fixes * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * Fix warnings that cause build to fail * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * Make DML include path global (#2882) * Make DML include path global * Add generated cppwinrt headers to winml_lib_common * Integrate changes to WindowsAI to make ADO Build (#2886) * Revert "CMake cross-generator fixes (#2790)" This reverts commit dbe7d97fa1ab155f1309bced87199527e8f35bd2. * add additional suppress warning in onnx_proto * ignore /wd4996 warning * DML execution provider fixes * Revert "Revert "CMake cross-generator fixes (#2790)"" This reverts commit 1ae7b4bcbc02edc881ad28685da98e095dfceb17. * Update func signature of custom op function overloads * common devicehelpers fixes * Add pch.h for winml_lib_common * re-add winml_lib_common_dir/inc to include path for winml_adapter * User/orilevari/dml redist shared folder (#2890) * move dml nuget package directory up one level to make it shared between build flavors * Merge conflict fix * Revert "Merge conflict fix" This reverts commit 142fa72cf9ce4344ad717b50b7ea2b8582aadc7c. * Revert "Merge remote-tracking branch 'origin/master' into windowsai" This reverts commit 6e2126d46e5e5f564d65da37dd4f70c93dd81165, reversing changes made to b3f5583dc9249834b947c8ea905f6a98060d5bd6. * Make winml_test_common free of test macros (#2902) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Don't generate files outside the build root (#2914) * Don't generate files outside the build root * Add onnxruntime_EXTERNAL_DEPENDENCIES to WinML * Add DML depedency on RESTORE_PACKAGES * User/orilevari/fix yaml merge bugs (#2918) * Add winml test source parameter into cmake function (#2919) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Add arguments to unittest cmake functions * remove comment * Revert "Revert "Merge remote-tracking branch 'origin/master' into windowsai"" This reverts commit ade5abe72a4234fdbc3623093c61c02c6b0bdc26. * Fix breaks from merge with ORT master * Brianma/linux (#2917) * don't include windows.h in cross-plat header * add default case for switch statement * signed/unsigned mismatch fix Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> * User/sheilk/winml adapter c api (#2891) * Create winml adapter c api * fix build * make it build * move adapter into onnxruntime core/session * entry point not exported * minor changes * make model metadata work * make tests pass * implement all the model reflection apis on the adapter c abi * update the new ort interface to create a lotus ennvironment with a logging sink * start adding ort env * move all winml code into adapter folder/lib to isolate it * ensure a single logging manager at a time * start refactoring session * refactor session creation interface * add cpu and dml session option methods to adapter * finish session init * stub out interfaces in ort lib to perform similar mechanics of iinference session * enable profiling, and enable schema override * update session register graph transformers * turn back on custom registry for custom ops * Add sync api * add last c api stubs * should build... but all feature values are broken since this is in flight to moving all implementation details into ivalue * remove ep adapter header * Implement DML execution provider functions from adapter (#2846) * Implement DML execution provider functions from adapter * Use functions in OnnxruntimeEngine.cpp * make map/sequence type_infos freeable, and start implementing ivalue * make it build again * implement value methods * implement remaining methods * remove com adapter abi * check dml session * cache the allocator on ivalue * check if resource is cpu/gpu when access its mutable data * update tensor * mismatched parentheses * fix tensor base and binding obj * it evaluates tensors! sometimes... * minor fixes * enable gpu evals * wrapper all existing winml adapter apis with API_IMPL to try catch (#2854) * update winml... tensor strings are broken, need to template tensorbase to do different things for strings * make tensor strings work with 2 copies in/2 copies out * Fix tensor string and allocator bug * make maps work again... needs some fixes still * Make it build! * enable map inputs * map outputs * unbound outputs for sequences and maps * User/xianz/merge windowsai (#2883) * Packaging pipeline changes for VS 2019 (#2711) * Tiny fix to codegen * Simplify cache implementation and avoid static variables that may carry over between models * Extend DML kernels (#2641) * Additional DML operators * Check unsupported attributes and inputs * Address PR comments * Add kernel capability function used for partitioning, and re-enable stride-based int64 support based on value range * Fix test failures * Build fix * PR comments * Update Nuphar tutorial notebook (#2721) 1. Reflect int8 GEMV improvements for multi-threading from #2696 2. Add notes on multi-threading control using OpenMP 3. Add samples of running multi-isa AOT, and show int8 GEMM differences between AVX and AVX2 4. Add rnn_benchmark example to resolve #1993 * Add schema for new Qops (#2611) * Add schema for new Qops * adding shape inference + qlinearaveragepool * plus review comments * plus review comments * updates per review comments * plus review comments * [server] Add supposed for model_name and model_version as cli parameter (#2708) * remove 64bit warning message from python validation. (#2727) * MLAS: ARM64 build fix (#2734) fix bad usage of vreinterpret to cast vector element types * Fix broken python docs links (#2740) * Fix build on Mac OS (#2731) mac os ld doesn't support --while-archive, correct option is -all_load * fix ngraph wheel (#2737) * fix ngraph wheel 1.1.0 onnxruntime_ngraph wheel doesn't work * remove libdnnl.so in nGraph Libs * make it easy to compare * Split onnxruntime server to a separated folder (#2744) * Fix build for Python 3.8 (#2747) * Fix build for Python 3.8 * Update protobuf to 3.11.2 (#1928) Update protobuf to 3.11.2 (#1928) * Change default optimization level to All (from Basic) (#2745) * change default optimization level to All (from Basic) * fix test * fix c# test * Update numpy to 1.18 (#2758) * Update numpy to 1.18 * Pipeline changes for python 3.8 (#2753) 1. Pipeline changes for python 3.8 2. Fix a regression in setup.py which was just introduced in the previous commit. Please notice, we still haven't made python 3.8 + Windows + CUDA work. * Add basic stacktrace output for posix debug builds. (#2749) * [NupharEP] fix a race condition when multiple sessions running different models concurrently (#2772) * Revert "Change default optimization level to All (from Basic) (#2745)" This reverts commit 56bb503c2f26474b6613bcb2a198691a11dcef38. * Fix typo in error message (#2736) * Rename MKL-DNN to DNNL to fix broken link (#2730) * Fix nightly build version number issue * Pass BUILD_BUILDNUMBER to linux docker * Disable featurizers in python packages * Import more featurizers (#2781) Make kernels non-template. Add input constraint for learnt data. Add min_max_scalar_transformer, robust_scalar_transformer, inputation_marker_transfomer, label_encoder_transformer, missing_dummies_transformer along with tests. Advance Featurizers library commit. * Implement a more stable softmax (#2715) * Implement a more stable SoftMax e^x is represented as infinity if x is large enough, like 100.f. Infinity divided by Infinity is a NAN. Thus, softmax gets a NAN if one or more item are large enough. A math transform as below is leveraged to get a stable softmax: e^xi/(e^x1 + ...e^xn) = e^(xi - max) / (e^(x1 - max) + ... + e^(xn - max)) And for convenience, force max to 0.f if all xi are negative * Contributing: Fix a typo (#2784) * ACL EP GEMM improvements (#2780) When it is posible we use a fully connected layer instead of the gemm implementation. This will let the library use the best implementation based on the input data. * ACL EP convolution improvements (#2774) Added the optimized implementation for depthwise convolution for both ACL v19.02 and ACL 19.05. Also the pointwise convolution seems to be more optimal in the CPU implementation so we opted for that instead. * Add script for release Nuget validation (#2719) * Initial commit * Nits * Disable a test temporarily * Change working directory * Test * Add download python step * Test update * More changes * Fix space issue * Fix * Verify nuget signing * Fix * Spaces * PR feedback * Nit * Fix * Fix * Remove temporary changes * add uint8 support to where op (#2792) * Improve bert optimization script: (#2712) (1) Move input int64=>int32 conversion to embed layer fusion. (2) Output epsilon attribute for LayerNormalization fusion. * add session creation time cost. (#2798) * ML.NET team needs featurizers within a package (#2789) Add auto ml featurizers to Windows, MacOS as well as to GPU packaging-pipelines. * Initialize max of softmax with lowest of float (#2786) * MLAS: update SGEMM threading parameters (#2808) * add interface to copy batch tensors. (#2807) * add interface to copy batch tensors. * onnxruntime * speed up Windows TRT CI (#2811) * don't run cuda tests if building with tensorrt * remove unnecessary build options for win trt ci * refactor win gpu tensorrt ci yml * --numpy_version=1.17 * update * update * azcopy and cuda path * Update test data (#2356) * Add timeseries imputer transformer featurizer kernel (#2813) Make kernels non-template. Add input constraint for learnt data. Fixup tests. Add two more featurizers along with tests. Tests fail. min_max_scalar_transformer robust_scalar_transformer Fix tests serialized stream by prepending version bytes. Add inputation_marker_transfomer and the test. Fix up float/double type designations. Added label_encoder_transformer along with a test. string_throw case is broken at the momement. Fix labelencodertransfomer_test.cc string_throw case Rename maxabsscalertransformer_test.cc Add MissingDummiesTransformer along with the test. Update manifest. Add TimeSeriesImputerTransformer definition, implementation and tests * Fix memory leak in TRT (#2815) * fix memory leak issue * revert EP_FAIL on enueueV2 * Add manifest missing comma * Run static code analyzer on most of our code (#2817) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * update quantization doc (#2783) * update documentation for quantization script * plus some spell corrections * Filter CPU case for IsFloat16Supported (#2802) * update default optimization level + fix gemm_activation fusion (#2791) * update defualt optimization level + fix gemm_activation fusion * fix typo * add unit test and incorporate review comments * fix test comment * Fix dnnl wheel package name (#2823) * Append '-dnnl' to whl package name when --use_dnnl * Update build.py * Update Ubuntu & TensorRT version in README (#2820) Dockerfile.tensorrt is using nvcr.io/nvidia/tensorrt:19.09-py3 as base Image, update Ubuntu and TensorRT version according to https://docs.nvidia.com/deeplearning/sdk/tensorrt-container-release-notes/rel_19-09.html#rel_19-09 * Merge fixes * Add OneHotEncoder and HashOneHotEncoder kernels. (#2830) Add defs and imlementation for OneHotEncoders, adjuist date_time_transformer kernel and test. Add OneHotEncoder kernel test. Add HashOneHotVectorizerTransformer unit test. This does not link due to multiple definitions of functions that are included into header from a CPP file. * Upgrade gtest to the latest version (#2827) WinML would like to update the googletest submodule. They want some newer features (namely GTEST_SKIP to skip tests programmatically and be able to skip entire fixtures easily) and would need to update the submodule version. However, because the new version of code hit a bug in gcc, even though the bug is already fixed in the latest gcc but we're using gcc 4.8.x and it won't get patched for the bug, so we have to do a compromise, change our code a little bit to make it work. The gcc bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=51213 * Add support for int64_t for topk CPU. Fixes github issue #2806. (#2833) * Ignore allocator type in ExecutionProviders allocator map. Make default initialization of OrtMemoryInfo more clearly invalid. (#2768) * Remove allocator type from the key comparison in ExecutionProviders. Remove usage of DummyArena as it's no longer necessary. * Fix x86 tests where arena allocator is disabled. Make initialization of OrtMemoryInfo clearer by adding Invalid enum value. * Make OrtValueNameIdxMap::MaxIdx more intuitive. * Convert ExternalProject Featurizers into git submodule (#2834) Add git submodule for Featurizer library. Update cmake to build for git submodule. * add domain check for nodes + update documentation (#2831) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Fix Linux CUDA nuget packaging pipeline break * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * update optimization doc for BERT related fusions (#2819) * Add bert related transformers to doc * Add execution provider and comment for bert optimizations * Add comment about accuracy impact of approximation * Fix warnings that cause build to fail * MLAS: enable threading for quantized GEMMs (#2844) * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * remove old winmladapter cpp Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * move sequence implementation into ort lib... still commented out... need to turn back on... * begin sequence implementation * make maps and sequences work * fix broken tests * remove dead code * misc cleanup * CR feedback * User/xianz/winml adapter c api (#2869) * wrapper all existing winml adapter apis with API_IMPL to try catch * Return HR or Throw for WinML adapter APIs if failed * undo macro wrapper for two places * Wrap error macros around ort apis, too. * address CR feedback #2 * add more api throw/return macros * Revert changes no longer needed * revert changes to cxx api * format winml lib.ort and winml adapter * remove static pheonix singleton Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * missing use_dml check in winml_adapter_session (#2930) * --use_dnnl flag was mangled in merge (#2931) * use dml macro not wrapping custom registry code (#2934) * Disable LNK4199 winml_dll to enable cuda builds (#2936) * Disable LNK4199 in winml_dll * linkler->linker * LearningModelSessionAPITestGpu.CreateSessionWithCastToFloat16InModel should return DXGI_ERROR_UNSUPPORTED when FP16 not supported (#2937) * Disable LNK4199 in winml_dll * linkler->linker * Need to return DXGI_ERROR_UNSUPPORTED when Model does not support fp16 * Publish build symbols (#2939) * Publish build symbols * Don't upload PDBs for .exe files * Make x86 build (#2943) * fix last remaining size_t/int64_t warnings->errors (#2948) * TensorString, Sequences and Maps use the first allocator, but should use the cpu default allocator. (#2952) * fix tensor string allcoator * clean up default allocator usage for strings in winml lib/api.ort Co-authored-by: Ryan Lai <ryalai96@gmail.com> * Handle tensor shape of zero (#2954) Co-authored-by: Ryan Lai <ryalai96@gmail.com> * CR feedback (#2970) * CR feedback * fix weird formatting on privacy readme * Add 'All rights reserved.' everywhere * readd all rights reserved to winml_provider_factory.h * remove extra space in comment * remove extra whitespace * fixes post master merge * remove winml from nuget gpu pipeline * set IR VERSION on generated_model in rnn_benchmark (#2972) * Fix slice conformance failures (#2908) Co-authored-by: Adrian Tsai <adtsai@microsoft.com> Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Paul McDaniel <paul_mcdaniel@hotmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Dwayne Robinson <fdwr@hotmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com>
2020-02-05 01:12:19 +00:00
[marshaling_behavior(agile)]
[dualapipartition(1)]
runtimeclass LearningModelEvaluationResult
{
//! The optional user supplied string that was attached to the Evaluate() call to connect the output results.
String CorrelationId{ get; };
//! If the evaluation failed, returns an error code for what caused the failure.
Int32 ErrorStatus{ get; };
//! True if the evaluation completed successfully.
//! If False, use ErrorStatus to find out what caused the failure.
Boolean Succeeded{ get; };
//! A set of features representing the output prediction along with probabilities.
Windows.Foundation.Collections.IMapView<String, IInspectable> Outputs{ get; };
}
//! \class LearningModelSessionOptions
//! \brief TODO:Docs
[contract(MachineLearningContract, 2)]
[dualapipartition(1)]
runtimeclass LearningModelSessionOptions
{
// default constructor
LearningModelSessionOptions();
//! The BatchSizeOverride option will allow the model compiler to use constant batch size performance optimizations when setting up the LearningModelSession.
//! The default value for the BatchSizeOverride will be 1 indicating a static batch size of 1.
//! BatchSizeOverride = 0 indicates that the batch size present in the model should be honored.
//! BatchSizeOverride > 0 indicates the size of batch that will be used to override the model batch size and optimize evaluations.
UInt32 BatchSizeOverride { get; set; };
[contract(MachineLearningContract, 4)]
{
//! The OverrideNamedDimension method will allow the model compiler to use constant batch size performance optimizations when setting up the LearningModelSession.
//! The caller can specify the size of the dimension for a given named dimension.
//! dimension = 0 indicates that the dimension present in the model should be honored.
//! dimension > 0 indicates the size of the dimension that will be used to override the model "name" dimension and optimize evaluations.
void OverrideNamedDimension(String name, UInt32 dimension);
}
Merge windowsai (winml layering) into master (#2956) * Initial Commit * Merged PR 3985217: add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc (#2346) add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc and violating our OS layering requirements. We linked against onecoreuap_apiset.lib in VB so we will continue doing this, but I am still unsure why not to link against onecore instead since that is where we ship. However, since Sheil is the owner of this code we will wait to discuss with him before changing anything. * Initial changes for layering * more snipping to get core into ort * update build instructions to include --build_shared_lib (#2358) * update build instructions to include --build_shared_lib * fix line breaks * Task 23998197: add winml_lib_core into onnnxruntime.dll (#2368) * Task 23998197: add winml_lib_core into onnnxruntime.dll * PR feedback build break on perf_test * return proper error when the model path isn't found (#2391) * LearningModelSession is cleaned up to use the adapter, and parts of b… (#2382) this is a big PR. we are going to move it up to layer_dev , which is still a L3 so we are still safe to do work there agile. we are going to move this into the L3 so that ryan can start doing intergration testing. we will pause for a full code review and integration test result prior to going into the L2. >>>> raw comments from previous commits >>> * LearningModelSession is cleaned up to use the adapter, and parts of binding are. * moved everything in the winmladapter made it all nano-com using, WRL to construct objects in the ORT side. base interfaces for everythign for winml to call cleaned up a bunch of winml to use the base interfaces. * more pieces * GetData across the abi. * renamed some namepsace cleaned up OrtValue cleaned up Tensor cleaned up custom ops. everything *but* learnignmodel should be clean * make sure it's building. winml.dll is still a monolith. * model moved over. everything builds clean. step ! * weak ref comment * Layer dev paulm (#2408) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * Layer dev paulm (#2414) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * Layer dev paulm (#2423) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * Layer dev paulm (#2424) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * Layer dev paulm (#2425) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Layer dev paulm (#2426) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * fixed map and sequence when passing stl types across the ABI . found a leak in nvidia driver, but skipped it. all winmlapitests pass now * Moved SessionOptions over to the abi * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * Register WinML TraceLogging provider on Onnxruntime.dll (#2455) * Register WinML TraceLogging provider on Onnxruntime.dll * Add ifdef to make sure trace logging provider has telemetry option when LAYERING_DONE * No need for ifdef for TraceLoggingOptionMicrosoftTelemetry * PR feedback * Move etw registration into lotus environment constructor and deresgister in lotus environment destructor * Brianma/cpuwinml (#2466) * allow building winml cpu without dml. * Brianma/breaks (#2469) * fix some more breaks * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * move dml checks out of winml and into the adapter * better error handling * Brianma/fi (#2470) * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * fix bad merge * Checking in a staging checkpoint point so that Ryan can work with me in parrallel * build break. * Brianma/testfails (#2473) * add missing ir version to dictvectorizer-string.onnx * add missing ir version to relu.onnx * add missing ir version to zipmap*onnx * add IR version to manually generated models * remove an unnecessary ifdef dml * Brianma/windowsai fi (#2475) * update dockerfiles/README (#2336) * Make elementwise op run 4 items per thread (#2335) Description: Describe your changes. Make elementwise op run 4 items per thread unroll for loop to leverage ILP remove unnessary N==0 check inside elementwise GPU kernel Motivation and Context Why is this change required? What problem does it solve? It can improve the performance of GPU elementwise ops. ~2% performance gain on popular NLP bert model. If it fixes an open issue, please link to the issue here. * Add CUDA GatherElements kernel (#2310) * Updates * Update test * Update * Updates * nits * PR feedback * Update * Update * PR feedback * PR comments * Update * Fix build * Fix build * Nits * Fix * Layer Normalization Fusion (#2319) basic layer normalization transform * Add FastGelu Cuda Op for Gelu and Add bias fusion (#2293) * Add FastGelu cuda op * Add AddBiasGelu for experiment * Revert "Add AddBiasGelu for experiment" This reverts commit 5c1ee019858c657e6bb75887265cb85675626e5b. * Add bias * Add unit tests * update comment * update script * fix build error * update coding style * update for CR feedback Enable half2 optimization only when cuda arch >= 7.0 * move _Tanh to common.cuh * implement CPU contrib OP Attention (#2333) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanUnusedInitializers. (#2320) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanupUnusedInitializers. This means initializers that have been replaced during graph optimizations are not left in the GraphProto when we save an optimized model. * Handle edge case where a model has an unused initializer with matching graph input by also removing the graph input. * Use non-const iterators in std::find_if calls to make centos build happy. * Nuget pipeline changes (#2305) 1. refactor the pipeline, remove some duplicated code 2. Move Windows_py_GPU_Wheels job to Win-GPU-CUDA10. We'll deprecated the "Win-GPU" pool 3. Delete cpu-nocontribops-esrp-pipeline.yml and cpu-nocontribops-pipeline.yml 4. In Linux nuget jobs, run "make install" before creating the package. So that extra RPAH info will be removed * Cuda Reverse Sequence Op, maping types of same size using same template function. (#2281) * Set ElementType to String type of node metadata, instead of byte[] (#2348) * Set ElementType to String type of node metadata, instead of byte[] * Fix spacing * Introduce PrimitiveType into a Type System along with an integer constant (#2307) Improve perf by avoiding GetType<T>() calls. Introduce MLTypeCallDispatcher to switch on Input Type. Add Tensor IsType<T>() fast method. * Fix/test dim value of 0 handling in a couple of places (#2337) * Update the CUDA Where implementation broadcasting logic to handle a dim with value of 0. Add unit test Also add unit test for unary op with dim value of 0 * Exclude ngraph from Where test with 0 dim. * Openvino EP R3.1 onnxrt server (#2357) * onnxrt server with OVEP * onnxrt server with OVEP * Update Dockerfile.server.openvino * onnxrt server OVEP fix reviews * onnxrt server OVEP fix reviews * Implement cuda nonzero op. (#2056) Implement cuda nonzero op. * Direct use python numpy array's memory if already contiguous. (#2355) * Direct use python numpy array's memory if already contiguous. This could greatly improve performance for session with large input, like big image 1920x1080 fastrcnn, 30~40% speed up could be achieved. * Add test case enforce contiguous/non-contiguos numpy array as inputs. * Add helper to create output to minimize binary size. (#2365) Add ConstEigenTensorMap typedef so we don't unnecessarily const_cast the const input Tensor. * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS (#2369) * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS * update * Add Tracelogging for profiling (#1639) Enabled only if onnxruntime_ENABLE_INSTRUMENT is ON * test bidaf with nuphar for avx target (#2370) increase nuphar test coverage a bit * Fix a bug in TLS refcount that may destabilized CUDA CI (#2374) * update output size calculation for resize (#2366) * change how output size is calculated for resize op * add tests for ver 10 resize * Extend OneHot CPU kernel to support more types (#2311) * Extend OneHot CPU kernel to support input int64_t, depth int32_t, output float * Skip BERT before the test data fix is picked up * Fix bug with Slice. Need to pass in flattened input dimensions so the initial offset into the input is calculated correctly. (#2372) * Add opset 11 version of Split to CUDA ops (#2376) Organize the CUDA ops definitions so all the opset 10 and 11 parts are together (same setup used for CPU ops) * Layer Norm Fusion Fix (#2379) * layer norm fusion fix * Add input shape check in code and unit tests * Fuse Add + Gelu (#2360) Implement the transformer to fuse add + gelu Implement the accurate kernel * Skip layer norm transform (#2350) * skip layer normalization transformer * Another try to stabilize CUDA CI (#2383) The root cause seems to be failure in CUDA dealloc when tear down. cudaFree return code was ignored before, so should the debug check. * fix BUILD.md typo (#2375) build.py: error: argument --config: invalid choice: 'RelWithDebugInfo' (choose from 'Debug', 'MinSizeRel', 'Release', 'RelWithDebInfo') * Fixed compilation with ngraph (#2388) * Fix reuse logic in allocation planner. (#2393) * Fix reuse logic in allocation planner. * PR comments * Add helpful comments * Don't allow reuse across string tensors. * [NupharEP] Multiple optimizations (#2380) Fuse transpose into MatMul Implement Pow and constant scalar simplification Vectorize ReduceMean Improve symbolic shape inference Minor updates for better debugging in fused function name * Avoid using the default logger in the graph lib and optimizers (#2361) 1. Use the session logger if it is available. 2. Don't disable warning 4100 globally. We should fix the warnings instead of disabling it. * Change CUDA implementation of Transpose to support all fixed size tensor types (#2387) * Change CUDA implementation of Transpose to not use a typed kernel so we can support more types with minimum binary size. Add support for 8, 16, 32 and 64 bit types. Add unit tests. Add method so the implementation can be called directly (will be used by CUDA Scan very soon). * Disable TensorRT for MLFloat16 and int8 unit tests. * Address PR comment and add support for calling cublas implementation if type is mlfloat16. * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. (#2398) * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. * [NupharEP] force some low/zero cost ops to be inlined (#2409) * fix cross compile bug (#2415) * Minor optimization: if a node has already been placed, there's no need to find a kernel for it. (#2417) * Add Reshape Fusion (#2395) * Add reshape fusion * Add some comments * update comments * update comment format * update according to feedback * update for recent logger change * fix build error * (1) Support both input and output edges in find path in graphutils (2) Add a test case of only one constant initializer of Concat input. (3) Refactor ReshapeFusion class to allow add more subgraph fusion in the future. * fix error * (1) loose constraint on initializer: non constant is allowed for reshape fusion. (2) Change versions type to vector. (3) Add logging. (4) Return false when multiple output edges matched in FindPath. Add comments. * only allow one direction (input or output) in FindPath * [NupharEP] Update notebook and docker image (#2416) Add BERT squad in Nuphar tutorial Enhance speed comparsion readability * Fix the issue in matmul_add_fusion (#2407) Fix the issue in matmul_add_fusion If Muatmul + Add has shape [K] * [K, N], reset it to [1, K] * [K, N] will make the output shape to [1, N] will also requires a reshape on the output. Fix: just remove the shape reset to not fuse it. Add a negative test case for matmul+add fusion * feat(treeregressor): Update TreeEnsembleRegressor for type support (#2389) Updates the `TreeEnsembleRegressor` to allow for `double`, `float`, `int64`, and `int32` inputs to match the upstream specification. Signed-off-by: Nick Groszewski <nicholas.groszewski@capitalone.com> * onnxrt server documentation update (#2396) * Added support for Pad-2 operator in OpenVINO-EP (#2405) * Add CUDA If operator. (#2377) * Add CUDA If operator. Uses CPU operator for implementation. By adding a CUDA version the inputs/outputs (with the exception of the 'cond' input) stay on GPU, and no other logic is required to avoid a copy to CPU across the control flow node. * Improved documentation for onnxruntime::utils::SwapByteOrderCopy(), added precondition check. * Fix the type constraints on CUDA If operator to exclude strings. (#2431) * add Im2col<uint8_t> (#2438) * Adjust codegen vectorization width from target (#2439) * Adjust codegen vectorization width from target * Add CUDA Scan operator. (#2403) * Add Scan CUDA op. Uses CPU implementation for logic. Added some device specific functors for handling when data needs to be manipulated on a different device. Added ability to override the materialization logic in the OrtValue slicer so DML can plugin their handling. * Fix Windows GPU C API packaging pipeline failure (#2440) Fix Windows GPU C API packaging pipeline failure (#2440) * Correctly handle implicit inputs for fused nodes (#2390) * Correctly handle implicit inputs for fused nodes Previously, nuphar's partitioning function didn't include node's implicit inputs into the inputs list of MetaDef, and hence a crash was triggered in the onnx graph checker. This commit fixed the issue. Furthermore, it also fixed a related issue where we didn't add implicit inputs into graph_inputs_excluding_initializers_ in Graph::SetGraphInputsOutputs. the issue was that graph_inputs_including_initializers_ populated by SetInputs (e.g. called by FunctionImpl::FunctionImpl) may contain implicit inputs which were not of any node's initializers in the graph. Because they were not part of any initializers, these implicit inputs couldn't be visited by going through all nodes' inputs. Consequently, they would *not* be added into graph_inputs_excluding_initializers_. We fixed the issue by first copying the populated graph_inputs_including_initializers_ into graph_inputs_excluding_initalizers_, which then had both initializers and non-initializers as its initial content. Later, we erase initializers from the list. In this way, we can ensure all implicit inputs to remain in graph_inputs_excluding_initializers_. * refined comments and fixed duplicates Address CR by revisiting comments in terms of implicit inputs Also fixed an issue by skipping duplicates while copying inputs from graph_inputs_including_initializers_. * address CR explain why we need to collect nodes' implicit inputs * don't rely on pointer values for iterating std::set Previously, openvino relied on iterating a set of NodeArg pointers to construct inputs and outputs for a fused graph. It could cause non-determinism. The reason was that although iterating std::set by itself is stable, pointer values of NodeArgs may vary. Consequently, we could end up visiting the set's elements in different orders for different runs for the same test, which resulted in constructing inputs (and outputs) with different orders to the fused graph. For example, for the same test, we may have inputs [A, B] in some runs but inputs[B, A] in others. Let's use std::string as the key type to avoid such nondeterminism. This commit also added implicit inputs into meta->inputs while returning the capability from the openvino provider. * Fixed another latent issue in openvino's GetCapability function The issue was that we couldn't simply erase fused_inputs and fused_outputs while iterating the nodes. For example, an output NodeArg may have multiple uses, and it's wrong if we erase it from fused_outputs when we encounter only one of its uses as input. * Remove DeviceAllocatorRegistry class (#2451) Remove DeviceAllocatorRegistry class * CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library. * Parallel Gelu with ParallelFor (#2399) Parallel Gelu to get better performance for Gelu * Clean up build.py (#2446) * Pull the latest image before running docker build * Fuse SkipLayerNorm with Bias (#2453) Fuse SkipLayerNorm with Bias * Allow more than one invocation of CreateEnv in the same process. (#2467) * Allow more than one invocation of CreateEnv in the same process. * Fix centos build * Symbolic shape inference improvements: (#2460) * Symbolic shape inference improvements: - add a mode to guess unknown ops' output rank - add support for GatherND - add support for If - fix a bug in get_int_values when then tensor rank > 1D, by treating it as no sympy data - add symbol to literal merge when ONNX silently merges dims - fix a bug in Concat when input dim is 0 - fix a bug in ConstantOfShape that computed dim is not updated - add support for dynamic shape in ConstantOfShape - fix a bug in Loop output shape that loop iterator dim is not inserted at dim 0 - add support for dynamic padding in Pad - add support for dynamic shape in Reshape - add support for Resize with opset > 10, by treating output dims as dynamic - fix a bug in Slice when starts/ends are dynamic - restrict input model to opset 7 and above - make output model optional to avoid disk write when testing Run model tests for symbolic shape inference Reduce 2GB docker image size of nuphar * add additional test data set for nuget pipeline (#2448) * add SAS token to download internal test data for nuget pipeline * update azure endpoint * fix keyvault download step * fix variable declaration for secret group * fix indentation * fix yaml syntax for variables * fix setting secrets for script * fix env synctax * Fix macos pipeline * attempt to add secrets to windows download data * fix mac and win data download * fix windows data download * update test data set url and location * Revert "Brianma/windowsai fi (#2475)" This reverts commit 5780b864a15513fda4eadbfc2b5345fefe70b5ec. * Add scenario tests (#2457) * Add scenario tests * Remove TODO from model license * Add winml_api test dependency * fix model load test. fi from master changed the constructor (#2483) * make api tests all pass (#2486) * fix bad merge * fix bad model merge * Layer dev paulm (#2492) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * Rename ambiguous header (#2489) * fix one more missing IR version model (#2500) * add missing IR version to 4 more models used by scenario tests (#2501) * Add CLI parameters to test runner, build WinML in ARM and x86 CI (#2479) * Support test parameters through CLI arguments * Add WinML do Windows x86/ARM CI builds * Code style fixes * Update googletest Remove GPUTEST macros everywhere now that GTEST_SKIP is supported * Refactor main.cpp * Build scenario tests without DML * Link scenario tests to DML when it's enabled (#2502) * Layer dev release pipeline (#2488) Adds winml binaries to existing cpu nuget package, and creates new gpu dml nuget package with winml binaries and DML EP. * Layer dev paulm (#2506) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * Remove usage of IOBinding in WinML and use C_API Run method (#2504) * remove usage of iobinding * Change data structure to use vector of Ort::Values * Polish bind input / output * Use C APIrun method * Update providers on evaluate getresults * Remove run and IObinding interface from WinMLAdapter * Remove use of IObinding * bind unbound outputs code moved to learningmodelbinding * clean up unneeded istensor adapter function * Fix comment * Check if session is closed before binding and clearing * PR feedback * Layer dev paulm (#2507) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * Make tests dependend on winml_dll (#2509) * add dml binaries to DirectML package and be more explicit about condition variables (#2520) * re-enable warnings for winml builds and fix the warnings that were hiding (#2526) * turn devmode back on for winml builds * fix some warnings. include protobuf in a way that disables some warnings * undo protobufhelpers changes and just ignore 4100 errors in pb code * attempt to isolate protobufhelpers errors * add template specialization for getting tensor proto data * Layer dev paulm (#2533) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * Spawn child process to run DeviceLostRecovery scenario test (#2530) * Spawn child process to run DeviceLostRecovery scenario test * Layer dev paulm (#2536) ori said yes * add missing namespace to winml_trace_logging_provider in lotusenvironment.h (#2542) * Handle exception thrown from all apis in WinMLAdapter (#2539) * various changes to unblock windowsai ADO build * Fix custom ops scenario tests (#2562) * Do not shutdown protobuf after ort environment gets destroyed. Lazy load lotus environment first time it is needed * comment typo * pr comment about calling phoenix singleton * Make lotus_environment static in winmladapter * Layer dev paulm (#2567) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * making windowsAI azure dev ops work. * code review comments. * revert changes * Cmake and preprocessor fixes that where uncovered by building on agents without DML available via SDK * Layer dev dml delayload (#2580) * Brianma/cpu (#2583) * don't include dml stuff in cpu builds * tests that link the image lib also need the telemetry lib now * Throw Winml_err_invalid_binding if binding gpu resource on cpu device (#2589) * Throw Winml_err_invalid_binding if binding gpu resource on cpu device * PR comments. No need to query executionprovider for is gpu device * User/xianz/ortthrow (#2596) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * User/xianz/ortthrow (#2599) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * add status error message * Remove uwp onsuspending winrt call because logruntimeperf is getting removed (#2630) * User/xianz/dedup telemetry (#2631) * investigate duplication of telemetry in winml and ort * remove winml telemetry events * telemetry executionProviderEvent * remove unneccessary file and refactor code little bit * Revert back TelemetryEvent, which send up ETW event. * merge changes from layer_dev to windowsai (#2638) * Remove underscore from googletest names (#2616) * Fix leaking memory allocator Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24278761 and https://microsoft.visualstudio.com/OS/_workitems/edit/24330198 * Explicitly initialize Ort::Value with nullptr * Cache WinML adapter * bad merge * define private version of dxcore enum that is added in 19H1 SDK. (#2654) * add comment for explaning private definition of dxcore d3d feature level ennum value. (#2672) * do not package directml.pdb for redist packages. (#2676) * Fix leaking operator registry (#2645) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24354916 * User/orilevari/windowsai master merge (#2674) merge resolutions included pulling in telemetry logic that was merged to master and not windowsai and dereferencing InferenceSession::sessionstate now that it is a unique pointer * Delete Ort Allocator in LearningModelBinding (#2653) * Delete OrtAllocator in LearningModelBinding * PR comments to make Ort::Allocator a smart pointer * Small comment change * PR feedback to clean up code * PR feedback on move semantics * Clean up std::move * Fix memory leaks (#2679) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24356109, https://microsoft.visualstudio.com/OS/_workitems/edit/24388361 and https://microsoft.visualstudio.com/OS/_workitems/edit/24388596 * various changes to properly organize and skip GPU tests. For now for No DML builds we will not run GPU tests at all. In the future we should adapt the tests to expect the appropiate errors. (#2695) * Windowsai without fi (#2701) * Disable Attention fusion tests when DISABLE_CONTRIB_OPS is defined (#2529) * Setup java ci (#2528) * Add provision in ORT for session options to be parsed when available via model file (#2449) * Initial commit * Fix gitmodules * Nits * Nits * Updates * Update * More changes * Updates * Update * Some updates * More changes * Update * Update * Merge * Update * Updates * More changes * Update * Fix nits * Updates * Fix warning * Fix build * Add comment * PR feedback * PR feedback * Updates * Updates * Update * More changes * Fix build break * Comment test for now * Updates * Updates * PR feedback * Updates * Nits * Add tests * Fix build * Fix build * Fix build * Fix build break * Fix build * Nits * PR feedback * More change * Expose GetSessionOptions in pybind logic and add unit test for python * Fix build * PR feedback * PR feedback * Revert "Disable thread pool creation when enabled OpenMP (#2485)" (#2535) This reverts commit 7c7d5a149c9ed52eec67304bae5c4b132166a8a1. * Add dynamic shape support in TensorRT execution provider (#2450) * remove onnx-tensorrt submodule * add new onnx-tensorrt submodule (experiment) for trt6 * update engine build for trt6 * update compile and compute for tensorrt6.0 * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * switch to onnx-tensorrt master for TensorRT6' * Update tensorrt_execution_provider.cc * Handle dynamic batch size and add memcpy in TensorRT EP * update test cases * Update tensorrt_execution_provider.cc * update onnx-tensorrt submodule * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.ubuntu_tensorrt * Update run_dockerbuild.sh * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update concat_op_test.cc * Update tensorrt_execution_provider.cc * Upgrade TensorRT to version 6.0.1.5 * Update onnxruntime_providers.cmake * Update CMakeLists.txt * Update reduction_ops_test.cc * Update install_ubuntu.sh * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.tensorrt * Update BUILD.md * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update onnxruntime_providers.cmake * Update install_ubuntu.sh * Update install_ubuntu.sh * Update gemm_test.cc * Update gather_op_test.cc * Update CMakeLists.txt * Removed submodule * update onnx-tensorrt submodule * update header file * Removed submodule * add submodule onnx-tensorrt kevin's branch shape-test' * add debugging code * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * merge master * Removed submodule * update onnx-tensorrt submodule * add more changes for dynamic shapes * Update tensorrt_execution_provider.cc * update for dynamic shape * update dynamic shape processing * fix logger issue * remove submodule onnx-tensorrt * add submodule onnx-tensorrt * add env variable min_subgraph_size * remove redundency * update document * use onnxruntime::make_unique * fix multi-run issue * remove some tests to save CI build time * Add dynamic shape test * Update TensorRT-ExecutionProvider.md * Add example of running Faster R-CNN model on TensorRT EP * Add more details on env variables * update environment variables * Update tensorrt_basic_test.cc * Update model tests * Update tensor_op_test.cc * remove --use_full_protobuf * Update build.py * User/xianz/telemetry (#2458) * enabme telemetry * enable telemetry * set enable telemetry as default * for debugging * remove log and set disable telemetry as default back * delete private file while testing * resolve comment: mainly add license header, rename macro and update docs * rewording in privacy.md * Fix integer overflow in cuda NonMaxSuppression implementation (#2540) * add test case that should pass but fail * fix nms * extract int_max_output_boxes_per_class * Introduce container type runtime checks and other improvements (#2522) Rework TensorSeq in a manner consistent with Tensor and SparseTensor in terms of type system setup. Reduce templating. Introduce helpers to ensure the same data type. Make OrtValue __dtor not virtual. Introduce ContainerChecker * Fix C API tests for centos and mac (#2544) * change c++14 to c++11 * add ld lib path for centos * enable csharp tests on macos * fix C API test on MacOS + fix manylinux dotnet install * fix manylinux dotnet install * fix lib link * Add back executable bit to build.py * Fix a bug handling negative begin pad values in Pad op (#2550) * Fix bug in Pad op * Update * DNNL CMAKE update (#2548) * Fix android build (#2558) * Update win-x86-ci.yml (#2557) Fix build pipeline break * Re-enable Windows C# tests (#2564) * disable onnx_test_runner -x invocations for dnnl (#2568) * Allow sequence length to be symbolic (#2559) * setup java ci mac (#2570) * make layernorm fusion to support opset 11 (#2545) * Fix a warning found in the latest VS release * Add more check on SkipLayerNorm and BiasGelu fusion (#2574) * Fix file not found error during docker build. (#2569) * Add ConvTranspose1D (#2578) * Ryanunderhill/packagename test (#2582) * [Nuphar EP] fixes for some object detection models (#2581) Update notebook tutorial with multi-threaded int8 GEMM from #2517 * EmbedLayerNormalization Fusion Improvement (#2553) Embedding layer norm fusion improvements - add more checks * Update version (#2584) * Temporarily exclude vgg19 test from Python backend test 1. temporarily exclude vgg19 test which comsumes too much memory, run out of memory on Upsquared device. Single test pass for vgg19, need furture investigation (#2588) 2. Update docker file to decrease the docker image size * Update docs for Android NNAPI EP (#2586) * Fix lto bug for protobuf and ubuntu * add path to build dir before test run (#2590) * Add missig env variables for mac pipeline test (#2595) * Fixed an issue in updating realized dims (#2597) when we update realized dims for scan's output, the sliced axis also needs to be inclusive, i.e. we should check with "dim >= insert_inclusive_axis", because the offset in the symbols are based on Scan sugraph. Otherwise, we would end up with shape mismatch later. * Java API for onnxruntime (#2215) * Add support for opset 11 in reshape fusion (#2592) Support opset verion 11 in reshape fusion * Rename automl python tools folder to featurizer_ops. (#2593) * Support opset 11 subgraph of Squad model in Embed Layer Normalization (#2605) Support opset 11 Squad model that is exported from PyTorch nightly. The embed layer uses Range op which is missed in the transformer. * symbolic shape inference: fix warnings in GPT-2 model (#2608) And revise nuphar perf test on BERT squad * Dump subgraph ID and fused graph ID (#2607) * Dump subgraph ID and fused graph ID Dump subgraph ID and fused graph ID for better debugging * Remove local static fused_count added a field global_fused_count_ to NupharExecutionProvider class * EmbedLayerNormalization Fusion For Dynamic Squad Model Opset 10 (#2613) Support subgraph of SQuAD model exported from pytorch with dynamic input axes * Allow providers to be set for InferenceSession at construction (#2606) * Remove unnecessary parameter in some places in GatherElements implementation (#2612) * Remove unnecessary parameter in some places * Update * Update * Make sure fenced tensor could not reuse other tensor. (#2561) Fix random error caused by this. * Improve Embed Layer Norm Fusion for SQuAD with static input shape (#2621) * fix float16 comparison in initializer (#2629) * epsilon attribute for layernormalization fusion (#2639) * removed unnecessary batch file and fix path (#2640) * Add shape inference to ConvTransposeWithDynamicPads schema (#2632) * Improve cuda expand() opeator's performance. (#2624) * Cuda pad optimize when no padding is needed. (#2625) * Shortcut cuda Pad() when no padding is needed. * Optimize cuda scatter() on 2D compatible. (#2628) * Optimize cuda scatter() on 2D compatible. * Add some comments. * fix build error for ARM (#2648) * Improve performance of resize() in Nearest mode (#2626) Special treatment for 2D, check same size as input image. And in 2d kernel, template use_expolation. * Fix memory exception in Layer Norm Fusion (#2644) * Windows CI changes(#2650) * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Revert "Windowsai without fi (#2701)" This reverts commit 285d4c85ff5c4e265f963208170304ef3461e684. * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Deref unique pointer for session_state * send shutdown event when dll is unloaded and EvaluationStop, SessionC… (#2704) * send shutdown event when dll is unloaded and EvaluationStop, SessionCreationStart Events. * Add EvalutationStart Event * add comment * use correct type for for loop (#2755) * ARM CI (#2759) * Set ARM agent pool * Set CMake generator to VS 2019 in ARM * Use system-wide CMake instead of custom version Our custom version is too old for VS 2019 * Use DML and build shared lib in ARM CI * Restore nuget packages in ARM CI * Disable DML * Refactor ARM debug/release builds * Use system packaged Python version * Remove hardcoded Python path * Downgrade Python to 3.7 for build * Remove explicit CMake path * Fix invalid JSON in cgmanifest.json (#2760) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * Remove usage of WHOLEARCHIVE in WinML CMake and add WinMLAdapterFactory (#2726) * Remove usage of WHOLEARCHIVE in WinMLAdapter CMake and add WinMLAdapterFactory * PR feedback, no need for dll(export) since using def file * PR comments * Small comment in gen_def.py * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * Move winml_provider_factory.h to proper location (#2801) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Filter CPU case for IsFloat16Supported (#2802) * Merge fixes * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * Fix warnings that cause build to fail * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * Make DML include path global (#2882) * Make DML include path global * Add generated cppwinrt headers to winml_lib_common * Integrate changes to WindowsAI to make ADO Build (#2886) * Revert "CMake cross-generator fixes (#2790)" This reverts commit dbe7d97fa1ab155f1309bced87199527e8f35bd2. * add additional suppress warning in onnx_proto * ignore /wd4996 warning * DML execution provider fixes * Revert "Revert "CMake cross-generator fixes (#2790)"" This reverts commit 1ae7b4bcbc02edc881ad28685da98e095dfceb17. * Update func signature of custom op function overloads * common devicehelpers fixes * Add pch.h for winml_lib_common * re-add winml_lib_common_dir/inc to include path for winml_adapter * User/orilevari/dml redist shared folder (#2890) * move dml nuget package directory up one level to make it shared between build flavors * Merge conflict fix * Revert "Merge conflict fix" This reverts commit 142fa72cf9ce4344ad717b50b7ea2b8582aadc7c. * Revert "Merge remote-tracking branch 'origin/master' into windowsai" This reverts commit 6e2126d46e5e5f564d65da37dd4f70c93dd81165, reversing changes made to b3f5583dc9249834b947c8ea905f6a98060d5bd6. * Make winml_test_common free of test macros (#2902) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Don't generate files outside the build root (#2914) * Don't generate files outside the build root * Add onnxruntime_EXTERNAL_DEPENDENCIES to WinML * Add DML depedency on RESTORE_PACKAGES * User/orilevari/fix yaml merge bugs (#2918) * Add winml test source parameter into cmake function (#2919) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Add arguments to unittest cmake functions * remove comment * Revert "Revert "Merge remote-tracking branch 'origin/master' into windowsai"" This reverts commit ade5abe72a4234fdbc3623093c61c02c6b0bdc26. * Fix breaks from merge with ORT master * Brianma/linux (#2917) * don't include windows.h in cross-plat header * add default case for switch statement * signed/unsigned mismatch fix Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> * User/sheilk/winml adapter c api (#2891) * Create winml adapter c api * fix build * make it build * move adapter into onnxruntime core/session * entry point not exported * minor changes * make model metadata work * make tests pass * implement all the model reflection apis on the adapter c abi * update the new ort interface to create a lotus ennvironment with a logging sink * start adding ort env * move all winml code into adapter folder/lib to isolate it * ensure a single logging manager at a time * start refactoring session * refactor session creation interface * add cpu and dml session option methods to adapter * finish session init * stub out interfaces in ort lib to perform similar mechanics of iinference session * enable profiling, and enable schema override * update session register graph transformers * turn back on custom registry for custom ops * Add sync api * add last c api stubs * should build... but all feature values are broken since this is in flight to moving all implementation details into ivalue * remove ep adapter header * Implement DML execution provider functions from adapter (#2846) * Implement DML execution provider functions from adapter * Use functions in OnnxruntimeEngine.cpp * make map/sequence type_infos freeable, and start implementing ivalue * make it build again * implement value methods * implement remaining methods * remove com adapter abi * check dml session * cache the allocator on ivalue * check if resource is cpu/gpu when access its mutable data * update tensor * mismatched parentheses * fix tensor base and binding obj * it evaluates tensors! sometimes... * minor fixes * enable gpu evals * wrapper all existing winml adapter apis with API_IMPL to try catch (#2854) * update winml... tensor strings are broken, need to template tensorbase to do different things for strings * make tensor strings work with 2 copies in/2 copies out * Fix tensor string and allocator bug * make maps work again... needs some fixes still * Make it build! * enable map inputs * map outputs * unbound outputs for sequences and maps * User/xianz/merge windowsai (#2883) * Packaging pipeline changes for VS 2019 (#2711) * Tiny fix to codegen * Simplify cache implementation and avoid static variables that may carry over between models * Extend DML kernels (#2641) * Additional DML operators * Check unsupported attributes and inputs * Address PR comments * Add kernel capability function used for partitioning, and re-enable stride-based int64 support based on value range * Fix test failures * Build fix * PR comments * Update Nuphar tutorial notebook (#2721) 1. Reflect int8 GEMV improvements for multi-threading from #2696 2. Add notes on multi-threading control using OpenMP 3. Add samples of running multi-isa AOT, and show int8 GEMM differences between AVX and AVX2 4. Add rnn_benchmark example to resolve #1993 * Add schema for new Qops (#2611) * Add schema for new Qops * adding shape inference + qlinearaveragepool * plus review comments * plus review comments * updates per review comments * plus review comments * [server] Add supposed for model_name and model_version as cli parameter (#2708) * remove 64bit warning message from python validation. (#2727) * MLAS: ARM64 build fix (#2734) fix bad usage of vreinterpret to cast vector element types * Fix broken python docs links (#2740) * Fix build on Mac OS (#2731) mac os ld doesn't support --while-archive, correct option is -all_load * fix ngraph wheel (#2737) * fix ngraph wheel 1.1.0 onnxruntime_ngraph wheel doesn't work * remove libdnnl.so in nGraph Libs * make it easy to compare * Split onnxruntime server to a separated folder (#2744) * Fix build for Python 3.8 (#2747) * Fix build for Python 3.8 * Update protobuf to 3.11.2 (#1928) Update protobuf to 3.11.2 (#1928) * Change default optimization level to All (from Basic) (#2745) * change default optimization level to All (from Basic) * fix test * fix c# test * Update numpy to 1.18 (#2758) * Update numpy to 1.18 * Pipeline changes for python 3.8 (#2753) 1. Pipeline changes for python 3.8 2. Fix a regression in setup.py which was just introduced in the previous commit. Please notice, we still haven't made python 3.8 + Windows + CUDA work. * Add basic stacktrace output for posix debug builds. (#2749) * [NupharEP] fix a race condition when multiple sessions running different models concurrently (#2772) * Revert "Change default optimization level to All (from Basic) (#2745)" This reverts commit 56bb503c2f26474b6613bcb2a198691a11dcef38. * Fix typo in error message (#2736) * Rename MKL-DNN to DNNL to fix broken link (#2730) * Fix nightly build version number issue * Pass BUILD_BUILDNUMBER to linux docker * Disable featurizers in python packages * Import more featurizers (#2781) Make kernels non-template. Add input constraint for learnt data. Add min_max_scalar_transformer, robust_scalar_transformer, inputation_marker_transfomer, label_encoder_transformer, missing_dummies_transformer along with tests. Advance Featurizers library commit. * Implement a more stable softmax (#2715) * Implement a more stable SoftMax e^x is represented as infinity if x is large enough, like 100.f. Infinity divided by Infinity is a NAN. Thus, softmax gets a NAN if one or more item are large enough. A math transform as below is leveraged to get a stable softmax: e^xi/(e^x1 + ...e^xn) = e^(xi - max) / (e^(x1 - max) + ... + e^(xn - max)) And for convenience, force max to 0.f if all xi are negative * Contributing: Fix a typo (#2784) * ACL EP GEMM improvements (#2780) When it is posible we use a fully connected layer instead of the gemm implementation. This will let the library use the best implementation based on the input data. * ACL EP convolution improvements (#2774) Added the optimized implementation for depthwise convolution for both ACL v19.02 and ACL 19.05. Also the pointwise convolution seems to be more optimal in the CPU implementation so we opted for that instead. * Add script for release Nuget validation (#2719) * Initial commit * Nits * Disable a test temporarily * Change working directory * Test * Add download python step * Test update * More changes * Fix space issue * Fix * Verify nuget signing * Fix * Spaces * PR feedback * Nit * Fix * Fix * Remove temporary changes * add uint8 support to where op (#2792) * Improve bert optimization script: (#2712) (1) Move input int64=>int32 conversion to embed layer fusion. (2) Output epsilon attribute for LayerNormalization fusion. * add session creation time cost. (#2798) * ML.NET team needs featurizers within a package (#2789) Add auto ml featurizers to Windows, MacOS as well as to GPU packaging-pipelines. * Initialize max of softmax with lowest of float (#2786) * MLAS: update SGEMM threading parameters (#2808) * add interface to copy batch tensors. (#2807) * add interface to copy batch tensors. * onnxruntime * speed up Windows TRT CI (#2811) * don't run cuda tests if building with tensorrt * remove unnecessary build options for win trt ci * refactor win gpu tensorrt ci yml * --numpy_version=1.17 * update * update * azcopy and cuda path * Update test data (#2356) * Add timeseries imputer transformer featurizer kernel (#2813) Make kernels non-template. Add input constraint for learnt data. Fixup tests. Add two more featurizers along with tests. Tests fail. min_max_scalar_transformer robust_scalar_transformer Fix tests serialized stream by prepending version bytes. Add inputation_marker_transfomer and the test. Fix up float/double type designations. Added label_encoder_transformer along with a test. string_throw case is broken at the momement. Fix labelencodertransfomer_test.cc string_throw case Rename maxabsscalertransformer_test.cc Add MissingDummiesTransformer along with the test. Update manifest. Add TimeSeriesImputerTransformer definition, implementation and tests * Fix memory leak in TRT (#2815) * fix memory leak issue * revert EP_FAIL on enueueV2 * Add manifest missing comma * Run static code analyzer on most of our code (#2817) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * update quantization doc (#2783) * update documentation for quantization script * plus some spell corrections * Filter CPU case for IsFloat16Supported (#2802) * update default optimization level + fix gemm_activation fusion (#2791) * update defualt optimization level + fix gemm_activation fusion * fix typo * add unit test and incorporate review comments * fix test comment * Fix dnnl wheel package name (#2823) * Append '-dnnl' to whl package name when --use_dnnl * Update build.py * Update Ubuntu & TensorRT version in README (#2820) Dockerfile.tensorrt is using nvcr.io/nvidia/tensorrt:19.09-py3 as base Image, update Ubuntu and TensorRT version according to https://docs.nvidia.com/deeplearning/sdk/tensorrt-container-release-notes/rel_19-09.html#rel_19-09 * Merge fixes * Add OneHotEncoder and HashOneHotEncoder kernels. (#2830) Add defs and imlementation for OneHotEncoders, adjuist date_time_transformer kernel and test. Add OneHotEncoder kernel test. Add HashOneHotVectorizerTransformer unit test. This does not link due to multiple definitions of functions that are included into header from a CPP file. * Upgrade gtest to the latest version (#2827) WinML would like to update the googletest submodule. They want some newer features (namely GTEST_SKIP to skip tests programmatically and be able to skip entire fixtures easily) and would need to update the submodule version. However, because the new version of code hit a bug in gcc, even though the bug is already fixed in the latest gcc but we're using gcc 4.8.x and it won't get patched for the bug, so we have to do a compromise, change our code a little bit to make it work. The gcc bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=51213 * Add support for int64_t for topk CPU. Fixes github issue #2806. (#2833) * Ignore allocator type in ExecutionProviders allocator map. Make default initialization of OrtMemoryInfo more clearly invalid. (#2768) * Remove allocator type from the key comparison in ExecutionProviders. Remove usage of DummyArena as it's no longer necessary. * Fix x86 tests where arena allocator is disabled. Make initialization of OrtMemoryInfo clearer by adding Invalid enum value. * Make OrtValueNameIdxMap::MaxIdx more intuitive. * Convert ExternalProject Featurizers into git submodule (#2834) Add git submodule for Featurizer library. Update cmake to build for git submodule. * add domain check for nodes + update documentation (#2831) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Fix Linux CUDA nuget packaging pipeline break * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * update optimization doc for BERT related fusions (#2819) * Add bert related transformers to doc * Add execution provider and comment for bert optimizations * Add comment about accuracy impact of approximation * Fix warnings that cause build to fail * MLAS: enable threading for quantized GEMMs (#2844) * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * remove old winmladapter cpp Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * move sequence implementation into ort lib... still commented out... need to turn back on... * begin sequence implementation * make maps and sequences work * fix broken tests * remove dead code * misc cleanup * CR feedback * User/xianz/winml adapter c api (#2869) * wrapper all existing winml adapter apis with API_IMPL to try catch * Return HR or Throw for WinML adapter APIs if failed * undo macro wrapper for two places * Wrap error macros around ort apis, too. * address CR feedback #2 * add more api throw/return macros * Revert changes no longer needed * revert changes to cxx api * format winml lib.ort and winml adapter * remove static pheonix singleton Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * missing use_dml check in winml_adapter_session (#2930) * --use_dnnl flag was mangled in merge (#2931) * use dml macro not wrapping custom registry code (#2934) * Disable LNK4199 winml_dll to enable cuda builds (#2936) * Disable LNK4199 in winml_dll * linkler->linker * LearningModelSessionAPITestGpu.CreateSessionWithCastToFloat16InModel should return DXGI_ERROR_UNSUPPORTED when FP16 not supported (#2937) * Disable LNK4199 in winml_dll * linkler->linker * Need to return DXGI_ERROR_UNSUPPORTED when Model does not support fp16 * Publish build symbols (#2939) * Publish build symbols * Don't upload PDBs for .exe files * Make x86 build (#2943) * fix last remaining size_t/int64_t warnings->errors (#2948) * TensorString, Sequences and Maps use the first allocator, but should use the cpu default allocator. (#2952) * fix tensor string allcoator * clean up default allocator usage for strings in winml lib/api.ort Co-authored-by: Ryan Lai <ryalai96@gmail.com> * Handle tensor shape of zero (#2954) Co-authored-by: Ryan Lai <ryalai96@gmail.com> * CR feedback (#2970) * CR feedback * fix weird formatting on privacy readme * Add 'All rights reserved.' everywhere * readd all rights reserved to winml_provider_factory.h * remove extra space in comment * remove extra whitespace * fixes post master merge * remove winml from nuget gpu pipeline * set IR VERSION on generated_model in rnn_benchmark (#2972) * Fix slice conformance failures (#2908) Co-authored-by: Adrian Tsai <adtsai@microsoft.com> Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Paul McDaniel <paul_mcdaniel@hotmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Dwayne Robinson <fdwr@hotmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com>
2020-02-05 01:12:19 +00:00
[contract(MachineLearningContract, 3)]
{
//! The CloseModelOnSessionCreation option will allow the LearningModelSession to take ownership of the LearningModel's
//! internal model representation. This will defunct the LearningModel session, but decreases the necessary peak working set.
//! CloseModelOnSessionCreation = True indicates that the model's internal data will be moved into the session during construction.
//! CloseModelOnSessionCreation = False indicates that the model's internal data will be copied into the session during construction.
Boolean CloseModelOnSessionCreation { get; set; };
}
}
//! \class LearningModelSession
//! \brief TODO:Docs
[contract(MachineLearningContract, 1)]
INBOX_ONLY([constructor_name("Windows.AI.MachineLearning.ILearningModelSessionFactory", 0f6b881d-1c9b-47b6-bfe0-f1cf62a67579)])
INBOX_ONLY([interface_name("Windows.AI.MachineLearning.ILearningModelSession", 8e58f8f6-b787-4c11-90f0-7129aeca74a9)])
Merge windowsai (winml layering) into master (#2956) * Initial Commit * Merged PR 3985217: add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc (#2346) add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc and violating our OS layering requirements. We linked against onecoreuap_apiset.lib in VB so we will continue doing this, but I am still unsure why not to link against onecore instead since that is where we ship. However, since Sheil is the owner of this code we will wait to discuss with him before changing anything. * Initial changes for layering * more snipping to get core into ort * update build instructions to include --build_shared_lib (#2358) * update build instructions to include --build_shared_lib * fix line breaks * Task 23998197: add winml_lib_core into onnnxruntime.dll (#2368) * Task 23998197: add winml_lib_core into onnnxruntime.dll * PR feedback build break on perf_test * return proper error when the model path isn't found (#2391) * LearningModelSession is cleaned up to use the adapter, and parts of b… (#2382) this is a big PR. we are going to move it up to layer_dev , which is still a L3 so we are still safe to do work there agile. we are going to move this into the L3 so that ryan can start doing intergration testing. we will pause for a full code review and integration test result prior to going into the L2. >>>> raw comments from previous commits >>> * LearningModelSession is cleaned up to use the adapter, and parts of binding are. * moved everything in the winmladapter made it all nano-com using, WRL to construct objects in the ORT side. base interfaces for everythign for winml to call cleaned up a bunch of winml to use the base interfaces. * more pieces * GetData across the abi. * renamed some namepsace cleaned up OrtValue cleaned up Tensor cleaned up custom ops. everything *but* learnignmodel should be clean * make sure it's building. winml.dll is still a monolith. * model moved over. everything builds clean. step ! * weak ref comment * Layer dev paulm (#2408) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * Layer dev paulm (#2414) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * Layer dev paulm (#2423) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * Layer dev paulm (#2424) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * Layer dev paulm (#2425) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Layer dev paulm (#2426) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * fixed map and sequence when passing stl types across the ABI . found a leak in nvidia driver, but skipped it. all winmlapitests pass now * Moved SessionOptions over to the abi * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * Register WinML TraceLogging provider on Onnxruntime.dll (#2455) * Register WinML TraceLogging provider on Onnxruntime.dll * Add ifdef to make sure trace logging provider has telemetry option when LAYERING_DONE * No need for ifdef for TraceLoggingOptionMicrosoftTelemetry * PR feedback * Move etw registration into lotus environment constructor and deresgister in lotus environment destructor * Brianma/cpuwinml (#2466) * allow building winml cpu without dml. * Brianma/breaks (#2469) * fix some more breaks * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * move dml checks out of winml and into the adapter * better error handling * Brianma/fi (#2470) * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * fix bad merge * Checking in a staging checkpoint point so that Ryan can work with me in parrallel * build break. * Brianma/testfails (#2473) * add missing ir version to dictvectorizer-string.onnx * add missing ir version to relu.onnx * add missing ir version to zipmap*onnx * add IR version to manually generated models * remove an unnecessary ifdef dml * Brianma/windowsai fi (#2475) * update dockerfiles/README (#2336) * Make elementwise op run 4 items per thread (#2335) Description: Describe your changes. Make elementwise op run 4 items per thread unroll for loop to leverage ILP remove unnessary N==0 check inside elementwise GPU kernel Motivation and Context Why is this change required? What problem does it solve? It can improve the performance of GPU elementwise ops. ~2% performance gain on popular NLP bert model. If it fixes an open issue, please link to the issue here. * Add CUDA GatherElements kernel (#2310) * Updates * Update test * Update * Updates * nits * PR feedback * Update * Update * PR feedback * PR comments * Update * Fix build * Fix build * Nits * Fix * Layer Normalization Fusion (#2319) basic layer normalization transform * Add FastGelu Cuda Op for Gelu and Add bias fusion (#2293) * Add FastGelu cuda op * Add AddBiasGelu for experiment * Revert "Add AddBiasGelu for experiment" This reverts commit 5c1ee019858c657e6bb75887265cb85675626e5b. * Add bias * Add unit tests * update comment * update script * fix build error * update coding style * update for CR feedback Enable half2 optimization only when cuda arch >= 7.0 * move _Tanh to common.cuh * implement CPU contrib OP Attention (#2333) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanUnusedInitializers. (#2320) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanupUnusedInitializers. This means initializers that have been replaced during graph optimizations are not left in the GraphProto when we save an optimized model. * Handle edge case where a model has an unused initializer with matching graph input by also removing the graph input. * Use non-const iterators in std::find_if calls to make centos build happy. * Nuget pipeline changes (#2305) 1. refactor the pipeline, remove some duplicated code 2. Move Windows_py_GPU_Wheels job to Win-GPU-CUDA10. We'll deprecated the "Win-GPU" pool 3. Delete cpu-nocontribops-esrp-pipeline.yml and cpu-nocontribops-pipeline.yml 4. In Linux nuget jobs, run "make install" before creating the package. So that extra RPAH info will be removed * Cuda Reverse Sequence Op, maping types of same size using same template function. (#2281) * Set ElementType to String type of node metadata, instead of byte[] (#2348) * Set ElementType to String type of node metadata, instead of byte[] * Fix spacing * Introduce PrimitiveType into a Type System along with an integer constant (#2307) Improve perf by avoiding GetType<T>() calls. Introduce MLTypeCallDispatcher to switch on Input Type. Add Tensor IsType<T>() fast method. * Fix/test dim value of 0 handling in a couple of places (#2337) * Update the CUDA Where implementation broadcasting logic to handle a dim with value of 0. Add unit test Also add unit test for unary op with dim value of 0 * Exclude ngraph from Where test with 0 dim. * Openvino EP R3.1 onnxrt server (#2357) * onnxrt server with OVEP * onnxrt server with OVEP * Update Dockerfile.server.openvino * onnxrt server OVEP fix reviews * onnxrt server OVEP fix reviews * Implement cuda nonzero op. (#2056) Implement cuda nonzero op. * Direct use python numpy array's memory if already contiguous. (#2355) * Direct use python numpy array's memory if already contiguous. This could greatly improve performance for session with large input, like big image 1920x1080 fastrcnn, 30~40% speed up could be achieved. * Add test case enforce contiguous/non-contiguos numpy array as inputs. * Add helper to create output to minimize binary size. (#2365) Add ConstEigenTensorMap typedef so we don't unnecessarily const_cast the const input Tensor. * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS (#2369) * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS * update * Add Tracelogging for profiling (#1639) Enabled only if onnxruntime_ENABLE_INSTRUMENT is ON * test bidaf with nuphar for avx target (#2370) increase nuphar test coverage a bit * Fix a bug in TLS refcount that may destabilized CUDA CI (#2374) * update output size calculation for resize (#2366) * change how output size is calculated for resize op * add tests for ver 10 resize * Extend OneHot CPU kernel to support more types (#2311) * Extend OneHot CPU kernel to support input int64_t, depth int32_t, output float * Skip BERT before the test data fix is picked up * Fix bug with Slice. Need to pass in flattened input dimensions so the initial offset into the input is calculated correctly. (#2372) * Add opset 11 version of Split to CUDA ops (#2376) Organize the CUDA ops definitions so all the opset 10 and 11 parts are together (same setup used for CPU ops) * Layer Norm Fusion Fix (#2379) * layer norm fusion fix * Add input shape check in code and unit tests * Fuse Add + Gelu (#2360) Implement the transformer to fuse add + gelu Implement the accurate kernel * Skip layer norm transform (#2350) * skip layer normalization transformer * Another try to stabilize CUDA CI (#2383) The root cause seems to be failure in CUDA dealloc when tear down. cudaFree return code was ignored before, so should the debug check. * fix BUILD.md typo (#2375) build.py: error: argument --config: invalid choice: 'RelWithDebugInfo' (choose from 'Debug', 'MinSizeRel', 'Release', 'RelWithDebInfo') * Fixed compilation with ngraph (#2388) * Fix reuse logic in allocation planner. (#2393) * Fix reuse logic in allocation planner. * PR comments * Add helpful comments * Don't allow reuse across string tensors. * [NupharEP] Multiple optimizations (#2380) Fuse transpose into MatMul Implement Pow and constant scalar simplification Vectorize ReduceMean Improve symbolic shape inference Minor updates for better debugging in fused function name * Avoid using the default logger in the graph lib and optimizers (#2361) 1. Use the session logger if it is available. 2. Don't disable warning 4100 globally. We should fix the warnings instead of disabling it. * Change CUDA implementation of Transpose to support all fixed size tensor types (#2387) * Change CUDA implementation of Transpose to not use a typed kernel so we can support more types with minimum binary size. Add support for 8, 16, 32 and 64 bit types. Add unit tests. Add method so the implementation can be called directly (will be used by CUDA Scan very soon). * Disable TensorRT for MLFloat16 and int8 unit tests. * Address PR comment and add support for calling cublas implementation if type is mlfloat16. * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. (#2398) * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. * [NupharEP] force some low/zero cost ops to be inlined (#2409) * fix cross compile bug (#2415) * Minor optimization: if a node has already been placed, there's no need to find a kernel for it. (#2417) * Add Reshape Fusion (#2395) * Add reshape fusion * Add some comments * update comments * update comment format * update according to feedback * update for recent logger change * fix build error * (1) Support both input and output edges in find path in graphutils (2) Add a test case of only one constant initializer of Concat input. (3) Refactor ReshapeFusion class to allow add more subgraph fusion in the future. * fix error * (1) loose constraint on initializer: non constant is allowed for reshape fusion. (2) Change versions type to vector. (3) Add logging. (4) Return false when multiple output edges matched in FindPath. Add comments. * only allow one direction (input or output) in FindPath * [NupharEP] Update notebook and docker image (#2416) Add BERT squad in Nuphar tutorial Enhance speed comparsion readability * Fix the issue in matmul_add_fusion (#2407) Fix the issue in matmul_add_fusion If Muatmul + Add has shape [K] * [K, N], reset it to [1, K] * [K, N] will make the output shape to [1, N] will also requires a reshape on the output. Fix: just remove the shape reset to not fuse it. Add a negative test case for matmul+add fusion * feat(treeregressor): Update TreeEnsembleRegressor for type support (#2389) Updates the `TreeEnsembleRegressor` to allow for `double`, `float`, `int64`, and `int32` inputs to match the upstream specification. Signed-off-by: Nick Groszewski <nicholas.groszewski@capitalone.com> * onnxrt server documentation update (#2396) * Added support for Pad-2 operator in OpenVINO-EP (#2405) * Add CUDA If operator. (#2377) * Add CUDA If operator. Uses CPU operator for implementation. By adding a CUDA version the inputs/outputs (with the exception of the 'cond' input) stay on GPU, and no other logic is required to avoid a copy to CPU across the control flow node. * Improved documentation for onnxruntime::utils::SwapByteOrderCopy(), added precondition check. * Fix the type constraints on CUDA If operator to exclude strings. (#2431) * add Im2col<uint8_t> (#2438) * Adjust codegen vectorization width from target (#2439) * Adjust codegen vectorization width from target * Add CUDA Scan operator. (#2403) * Add Scan CUDA op. Uses CPU implementation for logic. Added some device specific functors for handling when data needs to be manipulated on a different device. Added ability to override the materialization logic in the OrtValue slicer so DML can plugin their handling. * Fix Windows GPU C API packaging pipeline failure (#2440) Fix Windows GPU C API packaging pipeline failure (#2440) * Correctly handle implicit inputs for fused nodes (#2390) * Correctly handle implicit inputs for fused nodes Previously, nuphar's partitioning function didn't include node's implicit inputs into the inputs list of MetaDef, and hence a crash was triggered in the onnx graph checker. This commit fixed the issue. Furthermore, it also fixed a related issue where we didn't add implicit inputs into graph_inputs_excluding_initializers_ in Graph::SetGraphInputsOutputs. the issue was that graph_inputs_including_initializers_ populated by SetInputs (e.g. called by FunctionImpl::FunctionImpl) may contain implicit inputs which were not of any node's initializers in the graph. Because they were not part of any initializers, these implicit inputs couldn't be visited by going through all nodes' inputs. Consequently, they would *not* be added into graph_inputs_excluding_initializers_. We fixed the issue by first copying the populated graph_inputs_including_initializers_ into graph_inputs_excluding_initalizers_, which then had both initializers and non-initializers as its initial content. Later, we erase initializers from the list. In this way, we can ensure all implicit inputs to remain in graph_inputs_excluding_initializers_. * refined comments and fixed duplicates Address CR by revisiting comments in terms of implicit inputs Also fixed an issue by skipping duplicates while copying inputs from graph_inputs_including_initializers_. * address CR explain why we need to collect nodes' implicit inputs * don't rely on pointer values for iterating std::set Previously, openvino relied on iterating a set of NodeArg pointers to construct inputs and outputs for a fused graph. It could cause non-determinism. The reason was that although iterating std::set by itself is stable, pointer values of NodeArgs may vary. Consequently, we could end up visiting the set's elements in different orders for different runs for the same test, which resulted in constructing inputs (and outputs) with different orders to the fused graph. For example, for the same test, we may have inputs [A, B] in some runs but inputs[B, A] in others. Let's use std::string as the key type to avoid such nondeterminism. This commit also added implicit inputs into meta->inputs while returning the capability from the openvino provider. * Fixed another latent issue in openvino's GetCapability function The issue was that we couldn't simply erase fused_inputs and fused_outputs while iterating the nodes. For example, an output NodeArg may have multiple uses, and it's wrong if we erase it from fused_outputs when we encounter only one of its uses as input. * Remove DeviceAllocatorRegistry class (#2451) Remove DeviceAllocatorRegistry class * CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library. * Parallel Gelu with ParallelFor (#2399) Parallel Gelu to get better performance for Gelu * Clean up build.py (#2446) * Pull the latest image before running docker build * Fuse SkipLayerNorm with Bias (#2453) Fuse SkipLayerNorm with Bias * Allow more than one invocation of CreateEnv in the same process. (#2467) * Allow more than one invocation of CreateEnv in the same process. * Fix centos build * Symbolic shape inference improvements: (#2460) * Symbolic shape inference improvements: - add a mode to guess unknown ops' output rank - add support for GatherND - add support for If - fix a bug in get_int_values when then tensor rank > 1D, by treating it as no sympy data - add symbol to literal merge when ONNX silently merges dims - fix a bug in Concat when input dim is 0 - fix a bug in ConstantOfShape that computed dim is not updated - add support for dynamic shape in ConstantOfShape - fix a bug in Loop output shape that loop iterator dim is not inserted at dim 0 - add support for dynamic padding in Pad - add support for dynamic shape in Reshape - add support for Resize with opset > 10, by treating output dims as dynamic - fix a bug in Slice when starts/ends are dynamic - restrict input model to opset 7 and above - make output model optional to avoid disk write when testing Run model tests for symbolic shape inference Reduce 2GB docker image size of nuphar * add additional test data set for nuget pipeline (#2448) * add SAS token to download internal test data for nuget pipeline * update azure endpoint * fix keyvault download step * fix variable declaration for secret group * fix indentation * fix yaml syntax for variables * fix setting secrets for script * fix env synctax * Fix macos pipeline * attempt to add secrets to windows download data * fix mac and win data download * fix windows data download * update test data set url and location * Revert "Brianma/windowsai fi (#2475)" This reverts commit 5780b864a15513fda4eadbfc2b5345fefe70b5ec. * Add scenario tests (#2457) * Add scenario tests * Remove TODO from model license * Add winml_api test dependency * fix model load test. fi from master changed the constructor (#2483) * make api tests all pass (#2486) * fix bad merge * fix bad model merge * Layer dev paulm (#2492) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * Rename ambiguous header (#2489) * fix one more missing IR version model (#2500) * add missing IR version to 4 more models used by scenario tests (#2501) * Add CLI parameters to test runner, build WinML in ARM and x86 CI (#2479) * Support test parameters through CLI arguments * Add WinML do Windows x86/ARM CI builds * Code style fixes * Update googletest Remove GPUTEST macros everywhere now that GTEST_SKIP is supported * Refactor main.cpp * Build scenario tests without DML * Link scenario tests to DML when it's enabled (#2502) * Layer dev release pipeline (#2488) Adds winml binaries to existing cpu nuget package, and creates new gpu dml nuget package with winml binaries and DML EP. * Layer dev paulm (#2506) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * Remove usage of IOBinding in WinML and use C_API Run method (#2504) * remove usage of iobinding * Change data structure to use vector of Ort::Values * Polish bind input / output * Use C APIrun method * Update providers on evaluate getresults * Remove run and IObinding interface from WinMLAdapter * Remove use of IObinding * bind unbound outputs code moved to learningmodelbinding * clean up unneeded istensor adapter function * Fix comment * Check if session is closed before binding and clearing * PR feedback * Layer dev paulm (#2507) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * Make tests dependend on winml_dll (#2509) * add dml binaries to DirectML package and be more explicit about condition variables (#2520) * re-enable warnings for winml builds and fix the warnings that were hiding (#2526) * turn devmode back on for winml builds * fix some warnings. include protobuf in a way that disables some warnings * undo protobufhelpers changes and just ignore 4100 errors in pb code * attempt to isolate protobufhelpers errors * add template specialization for getting tensor proto data * Layer dev paulm (#2533) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * Spawn child process to run DeviceLostRecovery scenario test (#2530) * Spawn child process to run DeviceLostRecovery scenario test * Layer dev paulm (#2536) ori said yes * add missing namespace to winml_trace_logging_provider in lotusenvironment.h (#2542) * Handle exception thrown from all apis in WinMLAdapter (#2539) * various changes to unblock windowsai ADO build * Fix custom ops scenario tests (#2562) * Do not shutdown protobuf after ort environment gets destroyed. Lazy load lotus environment first time it is needed * comment typo * pr comment about calling phoenix singleton * Make lotus_environment static in winmladapter * Layer dev paulm (#2567) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * making windowsAI azure dev ops work. * code review comments. * revert changes * Cmake and preprocessor fixes that where uncovered by building on agents without DML available via SDK * Layer dev dml delayload (#2580) * Brianma/cpu (#2583) * don't include dml stuff in cpu builds * tests that link the image lib also need the telemetry lib now * Throw Winml_err_invalid_binding if binding gpu resource on cpu device (#2589) * Throw Winml_err_invalid_binding if binding gpu resource on cpu device * PR comments. No need to query executionprovider for is gpu device * User/xianz/ortthrow (#2596) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * User/xianz/ortthrow (#2599) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * add status error message * Remove uwp onsuspending winrt call because logruntimeperf is getting removed (#2630) * User/xianz/dedup telemetry (#2631) * investigate duplication of telemetry in winml and ort * remove winml telemetry events * telemetry executionProviderEvent * remove unneccessary file and refactor code little bit * Revert back TelemetryEvent, which send up ETW event. * merge changes from layer_dev to windowsai (#2638) * Remove underscore from googletest names (#2616) * Fix leaking memory allocator Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24278761 and https://microsoft.visualstudio.com/OS/_workitems/edit/24330198 * Explicitly initialize Ort::Value with nullptr * Cache WinML adapter * bad merge * define private version of dxcore enum that is added in 19H1 SDK. (#2654) * add comment for explaning private definition of dxcore d3d feature level ennum value. (#2672) * do not package directml.pdb for redist packages. (#2676) * Fix leaking operator registry (#2645) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24354916 * User/orilevari/windowsai master merge (#2674) merge resolutions included pulling in telemetry logic that was merged to master and not windowsai and dereferencing InferenceSession::sessionstate now that it is a unique pointer * Delete Ort Allocator in LearningModelBinding (#2653) * Delete OrtAllocator in LearningModelBinding * PR comments to make Ort::Allocator a smart pointer * Small comment change * PR feedback to clean up code * PR feedback on move semantics * Clean up std::move * Fix memory leaks (#2679) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24356109, https://microsoft.visualstudio.com/OS/_workitems/edit/24388361 and https://microsoft.visualstudio.com/OS/_workitems/edit/24388596 * various changes to properly organize and skip GPU tests. For now for No DML builds we will not run GPU tests at all. In the future we should adapt the tests to expect the appropiate errors. (#2695) * Windowsai without fi (#2701) * Disable Attention fusion tests when DISABLE_CONTRIB_OPS is defined (#2529) * Setup java ci (#2528) * Add provision in ORT for session options to be parsed when available via model file (#2449) * Initial commit * Fix gitmodules * Nits * Nits * Updates * Update * More changes * Updates * Update * Some updates * More changes * Update * Update * Merge * Update * Updates * More changes * Update * Fix nits * Updates * Fix warning * Fix build * Add comment * PR feedback * PR feedback * Updates * Updates * Update * More changes * Fix build break * Comment test for now * Updates * Updates * PR feedback * Updates * Nits * Add tests * Fix build * Fix build * Fix build * Fix build break * Fix build * Nits * PR feedback * More change * Expose GetSessionOptions in pybind logic and add unit test for python * Fix build * PR feedback * PR feedback * Revert "Disable thread pool creation when enabled OpenMP (#2485)" (#2535) This reverts commit 7c7d5a149c9ed52eec67304bae5c4b132166a8a1. * Add dynamic shape support in TensorRT execution provider (#2450) * remove onnx-tensorrt submodule * add new onnx-tensorrt submodule (experiment) for trt6 * update engine build for trt6 * update compile and compute for tensorrt6.0 * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * switch to onnx-tensorrt master for TensorRT6' * Update tensorrt_execution_provider.cc * Handle dynamic batch size and add memcpy in TensorRT EP * update test cases * Update tensorrt_execution_provider.cc * update onnx-tensorrt submodule * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.ubuntu_tensorrt * Update run_dockerbuild.sh * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update concat_op_test.cc * Update tensorrt_execution_provider.cc * Upgrade TensorRT to version 6.0.1.5 * Update onnxruntime_providers.cmake * Update CMakeLists.txt * Update reduction_ops_test.cc * Update install_ubuntu.sh * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.tensorrt * Update BUILD.md * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update onnxruntime_providers.cmake * Update install_ubuntu.sh * Update install_ubuntu.sh * Update gemm_test.cc * Update gather_op_test.cc * Update CMakeLists.txt * Removed submodule * update onnx-tensorrt submodule * update header file * Removed submodule * add submodule onnx-tensorrt kevin's branch shape-test' * add debugging code * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * merge master * Removed submodule * update onnx-tensorrt submodule * add more changes for dynamic shapes * Update tensorrt_execution_provider.cc * update for dynamic shape * update dynamic shape processing * fix logger issue * remove submodule onnx-tensorrt * add submodule onnx-tensorrt * add env variable min_subgraph_size * remove redundency * update document * use onnxruntime::make_unique * fix multi-run issue * remove some tests to save CI build time * Add dynamic shape test * Update TensorRT-ExecutionProvider.md * Add example of running Faster R-CNN model on TensorRT EP * Add more details on env variables * update environment variables * Update tensorrt_basic_test.cc * Update model tests * Update tensor_op_test.cc * remove --use_full_protobuf * Update build.py * User/xianz/telemetry (#2458) * enabme telemetry * enable telemetry * set enable telemetry as default * for debugging * remove log and set disable telemetry as default back * delete private file while testing * resolve comment: mainly add license header, rename macro and update docs * rewording in privacy.md * Fix integer overflow in cuda NonMaxSuppression implementation (#2540) * add test case that should pass but fail * fix nms * extract int_max_output_boxes_per_class * Introduce container type runtime checks and other improvements (#2522) Rework TensorSeq in a manner consistent with Tensor and SparseTensor in terms of type system setup. Reduce templating. Introduce helpers to ensure the same data type. Make OrtValue __dtor not virtual. Introduce ContainerChecker * Fix C API tests for centos and mac (#2544) * change c++14 to c++11 * add ld lib path for centos * enable csharp tests on macos * fix C API test on MacOS + fix manylinux dotnet install * fix manylinux dotnet install * fix lib link * Add back executable bit to build.py * Fix a bug handling negative begin pad values in Pad op (#2550) * Fix bug in Pad op * Update * DNNL CMAKE update (#2548) * Fix android build (#2558) * Update win-x86-ci.yml (#2557) Fix build pipeline break * Re-enable Windows C# tests (#2564) * disable onnx_test_runner -x invocations for dnnl (#2568) * Allow sequence length to be symbolic (#2559) * setup java ci mac (#2570) * make layernorm fusion to support opset 11 (#2545) * Fix a warning found in the latest VS release * Add more check on SkipLayerNorm and BiasGelu fusion (#2574) * Fix file not found error during docker build. (#2569) * Add ConvTranspose1D (#2578) * Ryanunderhill/packagename test (#2582) * [Nuphar EP] fixes for some object detection models (#2581) Update notebook tutorial with multi-threaded int8 GEMM from #2517 * EmbedLayerNormalization Fusion Improvement (#2553) Embedding layer norm fusion improvements - add more checks * Update version (#2584) * Temporarily exclude vgg19 test from Python backend test 1. temporarily exclude vgg19 test which comsumes too much memory, run out of memory on Upsquared device. Single test pass for vgg19, need furture investigation (#2588) 2. Update docker file to decrease the docker image size * Update docs for Android NNAPI EP (#2586) * Fix lto bug for protobuf and ubuntu * add path to build dir before test run (#2590) * Add missig env variables for mac pipeline test (#2595) * Fixed an issue in updating realized dims (#2597) when we update realized dims for scan's output, the sliced axis also needs to be inclusive, i.e. we should check with "dim >= insert_inclusive_axis", because the offset in the symbols are based on Scan sugraph. Otherwise, we would end up with shape mismatch later. * Java API for onnxruntime (#2215) * Add support for opset 11 in reshape fusion (#2592) Support opset verion 11 in reshape fusion * Rename automl python tools folder to featurizer_ops. (#2593) * Support opset 11 subgraph of Squad model in Embed Layer Normalization (#2605) Support opset 11 Squad model that is exported from PyTorch nightly. The embed layer uses Range op which is missed in the transformer. * symbolic shape inference: fix warnings in GPT-2 model (#2608) And revise nuphar perf test on BERT squad * Dump subgraph ID and fused graph ID (#2607) * Dump subgraph ID and fused graph ID Dump subgraph ID and fused graph ID for better debugging * Remove local static fused_count added a field global_fused_count_ to NupharExecutionProvider class * EmbedLayerNormalization Fusion For Dynamic Squad Model Opset 10 (#2613) Support subgraph of SQuAD model exported from pytorch with dynamic input axes * Allow providers to be set for InferenceSession at construction (#2606) * Remove unnecessary parameter in some places in GatherElements implementation (#2612) * Remove unnecessary parameter in some places * Update * Update * Make sure fenced tensor could not reuse other tensor. (#2561) Fix random error caused by this. * Improve Embed Layer Norm Fusion for SQuAD with static input shape (#2621) * fix float16 comparison in initializer (#2629) * epsilon attribute for layernormalization fusion (#2639) * removed unnecessary batch file and fix path (#2640) * Add shape inference to ConvTransposeWithDynamicPads schema (#2632) * Improve cuda expand() opeator's performance. (#2624) * Cuda pad optimize when no padding is needed. (#2625) * Shortcut cuda Pad() when no padding is needed. * Optimize cuda scatter() on 2D compatible. (#2628) * Optimize cuda scatter() on 2D compatible. * Add some comments. * fix build error for ARM (#2648) * Improve performance of resize() in Nearest mode (#2626) Special treatment for 2D, check same size as input image. And in 2d kernel, template use_expolation. * Fix memory exception in Layer Norm Fusion (#2644) * Windows CI changes(#2650) * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Revert "Windowsai without fi (#2701)" This reverts commit 285d4c85ff5c4e265f963208170304ef3461e684. * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Deref unique pointer for session_state * send shutdown event when dll is unloaded and EvaluationStop, SessionC… (#2704) * send shutdown event when dll is unloaded and EvaluationStop, SessionCreationStart Events. * Add EvalutationStart Event * add comment * use correct type for for loop (#2755) * ARM CI (#2759) * Set ARM agent pool * Set CMake generator to VS 2019 in ARM * Use system-wide CMake instead of custom version Our custom version is too old for VS 2019 * Use DML and build shared lib in ARM CI * Restore nuget packages in ARM CI * Disable DML * Refactor ARM debug/release builds * Use system packaged Python version * Remove hardcoded Python path * Downgrade Python to 3.7 for build * Remove explicit CMake path * Fix invalid JSON in cgmanifest.json (#2760) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * Remove usage of WHOLEARCHIVE in WinML CMake and add WinMLAdapterFactory (#2726) * Remove usage of WHOLEARCHIVE in WinMLAdapter CMake and add WinMLAdapterFactory * PR feedback, no need for dll(export) since using def file * PR comments * Small comment in gen_def.py * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * Move winml_provider_factory.h to proper location (#2801) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Filter CPU case for IsFloat16Supported (#2802) * Merge fixes * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * Fix warnings that cause build to fail * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * Make DML include path global (#2882) * Make DML include path global * Add generated cppwinrt headers to winml_lib_common * Integrate changes to WindowsAI to make ADO Build (#2886) * Revert "CMake cross-generator fixes (#2790)" This reverts commit dbe7d97fa1ab155f1309bced87199527e8f35bd2. * add additional suppress warning in onnx_proto * ignore /wd4996 warning * DML execution provider fixes * Revert "Revert "CMake cross-generator fixes (#2790)"" This reverts commit 1ae7b4bcbc02edc881ad28685da98e095dfceb17. * Update func signature of custom op function overloads * common devicehelpers fixes * Add pch.h for winml_lib_common * re-add winml_lib_common_dir/inc to include path for winml_adapter * User/orilevari/dml redist shared folder (#2890) * move dml nuget package directory up one level to make it shared between build flavors * Merge conflict fix * Revert "Merge conflict fix" This reverts commit 142fa72cf9ce4344ad717b50b7ea2b8582aadc7c. * Revert "Merge remote-tracking branch 'origin/master' into windowsai" This reverts commit 6e2126d46e5e5f564d65da37dd4f70c93dd81165, reversing changes made to b3f5583dc9249834b947c8ea905f6a98060d5bd6. * Make winml_test_common free of test macros (#2902) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Don't generate files outside the build root (#2914) * Don't generate files outside the build root * Add onnxruntime_EXTERNAL_DEPENDENCIES to WinML * Add DML depedency on RESTORE_PACKAGES * User/orilevari/fix yaml merge bugs (#2918) * Add winml test source parameter into cmake function (#2919) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Add arguments to unittest cmake functions * remove comment * Revert "Revert "Merge remote-tracking branch 'origin/master' into windowsai"" This reverts commit ade5abe72a4234fdbc3623093c61c02c6b0bdc26. * Fix breaks from merge with ORT master * Brianma/linux (#2917) * don't include windows.h in cross-plat header * add default case for switch statement * signed/unsigned mismatch fix Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> * User/sheilk/winml adapter c api (#2891) * Create winml adapter c api * fix build * make it build * move adapter into onnxruntime core/session * entry point not exported * minor changes * make model metadata work * make tests pass * implement all the model reflection apis on the adapter c abi * update the new ort interface to create a lotus ennvironment with a logging sink * start adding ort env * move all winml code into adapter folder/lib to isolate it * ensure a single logging manager at a time * start refactoring session * refactor session creation interface * add cpu and dml session option methods to adapter * finish session init * stub out interfaces in ort lib to perform similar mechanics of iinference session * enable profiling, and enable schema override * update session register graph transformers * turn back on custom registry for custom ops * Add sync api * add last c api stubs * should build... but all feature values are broken since this is in flight to moving all implementation details into ivalue * remove ep adapter header * Implement DML execution provider functions from adapter (#2846) * Implement DML execution provider functions from adapter * Use functions in OnnxruntimeEngine.cpp * make map/sequence type_infos freeable, and start implementing ivalue * make it build again * implement value methods * implement remaining methods * remove com adapter abi * check dml session * cache the allocator on ivalue * check if resource is cpu/gpu when access its mutable data * update tensor * mismatched parentheses * fix tensor base and binding obj * it evaluates tensors! sometimes... * minor fixes * enable gpu evals * wrapper all existing winml adapter apis with API_IMPL to try catch (#2854) * update winml... tensor strings are broken, need to template tensorbase to do different things for strings * make tensor strings work with 2 copies in/2 copies out * Fix tensor string and allocator bug * make maps work again... needs some fixes still * Make it build! * enable map inputs * map outputs * unbound outputs for sequences and maps * User/xianz/merge windowsai (#2883) * Packaging pipeline changes for VS 2019 (#2711) * Tiny fix to codegen * Simplify cache implementation and avoid static variables that may carry over between models * Extend DML kernels (#2641) * Additional DML operators * Check unsupported attributes and inputs * Address PR comments * Add kernel capability function used for partitioning, and re-enable stride-based int64 support based on value range * Fix test failures * Build fix * PR comments * Update Nuphar tutorial notebook (#2721) 1. Reflect int8 GEMV improvements for multi-threading from #2696 2. Add notes on multi-threading control using OpenMP 3. Add samples of running multi-isa AOT, and show int8 GEMM differences between AVX and AVX2 4. Add rnn_benchmark example to resolve #1993 * Add schema for new Qops (#2611) * Add schema for new Qops * adding shape inference + qlinearaveragepool * plus review comments * plus review comments * updates per review comments * plus review comments * [server] Add supposed for model_name and model_version as cli parameter (#2708) * remove 64bit warning message from python validation. (#2727) * MLAS: ARM64 build fix (#2734) fix bad usage of vreinterpret to cast vector element types * Fix broken python docs links (#2740) * Fix build on Mac OS (#2731) mac os ld doesn't support --while-archive, correct option is -all_load * fix ngraph wheel (#2737) * fix ngraph wheel 1.1.0 onnxruntime_ngraph wheel doesn't work * remove libdnnl.so in nGraph Libs * make it easy to compare * Split onnxruntime server to a separated folder (#2744) * Fix build for Python 3.8 (#2747) * Fix build for Python 3.8 * Update protobuf to 3.11.2 (#1928) Update protobuf to 3.11.2 (#1928) * Change default optimization level to All (from Basic) (#2745) * change default optimization level to All (from Basic) * fix test * fix c# test * Update numpy to 1.18 (#2758) * Update numpy to 1.18 * Pipeline changes for python 3.8 (#2753) 1. Pipeline changes for python 3.8 2. Fix a regression in setup.py which was just introduced in the previous commit. Please notice, we still haven't made python 3.8 + Windows + CUDA work. * Add basic stacktrace output for posix debug builds. (#2749) * [NupharEP] fix a race condition when multiple sessions running different models concurrently (#2772) * Revert "Change default optimization level to All (from Basic) (#2745)" This reverts commit 56bb503c2f26474b6613bcb2a198691a11dcef38. * Fix typo in error message (#2736) * Rename MKL-DNN to DNNL to fix broken link (#2730) * Fix nightly build version number issue * Pass BUILD_BUILDNUMBER to linux docker * Disable featurizers in python packages * Import more featurizers (#2781) Make kernels non-template. Add input constraint for learnt data. Add min_max_scalar_transformer, robust_scalar_transformer, inputation_marker_transfomer, label_encoder_transformer, missing_dummies_transformer along with tests. Advance Featurizers library commit. * Implement a more stable softmax (#2715) * Implement a more stable SoftMax e^x is represented as infinity if x is large enough, like 100.f. Infinity divided by Infinity is a NAN. Thus, softmax gets a NAN if one or more item are large enough. A math transform as below is leveraged to get a stable softmax: e^xi/(e^x1 + ...e^xn) = e^(xi - max) / (e^(x1 - max) + ... + e^(xn - max)) And for convenience, force max to 0.f if all xi are negative * Contributing: Fix a typo (#2784) * ACL EP GEMM improvements (#2780) When it is posible we use a fully connected layer instead of the gemm implementation. This will let the library use the best implementation based on the input data. * ACL EP convolution improvements (#2774) Added the optimized implementation for depthwise convolution for both ACL v19.02 and ACL 19.05. Also the pointwise convolution seems to be more optimal in the CPU implementation so we opted for that instead. * Add script for release Nuget validation (#2719) * Initial commit * Nits * Disable a test temporarily * Change working directory * Test * Add download python step * Test update * More changes * Fix space issue * Fix * Verify nuget signing * Fix * Spaces * PR feedback * Nit * Fix * Fix * Remove temporary changes * add uint8 support to where op (#2792) * Improve bert optimization script: (#2712) (1) Move input int64=>int32 conversion to embed layer fusion. (2) Output epsilon attribute for LayerNormalization fusion. * add session creation time cost. (#2798) * ML.NET team needs featurizers within a package (#2789) Add auto ml featurizers to Windows, MacOS as well as to GPU packaging-pipelines. * Initialize max of softmax with lowest of float (#2786) * MLAS: update SGEMM threading parameters (#2808) * add interface to copy batch tensors. (#2807) * add interface to copy batch tensors. * onnxruntime * speed up Windows TRT CI (#2811) * don't run cuda tests if building with tensorrt * remove unnecessary build options for win trt ci * refactor win gpu tensorrt ci yml * --numpy_version=1.17 * update * update * azcopy and cuda path * Update test data (#2356) * Add timeseries imputer transformer featurizer kernel (#2813) Make kernels non-template. Add input constraint for learnt data. Fixup tests. Add two more featurizers along with tests. Tests fail. min_max_scalar_transformer robust_scalar_transformer Fix tests serialized stream by prepending version bytes. Add inputation_marker_transfomer and the test. Fix up float/double type designations. Added label_encoder_transformer along with a test. string_throw case is broken at the momement. Fix labelencodertransfomer_test.cc string_throw case Rename maxabsscalertransformer_test.cc Add MissingDummiesTransformer along with the test. Update manifest. Add TimeSeriesImputerTransformer definition, implementation and tests * Fix memory leak in TRT (#2815) * fix memory leak issue * revert EP_FAIL on enueueV2 * Add manifest missing comma * Run static code analyzer on most of our code (#2817) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * update quantization doc (#2783) * update documentation for quantization script * plus some spell corrections * Filter CPU case for IsFloat16Supported (#2802) * update default optimization level + fix gemm_activation fusion (#2791) * update defualt optimization level + fix gemm_activation fusion * fix typo * add unit test and incorporate review comments * fix test comment * Fix dnnl wheel package name (#2823) * Append '-dnnl' to whl package name when --use_dnnl * Update build.py * Update Ubuntu & TensorRT version in README (#2820) Dockerfile.tensorrt is using nvcr.io/nvidia/tensorrt:19.09-py3 as base Image, update Ubuntu and TensorRT version according to https://docs.nvidia.com/deeplearning/sdk/tensorrt-container-release-notes/rel_19-09.html#rel_19-09 * Merge fixes * Add OneHotEncoder and HashOneHotEncoder kernels. (#2830) Add defs and imlementation for OneHotEncoders, adjuist date_time_transformer kernel and test. Add OneHotEncoder kernel test. Add HashOneHotVectorizerTransformer unit test. This does not link due to multiple definitions of functions that are included into header from a CPP file. * Upgrade gtest to the latest version (#2827) WinML would like to update the googletest submodule. They want some newer features (namely GTEST_SKIP to skip tests programmatically and be able to skip entire fixtures easily) and would need to update the submodule version. However, because the new version of code hit a bug in gcc, even though the bug is already fixed in the latest gcc but we're using gcc 4.8.x and it won't get patched for the bug, so we have to do a compromise, change our code a little bit to make it work. The gcc bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=51213 * Add support for int64_t for topk CPU. Fixes github issue #2806. (#2833) * Ignore allocator type in ExecutionProviders allocator map. Make default initialization of OrtMemoryInfo more clearly invalid. (#2768) * Remove allocator type from the key comparison in ExecutionProviders. Remove usage of DummyArena as it's no longer necessary. * Fix x86 tests where arena allocator is disabled. Make initialization of OrtMemoryInfo clearer by adding Invalid enum value. * Make OrtValueNameIdxMap::MaxIdx more intuitive. * Convert ExternalProject Featurizers into git submodule (#2834) Add git submodule for Featurizer library. Update cmake to build for git submodule. * add domain check for nodes + update documentation (#2831) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Fix Linux CUDA nuget packaging pipeline break * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * update optimization doc for BERT related fusions (#2819) * Add bert related transformers to doc * Add execution provider and comment for bert optimizations * Add comment about accuracy impact of approximation * Fix warnings that cause build to fail * MLAS: enable threading for quantized GEMMs (#2844) * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * remove old winmladapter cpp Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * move sequence implementation into ort lib... still commented out... need to turn back on... * begin sequence implementation * make maps and sequences work * fix broken tests * remove dead code * misc cleanup * CR feedback * User/xianz/winml adapter c api (#2869) * wrapper all existing winml adapter apis with API_IMPL to try catch * Return HR or Throw for WinML adapter APIs if failed * undo macro wrapper for two places * Wrap error macros around ort apis, too. * address CR feedback #2 * add more api throw/return macros * Revert changes no longer needed * revert changes to cxx api * format winml lib.ort and winml adapter * remove static pheonix singleton Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * missing use_dml check in winml_adapter_session (#2930) * --use_dnnl flag was mangled in merge (#2931) * use dml macro not wrapping custom registry code (#2934) * Disable LNK4199 winml_dll to enable cuda builds (#2936) * Disable LNK4199 in winml_dll * linkler->linker * LearningModelSessionAPITestGpu.CreateSessionWithCastToFloat16InModel should return DXGI_ERROR_UNSUPPORTED when FP16 not supported (#2937) * Disable LNK4199 in winml_dll * linkler->linker * Need to return DXGI_ERROR_UNSUPPORTED when Model does not support fp16 * Publish build symbols (#2939) * Publish build symbols * Don't upload PDBs for .exe files * Make x86 build (#2943) * fix last remaining size_t/int64_t warnings->errors (#2948) * TensorString, Sequences and Maps use the first allocator, but should use the cpu default allocator. (#2952) * fix tensor string allcoator * clean up default allocator usage for strings in winml lib/api.ort Co-authored-by: Ryan Lai <ryalai96@gmail.com> * Handle tensor shape of zero (#2954) Co-authored-by: Ryan Lai <ryalai96@gmail.com> * CR feedback (#2970) * CR feedback * fix weird formatting on privacy readme * Add 'All rights reserved.' everywhere * readd all rights reserved to winml_provider_factory.h * remove extra space in comment * remove extra whitespace * fixes post master merge * remove winml from nuget gpu pipeline * set IR VERSION on generated_model in rnn_benchmark (#2972) * Fix slice conformance failures (#2908) Co-authored-by: Adrian Tsai <adtsai@microsoft.com> Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Paul McDaniel <paul_mcdaniel@hotmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Dwayne Robinson <fdwr@hotmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com>
2020-02-05 01:12:19 +00:00
[threading(both)]
[marshaling_behavior(agile)]
[dualapipartition(1)]
runtimeclass LearningModelSession : Windows.Foundation.IClosable
{
//! Create a session, on the system default device, to evaluate the specified model on.
[method_name("CreateFromModel")] LearningModelSession(LearningModel model);
//! Create a session, on the provided device, to evaluate the specified model on.
[method_name("CreateFromModelOnDevice")] LearningModelSession(LearningModel model, LearningModelDevice deviceToRunOn);
//! Returns the machine learning model attached to the session.
LearningModel Model{ get; };
//! Returns the evaluation device that the session was created on.
LearningModelDevice Device{ get; };
//! Returns the list of properties set for model evaluation.
Windows.Foundation.Collections.IPropertySet EvaluationProperties{ get; };
//! Evaluate the machine learning model using the feature values already bound in 'bindings'. (asynchronous)
[remote_async]
Windows.Foundation.IAsyncOperation<LearningModelEvaluationResult> EvaluateAsync(LearningModelBinding bindings, String correlationId);
//! Evaluate the machine learning model using the feature values in the map 'features'. (asynchronous)
//! This method is an alternative to the separated bind then eval form that takes a LearningModelBinding.
//! It will take the passed in features, create a LearningModelBinding for you, bind the features, and then evaluate the model.
[remote_async]
Windows.Foundation.IAsyncOperation<LearningModelEvaluationResult> EvaluateFeaturesAsync(Windows.Foundation.Collections.IMap<String, IInspectable> features, String correlationId);
//! Evaluate the machine learning model using the feature values bound in 'bindings'.
LearningModelEvaluationResult Evaluate(LearningModelBinding bindings, String correlationId);
//! Evaluate the machine learning model using the feature values in the map 'features'.
//! This method is an alternative to the separated bind then eval form that takes a LearningModelBinding.
//! It will take the passed in features, create a LearningModelBinding for you, bind the features, and then evaluate the model.
LearningModelEvaluationResult EvaluateFeatures(Windows.Foundation.Collections.IMap<String, IInspectable> features, String correlationId);
[contract(MachineLearningContract, 2)]
{
//! Create a session, on the provided device, with the desired model compilation options, to evaluate the specified model on.
[method_name("CreateFromModelOnDeviceWithSessionOptions")] LearningModelSession(LearningModel model, LearningModelDevice deviceToRunOn, LearningModelSessionOptions learningModelSessionOptions);
}
}
//! \interface ILearningModelFeatureValue
//! \brief The instantiated value for a feature.
[contract(MachineLearningContract, 1)]
INBOX_ONLY([uuid(f51005db-4085-4dfe-9fed-95eb0c0cf75c)])
Merge windowsai (winml layering) into master (#2956) * Initial Commit * Merged PR 3985217: add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc (#2346) add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc and violating our OS layering requirements. We linked against onecoreuap_apiset.lib in VB so we will continue doing this, but I am still unsure why not to link against onecore instead since that is where we ship. However, since Sheil is the owner of this code we will wait to discuss with him before changing anything. * Initial changes for layering * more snipping to get core into ort * update build instructions to include --build_shared_lib (#2358) * update build instructions to include --build_shared_lib * fix line breaks * Task 23998197: add winml_lib_core into onnnxruntime.dll (#2368) * Task 23998197: add winml_lib_core into onnnxruntime.dll * PR feedback build break on perf_test * return proper error when the model path isn't found (#2391) * LearningModelSession is cleaned up to use the adapter, and parts of b… (#2382) this is a big PR. we are going to move it up to layer_dev , which is still a L3 so we are still safe to do work there agile. we are going to move this into the L3 so that ryan can start doing intergration testing. we will pause for a full code review and integration test result prior to going into the L2. >>>> raw comments from previous commits >>> * LearningModelSession is cleaned up to use the adapter, and parts of binding are. * moved everything in the winmladapter made it all nano-com using, WRL to construct objects in the ORT side. base interfaces for everythign for winml to call cleaned up a bunch of winml to use the base interfaces. * more pieces * GetData across the abi. * renamed some namepsace cleaned up OrtValue cleaned up Tensor cleaned up custom ops. everything *but* learnignmodel should be clean * make sure it's building. winml.dll is still a monolith. * model moved over. everything builds clean. step ! * weak ref comment * Layer dev paulm (#2408) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * Layer dev paulm (#2414) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * Layer dev paulm (#2423) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * Layer dev paulm (#2424) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * Layer dev paulm (#2425) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Layer dev paulm (#2426) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * fixed map and sequence when passing stl types across the ABI . found a leak in nvidia driver, but skipped it. all winmlapitests pass now * Moved SessionOptions over to the abi * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * Register WinML TraceLogging provider on Onnxruntime.dll (#2455) * Register WinML TraceLogging provider on Onnxruntime.dll * Add ifdef to make sure trace logging provider has telemetry option when LAYERING_DONE * No need for ifdef for TraceLoggingOptionMicrosoftTelemetry * PR feedback * Move etw registration into lotus environment constructor and deresgister in lotus environment destructor * Brianma/cpuwinml (#2466) * allow building winml cpu without dml. * Brianma/breaks (#2469) * fix some more breaks * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * move dml checks out of winml and into the adapter * better error handling * Brianma/fi (#2470) * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * fix bad merge * Checking in a staging checkpoint point so that Ryan can work with me in parrallel * build break. * Brianma/testfails (#2473) * add missing ir version to dictvectorizer-string.onnx * add missing ir version to relu.onnx * add missing ir version to zipmap*onnx * add IR version to manually generated models * remove an unnecessary ifdef dml * Brianma/windowsai fi (#2475) * update dockerfiles/README (#2336) * Make elementwise op run 4 items per thread (#2335) Description: Describe your changes. Make elementwise op run 4 items per thread unroll for loop to leverage ILP remove unnessary N==0 check inside elementwise GPU kernel Motivation and Context Why is this change required? What problem does it solve? It can improve the performance of GPU elementwise ops. ~2% performance gain on popular NLP bert model. If it fixes an open issue, please link to the issue here. * Add CUDA GatherElements kernel (#2310) * Updates * Update test * Update * Updates * nits * PR feedback * Update * Update * PR feedback * PR comments * Update * Fix build * Fix build * Nits * Fix * Layer Normalization Fusion (#2319) basic layer normalization transform * Add FastGelu Cuda Op for Gelu and Add bias fusion (#2293) * Add FastGelu cuda op * Add AddBiasGelu for experiment * Revert "Add AddBiasGelu for experiment" This reverts commit 5c1ee019858c657e6bb75887265cb85675626e5b. * Add bias * Add unit tests * update comment * update script * fix build error * update coding style * update for CR feedback Enable half2 optimization only when cuda arch >= 7.0 * move _Tanh to common.cuh * implement CPU contrib OP Attention (#2333) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanUnusedInitializers. (#2320) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanupUnusedInitializers. This means initializers that have been replaced during graph optimizations are not left in the GraphProto when we save an optimized model. * Handle edge case where a model has an unused initializer with matching graph input by also removing the graph input. * Use non-const iterators in std::find_if calls to make centos build happy. * Nuget pipeline changes (#2305) 1. refactor the pipeline, remove some duplicated code 2. Move Windows_py_GPU_Wheels job to Win-GPU-CUDA10. We'll deprecated the "Win-GPU" pool 3. Delete cpu-nocontribops-esrp-pipeline.yml and cpu-nocontribops-pipeline.yml 4. In Linux nuget jobs, run "make install" before creating the package. So that extra RPAH info will be removed * Cuda Reverse Sequence Op, maping types of same size using same template function. (#2281) * Set ElementType to String type of node metadata, instead of byte[] (#2348) * Set ElementType to String type of node metadata, instead of byte[] * Fix spacing * Introduce PrimitiveType into a Type System along with an integer constant (#2307) Improve perf by avoiding GetType<T>() calls. Introduce MLTypeCallDispatcher to switch on Input Type. Add Tensor IsType<T>() fast method. * Fix/test dim value of 0 handling in a couple of places (#2337) * Update the CUDA Where implementation broadcasting logic to handle a dim with value of 0. Add unit test Also add unit test for unary op with dim value of 0 * Exclude ngraph from Where test with 0 dim. * Openvino EP R3.1 onnxrt server (#2357) * onnxrt server with OVEP * onnxrt server with OVEP * Update Dockerfile.server.openvino * onnxrt server OVEP fix reviews * onnxrt server OVEP fix reviews * Implement cuda nonzero op. (#2056) Implement cuda nonzero op. * Direct use python numpy array's memory if already contiguous. (#2355) * Direct use python numpy array's memory if already contiguous. This could greatly improve performance for session with large input, like big image 1920x1080 fastrcnn, 30~40% speed up could be achieved. * Add test case enforce contiguous/non-contiguos numpy array as inputs. * Add helper to create output to minimize binary size. (#2365) Add ConstEigenTensorMap typedef so we don't unnecessarily const_cast the const input Tensor. * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS (#2369) * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS * update * Add Tracelogging for profiling (#1639) Enabled only if onnxruntime_ENABLE_INSTRUMENT is ON * test bidaf with nuphar for avx target (#2370) increase nuphar test coverage a bit * Fix a bug in TLS refcount that may destabilized CUDA CI (#2374) * update output size calculation for resize (#2366) * change how output size is calculated for resize op * add tests for ver 10 resize * Extend OneHot CPU kernel to support more types (#2311) * Extend OneHot CPU kernel to support input int64_t, depth int32_t, output float * Skip BERT before the test data fix is picked up * Fix bug with Slice. Need to pass in flattened input dimensions so the initial offset into the input is calculated correctly. (#2372) * Add opset 11 version of Split to CUDA ops (#2376) Organize the CUDA ops definitions so all the opset 10 and 11 parts are together (same setup used for CPU ops) * Layer Norm Fusion Fix (#2379) * layer norm fusion fix * Add input shape check in code and unit tests * Fuse Add + Gelu (#2360) Implement the transformer to fuse add + gelu Implement the accurate kernel * Skip layer norm transform (#2350) * skip layer normalization transformer * Another try to stabilize CUDA CI (#2383) The root cause seems to be failure in CUDA dealloc when tear down. cudaFree return code was ignored before, so should the debug check. * fix BUILD.md typo (#2375) build.py: error: argument --config: invalid choice: 'RelWithDebugInfo' (choose from 'Debug', 'MinSizeRel', 'Release', 'RelWithDebInfo') * Fixed compilation with ngraph (#2388) * Fix reuse logic in allocation planner. (#2393) * Fix reuse logic in allocation planner. * PR comments * Add helpful comments * Don't allow reuse across string tensors. * [NupharEP] Multiple optimizations (#2380) Fuse transpose into MatMul Implement Pow and constant scalar simplification Vectorize ReduceMean Improve symbolic shape inference Minor updates for better debugging in fused function name * Avoid using the default logger in the graph lib and optimizers (#2361) 1. Use the session logger if it is available. 2. Don't disable warning 4100 globally. We should fix the warnings instead of disabling it. * Change CUDA implementation of Transpose to support all fixed size tensor types (#2387) * Change CUDA implementation of Transpose to not use a typed kernel so we can support more types with minimum binary size. Add support for 8, 16, 32 and 64 bit types. Add unit tests. Add method so the implementation can be called directly (will be used by CUDA Scan very soon). * Disable TensorRT for MLFloat16 and int8 unit tests. * Address PR comment and add support for calling cublas implementation if type is mlfloat16. * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. (#2398) * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. * [NupharEP] force some low/zero cost ops to be inlined (#2409) * fix cross compile bug (#2415) * Minor optimization: if a node has already been placed, there's no need to find a kernel for it. (#2417) * Add Reshape Fusion (#2395) * Add reshape fusion * Add some comments * update comments * update comment format * update according to feedback * update for recent logger change * fix build error * (1) Support both input and output edges in find path in graphutils (2) Add a test case of only one constant initializer of Concat input. (3) Refactor ReshapeFusion class to allow add more subgraph fusion in the future. * fix error * (1) loose constraint on initializer: non constant is allowed for reshape fusion. (2) Change versions type to vector. (3) Add logging. (4) Return false when multiple output edges matched in FindPath. Add comments. * only allow one direction (input or output) in FindPath * [NupharEP] Update notebook and docker image (#2416) Add BERT squad in Nuphar tutorial Enhance speed comparsion readability * Fix the issue in matmul_add_fusion (#2407) Fix the issue in matmul_add_fusion If Muatmul + Add has shape [K] * [K, N], reset it to [1, K] * [K, N] will make the output shape to [1, N] will also requires a reshape on the output. Fix: just remove the shape reset to not fuse it. Add a negative test case for matmul+add fusion * feat(treeregressor): Update TreeEnsembleRegressor for type support (#2389) Updates the `TreeEnsembleRegressor` to allow for `double`, `float`, `int64`, and `int32` inputs to match the upstream specification. Signed-off-by: Nick Groszewski <nicholas.groszewski@capitalone.com> * onnxrt server documentation update (#2396) * Added support for Pad-2 operator in OpenVINO-EP (#2405) * Add CUDA If operator. (#2377) * Add CUDA If operator. Uses CPU operator for implementation. By adding a CUDA version the inputs/outputs (with the exception of the 'cond' input) stay on GPU, and no other logic is required to avoid a copy to CPU across the control flow node. * Improved documentation for onnxruntime::utils::SwapByteOrderCopy(), added precondition check. * Fix the type constraints on CUDA If operator to exclude strings. (#2431) * add Im2col<uint8_t> (#2438) * Adjust codegen vectorization width from target (#2439) * Adjust codegen vectorization width from target * Add CUDA Scan operator. (#2403) * Add Scan CUDA op. Uses CPU implementation for logic. Added some device specific functors for handling when data needs to be manipulated on a different device. Added ability to override the materialization logic in the OrtValue slicer so DML can plugin their handling. * Fix Windows GPU C API packaging pipeline failure (#2440) Fix Windows GPU C API packaging pipeline failure (#2440) * Correctly handle implicit inputs for fused nodes (#2390) * Correctly handle implicit inputs for fused nodes Previously, nuphar's partitioning function didn't include node's implicit inputs into the inputs list of MetaDef, and hence a crash was triggered in the onnx graph checker. This commit fixed the issue. Furthermore, it also fixed a related issue where we didn't add implicit inputs into graph_inputs_excluding_initializers_ in Graph::SetGraphInputsOutputs. the issue was that graph_inputs_including_initializers_ populated by SetInputs (e.g. called by FunctionImpl::FunctionImpl) may contain implicit inputs which were not of any node's initializers in the graph. Because they were not part of any initializers, these implicit inputs couldn't be visited by going through all nodes' inputs. Consequently, they would *not* be added into graph_inputs_excluding_initializers_. We fixed the issue by first copying the populated graph_inputs_including_initializers_ into graph_inputs_excluding_initalizers_, which then had both initializers and non-initializers as its initial content. Later, we erase initializers from the list. In this way, we can ensure all implicit inputs to remain in graph_inputs_excluding_initializers_. * refined comments and fixed duplicates Address CR by revisiting comments in terms of implicit inputs Also fixed an issue by skipping duplicates while copying inputs from graph_inputs_including_initializers_. * address CR explain why we need to collect nodes' implicit inputs * don't rely on pointer values for iterating std::set Previously, openvino relied on iterating a set of NodeArg pointers to construct inputs and outputs for a fused graph. It could cause non-determinism. The reason was that although iterating std::set by itself is stable, pointer values of NodeArgs may vary. Consequently, we could end up visiting the set's elements in different orders for different runs for the same test, which resulted in constructing inputs (and outputs) with different orders to the fused graph. For example, for the same test, we may have inputs [A, B] in some runs but inputs[B, A] in others. Let's use std::string as the key type to avoid such nondeterminism. This commit also added implicit inputs into meta->inputs while returning the capability from the openvino provider. * Fixed another latent issue in openvino's GetCapability function The issue was that we couldn't simply erase fused_inputs and fused_outputs while iterating the nodes. For example, an output NodeArg may have multiple uses, and it's wrong if we erase it from fused_outputs when we encounter only one of its uses as input. * Remove DeviceAllocatorRegistry class (#2451) Remove DeviceAllocatorRegistry class * CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library. * Parallel Gelu with ParallelFor (#2399) Parallel Gelu to get better performance for Gelu * Clean up build.py (#2446) * Pull the latest image before running docker build * Fuse SkipLayerNorm with Bias (#2453) Fuse SkipLayerNorm with Bias * Allow more than one invocation of CreateEnv in the same process. (#2467) * Allow more than one invocation of CreateEnv in the same process. * Fix centos build * Symbolic shape inference improvements: (#2460) * Symbolic shape inference improvements: - add a mode to guess unknown ops' output rank - add support for GatherND - add support for If - fix a bug in get_int_values when then tensor rank > 1D, by treating it as no sympy data - add symbol to literal merge when ONNX silently merges dims - fix a bug in Concat when input dim is 0 - fix a bug in ConstantOfShape that computed dim is not updated - add support for dynamic shape in ConstantOfShape - fix a bug in Loop output shape that loop iterator dim is not inserted at dim 0 - add support for dynamic padding in Pad - add support for dynamic shape in Reshape - add support for Resize with opset > 10, by treating output dims as dynamic - fix a bug in Slice when starts/ends are dynamic - restrict input model to opset 7 and above - make output model optional to avoid disk write when testing Run model tests for symbolic shape inference Reduce 2GB docker image size of nuphar * add additional test data set for nuget pipeline (#2448) * add SAS token to download internal test data for nuget pipeline * update azure endpoint * fix keyvault download step * fix variable declaration for secret group * fix indentation * fix yaml syntax for variables * fix setting secrets for script * fix env synctax * Fix macos pipeline * attempt to add secrets to windows download data * fix mac and win data download * fix windows data download * update test data set url and location * Revert "Brianma/windowsai fi (#2475)" This reverts commit 5780b864a15513fda4eadbfc2b5345fefe70b5ec. * Add scenario tests (#2457) * Add scenario tests * Remove TODO from model license * Add winml_api test dependency * fix model load test. fi from master changed the constructor (#2483) * make api tests all pass (#2486) * fix bad merge * fix bad model merge * Layer dev paulm (#2492) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * Rename ambiguous header (#2489) * fix one more missing IR version model (#2500) * add missing IR version to 4 more models used by scenario tests (#2501) * Add CLI parameters to test runner, build WinML in ARM and x86 CI (#2479) * Support test parameters through CLI arguments * Add WinML do Windows x86/ARM CI builds * Code style fixes * Update googletest Remove GPUTEST macros everywhere now that GTEST_SKIP is supported * Refactor main.cpp * Build scenario tests without DML * Link scenario tests to DML when it's enabled (#2502) * Layer dev release pipeline (#2488) Adds winml binaries to existing cpu nuget package, and creates new gpu dml nuget package with winml binaries and DML EP. * Layer dev paulm (#2506) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * Remove usage of IOBinding in WinML and use C_API Run method (#2504) * remove usage of iobinding * Change data structure to use vector of Ort::Values * Polish bind input / output * Use C APIrun method * Update providers on evaluate getresults * Remove run and IObinding interface from WinMLAdapter * Remove use of IObinding * bind unbound outputs code moved to learningmodelbinding * clean up unneeded istensor adapter function * Fix comment * Check if session is closed before binding and clearing * PR feedback * Layer dev paulm (#2507) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * Make tests dependend on winml_dll (#2509) * add dml binaries to DirectML package and be more explicit about condition variables (#2520) * re-enable warnings for winml builds and fix the warnings that were hiding (#2526) * turn devmode back on for winml builds * fix some warnings. include protobuf in a way that disables some warnings * undo protobufhelpers changes and just ignore 4100 errors in pb code * attempt to isolate protobufhelpers errors * add template specialization for getting tensor proto data * Layer dev paulm (#2533) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * Spawn child process to run DeviceLostRecovery scenario test (#2530) * Spawn child process to run DeviceLostRecovery scenario test * Layer dev paulm (#2536) ori said yes * add missing namespace to winml_trace_logging_provider in lotusenvironment.h (#2542) * Handle exception thrown from all apis in WinMLAdapter (#2539) * various changes to unblock windowsai ADO build * Fix custom ops scenario tests (#2562) * Do not shutdown protobuf after ort environment gets destroyed. Lazy load lotus environment first time it is needed * comment typo * pr comment about calling phoenix singleton * Make lotus_environment static in winmladapter * Layer dev paulm (#2567) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * making windowsAI azure dev ops work. * code review comments. * revert changes * Cmake and preprocessor fixes that where uncovered by building on agents without DML available via SDK * Layer dev dml delayload (#2580) * Brianma/cpu (#2583) * don't include dml stuff in cpu builds * tests that link the image lib also need the telemetry lib now * Throw Winml_err_invalid_binding if binding gpu resource on cpu device (#2589) * Throw Winml_err_invalid_binding if binding gpu resource on cpu device * PR comments. No need to query executionprovider for is gpu device * User/xianz/ortthrow (#2596) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * User/xianz/ortthrow (#2599) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * add status error message * Remove uwp onsuspending winrt call because logruntimeperf is getting removed (#2630) * User/xianz/dedup telemetry (#2631) * investigate duplication of telemetry in winml and ort * remove winml telemetry events * telemetry executionProviderEvent * remove unneccessary file and refactor code little bit * Revert back TelemetryEvent, which send up ETW event. * merge changes from layer_dev to windowsai (#2638) * Remove underscore from googletest names (#2616) * Fix leaking memory allocator Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24278761 and https://microsoft.visualstudio.com/OS/_workitems/edit/24330198 * Explicitly initialize Ort::Value with nullptr * Cache WinML adapter * bad merge * define private version of dxcore enum that is added in 19H1 SDK. (#2654) * add comment for explaning private definition of dxcore d3d feature level ennum value. (#2672) * do not package directml.pdb for redist packages. (#2676) * Fix leaking operator registry (#2645) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24354916 * User/orilevari/windowsai master merge (#2674) merge resolutions included pulling in telemetry logic that was merged to master and not windowsai and dereferencing InferenceSession::sessionstate now that it is a unique pointer * Delete Ort Allocator in LearningModelBinding (#2653) * Delete OrtAllocator in LearningModelBinding * PR comments to make Ort::Allocator a smart pointer * Small comment change * PR feedback to clean up code * PR feedback on move semantics * Clean up std::move * Fix memory leaks (#2679) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24356109, https://microsoft.visualstudio.com/OS/_workitems/edit/24388361 and https://microsoft.visualstudio.com/OS/_workitems/edit/24388596 * various changes to properly organize and skip GPU tests. For now for No DML builds we will not run GPU tests at all. In the future we should adapt the tests to expect the appropiate errors. (#2695) * Windowsai without fi (#2701) * Disable Attention fusion tests when DISABLE_CONTRIB_OPS is defined (#2529) * Setup java ci (#2528) * Add provision in ORT for session options to be parsed when available via model file (#2449) * Initial commit * Fix gitmodules * Nits * Nits * Updates * Update * More changes * Updates * Update * Some updates * More changes * Update * Update * Merge * Update * Updates * More changes * Update * Fix nits * Updates * Fix warning * Fix build * Add comment * PR feedback * PR feedback * Updates * Updates * Update * More changes * Fix build break * Comment test for now * Updates * Updates * PR feedback * Updates * Nits * Add tests * Fix build * Fix build * Fix build * Fix build break * Fix build * Nits * PR feedback * More change * Expose GetSessionOptions in pybind logic and add unit test for python * Fix build * PR feedback * PR feedback * Revert "Disable thread pool creation when enabled OpenMP (#2485)" (#2535) This reverts commit 7c7d5a149c9ed52eec67304bae5c4b132166a8a1. * Add dynamic shape support in TensorRT execution provider (#2450) * remove onnx-tensorrt submodule * add new onnx-tensorrt submodule (experiment) for trt6 * update engine build for trt6 * update compile and compute for tensorrt6.0 * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * switch to onnx-tensorrt master for TensorRT6' * Update tensorrt_execution_provider.cc * Handle dynamic batch size and add memcpy in TensorRT EP * update test cases * Update tensorrt_execution_provider.cc * update onnx-tensorrt submodule * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.ubuntu_tensorrt * Update run_dockerbuild.sh * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update concat_op_test.cc * Update tensorrt_execution_provider.cc * Upgrade TensorRT to version 6.0.1.5 * Update onnxruntime_providers.cmake * Update CMakeLists.txt * Update reduction_ops_test.cc * Update install_ubuntu.sh * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.tensorrt * Update BUILD.md * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update onnxruntime_providers.cmake * Update install_ubuntu.sh * Update install_ubuntu.sh * Update gemm_test.cc * Update gather_op_test.cc * Update CMakeLists.txt * Removed submodule * update onnx-tensorrt submodule * update header file * Removed submodule * add submodule onnx-tensorrt kevin's branch shape-test' * add debugging code * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * merge master * Removed submodule * update onnx-tensorrt submodule * add more changes for dynamic shapes * Update tensorrt_execution_provider.cc * update for dynamic shape * update dynamic shape processing * fix logger issue * remove submodule onnx-tensorrt * add submodule onnx-tensorrt * add env variable min_subgraph_size * remove redundency * update document * use onnxruntime::make_unique * fix multi-run issue * remove some tests to save CI build time * Add dynamic shape test * Update TensorRT-ExecutionProvider.md * Add example of running Faster R-CNN model on TensorRT EP * Add more details on env variables * update environment variables * Update tensorrt_basic_test.cc * Update model tests * Update tensor_op_test.cc * remove --use_full_protobuf * Update build.py * User/xianz/telemetry (#2458) * enabme telemetry * enable telemetry * set enable telemetry as default * for debugging * remove log and set disable telemetry as default back * delete private file while testing * resolve comment: mainly add license header, rename macro and update docs * rewording in privacy.md * Fix integer overflow in cuda NonMaxSuppression implementation (#2540) * add test case that should pass but fail * fix nms * extract int_max_output_boxes_per_class * Introduce container type runtime checks and other improvements (#2522) Rework TensorSeq in a manner consistent with Tensor and SparseTensor in terms of type system setup. Reduce templating. Introduce helpers to ensure the same data type. Make OrtValue __dtor not virtual. Introduce ContainerChecker * Fix C API tests for centos and mac (#2544) * change c++14 to c++11 * add ld lib path for centos * enable csharp tests on macos * fix C API test on MacOS + fix manylinux dotnet install * fix manylinux dotnet install * fix lib link * Add back executable bit to build.py * Fix a bug handling negative begin pad values in Pad op (#2550) * Fix bug in Pad op * Update * DNNL CMAKE update (#2548) * Fix android build (#2558) * Update win-x86-ci.yml (#2557) Fix build pipeline break * Re-enable Windows C# tests (#2564) * disable onnx_test_runner -x invocations for dnnl (#2568) * Allow sequence length to be symbolic (#2559) * setup java ci mac (#2570) * make layernorm fusion to support opset 11 (#2545) * Fix a warning found in the latest VS release * Add more check on SkipLayerNorm and BiasGelu fusion (#2574) * Fix file not found error during docker build. (#2569) * Add ConvTranspose1D (#2578) * Ryanunderhill/packagename test (#2582) * [Nuphar EP] fixes for some object detection models (#2581) Update notebook tutorial with multi-threaded int8 GEMM from #2517 * EmbedLayerNormalization Fusion Improvement (#2553) Embedding layer norm fusion improvements - add more checks * Update version (#2584) * Temporarily exclude vgg19 test from Python backend test 1. temporarily exclude vgg19 test which comsumes too much memory, run out of memory on Upsquared device. Single test pass for vgg19, need furture investigation (#2588) 2. Update docker file to decrease the docker image size * Update docs for Android NNAPI EP (#2586) * Fix lto bug for protobuf and ubuntu * add path to build dir before test run (#2590) * Add missig env variables for mac pipeline test (#2595) * Fixed an issue in updating realized dims (#2597) when we update realized dims for scan's output, the sliced axis also needs to be inclusive, i.e. we should check with "dim >= insert_inclusive_axis", because the offset in the symbols are based on Scan sugraph. Otherwise, we would end up with shape mismatch later. * Java API for onnxruntime (#2215) * Add support for opset 11 in reshape fusion (#2592) Support opset verion 11 in reshape fusion * Rename automl python tools folder to featurizer_ops. (#2593) * Support opset 11 subgraph of Squad model in Embed Layer Normalization (#2605) Support opset 11 Squad model that is exported from PyTorch nightly. The embed layer uses Range op which is missed in the transformer. * symbolic shape inference: fix warnings in GPT-2 model (#2608) And revise nuphar perf test on BERT squad * Dump subgraph ID and fused graph ID (#2607) * Dump subgraph ID and fused graph ID Dump subgraph ID and fused graph ID for better debugging * Remove local static fused_count added a field global_fused_count_ to NupharExecutionProvider class * EmbedLayerNormalization Fusion For Dynamic Squad Model Opset 10 (#2613) Support subgraph of SQuAD model exported from pytorch with dynamic input axes * Allow providers to be set for InferenceSession at construction (#2606) * Remove unnecessary parameter in some places in GatherElements implementation (#2612) * Remove unnecessary parameter in some places * Update * Update * Make sure fenced tensor could not reuse other tensor. (#2561) Fix random error caused by this. * Improve Embed Layer Norm Fusion for SQuAD with static input shape (#2621) * fix float16 comparison in initializer (#2629) * epsilon attribute for layernormalization fusion (#2639) * removed unnecessary batch file and fix path (#2640) * Add shape inference to ConvTransposeWithDynamicPads schema (#2632) * Improve cuda expand() opeator's performance. (#2624) * Cuda pad optimize when no padding is needed. (#2625) * Shortcut cuda Pad() when no padding is needed. * Optimize cuda scatter() on 2D compatible. (#2628) * Optimize cuda scatter() on 2D compatible. * Add some comments. * fix build error for ARM (#2648) * Improve performance of resize() in Nearest mode (#2626) Special treatment for 2D, check same size as input image. And in 2d kernel, template use_expolation. * Fix memory exception in Layer Norm Fusion (#2644) * Windows CI changes(#2650) * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Revert "Windowsai without fi (#2701)" This reverts commit 285d4c85ff5c4e265f963208170304ef3461e684. * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Deref unique pointer for session_state * send shutdown event when dll is unloaded and EvaluationStop, SessionC… (#2704) * send shutdown event when dll is unloaded and EvaluationStop, SessionCreationStart Events. * Add EvalutationStart Event * add comment * use correct type for for loop (#2755) * ARM CI (#2759) * Set ARM agent pool * Set CMake generator to VS 2019 in ARM * Use system-wide CMake instead of custom version Our custom version is too old for VS 2019 * Use DML and build shared lib in ARM CI * Restore nuget packages in ARM CI * Disable DML * Refactor ARM debug/release builds * Use system packaged Python version * Remove hardcoded Python path * Downgrade Python to 3.7 for build * Remove explicit CMake path * Fix invalid JSON in cgmanifest.json (#2760) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * Remove usage of WHOLEARCHIVE in WinML CMake and add WinMLAdapterFactory (#2726) * Remove usage of WHOLEARCHIVE in WinMLAdapter CMake and add WinMLAdapterFactory * PR feedback, no need for dll(export) since using def file * PR comments * Small comment in gen_def.py * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * Move winml_provider_factory.h to proper location (#2801) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Filter CPU case for IsFloat16Supported (#2802) * Merge fixes * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * Fix warnings that cause build to fail * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * Make DML include path global (#2882) * Make DML include path global * Add generated cppwinrt headers to winml_lib_common * Integrate changes to WindowsAI to make ADO Build (#2886) * Revert "CMake cross-generator fixes (#2790)" This reverts commit dbe7d97fa1ab155f1309bced87199527e8f35bd2. * add additional suppress warning in onnx_proto * ignore /wd4996 warning * DML execution provider fixes * Revert "Revert "CMake cross-generator fixes (#2790)"" This reverts commit 1ae7b4bcbc02edc881ad28685da98e095dfceb17. * Update func signature of custom op function overloads * common devicehelpers fixes * Add pch.h for winml_lib_common * re-add winml_lib_common_dir/inc to include path for winml_adapter * User/orilevari/dml redist shared folder (#2890) * move dml nuget package directory up one level to make it shared between build flavors * Merge conflict fix * Revert "Merge conflict fix" This reverts commit 142fa72cf9ce4344ad717b50b7ea2b8582aadc7c. * Revert "Merge remote-tracking branch 'origin/master' into windowsai" This reverts commit 6e2126d46e5e5f564d65da37dd4f70c93dd81165, reversing changes made to b3f5583dc9249834b947c8ea905f6a98060d5bd6. * Make winml_test_common free of test macros (#2902) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Don't generate files outside the build root (#2914) * Don't generate files outside the build root * Add onnxruntime_EXTERNAL_DEPENDENCIES to WinML * Add DML depedency on RESTORE_PACKAGES * User/orilevari/fix yaml merge bugs (#2918) * Add winml test source parameter into cmake function (#2919) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Add arguments to unittest cmake functions * remove comment * Revert "Revert "Merge remote-tracking branch 'origin/master' into windowsai"" This reverts commit ade5abe72a4234fdbc3623093c61c02c6b0bdc26. * Fix breaks from merge with ORT master * Brianma/linux (#2917) * don't include windows.h in cross-plat header * add default case for switch statement * signed/unsigned mismatch fix Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> * User/sheilk/winml adapter c api (#2891) * Create winml adapter c api * fix build * make it build * move adapter into onnxruntime core/session * entry point not exported * minor changes * make model metadata work * make tests pass * implement all the model reflection apis on the adapter c abi * update the new ort interface to create a lotus ennvironment with a logging sink * start adding ort env * move all winml code into adapter folder/lib to isolate it * ensure a single logging manager at a time * start refactoring session * refactor session creation interface * add cpu and dml session option methods to adapter * finish session init * stub out interfaces in ort lib to perform similar mechanics of iinference session * enable profiling, and enable schema override * update session register graph transformers * turn back on custom registry for custom ops * Add sync api * add last c api stubs * should build... but all feature values are broken since this is in flight to moving all implementation details into ivalue * remove ep adapter header * Implement DML execution provider functions from adapter (#2846) * Implement DML execution provider functions from adapter * Use functions in OnnxruntimeEngine.cpp * make map/sequence type_infos freeable, and start implementing ivalue * make it build again * implement value methods * implement remaining methods * remove com adapter abi * check dml session * cache the allocator on ivalue * check if resource is cpu/gpu when access its mutable data * update tensor * mismatched parentheses * fix tensor base and binding obj * it evaluates tensors! sometimes... * minor fixes * enable gpu evals * wrapper all existing winml adapter apis with API_IMPL to try catch (#2854) * update winml... tensor strings are broken, need to template tensorbase to do different things for strings * make tensor strings work with 2 copies in/2 copies out * Fix tensor string and allocator bug * make maps work again... needs some fixes still * Make it build! * enable map inputs * map outputs * unbound outputs for sequences and maps * User/xianz/merge windowsai (#2883) * Packaging pipeline changes for VS 2019 (#2711) * Tiny fix to codegen * Simplify cache implementation and avoid static variables that may carry over between models * Extend DML kernels (#2641) * Additional DML operators * Check unsupported attributes and inputs * Address PR comments * Add kernel capability function used for partitioning, and re-enable stride-based int64 support based on value range * Fix test failures * Build fix * PR comments * Update Nuphar tutorial notebook (#2721) 1. Reflect int8 GEMV improvements for multi-threading from #2696 2. Add notes on multi-threading control using OpenMP 3. Add samples of running multi-isa AOT, and show int8 GEMM differences between AVX and AVX2 4. Add rnn_benchmark example to resolve #1993 * Add schema for new Qops (#2611) * Add schema for new Qops * adding shape inference + qlinearaveragepool * plus review comments * plus review comments * updates per review comments * plus review comments * [server] Add supposed for model_name and model_version as cli parameter (#2708) * remove 64bit warning message from python validation. (#2727) * MLAS: ARM64 build fix (#2734) fix bad usage of vreinterpret to cast vector element types * Fix broken python docs links (#2740) * Fix build on Mac OS (#2731) mac os ld doesn't support --while-archive, correct option is -all_load * fix ngraph wheel (#2737) * fix ngraph wheel 1.1.0 onnxruntime_ngraph wheel doesn't work * remove libdnnl.so in nGraph Libs * make it easy to compare * Split onnxruntime server to a separated folder (#2744) * Fix build for Python 3.8 (#2747) * Fix build for Python 3.8 * Update protobuf to 3.11.2 (#1928) Update protobuf to 3.11.2 (#1928) * Change default optimization level to All (from Basic) (#2745) * change default optimization level to All (from Basic) * fix test * fix c# test * Update numpy to 1.18 (#2758) * Update numpy to 1.18 * Pipeline changes for python 3.8 (#2753) 1. Pipeline changes for python 3.8 2. Fix a regression in setup.py which was just introduced in the previous commit. Please notice, we still haven't made python 3.8 + Windows + CUDA work. * Add basic stacktrace output for posix debug builds. (#2749) * [NupharEP] fix a race condition when multiple sessions running different models concurrently (#2772) * Revert "Change default optimization level to All (from Basic) (#2745)" This reverts commit 56bb503c2f26474b6613bcb2a198691a11dcef38. * Fix typo in error message (#2736) * Rename MKL-DNN to DNNL to fix broken link (#2730) * Fix nightly build version number issue * Pass BUILD_BUILDNUMBER to linux docker * Disable featurizers in python packages * Import more featurizers (#2781) Make kernels non-template. Add input constraint for learnt data. Add min_max_scalar_transformer, robust_scalar_transformer, inputation_marker_transfomer, label_encoder_transformer, missing_dummies_transformer along with tests. Advance Featurizers library commit. * Implement a more stable softmax (#2715) * Implement a more stable SoftMax e^x is represented as infinity if x is large enough, like 100.f. Infinity divided by Infinity is a NAN. Thus, softmax gets a NAN if one or more item are large enough. A math transform as below is leveraged to get a stable softmax: e^xi/(e^x1 + ...e^xn) = e^(xi - max) / (e^(x1 - max) + ... + e^(xn - max)) And for convenience, force max to 0.f if all xi are negative * Contributing: Fix a typo (#2784) * ACL EP GEMM improvements (#2780) When it is posible we use a fully connected layer instead of the gemm implementation. This will let the library use the best implementation based on the input data. * ACL EP convolution improvements (#2774) Added the optimized implementation for depthwise convolution for both ACL v19.02 and ACL 19.05. Also the pointwise convolution seems to be more optimal in the CPU implementation so we opted for that instead. * Add script for release Nuget validation (#2719) * Initial commit * Nits * Disable a test temporarily * Change working directory * Test * Add download python step * Test update * More changes * Fix space issue * Fix * Verify nuget signing * Fix * Spaces * PR feedback * Nit * Fix * Fix * Remove temporary changes * add uint8 support to where op (#2792) * Improve bert optimization script: (#2712) (1) Move input int64=>int32 conversion to embed layer fusion. (2) Output epsilon attribute for LayerNormalization fusion. * add session creation time cost. (#2798) * ML.NET team needs featurizers within a package (#2789) Add auto ml featurizers to Windows, MacOS as well as to GPU packaging-pipelines. * Initialize max of softmax with lowest of float (#2786) * MLAS: update SGEMM threading parameters (#2808) * add interface to copy batch tensors. (#2807) * add interface to copy batch tensors. * onnxruntime * speed up Windows TRT CI (#2811) * don't run cuda tests if building with tensorrt * remove unnecessary build options for win trt ci * refactor win gpu tensorrt ci yml * --numpy_version=1.17 * update * update * azcopy and cuda path * Update test data (#2356) * Add timeseries imputer transformer featurizer kernel (#2813) Make kernels non-template. Add input constraint for learnt data. Fixup tests. Add two more featurizers along with tests. Tests fail. min_max_scalar_transformer robust_scalar_transformer Fix tests serialized stream by prepending version bytes. Add inputation_marker_transfomer and the test. Fix up float/double type designations. Added label_encoder_transformer along with a test. string_throw case is broken at the momement. Fix labelencodertransfomer_test.cc string_throw case Rename maxabsscalertransformer_test.cc Add MissingDummiesTransformer along with the test. Update manifest. Add TimeSeriesImputerTransformer definition, implementation and tests * Fix memory leak in TRT (#2815) * fix memory leak issue * revert EP_FAIL on enueueV2 * Add manifest missing comma * Run static code analyzer on most of our code (#2817) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * update quantization doc (#2783) * update documentation for quantization script * plus some spell corrections * Filter CPU case for IsFloat16Supported (#2802) * update default optimization level + fix gemm_activation fusion (#2791) * update defualt optimization level + fix gemm_activation fusion * fix typo * add unit test and incorporate review comments * fix test comment * Fix dnnl wheel package name (#2823) * Append '-dnnl' to whl package name when --use_dnnl * Update build.py * Update Ubuntu & TensorRT version in README (#2820) Dockerfile.tensorrt is using nvcr.io/nvidia/tensorrt:19.09-py3 as base Image, update Ubuntu and TensorRT version according to https://docs.nvidia.com/deeplearning/sdk/tensorrt-container-release-notes/rel_19-09.html#rel_19-09 * Merge fixes * Add OneHotEncoder and HashOneHotEncoder kernels. (#2830) Add defs and imlementation for OneHotEncoders, adjuist date_time_transformer kernel and test. Add OneHotEncoder kernel test. Add HashOneHotVectorizerTransformer unit test. This does not link due to multiple definitions of functions that are included into header from a CPP file. * Upgrade gtest to the latest version (#2827) WinML would like to update the googletest submodule. They want some newer features (namely GTEST_SKIP to skip tests programmatically and be able to skip entire fixtures easily) and would need to update the submodule version. However, because the new version of code hit a bug in gcc, even though the bug is already fixed in the latest gcc but we're using gcc 4.8.x and it won't get patched for the bug, so we have to do a compromise, change our code a little bit to make it work. The gcc bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=51213 * Add support for int64_t for topk CPU. Fixes github issue #2806. (#2833) * Ignore allocator type in ExecutionProviders allocator map. Make default initialization of OrtMemoryInfo more clearly invalid. (#2768) * Remove allocator type from the key comparison in ExecutionProviders. Remove usage of DummyArena as it's no longer necessary. * Fix x86 tests where arena allocator is disabled. Make initialization of OrtMemoryInfo clearer by adding Invalid enum value. * Make OrtValueNameIdxMap::MaxIdx more intuitive. * Convert ExternalProject Featurizers into git submodule (#2834) Add git submodule for Featurizer library. Update cmake to build for git submodule. * add domain check for nodes + update documentation (#2831) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Fix Linux CUDA nuget packaging pipeline break * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * update optimization doc for BERT related fusions (#2819) * Add bert related transformers to doc * Add execution provider and comment for bert optimizations * Add comment about accuracy impact of approximation * Fix warnings that cause build to fail * MLAS: enable threading for quantized GEMMs (#2844) * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * remove old winmladapter cpp Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * move sequence implementation into ort lib... still commented out... need to turn back on... * begin sequence implementation * make maps and sequences work * fix broken tests * remove dead code * misc cleanup * CR feedback * User/xianz/winml adapter c api (#2869) * wrapper all existing winml adapter apis with API_IMPL to try catch * Return HR or Throw for WinML adapter APIs if failed * undo macro wrapper for two places * Wrap error macros around ort apis, too. * address CR feedback #2 * add more api throw/return macros * Revert changes no longer needed * revert changes to cxx api * format winml lib.ort and winml adapter * remove static pheonix singleton Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * missing use_dml check in winml_adapter_session (#2930) * --use_dnnl flag was mangled in merge (#2931) * use dml macro not wrapping custom registry code (#2934) * Disable LNK4199 winml_dll to enable cuda builds (#2936) * Disable LNK4199 in winml_dll * linkler->linker * LearningModelSessionAPITestGpu.CreateSessionWithCastToFloat16InModel should return DXGI_ERROR_UNSUPPORTED when FP16 not supported (#2937) * Disable LNK4199 in winml_dll * linkler->linker * Need to return DXGI_ERROR_UNSUPPORTED when Model does not support fp16 * Publish build symbols (#2939) * Publish build symbols * Don't upload PDBs for .exe files * Make x86 build (#2943) * fix last remaining size_t/int64_t warnings->errors (#2948) * TensorString, Sequences and Maps use the first allocator, but should use the cpu default allocator. (#2952) * fix tensor string allcoator * clean up default allocator usage for strings in winml lib/api.ort Co-authored-by: Ryan Lai <ryalai96@gmail.com> * Handle tensor shape of zero (#2954) Co-authored-by: Ryan Lai <ryalai96@gmail.com> * CR feedback (#2970) * CR feedback * fix weird formatting on privacy readme * Add 'All rights reserved.' everywhere * readd all rights reserved to winml_provider_factory.h * remove extra space in comment * remove extra whitespace * fixes post master merge * remove winml from nuget gpu pipeline * set IR VERSION on generated_model in rnn_benchmark (#2972) * Fix slice conformance failures (#2908) Co-authored-by: Adrian Tsai <adtsai@microsoft.com> Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Paul McDaniel <paul_mcdaniel@hotmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Dwayne Robinson <fdwr@hotmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com>
2020-02-05 01:12:19 +00:00
interface ILearningModelFeatureValue
{
//! The data type of the feature.
LearningModelFeatureKind Kind{ get; };
};
//! \class LearningModelBinding
//! \brief Holderforassociationsbetweenmodelinputs/outputsandvariableinstances.
[contract(MachineLearningContract, 1)]
INBOX_ONLY([constructor_name("Windows.AI.MachineLearning.ILearningModelBindingFactory", c95f7a7a-e788-475e-8917-23aa381faf0b)])
INBOX_ONLY([interface_name("Windows.AI.MachineLearning.ILearningModelBinding", ea312f20-168f-4f8c-94fe-2e7ac31b4aa8)])
Merge windowsai (winml layering) into master (#2956) * Initial Commit * Merged PR 3985217: add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc (#2346) add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc and violating our OS layering requirements. We linked against onecoreuap_apiset.lib in VB so we will continue doing this, but I am still unsure why not to link against onecore instead since that is where we ship. However, since Sheil is the owner of this code we will wait to discuss with him before changing anything. * Initial changes for layering * more snipping to get core into ort * update build instructions to include --build_shared_lib (#2358) * update build instructions to include --build_shared_lib * fix line breaks * Task 23998197: add winml_lib_core into onnnxruntime.dll (#2368) * Task 23998197: add winml_lib_core into onnnxruntime.dll * PR feedback build break on perf_test * return proper error when the model path isn't found (#2391) * LearningModelSession is cleaned up to use the adapter, and parts of b… (#2382) this is a big PR. we are going to move it up to layer_dev , which is still a L3 so we are still safe to do work there agile. we are going to move this into the L3 so that ryan can start doing intergration testing. we will pause for a full code review and integration test result prior to going into the L2. >>>> raw comments from previous commits >>> * LearningModelSession is cleaned up to use the adapter, and parts of binding are. * moved everything in the winmladapter made it all nano-com using, WRL to construct objects in the ORT side. base interfaces for everythign for winml to call cleaned up a bunch of winml to use the base interfaces. * more pieces * GetData across the abi. * renamed some namepsace cleaned up OrtValue cleaned up Tensor cleaned up custom ops. everything *but* learnignmodel should be clean * make sure it's building. winml.dll is still a monolith. * model moved over. everything builds clean. step ! * weak ref comment * Layer dev paulm (#2408) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * Layer dev paulm (#2414) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * Layer dev paulm (#2423) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * Layer dev paulm (#2424) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * Layer dev paulm (#2425) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Layer dev paulm (#2426) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * fixed map and sequence when passing stl types across the ABI . found a leak in nvidia driver, but skipped it. all winmlapitests pass now * Moved SessionOptions over to the abi * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * Register WinML TraceLogging provider on Onnxruntime.dll (#2455) * Register WinML TraceLogging provider on Onnxruntime.dll * Add ifdef to make sure trace logging provider has telemetry option when LAYERING_DONE * No need for ifdef for TraceLoggingOptionMicrosoftTelemetry * PR feedback * Move etw registration into lotus environment constructor and deresgister in lotus environment destructor * Brianma/cpuwinml (#2466) * allow building winml cpu without dml. * Brianma/breaks (#2469) * fix some more breaks * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * move dml checks out of winml and into the adapter * better error handling * Brianma/fi (#2470) * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * fix bad merge * Checking in a staging checkpoint point so that Ryan can work with me in parrallel * build break. * Brianma/testfails (#2473) * add missing ir version to dictvectorizer-string.onnx * add missing ir version to relu.onnx * add missing ir version to zipmap*onnx * add IR version to manually generated models * remove an unnecessary ifdef dml * Brianma/windowsai fi (#2475) * update dockerfiles/README (#2336) * Make elementwise op run 4 items per thread (#2335) Description: Describe your changes. Make elementwise op run 4 items per thread unroll for loop to leverage ILP remove unnessary N==0 check inside elementwise GPU kernel Motivation and Context Why is this change required? What problem does it solve? It can improve the performance of GPU elementwise ops. ~2% performance gain on popular NLP bert model. If it fixes an open issue, please link to the issue here. * Add CUDA GatherElements kernel (#2310) * Updates * Update test * Update * Updates * nits * PR feedback * Update * Update * PR feedback * PR comments * Update * Fix build * Fix build * Nits * Fix * Layer Normalization Fusion (#2319) basic layer normalization transform * Add FastGelu Cuda Op for Gelu and Add bias fusion (#2293) * Add FastGelu cuda op * Add AddBiasGelu for experiment * Revert "Add AddBiasGelu for experiment" This reverts commit 5c1ee019858c657e6bb75887265cb85675626e5b. * Add bias * Add unit tests * update comment * update script * fix build error * update coding style * update for CR feedback Enable half2 optimization only when cuda arch >= 7.0 * move _Tanh to common.cuh * implement CPU contrib OP Attention (#2333) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanUnusedInitializers. (#2320) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanupUnusedInitializers. This means initializers that have been replaced during graph optimizations are not left in the GraphProto when we save an optimized model. * Handle edge case where a model has an unused initializer with matching graph input by also removing the graph input. * Use non-const iterators in std::find_if calls to make centos build happy. * Nuget pipeline changes (#2305) 1. refactor the pipeline, remove some duplicated code 2. Move Windows_py_GPU_Wheels job to Win-GPU-CUDA10. We'll deprecated the "Win-GPU" pool 3. Delete cpu-nocontribops-esrp-pipeline.yml and cpu-nocontribops-pipeline.yml 4. In Linux nuget jobs, run "make install" before creating the package. So that extra RPAH info will be removed * Cuda Reverse Sequence Op, maping types of same size using same template function. (#2281) * Set ElementType to String type of node metadata, instead of byte[] (#2348) * Set ElementType to String type of node metadata, instead of byte[] * Fix spacing * Introduce PrimitiveType into a Type System along with an integer constant (#2307) Improve perf by avoiding GetType<T>() calls. Introduce MLTypeCallDispatcher to switch on Input Type. Add Tensor IsType<T>() fast method. * Fix/test dim value of 0 handling in a couple of places (#2337) * Update the CUDA Where implementation broadcasting logic to handle a dim with value of 0. Add unit test Also add unit test for unary op with dim value of 0 * Exclude ngraph from Where test with 0 dim. * Openvino EP R3.1 onnxrt server (#2357) * onnxrt server with OVEP * onnxrt server with OVEP * Update Dockerfile.server.openvino * onnxrt server OVEP fix reviews * onnxrt server OVEP fix reviews * Implement cuda nonzero op. (#2056) Implement cuda nonzero op. * Direct use python numpy array's memory if already contiguous. (#2355) * Direct use python numpy array's memory if already contiguous. This could greatly improve performance for session with large input, like big image 1920x1080 fastrcnn, 30~40% speed up could be achieved. * Add test case enforce contiguous/non-contiguos numpy array as inputs. * Add helper to create output to minimize binary size. (#2365) Add ConstEigenTensorMap typedef so we don't unnecessarily const_cast the const input Tensor. * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS (#2369) * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS * update * Add Tracelogging for profiling (#1639) Enabled only if onnxruntime_ENABLE_INSTRUMENT is ON * test bidaf with nuphar for avx target (#2370) increase nuphar test coverage a bit * Fix a bug in TLS refcount that may destabilized CUDA CI (#2374) * update output size calculation for resize (#2366) * change how output size is calculated for resize op * add tests for ver 10 resize * Extend OneHot CPU kernel to support more types (#2311) * Extend OneHot CPU kernel to support input int64_t, depth int32_t, output float * Skip BERT before the test data fix is picked up * Fix bug with Slice. Need to pass in flattened input dimensions so the initial offset into the input is calculated correctly. (#2372) * Add opset 11 version of Split to CUDA ops (#2376) Organize the CUDA ops definitions so all the opset 10 and 11 parts are together (same setup used for CPU ops) * Layer Norm Fusion Fix (#2379) * layer norm fusion fix * Add input shape check in code and unit tests * Fuse Add + Gelu (#2360) Implement the transformer to fuse add + gelu Implement the accurate kernel * Skip layer norm transform (#2350) * skip layer normalization transformer * Another try to stabilize CUDA CI (#2383) The root cause seems to be failure in CUDA dealloc when tear down. cudaFree return code was ignored before, so should the debug check. * fix BUILD.md typo (#2375) build.py: error: argument --config: invalid choice: 'RelWithDebugInfo' (choose from 'Debug', 'MinSizeRel', 'Release', 'RelWithDebInfo') * Fixed compilation with ngraph (#2388) * Fix reuse logic in allocation planner. (#2393) * Fix reuse logic in allocation planner. * PR comments * Add helpful comments * Don't allow reuse across string tensors. * [NupharEP] Multiple optimizations (#2380) Fuse transpose into MatMul Implement Pow and constant scalar simplification Vectorize ReduceMean Improve symbolic shape inference Minor updates for better debugging in fused function name * Avoid using the default logger in the graph lib and optimizers (#2361) 1. Use the session logger if it is available. 2. Don't disable warning 4100 globally. We should fix the warnings instead of disabling it. * Change CUDA implementation of Transpose to support all fixed size tensor types (#2387) * Change CUDA implementation of Transpose to not use a typed kernel so we can support more types with minimum binary size. Add support for 8, 16, 32 and 64 bit types. Add unit tests. Add method so the implementation can be called directly (will be used by CUDA Scan very soon). * Disable TensorRT for MLFloat16 and int8 unit tests. * Address PR comment and add support for calling cublas implementation if type is mlfloat16. * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. (#2398) * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. * [NupharEP] force some low/zero cost ops to be inlined (#2409) * fix cross compile bug (#2415) * Minor optimization: if a node has already been placed, there's no need to find a kernel for it. (#2417) * Add Reshape Fusion (#2395) * Add reshape fusion * Add some comments * update comments * update comment format * update according to feedback * update for recent logger change * fix build error * (1) Support both input and output edges in find path in graphutils (2) Add a test case of only one constant initializer of Concat input. (3) Refactor ReshapeFusion class to allow add more subgraph fusion in the future. * fix error * (1) loose constraint on initializer: non constant is allowed for reshape fusion. (2) Change versions type to vector. (3) Add logging. (4) Return false when multiple output edges matched in FindPath. Add comments. * only allow one direction (input or output) in FindPath * [NupharEP] Update notebook and docker image (#2416) Add BERT squad in Nuphar tutorial Enhance speed comparsion readability * Fix the issue in matmul_add_fusion (#2407) Fix the issue in matmul_add_fusion If Muatmul + Add has shape [K] * [K, N], reset it to [1, K] * [K, N] will make the output shape to [1, N] will also requires a reshape on the output. Fix: just remove the shape reset to not fuse it. Add a negative test case for matmul+add fusion * feat(treeregressor): Update TreeEnsembleRegressor for type support (#2389) Updates the `TreeEnsembleRegressor` to allow for `double`, `float`, `int64`, and `int32` inputs to match the upstream specification. Signed-off-by: Nick Groszewski <nicholas.groszewski@capitalone.com> * onnxrt server documentation update (#2396) * Added support for Pad-2 operator in OpenVINO-EP (#2405) * Add CUDA If operator. (#2377) * Add CUDA If operator. Uses CPU operator for implementation. By adding a CUDA version the inputs/outputs (with the exception of the 'cond' input) stay on GPU, and no other logic is required to avoid a copy to CPU across the control flow node. * Improved documentation for onnxruntime::utils::SwapByteOrderCopy(), added precondition check. * Fix the type constraints on CUDA If operator to exclude strings. (#2431) * add Im2col<uint8_t> (#2438) * Adjust codegen vectorization width from target (#2439) * Adjust codegen vectorization width from target * Add CUDA Scan operator. (#2403) * Add Scan CUDA op. Uses CPU implementation for logic. Added some device specific functors for handling when data needs to be manipulated on a different device. Added ability to override the materialization logic in the OrtValue slicer so DML can plugin their handling. * Fix Windows GPU C API packaging pipeline failure (#2440) Fix Windows GPU C API packaging pipeline failure (#2440) * Correctly handle implicit inputs for fused nodes (#2390) * Correctly handle implicit inputs for fused nodes Previously, nuphar's partitioning function didn't include node's implicit inputs into the inputs list of MetaDef, and hence a crash was triggered in the onnx graph checker. This commit fixed the issue. Furthermore, it also fixed a related issue where we didn't add implicit inputs into graph_inputs_excluding_initializers_ in Graph::SetGraphInputsOutputs. the issue was that graph_inputs_including_initializers_ populated by SetInputs (e.g. called by FunctionImpl::FunctionImpl) may contain implicit inputs which were not of any node's initializers in the graph. Because they were not part of any initializers, these implicit inputs couldn't be visited by going through all nodes' inputs. Consequently, they would *not* be added into graph_inputs_excluding_initializers_. We fixed the issue by first copying the populated graph_inputs_including_initializers_ into graph_inputs_excluding_initalizers_, which then had both initializers and non-initializers as its initial content. Later, we erase initializers from the list. In this way, we can ensure all implicit inputs to remain in graph_inputs_excluding_initializers_. * refined comments and fixed duplicates Address CR by revisiting comments in terms of implicit inputs Also fixed an issue by skipping duplicates while copying inputs from graph_inputs_including_initializers_. * address CR explain why we need to collect nodes' implicit inputs * don't rely on pointer values for iterating std::set Previously, openvino relied on iterating a set of NodeArg pointers to construct inputs and outputs for a fused graph. It could cause non-determinism. The reason was that although iterating std::set by itself is stable, pointer values of NodeArgs may vary. Consequently, we could end up visiting the set's elements in different orders for different runs for the same test, which resulted in constructing inputs (and outputs) with different orders to the fused graph. For example, for the same test, we may have inputs [A, B] in some runs but inputs[B, A] in others. Let's use std::string as the key type to avoid such nondeterminism. This commit also added implicit inputs into meta->inputs while returning the capability from the openvino provider. * Fixed another latent issue in openvino's GetCapability function The issue was that we couldn't simply erase fused_inputs and fused_outputs while iterating the nodes. For example, an output NodeArg may have multiple uses, and it's wrong if we erase it from fused_outputs when we encounter only one of its uses as input. * Remove DeviceAllocatorRegistry class (#2451) Remove DeviceAllocatorRegistry class * CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library. * Parallel Gelu with ParallelFor (#2399) Parallel Gelu to get better performance for Gelu * Clean up build.py (#2446) * Pull the latest image before running docker build * Fuse SkipLayerNorm with Bias (#2453) Fuse SkipLayerNorm with Bias * Allow more than one invocation of CreateEnv in the same process. (#2467) * Allow more than one invocation of CreateEnv in the same process. * Fix centos build * Symbolic shape inference improvements: (#2460) * Symbolic shape inference improvements: - add a mode to guess unknown ops' output rank - add support for GatherND - add support for If - fix a bug in get_int_values when then tensor rank > 1D, by treating it as no sympy data - add symbol to literal merge when ONNX silently merges dims - fix a bug in Concat when input dim is 0 - fix a bug in ConstantOfShape that computed dim is not updated - add support for dynamic shape in ConstantOfShape - fix a bug in Loop output shape that loop iterator dim is not inserted at dim 0 - add support for dynamic padding in Pad - add support for dynamic shape in Reshape - add support for Resize with opset > 10, by treating output dims as dynamic - fix a bug in Slice when starts/ends are dynamic - restrict input model to opset 7 and above - make output model optional to avoid disk write when testing Run model tests for symbolic shape inference Reduce 2GB docker image size of nuphar * add additional test data set for nuget pipeline (#2448) * add SAS token to download internal test data for nuget pipeline * update azure endpoint * fix keyvault download step * fix variable declaration for secret group * fix indentation * fix yaml syntax for variables * fix setting secrets for script * fix env synctax * Fix macos pipeline * attempt to add secrets to windows download data * fix mac and win data download * fix windows data download * update test data set url and location * Revert "Brianma/windowsai fi (#2475)" This reverts commit 5780b864a15513fda4eadbfc2b5345fefe70b5ec. * Add scenario tests (#2457) * Add scenario tests * Remove TODO from model license * Add winml_api test dependency * fix model load test. fi from master changed the constructor (#2483) * make api tests all pass (#2486) * fix bad merge * fix bad model merge * Layer dev paulm (#2492) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * Rename ambiguous header (#2489) * fix one more missing IR version model (#2500) * add missing IR version to 4 more models used by scenario tests (#2501) * Add CLI parameters to test runner, build WinML in ARM and x86 CI (#2479) * Support test parameters through CLI arguments * Add WinML do Windows x86/ARM CI builds * Code style fixes * Update googletest Remove GPUTEST macros everywhere now that GTEST_SKIP is supported * Refactor main.cpp * Build scenario tests without DML * Link scenario tests to DML when it's enabled (#2502) * Layer dev release pipeline (#2488) Adds winml binaries to existing cpu nuget package, and creates new gpu dml nuget package with winml binaries and DML EP. * Layer dev paulm (#2506) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * Remove usage of IOBinding in WinML and use C_API Run method (#2504) * remove usage of iobinding * Change data structure to use vector of Ort::Values * Polish bind input / output * Use C APIrun method * Update providers on evaluate getresults * Remove run and IObinding interface from WinMLAdapter * Remove use of IObinding * bind unbound outputs code moved to learningmodelbinding * clean up unneeded istensor adapter function * Fix comment * Check if session is closed before binding and clearing * PR feedback * Layer dev paulm (#2507) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * Make tests dependend on winml_dll (#2509) * add dml binaries to DirectML package and be more explicit about condition variables (#2520) * re-enable warnings for winml builds and fix the warnings that were hiding (#2526) * turn devmode back on for winml builds * fix some warnings. include protobuf in a way that disables some warnings * undo protobufhelpers changes and just ignore 4100 errors in pb code * attempt to isolate protobufhelpers errors * add template specialization for getting tensor proto data * Layer dev paulm (#2533) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * Spawn child process to run DeviceLostRecovery scenario test (#2530) * Spawn child process to run DeviceLostRecovery scenario test * Layer dev paulm (#2536) ori said yes * add missing namespace to winml_trace_logging_provider in lotusenvironment.h (#2542) * Handle exception thrown from all apis in WinMLAdapter (#2539) * various changes to unblock windowsai ADO build * Fix custom ops scenario tests (#2562) * Do not shutdown protobuf after ort environment gets destroyed. Lazy load lotus environment first time it is needed * comment typo * pr comment about calling phoenix singleton * Make lotus_environment static in winmladapter * Layer dev paulm (#2567) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * making windowsAI azure dev ops work. * code review comments. * revert changes * Cmake and preprocessor fixes that where uncovered by building on agents without DML available via SDK * Layer dev dml delayload (#2580) * Brianma/cpu (#2583) * don't include dml stuff in cpu builds * tests that link the image lib also need the telemetry lib now * Throw Winml_err_invalid_binding if binding gpu resource on cpu device (#2589) * Throw Winml_err_invalid_binding if binding gpu resource on cpu device * PR comments. No need to query executionprovider for is gpu device * User/xianz/ortthrow (#2596) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * User/xianz/ortthrow (#2599) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * add status error message * Remove uwp onsuspending winrt call because logruntimeperf is getting removed (#2630) * User/xianz/dedup telemetry (#2631) * investigate duplication of telemetry in winml and ort * remove winml telemetry events * telemetry executionProviderEvent * remove unneccessary file and refactor code little bit * Revert back TelemetryEvent, which send up ETW event. * merge changes from layer_dev to windowsai (#2638) * Remove underscore from googletest names (#2616) * Fix leaking memory allocator Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24278761 and https://microsoft.visualstudio.com/OS/_workitems/edit/24330198 * Explicitly initialize Ort::Value with nullptr * Cache WinML adapter * bad merge * define private version of dxcore enum that is added in 19H1 SDK. (#2654) * add comment for explaning private definition of dxcore d3d feature level ennum value. (#2672) * do not package directml.pdb for redist packages. (#2676) * Fix leaking operator registry (#2645) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24354916 * User/orilevari/windowsai master merge (#2674) merge resolutions included pulling in telemetry logic that was merged to master and not windowsai and dereferencing InferenceSession::sessionstate now that it is a unique pointer * Delete Ort Allocator in LearningModelBinding (#2653) * Delete OrtAllocator in LearningModelBinding * PR comments to make Ort::Allocator a smart pointer * Small comment change * PR feedback to clean up code * PR feedback on move semantics * Clean up std::move * Fix memory leaks (#2679) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24356109, https://microsoft.visualstudio.com/OS/_workitems/edit/24388361 and https://microsoft.visualstudio.com/OS/_workitems/edit/24388596 * various changes to properly organize and skip GPU tests. For now for No DML builds we will not run GPU tests at all. In the future we should adapt the tests to expect the appropiate errors. (#2695) * Windowsai without fi (#2701) * Disable Attention fusion tests when DISABLE_CONTRIB_OPS is defined (#2529) * Setup java ci (#2528) * Add provision in ORT for session options to be parsed when available via model file (#2449) * Initial commit * Fix gitmodules * Nits * Nits * Updates * Update * More changes * Updates * Update * Some updates * More changes * Update * Update * Merge * Update * Updates * More changes * Update * Fix nits * Updates * Fix warning * Fix build * Add comment * PR feedback * PR feedback * Updates * Updates * Update * More changes * Fix build break * Comment test for now * Updates * Updates * PR feedback * Updates * Nits * Add tests * Fix build * Fix build * Fix build * Fix build break * Fix build * Nits * PR feedback * More change * Expose GetSessionOptions in pybind logic and add unit test for python * Fix build * PR feedback * PR feedback * Revert "Disable thread pool creation when enabled OpenMP (#2485)" (#2535) This reverts commit 7c7d5a149c9ed52eec67304bae5c4b132166a8a1. * Add dynamic shape support in TensorRT execution provider (#2450) * remove onnx-tensorrt submodule * add new onnx-tensorrt submodule (experiment) for trt6 * update engine build for trt6 * update compile and compute for tensorrt6.0 * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * switch to onnx-tensorrt master for TensorRT6' * Update tensorrt_execution_provider.cc * Handle dynamic batch size and add memcpy in TensorRT EP * update test cases * Update tensorrt_execution_provider.cc * update onnx-tensorrt submodule * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.ubuntu_tensorrt * Update run_dockerbuild.sh * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update concat_op_test.cc * Update tensorrt_execution_provider.cc * Upgrade TensorRT to version 6.0.1.5 * Update onnxruntime_providers.cmake * Update CMakeLists.txt * Update reduction_ops_test.cc * Update install_ubuntu.sh * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.tensorrt * Update BUILD.md * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update onnxruntime_providers.cmake * Update install_ubuntu.sh * Update install_ubuntu.sh * Update gemm_test.cc * Update gather_op_test.cc * Update CMakeLists.txt * Removed submodule * update onnx-tensorrt submodule * update header file * Removed submodule * add submodule onnx-tensorrt kevin's branch shape-test' * add debugging code * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * merge master * Removed submodule * update onnx-tensorrt submodule * add more changes for dynamic shapes * Update tensorrt_execution_provider.cc * update for dynamic shape * update dynamic shape processing * fix logger issue * remove submodule onnx-tensorrt * add submodule onnx-tensorrt * add env variable min_subgraph_size * remove redundency * update document * use onnxruntime::make_unique * fix multi-run issue * remove some tests to save CI build time * Add dynamic shape test * Update TensorRT-ExecutionProvider.md * Add example of running Faster R-CNN model on TensorRT EP * Add more details on env variables * update environment variables * Update tensorrt_basic_test.cc * Update model tests * Update tensor_op_test.cc * remove --use_full_protobuf * Update build.py * User/xianz/telemetry (#2458) * enabme telemetry * enable telemetry * set enable telemetry as default * for debugging * remove log and set disable telemetry as default back * delete private file while testing * resolve comment: mainly add license header, rename macro and update docs * rewording in privacy.md * Fix integer overflow in cuda NonMaxSuppression implementation (#2540) * add test case that should pass but fail * fix nms * extract int_max_output_boxes_per_class * Introduce container type runtime checks and other improvements (#2522) Rework TensorSeq in a manner consistent with Tensor and SparseTensor in terms of type system setup. Reduce templating. Introduce helpers to ensure the same data type. Make OrtValue __dtor not virtual. Introduce ContainerChecker * Fix C API tests for centos and mac (#2544) * change c++14 to c++11 * add ld lib path for centos * enable csharp tests on macos * fix C API test on MacOS + fix manylinux dotnet install * fix manylinux dotnet install * fix lib link * Add back executable bit to build.py * Fix a bug handling negative begin pad values in Pad op (#2550) * Fix bug in Pad op * Update * DNNL CMAKE update (#2548) * Fix android build (#2558) * Update win-x86-ci.yml (#2557) Fix build pipeline break * Re-enable Windows C# tests (#2564) * disable onnx_test_runner -x invocations for dnnl (#2568) * Allow sequence length to be symbolic (#2559) * setup java ci mac (#2570) * make layernorm fusion to support opset 11 (#2545) * Fix a warning found in the latest VS release * Add more check on SkipLayerNorm and BiasGelu fusion (#2574) * Fix file not found error during docker build. (#2569) * Add ConvTranspose1D (#2578) * Ryanunderhill/packagename test (#2582) * [Nuphar EP] fixes for some object detection models (#2581) Update notebook tutorial with multi-threaded int8 GEMM from #2517 * EmbedLayerNormalization Fusion Improvement (#2553) Embedding layer norm fusion improvements - add more checks * Update version (#2584) * Temporarily exclude vgg19 test from Python backend test 1. temporarily exclude vgg19 test which comsumes too much memory, run out of memory on Upsquared device. Single test pass for vgg19, need furture investigation (#2588) 2. Update docker file to decrease the docker image size * Update docs for Android NNAPI EP (#2586) * Fix lto bug for protobuf and ubuntu * add path to build dir before test run (#2590) * Add missig env variables for mac pipeline test (#2595) * Fixed an issue in updating realized dims (#2597) when we update realized dims for scan's output, the sliced axis also needs to be inclusive, i.e. we should check with "dim >= insert_inclusive_axis", because the offset in the symbols are based on Scan sugraph. Otherwise, we would end up with shape mismatch later. * Java API for onnxruntime (#2215) * Add support for opset 11 in reshape fusion (#2592) Support opset verion 11 in reshape fusion * Rename automl python tools folder to featurizer_ops. (#2593) * Support opset 11 subgraph of Squad model in Embed Layer Normalization (#2605) Support opset 11 Squad model that is exported from PyTorch nightly. The embed layer uses Range op which is missed in the transformer. * symbolic shape inference: fix warnings in GPT-2 model (#2608) And revise nuphar perf test on BERT squad * Dump subgraph ID and fused graph ID (#2607) * Dump subgraph ID and fused graph ID Dump subgraph ID and fused graph ID for better debugging * Remove local static fused_count added a field global_fused_count_ to NupharExecutionProvider class * EmbedLayerNormalization Fusion For Dynamic Squad Model Opset 10 (#2613) Support subgraph of SQuAD model exported from pytorch with dynamic input axes * Allow providers to be set for InferenceSession at construction (#2606) * Remove unnecessary parameter in some places in GatherElements implementation (#2612) * Remove unnecessary parameter in some places * Update * Update * Make sure fenced tensor could not reuse other tensor. (#2561) Fix random error caused by this. * Improve Embed Layer Norm Fusion for SQuAD with static input shape (#2621) * fix float16 comparison in initializer (#2629) * epsilon attribute for layernormalization fusion (#2639) * removed unnecessary batch file and fix path (#2640) * Add shape inference to ConvTransposeWithDynamicPads schema (#2632) * Improve cuda expand() opeator's performance. (#2624) * Cuda pad optimize when no padding is needed. (#2625) * Shortcut cuda Pad() when no padding is needed. * Optimize cuda scatter() on 2D compatible. (#2628) * Optimize cuda scatter() on 2D compatible. * Add some comments. * fix build error for ARM (#2648) * Improve performance of resize() in Nearest mode (#2626) Special treatment for 2D, check same size as input image. And in 2d kernel, template use_expolation. * Fix memory exception in Layer Norm Fusion (#2644) * Windows CI changes(#2650) * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Revert "Windowsai without fi (#2701)" This reverts commit 285d4c85ff5c4e265f963208170304ef3461e684. * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Deref unique pointer for session_state * send shutdown event when dll is unloaded and EvaluationStop, SessionC… (#2704) * send shutdown event when dll is unloaded and EvaluationStop, SessionCreationStart Events. * Add EvalutationStart Event * add comment * use correct type for for loop (#2755) * ARM CI (#2759) * Set ARM agent pool * Set CMake generator to VS 2019 in ARM * Use system-wide CMake instead of custom version Our custom version is too old for VS 2019 * Use DML and build shared lib in ARM CI * Restore nuget packages in ARM CI * Disable DML * Refactor ARM debug/release builds * Use system packaged Python version * Remove hardcoded Python path * Downgrade Python to 3.7 for build * Remove explicit CMake path * Fix invalid JSON in cgmanifest.json (#2760) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * Remove usage of WHOLEARCHIVE in WinML CMake and add WinMLAdapterFactory (#2726) * Remove usage of WHOLEARCHIVE in WinMLAdapter CMake and add WinMLAdapterFactory * PR feedback, no need for dll(export) since using def file * PR comments * Small comment in gen_def.py * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * Move winml_provider_factory.h to proper location (#2801) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Filter CPU case for IsFloat16Supported (#2802) * Merge fixes * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * Fix warnings that cause build to fail * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * Make DML include path global (#2882) * Make DML include path global * Add generated cppwinrt headers to winml_lib_common * Integrate changes to WindowsAI to make ADO Build (#2886) * Revert "CMake cross-generator fixes (#2790)" This reverts commit dbe7d97fa1ab155f1309bced87199527e8f35bd2. * add additional suppress warning in onnx_proto * ignore /wd4996 warning * DML execution provider fixes * Revert "Revert "CMake cross-generator fixes (#2790)"" This reverts commit 1ae7b4bcbc02edc881ad28685da98e095dfceb17. * Update func signature of custom op function overloads * common devicehelpers fixes * Add pch.h for winml_lib_common * re-add winml_lib_common_dir/inc to include path for winml_adapter * User/orilevari/dml redist shared folder (#2890) * move dml nuget package directory up one level to make it shared between build flavors * Merge conflict fix * Revert "Merge conflict fix" This reverts commit 142fa72cf9ce4344ad717b50b7ea2b8582aadc7c. * Revert "Merge remote-tracking branch 'origin/master' into windowsai" This reverts commit 6e2126d46e5e5f564d65da37dd4f70c93dd81165, reversing changes made to b3f5583dc9249834b947c8ea905f6a98060d5bd6. * Make winml_test_common free of test macros (#2902) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Don't generate files outside the build root (#2914) * Don't generate files outside the build root * Add onnxruntime_EXTERNAL_DEPENDENCIES to WinML * Add DML depedency on RESTORE_PACKAGES * User/orilevari/fix yaml merge bugs (#2918) * Add winml test source parameter into cmake function (#2919) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Add arguments to unittest cmake functions * remove comment * Revert "Revert "Merge remote-tracking branch 'origin/master' into windowsai"" This reverts commit ade5abe72a4234fdbc3623093c61c02c6b0bdc26. * Fix breaks from merge with ORT master * Brianma/linux (#2917) * don't include windows.h in cross-plat header * add default case for switch statement * signed/unsigned mismatch fix Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> * User/sheilk/winml adapter c api (#2891) * Create winml adapter c api * fix build * make it build * move adapter into onnxruntime core/session * entry point not exported * minor changes * make model metadata work * make tests pass * implement all the model reflection apis on the adapter c abi * update the new ort interface to create a lotus ennvironment with a logging sink * start adding ort env * move all winml code into adapter folder/lib to isolate it * ensure a single logging manager at a time * start refactoring session * refactor session creation interface * add cpu and dml session option methods to adapter * finish session init * stub out interfaces in ort lib to perform similar mechanics of iinference session * enable profiling, and enable schema override * update session register graph transformers * turn back on custom registry for custom ops * Add sync api * add last c api stubs * should build... but all feature values are broken since this is in flight to moving all implementation details into ivalue * remove ep adapter header * Implement DML execution provider functions from adapter (#2846) * Implement DML execution provider functions from adapter * Use functions in OnnxruntimeEngine.cpp * make map/sequence type_infos freeable, and start implementing ivalue * make it build again * implement value methods * implement remaining methods * remove com adapter abi * check dml session * cache the allocator on ivalue * check if resource is cpu/gpu when access its mutable data * update tensor * mismatched parentheses * fix tensor base and binding obj * it evaluates tensors! sometimes... * minor fixes * enable gpu evals * wrapper all existing winml adapter apis with API_IMPL to try catch (#2854) * update winml... tensor strings are broken, need to template tensorbase to do different things for strings * make tensor strings work with 2 copies in/2 copies out * Fix tensor string and allocator bug * make maps work again... needs some fixes still * Make it build! * enable map inputs * map outputs * unbound outputs for sequences and maps * User/xianz/merge windowsai (#2883) * Packaging pipeline changes for VS 2019 (#2711) * Tiny fix to codegen * Simplify cache implementation and avoid static variables that may carry over between models * Extend DML kernels (#2641) * Additional DML operators * Check unsupported attributes and inputs * Address PR comments * Add kernel capability function used for partitioning, and re-enable stride-based int64 support based on value range * Fix test failures * Build fix * PR comments * Update Nuphar tutorial notebook (#2721) 1. Reflect int8 GEMV improvements for multi-threading from #2696 2. Add notes on multi-threading control using OpenMP 3. Add samples of running multi-isa AOT, and show int8 GEMM differences between AVX and AVX2 4. Add rnn_benchmark example to resolve #1993 * Add schema for new Qops (#2611) * Add schema for new Qops * adding shape inference + qlinearaveragepool * plus review comments * plus review comments * updates per review comments * plus review comments * [server] Add supposed for model_name and model_version as cli parameter (#2708) * remove 64bit warning message from python validation. (#2727) * MLAS: ARM64 build fix (#2734) fix bad usage of vreinterpret to cast vector element types * Fix broken python docs links (#2740) * Fix build on Mac OS (#2731) mac os ld doesn't support --while-archive, correct option is -all_load * fix ngraph wheel (#2737) * fix ngraph wheel 1.1.0 onnxruntime_ngraph wheel doesn't work * remove libdnnl.so in nGraph Libs * make it easy to compare * Split onnxruntime server to a separated folder (#2744) * Fix build for Python 3.8 (#2747) * Fix build for Python 3.8 * Update protobuf to 3.11.2 (#1928) Update protobuf to 3.11.2 (#1928) * Change default optimization level to All (from Basic) (#2745) * change default optimization level to All (from Basic) * fix test * fix c# test * Update numpy to 1.18 (#2758) * Update numpy to 1.18 * Pipeline changes for python 3.8 (#2753) 1. Pipeline changes for python 3.8 2. Fix a regression in setup.py which was just introduced in the previous commit. Please notice, we still haven't made python 3.8 + Windows + CUDA work. * Add basic stacktrace output for posix debug builds. (#2749) * [NupharEP] fix a race condition when multiple sessions running different models concurrently (#2772) * Revert "Change default optimization level to All (from Basic) (#2745)" This reverts commit 56bb503c2f26474b6613bcb2a198691a11dcef38. * Fix typo in error message (#2736) * Rename MKL-DNN to DNNL to fix broken link (#2730) * Fix nightly build version number issue * Pass BUILD_BUILDNUMBER to linux docker * Disable featurizers in python packages * Import more featurizers (#2781) Make kernels non-template. Add input constraint for learnt data. Add min_max_scalar_transformer, robust_scalar_transformer, inputation_marker_transfomer, label_encoder_transformer, missing_dummies_transformer along with tests. Advance Featurizers library commit. * Implement a more stable softmax (#2715) * Implement a more stable SoftMax e^x is represented as infinity if x is large enough, like 100.f. Infinity divided by Infinity is a NAN. Thus, softmax gets a NAN if one or more item are large enough. A math transform as below is leveraged to get a stable softmax: e^xi/(e^x1 + ...e^xn) = e^(xi - max) / (e^(x1 - max) + ... + e^(xn - max)) And for convenience, force max to 0.f if all xi are negative * Contributing: Fix a typo (#2784) * ACL EP GEMM improvements (#2780) When it is posible we use a fully connected layer instead of the gemm implementation. This will let the library use the best implementation based on the input data. * ACL EP convolution improvements (#2774) Added the optimized implementation for depthwise convolution for both ACL v19.02 and ACL 19.05. Also the pointwise convolution seems to be more optimal in the CPU implementation so we opted for that instead. * Add script for release Nuget validation (#2719) * Initial commit * Nits * Disable a test temporarily * Change working directory * Test * Add download python step * Test update * More changes * Fix space issue * Fix * Verify nuget signing * Fix * Spaces * PR feedback * Nit * Fix * Fix * Remove temporary changes * add uint8 support to where op (#2792) * Improve bert optimization script: (#2712) (1) Move input int64=>int32 conversion to embed layer fusion. (2) Output epsilon attribute for LayerNormalization fusion. * add session creation time cost. (#2798) * ML.NET team needs featurizers within a package (#2789) Add auto ml featurizers to Windows, MacOS as well as to GPU packaging-pipelines. * Initialize max of softmax with lowest of float (#2786) * MLAS: update SGEMM threading parameters (#2808) * add interface to copy batch tensors. (#2807) * add interface to copy batch tensors. * onnxruntime * speed up Windows TRT CI (#2811) * don't run cuda tests if building with tensorrt * remove unnecessary build options for win trt ci * refactor win gpu tensorrt ci yml * --numpy_version=1.17 * update * update * azcopy and cuda path * Update test data (#2356) * Add timeseries imputer transformer featurizer kernel (#2813) Make kernels non-template. Add input constraint for learnt data. Fixup tests. Add two more featurizers along with tests. Tests fail. min_max_scalar_transformer robust_scalar_transformer Fix tests serialized stream by prepending version bytes. Add inputation_marker_transfomer and the test. Fix up float/double type designations. Added label_encoder_transformer along with a test. string_throw case is broken at the momement. Fix labelencodertransfomer_test.cc string_throw case Rename maxabsscalertransformer_test.cc Add MissingDummiesTransformer along with the test. Update manifest. Add TimeSeriesImputerTransformer definition, implementation and tests * Fix memory leak in TRT (#2815) * fix memory leak issue * revert EP_FAIL on enueueV2 * Add manifest missing comma * Run static code analyzer on most of our code (#2817) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * update quantization doc (#2783) * update documentation for quantization script * plus some spell corrections * Filter CPU case for IsFloat16Supported (#2802) * update default optimization level + fix gemm_activation fusion (#2791) * update defualt optimization level + fix gemm_activation fusion * fix typo * add unit test and incorporate review comments * fix test comment * Fix dnnl wheel package name (#2823) * Append '-dnnl' to whl package name when --use_dnnl * Update build.py * Update Ubuntu & TensorRT version in README (#2820) Dockerfile.tensorrt is using nvcr.io/nvidia/tensorrt:19.09-py3 as base Image, update Ubuntu and TensorRT version according to https://docs.nvidia.com/deeplearning/sdk/tensorrt-container-release-notes/rel_19-09.html#rel_19-09 * Merge fixes * Add OneHotEncoder and HashOneHotEncoder kernels. (#2830) Add defs and imlementation for OneHotEncoders, adjuist date_time_transformer kernel and test. Add OneHotEncoder kernel test. Add HashOneHotVectorizerTransformer unit test. This does not link due to multiple definitions of functions that are included into header from a CPP file. * Upgrade gtest to the latest version (#2827) WinML would like to update the googletest submodule. They want some newer features (namely GTEST_SKIP to skip tests programmatically and be able to skip entire fixtures easily) and would need to update the submodule version. However, because the new version of code hit a bug in gcc, even though the bug is already fixed in the latest gcc but we're using gcc 4.8.x and it won't get patched for the bug, so we have to do a compromise, change our code a little bit to make it work. The gcc bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=51213 * Add support for int64_t for topk CPU. Fixes github issue #2806. (#2833) * Ignore allocator type in ExecutionProviders allocator map. Make default initialization of OrtMemoryInfo more clearly invalid. (#2768) * Remove allocator type from the key comparison in ExecutionProviders. Remove usage of DummyArena as it's no longer necessary. * Fix x86 tests where arena allocator is disabled. Make initialization of OrtMemoryInfo clearer by adding Invalid enum value. * Make OrtValueNameIdxMap::MaxIdx more intuitive. * Convert ExternalProject Featurizers into git submodule (#2834) Add git submodule for Featurizer library. Update cmake to build for git submodule. * add domain check for nodes + update documentation (#2831) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Fix Linux CUDA nuget packaging pipeline break * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * update optimization doc for BERT related fusions (#2819) * Add bert related transformers to doc * Add execution provider and comment for bert optimizations * Add comment about accuracy impact of approximation * Fix warnings that cause build to fail * MLAS: enable threading for quantized GEMMs (#2844) * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * remove old winmladapter cpp Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * move sequence implementation into ort lib... still commented out... need to turn back on... * begin sequence implementation * make maps and sequences work * fix broken tests * remove dead code * misc cleanup * CR feedback * User/xianz/winml adapter c api (#2869) * wrapper all existing winml adapter apis with API_IMPL to try catch * Return HR or Throw for WinML adapter APIs if failed * undo macro wrapper for two places * Wrap error macros around ort apis, too. * address CR feedback #2 * add more api throw/return macros * Revert changes no longer needed * revert changes to cxx api * format winml lib.ort and winml adapter * remove static pheonix singleton Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * missing use_dml check in winml_adapter_session (#2930) * --use_dnnl flag was mangled in merge (#2931) * use dml macro not wrapping custom registry code (#2934) * Disable LNK4199 winml_dll to enable cuda builds (#2936) * Disable LNK4199 in winml_dll * linkler->linker * LearningModelSessionAPITestGpu.CreateSessionWithCastToFloat16InModel should return DXGI_ERROR_UNSUPPORTED when FP16 not supported (#2937) * Disable LNK4199 in winml_dll * linkler->linker * Need to return DXGI_ERROR_UNSUPPORTED when Model does not support fp16 * Publish build symbols (#2939) * Publish build symbols * Don't upload PDBs for .exe files * Make x86 build (#2943) * fix last remaining size_t/int64_t warnings->errors (#2948) * TensorString, Sequences and Maps use the first allocator, but should use the cpu default allocator. (#2952) * fix tensor string allcoator * clean up default allocator usage for strings in winml lib/api.ort Co-authored-by: Ryan Lai <ryalai96@gmail.com> * Handle tensor shape of zero (#2954) Co-authored-by: Ryan Lai <ryalai96@gmail.com> * CR feedback (#2970) * CR feedback * fix weird formatting on privacy readme * Add 'All rights reserved.' everywhere * readd all rights reserved to winml_provider_factory.h * remove extra space in comment * remove extra whitespace * fixes post master merge * remove winml from nuget gpu pipeline * set IR VERSION on generated_model in rnn_benchmark (#2972) * Fix slice conformance failures (#2908) Co-authored-by: Adrian Tsai <adtsai@microsoft.com> Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Paul McDaniel <paul_mcdaniel@hotmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Dwayne Robinson <fdwr@hotmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com>
2020-02-05 01:12:19 +00:00
[threading(both)]
[marshaling_behavior(agile)]
[dualapipartition(1)]
runtimeclass LearningModelBinding : Windows.Foundation.Collections.IMapView <String, IInspectable>
{
//! Create a LearningModelBinding from the specified LearningModelSession.
//! During evaluation, the specified adapter will be used to create resources and queue work during execution.
[method_name("CreateFromSession")] LearningModelBinding(LearningModelSession session);
//! Bind a value to the specified feature.
void Bind(String name, IInspectable value);
//! TODO:Docs
[method_name("BindWithProperties")] void Bind(String name, IInspectable value, Windows.Foundation.Collections.IPropertySet props);
//! Remove all bindings.
void Clear();
}
//! \enum TensorKind
//! \brief Defines the list of supported tensor data types.
[contract(MachineLearningContract, 1)]
enum TensorKind
{
//! Supported by ONNX, but should never happen and is invalid for Windows ML.
Undefined = 0,
//! The tensor type is 32bit float.
Float,
//! The tensor type is 8bit unsigned int.
UInt8,
//! The tensor type is 8bit signed int.
Int8,
//! The tensor type is 16bit unsigned int.
UInt16,
//! The tensor type is 16bit signed int.
Int16,
//! The tensor type is 32bit signed int.
Int32,
//! The tensor type is 64bit signed int.
Int64,
//! The tensor type is String.
String,
//! The tensor type is Boolean.
Boolean,
//! The tensor type is 16bit float.
Float16,
//! The tensor type is 64bit float.
Double,
//! The tensor type is 32bit unsigned int.
UInt32,
//! The tensor type is 64bit unsigned int.
UInt64,
//! Supported by ONNX, but is not supported by Windows ML.
Complex64,
//! Supported by ONNX, but is not supported by Windows ML.
Complex128
};
//! \class MapFeatureDescriptor
//! \brief TODO:Docs
[contract(MachineLearningContract, 1)]
INBOX_ONLY([interface_name("Windows.AI.MachineLearning.IMapFeatureDescriptor", 530424bd-a257-436d-9e60-c2981f7cc5c4)])
Merge windowsai (winml layering) into master (#2956) * Initial Commit * Merged PR 3985217: add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc (#2346) add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc and violating our OS layering requirements. We linked against onecoreuap_apiset.lib in VB so we will continue doing this, but I am still unsure why not to link against onecore instead since that is where we ship. However, since Sheil is the owner of this code we will wait to discuss with him before changing anything. * Initial changes for layering * more snipping to get core into ort * update build instructions to include --build_shared_lib (#2358) * update build instructions to include --build_shared_lib * fix line breaks * Task 23998197: add winml_lib_core into onnnxruntime.dll (#2368) * Task 23998197: add winml_lib_core into onnnxruntime.dll * PR feedback build break on perf_test * return proper error when the model path isn't found (#2391) * LearningModelSession is cleaned up to use the adapter, and parts of b… (#2382) this is a big PR. we are going to move it up to layer_dev , which is still a L3 so we are still safe to do work there agile. we are going to move this into the L3 so that ryan can start doing intergration testing. we will pause for a full code review and integration test result prior to going into the L2. >>>> raw comments from previous commits >>> * LearningModelSession is cleaned up to use the adapter, and parts of binding are. * moved everything in the winmladapter made it all nano-com using, WRL to construct objects in the ORT side. base interfaces for everythign for winml to call cleaned up a bunch of winml to use the base interfaces. * more pieces * GetData across the abi. * renamed some namepsace cleaned up OrtValue cleaned up Tensor cleaned up custom ops. everything *but* learnignmodel should be clean * make sure it's building. winml.dll is still a monolith. * model moved over. everything builds clean. step ! * weak ref comment * Layer dev paulm (#2408) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * Layer dev paulm (#2414) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * Layer dev paulm (#2423) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * Layer dev paulm (#2424) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * Layer dev paulm (#2425) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Layer dev paulm (#2426) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * fixed map and sequence when passing stl types across the ABI . found a leak in nvidia driver, but skipped it. all winmlapitests pass now * Moved SessionOptions over to the abi * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * Register WinML TraceLogging provider on Onnxruntime.dll (#2455) * Register WinML TraceLogging provider on Onnxruntime.dll * Add ifdef to make sure trace logging provider has telemetry option when LAYERING_DONE * No need for ifdef for TraceLoggingOptionMicrosoftTelemetry * PR feedback * Move etw registration into lotus environment constructor and deresgister in lotus environment destructor * Brianma/cpuwinml (#2466) * allow building winml cpu without dml. * Brianma/breaks (#2469) * fix some more breaks * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * move dml checks out of winml and into the adapter * better error handling * Brianma/fi (#2470) * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * fix bad merge * Checking in a staging checkpoint point so that Ryan can work with me in parrallel * build break. * Brianma/testfails (#2473) * add missing ir version to dictvectorizer-string.onnx * add missing ir version to relu.onnx * add missing ir version to zipmap*onnx * add IR version to manually generated models * remove an unnecessary ifdef dml * Brianma/windowsai fi (#2475) * update dockerfiles/README (#2336) * Make elementwise op run 4 items per thread (#2335) Description: Describe your changes. Make elementwise op run 4 items per thread unroll for loop to leverage ILP remove unnessary N==0 check inside elementwise GPU kernel Motivation and Context Why is this change required? What problem does it solve? It can improve the performance of GPU elementwise ops. ~2% performance gain on popular NLP bert model. If it fixes an open issue, please link to the issue here. * Add CUDA GatherElements kernel (#2310) * Updates * Update test * Update * Updates * nits * PR feedback * Update * Update * PR feedback * PR comments * Update * Fix build * Fix build * Nits * Fix * Layer Normalization Fusion (#2319) basic layer normalization transform * Add FastGelu Cuda Op for Gelu and Add bias fusion (#2293) * Add FastGelu cuda op * Add AddBiasGelu for experiment * Revert "Add AddBiasGelu for experiment" This reverts commit 5c1ee019858c657e6bb75887265cb85675626e5b. * Add bias * Add unit tests * update comment * update script * fix build error * update coding style * update for CR feedback Enable half2 optimization only when cuda arch >= 7.0 * move _Tanh to common.cuh * implement CPU contrib OP Attention (#2333) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanUnusedInitializers. (#2320) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanupUnusedInitializers. This means initializers that have been replaced during graph optimizations are not left in the GraphProto when we save an optimized model. * Handle edge case where a model has an unused initializer with matching graph input by also removing the graph input. * Use non-const iterators in std::find_if calls to make centos build happy. * Nuget pipeline changes (#2305) 1. refactor the pipeline, remove some duplicated code 2. Move Windows_py_GPU_Wheels job to Win-GPU-CUDA10. We'll deprecated the "Win-GPU" pool 3. Delete cpu-nocontribops-esrp-pipeline.yml and cpu-nocontribops-pipeline.yml 4. In Linux nuget jobs, run "make install" before creating the package. So that extra RPAH info will be removed * Cuda Reverse Sequence Op, maping types of same size using same template function. (#2281) * Set ElementType to String type of node metadata, instead of byte[] (#2348) * Set ElementType to String type of node metadata, instead of byte[] * Fix spacing * Introduce PrimitiveType into a Type System along with an integer constant (#2307) Improve perf by avoiding GetType<T>() calls. Introduce MLTypeCallDispatcher to switch on Input Type. Add Tensor IsType<T>() fast method. * Fix/test dim value of 0 handling in a couple of places (#2337) * Update the CUDA Where implementation broadcasting logic to handle a dim with value of 0. Add unit test Also add unit test for unary op with dim value of 0 * Exclude ngraph from Where test with 0 dim. * Openvino EP R3.1 onnxrt server (#2357) * onnxrt server with OVEP * onnxrt server with OVEP * Update Dockerfile.server.openvino * onnxrt server OVEP fix reviews * onnxrt server OVEP fix reviews * Implement cuda nonzero op. (#2056) Implement cuda nonzero op. * Direct use python numpy array's memory if already contiguous. (#2355) * Direct use python numpy array's memory if already contiguous. This could greatly improve performance for session with large input, like big image 1920x1080 fastrcnn, 30~40% speed up could be achieved. * Add test case enforce contiguous/non-contiguos numpy array as inputs. * Add helper to create output to minimize binary size. (#2365) Add ConstEigenTensorMap typedef so we don't unnecessarily const_cast the const input Tensor. * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS (#2369) * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS * update * Add Tracelogging for profiling (#1639) Enabled only if onnxruntime_ENABLE_INSTRUMENT is ON * test bidaf with nuphar for avx target (#2370) increase nuphar test coverage a bit * Fix a bug in TLS refcount that may destabilized CUDA CI (#2374) * update output size calculation for resize (#2366) * change how output size is calculated for resize op * add tests for ver 10 resize * Extend OneHot CPU kernel to support more types (#2311) * Extend OneHot CPU kernel to support input int64_t, depth int32_t, output float * Skip BERT before the test data fix is picked up * Fix bug with Slice. Need to pass in flattened input dimensions so the initial offset into the input is calculated correctly. (#2372) * Add opset 11 version of Split to CUDA ops (#2376) Organize the CUDA ops definitions so all the opset 10 and 11 parts are together (same setup used for CPU ops) * Layer Norm Fusion Fix (#2379) * layer norm fusion fix * Add input shape check in code and unit tests * Fuse Add + Gelu (#2360) Implement the transformer to fuse add + gelu Implement the accurate kernel * Skip layer norm transform (#2350) * skip layer normalization transformer * Another try to stabilize CUDA CI (#2383) The root cause seems to be failure in CUDA dealloc when tear down. cudaFree return code was ignored before, so should the debug check. * fix BUILD.md typo (#2375) build.py: error: argument --config: invalid choice: 'RelWithDebugInfo' (choose from 'Debug', 'MinSizeRel', 'Release', 'RelWithDebInfo') * Fixed compilation with ngraph (#2388) * Fix reuse logic in allocation planner. (#2393) * Fix reuse logic in allocation planner. * PR comments * Add helpful comments * Don't allow reuse across string tensors. * [NupharEP] Multiple optimizations (#2380) Fuse transpose into MatMul Implement Pow and constant scalar simplification Vectorize ReduceMean Improve symbolic shape inference Minor updates for better debugging in fused function name * Avoid using the default logger in the graph lib and optimizers (#2361) 1. Use the session logger if it is available. 2. Don't disable warning 4100 globally. We should fix the warnings instead of disabling it. * Change CUDA implementation of Transpose to support all fixed size tensor types (#2387) * Change CUDA implementation of Transpose to not use a typed kernel so we can support more types with minimum binary size. Add support for 8, 16, 32 and 64 bit types. Add unit tests. Add method so the implementation can be called directly (will be used by CUDA Scan very soon). * Disable TensorRT for MLFloat16 and int8 unit tests. * Address PR comment and add support for calling cublas implementation if type is mlfloat16. * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. (#2398) * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. * [NupharEP] force some low/zero cost ops to be inlined (#2409) * fix cross compile bug (#2415) * Minor optimization: if a node has already been placed, there's no need to find a kernel for it. (#2417) * Add Reshape Fusion (#2395) * Add reshape fusion * Add some comments * update comments * update comment format * update according to feedback * update for recent logger change * fix build error * (1) Support both input and output edges in find path in graphutils (2) Add a test case of only one constant initializer of Concat input. (3) Refactor ReshapeFusion class to allow add more subgraph fusion in the future. * fix error * (1) loose constraint on initializer: non constant is allowed for reshape fusion. (2) Change versions type to vector. (3) Add logging. (4) Return false when multiple output edges matched in FindPath. Add comments. * only allow one direction (input or output) in FindPath * [NupharEP] Update notebook and docker image (#2416) Add BERT squad in Nuphar tutorial Enhance speed comparsion readability * Fix the issue in matmul_add_fusion (#2407) Fix the issue in matmul_add_fusion If Muatmul + Add has shape [K] * [K, N], reset it to [1, K] * [K, N] will make the output shape to [1, N] will also requires a reshape on the output. Fix: just remove the shape reset to not fuse it. Add a negative test case for matmul+add fusion * feat(treeregressor): Update TreeEnsembleRegressor for type support (#2389) Updates the `TreeEnsembleRegressor` to allow for `double`, `float`, `int64`, and `int32` inputs to match the upstream specification. Signed-off-by: Nick Groszewski <nicholas.groszewski@capitalone.com> * onnxrt server documentation update (#2396) * Added support for Pad-2 operator in OpenVINO-EP (#2405) * Add CUDA If operator. (#2377) * Add CUDA If operator. Uses CPU operator for implementation. By adding a CUDA version the inputs/outputs (with the exception of the 'cond' input) stay on GPU, and no other logic is required to avoid a copy to CPU across the control flow node. * Improved documentation for onnxruntime::utils::SwapByteOrderCopy(), added precondition check. * Fix the type constraints on CUDA If operator to exclude strings. (#2431) * add Im2col<uint8_t> (#2438) * Adjust codegen vectorization width from target (#2439) * Adjust codegen vectorization width from target * Add CUDA Scan operator. (#2403) * Add Scan CUDA op. Uses CPU implementation for logic. Added some device specific functors for handling when data needs to be manipulated on a different device. Added ability to override the materialization logic in the OrtValue slicer so DML can plugin their handling. * Fix Windows GPU C API packaging pipeline failure (#2440) Fix Windows GPU C API packaging pipeline failure (#2440) * Correctly handle implicit inputs for fused nodes (#2390) * Correctly handle implicit inputs for fused nodes Previously, nuphar's partitioning function didn't include node's implicit inputs into the inputs list of MetaDef, and hence a crash was triggered in the onnx graph checker. This commit fixed the issue. Furthermore, it also fixed a related issue where we didn't add implicit inputs into graph_inputs_excluding_initializers_ in Graph::SetGraphInputsOutputs. the issue was that graph_inputs_including_initializers_ populated by SetInputs (e.g. called by FunctionImpl::FunctionImpl) may contain implicit inputs which were not of any node's initializers in the graph. Because they were not part of any initializers, these implicit inputs couldn't be visited by going through all nodes' inputs. Consequently, they would *not* be added into graph_inputs_excluding_initializers_. We fixed the issue by first copying the populated graph_inputs_including_initializers_ into graph_inputs_excluding_initalizers_, which then had both initializers and non-initializers as its initial content. Later, we erase initializers from the list. In this way, we can ensure all implicit inputs to remain in graph_inputs_excluding_initializers_. * refined comments and fixed duplicates Address CR by revisiting comments in terms of implicit inputs Also fixed an issue by skipping duplicates while copying inputs from graph_inputs_including_initializers_. * address CR explain why we need to collect nodes' implicit inputs * don't rely on pointer values for iterating std::set Previously, openvino relied on iterating a set of NodeArg pointers to construct inputs and outputs for a fused graph. It could cause non-determinism. The reason was that although iterating std::set by itself is stable, pointer values of NodeArgs may vary. Consequently, we could end up visiting the set's elements in different orders for different runs for the same test, which resulted in constructing inputs (and outputs) with different orders to the fused graph. For example, for the same test, we may have inputs [A, B] in some runs but inputs[B, A] in others. Let's use std::string as the key type to avoid such nondeterminism. This commit also added implicit inputs into meta->inputs while returning the capability from the openvino provider. * Fixed another latent issue in openvino's GetCapability function The issue was that we couldn't simply erase fused_inputs and fused_outputs while iterating the nodes. For example, an output NodeArg may have multiple uses, and it's wrong if we erase it from fused_outputs when we encounter only one of its uses as input. * Remove DeviceAllocatorRegistry class (#2451) Remove DeviceAllocatorRegistry class * CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library. * Parallel Gelu with ParallelFor (#2399) Parallel Gelu to get better performance for Gelu * Clean up build.py (#2446) * Pull the latest image before running docker build * Fuse SkipLayerNorm with Bias (#2453) Fuse SkipLayerNorm with Bias * Allow more than one invocation of CreateEnv in the same process. (#2467) * Allow more than one invocation of CreateEnv in the same process. * Fix centos build * Symbolic shape inference improvements: (#2460) * Symbolic shape inference improvements: - add a mode to guess unknown ops' output rank - add support for GatherND - add support for If - fix a bug in get_int_values when then tensor rank > 1D, by treating it as no sympy data - add symbol to literal merge when ONNX silently merges dims - fix a bug in Concat when input dim is 0 - fix a bug in ConstantOfShape that computed dim is not updated - add support for dynamic shape in ConstantOfShape - fix a bug in Loop output shape that loop iterator dim is not inserted at dim 0 - add support for dynamic padding in Pad - add support for dynamic shape in Reshape - add support for Resize with opset > 10, by treating output dims as dynamic - fix a bug in Slice when starts/ends are dynamic - restrict input model to opset 7 and above - make output model optional to avoid disk write when testing Run model tests for symbolic shape inference Reduce 2GB docker image size of nuphar * add additional test data set for nuget pipeline (#2448) * add SAS token to download internal test data for nuget pipeline * update azure endpoint * fix keyvault download step * fix variable declaration for secret group * fix indentation * fix yaml syntax for variables * fix setting secrets for script * fix env synctax * Fix macos pipeline * attempt to add secrets to windows download data * fix mac and win data download * fix windows data download * update test data set url and location * Revert "Brianma/windowsai fi (#2475)" This reverts commit 5780b864a15513fda4eadbfc2b5345fefe70b5ec. * Add scenario tests (#2457) * Add scenario tests * Remove TODO from model license * Add winml_api test dependency * fix model load test. fi from master changed the constructor (#2483) * make api tests all pass (#2486) * fix bad merge * fix bad model merge * Layer dev paulm (#2492) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * Rename ambiguous header (#2489) * fix one more missing IR version model (#2500) * add missing IR version to 4 more models used by scenario tests (#2501) * Add CLI parameters to test runner, build WinML in ARM and x86 CI (#2479) * Support test parameters through CLI arguments * Add WinML do Windows x86/ARM CI builds * Code style fixes * Update googletest Remove GPUTEST macros everywhere now that GTEST_SKIP is supported * Refactor main.cpp * Build scenario tests without DML * Link scenario tests to DML when it's enabled (#2502) * Layer dev release pipeline (#2488) Adds winml binaries to existing cpu nuget package, and creates new gpu dml nuget package with winml binaries and DML EP. * Layer dev paulm (#2506) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * Remove usage of IOBinding in WinML and use C_API Run method (#2504) * remove usage of iobinding * Change data structure to use vector of Ort::Values * Polish bind input / output * Use C APIrun method * Update providers on evaluate getresults * Remove run and IObinding interface from WinMLAdapter * Remove use of IObinding * bind unbound outputs code moved to learningmodelbinding * clean up unneeded istensor adapter function * Fix comment * Check if session is closed before binding and clearing * PR feedback * Layer dev paulm (#2507) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * Make tests dependend on winml_dll (#2509) * add dml binaries to DirectML package and be more explicit about condition variables (#2520) * re-enable warnings for winml builds and fix the warnings that were hiding (#2526) * turn devmode back on for winml builds * fix some warnings. include protobuf in a way that disables some warnings * undo protobufhelpers changes and just ignore 4100 errors in pb code * attempt to isolate protobufhelpers errors * add template specialization for getting tensor proto data * Layer dev paulm (#2533) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * Spawn child process to run DeviceLostRecovery scenario test (#2530) * Spawn child process to run DeviceLostRecovery scenario test * Layer dev paulm (#2536) ori said yes * add missing namespace to winml_trace_logging_provider in lotusenvironment.h (#2542) * Handle exception thrown from all apis in WinMLAdapter (#2539) * various changes to unblock windowsai ADO build * Fix custom ops scenario tests (#2562) * Do not shutdown protobuf after ort environment gets destroyed. Lazy load lotus environment first time it is needed * comment typo * pr comment about calling phoenix singleton * Make lotus_environment static in winmladapter * Layer dev paulm (#2567) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * making windowsAI azure dev ops work. * code review comments. * revert changes * Cmake and preprocessor fixes that where uncovered by building on agents without DML available via SDK * Layer dev dml delayload (#2580) * Brianma/cpu (#2583) * don't include dml stuff in cpu builds * tests that link the image lib also need the telemetry lib now * Throw Winml_err_invalid_binding if binding gpu resource on cpu device (#2589) * Throw Winml_err_invalid_binding if binding gpu resource on cpu device * PR comments. No need to query executionprovider for is gpu device * User/xianz/ortthrow (#2596) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * User/xianz/ortthrow (#2599) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * add status error message * Remove uwp onsuspending winrt call because logruntimeperf is getting removed (#2630) * User/xianz/dedup telemetry (#2631) * investigate duplication of telemetry in winml and ort * remove winml telemetry events * telemetry executionProviderEvent * remove unneccessary file and refactor code little bit * Revert back TelemetryEvent, which send up ETW event. * merge changes from layer_dev to windowsai (#2638) * Remove underscore from googletest names (#2616) * Fix leaking memory allocator Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24278761 and https://microsoft.visualstudio.com/OS/_workitems/edit/24330198 * Explicitly initialize Ort::Value with nullptr * Cache WinML adapter * bad merge * define private version of dxcore enum that is added in 19H1 SDK. (#2654) * add comment for explaning private definition of dxcore d3d feature level ennum value. (#2672) * do not package directml.pdb for redist packages. (#2676) * Fix leaking operator registry (#2645) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24354916 * User/orilevari/windowsai master merge (#2674) merge resolutions included pulling in telemetry logic that was merged to master and not windowsai and dereferencing InferenceSession::sessionstate now that it is a unique pointer * Delete Ort Allocator in LearningModelBinding (#2653) * Delete OrtAllocator in LearningModelBinding * PR comments to make Ort::Allocator a smart pointer * Small comment change * PR feedback to clean up code * PR feedback on move semantics * Clean up std::move * Fix memory leaks (#2679) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24356109, https://microsoft.visualstudio.com/OS/_workitems/edit/24388361 and https://microsoft.visualstudio.com/OS/_workitems/edit/24388596 * various changes to properly organize and skip GPU tests. For now for No DML builds we will not run GPU tests at all. In the future we should adapt the tests to expect the appropiate errors. (#2695) * Windowsai without fi (#2701) * Disable Attention fusion tests when DISABLE_CONTRIB_OPS is defined (#2529) * Setup java ci (#2528) * Add provision in ORT for session options to be parsed when available via model file (#2449) * Initial commit * Fix gitmodules * Nits * Nits * Updates * Update * More changes * Updates * Update * Some updates * More changes * Update * Update * Merge * Update * Updates * More changes * Update * Fix nits * Updates * Fix warning * Fix build * Add comment * PR feedback * PR feedback * Updates * Updates * Update * More changes * Fix build break * Comment test for now * Updates * Updates * PR feedback * Updates * Nits * Add tests * Fix build * Fix build * Fix build * Fix build break * Fix build * Nits * PR feedback * More change * Expose GetSessionOptions in pybind logic and add unit test for python * Fix build * PR feedback * PR feedback * Revert "Disable thread pool creation when enabled OpenMP (#2485)" (#2535) This reverts commit 7c7d5a149c9ed52eec67304bae5c4b132166a8a1. * Add dynamic shape support in TensorRT execution provider (#2450) * remove onnx-tensorrt submodule * add new onnx-tensorrt submodule (experiment) for trt6 * update engine build for trt6 * update compile and compute for tensorrt6.0 * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * switch to onnx-tensorrt master for TensorRT6' * Update tensorrt_execution_provider.cc * Handle dynamic batch size and add memcpy in TensorRT EP * update test cases * Update tensorrt_execution_provider.cc * update onnx-tensorrt submodule * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.ubuntu_tensorrt * Update run_dockerbuild.sh * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update concat_op_test.cc * Update tensorrt_execution_provider.cc * Upgrade TensorRT to version 6.0.1.5 * Update onnxruntime_providers.cmake * Update CMakeLists.txt * Update reduction_ops_test.cc * Update install_ubuntu.sh * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.tensorrt * Update BUILD.md * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update onnxruntime_providers.cmake * Update install_ubuntu.sh * Update install_ubuntu.sh * Update gemm_test.cc * Update gather_op_test.cc * Update CMakeLists.txt * Removed submodule * update onnx-tensorrt submodule * update header file * Removed submodule * add submodule onnx-tensorrt kevin's branch shape-test' * add debugging code * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * merge master * Removed submodule * update onnx-tensorrt submodule * add more changes for dynamic shapes * Update tensorrt_execution_provider.cc * update for dynamic shape * update dynamic shape processing * fix logger issue * remove submodule onnx-tensorrt * add submodule onnx-tensorrt * add env variable min_subgraph_size * remove redundency * update document * use onnxruntime::make_unique * fix multi-run issue * remove some tests to save CI build time * Add dynamic shape test * Update TensorRT-ExecutionProvider.md * Add example of running Faster R-CNN model on TensorRT EP * Add more details on env variables * update environment variables * Update tensorrt_basic_test.cc * Update model tests * Update tensor_op_test.cc * remove --use_full_protobuf * Update build.py * User/xianz/telemetry (#2458) * enabme telemetry * enable telemetry * set enable telemetry as default * for debugging * remove log and set disable telemetry as default back * delete private file while testing * resolve comment: mainly add license header, rename macro and update docs * rewording in privacy.md * Fix integer overflow in cuda NonMaxSuppression implementation (#2540) * add test case that should pass but fail * fix nms * extract int_max_output_boxes_per_class * Introduce container type runtime checks and other improvements (#2522) Rework TensorSeq in a manner consistent with Tensor and SparseTensor in terms of type system setup. Reduce templating. Introduce helpers to ensure the same data type. Make OrtValue __dtor not virtual. Introduce ContainerChecker * Fix C API tests for centos and mac (#2544) * change c++14 to c++11 * add ld lib path for centos * enable csharp tests on macos * fix C API test on MacOS + fix manylinux dotnet install * fix manylinux dotnet install * fix lib link * Add back executable bit to build.py * Fix a bug handling negative begin pad values in Pad op (#2550) * Fix bug in Pad op * Update * DNNL CMAKE update (#2548) * Fix android build (#2558) * Update win-x86-ci.yml (#2557) Fix build pipeline break * Re-enable Windows C# tests (#2564) * disable onnx_test_runner -x invocations for dnnl (#2568) * Allow sequence length to be symbolic (#2559) * setup java ci mac (#2570) * make layernorm fusion to support opset 11 (#2545) * Fix a warning found in the latest VS release * Add more check on SkipLayerNorm and BiasGelu fusion (#2574) * Fix file not found error during docker build. (#2569) * Add ConvTranspose1D (#2578) * Ryanunderhill/packagename test (#2582) * [Nuphar EP] fixes for some object detection models (#2581) Update notebook tutorial with multi-threaded int8 GEMM from #2517 * EmbedLayerNormalization Fusion Improvement (#2553) Embedding layer norm fusion improvements - add more checks * Update version (#2584) * Temporarily exclude vgg19 test from Python backend test 1. temporarily exclude vgg19 test which comsumes too much memory, run out of memory on Upsquared device. Single test pass for vgg19, need furture investigation (#2588) 2. Update docker file to decrease the docker image size * Update docs for Android NNAPI EP (#2586) * Fix lto bug for protobuf and ubuntu * add path to build dir before test run (#2590) * Add missig env variables for mac pipeline test (#2595) * Fixed an issue in updating realized dims (#2597) when we update realized dims for scan's output, the sliced axis also needs to be inclusive, i.e. we should check with "dim >= insert_inclusive_axis", because the offset in the symbols are based on Scan sugraph. Otherwise, we would end up with shape mismatch later. * Java API for onnxruntime (#2215) * Add support for opset 11 in reshape fusion (#2592) Support opset verion 11 in reshape fusion * Rename automl python tools folder to featurizer_ops. (#2593) * Support opset 11 subgraph of Squad model in Embed Layer Normalization (#2605) Support opset 11 Squad model that is exported from PyTorch nightly. The embed layer uses Range op which is missed in the transformer. * symbolic shape inference: fix warnings in GPT-2 model (#2608) And revise nuphar perf test on BERT squad * Dump subgraph ID and fused graph ID (#2607) * Dump subgraph ID and fused graph ID Dump subgraph ID and fused graph ID for better debugging * Remove local static fused_count added a field global_fused_count_ to NupharExecutionProvider class * EmbedLayerNormalization Fusion For Dynamic Squad Model Opset 10 (#2613) Support subgraph of SQuAD model exported from pytorch with dynamic input axes * Allow providers to be set for InferenceSession at construction (#2606) * Remove unnecessary parameter in some places in GatherElements implementation (#2612) * Remove unnecessary parameter in some places * Update * Update * Make sure fenced tensor could not reuse other tensor. (#2561) Fix random error caused by this. * Improve Embed Layer Norm Fusion for SQuAD with static input shape (#2621) * fix float16 comparison in initializer (#2629) * epsilon attribute for layernormalization fusion (#2639) * removed unnecessary batch file and fix path (#2640) * Add shape inference to ConvTransposeWithDynamicPads schema (#2632) * Improve cuda expand() opeator's performance. (#2624) * Cuda pad optimize when no padding is needed. (#2625) * Shortcut cuda Pad() when no padding is needed. * Optimize cuda scatter() on 2D compatible. (#2628) * Optimize cuda scatter() on 2D compatible. * Add some comments. * fix build error for ARM (#2648) * Improve performance of resize() in Nearest mode (#2626) Special treatment for 2D, check same size as input image. And in 2d kernel, template use_expolation. * Fix memory exception in Layer Norm Fusion (#2644) * Windows CI changes(#2650) * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Revert "Windowsai without fi (#2701)" This reverts commit 285d4c85ff5c4e265f963208170304ef3461e684. * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Deref unique pointer for session_state * send shutdown event when dll is unloaded and EvaluationStop, SessionC… (#2704) * send shutdown event when dll is unloaded and EvaluationStop, SessionCreationStart Events. * Add EvalutationStart Event * add comment * use correct type for for loop (#2755) * ARM CI (#2759) * Set ARM agent pool * Set CMake generator to VS 2019 in ARM * Use system-wide CMake instead of custom version Our custom version is too old for VS 2019 * Use DML and build shared lib in ARM CI * Restore nuget packages in ARM CI * Disable DML * Refactor ARM debug/release builds * Use system packaged Python version * Remove hardcoded Python path * Downgrade Python to 3.7 for build * Remove explicit CMake path * Fix invalid JSON in cgmanifest.json (#2760) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * Remove usage of WHOLEARCHIVE in WinML CMake and add WinMLAdapterFactory (#2726) * Remove usage of WHOLEARCHIVE in WinMLAdapter CMake and add WinMLAdapterFactory * PR feedback, no need for dll(export) since using def file * PR comments * Small comment in gen_def.py * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * Move winml_provider_factory.h to proper location (#2801) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Filter CPU case for IsFloat16Supported (#2802) * Merge fixes * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * Fix warnings that cause build to fail * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * Make DML include path global (#2882) * Make DML include path global * Add generated cppwinrt headers to winml_lib_common * Integrate changes to WindowsAI to make ADO Build (#2886) * Revert "CMake cross-generator fixes (#2790)" This reverts commit dbe7d97fa1ab155f1309bced87199527e8f35bd2. * add additional suppress warning in onnx_proto * ignore /wd4996 warning * DML execution provider fixes * Revert "Revert "CMake cross-generator fixes (#2790)"" This reverts commit 1ae7b4bcbc02edc881ad28685da98e095dfceb17. * Update func signature of custom op function overloads * common devicehelpers fixes * Add pch.h for winml_lib_common * re-add winml_lib_common_dir/inc to include path for winml_adapter * User/orilevari/dml redist shared folder (#2890) * move dml nuget package directory up one level to make it shared between build flavors * Merge conflict fix * Revert "Merge conflict fix" This reverts commit 142fa72cf9ce4344ad717b50b7ea2b8582aadc7c. * Revert "Merge remote-tracking branch 'origin/master' into windowsai" This reverts commit 6e2126d46e5e5f564d65da37dd4f70c93dd81165, reversing changes made to b3f5583dc9249834b947c8ea905f6a98060d5bd6. * Make winml_test_common free of test macros (#2902) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Don't generate files outside the build root (#2914) * Don't generate files outside the build root * Add onnxruntime_EXTERNAL_DEPENDENCIES to WinML * Add DML depedency on RESTORE_PACKAGES * User/orilevari/fix yaml merge bugs (#2918) * Add winml test source parameter into cmake function (#2919) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Add arguments to unittest cmake functions * remove comment * Revert "Revert "Merge remote-tracking branch 'origin/master' into windowsai"" This reverts commit ade5abe72a4234fdbc3623093c61c02c6b0bdc26. * Fix breaks from merge with ORT master * Brianma/linux (#2917) * don't include windows.h in cross-plat header * add default case for switch statement * signed/unsigned mismatch fix Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> * User/sheilk/winml adapter c api (#2891) * Create winml adapter c api * fix build * make it build * move adapter into onnxruntime core/session * entry point not exported * minor changes * make model metadata work * make tests pass * implement all the model reflection apis on the adapter c abi * update the new ort interface to create a lotus ennvironment with a logging sink * start adding ort env * move all winml code into adapter folder/lib to isolate it * ensure a single logging manager at a time * start refactoring session * refactor session creation interface * add cpu and dml session option methods to adapter * finish session init * stub out interfaces in ort lib to perform similar mechanics of iinference session * enable profiling, and enable schema override * update session register graph transformers * turn back on custom registry for custom ops * Add sync api * add last c api stubs * should build... but all feature values are broken since this is in flight to moving all implementation details into ivalue * remove ep adapter header * Implement DML execution provider functions from adapter (#2846) * Implement DML execution provider functions from adapter * Use functions in OnnxruntimeEngine.cpp * make map/sequence type_infos freeable, and start implementing ivalue * make it build again * implement value methods * implement remaining methods * remove com adapter abi * check dml session * cache the allocator on ivalue * check if resource is cpu/gpu when access its mutable data * update tensor * mismatched parentheses * fix tensor base and binding obj * it evaluates tensors! sometimes... * minor fixes * enable gpu evals * wrapper all existing winml adapter apis with API_IMPL to try catch (#2854) * update winml... tensor strings are broken, need to template tensorbase to do different things for strings * make tensor strings work with 2 copies in/2 copies out * Fix tensor string and allocator bug * make maps work again... needs some fixes still * Make it build! * enable map inputs * map outputs * unbound outputs for sequences and maps * User/xianz/merge windowsai (#2883) * Packaging pipeline changes for VS 2019 (#2711) * Tiny fix to codegen * Simplify cache implementation and avoid static variables that may carry over between models * Extend DML kernels (#2641) * Additional DML operators * Check unsupported attributes and inputs * Address PR comments * Add kernel capability function used for partitioning, and re-enable stride-based int64 support based on value range * Fix test failures * Build fix * PR comments * Update Nuphar tutorial notebook (#2721) 1. Reflect int8 GEMV improvements for multi-threading from #2696 2. Add notes on multi-threading control using OpenMP 3. Add samples of running multi-isa AOT, and show int8 GEMM differences between AVX and AVX2 4. Add rnn_benchmark example to resolve #1993 * Add schema for new Qops (#2611) * Add schema for new Qops * adding shape inference + qlinearaveragepool * plus review comments * plus review comments * updates per review comments * plus review comments * [server] Add supposed for model_name and model_version as cli parameter (#2708) * remove 64bit warning message from python validation. (#2727) * MLAS: ARM64 build fix (#2734) fix bad usage of vreinterpret to cast vector element types * Fix broken python docs links (#2740) * Fix build on Mac OS (#2731) mac os ld doesn't support --while-archive, correct option is -all_load * fix ngraph wheel (#2737) * fix ngraph wheel 1.1.0 onnxruntime_ngraph wheel doesn't work * remove libdnnl.so in nGraph Libs * make it easy to compare * Split onnxruntime server to a separated folder (#2744) * Fix build for Python 3.8 (#2747) * Fix build for Python 3.8 * Update protobuf to 3.11.2 (#1928) Update protobuf to 3.11.2 (#1928) * Change default optimization level to All (from Basic) (#2745) * change default optimization level to All (from Basic) * fix test * fix c# test * Update numpy to 1.18 (#2758) * Update numpy to 1.18 * Pipeline changes for python 3.8 (#2753) 1. Pipeline changes for python 3.8 2. Fix a regression in setup.py which was just introduced in the previous commit. Please notice, we still haven't made python 3.8 + Windows + CUDA work. * Add basic stacktrace output for posix debug builds. (#2749) * [NupharEP] fix a race condition when multiple sessions running different models concurrently (#2772) * Revert "Change default optimization level to All (from Basic) (#2745)" This reverts commit 56bb503c2f26474b6613bcb2a198691a11dcef38. * Fix typo in error message (#2736) * Rename MKL-DNN to DNNL to fix broken link (#2730) * Fix nightly build version number issue * Pass BUILD_BUILDNUMBER to linux docker * Disable featurizers in python packages * Import more featurizers (#2781) Make kernels non-template. Add input constraint for learnt data. Add min_max_scalar_transformer, robust_scalar_transformer, inputation_marker_transfomer, label_encoder_transformer, missing_dummies_transformer along with tests. Advance Featurizers library commit. * Implement a more stable softmax (#2715) * Implement a more stable SoftMax e^x is represented as infinity if x is large enough, like 100.f. Infinity divided by Infinity is a NAN. Thus, softmax gets a NAN if one or more item are large enough. A math transform as below is leveraged to get a stable softmax: e^xi/(e^x1 + ...e^xn) = e^(xi - max) / (e^(x1 - max) + ... + e^(xn - max)) And for convenience, force max to 0.f if all xi are negative * Contributing: Fix a typo (#2784) * ACL EP GEMM improvements (#2780) When it is posible we use a fully connected layer instead of the gemm implementation. This will let the library use the best implementation based on the input data. * ACL EP convolution improvements (#2774) Added the optimized implementation for depthwise convolution for both ACL v19.02 and ACL 19.05. Also the pointwise convolution seems to be more optimal in the CPU implementation so we opted for that instead. * Add script for release Nuget validation (#2719) * Initial commit * Nits * Disable a test temporarily * Change working directory * Test * Add download python step * Test update * More changes * Fix space issue * Fix * Verify nuget signing * Fix * Spaces * PR feedback * Nit * Fix * Fix * Remove temporary changes * add uint8 support to where op (#2792) * Improve bert optimization script: (#2712) (1) Move input int64=>int32 conversion to embed layer fusion. (2) Output epsilon attribute for LayerNormalization fusion. * add session creation time cost. (#2798) * ML.NET team needs featurizers within a package (#2789) Add auto ml featurizers to Windows, MacOS as well as to GPU packaging-pipelines. * Initialize max of softmax with lowest of float (#2786) * MLAS: update SGEMM threading parameters (#2808) * add interface to copy batch tensors. (#2807) * add interface to copy batch tensors. * onnxruntime * speed up Windows TRT CI (#2811) * don't run cuda tests if building with tensorrt * remove unnecessary build options for win trt ci * refactor win gpu tensorrt ci yml * --numpy_version=1.17 * update * update * azcopy and cuda path * Update test data (#2356) * Add timeseries imputer transformer featurizer kernel (#2813) Make kernels non-template. Add input constraint for learnt data. Fixup tests. Add two more featurizers along with tests. Tests fail. min_max_scalar_transformer robust_scalar_transformer Fix tests serialized stream by prepending version bytes. Add inputation_marker_transfomer and the test. Fix up float/double type designations. Added label_encoder_transformer along with a test. string_throw case is broken at the momement. Fix labelencodertransfomer_test.cc string_throw case Rename maxabsscalertransformer_test.cc Add MissingDummiesTransformer along with the test. Update manifest. Add TimeSeriesImputerTransformer definition, implementation and tests * Fix memory leak in TRT (#2815) * fix memory leak issue * revert EP_FAIL on enueueV2 * Add manifest missing comma * Run static code analyzer on most of our code (#2817) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * update quantization doc (#2783) * update documentation for quantization script * plus some spell corrections * Filter CPU case for IsFloat16Supported (#2802) * update default optimization level + fix gemm_activation fusion (#2791) * update defualt optimization level + fix gemm_activation fusion * fix typo * add unit test and incorporate review comments * fix test comment * Fix dnnl wheel package name (#2823) * Append '-dnnl' to whl package name when --use_dnnl * Update build.py * Update Ubuntu & TensorRT version in README (#2820) Dockerfile.tensorrt is using nvcr.io/nvidia/tensorrt:19.09-py3 as base Image, update Ubuntu and TensorRT version according to https://docs.nvidia.com/deeplearning/sdk/tensorrt-container-release-notes/rel_19-09.html#rel_19-09 * Merge fixes * Add OneHotEncoder and HashOneHotEncoder kernels. (#2830) Add defs and imlementation for OneHotEncoders, adjuist date_time_transformer kernel and test. Add OneHotEncoder kernel test. Add HashOneHotVectorizerTransformer unit test. This does not link due to multiple definitions of functions that are included into header from a CPP file. * Upgrade gtest to the latest version (#2827) WinML would like to update the googletest submodule. They want some newer features (namely GTEST_SKIP to skip tests programmatically and be able to skip entire fixtures easily) and would need to update the submodule version. However, because the new version of code hit a bug in gcc, even though the bug is already fixed in the latest gcc but we're using gcc 4.8.x and it won't get patched for the bug, so we have to do a compromise, change our code a little bit to make it work. The gcc bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=51213 * Add support for int64_t for topk CPU. Fixes github issue #2806. (#2833) * Ignore allocator type in ExecutionProviders allocator map. Make default initialization of OrtMemoryInfo more clearly invalid. (#2768) * Remove allocator type from the key comparison in ExecutionProviders. Remove usage of DummyArena as it's no longer necessary. * Fix x86 tests where arena allocator is disabled. Make initialization of OrtMemoryInfo clearer by adding Invalid enum value. * Make OrtValueNameIdxMap::MaxIdx more intuitive. * Convert ExternalProject Featurizers into git submodule (#2834) Add git submodule for Featurizer library. Update cmake to build for git submodule. * add domain check for nodes + update documentation (#2831) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Fix Linux CUDA nuget packaging pipeline break * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * update optimization doc for BERT related fusions (#2819) * Add bert related transformers to doc * Add execution provider and comment for bert optimizations * Add comment about accuracy impact of approximation * Fix warnings that cause build to fail * MLAS: enable threading for quantized GEMMs (#2844) * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * remove old winmladapter cpp Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * move sequence implementation into ort lib... still commented out... need to turn back on... * begin sequence implementation * make maps and sequences work * fix broken tests * remove dead code * misc cleanup * CR feedback * User/xianz/winml adapter c api (#2869) * wrapper all existing winml adapter apis with API_IMPL to try catch * Return HR or Throw for WinML adapter APIs if failed * undo macro wrapper for two places * Wrap error macros around ort apis, too. * address CR feedback #2 * add more api throw/return macros * Revert changes no longer needed * revert changes to cxx api * format winml lib.ort and winml adapter * remove static pheonix singleton Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * missing use_dml check in winml_adapter_session (#2930) * --use_dnnl flag was mangled in merge (#2931) * use dml macro not wrapping custom registry code (#2934) * Disable LNK4199 winml_dll to enable cuda builds (#2936) * Disable LNK4199 in winml_dll * linkler->linker * LearningModelSessionAPITestGpu.CreateSessionWithCastToFloat16InModel should return DXGI_ERROR_UNSUPPORTED when FP16 not supported (#2937) * Disable LNK4199 in winml_dll * linkler->linker * Need to return DXGI_ERROR_UNSUPPORTED when Model does not support fp16 * Publish build symbols (#2939) * Publish build symbols * Don't upload PDBs for .exe files * Make x86 build (#2943) * fix last remaining size_t/int64_t warnings->errors (#2948) * TensorString, Sequences and Maps use the first allocator, but should use the cpu default allocator. (#2952) * fix tensor string allcoator * clean up default allocator usage for strings in winml lib/api.ort Co-authored-by: Ryan Lai <ryalai96@gmail.com> * Handle tensor shape of zero (#2954) Co-authored-by: Ryan Lai <ryalai96@gmail.com> * CR feedback (#2970) * CR feedback * fix weird formatting on privacy readme * Add 'All rights reserved.' everywhere * readd all rights reserved to winml_provider_factory.h * remove extra space in comment * remove extra whitespace * fixes post master merge * remove winml from nuget gpu pipeline * set IR VERSION on generated_model in rnn_benchmark (#2972) * Fix slice conformance failures (#2908) Co-authored-by: Adrian Tsai <adtsai@microsoft.com> Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Paul McDaniel <paul_mcdaniel@hotmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Dwayne Robinson <fdwr@hotmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com>
2020-02-05 01:12:19 +00:00
[marshaling_behavior(agile)]
[dualapipartition(1)]
runtimeclass MapFeatureDescriptor : ILearningModelFeatureDescriptor
{
//! Returns the data type of the map's key.
TensorKind KeyKind{ get; };
//! Returns the properties of the map's value.
ILearningModelFeatureDescriptor ValueDescriptor{ get; };
}
//! \class SequenceFeatureDescriptor
//! \brief TODO:Docs
[contract(MachineLearningContract, 1)]
INBOX_ONLY([interface_name("Windows.AI.MachineLearning.ISequenceFeatureDescriptor", 84f6945a-562b-4d62-a851-739aced96668)])
Merge windowsai (winml layering) into master (#2956) * Initial Commit * Merged PR 3985217: add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc (#2346) add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc and violating our OS layering requirements. We linked against onecoreuap_apiset.lib in VB so we will continue doing this, but I am still unsure why not to link against onecore instead since that is where we ship. However, since Sheil is the owner of this code we will wait to discuss with him before changing anything. * Initial changes for layering * more snipping to get core into ort * update build instructions to include --build_shared_lib (#2358) * update build instructions to include --build_shared_lib * fix line breaks * Task 23998197: add winml_lib_core into onnnxruntime.dll (#2368) * Task 23998197: add winml_lib_core into onnnxruntime.dll * PR feedback build break on perf_test * return proper error when the model path isn't found (#2391) * LearningModelSession is cleaned up to use the adapter, and parts of b… (#2382) this is a big PR. we are going to move it up to layer_dev , which is still a L3 so we are still safe to do work there agile. we are going to move this into the L3 so that ryan can start doing intergration testing. we will pause for a full code review and integration test result prior to going into the L2. >>>> raw comments from previous commits >>> * LearningModelSession is cleaned up to use the adapter, and parts of binding are. * moved everything in the winmladapter made it all nano-com using, WRL to construct objects in the ORT side. base interfaces for everythign for winml to call cleaned up a bunch of winml to use the base interfaces. * more pieces * GetData across the abi. * renamed some namepsace cleaned up OrtValue cleaned up Tensor cleaned up custom ops. everything *but* learnignmodel should be clean * make sure it's building. winml.dll is still a monolith. * model moved over. everything builds clean. step ! * weak ref comment * Layer dev paulm (#2408) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * Layer dev paulm (#2414) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * Layer dev paulm (#2423) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * Layer dev paulm (#2424) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * Layer dev paulm (#2425) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Layer dev paulm (#2426) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * fixed map and sequence when passing stl types across the ABI . found a leak in nvidia driver, but skipped it. all winmlapitests pass now * Moved SessionOptions over to the abi * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * Register WinML TraceLogging provider on Onnxruntime.dll (#2455) * Register WinML TraceLogging provider on Onnxruntime.dll * Add ifdef to make sure trace logging provider has telemetry option when LAYERING_DONE * No need for ifdef for TraceLoggingOptionMicrosoftTelemetry * PR feedback * Move etw registration into lotus environment constructor and deresgister in lotus environment destructor * Brianma/cpuwinml (#2466) * allow building winml cpu without dml. * Brianma/breaks (#2469) * fix some more breaks * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * move dml checks out of winml and into the adapter * better error handling * Brianma/fi (#2470) * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * fix bad merge * Checking in a staging checkpoint point so that Ryan can work with me in parrallel * build break. * Brianma/testfails (#2473) * add missing ir version to dictvectorizer-string.onnx * add missing ir version to relu.onnx * add missing ir version to zipmap*onnx * add IR version to manually generated models * remove an unnecessary ifdef dml * Brianma/windowsai fi (#2475) * update dockerfiles/README (#2336) * Make elementwise op run 4 items per thread (#2335) Description: Describe your changes. Make elementwise op run 4 items per thread unroll for loop to leverage ILP remove unnessary N==0 check inside elementwise GPU kernel Motivation and Context Why is this change required? What problem does it solve? It can improve the performance of GPU elementwise ops. ~2% performance gain on popular NLP bert model. If it fixes an open issue, please link to the issue here. * Add CUDA GatherElements kernel (#2310) * Updates * Update test * Update * Updates * nits * PR feedback * Update * Update * PR feedback * PR comments * Update * Fix build * Fix build * Nits * Fix * Layer Normalization Fusion (#2319) basic layer normalization transform * Add FastGelu Cuda Op for Gelu and Add bias fusion (#2293) * Add FastGelu cuda op * Add AddBiasGelu for experiment * Revert "Add AddBiasGelu for experiment" This reverts commit 5c1ee019858c657e6bb75887265cb85675626e5b. * Add bias * Add unit tests * update comment * update script * fix build error * update coding style * update for CR feedback Enable half2 optimization only when cuda arch >= 7.0 * move _Tanh to common.cuh * implement CPU contrib OP Attention (#2333) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanUnusedInitializers. (#2320) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanupUnusedInitializers. This means initializers that have been replaced during graph optimizations are not left in the GraphProto when we save an optimized model. * Handle edge case where a model has an unused initializer with matching graph input by also removing the graph input. * Use non-const iterators in std::find_if calls to make centos build happy. * Nuget pipeline changes (#2305) 1. refactor the pipeline, remove some duplicated code 2. Move Windows_py_GPU_Wheels job to Win-GPU-CUDA10. We'll deprecated the "Win-GPU" pool 3. Delete cpu-nocontribops-esrp-pipeline.yml and cpu-nocontribops-pipeline.yml 4. In Linux nuget jobs, run "make install" before creating the package. So that extra RPAH info will be removed * Cuda Reverse Sequence Op, maping types of same size using same template function. (#2281) * Set ElementType to String type of node metadata, instead of byte[] (#2348) * Set ElementType to String type of node metadata, instead of byte[] * Fix spacing * Introduce PrimitiveType into a Type System along with an integer constant (#2307) Improve perf by avoiding GetType<T>() calls. Introduce MLTypeCallDispatcher to switch on Input Type. Add Tensor IsType<T>() fast method. * Fix/test dim value of 0 handling in a couple of places (#2337) * Update the CUDA Where implementation broadcasting logic to handle a dim with value of 0. Add unit test Also add unit test for unary op with dim value of 0 * Exclude ngraph from Where test with 0 dim. * Openvino EP R3.1 onnxrt server (#2357) * onnxrt server with OVEP * onnxrt server with OVEP * Update Dockerfile.server.openvino * onnxrt server OVEP fix reviews * onnxrt server OVEP fix reviews * Implement cuda nonzero op. (#2056) Implement cuda nonzero op. * Direct use python numpy array's memory if already contiguous. (#2355) * Direct use python numpy array's memory if already contiguous. This could greatly improve performance for session with large input, like big image 1920x1080 fastrcnn, 30~40% speed up could be achieved. * Add test case enforce contiguous/non-contiguos numpy array as inputs. * Add helper to create output to minimize binary size. (#2365) Add ConstEigenTensorMap typedef so we don't unnecessarily const_cast the const input Tensor. * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS (#2369) * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS * update * Add Tracelogging for profiling (#1639) Enabled only if onnxruntime_ENABLE_INSTRUMENT is ON * test bidaf with nuphar for avx target (#2370) increase nuphar test coverage a bit * Fix a bug in TLS refcount that may destabilized CUDA CI (#2374) * update output size calculation for resize (#2366) * change how output size is calculated for resize op * add tests for ver 10 resize * Extend OneHot CPU kernel to support more types (#2311) * Extend OneHot CPU kernel to support input int64_t, depth int32_t, output float * Skip BERT before the test data fix is picked up * Fix bug with Slice. Need to pass in flattened input dimensions so the initial offset into the input is calculated correctly. (#2372) * Add opset 11 version of Split to CUDA ops (#2376) Organize the CUDA ops definitions so all the opset 10 and 11 parts are together (same setup used for CPU ops) * Layer Norm Fusion Fix (#2379) * layer norm fusion fix * Add input shape check in code and unit tests * Fuse Add + Gelu (#2360) Implement the transformer to fuse add + gelu Implement the accurate kernel * Skip layer norm transform (#2350) * skip layer normalization transformer * Another try to stabilize CUDA CI (#2383) The root cause seems to be failure in CUDA dealloc when tear down. cudaFree return code was ignored before, so should the debug check. * fix BUILD.md typo (#2375) build.py: error: argument --config: invalid choice: 'RelWithDebugInfo' (choose from 'Debug', 'MinSizeRel', 'Release', 'RelWithDebInfo') * Fixed compilation with ngraph (#2388) * Fix reuse logic in allocation planner. (#2393) * Fix reuse logic in allocation planner. * PR comments * Add helpful comments * Don't allow reuse across string tensors. * [NupharEP] Multiple optimizations (#2380) Fuse transpose into MatMul Implement Pow and constant scalar simplification Vectorize ReduceMean Improve symbolic shape inference Minor updates for better debugging in fused function name * Avoid using the default logger in the graph lib and optimizers (#2361) 1. Use the session logger if it is available. 2. Don't disable warning 4100 globally. We should fix the warnings instead of disabling it. * Change CUDA implementation of Transpose to support all fixed size tensor types (#2387) * Change CUDA implementation of Transpose to not use a typed kernel so we can support more types with minimum binary size. Add support for 8, 16, 32 and 64 bit types. Add unit tests. Add method so the implementation can be called directly (will be used by CUDA Scan very soon). * Disable TensorRT for MLFloat16 and int8 unit tests. * Address PR comment and add support for calling cublas implementation if type is mlfloat16. * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. (#2398) * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. * [NupharEP] force some low/zero cost ops to be inlined (#2409) * fix cross compile bug (#2415) * Minor optimization: if a node has already been placed, there's no need to find a kernel for it. (#2417) * Add Reshape Fusion (#2395) * Add reshape fusion * Add some comments * update comments * update comment format * update according to feedback * update for recent logger change * fix build error * (1) Support both input and output edges in find path in graphutils (2) Add a test case of only one constant initializer of Concat input. (3) Refactor ReshapeFusion class to allow add more subgraph fusion in the future. * fix error * (1) loose constraint on initializer: non constant is allowed for reshape fusion. (2) Change versions type to vector. (3) Add logging. (4) Return false when multiple output edges matched in FindPath. Add comments. * only allow one direction (input or output) in FindPath * [NupharEP] Update notebook and docker image (#2416) Add BERT squad in Nuphar tutorial Enhance speed comparsion readability * Fix the issue in matmul_add_fusion (#2407) Fix the issue in matmul_add_fusion If Muatmul + Add has shape [K] * [K, N], reset it to [1, K] * [K, N] will make the output shape to [1, N] will also requires a reshape on the output. Fix: just remove the shape reset to not fuse it. Add a negative test case for matmul+add fusion * feat(treeregressor): Update TreeEnsembleRegressor for type support (#2389) Updates the `TreeEnsembleRegressor` to allow for `double`, `float`, `int64`, and `int32` inputs to match the upstream specification. Signed-off-by: Nick Groszewski <nicholas.groszewski@capitalone.com> * onnxrt server documentation update (#2396) * Added support for Pad-2 operator in OpenVINO-EP (#2405) * Add CUDA If operator. (#2377) * Add CUDA If operator. Uses CPU operator for implementation. By adding a CUDA version the inputs/outputs (with the exception of the 'cond' input) stay on GPU, and no other logic is required to avoid a copy to CPU across the control flow node. * Improved documentation for onnxruntime::utils::SwapByteOrderCopy(), added precondition check. * Fix the type constraints on CUDA If operator to exclude strings. (#2431) * add Im2col<uint8_t> (#2438) * Adjust codegen vectorization width from target (#2439) * Adjust codegen vectorization width from target * Add CUDA Scan operator. (#2403) * Add Scan CUDA op. Uses CPU implementation for logic. Added some device specific functors for handling when data needs to be manipulated on a different device. Added ability to override the materialization logic in the OrtValue slicer so DML can plugin their handling. * Fix Windows GPU C API packaging pipeline failure (#2440) Fix Windows GPU C API packaging pipeline failure (#2440) * Correctly handle implicit inputs for fused nodes (#2390) * Correctly handle implicit inputs for fused nodes Previously, nuphar's partitioning function didn't include node's implicit inputs into the inputs list of MetaDef, and hence a crash was triggered in the onnx graph checker. This commit fixed the issue. Furthermore, it also fixed a related issue where we didn't add implicit inputs into graph_inputs_excluding_initializers_ in Graph::SetGraphInputsOutputs. the issue was that graph_inputs_including_initializers_ populated by SetInputs (e.g. called by FunctionImpl::FunctionImpl) may contain implicit inputs which were not of any node's initializers in the graph. Because they were not part of any initializers, these implicit inputs couldn't be visited by going through all nodes' inputs. Consequently, they would *not* be added into graph_inputs_excluding_initializers_. We fixed the issue by first copying the populated graph_inputs_including_initializers_ into graph_inputs_excluding_initalizers_, which then had both initializers and non-initializers as its initial content. Later, we erase initializers from the list. In this way, we can ensure all implicit inputs to remain in graph_inputs_excluding_initializers_. * refined comments and fixed duplicates Address CR by revisiting comments in terms of implicit inputs Also fixed an issue by skipping duplicates while copying inputs from graph_inputs_including_initializers_. * address CR explain why we need to collect nodes' implicit inputs * don't rely on pointer values for iterating std::set Previously, openvino relied on iterating a set of NodeArg pointers to construct inputs and outputs for a fused graph. It could cause non-determinism. The reason was that although iterating std::set by itself is stable, pointer values of NodeArgs may vary. Consequently, we could end up visiting the set's elements in different orders for different runs for the same test, which resulted in constructing inputs (and outputs) with different orders to the fused graph. For example, for the same test, we may have inputs [A, B] in some runs but inputs[B, A] in others. Let's use std::string as the key type to avoid such nondeterminism. This commit also added implicit inputs into meta->inputs while returning the capability from the openvino provider. * Fixed another latent issue in openvino's GetCapability function The issue was that we couldn't simply erase fused_inputs and fused_outputs while iterating the nodes. For example, an output NodeArg may have multiple uses, and it's wrong if we erase it from fused_outputs when we encounter only one of its uses as input. * Remove DeviceAllocatorRegistry class (#2451) Remove DeviceAllocatorRegistry class * CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library. * Parallel Gelu with ParallelFor (#2399) Parallel Gelu to get better performance for Gelu * Clean up build.py (#2446) * Pull the latest image before running docker build * Fuse SkipLayerNorm with Bias (#2453) Fuse SkipLayerNorm with Bias * Allow more than one invocation of CreateEnv in the same process. (#2467) * Allow more than one invocation of CreateEnv in the same process. * Fix centos build * Symbolic shape inference improvements: (#2460) * Symbolic shape inference improvements: - add a mode to guess unknown ops' output rank - add support for GatherND - add support for If - fix a bug in get_int_values when then tensor rank > 1D, by treating it as no sympy data - add symbol to literal merge when ONNX silently merges dims - fix a bug in Concat when input dim is 0 - fix a bug in ConstantOfShape that computed dim is not updated - add support for dynamic shape in ConstantOfShape - fix a bug in Loop output shape that loop iterator dim is not inserted at dim 0 - add support for dynamic padding in Pad - add support for dynamic shape in Reshape - add support for Resize with opset > 10, by treating output dims as dynamic - fix a bug in Slice when starts/ends are dynamic - restrict input model to opset 7 and above - make output model optional to avoid disk write when testing Run model tests for symbolic shape inference Reduce 2GB docker image size of nuphar * add additional test data set for nuget pipeline (#2448) * add SAS token to download internal test data for nuget pipeline * update azure endpoint * fix keyvault download step * fix variable declaration for secret group * fix indentation * fix yaml syntax for variables * fix setting secrets for script * fix env synctax * Fix macos pipeline * attempt to add secrets to windows download data * fix mac and win data download * fix windows data download * update test data set url and location * Revert "Brianma/windowsai fi (#2475)" This reverts commit 5780b864a15513fda4eadbfc2b5345fefe70b5ec. * Add scenario tests (#2457) * Add scenario tests * Remove TODO from model license * Add winml_api test dependency * fix model load test. fi from master changed the constructor (#2483) * make api tests all pass (#2486) * fix bad merge * fix bad model merge * Layer dev paulm (#2492) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * Rename ambiguous header (#2489) * fix one more missing IR version model (#2500) * add missing IR version to 4 more models used by scenario tests (#2501) * Add CLI parameters to test runner, build WinML in ARM and x86 CI (#2479) * Support test parameters through CLI arguments * Add WinML do Windows x86/ARM CI builds * Code style fixes * Update googletest Remove GPUTEST macros everywhere now that GTEST_SKIP is supported * Refactor main.cpp * Build scenario tests without DML * Link scenario tests to DML when it's enabled (#2502) * Layer dev release pipeline (#2488) Adds winml binaries to existing cpu nuget package, and creates new gpu dml nuget package with winml binaries and DML EP. * Layer dev paulm (#2506) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * Remove usage of IOBinding in WinML and use C_API Run method (#2504) * remove usage of iobinding * Change data structure to use vector of Ort::Values * Polish bind input / output * Use C APIrun method * Update providers on evaluate getresults * Remove run and IObinding interface from WinMLAdapter * Remove use of IObinding * bind unbound outputs code moved to learningmodelbinding * clean up unneeded istensor adapter function * Fix comment * Check if session is closed before binding and clearing * PR feedback * Layer dev paulm (#2507) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * Make tests dependend on winml_dll (#2509) * add dml binaries to DirectML package and be more explicit about condition variables (#2520) * re-enable warnings for winml builds and fix the warnings that were hiding (#2526) * turn devmode back on for winml builds * fix some warnings. include protobuf in a way that disables some warnings * undo protobufhelpers changes and just ignore 4100 errors in pb code * attempt to isolate protobufhelpers errors * add template specialization for getting tensor proto data * Layer dev paulm (#2533) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * Spawn child process to run DeviceLostRecovery scenario test (#2530) * Spawn child process to run DeviceLostRecovery scenario test * Layer dev paulm (#2536) ori said yes * add missing namespace to winml_trace_logging_provider in lotusenvironment.h (#2542) * Handle exception thrown from all apis in WinMLAdapter (#2539) * various changes to unblock windowsai ADO build * Fix custom ops scenario tests (#2562) * Do not shutdown protobuf after ort environment gets destroyed. Lazy load lotus environment first time it is needed * comment typo * pr comment about calling phoenix singleton * Make lotus_environment static in winmladapter * Layer dev paulm (#2567) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * making windowsAI azure dev ops work. * code review comments. * revert changes * Cmake and preprocessor fixes that where uncovered by building on agents without DML available via SDK * Layer dev dml delayload (#2580) * Brianma/cpu (#2583) * don't include dml stuff in cpu builds * tests that link the image lib also need the telemetry lib now * Throw Winml_err_invalid_binding if binding gpu resource on cpu device (#2589) * Throw Winml_err_invalid_binding if binding gpu resource on cpu device * PR comments. No need to query executionprovider for is gpu device * User/xianz/ortthrow (#2596) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * User/xianz/ortthrow (#2599) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * add status error message * Remove uwp onsuspending winrt call because logruntimeperf is getting removed (#2630) * User/xianz/dedup telemetry (#2631) * investigate duplication of telemetry in winml and ort * remove winml telemetry events * telemetry executionProviderEvent * remove unneccessary file and refactor code little bit * Revert back TelemetryEvent, which send up ETW event. * merge changes from layer_dev to windowsai (#2638) * Remove underscore from googletest names (#2616) * Fix leaking memory allocator Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24278761 and https://microsoft.visualstudio.com/OS/_workitems/edit/24330198 * Explicitly initialize Ort::Value with nullptr * Cache WinML adapter * bad merge * define private version of dxcore enum that is added in 19H1 SDK. (#2654) * add comment for explaning private definition of dxcore d3d feature level ennum value. (#2672) * do not package directml.pdb for redist packages. (#2676) * Fix leaking operator registry (#2645) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24354916 * User/orilevari/windowsai master merge (#2674) merge resolutions included pulling in telemetry logic that was merged to master and not windowsai and dereferencing InferenceSession::sessionstate now that it is a unique pointer * Delete Ort Allocator in LearningModelBinding (#2653) * Delete OrtAllocator in LearningModelBinding * PR comments to make Ort::Allocator a smart pointer * Small comment change * PR feedback to clean up code * PR feedback on move semantics * Clean up std::move * Fix memory leaks (#2679) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24356109, https://microsoft.visualstudio.com/OS/_workitems/edit/24388361 and https://microsoft.visualstudio.com/OS/_workitems/edit/24388596 * various changes to properly organize and skip GPU tests. For now for No DML builds we will not run GPU tests at all. In the future we should adapt the tests to expect the appropiate errors. (#2695) * Windowsai without fi (#2701) * Disable Attention fusion tests when DISABLE_CONTRIB_OPS is defined (#2529) * Setup java ci (#2528) * Add provision in ORT for session options to be parsed when available via model file (#2449) * Initial commit * Fix gitmodules * Nits * Nits * Updates * Update * More changes * Updates * Update * Some updates * More changes * Update * Update * Merge * Update * Updates * More changes * Update * Fix nits * Updates * Fix warning * Fix build * Add comment * PR feedback * PR feedback * Updates * Updates * Update * More changes * Fix build break * Comment test for now * Updates * Updates * PR feedback * Updates * Nits * Add tests * Fix build * Fix build * Fix build * Fix build break * Fix build * Nits * PR feedback * More change * Expose GetSessionOptions in pybind logic and add unit test for python * Fix build * PR feedback * PR feedback * Revert "Disable thread pool creation when enabled OpenMP (#2485)" (#2535) This reverts commit 7c7d5a149c9ed52eec67304bae5c4b132166a8a1. * Add dynamic shape support in TensorRT execution provider (#2450) * remove onnx-tensorrt submodule * add new onnx-tensorrt submodule (experiment) for trt6 * update engine build for trt6 * update compile and compute for tensorrt6.0 * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * switch to onnx-tensorrt master for TensorRT6' * Update tensorrt_execution_provider.cc * Handle dynamic batch size and add memcpy in TensorRT EP * update test cases * Update tensorrt_execution_provider.cc * update onnx-tensorrt submodule * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.ubuntu_tensorrt * Update run_dockerbuild.sh * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update concat_op_test.cc * Update tensorrt_execution_provider.cc * Upgrade TensorRT to version 6.0.1.5 * Update onnxruntime_providers.cmake * Update CMakeLists.txt * Update reduction_ops_test.cc * Update install_ubuntu.sh * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.tensorrt * Update BUILD.md * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update onnxruntime_providers.cmake * Update install_ubuntu.sh * Update install_ubuntu.sh * Update gemm_test.cc * Update gather_op_test.cc * Update CMakeLists.txt * Removed submodule * update onnx-tensorrt submodule * update header file * Removed submodule * add submodule onnx-tensorrt kevin's branch shape-test' * add debugging code * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * merge master * Removed submodule * update onnx-tensorrt submodule * add more changes for dynamic shapes * Update tensorrt_execution_provider.cc * update for dynamic shape * update dynamic shape processing * fix logger issue * remove submodule onnx-tensorrt * add submodule onnx-tensorrt * add env variable min_subgraph_size * remove redundency * update document * use onnxruntime::make_unique * fix multi-run issue * remove some tests to save CI build time * Add dynamic shape test * Update TensorRT-ExecutionProvider.md * Add example of running Faster R-CNN model on TensorRT EP * Add more details on env variables * update environment variables * Update tensorrt_basic_test.cc * Update model tests * Update tensor_op_test.cc * remove --use_full_protobuf * Update build.py * User/xianz/telemetry (#2458) * enabme telemetry * enable telemetry * set enable telemetry as default * for debugging * remove log and set disable telemetry as default back * delete private file while testing * resolve comment: mainly add license header, rename macro and update docs * rewording in privacy.md * Fix integer overflow in cuda NonMaxSuppression implementation (#2540) * add test case that should pass but fail * fix nms * extract int_max_output_boxes_per_class * Introduce container type runtime checks and other improvements (#2522) Rework TensorSeq in a manner consistent with Tensor and SparseTensor in terms of type system setup. Reduce templating. Introduce helpers to ensure the same data type. Make OrtValue __dtor not virtual. Introduce ContainerChecker * Fix C API tests for centos and mac (#2544) * change c++14 to c++11 * add ld lib path for centos * enable csharp tests on macos * fix C API test on MacOS + fix manylinux dotnet install * fix manylinux dotnet install * fix lib link * Add back executable bit to build.py * Fix a bug handling negative begin pad values in Pad op (#2550) * Fix bug in Pad op * Update * DNNL CMAKE update (#2548) * Fix android build (#2558) * Update win-x86-ci.yml (#2557) Fix build pipeline break * Re-enable Windows C# tests (#2564) * disable onnx_test_runner -x invocations for dnnl (#2568) * Allow sequence length to be symbolic (#2559) * setup java ci mac (#2570) * make layernorm fusion to support opset 11 (#2545) * Fix a warning found in the latest VS release * Add more check on SkipLayerNorm and BiasGelu fusion (#2574) * Fix file not found error during docker build. (#2569) * Add ConvTranspose1D (#2578) * Ryanunderhill/packagename test (#2582) * [Nuphar EP] fixes for some object detection models (#2581) Update notebook tutorial with multi-threaded int8 GEMM from #2517 * EmbedLayerNormalization Fusion Improvement (#2553) Embedding layer norm fusion improvements - add more checks * Update version (#2584) * Temporarily exclude vgg19 test from Python backend test 1. temporarily exclude vgg19 test which comsumes too much memory, run out of memory on Upsquared device. Single test pass for vgg19, need furture investigation (#2588) 2. Update docker file to decrease the docker image size * Update docs for Android NNAPI EP (#2586) * Fix lto bug for protobuf and ubuntu * add path to build dir before test run (#2590) * Add missig env variables for mac pipeline test (#2595) * Fixed an issue in updating realized dims (#2597) when we update realized dims for scan's output, the sliced axis also needs to be inclusive, i.e. we should check with "dim >= insert_inclusive_axis", because the offset in the symbols are based on Scan sugraph. Otherwise, we would end up with shape mismatch later. * Java API for onnxruntime (#2215) * Add support for opset 11 in reshape fusion (#2592) Support opset verion 11 in reshape fusion * Rename automl python tools folder to featurizer_ops. (#2593) * Support opset 11 subgraph of Squad model in Embed Layer Normalization (#2605) Support opset 11 Squad model that is exported from PyTorch nightly. The embed layer uses Range op which is missed in the transformer. * symbolic shape inference: fix warnings in GPT-2 model (#2608) And revise nuphar perf test on BERT squad * Dump subgraph ID and fused graph ID (#2607) * Dump subgraph ID and fused graph ID Dump subgraph ID and fused graph ID for better debugging * Remove local static fused_count added a field global_fused_count_ to NupharExecutionProvider class * EmbedLayerNormalization Fusion For Dynamic Squad Model Opset 10 (#2613) Support subgraph of SQuAD model exported from pytorch with dynamic input axes * Allow providers to be set for InferenceSession at construction (#2606) * Remove unnecessary parameter in some places in GatherElements implementation (#2612) * Remove unnecessary parameter in some places * Update * Update * Make sure fenced tensor could not reuse other tensor. (#2561) Fix random error caused by this. * Improve Embed Layer Norm Fusion for SQuAD with static input shape (#2621) * fix float16 comparison in initializer (#2629) * epsilon attribute for layernormalization fusion (#2639) * removed unnecessary batch file and fix path (#2640) * Add shape inference to ConvTransposeWithDynamicPads schema (#2632) * Improve cuda expand() opeator's performance. (#2624) * Cuda pad optimize when no padding is needed. (#2625) * Shortcut cuda Pad() when no padding is needed. * Optimize cuda scatter() on 2D compatible. (#2628) * Optimize cuda scatter() on 2D compatible. * Add some comments. * fix build error for ARM (#2648) * Improve performance of resize() in Nearest mode (#2626) Special treatment for 2D, check same size as input image. And in 2d kernel, template use_expolation. * Fix memory exception in Layer Norm Fusion (#2644) * Windows CI changes(#2650) * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Revert "Windowsai without fi (#2701)" This reverts commit 285d4c85ff5c4e265f963208170304ef3461e684. * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Deref unique pointer for session_state * send shutdown event when dll is unloaded and EvaluationStop, SessionC… (#2704) * send shutdown event when dll is unloaded and EvaluationStop, SessionCreationStart Events. * Add EvalutationStart Event * add comment * use correct type for for loop (#2755) * ARM CI (#2759) * Set ARM agent pool * Set CMake generator to VS 2019 in ARM * Use system-wide CMake instead of custom version Our custom version is too old for VS 2019 * Use DML and build shared lib in ARM CI * Restore nuget packages in ARM CI * Disable DML * Refactor ARM debug/release builds * Use system packaged Python version * Remove hardcoded Python path * Downgrade Python to 3.7 for build * Remove explicit CMake path * Fix invalid JSON in cgmanifest.json (#2760) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * Remove usage of WHOLEARCHIVE in WinML CMake and add WinMLAdapterFactory (#2726) * Remove usage of WHOLEARCHIVE in WinMLAdapter CMake and add WinMLAdapterFactory * PR feedback, no need for dll(export) since using def file * PR comments * Small comment in gen_def.py * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * Move winml_provider_factory.h to proper location (#2801) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Filter CPU case for IsFloat16Supported (#2802) * Merge fixes * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * Fix warnings that cause build to fail * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * Make DML include path global (#2882) * Make DML include path global * Add generated cppwinrt headers to winml_lib_common * Integrate changes to WindowsAI to make ADO Build (#2886) * Revert "CMake cross-generator fixes (#2790)" This reverts commit dbe7d97fa1ab155f1309bced87199527e8f35bd2. * add additional suppress warning in onnx_proto * ignore /wd4996 warning * DML execution provider fixes * Revert "Revert "CMake cross-generator fixes (#2790)"" This reverts commit 1ae7b4bcbc02edc881ad28685da98e095dfceb17. * Update func signature of custom op function overloads * common devicehelpers fixes * Add pch.h for winml_lib_common * re-add winml_lib_common_dir/inc to include path for winml_adapter * User/orilevari/dml redist shared folder (#2890) * move dml nuget package directory up one level to make it shared between build flavors * Merge conflict fix * Revert "Merge conflict fix" This reverts commit 142fa72cf9ce4344ad717b50b7ea2b8582aadc7c. * Revert "Merge remote-tracking branch 'origin/master' into windowsai" This reverts commit 6e2126d46e5e5f564d65da37dd4f70c93dd81165, reversing changes made to b3f5583dc9249834b947c8ea905f6a98060d5bd6. * Make winml_test_common free of test macros (#2902) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Don't generate files outside the build root (#2914) * Don't generate files outside the build root * Add onnxruntime_EXTERNAL_DEPENDENCIES to WinML * Add DML depedency on RESTORE_PACKAGES * User/orilevari/fix yaml merge bugs (#2918) * Add winml test source parameter into cmake function (#2919) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Add arguments to unittest cmake functions * remove comment * Revert "Revert "Merge remote-tracking branch 'origin/master' into windowsai"" This reverts commit ade5abe72a4234fdbc3623093c61c02c6b0bdc26. * Fix breaks from merge with ORT master * Brianma/linux (#2917) * don't include windows.h in cross-plat header * add default case for switch statement * signed/unsigned mismatch fix Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> * User/sheilk/winml adapter c api (#2891) * Create winml adapter c api * fix build * make it build * move adapter into onnxruntime core/session * entry point not exported * minor changes * make model metadata work * make tests pass * implement all the model reflection apis on the adapter c abi * update the new ort interface to create a lotus ennvironment with a logging sink * start adding ort env * move all winml code into adapter folder/lib to isolate it * ensure a single logging manager at a time * start refactoring session * refactor session creation interface * add cpu and dml session option methods to adapter * finish session init * stub out interfaces in ort lib to perform similar mechanics of iinference session * enable profiling, and enable schema override * update session register graph transformers * turn back on custom registry for custom ops * Add sync api * add last c api stubs * should build... but all feature values are broken since this is in flight to moving all implementation details into ivalue * remove ep adapter header * Implement DML execution provider functions from adapter (#2846) * Implement DML execution provider functions from adapter * Use functions in OnnxruntimeEngine.cpp * make map/sequence type_infos freeable, and start implementing ivalue * make it build again * implement value methods * implement remaining methods * remove com adapter abi * check dml session * cache the allocator on ivalue * check if resource is cpu/gpu when access its mutable data * update tensor * mismatched parentheses * fix tensor base and binding obj * it evaluates tensors! sometimes... * minor fixes * enable gpu evals * wrapper all existing winml adapter apis with API_IMPL to try catch (#2854) * update winml... tensor strings are broken, need to template tensorbase to do different things for strings * make tensor strings work with 2 copies in/2 copies out * Fix tensor string and allocator bug * make maps work again... needs some fixes still * Make it build! * enable map inputs * map outputs * unbound outputs for sequences and maps * User/xianz/merge windowsai (#2883) * Packaging pipeline changes for VS 2019 (#2711) * Tiny fix to codegen * Simplify cache implementation and avoid static variables that may carry over between models * Extend DML kernels (#2641) * Additional DML operators * Check unsupported attributes and inputs * Address PR comments * Add kernel capability function used for partitioning, and re-enable stride-based int64 support based on value range * Fix test failures * Build fix * PR comments * Update Nuphar tutorial notebook (#2721) 1. Reflect int8 GEMV improvements for multi-threading from #2696 2. Add notes on multi-threading control using OpenMP 3. Add samples of running multi-isa AOT, and show int8 GEMM differences between AVX and AVX2 4. Add rnn_benchmark example to resolve #1993 * Add schema for new Qops (#2611) * Add schema for new Qops * adding shape inference + qlinearaveragepool * plus review comments * plus review comments * updates per review comments * plus review comments * [server] Add supposed for model_name and model_version as cli parameter (#2708) * remove 64bit warning message from python validation. (#2727) * MLAS: ARM64 build fix (#2734) fix bad usage of vreinterpret to cast vector element types * Fix broken python docs links (#2740) * Fix build on Mac OS (#2731) mac os ld doesn't support --while-archive, correct option is -all_load * fix ngraph wheel (#2737) * fix ngraph wheel 1.1.0 onnxruntime_ngraph wheel doesn't work * remove libdnnl.so in nGraph Libs * make it easy to compare * Split onnxruntime server to a separated folder (#2744) * Fix build for Python 3.8 (#2747) * Fix build for Python 3.8 * Update protobuf to 3.11.2 (#1928) Update protobuf to 3.11.2 (#1928) * Change default optimization level to All (from Basic) (#2745) * change default optimization level to All (from Basic) * fix test * fix c# test * Update numpy to 1.18 (#2758) * Update numpy to 1.18 * Pipeline changes for python 3.8 (#2753) 1. Pipeline changes for python 3.8 2. Fix a regression in setup.py which was just introduced in the previous commit. Please notice, we still haven't made python 3.8 + Windows + CUDA work. * Add basic stacktrace output for posix debug builds. (#2749) * [NupharEP] fix a race condition when multiple sessions running different models concurrently (#2772) * Revert "Change default optimization level to All (from Basic) (#2745)" This reverts commit 56bb503c2f26474b6613bcb2a198691a11dcef38. * Fix typo in error message (#2736) * Rename MKL-DNN to DNNL to fix broken link (#2730) * Fix nightly build version number issue * Pass BUILD_BUILDNUMBER to linux docker * Disable featurizers in python packages * Import more featurizers (#2781) Make kernels non-template. Add input constraint for learnt data. Add min_max_scalar_transformer, robust_scalar_transformer, inputation_marker_transfomer, label_encoder_transformer, missing_dummies_transformer along with tests. Advance Featurizers library commit. * Implement a more stable softmax (#2715) * Implement a more stable SoftMax e^x is represented as infinity if x is large enough, like 100.f. Infinity divided by Infinity is a NAN. Thus, softmax gets a NAN if one or more item are large enough. A math transform as below is leveraged to get a stable softmax: e^xi/(e^x1 + ...e^xn) = e^(xi - max) / (e^(x1 - max) + ... + e^(xn - max)) And for convenience, force max to 0.f if all xi are negative * Contributing: Fix a typo (#2784) * ACL EP GEMM improvements (#2780) When it is posible we use a fully connected layer instead of the gemm implementation. This will let the library use the best implementation based on the input data. * ACL EP convolution improvements (#2774) Added the optimized implementation for depthwise convolution for both ACL v19.02 and ACL 19.05. Also the pointwise convolution seems to be more optimal in the CPU implementation so we opted for that instead. * Add script for release Nuget validation (#2719) * Initial commit * Nits * Disable a test temporarily * Change working directory * Test * Add download python step * Test update * More changes * Fix space issue * Fix * Verify nuget signing * Fix * Spaces * PR feedback * Nit * Fix * Fix * Remove temporary changes * add uint8 support to where op (#2792) * Improve bert optimization script: (#2712) (1) Move input int64=>int32 conversion to embed layer fusion. (2) Output epsilon attribute for LayerNormalization fusion. * add session creation time cost. (#2798) * ML.NET team needs featurizers within a package (#2789) Add auto ml featurizers to Windows, MacOS as well as to GPU packaging-pipelines. * Initialize max of softmax with lowest of float (#2786) * MLAS: update SGEMM threading parameters (#2808) * add interface to copy batch tensors. (#2807) * add interface to copy batch tensors. * onnxruntime * speed up Windows TRT CI (#2811) * don't run cuda tests if building with tensorrt * remove unnecessary build options for win trt ci * refactor win gpu tensorrt ci yml * --numpy_version=1.17 * update * update * azcopy and cuda path * Update test data (#2356) * Add timeseries imputer transformer featurizer kernel (#2813) Make kernels non-template. Add input constraint for learnt data. Fixup tests. Add two more featurizers along with tests. Tests fail. min_max_scalar_transformer robust_scalar_transformer Fix tests serialized stream by prepending version bytes. Add inputation_marker_transfomer and the test. Fix up float/double type designations. Added label_encoder_transformer along with a test. string_throw case is broken at the momement. Fix labelencodertransfomer_test.cc string_throw case Rename maxabsscalertransformer_test.cc Add MissingDummiesTransformer along with the test. Update manifest. Add TimeSeriesImputerTransformer definition, implementation and tests * Fix memory leak in TRT (#2815) * fix memory leak issue * revert EP_FAIL on enueueV2 * Add manifest missing comma * Run static code analyzer on most of our code (#2817) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * update quantization doc (#2783) * update documentation for quantization script * plus some spell corrections * Filter CPU case for IsFloat16Supported (#2802) * update default optimization level + fix gemm_activation fusion (#2791) * update defualt optimization level + fix gemm_activation fusion * fix typo * add unit test and incorporate review comments * fix test comment * Fix dnnl wheel package name (#2823) * Append '-dnnl' to whl package name when --use_dnnl * Update build.py * Update Ubuntu & TensorRT version in README (#2820) Dockerfile.tensorrt is using nvcr.io/nvidia/tensorrt:19.09-py3 as base Image, update Ubuntu and TensorRT version according to https://docs.nvidia.com/deeplearning/sdk/tensorrt-container-release-notes/rel_19-09.html#rel_19-09 * Merge fixes * Add OneHotEncoder and HashOneHotEncoder kernels. (#2830) Add defs and imlementation for OneHotEncoders, adjuist date_time_transformer kernel and test. Add OneHotEncoder kernel test. Add HashOneHotVectorizerTransformer unit test. This does not link due to multiple definitions of functions that are included into header from a CPP file. * Upgrade gtest to the latest version (#2827) WinML would like to update the googletest submodule. They want some newer features (namely GTEST_SKIP to skip tests programmatically and be able to skip entire fixtures easily) and would need to update the submodule version. However, because the new version of code hit a bug in gcc, even though the bug is already fixed in the latest gcc but we're using gcc 4.8.x and it won't get patched for the bug, so we have to do a compromise, change our code a little bit to make it work. The gcc bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=51213 * Add support for int64_t for topk CPU. Fixes github issue #2806. (#2833) * Ignore allocator type in ExecutionProviders allocator map. Make default initialization of OrtMemoryInfo more clearly invalid. (#2768) * Remove allocator type from the key comparison in ExecutionProviders. Remove usage of DummyArena as it's no longer necessary. * Fix x86 tests where arena allocator is disabled. Make initialization of OrtMemoryInfo clearer by adding Invalid enum value. * Make OrtValueNameIdxMap::MaxIdx more intuitive. * Convert ExternalProject Featurizers into git submodule (#2834) Add git submodule for Featurizer library. Update cmake to build for git submodule. * add domain check for nodes + update documentation (#2831) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Fix Linux CUDA nuget packaging pipeline break * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * update optimization doc for BERT related fusions (#2819) * Add bert related transformers to doc * Add execution provider and comment for bert optimizations * Add comment about accuracy impact of approximation * Fix warnings that cause build to fail * MLAS: enable threading for quantized GEMMs (#2844) * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * remove old winmladapter cpp Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * move sequence implementation into ort lib... still commented out... need to turn back on... * begin sequence implementation * make maps and sequences work * fix broken tests * remove dead code * misc cleanup * CR feedback * User/xianz/winml adapter c api (#2869) * wrapper all existing winml adapter apis with API_IMPL to try catch * Return HR or Throw for WinML adapter APIs if failed * undo macro wrapper for two places * Wrap error macros around ort apis, too. * address CR feedback #2 * add more api throw/return macros * Revert changes no longer needed * revert changes to cxx api * format winml lib.ort and winml adapter * remove static pheonix singleton Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * missing use_dml check in winml_adapter_session (#2930) * --use_dnnl flag was mangled in merge (#2931) * use dml macro not wrapping custom registry code (#2934) * Disable LNK4199 winml_dll to enable cuda builds (#2936) * Disable LNK4199 in winml_dll * linkler->linker * LearningModelSessionAPITestGpu.CreateSessionWithCastToFloat16InModel should return DXGI_ERROR_UNSUPPORTED when FP16 not supported (#2937) * Disable LNK4199 in winml_dll * linkler->linker * Need to return DXGI_ERROR_UNSUPPORTED when Model does not support fp16 * Publish build symbols (#2939) * Publish build symbols * Don't upload PDBs for .exe files * Make x86 build (#2943) * fix last remaining size_t/int64_t warnings->errors (#2948) * TensorString, Sequences and Maps use the first allocator, but should use the cpu default allocator. (#2952) * fix tensor string allcoator * clean up default allocator usage for strings in winml lib/api.ort Co-authored-by: Ryan Lai <ryalai96@gmail.com> * Handle tensor shape of zero (#2954) Co-authored-by: Ryan Lai <ryalai96@gmail.com> * CR feedback (#2970) * CR feedback * fix weird formatting on privacy readme * Add 'All rights reserved.' everywhere * readd all rights reserved to winml_provider_factory.h * remove extra space in comment * remove extra whitespace * fixes post master merge * remove winml from nuget gpu pipeline * set IR VERSION on generated_model in rnn_benchmark (#2972) * Fix slice conformance failures (#2908) Co-authored-by: Adrian Tsai <adtsai@microsoft.com> Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Paul McDaniel <paul_mcdaniel@hotmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Dwayne Robinson <fdwr@hotmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com>
2020-02-05 01:12:19 +00:00
[marshaling_behavior(agile)]
[dualapipartition(1)]
runtimeclass SequenceFeatureDescriptor : ILearningModelFeatureDescriptor
{
//! Gets the properties of the specified feature.
ILearningModelFeatureDescriptor ElementDescriptor{ get; };
}
//! \class TensorFeatureDescriptor
//! \brief TODO:Docs
[contract(MachineLearningContract, 1)]
INBOX_ONLY([interface_name("Windows.AI.MachineLearning.ITensorFeatureDescriptor", 74455c80-946a-4310-a19c-ee0af028fce4)])
Merge windowsai (winml layering) into master (#2956) * Initial Commit * Merged PR 3985217: add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc (#2346) add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc and violating our OS layering requirements. We linked against onecoreuap_apiset.lib in VB so we will continue doing this, but I am still unsure why not to link against onecore instead since that is where we ship. However, since Sheil is the owner of this code we will wait to discuss with him before changing anything. * Initial changes for layering * more snipping to get core into ort * update build instructions to include --build_shared_lib (#2358) * update build instructions to include --build_shared_lib * fix line breaks * Task 23998197: add winml_lib_core into onnnxruntime.dll (#2368) * Task 23998197: add winml_lib_core into onnnxruntime.dll * PR feedback build break on perf_test * return proper error when the model path isn't found (#2391) * LearningModelSession is cleaned up to use the adapter, and parts of b… (#2382) this is a big PR. we are going to move it up to layer_dev , which is still a L3 so we are still safe to do work there agile. we are going to move this into the L3 so that ryan can start doing intergration testing. we will pause for a full code review and integration test result prior to going into the L2. >>>> raw comments from previous commits >>> * LearningModelSession is cleaned up to use the adapter, and parts of binding are. * moved everything in the winmladapter made it all nano-com using, WRL to construct objects in the ORT side. base interfaces for everythign for winml to call cleaned up a bunch of winml to use the base interfaces. * more pieces * GetData across the abi. * renamed some namepsace cleaned up OrtValue cleaned up Tensor cleaned up custom ops. everything *but* learnignmodel should be clean * make sure it's building. winml.dll is still a monolith. * model moved over. everything builds clean. step ! * weak ref comment * Layer dev paulm (#2408) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * Layer dev paulm (#2414) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * Layer dev paulm (#2423) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * Layer dev paulm (#2424) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * Layer dev paulm (#2425) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Layer dev paulm (#2426) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * fixed map and sequence when passing stl types across the ABI . found a leak in nvidia driver, but skipped it. all winmlapitests pass now * Moved SessionOptions over to the abi * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * Register WinML TraceLogging provider on Onnxruntime.dll (#2455) * Register WinML TraceLogging provider on Onnxruntime.dll * Add ifdef to make sure trace logging provider has telemetry option when LAYERING_DONE * No need for ifdef for TraceLoggingOptionMicrosoftTelemetry * PR feedback * Move etw registration into lotus environment constructor and deresgister in lotus environment destructor * Brianma/cpuwinml (#2466) * allow building winml cpu without dml. * Brianma/breaks (#2469) * fix some more breaks * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * move dml checks out of winml and into the adapter * better error handling * Brianma/fi (#2470) * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * fix bad merge * Checking in a staging checkpoint point so that Ryan can work with me in parrallel * build break. * Brianma/testfails (#2473) * add missing ir version to dictvectorizer-string.onnx * add missing ir version to relu.onnx * add missing ir version to zipmap*onnx * add IR version to manually generated models * remove an unnecessary ifdef dml * Brianma/windowsai fi (#2475) * update dockerfiles/README (#2336) * Make elementwise op run 4 items per thread (#2335) Description: Describe your changes. Make elementwise op run 4 items per thread unroll for loop to leverage ILP remove unnessary N==0 check inside elementwise GPU kernel Motivation and Context Why is this change required? What problem does it solve? It can improve the performance of GPU elementwise ops. ~2% performance gain on popular NLP bert model. If it fixes an open issue, please link to the issue here. * Add CUDA GatherElements kernel (#2310) * Updates * Update test * Update * Updates * nits * PR feedback * Update * Update * PR feedback * PR comments * Update * Fix build * Fix build * Nits * Fix * Layer Normalization Fusion (#2319) basic layer normalization transform * Add FastGelu Cuda Op for Gelu and Add bias fusion (#2293) * Add FastGelu cuda op * Add AddBiasGelu for experiment * Revert "Add AddBiasGelu for experiment" This reverts commit 5c1ee019858c657e6bb75887265cb85675626e5b. * Add bias * Add unit tests * update comment * update script * fix build error * update coding style * update for CR feedback Enable half2 optimization only when cuda arch >= 7.0 * move _Tanh to common.cuh * implement CPU contrib OP Attention (#2333) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanUnusedInitializers. (#2320) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanupUnusedInitializers. This means initializers that have been replaced during graph optimizations are not left in the GraphProto when we save an optimized model. * Handle edge case where a model has an unused initializer with matching graph input by also removing the graph input. * Use non-const iterators in std::find_if calls to make centos build happy. * Nuget pipeline changes (#2305) 1. refactor the pipeline, remove some duplicated code 2. Move Windows_py_GPU_Wheels job to Win-GPU-CUDA10. We'll deprecated the "Win-GPU" pool 3. Delete cpu-nocontribops-esrp-pipeline.yml and cpu-nocontribops-pipeline.yml 4. In Linux nuget jobs, run "make install" before creating the package. So that extra RPAH info will be removed * Cuda Reverse Sequence Op, maping types of same size using same template function. (#2281) * Set ElementType to String type of node metadata, instead of byte[] (#2348) * Set ElementType to String type of node metadata, instead of byte[] * Fix spacing * Introduce PrimitiveType into a Type System along with an integer constant (#2307) Improve perf by avoiding GetType<T>() calls. Introduce MLTypeCallDispatcher to switch on Input Type. Add Tensor IsType<T>() fast method. * Fix/test dim value of 0 handling in a couple of places (#2337) * Update the CUDA Where implementation broadcasting logic to handle a dim with value of 0. Add unit test Also add unit test for unary op with dim value of 0 * Exclude ngraph from Where test with 0 dim. * Openvino EP R3.1 onnxrt server (#2357) * onnxrt server with OVEP * onnxrt server with OVEP * Update Dockerfile.server.openvino * onnxrt server OVEP fix reviews * onnxrt server OVEP fix reviews * Implement cuda nonzero op. (#2056) Implement cuda nonzero op. * Direct use python numpy array's memory if already contiguous. (#2355) * Direct use python numpy array's memory if already contiguous. This could greatly improve performance for session with large input, like big image 1920x1080 fastrcnn, 30~40% speed up could be achieved. * Add test case enforce contiguous/non-contiguos numpy array as inputs. * Add helper to create output to minimize binary size. (#2365) Add ConstEigenTensorMap typedef so we don't unnecessarily const_cast the const input Tensor. * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS (#2369) * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS * update * Add Tracelogging for profiling (#1639) Enabled only if onnxruntime_ENABLE_INSTRUMENT is ON * test bidaf with nuphar for avx target (#2370) increase nuphar test coverage a bit * Fix a bug in TLS refcount that may destabilized CUDA CI (#2374) * update output size calculation for resize (#2366) * change how output size is calculated for resize op * add tests for ver 10 resize * Extend OneHot CPU kernel to support more types (#2311) * Extend OneHot CPU kernel to support input int64_t, depth int32_t, output float * Skip BERT before the test data fix is picked up * Fix bug with Slice. Need to pass in flattened input dimensions so the initial offset into the input is calculated correctly. (#2372) * Add opset 11 version of Split to CUDA ops (#2376) Organize the CUDA ops definitions so all the opset 10 and 11 parts are together (same setup used for CPU ops) * Layer Norm Fusion Fix (#2379) * layer norm fusion fix * Add input shape check in code and unit tests * Fuse Add + Gelu (#2360) Implement the transformer to fuse add + gelu Implement the accurate kernel * Skip layer norm transform (#2350) * skip layer normalization transformer * Another try to stabilize CUDA CI (#2383) The root cause seems to be failure in CUDA dealloc when tear down. cudaFree return code was ignored before, so should the debug check. * fix BUILD.md typo (#2375) build.py: error: argument --config: invalid choice: 'RelWithDebugInfo' (choose from 'Debug', 'MinSizeRel', 'Release', 'RelWithDebInfo') * Fixed compilation with ngraph (#2388) * Fix reuse logic in allocation planner. (#2393) * Fix reuse logic in allocation planner. * PR comments * Add helpful comments * Don't allow reuse across string tensors. * [NupharEP] Multiple optimizations (#2380) Fuse transpose into MatMul Implement Pow and constant scalar simplification Vectorize ReduceMean Improve symbolic shape inference Minor updates for better debugging in fused function name * Avoid using the default logger in the graph lib and optimizers (#2361) 1. Use the session logger if it is available. 2. Don't disable warning 4100 globally. We should fix the warnings instead of disabling it. * Change CUDA implementation of Transpose to support all fixed size tensor types (#2387) * Change CUDA implementation of Transpose to not use a typed kernel so we can support more types with minimum binary size. Add support for 8, 16, 32 and 64 bit types. Add unit tests. Add method so the implementation can be called directly (will be used by CUDA Scan very soon). * Disable TensorRT for MLFloat16 and int8 unit tests. * Address PR comment and add support for calling cublas implementation if type is mlfloat16. * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. (#2398) * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. * [NupharEP] force some low/zero cost ops to be inlined (#2409) * fix cross compile bug (#2415) * Minor optimization: if a node has already been placed, there's no need to find a kernel for it. (#2417) * Add Reshape Fusion (#2395) * Add reshape fusion * Add some comments * update comments * update comment format * update according to feedback * update for recent logger change * fix build error * (1) Support both input and output edges in find path in graphutils (2) Add a test case of only one constant initializer of Concat input. (3) Refactor ReshapeFusion class to allow add more subgraph fusion in the future. * fix error * (1) loose constraint on initializer: non constant is allowed for reshape fusion. (2) Change versions type to vector. (3) Add logging. (4) Return false when multiple output edges matched in FindPath. Add comments. * only allow one direction (input or output) in FindPath * [NupharEP] Update notebook and docker image (#2416) Add BERT squad in Nuphar tutorial Enhance speed comparsion readability * Fix the issue in matmul_add_fusion (#2407) Fix the issue in matmul_add_fusion If Muatmul + Add has shape [K] * [K, N], reset it to [1, K] * [K, N] will make the output shape to [1, N] will also requires a reshape on the output. Fix: just remove the shape reset to not fuse it. Add a negative test case for matmul+add fusion * feat(treeregressor): Update TreeEnsembleRegressor for type support (#2389) Updates the `TreeEnsembleRegressor` to allow for `double`, `float`, `int64`, and `int32` inputs to match the upstream specification. Signed-off-by: Nick Groszewski <nicholas.groszewski@capitalone.com> * onnxrt server documentation update (#2396) * Added support for Pad-2 operator in OpenVINO-EP (#2405) * Add CUDA If operator. (#2377) * Add CUDA If operator. Uses CPU operator for implementation. By adding a CUDA version the inputs/outputs (with the exception of the 'cond' input) stay on GPU, and no other logic is required to avoid a copy to CPU across the control flow node. * Improved documentation for onnxruntime::utils::SwapByteOrderCopy(), added precondition check. * Fix the type constraints on CUDA If operator to exclude strings. (#2431) * add Im2col<uint8_t> (#2438) * Adjust codegen vectorization width from target (#2439) * Adjust codegen vectorization width from target * Add CUDA Scan operator. (#2403) * Add Scan CUDA op. Uses CPU implementation for logic. Added some device specific functors for handling when data needs to be manipulated on a different device. Added ability to override the materialization logic in the OrtValue slicer so DML can plugin their handling. * Fix Windows GPU C API packaging pipeline failure (#2440) Fix Windows GPU C API packaging pipeline failure (#2440) * Correctly handle implicit inputs for fused nodes (#2390) * Correctly handle implicit inputs for fused nodes Previously, nuphar's partitioning function didn't include node's implicit inputs into the inputs list of MetaDef, and hence a crash was triggered in the onnx graph checker. This commit fixed the issue. Furthermore, it also fixed a related issue where we didn't add implicit inputs into graph_inputs_excluding_initializers_ in Graph::SetGraphInputsOutputs. the issue was that graph_inputs_including_initializers_ populated by SetInputs (e.g. called by FunctionImpl::FunctionImpl) may contain implicit inputs which were not of any node's initializers in the graph. Because they were not part of any initializers, these implicit inputs couldn't be visited by going through all nodes' inputs. Consequently, they would *not* be added into graph_inputs_excluding_initializers_. We fixed the issue by first copying the populated graph_inputs_including_initializers_ into graph_inputs_excluding_initalizers_, which then had both initializers and non-initializers as its initial content. Later, we erase initializers from the list. In this way, we can ensure all implicit inputs to remain in graph_inputs_excluding_initializers_. * refined comments and fixed duplicates Address CR by revisiting comments in terms of implicit inputs Also fixed an issue by skipping duplicates while copying inputs from graph_inputs_including_initializers_. * address CR explain why we need to collect nodes' implicit inputs * don't rely on pointer values for iterating std::set Previously, openvino relied on iterating a set of NodeArg pointers to construct inputs and outputs for a fused graph. It could cause non-determinism. The reason was that although iterating std::set by itself is stable, pointer values of NodeArgs may vary. Consequently, we could end up visiting the set's elements in different orders for different runs for the same test, which resulted in constructing inputs (and outputs) with different orders to the fused graph. For example, for the same test, we may have inputs [A, B] in some runs but inputs[B, A] in others. Let's use std::string as the key type to avoid such nondeterminism. This commit also added implicit inputs into meta->inputs while returning the capability from the openvino provider. * Fixed another latent issue in openvino's GetCapability function The issue was that we couldn't simply erase fused_inputs and fused_outputs while iterating the nodes. For example, an output NodeArg may have multiple uses, and it's wrong if we erase it from fused_outputs when we encounter only one of its uses as input. * Remove DeviceAllocatorRegistry class (#2451) Remove DeviceAllocatorRegistry class * CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library. * Parallel Gelu with ParallelFor (#2399) Parallel Gelu to get better performance for Gelu * Clean up build.py (#2446) * Pull the latest image before running docker build * Fuse SkipLayerNorm with Bias (#2453) Fuse SkipLayerNorm with Bias * Allow more than one invocation of CreateEnv in the same process. (#2467) * Allow more than one invocation of CreateEnv in the same process. * Fix centos build * Symbolic shape inference improvements: (#2460) * Symbolic shape inference improvements: - add a mode to guess unknown ops' output rank - add support for GatherND - add support for If - fix a bug in get_int_values when then tensor rank > 1D, by treating it as no sympy data - add symbol to literal merge when ONNX silently merges dims - fix a bug in Concat when input dim is 0 - fix a bug in ConstantOfShape that computed dim is not updated - add support for dynamic shape in ConstantOfShape - fix a bug in Loop output shape that loop iterator dim is not inserted at dim 0 - add support for dynamic padding in Pad - add support for dynamic shape in Reshape - add support for Resize with opset > 10, by treating output dims as dynamic - fix a bug in Slice when starts/ends are dynamic - restrict input model to opset 7 and above - make output model optional to avoid disk write when testing Run model tests for symbolic shape inference Reduce 2GB docker image size of nuphar * add additional test data set for nuget pipeline (#2448) * add SAS token to download internal test data for nuget pipeline * update azure endpoint * fix keyvault download step * fix variable declaration for secret group * fix indentation * fix yaml syntax for variables * fix setting secrets for script * fix env synctax * Fix macos pipeline * attempt to add secrets to windows download data * fix mac and win data download * fix windows data download * update test data set url and location * Revert "Brianma/windowsai fi (#2475)" This reverts commit 5780b864a15513fda4eadbfc2b5345fefe70b5ec. * Add scenario tests (#2457) * Add scenario tests * Remove TODO from model license * Add winml_api test dependency * fix model load test. fi from master changed the constructor (#2483) * make api tests all pass (#2486) * fix bad merge * fix bad model merge * Layer dev paulm (#2492) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * Rename ambiguous header (#2489) * fix one more missing IR version model (#2500) * add missing IR version to 4 more models used by scenario tests (#2501) * Add CLI parameters to test runner, build WinML in ARM and x86 CI (#2479) * Support test parameters through CLI arguments * Add WinML do Windows x86/ARM CI builds * Code style fixes * Update googletest Remove GPUTEST macros everywhere now that GTEST_SKIP is supported * Refactor main.cpp * Build scenario tests without DML * Link scenario tests to DML when it's enabled (#2502) * Layer dev release pipeline (#2488) Adds winml binaries to existing cpu nuget package, and creates new gpu dml nuget package with winml binaries and DML EP. * Layer dev paulm (#2506) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * Remove usage of IOBinding in WinML and use C_API Run method (#2504) * remove usage of iobinding * Change data structure to use vector of Ort::Values * Polish bind input / output * Use C APIrun method * Update providers on evaluate getresults * Remove run and IObinding interface from WinMLAdapter * Remove use of IObinding * bind unbound outputs code moved to learningmodelbinding * clean up unneeded istensor adapter function * Fix comment * Check if session is closed before binding and clearing * PR feedback * Layer dev paulm (#2507) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * Make tests dependend on winml_dll (#2509) * add dml binaries to DirectML package and be more explicit about condition variables (#2520) * re-enable warnings for winml builds and fix the warnings that were hiding (#2526) * turn devmode back on for winml builds * fix some warnings. include protobuf in a way that disables some warnings * undo protobufhelpers changes and just ignore 4100 errors in pb code * attempt to isolate protobufhelpers errors * add template specialization for getting tensor proto data * Layer dev paulm (#2533) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * Spawn child process to run DeviceLostRecovery scenario test (#2530) * Spawn child process to run DeviceLostRecovery scenario test * Layer dev paulm (#2536) ori said yes * add missing namespace to winml_trace_logging_provider in lotusenvironment.h (#2542) * Handle exception thrown from all apis in WinMLAdapter (#2539) * various changes to unblock windowsai ADO build * Fix custom ops scenario tests (#2562) * Do not shutdown protobuf after ort environment gets destroyed. Lazy load lotus environment first time it is needed * comment typo * pr comment about calling phoenix singleton * Make lotus_environment static in winmladapter * Layer dev paulm (#2567) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * making windowsAI azure dev ops work. * code review comments. * revert changes * Cmake and preprocessor fixes that where uncovered by building on agents without DML available via SDK * Layer dev dml delayload (#2580) * Brianma/cpu (#2583) * don't include dml stuff in cpu builds * tests that link the image lib also need the telemetry lib now * Throw Winml_err_invalid_binding if binding gpu resource on cpu device (#2589) * Throw Winml_err_invalid_binding if binding gpu resource on cpu device * PR comments. No need to query executionprovider for is gpu device * User/xianz/ortthrow (#2596) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * User/xianz/ortthrow (#2599) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * add status error message * Remove uwp onsuspending winrt call because logruntimeperf is getting removed (#2630) * User/xianz/dedup telemetry (#2631) * investigate duplication of telemetry in winml and ort * remove winml telemetry events * telemetry executionProviderEvent * remove unneccessary file and refactor code little bit * Revert back TelemetryEvent, which send up ETW event. * merge changes from layer_dev to windowsai (#2638) * Remove underscore from googletest names (#2616) * Fix leaking memory allocator Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24278761 and https://microsoft.visualstudio.com/OS/_workitems/edit/24330198 * Explicitly initialize Ort::Value with nullptr * Cache WinML adapter * bad merge * define private version of dxcore enum that is added in 19H1 SDK. (#2654) * add comment for explaning private definition of dxcore d3d feature level ennum value. (#2672) * do not package directml.pdb for redist packages. (#2676) * Fix leaking operator registry (#2645) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24354916 * User/orilevari/windowsai master merge (#2674) merge resolutions included pulling in telemetry logic that was merged to master and not windowsai and dereferencing InferenceSession::sessionstate now that it is a unique pointer * Delete Ort Allocator in LearningModelBinding (#2653) * Delete OrtAllocator in LearningModelBinding * PR comments to make Ort::Allocator a smart pointer * Small comment change * PR feedback to clean up code * PR feedback on move semantics * Clean up std::move * Fix memory leaks (#2679) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24356109, https://microsoft.visualstudio.com/OS/_workitems/edit/24388361 and https://microsoft.visualstudio.com/OS/_workitems/edit/24388596 * various changes to properly organize and skip GPU tests. For now for No DML builds we will not run GPU tests at all. In the future we should adapt the tests to expect the appropiate errors. (#2695) * Windowsai without fi (#2701) * Disable Attention fusion tests when DISABLE_CONTRIB_OPS is defined (#2529) * Setup java ci (#2528) * Add provision in ORT for session options to be parsed when available via model file (#2449) * Initial commit * Fix gitmodules * Nits * Nits * Updates * Update * More changes * Updates * Update * Some updates * More changes * Update * Update * Merge * Update * Updates * More changes * Update * Fix nits * Updates * Fix warning * Fix build * Add comment * PR feedback * PR feedback * Updates * Updates * Update * More changes * Fix build break * Comment test for now * Updates * Updates * PR feedback * Updates * Nits * Add tests * Fix build * Fix build * Fix build * Fix build break * Fix build * Nits * PR feedback * More change * Expose GetSessionOptions in pybind logic and add unit test for python * Fix build * PR feedback * PR feedback * Revert "Disable thread pool creation when enabled OpenMP (#2485)" (#2535) This reverts commit 7c7d5a149c9ed52eec67304bae5c4b132166a8a1. * Add dynamic shape support in TensorRT execution provider (#2450) * remove onnx-tensorrt submodule * add new onnx-tensorrt submodule (experiment) for trt6 * update engine build for trt6 * update compile and compute for tensorrt6.0 * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * switch to onnx-tensorrt master for TensorRT6' * Update tensorrt_execution_provider.cc * Handle dynamic batch size and add memcpy in TensorRT EP * update test cases * Update tensorrt_execution_provider.cc * update onnx-tensorrt submodule * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.ubuntu_tensorrt * Update run_dockerbuild.sh * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update concat_op_test.cc * Update tensorrt_execution_provider.cc * Upgrade TensorRT to version 6.0.1.5 * Update onnxruntime_providers.cmake * Update CMakeLists.txt * Update reduction_ops_test.cc * Update install_ubuntu.sh * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.tensorrt * Update BUILD.md * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update onnxruntime_providers.cmake * Update install_ubuntu.sh * Update install_ubuntu.sh * Update gemm_test.cc * Update gather_op_test.cc * Update CMakeLists.txt * Removed submodule * update onnx-tensorrt submodule * update header file * Removed submodule * add submodule onnx-tensorrt kevin's branch shape-test' * add debugging code * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * merge master * Removed submodule * update onnx-tensorrt submodule * add more changes for dynamic shapes * Update tensorrt_execution_provider.cc * update for dynamic shape * update dynamic shape processing * fix logger issue * remove submodule onnx-tensorrt * add submodule onnx-tensorrt * add env variable min_subgraph_size * remove redundency * update document * use onnxruntime::make_unique * fix multi-run issue * remove some tests to save CI build time * Add dynamic shape test * Update TensorRT-ExecutionProvider.md * Add example of running Faster R-CNN model on TensorRT EP * Add more details on env variables * update environment variables * Update tensorrt_basic_test.cc * Update model tests * Update tensor_op_test.cc * remove --use_full_protobuf * Update build.py * User/xianz/telemetry (#2458) * enabme telemetry * enable telemetry * set enable telemetry as default * for debugging * remove log and set disable telemetry as default back * delete private file while testing * resolve comment: mainly add license header, rename macro and update docs * rewording in privacy.md * Fix integer overflow in cuda NonMaxSuppression implementation (#2540) * add test case that should pass but fail * fix nms * extract int_max_output_boxes_per_class * Introduce container type runtime checks and other improvements (#2522) Rework TensorSeq in a manner consistent with Tensor and SparseTensor in terms of type system setup. Reduce templating. Introduce helpers to ensure the same data type. Make OrtValue __dtor not virtual. Introduce ContainerChecker * Fix C API tests for centos and mac (#2544) * change c++14 to c++11 * add ld lib path for centos * enable csharp tests on macos * fix C API test on MacOS + fix manylinux dotnet install * fix manylinux dotnet install * fix lib link * Add back executable bit to build.py * Fix a bug handling negative begin pad values in Pad op (#2550) * Fix bug in Pad op * Update * DNNL CMAKE update (#2548) * Fix android build (#2558) * Update win-x86-ci.yml (#2557) Fix build pipeline break * Re-enable Windows C# tests (#2564) * disable onnx_test_runner -x invocations for dnnl (#2568) * Allow sequence length to be symbolic (#2559) * setup java ci mac (#2570) * make layernorm fusion to support opset 11 (#2545) * Fix a warning found in the latest VS release * Add more check on SkipLayerNorm and BiasGelu fusion (#2574) * Fix file not found error during docker build. (#2569) * Add ConvTranspose1D (#2578) * Ryanunderhill/packagename test (#2582) * [Nuphar EP] fixes for some object detection models (#2581) Update notebook tutorial with multi-threaded int8 GEMM from #2517 * EmbedLayerNormalization Fusion Improvement (#2553) Embedding layer norm fusion improvements - add more checks * Update version (#2584) * Temporarily exclude vgg19 test from Python backend test 1. temporarily exclude vgg19 test which comsumes too much memory, run out of memory on Upsquared device. Single test pass for vgg19, need furture investigation (#2588) 2. Update docker file to decrease the docker image size * Update docs for Android NNAPI EP (#2586) * Fix lto bug for protobuf and ubuntu * add path to build dir before test run (#2590) * Add missig env variables for mac pipeline test (#2595) * Fixed an issue in updating realized dims (#2597) when we update realized dims for scan's output, the sliced axis also needs to be inclusive, i.e. we should check with "dim >= insert_inclusive_axis", because the offset in the symbols are based on Scan sugraph. Otherwise, we would end up with shape mismatch later. * Java API for onnxruntime (#2215) * Add support for opset 11 in reshape fusion (#2592) Support opset verion 11 in reshape fusion * Rename automl python tools folder to featurizer_ops. (#2593) * Support opset 11 subgraph of Squad model in Embed Layer Normalization (#2605) Support opset 11 Squad model that is exported from PyTorch nightly. The embed layer uses Range op which is missed in the transformer. * symbolic shape inference: fix warnings in GPT-2 model (#2608) And revise nuphar perf test on BERT squad * Dump subgraph ID and fused graph ID (#2607) * Dump subgraph ID and fused graph ID Dump subgraph ID and fused graph ID for better debugging * Remove local static fused_count added a field global_fused_count_ to NupharExecutionProvider class * EmbedLayerNormalization Fusion For Dynamic Squad Model Opset 10 (#2613) Support subgraph of SQuAD model exported from pytorch with dynamic input axes * Allow providers to be set for InferenceSession at construction (#2606) * Remove unnecessary parameter in some places in GatherElements implementation (#2612) * Remove unnecessary parameter in some places * Update * Update * Make sure fenced tensor could not reuse other tensor. (#2561) Fix random error caused by this. * Improve Embed Layer Norm Fusion for SQuAD with static input shape (#2621) * fix float16 comparison in initializer (#2629) * epsilon attribute for layernormalization fusion (#2639) * removed unnecessary batch file and fix path (#2640) * Add shape inference to ConvTransposeWithDynamicPads schema (#2632) * Improve cuda expand() opeator's performance. (#2624) * Cuda pad optimize when no padding is needed. (#2625) * Shortcut cuda Pad() when no padding is needed. * Optimize cuda scatter() on 2D compatible. (#2628) * Optimize cuda scatter() on 2D compatible. * Add some comments. * fix build error for ARM (#2648) * Improve performance of resize() in Nearest mode (#2626) Special treatment for 2D, check same size as input image. And in 2d kernel, template use_expolation. * Fix memory exception in Layer Norm Fusion (#2644) * Windows CI changes(#2650) * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Revert "Windowsai without fi (#2701)" This reverts commit 285d4c85ff5c4e265f963208170304ef3461e684. * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Deref unique pointer for session_state * send shutdown event when dll is unloaded and EvaluationStop, SessionC… (#2704) * send shutdown event when dll is unloaded and EvaluationStop, SessionCreationStart Events. * Add EvalutationStart Event * add comment * use correct type for for loop (#2755) * ARM CI (#2759) * Set ARM agent pool * Set CMake generator to VS 2019 in ARM * Use system-wide CMake instead of custom version Our custom version is too old for VS 2019 * Use DML and build shared lib in ARM CI * Restore nuget packages in ARM CI * Disable DML * Refactor ARM debug/release builds * Use system packaged Python version * Remove hardcoded Python path * Downgrade Python to 3.7 for build * Remove explicit CMake path * Fix invalid JSON in cgmanifest.json (#2760) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * Remove usage of WHOLEARCHIVE in WinML CMake and add WinMLAdapterFactory (#2726) * Remove usage of WHOLEARCHIVE in WinMLAdapter CMake and add WinMLAdapterFactory * PR feedback, no need for dll(export) since using def file * PR comments * Small comment in gen_def.py * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * Move winml_provider_factory.h to proper location (#2801) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Filter CPU case for IsFloat16Supported (#2802) * Merge fixes * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * Fix warnings that cause build to fail * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * Make DML include path global (#2882) * Make DML include path global * Add generated cppwinrt headers to winml_lib_common * Integrate changes to WindowsAI to make ADO Build (#2886) * Revert "CMake cross-generator fixes (#2790)" This reverts commit dbe7d97fa1ab155f1309bced87199527e8f35bd2. * add additional suppress warning in onnx_proto * ignore /wd4996 warning * DML execution provider fixes * Revert "Revert "CMake cross-generator fixes (#2790)"" This reverts commit 1ae7b4bcbc02edc881ad28685da98e095dfceb17. * Update func signature of custom op function overloads * common devicehelpers fixes * Add pch.h for winml_lib_common * re-add winml_lib_common_dir/inc to include path for winml_adapter * User/orilevari/dml redist shared folder (#2890) * move dml nuget package directory up one level to make it shared between build flavors * Merge conflict fix * Revert "Merge conflict fix" This reverts commit 142fa72cf9ce4344ad717b50b7ea2b8582aadc7c. * Revert "Merge remote-tracking branch 'origin/master' into windowsai" This reverts commit 6e2126d46e5e5f564d65da37dd4f70c93dd81165, reversing changes made to b3f5583dc9249834b947c8ea905f6a98060d5bd6. * Make winml_test_common free of test macros (#2902) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Don't generate files outside the build root (#2914) * Don't generate files outside the build root * Add onnxruntime_EXTERNAL_DEPENDENCIES to WinML * Add DML depedency on RESTORE_PACKAGES * User/orilevari/fix yaml merge bugs (#2918) * Add winml test source parameter into cmake function (#2919) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Add arguments to unittest cmake functions * remove comment * Revert "Revert "Merge remote-tracking branch 'origin/master' into windowsai"" This reverts commit ade5abe72a4234fdbc3623093c61c02c6b0bdc26. * Fix breaks from merge with ORT master * Brianma/linux (#2917) * don't include windows.h in cross-plat header * add default case for switch statement * signed/unsigned mismatch fix Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> * User/sheilk/winml adapter c api (#2891) * Create winml adapter c api * fix build * make it build * move adapter into onnxruntime core/session * entry point not exported * minor changes * make model metadata work * make tests pass * implement all the model reflection apis on the adapter c abi * update the new ort interface to create a lotus ennvironment with a logging sink * start adding ort env * move all winml code into adapter folder/lib to isolate it * ensure a single logging manager at a time * start refactoring session * refactor session creation interface * add cpu and dml session option methods to adapter * finish session init * stub out interfaces in ort lib to perform similar mechanics of iinference session * enable profiling, and enable schema override * update session register graph transformers * turn back on custom registry for custom ops * Add sync api * add last c api stubs * should build... but all feature values are broken since this is in flight to moving all implementation details into ivalue * remove ep adapter header * Implement DML execution provider functions from adapter (#2846) * Implement DML execution provider functions from adapter * Use functions in OnnxruntimeEngine.cpp * make map/sequence type_infos freeable, and start implementing ivalue * make it build again * implement value methods * implement remaining methods * remove com adapter abi * check dml session * cache the allocator on ivalue * check if resource is cpu/gpu when access its mutable data * update tensor * mismatched parentheses * fix tensor base and binding obj * it evaluates tensors! sometimes... * minor fixes * enable gpu evals * wrapper all existing winml adapter apis with API_IMPL to try catch (#2854) * update winml... tensor strings are broken, need to template tensorbase to do different things for strings * make tensor strings work with 2 copies in/2 copies out * Fix tensor string and allocator bug * make maps work again... needs some fixes still * Make it build! * enable map inputs * map outputs * unbound outputs for sequences and maps * User/xianz/merge windowsai (#2883) * Packaging pipeline changes for VS 2019 (#2711) * Tiny fix to codegen * Simplify cache implementation and avoid static variables that may carry over between models * Extend DML kernels (#2641) * Additional DML operators * Check unsupported attributes and inputs * Address PR comments * Add kernel capability function used for partitioning, and re-enable stride-based int64 support based on value range * Fix test failures * Build fix * PR comments * Update Nuphar tutorial notebook (#2721) 1. Reflect int8 GEMV improvements for multi-threading from #2696 2. Add notes on multi-threading control using OpenMP 3. Add samples of running multi-isa AOT, and show int8 GEMM differences between AVX and AVX2 4. Add rnn_benchmark example to resolve #1993 * Add schema for new Qops (#2611) * Add schema for new Qops * adding shape inference + qlinearaveragepool * plus review comments * plus review comments * updates per review comments * plus review comments * [server] Add supposed for model_name and model_version as cli parameter (#2708) * remove 64bit warning message from python validation. (#2727) * MLAS: ARM64 build fix (#2734) fix bad usage of vreinterpret to cast vector element types * Fix broken python docs links (#2740) * Fix build on Mac OS (#2731) mac os ld doesn't support --while-archive, correct option is -all_load * fix ngraph wheel (#2737) * fix ngraph wheel 1.1.0 onnxruntime_ngraph wheel doesn't work * remove libdnnl.so in nGraph Libs * make it easy to compare * Split onnxruntime server to a separated folder (#2744) * Fix build for Python 3.8 (#2747) * Fix build for Python 3.8 * Update protobuf to 3.11.2 (#1928) Update protobuf to 3.11.2 (#1928) * Change default optimization level to All (from Basic) (#2745) * change default optimization level to All (from Basic) * fix test * fix c# test * Update numpy to 1.18 (#2758) * Update numpy to 1.18 * Pipeline changes for python 3.8 (#2753) 1. Pipeline changes for python 3.8 2. Fix a regression in setup.py which was just introduced in the previous commit. Please notice, we still haven't made python 3.8 + Windows + CUDA work. * Add basic stacktrace output for posix debug builds. (#2749) * [NupharEP] fix a race condition when multiple sessions running different models concurrently (#2772) * Revert "Change default optimization level to All (from Basic) (#2745)" This reverts commit 56bb503c2f26474b6613bcb2a198691a11dcef38. * Fix typo in error message (#2736) * Rename MKL-DNN to DNNL to fix broken link (#2730) * Fix nightly build version number issue * Pass BUILD_BUILDNUMBER to linux docker * Disable featurizers in python packages * Import more featurizers (#2781) Make kernels non-template. Add input constraint for learnt data. Add min_max_scalar_transformer, robust_scalar_transformer, inputation_marker_transfomer, label_encoder_transformer, missing_dummies_transformer along with tests. Advance Featurizers library commit. * Implement a more stable softmax (#2715) * Implement a more stable SoftMax e^x is represented as infinity if x is large enough, like 100.f. Infinity divided by Infinity is a NAN. Thus, softmax gets a NAN if one or more item are large enough. A math transform as below is leveraged to get a stable softmax: e^xi/(e^x1 + ...e^xn) = e^(xi - max) / (e^(x1 - max) + ... + e^(xn - max)) And for convenience, force max to 0.f if all xi are negative * Contributing: Fix a typo (#2784) * ACL EP GEMM improvements (#2780) When it is posible we use a fully connected layer instead of the gemm implementation. This will let the library use the best implementation based on the input data. * ACL EP convolution improvements (#2774) Added the optimized implementation for depthwise convolution for both ACL v19.02 and ACL 19.05. Also the pointwise convolution seems to be more optimal in the CPU implementation so we opted for that instead. * Add script for release Nuget validation (#2719) * Initial commit * Nits * Disable a test temporarily * Change working directory * Test * Add download python step * Test update * More changes * Fix space issue * Fix * Verify nuget signing * Fix * Spaces * PR feedback * Nit * Fix * Fix * Remove temporary changes * add uint8 support to where op (#2792) * Improve bert optimization script: (#2712) (1) Move input int64=>int32 conversion to embed layer fusion. (2) Output epsilon attribute for LayerNormalization fusion. * add session creation time cost. (#2798) * ML.NET team needs featurizers within a package (#2789) Add auto ml featurizers to Windows, MacOS as well as to GPU packaging-pipelines. * Initialize max of softmax with lowest of float (#2786) * MLAS: update SGEMM threading parameters (#2808) * add interface to copy batch tensors. (#2807) * add interface to copy batch tensors. * onnxruntime * speed up Windows TRT CI (#2811) * don't run cuda tests if building with tensorrt * remove unnecessary build options for win trt ci * refactor win gpu tensorrt ci yml * --numpy_version=1.17 * update * update * azcopy and cuda path * Update test data (#2356) * Add timeseries imputer transformer featurizer kernel (#2813) Make kernels non-template. Add input constraint for learnt data. Fixup tests. Add two more featurizers along with tests. Tests fail. min_max_scalar_transformer robust_scalar_transformer Fix tests serialized stream by prepending version bytes. Add inputation_marker_transfomer and the test. Fix up float/double type designations. Added label_encoder_transformer along with a test. string_throw case is broken at the momement. Fix labelencodertransfomer_test.cc string_throw case Rename maxabsscalertransformer_test.cc Add MissingDummiesTransformer along with the test. Update manifest. Add TimeSeriesImputerTransformer definition, implementation and tests * Fix memory leak in TRT (#2815) * fix memory leak issue * revert EP_FAIL on enueueV2 * Add manifest missing comma * Run static code analyzer on most of our code (#2817) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * update quantization doc (#2783) * update documentation for quantization script * plus some spell corrections * Filter CPU case for IsFloat16Supported (#2802) * update default optimization level + fix gemm_activation fusion (#2791) * update defualt optimization level + fix gemm_activation fusion * fix typo * add unit test and incorporate review comments * fix test comment * Fix dnnl wheel package name (#2823) * Append '-dnnl' to whl package name when --use_dnnl * Update build.py * Update Ubuntu & TensorRT version in README (#2820) Dockerfile.tensorrt is using nvcr.io/nvidia/tensorrt:19.09-py3 as base Image, update Ubuntu and TensorRT version according to https://docs.nvidia.com/deeplearning/sdk/tensorrt-container-release-notes/rel_19-09.html#rel_19-09 * Merge fixes * Add OneHotEncoder and HashOneHotEncoder kernels. (#2830) Add defs and imlementation for OneHotEncoders, adjuist date_time_transformer kernel and test. Add OneHotEncoder kernel test. Add HashOneHotVectorizerTransformer unit test. This does not link due to multiple definitions of functions that are included into header from a CPP file. * Upgrade gtest to the latest version (#2827) WinML would like to update the googletest submodule. They want some newer features (namely GTEST_SKIP to skip tests programmatically and be able to skip entire fixtures easily) and would need to update the submodule version. However, because the new version of code hit a bug in gcc, even though the bug is already fixed in the latest gcc but we're using gcc 4.8.x and it won't get patched for the bug, so we have to do a compromise, change our code a little bit to make it work. The gcc bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=51213 * Add support for int64_t for topk CPU. Fixes github issue #2806. (#2833) * Ignore allocator type in ExecutionProviders allocator map. Make default initialization of OrtMemoryInfo more clearly invalid. (#2768) * Remove allocator type from the key comparison in ExecutionProviders. Remove usage of DummyArena as it's no longer necessary. * Fix x86 tests where arena allocator is disabled. Make initialization of OrtMemoryInfo clearer by adding Invalid enum value. * Make OrtValueNameIdxMap::MaxIdx more intuitive. * Convert ExternalProject Featurizers into git submodule (#2834) Add git submodule for Featurizer library. Update cmake to build for git submodule. * add domain check for nodes + update documentation (#2831) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Fix Linux CUDA nuget packaging pipeline break * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * update optimization doc for BERT related fusions (#2819) * Add bert related transformers to doc * Add execution provider and comment for bert optimizations * Add comment about accuracy impact of approximation * Fix warnings that cause build to fail * MLAS: enable threading for quantized GEMMs (#2844) * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * remove old winmladapter cpp Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * move sequence implementation into ort lib... still commented out... need to turn back on... * begin sequence implementation * make maps and sequences work * fix broken tests * remove dead code * misc cleanup * CR feedback * User/xianz/winml adapter c api (#2869) * wrapper all existing winml adapter apis with API_IMPL to try catch * Return HR or Throw for WinML adapter APIs if failed * undo macro wrapper for two places * Wrap error macros around ort apis, too. * address CR feedback #2 * add more api throw/return macros * Revert changes no longer needed * revert changes to cxx api * format winml lib.ort and winml adapter * remove static pheonix singleton Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * missing use_dml check in winml_adapter_session (#2930) * --use_dnnl flag was mangled in merge (#2931) * use dml macro not wrapping custom registry code (#2934) * Disable LNK4199 winml_dll to enable cuda builds (#2936) * Disable LNK4199 in winml_dll * linkler->linker * LearningModelSessionAPITestGpu.CreateSessionWithCastToFloat16InModel should return DXGI_ERROR_UNSUPPORTED when FP16 not supported (#2937) * Disable LNK4199 in winml_dll * linkler->linker * Need to return DXGI_ERROR_UNSUPPORTED when Model does not support fp16 * Publish build symbols (#2939) * Publish build symbols * Don't upload PDBs for .exe files * Make x86 build (#2943) * fix last remaining size_t/int64_t warnings->errors (#2948) * TensorString, Sequences and Maps use the first allocator, but should use the cpu default allocator. (#2952) * fix tensor string allcoator * clean up default allocator usage for strings in winml lib/api.ort Co-authored-by: Ryan Lai <ryalai96@gmail.com> * Handle tensor shape of zero (#2954) Co-authored-by: Ryan Lai <ryalai96@gmail.com> * CR feedback (#2970) * CR feedback * fix weird formatting on privacy readme * Add 'All rights reserved.' everywhere * readd all rights reserved to winml_provider_factory.h * remove extra space in comment * remove extra whitespace * fixes post master merge * remove winml from nuget gpu pipeline * set IR VERSION on generated_model in rnn_benchmark (#2972) * Fix slice conformance failures (#2908) Co-authored-by: Adrian Tsai <adtsai@microsoft.com> Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Paul McDaniel <paul_mcdaniel@hotmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Dwayne Robinson <fdwr@hotmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com>
2020-02-05 01:12:19 +00:00
[marshaling_behavior(agile)]
[dualapipartition(1)]
runtimeclass TensorFeatureDescriptor : ILearningModelFeatureDescriptor
{
//! Returns the data type of the tensor.
TensorKind TensorKind{ get; };
//! Returns the count and size of each dimension.
Windows.Foundation.Collections.IVectorView<Int64> Shape{ get; };
}
Sync ORTModule branch with master and fix tests (#6526) * Deprecate Python global configuration functions [Part 1] (#5923) Enable options to be set via execution provider (EP)-specific options and log deprecation warning from current global configuration functions. * remove dnnl_dll_path from post build copy (#6142) * Model Fusion For Bart (#6105) Fusion fix for Bart models * Unify IExecutionProvider and IExecutionProviderFactory interfaces (#6108) * Remove Provider_IExecutionProvider and make the internal IExecutionProvider usable by shared providers * Change Provider_IExecutionProviderFactory to be the core version. * Enable running the mnist_training sample without cuda (#6085) Signed-off-by: George Nash <george.nash@intel.com> * nnapi add min max support (#6117) * Fix CUDA test hang: (#6138) - Make condition check in `CUDAAllocatorTest` to ensure CUDA device is present. * Fix TensorRT kernel conflict issue for subgraphs of control flow operators (#6115) * add static subgraph kernel index * change kernel naming to avoid conflicts * Add gradient registration for Abs. (#6139) * Partition initial optimizer state for Zero-1 (#6093) * Initial changes * Working changes * Working changes * Cleanup * fix windows CI * Review comments * review comments * Fix edge case in BFCArena where allocation failures could lead to an infinite loop. (#6145) #4656 * Revert "work around of the build break in mac (#6069)" (#6150) This reverts commit 3cae28699bed5de1fcaadb219fa69bae0fc3cee8. * Fix clean_docker_image_cache.py detection of image pushes. (#6151) Fix clean_docker_image_cache.py detection of image pushes. They were being ignored because the expected HTTP status code was wrong. For pushes, it's 201 instead of 200. * MLAS: add NEON version of int8 depthwise convolution (#6152) * Using a map of of ops to stages as input of partition function. (#5940) * New partition algorithm running before AD * Convert cut_group_info into device map. Work in progress -- works for bert-tiny with pp=2 * Removing code for partition of bwd graphs * Remove old code * Adding some verification code * Handle Shared Initializer * Renaming rank with stage * Added first unit test * new test * redundant check * undo change in bert * Moved cut-based partition to testing utils file Co-authored-by: xzhu1900 Co-authored-by: wschin * New conversion function and tests * minor * remove test that is not needed2 * improve GetDeviceAssignment and PR comments * minor changes * PR comments * improving documentation and variable naming * add documentation * Variable naming and docs * more doc improvements * more doc improvements * missing static cast * Fix test file for windows * Fix test file for windows * Fix test file for windows * stage id is not the same as rank id * PR comments * PR comments * More comments * More comments * Minor fix to satisfy c++14 (#6162) * Deprecating Horovod and refactored Adasum computations (#5468) deprecated horovod submodule refactored adasum logic to be ort-native added tests for native kernel and e2e tests * Update TensorRT-ExecutionProvider.md (#6161) * Bugfix for topk cuda kernel (#6164) * fix the issue that std::numeric_limits cannot handle half type * adding a test Co-authored-by: Du Li <duli@OrtTrainingDev4.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net> * Revert "Fuse MatMulIntegerToFloat only when scales are scalar (#6008)" (#6169) This reverts commit f2dcba7afe0d42ebdaaef0c6cdf913a1156c9e98. * Remove ignored build warnings for pybind on Mac (#6165) * save_checkpoint, load_checkpoint and aggregate_checkpoints (#6136) * save_checkpoint and load_checkpoint implementations * checkpoint aggregation logic * unit tests for save_checkpoint, load_checkpoint and aggregate_checkpoints * Don't try to bind unused inputs in the Training frontend (#6166) * Update documentation for contributing a PR and add deprecation notices for PyOp and ORT server. (#6172) * aggregate model states only for the case when mixed precision was true (#6176) * [NNAPI EP] Enable per-channel quantization for QlinearConv (#6155) * Enable qlinearconv per-channel quantization * Fix the android CI test failure * Add Android Version Check for Per-Channel Quant * Address PR comments * Fix some minor issues * Add verification of per-channel zero points * Make the error tolerance configurable * Fix typo in BERT pretraining script (#6175) A misplaced `}` meant that the `'enable_adasum'` option was interpreted incorrectly, causing the test to fail. * Update get_docker_image.py to enable use without image cache container registry. (#6177) Update get_docker_image.py to enable use without image cache container registry. * Helper for compiling EP to generate deterministic unique ids for use in MetaDef names (#6156) * Create a helper for generating unique ids that can be used by an EP that creates compiled nodes and needs ids to be deterministic for a model when used in multiple sessions. Added to IExecutionProvider as this can potentially be used by all compiling EPs and is more robust than a simplistic counter (although EP implementer is free to choose either approach). * Restructure the helper so it can be called across the EP bridge. Add ability to call id generation helper from EP bridge - convert DNNL EP to use helper to validate Address issue where a new Model may be loaded into the same address as a previous one. - hash the bytes in the Graph instance (1728 bytes currently) to use as the key to the full hash for the model Add lock around id generation to ensure no issues if multiple sessions partitions graphs at exactly the same time. - Extremely unlikely but would be hard to debug and the locking cost is not an issue as it's only incurred during graph partitioning and not execution. * Backend APIs for checkpointing (#5803) * Add backend API GetOptimizerState and GetModelState * add GetPartitionInfoMap * Android coverage dashboard (#6163) * Write the report to a file. * Post code coverage to the Dashboard database. * Add usage details of unified MCR container image (#6182) Going forward, a single unifed docker image will be published in MCR. The hardware accelerator target choice will have to be made in the application using OpenVINO EP's runtime config options. * improve perf for softmax (#6128) * improve perf for both gathergrad and softmax * revert the change in gathergrad and will be done in another PR. * address comments from code review. * Tune fast Gelu to use exp(x) instead of tanh(x) on Rocm platform (#6174) * tune fast gelu to use exp(x) instead of tanh(x) on rocm * update to use expression 2/(1+exp(-2x))-1 for stability * Add Status.csv to EP Perf Tool (#6167) * merge master, keep postprocess status commit * download float16.py everytime * removing hardcoded values * Lochi/quantization tool for trt (#6103) * Initial implementation of generating calibration dynamic range table * Initialize validation support for Quantization * Initialize validation support for Quantization (cont.) * Improve validation support for Quantization * Improve validation support for Quantization * Rewrite/Refine for calibration and validation * Rewrite/Refine for calibration and validation (cont.) * Refine code * Refine code * Add data reader for BERT * Add flatbuffers to serialize calibration table * Refine code and add BERT evaluation * Refine the code * minor modification * Add preprocess/postprocess of vision team yolov3 and refine the code * Update annotation * Make bbox cooridates more accurate * Fix bug * Add support of batch processing * Batch processing for model zoo yolov3 * Add batch inference for evaluation * Refine the code * Add README * Add comments * Refine the code for PR * Remove batch support checking in data_reader and refine the code * Refine the code for PR * Refine the code for PR review Co-authored-by: Olivia Jain <oljain@microsoft.com> * Implement ScatterND for CUDA EP (#6184) * Condition fix in Resize operator (#6193) * Clean up checkpoint tests to use the new checkpoint functions (#6188) * add deprecation warning for old checkpoint functions * update all the distributed checkpoint tests to use new checkpoint functions * Implement comparing outputs that are sequence of maps of strings to floats (#6180) * Implement conversion from ortvalue to Itensor for string tensors and comparing sequence of maps of strings to floats * PR comments * Dockerfile to build onnxruntime with ROCm 4.0 * Add ability to skip GPU tests based on GPU adapter name (#6198) * Implement conversion from ortvalue to Itensor for string tensors and comparing sequence of maps of strings to floats * PR comments * Add ability to skip gpu tests according to adapter description * spacing * spacing * spacing * Openvino ep 2021.2 (#6196) * Enabling fasterrcnn variant and vehicle detector * changes for 2021_2 branch * yolov3_pytorch commit * fixed braces in basic_backend.cc * ci information added * faster rcnn variant and vehicle detector changes were made in 2021.1 and not in 2021.2 * some changes to support unit tests * disable some tests which are failing * fix myriad tests for vehicle detector * Did some cleanup *cleaned up comments *Disabled Add_Broadcast_0x1 and Add_Broadcast_1x0 tests on MYRIAD_FP16 backend due to a bug *cleaned up capability_2021_2.cc file *Removed extra conditions which were added for some validation in backend_utils Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com> * yolov3 pytorch workaround to ensure that the output names are matched * gemmoptest fixed on myriad * Fixed MYRIADX CPP Test Failures *Expand,GatherND,Range,Round op's are only supported in model *where op with float input data types are not supported and fixed *Scatter and ScatterElements op's with negative axis are fixed *Reshape op with 0 dim value are not supported and fixed *Disabled InstanceNorm_2 test on MYRIADX Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com> * make changes to yolov3 pytorch * Fixed python unit tests *Fixed failing python tests on vpu, GPU and CPU Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com> * Fixes POW op failures on GPU_FP16 Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com> * Clean up capability_2021_2.cc Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com> * Updated docx for MultiThreading option *Added extra info on setting the num_of_threads option using the API and it's actual usage Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com> * fixed slice and removed extra prints * Disabled failing python tests Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com> * Minor changes added in capabilty_2021_2 Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com> * made changes to slice to avoid failures * Disabling FP16 support for GPU_FP32 ->Inferencing an FP16 model on GPU_FP32 leads to accuracy mismatches. so, we would rather use GPU_FP16 to infer an FP16 model on GPU Device Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com> * Updated docx for Inferencing a FP16 Model Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com> * fix for mask rcnn * Script for installing openvino from source * Updated with openvino 2021.2 online installation * code comment fixes fixed accuracy mismatch for div * Update OpenvinoEP-ExecutionProvider.md updated for 2021.2 branch * Update README.md updated dockerfile documentation * Update BUILD.md build.md update documentation * permissiong change of install_openvino.sh * made changes to align with microsoft onnxruntime changes * Updated with ov 2021.2.200 Co-authored-by: suryasidd <surya.siddharth.pemmaraju@intel.com> Co-authored-by: sfatimar <sahar.fatima@intel/com> Co-authored-by: MaajidKhan <n.maajidkhan@gmail.com> Co-authored-by: mohdansx <mohdx.ansari@intel.com> * Fix a memory leak in test_inference.cc (#6201) * Fix a memory leak in test_inference.cc * Use TArray in AMD element-wise kernels, rather than manually copying memory to device. * Remove most ROCm-specific element-wise code and reuse CUDA element-wise code. * Minor change to improve performance for operator Pad. (#5537) * small improvment for pad * Support double for operators Log, Reciprocal, Sum (CPU) (#6032) * Support double for operators Log, Reciprocal, Sum * remove tesdt erf_double * Support double for operators Where, LpNormalisation (#6034) * Support double for operators Relu, Tanh, Sigmoid (#6221) * Fix ImportError in build.py (#6231) There is a possible ImportError where build.py can import the wrong 'util' package if there are others present in `sys.path` already * Removed executor todo that looks dead. (#6234) * Remove MKLML/openblas/jemalloc build config (#6212) * Remove python 3.5 * Update the readme file * Upgrade build.py to assert for python 3.6+ Upgrade build.py to assert for python 3.6+ as python 3.5 cannot build anymore todays master. * Support MLFloat16 type in Pow opset-12 CUDA kernel (#6233) * MLAS: handle MlasGemm(M/N/K==0) cases (#6238) * Support double for operator TopK + fix one bug in TopK implementation for GPU for double (#6220) * Support double for operator TopK * add static classes for topk/double * fix cast issue in topk * Support double for operator Gemm + fix bug in gemm implementation for cuda, rocm when sizeof(type) != sizeof(float) (#6223) * Support double for operator Gemm * fix type size while copying data in gemm operator for GPU * fix type in gemm implementation for rocm * Support double for operator ReduceMean, ReduceLogSumExp (#6217) * Support double for operators ReduceMean, ReduceLogSumExp * Support double for operator ArgMin (#6222) * Support double for operator ArgMin * add test specifically for double * add new test on pai-excluded-tests.txt * Update BUILD.md * Update manylinux docker image to the latest (#6242) * Fix allocator issue for TensorRT IOBinding (#6240) * Fix issue: https://github.com/microsoft/onnxruntime/issues/6094 Root cause: we didn't expose the OrtMemoryInfo for TRT, so it will cause issue if user want use IObinding for Tensorrt. Short term fix, add the OrtMemoryInfo for TRT. Long term should unify the allocator for CUDA and TRT * Tune BiasGeluGradDx kernel in approximation mode to avoid tanh(...) on Rocm (#6239) * bias gelu grad use exp(...) instead * update cuda to rocm * missing semicolon * comment * remove dockerfile * missing factor of two * Refactor EP Perf Tool (#6202) * merge master, keep postprocess status commit * download float16.py everytime * using variables to reference eps * adding ACL EP to ep perf tool * accuracy with absolute tolerance configurable * add acl to dict + remove commented line * Documentation for distributed CI tests pipeline (#6140) * Remove a debug log in provider_test_utils.cc (#6200) * Add the Concat Slice Elimination transform, fix constant_folding transform (#5457) * Add concat slice transform + test * Cosmetic improvements in concat slice transform * Remove unrelated file, fix comment, fix constant folding bug * Add test onnx graph * fix windows build * Review comments * review comment Co-authored-by: Aishwarya <aibhanda@OrtTrainingDev4.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net> * Add MakeStringLite which uses current locale, update some MakeString call sites to use it instead. (#6252) * Add MakeStringLite which uses current locale, update macros to use that to generate messages. * Convert calls to MakeStringLite(). * Liqun/speech model loop to scan (#6070) Provide a tool to convert Loop to Scan for Nuphar performance Fix Nuphar CI pipeline failures. Co-authored-by: liqun <liqun@OrtTrainingDev4.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net> * model parallel refinement (#6244) * Megatron Transformation as a seperate step * remove useless header * clang formating * Re-Structure megatron transformer for subsquent changes * fix comments * Allow querying a GraphProto's doc_string as part of ModelMetadata (#6248) * Fix Linux/Mac error message on input type mismatch (#6256) * add bfloat16 to gathergrad type constrains (#6267) Co-authored-by: Cheng Tang <chenta@microsoft.com> * Fix VS 2017 build break (#6276) * Deprecate Python global configuration functions [Part 2] (#6171) Update Python API to allow more flexibility for setting providers and provider options. The providers argument (InferenceSession/TrainingSession constructors, InferenceSession.set_providers()) now also accepts a tuple of (name, options dict). Fix get_available_providers() API (and the corresponding function in the C API) to return the providers in default priority order. Now it can be used as a starting point for the providers argument and maintain the default priority order. Convert some usages of the deprecated global configuration functions to use EP-specific options instead. Update some EP-specific option parsing to fail on unknown options. Other clean up. * Add script to preprocess python documentation before publishing (#6129) * add script to preprocessing python documentation before publishing * rename past to past_key_values for GPT-2 (#6269) rename past to past_key_values for transformers 4.* * Rename MakeString and ParseString functions. (#6272) Rename MakeString to MakeStringWithClassicLocale, MakeStringLite to MakeString, *ParseString to *ParseStringWithClassicLocale. Add missing pass-through versions of MakeStringWithClassicLocale for string types. * Increase timeout for Linux GPU CUDA11 build. (#6280) * Add helper to compare model with different precision (#6270) * add parity_check_helper.py * add real example * remove lines * Fix Min/Max CPU kernels for float16 type (#6205) * fix data_ptr assertion error for past_sequence_length=0 in GPT-2 (#6284) fix io binding crash for past_sequence_length=0 * A list of changes in transformers tool (#6224) * longformer fp16 e2e * add fp16/fp32 parity check helper file * excludes nodes with subgraph in profiling * use onnxconverter_common to do fp32->fp16 * add version check for onnxconverter_common * remove helper file * add pkg installation on notebooks and script * Workaround for static_cast<double>(half) * Add workaround to remove ROCm-specific binary-elementwise files. * Update nuget build (#6297) 1. Update the ProtoSrc path. The old one is not used anymore. 2. Regenerate OnnxMl.cs 3. Delete some unused code in tools/ci_build/build.py 4. Avoid set intra_op_param.thread_pool_size in ModelTests in OpenMP build. 5. Fix a typo in the C API pipeline. * Enable ONNX backend test of SequenceProto input/output (#6043) * assert sequence tensor and remove skips * update testdata json * use ONNX 1.8 in cgmanifest.json * use previous commit to workaround * update ONNX commit ID in docker * skip test_maxpool_2d_dilations test for now * update function name * add --sequence_lengths option (#6285) * more dtype for Equal CUDA kernel (#6288) Co-authored-by: Vincent Wang <weicwang@microsoft.com> * Force reinstall onnx python package on Windows (#6309) * update transformers required package versions (#6315) * Remove abs in LpPool (#6303) * Support 1D input for Conv + Mul/Add fusion optimizer with test (#6295) * Support 1D input (N C H) for Conv + Mul/Add fusion optimizer with test cases and test models. * Add longformer to python package (#6314) * add longformer to python package * move test related script and data to a new folder * Avoid false sharing on thread pool data structures (#6298) Description: This change adds alignment and padding to avoid false sharing on fields in the thread pool. It also adds a new microbenchmark to profile thread-pool performance over short loops. Motivation and Context MobileNet on a 2*12-core system showed a performance gap between the ORT thread pool and OpenMP. One cause appeared to be false sharing on fields in the thread pool: ThreadPoolParallelSection::tasks_finished (which the main thread spins on waiting for workers to complete a loop), and the RunQueue::front_ and back_ fields (used respectively by the worker thread and the main thread). The additional micro-benchmark BM_ThreadPoolSimpleParallelFor tests performance of loops of different sizes at different thread counts. The results below are on a machine with 2*14-core processors (E5-2690 v4) running with 1, 14, 15, and 28 threads. For each test, the microbenchmark has N threads run a loop with N iterations; hence a perfect result is for the time taken to be constant as additional threads are added (although we will also see power management effects helping at very low thread counts). The loop durations (100000, 10000, 1000) correspond roughly to 200us, 20us, and 2us on this machine. Before change: BM_ThreadPoolSimpleParallelFor/1/1/100000/real_time 17153 us 17154 us 32 BM_ThreadPoolSimpleParallelFor/14/14/100000/real_time 22553 us 22553 us 30 BM_ThreadPoolSimpleParallelFor/15/15/100000/real_time 21521 us 21521 us 29 BM_ThreadPoolSimpleParallelFor/28/28/100000/real_time 24111 us 24111 us 24 BM_ThreadPoolSimpleParallelFor/1/1/10000/real_time 1719 us 1719 us 407 BM_ThreadPoolSimpleParallelFor/14/14/10000/real_time 3409 us 3409 us 200 BM_ThreadPoolSimpleParallelFor/15/15/10000/real_time 3541 us 3541 us 201 BM_ThreadPoolSimpleParallelFor/28/28/10000/real_time 4576 us 4576 us 151 BM_ThreadPoolSimpleParallelFor/1/1/1000/real_time 174 us 174 us 4017 BM_ThreadPoolSimpleParallelFor/14/14/1000/real_time 1586 us 1586 us 402 BM_ThreadPoolSimpleParallelFor/15/15/1000/real_time 1586 us 1586 us 397 BM_ThreadPoolSimpleParallelFor/28/28/1000/real_time 2864 us 2864 us 232 After change: BM_ThreadPoolSimpleParallelFor/1/1/100000/real_time 17160 us 17160 us 33 BM_ThreadPoolSimpleParallelFor/14/14/100000/real_time 20989 us 20989 us 31 BM_ThreadPoolSimpleParallelFor/15/15/100000/real_time 22286 us 22286 us 31 BM_ThreadPoolSimpleParallelFor/28/28/100000/real_time 24631 us 24631 us 25 BM_ThreadPoolSimpleParallelFor/1/1/10000/real_time 1718 us 1718 us 407 BM_ThreadPoolSimpleParallelFor/14/14/10000/real_time 2868 us 2868 us 242 BM_ThreadPoolSimpleParallelFor/15/15/10000/real_time 2907 us 2907 us 240 BM_ThreadPoolSimpleParallelFor/28/28/10000/real_time 3872 us 3872 us 186 BM_ThreadPoolSimpleParallelFor/1/1/1000/real_time 175 us 175 us 3938 BM_ThreadPoolSimpleParallelFor/14/14/1000/real_time 933 us 933 us 659 BM_ThreadPoolSimpleParallelFor/15/15/1000/real_time 912 us 912 us 591 BM_ThreadPoolSimpleParallelFor/28/28/1000/real_time 1976 us 1976 us 317 * fix opset imports for function body (#6287) * fix function opsets * add tests and update onnx * changes per review comments * add comments * plus updates * build fix * Remove false positive prefast warning from threadpool (#6324) * Java: add Semmle to Java publishing pipelines (#6326) Add Semmle to Java API pipeline Add security results publishing and add Java GPU. * Quantization support for split operator with its NHWC support (#6107) * Make split working for quantization. * NHWC transformer support for split operator * Refactor some according to Feedback. Will add test cases soon. * Fix build error on windows. * Add test case for split op on uint8_t support * Add nhwc_transformer_test for split uint8_t support * Some change according to PR feedbacks. * Liqun/enable pipeline parallel test (#6331) enable pipeline parallel test Co-authored-by: liqun <liqun@OrtTrainingDev4.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net> * Use onnxruntime_USE_FULL_PROTOBUF=OFF for the cuda execution provider (#6340) This removes a special case of the cuda EP. * MLAS: add fallback implementation for quantized GEMM (#6335) Add a non-vectorized version of the kernel used for the quantized version of MlasGemm. * Delete float16.py (#6336) No longer needed. Also doesn't pass policheck. * Enable add + softmax fusion for Rocm platform (#6259) * add bias softmax; tests appear to pass * check fusion occurs for rocm as well * check for rocm provider compatible as well * build for cpu scenario as well * try again; broader cope * proper scope on kGpuExecutionProvider * been editing wrong file * remove commented #include lines * try again due to mac os ci error * try again * test fusion both cuda and rocm to avoid mac ci error * add external data support to tensor proto utils (#6257) * update unpack tensor utilities to support loading external data * more updates * fix test * fix nuphar build * minor build fix * add tests * fix Android CI * fix warning * fix DML build failure and some warnings * more updates * more updates * plus few updates * plus some refactoring * changes per review * plus some change * remove temp code * plus updates to safeint usage * build fix * fix for safeint * changed wording. (#6337) * Remove OpSchema dummy definition. Only needed for Function now, and we can just exclude the method in Function (#6321) * remove gemmlowp submodule (#6341) * [NNAPI] Add pow support (#6310) * Add support for running Android emulator from build.py on Windows. (#6317) * fix the pipeline failure (#6346) * Train BERT Using BFloat16 on A100 (#6090) * traing bert using bf16 * Adam support bf16 * bugfix * add fusedmatmul support * fix after merge from master. * bugfix * bugfix after merge from master * fast reduction for bf16. * resolve comments * fix win build * bugfix * change header file. Co-authored-by: Vincent Wang <weicwang@microsoft.com> * Fix DerefNullPtr issues raised by SDLNativeRules. (#6348) * update quantize to support basic optimization and e2e example for image classification (#6313) update the resnet50-v1 to standard one from onnx zoo. add an example for mobilenet run basic optimization before quantization fix a bug in Clip * Enable graph save for orttrainer (#6333) * Enable graph save for orttrainer * Fix CI * Update orttraining/orttraining/python/training/orttrainer_options.py * Update orttraining/orttraining/python/training/orttrainer_options.py * Update orttraining/orttraining/python/training/orttrainer_options.py * Update orttraining/orttraining/python/training/orttrainer_options.py * Update orttraining/orttraining/python/training/orttrainer_options.py Co-authored-by: Thiago Crepaldi <thiago.crepaldi@microsoft.com> * Add PREfast to python packaging pipeline (#6343) * Add PREfast to python packaging pipeline * fix longformer benchmark io_binding output_buffers (#6345) * fix longformer benchmark io_binding output_buffers * format * import benchmark_helper from parent directory. * Use readelf for minimal build binary size checks. (#6338) * Use readelf for minimal build binary size checks. The on-disk size grows in 4KB chunks which makes it hard to see how much growth an individual checkin causes. Only downside is that the sum of the sections is larger than the on-disk size (assumably things get packed smaller on disk and some of the section alignment constraints can be ignored) * Remove unused function * Java: Set C language warnings to W4 and adjust JNI code (#6347) Set /W3 for C language and fix up JNI warnings. * Pipeline Parallel Experimental Python API (#5815) * Add create session to WinML telemetry to track WinML Usage (#6356) * Fix one more SDL warning (#6359) * fix -Wdangling-gsl (#6357) * Add python example of TensorRT INT8 inference on ResNet model (#6255) * add trt int8 example on resnet model * Update e2e_tensorrt_resnet_example.py * remove keras dependency and update class names * move ImageNetDataReader and ImageClassificationEvaluator to tensorrt resnet example * simplify e2e_tensorrt_resnet_example.py * Update preprocessing.py * merge tensorrt_calibrate * Update calibrate.py * Update calibrate.py * generalize calibrate * Update calibrate.py * fix issues * fix formating * remove augment_all * This added telemetry isn't needed (#6363) * Wezuo/memory analysis (#5658) * merged alloc_plan * pass compilation * Start running, incorrect allocation memory info * add in comments * fix a bug of recording pattern too early. * debugging lifetime * fix lifetime * passed mnist * in process of visualization * Add code to generate chrome trace for allocations. * in process of collecting fragmentation * before rebuild * passed mnist * passed bert tiny * fix the inplace reuse * fix the exception of weight in pinned memory * add guards to ensure the tensor is in AllocPlan * add customized profiling * debugging * debugging * fix the reuse of differnt location type * add rank * add the rank * add fragmentation * add time_step_trace * Add summary for each execution step (total bytes, used/free bytes). * add top k * change type of top k parameter * remove prints * change heap to set{ * add the name pattern * add the useage for pattern * add partition * change to static class * add custom group * remove const * update memory_info * in process of adding it as runtime config * change the memory profiling to be an argument * add some comments * add checks to recored meomry_info in traaining session * set the "local rank setting" to correct argument. * addressing comments * format adjustment * formatting * remove alloc_interval * update memory_info.cc to skip session when there is no tensor for a particular memory type * fix memory_info multiple iteration seg-fault * consolidate mainz changes * fixed some minor errors * guard by ORT_MINIMAL_BUILD * add ORT_MEMORY_PROFILE flag * added compiler flag to turn on/off memory profiling related code * clean up the code regarding comments * add comments * revoke the onnx version * clean up the code to match master * clean up the code to match master * clean up the code to match master Co-authored-by: Jesse Benson <benson.jesse@gmail.com> Co-authored-by: Wei Zuo <wezuo@OrtTrainingDev3.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net> Co-authored-by: wezuo <wezuo@az-eus-v100-32gb-5-worker-mgtbby.eastus.cloudapp.azure.com> Co-authored-by: wezuo <wezuo@az-eus-v100-32gb-5-worker-yclzsf.eastus.cloudapp.azure.com> * Support MLFloat16 in CumSum Cuda op for Opset 14 (#6355) * Add CumSum-14 for Cuda * fix convert_common version retrival (#6382) * Refine auto_pad based pad computation in ConvTranspose (#6305) * Fix SDL warning (#6390) * Add max_norm for gradient clipping. (#6289) * add max_norm as user option for gradient clipping * add adam and lamb test cases for clip norm * add frontend tests * Add the custom op project information (#6334) * Dont use default string marshalling in C# (#6219) * Fix Windows x86 compiler warnings in the optimizers project (#6377) * [Perf] Optimize Tile CPU and CUDA kernels for a corner case (#6376) * Unblock Android CI code coverage failure (#6393) * fix build on cuda11 (#6394) Co-authored-by: Vincent Wang <weicwang@microsoft.com> * Load the model path correctly (#6369) * Fix some compile warnings (#6316) * OpenVino docker file changes to bypass privileged mode Description: Builds and installs libusb without UDEV support, which is used for communicating with the VPU device. Motivation and Context This enables the resulting docker container to be run without '--privileged' and '--network host' options which may not be suitable in deployment environments. * Megatron checkpointing (#6293) * Add bart fairseq run script * Add frontend change to enable megatron * Initial changes for checkpointing * Megatron optim state loading, checkpoint aggregation, frontend distributed tests for H, D+H * Add load_checkpoint changes * Fix CI * Cleanup * Fix CI * review comments * review comments * review comments: * Fix generate_submodule_cgmanifest.py Windows issues. (#6404) * Continue memory planning when unknown shape tensor is encountered. (#6413) * Reintroduce experimental api changes and fix remote build break (#6385) Co-authored-by: Ori Levari <orlevari@microsoft.com> * Add support for custom ops to minimal build. (#6228) * Add support for custom ops to minimal build. Cost is only ~8KB so including in base minimal build. * enable pipeline to run quantization tests (#6416) * enable pipeline to run quantization tests setup test pipeline for quantization * Minor cmake change (#6431) * Liqun/liqun/enable pipeline parallel test2 (#6399) * enable data and pipeline parallism test Co-authored-by: liqun <liqun@OrtTrainingDev4.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net> * Farewell TrainableDropout (#5793) * Deprecate TrainableDropout kernel. * Update bert_toy_postprocessed.onnx to opset 12. * Add more dropout tests. * Fix BiasDropout kernel. Co-authored-by: Ubuntu <OrtTrainingDev3@OrtTrainingDev3.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net> Co-authored-by: Sherlock Huang <bahuang@OrtTrainingDev3.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net> Co-authored-by: Sergii Dymchenko <sedymche@microsoft.com> * fix null dereference warning (#6437) * Expose graph ModelPath to TensorRT shared library (#6353) * Update graph_viewer.cc * Update tensorrt_execution_provider.cc * Update graph_viewer.h * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update provider_api.h * Update provider_bridge_ort.cc * Update provider_interfaces.h * Update provider_interfaces.h * expose GraphViewer ModelPath API to TRT shared lib * add modelpath to compile * update * add model_path to onnx tensorrt parser * use GenerateMetaDefId to generate unique TRT kernel name * use GenerateMetaDefId to generate unique TRT engine name * fix issue * Update tensorrt_execution_provider.cc * remove GetVecHash * Update tensorrt_execution_provider.h * convert wchar_t to char for tensorrt parser * update tensorrt parser to include latest changes * fix issues * Update tensorrt_execution_provider.cc * merge trt parser latest change * add PROVIDER_DISALLOW_ALL(Path) * add tool for generating test data for longformer (#6415) * only build experimental api in redist (#6465) Co-authored-by: Sheil Kumar <sheilk@microsoft.com> * Add an option to save the training graph after optimization (#6410) * expose optimized_model_filepath in SessionOptions as `debug.graph_save_paths.model_with_training_graph_after_optimization_path` in `ORTTrainerOptions` * Share allocator between CUDA EP & TRT EP. (#6332) * Share allocator between CUDA EP & TRT EP. limitation: 1. Does not cover the per-thread allocator created by CUDA EP, still need to figure out the way to remove it 2. Need to have more identifiers to make it able to share CPU allocator across all EPs * fix max norm clipping test in python packaging pipeline test (#6468) * fix python packaging pipeline * make clip norm test compatabile with both V100 and M60 GPUs * Initial version of CoreML EP (#6392) * Bug 31463811: Servicing: Redist (Nuget) conflicts with Microsoft.AI.MachineLearning starting 21H1+ (#6460) * update load library code to have the fullly qualified path * make it work for syswow32 * git Revert "make it work for syswow32" This reverts commit b9f594341b7cf07241b18d0c376af905edcabae3. Co-authored-by: Sheil Kumar <sheilk@microsoft.com> * dequantize 1st input of lstm back if it is quantized (#6444) * [java] Adds support for OrtEnvironment thread pools (#6406) * Updates for Gradle 7. * Adding support for OrtThreadingOptions into the Java API. * Fixing a typo in the JNI code. * Adding a test for the environment's thread pool. * Fix cuda test, add comment to failure. * Updating build.gradle * fix SDL native rule warning #6246 (#6461) * fix SDL rule (#6464) * use tickcount64 (#6447) Co-authored-by: Ori Levari <orlevari@microsoft.com> * Update pypi package metadata (#6354) * Update setup file data * add missing comma * remove python 3.5 * fix typo bracket * Delete nuget extra configs (#6477) * Op kernel type reduction infrastructure. (#6466) Add infrastructure to support type reduction in Op kernel implementations. Update Cast and IsInf CPU kernels to use it. * Fixing a leak in OnnxSequences with String keys or values. (#6473) * Increase the distributes tests pipeline timeout to 120 minutes (#6479) * [CoreML EP] Add CI for CoreML EP (macOS) and add coreml_flags for EP options (#6481) * Add macos coreml CI and coreml_flags * Move save debuggubg model to use environment var * Move pipeline off from macos CI template * Fix an issue building using unix make, add parallel to build script * Fixed build break for shared_lib and cmpile warning * Fix a compile warning * test * Revert the accidental push from another branch This reverts commit 472029ba25d50f9508474c9eeceb3454cead7877. * Add ability to track per operator types in reduced build config. (#6428) * Add ability to generate configuration that includes required types for individual operators, to allow build size reduction based on that. - Add python bindings for ORT format models - Add script to update bindings and help info - Add parsing of ORT format models - Add ability to enable type reduction to config generation - Update build.py to only allow operator/type reduction via config - simpler to require config to be generated first - can't mix a type aware (ORT format model only) and non-type aware config as that may result in insufficient types being enabled - Add script to create reduced build config - Update CIs * merge e2e with distributed pipeline (#6443) merge e2e with distributed pipeline * Fix test breaks in Windows ingestion pipeline (#6476) * fix various build breaks with Windows build * fix runtime errors loading libraries from system32 * add build_inbox check to winml_test_common * use raw string * cleanup * fix dll load Co-authored-by: Sheil Kumar <sheilk@microsoft.com> * Speed up the Mac CI runs (#6483) * expose learningmodelpixelrange property (#5877) * Fix of support api version bug for [de]quantize (#6492) * SDL fixes: add proper casts/format specifiers (#6446) * SDL annotation fixes (#6448) Co-authored-by: Ori Levari <orlevari@microsoft.com> * [OpenVINO-EP] Remove support for OpenVINO 2020.2 (#6493) * Removed OpenVINO 2020.2 support * Updated documentation and build.py * Removed unnecessary libraries from setup.py * Support pad operator in quantization and quantized nhwc transformer. Fix Pad operator bug. (#6325) Support pad operator in quantization tool. Support pad operator in quantized nhwc transformer. Fix pad() operator bug when pad input's inner(right) most axis value is zero for Edge and Reflect mode, it copied wrong value to the cells to be padded. Note the Constant mode will not trigger this bug, as Edge/Reflect need copy value from the already copied array while Constant mode only fill specified value. Add more test cases to cover pad() operator bug fixed here. Fix quantization tools uint8/int8 value overflow issue when quantize weights in python. * Improve work distribution for Expand operator, and sharded LoopCounter configuration (#6454) Description: This PR makes two changes identified while looking at a PGAN model. First, it uses ThreadPool::TryParallelFor for the main parallel loops in the Expand operator. This lets the thread pool decide on the granularity at which to distribute work (unlike TrySimpleParallelFor). Profiling showed high costs when running "simple" loops with 4M iterations each of which copied only 4 bytes. Second, it updates the sharded loop counter in the thread pool so that the number of shards is capped by the number of threads. This helps make the performance of any other high-contention "simple" loops more robust at low thread counts by letting each thread work on its own "home" shard for longer. Motivation and Context Profiling showed a PGAN model taking 2x+ longer with the non-OpenMP build. The root cause was that the OpenMP build uses simple static scheduling of loop iterations, while the non-OpenMP build uses dynamic scheduling. The combination of large numbers of tiny iterations is less significant with static scheduling --- although still desirable to avoid, given that each iteration incurs a std::function invocation. * Update document of transformer optimization (#6487) * nuphar test to avoid test data download to improve passing rate (#6467) nuphar test to avoid test data download to improve passing rate * Fuse cuda conv with activation (#6351) * optimize cuda conv by fused activation * remove needless print out * exclude test from cpu * handle status error from cudnn 8.x * add reference to base class * add hipify * [CoreML EP] Add support for some activations/Transpose, move some shared helpers from NNAPI to shared space (#6498) * Init change * Move some helper from nnapi ep to shared * Add transpose support * Fix trt ci build break * Refine transformers profiler output (#6502) * output nodes in the original order; grouped by node name * add document for profiler * Update to match new test setup. (#6496) * Update to match new test setup. * Add Gemm(7) manually for now. Will fix properly on Monday. It's used by mnist.ort as that is created by optimizing mnist.onnx to level 1 causing 2 nodes to be replaced by a Gemm and the op to be missing from the required list as that is created using the original onnx model. * Enable dense sequence optimized version of Pytorch exported BERT-L on AMD GPU (#6504) * Permit dense seq optimization on BERT-L pytorch export by enabling ReduceSumTraining, Equal, and NonZero on AMD * enable Equal tests * enable fast_matrix_reduction test case * Optimize GatherGrad for AMD GPU (#6381) * optimize gathergrad * address comments Co-authored-by: Weixing Zhang <wezhan@microsoft.com> * add explicit barriers for buffer overread and overrwrite (#6484) Co-authored-by: Ori Levari <orlevari@microsoft.com> * fix sdl bugs for uninitialized variables and returns (#6450) Co-authored-by: Ori Levari <orlevari@microsoft.com> * handle hr error conditions (#6449) Co-authored-by: Ori Levari <orlevari@microsoft.com> * Dnnl training (#6045) * Add ReluGrad and ConvGrad ops for the dnnl provider * the mnist sample is updated to add the --use_dnnl option that will cause the sample to use the dnnl execution provider for nodes that exist in dnnl provider. * Added the ability to find forward ops. Dnnl backward gradient ops require the forward primitive description and workspace from the forward operation. * Enable specifying the execution provider for Gradient Checker Tests * Prevent memory leak when running dnnl_provider in training mode Prevent creating a SubgraphPrimitivePool when the code is built with the ENABLE_TRAINING build flag. Instead create a SubgraphPrimitive directly. The SubgraphPrimitivePool was causing a pool of SubgraphPrimitives to be stashed in a map for reuse. Due to the way the Training Loop uses threads the pool of SubgraphPrimitives were not being reuse instead a new pool of SubgraphPrimitives being created each run. The old pool was not instantly freed. This behavior could be a language error when using thread_local memory. Signed-off-by: George Nash <george.nash@intel.com> * Added fixes to maxpoolgrad and memory leak. Maxpoolgrad will now pass all unit tests. With the conv and convgrad disabled for dnnl, mnist is able to train till 95% Signed-off-by: Chethan Palangotu Keshava <chethan.palangotu.keshava@intel.com> * Fixed misc issues when testing training code with dnnl provider * fix conv_grad dnnl tests with dilation to run dnnl execution provider * update mnist training sample to accept convolution type models convolution models require the input shape to be {1, 28, 28} instead of the flat {728} image that is used for the gemm models this will enable models that require the different shape by adding `--model_type conv` to the command line when running the mnist sample. (while testing a workaround was used see #4762) * Disable weight caching in dnnl conv operator when using training When training we can not use cached weights because the weight will be updated each run. This re-enables dnnl Conv and ConvGrad Ops. The weight caching was the source of the error from Conv when training. * Fix issues found when building grad ops on Linux * The dnnl_convgrad code was over using the scope operator causing a compilation problem. * The dnnl_maxpoolgrad code had a logic error that is was comparing with the source description when it should have been comparing with the destination despription. * Update BUILD.md so it shows DNNL for training * Updated the table of contents. Since the same providers are listed twice. Once for Infrance and again for Training an HTML anchor was added to distinguish the second header from the first for the TOC. * Fix build failure when not using --enable-training build option * reorganize the gradient operators so they are grouped together * Fix issues found when running onnx_backend_test_series.py * Pooling code only supports 2 outputs when built with --enable-training * Address code review feedback * class member variables end in underscore_ * use dst instead of dist to match pattern use elsewhere in DNNL code. * Remove workaround that was introduced to handle problems running convolution based training models. See issue #4762 Signed-off-by: George Nash <george.nash@intel.com> * Isolate training code and code cleanup * Do not build if dnnl_gpu_runtime if enable_training is set training code does not support dnnl_gpu_runtime yet. * Isolated Training code inside ifdefs so that they wont affect project if built without training enabled * Inadvertant changes in whitespace were removed to make code review simpler * Undid some code reordering that was not needed * comments added to closing #endif statments to simplify reading complex ifdefs * Modified the GetPrimitiveDesc functions to return shared_ptr instead of raw pointer. This matches what was done in Pool code and is safer memory code. Signed-off-by: George Nash <george.nash@intel.com> * Address code review issues - whitespace changes caused by running clang-format on the code - Several spelling errors fixed - Removed/changed some ifdefs to improve readability - other misc. changes in responce to code review. Signed-off-by: George Nash <george.nash@intel.com> * Code changes to address code review - Simplify iteration code using `auto` keyword - remove C style cast that was not needed - remove instance variable that was not needed [relugrad.h] - added the execution providers to `ComputeGradientErrorInternal()` and `ComputeTheoreticalJacobianTranspose()` instead of using a pointer to an instance varaible [gradient_checker.h/.cc] Signed-off-by: George Nash <george.nash@intel.com> * Combined the default gradient ops test and dnnl gradient ops test for ConvGrad and MaxPoolGrad into one function with the help of a helper function. This will reduce repeated code. Signed-off-by: Palangotu Keshava, Chethan's avatarChethan Palangotu Keshava <chethan.palangotu.keshava@intel.com> * Replaced the stack used by convgrad to vector so that the vector(used as stack) can be easily cleared everytime the graph is created. This will prevent memory leak from convolution kernels being pushed constantly onto the stack. Signed-off-by: chethan.palangotu.keshava@intel.com * Code clean up and formating updates - Removed empty else statment - updated indentation of code that was causing double curly brackets to look unususal - Changed check for NumDimensions to Size in Relu and ReluGrad error checking code. - isolated training code Signed-off-by: George Nash <george.nash@intel.com> * Restore inadvertantly removed ConvGrad tests When combining the DNNL and CPU version of the ConvGrad tests two test were inadvertantly excluded. This adds back the Conv3d and Conv3d with strides test cases. Signed-off-by: George Nash <george.nash@intel.com> * Add validation to ConvGrad This validates the dimensions of the ConvGrad match the passed in Convolution forward primitive description. The current code for DNNL ConvGrad makes the assumption that the ConvGrad nodes will be visited in the reverse order from the corresponding Conv nodes The added validation will return an error if this assumption is not true. Signed-off-by: George Nash <george.nash@intel.com> * Do not create new execution providers in provider_test_utils This removes the code that generated new execution providers in the OpTester::Run function. This was added because the std::move was leaving the `entry` value empty so subsequent calls would cause a segfault. Problem is this potentially changed the execution_provider because it would create the default provider dropping any custom arguments. When the now removed code was originally added the std::move was causing crashes when the GradientChecker unit tests were run. However, it is no longer causing problems even with the code removed. Signed-off-by: George Nash <george.nash@intel.com> * Change the forward conv stack to a forward conv map This changes how the forward conv kernel is mapped to the bwd ConvGrad kernel the problematic stack is no longer used. The convolution stack made the assumption that the corresponding ConvGrad operator would be visited in reverse order of the forward Conv operators. This was always problematic and was unlikely to work for inception models. Important changes: - The weight_name is added to the ConvGrad dnnl_node making it possible to use the weight_name as a lookup key to find the Conv forward Kernel - the `std::vector fwd_conv_stack_` has been replaced with a `std::map fwd_conv_kernel_map_` - Although it is not needed lock_guards were added when writing to and reading from the fwd_conv_kernel_map_ as well as the fwd_kernel_map_. These should always be accessed by a single thread when preparing the dnnl subgraphs so the guard should not be needed but its added just in case. - Updated the comments ConvGrad.h code to no longer mention the stack. The error check is not removed. It will be good to verify there are no errors as we continue to test against more models. Signed-off-by: George Nash <george.nash@intel.com> Co-authored-by: Chethan Palangotu Keshava <chethan.palangotu.keshava@intel.com> Co-authored-by: unknown <63478620+jeyblu@users.noreply.github.com> * Lochi/refactor yolov3 quantization (#6290) * Refactor the code and move data reader, preprocessing, evaluation to E2E_example_mode * Refactor the code. Move data reader, preprocessing, evaluation to model specific example under E2E_example_mode * refactor code * Move yolov3 example to specific folder and add additional pre/post processing * Print a warning message for using newer c_api header on old binary (#6507) * Fix issues with ArmNN build setup (#6495) * ArmNN build fixes * Update BUILD.md to document that the ACL paths must be specified to build ArmNN * Fix CUDA build error. We don't setup the link libraries correctly/consistently so improve that. * Fix Windows CI builds by updating test scripts to work with numpy 1.20. (#6518) * Update onnxruntime_test_python.py to work with numpy 1.20. Some aliases are deprecated in favor of the built-in python types. See https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations np.array with bytes for entries and dtype of np.void no longer automatically pads. Change a test to adjust for that. * Fix another test script * Fix ORTModule branch for orttraining-* pipelines * Update pytorch nightly version dependency Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Cecilia Liu <ziyue.liu7@gmail.com> Co-authored-by: Ryan Hill <38674843+RyanUnderhill@users.noreply.github.com> Co-authored-by: George Nash <george.nash@intel.com> Co-authored-by: Guoyu Wang <62914304+gwang-msft@users.noreply.github.com> Co-authored-by: Yateng Hong <toothache9010@gmail.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Derek Murray <Derek.Murray@microsoft.com> Co-authored-by: ashbhandare <ash.bhandare@gmail.com> Co-authored-by: Scott McKay <skottmckay@gmail.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Juliana Franco <jufranc@microsoft.com> Co-authored-by: Pranav Sharma <prs@microsoft.com> Co-authored-by: Tixxx <tix@microsoft.com> Co-authored-by: Jay Rodge <jayrodge@live.com> Co-authored-by: Du Li <duli1@microsoft.com> Co-authored-by: Du Li <duli@OrtTrainingDev4.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: baijumeswani <bmeswani@microsoft.com> Co-authored-by: Sergii Dymchenko <sedymche@microsoft.com> Co-authored-by: jingyanwangms <47403504+jingyanwangms@users.noreply.github.com> Co-authored-by: satyajandhyala <satya.k.jandhyala@gmail.com> Co-authored-by: S. Manohar Karlapalem <manohar.karlapalem@intel.com> Co-authored-by: Weixing Zhang <weixingzhang@users.noreply.github.com> Co-authored-by: Suffian Khan <sukha@microsoft.com> Co-authored-by: Olivia Jain <oljain@microsoft.com> Co-authored-by: Chi Lo <54722500+chilo-ms@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Ryan Lai <rylai@microsoft.com> Co-authored-by: Jesse Benson <jesseb@microsoft.com> Co-authored-by: sfatimar <64512376+sfatimar@users.noreply.github.com> Co-authored-by: suryasidd <surya.siddharth.pemmaraju@intel.com> Co-authored-by: sfatimar <sahar.fatima@intel/com> Co-authored-by: MaajidKhan <n.maajidkhan@gmail.com> Co-authored-by: mohdansx <mohdx.ansari@intel.com> Co-authored-by: Xavier Dupré <xadupre@users.noreply.github.com> Co-authored-by: Michael Goin <mgoin@vols.utk.edu> Co-authored-by: Michael Giba <michaelgiba@gmail.com> Co-authored-by: William Tambellini <wtambellini@sdl.com> Co-authored-by: Hector Li <hecli@microsoft.com> Co-authored-by: Aishwarya <aibhanda@OrtTrainingDev4.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net> Co-authored-by: liqunfu <liqfu@microsoft.com> Co-authored-by: liqun <liqun@OrtTrainingDev4.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net> Co-authored-by: pengwa <pengwa@microsoft.com> Co-authored-by: Tang, Cheng <souptc@gmail.com> Co-authored-by: Cheng Tang <chenta@microsoft.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ye Wang <52801275+wangyems@users.noreply.github.com> Co-authored-by: Chun-Wei Chen <jacky82226@gmail.com> Co-authored-by: Vincent Wang <wangwchpku@outlook.com> Co-authored-by: Vincent Wang <weicwang@microsoft.com> Co-authored-by: Luyao Ren <375833274@qq.com> Co-authored-by: Zhang Lei <zhang.huanning@hotmail.com> Co-authored-by: Tim Harris <tiharr@microsoft.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Alberto Magni <49027342+alberto-magni@users.noreply.github.com> Co-authored-by: Wei-Sheng Chin <wschin@outlook.com> Co-authored-by: wezuo <49965641+wezuo@users.noreply.github.com> Co-authored-by: Jesse Benson <benson.jesse@gmail.com> Co-authored-by: Wei Zuo <wezuo@OrtTrainingDev3.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net> Co-authored-by: wezuo <wezuo@az-eus-v100-32gb-5-worker-mgtbby.eastus.cloudapp.azure.com> Co-authored-by: wezuo <wezuo@az-eus-v100-32gb-5-worker-yclzsf.eastus.cloudapp.azure.com> Co-authored-by: Wenbing Li <10278425+wenbingl@users.noreply.github.com> Co-authored-by: Martin Man <supermt@gmail.com> Co-authored-by: M. Zeeshan Siddiqui <mzs@microsoft.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Ori Levari <orlevari@microsoft.com> Co-authored-by: Ubuntu <OrtTrainingDev3@OrtTrainingDev3.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net> Co-authored-by: Sherlock Huang <bahuang@OrtTrainingDev3.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net> Co-authored-by: Sheil Kumar <smk2007@gmail.com> Co-authored-by: Sheil Kumar <sheilk@microsoft.com> Co-authored-by: Ryota Tomioka <ryoto@microsoft.com> Co-authored-by: Adam Pocock <adam.pocock@oracle.com> Co-authored-by: Yulong Wang <f.s@qq.com> Co-authored-by: Faith Xu <faxu@microsoft.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: suryasidd <48925384+suryasidd@users.noreply.github.com> Co-authored-by: RandySheriffH <48490400+RandySheriffH@users.noreply.github.com> Co-authored-by: Weixing Zhang <wezhan@microsoft.com> Co-authored-by: Chethan Palangotu Keshava <chethan.palangotu.keshava@intel.com> Co-authored-by: unknown <63478620+jeyblu@users.noreply.github.com>
2021-02-02 16:59:56 +00:00
//! \enum LearningModelPixelRange
//! \brief Defines the list of image nominal pixel range specified in model metadata.
[contract(MachineLearningContract, 5)]
enum LearningModelPixelRange
{
//! [0...255] for 8bpp(8 Bits per Pixel) samples
ZeroTo255= 0,
//! [0...1] pixel data is stored normalized
ZeroToOne,
//! [-1...1] pixel data is stored normalized
MinusOneToOne
};
Merge windowsai (winml layering) into master (#2956) * Initial Commit * Merged PR 3985217: add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc (#2346) add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc and violating our OS layering requirements. We linked against onecoreuap_apiset.lib in VB so we will continue doing this, but I am still unsure why not to link against onecore instead since that is where we ship. However, since Sheil is the owner of this code we will wait to discuss with him before changing anything. * Initial changes for layering * more snipping to get core into ort * update build instructions to include --build_shared_lib (#2358) * update build instructions to include --build_shared_lib * fix line breaks * Task 23998197: add winml_lib_core into onnnxruntime.dll (#2368) * Task 23998197: add winml_lib_core into onnnxruntime.dll * PR feedback build break on perf_test * return proper error when the model path isn't found (#2391) * LearningModelSession is cleaned up to use the adapter, and parts of b… (#2382) this is a big PR. we are going to move it up to layer_dev , which is still a L3 so we are still safe to do work there agile. we are going to move this into the L3 so that ryan can start doing intergration testing. we will pause for a full code review and integration test result prior to going into the L2. >>>> raw comments from previous commits >>> * LearningModelSession is cleaned up to use the adapter, and parts of binding are. * moved everything in the winmladapter made it all nano-com using, WRL to construct objects in the ORT side. base interfaces for everythign for winml to call cleaned up a bunch of winml to use the base interfaces. * more pieces * GetData across the abi. * renamed some namepsace cleaned up OrtValue cleaned up Tensor cleaned up custom ops. everything *but* learnignmodel should be clean * make sure it's building. winml.dll is still a monolith. * model moved over. everything builds clean. step ! * weak ref comment * Layer dev paulm (#2408) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * Layer dev paulm (#2414) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * Layer dev paulm (#2423) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * Layer dev paulm (#2424) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * Layer dev paulm (#2425) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Layer dev paulm (#2426) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * fixed map and sequence when passing stl types across the ABI . found a leak in nvidia driver, but skipped it. all winmlapitests pass now * Moved SessionOptions over to the abi * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * Register WinML TraceLogging provider on Onnxruntime.dll (#2455) * Register WinML TraceLogging provider on Onnxruntime.dll * Add ifdef to make sure trace logging provider has telemetry option when LAYERING_DONE * No need for ifdef for TraceLoggingOptionMicrosoftTelemetry * PR feedback * Move etw registration into lotus environment constructor and deresgister in lotus environment destructor * Brianma/cpuwinml (#2466) * allow building winml cpu without dml. * Brianma/breaks (#2469) * fix some more breaks * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * move dml checks out of winml and into the adapter * better error handling * Brianma/fi (#2470) * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * fix bad merge * Checking in a staging checkpoint point so that Ryan can work with me in parrallel * build break. * Brianma/testfails (#2473) * add missing ir version to dictvectorizer-string.onnx * add missing ir version to relu.onnx * add missing ir version to zipmap*onnx * add IR version to manually generated models * remove an unnecessary ifdef dml * Brianma/windowsai fi (#2475) * update dockerfiles/README (#2336) * Make elementwise op run 4 items per thread (#2335) Description: Describe your changes. Make elementwise op run 4 items per thread unroll for loop to leverage ILP remove unnessary N==0 check inside elementwise GPU kernel Motivation and Context Why is this change required? What problem does it solve? It can improve the performance of GPU elementwise ops. ~2% performance gain on popular NLP bert model. If it fixes an open issue, please link to the issue here. * Add CUDA GatherElements kernel (#2310) * Updates * Update test * Update * Updates * nits * PR feedback * Update * Update * PR feedback * PR comments * Update * Fix build * Fix build * Nits * Fix * Layer Normalization Fusion (#2319) basic layer normalization transform * Add FastGelu Cuda Op for Gelu and Add bias fusion (#2293) * Add FastGelu cuda op * Add AddBiasGelu for experiment * Revert "Add AddBiasGelu for experiment" This reverts commit 5c1ee019858c657e6bb75887265cb85675626e5b. * Add bias * Add unit tests * update comment * update script * fix build error * update coding style * update for CR feedback Enable half2 optimization only when cuda arch >= 7.0 * move _Tanh to common.cuh * implement CPU contrib OP Attention (#2333) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanUnusedInitializers. (#2320) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanupUnusedInitializers. This means initializers that have been replaced during graph optimizations are not left in the GraphProto when we save an optimized model. * Handle edge case where a model has an unused initializer with matching graph input by also removing the graph input. * Use non-const iterators in std::find_if calls to make centos build happy. * Nuget pipeline changes (#2305) 1. refactor the pipeline, remove some duplicated code 2. Move Windows_py_GPU_Wheels job to Win-GPU-CUDA10. We'll deprecated the "Win-GPU" pool 3. Delete cpu-nocontribops-esrp-pipeline.yml and cpu-nocontribops-pipeline.yml 4. In Linux nuget jobs, run "make install" before creating the package. So that extra RPAH info will be removed * Cuda Reverse Sequence Op, maping types of same size using same template function. (#2281) * Set ElementType to String type of node metadata, instead of byte[] (#2348) * Set ElementType to String type of node metadata, instead of byte[] * Fix spacing * Introduce PrimitiveType into a Type System along with an integer constant (#2307) Improve perf by avoiding GetType<T>() calls. Introduce MLTypeCallDispatcher to switch on Input Type. Add Tensor IsType<T>() fast method. * Fix/test dim value of 0 handling in a couple of places (#2337) * Update the CUDA Where implementation broadcasting logic to handle a dim with value of 0. Add unit test Also add unit test for unary op with dim value of 0 * Exclude ngraph from Where test with 0 dim. * Openvino EP R3.1 onnxrt server (#2357) * onnxrt server with OVEP * onnxrt server with OVEP * Update Dockerfile.server.openvino * onnxrt server OVEP fix reviews * onnxrt server OVEP fix reviews * Implement cuda nonzero op. (#2056) Implement cuda nonzero op. * Direct use python numpy array's memory if already contiguous. (#2355) * Direct use python numpy array's memory if already contiguous. This could greatly improve performance for session with large input, like big image 1920x1080 fastrcnn, 30~40% speed up could be achieved. * Add test case enforce contiguous/non-contiguos numpy array as inputs. * Add helper to create output to minimize binary size. (#2365) Add ConstEigenTensorMap typedef so we don't unnecessarily const_cast the const input Tensor. * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS (#2369) * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS * update * Add Tracelogging for profiling (#1639) Enabled only if onnxruntime_ENABLE_INSTRUMENT is ON * test bidaf with nuphar for avx target (#2370) increase nuphar test coverage a bit * Fix a bug in TLS refcount that may destabilized CUDA CI (#2374) * update output size calculation for resize (#2366) * change how output size is calculated for resize op * add tests for ver 10 resize * Extend OneHot CPU kernel to support more types (#2311) * Extend OneHot CPU kernel to support input int64_t, depth int32_t, output float * Skip BERT before the test data fix is picked up * Fix bug with Slice. Need to pass in flattened input dimensions so the initial offset into the input is calculated correctly. (#2372) * Add opset 11 version of Split to CUDA ops (#2376) Organize the CUDA ops definitions so all the opset 10 and 11 parts are together (same setup used for CPU ops) * Layer Norm Fusion Fix (#2379) * layer norm fusion fix * Add input shape check in code and unit tests * Fuse Add + Gelu (#2360) Implement the transformer to fuse add + gelu Implement the accurate kernel * Skip layer norm transform (#2350) * skip layer normalization transformer * Another try to stabilize CUDA CI (#2383) The root cause seems to be failure in CUDA dealloc when tear down. cudaFree return code was ignored before, so should the debug check. * fix BUILD.md typo (#2375) build.py: error: argument --config: invalid choice: 'RelWithDebugInfo' (choose from 'Debug', 'MinSizeRel', 'Release', 'RelWithDebInfo') * Fixed compilation with ngraph (#2388) * Fix reuse logic in allocation planner. (#2393) * Fix reuse logic in allocation planner. * PR comments * Add helpful comments * Don't allow reuse across string tensors. * [NupharEP] Multiple optimizations (#2380) Fuse transpose into MatMul Implement Pow and constant scalar simplification Vectorize ReduceMean Improve symbolic shape inference Minor updates for better debugging in fused function name * Avoid using the default logger in the graph lib and optimizers (#2361) 1. Use the session logger if it is available. 2. Don't disable warning 4100 globally. We should fix the warnings instead of disabling it. * Change CUDA implementation of Transpose to support all fixed size tensor types (#2387) * Change CUDA implementation of Transpose to not use a typed kernel so we can support more types with minimum binary size. Add support for 8, 16, 32 and 64 bit types. Add unit tests. Add method so the implementation can be called directly (will be used by CUDA Scan very soon). * Disable TensorRT for MLFloat16 and int8 unit tests. * Address PR comment and add support for calling cublas implementation if type is mlfloat16. * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. (#2398) * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. * [NupharEP] force some low/zero cost ops to be inlined (#2409) * fix cross compile bug (#2415) * Minor optimization: if a node has already been placed, there's no need to find a kernel for it. (#2417) * Add Reshape Fusion (#2395) * Add reshape fusion * Add some comments * update comments * update comment format * update according to feedback * update for recent logger change * fix build error * (1) Support both input and output edges in find path in graphutils (2) Add a test case of only one constant initializer of Concat input. (3) Refactor ReshapeFusion class to allow add more subgraph fusion in the future. * fix error * (1) loose constraint on initializer: non constant is allowed for reshape fusion. (2) Change versions type to vector. (3) Add logging. (4) Return false when multiple output edges matched in FindPath. Add comments. * only allow one direction (input or output) in FindPath * [NupharEP] Update notebook and docker image (#2416) Add BERT squad in Nuphar tutorial Enhance speed comparsion readability * Fix the issue in matmul_add_fusion (#2407) Fix the issue in matmul_add_fusion If Muatmul + Add has shape [K] * [K, N], reset it to [1, K] * [K, N] will make the output shape to [1, N] will also requires a reshape on the output. Fix: just remove the shape reset to not fuse it. Add a negative test case for matmul+add fusion * feat(treeregressor): Update TreeEnsembleRegressor for type support (#2389) Updates the `TreeEnsembleRegressor` to allow for `double`, `float`, `int64`, and `int32` inputs to match the upstream specification. Signed-off-by: Nick Groszewski <nicholas.groszewski@capitalone.com> * onnxrt server documentation update (#2396) * Added support for Pad-2 operator in OpenVINO-EP (#2405) * Add CUDA If operator. (#2377) * Add CUDA If operator. Uses CPU operator for implementation. By adding a CUDA version the inputs/outputs (with the exception of the 'cond' input) stay on GPU, and no other logic is required to avoid a copy to CPU across the control flow node. * Improved documentation for onnxruntime::utils::SwapByteOrderCopy(), added precondition check. * Fix the type constraints on CUDA If operator to exclude strings. (#2431) * add Im2col<uint8_t> (#2438) * Adjust codegen vectorization width from target (#2439) * Adjust codegen vectorization width from target * Add CUDA Scan operator. (#2403) * Add Scan CUDA op. Uses CPU implementation for logic. Added some device specific functors for handling when data needs to be manipulated on a different device. Added ability to override the materialization logic in the OrtValue slicer so DML can plugin their handling. * Fix Windows GPU C API packaging pipeline failure (#2440) Fix Windows GPU C API packaging pipeline failure (#2440) * Correctly handle implicit inputs for fused nodes (#2390) * Correctly handle implicit inputs for fused nodes Previously, nuphar's partitioning function didn't include node's implicit inputs into the inputs list of MetaDef, and hence a crash was triggered in the onnx graph checker. This commit fixed the issue. Furthermore, it also fixed a related issue where we didn't add implicit inputs into graph_inputs_excluding_initializers_ in Graph::SetGraphInputsOutputs. the issue was that graph_inputs_including_initializers_ populated by SetInputs (e.g. called by FunctionImpl::FunctionImpl) may contain implicit inputs which were not of any node's initializers in the graph. Because they were not part of any initializers, these implicit inputs couldn't be visited by going through all nodes' inputs. Consequently, they would *not* be added into graph_inputs_excluding_initializers_. We fixed the issue by first copying the populated graph_inputs_including_initializers_ into graph_inputs_excluding_initalizers_, which then had both initializers and non-initializers as its initial content. Later, we erase initializers from the list. In this way, we can ensure all implicit inputs to remain in graph_inputs_excluding_initializers_. * refined comments and fixed duplicates Address CR by revisiting comments in terms of implicit inputs Also fixed an issue by skipping duplicates while copying inputs from graph_inputs_including_initializers_. * address CR explain why we need to collect nodes' implicit inputs * don't rely on pointer values for iterating std::set Previously, openvino relied on iterating a set of NodeArg pointers to construct inputs and outputs for a fused graph. It could cause non-determinism. The reason was that although iterating std::set by itself is stable, pointer values of NodeArgs may vary. Consequently, we could end up visiting the set's elements in different orders for different runs for the same test, which resulted in constructing inputs (and outputs) with different orders to the fused graph. For example, for the same test, we may have inputs [A, B] in some runs but inputs[B, A] in others. Let's use std::string as the key type to avoid such nondeterminism. This commit also added implicit inputs into meta->inputs while returning the capability from the openvino provider. * Fixed another latent issue in openvino's GetCapability function The issue was that we couldn't simply erase fused_inputs and fused_outputs while iterating the nodes. For example, an output NodeArg may have multiple uses, and it's wrong if we erase it from fused_outputs when we encounter only one of its uses as input. * Remove DeviceAllocatorRegistry class (#2451) Remove DeviceAllocatorRegistry class * CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library. * Parallel Gelu with ParallelFor (#2399) Parallel Gelu to get better performance for Gelu * Clean up build.py (#2446) * Pull the latest image before running docker build * Fuse SkipLayerNorm with Bias (#2453) Fuse SkipLayerNorm with Bias * Allow more than one invocation of CreateEnv in the same process. (#2467) * Allow more than one invocation of CreateEnv in the same process. * Fix centos build * Symbolic shape inference improvements: (#2460) * Symbolic shape inference improvements: - add a mode to guess unknown ops' output rank - add support for GatherND - add support for If - fix a bug in get_int_values when then tensor rank > 1D, by treating it as no sympy data - add symbol to literal merge when ONNX silently merges dims - fix a bug in Concat when input dim is 0 - fix a bug in ConstantOfShape that computed dim is not updated - add support for dynamic shape in ConstantOfShape - fix a bug in Loop output shape that loop iterator dim is not inserted at dim 0 - add support for dynamic padding in Pad - add support for dynamic shape in Reshape - add support for Resize with opset > 10, by treating output dims as dynamic - fix a bug in Slice when starts/ends are dynamic - restrict input model to opset 7 and above - make output model optional to avoid disk write when testing Run model tests for symbolic shape inference Reduce 2GB docker image size of nuphar * add additional test data set for nuget pipeline (#2448) * add SAS token to download internal test data for nuget pipeline * update azure endpoint * fix keyvault download step * fix variable declaration for secret group * fix indentation * fix yaml syntax for variables * fix setting secrets for script * fix env synctax * Fix macos pipeline * attempt to add secrets to windows download data * fix mac and win data download * fix windows data download * update test data set url and location * Revert "Brianma/windowsai fi (#2475)" This reverts commit 5780b864a15513fda4eadbfc2b5345fefe70b5ec. * Add scenario tests (#2457) * Add scenario tests * Remove TODO from model license * Add winml_api test dependency * fix model load test. fi from master changed the constructor (#2483) * make api tests all pass (#2486) * fix bad merge * fix bad model merge * Layer dev paulm (#2492) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * Rename ambiguous header (#2489) * fix one more missing IR version model (#2500) * add missing IR version to 4 more models used by scenario tests (#2501) * Add CLI parameters to test runner, build WinML in ARM and x86 CI (#2479) * Support test parameters through CLI arguments * Add WinML do Windows x86/ARM CI builds * Code style fixes * Update googletest Remove GPUTEST macros everywhere now that GTEST_SKIP is supported * Refactor main.cpp * Build scenario tests without DML * Link scenario tests to DML when it's enabled (#2502) * Layer dev release pipeline (#2488) Adds winml binaries to existing cpu nuget package, and creates new gpu dml nuget package with winml binaries and DML EP. * Layer dev paulm (#2506) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * Remove usage of IOBinding in WinML and use C_API Run method (#2504) * remove usage of iobinding * Change data structure to use vector of Ort::Values * Polish bind input / output * Use C APIrun method * Update providers on evaluate getresults * Remove run and IObinding interface from WinMLAdapter * Remove use of IObinding * bind unbound outputs code moved to learningmodelbinding * clean up unneeded istensor adapter function * Fix comment * Check if session is closed before binding and clearing * PR feedback * Layer dev paulm (#2507) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * Make tests dependend on winml_dll (#2509) * add dml binaries to DirectML package and be more explicit about condition variables (#2520) * re-enable warnings for winml builds and fix the warnings that were hiding (#2526) * turn devmode back on for winml builds * fix some warnings. include protobuf in a way that disables some warnings * undo protobufhelpers changes and just ignore 4100 errors in pb code * attempt to isolate protobufhelpers errors * add template specialization for getting tensor proto data * Layer dev paulm (#2533) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * Spawn child process to run DeviceLostRecovery scenario test (#2530) * Spawn child process to run DeviceLostRecovery scenario test * Layer dev paulm (#2536) ori said yes * add missing namespace to winml_trace_logging_provider in lotusenvironment.h (#2542) * Handle exception thrown from all apis in WinMLAdapter (#2539) * various changes to unblock windowsai ADO build * Fix custom ops scenario tests (#2562) * Do not shutdown protobuf after ort environment gets destroyed. Lazy load lotus environment first time it is needed * comment typo * pr comment about calling phoenix singleton * Make lotus_environment static in winmladapter * Layer dev paulm (#2567) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * making windowsAI azure dev ops work. * code review comments. * revert changes * Cmake and preprocessor fixes that where uncovered by building on agents without DML available via SDK * Layer dev dml delayload (#2580) * Brianma/cpu (#2583) * don't include dml stuff in cpu builds * tests that link the image lib also need the telemetry lib now * Throw Winml_err_invalid_binding if binding gpu resource on cpu device (#2589) * Throw Winml_err_invalid_binding if binding gpu resource on cpu device * PR comments. No need to query executionprovider for is gpu device * User/xianz/ortthrow (#2596) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * User/xianz/ortthrow (#2599) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * add status error message * Remove uwp onsuspending winrt call because logruntimeperf is getting removed (#2630) * User/xianz/dedup telemetry (#2631) * investigate duplication of telemetry in winml and ort * remove winml telemetry events * telemetry executionProviderEvent * remove unneccessary file and refactor code little bit * Revert back TelemetryEvent, which send up ETW event. * merge changes from layer_dev to windowsai (#2638) * Remove underscore from googletest names (#2616) * Fix leaking memory allocator Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24278761 and https://microsoft.visualstudio.com/OS/_workitems/edit/24330198 * Explicitly initialize Ort::Value with nullptr * Cache WinML adapter * bad merge * define private version of dxcore enum that is added in 19H1 SDK. (#2654) * add comment for explaning private definition of dxcore d3d feature level ennum value. (#2672) * do not package directml.pdb for redist packages. (#2676) * Fix leaking operator registry (#2645) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24354916 * User/orilevari/windowsai master merge (#2674) merge resolutions included pulling in telemetry logic that was merged to master and not windowsai and dereferencing InferenceSession::sessionstate now that it is a unique pointer * Delete Ort Allocator in LearningModelBinding (#2653) * Delete OrtAllocator in LearningModelBinding * PR comments to make Ort::Allocator a smart pointer * Small comment change * PR feedback to clean up code * PR feedback on move semantics * Clean up std::move * Fix memory leaks (#2679) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24356109, https://microsoft.visualstudio.com/OS/_workitems/edit/24388361 and https://microsoft.visualstudio.com/OS/_workitems/edit/24388596 * various changes to properly organize and skip GPU tests. For now for No DML builds we will not run GPU tests at all. In the future we should adapt the tests to expect the appropiate errors. (#2695) * Windowsai without fi (#2701) * Disable Attention fusion tests when DISABLE_CONTRIB_OPS is defined (#2529) * Setup java ci (#2528) * Add provision in ORT for session options to be parsed when available via model file (#2449) * Initial commit * Fix gitmodules * Nits * Nits * Updates * Update * More changes * Updates * Update * Some updates * More changes * Update * Update * Merge * Update * Updates * More changes * Update * Fix nits * Updates * Fix warning * Fix build * Add comment * PR feedback * PR feedback * Updates * Updates * Update * More changes * Fix build break * Comment test for now * Updates * Updates * PR feedback * Updates * Nits * Add tests * Fix build * Fix build * Fix build * Fix build break * Fix build * Nits * PR feedback * More change * Expose GetSessionOptions in pybind logic and add unit test for python * Fix build * PR feedback * PR feedback * Revert "Disable thread pool creation when enabled OpenMP (#2485)" (#2535) This reverts commit 7c7d5a149c9ed52eec67304bae5c4b132166a8a1. * Add dynamic shape support in TensorRT execution provider (#2450) * remove onnx-tensorrt submodule * add new onnx-tensorrt submodule (experiment) for trt6 * update engine build for trt6 * update compile and compute for tensorrt6.0 * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * switch to onnx-tensorrt master for TensorRT6' * Update tensorrt_execution_provider.cc * Handle dynamic batch size and add memcpy in TensorRT EP * update test cases * Update tensorrt_execution_provider.cc * update onnx-tensorrt submodule * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.ubuntu_tensorrt * Update run_dockerbuild.sh * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update concat_op_test.cc * Update tensorrt_execution_provider.cc * Upgrade TensorRT to version 6.0.1.5 * Update onnxruntime_providers.cmake * Update CMakeLists.txt * Update reduction_ops_test.cc * Update install_ubuntu.sh * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.tensorrt * Update BUILD.md * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update onnxruntime_providers.cmake * Update install_ubuntu.sh * Update install_ubuntu.sh * Update gemm_test.cc * Update gather_op_test.cc * Update CMakeLists.txt * Removed submodule * update onnx-tensorrt submodule * update header file * Removed submodule * add submodule onnx-tensorrt kevin's branch shape-test' * add debugging code * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * merge master * Removed submodule * update onnx-tensorrt submodule * add more changes for dynamic shapes * Update tensorrt_execution_provider.cc * update for dynamic shape * update dynamic shape processing * fix logger issue * remove submodule onnx-tensorrt * add submodule onnx-tensorrt * add env variable min_subgraph_size * remove redundency * update document * use onnxruntime::make_unique * fix multi-run issue * remove some tests to save CI build time * Add dynamic shape test * Update TensorRT-ExecutionProvider.md * Add example of running Faster R-CNN model on TensorRT EP * Add more details on env variables * update environment variables * Update tensorrt_basic_test.cc * Update model tests * Update tensor_op_test.cc * remove --use_full_protobuf * Update build.py * User/xianz/telemetry (#2458) * enabme telemetry * enable telemetry * set enable telemetry as default * for debugging * remove log and set disable telemetry as default back * delete private file while testing * resolve comment: mainly add license header, rename macro and update docs * rewording in privacy.md * Fix integer overflow in cuda NonMaxSuppression implementation (#2540) * add test case that should pass but fail * fix nms * extract int_max_output_boxes_per_class * Introduce container type runtime checks and other improvements (#2522) Rework TensorSeq in a manner consistent with Tensor and SparseTensor in terms of type system setup. Reduce templating. Introduce helpers to ensure the same data type. Make OrtValue __dtor not virtual. Introduce ContainerChecker * Fix C API tests for centos and mac (#2544) * change c++14 to c++11 * add ld lib path for centos * enable csharp tests on macos * fix C API test on MacOS + fix manylinux dotnet install * fix manylinux dotnet install * fix lib link * Add back executable bit to build.py * Fix a bug handling negative begin pad values in Pad op (#2550) * Fix bug in Pad op * Update * DNNL CMAKE update (#2548) * Fix android build (#2558) * Update win-x86-ci.yml (#2557) Fix build pipeline break * Re-enable Windows C# tests (#2564) * disable onnx_test_runner -x invocations for dnnl (#2568) * Allow sequence length to be symbolic (#2559) * setup java ci mac (#2570) * make layernorm fusion to support opset 11 (#2545) * Fix a warning found in the latest VS release * Add more check on SkipLayerNorm and BiasGelu fusion (#2574) * Fix file not found error during docker build. (#2569) * Add ConvTranspose1D (#2578) * Ryanunderhill/packagename test (#2582) * [Nuphar EP] fixes for some object detection models (#2581) Update notebook tutorial with multi-threaded int8 GEMM from #2517 * EmbedLayerNormalization Fusion Improvement (#2553) Embedding layer norm fusion improvements - add more checks * Update version (#2584) * Temporarily exclude vgg19 test from Python backend test 1. temporarily exclude vgg19 test which comsumes too much memory, run out of memory on Upsquared device. Single test pass for vgg19, need furture investigation (#2588) 2. Update docker file to decrease the docker image size * Update docs for Android NNAPI EP (#2586) * Fix lto bug for protobuf and ubuntu * add path to build dir before test run (#2590) * Add missig env variables for mac pipeline test (#2595) * Fixed an issue in updating realized dims (#2597) when we update realized dims for scan's output, the sliced axis also needs to be inclusive, i.e. we should check with "dim >= insert_inclusive_axis", because the offset in the symbols are based on Scan sugraph. Otherwise, we would end up with shape mismatch later. * Java API for onnxruntime (#2215) * Add support for opset 11 in reshape fusion (#2592) Support opset verion 11 in reshape fusion * Rename automl python tools folder to featurizer_ops. (#2593) * Support opset 11 subgraph of Squad model in Embed Layer Normalization (#2605) Support opset 11 Squad model that is exported from PyTorch nightly. The embed layer uses Range op which is missed in the transformer. * symbolic shape inference: fix warnings in GPT-2 model (#2608) And revise nuphar perf test on BERT squad * Dump subgraph ID and fused graph ID (#2607) * Dump subgraph ID and fused graph ID Dump subgraph ID and fused graph ID for better debugging * Remove local static fused_count added a field global_fused_count_ to NupharExecutionProvider class * EmbedLayerNormalization Fusion For Dynamic Squad Model Opset 10 (#2613) Support subgraph of SQuAD model exported from pytorch with dynamic input axes * Allow providers to be set for InferenceSession at construction (#2606) * Remove unnecessary parameter in some places in GatherElements implementation (#2612) * Remove unnecessary parameter in some places * Update * Update * Make sure fenced tensor could not reuse other tensor. (#2561) Fix random error caused by this. * Improve Embed Layer Norm Fusion for SQuAD with static input shape (#2621) * fix float16 comparison in initializer (#2629) * epsilon attribute for layernormalization fusion (#2639) * removed unnecessary batch file and fix path (#2640) * Add shape inference to ConvTransposeWithDynamicPads schema (#2632) * Improve cuda expand() opeator's performance. (#2624) * Cuda pad optimize when no padding is needed. (#2625) * Shortcut cuda Pad() when no padding is needed. * Optimize cuda scatter() on 2D compatible. (#2628) * Optimize cuda scatter() on 2D compatible. * Add some comments. * fix build error for ARM (#2648) * Improve performance of resize() in Nearest mode (#2626) Special treatment for 2D, check same size as input image. And in 2d kernel, template use_expolation. * Fix memory exception in Layer Norm Fusion (#2644) * Windows CI changes(#2650) * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Revert "Windowsai without fi (#2701)" This reverts commit 285d4c85ff5c4e265f963208170304ef3461e684. * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Deref unique pointer for session_state * send shutdown event when dll is unloaded and EvaluationStop, SessionC… (#2704) * send shutdown event when dll is unloaded and EvaluationStop, SessionCreationStart Events. * Add EvalutationStart Event * add comment * use correct type for for loop (#2755) * ARM CI (#2759) * Set ARM agent pool * Set CMake generator to VS 2019 in ARM * Use system-wide CMake instead of custom version Our custom version is too old for VS 2019 * Use DML and build shared lib in ARM CI * Restore nuget packages in ARM CI * Disable DML * Refactor ARM debug/release builds * Use system packaged Python version * Remove hardcoded Python path * Downgrade Python to 3.7 for build * Remove explicit CMake path * Fix invalid JSON in cgmanifest.json (#2760) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * Remove usage of WHOLEARCHIVE in WinML CMake and add WinMLAdapterFactory (#2726) * Remove usage of WHOLEARCHIVE in WinMLAdapter CMake and add WinMLAdapterFactory * PR feedback, no need for dll(export) since using def file * PR comments * Small comment in gen_def.py * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * Move winml_provider_factory.h to proper location (#2801) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Filter CPU case for IsFloat16Supported (#2802) * Merge fixes * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * Fix warnings that cause build to fail * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * Make DML include path global (#2882) * Make DML include path global * Add generated cppwinrt headers to winml_lib_common * Integrate changes to WindowsAI to make ADO Build (#2886) * Revert "CMake cross-generator fixes (#2790)" This reverts commit dbe7d97fa1ab155f1309bced87199527e8f35bd2. * add additional suppress warning in onnx_proto * ignore /wd4996 warning * DML execution provider fixes * Revert "Revert "CMake cross-generator fixes (#2790)"" This reverts commit 1ae7b4bcbc02edc881ad28685da98e095dfceb17. * Update func signature of custom op function overloads * common devicehelpers fixes * Add pch.h for winml_lib_common * re-add winml_lib_common_dir/inc to include path for winml_adapter * User/orilevari/dml redist shared folder (#2890) * move dml nuget package directory up one level to make it shared between build flavors * Merge conflict fix * Revert "Merge conflict fix" This reverts commit 142fa72cf9ce4344ad717b50b7ea2b8582aadc7c. * Revert "Merge remote-tracking branch 'origin/master' into windowsai" This reverts commit 6e2126d46e5e5f564d65da37dd4f70c93dd81165, reversing changes made to b3f5583dc9249834b947c8ea905f6a98060d5bd6. * Make winml_test_common free of test macros (#2902) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Don't generate files outside the build root (#2914) * Don't generate files outside the build root * Add onnxruntime_EXTERNAL_DEPENDENCIES to WinML * Add DML depedency on RESTORE_PACKAGES * User/orilevari/fix yaml merge bugs (#2918) * Add winml test source parameter into cmake function (#2919) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Add arguments to unittest cmake functions * remove comment * Revert "Revert "Merge remote-tracking branch 'origin/master' into windowsai"" This reverts commit ade5abe72a4234fdbc3623093c61c02c6b0bdc26. * Fix breaks from merge with ORT master * Brianma/linux (#2917) * don't include windows.h in cross-plat header * add default case for switch statement * signed/unsigned mismatch fix Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> * User/sheilk/winml adapter c api (#2891) * Create winml adapter c api * fix build * make it build * move adapter into onnxruntime core/session * entry point not exported * minor changes * make model metadata work * make tests pass * implement all the model reflection apis on the adapter c abi * update the new ort interface to create a lotus ennvironment with a logging sink * start adding ort env * move all winml code into adapter folder/lib to isolate it * ensure a single logging manager at a time * start refactoring session * refactor session creation interface * add cpu and dml session option methods to adapter * finish session init * stub out interfaces in ort lib to perform similar mechanics of iinference session * enable profiling, and enable schema override * update session register graph transformers * turn back on custom registry for custom ops * Add sync api * add last c api stubs * should build... but all feature values are broken since this is in flight to moving all implementation details into ivalue * remove ep adapter header * Implement DML execution provider functions from adapter (#2846) * Implement DML execution provider functions from adapter * Use functions in OnnxruntimeEngine.cpp * make map/sequence type_infos freeable, and start implementing ivalue * make it build again * implement value methods * implement remaining methods * remove com adapter abi * check dml session * cache the allocator on ivalue * check if resource is cpu/gpu when access its mutable data * update tensor * mismatched parentheses * fix tensor base and binding obj * it evaluates tensors! sometimes... * minor fixes * enable gpu evals * wrapper all existing winml adapter apis with API_IMPL to try catch (#2854) * update winml... tensor strings are broken, need to template tensorbase to do different things for strings * make tensor strings work with 2 copies in/2 copies out * Fix tensor string and allocator bug * make maps work again... needs some fixes still * Make it build! * enable map inputs * map outputs * unbound outputs for sequences and maps * User/xianz/merge windowsai (#2883) * Packaging pipeline changes for VS 2019 (#2711) * Tiny fix to codegen * Simplify cache implementation and avoid static variables that may carry over between models * Extend DML kernels (#2641) * Additional DML operators * Check unsupported attributes and inputs * Address PR comments * Add kernel capability function used for partitioning, and re-enable stride-based int64 support based on value range * Fix test failures * Build fix * PR comments * Update Nuphar tutorial notebook (#2721) 1. Reflect int8 GEMV improvements for multi-threading from #2696 2. Add notes on multi-threading control using OpenMP 3. Add samples of running multi-isa AOT, and show int8 GEMM differences between AVX and AVX2 4. Add rnn_benchmark example to resolve #1993 * Add schema for new Qops (#2611) * Add schema for new Qops * adding shape inference + qlinearaveragepool * plus review comments * plus review comments * updates per review comments * plus review comments * [server] Add supposed for model_name and model_version as cli parameter (#2708) * remove 64bit warning message from python validation. (#2727) * MLAS: ARM64 build fix (#2734) fix bad usage of vreinterpret to cast vector element types * Fix broken python docs links (#2740) * Fix build on Mac OS (#2731) mac os ld doesn't support --while-archive, correct option is -all_load * fix ngraph wheel (#2737) * fix ngraph wheel 1.1.0 onnxruntime_ngraph wheel doesn't work * remove libdnnl.so in nGraph Libs * make it easy to compare * Split onnxruntime server to a separated folder (#2744) * Fix build for Python 3.8 (#2747) * Fix build for Python 3.8 * Update protobuf to 3.11.2 (#1928) Update protobuf to 3.11.2 (#1928) * Change default optimization level to All (from Basic) (#2745) * change default optimization level to All (from Basic) * fix test * fix c# test * Update numpy to 1.18 (#2758) * Update numpy to 1.18 * Pipeline changes for python 3.8 (#2753) 1. Pipeline changes for python 3.8 2. Fix a regression in setup.py which was just introduced in the previous commit. Please notice, we still haven't made python 3.8 + Windows + CUDA work. * Add basic stacktrace output for posix debug builds. (#2749) * [NupharEP] fix a race condition when multiple sessions running different models concurrently (#2772) * Revert "Change default optimization level to All (from Basic) (#2745)" This reverts commit 56bb503c2f26474b6613bcb2a198691a11dcef38. * Fix typo in error message (#2736) * Rename MKL-DNN to DNNL to fix broken link (#2730) * Fix nightly build version number issue * Pass BUILD_BUILDNUMBER to linux docker * Disable featurizers in python packages * Import more featurizers (#2781) Make kernels non-template. Add input constraint for learnt data. Add min_max_scalar_transformer, robust_scalar_transformer, inputation_marker_transfomer, label_encoder_transformer, missing_dummies_transformer along with tests. Advance Featurizers library commit. * Implement a more stable softmax (#2715) * Implement a more stable SoftMax e^x is represented as infinity if x is large enough, like 100.f. Infinity divided by Infinity is a NAN. Thus, softmax gets a NAN if one or more item are large enough. A math transform as below is leveraged to get a stable softmax: e^xi/(e^x1 + ...e^xn) = e^(xi - max) / (e^(x1 - max) + ... + e^(xn - max)) And for convenience, force max to 0.f if all xi are negative * Contributing: Fix a typo (#2784) * ACL EP GEMM improvements (#2780) When it is posible we use a fully connected layer instead of the gemm implementation. This will let the library use the best implementation based on the input data. * ACL EP convolution improvements (#2774) Added the optimized implementation for depthwise convolution for both ACL v19.02 and ACL 19.05. Also the pointwise convolution seems to be more optimal in the CPU implementation so we opted for that instead. * Add script for release Nuget validation (#2719) * Initial commit * Nits * Disable a test temporarily * Change working directory * Test * Add download python step * Test update * More changes * Fix space issue * Fix * Verify nuget signing * Fix * Spaces * PR feedback * Nit * Fix * Fix * Remove temporary changes * add uint8 support to where op (#2792) * Improve bert optimization script: (#2712) (1) Move input int64=>int32 conversion to embed layer fusion. (2) Output epsilon attribute for LayerNormalization fusion. * add session creation time cost. (#2798) * ML.NET team needs featurizers within a package (#2789) Add auto ml featurizers to Windows, MacOS as well as to GPU packaging-pipelines. * Initialize max of softmax with lowest of float (#2786) * MLAS: update SGEMM threading parameters (#2808) * add interface to copy batch tensors. (#2807) * add interface to copy batch tensors. * onnxruntime * speed up Windows TRT CI (#2811) * don't run cuda tests if building with tensorrt * remove unnecessary build options for win trt ci * refactor win gpu tensorrt ci yml * --numpy_version=1.17 * update * update * azcopy and cuda path * Update test data (#2356) * Add timeseries imputer transformer featurizer kernel (#2813) Make kernels non-template. Add input constraint for learnt data. Fixup tests. Add two more featurizers along with tests. Tests fail. min_max_scalar_transformer robust_scalar_transformer Fix tests serialized stream by prepending version bytes. Add inputation_marker_transfomer and the test. Fix up float/double type designations. Added label_encoder_transformer along with a test. string_throw case is broken at the momement. Fix labelencodertransfomer_test.cc string_throw case Rename maxabsscalertransformer_test.cc Add MissingDummiesTransformer along with the test. Update manifest. Add TimeSeriesImputerTransformer definition, implementation and tests * Fix memory leak in TRT (#2815) * fix memory leak issue * revert EP_FAIL on enueueV2 * Add manifest missing comma * Run static code analyzer on most of our code (#2817) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * update quantization doc (#2783) * update documentation for quantization script * plus some spell corrections * Filter CPU case for IsFloat16Supported (#2802) * update default optimization level + fix gemm_activation fusion (#2791) * update defualt optimization level + fix gemm_activation fusion * fix typo * add unit test and incorporate review comments * fix test comment * Fix dnnl wheel package name (#2823) * Append '-dnnl' to whl package name when --use_dnnl * Update build.py * Update Ubuntu & TensorRT version in README (#2820) Dockerfile.tensorrt is using nvcr.io/nvidia/tensorrt:19.09-py3 as base Image, update Ubuntu and TensorRT version according to https://docs.nvidia.com/deeplearning/sdk/tensorrt-container-release-notes/rel_19-09.html#rel_19-09 * Merge fixes * Add OneHotEncoder and HashOneHotEncoder kernels. (#2830) Add defs and imlementation for OneHotEncoders, adjuist date_time_transformer kernel and test. Add OneHotEncoder kernel test. Add HashOneHotVectorizerTransformer unit test. This does not link due to multiple definitions of functions that are included into header from a CPP file. * Upgrade gtest to the latest version (#2827) WinML would like to update the googletest submodule. They want some newer features (namely GTEST_SKIP to skip tests programmatically and be able to skip entire fixtures easily) and would need to update the submodule version. However, because the new version of code hit a bug in gcc, even though the bug is already fixed in the latest gcc but we're using gcc 4.8.x and it won't get patched for the bug, so we have to do a compromise, change our code a little bit to make it work. The gcc bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=51213 * Add support for int64_t for topk CPU. Fixes github issue #2806. (#2833) * Ignore allocator type in ExecutionProviders allocator map. Make default initialization of OrtMemoryInfo more clearly invalid. (#2768) * Remove allocator type from the key comparison in ExecutionProviders. Remove usage of DummyArena as it's no longer necessary. * Fix x86 tests where arena allocator is disabled. Make initialization of OrtMemoryInfo clearer by adding Invalid enum value. * Make OrtValueNameIdxMap::MaxIdx more intuitive. * Convert ExternalProject Featurizers into git submodule (#2834) Add git submodule for Featurizer library. Update cmake to build for git submodule. * add domain check for nodes + update documentation (#2831) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Fix Linux CUDA nuget packaging pipeline break * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * update optimization doc for BERT related fusions (#2819) * Add bert related transformers to doc * Add execution provider and comment for bert optimizations * Add comment about accuracy impact of approximation * Fix warnings that cause build to fail * MLAS: enable threading for quantized GEMMs (#2844) * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * remove old winmladapter cpp Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * move sequence implementation into ort lib... still commented out... need to turn back on... * begin sequence implementation * make maps and sequences work * fix broken tests * remove dead code * misc cleanup * CR feedback * User/xianz/winml adapter c api (#2869) * wrapper all existing winml adapter apis with API_IMPL to try catch * Return HR or Throw for WinML adapter APIs if failed * undo macro wrapper for two places * Wrap error macros around ort apis, too. * address CR feedback #2 * add more api throw/return macros * Revert changes no longer needed * revert changes to cxx api * format winml lib.ort and winml adapter * remove static pheonix singleton Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * missing use_dml check in winml_adapter_session (#2930) * --use_dnnl flag was mangled in merge (#2931) * use dml macro not wrapping custom registry code (#2934) * Disable LNK4199 winml_dll to enable cuda builds (#2936) * Disable LNK4199 in winml_dll * linkler->linker * LearningModelSessionAPITestGpu.CreateSessionWithCastToFloat16InModel should return DXGI_ERROR_UNSUPPORTED when FP16 not supported (#2937) * Disable LNK4199 in winml_dll * linkler->linker * Need to return DXGI_ERROR_UNSUPPORTED when Model does not support fp16 * Publish build symbols (#2939) * Publish build symbols * Don't upload PDBs for .exe files * Make x86 build (#2943) * fix last remaining size_t/int64_t warnings->errors (#2948) * TensorString, Sequences and Maps use the first allocator, but should use the cpu default allocator. (#2952) * fix tensor string allcoator * clean up default allocator usage for strings in winml lib/api.ort Co-authored-by: Ryan Lai <ryalai96@gmail.com> * Handle tensor shape of zero (#2954) Co-authored-by: Ryan Lai <ryalai96@gmail.com> * CR feedback (#2970) * CR feedback * fix weird formatting on privacy readme * Add 'All rights reserved.' everywhere * readd all rights reserved to winml_provider_factory.h * remove extra space in comment * remove extra whitespace * fixes post master merge * remove winml from nuget gpu pipeline * set IR VERSION on generated_model in rnn_benchmark (#2972) * Fix slice conformance failures (#2908) Co-authored-by: Adrian Tsai <adtsai@microsoft.com> Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Paul McDaniel <paul_mcdaniel@hotmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Dwayne Robinson <fdwr@hotmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com>
2020-02-05 01:12:19 +00:00
//! \class ImageFeatureDescriptor
//! \brief TODO:Docs
[contract(MachineLearningContract, 1)]
INBOX_ONLY([interface_name("Windows.AI.MachineLearning.IImageFeatureDescriptor", 365585a5-171a-4a2a-985f-265159d3895a)])
Merge windowsai (winml layering) into master (#2956) * Initial Commit * Merged PR 3985217: add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc (#2346) add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc and violating our OS layering requirements. We linked against onecoreuap_apiset.lib in VB so we will continue doing this, but I am still unsure why not to link against onecore instead since that is where we ship. However, since Sheil is the owner of this code we will wait to discuss with him before changing anything. * Initial changes for layering * more snipping to get core into ort * update build instructions to include --build_shared_lib (#2358) * update build instructions to include --build_shared_lib * fix line breaks * Task 23998197: add winml_lib_core into onnnxruntime.dll (#2368) * Task 23998197: add winml_lib_core into onnnxruntime.dll * PR feedback build break on perf_test * return proper error when the model path isn't found (#2391) * LearningModelSession is cleaned up to use the adapter, and parts of b… (#2382) this is a big PR. we are going to move it up to layer_dev , which is still a L3 so we are still safe to do work there agile. we are going to move this into the L3 so that ryan can start doing intergration testing. we will pause for a full code review and integration test result prior to going into the L2. >>>> raw comments from previous commits >>> * LearningModelSession is cleaned up to use the adapter, and parts of binding are. * moved everything in the winmladapter made it all nano-com using, WRL to construct objects in the ORT side. base interfaces for everythign for winml to call cleaned up a bunch of winml to use the base interfaces. * more pieces * GetData across the abi. * renamed some namepsace cleaned up OrtValue cleaned up Tensor cleaned up custom ops. everything *but* learnignmodel should be clean * make sure it's building. winml.dll is still a monolith. * model moved over. everything builds clean. step ! * weak ref comment * Layer dev paulm (#2408) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * Layer dev paulm (#2414) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * Layer dev paulm (#2423) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * Layer dev paulm (#2424) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * Layer dev paulm (#2425) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Layer dev paulm (#2426) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * fixed map and sequence when passing stl types across the ABI . found a leak in nvidia driver, but skipped it. all winmlapitests pass now * Moved SessionOptions over to the abi * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * Register WinML TraceLogging provider on Onnxruntime.dll (#2455) * Register WinML TraceLogging provider on Onnxruntime.dll * Add ifdef to make sure trace logging provider has telemetry option when LAYERING_DONE * No need for ifdef for TraceLoggingOptionMicrosoftTelemetry * PR feedback * Move etw registration into lotus environment constructor and deresgister in lotus environment destructor * Brianma/cpuwinml (#2466) * allow building winml cpu without dml. * Brianma/breaks (#2469) * fix some more breaks * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * move dml checks out of winml and into the adapter * better error handling * Brianma/fi (#2470) * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * fix bad merge * Checking in a staging checkpoint point so that Ryan can work with me in parrallel * build break. * Brianma/testfails (#2473) * add missing ir version to dictvectorizer-string.onnx * add missing ir version to relu.onnx * add missing ir version to zipmap*onnx * add IR version to manually generated models * remove an unnecessary ifdef dml * Brianma/windowsai fi (#2475) * update dockerfiles/README (#2336) * Make elementwise op run 4 items per thread (#2335) Description: Describe your changes. Make elementwise op run 4 items per thread unroll for loop to leverage ILP remove unnessary N==0 check inside elementwise GPU kernel Motivation and Context Why is this change required? What problem does it solve? It can improve the performance of GPU elementwise ops. ~2% performance gain on popular NLP bert model. If it fixes an open issue, please link to the issue here. * Add CUDA GatherElements kernel (#2310) * Updates * Update test * Update * Updates * nits * PR feedback * Update * Update * PR feedback * PR comments * Update * Fix build * Fix build * Nits * Fix * Layer Normalization Fusion (#2319) basic layer normalization transform * Add FastGelu Cuda Op for Gelu and Add bias fusion (#2293) * Add FastGelu cuda op * Add AddBiasGelu for experiment * Revert "Add AddBiasGelu for experiment" This reverts commit 5c1ee019858c657e6bb75887265cb85675626e5b. * Add bias * Add unit tests * update comment * update script * fix build error * update coding style * update for CR feedback Enable half2 optimization only when cuda arch >= 7.0 * move _Tanh to common.cuh * implement CPU contrib OP Attention (#2333) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanUnusedInitializers. (#2320) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanupUnusedInitializers. This means initializers that have been replaced during graph optimizations are not left in the GraphProto when we save an optimized model. * Handle edge case where a model has an unused initializer with matching graph input by also removing the graph input. * Use non-const iterators in std::find_if calls to make centos build happy. * Nuget pipeline changes (#2305) 1. refactor the pipeline, remove some duplicated code 2. Move Windows_py_GPU_Wheels job to Win-GPU-CUDA10. We'll deprecated the "Win-GPU" pool 3. Delete cpu-nocontribops-esrp-pipeline.yml and cpu-nocontribops-pipeline.yml 4. In Linux nuget jobs, run "make install" before creating the package. So that extra RPAH info will be removed * Cuda Reverse Sequence Op, maping types of same size using same template function. (#2281) * Set ElementType to String type of node metadata, instead of byte[] (#2348) * Set ElementType to String type of node metadata, instead of byte[] * Fix spacing * Introduce PrimitiveType into a Type System along with an integer constant (#2307) Improve perf by avoiding GetType<T>() calls. Introduce MLTypeCallDispatcher to switch on Input Type. Add Tensor IsType<T>() fast method. * Fix/test dim value of 0 handling in a couple of places (#2337) * Update the CUDA Where implementation broadcasting logic to handle a dim with value of 0. Add unit test Also add unit test for unary op with dim value of 0 * Exclude ngraph from Where test with 0 dim. * Openvino EP R3.1 onnxrt server (#2357) * onnxrt server with OVEP * onnxrt server with OVEP * Update Dockerfile.server.openvino * onnxrt server OVEP fix reviews * onnxrt server OVEP fix reviews * Implement cuda nonzero op. (#2056) Implement cuda nonzero op. * Direct use python numpy array's memory if already contiguous. (#2355) * Direct use python numpy array's memory if already contiguous. This could greatly improve performance for session with large input, like big image 1920x1080 fastrcnn, 30~40% speed up could be achieved. * Add test case enforce contiguous/non-contiguos numpy array as inputs. * Add helper to create output to minimize binary size. (#2365) Add ConstEigenTensorMap typedef so we don't unnecessarily const_cast the const input Tensor. * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS (#2369) * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS * update * Add Tracelogging for profiling (#1639) Enabled only if onnxruntime_ENABLE_INSTRUMENT is ON * test bidaf with nuphar for avx target (#2370) increase nuphar test coverage a bit * Fix a bug in TLS refcount that may destabilized CUDA CI (#2374) * update output size calculation for resize (#2366) * change how output size is calculated for resize op * add tests for ver 10 resize * Extend OneHot CPU kernel to support more types (#2311) * Extend OneHot CPU kernel to support input int64_t, depth int32_t, output float * Skip BERT before the test data fix is picked up * Fix bug with Slice. Need to pass in flattened input dimensions so the initial offset into the input is calculated correctly. (#2372) * Add opset 11 version of Split to CUDA ops (#2376) Organize the CUDA ops definitions so all the opset 10 and 11 parts are together (same setup used for CPU ops) * Layer Norm Fusion Fix (#2379) * layer norm fusion fix * Add input shape check in code and unit tests * Fuse Add + Gelu (#2360) Implement the transformer to fuse add + gelu Implement the accurate kernel * Skip layer norm transform (#2350) * skip layer normalization transformer * Another try to stabilize CUDA CI (#2383) The root cause seems to be failure in CUDA dealloc when tear down. cudaFree return code was ignored before, so should the debug check. * fix BUILD.md typo (#2375) build.py: error: argument --config: invalid choice: 'RelWithDebugInfo' (choose from 'Debug', 'MinSizeRel', 'Release', 'RelWithDebInfo') * Fixed compilation with ngraph (#2388) * Fix reuse logic in allocation planner. (#2393) * Fix reuse logic in allocation planner. * PR comments * Add helpful comments * Don't allow reuse across string tensors. * [NupharEP] Multiple optimizations (#2380) Fuse transpose into MatMul Implement Pow and constant scalar simplification Vectorize ReduceMean Improve symbolic shape inference Minor updates for better debugging in fused function name * Avoid using the default logger in the graph lib and optimizers (#2361) 1. Use the session logger if it is available. 2. Don't disable warning 4100 globally. We should fix the warnings instead of disabling it. * Change CUDA implementation of Transpose to support all fixed size tensor types (#2387) * Change CUDA implementation of Transpose to not use a typed kernel so we can support more types with minimum binary size. Add support for 8, 16, 32 and 64 bit types. Add unit tests. Add method so the implementation can be called directly (will be used by CUDA Scan very soon). * Disable TensorRT for MLFloat16 and int8 unit tests. * Address PR comment and add support for calling cublas implementation if type is mlfloat16. * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. (#2398) * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. * [NupharEP] force some low/zero cost ops to be inlined (#2409) * fix cross compile bug (#2415) * Minor optimization: if a node has already been placed, there's no need to find a kernel for it. (#2417) * Add Reshape Fusion (#2395) * Add reshape fusion * Add some comments * update comments * update comment format * update according to feedback * update for recent logger change * fix build error * (1) Support both input and output edges in find path in graphutils (2) Add a test case of only one constant initializer of Concat input. (3) Refactor ReshapeFusion class to allow add more subgraph fusion in the future. * fix error * (1) loose constraint on initializer: non constant is allowed for reshape fusion. (2) Change versions type to vector. (3) Add logging. (4) Return false when multiple output edges matched in FindPath. Add comments. * only allow one direction (input or output) in FindPath * [NupharEP] Update notebook and docker image (#2416) Add BERT squad in Nuphar tutorial Enhance speed comparsion readability * Fix the issue in matmul_add_fusion (#2407) Fix the issue in matmul_add_fusion If Muatmul + Add has shape [K] * [K, N], reset it to [1, K] * [K, N] will make the output shape to [1, N] will also requires a reshape on the output. Fix: just remove the shape reset to not fuse it. Add a negative test case for matmul+add fusion * feat(treeregressor): Update TreeEnsembleRegressor for type support (#2389) Updates the `TreeEnsembleRegressor` to allow for `double`, `float`, `int64`, and `int32` inputs to match the upstream specification. Signed-off-by: Nick Groszewski <nicholas.groszewski@capitalone.com> * onnxrt server documentation update (#2396) * Added support for Pad-2 operator in OpenVINO-EP (#2405) * Add CUDA If operator. (#2377) * Add CUDA If operator. Uses CPU operator for implementation. By adding a CUDA version the inputs/outputs (with the exception of the 'cond' input) stay on GPU, and no other logic is required to avoid a copy to CPU across the control flow node. * Improved documentation for onnxruntime::utils::SwapByteOrderCopy(), added precondition check. * Fix the type constraints on CUDA If operator to exclude strings. (#2431) * add Im2col<uint8_t> (#2438) * Adjust codegen vectorization width from target (#2439) * Adjust codegen vectorization width from target * Add CUDA Scan operator. (#2403) * Add Scan CUDA op. Uses CPU implementation for logic. Added some device specific functors for handling when data needs to be manipulated on a different device. Added ability to override the materialization logic in the OrtValue slicer so DML can plugin their handling. * Fix Windows GPU C API packaging pipeline failure (#2440) Fix Windows GPU C API packaging pipeline failure (#2440) * Correctly handle implicit inputs for fused nodes (#2390) * Correctly handle implicit inputs for fused nodes Previously, nuphar's partitioning function didn't include node's implicit inputs into the inputs list of MetaDef, and hence a crash was triggered in the onnx graph checker. This commit fixed the issue. Furthermore, it also fixed a related issue where we didn't add implicit inputs into graph_inputs_excluding_initializers_ in Graph::SetGraphInputsOutputs. the issue was that graph_inputs_including_initializers_ populated by SetInputs (e.g. called by FunctionImpl::FunctionImpl) may contain implicit inputs which were not of any node's initializers in the graph. Because they were not part of any initializers, these implicit inputs couldn't be visited by going through all nodes' inputs. Consequently, they would *not* be added into graph_inputs_excluding_initializers_. We fixed the issue by first copying the populated graph_inputs_including_initializers_ into graph_inputs_excluding_initalizers_, which then had both initializers and non-initializers as its initial content. Later, we erase initializers from the list. In this way, we can ensure all implicit inputs to remain in graph_inputs_excluding_initializers_. * refined comments and fixed duplicates Address CR by revisiting comments in terms of implicit inputs Also fixed an issue by skipping duplicates while copying inputs from graph_inputs_including_initializers_. * address CR explain why we need to collect nodes' implicit inputs * don't rely on pointer values for iterating std::set Previously, openvino relied on iterating a set of NodeArg pointers to construct inputs and outputs for a fused graph. It could cause non-determinism. The reason was that although iterating std::set by itself is stable, pointer values of NodeArgs may vary. Consequently, we could end up visiting the set's elements in different orders for different runs for the same test, which resulted in constructing inputs (and outputs) with different orders to the fused graph. For example, for the same test, we may have inputs [A, B] in some runs but inputs[B, A] in others. Let's use std::string as the key type to avoid such nondeterminism. This commit also added implicit inputs into meta->inputs while returning the capability from the openvino provider. * Fixed another latent issue in openvino's GetCapability function The issue was that we couldn't simply erase fused_inputs and fused_outputs while iterating the nodes. For example, an output NodeArg may have multiple uses, and it's wrong if we erase it from fused_outputs when we encounter only one of its uses as input. * Remove DeviceAllocatorRegistry class (#2451) Remove DeviceAllocatorRegistry class * CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library. * Parallel Gelu with ParallelFor (#2399) Parallel Gelu to get better performance for Gelu * Clean up build.py (#2446) * Pull the latest image before running docker build * Fuse SkipLayerNorm with Bias (#2453) Fuse SkipLayerNorm with Bias * Allow more than one invocation of CreateEnv in the same process. (#2467) * Allow more than one invocation of CreateEnv in the same process. * Fix centos build * Symbolic shape inference improvements: (#2460) * Symbolic shape inference improvements: - add a mode to guess unknown ops' output rank - add support for GatherND - add support for If - fix a bug in get_int_values when then tensor rank > 1D, by treating it as no sympy data - add symbol to literal merge when ONNX silently merges dims - fix a bug in Concat when input dim is 0 - fix a bug in ConstantOfShape that computed dim is not updated - add support for dynamic shape in ConstantOfShape - fix a bug in Loop output shape that loop iterator dim is not inserted at dim 0 - add support for dynamic padding in Pad - add support for dynamic shape in Reshape - add support for Resize with opset > 10, by treating output dims as dynamic - fix a bug in Slice when starts/ends are dynamic - restrict input model to opset 7 and above - make output model optional to avoid disk write when testing Run model tests for symbolic shape inference Reduce 2GB docker image size of nuphar * add additional test data set for nuget pipeline (#2448) * add SAS token to download internal test data for nuget pipeline * update azure endpoint * fix keyvault download step * fix variable declaration for secret group * fix indentation * fix yaml syntax for variables * fix setting secrets for script * fix env synctax * Fix macos pipeline * attempt to add secrets to windows download data * fix mac and win data download * fix windows data download * update test data set url and location * Revert "Brianma/windowsai fi (#2475)" This reverts commit 5780b864a15513fda4eadbfc2b5345fefe70b5ec. * Add scenario tests (#2457) * Add scenario tests * Remove TODO from model license * Add winml_api test dependency * fix model load test. fi from master changed the constructor (#2483) * make api tests all pass (#2486) * fix bad merge * fix bad model merge * Layer dev paulm (#2492) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * Rename ambiguous header (#2489) * fix one more missing IR version model (#2500) * add missing IR version to 4 more models used by scenario tests (#2501) * Add CLI parameters to test runner, build WinML in ARM and x86 CI (#2479) * Support test parameters through CLI arguments * Add WinML do Windows x86/ARM CI builds * Code style fixes * Update googletest Remove GPUTEST macros everywhere now that GTEST_SKIP is supported * Refactor main.cpp * Build scenario tests without DML * Link scenario tests to DML when it's enabled (#2502) * Layer dev release pipeline (#2488) Adds winml binaries to existing cpu nuget package, and creates new gpu dml nuget package with winml binaries and DML EP. * Layer dev paulm (#2506) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * Remove usage of IOBinding in WinML and use C_API Run method (#2504) * remove usage of iobinding * Change data structure to use vector of Ort::Values * Polish bind input / output * Use C APIrun method * Update providers on evaluate getresults * Remove run and IObinding interface from WinMLAdapter * Remove use of IObinding * bind unbound outputs code moved to learningmodelbinding * clean up unneeded istensor adapter function * Fix comment * Check if session is closed before binding and clearing * PR feedback * Layer dev paulm (#2507) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * Make tests dependend on winml_dll (#2509) * add dml binaries to DirectML package and be more explicit about condition variables (#2520) * re-enable warnings for winml builds and fix the warnings that were hiding (#2526) * turn devmode back on for winml builds * fix some warnings. include protobuf in a way that disables some warnings * undo protobufhelpers changes and just ignore 4100 errors in pb code * attempt to isolate protobufhelpers errors * add template specialization for getting tensor proto data * Layer dev paulm (#2533) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * Spawn child process to run DeviceLostRecovery scenario test (#2530) * Spawn child process to run DeviceLostRecovery scenario test * Layer dev paulm (#2536) ori said yes * add missing namespace to winml_trace_logging_provider in lotusenvironment.h (#2542) * Handle exception thrown from all apis in WinMLAdapter (#2539) * various changes to unblock windowsai ADO build * Fix custom ops scenario tests (#2562) * Do not shutdown protobuf after ort environment gets destroyed. Lazy load lotus environment first time it is needed * comment typo * pr comment about calling phoenix singleton * Make lotus_environment static in winmladapter * Layer dev paulm (#2567) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * making windowsAI azure dev ops work. * code review comments. * revert changes * Cmake and preprocessor fixes that where uncovered by building on agents without DML available via SDK * Layer dev dml delayload (#2580) * Brianma/cpu (#2583) * don't include dml stuff in cpu builds * tests that link the image lib also need the telemetry lib now * Throw Winml_err_invalid_binding if binding gpu resource on cpu device (#2589) * Throw Winml_err_invalid_binding if binding gpu resource on cpu device * PR comments. No need to query executionprovider for is gpu device * User/xianz/ortthrow (#2596) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * User/xianz/ortthrow (#2599) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * add status error message * Remove uwp onsuspending winrt call because logruntimeperf is getting removed (#2630) * User/xianz/dedup telemetry (#2631) * investigate duplication of telemetry in winml and ort * remove winml telemetry events * telemetry executionProviderEvent * remove unneccessary file and refactor code little bit * Revert back TelemetryEvent, which send up ETW event. * merge changes from layer_dev to windowsai (#2638) * Remove underscore from googletest names (#2616) * Fix leaking memory allocator Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24278761 and https://microsoft.visualstudio.com/OS/_workitems/edit/24330198 * Explicitly initialize Ort::Value with nullptr * Cache WinML adapter * bad merge * define private version of dxcore enum that is added in 19H1 SDK. (#2654) * add comment for explaning private definition of dxcore d3d feature level ennum value. (#2672) * do not package directml.pdb for redist packages. (#2676) * Fix leaking operator registry (#2645) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24354916 * User/orilevari/windowsai master merge (#2674) merge resolutions included pulling in telemetry logic that was merged to master and not windowsai and dereferencing InferenceSession::sessionstate now that it is a unique pointer * Delete Ort Allocator in LearningModelBinding (#2653) * Delete OrtAllocator in LearningModelBinding * PR comments to make Ort::Allocator a smart pointer * Small comment change * PR feedback to clean up code * PR feedback on move semantics * Clean up std::move * Fix memory leaks (#2679) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24356109, https://microsoft.visualstudio.com/OS/_workitems/edit/24388361 and https://microsoft.visualstudio.com/OS/_workitems/edit/24388596 * various changes to properly organize and skip GPU tests. For now for No DML builds we will not run GPU tests at all. In the future we should adapt the tests to expect the appropiate errors. (#2695) * Windowsai without fi (#2701) * Disable Attention fusion tests when DISABLE_CONTRIB_OPS is defined (#2529) * Setup java ci (#2528) * Add provision in ORT for session options to be parsed when available via model file (#2449) * Initial commit * Fix gitmodules * Nits * Nits * Updates * Update * More changes * Updates * Update * Some updates * More changes * Update * Update * Merge * Update * Updates * More changes * Update * Fix nits * Updates * Fix warning * Fix build * Add comment * PR feedback * PR feedback * Updates * Updates * Update * More changes * Fix build break * Comment test for now * Updates * Updates * PR feedback * Updates * Nits * Add tests * Fix build * Fix build * Fix build * Fix build break * Fix build * Nits * PR feedback * More change * Expose GetSessionOptions in pybind logic and add unit test for python * Fix build * PR feedback * PR feedback * Revert "Disable thread pool creation when enabled OpenMP (#2485)" (#2535) This reverts commit 7c7d5a149c9ed52eec67304bae5c4b132166a8a1. * Add dynamic shape support in TensorRT execution provider (#2450) * remove onnx-tensorrt submodule * add new onnx-tensorrt submodule (experiment) for trt6 * update engine build for trt6 * update compile and compute for tensorrt6.0 * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * switch to onnx-tensorrt master for TensorRT6' * Update tensorrt_execution_provider.cc * Handle dynamic batch size and add memcpy in TensorRT EP * update test cases * Update tensorrt_execution_provider.cc * update onnx-tensorrt submodule * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.ubuntu_tensorrt * Update run_dockerbuild.sh * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update concat_op_test.cc * Update tensorrt_execution_provider.cc * Upgrade TensorRT to version 6.0.1.5 * Update onnxruntime_providers.cmake * Update CMakeLists.txt * Update reduction_ops_test.cc * Update install_ubuntu.sh * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.tensorrt * Update BUILD.md * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update onnxruntime_providers.cmake * Update install_ubuntu.sh * Update install_ubuntu.sh * Update gemm_test.cc * Update gather_op_test.cc * Update CMakeLists.txt * Removed submodule * update onnx-tensorrt submodule * update header file * Removed submodule * add submodule onnx-tensorrt kevin's branch shape-test' * add debugging code * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * merge master * Removed submodule * update onnx-tensorrt submodule * add more changes for dynamic shapes * Update tensorrt_execution_provider.cc * update for dynamic shape * update dynamic shape processing * fix logger issue * remove submodule onnx-tensorrt * add submodule onnx-tensorrt * add env variable min_subgraph_size * remove redundency * update document * use onnxruntime::make_unique * fix multi-run issue * remove some tests to save CI build time * Add dynamic shape test * Update TensorRT-ExecutionProvider.md * Add example of running Faster R-CNN model on TensorRT EP * Add more details on env variables * update environment variables * Update tensorrt_basic_test.cc * Update model tests * Update tensor_op_test.cc * remove --use_full_protobuf * Update build.py * User/xianz/telemetry (#2458) * enabme telemetry * enable telemetry * set enable telemetry as default * for debugging * remove log and set disable telemetry as default back * delete private file while testing * resolve comment: mainly add license header, rename macro and update docs * rewording in privacy.md * Fix integer overflow in cuda NonMaxSuppression implementation (#2540) * add test case that should pass but fail * fix nms * extract int_max_output_boxes_per_class * Introduce container type runtime checks and other improvements (#2522) Rework TensorSeq in a manner consistent with Tensor and SparseTensor in terms of type system setup. Reduce templating. Introduce helpers to ensure the same data type. Make OrtValue __dtor not virtual. Introduce ContainerChecker * Fix C API tests for centos and mac (#2544) * change c++14 to c++11 * add ld lib path for centos * enable csharp tests on macos * fix C API test on MacOS + fix manylinux dotnet install * fix manylinux dotnet install * fix lib link * Add back executable bit to build.py * Fix a bug handling negative begin pad values in Pad op (#2550) * Fix bug in Pad op * Update * DNNL CMAKE update (#2548) * Fix android build (#2558) * Update win-x86-ci.yml (#2557) Fix build pipeline break * Re-enable Windows C# tests (#2564) * disable onnx_test_runner -x invocations for dnnl (#2568) * Allow sequence length to be symbolic (#2559) * setup java ci mac (#2570) * make layernorm fusion to support opset 11 (#2545) * Fix a warning found in the latest VS release * Add more check on SkipLayerNorm and BiasGelu fusion (#2574) * Fix file not found error during docker build. (#2569) * Add ConvTranspose1D (#2578) * Ryanunderhill/packagename test (#2582) * [Nuphar EP] fixes for some object detection models (#2581) Update notebook tutorial with multi-threaded int8 GEMM from #2517 * EmbedLayerNormalization Fusion Improvement (#2553) Embedding layer norm fusion improvements - add more checks * Update version (#2584) * Temporarily exclude vgg19 test from Python backend test 1. temporarily exclude vgg19 test which comsumes too much memory, run out of memory on Upsquared device. Single test pass for vgg19, need furture investigation (#2588) 2. Update docker file to decrease the docker image size * Update docs for Android NNAPI EP (#2586) * Fix lto bug for protobuf and ubuntu * add path to build dir before test run (#2590) * Add missig env variables for mac pipeline test (#2595) * Fixed an issue in updating realized dims (#2597) when we update realized dims for scan's output, the sliced axis also needs to be inclusive, i.e. we should check with "dim >= insert_inclusive_axis", because the offset in the symbols are based on Scan sugraph. Otherwise, we would end up with shape mismatch later. * Java API for onnxruntime (#2215) * Add support for opset 11 in reshape fusion (#2592) Support opset verion 11 in reshape fusion * Rename automl python tools folder to featurizer_ops. (#2593) * Support opset 11 subgraph of Squad model in Embed Layer Normalization (#2605) Support opset 11 Squad model that is exported from PyTorch nightly. The embed layer uses Range op which is missed in the transformer. * symbolic shape inference: fix warnings in GPT-2 model (#2608) And revise nuphar perf test on BERT squad * Dump subgraph ID and fused graph ID (#2607) * Dump subgraph ID and fused graph ID Dump subgraph ID and fused graph ID for better debugging * Remove local static fused_count added a field global_fused_count_ to NupharExecutionProvider class * EmbedLayerNormalization Fusion For Dynamic Squad Model Opset 10 (#2613) Support subgraph of SQuAD model exported from pytorch with dynamic input axes * Allow providers to be set for InferenceSession at construction (#2606) * Remove unnecessary parameter in some places in GatherElements implementation (#2612) * Remove unnecessary parameter in some places * Update * Update * Make sure fenced tensor could not reuse other tensor. (#2561) Fix random error caused by this. * Improve Embed Layer Norm Fusion for SQuAD with static input shape (#2621) * fix float16 comparison in initializer (#2629) * epsilon attribute for layernormalization fusion (#2639) * removed unnecessary batch file and fix path (#2640) * Add shape inference to ConvTransposeWithDynamicPads schema (#2632) * Improve cuda expand() opeator's performance. (#2624) * Cuda pad optimize when no padding is needed. (#2625) * Shortcut cuda Pad() when no padding is needed. * Optimize cuda scatter() on 2D compatible. (#2628) * Optimize cuda scatter() on 2D compatible. * Add some comments. * fix build error for ARM (#2648) * Improve performance of resize() in Nearest mode (#2626) Special treatment for 2D, check same size as input image. And in 2d kernel, template use_expolation. * Fix memory exception in Layer Norm Fusion (#2644) * Windows CI changes(#2650) * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Revert "Windowsai without fi (#2701)" This reverts commit 285d4c85ff5c4e265f963208170304ef3461e684. * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Deref unique pointer for session_state * send shutdown event when dll is unloaded and EvaluationStop, SessionC… (#2704) * send shutdown event when dll is unloaded and EvaluationStop, SessionCreationStart Events. * Add EvalutationStart Event * add comment * use correct type for for loop (#2755) * ARM CI (#2759) * Set ARM agent pool * Set CMake generator to VS 2019 in ARM * Use system-wide CMake instead of custom version Our custom version is too old for VS 2019 * Use DML and build shared lib in ARM CI * Restore nuget packages in ARM CI * Disable DML * Refactor ARM debug/release builds * Use system packaged Python version * Remove hardcoded Python path * Downgrade Python to 3.7 for build * Remove explicit CMake path * Fix invalid JSON in cgmanifest.json (#2760) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * Remove usage of WHOLEARCHIVE in WinML CMake and add WinMLAdapterFactory (#2726) * Remove usage of WHOLEARCHIVE in WinMLAdapter CMake and add WinMLAdapterFactory * PR feedback, no need for dll(export) since using def file * PR comments * Small comment in gen_def.py * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * Move winml_provider_factory.h to proper location (#2801) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Filter CPU case for IsFloat16Supported (#2802) * Merge fixes * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * Fix warnings that cause build to fail * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * Make DML include path global (#2882) * Make DML include path global * Add generated cppwinrt headers to winml_lib_common * Integrate changes to WindowsAI to make ADO Build (#2886) * Revert "CMake cross-generator fixes (#2790)" This reverts commit dbe7d97fa1ab155f1309bced87199527e8f35bd2. * add additional suppress warning in onnx_proto * ignore /wd4996 warning * DML execution provider fixes * Revert "Revert "CMake cross-generator fixes (#2790)"" This reverts commit 1ae7b4bcbc02edc881ad28685da98e095dfceb17. * Update func signature of custom op function overloads * common devicehelpers fixes * Add pch.h for winml_lib_common * re-add winml_lib_common_dir/inc to include path for winml_adapter * User/orilevari/dml redist shared folder (#2890) * move dml nuget package directory up one level to make it shared between build flavors * Merge conflict fix * Revert "Merge conflict fix" This reverts commit 142fa72cf9ce4344ad717b50b7ea2b8582aadc7c. * Revert "Merge remote-tracking branch 'origin/master' into windowsai" This reverts commit 6e2126d46e5e5f564d65da37dd4f70c93dd81165, reversing changes made to b3f5583dc9249834b947c8ea905f6a98060d5bd6. * Make winml_test_common free of test macros (#2902) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Don't generate files outside the build root (#2914) * Don't generate files outside the build root * Add onnxruntime_EXTERNAL_DEPENDENCIES to WinML * Add DML depedency on RESTORE_PACKAGES * User/orilevari/fix yaml merge bugs (#2918) * Add winml test source parameter into cmake function (#2919) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Add arguments to unittest cmake functions * remove comment * Revert "Revert "Merge remote-tracking branch 'origin/master' into windowsai"" This reverts commit ade5abe72a4234fdbc3623093c61c02c6b0bdc26. * Fix breaks from merge with ORT master * Brianma/linux (#2917) * don't include windows.h in cross-plat header * add default case for switch statement * signed/unsigned mismatch fix Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> * User/sheilk/winml adapter c api (#2891) * Create winml adapter c api * fix build * make it build * move adapter into onnxruntime core/session * entry point not exported * minor changes * make model metadata work * make tests pass * implement all the model reflection apis on the adapter c abi * update the new ort interface to create a lotus ennvironment with a logging sink * start adding ort env * move all winml code into adapter folder/lib to isolate it * ensure a single logging manager at a time * start refactoring session * refactor session creation interface * add cpu and dml session option methods to adapter * finish session init * stub out interfaces in ort lib to perform similar mechanics of iinference session * enable profiling, and enable schema override * update session register graph transformers * turn back on custom registry for custom ops * Add sync api * add last c api stubs * should build... but all feature values are broken since this is in flight to moving all implementation details into ivalue * remove ep adapter header * Implement DML execution provider functions from adapter (#2846) * Implement DML execution provider functions from adapter * Use functions in OnnxruntimeEngine.cpp * make map/sequence type_infos freeable, and start implementing ivalue * make it build again * implement value methods * implement remaining methods * remove com adapter abi * check dml session * cache the allocator on ivalue * check if resource is cpu/gpu when access its mutable data * update tensor * mismatched parentheses * fix tensor base and binding obj * it evaluates tensors! sometimes... * minor fixes * enable gpu evals * wrapper all existing winml adapter apis with API_IMPL to try catch (#2854) * update winml... tensor strings are broken, need to template tensorbase to do different things for strings * make tensor strings work with 2 copies in/2 copies out * Fix tensor string and allocator bug * make maps work again... needs some fixes still * Make it build! * enable map inputs * map outputs * unbound outputs for sequences and maps * User/xianz/merge windowsai (#2883) * Packaging pipeline changes for VS 2019 (#2711) * Tiny fix to codegen * Simplify cache implementation and avoid static variables that may carry over between models * Extend DML kernels (#2641) * Additional DML operators * Check unsupported attributes and inputs * Address PR comments * Add kernel capability function used for partitioning, and re-enable stride-based int64 support based on value range * Fix test failures * Build fix * PR comments * Update Nuphar tutorial notebook (#2721) 1. Reflect int8 GEMV improvements for multi-threading from #2696 2. Add notes on multi-threading control using OpenMP 3. Add samples of running multi-isa AOT, and show int8 GEMM differences between AVX and AVX2 4. Add rnn_benchmark example to resolve #1993 * Add schema for new Qops (#2611) * Add schema for new Qops * adding shape inference + qlinearaveragepool * plus review comments * plus review comments * updates per review comments * plus review comments * [server] Add supposed for model_name and model_version as cli parameter (#2708) * remove 64bit warning message from python validation. (#2727) * MLAS: ARM64 build fix (#2734) fix bad usage of vreinterpret to cast vector element types * Fix broken python docs links (#2740) * Fix build on Mac OS (#2731) mac os ld doesn't support --while-archive, correct option is -all_load * fix ngraph wheel (#2737) * fix ngraph wheel 1.1.0 onnxruntime_ngraph wheel doesn't work * remove libdnnl.so in nGraph Libs * make it easy to compare * Split onnxruntime server to a separated folder (#2744) * Fix build for Python 3.8 (#2747) * Fix build for Python 3.8 * Update protobuf to 3.11.2 (#1928) Update protobuf to 3.11.2 (#1928) * Change default optimization level to All (from Basic) (#2745) * change default optimization level to All (from Basic) * fix test * fix c# test * Update numpy to 1.18 (#2758) * Update numpy to 1.18 * Pipeline changes for python 3.8 (#2753) 1. Pipeline changes for python 3.8 2. Fix a regression in setup.py which was just introduced in the previous commit. Please notice, we still haven't made python 3.8 + Windows + CUDA work. * Add basic stacktrace output for posix debug builds. (#2749) * [NupharEP] fix a race condition when multiple sessions running different models concurrently (#2772) * Revert "Change default optimization level to All (from Basic) (#2745)" This reverts commit 56bb503c2f26474b6613bcb2a198691a11dcef38. * Fix typo in error message (#2736) * Rename MKL-DNN to DNNL to fix broken link (#2730) * Fix nightly build version number issue * Pass BUILD_BUILDNUMBER to linux docker * Disable featurizers in python packages * Import more featurizers (#2781) Make kernels non-template. Add input constraint for learnt data. Add min_max_scalar_transformer, robust_scalar_transformer, inputation_marker_transfomer, label_encoder_transformer, missing_dummies_transformer along with tests. Advance Featurizers library commit. * Implement a more stable softmax (#2715) * Implement a more stable SoftMax e^x is represented as infinity if x is large enough, like 100.f. Infinity divided by Infinity is a NAN. Thus, softmax gets a NAN if one or more item are large enough. A math transform as below is leveraged to get a stable softmax: e^xi/(e^x1 + ...e^xn) = e^(xi - max) / (e^(x1 - max) + ... + e^(xn - max)) And for convenience, force max to 0.f if all xi are negative * Contributing: Fix a typo (#2784) * ACL EP GEMM improvements (#2780) When it is posible we use a fully connected layer instead of the gemm implementation. This will let the library use the best implementation based on the input data. * ACL EP convolution improvements (#2774) Added the optimized implementation for depthwise convolution for both ACL v19.02 and ACL 19.05. Also the pointwise convolution seems to be more optimal in the CPU implementation so we opted for that instead. * Add script for release Nuget validation (#2719) * Initial commit * Nits * Disable a test temporarily * Change working directory * Test * Add download python step * Test update * More changes * Fix space issue * Fix * Verify nuget signing * Fix * Spaces * PR feedback * Nit * Fix * Fix * Remove temporary changes * add uint8 support to where op (#2792) * Improve bert optimization script: (#2712) (1) Move input int64=>int32 conversion to embed layer fusion. (2) Output epsilon attribute for LayerNormalization fusion. * add session creation time cost. (#2798) * ML.NET team needs featurizers within a package (#2789) Add auto ml featurizers to Windows, MacOS as well as to GPU packaging-pipelines. * Initialize max of softmax with lowest of float (#2786) * MLAS: update SGEMM threading parameters (#2808) * add interface to copy batch tensors. (#2807) * add interface to copy batch tensors. * onnxruntime * speed up Windows TRT CI (#2811) * don't run cuda tests if building with tensorrt * remove unnecessary build options for win trt ci * refactor win gpu tensorrt ci yml * --numpy_version=1.17 * update * update * azcopy and cuda path * Update test data (#2356) * Add timeseries imputer transformer featurizer kernel (#2813) Make kernels non-template. Add input constraint for learnt data. Fixup tests. Add two more featurizers along with tests. Tests fail. min_max_scalar_transformer robust_scalar_transformer Fix tests serialized stream by prepending version bytes. Add inputation_marker_transfomer and the test. Fix up float/double type designations. Added label_encoder_transformer along with a test. string_throw case is broken at the momement. Fix labelencodertransfomer_test.cc string_throw case Rename maxabsscalertransformer_test.cc Add MissingDummiesTransformer along with the test. Update manifest. Add TimeSeriesImputerTransformer definition, implementation and tests * Fix memory leak in TRT (#2815) * fix memory leak issue * revert EP_FAIL on enueueV2 * Add manifest missing comma * Run static code analyzer on most of our code (#2817) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * update quantization doc (#2783) * update documentation for quantization script * plus some spell corrections * Filter CPU case for IsFloat16Supported (#2802) * update default optimization level + fix gemm_activation fusion (#2791) * update defualt optimization level + fix gemm_activation fusion * fix typo * add unit test and incorporate review comments * fix test comment * Fix dnnl wheel package name (#2823) * Append '-dnnl' to whl package name when --use_dnnl * Update build.py * Update Ubuntu & TensorRT version in README (#2820) Dockerfile.tensorrt is using nvcr.io/nvidia/tensorrt:19.09-py3 as base Image, update Ubuntu and TensorRT version according to https://docs.nvidia.com/deeplearning/sdk/tensorrt-container-release-notes/rel_19-09.html#rel_19-09 * Merge fixes * Add OneHotEncoder and HashOneHotEncoder kernels. (#2830) Add defs and imlementation for OneHotEncoders, adjuist date_time_transformer kernel and test. Add OneHotEncoder kernel test. Add HashOneHotVectorizerTransformer unit test. This does not link due to multiple definitions of functions that are included into header from a CPP file. * Upgrade gtest to the latest version (#2827) WinML would like to update the googletest submodule. They want some newer features (namely GTEST_SKIP to skip tests programmatically and be able to skip entire fixtures easily) and would need to update the submodule version. However, because the new version of code hit a bug in gcc, even though the bug is already fixed in the latest gcc but we're using gcc 4.8.x and it won't get patched for the bug, so we have to do a compromise, change our code a little bit to make it work. The gcc bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=51213 * Add support for int64_t for topk CPU. Fixes github issue #2806. (#2833) * Ignore allocator type in ExecutionProviders allocator map. Make default initialization of OrtMemoryInfo more clearly invalid. (#2768) * Remove allocator type from the key comparison in ExecutionProviders. Remove usage of DummyArena as it's no longer necessary. * Fix x86 tests where arena allocator is disabled. Make initialization of OrtMemoryInfo clearer by adding Invalid enum value. * Make OrtValueNameIdxMap::MaxIdx more intuitive. * Convert ExternalProject Featurizers into git submodule (#2834) Add git submodule for Featurizer library. Update cmake to build for git submodule. * add domain check for nodes + update documentation (#2831) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Fix Linux CUDA nuget packaging pipeline break * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * update optimization doc for BERT related fusions (#2819) * Add bert related transformers to doc * Add execution provider and comment for bert optimizations * Add comment about accuracy impact of approximation * Fix warnings that cause build to fail * MLAS: enable threading for quantized GEMMs (#2844) * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * remove old winmladapter cpp Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * move sequence implementation into ort lib... still commented out... need to turn back on... * begin sequence implementation * make maps and sequences work * fix broken tests * remove dead code * misc cleanup * CR feedback * User/xianz/winml adapter c api (#2869) * wrapper all existing winml adapter apis with API_IMPL to try catch * Return HR or Throw for WinML adapter APIs if failed * undo macro wrapper for two places * Wrap error macros around ort apis, too. * address CR feedback #2 * add more api throw/return macros * Revert changes no longer needed * revert changes to cxx api * format winml lib.ort and winml adapter * remove static pheonix singleton Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * missing use_dml check in winml_adapter_session (#2930) * --use_dnnl flag was mangled in merge (#2931) * use dml macro not wrapping custom registry code (#2934) * Disable LNK4199 winml_dll to enable cuda builds (#2936) * Disable LNK4199 in winml_dll * linkler->linker * LearningModelSessionAPITestGpu.CreateSessionWithCastToFloat16InModel should return DXGI_ERROR_UNSUPPORTED when FP16 not supported (#2937) * Disable LNK4199 in winml_dll * linkler->linker * Need to return DXGI_ERROR_UNSUPPORTED when Model does not support fp16 * Publish build symbols (#2939) * Publish build symbols * Don't upload PDBs for .exe files * Make x86 build (#2943) * fix last remaining size_t/int64_t warnings->errors (#2948) * TensorString, Sequences and Maps use the first allocator, but should use the cpu default allocator. (#2952) * fix tensor string allcoator * clean up default allocator usage for strings in winml lib/api.ort Co-authored-by: Ryan Lai <ryalai96@gmail.com> * Handle tensor shape of zero (#2954) Co-authored-by: Ryan Lai <ryalai96@gmail.com> * CR feedback (#2970) * CR feedback * fix weird formatting on privacy readme * Add 'All rights reserved.' everywhere * readd all rights reserved to winml_provider_factory.h * remove extra space in comment * remove extra whitespace * fixes post master merge * remove winml from nuget gpu pipeline * set IR VERSION on generated_model in rnn_benchmark (#2972) * Fix slice conformance failures (#2908) Co-authored-by: Adrian Tsai <adtsai@microsoft.com> Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Paul McDaniel <paul_mcdaniel@hotmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Dwayne Robinson <fdwr@hotmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com>
2020-02-05 01:12:19 +00:00
[marshaling_behavior(agile)]
[dualapipartition(1)]
runtimeclass ImageFeatureDescriptor : ILearningModelFeatureDescriptor
{
//! Specifies the pixel format (channel ordering, bit depth, and data type) of the pixel data.
Windows.Graphics.Imaging.BitmapPixelFormat BitmapPixelFormat{ get; };
//! Specifies the alpha mode of the pixel data.
Windows.Graphics.Imaging.BitmapAlphaMode BitmapAlphaMode{ get; };
Sync ORTModule branch with master and fix tests (#6526) * Deprecate Python global configuration functions [Part 1] (#5923) Enable options to be set via execution provider (EP)-specific options and log deprecation warning from current global configuration functions. * remove dnnl_dll_path from post build copy (#6142) * Model Fusion For Bart (#6105) Fusion fix for Bart models * Unify IExecutionProvider and IExecutionProviderFactory interfaces (#6108) * Remove Provider_IExecutionProvider and make the internal IExecutionProvider usable by shared providers * Change Provider_IExecutionProviderFactory to be the core version. * Enable running the mnist_training sample without cuda (#6085) Signed-off-by: George Nash <george.nash@intel.com> * nnapi add min max support (#6117) * Fix CUDA test hang: (#6138) - Make condition check in `CUDAAllocatorTest` to ensure CUDA device is present. * Fix TensorRT kernel conflict issue for subgraphs of control flow operators (#6115) * add static subgraph kernel index * change kernel naming to avoid conflicts * Add gradient registration for Abs. (#6139) * Partition initial optimizer state for Zero-1 (#6093) * Initial changes * Working changes * Working changes * Cleanup * fix windows CI * Review comments * review comments * Fix edge case in BFCArena where allocation failures could lead to an infinite loop. (#6145) #4656 * Revert "work around of the build break in mac (#6069)" (#6150) This reverts commit 3cae28699bed5de1fcaadb219fa69bae0fc3cee8. * Fix clean_docker_image_cache.py detection of image pushes. (#6151) Fix clean_docker_image_cache.py detection of image pushes. They were being ignored because the expected HTTP status code was wrong. For pushes, it's 201 instead of 200. * MLAS: add NEON version of int8 depthwise convolution (#6152) * Using a map of of ops to stages as input of partition function. (#5940) * New partition algorithm running before AD * Convert cut_group_info into device map. Work in progress -- works for bert-tiny with pp=2 * Removing code for partition of bwd graphs * Remove old code * Adding some verification code * Handle Shared Initializer * Renaming rank with stage * Added first unit test * new test * redundant check * undo change in bert * Moved cut-based partition to testing utils file Co-authored-by: xzhu1900 Co-authored-by: wschin * New conversion function and tests * minor * remove test that is not needed2 * improve GetDeviceAssignment and PR comments * minor changes * PR comments * improving documentation and variable naming * add documentation * Variable naming and docs * more doc improvements * more doc improvements * missing static cast * Fix test file for windows * Fix test file for windows * Fix test file for windows * stage id is not the same as rank id * PR comments * PR comments * More comments * More comments * Minor fix to satisfy c++14 (#6162) * Deprecating Horovod and refactored Adasum computations (#5468) deprecated horovod submodule refactored adasum logic to be ort-native added tests for native kernel and e2e tests * Update TensorRT-ExecutionProvider.md (#6161) * Bugfix for topk cuda kernel (#6164) * fix the issue that std::numeric_limits cannot handle half type * adding a test Co-authored-by: Du Li <duli@OrtTrainingDev4.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net> * Revert "Fuse MatMulIntegerToFloat only when scales are scalar (#6008)" (#6169) This reverts commit f2dcba7afe0d42ebdaaef0c6cdf913a1156c9e98. * Remove ignored build warnings for pybind on Mac (#6165) * save_checkpoint, load_checkpoint and aggregate_checkpoints (#6136) * save_checkpoint and load_checkpoint implementations * checkpoint aggregation logic * unit tests for save_checkpoint, load_checkpoint and aggregate_checkpoints * Don't try to bind unused inputs in the Training frontend (#6166) * Update documentation for contributing a PR and add deprecation notices for PyOp and ORT server. (#6172) * aggregate model states only for the case when mixed precision was true (#6176) * [NNAPI EP] Enable per-channel quantization for QlinearConv (#6155) * Enable qlinearconv per-channel quantization * Fix the android CI test failure * Add Android Version Check for Per-Channel Quant * Address PR comments * Fix some minor issues * Add verification of per-channel zero points * Make the error tolerance configurable * Fix typo in BERT pretraining script (#6175) A misplaced `}` meant that the `'enable_adasum'` option was interpreted incorrectly, causing the test to fail. * Update get_docker_image.py to enable use without image cache container registry. (#6177) Update get_docker_image.py to enable use without image cache container registry. * Helper for compiling EP to generate deterministic unique ids for use in MetaDef names (#6156) * Create a helper for generating unique ids that can be used by an EP that creates compiled nodes and needs ids to be deterministic for a model when used in multiple sessions. Added to IExecutionProvider as this can potentially be used by all compiling EPs and is more robust than a simplistic counter (although EP implementer is free to choose either approach). * Restructure the helper so it can be called across the EP bridge. Add ability to call id generation helper from EP bridge - convert DNNL EP to use helper to validate Address issue where a new Model may be loaded into the same address as a previous one. - hash the bytes in the Graph instance (1728 bytes currently) to use as the key to the full hash for the model Add lock around id generation to ensure no issues if multiple sessions partitions graphs at exactly the same time. - Extremely unlikely but would be hard to debug and the locking cost is not an issue as it's only incurred during graph partitioning and not execution. * Backend APIs for checkpointing (#5803) * Add backend API GetOptimizerState and GetModelState * add GetPartitionInfoMap * Android coverage dashboard (#6163) * Write the report to a file. * Post code coverage to the Dashboard database. * Add usage details of unified MCR container image (#6182) Going forward, a single unifed docker image will be published in MCR. The hardware accelerator target choice will have to be made in the application using OpenVINO EP's runtime config options. * improve perf for softmax (#6128) * improve perf for both gathergrad and softmax * revert the change in gathergrad and will be done in another PR. * address comments from code review. * Tune fast Gelu to use exp(x) instead of tanh(x) on Rocm platform (#6174) * tune fast gelu to use exp(x) instead of tanh(x) on rocm * update to use expression 2/(1+exp(-2x))-1 for stability * Add Status.csv to EP Perf Tool (#6167) * merge master, keep postprocess status commit * download float16.py everytime * removing hardcoded values * Lochi/quantization tool for trt (#6103) * Initial implementation of generating calibration dynamic range table * Initialize validation support for Quantization * Initialize validation support for Quantization (cont.) * Improve validation support for Quantization * Improve validation support for Quantization * Rewrite/Refine for calibration and validation * Rewrite/Refine for calibration and validation (cont.) * Refine code * Refine code * Add data reader for BERT * Add flatbuffers to serialize calibration table * Refine code and add BERT evaluation * Refine the code * minor modification * Add preprocess/postprocess of vision team yolov3 and refine the code * Update annotation * Make bbox cooridates more accurate * Fix bug * Add support of batch processing * Batch processing for model zoo yolov3 * Add batch inference for evaluation * Refine the code * Add README * Add comments * Refine the code for PR * Remove batch support checking in data_reader and refine the code * Refine the code for PR * Refine the code for PR review Co-authored-by: Olivia Jain <oljain@microsoft.com> * Implement ScatterND for CUDA EP (#6184) * Condition fix in Resize operator (#6193) * Clean up checkpoint tests to use the new checkpoint functions (#6188) * add deprecation warning for old checkpoint functions * update all the distributed checkpoint tests to use new checkpoint functions * Implement comparing outputs that are sequence of maps of strings to floats (#6180) * Implement conversion from ortvalue to Itensor for string tensors and comparing sequence of maps of strings to floats * PR comments * Dockerfile to build onnxruntime with ROCm 4.0 * Add ability to skip GPU tests based on GPU adapter name (#6198) * Implement conversion from ortvalue to Itensor for string tensors and comparing sequence of maps of strings to floats * PR comments * Add ability to skip gpu tests according to adapter description * spacing * spacing * spacing * Openvino ep 2021.2 (#6196) * Enabling fasterrcnn variant and vehicle detector * changes for 2021_2 branch * yolov3_pytorch commit * fixed braces in basic_backend.cc * ci information added * faster rcnn variant and vehicle detector changes were made in 2021.1 and not in 2021.2 * some changes to support unit tests * disable some tests which are failing * fix myriad tests for vehicle detector * Did some cleanup *cleaned up comments *Disabled Add_Broadcast_0x1 and Add_Broadcast_1x0 tests on MYRIAD_FP16 backend due to a bug *cleaned up capability_2021_2.cc file *Removed extra conditions which were added for some validation in backend_utils Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com> * yolov3 pytorch workaround to ensure that the output names are matched * gemmoptest fixed on myriad * Fixed MYRIADX CPP Test Failures *Expand,GatherND,Range,Round op's are only supported in model *where op with float input data types are not supported and fixed *Scatter and ScatterElements op's with negative axis are fixed *Reshape op with 0 dim value are not supported and fixed *Disabled InstanceNorm_2 test on MYRIADX Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com> * make changes to yolov3 pytorch * Fixed python unit tests *Fixed failing python tests on vpu, GPU and CPU Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com> * Fixes POW op failures on GPU_FP16 Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com> * Clean up capability_2021_2.cc Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com> * Updated docx for MultiThreading option *Added extra info on setting the num_of_threads option using the API and it's actual usage Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com> * fixed slice and removed extra prints * Disabled failing python tests Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com> * Minor changes added in capabilty_2021_2 Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com> * made changes to slice to avoid failures * Disabling FP16 support for GPU_FP32 ->Inferencing an FP16 model on GPU_FP32 leads to accuracy mismatches. so, we would rather use GPU_FP16 to infer an FP16 model on GPU Device Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com> * Updated docx for Inferencing a FP16 Model Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com> * fix for mask rcnn * Script for installing openvino from source * Updated with openvino 2021.2 online installation * code comment fixes fixed accuracy mismatch for div * Update OpenvinoEP-ExecutionProvider.md updated for 2021.2 branch * Update README.md updated dockerfile documentation * Update BUILD.md build.md update documentation * permissiong change of install_openvino.sh * made changes to align with microsoft onnxruntime changes * Updated with ov 2021.2.200 Co-authored-by: suryasidd <surya.siddharth.pemmaraju@intel.com> Co-authored-by: sfatimar <sahar.fatima@intel/com> Co-authored-by: MaajidKhan <n.maajidkhan@gmail.com> Co-authored-by: mohdansx <mohdx.ansari@intel.com> * Fix a memory leak in test_inference.cc (#6201) * Fix a memory leak in test_inference.cc * Use TArray in AMD element-wise kernels, rather than manually copying memory to device. * Remove most ROCm-specific element-wise code and reuse CUDA element-wise code. * Minor change to improve performance for operator Pad. (#5537) * small improvment for pad * Support double for operators Log, Reciprocal, Sum (CPU) (#6032) * Support double for operators Log, Reciprocal, Sum * remove tesdt erf_double * Support double for operators Where, LpNormalisation (#6034) * Support double for operators Relu, Tanh, Sigmoid (#6221) * Fix ImportError in build.py (#6231) There is a possible ImportError where build.py can import the wrong 'util' package if there are others present in `sys.path` already * Removed executor todo that looks dead. (#6234) * Remove MKLML/openblas/jemalloc build config (#6212) * Remove python 3.5 * Update the readme file * Upgrade build.py to assert for python 3.6+ Upgrade build.py to assert for python 3.6+ as python 3.5 cannot build anymore todays master. * Support MLFloat16 type in Pow opset-12 CUDA kernel (#6233) * MLAS: handle MlasGemm(M/N/K==0) cases (#6238) * Support double for operator TopK + fix one bug in TopK implementation for GPU for double (#6220) * Support double for operator TopK * add static classes for topk/double * fix cast issue in topk * Support double for operator Gemm + fix bug in gemm implementation for cuda, rocm when sizeof(type) != sizeof(float) (#6223) * Support double for operator Gemm * fix type size while copying data in gemm operator for GPU * fix type in gemm implementation for rocm * Support double for operator ReduceMean, ReduceLogSumExp (#6217) * Support double for operators ReduceMean, ReduceLogSumExp * Support double for operator ArgMin (#6222) * Support double for operator ArgMin * add test specifically for double * add new test on pai-excluded-tests.txt * Update BUILD.md * Update manylinux docker image to the latest (#6242) * Fix allocator issue for TensorRT IOBinding (#6240) * Fix issue: https://github.com/microsoft/onnxruntime/issues/6094 Root cause: we didn't expose the OrtMemoryInfo for TRT, so it will cause issue if user want use IObinding for Tensorrt. Short term fix, add the OrtMemoryInfo for TRT. Long term should unify the allocator for CUDA and TRT * Tune BiasGeluGradDx kernel in approximation mode to avoid tanh(...) on Rocm (#6239) * bias gelu grad use exp(...) instead * update cuda to rocm * missing semicolon * comment * remove dockerfile * missing factor of two * Refactor EP Perf Tool (#6202) * merge master, keep postprocess status commit * download float16.py everytime * using variables to reference eps * adding ACL EP to ep perf tool * accuracy with absolute tolerance configurable * add acl to dict + remove commented line * Documentation for distributed CI tests pipeline (#6140) * Remove a debug log in provider_test_utils.cc (#6200) * Add the Concat Slice Elimination transform, fix constant_folding transform (#5457) * Add concat slice transform + test * Cosmetic improvements in concat slice transform * Remove unrelated file, fix comment, fix constant folding bug * Add test onnx graph * fix windows build * Review comments * review comment Co-authored-by: Aishwarya <aibhanda@OrtTrainingDev4.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net> * Add MakeStringLite which uses current locale, update some MakeString call sites to use it instead. (#6252) * Add MakeStringLite which uses current locale, update macros to use that to generate messages. * Convert calls to MakeStringLite(). * Liqun/speech model loop to scan (#6070) Provide a tool to convert Loop to Scan for Nuphar performance Fix Nuphar CI pipeline failures. Co-authored-by: liqun <liqun@OrtTrainingDev4.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net> * model parallel refinement (#6244) * Megatron Transformation as a seperate step * remove useless header * clang formating * Re-Structure megatron transformer for subsquent changes * fix comments * Allow querying a GraphProto's doc_string as part of ModelMetadata (#6248) * Fix Linux/Mac error message on input type mismatch (#6256) * add bfloat16 to gathergrad type constrains (#6267) Co-authored-by: Cheng Tang <chenta@microsoft.com> * Fix VS 2017 build break (#6276) * Deprecate Python global configuration functions [Part 2] (#6171) Update Python API to allow more flexibility for setting providers and provider options. The providers argument (InferenceSession/TrainingSession constructors, InferenceSession.set_providers()) now also accepts a tuple of (name, options dict). Fix get_available_providers() API (and the corresponding function in the C API) to return the providers in default priority order. Now it can be used as a starting point for the providers argument and maintain the default priority order. Convert some usages of the deprecated global configuration functions to use EP-specific options instead. Update some EP-specific option parsing to fail on unknown options. Other clean up. * Add script to preprocess python documentation before publishing (#6129) * add script to preprocessing python documentation before publishing * rename past to past_key_values for GPT-2 (#6269) rename past to past_key_values for transformers 4.* * Rename MakeString and ParseString functions. (#6272) Rename MakeString to MakeStringWithClassicLocale, MakeStringLite to MakeString, *ParseString to *ParseStringWithClassicLocale. Add missing pass-through versions of MakeStringWithClassicLocale for string types. * Increase timeout for Linux GPU CUDA11 build. (#6280) * Add helper to compare model with different precision (#6270) * add parity_check_helper.py * add real example * remove lines * Fix Min/Max CPU kernels for float16 type (#6205) * fix data_ptr assertion error for past_sequence_length=0 in GPT-2 (#6284) fix io binding crash for past_sequence_length=0 * A list of changes in transformers tool (#6224) * longformer fp16 e2e * add fp16/fp32 parity check helper file * excludes nodes with subgraph in profiling * use onnxconverter_common to do fp32->fp16 * add version check for onnxconverter_common * remove helper file * add pkg installation on notebooks and script * Workaround for static_cast<double>(half) * Add workaround to remove ROCm-specific binary-elementwise files. * Update nuget build (#6297) 1. Update the ProtoSrc path. The old one is not used anymore. 2. Regenerate OnnxMl.cs 3. Delete some unused code in tools/ci_build/build.py 4. Avoid set intra_op_param.thread_pool_size in ModelTests in OpenMP build. 5. Fix a typo in the C API pipeline. * Enable ONNX backend test of SequenceProto input/output (#6043) * assert sequence tensor and remove skips * update testdata json * use ONNX 1.8 in cgmanifest.json * use previous commit to workaround * update ONNX commit ID in docker * skip test_maxpool_2d_dilations test for now * update function name * add --sequence_lengths option (#6285) * more dtype for Equal CUDA kernel (#6288) Co-authored-by: Vincent Wang <weicwang@microsoft.com> * Force reinstall onnx python package on Windows (#6309) * update transformers required package versions (#6315) * Remove abs in LpPool (#6303) * Support 1D input for Conv + Mul/Add fusion optimizer with test (#6295) * Support 1D input (N C H) for Conv + Mul/Add fusion optimizer with test cases and test models. * Add longformer to python package (#6314) * add longformer to python package * move test related script and data to a new folder * Avoid false sharing on thread pool data structures (#6298) Description: This change adds alignment and padding to avoid false sharing on fields in the thread pool. It also adds a new microbenchmark to profile thread-pool performance over short loops. Motivation and Context MobileNet on a 2*12-core system showed a performance gap between the ORT thread pool and OpenMP. One cause appeared to be false sharing on fields in the thread pool: ThreadPoolParallelSection::tasks_finished (which the main thread spins on waiting for workers to complete a loop), and the RunQueue::front_ and back_ fields (used respectively by the worker thread and the main thread). The additional micro-benchmark BM_ThreadPoolSimpleParallelFor tests performance of loops of different sizes at different thread counts. The results below are on a machine with 2*14-core processors (E5-2690 v4) running with 1, 14, 15, and 28 threads. For each test, the microbenchmark has N threads run a loop with N iterations; hence a perfect result is for the time taken to be constant as additional threads are added (although we will also see power management effects helping at very low thread counts). The loop durations (100000, 10000, 1000) correspond roughly to 200us, 20us, and 2us on this machine. Before change: BM_ThreadPoolSimpleParallelFor/1/1/100000/real_time 17153 us 17154 us 32 BM_ThreadPoolSimpleParallelFor/14/14/100000/real_time 22553 us 22553 us 30 BM_ThreadPoolSimpleParallelFor/15/15/100000/real_time 21521 us 21521 us 29 BM_ThreadPoolSimpleParallelFor/28/28/100000/real_time 24111 us 24111 us 24 BM_ThreadPoolSimpleParallelFor/1/1/10000/real_time 1719 us 1719 us 407 BM_ThreadPoolSimpleParallelFor/14/14/10000/real_time 3409 us 3409 us 200 BM_ThreadPoolSimpleParallelFor/15/15/10000/real_time 3541 us 3541 us 201 BM_ThreadPoolSimpleParallelFor/28/28/10000/real_time 4576 us 4576 us 151 BM_ThreadPoolSimpleParallelFor/1/1/1000/real_time 174 us 174 us 4017 BM_ThreadPoolSimpleParallelFor/14/14/1000/real_time 1586 us 1586 us 402 BM_ThreadPoolSimpleParallelFor/15/15/1000/real_time 1586 us 1586 us 397 BM_ThreadPoolSimpleParallelFor/28/28/1000/real_time 2864 us 2864 us 232 After change: BM_ThreadPoolSimpleParallelFor/1/1/100000/real_time 17160 us 17160 us 33 BM_ThreadPoolSimpleParallelFor/14/14/100000/real_time 20989 us 20989 us 31 BM_ThreadPoolSimpleParallelFor/15/15/100000/real_time 22286 us 22286 us 31 BM_ThreadPoolSimpleParallelFor/28/28/100000/real_time 24631 us 24631 us 25 BM_ThreadPoolSimpleParallelFor/1/1/10000/real_time 1718 us 1718 us 407 BM_ThreadPoolSimpleParallelFor/14/14/10000/real_time 2868 us 2868 us 242 BM_ThreadPoolSimpleParallelFor/15/15/10000/real_time 2907 us 2907 us 240 BM_ThreadPoolSimpleParallelFor/28/28/10000/real_time 3872 us 3872 us 186 BM_ThreadPoolSimpleParallelFor/1/1/1000/real_time 175 us 175 us 3938 BM_ThreadPoolSimpleParallelFor/14/14/1000/real_time 933 us 933 us 659 BM_ThreadPoolSimpleParallelFor/15/15/1000/real_time 912 us 912 us 591 BM_ThreadPoolSimpleParallelFor/28/28/1000/real_time 1976 us 1976 us 317 * fix opset imports for function body (#6287) * fix function opsets * add tests and update onnx * changes per review comments * add comments * plus updates * build fix * Remove false positive prefast warning from threadpool (#6324) * Java: add Semmle to Java publishing pipelines (#6326) Add Semmle to Java API pipeline Add security results publishing and add Java GPU. * Quantization support for split operator with its NHWC support (#6107) * Make split working for quantization. * NHWC transformer support for split operator * Refactor some according to Feedback. Will add test cases soon. * Fix build error on windows. * Add test case for split op on uint8_t support * Add nhwc_transformer_test for split uint8_t support * Some change according to PR feedbacks. * Liqun/enable pipeline parallel test (#6331) enable pipeline parallel test Co-authored-by: liqun <liqun@OrtTrainingDev4.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net> * Use onnxruntime_USE_FULL_PROTOBUF=OFF for the cuda execution provider (#6340) This removes a special case of the cuda EP. * MLAS: add fallback implementation for quantized GEMM (#6335) Add a non-vectorized version of the kernel used for the quantized version of MlasGemm. * Delete float16.py (#6336) No longer needed. Also doesn't pass policheck. * Enable add + softmax fusion for Rocm platform (#6259) * add bias softmax; tests appear to pass * check fusion occurs for rocm as well * check for rocm provider compatible as well * build for cpu scenario as well * try again; broader cope * proper scope on kGpuExecutionProvider * been editing wrong file * remove commented #include lines * try again due to mac os ci error * try again * test fusion both cuda and rocm to avoid mac ci error * add external data support to tensor proto utils (#6257) * update unpack tensor utilities to support loading external data * more updates * fix test * fix nuphar build * minor build fix * add tests * fix Android CI * fix warning * fix DML build failure and some warnings * more updates * more updates * plus few updates * plus some refactoring * changes per review * plus some change * remove temp code * plus updates to safeint usage * build fix * fix for safeint * changed wording. (#6337) * Remove OpSchema dummy definition. Only needed for Function now, and we can just exclude the method in Function (#6321) * remove gemmlowp submodule (#6341) * [NNAPI] Add pow support (#6310) * Add support for running Android emulator from build.py on Windows. (#6317) * fix the pipeline failure (#6346) * Train BERT Using BFloat16 on A100 (#6090) * traing bert using bf16 * Adam support bf16 * bugfix * add fusedmatmul support * fix after merge from master. * bugfix * bugfix after merge from master * fast reduction for bf16. * resolve comments * fix win build * bugfix * change header file. Co-authored-by: Vincent Wang <weicwang@microsoft.com> * Fix DerefNullPtr issues raised by SDLNativeRules. (#6348) * update quantize to support basic optimization and e2e example for image classification (#6313) update the resnet50-v1 to standard one from onnx zoo. add an example for mobilenet run basic optimization before quantization fix a bug in Clip * Enable graph save for orttrainer (#6333) * Enable graph save for orttrainer * Fix CI * Update orttraining/orttraining/python/training/orttrainer_options.py * Update orttraining/orttraining/python/training/orttrainer_options.py * Update orttraining/orttraining/python/training/orttrainer_options.py * Update orttraining/orttraining/python/training/orttrainer_options.py * Update orttraining/orttraining/python/training/orttrainer_options.py Co-authored-by: Thiago Crepaldi <thiago.crepaldi@microsoft.com> * Add PREfast to python packaging pipeline (#6343) * Add PREfast to python packaging pipeline * fix longformer benchmark io_binding output_buffers (#6345) * fix longformer benchmark io_binding output_buffers * format * import benchmark_helper from parent directory. * Use readelf for minimal build binary size checks. (#6338) * Use readelf for minimal build binary size checks. The on-disk size grows in 4KB chunks which makes it hard to see how much growth an individual checkin causes. Only downside is that the sum of the sections is larger than the on-disk size (assumably things get packed smaller on disk and some of the section alignment constraints can be ignored) * Remove unused function * Java: Set C language warnings to W4 and adjust JNI code (#6347) Set /W3 for C language and fix up JNI warnings. * Pipeline Parallel Experimental Python API (#5815) * Add create session to WinML telemetry to track WinML Usage (#6356) * Fix one more SDL warning (#6359) * fix -Wdangling-gsl (#6357) * Add python example of TensorRT INT8 inference on ResNet model (#6255) * add trt int8 example on resnet model * Update e2e_tensorrt_resnet_example.py * remove keras dependency and update class names * move ImageNetDataReader and ImageClassificationEvaluator to tensorrt resnet example * simplify e2e_tensorrt_resnet_example.py * Update preprocessing.py * merge tensorrt_calibrate * Update calibrate.py * Update calibrate.py * generalize calibrate * Update calibrate.py * fix issues * fix formating * remove augment_all * This added telemetry isn't needed (#6363) * Wezuo/memory analysis (#5658) * merged alloc_plan * pass compilation * Start running, incorrect allocation memory info * add in comments * fix a bug of recording pattern too early. * debugging lifetime * fix lifetime * passed mnist * in process of visualization * Add code to generate chrome trace for allocations. * in process of collecting fragmentation * before rebuild * passed mnist * passed bert tiny * fix the inplace reuse * fix the exception of weight in pinned memory * add guards to ensure the tensor is in AllocPlan * add customized profiling * debugging * debugging * fix the reuse of differnt location type * add rank * add the rank * add fragmentation * add time_step_trace * Add summary for each execution step (total bytes, used/free bytes). * add top k * change type of top k parameter * remove prints * change heap to set{ * add the name pattern * add the useage for pattern * add partition * change to static class * add custom group * remove const * update memory_info * in process of adding it as runtime config * change the memory profiling to be an argument * add some comments * add checks to recored meomry_info in traaining session * set the "local rank setting" to correct argument. * addressing comments * format adjustment * formatting * remove alloc_interval * update memory_info.cc to skip session when there is no tensor for a particular memory type * fix memory_info multiple iteration seg-fault * consolidate mainz changes * fixed some minor errors * guard by ORT_MINIMAL_BUILD * add ORT_MEMORY_PROFILE flag * added compiler flag to turn on/off memory profiling related code * clean up the code regarding comments * add comments * revoke the onnx version * clean up the code to match master * clean up the code to match master * clean up the code to match master Co-authored-by: Jesse Benson <benson.jesse@gmail.com> Co-authored-by: Wei Zuo <wezuo@OrtTrainingDev3.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net> Co-authored-by: wezuo <wezuo@az-eus-v100-32gb-5-worker-mgtbby.eastus.cloudapp.azure.com> Co-authored-by: wezuo <wezuo@az-eus-v100-32gb-5-worker-yclzsf.eastus.cloudapp.azure.com> * Support MLFloat16 in CumSum Cuda op for Opset 14 (#6355) * Add CumSum-14 for Cuda * fix convert_common version retrival (#6382) * Refine auto_pad based pad computation in ConvTranspose (#6305) * Fix SDL warning (#6390) * Add max_norm for gradient clipping. (#6289) * add max_norm as user option for gradient clipping * add adam and lamb test cases for clip norm * add frontend tests * Add the custom op project information (#6334) * Dont use default string marshalling in C# (#6219) * Fix Windows x86 compiler warnings in the optimizers project (#6377) * [Perf] Optimize Tile CPU and CUDA kernels for a corner case (#6376) * Unblock Android CI code coverage failure (#6393) * fix build on cuda11 (#6394) Co-authored-by: Vincent Wang <weicwang@microsoft.com> * Load the model path correctly (#6369) * Fix some compile warnings (#6316) * OpenVino docker file changes to bypass privileged mode Description: Builds and installs libusb without UDEV support, which is used for communicating with the VPU device. Motivation and Context This enables the resulting docker container to be run without '--privileged' and '--network host' options which may not be suitable in deployment environments. * Megatron checkpointing (#6293) * Add bart fairseq run script * Add frontend change to enable megatron * Initial changes for checkpointing * Megatron optim state loading, checkpoint aggregation, frontend distributed tests for H, D+H * Add load_checkpoint changes * Fix CI * Cleanup * Fix CI * review comments * review comments * review comments: * Fix generate_submodule_cgmanifest.py Windows issues. (#6404) * Continue memory planning when unknown shape tensor is encountered. (#6413) * Reintroduce experimental api changes and fix remote build break (#6385) Co-authored-by: Ori Levari <orlevari@microsoft.com> * Add support for custom ops to minimal build. (#6228) * Add support for custom ops to minimal build. Cost is only ~8KB so including in base minimal build. * enable pipeline to run quantization tests (#6416) * enable pipeline to run quantization tests setup test pipeline for quantization * Minor cmake change (#6431) * Liqun/liqun/enable pipeline parallel test2 (#6399) * enable data and pipeline parallism test Co-authored-by: liqun <liqun@OrtTrainingDev4.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net> * Farewell TrainableDropout (#5793) * Deprecate TrainableDropout kernel. * Update bert_toy_postprocessed.onnx to opset 12. * Add more dropout tests. * Fix BiasDropout kernel. Co-authored-by: Ubuntu <OrtTrainingDev3@OrtTrainingDev3.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net> Co-authored-by: Sherlock Huang <bahuang@OrtTrainingDev3.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net> Co-authored-by: Sergii Dymchenko <sedymche@microsoft.com> * fix null dereference warning (#6437) * Expose graph ModelPath to TensorRT shared library (#6353) * Update graph_viewer.cc * Update tensorrt_execution_provider.cc * Update graph_viewer.h * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update provider_api.h * Update provider_bridge_ort.cc * Update provider_interfaces.h * Update provider_interfaces.h * expose GraphViewer ModelPath API to TRT shared lib * add modelpath to compile * update * add model_path to onnx tensorrt parser * use GenerateMetaDefId to generate unique TRT kernel name * use GenerateMetaDefId to generate unique TRT engine name * fix issue * Update tensorrt_execution_provider.cc * remove GetVecHash * Update tensorrt_execution_provider.h * convert wchar_t to char for tensorrt parser * update tensorrt parser to include latest changes * fix issues * Update tensorrt_execution_provider.cc * merge trt parser latest change * add PROVIDER_DISALLOW_ALL(Path) * add tool for generating test data for longformer (#6415) * only build experimental api in redist (#6465) Co-authored-by: Sheil Kumar <sheilk@microsoft.com> * Add an option to save the training graph after optimization (#6410) * expose optimized_model_filepath in SessionOptions as `debug.graph_save_paths.model_with_training_graph_after_optimization_path` in `ORTTrainerOptions` * Share allocator between CUDA EP & TRT EP. (#6332) * Share allocator between CUDA EP & TRT EP. limitation: 1. Does not cover the per-thread allocator created by CUDA EP, still need to figure out the way to remove it 2. Need to have more identifiers to make it able to share CPU allocator across all EPs * fix max norm clipping test in python packaging pipeline test (#6468) * fix python packaging pipeline * make clip norm test compatabile with both V100 and M60 GPUs * Initial version of CoreML EP (#6392) * Bug 31463811: Servicing: Redist (Nuget) conflicts with Microsoft.AI.MachineLearning starting 21H1+ (#6460) * update load library code to have the fullly qualified path * make it work for syswow32 * git Revert "make it work for syswow32" This reverts commit b9f594341b7cf07241b18d0c376af905edcabae3. Co-authored-by: Sheil Kumar <sheilk@microsoft.com> * dequantize 1st input of lstm back if it is quantized (#6444) * [java] Adds support for OrtEnvironment thread pools (#6406) * Updates for Gradle 7. * Adding support for OrtThreadingOptions into the Java API. * Fixing a typo in the JNI code. * Adding a test for the environment's thread pool. * Fix cuda test, add comment to failure. * Updating build.gradle * fix SDL native rule warning #6246 (#6461) * fix SDL rule (#6464) * use tickcount64 (#6447) Co-authored-by: Ori Levari <orlevari@microsoft.com> * Update pypi package metadata (#6354) * Update setup file data * add missing comma * remove python 3.5 * fix typo bracket * Delete nuget extra configs (#6477) * Op kernel type reduction infrastructure. (#6466) Add infrastructure to support type reduction in Op kernel implementations. Update Cast and IsInf CPU kernels to use it. * Fixing a leak in OnnxSequences with String keys or values. (#6473) * Increase the distributes tests pipeline timeout to 120 minutes (#6479) * [CoreML EP] Add CI for CoreML EP (macOS) and add coreml_flags for EP options (#6481) * Add macos coreml CI and coreml_flags * Move save debuggubg model to use environment var * Move pipeline off from macos CI template * Fix an issue building using unix make, add parallel to build script * Fixed build break for shared_lib and cmpile warning * Fix a compile warning * test * Revert the accidental push from another branch This reverts commit 472029ba25d50f9508474c9eeceb3454cead7877. * Add ability to track per operator types in reduced build config. (#6428) * Add ability to generate configuration that includes required types for individual operators, to allow build size reduction based on that. - Add python bindings for ORT format models - Add script to update bindings and help info - Add parsing of ORT format models - Add ability to enable type reduction to config generation - Update build.py to only allow operator/type reduction via config - simpler to require config to be generated first - can't mix a type aware (ORT format model only) and non-type aware config as that may result in insufficient types being enabled - Add script to create reduced build config - Update CIs * merge e2e with distributed pipeline (#6443) merge e2e with distributed pipeline * Fix test breaks in Windows ingestion pipeline (#6476) * fix various build breaks with Windows build * fix runtime errors loading libraries from system32 * add build_inbox check to winml_test_common * use raw string * cleanup * fix dll load Co-authored-by: Sheil Kumar <sheilk@microsoft.com> * Speed up the Mac CI runs (#6483) * expose learningmodelpixelrange property (#5877) * Fix of support api version bug for [de]quantize (#6492) * SDL fixes: add proper casts/format specifiers (#6446) * SDL annotation fixes (#6448) Co-authored-by: Ori Levari <orlevari@microsoft.com> * [OpenVINO-EP] Remove support for OpenVINO 2020.2 (#6493) * Removed OpenVINO 2020.2 support * Updated documentation and build.py * Removed unnecessary libraries from setup.py * Support pad operator in quantization and quantized nhwc transformer. Fix Pad operator bug. (#6325) Support pad operator in quantization tool. Support pad operator in quantized nhwc transformer. Fix pad() operator bug when pad input's inner(right) most axis value is zero for Edge and Reflect mode, it copied wrong value to the cells to be padded. Note the Constant mode will not trigger this bug, as Edge/Reflect need copy value from the already copied array while Constant mode only fill specified value. Add more test cases to cover pad() operator bug fixed here. Fix quantization tools uint8/int8 value overflow issue when quantize weights in python. * Improve work distribution for Expand operator, and sharded LoopCounter configuration (#6454) Description: This PR makes two changes identified while looking at a PGAN model. First, it uses ThreadPool::TryParallelFor for the main parallel loops in the Expand operator. This lets the thread pool decide on the granularity at which to distribute work (unlike TrySimpleParallelFor). Profiling showed high costs when running "simple" loops with 4M iterations each of which copied only 4 bytes. Second, it updates the sharded loop counter in the thread pool so that the number of shards is capped by the number of threads. This helps make the performance of any other high-contention "simple" loops more robust at low thread counts by letting each thread work on its own "home" shard for longer. Motivation and Context Profiling showed a PGAN model taking 2x+ longer with the non-OpenMP build. The root cause was that the OpenMP build uses simple static scheduling of loop iterations, while the non-OpenMP build uses dynamic scheduling. The combination of large numbers of tiny iterations is less significant with static scheduling --- although still desirable to avoid, given that each iteration incurs a std::function invocation. * Update document of transformer optimization (#6487) * nuphar test to avoid test data download to improve passing rate (#6467) nuphar test to avoid test data download to improve passing rate * Fuse cuda conv with activation (#6351) * optimize cuda conv by fused activation * remove needless print out * exclude test from cpu * handle status error from cudnn 8.x * add reference to base class * add hipify * [CoreML EP] Add support for some activations/Transpose, move some shared helpers from NNAPI to shared space (#6498) * Init change * Move some helper from nnapi ep to shared * Add transpose support * Fix trt ci build break * Refine transformers profiler output (#6502) * output nodes in the original order; grouped by node name * add document for profiler * Update to match new test setup. (#6496) * Update to match new test setup. * Add Gemm(7) manually for now. Will fix properly on Monday. It's used by mnist.ort as that is created by optimizing mnist.onnx to level 1 causing 2 nodes to be replaced by a Gemm and the op to be missing from the required list as that is created using the original onnx model. * Enable dense sequence optimized version of Pytorch exported BERT-L on AMD GPU (#6504) * Permit dense seq optimization on BERT-L pytorch export by enabling ReduceSumTraining, Equal, and NonZero on AMD * enable Equal tests * enable fast_matrix_reduction test case * Optimize GatherGrad for AMD GPU (#6381) * optimize gathergrad * address comments Co-authored-by: Weixing Zhang <wezhan@microsoft.com> * add explicit barriers for buffer overread and overrwrite (#6484) Co-authored-by: Ori Levari <orlevari@microsoft.com> * fix sdl bugs for uninitialized variables and returns (#6450) Co-authored-by: Ori Levari <orlevari@microsoft.com> * handle hr error conditions (#6449) Co-authored-by: Ori Levari <orlevari@microsoft.com> * Dnnl training (#6045) * Add ReluGrad and ConvGrad ops for the dnnl provider * the mnist sample is updated to add the --use_dnnl option that will cause the sample to use the dnnl execution provider for nodes that exist in dnnl provider. * Added the ability to find forward ops. Dnnl backward gradient ops require the forward primitive description and workspace from the forward operation. * Enable specifying the execution provider for Gradient Checker Tests * Prevent memory leak when running dnnl_provider in training mode Prevent creating a SubgraphPrimitivePool when the code is built with the ENABLE_TRAINING build flag. Instead create a SubgraphPrimitive directly. The SubgraphPrimitivePool was causing a pool of SubgraphPrimitives to be stashed in a map for reuse. Due to the way the Training Loop uses threads the pool of SubgraphPrimitives were not being reuse instead a new pool of SubgraphPrimitives being created each run. The old pool was not instantly freed. This behavior could be a language error when using thread_local memory. Signed-off-by: George Nash <george.nash@intel.com> * Added fixes to maxpoolgrad and memory leak. Maxpoolgrad will now pass all unit tests. With the conv and convgrad disabled for dnnl, mnist is able to train till 95% Signed-off-by: Chethan Palangotu Keshava <chethan.palangotu.keshava@intel.com> * Fixed misc issues when testing training code with dnnl provider * fix conv_grad dnnl tests with dilation to run dnnl execution provider * update mnist training sample to accept convolution type models convolution models require the input shape to be {1, 28, 28} instead of the flat {728} image that is used for the gemm models this will enable models that require the different shape by adding `--model_type conv` to the command line when running the mnist sample. (while testing a workaround was used see #4762) * Disable weight caching in dnnl conv operator when using training When training we can not use cached weights because the weight will be updated each run. This re-enables dnnl Conv and ConvGrad Ops. The weight caching was the source of the error from Conv when training. * Fix issues found when building grad ops on Linux * The dnnl_convgrad code was over using the scope operator causing a compilation problem. * The dnnl_maxpoolgrad code had a logic error that is was comparing with the source description when it should have been comparing with the destination despription. * Update BUILD.md so it shows DNNL for training * Updated the table of contents. Since the same providers are listed twice. Once for Infrance and again for Training an HTML anchor was added to distinguish the second header from the first for the TOC. * Fix build failure when not using --enable-training build option * reorganize the gradient operators so they are grouped together * Fix issues found when running onnx_backend_test_series.py * Pooling code only supports 2 outputs when built with --enable-training * Address code review feedback * class member variables end in underscore_ * use dst instead of dist to match pattern use elsewhere in DNNL code. * Remove workaround that was introduced to handle problems running convolution based training models. See issue #4762 Signed-off-by: George Nash <george.nash@intel.com> * Isolate training code and code cleanup * Do not build if dnnl_gpu_runtime if enable_training is set training code does not support dnnl_gpu_runtime yet. * Isolated Training code inside ifdefs so that they wont affect project if built without training enabled * Inadvertant changes in whitespace were removed to make code review simpler * Undid some code reordering that was not needed * comments added to closing #endif statments to simplify reading complex ifdefs * Modified the GetPrimitiveDesc functions to return shared_ptr instead of raw pointer. This matches what was done in Pool code and is safer memory code. Signed-off-by: George Nash <george.nash@intel.com> * Address code review issues - whitespace changes caused by running clang-format on the code - Several spelling errors fixed - Removed/changed some ifdefs to improve readability - other misc. changes in responce to code review. Signed-off-by: George Nash <george.nash@intel.com> * Code changes to address code review - Simplify iteration code using `auto` keyword - remove C style cast that was not needed - remove instance variable that was not needed [relugrad.h] - added the execution providers to `ComputeGradientErrorInternal()` and `ComputeTheoreticalJacobianTranspose()` instead of using a pointer to an instance varaible [gradient_checker.h/.cc] Signed-off-by: George Nash <george.nash@intel.com> * Combined the default gradient ops test and dnnl gradient ops test for ConvGrad and MaxPoolGrad into one function with the help of a helper function. This will reduce repeated code. Signed-off-by: Palangotu Keshava, Chethan's avatarChethan Palangotu Keshava <chethan.palangotu.keshava@intel.com> * Replaced the stack used by convgrad to vector so that the vector(used as stack) can be easily cleared everytime the graph is created. This will prevent memory leak from convolution kernels being pushed constantly onto the stack. Signed-off-by: chethan.palangotu.keshava@intel.com * Code clean up and formating updates - Removed empty else statment - updated indentation of code that was causing double curly brackets to look unususal - Changed check for NumDimensions to Size in Relu and ReluGrad error checking code. - isolated training code Signed-off-by: George Nash <george.nash@intel.com> * Restore inadvertantly removed ConvGrad tests When combining the DNNL and CPU version of the ConvGrad tests two test were inadvertantly excluded. This adds back the Conv3d and Conv3d with strides test cases. Signed-off-by: George Nash <george.nash@intel.com> * Add validation to ConvGrad This validates the dimensions of the ConvGrad match the passed in Convolution forward primitive description. The current code for DNNL ConvGrad makes the assumption that the ConvGrad nodes will be visited in the reverse order from the corresponding Conv nodes The added validation will return an error if this assumption is not true. Signed-off-by: George Nash <george.nash@intel.com> * Do not create new execution providers in provider_test_utils This removes the code that generated new execution providers in the OpTester::Run function. This was added because the std::move was leaving the `entry` value empty so subsequent calls would cause a segfault. Problem is this potentially changed the execution_provider because it would create the default provider dropping any custom arguments. When the now removed code was originally added the std::move was causing crashes when the GradientChecker unit tests were run. However, it is no longer causing problems even with the code removed. Signed-off-by: George Nash <george.nash@intel.com> * Change the forward conv stack to a forward conv map This changes how the forward conv kernel is mapped to the bwd ConvGrad kernel the problematic stack is no longer used. The convolution stack made the assumption that the corresponding ConvGrad operator would be visited in reverse order of the forward Conv operators. This was always problematic and was unlikely to work for inception models. Important changes: - The weight_name is added to the ConvGrad dnnl_node making it possible to use the weight_name as a lookup key to find the Conv forward Kernel - the `std::vector fwd_conv_stack_` has been replaced with a `std::map fwd_conv_kernel_map_` - Although it is not needed lock_guards were added when writing to and reading from the fwd_conv_kernel_map_ as well as the fwd_kernel_map_. These should always be accessed by a single thread when preparing the dnnl subgraphs so the guard should not be needed but its added just in case. - Updated the comments ConvGrad.h code to no longer mention the stack. The error check is not removed. It will be good to verify there are no errors as we continue to test against more models. Signed-off-by: George Nash <george.nash@intel.com> Co-authored-by: Chethan Palangotu Keshava <chethan.palangotu.keshava@intel.com> Co-authored-by: unknown <63478620+jeyblu@users.noreply.github.com> * Lochi/refactor yolov3 quantization (#6290) * Refactor the code and move data reader, preprocessing, evaluation to E2E_example_mode * Refactor the code. Move data reader, preprocessing, evaluation to model specific example under E2E_example_mode * refactor code * Move yolov3 example to specific folder and add additional pre/post processing * Print a warning message for using newer c_api header on old binary (#6507) * Fix issues with ArmNN build setup (#6495) * ArmNN build fixes * Update BUILD.md to document that the ACL paths must be specified to build ArmNN * Fix CUDA build error. We don't setup the link libraries correctly/consistently so improve that. * Fix Windows CI builds by updating test scripts to work with numpy 1.20. (#6518) * Update onnxruntime_test_python.py to work with numpy 1.20. Some aliases are deprecated in favor of the built-in python types. See https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations np.array with bytes for entries and dtype of np.void no longer automatically pads. Change a test to adjust for that. * Fix another test script * Fix ORTModule branch for orttraining-* pipelines * Update pytorch nightly version dependency Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Cecilia Liu <ziyue.liu7@gmail.com> Co-authored-by: Ryan Hill <38674843+RyanUnderhill@users.noreply.github.com> Co-authored-by: George Nash <george.nash@intel.com> Co-authored-by: Guoyu Wang <62914304+gwang-msft@users.noreply.github.com> Co-authored-by: Yateng Hong <toothache9010@gmail.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Derek Murray <Derek.Murray@microsoft.com> Co-authored-by: ashbhandare <ash.bhandare@gmail.com> Co-authored-by: Scott McKay <skottmckay@gmail.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Juliana Franco <jufranc@microsoft.com> Co-authored-by: Pranav Sharma <prs@microsoft.com> Co-authored-by: Tixxx <tix@microsoft.com> Co-authored-by: Jay Rodge <jayrodge@live.com> Co-authored-by: Du Li <duli1@microsoft.com> Co-authored-by: Du Li <duli@OrtTrainingDev4.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: baijumeswani <bmeswani@microsoft.com> Co-authored-by: Sergii Dymchenko <sedymche@microsoft.com> Co-authored-by: jingyanwangms <47403504+jingyanwangms@users.noreply.github.com> Co-authored-by: satyajandhyala <satya.k.jandhyala@gmail.com> Co-authored-by: S. Manohar Karlapalem <manohar.karlapalem@intel.com> Co-authored-by: Weixing Zhang <weixingzhang@users.noreply.github.com> Co-authored-by: Suffian Khan <sukha@microsoft.com> Co-authored-by: Olivia Jain <oljain@microsoft.com> Co-authored-by: Chi Lo <54722500+chilo-ms@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Ryan Lai <rylai@microsoft.com> Co-authored-by: Jesse Benson <jesseb@microsoft.com> Co-authored-by: sfatimar <64512376+sfatimar@users.noreply.github.com> Co-authored-by: suryasidd <surya.siddharth.pemmaraju@intel.com> Co-authored-by: sfatimar <sahar.fatima@intel/com> Co-authored-by: MaajidKhan <n.maajidkhan@gmail.com> Co-authored-by: mohdansx <mohdx.ansari@intel.com> Co-authored-by: Xavier Dupré <xadupre@users.noreply.github.com> Co-authored-by: Michael Goin <mgoin@vols.utk.edu> Co-authored-by: Michael Giba <michaelgiba@gmail.com> Co-authored-by: William Tambellini <wtambellini@sdl.com> Co-authored-by: Hector Li <hecli@microsoft.com> Co-authored-by: Aishwarya <aibhanda@OrtTrainingDev4.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net> Co-authored-by: liqunfu <liqfu@microsoft.com> Co-authored-by: liqun <liqun@OrtTrainingDev4.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net> Co-authored-by: pengwa <pengwa@microsoft.com> Co-authored-by: Tang, Cheng <souptc@gmail.com> Co-authored-by: Cheng Tang <chenta@microsoft.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ye Wang <52801275+wangyems@users.noreply.github.com> Co-authored-by: Chun-Wei Chen <jacky82226@gmail.com> Co-authored-by: Vincent Wang <wangwchpku@outlook.com> Co-authored-by: Vincent Wang <weicwang@microsoft.com> Co-authored-by: Luyao Ren <375833274@qq.com> Co-authored-by: Zhang Lei <zhang.huanning@hotmail.com> Co-authored-by: Tim Harris <tiharr@microsoft.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Alberto Magni <49027342+alberto-magni@users.noreply.github.com> Co-authored-by: Wei-Sheng Chin <wschin@outlook.com> Co-authored-by: wezuo <49965641+wezuo@users.noreply.github.com> Co-authored-by: Jesse Benson <benson.jesse@gmail.com> Co-authored-by: Wei Zuo <wezuo@OrtTrainingDev3.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net> Co-authored-by: wezuo <wezuo@az-eus-v100-32gb-5-worker-mgtbby.eastus.cloudapp.azure.com> Co-authored-by: wezuo <wezuo@az-eus-v100-32gb-5-worker-yclzsf.eastus.cloudapp.azure.com> Co-authored-by: Wenbing Li <10278425+wenbingl@users.noreply.github.com> Co-authored-by: Martin Man <supermt@gmail.com> Co-authored-by: M. Zeeshan Siddiqui <mzs@microsoft.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Ori Levari <orlevari@microsoft.com> Co-authored-by: Ubuntu <OrtTrainingDev3@OrtTrainingDev3.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net> Co-authored-by: Sherlock Huang <bahuang@OrtTrainingDev3.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net> Co-authored-by: Sheil Kumar <smk2007@gmail.com> Co-authored-by: Sheil Kumar <sheilk@microsoft.com> Co-authored-by: Ryota Tomioka <ryoto@microsoft.com> Co-authored-by: Adam Pocock <adam.pocock@oracle.com> Co-authored-by: Yulong Wang <f.s@qq.com> Co-authored-by: Faith Xu <faxu@microsoft.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: suryasidd <48925384+suryasidd@users.noreply.github.com> Co-authored-by: RandySheriffH <48490400+RandySheriffH@users.noreply.github.com> Co-authored-by: Weixing Zhang <wezhan@microsoft.com> Co-authored-by: Chethan Palangotu Keshava <chethan.palangotu.keshava@intel.com> Co-authored-by: unknown <63478620+jeyblu@users.noreply.github.com>
2021-02-02 16:59:56 +00:00
[contract(MachineLearningContract, 5)]
{
//Specifies the nominal pixel range of pixel data.
LearningModelPixelRange PixelRange{ get; };
}
Merge windowsai (winml layering) into master (#2956) * Initial Commit * Merged PR 3985217: add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc (#2346) add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc and violating our OS layering requirements. We linked against onecoreuap_apiset.lib in VB so we will continue doing this, but I am still unsure why not to link against onecore instead since that is where we ship. However, since Sheil is the owner of this code we will wait to discuss with him before changing anything. * Initial changes for layering * more snipping to get core into ort * update build instructions to include --build_shared_lib (#2358) * update build instructions to include --build_shared_lib * fix line breaks * Task 23998197: add winml_lib_core into onnnxruntime.dll (#2368) * Task 23998197: add winml_lib_core into onnnxruntime.dll * PR feedback build break on perf_test * return proper error when the model path isn't found (#2391) * LearningModelSession is cleaned up to use the adapter, and parts of b… (#2382) this is a big PR. we are going to move it up to layer_dev , which is still a L3 so we are still safe to do work there agile. we are going to move this into the L3 so that ryan can start doing intergration testing. we will pause for a full code review and integration test result prior to going into the L2. >>>> raw comments from previous commits >>> * LearningModelSession is cleaned up to use the adapter, and parts of binding are. * moved everything in the winmladapter made it all nano-com using, WRL to construct objects in the ORT side. base interfaces for everythign for winml to call cleaned up a bunch of winml to use the base interfaces. * more pieces * GetData across the abi. * renamed some namepsace cleaned up OrtValue cleaned up Tensor cleaned up custom ops. everything *but* learnignmodel should be clean * make sure it's building. winml.dll is still a monolith. * model moved over. everything builds clean. step ! * weak ref comment * Layer dev paulm (#2408) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * Layer dev paulm (#2414) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * Layer dev paulm (#2423) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * Layer dev paulm (#2424) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * Layer dev paulm (#2425) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Layer dev paulm (#2426) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * fixed map and sequence when passing stl types across the ABI . found a leak in nvidia driver, but skipped it. all winmlapitests pass now * Moved SessionOptions over to the abi * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * Register WinML TraceLogging provider on Onnxruntime.dll (#2455) * Register WinML TraceLogging provider on Onnxruntime.dll * Add ifdef to make sure trace logging provider has telemetry option when LAYERING_DONE * No need for ifdef for TraceLoggingOptionMicrosoftTelemetry * PR feedback * Move etw registration into lotus environment constructor and deresgister in lotus environment destructor * Brianma/cpuwinml (#2466) * allow building winml cpu without dml. * Brianma/breaks (#2469) * fix some more breaks * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * move dml checks out of winml and into the adapter * better error handling * Brianma/fi (#2470) * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * fix bad merge * Checking in a staging checkpoint point so that Ryan can work with me in parrallel * build break. * Brianma/testfails (#2473) * add missing ir version to dictvectorizer-string.onnx * add missing ir version to relu.onnx * add missing ir version to zipmap*onnx * add IR version to manually generated models * remove an unnecessary ifdef dml * Brianma/windowsai fi (#2475) * update dockerfiles/README (#2336) * Make elementwise op run 4 items per thread (#2335) Description: Describe your changes. Make elementwise op run 4 items per thread unroll for loop to leverage ILP remove unnessary N==0 check inside elementwise GPU kernel Motivation and Context Why is this change required? What problem does it solve? It can improve the performance of GPU elementwise ops. ~2% performance gain on popular NLP bert model. If it fixes an open issue, please link to the issue here. * Add CUDA GatherElements kernel (#2310) * Updates * Update test * Update * Updates * nits * PR feedback * Update * Update * PR feedback * PR comments * Update * Fix build * Fix build * Nits * Fix * Layer Normalization Fusion (#2319) basic layer normalization transform * Add FastGelu Cuda Op for Gelu and Add bias fusion (#2293) * Add FastGelu cuda op * Add AddBiasGelu for experiment * Revert "Add AddBiasGelu for experiment" This reverts commit 5c1ee019858c657e6bb75887265cb85675626e5b. * Add bias * Add unit tests * update comment * update script * fix build error * update coding style * update for CR feedback Enable half2 optimization only when cuda arch >= 7.0 * move _Tanh to common.cuh * implement CPU contrib OP Attention (#2333) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanUnusedInitializers. (#2320) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanupUnusedInitializers. This means initializers that have been replaced during graph optimizations are not left in the GraphProto when we save an optimized model. * Handle edge case where a model has an unused initializer with matching graph input by also removing the graph input. * Use non-const iterators in std::find_if calls to make centos build happy. * Nuget pipeline changes (#2305) 1. refactor the pipeline, remove some duplicated code 2. Move Windows_py_GPU_Wheels job to Win-GPU-CUDA10. We'll deprecated the "Win-GPU" pool 3. Delete cpu-nocontribops-esrp-pipeline.yml and cpu-nocontribops-pipeline.yml 4. In Linux nuget jobs, run "make install" before creating the package. So that extra RPAH info will be removed * Cuda Reverse Sequence Op, maping types of same size using same template function. (#2281) * Set ElementType to String type of node metadata, instead of byte[] (#2348) * Set ElementType to String type of node metadata, instead of byte[] * Fix spacing * Introduce PrimitiveType into a Type System along with an integer constant (#2307) Improve perf by avoiding GetType<T>() calls. Introduce MLTypeCallDispatcher to switch on Input Type. Add Tensor IsType<T>() fast method. * Fix/test dim value of 0 handling in a couple of places (#2337) * Update the CUDA Where implementation broadcasting logic to handle a dim with value of 0. Add unit test Also add unit test for unary op with dim value of 0 * Exclude ngraph from Where test with 0 dim. * Openvino EP R3.1 onnxrt server (#2357) * onnxrt server with OVEP * onnxrt server with OVEP * Update Dockerfile.server.openvino * onnxrt server OVEP fix reviews * onnxrt server OVEP fix reviews * Implement cuda nonzero op. (#2056) Implement cuda nonzero op. * Direct use python numpy array's memory if already contiguous. (#2355) * Direct use python numpy array's memory if already contiguous. This could greatly improve performance for session with large input, like big image 1920x1080 fastrcnn, 30~40% speed up could be achieved. * Add test case enforce contiguous/non-contiguos numpy array as inputs. * Add helper to create output to minimize binary size. (#2365) Add ConstEigenTensorMap typedef so we don't unnecessarily const_cast the const input Tensor. * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS (#2369) * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS * update * Add Tracelogging for profiling (#1639) Enabled only if onnxruntime_ENABLE_INSTRUMENT is ON * test bidaf with nuphar for avx target (#2370) increase nuphar test coverage a bit * Fix a bug in TLS refcount that may destabilized CUDA CI (#2374) * update output size calculation for resize (#2366) * change how output size is calculated for resize op * add tests for ver 10 resize * Extend OneHot CPU kernel to support more types (#2311) * Extend OneHot CPU kernel to support input int64_t, depth int32_t, output float * Skip BERT before the test data fix is picked up * Fix bug with Slice. Need to pass in flattened input dimensions so the initial offset into the input is calculated correctly. (#2372) * Add opset 11 version of Split to CUDA ops (#2376) Organize the CUDA ops definitions so all the opset 10 and 11 parts are together (same setup used for CPU ops) * Layer Norm Fusion Fix (#2379) * layer norm fusion fix * Add input shape check in code and unit tests * Fuse Add + Gelu (#2360) Implement the transformer to fuse add + gelu Implement the accurate kernel * Skip layer norm transform (#2350) * skip layer normalization transformer * Another try to stabilize CUDA CI (#2383) The root cause seems to be failure in CUDA dealloc when tear down. cudaFree return code was ignored before, so should the debug check. * fix BUILD.md typo (#2375) build.py: error: argument --config: invalid choice: 'RelWithDebugInfo' (choose from 'Debug', 'MinSizeRel', 'Release', 'RelWithDebInfo') * Fixed compilation with ngraph (#2388) * Fix reuse logic in allocation planner. (#2393) * Fix reuse logic in allocation planner. * PR comments * Add helpful comments * Don't allow reuse across string tensors. * [NupharEP] Multiple optimizations (#2380) Fuse transpose into MatMul Implement Pow and constant scalar simplification Vectorize ReduceMean Improve symbolic shape inference Minor updates for better debugging in fused function name * Avoid using the default logger in the graph lib and optimizers (#2361) 1. Use the session logger if it is available. 2. Don't disable warning 4100 globally. We should fix the warnings instead of disabling it. * Change CUDA implementation of Transpose to support all fixed size tensor types (#2387) * Change CUDA implementation of Transpose to not use a typed kernel so we can support more types with minimum binary size. Add support for 8, 16, 32 and 64 bit types. Add unit tests. Add method so the implementation can be called directly (will be used by CUDA Scan very soon). * Disable TensorRT for MLFloat16 and int8 unit tests. * Address PR comment and add support for calling cublas implementation if type is mlfloat16. * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. (#2398) * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. * [NupharEP] force some low/zero cost ops to be inlined (#2409) * fix cross compile bug (#2415) * Minor optimization: if a node has already been placed, there's no need to find a kernel for it. (#2417) * Add Reshape Fusion (#2395) * Add reshape fusion * Add some comments * update comments * update comment format * update according to feedback * update for recent logger change * fix build error * (1) Support both input and output edges in find path in graphutils (2) Add a test case of only one constant initializer of Concat input. (3) Refactor ReshapeFusion class to allow add more subgraph fusion in the future. * fix error * (1) loose constraint on initializer: non constant is allowed for reshape fusion. (2) Change versions type to vector. (3) Add logging. (4) Return false when multiple output edges matched in FindPath. Add comments. * only allow one direction (input or output) in FindPath * [NupharEP] Update notebook and docker image (#2416) Add BERT squad in Nuphar tutorial Enhance speed comparsion readability * Fix the issue in matmul_add_fusion (#2407) Fix the issue in matmul_add_fusion If Muatmul + Add has shape [K] * [K, N], reset it to [1, K] * [K, N] will make the output shape to [1, N] will also requires a reshape on the output. Fix: just remove the shape reset to not fuse it. Add a negative test case for matmul+add fusion * feat(treeregressor): Update TreeEnsembleRegressor for type support (#2389) Updates the `TreeEnsembleRegressor` to allow for `double`, `float`, `int64`, and `int32` inputs to match the upstream specification. Signed-off-by: Nick Groszewski <nicholas.groszewski@capitalone.com> * onnxrt server documentation update (#2396) * Added support for Pad-2 operator in OpenVINO-EP (#2405) * Add CUDA If operator. (#2377) * Add CUDA If operator. Uses CPU operator for implementation. By adding a CUDA version the inputs/outputs (with the exception of the 'cond' input) stay on GPU, and no other logic is required to avoid a copy to CPU across the control flow node. * Improved documentation for onnxruntime::utils::SwapByteOrderCopy(), added precondition check. * Fix the type constraints on CUDA If operator to exclude strings. (#2431) * add Im2col<uint8_t> (#2438) * Adjust codegen vectorization width from target (#2439) * Adjust codegen vectorization width from target * Add CUDA Scan operator. (#2403) * Add Scan CUDA op. Uses CPU implementation for logic. Added some device specific functors for handling when data needs to be manipulated on a different device. Added ability to override the materialization logic in the OrtValue slicer so DML can plugin their handling. * Fix Windows GPU C API packaging pipeline failure (#2440) Fix Windows GPU C API packaging pipeline failure (#2440) * Correctly handle implicit inputs for fused nodes (#2390) * Correctly handle implicit inputs for fused nodes Previously, nuphar's partitioning function didn't include node's implicit inputs into the inputs list of MetaDef, and hence a crash was triggered in the onnx graph checker. This commit fixed the issue. Furthermore, it also fixed a related issue where we didn't add implicit inputs into graph_inputs_excluding_initializers_ in Graph::SetGraphInputsOutputs. the issue was that graph_inputs_including_initializers_ populated by SetInputs (e.g. called by FunctionImpl::FunctionImpl) may contain implicit inputs which were not of any node's initializers in the graph. Because they were not part of any initializers, these implicit inputs couldn't be visited by going through all nodes' inputs. Consequently, they would *not* be added into graph_inputs_excluding_initializers_. We fixed the issue by first copying the populated graph_inputs_including_initializers_ into graph_inputs_excluding_initalizers_, which then had both initializers and non-initializers as its initial content. Later, we erase initializers from the list. In this way, we can ensure all implicit inputs to remain in graph_inputs_excluding_initializers_. * refined comments and fixed duplicates Address CR by revisiting comments in terms of implicit inputs Also fixed an issue by skipping duplicates while copying inputs from graph_inputs_including_initializers_. * address CR explain why we need to collect nodes' implicit inputs * don't rely on pointer values for iterating std::set Previously, openvino relied on iterating a set of NodeArg pointers to construct inputs and outputs for a fused graph. It could cause non-determinism. The reason was that although iterating std::set by itself is stable, pointer values of NodeArgs may vary. Consequently, we could end up visiting the set's elements in different orders for different runs for the same test, which resulted in constructing inputs (and outputs) with different orders to the fused graph. For example, for the same test, we may have inputs [A, B] in some runs but inputs[B, A] in others. Let's use std::string as the key type to avoid such nondeterminism. This commit also added implicit inputs into meta->inputs while returning the capability from the openvino provider. * Fixed another latent issue in openvino's GetCapability function The issue was that we couldn't simply erase fused_inputs and fused_outputs while iterating the nodes. For example, an output NodeArg may have multiple uses, and it's wrong if we erase it from fused_outputs when we encounter only one of its uses as input. * Remove DeviceAllocatorRegistry class (#2451) Remove DeviceAllocatorRegistry class * CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library. * Parallel Gelu with ParallelFor (#2399) Parallel Gelu to get better performance for Gelu * Clean up build.py (#2446) * Pull the latest image before running docker build * Fuse SkipLayerNorm with Bias (#2453) Fuse SkipLayerNorm with Bias * Allow more than one invocation of CreateEnv in the same process. (#2467) * Allow more than one invocation of CreateEnv in the same process. * Fix centos build * Symbolic shape inference improvements: (#2460) * Symbolic shape inference improvements: - add a mode to guess unknown ops' output rank - add support for GatherND - add support for If - fix a bug in get_int_values when then tensor rank > 1D, by treating it as no sympy data - add symbol to literal merge when ONNX silently merges dims - fix a bug in Concat when input dim is 0 - fix a bug in ConstantOfShape that computed dim is not updated - add support for dynamic shape in ConstantOfShape - fix a bug in Loop output shape that loop iterator dim is not inserted at dim 0 - add support for dynamic padding in Pad - add support for dynamic shape in Reshape - add support for Resize with opset > 10, by treating output dims as dynamic - fix a bug in Slice when starts/ends are dynamic - restrict input model to opset 7 and above - make output model optional to avoid disk write when testing Run model tests for symbolic shape inference Reduce 2GB docker image size of nuphar * add additional test data set for nuget pipeline (#2448) * add SAS token to download internal test data for nuget pipeline * update azure endpoint * fix keyvault download step * fix variable declaration for secret group * fix indentation * fix yaml syntax for variables * fix setting secrets for script * fix env synctax * Fix macos pipeline * attempt to add secrets to windows download data * fix mac and win data download * fix windows data download * update test data set url and location * Revert "Brianma/windowsai fi (#2475)" This reverts commit 5780b864a15513fda4eadbfc2b5345fefe70b5ec. * Add scenario tests (#2457) * Add scenario tests * Remove TODO from model license * Add winml_api test dependency * fix model load test. fi from master changed the constructor (#2483) * make api tests all pass (#2486) * fix bad merge * fix bad model merge * Layer dev paulm (#2492) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * Rename ambiguous header (#2489) * fix one more missing IR version model (#2500) * add missing IR version to 4 more models used by scenario tests (#2501) * Add CLI parameters to test runner, build WinML in ARM and x86 CI (#2479) * Support test parameters through CLI arguments * Add WinML do Windows x86/ARM CI builds * Code style fixes * Update googletest Remove GPUTEST macros everywhere now that GTEST_SKIP is supported * Refactor main.cpp * Build scenario tests without DML * Link scenario tests to DML when it's enabled (#2502) * Layer dev release pipeline (#2488) Adds winml binaries to existing cpu nuget package, and creates new gpu dml nuget package with winml binaries and DML EP. * Layer dev paulm (#2506) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * Remove usage of IOBinding in WinML and use C_API Run method (#2504) * remove usage of iobinding * Change data structure to use vector of Ort::Values * Polish bind input / output * Use C APIrun method * Update providers on evaluate getresults * Remove run and IObinding interface from WinMLAdapter * Remove use of IObinding * bind unbound outputs code moved to learningmodelbinding * clean up unneeded istensor adapter function * Fix comment * Check if session is closed before binding and clearing * PR feedback * Layer dev paulm (#2507) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * Make tests dependend on winml_dll (#2509) * add dml binaries to DirectML package and be more explicit about condition variables (#2520) * re-enable warnings for winml builds and fix the warnings that were hiding (#2526) * turn devmode back on for winml builds * fix some warnings. include protobuf in a way that disables some warnings * undo protobufhelpers changes and just ignore 4100 errors in pb code * attempt to isolate protobufhelpers errors * add template specialization for getting tensor proto data * Layer dev paulm (#2533) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * Spawn child process to run DeviceLostRecovery scenario test (#2530) * Spawn child process to run DeviceLostRecovery scenario test * Layer dev paulm (#2536) ori said yes * add missing namespace to winml_trace_logging_provider in lotusenvironment.h (#2542) * Handle exception thrown from all apis in WinMLAdapter (#2539) * various changes to unblock windowsai ADO build * Fix custom ops scenario tests (#2562) * Do not shutdown protobuf after ort environment gets destroyed. Lazy load lotus environment first time it is needed * comment typo * pr comment about calling phoenix singleton * Make lotus_environment static in winmladapter * Layer dev paulm (#2567) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * making windowsAI azure dev ops work. * code review comments. * revert changes * Cmake and preprocessor fixes that where uncovered by building on agents without DML available via SDK * Layer dev dml delayload (#2580) * Brianma/cpu (#2583) * don't include dml stuff in cpu builds * tests that link the image lib also need the telemetry lib now * Throw Winml_err_invalid_binding if binding gpu resource on cpu device (#2589) * Throw Winml_err_invalid_binding if binding gpu resource on cpu device * PR comments. No need to query executionprovider for is gpu device * User/xianz/ortthrow (#2596) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * User/xianz/ortthrow (#2599) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * add status error message * Remove uwp onsuspending winrt call because logruntimeperf is getting removed (#2630) * User/xianz/dedup telemetry (#2631) * investigate duplication of telemetry in winml and ort * remove winml telemetry events * telemetry executionProviderEvent * remove unneccessary file and refactor code little bit * Revert back TelemetryEvent, which send up ETW event. * merge changes from layer_dev to windowsai (#2638) * Remove underscore from googletest names (#2616) * Fix leaking memory allocator Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24278761 and https://microsoft.visualstudio.com/OS/_workitems/edit/24330198 * Explicitly initialize Ort::Value with nullptr * Cache WinML adapter * bad merge * define private version of dxcore enum that is added in 19H1 SDK. (#2654) * add comment for explaning private definition of dxcore d3d feature level ennum value. (#2672) * do not package directml.pdb for redist packages. (#2676) * Fix leaking operator registry (#2645) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24354916 * User/orilevari/windowsai master merge (#2674) merge resolutions included pulling in telemetry logic that was merged to master and not windowsai and dereferencing InferenceSession::sessionstate now that it is a unique pointer * Delete Ort Allocator in LearningModelBinding (#2653) * Delete OrtAllocator in LearningModelBinding * PR comments to make Ort::Allocator a smart pointer * Small comment change * PR feedback to clean up code * PR feedback on move semantics * Clean up std::move * Fix memory leaks (#2679) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24356109, https://microsoft.visualstudio.com/OS/_workitems/edit/24388361 and https://microsoft.visualstudio.com/OS/_workitems/edit/24388596 * various changes to properly organize and skip GPU tests. For now for No DML builds we will not run GPU tests at all. In the future we should adapt the tests to expect the appropiate errors. (#2695) * Windowsai without fi (#2701) * Disable Attention fusion tests when DISABLE_CONTRIB_OPS is defined (#2529) * Setup java ci (#2528) * Add provision in ORT for session options to be parsed when available via model file (#2449) * Initial commit * Fix gitmodules * Nits * Nits * Updates * Update * More changes * Updates * Update * Some updates * More changes * Update * Update * Merge * Update * Updates * More changes * Update * Fix nits * Updates * Fix warning * Fix build * Add comment * PR feedback * PR feedback * Updates * Updates * Update * More changes * Fix build break * Comment test for now * Updates * Updates * PR feedback * Updates * Nits * Add tests * Fix build * Fix build * Fix build * Fix build break * Fix build * Nits * PR feedback * More change * Expose GetSessionOptions in pybind logic and add unit test for python * Fix build * PR feedback * PR feedback * Revert "Disable thread pool creation when enabled OpenMP (#2485)" (#2535) This reverts commit 7c7d5a149c9ed52eec67304bae5c4b132166a8a1. * Add dynamic shape support in TensorRT execution provider (#2450) * remove onnx-tensorrt submodule * add new onnx-tensorrt submodule (experiment) for trt6 * update engine build for trt6 * update compile and compute for tensorrt6.0 * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * switch to onnx-tensorrt master for TensorRT6' * Update tensorrt_execution_provider.cc * Handle dynamic batch size and add memcpy in TensorRT EP * update test cases * Update tensorrt_execution_provider.cc * update onnx-tensorrt submodule * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.ubuntu_tensorrt * Update run_dockerbuild.sh * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update concat_op_test.cc * Update tensorrt_execution_provider.cc * Upgrade TensorRT to version 6.0.1.5 * Update onnxruntime_providers.cmake * Update CMakeLists.txt * Update reduction_ops_test.cc * Update install_ubuntu.sh * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.tensorrt * Update BUILD.md * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update onnxruntime_providers.cmake * Update install_ubuntu.sh * Update install_ubuntu.sh * Update gemm_test.cc * Update gather_op_test.cc * Update CMakeLists.txt * Removed submodule * update onnx-tensorrt submodule * update header file * Removed submodule * add submodule onnx-tensorrt kevin's branch shape-test' * add debugging code * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * merge master * Removed submodule * update onnx-tensorrt submodule * add more changes for dynamic shapes * Update tensorrt_execution_provider.cc * update for dynamic shape * update dynamic shape processing * fix logger issue * remove submodule onnx-tensorrt * add submodule onnx-tensorrt * add env variable min_subgraph_size * remove redundency * update document * use onnxruntime::make_unique * fix multi-run issue * remove some tests to save CI build time * Add dynamic shape test * Update TensorRT-ExecutionProvider.md * Add example of running Faster R-CNN model on TensorRT EP * Add more details on env variables * update environment variables * Update tensorrt_basic_test.cc * Update model tests * Update tensor_op_test.cc * remove --use_full_protobuf * Update build.py * User/xianz/telemetry (#2458) * enabme telemetry * enable telemetry * set enable telemetry as default * for debugging * remove log and set disable telemetry as default back * delete private file while testing * resolve comment: mainly add license header, rename macro and update docs * rewording in privacy.md * Fix integer overflow in cuda NonMaxSuppression implementation (#2540) * add test case that should pass but fail * fix nms * extract int_max_output_boxes_per_class * Introduce container type runtime checks and other improvements (#2522) Rework TensorSeq in a manner consistent with Tensor and SparseTensor in terms of type system setup. Reduce templating. Introduce helpers to ensure the same data type. Make OrtValue __dtor not virtual. Introduce ContainerChecker * Fix C API tests for centos and mac (#2544) * change c++14 to c++11 * add ld lib path for centos * enable csharp tests on macos * fix C API test on MacOS + fix manylinux dotnet install * fix manylinux dotnet install * fix lib link * Add back executable bit to build.py * Fix a bug handling negative begin pad values in Pad op (#2550) * Fix bug in Pad op * Update * DNNL CMAKE update (#2548) * Fix android build (#2558) * Update win-x86-ci.yml (#2557) Fix build pipeline break * Re-enable Windows C# tests (#2564) * disable onnx_test_runner -x invocations for dnnl (#2568) * Allow sequence length to be symbolic (#2559) * setup java ci mac (#2570) * make layernorm fusion to support opset 11 (#2545) * Fix a warning found in the latest VS release * Add more check on SkipLayerNorm and BiasGelu fusion (#2574) * Fix file not found error during docker build. (#2569) * Add ConvTranspose1D (#2578) * Ryanunderhill/packagename test (#2582) * [Nuphar EP] fixes for some object detection models (#2581) Update notebook tutorial with multi-threaded int8 GEMM from #2517 * EmbedLayerNormalization Fusion Improvement (#2553) Embedding layer norm fusion improvements - add more checks * Update version (#2584) * Temporarily exclude vgg19 test from Python backend test 1. temporarily exclude vgg19 test which comsumes too much memory, run out of memory on Upsquared device. Single test pass for vgg19, need furture investigation (#2588) 2. Update docker file to decrease the docker image size * Update docs for Android NNAPI EP (#2586) * Fix lto bug for protobuf and ubuntu * add path to build dir before test run (#2590) * Add missig env variables for mac pipeline test (#2595) * Fixed an issue in updating realized dims (#2597) when we update realized dims for scan's output, the sliced axis also needs to be inclusive, i.e. we should check with "dim >= insert_inclusive_axis", because the offset in the symbols are based on Scan sugraph. Otherwise, we would end up with shape mismatch later. * Java API for onnxruntime (#2215) * Add support for opset 11 in reshape fusion (#2592) Support opset verion 11 in reshape fusion * Rename automl python tools folder to featurizer_ops. (#2593) * Support opset 11 subgraph of Squad model in Embed Layer Normalization (#2605) Support opset 11 Squad model that is exported from PyTorch nightly. The embed layer uses Range op which is missed in the transformer. * symbolic shape inference: fix warnings in GPT-2 model (#2608) And revise nuphar perf test on BERT squad * Dump subgraph ID and fused graph ID (#2607) * Dump subgraph ID and fused graph ID Dump subgraph ID and fused graph ID for better debugging * Remove local static fused_count added a field global_fused_count_ to NupharExecutionProvider class * EmbedLayerNormalization Fusion For Dynamic Squad Model Opset 10 (#2613) Support subgraph of SQuAD model exported from pytorch with dynamic input axes * Allow providers to be set for InferenceSession at construction (#2606) * Remove unnecessary parameter in some places in GatherElements implementation (#2612) * Remove unnecessary parameter in some places * Update * Update * Make sure fenced tensor could not reuse other tensor. (#2561) Fix random error caused by this. * Improve Embed Layer Norm Fusion for SQuAD with static input shape (#2621) * fix float16 comparison in initializer (#2629) * epsilon attribute for layernormalization fusion (#2639) * removed unnecessary batch file and fix path (#2640) * Add shape inference to ConvTransposeWithDynamicPads schema (#2632) * Improve cuda expand() opeator's performance. (#2624) * Cuda pad optimize when no padding is needed. (#2625) * Shortcut cuda Pad() when no padding is needed. * Optimize cuda scatter() on 2D compatible. (#2628) * Optimize cuda scatter() on 2D compatible. * Add some comments. * fix build error for ARM (#2648) * Improve performance of resize() in Nearest mode (#2626) Special treatment for 2D, check same size as input image. And in 2d kernel, template use_expolation. * Fix memory exception in Layer Norm Fusion (#2644) * Windows CI changes(#2650) * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Revert "Windowsai without fi (#2701)" This reverts commit 285d4c85ff5c4e265f963208170304ef3461e684. * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Deref unique pointer for session_state * send shutdown event when dll is unloaded and EvaluationStop, SessionC… (#2704) * send shutdown event when dll is unloaded and EvaluationStop, SessionCreationStart Events. * Add EvalutationStart Event * add comment * use correct type for for loop (#2755) * ARM CI (#2759) * Set ARM agent pool * Set CMake generator to VS 2019 in ARM * Use system-wide CMake instead of custom version Our custom version is too old for VS 2019 * Use DML and build shared lib in ARM CI * Restore nuget packages in ARM CI * Disable DML * Refactor ARM debug/release builds * Use system packaged Python version * Remove hardcoded Python path * Downgrade Python to 3.7 for build * Remove explicit CMake path * Fix invalid JSON in cgmanifest.json (#2760) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * Remove usage of WHOLEARCHIVE in WinML CMake and add WinMLAdapterFactory (#2726) * Remove usage of WHOLEARCHIVE in WinMLAdapter CMake and add WinMLAdapterFactory * PR feedback, no need for dll(export) since using def file * PR comments * Small comment in gen_def.py * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * Move winml_provider_factory.h to proper location (#2801) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Filter CPU case for IsFloat16Supported (#2802) * Merge fixes * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * Fix warnings that cause build to fail * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * Make DML include path global (#2882) * Make DML include path global * Add generated cppwinrt headers to winml_lib_common * Integrate changes to WindowsAI to make ADO Build (#2886) * Revert "CMake cross-generator fixes (#2790)" This reverts commit dbe7d97fa1ab155f1309bced87199527e8f35bd2. * add additional suppress warning in onnx_proto * ignore /wd4996 warning * DML execution provider fixes * Revert "Revert "CMake cross-generator fixes (#2790)"" This reverts commit 1ae7b4bcbc02edc881ad28685da98e095dfceb17. * Update func signature of custom op function overloads * common devicehelpers fixes * Add pch.h for winml_lib_common * re-add winml_lib_common_dir/inc to include path for winml_adapter * User/orilevari/dml redist shared folder (#2890) * move dml nuget package directory up one level to make it shared between build flavors * Merge conflict fix * Revert "Merge conflict fix" This reverts commit 142fa72cf9ce4344ad717b50b7ea2b8582aadc7c. * Revert "Merge remote-tracking branch 'origin/master' into windowsai" This reverts commit 6e2126d46e5e5f564d65da37dd4f70c93dd81165, reversing changes made to b3f5583dc9249834b947c8ea905f6a98060d5bd6. * Make winml_test_common free of test macros (#2902) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Don't generate files outside the build root (#2914) * Don't generate files outside the build root * Add onnxruntime_EXTERNAL_DEPENDENCIES to WinML * Add DML depedency on RESTORE_PACKAGES * User/orilevari/fix yaml merge bugs (#2918) * Add winml test source parameter into cmake function (#2919) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Add arguments to unittest cmake functions * remove comment * Revert "Revert "Merge remote-tracking branch 'origin/master' into windowsai"" This reverts commit ade5abe72a4234fdbc3623093c61c02c6b0bdc26. * Fix breaks from merge with ORT master * Brianma/linux (#2917) * don't include windows.h in cross-plat header * add default case for switch statement * signed/unsigned mismatch fix Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> * User/sheilk/winml adapter c api (#2891) * Create winml adapter c api * fix build * make it build * move adapter into onnxruntime core/session * entry point not exported * minor changes * make model metadata work * make tests pass * implement all the model reflection apis on the adapter c abi * update the new ort interface to create a lotus ennvironment with a logging sink * start adding ort env * move all winml code into adapter folder/lib to isolate it * ensure a single logging manager at a time * start refactoring session * refactor session creation interface * add cpu and dml session option methods to adapter * finish session init * stub out interfaces in ort lib to perform similar mechanics of iinference session * enable profiling, and enable schema override * update session register graph transformers * turn back on custom registry for custom ops * Add sync api * add last c api stubs * should build... but all feature values are broken since this is in flight to moving all implementation details into ivalue * remove ep adapter header * Implement DML execution provider functions from adapter (#2846) * Implement DML execution provider functions from adapter * Use functions in OnnxruntimeEngine.cpp * make map/sequence type_infos freeable, and start implementing ivalue * make it build again * implement value methods * implement remaining methods * remove com adapter abi * check dml session * cache the allocator on ivalue * check if resource is cpu/gpu when access its mutable data * update tensor * mismatched parentheses * fix tensor base and binding obj * it evaluates tensors! sometimes... * minor fixes * enable gpu evals * wrapper all existing winml adapter apis with API_IMPL to try catch (#2854) * update winml... tensor strings are broken, need to template tensorbase to do different things for strings * make tensor strings work with 2 copies in/2 copies out * Fix tensor string and allocator bug * make maps work again... needs some fixes still * Make it build! * enable map inputs * map outputs * unbound outputs for sequences and maps * User/xianz/merge windowsai (#2883) * Packaging pipeline changes for VS 2019 (#2711) * Tiny fix to codegen * Simplify cache implementation and avoid static variables that may carry over between models * Extend DML kernels (#2641) * Additional DML operators * Check unsupported attributes and inputs * Address PR comments * Add kernel capability function used for partitioning, and re-enable stride-based int64 support based on value range * Fix test failures * Build fix * PR comments * Update Nuphar tutorial notebook (#2721) 1. Reflect int8 GEMV improvements for multi-threading from #2696 2. Add notes on multi-threading control using OpenMP 3. Add samples of running multi-isa AOT, and show int8 GEMM differences between AVX and AVX2 4. Add rnn_benchmark example to resolve #1993 * Add schema for new Qops (#2611) * Add schema for new Qops * adding shape inference + qlinearaveragepool * plus review comments * plus review comments * updates per review comments * plus review comments * [server] Add supposed for model_name and model_version as cli parameter (#2708) * remove 64bit warning message from python validation. (#2727) * MLAS: ARM64 build fix (#2734) fix bad usage of vreinterpret to cast vector element types * Fix broken python docs links (#2740) * Fix build on Mac OS (#2731) mac os ld doesn't support --while-archive, correct option is -all_load * fix ngraph wheel (#2737) * fix ngraph wheel 1.1.0 onnxruntime_ngraph wheel doesn't work * remove libdnnl.so in nGraph Libs * make it easy to compare * Split onnxruntime server to a separated folder (#2744) * Fix build for Python 3.8 (#2747) * Fix build for Python 3.8 * Update protobuf to 3.11.2 (#1928) Update protobuf to 3.11.2 (#1928) * Change default optimization level to All (from Basic) (#2745) * change default optimization level to All (from Basic) * fix test * fix c# test * Update numpy to 1.18 (#2758) * Update numpy to 1.18 * Pipeline changes for python 3.8 (#2753) 1. Pipeline changes for python 3.8 2. Fix a regression in setup.py which was just introduced in the previous commit. Please notice, we still haven't made python 3.8 + Windows + CUDA work. * Add basic stacktrace output for posix debug builds. (#2749) * [NupharEP] fix a race condition when multiple sessions running different models concurrently (#2772) * Revert "Change default optimization level to All (from Basic) (#2745)" This reverts commit 56bb503c2f26474b6613bcb2a198691a11dcef38. * Fix typo in error message (#2736) * Rename MKL-DNN to DNNL to fix broken link (#2730) * Fix nightly build version number issue * Pass BUILD_BUILDNUMBER to linux docker * Disable featurizers in python packages * Import more featurizers (#2781) Make kernels non-template. Add input constraint for learnt data. Add min_max_scalar_transformer, robust_scalar_transformer, inputation_marker_transfomer, label_encoder_transformer, missing_dummies_transformer along with tests. Advance Featurizers library commit. * Implement a more stable softmax (#2715) * Implement a more stable SoftMax e^x is represented as infinity if x is large enough, like 100.f. Infinity divided by Infinity is a NAN. Thus, softmax gets a NAN if one or more item are large enough. A math transform as below is leveraged to get a stable softmax: e^xi/(e^x1 + ...e^xn) = e^(xi - max) / (e^(x1 - max) + ... + e^(xn - max)) And for convenience, force max to 0.f if all xi are negative * Contributing: Fix a typo (#2784) * ACL EP GEMM improvements (#2780) When it is posible we use a fully connected layer instead of the gemm implementation. This will let the library use the best implementation based on the input data. * ACL EP convolution improvements (#2774) Added the optimized implementation for depthwise convolution for both ACL v19.02 and ACL 19.05. Also the pointwise convolution seems to be more optimal in the CPU implementation so we opted for that instead. * Add script for release Nuget validation (#2719) * Initial commit * Nits * Disable a test temporarily * Change working directory * Test * Add download python step * Test update * More changes * Fix space issue * Fix * Verify nuget signing * Fix * Spaces * PR feedback * Nit * Fix * Fix * Remove temporary changes * add uint8 support to where op (#2792) * Improve bert optimization script: (#2712) (1) Move input int64=>int32 conversion to embed layer fusion. (2) Output epsilon attribute for LayerNormalization fusion. * add session creation time cost. (#2798) * ML.NET team needs featurizers within a package (#2789) Add auto ml featurizers to Windows, MacOS as well as to GPU packaging-pipelines. * Initialize max of softmax with lowest of float (#2786) * MLAS: update SGEMM threading parameters (#2808) * add interface to copy batch tensors. (#2807) * add interface to copy batch tensors. * onnxruntime * speed up Windows TRT CI (#2811) * don't run cuda tests if building with tensorrt * remove unnecessary build options for win trt ci * refactor win gpu tensorrt ci yml * --numpy_version=1.17 * update * update * azcopy and cuda path * Update test data (#2356) * Add timeseries imputer transformer featurizer kernel (#2813) Make kernels non-template. Add input constraint for learnt data. Fixup tests. Add two more featurizers along with tests. Tests fail. min_max_scalar_transformer robust_scalar_transformer Fix tests serialized stream by prepending version bytes. Add inputation_marker_transfomer and the test. Fix up float/double type designations. Added label_encoder_transformer along with a test. string_throw case is broken at the momement. Fix labelencodertransfomer_test.cc string_throw case Rename maxabsscalertransformer_test.cc Add MissingDummiesTransformer along with the test. Update manifest. Add TimeSeriesImputerTransformer definition, implementation and tests * Fix memory leak in TRT (#2815) * fix memory leak issue * revert EP_FAIL on enueueV2 * Add manifest missing comma * Run static code analyzer on most of our code (#2817) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * update quantization doc (#2783) * update documentation for quantization script * plus some spell corrections * Filter CPU case for IsFloat16Supported (#2802) * update default optimization level + fix gemm_activation fusion (#2791) * update defualt optimization level + fix gemm_activation fusion * fix typo * add unit test and incorporate review comments * fix test comment * Fix dnnl wheel package name (#2823) * Append '-dnnl' to whl package name when --use_dnnl * Update build.py * Update Ubuntu & TensorRT version in README (#2820) Dockerfile.tensorrt is using nvcr.io/nvidia/tensorrt:19.09-py3 as base Image, update Ubuntu and TensorRT version according to https://docs.nvidia.com/deeplearning/sdk/tensorrt-container-release-notes/rel_19-09.html#rel_19-09 * Merge fixes * Add OneHotEncoder and HashOneHotEncoder kernels. (#2830) Add defs and imlementation for OneHotEncoders, adjuist date_time_transformer kernel and test. Add OneHotEncoder kernel test. Add HashOneHotVectorizerTransformer unit test. This does not link due to multiple definitions of functions that are included into header from a CPP file. * Upgrade gtest to the latest version (#2827) WinML would like to update the googletest submodule. They want some newer features (namely GTEST_SKIP to skip tests programmatically and be able to skip entire fixtures easily) and would need to update the submodule version. However, because the new version of code hit a bug in gcc, even though the bug is already fixed in the latest gcc but we're using gcc 4.8.x and it won't get patched for the bug, so we have to do a compromise, change our code a little bit to make it work. The gcc bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=51213 * Add support for int64_t for topk CPU. Fixes github issue #2806. (#2833) * Ignore allocator type in ExecutionProviders allocator map. Make default initialization of OrtMemoryInfo more clearly invalid. (#2768) * Remove allocator type from the key comparison in ExecutionProviders. Remove usage of DummyArena as it's no longer necessary. * Fix x86 tests where arena allocator is disabled. Make initialization of OrtMemoryInfo clearer by adding Invalid enum value. * Make OrtValueNameIdxMap::MaxIdx more intuitive. * Convert ExternalProject Featurizers into git submodule (#2834) Add git submodule for Featurizer library. Update cmake to build for git submodule. * add domain check for nodes + update documentation (#2831) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Fix Linux CUDA nuget packaging pipeline break * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * update optimization doc for BERT related fusions (#2819) * Add bert related transformers to doc * Add execution provider and comment for bert optimizations * Add comment about accuracy impact of approximation * Fix warnings that cause build to fail * MLAS: enable threading for quantized GEMMs (#2844) * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * remove old winmladapter cpp Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * move sequence implementation into ort lib... still commented out... need to turn back on... * begin sequence implementation * make maps and sequences work * fix broken tests * remove dead code * misc cleanup * CR feedback * User/xianz/winml adapter c api (#2869) * wrapper all existing winml adapter apis with API_IMPL to try catch * Return HR or Throw for WinML adapter APIs if failed * undo macro wrapper for two places * Wrap error macros around ort apis, too. * address CR feedback #2 * add more api throw/return macros * Revert changes no longer needed * revert changes to cxx api * format winml lib.ort and winml adapter * remove static pheonix singleton Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * missing use_dml check in winml_adapter_session (#2930) * --use_dnnl flag was mangled in merge (#2931) * use dml macro not wrapping custom registry code (#2934) * Disable LNK4199 winml_dll to enable cuda builds (#2936) * Disable LNK4199 in winml_dll * linkler->linker * LearningModelSessionAPITestGpu.CreateSessionWithCastToFloat16InModel should return DXGI_ERROR_UNSUPPORTED when FP16 not supported (#2937) * Disable LNK4199 in winml_dll * linkler->linker * Need to return DXGI_ERROR_UNSUPPORTED when Model does not support fp16 * Publish build symbols (#2939) * Publish build symbols * Don't upload PDBs for .exe files * Make x86 build (#2943) * fix last remaining size_t/int64_t warnings->errors (#2948) * TensorString, Sequences and Maps use the first allocator, but should use the cpu default allocator. (#2952) * fix tensor string allcoator * clean up default allocator usage for strings in winml lib/api.ort Co-authored-by: Ryan Lai <ryalai96@gmail.com> * Handle tensor shape of zero (#2954) Co-authored-by: Ryan Lai <ryalai96@gmail.com> * CR feedback (#2970) * CR feedback * fix weird formatting on privacy readme * Add 'All rights reserved.' everywhere * readd all rights reserved to winml_provider_factory.h * remove extra space in comment * remove extra whitespace * fixes post master merge * remove winml from nuget gpu pipeline * set IR VERSION on generated_model in rnn_benchmark (#2972) * Fix slice conformance failures (#2908) Co-authored-by: Adrian Tsai <adtsai@microsoft.com> Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Paul McDaniel <paul_mcdaniel@hotmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Dwayne Robinson <fdwr@hotmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com>
2020-02-05 01:12:19 +00:00
//! The width of the image.
UInt32 Width{ get; };
//! The height of the image.
UInt32 Height{ get; };
}
//! \interface ITensor
//! \brief TODO:Docs
[contract(MachineLearningContract, 1)]
INBOX_ONLY([uuid(05489593-a305-4a25-ad09-440119b4b7f6)])
Merge windowsai (winml layering) into master (#2956) * Initial Commit * Merged PR 3985217: add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc (#2346) add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc and violating our OS layering requirements. We linked against onecoreuap_apiset.lib in VB so we will continue doing this, but I am still unsure why not to link against onecore instead since that is where we ship. However, since Sheil is the owner of this code we will wait to discuss with him before changing anything. * Initial changes for layering * more snipping to get core into ort * update build instructions to include --build_shared_lib (#2358) * update build instructions to include --build_shared_lib * fix line breaks * Task 23998197: add winml_lib_core into onnnxruntime.dll (#2368) * Task 23998197: add winml_lib_core into onnnxruntime.dll * PR feedback build break on perf_test * return proper error when the model path isn't found (#2391) * LearningModelSession is cleaned up to use the adapter, and parts of b… (#2382) this is a big PR. we are going to move it up to layer_dev , which is still a L3 so we are still safe to do work there agile. we are going to move this into the L3 so that ryan can start doing intergration testing. we will pause for a full code review and integration test result prior to going into the L2. >>>> raw comments from previous commits >>> * LearningModelSession is cleaned up to use the adapter, and parts of binding are. * moved everything in the winmladapter made it all nano-com using, WRL to construct objects in the ORT side. base interfaces for everythign for winml to call cleaned up a bunch of winml to use the base interfaces. * more pieces * GetData across the abi. * renamed some namepsace cleaned up OrtValue cleaned up Tensor cleaned up custom ops. everything *but* learnignmodel should be clean * make sure it's building. winml.dll is still a monolith. * model moved over. everything builds clean. step ! * weak ref comment * Layer dev paulm (#2408) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * Layer dev paulm (#2414) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * Layer dev paulm (#2423) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * Layer dev paulm (#2424) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * Layer dev paulm (#2425) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Layer dev paulm (#2426) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * fixed map and sequence when passing stl types across the ABI . found a leak in nvidia driver, but skipped it. all winmlapitests pass now * Moved SessionOptions over to the abi * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * Register WinML TraceLogging provider on Onnxruntime.dll (#2455) * Register WinML TraceLogging provider on Onnxruntime.dll * Add ifdef to make sure trace logging provider has telemetry option when LAYERING_DONE * No need for ifdef for TraceLoggingOptionMicrosoftTelemetry * PR feedback * Move etw registration into lotus environment constructor and deresgister in lotus environment destructor * Brianma/cpuwinml (#2466) * allow building winml cpu without dml. * Brianma/breaks (#2469) * fix some more breaks * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * move dml checks out of winml and into the adapter * better error handling * Brianma/fi (#2470) * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * fix bad merge * Checking in a staging checkpoint point so that Ryan can work with me in parrallel * build break. * Brianma/testfails (#2473) * add missing ir version to dictvectorizer-string.onnx * add missing ir version to relu.onnx * add missing ir version to zipmap*onnx * add IR version to manually generated models * remove an unnecessary ifdef dml * Brianma/windowsai fi (#2475) * update dockerfiles/README (#2336) * Make elementwise op run 4 items per thread (#2335) Description: Describe your changes. Make elementwise op run 4 items per thread unroll for loop to leverage ILP remove unnessary N==0 check inside elementwise GPU kernel Motivation and Context Why is this change required? What problem does it solve? It can improve the performance of GPU elementwise ops. ~2% performance gain on popular NLP bert model. If it fixes an open issue, please link to the issue here. * Add CUDA GatherElements kernel (#2310) * Updates * Update test * Update * Updates * nits * PR feedback * Update * Update * PR feedback * PR comments * Update * Fix build * Fix build * Nits * Fix * Layer Normalization Fusion (#2319) basic layer normalization transform * Add FastGelu Cuda Op for Gelu and Add bias fusion (#2293) * Add FastGelu cuda op * Add AddBiasGelu for experiment * Revert "Add AddBiasGelu for experiment" This reverts commit 5c1ee019858c657e6bb75887265cb85675626e5b. * Add bias * Add unit tests * update comment * update script * fix build error * update coding style * update for CR feedback Enable half2 optimization only when cuda arch >= 7.0 * move _Tanh to common.cuh * implement CPU contrib OP Attention (#2333) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanUnusedInitializers. (#2320) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanupUnusedInitializers. This means initializers that have been replaced during graph optimizations are not left in the GraphProto when we save an optimized model. * Handle edge case where a model has an unused initializer with matching graph input by also removing the graph input. * Use non-const iterators in std::find_if calls to make centos build happy. * Nuget pipeline changes (#2305) 1. refactor the pipeline, remove some duplicated code 2. Move Windows_py_GPU_Wheels job to Win-GPU-CUDA10. We'll deprecated the "Win-GPU" pool 3. Delete cpu-nocontribops-esrp-pipeline.yml and cpu-nocontribops-pipeline.yml 4. In Linux nuget jobs, run "make install" before creating the package. So that extra RPAH info will be removed * Cuda Reverse Sequence Op, maping types of same size using same template function. (#2281) * Set ElementType to String type of node metadata, instead of byte[] (#2348) * Set ElementType to String type of node metadata, instead of byte[] * Fix spacing * Introduce PrimitiveType into a Type System along with an integer constant (#2307) Improve perf by avoiding GetType<T>() calls. Introduce MLTypeCallDispatcher to switch on Input Type. Add Tensor IsType<T>() fast method. * Fix/test dim value of 0 handling in a couple of places (#2337) * Update the CUDA Where implementation broadcasting logic to handle a dim with value of 0. Add unit test Also add unit test for unary op with dim value of 0 * Exclude ngraph from Where test with 0 dim. * Openvino EP R3.1 onnxrt server (#2357) * onnxrt server with OVEP * onnxrt server with OVEP * Update Dockerfile.server.openvino * onnxrt server OVEP fix reviews * onnxrt server OVEP fix reviews * Implement cuda nonzero op. (#2056) Implement cuda nonzero op. * Direct use python numpy array's memory if already contiguous. (#2355) * Direct use python numpy array's memory if already contiguous. This could greatly improve performance for session with large input, like big image 1920x1080 fastrcnn, 30~40% speed up could be achieved. * Add test case enforce contiguous/non-contiguos numpy array as inputs. * Add helper to create output to minimize binary size. (#2365) Add ConstEigenTensorMap typedef so we don't unnecessarily const_cast the const input Tensor. * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS (#2369) * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS * update * Add Tracelogging for profiling (#1639) Enabled only if onnxruntime_ENABLE_INSTRUMENT is ON * test bidaf with nuphar for avx target (#2370) increase nuphar test coverage a bit * Fix a bug in TLS refcount that may destabilized CUDA CI (#2374) * update output size calculation for resize (#2366) * change how output size is calculated for resize op * add tests for ver 10 resize * Extend OneHot CPU kernel to support more types (#2311) * Extend OneHot CPU kernel to support input int64_t, depth int32_t, output float * Skip BERT before the test data fix is picked up * Fix bug with Slice. Need to pass in flattened input dimensions so the initial offset into the input is calculated correctly. (#2372) * Add opset 11 version of Split to CUDA ops (#2376) Organize the CUDA ops definitions so all the opset 10 and 11 parts are together (same setup used for CPU ops) * Layer Norm Fusion Fix (#2379) * layer norm fusion fix * Add input shape check in code and unit tests * Fuse Add + Gelu (#2360) Implement the transformer to fuse add + gelu Implement the accurate kernel * Skip layer norm transform (#2350) * skip layer normalization transformer * Another try to stabilize CUDA CI (#2383) The root cause seems to be failure in CUDA dealloc when tear down. cudaFree return code was ignored before, so should the debug check. * fix BUILD.md typo (#2375) build.py: error: argument --config: invalid choice: 'RelWithDebugInfo' (choose from 'Debug', 'MinSizeRel', 'Release', 'RelWithDebInfo') * Fixed compilation with ngraph (#2388) * Fix reuse logic in allocation planner. (#2393) * Fix reuse logic in allocation planner. * PR comments * Add helpful comments * Don't allow reuse across string tensors. * [NupharEP] Multiple optimizations (#2380) Fuse transpose into MatMul Implement Pow and constant scalar simplification Vectorize ReduceMean Improve symbolic shape inference Minor updates for better debugging in fused function name * Avoid using the default logger in the graph lib and optimizers (#2361) 1. Use the session logger if it is available. 2. Don't disable warning 4100 globally. We should fix the warnings instead of disabling it. * Change CUDA implementation of Transpose to support all fixed size tensor types (#2387) * Change CUDA implementation of Transpose to not use a typed kernel so we can support more types with minimum binary size. Add support for 8, 16, 32 and 64 bit types. Add unit tests. Add method so the implementation can be called directly (will be used by CUDA Scan very soon). * Disable TensorRT for MLFloat16 and int8 unit tests. * Address PR comment and add support for calling cublas implementation if type is mlfloat16. * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. (#2398) * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. * [NupharEP] force some low/zero cost ops to be inlined (#2409) * fix cross compile bug (#2415) * Minor optimization: if a node has already been placed, there's no need to find a kernel for it. (#2417) * Add Reshape Fusion (#2395) * Add reshape fusion * Add some comments * update comments * update comment format * update according to feedback * update for recent logger change * fix build error * (1) Support both input and output edges in find path in graphutils (2) Add a test case of only one constant initializer of Concat input. (3) Refactor ReshapeFusion class to allow add more subgraph fusion in the future. * fix error * (1) loose constraint on initializer: non constant is allowed for reshape fusion. (2) Change versions type to vector. (3) Add logging. (4) Return false when multiple output edges matched in FindPath. Add comments. * only allow one direction (input or output) in FindPath * [NupharEP] Update notebook and docker image (#2416) Add BERT squad in Nuphar tutorial Enhance speed comparsion readability * Fix the issue in matmul_add_fusion (#2407) Fix the issue in matmul_add_fusion If Muatmul + Add has shape [K] * [K, N], reset it to [1, K] * [K, N] will make the output shape to [1, N] will also requires a reshape on the output. Fix: just remove the shape reset to not fuse it. Add a negative test case for matmul+add fusion * feat(treeregressor): Update TreeEnsembleRegressor for type support (#2389) Updates the `TreeEnsembleRegressor` to allow for `double`, `float`, `int64`, and `int32` inputs to match the upstream specification. Signed-off-by: Nick Groszewski <nicholas.groszewski@capitalone.com> * onnxrt server documentation update (#2396) * Added support for Pad-2 operator in OpenVINO-EP (#2405) * Add CUDA If operator. (#2377) * Add CUDA If operator. Uses CPU operator for implementation. By adding a CUDA version the inputs/outputs (with the exception of the 'cond' input) stay on GPU, and no other logic is required to avoid a copy to CPU across the control flow node. * Improved documentation for onnxruntime::utils::SwapByteOrderCopy(), added precondition check. * Fix the type constraints on CUDA If operator to exclude strings. (#2431) * add Im2col<uint8_t> (#2438) * Adjust codegen vectorization width from target (#2439) * Adjust codegen vectorization width from target * Add CUDA Scan operator. (#2403) * Add Scan CUDA op. Uses CPU implementation for logic. Added some device specific functors for handling when data needs to be manipulated on a different device. Added ability to override the materialization logic in the OrtValue slicer so DML can plugin their handling. * Fix Windows GPU C API packaging pipeline failure (#2440) Fix Windows GPU C API packaging pipeline failure (#2440) * Correctly handle implicit inputs for fused nodes (#2390) * Correctly handle implicit inputs for fused nodes Previously, nuphar's partitioning function didn't include node's implicit inputs into the inputs list of MetaDef, and hence a crash was triggered in the onnx graph checker. This commit fixed the issue. Furthermore, it also fixed a related issue where we didn't add implicit inputs into graph_inputs_excluding_initializers_ in Graph::SetGraphInputsOutputs. the issue was that graph_inputs_including_initializers_ populated by SetInputs (e.g. called by FunctionImpl::FunctionImpl) may contain implicit inputs which were not of any node's initializers in the graph. Because they were not part of any initializers, these implicit inputs couldn't be visited by going through all nodes' inputs. Consequently, they would *not* be added into graph_inputs_excluding_initializers_. We fixed the issue by first copying the populated graph_inputs_including_initializers_ into graph_inputs_excluding_initalizers_, which then had both initializers and non-initializers as its initial content. Later, we erase initializers from the list. In this way, we can ensure all implicit inputs to remain in graph_inputs_excluding_initializers_. * refined comments and fixed duplicates Address CR by revisiting comments in terms of implicit inputs Also fixed an issue by skipping duplicates while copying inputs from graph_inputs_including_initializers_. * address CR explain why we need to collect nodes' implicit inputs * don't rely on pointer values for iterating std::set Previously, openvino relied on iterating a set of NodeArg pointers to construct inputs and outputs for a fused graph. It could cause non-determinism. The reason was that although iterating std::set by itself is stable, pointer values of NodeArgs may vary. Consequently, we could end up visiting the set's elements in different orders for different runs for the same test, which resulted in constructing inputs (and outputs) with different orders to the fused graph. For example, for the same test, we may have inputs [A, B] in some runs but inputs[B, A] in others. Let's use std::string as the key type to avoid such nondeterminism. This commit also added implicit inputs into meta->inputs while returning the capability from the openvino provider. * Fixed another latent issue in openvino's GetCapability function The issue was that we couldn't simply erase fused_inputs and fused_outputs while iterating the nodes. For example, an output NodeArg may have multiple uses, and it's wrong if we erase it from fused_outputs when we encounter only one of its uses as input. * Remove DeviceAllocatorRegistry class (#2451) Remove DeviceAllocatorRegistry class * CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library. * Parallel Gelu with ParallelFor (#2399) Parallel Gelu to get better performance for Gelu * Clean up build.py (#2446) * Pull the latest image before running docker build * Fuse SkipLayerNorm with Bias (#2453) Fuse SkipLayerNorm with Bias * Allow more than one invocation of CreateEnv in the same process. (#2467) * Allow more than one invocation of CreateEnv in the same process. * Fix centos build * Symbolic shape inference improvements: (#2460) * Symbolic shape inference improvements: - add a mode to guess unknown ops' output rank - add support for GatherND - add support for If - fix a bug in get_int_values when then tensor rank > 1D, by treating it as no sympy data - add symbol to literal merge when ONNX silently merges dims - fix a bug in Concat when input dim is 0 - fix a bug in ConstantOfShape that computed dim is not updated - add support for dynamic shape in ConstantOfShape - fix a bug in Loop output shape that loop iterator dim is not inserted at dim 0 - add support for dynamic padding in Pad - add support for dynamic shape in Reshape - add support for Resize with opset > 10, by treating output dims as dynamic - fix a bug in Slice when starts/ends are dynamic - restrict input model to opset 7 and above - make output model optional to avoid disk write when testing Run model tests for symbolic shape inference Reduce 2GB docker image size of nuphar * add additional test data set for nuget pipeline (#2448) * add SAS token to download internal test data for nuget pipeline * update azure endpoint * fix keyvault download step * fix variable declaration for secret group * fix indentation * fix yaml syntax for variables * fix setting secrets for script * fix env synctax * Fix macos pipeline * attempt to add secrets to windows download data * fix mac and win data download * fix windows data download * update test data set url and location * Revert "Brianma/windowsai fi (#2475)" This reverts commit 5780b864a15513fda4eadbfc2b5345fefe70b5ec. * Add scenario tests (#2457) * Add scenario tests * Remove TODO from model license * Add winml_api test dependency * fix model load test. fi from master changed the constructor (#2483) * make api tests all pass (#2486) * fix bad merge * fix bad model merge * Layer dev paulm (#2492) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * Rename ambiguous header (#2489) * fix one more missing IR version model (#2500) * add missing IR version to 4 more models used by scenario tests (#2501) * Add CLI parameters to test runner, build WinML in ARM and x86 CI (#2479) * Support test parameters through CLI arguments * Add WinML do Windows x86/ARM CI builds * Code style fixes * Update googletest Remove GPUTEST macros everywhere now that GTEST_SKIP is supported * Refactor main.cpp * Build scenario tests without DML * Link scenario tests to DML when it's enabled (#2502) * Layer dev release pipeline (#2488) Adds winml binaries to existing cpu nuget package, and creates new gpu dml nuget package with winml binaries and DML EP. * Layer dev paulm (#2506) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * Remove usage of IOBinding in WinML and use C_API Run method (#2504) * remove usage of iobinding * Change data structure to use vector of Ort::Values * Polish bind input / output * Use C APIrun method * Update providers on evaluate getresults * Remove run and IObinding interface from WinMLAdapter * Remove use of IObinding * bind unbound outputs code moved to learningmodelbinding * clean up unneeded istensor adapter function * Fix comment * Check if session is closed before binding and clearing * PR feedback * Layer dev paulm (#2507) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * Make tests dependend on winml_dll (#2509) * add dml binaries to DirectML package and be more explicit about condition variables (#2520) * re-enable warnings for winml builds and fix the warnings that were hiding (#2526) * turn devmode back on for winml builds * fix some warnings. include protobuf in a way that disables some warnings * undo protobufhelpers changes and just ignore 4100 errors in pb code * attempt to isolate protobufhelpers errors * add template specialization for getting tensor proto data * Layer dev paulm (#2533) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * Spawn child process to run DeviceLostRecovery scenario test (#2530) * Spawn child process to run DeviceLostRecovery scenario test * Layer dev paulm (#2536) ori said yes * add missing namespace to winml_trace_logging_provider in lotusenvironment.h (#2542) * Handle exception thrown from all apis in WinMLAdapter (#2539) * various changes to unblock windowsai ADO build * Fix custom ops scenario tests (#2562) * Do not shutdown protobuf after ort environment gets destroyed. Lazy load lotus environment first time it is needed * comment typo * pr comment about calling phoenix singleton * Make lotus_environment static in winmladapter * Layer dev paulm (#2567) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * making windowsAI azure dev ops work. * code review comments. * revert changes * Cmake and preprocessor fixes that where uncovered by building on agents without DML available via SDK * Layer dev dml delayload (#2580) * Brianma/cpu (#2583) * don't include dml stuff in cpu builds * tests that link the image lib also need the telemetry lib now * Throw Winml_err_invalid_binding if binding gpu resource on cpu device (#2589) * Throw Winml_err_invalid_binding if binding gpu resource on cpu device * PR comments. No need to query executionprovider for is gpu device * User/xianz/ortthrow (#2596) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * User/xianz/ortthrow (#2599) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * add status error message * Remove uwp onsuspending winrt call because logruntimeperf is getting removed (#2630) * User/xianz/dedup telemetry (#2631) * investigate duplication of telemetry in winml and ort * remove winml telemetry events * telemetry executionProviderEvent * remove unneccessary file and refactor code little bit * Revert back TelemetryEvent, which send up ETW event. * merge changes from layer_dev to windowsai (#2638) * Remove underscore from googletest names (#2616) * Fix leaking memory allocator Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24278761 and https://microsoft.visualstudio.com/OS/_workitems/edit/24330198 * Explicitly initialize Ort::Value with nullptr * Cache WinML adapter * bad merge * define private version of dxcore enum that is added in 19H1 SDK. (#2654) * add comment for explaning private definition of dxcore d3d feature level ennum value. (#2672) * do not package directml.pdb for redist packages. (#2676) * Fix leaking operator registry (#2645) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24354916 * User/orilevari/windowsai master merge (#2674) merge resolutions included pulling in telemetry logic that was merged to master and not windowsai and dereferencing InferenceSession::sessionstate now that it is a unique pointer * Delete Ort Allocator in LearningModelBinding (#2653) * Delete OrtAllocator in LearningModelBinding * PR comments to make Ort::Allocator a smart pointer * Small comment change * PR feedback to clean up code * PR feedback on move semantics * Clean up std::move * Fix memory leaks (#2679) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24356109, https://microsoft.visualstudio.com/OS/_workitems/edit/24388361 and https://microsoft.visualstudio.com/OS/_workitems/edit/24388596 * various changes to properly organize and skip GPU tests. For now for No DML builds we will not run GPU tests at all. In the future we should adapt the tests to expect the appropiate errors. (#2695) * Windowsai without fi (#2701) * Disable Attention fusion tests when DISABLE_CONTRIB_OPS is defined (#2529) * Setup java ci (#2528) * Add provision in ORT for session options to be parsed when available via model file (#2449) * Initial commit * Fix gitmodules * Nits * Nits * Updates * Update * More changes * Updates * Update * Some updates * More changes * Update * Update * Merge * Update * Updates * More changes * Update * Fix nits * Updates * Fix warning * Fix build * Add comment * PR feedback * PR feedback * Updates * Updates * Update * More changes * Fix build break * Comment test for now * Updates * Updates * PR feedback * Updates * Nits * Add tests * Fix build * Fix build * Fix build * Fix build break * Fix build * Nits * PR feedback * More change * Expose GetSessionOptions in pybind logic and add unit test for python * Fix build * PR feedback * PR feedback * Revert "Disable thread pool creation when enabled OpenMP (#2485)" (#2535) This reverts commit 7c7d5a149c9ed52eec67304bae5c4b132166a8a1. * Add dynamic shape support in TensorRT execution provider (#2450) * remove onnx-tensorrt submodule * add new onnx-tensorrt submodule (experiment) for trt6 * update engine build for trt6 * update compile and compute for tensorrt6.0 * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * switch to onnx-tensorrt master for TensorRT6' * Update tensorrt_execution_provider.cc * Handle dynamic batch size and add memcpy in TensorRT EP * update test cases * Update tensorrt_execution_provider.cc * update onnx-tensorrt submodule * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.ubuntu_tensorrt * Update run_dockerbuild.sh * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update concat_op_test.cc * Update tensorrt_execution_provider.cc * Upgrade TensorRT to version 6.0.1.5 * Update onnxruntime_providers.cmake * Update CMakeLists.txt * Update reduction_ops_test.cc * Update install_ubuntu.sh * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.tensorrt * Update BUILD.md * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update onnxruntime_providers.cmake * Update install_ubuntu.sh * Update install_ubuntu.sh * Update gemm_test.cc * Update gather_op_test.cc * Update CMakeLists.txt * Removed submodule * update onnx-tensorrt submodule * update header file * Removed submodule * add submodule onnx-tensorrt kevin's branch shape-test' * add debugging code * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * merge master * Removed submodule * update onnx-tensorrt submodule * add more changes for dynamic shapes * Update tensorrt_execution_provider.cc * update for dynamic shape * update dynamic shape processing * fix logger issue * remove submodule onnx-tensorrt * add submodule onnx-tensorrt * add env variable min_subgraph_size * remove redundency * update document * use onnxruntime::make_unique * fix multi-run issue * remove some tests to save CI build time * Add dynamic shape test * Update TensorRT-ExecutionProvider.md * Add example of running Faster R-CNN model on TensorRT EP * Add more details on env variables * update environment variables * Update tensorrt_basic_test.cc * Update model tests * Update tensor_op_test.cc * remove --use_full_protobuf * Update build.py * User/xianz/telemetry (#2458) * enabme telemetry * enable telemetry * set enable telemetry as default * for debugging * remove log and set disable telemetry as default back * delete private file while testing * resolve comment: mainly add license header, rename macro and update docs * rewording in privacy.md * Fix integer overflow in cuda NonMaxSuppression implementation (#2540) * add test case that should pass but fail * fix nms * extract int_max_output_boxes_per_class * Introduce container type runtime checks and other improvements (#2522) Rework TensorSeq in a manner consistent with Tensor and SparseTensor in terms of type system setup. Reduce templating. Introduce helpers to ensure the same data type. Make OrtValue __dtor not virtual. Introduce ContainerChecker * Fix C API tests for centos and mac (#2544) * change c++14 to c++11 * add ld lib path for centos * enable csharp tests on macos * fix C API test on MacOS + fix manylinux dotnet install * fix manylinux dotnet install * fix lib link * Add back executable bit to build.py * Fix a bug handling negative begin pad values in Pad op (#2550) * Fix bug in Pad op * Update * DNNL CMAKE update (#2548) * Fix android build (#2558) * Update win-x86-ci.yml (#2557) Fix build pipeline break * Re-enable Windows C# tests (#2564) * disable onnx_test_runner -x invocations for dnnl (#2568) * Allow sequence length to be symbolic (#2559) * setup java ci mac (#2570) * make layernorm fusion to support opset 11 (#2545) * Fix a warning found in the latest VS release * Add more check on SkipLayerNorm and BiasGelu fusion (#2574) * Fix file not found error during docker build. (#2569) * Add ConvTranspose1D (#2578) * Ryanunderhill/packagename test (#2582) * [Nuphar EP] fixes for some object detection models (#2581) Update notebook tutorial with multi-threaded int8 GEMM from #2517 * EmbedLayerNormalization Fusion Improvement (#2553) Embedding layer norm fusion improvements - add more checks * Update version (#2584) * Temporarily exclude vgg19 test from Python backend test 1. temporarily exclude vgg19 test which comsumes too much memory, run out of memory on Upsquared device. Single test pass for vgg19, need furture investigation (#2588) 2. Update docker file to decrease the docker image size * Update docs for Android NNAPI EP (#2586) * Fix lto bug for protobuf and ubuntu * add path to build dir before test run (#2590) * Add missig env variables for mac pipeline test (#2595) * Fixed an issue in updating realized dims (#2597) when we update realized dims for scan's output, the sliced axis also needs to be inclusive, i.e. we should check with "dim >= insert_inclusive_axis", because the offset in the symbols are based on Scan sugraph. Otherwise, we would end up with shape mismatch later. * Java API for onnxruntime (#2215) * Add support for opset 11 in reshape fusion (#2592) Support opset verion 11 in reshape fusion * Rename automl python tools folder to featurizer_ops. (#2593) * Support opset 11 subgraph of Squad model in Embed Layer Normalization (#2605) Support opset 11 Squad model that is exported from PyTorch nightly. The embed layer uses Range op which is missed in the transformer. * symbolic shape inference: fix warnings in GPT-2 model (#2608) And revise nuphar perf test on BERT squad * Dump subgraph ID and fused graph ID (#2607) * Dump subgraph ID and fused graph ID Dump subgraph ID and fused graph ID for better debugging * Remove local static fused_count added a field global_fused_count_ to NupharExecutionProvider class * EmbedLayerNormalization Fusion For Dynamic Squad Model Opset 10 (#2613) Support subgraph of SQuAD model exported from pytorch with dynamic input axes * Allow providers to be set for InferenceSession at construction (#2606) * Remove unnecessary parameter in some places in GatherElements implementation (#2612) * Remove unnecessary parameter in some places * Update * Update * Make sure fenced tensor could not reuse other tensor. (#2561) Fix random error caused by this. * Improve Embed Layer Norm Fusion for SQuAD with static input shape (#2621) * fix float16 comparison in initializer (#2629) * epsilon attribute for layernormalization fusion (#2639) * removed unnecessary batch file and fix path (#2640) * Add shape inference to ConvTransposeWithDynamicPads schema (#2632) * Improve cuda expand() opeator's performance. (#2624) * Cuda pad optimize when no padding is needed. (#2625) * Shortcut cuda Pad() when no padding is needed. * Optimize cuda scatter() on 2D compatible. (#2628) * Optimize cuda scatter() on 2D compatible. * Add some comments. * fix build error for ARM (#2648) * Improve performance of resize() in Nearest mode (#2626) Special treatment for 2D, check same size as input image. And in 2d kernel, template use_expolation. * Fix memory exception in Layer Norm Fusion (#2644) * Windows CI changes(#2650) * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Revert "Windowsai without fi (#2701)" This reverts commit 285d4c85ff5c4e265f963208170304ef3461e684. * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Deref unique pointer for session_state * send shutdown event when dll is unloaded and EvaluationStop, SessionC… (#2704) * send shutdown event when dll is unloaded and EvaluationStop, SessionCreationStart Events. * Add EvalutationStart Event * add comment * use correct type for for loop (#2755) * ARM CI (#2759) * Set ARM agent pool * Set CMake generator to VS 2019 in ARM * Use system-wide CMake instead of custom version Our custom version is too old for VS 2019 * Use DML and build shared lib in ARM CI * Restore nuget packages in ARM CI * Disable DML * Refactor ARM debug/release builds * Use system packaged Python version * Remove hardcoded Python path * Downgrade Python to 3.7 for build * Remove explicit CMake path * Fix invalid JSON in cgmanifest.json (#2760) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * Remove usage of WHOLEARCHIVE in WinML CMake and add WinMLAdapterFactory (#2726) * Remove usage of WHOLEARCHIVE in WinMLAdapter CMake and add WinMLAdapterFactory * PR feedback, no need for dll(export) since using def file * PR comments * Small comment in gen_def.py * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * Move winml_provider_factory.h to proper location (#2801) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Filter CPU case for IsFloat16Supported (#2802) * Merge fixes * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * Fix warnings that cause build to fail * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * Make DML include path global (#2882) * Make DML include path global * Add generated cppwinrt headers to winml_lib_common * Integrate changes to WindowsAI to make ADO Build (#2886) * Revert "CMake cross-generator fixes (#2790)" This reverts commit dbe7d97fa1ab155f1309bced87199527e8f35bd2. * add additional suppress warning in onnx_proto * ignore /wd4996 warning * DML execution provider fixes * Revert "Revert "CMake cross-generator fixes (#2790)"" This reverts commit 1ae7b4bcbc02edc881ad28685da98e095dfceb17. * Update func signature of custom op function overloads * common devicehelpers fixes * Add pch.h for winml_lib_common * re-add winml_lib_common_dir/inc to include path for winml_adapter * User/orilevari/dml redist shared folder (#2890) * move dml nuget package directory up one level to make it shared between build flavors * Merge conflict fix * Revert "Merge conflict fix" This reverts commit 142fa72cf9ce4344ad717b50b7ea2b8582aadc7c. * Revert "Merge remote-tracking branch 'origin/master' into windowsai" This reverts commit 6e2126d46e5e5f564d65da37dd4f70c93dd81165, reversing changes made to b3f5583dc9249834b947c8ea905f6a98060d5bd6. * Make winml_test_common free of test macros (#2902) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Don't generate files outside the build root (#2914) * Don't generate files outside the build root * Add onnxruntime_EXTERNAL_DEPENDENCIES to WinML * Add DML depedency on RESTORE_PACKAGES * User/orilevari/fix yaml merge bugs (#2918) * Add winml test source parameter into cmake function (#2919) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Add arguments to unittest cmake functions * remove comment * Revert "Revert "Merge remote-tracking branch 'origin/master' into windowsai"" This reverts commit ade5abe72a4234fdbc3623093c61c02c6b0bdc26. * Fix breaks from merge with ORT master * Brianma/linux (#2917) * don't include windows.h in cross-plat header * add default case for switch statement * signed/unsigned mismatch fix Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> * User/sheilk/winml adapter c api (#2891) * Create winml adapter c api * fix build * make it build * move adapter into onnxruntime core/session * entry point not exported * minor changes * make model metadata work * make tests pass * implement all the model reflection apis on the adapter c abi * update the new ort interface to create a lotus ennvironment with a logging sink * start adding ort env * move all winml code into adapter folder/lib to isolate it * ensure a single logging manager at a time * start refactoring session * refactor session creation interface * add cpu and dml session option methods to adapter * finish session init * stub out interfaces in ort lib to perform similar mechanics of iinference session * enable profiling, and enable schema override * update session register graph transformers * turn back on custom registry for custom ops * Add sync api * add last c api stubs * should build... but all feature values are broken since this is in flight to moving all implementation details into ivalue * remove ep adapter header * Implement DML execution provider functions from adapter (#2846) * Implement DML execution provider functions from adapter * Use functions in OnnxruntimeEngine.cpp * make map/sequence type_infos freeable, and start implementing ivalue * make it build again * implement value methods * implement remaining methods * remove com adapter abi * check dml session * cache the allocator on ivalue * check if resource is cpu/gpu when access its mutable data * update tensor * mismatched parentheses * fix tensor base and binding obj * it evaluates tensors! sometimes... * minor fixes * enable gpu evals * wrapper all existing winml adapter apis with API_IMPL to try catch (#2854) * update winml... tensor strings are broken, need to template tensorbase to do different things for strings * make tensor strings work with 2 copies in/2 copies out * Fix tensor string and allocator bug * make maps work again... needs some fixes still * Make it build! * enable map inputs * map outputs * unbound outputs for sequences and maps * User/xianz/merge windowsai (#2883) * Packaging pipeline changes for VS 2019 (#2711) * Tiny fix to codegen * Simplify cache implementation and avoid static variables that may carry over between models * Extend DML kernels (#2641) * Additional DML operators * Check unsupported attributes and inputs * Address PR comments * Add kernel capability function used for partitioning, and re-enable stride-based int64 support based on value range * Fix test failures * Build fix * PR comments * Update Nuphar tutorial notebook (#2721) 1. Reflect int8 GEMV improvements for multi-threading from #2696 2. Add notes on multi-threading control using OpenMP 3. Add samples of running multi-isa AOT, and show int8 GEMM differences between AVX and AVX2 4. Add rnn_benchmark example to resolve #1993 * Add schema for new Qops (#2611) * Add schema for new Qops * adding shape inference + qlinearaveragepool * plus review comments * plus review comments * updates per review comments * plus review comments * [server] Add supposed for model_name and model_version as cli parameter (#2708) * remove 64bit warning message from python validation. (#2727) * MLAS: ARM64 build fix (#2734) fix bad usage of vreinterpret to cast vector element types * Fix broken python docs links (#2740) * Fix build on Mac OS (#2731) mac os ld doesn't support --while-archive, correct option is -all_load * fix ngraph wheel (#2737) * fix ngraph wheel 1.1.0 onnxruntime_ngraph wheel doesn't work * remove libdnnl.so in nGraph Libs * make it easy to compare * Split onnxruntime server to a separated folder (#2744) * Fix build for Python 3.8 (#2747) * Fix build for Python 3.8 * Update protobuf to 3.11.2 (#1928) Update protobuf to 3.11.2 (#1928) * Change default optimization level to All (from Basic) (#2745) * change default optimization level to All (from Basic) * fix test * fix c# test * Update numpy to 1.18 (#2758) * Update numpy to 1.18 * Pipeline changes for python 3.8 (#2753) 1. Pipeline changes for python 3.8 2. Fix a regression in setup.py which was just introduced in the previous commit. Please notice, we still haven't made python 3.8 + Windows + CUDA work. * Add basic stacktrace output for posix debug builds. (#2749) * [NupharEP] fix a race condition when multiple sessions running different models concurrently (#2772) * Revert "Change default optimization level to All (from Basic) (#2745)" This reverts commit 56bb503c2f26474b6613bcb2a198691a11dcef38. * Fix typo in error message (#2736) * Rename MKL-DNN to DNNL to fix broken link (#2730) * Fix nightly build version number issue * Pass BUILD_BUILDNUMBER to linux docker * Disable featurizers in python packages * Import more featurizers (#2781) Make kernels non-template. Add input constraint for learnt data. Add min_max_scalar_transformer, robust_scalar_transformer, inputation_marker_transfomer, label_encoder_transformer, missing_dummies_transformer along with tests. Advance Featurizers library commit. * Implement a more stable softmax (#2715) * Implement a more stable SoftMax e^x is represented as infinity if x is large enough, like 100.f. Infinity divided by Infinity is a NAN. Thus, softmax gets a NAN if one or more item are large enough. A math transform as below is leveraged to get a stable softmax: e^xi/(e^x1 + ...e^xn) = e^(xi - max) / (e^(x1 - max) + ... + e^(xn - max)) And for convenience, force max to 0.f if all xi are negative * Contributing: Fix a typo (#2784) * ACL EP GEMM improvements (#2780) When it is posible we use a fully connected layer instead of the gemm implementation. This will let the library use the best implementation based on the input data. * ACL EP convolution improvements (#2774) Added the optimized implementation for depthwise convolution for both ACL v19.02 and ACL 19.05. Also the pointwise convolution seems to be more optimal in the CPU implementation so we opted for that instead. * Add script for release Nuget validation (#2719) * Initial commit * Nits * Disable a test temporarily * Change working directory * Test * Add download python step * Test update * More changes * Fix space issue * Fix * Verify nuget signing * Fix * Spaces * PR feedback * Nit * Fix * Fix * Remove temporary changes * add uint8 support to where op (#2792) * Improve bert optimization script: (#2712) (1) Move input int64=>int32 conversion to embed layer fusion. (2) Output epsilon attribute for LayerNormalization fusion. * add session creation time cost. (#2798) * ML.NET team needs featurizers within a package (#2789) Add auto ml featurizers to Windows, MacOS as well as to GPU packaging-pipelines. * Initialize max of softmax with lowest of float (#2786) * MLAS: update SGEMM threading parameters (#2808) * add interface to copy batch tensors. (#2807) * add interface to copy batch tensors. * onnxruntime * speed up Windows TRT CI (#2811) * don't run cuda tests if building with tensorrt * remove unnecessary build options for win trt ci * refactor win gpu tensorrt ci yml * --numpy_version=1.17 * update * update * azcopy and cuda path * Update test data (#2356) * Add timeseries imputer transformer featurizer kernel (#2813) Make kernels non-template. Add input constraint for learnt data. Fixup tests. Add two more featurizers along with tests. Tests fail. min_max_scalar_transformer robust_scalar_transformer Fix tests serialized stream by prepending version bytes. Add inputation_marker_transfomer and the test. Fix up float/double type designations. Added label_encoder_transformer along with a test. string_throw case is broken at the momement. Fix labelencodertransfomer_test.cc string_throw case Rename maxabsscalertransformer_test.cc Add MissingDummiesTransformer along with the test. Update manifest. Add TimeSeriesImputerTransformer definition, implementation and tests * Fix memory leak in TRT (#2815) * fix memory leak issue * revert EP_FAIL on enueueV2 * Add manifest missing comma * Run static code analyzer on most of our code (#2817) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * update quantization doc (#2783) * update documentation for quantization script * plus some spell corrections * Filter CPU case for IsFloat16Supported (#2802) * update default optimization level + fix gemm_activation fusion (#2791) * update defualt optimization level + fix gemm_activation fusion * fix typo * add unit test and incorporate review comments * fix test comment * Fix dnnl wheel package name (#2823) * Append '-dnnl' to whl package name when --use_dnnl * Update build.py * Update Ubuntu & TensorRT version in README (#2820) Dockerfile.tensorrt is using nvcr.io/nvidia/tensorrt:19.09-py3 as base Image, update Ubuntu and TensorRT version according to https://docs.nvidia.com/deeplearning/sdk/tensorrt-container-release-notes/rel_19-09.html#rel_19-09 * Merge fixes * Add OneHotEncoder and HashOneHotEncoder kernels. (#2830) Add defs and imlementation for OneHotEncoders, adjuist date_time_transformer kernel and test. Add OneHotEncoder kernel test. Add HashOneHotVectorizerTransformer unit test. This does not link due to multiple definitions of functions that are included into header from a CPP file. * Upgrade gtest to the latest version (#2827) WinML would like to update the googletest submodule. They want some newer features (namely GTEST_SKIP to skip tests programmatically and be able to skip entire fixtures easily) and would need to update the submodule version. However, because the new version of code hit a bug in gcc, even though the bug is already fixed in the latest gcc but we're using gcc 4.8.x and it won't get patched for the bug, so we have to do a compromise, change our code a little bit to make it work. The gcc bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=51213 * Add support for int64_t for topk CPU. Fixes github issue #2806. (#2833) * Ignore allocator type in ExecutionProviders allocator map. Make default initialization of OrtMemoryInfo more clearly invalid. (#2768) * Remove allocator type from the key comparison in ExecutionProviders. Remove usage of DummyArena as it's no longer necessary. * Fix x86 tests where arena allocator is disabled. Make initialization of OrtMemoryInfo clearer by adding Invalid enum value. * Make OrtValueNameIdxMap::MaxIdx more intuitive. * Convert ExternalProject Featurizers into git submodule (#2834) Add git submodule for Featurizer library. Update cmake to build for git submodule. * add domain check for nodes + update documentation (#2831) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Fix Linux CUDA nuget packaging pipeline break * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * update optimization doc for BERT related fusions (#2819) * Add bert related transformers to doc * Add execution provider and comment for bert optimizations * Add comment about accuracy impact of approximation * Fix warnings that cause build to fail * MLAS: enable threading for quantized GEMMs (#2844) * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * remove old winmladapter cpp Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * move sequence implementation into ort lib... still commented out... need to turn back on... * begin sequence implementation * make maps and sequences work * fix broken tests * remove dead code * misc cleanup * CR feedback * User/xianz/winml adapter c api (#2869) * wrapper all existing winml adapter apis with API_IMPL to try catch * Return HR or Throw for WinML adapter APIs if failed * undo macro wrapper for two places * Wrap error macros around ort apis, too. * address CR feedback #2 * add more api throw/return macros * Revert changes no longer needed * revert changes to cxx api * format winml lib.ort and winml adapter * remove static pheonix singleton Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * missing use_dml check in winml_adapter_session (#2930) * --use_dnnl flag was mangled in merge (#2931) * use dml macro not wrapping custom registry code (#2934) * Disable LNK4199 winml_dll to enable cuda builds (#2936) * Disable LNK4199 in winml_dll * linkler->linker * LearningModelSessionAPITestGpu.CreateSessionWithCastToFloat16InModel should return DXGI_ERROR_UNSUPPORTED when FP16 not supported (#2937) * Disable LNK4199 in winml_dll * linkler->linker * Need to return DXGI_ERROR_UNSUPPORTED when Model does not support fp16 * Publish build symbols (#2939) * Publish build symbols * Don't upload PDBs for .exe files * Make x86 build (#2943) * fix last remaining size_t/int64_t warnings->errors (#2948) * TensorString, Sequences and Maps use the first allocator, but should use the cpu default allocator. (#2952) * fix tensor string allcoator * clean up default allocator usage for strings in winml lib/api.ort Co-authored-by: Ryan Lai <ryalai96@gmail.com> * Handle tensor shape of zero (#2954) Co-authored-by: Ryan Lai <ryalai96@gmail.com> * CR feedback (#2970) * CR feedback * fix weird formatting on privacy readme * Add 'All rights reserved.' everywhere * readd all rights reserved to winml_provider_factory.h * remove extra space in comment * remove extra whitespace * fixes post master merge * remove winml from nuget gpu pipeline * set IR VERSION on generated_model in rnn_benchmark (#2972) * Fix slice conformance failures (#2908) Co-authored-by: Adrian Tsai <adtsai@microsoft.com> Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Paul McDaniel <paul_mcdaniel@hotmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Dwayne Robinson <fdwr@hotmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com>
2020-02-05 01:12:19 +00:00
interface ITensor : IInspectable requires ILearningModelFeatureValue
{
//! Returns the data type of the tensor.
TensorKind TensorKind{ get; };
//! TODO:Docs
Windows.Foundation.Collections.IVectorView<Int64> Shape{ get; };
}
//! \class TensorFloat
//! \brief A 32bit float tensor object.
[contract(MachineLearningContract, 1)]
INBOX_ONLY([static_name("Windows.AI.MachineLearning.ITensorFloatStatics", dbcd395b-3ba3-452f-b10d-3c135e573fa9)])
INBOX_ONLY([interface_name("Windows.AI.MachineLearning.ITensorFloat", f2282d82-aa02-42c8-a0c8-df1efc9676e1)])
Merge windowsai (winml layering) into master (#2956) * Initial Commit * Merged PR 3985217: add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc (#2346) add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc and violating our OS layering requirements. We linked against onecoreuap_apiset.lib in VB so we will continue doing this, but I am still unsure why not to link against onecore instead since that is where we ship. However, since Sheil is the owner of this code we will wait to discuss with him before changing anything. * Initial changes for layering * more snipping to get core into ort * update build instructions to include --build_shared_lib (#2358) * update build instructions to include --build_shared_lib * fix line breaks * Task 23998197: add winml_lib_core into onnnxruntime.dll (#2368) * Task 23998197: add winml_lib_core into onnnxruntime.dll * PR feedback build break on perf_test * return proper error when the model path isn't found (#2391) * LearningModelSession is cleaned up to use the adapter, and parts of b… (#2382) this is a big PR. we are going to move it up to layer_dev , which is still a L3 so we are still safe to do work there agile. we are going to move this into the L3 so that ryan can start doing intergration testing. we will pause for a full code review and integration test result prior to going into the L2. >>>> raw comments from previous commits >>> * LearningModelSession is cleaned up to use the adapter, and parts of binding are. * moved everything in the winmladapter made it all nano-com using, WRL to construct objects in the ORT side. base interfaces for everythign for winml to call cleaned up a bunch of winml to use the base interfaces. * more pieces * GetData across the abi. * renamed some namepsace cleaned up OrtValue cleaned up Tensor cleaned up custom ops. everything *but* learnignmodel should be clean * make sure it's building. winml.dll is still a monolith. * model moved over. everything builds clean. step ! * weak ref comment * Layer dev paulm (#2408) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * Layer dev paulm (#2414) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * Layer dev paulm (#2423) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * Layer dev paulm (#2424) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * Layer dev paulm (#2425) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Layer dev paulm (#2426) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * fixed map and sequence when passing stl types across the ABI . found a leak in nvidia driver, but skipped it. all winmlapitests pass now * Moved SessionOptions over to the abi * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * Register WinML TraceLogging provider on Onnxruntime.dll (#2455) * Register WinML TraceLogging provider on Onnxruntime.dll * Add ifdef to make sure trace logging provider has telemetry option when LAYERING_DONE * No need for ifdef for TraceLoggingOptionMicrosoftTelemetry * PR feedback * Move etw registration into lotus environment constructor and deresgister in lotus environment destructor * Brianma/cpuwinml (#2466) * allow building winml cpu without dml. * Brianma/breaks (#2469) * fix some more breaks * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * move dml checks out of winml and into the adapter * better error handling * Brianma/fi (#2470) * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * fix bad merge * Checking in a staging checkpoint point so that Ryan can work with me in parrallel * build break. * Brianma/testfails (#2473) * add missing ir version to dictvectorizer-string.onnx * add missing ir version to relu.onnx * add missing ir version to zipmap*onnx * add IR version to manually generated models * remove an unnecessary ifdef dml * Brianma/windowsai fi (#2475) * update dockerfiles/README (#2336) * Make elementwise op run 4 items per thread (#2335) Description: Describe your changes. Make elementwise op run 4 items per thread unroll for loop to leverage ILP remove unnessary N==0 check inside elementwise GPU kernel Motivation and Context Why is this change required? What problem does it solve? It can improve the performance of GPU elementwise ops. ~2% performance gain on popular NLP bert model. If it fixes an open issue, please link to the issue here. * Add CUDA GatherElements kernel (#2310) * Updates * Update test * Update * Updates * nits * PR feedback * Update * Update * PR feedback * PR comments * Update * Fix build * Fix build * Nits * Fix * Layer Normalization Fusion (#2319) basic layer normalization transform * Add FastGelu Cuda Op for Gelu and Add bias fusion (#2293) * Add FastGelu cuda op * Add AddBiasGelu for experiment * Revert "Add AddBiasGelu for experiment" This reverts commit 5c1ee019858c657e6bb75887265cb85675626e5b. * Add bias * Add unit tests * update comment * update script * fix build error * update coding style * update for CR feedback Enable half2 optimization only when cuda arch >= 7.0 * move _Tanh to common.cuh * implement CPU contrib OP Attention (#2333) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanUnusedInitializers. (#2320) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanupUnusedInitializers. This means initializers that have been replaced during graph optimizations are not left in the GraphProto when we save an optimized model. * Handle edge case where a model has an unused initializer with matching graph input by also removing the graph input. * Use non-const iterators in std::find_if calls to make centos build happy. * Nuget pipeline changes (#2305) 1. refactor the pipeline, remove some duplicated code 2. Move Windows_py_GPU_Wheels job to Win-GPU-CUDA10. We'll deprecated the "Win-GPU" pool 3. Delete cpu-nocontribops-esrp-pipeline.yml and cpu-nocontribops-pipeline.yml 4. In Linux nuget jobs, run "make install" before creating the package. So that extra RPAH info will be removed * Cuda Reverse Sequence Op, maping types of same size using same template function. (#2281) * Set ElementType to String type of node metadata, instead of byte[] (#2348) * Set ElementType to String type of node metadata, instead of byte[] * Fix spacing * Introduce PrimitiveType into a Type System along with an integer constant (#2307) Improve perf by avoiding GetType<T>() calls. Introduce MLTypeCallDispatcher to switch on Input Type. Add Tensor IsType<T>() fast method. * Fix/test dim value of 0 handling in a couple of places (#2337) * Update the CUDA Where implementation broadcasting logic to handle a dim with value of 0. Add unit test Also add unit test for unary op with dim value of 0 * Exclude ngraph from Where test with 0 dim. * Openvino EP R3.1 onnxrt server (#2357) * onnxrt server with OVEP * onnxrt server with OVEP * Update Dockerfile.server.openvino * onnxrt server OVEP fix reviews * onnxrt server OVEP fix reviews * Implement cuda nonzero op. (#2056) Implement cuda nonzero op. * Direct use python numpy array's memory if already contiguous. (#2355) * Direct use python numpy array's memory if already contiguous. This could greatly improve performance for session with large input, like big image 1920x1080 fastrcnn, 30~40% speed up could be achieved. * Add test case enforce contiguous/non-contiguos numpy array as inputs. * Add helper to create output to minimize binary size. (#2365) Add ConstEigenTensorMap typedef so we don't unnecessarily const_cast the const input Tensor. * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS (#2369) * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS * update * Add Tracelogging for profiling (#1639) Enabled only if onnxruntime_ENABLE_INSTRUMENT is ON * test bidaf with nuphar for avx target (#2370) increase nuphar test coverage a bit * Fix a bug in TLS refcount that may destabilized CUDA CI (#2374) * update output size calculation for resize (#2366) * change how output size is calculated for resize op * add tests for ver 10 resize * Extend OneHot CPU kernel to support more types (#2311) * Extend OneHot CPU kernel to support input int64_t, depth int32_t, output float * Skip BERT before the test data fix is picked up * Fix bug with Slice. Need to pass in flattened input dimensions so the initial offset into the input is calculated correctly. (#2372) * Add opset 11 version of Split to CUDA ops (#2376) Organize the CUDA ops definitions so all the opset 10 and 11 parts are together (same setup used for CPU ops) * Layer Norm Fusion Fix (#2379) * layer norm fusion fix * Add input shape check in code and unit tests * Fuse Add + Gelu (#2360) Implement the transformer to fuse add + gelu Implement the accurate kernel * Skip layer norm transform (#2350) * skip layer normalization transformer * Another try to stabilize CUDA CI (#2383) The root cause seems to be failure in CUDA dealloc when tear down. cudaFree return code was ignored before, so should the debug check. * fix BUILD.md typo (#2375) build.py: error: argument --config: invalid choice: 'RelWithDebugInfo' (choose from 'Debug', 'MinSizeRel', 'Release', 'RelWithDebInfo') * Fixed compilation with ngraph (#2388) * Fix reuse logic in allocation planner. (#2393) * Fix reuse logic in allocation planner. * PR comments * Add helpful comments * Don't allow reuse across string tensors. * [NupharEP] Multiple optimizations (#2380) Fuse transpose into MatMul Implement Pow and constant scalar simplification Vectorize ReduceMean Improve symbolic shape inference Minor updates for better debugging in fused function name * Avoid using the default logger in the graph lib and optimizers (#2361) 1. Use the session logger if it is available. 2. Don't disable warning 4100 globally. We should fix the warnings instead of disabling it. * Change CUDA implementation of Transpose to support all fixed size tensor types (#2387) * Change CUDA implementation of Transpose to not use a typed kernel so we can support more types with minimum binary size. Add support for 8, 16, 32 and 64 bit types. Add unit tests. Add method so the implementation can be called directly (will be used by CUDA Scan very soon). * Disable TensorRT for MLFloat16 and int8 unit tests. * Address PR comment and add support for calling cublas implementation if type is mlfloat16. * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. (#2398) * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. * [NupharEP] force some low/zero cost ops to be inlined (#2409) * fix cross compile bug (#2415) * Minor optimization: if a node has already been placed, there's no need to find a kernel for it. (#2417) * Add Reshape Fusion (#2395) * Add reshape fusion * Add some comments * update comments * update comment format * update according to feedback * update for recent logger change * fix build error * (1) Support both input and output edges in find path in graphutils (2) Add a test case of only one constant initializer of Concat input. (3) Refactor ReshapeFusion class to allow add more subgraph fusion in the future. * fix error * (1) loose constraint on initializer: non constant is allowed for reshape fusion. (2) Change versions type to vector. (3) Add logging. (4) Return false when multiple output edges matched in FindPath. Add comments. * only allow one direction (input or output) in FindPath * [NupharEP] Update notebook and docker image (#2416) Add BERT squad in Nuphar tutorial Enhance speed comparsion readability * Fix the issue in matmul_add_fusion (#2407) Fix the issue in matmul_add_fusion If Muatmul + Add has shape [K] * [K, N], reset it to [1, K] * [K, N] will make the output shape to [1, N] will also requires a reshape on the output. Fix: just remove the shape reset to not fuse it. Add a negative test case for matmul+add fusion * feat(treeregressor): Update TreeEnsembleRegressor for type support (#2389) Updates the `TreeEnsembleRegressor` to allow for `double`, `float`, `int64`, and `int32` inputs to match the upstream specification. Signed-off-by: Nick Groszewski <nicholas.groszewski@capitalone.com> * onnxrt server documentation update (#2396) * Added support for Pad-2 operator in OpenVINO-EP (#2405) * Add CUDA If operator. (#2377) * Add CUDA If operator. Uses CPU operator for implementation. By adding a CUDA version the inputs/outputs (with the exception of the 'cond' input) stay on GPU, and no other logic is required to avoid a copy to CPU across the control flow node. * Improved documentation for onnxruntime::utils::SwapByteOrderCopy(), added precondition check. * Fix the type constraints on CUDA If operator to exclude strings. (#2431) * add Im2col<uint8_t> (#2438) * Adjust codegen vectorization width from target (#2439) * Adjust codegen vectorization width from target * Add CUDA Scan operator. (#2403) * Add Scan CUDA op. Uses CPU implementation for logic. Added some device specific functors for handling when data needs to be manipulated on a different device. Added ability to override the materialization logic in the OrtValue slicer so DML can plugin their handling. * Fix Windows GPU C API packaging pipeline failure (#2440) Fix Windows GPU C API packaging pipeline failure (#2440) * Correctly handle implicit inputs for fused nodes (#2390) * Correctly handle implicit inputs for fused nodes Previously, nuphar's partitioning function didn't include node's implicit inputs into the inputs list of MetaDef, and hence a crash was triggered in the onnx graph checker. This commit fixed the issue. Furthermore, it also fixed a related issue where we didn't add implicit inputs into graph_inputs_excluding_initializers_ in Graph::SetGraphInputsOutputs. the issue was that graph_inputs_including_initializers_ populated by SetInputs (e.g. called by FunctionImpl::FunctionImpl) may contain implicit inputs which were not of any node's initializers in the graph. Because they were not part of any initializers, these implicit inputs couldn't be visited by going through all nodes' inputs. Consequently, they would *not* be added into graph_inputs_excluding_initializers_. We fixed the issue by first copying the populated graph_inputs_including_initializers_ into graph_inputs_excluding_initalizers_, which then had both initializers and non-initializers as its initial content. Later, we erase initializers from the list. In this way, we can ensure all implicit inputs to remain in graph_inputs_excluding_initializers_. * refined comments and fixed duplicates Address CR by revisiting comments in terms of implicit inputs Also fixed an issue by skipping duplicates while copying inputs from graph_inputs_including_initializers_. * address CR explain why we need to collect nodes' implicit inputs * don't rely on pointer values for iterating std::set Previously, openvino relied on iterating a set of NodeArg pointers to construct inputs and outputs for a fused graph. It could cause non-determinism. The reason was that although iterating std::set by itself is stable, pointer values of NodeArgs may vary. Consequently, we could end up visiting the set's elements in different orders for different runs for the same test, which resulted in constructing inputs (and outputs) with different orders to the fused graph. For example, for the same test, we may have inputs [A, B] in some runs but inputs[B, A] in others. Let's use std::string as the key type to avoid such nondeterminism. This commit also added implicit inputs into meta->inputs while returning the capability from the openvino provider. * Fixed another latent issue in openvino's GetCapability function The issue was that we couldn't simply erase fused_inputs and fused_outputs while iterating the nodes. For example, an output NodeArg may have multiple uses, and it's wrong if we erase it from fused_outputs when we encounter only one of its uses as input. * Remove DeviceAllocatorRegistry class (#2451) Remove DeviceAllocatorRegistry class * CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library. * Parallel Gelu with ParallelFor (#2399) Parallel Gelu to get better performance for Gelu * Clean up build.py (#2446) * Pull the latest image before running docker build * Fuse SkipLayerNorm with Bias (#2453) Fuse SkipLayerNorm with Bias * Allow more than one invocation of CreateEnv in the same process. (#2467) * Allow more than one invocation of CreateEnv in the same process. * Fix centos build * Symbolic shape inference improvements: (#2460) * Symbolic shape inference improvements: - add a mode to guess unknown ops' output rank - add support for GatherND - add support for If - fix a bug in get_int_values when then tensor rank > 1D, by treating it as no sympy data - add symbol to literal merge when ONNX silently merges dims - fix a bug in Concat when input dim is 0 - fix a bug in ConstantOfShape that computed dim is not updated - add support for dynamic shape in ConstantOfShape - fix a bug in Loop output shape that loop iterator dim is not inserted at dim 0 - add support for dynamic padding in Pad - add support for dynamic shape in Reshape - add support for Resize with opset > 10, by treating output dims as dynamic - fix a bug in Slice when starts/ends are dynamic - restrict input model to opset 7 and above - make output model optional to avoid disk write when testing Run model tests for symbolic shape inference Reduce 2GB docker image size of nuphar * add additional test data set for nuget pipeline (#2448) * add SAS token to download internal test data for nuget pipeline * update azure endpoint * fix keyvault download step * fix variable declaration for secret group * fix indentation * fix yaml syntax for variables * fix setting secrets for script * fix env synctax * Fix macos pipeline * attempt to add secrets to windows download data * fix mac and win data download * fix windows data download * update test data set url and location * Revert "Brianma/windowsai fi (#2475)" This reverts commit 5780b864a15513fda4eadbfc2b5345fefe70b5ec. * Add scenario tests (#2457) * Add scenario tests * Remove TODO from model license * Add winml_api test dependency * fix model load test. fi from master changed the constructor (#2483) * make api tests all pass (#2486) * fix bad merge * fix bad model merge * Layer dev paulm (#2492) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * Rename ambiguous header (#2489) * fix one more missing IR version model (#2500) * add missing IR version to 4 more models used by scenario tests (#2501) * Add CLI parameters to test runner, build WinML in ARM and x86 CI (#2479) * Support test parameters through CLI arguments * Add WinML do Windows x86/ARM CI builds * Code style fixes * Update googletest Remove GPUTEST macros everywhere now that GTEST_SKIP is supported * Refactor main.cpp * Build scenario tests without DML * Link scenario tests to DML when it's enabled (#2502) * Layer dev release pipeline (#2488) Adds winml binaries to existing cpu nuget package, and creates new gpu dml nuget package with winml binaries and DML EP. * Layer dev paulm (#2506) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * Remove usage of IOBinding in WinML and use C_API Run method (#2504) * remove usage of iobinding * Change data structure to use vector of Ort::Values * Polish bind input / output * Use C APIrun method * Update providers on evaluate getresults * Remove run and IObinding interface from WinMLAdapter * Remove use of IObinding * bind unbound outputs code moved to learningmodelbinding * clean up unneeded istensor adapter function * Fix comment * Check if session is closed before binding and clearing * PR feedback * Layer dev paulm (#2507) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * Make tests dependend on winml_dll (#2509) * add dml binaries to DirectML package and be more explicit about condition variables (#2520) * re-enable warnings for winml builds and fix the warnings that were hiding (#2526) * turn devmode back on for winml builds * fix some warnings. include protobuf in a way that disables some warnings * undo protobufhelpers changes and just ignore 4100 errors in pb code * attempt to isolate protobufhelpers errors * add template specialization for getting tensor proto data * Layer dev paulm (#2533) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * Spawn child process to run DeviceLostRecovery scenario test (#2530) * Spawn child process to run DeviceLostRecovery scenario test * Layer dev paulm (#2536) ori said yes * add missing namespace to winml_trace_logging_provider in lotusenvironment.h (#2542) * Handle exception thrown from all apis in WinMLAdapter (#2539) * various changes to unblock windowsai ADO build * Fix custom ops scenario tests (#2562) * Do not shutdown protobuf after ort environment gets destroyed. Lazy load lotus environment first time it is needed * comment typo * pr comment about calling phoenix singleton * Make lotus_environment static in winmladapter * Layer dev paulm (#2567) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * making windowsAI azure dev ops work. * code review comments. * revert changes * Cmake and preprocessor fixes that where uncovered by building on agents without DML available via SDK * Layer dev dml delayload (#2580) * Brianma/cpu (#2583) * don't include dml stuff in cpu builds * tests that link the image lib also need the telemetry lib now * Throw Winml_err_invalid_binding if binding gpu resource on cpu device (#2589) * Throw Winml_err_invalid_binding if binding gpu resource on cpu device * PR comments. No need to query executionprovider for is gpu device * User/xianz/ortthrow (#2596) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * User/xianz/ortthrow (#2599) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * add status error message * Remove uwp onsuspending winrt call because logruntimeperf is getting removed (#2630) * User/xianz/dedup telemetry (#2631) * investigate duplication of telemetry in winml and ort * remove winml telemetry events * telemetry executionProviderEvent * remove unneccessary file and refactor code little bit * Revert back TelemetryEvent, which send up ETW event. * merge changes from layer_dev to windowsai (#2638) * Remove underscore from googletest names (#2616) * Fix leaking memory allocator Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24278761 and https://microsoft.visualstudio.com/OS/_workitems/edit/24330198 * Explicitly initialize Ort::Value with nullptr * Cache WinML adapter * bad merge * define private version of dxcore enum that is added in 19H1 SDK. (#2654) * add comment for explaning private definition of dxcore d3d feature level ennum value. (#2672) * do not package directml.pdb for redist packages. (#2676) * Fix leaking operator registry (#2645) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24354916 * User/orilevari/windowsai master merge (#2674) merge resolutions included pulling in telemetry logic that was merged to master and not windowsai and dereferencing InferenceSession::sessionstate now that it is a unique pointer * Delete Ort Allocator in LearningModelBinding (#2653) * Delete OrtAllocator in LearningModelBinding * PR comments to make Ort::Allocator a smart pointer * Small comment change * PR feedback to clean up code * PR feedback on move semantics * Clean up std::move * Fix memory leaks (#2679) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24356109, https://microsoft.visualstudio.com/OS/_workitems/edit/24388361 and https://microsoft.visualstudio.com/OS/_workitems/edit/24388596 * various changes to properly organize and skip GPU tests. For now for No DML builds we will not run GPU tests at all. In the future we should adapt the tests to expect the appropiate errors. (#2695) * Windowsai without fi (#2701) * Disable Attention fusion tests when DISABLE_CONTRIB_OPS is defined (#2529) * Setup java ci (#2528) * Add provision in ORT for session options to be parsed when available via model file (#2449) * Initial commit * Fix gitmodules * Nits * Nits * Updates * Update * More changes * Updates * Update * Some updates * More changes * Update * Update * Merge * Update * Updates * More changes * Update * Fix nits * Updates * Fix warning * Fix build * Add comment * PR feedback * PR feedback * Updates * Updates * Update * More changes * Fix build break * Comment test for now * Updates * Updates * PR feedback * Updates * Nits * Add tests * Fix build * Fix build * Fix build * Fix build break * Fix build * Nits * PR feedback * More change * Expose GetSessionOptions in pybind logic and add unit test for python * Fix build * PR feedback * PR feedback * Revert "Disable thread pool creation when enabled OpenMP (#2485)" (#2535) This reverts commit 7c7d5a149c9ed52eec67304bae5c4b132166a8a1. * Add dynamic shape support in TensorRT execution provider (#2450) * remove onnx-tensorrt submodule * add new onnx-tensorrt submodule (experiment) for trt6 * update engine build for trt6 * update compile and compute for tensorrt6.0 * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * switch to onnx-tensorrt master for TensorRT6' * Update tensorrt_execution_provider.cc * Handle dynamic batch size and add memcpy in TensorRT EP * update test cases * Update tensorrt_execution_provider.cc * update onnx-tensorrt submodule * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.ubuntu_tensorrt * Update run_dockerbuild.sh * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update concat_op_test.cc * Update tensorrt_execution_provider.cc * Upgrade TensorRT to version 6.0.1.5 * Update onnxruntime_providers.cmake * Update CMakeLists.txt * Update reduction_ops_test.cc * Update install_ubuntu.sh * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.tensorrt * Update BUILD.md * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update onnxruntime_providers.cmake * Update install_ubuntu.sh * Update install_ubuntu.sh * Update gemm_test.cc * Update gather_op_test.cc * Update CMakeLists.txt * Removed submodule * update onnx-tensorrt submodule * update header file * Removed submodule * add submodule onnx-tensorrt kevin's branch shape-test' * add debugging code * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * merge master * Removed submodule * update onnx-tensorrt submodule * add more changes for dynamic shapes * Update tensorrt_execution_provider.cc * update for dynamic shape * update dynamic shape processing * fix logger issue * remove submodule onnx-tensorrt * add submodule onnx-tensorrt * add env variable min_subgraph_size * remove redundency * update document * use onnxruntime::make_unique * fix multi-run issue * remove some tests to save CI build time * Add dynamic shape test * Update TensorRT-ExecutionProvider.md * Add example of running Faster R-CNN model on TensorRT EP * Add more details on env variables * update environment variables * Update tensorrt_basic_test.cc * Update model tests * Update tensor_op_test.cc * remove --use_full_protobuf * Update build.py * User/xianz/telemetry (#2458) * enabme telemetry * enable telemetry * set enable telemetry as default * for debugging * remove log and set disable telemetry as default back * delete private file while testing * resolve comment: mainly add license header, rename macro and update docs * rewording in privacy.md * Fix integer overflow in cuda NonMaxSuppression implementation (#2540) * add test case that should pass but fail * fix nms * extract int_max_output_boxes_per_class * Introduce container type runtime checks and other improvements (#2522) Rework TensorSeq in a manner consistent with Tensor and SparseTensor in terms of type system setup. Reduce templating. Introduce helpers to ensure the same data type. Make OrtValue __dtor not virtual. Introduce ContainerChecker * Fix C API tests for centos and mac (#2544) * change c++14 to c++11 * add ld lib path for centos * enable csharp tests on macos * fix C API test on MacOS + fix manylinux dotnet install * fix manylinux dotnet install * fix lib link * Add back executable bit to build.py * Fix a bug handling negative begin pad values in Pad op (#2550) * Fix bug in Pad op * Update * DNNL CMAKE update (#2548) * Fix android build (#2558) * Update win-x86-ci.yml (#2557) Fix build pipeline break * Re-enable Windows C# tests (#2564) * disable onnx_test_runner -x invocations for dnnl (#2568) * Allow sequence length to be symbolic (#2559) * setup java ci mac (#2570) * make layernorm fusion to support opset 11 (#2545) * Fix a warning found in the latest VS release * Add more check on SkipLayerNorm and BiasGelu fusion (#2574) * Fix file not found error during docker build. (#2569) * Add ConvTranspose1D (#2578) * Ryanunderhill/packagename test (#2582) * [Nuphar EP] fixes for some object detection models (#2581) Update notebook tutorial with multi-threaded int8 GEMM from #2517 * EmbedLayerNormalization Fusion Improvement (#2553) Embedding layer norm fusion improvements - add more checks * Update version (#2584) * Temporarily exclude vgg19 test from Python backend test 1. temporarily exclude vgg19 test which comsumes too much memory, run out of memory on Upsquared device. Single test pass for vgg19, need furture investigation (#2588) 2. Update docker file to decrease the docker image size * Update docs for Android NNAPI EP (#2586) * Fix lto bug for protobuf and ubuntu * add path to build dir before test run (#2590) * Add missig env variables for mac pipeline test (#2595) * Fixed an issue in updating realized dims (#2597) when we update realized dims for scan's output, the sliced axis also needs to be inclusive, i.e. we should check with "dim >= insert_inclusive_axis", because the offset in the symbols are based on Scan sugraph. Otherwise, we would end up with shape mismatch later. * Java API for onnxruntime (#2215) * Add support for opset 11 in reshape fusion (#2592) Support opset verion 11 in reshape fusion * Rename automl python tools folder to featurizer_ops. (#2593) * Support opset 11 subgraph of Squad model in Embed Layer Normalization (#2605) Support opset 11 Squad model that is exported from PyTorch nightly. The embed layer uses Range op which is missed in the transformer. * symbolic shape inference: fix warnings in GPT-2 model (#2608) And revise nuphar perf test on BERT squad * Dump subgraph ID and fused graph ID (#2607) * Dump subgraph ID and fused graph ID Dump subgraph ID and fused graph ID for better debugging * Remove local static fused_count added a field global_fused_count_ to NupharExecutionProvider class * EmbedLayerNormalization Fusion For Dynamic Squad Model Opset 10 (#2613) Support subgraph of SQuAD model exported from pytorch with dynamic input axes * Allow providers to be set for InferenceSession at construction (#2606) * Remove unnecessary parameter in some places in GatherElements implementation (#2612) * Remove unnecessary parameter in some places * Update * Update * Make sure fenced tensor could not reuse other tensor. (#2561) Fix random error caused by this. * Improve Embed Layer Norm Fusion for SQuAD with static input shape (#2621) * fix float16 comparison in initializer (#2629) * epsilon attribute for layernormalization fusion (#2639) * removed unnecessary batch file and fix path (#2640) * Add shape inference to ConvTransposeWithDynamicPads schema (#2632) * Improve cuda expand() opeator's performance. (#2624) * Cuda pad optimize when no padding is needed. (#2625) * Shortcut cuda Pad() when no padding is needed. * Optimize cuda scatter() on 2D compatible. (#2628) * Optimize cuda scatter() on 2D compatible. * Add some comments. * fix build error for ARM (#2648) * Improve performance of resize() in Nearest mode (#2626) Special treatment for 2D, check same size as input image. And in 2d kernel, template use_expolation. * Fix memory exception in Layer Norm Fusion (#2644) * Windows CI changes(#2650) * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Revert "Windowsai without fi (#2701)" This reverts commit 285d4c85ff5c4e265f963208170304ef3461e684. * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Deref unique pointer for session_state * send shutdown event when dll is unloaded and EvaluationStop, SessionC… (#2704) * send shutdown event when dll is unloaded and EvaluationStop, SessionCreationStart Events. * Add EvalutationStart Event * add comment * use correct type for for loop (#2755) * ARM CI (#2759) * Set ARM agent pool * Set CMake generator to VS 2019 in ARM * Use system-wide CMake instead of custom version Our custom version is too old for VS 2019 * Use DML and build shared lib in ARM CI * Restore nuget packages in ARM CI * Disable DML * Refactor ARM debug/release builds * Use system packaged Python version * Remove hardcoded Python path * Downgrade Python to 3.7 for build * Remove explicit CMake path * Fix invalid JSON in cgmanifest.json (#2760) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * Remove usage of WHOLEARCHIVE in WinML CMake and add WinMLAdapterFactory (#2726) * Remove usage of WHOLEARCHIVE in WinMLAdapter CMake and add WinMLAdapterFactory * PR feedback, no need for dll(export) since using def file * PR comments * Small comment in gen_def.py * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * Move winml_provider_factory.h to proper location (#2801) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Filter CPU case for IsFloat16Supported (#2802) * Merge fixes * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * Fix warnings that cause build to fail * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * Make DML include path global (#2882) * Make DML include path global * Add generated cppwinrt headers to winml_lib_common * Integrate changes to WindowsAI to make ADO Build (#2886) * Revert "CMake cross-generator fixes (#2790)" This reverts commit dbe7d97fa1ab155f1309bced87199527e8f35bd2. * add additional suppress warning in onnx_proto * ignore /wd4996 warning * DML execution provider fixes * Revert "Revert "CMake cross-generator fixes (#2790)"" This reverts commit 1ae7b4bcbc02edc881ad28685da98e095dfceb17. * Update func signature of custom op function overloads * common devicehelpers fixes * Add pch.h for winml_lib_common * re-add winml_lib_common_dir/inc to include path for winml_adapter * User/orilevari/dml redist shared folder (#2890) * move dml nuget package directory up one level to make it shared between build flavors * Merge conflict fix * Revert "Merge conflict fix" This reverts commit 142fa72cf9ce4344ad717b50b7ea2b8582aadc7c. * Revert "Merge remote-tracking branch 'origin/master' into windowsai" This reverts commit 6e2126d46e5e5f564d65da37dd4f70c93dd81165, reversing changes made to b3f5583dc9249834b947c8ea905f6a98060d5bd6. * Make winml_test_common free of test macros (#2902) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Don't generate files outside the build root (#2914) * Don't generate files outside the build root * Add onnxruntime_EXTERNAL_DEPENDENCIES to WinML * Add DML depedency on RESTORE_PACKAGES * User/orilevari/fix yaml merge bugs (#2918) * Add winml test source parameter into cmake function (#2919) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Add arguments to unittest cmake functions * remove comment * Revert "Revert "Merge remote-tracking branch 'origin/master' into windowsai"" This reverts commit ade5abe72a4234fdbc3623093c61c02c6b0bdc26. * Fix breaks from merge with ORT master * Brianma/linux (#2917) * don't include windows.h in cross-plat header * add default case for switch statement * signed/unsigned mismatch fix Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> * User/sheilk/winml adapter c api (#2891) * Create winml adapter c api * fix build * make it build * move adapter into onnxruntime core/session * entry point not exported * minor changes * make model metadata work * make tests pass * implement all the model reflection apis on the adapter c abi * update the new ort interface to create a lotus ennvironment with a logging sink * start adding ort env * move all winml code into adapter folder/lib to isolate it * ensure a single logging manager at a time * start refactoring session * refactor session creation interface * add cpu and dml session option methods to adapter * finish session init * stub out interfaces in ort lib to perform similar mechanics of iinference session * enable profiling, and enable schema override * update session register graph transformers * turn back on custom registry for custom ops * Add sync api * add last c api stubs * should build... but all feature values are broken since this is in flight to moving all implementation details into ivalue * remove ep adapter header * Implement DML execution provider functions from adapter (#2846) * Implement DML execution provider functions from adapter * Use functions in OnnxruntimeEngine.cpp * make map/sequence type_infos freeable, and start implementing ivalue * make it build again * implement value methods * implement remaining methods * remove com adapter abi * check dml session * cache the allocator on ivalue * check if resource is cpu/gpu when access its mutable data * update tensor * mismatched parentheses * fix tensor base and binding obj * it evaluates tensors! sometimes... * minor fixes * enable gpu evals * wrapper all existing winml adapter apis with API_IMPL to try catch (#2854) * update winml... tensor strings are broken, need to template tensorbase to do different things for strings * make tensor strings work with 2 copies in/2 copies out * Fix tensor string and allocator bug * make maps work again... needs some fixes still * Make it build! * enable map inputs * map outputs * unbound outputs for sequences and maps * User/xianz/merge windowsai (#2883) * Packaging pipeline changes for VS 2019 (#2711) * Tiny fix to codegen * Simplify cache implementation and avoid static variables that may carry over between models * Extend DML kernels (#2641) * Additional DML operators * Check unsupported attributes and inputs * Address PR comments * Add kernel capability function used for partitioning, and re-enable stride-based int64 support based on value range * Fix test failures * Build fix * PR comments * Update Nuphar tutorial notebook (#2721) 1. Reflect int8 GEMV improvements for multi-threading from #2696 2. Add notes on multi-threading control using OpenMP 3. Add samples of running multi-isa AOT, and show int8 GEMM differences between AVX and AVX2 4. Add rnn_benchmark example to resolve #1993 * Add schema for new Qops (#2611) * Add schema for new Qops * adding shape inference + qlinearaveragepool * plus review comments * plus review comments * updates per review comments * plus review comments * [server] Add supposed for model_name and model_version as cli parameter (#2708) * remove 64bit warning message from python validation. (#2727) * MLAS: ARM64 build fix (#2734) fix bad usage of vreinterpret to cast vector element types * Fix broken python docs links (#2740) * Fix build on Mac OS (#2731) mac os ld doesn't support --while-archive, correct option is -all_load * fix ngraph wheel (#2737) * fix ngraph wheel 1.1.0 onnxruntime_ngraph wheel doesn't work * remove libdnnl.so in nGraph Libs * make it easy to compare * Split onnxruntime server to a separated folder (#2744) * Fix build for Python 3.8 (#2747) * Fix build for Python 3.8 * Update protobuf to 3.11.2 (#1928) Update protobuf to 3.11.2 (#1928) * Change default optimization level to All (from Basic) (#2745) * change default optimization level to All (from Basic) * fix test * fix c# test * Update numpy to 1.18 (#2758) * Update numpy to 1.18 * Pipeline changes for python 3.8 (#2753) 1. Pipeline changes for python 3.8 2. Fix a regression in setup.py which was just introduced in the previous commit. Please notice, we still haven't made python 3.8 + Windows + CUDA work. * Add basic stacktrace output for posix debug builds. (#2749) * [NupharEP] fix a race condition when multiple sessions running different models concurrently (#2772) * Revert "Change default optimization level to All (from Basic) (#2745)" This reverts commit 56bb503c2f26474b6613bcb2a198691a11dcef38. * Fix typo in error message (#2736) * Rename MKL-DNN to DNNL to fix broken link (#2730) * Fix nightly build version number issue * Pass BUILD_BUILDNUMBER to linux docker * Disable featurizers in python packages * Import more featurizers (#2781) Make kernels non-template. Add input constraint for learnt data. Add min_max_scalar_transformer, robust_scalar_transformer, inputation_marker_transfomer, label_encoder_transformer, missing_dummies_transformer along with tests. Advance Featurizers library commit. * Implement a more stable softmax (#2715) * Implement a more stable SoftMax e^x is represented as infinity if x is large enough, like 100.f. Infinity divided by Infinity is a NAN. Thus, softmax gets a NAN if one or more item are large enough. A math transform as below is leveraged to get a stable softmax: e^xi/(e^x1 + ...e^xn) = e^(xi - max) / (e^(x1 - max) + ... + e^(xn - max)) And for convenience, force max to 0.f if all xi are negative * Contributing: Fix a typo (#2784) * ACL EP GEMM improvements (#2780) When it is posible we use a fully connected layer instead of the gemm implementation. This will let the library use the best implementation based on the input data. * ACL EP convolution improvements (#2774) Added the optimized implementation for depthwise convolution for both ACL v19.02 and ACL 19.05. Also the pointwise convolution seems to be more optimal in the CPU implementation so we opted for that instead. * Add script for release Nuget validation (#2719) * Initial commit * Nits * Disable a test temporarily * Change working directory * Test * Add download python step * Test update * More changes * Fix space issue * Fix * Verify nuget signing * Fix * Spaces * PR feedback * Nit * Fix * Fix * Remove temporary changes * add uint8 support to where op (#2792) * Improve bert optimization script: (#2712) (1) Move input int64=>int32 conversion to embed layer fusion. (2) Output epsilon attribute for LayerNormalization fusion. * add session creation time cost. (#2798) * ML.NET team needs featurizers within a package (#2789) Add auto ml featurizers to Windows, MacOS as well as to GPU packaging-pipelines. * Initialize max of softmax with lowest of float (#2786) * MLAS: update SGEMM threading parameters (#2808) * add interface to copy batch tensors. (#2807) * add interface to copy batch tensors. * onnxruntime * speed up Windows TRT CI (#2811) * don't run cuda tests if building with tensorrt * remove unnecessary build options for win trt ci * refactor win gpu tensorrt ci yml * --numpy_version=1.17 * update * update * azcopy and cuda path * Update test data (#2356) * Add timeseries imputer transformer featurizer kernel (#2813) Make kernels non-template. Add input constraint for learnt data. Fixup tests. Add two more featurizers along with tests. Tests fail. min_max_scalar_transformer robust_scalar_transformer Fix tests serialized stream by prepending version bytes. Add inputation_marker_transfomer and the test. Fix up float/double type designations. Added label_encoder_transformer along with a test. string_throw case is broken at the momement. Fix labelencodertransfomer_test.cc string_throw case Rename maxabsscalertransformer_test.cc Add MissingDummiesTransformer along with the test. Update manifest. Add TimeSeriesImputerTransformer definition, implementation and tests * Fix memory leak in TRT (#2815) * fix memory leak issue * revert EP_FAIL on enueueV2 * Add manifest missing comma * Run static code analyzer on most of our code (#2817) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * update quantization doc (#2783) * update documentation for quantization script * plus some spell corrections * Filter CPU case for IsFloat16Supported (#2802) * update default optimization level + fix gemm_activation fusion (#2791) * update defualt optimization level + fix gemm_activation fusion * fix typo * add unit test and incorporate review comments * fix test comment * Fix dnnl wheel package name (#2823) * Append '-dnnl' to whl package name when --use_dnnl * Update build.py * Update Ubuntu & TensorRT version in README (#2820) Dockerfile.tensorrt is using nvcr.io/nvidia/tensorrt:19.09-py3 as base Image, update Ubuntu and TensorRT version according to https://docs.nvidia.com/deeplearning/sdk/tensorrt-container-release-notes/rel_19-09.html#rel_19-09 * Merge fixes * Add OneHotEncoder and HashOneHotEncoder kernels. (#2830) Add defs and imlementation for OneHotEncoders, adjuist date_time_transformer kernel and test. Add OneHotEncoder kernel test. Add HashOneHotVectorizerTransformer unit test. This does not link due to multiple definitions of functions that are included into header from a CPP file. * Upgrade gtest to the latest version (#2827) WinML would like to update the googletest submodule. They want some newer features (namely GTEST_SKIP to skip tests programmatically and be able to skip entire fixtures easily) and would need to update the submodule version. However, because the new version of code hit a bug in gcc, even though the bug is already fixed in the latest gcc but we're using gcc 4.8.x and it won't get patched for the bug, so we have to do a compromise, change our code a little bit to make it work. The gcc bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=51213 * Add support for int64_t for topk CPU. Fixes github issue #2806. (#2833) * Ignore allocator type in ExecutionProviders allocator map. Make default initialization of OrtMemoryInfo more clearly invalid. (#2768) * Remove allocator type from the key comparison in ExecutionProviders. Remove usage of DummyArena as it's no longer necessary. * Fix x86 tests where arena allocator is disabled. Make initialization of OrtMemoryInfo clearer by adding Invalid enum value. * Make OrtValueNameIdxMap::MaxIdx more intuitive. * Convert ExternalProject Featurizers into git submodule (#2834) Add git submodule for Featurizer library. Update cmake to build for git submodule. * add domain check for nodes + update documentation (#2831) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Fix Linux CUDA nuget packaging pipeline break * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * update optimization doc for BERT related fusions (#2819) * Add bert related transformers to doc * Add execution provider and comment for bert optimizations * Add comment about accuracy impact of approximation * Fix warnings that cause build to fail * MLAS: enable threading for quantized GEMMs (#2844) * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * remove old winmladapter cpp Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * move sequence implementation into ort lib... still commented out... need to turn back on... * begin sequence implementation * make maps and sequences work * fix broken tests * remove dead code * misc cleanup * CR feedback * User/xianz/winml adapter c api (#2869) * wrapper all existing winml adapter apis with API_IMPL to try catch * Return HR or Throw for WinML adapter APIs if failed * undo macro wrapper for two places * Wrap error macros around ort apis, too. * address CR feedback #2 * add more api throw/return macros * Revert changes no longer needed * revert changes to cxx api * format winml lib.ort and winml adapter * remove static pheonix singleton Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * missing use_dml check in winml_adapter_session (#2930) * --use_dnnl flag was mangled in merge (#2931) * use dml macro not wrapping custom registry code (#2934) * Disable LNK4199 winml_dll to enable cuda builds (#2936) * Disable LNK4199 in winml_dll * linkler->linker * LearningModelSessionAPITestGpu.CreateSessionWithCastToFloat16InModel should return DXGI_ERROR_UNSUPPORTED when FP16 not supported (#2937) * Disable LNK4199 in winml_dll * linkler->linker * Need to return DXGI_ERROR_UNSUPPORTED when Model does not support fp16 * Publish build symbols (#2939) * Publish build symbols * Don't upload PDBs for .exe files * Make x86 build (#2943) * fix last remaining size_t/int64_t warnings->errors (#2948) * TensorString, Sequences and Maps use the first allocator, but should use the cpu default allocator. (#2952) * fix tensor string allcoator * clean up default allocator usage for strings in winml lib/api.ort Co-authored-by: Ryan Lai <ryalai96@gmail.com> * Handle tensor shape of zero (#2954) Co-authored-by: Ryan Lai <ryalai96@gmail.com> * CR feedback (#2970) * CR feedback * fix weird formatting on privacy readme * Add 'All rights reserved.' everywhere * readd all rights reserved to winml_provider_factory.h * remove extra space in comment * remove extra whitespace * fixes post master merge * remove winml from nuget gpu pipeline * set IR VERSION on generated_model in rnn_benchmark (#2972) * Fix slice conformance failures (#2908) Co-authored-by: Adrian Tsai <adtsai@microsoft.com> Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Paul McDaniel <paul_mcdaniel@hotmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Dwayne Robinson <fdwr@hotmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com>
2020-02-05 01:12:19 +00:00
[threading(both)]
[marshaling_behavior(agile)]
[dualapipartition(1)]
runtimeclass TensorFloat : ITensor, [contract(MachineLearningContract, 2)]Windows.Foundation.IMemoryBuffer
{
//! Creates a tensor object without allocating a buffer.
static TensorFloat Create();
//! Creates a tensor object and allocates a buffer of size 'shape'.
static TensorFloat Create(Windows.Foundation.Collections.IIterable<Int64> shape);
//! Creates a tensor object, allocates a buffer of size 'shape', and copies all of 'data' into it.
static TensorFloat CreateFromArray(Windows.Foundation.Collections.IIterable<Int64> shape, Single[] data);
//! Creates a tensor object, allocates a buffer of size 'shape', and copies all of 'data' into it.
static TensorFloat CreateFromIterable(Windows.Foundation.Collections.IIterable<Int64> shape, Windows.Foundation.Collections.IIterable<Single> data);
//! Returns a read only view of the data.
Windows.Foundation.Collections.IVectorView <Single> GetAsVectorView();
[contract(MachineLearningContract, 2)]
{
//! Creates a tensor object, allocates a buffer of size 'shape', and copies all of 'data' into it.
static TensorFloat CreateFromShapeArrayAndDataArray(Int64[] shape, Single[] data);
//! Creates a tensor object of size 'shape', and uses the data in 'buffer'.
static TensorFloat CreateFromBuffer(Int64[] shape, Windows.Storage.Streams.IBuffer buffer);
}
}
[contract(MachineLearningContract, 1)]
INBOX_ONLY([static_name("Windows.AI.MachineLearning.ITensorFloat16BitStatics", a52db6f5-318a-44d4-820b-0cdc7054a84a)])
INBOX_ONLY([interface_name("Windows.AI.MachineLearning.ITensorFloat16Bit", 0ab994fc-5b89-4c3c-b5e4-5282a5316c0a)])
Merge windowsai (winml layering) into master (#2956) * Initial Commit * Merged PR 3985217: add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc (#2346) add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc and violating our OS layering requirements. We linked against onecoreuap_apiset.lib in VB so we will continue doing this, but I am still unsure why not to link against onecore instead since that is where we ship. However, since Sheil is the owner of this code we will wait to discuss with him before changing anything. * Initial changes for layering * more snipping to get core into ort * update build instructions to include --build_shared_lib (#2358) * update build instructions to include --build_shared_lib * fix line breaks * Task 23998197: add winml_lib_core into onnnxruntime.dll (#2368) * Task 23998197: add winml_lib_core into onnnxruntime.dll * PR feedback build break on perf_test * return proper error when the model path isn't found (#2391) * LearningModelSession is cleaned up to use the adapter, and parts of b… (#2382) this is a big PR. we are going to move it up to layer_dev , which is still a L3 so we are still safe to do work there agile. we are going to move this into the L3 so that ryan can start doing intergration testing. we will pause for a full code review and integration test result prior to going into the L2. >>>> raw comments from previous commits >>> * LearningModelSession is cleaned up to use the adapter, and parts of binding are. * moved everything in the winmladapter made it all nano-com using, WRL to construct objects in the ORT side. base interfaces for everythign for winml to call cleaned up a bunch of winml to use the base interfaces. * more pieces * GetData across the abi. * renamed some namepsace cleaned up OrtValue cleaned up Tensor cleaned up custom ops. everything *but* learnignmodel should be clean * make sure it's building. winml.dll is still a monolith. * model moved over. everything builds clean. step ! * weak ref comment * Layer dev paulm (#2408) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * Layer dev paulm (#2414) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * Layer dev paulm (#2423) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * Layer dev paulm (#2424) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * Layer dev paulm (#2425) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Layer dev paulm (#2426) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * fixed map and sequence when passing stl types across the ABI . found a leak in nvidia driver, but skipped it. all winmlapitests pass now * Moved SessionOptions over to the abi * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * Register WinML TraceLogging provider on Onnxruntime.dll (#2455) * Register WinML TraceLogging provider on Onnxruntime.dll * Add ifdef to make sure trace logging provider has telemetry option when LAYERING_DONE * No need for ifdef for TraceLoggingOptionMicrosoftTelemetry * PR feedback * Move etw registration into lotus environment constructor and deresgister in lotus environment destructor * Brianma/cpuwinml (#2466) * allow building winml cpu without dml. * Brianma/breaks (#2469) * fix some more breaks * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * move dml checks out of winml and into the adapter * better error handling * Brianma/fi (#2470) * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * fix bad merge * Checking in a staging checkpoint point so that Ryan can work with me in parrallel * build break. * Brianma/testfails (#2473) * add missing ir version to dictvectorizer-string.onnx * add missing ir version to relu.onnx * add missing ir version to zipmap*onnx * add IR version to manually generated models * remove an unnecessary ifdef dml * Brianma/windowsai fi (#2475) * update dockerfiles/README (#2336) * Make elementwise op run 4 items per thread (#2335) Description: Describe your changes. Make elementwise op run 4 items per thread unroll for loop to leverage ILP remove unnessary N==0 check inside elementwise GPU kernel Motivation and Context Why is this change required? What problem does it solve? It can improve the performance of GPU elementwise ops. ~2% performance gain on popular NLP bert model. If it fixes an open issue, please link to the issue here. * Add CUDA GatherElements kernel (#2310) * Updates * Update test * Update * Updates * nits * PR feedback * Update * Update * PR feedback * PR comments * Update * Fix build * Fix build * Nits * Fix * Layer Normalization Fusion (#2319) basic layer normalization transform * Add FastGelu Cuda Op for Gelu and Add bias fusion (#2293) * Add FastGelu cuda op * Add AddBiasGelu for experiment * Revert "Add AddBiasGelu for experiment" This reverts commit 5c1ee019858c657e6bb75887265cb85675626e5b. * Add bias * Add unit tests * update comment * update script * fix build error * update coding style * update for CR feedback Enable half2 optimization only when cuda arch >= 7.0 * move _Tanh to common.cuh * implement CPU contrib OP Attention (#2333) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanUnusedInitializers. (#2320) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanupUnusedInitializers. This means initializers that have been replaced during graph optimizations are not left in the GraphProto when we save an optimized model. * Handle edge case where a model has an unused initializer with matching graph input by also removing the graph input. * Use non-const iterators in std::find_if calls to make centos build happy. * Nuget pipeline changes (#2305) 1. refactor the pipeline, remove some duplicated code 2. Move Windows_py_GPU_Wheels job to Win-GPU-CUDA10. We'll deprecated the "Win-GPU" pool 3. Delete cpu-nocontribops-esrp-pipeline.yml and cpu-nocontribops-pipeline.yml 4. In Linux nuget jobs, run "make install" before creating the package. So that extra RPAH info will be removed * Cuda Reverse Sequence Op, maping types of same size using same template function. (#2281) * Set ElementType to String type of node metadata, instead of byte[] (#2348) * Set ElementType to String type of node metadata, instead of byte[] * Fix spacing * Introduce PrimitiveType into a Type System along with an integer constant (#2307) Improve perf by avoiding GetType<T>() calls. Introduce MLTypeCallDispatcher to switch on Input Type. Add Tensor IsType<T>() fast method. * Fix/test dim value of 0 handling in a couple of places (#2337) * Update the CUDA Where implementation broadcasting logic to handle a dim with value of 0. Add unit test Also add unit test for unary op with dim value of 0 * Exclude ngraph from Where test with 0 dim. * Openvino EP R3.1 onnxrt server (#2357) * onnxrt server with OVEP * onnxrt server with OVEP * Update Dockerfile.server.openvino * onnxrt server OVEP fix reviews * onnxrt server OVEP fix reviews * Implement cuda nonzero op. (#2056) Implement cuda nonzero op. * Direct use python numpy array's memory if already contiguous. (#2355) * Direct use python numpy array's memory if already contiguous. This could greatly improve performance for session with large input, like big image 1920x1080 fastrcnn, 30~40% speed up could be achieved. * Add test case enforce contiguous/non-contiguos numpy array as inputs. * Add helper to create output to minimize binary size. (#2365) Add ConstEigenTensorMap typedef so we don't unnecessarily const_cast the const input Tensor. * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS (#2369) * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS * update * Add Tracelogging for profiling (#1639) Enabled only if onnxruntime_ENABLE_INSTRUMENT is ON * test bidaf with nuphar for avx target (#2370) increase nuphar test coverage a bit * Fix a bug in TLS refcount that may destabilized CUDA CI (#2374) * update output size calculation for resize (#2366) * change how output size is calculated for resize op * add tests for ver 10 resize * Extend OneHot CPU kernel to support more types (#2311) * Extend OneHot CPU kernel to support input int64_t, depth int32_t, output float * Skip BERT before the test data fix is picked up * Fix bug with Slice. Need to pass in flattened input dimensions so the initial offset into the input is calculated correctly. (#2372) * Add opset 11 version of Split to CUDA ops (#2376) Organize the CUDA ops definitions so all the opset 10 and 11 parts are together (same setup used for CPU ops) * Layer Norm Fusion Fix (#2379) * layer norm fusion fix * Add input shape check in code and unit tests * Fuse Add + Gelu (#2360) Implement the transformer to fuse add + gelu Implement the accurate kernel * Skip layer norm transform (#2350) * skip layer normalization transformer * Another try to stabilize CUDA CI (#2383) The root cause seems to be failure in CUDA dealloc when tear down. cudaFree return code was ignored before, so should the debug check. * fix BUILD.md typo (#2375) build.py: error: argument --config: invalid choice: 'RelWithDebugInfo' (choose from 'Debug', 'MinSizeRel', 'Release', 'RelWithDebInfo') * Fixed compilation with ngraph (#2388) * Fix reuse logic in allocation planner. (#2393) * Fix reuse logic in allocation planner. * PR comments * Add helpful comments * Don't allow reuse across string tensors. * [NupharEP] Multiple optimizations (#2380) Fuse transpose into MatMul Implement Pow and constant scalar simplification Vectorize ReduceMean Improve symbolic shape inference Minor updates for better debugging in fused function name * Avoid using the default logger in the graph lib and optimizers (#2361) 1. Use the session logger if it is available. 2. Don't disable warning 4100 globally. We should fix the warnings instead of disabling it. * Change CUDA implementation of Transpose to support all fixed size tensor types (#2387) * Change CUDA implementation of Transpose to not use a typed kernel so we can support more types with minimum binary size. Add support for 8, 16, 32 and 64 bit types. Add unit tests. Add method so the implementation can be called directly (will be used by CUDA Scan very soon). * Disable TensorRT for MLFloat16 and int8 unit tests. * Address PR comment and add support for calling cublas implementation if type is mlfloat16. * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. (#2398) * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. * [NupharEP] force some low/zero cost ops to be inlined (#2409) * fix cross compile bug (#2415) * Minor optimization: if a node has already been placed, there's no need to find a kernel for it. (#2417) * Add Reshape Fusion (#2395) * Add reshape fusion * Add some comments * update comments * update comment format * update according to feedback * update for recent logger change * fix build error * (1) Support both input and output edges in find path in graphutils (2) Add a test case of only one constant initializer of Concat input. (3) Refactor ReshapeFusion class to allow add more subgraph fusion in the future. * fix error * (1) loose constraint on initializer: non constant is allowed for reshape fusion. (2) Change versions type to vector. (3) Add logging. (4) Return false when multiple output edges matched in FindPath. Add comments. * only allow one direction (input or output) in FindPath * [NupharEP] Update notebook and docker image (#2416) Add BERT squad in Nuphar tutorial Enhance speed comparsion readability * Fix the issue in matmul_add_fusion (#2407) Fix the issue in matmul_add_fusion If Muatmul + Add has shape [K] * [K, N], reset it to [1, K] * [K, N] will make the output shape to [1, N] will also requires a reshape on the output. Fix: just remove the shape reset to not fuse it. Add a negative test case for matmul+add fusion * feat(treeregressor): Update TreeEnsembleRegressor for type support (#2389) Updates the `TreeEnsembleRegressor` to allow for `double`, `float`, `int64`, and `int32` inputs to match the upstream specification. Signed-off-by: Nick Groszewski <nicholas.groszewski@capitalone.com> * onnxrt server documentation update (#2396) * Added support for Pad-2 operator in OpenVINO-EP (#2405) * Add CUDA If operator. (#2377) * Add CUDA If operator. Uses CPU operator for implementation. By adding a CUDA version the inputs/outputs (with the exception of the 'cond' input) stay on GPU, and no other logic is required to avoid a copy to CPU across the control flow node. * Improved documentation for onnxruntime::utils::SwapByteOrderCopy(), added precondition check. * Fix the type constraints on CUDA If operator to exclude strings. (#2431) * add Im2col<uint8_t> (#2438) * Adjust codegen vectorization width from target (#2439) * Adjust codegen vectorization width from target * Add CUDA Scan operator. (#2403) * Add Scan CUDA op. Uses CPU implementation for logic. Added some device specific functors for handling when data needs to be manipulated on a different device. Added ability to override the materialization logic in the OrtValue slicer so DML can plugin their handling. * Fix Windows GPU C API packaging pipeline failure (#2440) Fix Windows GPU C API packaging pipeline failure (#2440) * Correctly handle implicit inputs for fused nodes (#2390) * Correctly handle implicit inputs for fused nodes Previously, nuphar's partitioning function didn't include node's implicit inputs into the inputs list of MetaDef, and hence a crash was triggered in the onnx graph checker. This commit fixed the issue. Furthermore, it also fixed a related issue where we didn't add implicit inputs into graph_inputs_excluding_initializers_ in Graph::SetGraphInputsOutputs. the issue was that graph_inputs_including_initializers_ populated by SetInputs (e.g. called by FunctionImpl::FunctionImpl) may contain implicit inputs which were not of any node's initializers in the graph. Because they were not part of any initializers, these implicit inputs couldn't be visited by going through all nodes' inputs. Consequently, they would *not* be added into graph_inputs_excluding_initializers_. We fixed the issue by first copying the populated graph_inputs_including_initializers_ into graph_inputs_excluding_initalizers_, which then had both initializers and non-initializers as its initial content. Later, we erase initializers from the list. In this way, we can ensure all implicit inputs to remain in graph_inputs_excluding_initializers_. * refined comments and fixed duplicates Address CR by revisiting comments in terms of implicit inputs Also fixed an issue by skipping duplicates while copying inputs from graph_inputs_including_initializers_. * address CR explain why we need to collect nodes' implicit inputs * don't rely on pointer values for iterating std::set Previously, openvino relied on iterating a set of NodeArg pointers to construct inputs and outputs for a fused graph. It could cause non-determinism. The reason was that although iterating std::set by itself is stable, pointer values of NodeArgs may vary. Consequently, we could end up visiting the set's elements in different orders for different runs for the same test, which resulted in constructing inputs (and outputs) with different orders to the fused graph. For example, for the same test, we may have inputs [A, B] in some runs but inputs[B, A] in others. Let's use std::string as the key type to avoid such nondeterminism. This commit also added implicit inputs into meta->inputs while returning the capability from the openvino provider. * Fixed another latent issue in openvino's GetCapability function The issue was that we couldn't simply erase fused_inputs and fused_outputs while iterating the nodes. For example, an output NodeArg may have multiple uses, and it's wrong if we erase it from fused_outputs when we encounter only one of its uses as input. * Remove DeviceAllocatorRegistry class (#2451) Remove DeviceAllocatorRegistry class * CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library. * Parallel Gelu with ParallelFor (#2399) Parallel Gelu to get better performance for Gelu * Clean up build.py (#2446) * Pull the latest image before running docker build * Fuse SkipLayerNorm with Bias (#2453) Fuse SkipLayerNorm with Bias * Allow more than one invocation of CreateEnv in the same process. (#2467) * Allow more than one invocation of CreateEnv in the same process. * Fix centos build * Symbolic shape inference improvements: (#2460) * Symbolic shape inference improvements: - add a mode to guess unknown ops' output rank - add support for GatherND - add support for If - fix a bug in get_int_values when then tensor rank > 1D, by treating it as no sympy data - add symbol to literal merge when ONNX silently merges dims - fix a bug in Concat when input dim is 0 - fix a bug in ConstantOfShape that computed dim is not updated - add support for dynamic shape in ConstantOfShape - fix a bug in Loop output shape that loop iterator dim is not inserted at dim 0 - add support for dynamic padding in Pad - add support for dynamic shape in Reshape - add support for Resize with opset > 10, by treating output dims as dynamic - fix a bug in Slice when starts/ends are dynamic - restrict input model to opset 7 and above - make output model optional to avoid disk write when testing Run model tests for symbolic shape inference Reduce 2GB docker image size of nuphar * add additional test data set for nuget pipeline (#2448) * add SAS token to download internal test data for nuget pipeline * update azure endpoint * fix keyvault download step * fix variable declaration for secret group * fix indentation * fix yaml syntax for variables * fix setting secrets for script * fix env synctax * Fix macos pipeline * attempt to add secrets to windows download data * fix mac and win data download * fix windows data download * update test data set url and location * Revert "Brianma/windowsai fi (#2475)" This reverts commit 5780b864a15513fda4eadbfc2b5345fefe70b5ec. * Add scenario tests (#2457) * Add scenario tests * Remove TODO from model license * Add winml_api test dependency * fix model load test. fi from master changed the constructor (#2483) * make api tests all pass (#2486) * fix bad merge * fix bad model merge * Layer dev paulm (#2492) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * Rename ambiguous header (#2489) * fix one more missing IR version model (#2500) * add missing IR version to 4 more models used by scenario tests (#2501) * Add CLI parameters to test runner, build WinML in ARM and x86 CI (#2479) * Support test parameters through CLI arguments * Add WinML do Windows x86/ARM CI builds * Code style fixes * Update googletest Remove GPUTEST macros everywhere now that GTEST_SKIP is supported * Refactor main.cpp * Build scenario tests without DML * Link scenario tests to DML when it's enabled (#2502) * Layer dev release pipeline (#2488) Adds winml binaries to existing cpu nuget package, and creates new gpu dml nuget package with winml binaries and DML EP. * Layer dev paulm (#2506) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * Remove usage of IOBinding in WinML and use C_API Run method (#2504) * remove usage of iobinding * Change data structure to use vector of Ort::Values * Polish bind input / output * Use C APIrun method * Update providers on evaluate getresults * Remove run and IObinding interface from WinMLAdapter * Remove use of IObinding * bind unbound outputs code moved to learningmodelbinding * clean up unneeded istensor adapter function * Fix comment * Check if session is closed before binding and clearing * PR feedback * Layer dev paulm (#2507) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * Make tests dependend on winml_dll (#2509) * add dml binaries to DirectML package and be more explicit about condition variables (#2520) * re-enable warnings for winml builds and fix the warnings that were hiding (#2526) * turn devmode back on for winml builds * fix some warnings. include protobuf in a way that disables some warnings * undo protobufhelpers changes and just ignore 4100 errors in pb code * attempt to isolate protobufhelpers errors * add template specialization for getting tensor proto data * Layer dev paulm (#2533) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * Spawn child process to run DeviceLostRecovery scenario test (#2530) * Spawn child process to run DeviceLostRecovery scenario test * Layer dev paulm (#2536) ori said yes * add missing namespace to winml_trace_logging_provider in lotusenvironment.h (#2542) * Handle exception thrown from all apis in WinMLAdapter (#2539) * various changes to unblock windowsai ADO build * Fix custom ops scenario tests (#2562) * Do not shutdown protobuf after ort environment gets destroyed. Lazy load lotus environment first time it is needed * comment typo * pr comment about calling phoenix singleton * Make lotus_environment static in winmladapter * Layer dev paulm (#2567) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * making windowsAI azure dev ops work. * code review comments. * revert changes * Cmake and preprocessor fixes that where uncovered by building on agents without DML available via SDK * Layer dev dml delayload (#2580) * Brianma/cpu (#2583) * don't include dml stuff in cpu builds * tests that link the image lib also need the telemetry lib now * Throw Winml_err_invalid_binding if binding gpu resource on cpu device (#2589) * Throw Winml_err_invalid_binding if binding gpu resource on cpu device * PR comments. No need to query executionprovider for is gpu device * User/xianz/ortthrow (#2596) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * User/xianz/ortthrow (#2599) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * add status error message * Remove uwp onsuspending winrt call because logruntimeperf is getting removed (#2630) * User/xianz/dedup telemetry (#2631) * investigate duplication of telemetry in winml and ort * remove winml telemetry events * telemetry executionProviderEvent * remove unneccessary file and refactor code little bit * Revert back TelemetryEvent, which send up ETW event. * merge changes from layer_dev to windowsai (#2638) * Remove underscore from googletest names (#2616) * Fix leaking memory allocator Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24278761 and https://microsoft.visualstudio.com/OS/_workitems/edit/24330198 * Explicitly initialize Ort::Value with nullptr * Cache WinML adapter * bad merge * define private version of dxcore enum that is added in 19H1 SDK. (#2654) * add comment for explaning private definition of dxcore d3d feature level ennum value. (#2672) * do not package directml.pdb for redist packages. (#2676) * Fix leaking operator registry (#2645) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24354916 * User/orilevari/windowsai master merge (#2674) merge resolutions included pulling in telemetry logic that was merged to master and not windowsai and dereferencing InferenceSession::sessionstate now that it is a unique pointer * Delete Ort Allocator in LearningModelBinding (#2653) * Delete OrtAllocator in LearningModelBinding * PR comments to make Ort::Allocator a smart pointer * Small comment change * PR feedback to clean up code * PR feedback on move semantics * Clean up std::move * Fix memory leaks (#2679) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24356109, https://microsoft.visualstudio.com/OS/_workitems/edit/24388361 and https://microsoft.visualstudio.com/OS/_workitems/edit/24388596 * various changes to properly organize and skip GPU tests. For now for No DML builds we will not run GPU tests at all. In the future we should adapt the tests to expect the appropiate errors. (#2695) * Windowsai without fi (#2701) * Disable Attention fusion tests when DISABLE_CONTRIB_OPS is defined (#2529) * Setup java ci (#2528) * Add provision in ORT for session options to be parsed when available via model file (#2449) * Initial commit * Fix gitmodules * Nits * Nits * Updates * Update * More changes * Updates * Update * Some updates * More changes * Update * Update * Merge * Update * Updates * More changes * Update * Fix nits * Updates * Fix warning * Fix build * Add comment * PR feedback * PR feedback * Updates * Updates * Update * More changes * Fix build break * Comment test for now * Updates * Updates * PR feedback * Updates * Nits * Add tests * Fix build * Fix build * Fix build * Fix build break * Fix build * Nits * PR feedback * More change * Expose GetSessionOptions in pybind logic and add unit test for python * Fix build * PR feedback * PR feedback * Revert "Disable thread pool creation when enabled OpenMP (#2485)" (#2535) This reverts commit 7c7d5a149c9ed52eec67304bae5c4b132166a8a1. * Add dynamic shape support in TensorRT execution provider (#2450) * remove onnx-tensorrt submodule * add new onnx-tensorrt submodule (experiment) for trt6 * update engine build for trt6 * update compile and compute for tensorrt6.0 * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * switch to onnx-tensorrt master for TensorRT6' * Update tensorrt_execution_provider.cc * Handle dynamic batch size and add memcpy in TensorRT EP * update test cases * Update tensorrt_execution_provider.cc * update onnx-tensorrt submodule * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.ubuntu_tensorrt * Update run_dockerbuild.sh * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update concat_op_test.cc * Update tensorrt_execution_provider.cc * Upgrade TensorRT to version 6.0.1.5 * Update onnxruntime_providers.cmake * Update CMakeLists.txt * Update reduction_ops_test.cc * Update install_ubuntu.sh * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.tensorrt * Update BUILD.md * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update onnxruntime_providers.cmake * Update install_ubuntu.sh * Update install_ubuntu.sh * Update gemm_test.cc * Update gather_op_test.cc * Update CMakeLists.txt * Removed submodule * update onnx-tensorrt submodule * update header file * Removed submodule * add submodule onnx-tensorrt kevin's branch shape-test' * add debugging code * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * merge master * Removed submodule * update onnx-tensorrt submodule * add more changes for dynamic shapes * Update tensorrt_execution_provider.cc * update for dynamic shape * update dynamic shape processing * fix logger issue * remove submodule onnx-tensorrt * add submodule onnx-tensorrt * add env variable min_subgraph_size * remove redundency * update document * use onnxruntime::make_unique * fix multi-run issue * remove some tests to save CI build time * Add dynamic shape test * Update TensorRT-ExecutionProvider.md * Add example of running Faster R-CNN model on TensorRT EP * Add more details on env variables * update environment variables * Update tensorrt_basic_test.cc * Update model tests * Update tensor_op_test.cc * remove --use_full_protobuf * Update build.py * User/xianz/telemetry (#2458) * enabme telemetry * enable telemetry * set enable telemetry as default * for debugging * remove log and set disable telemetry as default back * delete private file while testing * resolve comment: mainly add license header, rename macro and update docs * rewording in privacy.md * Fix integer overflow in cuda NonMaxSuppression implementation (#2540) * add test case that should pass but fail * fix nms * extract int_max_output_boxes_per_class * Introduce container type runtime checks and other improvements (#2522) Rework TensorSeq in a manner consistent with Tensor and SparseTensor in terms of type system setup. Reduce templating. Introduce helpers to ensure the same data type. Make OrtValue __dtor not virtual. Introduce ContainerChecker * Fix C API tests for centos and mac (#2544) * change c++14 to c++11 * add ld lib path for centos * enable csharp tests on macos * fix C API test on MacOS + fix manylinux dotnet install * fix manylinux dotnet install * fix lib link * Add back executable bit to build.py * Fix a bug handling negative begin pad values in Pad op (#2550) * Fix bug in Pad op * Update * DNNL CMAKE update (#2548) * Fix android build (#2558) * Update win-x86-ci.yml (#2557) Fix build pipeline break * Re-enable Windows C# tests (#2564) * disable onnx_test_runner -x invocations for dnnl (#2568) * Allow sequence length to be symbolic (#2559) * setup java ci mac (#2570) * make layernorm fusion to support opset 11 (#2545) * Fix a warning found in the latest VS release * Add more check on SkipLayerNorm and BiasGelu fusion (#2574) * Fix file not found error during docker build. (#2569) * Add ConvTranspose1D (#2578) * Ryanunderhill/packagename test (#2582) * [Nuphar EP] fixes for some object detection models (#2581) Update notebook tutorial with multi-threaded int8 GEMM from #2517 * EmbedLayerNormalization Fusion Improvement (#2553) Embedding layer norm fusion improvements - add more checks * Update version (#2584) * Temporarily exclude vgg19 test from Python backend test 1. temporarily exclude vgg19 test which comsumes too much memory, run out of memory on Upsquared device. Single test pass for vgg19, need furture investigation (#2588) 2. Update docker file to decrease the docker image size * Update docs for Android NNAPI EP (#2586) * Fix lto bug for protobuf and ubuntu * add path to build dir before test run (#2590) * Add missig env variables for mac pipeline test (#2595) * Fixed an issue in updating realized dims (#2597) when we update realized dims for scan's output, the sliced axis also needs to be inclusive, i.e. we should check with "dim >= insert_inclusive_axis", because the offset in the symbols are based on Scan sugraph. Otherwise, we would end up with shape mismatch later. * Java API for onnxruntime (#2215) * Add support for opset 11 in reshape fusion (#2592) Support opset verion 11 in reshape fusion * Rename automl python tools folder to featurizer_ops. (#2593) * Support opset 11 subgraph of Squad model in Embed Layer Normalization (#2605) Support opset 11 Squad model that is exported from PyTorch nightly. The embed layer uses Range op which is missed in the transformer. * symbolic shape inference: fix warnings in GPT-2 model (#2608) And revise nuphar perf test on BERT squad * Dump subgraph ID and fused graph ID (#2607) * Dump subgraph ID and fused graph ID Dump subgraph ID and fused graph ID for better debugging * Remove local static fused_count added a field global_fused_count_ to NupharExecutionProvider class * EmbedLayerNormalization Fusion For Dynamic Squad Model Opset 10 (#2613) Support subgraph of SQuAD model exported from pytorch with dynamic input axes * Allow providers to be set for InferenceSession at construction (#2606) * Remove unnecessary parameter in some places in GatherElements implementation (#2612) * Remove unnecessary parameter in some places * Update * Update * Make sure fenced tensor could not reuse other tensor. (#2561) Fix random error caused by this. * Improve Embed Layer Norm Fusion for SQuAD with static input shape (#2621) * fix float16 comparison in initializer (#2629) * epsilon attribute for layernormalization fusion (#2639) * removed unnecessary batch file and fix path (#2640) * Add shape inference to ConvTransposeWithDynamicPads schema (#2632) * Improve cuda expand() opeator's performance. (#2624) * Cuda pad optimize when no padding is needed. (#2625) * Shortcut cuda Pad() when no padding is needed. * Optimize cuda scatter() on 2D compatible. (#2628) * Optimize cuda scatter() on 2D compatible. * Add some comments. * fix build error for ARM (#2648) * Improve performance of resize() in Nearest mode (#2626) Special treatment for 2D, check same size as input image. And in 2d kernel, template use_expolation. * Fix memory exception in Layer Norm Fusion (#2644) * Windows CI changes(#2650) * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Revert "Windowsai without fi (#2701)" This reverts commit 285d4c85ff5c4e265f963208170304ef3461e684. * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Deref unique pointer for session_state * send shutdown event when dll is unloaded and EvaluationStop, SessionC… (#2704) * send shutdown event when dll is unloaded and EvaluationStop, SessionCreationStart Events. * Add EvalutationStart Event * add comment * use correct type for for loop (#2755) * ARM CI (#2759) * Set ARM agent pool * Set CMake generator to VS 2019 in ARM * Use system-wide CMake instead of custom version Our custom version is too old for VS 2019 * Use DML and build shared lib in ARM CI * Restore nuget packages in ARM CI * Disable DML * Refactor ARM debug/release builds * Use system packaged Python version * Remove hardcoded Python path * Downgrade Python to 3.7 for build * Remove explicit CMake path * Fix invalid JSON in cgmanifest.json (#2760) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * Remove usage of WHOLEARCHIVE in WinML CMake and add WinMLAdapterFactory (#2726) * Remove usage of WHOLEARCHIVE in WinMLAdapter CMake and add WinMLAdapterFactory * PR feedback, no need for dll(export) since using def file * PR comments * Small comment in gen_def.py * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * Move winml_provider_factory.h to proper location (#2801) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Filter CPU case for IsFloat16Supported (#2802) * Merge fixes * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * Fix warnings that cause build to fail * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * Make DML include path global (#2882) * Make DML include path global * Add generated cppwinrt headers to winml_lib_common * Integrate changes to WindowsAI to make ADO Build (#2886) * Revert "CMake cross-generator fixes (#2790)" This reverts commit dbe7d97fa1ab155f1309bced87199527e8f35bd2. * add additional suppress warning in onnx_proto * ignore /wd4996 warning * DML execution provider fixes * Revert "Revert "CMake cross-generator fixes (#2790)"" This reverts commit 1ae7b4bcbc02edc881ad28685da98e095dfceb17. * Update func signature of custom op function overloads * common devicehelpers fixes * Add pch.h for winml_lib_common * re-add winml_lib_common_dir/inc to include path for winml_adapter * User/orilevari/dml redist shared folder (#2890) * move dml nuget package directory up one level to make it shared between build flavors * Merge conflict fix * Revert "Merge conflict fix" This reverts commit 142fa72cf9ce4344ad717b50b7ea2b8582aadc7c. * Revert "Merge remote-tracking branch 'origin/master' into windowsai" This reverts commit 6e2126d46e5e5f564d65da37dd4f70c93dd81165, reversing changes made to b3f5583dc9249834b947c8ea905f6a98060d5bd6. * Make winml_test_common free of test macros (#2902) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Don't generate files outside the build root (#2914) * Don't generate files outside the build root * Add onnxruntime_EXTERNAL_DEPENDENCIES to WinML * Add DML depedency on RESTORE_PACKAGES * User/orilevari/fix yaml merge bugs (#2918) * Add winml test source parameter into cmake function (#2919) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Add arguments to unittest cmake functions * remove comment * Revert "Revert "Merge remote-tracking branch 'origin/master' into windowsai"" This reverts commit ade5abe72a4234fdbc3623093c61c02c6b0bdc26. * Fix breaks from merge with ORT master * Brianma/linux (#2917) * don't include windows.h in cross-plat header * add default case for switch statement * signed/unsigned mismatch fix Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> * User/sheilk/winml adapter c api (#2891) * Create winml adapter c api * fix build * make it build * move adapter into onnxruntime core/session * entry point not exported * minor changes * make model metadata work * make tests pass * implement all the model reflection apis on the adapter c abi * update the new ort interface to create a lotus ennvironment with a logging sink * start adding ort env * move all winml code into adapter folder/lib to isolate it * ensure a single logging manager at a time * start refactoring session * refactor session creation interface * add cpu and dml session option methods to adapter * finish session init * stub out interfaces in ort lib to perform similar mechanics of iinference session * enable profiling, and enable schema override * update session register graph transformers * turn back on custom registry for custom ops * Add sync api * add last c api stubs * should build... but all feature values are broken since this is in flight to moving all implementation details into ivalue * remove ep adapter header * Implement DML execution provider functions from adapter (#2846) * Implement DML execution provider functions from adapter * Use functions in OnnxruntimeEngine.cpp * make map/sequence type_infos freeable, and start implementing ivalue * make it build again * implement value methods * implement remaining methods * remove com adapter abi * check dml session * cache the allocator on ivalue * check if resource is cpu/gpu when access its mutable data * update tensor * mismatched parentheses * fix tensor base and binding obj * it evaluates tensors! sometimes... * minor fixes * enable gpu evals * wrapper all existing winml adapter apis with API_IMPL to try catch (#2854) * update winml... tensor strings are broken, need to template tensorbase to do different things for strings * make tensor strings work with 2 copies in/2 copies out * Fix tensor string and allocator bug * make maps work again... needs some fixes still * Make it build! * enable map inputs * map outputs * unbound outputs for sequences and maps * User/xianz/merge windowsai (#2883) * Packaging pipeline changes for VS 2019 (#2711) * Tiny fix to codegen * Simplify cache implementation and avoid static variables that may carry over between models * Extend DML kernels (#2641) * Additional DML operators * Check unsupported attributes and inputs * Address PR comments * Add kernel capability function used for partitioning, and re-enable stride-based int64 support based on value range * Fix test failures * Build fix * PR comments * Update Nuphar tutorial notebook (#2721) 1. Reflect int8 GEMV improvements for multi-threading from #2696 2. Add notes on multi-threading control using OpenMP 3. Add samples of running multi-isa AOT, and show int8 GEMM differences between AVX and AVX2 4. Add rnn_benchmark example to resolve #1993 * Add schema for new Qops (#2611) * Add schema for new Qops * adding shape inference + qlinearaveragepool * plus review comments * plus review comments * updates per review comments * plus review comments * [server] Add supposed for model_name and model_version as cli parameter (#2708) * remove 64bit warning message from python validation. (#2727) * MLAS: ARM64 build fix (#2734) fix bad usage of vreinterpret to cast vector element types * Fix broken python docs links (#2740) * Fix build on Mac OS (#2731) mac os ld doesn't support --while-archive, correct option is -all_load * fix ngraph wheel (#2737) * fix ngraph wheel 1.1.0 onnxruntime_ngraph wheel doesn't work * remove libdnnl.so in nGraph Libs * make it easy to compare * Split onnxruntime server to a separated folder (#2744) * Fix build for Python 3.8 (#2747) * Fix build for Python 3.8 * Update protobuf to 3.11.2 (#1928) Update protobuf to 3.11.2 (#1928) * Change default optimization level to All (from Basic) (#2745) * change default optimization level to All (from Basic) * fix test * fix c# test * Update numpy to 1.18 (#2758) * Update numpy to 1.18 * Pipeline changes for python 3.8 (#2753) 1. Pipeline changes for python 3.8 2. Fix a regression in setup.py which was just introduced in the previous commit. Please notice, we still haven't made python 3.8 + Windows + CUDA work. * Add basic stacktrace output for posix debug builds. (#2749) * [NupharEP] fix a race condition when multiple sessions running different models concurrently (#2772) * Revert "Change default optimization level to All (from Basic) (#2745)" This reverts commit 56bb503c2f26474b6613bcb2a198691a11dcef38. * Fix typo in error message (#2736) * Rename MKL-DNN to DNNL to fix broken link (#2730) * Fix nightly build version number issue * Pass BUILD_BUILDNUMBER to linux docker * Disable featurizers in python packages * Import more featurizers (#2781) Make kernels non-template. Add input constraint for learnt data. Add min_max_scalar_transformer, robust_scalar_transformer, inputation_marker_transfomer, label_encoder_transformer, missing_dummies_transformer along with tests. Advance Featurizers library commit. * Implement a more stable softmax (#2715) * Implement a more stable SoftMax e^x is represented as infinity if x is large enough, like 100.f. Infinity divided by Infinity is a NAN. Thus, softmax gets a NAN if one or more item are large enough. A math transform as below is leveraged to get a stable softmax: e^xi/(e^x1 + ...e^xn) = e^(xi - max) / (e^(x1 - max) + ... + e^(xn - max)) And for convenience, force max to 0.f if all xi are negative * Contributing: Fix a typo (#2784) * ACL EP GEMM improvements (#2780) When it is posible we use a fully connected layer instead of the gemm implementation. This will let the library use the best implementation based on the input data. * ACL EP convolution improvements (#2774) Added the optimized implementation for depthwise convolution for both ACL v19.02 and ACL 19.05. Also the pointwise convolution seems to be more optimal in the CPU implementation so we opted for that instead. * Add script for release Nuget validation (#2719) * Initial commit * Nits * Disable a test temporarily * Change working directory * Test * Add download python step * Test update * More changes * Fix space issue * Fix * Verify nuget signing * Fix * Spaces * PR feedback * Nit * Fix * Fix * Remove temporary changes * add uint8 support to where op (#2792) * Improve bert optimization script: (#2712) (1) Move input int64=>int32 conversion to embed layer fusion. (2) Output epsilon attribute for LayerNormalization fusion. * add session creation time cost. (#2798) * ML.NET team needs featurizers within a package (#2789) Add auto ml featurizers to Windows, MacOS as well as to GPU packaging-pipelines. * Initialize max of softmax with lowest of float (#2786) * MLAS: update SGEMM threading parameters (#2808) * add interface to copy batch tensors. (#2807) * add interface to copy batch tensors. * onnxruntime * speed up Windows TRT CI (#2811) * don't run cuda tests if building with tensorrt * remove unnecessary build options for win trt ci * refactor win gpu tensorrt ci yml * --numpy_version=1.17 * update * update * azcopy and cuda path * Update test data (#2356) * Add timeseries imputer transformer featurizer kernel (#2813) Make kernels non-template. Add input constraint for learnt data. Fixup tests. Add two more featurizers along with tests. Tests fail. min_max_scalar_transformer robust_scalar_transformer Fix tests serialized stream by prepending version bytes. Add inputation_marker_transfomer and the test. Fix up float/double type designations. Added label_encoder_transformer along with a test. string_throw case is broken at the momement. Fix labelencodertransfomer_test.cc string_throw case Rename maxabsscalertransformer_test.cc Add MissingDummiesTransformer along with the test. Update manifest. Add TimeSeriesImputerTransformer definition, implementation and tests * Fix memory leak in TRT (#2815) * fix memory leak issue * revert EP_FAIL on enueueV2 * Add manifest missing comma * Run static code analyzer on most of our code (#2817) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * update quantization doc (#2783) * update documentation for quantization script * plus some spell corrections * Filter CPU case for IsFloat16Supported (#2802) * update default optimization level + fix gemm_activation fusion (#2791) * update defualt optimization level + fix gemm_activation fusion * fix typo * add unit test and incorporate review comments * fix test comment * Fix dnnl wheel package name (#2823) * Append '-dnnl' to whl package name when --use_dnnl * Update build.py * Update Ubuntu & TensorRT version in README (#2820) Dockerfile.tensorrt is using nvcr.io/nvidia/tensorrt:19.09-py3 as base Image, update Ubuntu and TensorRT version according to https://docs.nvidia.com/deeplearning/sdk/tensorrt-container-release-notes/rel_19-09.html#rel_19-09 * Merge fixes * Add OneHotEncoder and HashOneHotEncoder kernels. (#2830) Add defs and imlementation for OneHotEncoders, adjuist date_time_transformer kernel and test. Add OneHotEncoder kernel test. Add HashOneHotVectorizerTransformer unit test. This does not link due to multiple definitions of functions that are included into header from a CPP file. * Upgrade gtest to the latest version (#2827) WinML would like to update the googletest submodule. They want some newer features (namely GTEST_SKIP to skip tests programmatically and be able to skip entire fixtures easily) and would need to update the submodule version. However, because the new version of code hit a bug in gcc, even though the bug is already fixed in the latest gcc but we're using gcc 4.8.x and it won't get patched for the bug, so we have to do a compromise, change our code a little bit to make it work. The gcc bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=51213 * Add support for int64_t for topk CPU. Fixes github issue #2806. (#2833) * Ignore allocator type in ExecutionProviders allocator map. Make default initialization of OrtMemoryInfo more clearly invalid. (#2768) * Remove allocator type from the key comparison in ExecutionProviders. Remove usage of DummyArena as it's no longer necessary. * Fix x86 tests where arena allocator is disabled. Make initialization of OrtMemoryInfo clearer by adding Invalid enum value. * Make OrtValueNameIdxMap::MaxIdx more intuitive. * Convert ExternalProject Featurizers into git submodule (#2834) Add git submodule for Featurizer library. Update cmake to build for git submodule. * add domain check for nodes + update documentation (#2831) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Fix Linux CUDA nuget packaging pipeline break * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * update optimization doc for BERT related fusions (#2819) * Add bert related transformers to doc * Add execution provider and comment for bert optimizations * Add comment about accuracy impact of approximation * Fix warnings that cause build to fail * MLAS: enable threading for quantized GEMMs (#2844) * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * remove old winmladapter cpp Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * move sequence implementation into ort lib... still commented out... need to turn back on... * begin sequence implementation * make maps and sequences work * fix broken tests * remove dead code * misc cleanup * CR feedback * User/xianz/winml adapter c api (#2869) * wrapper all existing winml adapter apis with API_IMPL to try catch * Return HR or Throw for WinML adapter APIs if failed * undo macro wrapper for two places * Wrap error macros around ort apis, too. * address CR feedback #2 * add more api throw/return macros * Revert changes no longer needed * revert changes to cxx api * format winml lib.ort and winml adapter * remove static pheonix singleton Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * missing use_dml check in winml_adapter_session (#2930) * --use_dnnl flag was mangled in merge (#2931) * use dml macro not wrapping custom registry code (#2934) * Disable LNK4199 winml_dll to enable cuda builds (#2936) * Disable LNK4199 in winml_dll * linkler->linker * LearningModelSessionAPITestGpu.CreateSessionWithCastToFloat16InModel should return DXGI_ERROR_UNSUPPORTED when FP16 not supported (#2937) * Disable LNK4199 in winml_dll * linkler->linker * Need to return DXGI_ERROR_UNSUPPORTED when Model does not support fp16 * Publish build symbols (#2939) * Publish build symbols * Don't upload PDBs for .exe files * Make x86 build (#2943) * fix last remaining size_t/int64_t warnings->errors (#2948) * TensorString, Sequences and Maps use the first allocator, but should use the cpu default allocator. (#2952) * fix tensor string allcoator * clean up default allocator usage for strings in winml lib/api.ort Co-authored-by: Ryan Lai <ryalai96@gmail.com> * Handle tensor shape of zero (#2954) Co-authored-by: Ryan Lai <ryalai96@gmail.com> * CR feedback (#2970) * CR feedback * fix weird formatting on privacy readme * Add 'All rights reserved.' everywhere * readd all rights reserved to winml_provider_factory.h * remove extra space in comment * remove extra whitespace * fixes post master merge * remove winml from nuget gpu pipeline * set IR VERSION on generated_model in rnn_benchmark (#2972) * Fix slice conformance failures (#2908) Co-authored-by: Adrian Tsai <adtsai@microsoft.com> Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Paul McDaniel <paul_mcdaniel@hotmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Dwayne Robinson <fdwr@hotmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com>
2020-02-05 01:12:19 +00:00
[threading(both)]
[marshaling_behavior(agile)]
[dualapipartition(1)]
runtimeclass TensorFloat16Bit : ITensor, [contract(MachineLearningContract, 2)]Windows.Foundation.IMemoryBuffer
{
static TensorFloat16Bit Create();
static TensorFloat16Bit Create(Windows.Foundation.Collections.IIterable<Int64> shape);
static TensorFloat16Bit CreateFromArray(Windows.Foundation.Collections.IIterable<Int64> shape, Single[] data);
static TensorFloat16Bit CreateFromIterable(Windows.Foundation.Collections.IIterable<Int64> shape, Windows.Foundation.Collections.IIterable<Single> data);
Windows.Foundation.Collections.IVectorView <Single> GetAsVectorView();
[contract(MachineLearningContract, 2)]
{
//! Creates a tensor object, allocates a buffer of size 'shape', and copies all of 'data' into it.
static TensorFloat16Bit CreateFromShapeArrayAndDataArray(Int64[] shape, Single[] data);
//! Creates a tensor object, creates a tensor of size 'shape', and uses the data in 'buffer'.
//! 'buffer' contains a packed array of 16bit floating point values.
static TensorFloat16Bit CreateFromBuffer(Int64[] shape, Windows.Storage.Streams.IBuffer buffer);
}
}
[contract(MachineLearningContract, 1)]
INBOX_ONLY([static_name("Windows.AI.MachineLearning.ITensorUInt8BitStatics", 05f67583-bc24-4220-8a41-2dcd8c5ed33c)])
INBOX_ONLY([interface_name("Windows.AI.MachineLearning.ITensorUInt8Bit", 58e1ae27-622b-48e3-be22-d867aed1daac)])
Merge windowsai (winml layering) into master (#2956) * Initial Commit * Merged PR 3985217: add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc (#2346) add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc and violating our OS layering requirements. We linked against onecoreuap_apiset.lib in VB so we will continue doing this, but I am still unsure why not to link against onecore instead since that is where we ship. However, since Sheil is the owner of this code we will wait to discuss with him before changing anything. * Initial changes for layering * more snipping to get core into ort * update build instructions to include --build_shared_lib (#2358) * update build instructions to include --build_shared_lib * fix line breaks * Task 23998197: add winml_lib_core into onnnxruntime.dll (#2368) * Task 23998197: add winml_lib_core into onnnxruntime.dll * PR feedback build break on perf_test * return proper error when the model path isn't found (#2391) * LearningModelSession is cleaned up to use the adapter, and parts of b… (#2382) this is a big PR. we are going to move it up to layer_dev , which is still a L3 so we are still safe to do work there agile. we are going to move this into the L3 so that ryan can start doing intergration testing. we will pause for a full code review and integration test result prior to going into the L2. >>>> raw comments from previous commits >>> * LearningModelSession is cleaned up to use the adapter, and parts of binding are. * moved everything in the winmladapter made it all nano-com using, WRL to construct objects in the ORT side. base interfaces for everythign for winml to call cleaned up a bunch of winml to use the base interfaces. * more pieces * GetData across the abi. * renamed some namepsace cleaned up OrtValue cleaned up Tensor cleaned up custom ops. everything *but* learnignmodel should be clean * make sure it's building. winml.dll is still a monolith. * model moved over. everything builds clean. step ! * weak ref comment * Layer dev paulm (#2408) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * Layer dev paulm (#2414) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * Layer dev paulm (#2423) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * Layer dev paulm (#2424) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * Layer dev paulm (#2425) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Layer dev paulm (#2426) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * fixed map and sequence when passing stl types across the ABI . found a leak in nvidia driver, but skipped it. all winmlapitests pass now * Moved SessionOptions over to the abi * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * Register WinML TraceLogging provider on Onnxruntime.dll (#2455) * Register WinML TraceLogging provider on Onnxruntime.dll * Add ifdef to make sure trace logging provider has telemetry option when LAYERING_DONE * No need for ifdef for TraceLoggingOptionMicrosoftTelemetry * PR feedback * Move etw registration into lotus environment constructor and deresgister in lotus environment destructor * Brianma/cpuwinml (#2466) * allow building winml cpu without dml. * Brianma/breaks (#2469) * fix some more breaks * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * move dml checks out of winml and into the adapter * better error handling * Brianma/fi (#2470) * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * fix bad merge * Checking in a staging checkpoint point so that Ryan can work with me in parrallel * build break. * Brianma/testfails (#2473) * add missing ir version to dictvectorizer-string.onnx * add missing ir version to relu.onnx * add missing ir version to zipmap*onnx * add IR version to manually generated models * remove an unnecessary ifdef dml * Brianma/windowsai fi (#2475) * update dockerfiles/README (#2336) * Make elementwise op run 4 items per thread (#2335) Description: Describe your changes. Make elementwise op run 4 items per thread unroll for loop to leverage ILP remove unnessary N==0 check inside elementwise GPU kernel Motivation and Context Why is this change required? What problem does it solve? It can improve the performance of GPU elementwise ops. ~2% performance gain on popular NLP bert model. If it fixes an open issue, please link to the issue here. * Add CUDA GatherElements kernel (#2310) * Updates * Update test * Update * Updates * nits * PR feedback * Update * Update * PR feedback * PR comments * Update * Fix build * Fix build * Nits * Fix * Layer Normalization Fusion (#2319) basic layer normalization transform * Add FastGelu Cuda Op for Gelu and Add bias fusion (#2293) * Add FastGelu cuda op * Add AddBiasGelu for experiment * Revert "Add AddBiasGelu for experiment" This reverts commit 5c1ee019858c657e6bb75887265cb85675626e5b. * Add bias * Add unit tests * update comment * update script * fix build error * update coding style * update for CR feedback Enable half2 optimization only when cuda arch >= 7.0 * move _Tanh to common.cuh * implement CPU contrib OP Attention (#2333) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanUnusedInitializers. (#2320) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanupUnusedInitializers. This means initializers that have been replaced during graph optimizations are not left in the GraphProto when we save an optimized model. * Handle edge case where a model has an unused initializer with matching graph input by also removing the graph input. * Use non-const iterators in std::find_if calls to make centos build happy. * Nuget pipeline changes (#2305) 1. refactor the pipeline, remove some duplicated code 2. Move Windows_py_GPU_Wheels job to Win-GPU-CUDA10. We'll deprecated the "Win-GPU" pool 3. Delete cpu-nocontribops-esrp-pipeline.yml and cpu-nocontribops-pipeline.yml 4. In Linux nuget jobs, run "make install" before creating the package. So that extra RPAH info will be removed * Cuda Reverse Sequence Op, maping types of same size using same template function. (#2281) * Set ElementType to String type of node metadata, instead of byte[] (#2348) * Set ElementType to String type of node metadata, instead of byte[] * Fix spacing * Introduce PrimitiveType into a Type System along with an integer constant (#2307) Improve perf by avoiding GetType<T>() calls. Introduce MLTypeCallDispatcher to switch on Input Type. Add Tensor IsType<T>() fast method. * Fix/test dim value of 0 handling in a couple of places (#2337) * Update the CUDA Where implementation broadcasting logic to handle a dim with value of 0. Add unit test Also add unit test for unary op with dim value of 0 * Exclude ngraph from Where test with 0 dim. * Openvino EP R3.1 onnxrt server (#2357) * onnxrt server with OVEP * onnxrt server with OVEP * Update Dockerfile.server.openvino * onnxrt server OVEP fix reviews * onnxrt server OVEP fix reviews * Implement cuda nonzero op. (#2056) Implement cuda nonzero op. * Direct use python numpy array's memory if already contiguous. (#2355) * Direct use python numpy array's memory if already contiguous. This could greatly improve performance for session with large input, like big image 1920x1080 fastrcnn, 30~40% speed up could be achieved. * Add test case enforce contiguous/non-contiguos numpy array as inputs. * Add helper to create output to minimize binary size. (#2365) Add ConstEigenTensorMap typedef so we don't unnecessarily const_cast the const input Tensor. * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS (#2369) * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS * update * Add Tracelogging for profiling (#1639) Enabled only if onnxruntime_ENABLE_INSTRUMENT is ON * test bidaf with nuphar for avx target (#2370) increase nuphar test coverage a bit * Fix a bug in TLS refcount that may destabilized CUDA CI (#2374) * update output size calculation for resize (#2366) * change how output size is calculated for resize op * add tests for ver 10 resize * Extend OneHot CPU kernel to support more types (#2311) * Extend OneHot CPU kernel to support input int64_t, depth int32_t, output float * Skip BERT before the test data fix is picked up * Fix bug with Slice. Need to pass in flattened input dimensions so the initial offset into the input is calculated correctly. (#2372) * Add opset 11 version of Split to CUDA ops (#2376) Organize the CUDA ops definitions so all the opset 10 and 11 parts are together (same setup used for CPU ops) * Layer Norm Fusion Fix (#2379) * layer norm fusion fix * Add input shape check in code and unit tests * Fuse Add + Gelu (#2360) Implement the transformer to fuse add + gelu Implement the accurate kernel * Skip layer norm transform (#2350) * skip layer normalization transformer * Another try to stabilize CUDA CI (#2383) The root cause seems to be failure in CUDA dealloc when tear down. cudaFree return code was ignored before, so should the debug check. * fix BUILD.md typo (#2375) build.py: error: argument --config: invalid choice: 'RelWithDebugInfo' (choose from 'Debug', 'MinSizeRel', 'Release', 'RelWithDebInfo') * Fixed compilation with ngraph (#2388) * Fix reuse logic in allocation planner. (#2393) * Fix reuse logic in allocation planner. * PR comments * Add helpful comments * Don't allow reuse across string tensors. * [NupharEP] Multiple optimizations (#2380) Fuse transpose into MatMul Implement Pow and constant scalar simplification Vectorize ReduceMean Improve symbolic shape inference Minor updates for better debugging in fused function name * Avoid using the default logger in the graph lib and optimizers (#2361) 1. Use the session logger if it is available. 2. Don't disable warning 4100 globally. We should fix the warnings instead of disabling it. * Change CUDA implementation of Transpose to support all fixed size tensor types (#2387) * Change CUDA implementation of Transpose to not use a typed kernel so we can support more types with minimum binary size. Add support for 8, 16, 32 and 64 bit types. Add unit tests. Add method so the implementation can be called directly (will be used by CUDA Scan very soon). * Disable TensorRT for MLFloat16 and int8 unit tests. * Address PR comment and add support for calling cublas implementation if type is mlfloat16. * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. (#2398) * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. * [NupharEP] force some low/zero cost ops to be inlined (#2409) * fix cross compile bug (#2415) * Minor optimization: if a node has already been placed, there's no need to find a kernel for it. (#2417) * Add Reshape Fusion (#2395) * Add reshape fusion * Add some comments * update comments * update comment format * update according to feedback * update for recent logger change * fix build error * (1) Support both input and output edges in find path in graphutils (2) Add a test case of only one constant initializer of Concat input. (3) Refactor ReshapeFusion class to allow add more subgraph fusion in the future. * fix error * (1) loose constraint on initializer: non constant is allowed for reshape fusion. (2) Change versions type to vector. (3) Add logging. (4) Return false when multiple output edges matched in FindPath. Add comments. * only allow one direction (input or output) in FindPath * [NupharEP] Update notebook and docker image (#2416) Add BERT squad in Nuphar tutorial Enhance speed comparsion readability * Fix the issue in matmul_add_fusion (#2407) Fix the issue in matmul_add_fusion If Muatmul + Add has shape [K] * [K, N], reset it to [1, K] * [K, N] will make the output shape to [1, N] will also requires a reshape on the output. Fix: just remove the shape reset to not fuse it. Add a negative test case for matmul+add fusion * feat(treeregressor): Update TreeEnsembleRegressor for type support (#2389) Updates the `TreeEnsembleRegressor` to allow for `double`, `float`, `int64`, and `int32` inputs to match the upstream specification. Signed-off-by: Nick Groszewski <nicholas.groszewski@capitalone.com> * onnxrt server documentation update (#2396) * Added support for Pad-2 operator in OpenVINO-EP (#2405) * Add CUDA If operator. (#2377) * Add CUDA If operator. Uses CPU operator for implementation. By adding a CUDA version the inputs/outputs (with the exception of the 'cond' input) stay on GPU, and no other logic is required to avoid a copy to CPU across the control flow node. * Improved documentation for onnxruntime::utils::SwapByteOrderCopy(), added precondition check. * Fix the type constraints on CUDA If operator to exclude strings. (#2431) * add Im2col<uint8_t> (#2438) * Adjust codegen vectorization width from target (#2439) * Adjust codegen vectorization width from target * Add CUDA Scan operator. (#2403) * Add Scan CUDA op. Uses CPU implementation for logic. Added some device specific functors for handling when data needs to be manipulated on a different device. Added ability to override the materialization logic in the OrtValue slicer so DML can plugin their handling. * Fix Windows GPU C API packaging pipeline failure (#2440) Fix Windows GPU C API packaging pipeline failure (#2440) * Correctly handle implicit inputs for fused nodes (#2390) * Correctly handle implicit inputs for fused nodes Previously, nuphar's partitioning function didn't include node's implicit inputs into the inputs list of MetaDef, and hence a crash was triggered in the onnx graph checker. This commit fixed the issue. Furthermore, it also fixed a related issue where we didn't add implicit inputs into graph_inputs_excluding_initializers_ in Graph::SetGraphInputsOutputs. the issue was that graph_inputs_including_initializers_ populated by SetInputs (e.g. called by FunctionImpl::FunctionImpl) may contain implicit inputs which were not of any node's initializers in the graph. Because they were not part of any initializers, these implicit inputs couldn't be visited by going through all nodes' inputs. Consequently, they would *not* be added into graph_inputs_excluding_initializers_. We fixed the issue by first copying the populated graph_inputs_including_initializers_ into graph_inputs_excluding_initalizers_, which then had both initializers and non-initializers as its initial content. Later, we erase initializers from the list. In this way, we can ensure all implicit inputs to remain in graph_inputs_excluding_initializers_. * refined comments and fixed duplicates Address CR by revisiting comments in terms of implicit inputs Also fixed an issue by skipping duplicates while copying inputs from graph_inputs_including_initializers_. * address CR explain why we need to collect nodes' implicit inputs * don't rely on pointer values for iterating std::set Previously, openvino relied on iterating a set of NodeArg pointers to construct inputs and outputs for a fused graph. It could cause non-determinism. The reason was that although iterating std::set by itself is stable, pointer values of NodeArgs may vary. Consequently, we could end up visiting the set's elements in different orders for different runs for the same test, which resulted in constructing inputs (and outputs) with different orders to the fused graph. For example, for the same test, we may have inputs [A, B] in some runs but inputs[B, A] in others. Let's use std::string as the key type to avoid such nondeterminism. This commit also added implicit inputs into meta->inputs while returning the capability from the openvino provider. * Fixed another latent issue in openvino's GetCapability function The issue was that we couldn't simply erase fused_inputs and fused_outputs while iterating the nodes. For example, an output NodeArg may have multiple uses, and it's wrong if we erase it from fused_outputs when we encounter only one of its uses as input. * Remove DeviceAllocatorRegistry class (#2451) Remove DeviceAllocatorRegistry class * CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library. * Parallel Gelu with ParallelFor (#2399) Parallel Gelu to get better performance for Gelu * Clean up build.py (#2446) * Pull the latest image before running docker build * Fuse SkipLayerNorm with Bias (#2453) Fuse SkipLayerNorm with Bias * Allow more than one invocation of CreateEnv in the same process. (#2467) * Allow more than one invocation of CreateEnv in the same process. * Fix centos build * Symbolic shape inference improvements: (#2460) * Symbolic shape inference improvements: - add a mode to guess unknown ops' output rank - add support for GatherND - add support for If - fix a bug in get_int_values when then tensor rank > 1D, by treating it as no sympy data - add symbol to literal merge when ONNX silently merges dims - fix a bug in Concat when input dim is 0 - fix a bug in ConstantOfShape that computed dim is not updated - add support for dynamic shape in ConstantOfShape - fix a bug in Loop output shape that loop iterator dim is not inserted at dim 0 - add support for dynamic padding in Pad - add support for dynamic shape in Reshape - add support for Resize with opset > 10, by treating output dims as dynamic - fix a bug in Slice when starts/ends are dynamic - restrict input model to opset 7 and above - make output model optional to avoid disk write when testing Run model tests for symbolic shape inference Reduce 2GB docker image size of nuphar * add additional test data set for nuget pipeline (#2448) * add SAS token to download internal test data for nuget pipeline * update azure endpoint * fix keyvault download step * fix variable declaration for secret group * fix indentation * fix yaml syntax for variables * fix setting secrets for script * fix env synctax * Fix macos pipeline * attempt to add secrets to windows download data * fix mac and win data download * fix windows data download * update test data set url and location * Revert "Brianma/windowsai fi (#2475)" This reverts commit 5780b864a15513fda4eadbfc2b5345fefe70b5ec. * Add scenario tests (#2457) * Add scenario tests * Remove TODO from model license * Add winml_api test dependency * fix model load test. fi from master changed the constructor (#2483) * make api tests all pass (#2486) * fix bad merge * fix bad model merge * Layer dev paulm (#2492) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * Rename ambiguous header (#2489) * fix one more missing IR version model (#2500) * add missing IR version to 4 more models used by scenario tests (#2501) * Add CLI parameters to test runner, build WinML in ARM and x86 CI (#2479) * Support test parameters through CLI arguments * Add WinML do Windows x86/ARM CI builds * Code style fixes * Update googletest Remove GPUTEST macros everywhere now that GTEST_SKIP is supported * Refactor main.cpp * Build scenario tests without DML * Link scenario tests to DML when it's enabled (#2502) * Layer dev release pipeline (#2488) Adds winml binaries to existing cpu nuget package, and creates new gpu dml nuget package with winml binaries and DML EP. * Layer dev paulm (#2506) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * Remove usage of IOBinding in WinML and use C_API Run method (#2504) * remove usage of iobinding * Change data structure to use vector of Ort::Values * Polish bind input / output * Use C APIrun method * Update providers on evaluate getresults * Remove run and IObinding interface from WinMLAdapter * Remove use of IObinding * bind unbound outputs code moved to learningmodelbinding * clean up unneeded istensor adapter function * Fix comment * Check if session is closed before binding and clearing * PR feedback * Layer dev paulm (#2507) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * Make tests dependend on winml_dll (#2509) * add dml binaries to DirectML package and be more explicit about condition variables (#2520) * re-enable warnings for winml builds and fix the warnings that were hiding (#2526) * turn devmode back on for winml builds * fix some warnings. include protobuf in a way that disables some warnings * undo protobufhelpers changes and just ignore 4100 errors in pb code * attempt to isolate protobufhelpers errors * add template specialization for getting tensor proto data * Layer dev paulm (#2533) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * Spawn child process to run DeviceLostRecovery scenario test (#2530) * Spawn child process to run DeviceLostRecovery scenario test * Layer dev paulm (#2536) ori said yes * add missing namespace to winml_trace_logging_provider in lotusenvironment.h (#2542) * Handle exception thrown from all apis in WinMLAdapter (#2539) * various changes to unblock windowsai ADO build * Fix custom ops scenario tests (#2562) * Do not shutdown protobuf after ort environment gets destroyed. Lazy load lotus environment first time it is needed * comment typo * pr comment about calling phoenix singleton * Make lotus_environment static in winmladapter * Layer dev paulm (#2567) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * making windowsAI azure dev ops work. * code review comments. * revert changes * Cmake and preprocessor fixes that where uncovered by building on agents without DML available via SDK * Layer dev dml delayload (#2580) * Brianma/cpu (#2583) * don't include dml stuff in cpu builds * tests that link the image lib also need the telemetry lib now * Throw Winml_err_invalid_binding if binding gpu resource on cpu device (#2589) * Throw Winml_err_invalid_binding if binding gpu resource on cpu device * PR comments. No need to query executionprovider for is gpu device * User/xianz/ortthrow (#2596) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * User/xianz/ortthrow (#2599) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * add status error message * Remove uwp onsuspending winrt call because logruntimeperf is getting removed (#2630) * User/xianz/dedup telemetry (#2631) * investigate duplication of telemetry in winml and ort * remove winml telemetry events * telemetry executionProviderEvent * remove unneccessary file and refactor code little bit * Revert back TelemetryEvent, which send up ETW event. * merge changes from layer_dev to windowsai (#2638) * Remove underscore from googletest names (#2616) * Fix leaking memory allocator Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24278761 and https://microsoft.visualstudio.com/OS/_workitems/edit/24330198 * Explicitly initialize Ort::Value with nullptr * Cache WinML adapter * bad merge * define private version of dxcore enum that is added in 19H1 SDK. (#2654) * add comment for explaning private definition of dxcore d3d feature level ennum value. (#2672) * do not package directml.pdb for redist packages. (#2676) * Fix leaking operator registry (#2645) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24354916 * User/orilevari/windowsai master merge (#2674) merge resolutions included pulling in telemetry logic that was merged to master and not windowsai and dereferencing InferenceSession::sessionstate now that it is a unique pointer * Delete Ort Allocator in LearningModelBinding (#2653) * Delete OrtAllocator in LearningModelBinding * PR comments to make Ort::Allocator a smart pointer * Small comment change * PR feedback to clean up code * PR feedback on move semantics * Clean up std::move * Fix memory leaks (#2679) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24356109, https://microsoft.visualstudio.com/OS/_workitems/edit/24388361 and https://microsoft.visualstudio.com/OS/_workitems/edit/24388596 * various changes to properly organize and skip GPU tests. For now for No DML builds we will not run GPU tests at all. In the future we should adapt the tests to expect the appropiate errors. (#2695) * Windowsai without fi (#2701) * Disable Attention fusion tests when DISABLE_CONTRIB_OPS is defined (#2529) * Setup java ci (#2528) * Add provision in ORT for session options to be parsed when available via model file (#2449) * Initial commit * Fix gitmodules * Nits * Nits * Updates * Update * More changes * Updates * Update * Some updates * More changes * Update * Update * Merge * Update * Updates * More changes * Update * Fix nits * Updates * Fix warning * Fix build * Add comment * PR feedback * PR feedback * Updates * Updates * Update * More changes * Fix build break * Comment test for now * Updates * Updates * PR feedback * Updates * Nits * Add tests * Fix build * Fix build * Fix build * Fix build break * Fix build * Nits * PR feedback * More change * Expose GetSessionOptions in pybind logic and add unit test for python * Fix build * PR feedback * PR feedback * Revert "Disable thread pool creation when enabled OpenMP (#2485)" (#2535) This reverts commit 7c7d5a149c9ed52eec67304bae5c4b132166a8a1. * Add dynamic shape support in TensorRT execution provider (#2450) * remove onnx-tensorrt submodule * add new onnx-tensorrt submodule (experiment) for trt6 * update engine build for trt6 * update compile and compute for tensorrt6.0 * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * switch to onnx-tensorrt master for TensorRT6' * Update tensorrt_execution_provider.cc * Handle dynamic batch size and add memcpy in TensorRT EP * update test cases * Update tensorrt_execution_provider.cc * update onnx-tensorrt submodule * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.ubuntu_tensorrt * Update run_dockerbuild.sh * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update concat_op_test.cc * Update tensorrt_execution_provider.cc * Upgrade TensorRT to version 6.0.1.5 * Update onnxruntime_providers.cmake * Update CMakeLists.txt * Update reduction_ops_test.cc * Update install_ubuntu.sh * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.tensorrt * Update BUILD.md * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update onnxruntime_providers.cmake * Update install_ubuntu.sh * Update install_ubuntu.sh * Update gemm_test.cc * Update gather_op_test.cc * Update CMakeLists.txt * Removed submodule * update onnx-tensorrt submodule * update header file * Removed submodule * add submodule onnx-tensorrt kevin's branch shape-test' * add debugging code * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * merge master * Removed submodule * update onnx-tensorrt submodule * add more changes for dynamic shapes * Update tensorrt_execution_provider.cc * update for dynamic shape * update dynamic shape processing * fix logger issue * remove submodule onnx-tensorrt * add submodule onnx-tensorrt * add env variable min_subgraph_size * remove redundency * update document * use onnxruntime::make_unique * fix multi-run issue * remove some tests to save CI build time * Add dynamic shape test * Update TensorRT-ExecutionProvider.md * Add example of running Faster R-CNN model on TensorRT EP * Add more details on env variables * update environment variables * Update tensorrt_basic_test.cc * Update model tests * Update tensor_op_test.cc * remove --use_full_protobuf * Update build.py * User/xianz/telemetry (#2458) * enabme telemetry * enable telemetry * set enable telemetry as default * for debugging * remove log and set disable telemetry as default back * delete private file while testing * resolve comment: mainly add license header, rename macro and update docs * rewording in privacy.md * Fix integer overflow in cuda NonMaxSuppression implementation (#2540) * add test case that should pass but fail * fix nms * extract int_max_output_boxes_per_class * Introduce container type runtime checks and other improvements (#2522) Rework TensorSeq in a manner consistent with Tensor and SparseTensor in terms of type system setup. Reduce templating. Introduce helpers to ensure the same data type. Make OrtValue __dtor not virtual. Introduce ContainerChecker * Fix C API tests for centos and mac (#2544) * change c++14 to c++11 * add ld lib path for centos * enable csharp tests on macos * fix C API test on MacOS + fix manylinux dotnet install * fix manylinux dotnet install * fix lib link * Add back executable bit to build.py * Fix a bug handling negative begin pad values in Pad op (#2550) * Fix bug in Pad op * Update * DNNL CMAKE update (#2548) * Fix android build (#2558) * Update win-x86-ci.yml (#2557) Fix build pipeline break * Re-enable Windows C# tests (#2564) * disable onnx_test_runner -x invocations for dnnl (#2568) * Allow sequence length to be symbolic (#2559) * setup java ci mac (#2570) * make layernorm fusion to support opset 11 (#2545) * Fix a warning found in the latest VS release * Add more check on SkipLayerNorm and BiasGelu fusion (#2574) * Fix file not found error during docker build. (#2569) * Add ConvTranspose1D (#2578) * Ryanunderhill/packagename test (#2582) * [Nuphar EP] fixes for some object detection models (#2581) Update notebook tutorial with multi-threaded int8 GEMM from #2517 * EmbedLayerNormalization Fusion Improvement (#2553) Embedding layer norm fusion improvements - add more checks * Update version (#2584) * Temporarily exclude vgg19 test from Python backend test 1. temporarily exclude vgg19 test which comsumes too much memory, run out of memory on Upsquared device. Single test pass for vgg19, need furture investigation (#2588) 2. Update docker file to decrease the docker image size * Update docs for Android NNAPI EP (#2586) * Fix lto bug for protobuf and ubuntu * add path to build dir before test run (#2590) * Add missig env variables for mac pipeline test (#2595) * Fixed an issue in updating realized dims (#2597) when we update realized dims for scan's output, the sliced axis also needs to be inclusive, i.e. we should check with "dim >= insert_inclusive_axis", because the offset in the symbols are based on Scan sugraph. Otherwise, we would end up with shape mismatch later. * Java API for onnxruntime (#2215) * Add support for opset 11 in reshape fusion (#2592) Support opset verion 11 in reshape fusion * Rename automl python tools folder to featurizer_ops. (#2593) * Support opset 11 subgraph of Squad model in Embed Layer Normalization (#2605) Support opset 11 Squad model that is exported from PyTorch nightly. The embed layer uses Range op which is missed in the transformer. * symbolic shape inference: fix warnings in GPT-2 model (#2608) And revise nuphar perf test on BERT squad * Dump subgraph ID and fused graph ID (#2607) * Dump subgraph ID and fused graph ID Dump subgraph ID and fused graph ID for better debugging * Remove local static fused_count added a field global_fused_count_ to NupharExecutionProvider class * EmbedLayerNormalization Fusion For Dynamic Squad Model Opset 10 (#2613) Support subgraph of SQuAD model exported from pytorch with dynamic input axes * Allow providers to be set for InferenceSession at construction (#2606) * Remove unnecessary parameter in some places in GatherElements implementation (#2612) * Remove unnecessary parameter in some places * Update * Update * Make sure fenced tensor could not reuse other tensor. (#2561) Fix random error caused by this. * Improve Embed Layer Norm Fusion for SQuAD with static input shape (#2621) * fix float16 comparison in initializer (#2629) * epsilon attribute for layernormalization fusion (#2639) * removed unnecessary batch file and fix path (#2640) * Add shape inference to ConvTransposeWithDynamicPads schema (#2632) * Improve cuda expand() opeator's performance. (#2624) * Cuda pad optimize when no padding is needed. (#2625) * Shortcut cuda Pad() when no padding is needed. * Optimize cuda scatter() on 2D compatible. (#2628) * Optimize cuda scatter() on 2D compatible. * Add some comments. * fix build error for ARM (#2648) * Improve performance of resize() in Nearest mode (#2626) Special treatment for 2D, check same size as input image. And in 2d kernel, template use_expolation. * Fix memory exception in Layer Norm Fusion (#2644) * Windows CI changes(#2650) * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Revert "Windowsai without fi (#2701)" This reverts commit 285d4c85ff5c4e265f963208170304ef3461e684. * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Deref unique pointer for session_state * send shutdown event when dll is unloaded and EvaluationStop, SessionC… (#2704) * send shutdown event when dll is unloaded and EvaluationStop, SessionCreationStart Events. * Add EvalutationStart Event * add comment * use correct type for for loop (#2755) * ARM CI (#2759) * Set ARM agent pool * Set CMake generator to VS 2019 in ARM * Use system-wide CMake instead of custom version Our custom version is too old for VS 2019 * Use DML and build shared lib in ARM CI * Restore nuget packages in ARM CI * Disable DML * Refactor ARM debug/release builds * Use system packaged Python version * Remove hardcoded Python path * Downgrade Python to 3.7 for build * Remove explicit CMake path * Fix invalid JSON in cgmanifest.json (#2760) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * Remove usage of WHOLEARCHIVE in WinML CMake and add WinMLAdapterFactory (#2726) * Remove usage of WHOLEARCHIVE in WinMLAdapter CMake and add WinMLAdapterFactory * PR feedback, no need for dll(export) since using def file * PR comments * Small comment in gen_def.py * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * Move winml_provider_factory.h to proper location (#2801) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Filter CPU case for IsFloat16Supported (#2802) * Merge fixes * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * Fix warnings that cause build to fail * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * Make DML include path global (#2882) * Make DML include path global * Add generated cppwinrt headers to winml_lib_common * Integrate changes to WindowsAI to make ADO Build (#2886) * Revert "CMake cross-generator fixes (#2790)" This reverts commit dbe7d97fa1ab155f1309bced87199527e8f35bd2. * add additional suppress warning in onnx_proto * ignore /wd4996 warning * DML execution provider fixes * Revert "Revert "CMake cross-generator fixes (#2790)"" This reverts commit 1ae7b4bcbc02edc881ad28685da98e095dfceb17. * Update func signature of custom op function overloads * common devicehelpers fixes * Add pch.h for winml_lib_common * re-add winml_lib_common_dir/inc to include path for winml_adapter * User/orilevari/dml redist shared folder (#2890) * move dml nuget package directory up one level to make it shared between build flavors * Merge conflict fix * Revert "Merge conflict fix" This reverts commit 142fa72cf9ce4344ad717b50b7ea2b8582aadc7c. * Revert "Merge remote-tracking branch 'origin/master' into windowsai" This reverts commit 6e2126d46e5e5f564d65da37dd4f70c93dd81165, reversing changes made to b3f5583dc9249834b947c8ea905f6a98060d5bd6. * Make winml_test_common free of test macros (#2902) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Don't generate files outside the build root (#2914) * Don't generate files outside the build root * Add onnxruntime_EXTERNAL_DEPENDENCIES to WinML * Add DML depedency on RESTORE_PACKAGES * User/orilevari/fix yaml merge bugs (#2918) * Add winml test source parameter into cmake function (#2919) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Add arguments to unittest cmake functions * remove comment * Revert "Revert "Merge remote-tracking branch 'origin/master' into windowsai"" This reverts commit ade5abe72a4234fdbc3623093c61c02c6b0bdc26. * Fix breaks from merge with ORT master * Brianma/linux (#2917) * don't include windows.h in cross-plat header * add default case for switch statement * signed/unsigned mismatch fix Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> * User/sheilk/winml adapter c api (#2891) * Create winml adapter c api * fix build * make it build * move adapter into onnxruntime core/session * entry point not exported * minor changes * make model metadata work * make tests pass * implement all the model reflection apis on the adapter c abi * update the new ort interface to create a lotus ennvironment with a logging sink * start adding ort env * move all winml code into adapter folder/lib to isolate it * ensure a single logging manager at a time * start refactoring session * refactor session creation interface * add cpu and dml session option methods to adapter * finish session init * stub out interfaces in ort lib to perform similar mechanics of iinference session * enable profiling, and enable schema override * update session register graph transformers * turn back on custom registry for custom ops * Add sync api * add last c api stubs * should build... but all feature values are broken since this is in flight to moving all implementation details into ivalue * remove ep adapter header * Implement DML execution provider functions from adapter (#2846) * Implement DML execution provider functions from adapter * Use functions in OnnxruntimeEngine.cpp * make map/sequence type_infos freeable, and start implementing ivalue * make it build again * implement value methods * implement remaining methods * remove com adapter abi * check dml session * cache the allocator on ivalue * check if resource is cpu/gpu when access its mutable data * update tensor * mismatched parentheses * fix tensor base and binding obj * it evaluates tensors! sometimes... * minor fixes * enable gpu evals * wrapper all existing winml adapter apis with API_IMPL to try catch (#2854) * update winml... tensor strings are broken, need to template tensorbase to do different things for strings * make tensor strings work with 2 copies in/2 copies out * Fix tensor string and allocator bug * make maps work again... needs some fixes still * Make it build! * enable map inputs * map outputs * unbound outputs for sequences and maps * User/xianz/merge windowsai (#2883) * Packaging pipeline changes for VS 2019 (#2711) * Tiny fix to codegen * Simplify cache implementation and avoid static variables that may carry over between models * Extend DML kernels (#2641) * Additional DML operators * Check unsupported attributes and inputs * Address PR comments * Add kernel capability function used for partitioning, and re-enable stride-based int64 support based on value range * Fix test failures * Build fix * PR comments * Update Nuphar tutorial notebook (#2721) 1. Reflect int8 GEMV improvements for multi-threading from #2696 2. Add notes on multi-threading control using OpenMP 3. Add samples of running multi-isa AOT, and show int8 GEMM differences between AVX and AVX2 4. Add rnn_benchmark example to resolve #1993 * Add schema for new Qops (#2611) * Add schema for new Qops * adding shape inference + qlinearaveragepool * plus review comments * plus review comments * updates per review comments * plus review comments * [server] Add supposed for model_name and model_version as cli parameter (#2708) * remove 64bit warning message from python validation. (#2727) * MLAS: ARM64 build fix (#2734) fix bad usage of vreinterpret to cast vector element types * Fix broken python docs links (#2740) * Fix build on Mac OS (#2731) mac os ld doesn't support --while-archive, correct option is -all_load * fix ngraph wheel (#2737) * fix ngraph wheel 1.1.0 onnxruntime_ngraph wheel doesn't work * remove libdnnl.so in nGraph Libs * make it easy to compare * Split onnxruntime server to a separated folder (#2744) * Fix build for Python 3.8 (#2747) * Fix build for Python 3.8 * Update protobuf to 3.11.2 (#1928) Update protobuf to 3.11.2 (#1928) * Change default optimization level to All (from Basic) (#2745) * change default optimization level to All (from Basic) * fix test * fix c# test * Update numpy to 1.18 (#2758) * Update numpy to 1.18 * Pipeline changes for python 3.8 (#2753) 1. Pipeline changes for python 3.8 2. Fix a regression in setup.py which was just introduced in the previous commit. Please notice, we still haven't made python 3.8 + Windows + CUDA work. * Add basic stacktrace output for posix debug builds. (#2749) * [NupharEP] fix a race condition when multiple sessions running different models concurrently (#2772) * Revert "Change default optimization level to All (from Basic) (#2745)" This reverts commit 56bb503c2f26474b6613bcb2a198691a11dcef38. * Fix typo in error message (#2736) * Rename MKL-DNN to DNNL to fix broken link (#2730) * Fix nightly build version number issue * Pass BUILD_BUILDNUMBER to linux docker * Disable featurizers in python packages * Import more featurizers (#2781) Make kernels non-template. Add input constraint for learnt data. Add min_max_scalar_transformer, robust_scalar_transformer, inputation_marker_transfomer, label_encoder_transformer, missing_dummies_transformer along with tests. Advance Featurizers library commit. * Implement a more stable softmax (#2715) * Implement a more stable SoftMax e^x is represented as infinity if x is large enough, like 100.f. Infinity divided by Infinity is a NAN. Thus, softmax gets a NAN if one or more item are large enough. A math transform as below is leveraged to get a stable softmax: e^xi/(e^x1 + ...e^xn) = e^(xi - max) / (e^(x1 - max) + ... + e^(xn - max)) And for convenience, force max to 0.f if all xi are negative * Contributing: Fix a typo (#2784) * ACL EP GEMM improvements (#2780) When it is posible we use a fully connected layer instead of the gemm implementation. This will let the library use the best implementation based on the input data. * ACL EP convolution improvements (#2774) Added the optimized implementation for depthwise convolution for both ACL v19.02 and ACL 19.05. Also the pointwise convolution seems to be more optimal in the CPU implementation so we opted for that instead. * Add script for release Nuget validation (#2719) * Initial commit * Nits * Disable a test temporarily * Change working directory * Test * Add download python step * Test update * More changes * Fix space issue * Fix * Verify nuget signing * Fix * Spaces * PR feedback * Nit * Fix * Fix * Remove temporary changes * add uint8 support to where op (#2792) * Improve bert optimization script: (#2712) (1) Move input int64=>int32 conversion to embed layer fusion. (2) Output epsilon attribute for LayerNormalization fusion. * add session creation time cost. (#2798) * ML.NET team needs featurizers within a package (#2789) Add auto ml featurizers to Windows, MacOS as well as to GPU packaging-pipelines. * Initialize max of softmax with lowest of float (#2786) * MLAS: update SGEMM threading parameters (#2808) * add interface to copy batch tensors. (#2807) * add interface to copy batch tensors. * onnxruntime * speed up Windows TRT CI (#2811) * don't run cuda tests if building with tensorrt * remove unnecessary build options for win trt ci * refactor win gpu tensorrt ci yml * --numpy_version=1.17 * update * update * azcopy and cuda path * Update test data (#2356) * Add timeseries imputer transformer featurizer kernel (#2813) Make kernels non-template. Add input constraint for learnt data. Fixup tests. Add two more featurizers along with tests. Tests fail. min_max_scalar_transformer robust_scalar_transformer Fix tests serialized stream by prepending version bytes. Add inputation_marker_transfomer and the test. Fix up float/double type designations. Added label_encoder_transformer along with a test. string_throw case is broken at the momement. Fix labelencodertransfomer_test.cc string_throw case Rename maxabsscalertransformer_test.cc Add MissingDummiesTransformer along with the test. Update manifest. Add TimeSeriesImputerTransformer definition, implementation and tests * Fix memory leak in TRT (#2815) * fix memory leak issue * revert EP_FAIL on enueueV2 * Add manifest missing comma * Run static code analyzer on most of our code (#2817) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * update quantization doc (#2783) * update documentation for quantization script * plus some spell corrections * Filter CPU case for IsFloat16Supported (#2802) * update default optimization level + fix gemm_activation fusion (#2791) * update defualt optimization level + fix gemm_activation fusion * fix typo * add unit test and incorporate review comments * fix test comment * Fix dnnl wheel package name (#2823) * Append '-dnnl' to whl package name when --use_dnnl * Update build.py * Update Ubuntu & TensorRT version in README (#2820) Dockerfile.tensorrt is using nvcr.io/nvidia/tensorrt:19.09-py3 as base Image, update Ubuntu and TensorRT version according to https://docs.nvidia.com/deeplearning/sdk/tensorrt-container-release-notes/rel_19-09.html#rel_19-09 * Merge fixes * Add OneHotEncoder and HashOneHotEncoder kernels. (#2830) Add defs and imlementation for OneHotEncoders, adjuist date_time_transformer kernel and test. Add OneHotEncoder kernel test. Add HashOneHotVectorizerTransformer unit test. This does not link due to multiple definitions of functions that are included into header from a CPP file. * Upgrade gtest to the latest version (#2827) WinML would like to update the googletest submodule. They want some newer features (namely GTEST_SKIP to skip tests programmatically and be able to skip entire fixtures easily) and would need to update the submodule version. However, because the new version of code hit a bug in gcc, even though the bug is already fixed in the latest gcc but we're using gcc 4.8.x and it won't get patched for the bug, so we have to do a compromise, change our code a little bit to make it work. The gcc bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=51213 * Add support for int64_t for topk CPU. Fixes github issue #2806. (#2833) * Ignore allocator type in ExecutionProviders allocator map. Make default initialization of OrtMemoryInfo more clearly invalid. (#2768) * Remove allocator type from the key comparison in ExecutionProviders. Remove usage of DummyArena as it's no longer necessary. * Fix x86 tests where arena allocator is disabled. Make initialization of OrtMemoryInfo clearer by adding Invalid enum value. * Make OrtValueNameIdxMap::MaxIdx more intuitive. * Convert ExternalProject Featurizers into git submodule (#2834) Add git submodule for Featurizer library. Update cmake to build for git submodule. * add domain check for nodes + update documentation (#2831) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Fix Linux CUDA nuget packaging pipeline break * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * update optimization doc for BERT related fusions (#2819) * Add bert related transformers to doc * Add execution provider and comment for bert optimizations * Add comment about accuracy impact of approximation * Fix warnings that cause build to fail * MLAS: enable threading for quantized GEMMs (#2844) * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * remove old winmladapter cpp Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * move sequence implementation into ort lib... still commented out... need to turn back on... * begin sequence implementation * make maps and sequences work * fix broken tests * remove dead code * misc cleanup * CR feedback * User/xianz/winml adapter c api (#2869) * wrapper all existing winml adapter apis with API_IMPL to try catch * Return HR or Throw for WinML adapter APIs if failed * undo macro wrapper for two places * Wrap error macros around ort apis, too. * address CR feedback #2 * add more api throw/return macros * Revert changes no longer needed * revert changes to cxx api * format winml lib.ort and winml adapter * remove static pheonix singleton Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * missing use_dml check in winml_adapter_session (#2930) * --use_dnnl flag was mangled in merge (#2931) * use dml macro not wrapping custom registry code (#2934) * Disable LNK4199 winml_dll to enable cuda builds (#2936) * Disable LNK4199 in winml_dll * linkler->linker * LearningModelSessionAPITestGpu.CreateSessionWithCastToFloat16InModel should return DXGI_ERROR_UNSUPPORTED when FP16 not supported (#2937) * Disable LNK4199 in winml_dll * linkler->linker * Need to return DXGI_ERROR_UNSUPPORTED when Model does not support fp16 * Publish build symbols (#2939) * Publish build symbols * Don't upload PDBs for .exe files * Make x86 build (#2943) * fix last remaining size_t/int64_t warnings->errors (#2948) * TensorString, Sequences and Maps use the first allocator, but should use the cpu default allocator. (#2952) * fix tensor string allcoator * clean up default allocator usage for strings in winml lib/api.ort Co-authored-by: Ryan Lai <ryalai96@gmail.com> * Handle tensor shape of zero (#2954) Co-authored-by: Ryan Lai <ryalai96@gmail.com> * CR feedback (#2970) * CR feedback * fix weird formatting on privacy readme * Add 'All rights reserved.' everywhere * readd all rights reserved to winml_provider_factory.h * remove extra space in comment * remove extra whitespace * fixes post master merge * remove winml from nuget gpu pipeline * set IR VERSION on generated_model in rnn_benchmark (#2972) * Fix slice conformance failures (#2908) Co-authored-by: Adrian Tsai <adtsai@microsoft.com> Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Paul McDaniel <paul_mcdaniel@hotmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Dwayne Robinson <fdwr@hotmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com>
2020-02-05 01:12:19 +00:00
[threading(both)]
[marshaling_behavior(agile)]
[dualapipartition(1)]
runtimeclass TensorUInt8Bit : ITensor, [contract(MachineLearningContract, 2)]Windows.Foundation.IMemoryBuffer
{
static TensorUInt8Bit Create();
static TensorUInt8Bit Create(Windows.Foundation.Collections.IIterable<Int64> shape);
static TensorUInt8Bit CreateFromArray(Windows.Foundation.Collections.IIterable<Int64> shape, UInt8[] data);
static TensorUInt8Bit CreateFromIterable(Windows.Foundation.Collections.IIterable<Int64> shape, Windows.Foundation.Collections.IIterable<UInt8> data);
Windows.Foundation.Collections.IVectorView <UInt8> GetAsVectorView();
[contract(MachineLearningContract, 2)]
{
//! Creates a tensor object, allocates a buffer of size 'shape', and copies all of 'data' into it.
static TensorUInt8Bit CreateFromShapeArrayAndDataArray(Int64[] shape, UInt8[] data);
//! Creates a tensor object, creates a tensor of size 'shape', and uses the data in 'buffer'.
//! 'buffer' contains a packed array of 8bit uint8 values.
static TensorUInt8Bit CreateFromBuffer(Int64[] shape, Windows.Storage.Streams.IBuffer buffer);
}
}
[contract(MachineLearningContract, 1)]
INBOX_ONLY([static_name("Windows.AI.MachineLearning.ITensorInt8BitStatics", b1a12284-095c-4c76-a661-ac4cee1f3e8b)])
INBOX_ONLY([interface_name("Windows.AI.MachineLearning.ITensorInt8Bit", cddd97c5-ffd8-4fef-aefb-30e1a485b2ee)])
Merge windowsai (winml layering) into master (#2956) * Initial Commit * Merged PR 3985217: add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc (#2346) add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc and violating our OS layering requirements. We linked against onecoreuap_apiset.lib in VB so we will continue doing this, but I am still unsure why not to link against onecore instead since that is where we ship. However, since Sheil is the owner of this code we will wait to discuss with him before changing anything. * Initial changes for layering * more snipping to get core into ort * update build instructions to include --build_shared_lib (#2358) * update build instructions to include --build_shared_lib * fix line breaks * Task 23998197: add winml_lib_core into onnnxruntime.dll (#2368) * Task 23998197: add winml_lib_core into onnnxruntime.dll * PR feedback build break on perf_test * return proper error when the model path isn't found (#2391) * LearningModelSession is cleaned up to use the adapter, and parts of b… (#2382) this is a big PR. we are going to move it up to layer_dev , which is still a L3 so we are still safe to do work there agile. we are going to move this into the L3 so that ryan can start doing intergration testing. we will pause for a full code review and integration test result prior to going into the L2. >>>> raw comments from previous commits >>> * LearningModelSession is cleaned up to use the adapter, and parts of binding are. * moved everything in the winmladapter made it all nano-com using, WRL to construct objects in the ORT side. base interfaces for everythign for winml to call cleaned up a bunch of winml to use the base interfaces. * more pieces * GetData across the abi. * renamed some namepsace cleaned up OrtValue cleaned up Tensor cleaned up custom ops. everything *but* learnignmodel should be clean * make sure it's building. winml.dll is still a monolith. * model moved over. everything builds clean. step ! * weak ref comment * Layer dev paulm (#2408) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * Layer dev paulm (#2414) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * Layer dev paulm (#2423) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * Layer dev paulm (#2424) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * Layer dev paulm (#2425) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Layer dev paulm (#2426) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * fixed map and sequence when passing stl types across the ABI . found a leak in nvidia driver, but skipped it. all winmlapitests pass now * Moved SessionOptions over to the abi * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * Register WinML TraceLogging provider on Onnxruntime.dll (#2455) * Register WinML TraceLogging provider on Onnxruntime.dll * Add ifdef to make sure trace logging provider has telemetry option when LAYERING_DONE * No need for ifdef for TraceLoggingOptionMicrosoftTelemetry * PR feedback * Move etw registration into lotus environment constructor and deresgister in lotus environment destructor * Brianma/cpuwinml (#2466) * allow building winml cpu without dml. * Brianma/breaks (#2469) * fix some more breaks * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * move dml checks out of winml and into the adapter * better error handling * Brianma/fi (#2470) * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * fix bad merge * Checking in a staging checkpoint point so that Ryan can work with me in parrallel * build break. * Brianma/testfails (#2473) * add missing ir version to dictvectorizer-string.onnx * add missing ir version to relu.onnx * add missing ir version to zipmap*onnx * add IR version to manually generated models * remove an unnecessary ifdef dml * Brianma/windowsai fi (#2475) * update dockerfiles/README (#2336) * Make elementwise op run 4 items per thread (#2335) Description: Describe your changes. Make elementwise op run 4 items per thread unroll for loop to leverage ILP remove unnessary N==0 check inside elementwise GPU kernel Motivation and Context Why is this change required? What problem does it solve? It can improve the performance of GPU elementwise ops. ~2% performance gain on popular NLP bert model. If it fixes an open issue, please link to the issue here. * Add CUDA GatherElements kernel (#2310) * Updates * Update test * Update * Updates * nits * PR feedback * Update * Update * PR feedback * PR comments * Update * Fix build * Fix build * Nits * Fix * Layer Normalization Fusion (#2319) basic layer normalization transform * Add FastGelu Cuda Op for Gelu and Add bias fusion (#2293) * Add FastGelu cuda op * Add AddBiasGelu for experiment * Revert "Add AddBiasGelu for experiment" This reverts commit 5c1ee019858c657e6bb75887265cb85675626e5b. * Add bias * Add unit tests * update comment * update script * fix build error * update coding style * update for CR feedback Enable half2 optimization only when cuda arch >= 7.0 * move _Tanh to common.cuh * implement CPU contrib OP Attention (#2333) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanUnusedInitializers. (#2320) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanupUnusedInitializers. This means initializers that have been replaced during graph optimizations are not left in the GraphProto when we save an optimized model. * Handle edge case where a model has an unused initializer with matching graph input by also removing the graph input. * Use non-const iterators in std::find_if calls to make centos build happy. * Nuget pipeline changes (#2305) 1. refactor the pipeline, remove some duplicated code 2. Move Windows_py_GPU_Wheels job to Win-GPU-CUDA10. We'll deprecated the "Win-GPU" pool 3. Delete cpu-nocontribops-esrp-pipeline.yml and cpu-nocontribops-pipeline.yml 4. In Linux nuget jobs, run "make install" before creating the package. So that extra RPAH info will be removed * Cuda Reverse Sequence Op, maping types of same size using same template function. (#2281) * Set ElementType to String type of node metadata, instead of byte[] (#2348) * Set ElementType to String type of node metadata, instead of byte[] * Fix spacing * Introduce PrimitiveType into a Type System along with an integer constant (#2307) Improve perf by avoiding GetType<T>() calls. Introduce MLTypeCallDispatcher to switch on Input Type. Add Tensor IsType<T>() fast method. * Fix/test dim value of 0 handling in a couple of places (#2337) * Update the CUDA Where implementation broadcasting logic to handle a dim with value of 0. Add unit test Also add unit test for unary op with dim value of 0 * Exclude ngraph from Where test with 0 dim. * Openvino EP R3.1 onnxrt server (#2357) * onnxrt server with OVEP * onnxrt server with OVEP * Update Dockerfile.server.openvino * onnxrt server OVEP fix reviews * onnxrt server OVEP fix reviews * Implement cuda nonzero op. (#2056) Implement cuda nonzero op. * Direct use python numpy array's memory if already contiguous. (#2355) * Direct use python numpy array's memory if already contiguous. This could greatly improve performance for session with large input, like big image 1920x1080 fastrcnn, 30~40% speed up could be achieved. * Add test case enforce contiguous/non-contiguos numpy array as inputs. * Add helper to create output to minimize binary size. (#2365) Add ConstEigenTensorMap typedef so we don't unnecessarily const_cast the const input Tensor. * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS (#2369) * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS * update * Add Tracelogging for profiling (#1639) Enabled only if onnxruntime_ENABLE_INSTRUMENT is ON * test bidaf with nuphar for avx target (#2370) increase nuphar test coverage a bit * Fix a bug in TLS refcount that may destabilized CUDA CI (#2374) * update output size calculation for resize (#2366) * change how output size is calculated for resize op * add tests for ver 10 resize * Extend OneHot CPU kernel to support more types (#2311) * Extend OneHot CPU kernel to support input int64_t, depth int32_t, output float * Skip BERT before the test data fix is picked up * Fix bug with Slice. Need to pass in flattened input dimensions so the initial offset into the input is calculated correctly. (#2372) * Add opset 11 version of Split to CUDA ops (#2376) Organize the CUDA ops definitions so all the opset 10 and 11 parts are together (same setup used for CPU ops) * Layer Norm Fusion Fix (#2379) * layer norm fusion fix * Add input shape check in code and unit tests * Fuse Add + Gelu (#2360) Implement the transformer to fuse add + gelu Implement the accurate kernel * Skip layer norm transform (#2350) * skip layer normalization transformer * Another try to stabilize CUDA CI (#2383) The root cause seems to be failure in CUDA dealloc when tear down. cudaFree return code was ignored before, so should the debug check. * fix BUILD.md typo (#2375) build.py: error: argument --config: invalid choice: 'RelWithDebugInfo' (choose from 'Debug', 'MinSizeRel', 'Release', 'RelWithDebInfo') * Fixed compilation with ngraph (#2388) * Fix reuse logic in allocation planner. (#2393) * Fix reuse logic in allocation planner. * PR comments * Add helpful comments * Don't allow reuse across string tensors. * [NupharEP] Multiple optimizations (#2380) Fuse transpose into MatMul Implement Pow and constant scalar simplification Vectorize ReduceMean Improve symbolic shape inference Minor updates for better debugging in fused function name * Avoid using the default logger in the graph lib and optimizers (#2361) 1. Use the session logger if it is available. 2. Don't disable warning 4100 globally. We should fix the warnings instead of disabling it. * Change CUDA implementation of Transpose to support all fixed size tensor types (#2387) * Change CUDA implementation of Transpose to not use a typed kernel so we can support more types with minimum binary size. Add support for 8, 16, 32 and 64 bit types. Add unit tests. Add method so the implementation can be called directly (will be used by CUDA Scan very soon). * Disable TensorRT for MLFloat16 and int8 unit tests. * Address PR comment and add support for calling cublas implementation if type is mlfloat16. * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. (#2398) * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. * [NupharEP] force some low/zero cost ops to be inlined (#2409) * fix cross compile bug (#2415) * Minor optimization: if a node has already been placed, there's no need to find a kernel for it. (#2417) * Add Reshape Fusion (#2395) * Add reshape fusion * Add some comments * update comments * update comment format * update according to feedback * update for recent logger change * fix build error * (1) Support both input and output edges in find path in graphutils (2) Add a test case of only one constant initializer of Concat input. (3) Refactor ReshapeFusion class to allow add more subgraph fusion in the future. * fix error * (1) loose constraint on initializer: non constant is allowed for reshape fusion. (2) Change versions type to vector. (3) Add logging. (4) Return false when multiple output edges matched in FindPath. Add comments. * only allow one direction (input or output) in FindPath * [NupharEP] Update notebook and docker image (#2416) Add BERT squad in Nuphar tutorial Enhance speed comparsion readability * Fix the issue in matmul_add_fusion (#2407) Fix the issue in matmul_add_fusion If Muatmul + Add has shape [K] * [K, N], reset it to [1, K] * [K, N] will make the output shape to [1, N] will also requires a reshape on the output. Fix: just remove the shape reset to not fuse it. Add a negative test case for matmul+add fusion * feat(treeregressor): Update TreeEnsembleRegressor for type support (#2389) Updates the `TreeEnsembleRegressor` to allow for `double`, `float`, `int64`, and `int32` inputs to match the upstream specification. Signed-off-by: Nick Groszewski <nicholas.groszewski@capitalone.com> * onnxrt server documentation update (#2396) * Added support for Pad-2 operator in OpenVINO-EP (#2405) * Add CUDA If operator. (#2377) * Add CUDA If operator. Uses CPU operator for implementation. By adding a CUDA version the inputs/outputs (with the exception of the 'cond' input) stay on GPU, and no other logic is required to avoid a copy to CPU across the control flow node. * Improved documentation for onnxruntime::utils::SwapByteOrderCopy(), added precondition check. * Fix the type constraints on CUDA If operator to exclude strings. (#2431) * add Im2col<uint8_t> (#2438) * Adjust codegen vectorization width from target (#2439) * Adjust codegen vectorization width from target * Add CUDA Scan operator. (#2403) * Add Scan CUDA op. Uses CPU implementation for logic. Added some device specific functors for handling when data needs to be manipulated on a different device. Added ability to override the materialization logic in the OrtValue slicer so DML can plugin their handling. * Fix Windows GPU C API packaging pipeline failure (#2440) Fix Windows GPU C API packaging pipeline failure (#2440) * Correctly handle implicit inputs for fused nodes (#2390) * Correctly handle implicit inputs for fused nodes Previously, nuphar's partitioning function didn't include node's implicit inputs into the inputs list of MetaDef, and hence a crash was triggered in the onnx graph checker. This commit fixed the issue. Furthermore, it also fixed a related issue where we didn't add implicit inputs into graph_inputs_excluding_initializers_ in Graph::SetGraphInputsOutputs. the issue was that graph_inputs_including_initializers_ populated by SetInputs (e.g. called by FunctionImpl::FunctionImpl) may contain implicit inputs which were not of any node's initializers in the graph. Because they were not part of any initializers, these implicit inputs couldn't be visited by going through all nodes' inputs. Consequently, they would *not* be added into graph_inputs_excluding_initializers_. We fixed the issue by first copying the populated graph_inputs_including_initializers_ into graph_inputs_excluding_initalizers_, which then had both initializers and non-initializers as its initial content. Later, we erase initializers from the list. In this way, we can ensure all implicit inputs to remain in graph_inputs_excluding_initializers_. * refined comments and fixed duplicates Address CR by revisiting comments in terms of implicit inputs Also fixed an issue by skipping duplicates while copying inputs from graph_inputs_including_initializers_. * address CR explain why we need to collect nodes' implicit inputs * don't rely on pointer values for iterating std::set Previously, openvino relied on iterating a set of NodeArg pointers to construct inputs and outputs for a fused graph. It could cause non-determinism. The reason was that although iterating std::set by itself is stable, pointer values of NodeArgs may vary. Consequently, we could end up visiting the set's elements in different orders for different runs for the same test, which resulted in constructing inputs (and outputs) with different orders to the fused graph. For example, for the same test, we may have inputs [A, B] in some runs but inputs[B, A] in others. Let's use std::string as the key type to avoid such nondeterminism. This commit also added implicit inputs into meta->inputs while returning the capability from the openvino provider. * Fixed another latent issue in openvino's GetCapability function The issue was that we couldn't simply erase fused_inputs and fused_outputs while iterating the nodes. For example, an output NodeArg may have multiple uses, and it's wrong if we erase it from fused_outputs when we encounter only one of its uses as input. * Remove DeviceAllocatorRegistry class (#2451) Remove DeviceAllocatorRegistry class * CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library. * Parallel Gelu with ParallelFor (#2399) Parallel Gelu to get better performance for Gelu * Clean up build.py (#2446) * Pull the latest image before running docker build * Fuse SkipLayerNorm with Bias (#2453) Fuse SkipLayerNorm with Bias * Allow more than one invocation of CreateEnv in the same process. (#2467) * Allow more than one invocation of CreateEnv in the same process. * Fix centos build * Symbolic shape inference improvements: (#2460) * Symbolic shape inference improvements: - add a mode to guess unknown ops' output rank - add support for GatherND - add support for If - fix a bug in get_int_values when then tensor rank > 1D, by treating it as no sympy data - add symbol to literal merge when ONNX silently merges dims - fix a bug in Concat when input dim is 0 - fix a bug in ConstantOfShape that computed dim is not updated - add support for dynamic shape in ConstantOfShape - fix a bug in Loop output shape that loop iterator dim is not inserted at dim 0 - add support for dynamic padding in Pad - add support for dynamic shape in Reshape - add support for Resize with opset > 10, by treating output dims as dynamic - fix a bug in Slice when starts/ends are dynamic - restrict input model to opset 7 and above - make output model optional to avoid disk write when testing Run model tests for symbolic shape inference Reduce 2GB docker image size of nuphar * add additional test data set for nuget pipeline (#2448) * add SAS token to download internal test data for nuget pipeline * update azure endpoint * fix keyvault download step * fix variable declaration for secret group * fix indentation * fix yaml syntax for variables * fix setting secrets for script * fix env synctax * Fix macos pipeline * attempt to add secrets to windows download data * fix mac and win data download * fix windows data download * update test data set url and location * Revert "Brianma/windowsai fi (#2475)" This reverts commit 5780b864a15513fda4eadbfc2b5345fefe70b5ec. * Add scenario tests (#2457) * Add scenario tests * Remove TODO from model license * Add winml_api test dependency * fix model load test. fi from master changed the constructor (#2483) * make api tests all pass (#2486) * fix bad merge * fix bad model merge * Layer dev paulm (#2492) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * Rename ambiguous header (#2489) * fix one more missing IR version model (#2500) * add missing IR version to 4 more models used by scenario tests (#2501) * Add CLI parameters to test runner, build WinML in ARM and x86 CI (#2479) * Support test parameters through CLI arguments * Add WinML do Windows x86/ARM CI builds * Code style fixes * Update googletest Remove GPUTEST macros everywhere now that GTEST_SKIP is supported * Refactor main.cpp * Build scenario tests without DML * Link scenario tests to DML when it's enabled (#2502) * Layer dev release pipeline (#2488) Adds winml binaries to existing cpu nuget package, and creates new gpu dml nuget package with winml binaries and DML EP. * Layer dev paulm (#2506) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * Remove usage of IOBinding in WinML and use C_API Run method (#2504) * remove usage of iobinding * Change data structure to use vector of Ort::Values * Polish bind input / output * Use C APIrun method * Update providers on evaluate getresults * Remove run and IObinding interface from WinMLAdapter * Remove use of IObinding * bind unbound outputs code moved to learningmodelbinding * clean up unneeded istensor adapter function * Fix comment * Check if session is closed before binding and clearing * PR feedback * Layer dev paulm (#2507) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * Make tests dependend on winml_dll (#2509) * add dml binaries to DirectML package and be more explicit about condition variables (#2520) * re-enable warnings for winml builds and fix the warnings that were hiding (#2526) * turn devmode back on for winml builds * fix some warnings. include protobuf in a way that disables some warnings * undo protobufhelpers changes and just ignore 4100 errors in pb code * attempt to isolate protobufhelpers errors * add template specialization for getting tensor proto data * Layer dev paulm (#2533) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * Spawn child process to run DeviceLostRecovery scenario test (#2530) * Spawn child process to run DeviceLostRecovery scenario test * Layer dev paulm (#2536) ori said yes * add missing namespace to winml_trace_logging_provider in lotusenvironment.h (#2542) * Handle exception thrown from all apis in WinMLAdapter (#2539) * various changes to unblock windowsai ADO build * Fix custom ops scenario tests (#2562) * Do not shutdown protobuf after ort environment gets destroyed. Lazy load lotus environment first time it is needed * comment typo * pr comment about calling phoenix singleton * Make lotus_environment static in winmladapter * Layer dev paulm (#2567) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * making windowsAI azure dev ops work. * code review comments. * revert changes * Cmake and preprocessor fixes that where uncovered by building on agents without DML available via SDK * Layer dev dml delayload (#2580) * Brianma/cpu (#2583) * don't include dml stuff in cpu builds * tests that link the image lib also need the telemetry lib now * Throw Winml_err_invalid_binding if binding gpu resource on cpu device (#2589) * Throw Winml_err_invalid_binding if binding gpu resource on cpu device * PR comments. No need to query executionprovider for is gpu device * User/xianz/ortthrow (#2596) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * User/xianz/ortthrow (#2599) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * add status error message * Remove uwp onsuspending winrt call because logruntimeperf is getting removed (#2630) * User/xianz/dedup telemetry (#2631) * investigate duplication of telemetry in winml and ort * remove winml telemetry events * telemetry executionProviderEvent * remove unneccessary file and refactor code little bit * Revert back TelemetryEvent, which send up ETW event. * merge changes from layer_dev to windowsai (#2638) * Remove underscore from googletest names (#2616) * Fix leaking memory allocator Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24278761 and https://microsoft.visualstudio.com/OS/_workitems/edit/24330198 * Explicitly initialize Ort::Value with nullptr * Cache WinML adapter * bad merge * define private version of dxcore enum that is added in 19H1 SDK. (#2654) * add comment for explaning private definition of dxcore d3d feature level ennum value. (#2672) * do not package directml.pdb for redist packages. (#2676) * Fix leaking operator registry (#2645) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24354916 * User/orilevari/windowsai master merge (#2674) merge resolutions included pulling in telemetry logic that was merged to master and not windowsai and dereferencing InferenceSession::sessionstate now that it is a unique pointer * Delete Ort Allocator in LearningModelBinding (#2653) * Delete OrtAllocator in LearningModelBinding * PR comments to make Ort::Allocator a smart pointer * Small comment change * PR feedback to clean up code * PR feedback on move semantics * Clean up std::move * Fix memory leaks (#2679) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24356109, https://microsoft.visualstudio.com/OS/_workitems/edit/24388361 and https://microsoft.visualstudio.com/OS/_workitems/edit/24388596 * various changes to properly organize and skip GPU tests. For now for No DML builds we will not run GPU tests at all. In the future we should adapt the tests to expect the appropiate errors. (#2695) * Windowsai without fi (#2701) * Disable Attention fusion tests when DISABLE_CONTRIB_OPS is defined (#2529) * Setup java ci (#2528) * Add provision in ORT for session options to be parsed when available via model file (#2449) * Initial commit * Fix gitmodules * Nits * Nits * Updates * Update * More changes * Updates * Update * Some updates * More changes * Update * Update * Merge * Update * Updates * More changes * Update * Fix nits * Updates * Fix warning * Fix build * Add comment * PR feedback * PR feedback * Updates * Updates * Update * More changes * Fix build break * Comment test for now * Updates * Updates * PR feedback * Updates * Nits * Add tests * Fix build * Fix build * Fix build * Fix build break * Fix build * Nits * PR feedback * More change * Expose GetSessionOptions in pybind logic and add unit test for python * Fix build * PR feedback * PR feedback * Revert "Disable thread pool creation when enabled OpenMP (#2485)" (#2535) This reverts commit 7c7d5a149c9ed52eec67304bae5c4b132166a8a1. * Add dynamic shape support in TensorRT execution provider (#2450) * remove onnx-tensorrt submodule * add new onnx-tensorrt submodule (experiment) for trt6 * update engine build for trt6 * update compile and compute for tensorrt6.0 * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * switch to onnx-tensorrt master for TensorRT6' * Update tensorrt_execution_provider.cc * Handle dynamic batch size and add memcpy in TensorRT EP * update test cases * Update tensorrt_execution_provider.cc * update onnx-tensorrt submodule * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.ubuntu_tensorrt * Update run_dockerbuild.sh * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update concat_op_test.cc * Update tensorrt_execution_provider.cc * Upgrade TensorRT to version 6.0.1.5 * Update onnxruntime_providers.cmake * Update CMakeLists.txt * Update reduction_ops_test.cc * Update install_ubuntu.sh * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.tensorrt * Update BUILD.md * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update onnxruntime_providers.cmake * Update install_ubuntu.sh * Update install_ubuntu.sh * Update gemm_test.cc * Update gather_op_test.cc * Update CMakeLists.txt * Removed submodule * update onnx-tensorrt submodule * update header file * Removed submodule * add submodule onnx-tensorrt kevin's branch shape-test' * add debugging code * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * merge master * Removed submodule * update onnx-tensorrt submodule * add more changes for dynamic shapes * Update tensorrt_execution_provider.cc * update for dynamic shape * update dynamic shape processing * fix logger issue * remove submodule onnx-tensorrt * add submodule onnx-tensorrt * add env variable min_subgraph_size * remove redundency * update document * use onnxruntime::make_unique * fix multi-run issue * remove some tests to save CI build time * Add dynamic shape test * Update TensorRT-ExecutionProvider.md * Add example of running Faster R-CNN model on TensorRT EP * Add more details on env variables * update environment variables * Update tensorrt_basic_test.cc * Update model tests * Update tensor_op_test.cc * remove --use_full_protobuf * Update build.py * User/xianz/telemetry (#2458) * enabme telemetry * enable telemetry * set enable telemetry as default * for debugging * remove log and set disable telemetry as default back * delete private file while testing * resolve comment: mainly add license header, rename macro and update docs * rewording in privacy.md * Fix integer overflow in cuda NonMaxSuppression implementation (#2540) * add test case that should pass but fail * fix nms * extract int_max_output_boxes_per_class * Introduce container type runtime checks and other improvements (#2522) Rework TensorSeq in a manner consistent with Tensor and SparseTensor in terms of type system setup. Reduce templating. Introduce helpers to ensure the same data type. Make OrtValue __dtor not virtual. Introduce ContainerChecker * Fix C API tests for centos and mac (#2544) * change c++14 to c++11 * add ld lib path for centos * enable csharp tests on macos * fix C API test on MacOS + fix manylinux dotnet install * fix manylinux dotnet install * fix lib link * Add back executable bit to build.py * Fix a bug handling negative begin pad values in Pad op (#2550) * Fix bug in Pad op * Update * DNNL CMAKE update (#2548) * Fix android build (#2558) * Update win-x86-ci.yml (#2557) Fix build pipeline break * Re-enable Windows C# tests (#2564) * disable onnx_test_runner -x invocations for dnnl (#2568) * Allow sequence length to be symbolic (#2559) * setup java ci mac (#2570) * make layernorm fusion to support opset 11 (#2545) * Fix a warning found in the latest VS release * Add more check on SkipLayerNorm and BiasGelu fusion (#2574) * Fix file not found error during docker build. (#2569) * Add ConvTranspose1D (#2578) * Ryanunderhill/packagename test (#2582) * [Nuphar EP] fixes for some object detection models (#2581) Update notebook tutorial with multi-threaded int8 GEMM from #2517 * EmbedLayerNormalization Fusion Improvement (#2553) Embedding layer norm fusion improvements - add more checks * Update version (#2584) * Temporarily exclude vgg19 test from Python backend test 1. temporarily exclude vgg19 test which comsumes too much memory, run out of memory on Upsquared device. Single test pass for vgg19, need furture investigation (#2588) 2. Update docker file to decrease the docker image size * Update docs for Android NNAPI EP (#2586) * Fix lto bug for protobuf and ubuntu * add path to build dir before test run (#2590) * Add missig env variables for mac pipeline test (#2595) * Fixed an issue in updating realized dims (#2597) when we update realized dims for scan's output, the sliced axis also needs to be inclusive, i.e. we should check with "dim >= insert_inclusive_axis", because the offset in the symbols are based on Scan sugraph. Otherwise, we would end up with shape mismatch later. * Java API for onnxruntime (#2215) * Add support for opset 11 in reshape fusion (#2592) Support opset verion 11 in reshape fusion * Rename automl python tools folder to featurizer_ops. (#2593) * Support opset 11 subgraph of Squad model in Embed Layer Normalization (#2605) Support opset 11 Squad model that is exported from PyTorch nightly. The embed layer uses Range op which is missed in the transformer. * symbolic shape inference: fix warnings in GPT-2 model (#2608) And revise nuphar perf test on BERT squad * Dump subgraph ID and fused graph ID (#2607) * Dump subgraph ID and fused graph ID Dump subgraph ID and fused graph ID for better debugging * Remove local static fused_count added a field global_fused_count_ to NupharExecutionProvider class * EmbedLayerNormalization Fusion For Dynamic Squad Model Opset 10 (#2613) Support subgraph of SQuAD model exported from pytorch with dynamic input axes * Allow providers to be set for InferenceSession at construction (#2606) * Remove unnecessary parameter in some places in GatherElements implementation (#2612) * Remove unnecessary parameter in some places * Update * Update * Make sure fenced tensor could not reuse other tensor. (#2561) Fix random error caused by this. * Improve Embed Layer Norm Fusion for SQuAD with static input shape (#2621) * fix float16 comparison in initializer (#2629) * epsilon attribute for layernormalization fusion (#2639) * removed unnecessary batch file and fix path (#2640) * Add shape inference to ConvTransposeWithDynamicPads schema (#2632) * Improve cuda expand() opeator's performance. (#2624) * Cuda pad optimize when no padding is needed. (#2625) * Shortcut cuda Pad() when no padding is needed. * Optimize cuda scatter() on 2D compatible. (#2628) * Optimize cuda scatter() on 2D compatible. * Add some comments. * fix build error for ARM (#2648) * Improve performance of resize() in Nearest mode (#2626) Special treatment for 2D, check same size as input image. And in 2d kernel, template use_expolation. * Fix memory exception in Layer Norm Fusion (#2644) * Windows CI changes(#2650) * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Revert "Windowsai without fi (#2701)" This reverts commit 285d4c85ff5c4e265f963208170304ef3461e684. * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Deref unique pointer for session_state * send shutdown event when dll is unloaded and EvaluationStop, SessionC… (#2704) * send shutdown event when dll is unloaded and EvaluationStop, SessionCreationStart Events. * Add EvalutationStart Event * add comment * use correct type for for loop (#2755) * ARM CI (#2759) * Set ARM agent pool * Set CMake generator to VS 2019 in ARM * Use system-wide CMake instead of custom version Our custom version is too old for VS 2019 * Use DML and build shared lib in ARM CI * Restore nuget packages in ARM CI * Disable DML * Refactor ARM debug/release builds * Use system packaged Python version * Remove hardcoded Python path * Downgrade Python to 3.7 for build * Remove explicit CMake path * Fix invalid JSON in cgmanifest.json (#2760) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * Remove usage of WHOLEARCHIVE in WinML CMake and add WinMLAdapterFactory (#2726) * Remove usage of WHOLEARCHIVE in WinMLAdapter CMake and add WinMLAdapterFactory * PR feedback, no need for dll(export) since using def file * PR comments * Small comment in gen_def.py * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * Move winml_provider_factory.h to proper location (#2801) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Filter CPU case for IsFloat16Supported (#2802) * Merge fixes * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * Fix warnings that cause build to fail * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * Make DML include path global (#2882) * Make DML include path global * Add generated cppwinrt headers to winml_lib_common * Integrate changes to WindowsAI to make ADO Build (#2886) * Revert "CMake cross-generator fixes (#2790)" This reverts commit dbe7d97fa1ab155f1309bced87199527e8f35bd2. * add additional suppress warning in onnx_proto * ignore /wd4996 warning * DML execution provider fixes * Revert "Revert "CMake cross-generator fixes (#2790)"" This reverts commit 1ae7b4bcbc02edc881ad28685da98e095dfceb17. * Update func signature of custom op function overloads * common devicehelpers fixes * Add pch.h for winml_lib_common * re-add winml_lib_common_dir/inc to include path for winml_adapter * User/orilevari/dml redist shared folder (#2890) * move dml nuget package directory up one level to make it shared between build flavors * Merge conflict fix * Revert "Merge conflict fix" This reverts commit 142fa72cf9ce4344ad717b50b7ea2b8582aadc7c. * Revert "Merge remote-tracking branch 'origin/master' into windowsai" This reverts commit 6e2126d46e5e5f564d65da37dd4f70c93dd81165, reversing changes made to b3f5583dc9249834b947c8ea905f6a98060d5bd6. * Make winml_test_common free of test macros (#2902) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Don't generate files outside the build root (#2914) * Don't generate files outside the build root * Add onnxruntime_EXTERNAL_DEPENDENCIES to WinML * Add DML depedency on RESTORE_PACKAGES * User/orilevari/fix yaml merge bugs (#2918) * Add winml test source parameter into cmake function (#2919) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Add arguments to unittest cmake functions * remove comment * Revert "Revert "Merge remote-tracking branch 'origin/master' into windowsai"" This reverts commit ade5abe72a4234fdbc3623093c61c02c6b0bdc26. * Fix breaks from merge with ORT master * Brianma/linux (#2917) * don't include windows.h in cross-plat header * add default case for switch statement * signed/unsigned mismatch fix Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> * User/sheilk/winml adapter c api (#2891) * Create winml adapter c api * fix build * make it build * move adapter into onnxruntime core/session * entry point not exported * minor changes * make model metadata work * make tests pass * implement all the model reflection apis on the adapter c abi * update the new ort interface to create a lotus ennvironment with a logging sink * start adding ort env * move all winml code into adapter folder/lib to isolate it * ensure a single logging manager at a time * start refactoring session * refactor session creation interface * add cpu and dml session option methods to adapter * finish session init * stub out interfaces in ort lib to perform similar mechanics of iinference session * enable profiling, and enable schema override * update session register graph transformers * turn back on custom registry for custom ops * Add sync api * add last c api stubs * should build... but all feature values are broken since this is in flight to moving all implementation details into ivalue * remove ep adapter header * Implement DML execution provider functions from adapter (#2846) * Implement DML execution provider functions from adapter * Use functions in OnnxruntimeEngine.cpp * make map/sequence type_infos freeable, and start implementing ivalue * make it build again * implement value methods * implement remaining methods * remove com adapter abi * check dml session * cache the allocator on ivalue * check if resource is cpu/gpu when access its mutable data * update tensor * mismatched parentheses * fix tensor base and binding obj * it evaluates tensors! sometimes... * minor fixes * enable gpu evals * wrapper all existing winml adapter apis with API_IMPL to try catch (#2854) * update winml... tensor strings are broken, need to template tensorbase to do different things for strings * make tensor strings work with 2 copies in/2 copies out * Fix tensor string and allocator bug * make maps work again... needs some fixes still * Make it build! * enable map inputs * map outputs * unbound outputs for sequences and maps * User/xianz/merge windowsai (#2883) * Packaging pipeline changes for VS 2019 (#2711) * Tiny fix to codegen * Simplify cache implementation and avoid static variables that may carry over between models * Extend DML kernels (#2641) * Additional DML operators * Check unsupported attributes and inputs * Address PR comments * Add kernel capability function used for partitioning, and re-enable stride-based int64 support based on value range * Fix test failures * Build fix * PR comments * Update Nuphar tutorial notebook (#2721) 1. Reflect int8 GEMV improvements for multi-threading from #2696 2. Add notes on multi-threading control using OpenMP 3. Add samples of running multi-isa AOT, and show int8 GEMM differences between AVX and AVX2 4. Add rnn_benchmark example to resolve #1993 * Add schema for new Qops (#2611) * Add schema for new Qops * adding shape inference + qlinearaveragepool * plus review comments * plus review comments * updates per review comments * plus review comments * [server] Add supposed for model_name and model_version as cli parameter (#2708) * remove 64bit warning message from python validation. (#2727) * MLAS: ARM64 build fix (#2734) fix bad usage of vreinterpret to cast vector element types * Fix broken python docs links (#2740) * Fix build on Mac OS (#2731) mac os ld doesn't support --while-archive, correct option is -all_load * fix ngraph wheel (#2737) * fix ngraph wheel 1.1.0 onnxruntime_ngraph wheel doesn't work * remove libdnnl.so in nGraph Libs * make it easy to compare * Split onnxruntime server to a separated folder (#2744) * Fix build for Python 3.8 (#2747) * Fix build for Python 3.8 * Update protobuf to 3.11.2 (#1928) Update protobuf to 3.11.2 (#1928) * Change default optimization level to All (from Basic) (#2745) * change default optimization level to All (from Basic) * fix test * fix c# test * Update numpy to 1.18 (#2758) * Update numpy to 1.18 * Pipeline changes for python 3.8 (#2753) 1. Pipeline changes for python 3.8 2. Fix a regression in setup.py which was just introduced in the previous commit. Please notice, we still haven't made python 3.8 + Windows + CUDA work. * Add basic stacktrace output for posix debug builds. (#2749) * [NupharEP] fix a race condition when multiple sessions running different models concurrently (#2772) * Revert "Change default optimization level to All (from Basic) (#2745)" This reverts commit 56bb503c2f26474b6613bcb2a198691a11dcef38. * Fix typo in error message (#2736) * Rename MKL-DNN to DNNL to fix broken link (#2730) * Fix nightly build version number issue * Pass BUILD_BUILDNUMBER to linux docker * Disable featurizers in python packages * Import more featurizers (#2781) Make kernels non-template. Add input constraint for learnt data. Add min_max_scalar_transformer, robust_scalar_transformer, inputation_marker_transfomer, label_encoder_transformer, missing_dummies_transformer along with tests. Advance Featurizers library commit. * Implement a more stable softmax (#2715) * Implement a more stable SoftMax e^x is represented as infinity if x is large enough, like 100.f. Infinity divided by Infinity is a NAN. Thus, softmax gets a NAN if one or more item are large enough. A math transform as below is leveraged to get a stable softmax: e^xi/(e^x1 + ...e^xn) = e^(xi - max) / (e^(x1 - max) + ... + e^(xn - max)) And for convenience, force max to 0.f if all xi are negative * Contributing: Fix a typo (#2784) * ACL EP GEMM improvements (#2780) When it is posible we use a fully connected layer instead of the gemm implementation. This will let the library use the best implementation based on the input data. * ACL EP convolution improvements (#2774) Added the optimized implementation for depthwise convolution for both ACL v19.02 and ACL 19.05. Also the pointwise convolution seems to be more optimal in the CPU implementation so we opted for that instead. * Add script for release Nuget validation (#2719) * Initial commit * Nits * Disable a test temporarily * Change working directory * Test * Add download python step * Test update * More changes * Fix space issue * Fix * Verify nuget signing * Fix * Spaces * PR feedback * Nit * Fix * Fix * Remove temporary changes * add uint8 support to where op (#2792) * Improve bert optimization script: (#2712) (1) Move input int64=>int32 conversion to embed layer fusion. (2) Output epsilon attribute for LayerNormalization fusion. * add session creation time cost. (#2798) * ML.NET team needs featurizers within a package (#2789) Add auto ml featurizers to Windows, MacOS as well as to GPU packaging-pipelines. * Initialize max of softmax with lowest of float (#2786) * MLAS: update SGEMM threading parameters (#2808) * add interface to copy batch tensors. (#2807) * add interface to copy batch tensors. * onnxruntime * speed up Windows TRT CI (#2811) * don't run cuda tests if building with tensorrt * remove unnecessary build options for win trt ci * refactor win gpu tensorrt ci yml * --numpy_version=1.17 * update * update * azcopy and cuda path * Update test data (#2356) * Add timeseries imputer transformer featurizer kernel (#2813) Make kernels non-template. Add input constraint for learnt data. Fixup tests. Add two more featurizers along with tests. Tests fail. min_max_scalar_transformer robust_scalar_transformer Fix tests serialized stream by prepending version bytes. Add inputation_marker_transfomer and the test. Fix up float/double type designations. Added label_encoder_transformer along with a test. string_throw case is broken at the momement. Fix labelencodertransfomer_test.cc string_throw case Rename maxabsscalertransformer_test.cc Add MissingDummiesTransformer along with the test. Update manifest. Add TimeSeriesImputerTransformer definition, implementation and tests * Fix memory leak in TRT (#2815) * fix memory leak issue * revert EP_FAIL on enueueV2 * Add manifest missing comma * Run static code analyzer on most of our code (#2817) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * update quantization doc (#2783) * update documentation for quantization script * plus some spell corrections * Filter CPU case for IsFloat16Supported (#2802) * update default optimization level + fix gemm_activation fusion (#2791) * update defualt optimization level + fix gemm_activation fusion * fix typo * add unit test and incorporate review comments * fix test comment * Fix dnnl wheel package name (#2823) * Append '-dnnl' to whl package name when --use_dnnl * Update build.py * Update Ubuntu & TensorRT version in README (#2820) Dockerfile.tensorrt is using nvcr.io/nvidia/tensorrt:19.09-py3 as base Image, update Ubuntu and TensorRT version according to https://docs.nvidia.com/deeplearning/sdk/tensorrt-container-release-notes/rel_19-09.html#rel_19-09 * Merge fixes * Add OneHotEncoder and HashOneHotEncoder kernels. (#2830) Add defs and imlementation for OneHotEncoders, adjuist date_time_transformer kernel and test. Add OneHotEncoder kernel test. Add HashOneHotVectorizerTransformer unit test. This does not link due to multiple definitions of functions that are included into header from a CPP file. * Upgrade gtest to the latest version (#2827) WinML would like to update the googletest submodule. They want some newer features (namely GTEST_SKIP to skip tests programmatically and be able to skip entire fixtures easily) and would need to update the submodule version. However, because the new version of code hit a bug in gcc, even though the bug is already fixed in the latest gcc but we're using gcc 4.8.x and it won't get patched for the bug, so we have to do a compromise, change our code a little bit to make it work. The gcc bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=51213 * Add support for int64_t for topk CPU. Fixes github issue #2806. (#2833) * Ignore allocator type in ExecutionProviders allocator map. Make default initialization of OrtMemoryInfo more clearly invalid. (#2768) * Remove allocator type from the key comparison in ExecutionProviders. Remove usage of DummyArena as it's no longer necessary. * Fix x86 tests where arena allocator is disabled. Make initialization of OrtMemoryInfo clearer by adding Invalid enum value. * Make OrtValueNameIdxMap::MaxIdx more intuitive. * Convert ExternalProject Featurizers into git submodule (#2834) Add git submodule for Featurizer library. Update cmake to build for git submodule. * add domain check for nodes + update documentation (#2831) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Fix Linux CUDA nuget packaging pipeline break * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * update optimization doc for BERT related fusions (#2819) * Add bert related transformers to doc * Add execution provider and comment for bert optimizations * Add comment about accuracy impact of approximation * Fix warnings that cause build to fail * MLAS: enable threading for quantized GEMMs (#2844) * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * remove old winmladapter cpp Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * move sequence implementation into ort lib... still commented out... need to turn back on... * begin sequence implementation * make maps and sequences work * fix broken tests * remove dead code * misc cleanup * CR feedback * User/xianz/winml adapter c api (#2869) * wrapper all existing winml adapter apis with API_IMPL to try catch * Return HR or Throw for WinML adapter APIs if failed * undo macro wrapper for two places * Wrap error macros around ort apis, too. * address CR feedback #2 * add more api throw/return macros * Revert changes no longer needed * revert changes to cxx api * format winml lib.ort and winml adapter * remove static pheonix singleton Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * missing use_dml check in winml_adapter_session (#2930) * --use_dnnl flag was mangled in merge (#2931) * use dml macro not wrapping custom registry code (#2934) * Disable LNK4199 winml_dll to enable cuda builds (#2936) * Disable LNK4199 in winml_dll * linkler->linker * LearningModelSessionAPITestGpu.CreateSessionWithCastToFloat16InModel should return DXGI_ERROR_UNSUPPORTED when FP16 not supported (#2937) * Disable LNK4199 in winml_dll * linkler->linker * Need to return DXGI_ERROR_UNSUPPORTED when Model does not support fp16 * Publish build symbols (#2939) * Publish build symbols * Don't upload PDBs for .exe files * Make x86 build (#2943) * fix last remaining size_t/int64_t warnings->errors (#2948) * TensorString, Sequences and Maps use the first allocator, but should use the cpu default allocator. (#2952) * fix tensor string allcoator * clean up default allocator usage for strings in winml lib/api.ort Co-authored-by: Ryan Lai <ryalai96@gmail.com> * Handle tensor shape of zero (#2954) Co-authored-by: Ryan Lai <ryalai96@gmail.com> * CR feedback (#2970) * CR feedback * fix weird formatting on privacy readme * Add 'All rights reserved.' everywhere * readd all rights reserved to winml_provider_factory.h * remove extra space in comment * remove extra whitespace * fixes post master merge * remove winml from nuget gpu pipeline * set IR VERSION on generated_model in rnn_benchmark (#2972) * Fix slice conformance failures (#2908) Co-authored-by: Adrian Tsai <adtsai@microsoft.com> Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Paul McDaniel <paul_mcdaniel@hotmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Dwayne Robinson <fdwr@hotmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com>
2020-02-05 01:12:19 +00:00
[threading(both)]
[marshaling_behavior(agile)]
[dualapipartition(1)]
runtimeclass TensorInt8Bit : ITensor, [contract(MachineLearningContract, 2)]Windows.Foundation.IMemoryBuffer
{
static TensorInt8Bit Create();
static TensorInt8Bit Create(Windows.Foundation.Collections.IIterable<Int64> shape);
static TensorInt8Bit CreateFromArray(Windows.Foundation.Collections.IIterable<Int64> shape, BYTE[] data);
static TensorInt8Bit CreateFromIterable(Windows.Foundation.Collections.IIterable<Int64> shape, Windows.Foundation.Collections.IIterable<BYTE> data);
Windows.Foundation.Collections.IVectorView <BYTE> GetAsVectorView();
[contract(MachineLearningContract, 2)]
{
//! Creates a tensor object, allocates a buffer of size 'shape', and copies all of 'data' into it.
static TensorInt8Bit CreateFromShapeArrayAndDataArray(Int64[] shape, BYTE[] data);
//! Creates a tensor object, creates a tensor of size 'shape', and uses the data in 'buffer'.
static TensorInt8Bit CreateFromBuffer(Int64[] shape, Windows.Storage.Streams.IBuffer buffer);
}
}
[contract(MachineLearningContract, 1)]
INBOX_ONLY([static_name("Windows.AI.MachineLearning.ITensorUInt16BitStatics", 5df745dd-028a-481a-a27c-c7e6435e52dd)])
INBOX_ONLY([interface_name("Windows.AI.MachineLearning.ITensorUInt16Bit", 68140f4b-23c0-42f3-81f6-a891c011bc3f)])
Merge windowsai (winml layering) into master (#2956) * Initial Commit * Merged PR 3985217: add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc (#2346) add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc and violating our OS layering requirements. We linked against onecoreuap_apiset.lib in VB so we will continue doing this, but I am still unsure why not to link against onecore instead since that is where we ship. However, since Sheil is the owner of this code we will wait to discuss with him before changing anything. * Initial changes for layering * more snipping to get core into ort * update build instructions to include --build_shared_lib (#2358) * update build instructions to include --build_shared_lib * fix line breaks * Task 23998197: add winml_lib_core into onnnxruntime.dll (#2368) * Task 23998197: add winml_lib_core into onnnxruntime.dll * PR feedback build break on perf_test * return proper error when the model path isn't found (#2391) * LearningModelSession is cleaned up to use the adapter, and parts of b… (#2382) this is a big PR. we are going to move it up to layer_dev , which is still a L3 so we are still safe to do work there agile. we are going to move this into the L3 so that ryan can start doing intergration testing. we will pause for a full code review and integration test result prior to going into the L2. >>>> raw comments from previous commits >>> * LearningModelSession is cleaned up to use the adapter, and parts of binding are. * moved everything in the winmladapter made it all nano-com using, WRL to construct objects in the ORT side. base interfaces for everythign for winml to call cleaned up a bunch of winml to use the base interfaces. * more pieces * GetData across the abi. * renamed some namepsace cleaned up OrtValue cleaned up Tensor cleaned up custom ops. everything *but* learnignmodel should be clean * make sure it's building. winml.dll is still a monolith. * model moved over. everything builds clean. step ! * weak ref comment * Layer dev paulm (#2408) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * Layer dev paulm (#2414) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * Layer dev paulm (#2423) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * Layer dev paulm (#2424) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * Layer dev paulm (#2425) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Layer dev paulm (#2426) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * fixed map and sequence when passing stl types across the ABI . found a leak in nvidia driver, but skipped it. all winmlapitests pass now * Moved SessionOptions over to the abi * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * Register WinML TraceLogging provider on Onnxruntime.dll (#2455) * Register WinML TraceLogging provider on Onnxruntime.dll * Add ifdef to make sure trace logging provider has telemetry option when LAYERING_DONE * No need for ifdef for TraceLoggingOptionMicrosoftTelemetry * PR feedback * Move etw registration into lotus environment constructor and deresgister in lotus environment destructor * Brianma/cpuwinml (#2466) * allow building winml cpu without dml. * Brianma/breaks (#2469) * fix some more breaks * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * move dml checks out of winml and into the adapter * better error handling * Brianma/fi (#2470) * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * fix bad merge * Checking in a staging checkpoint point so that Ryan can work with me in parrallel * build break. * Brianma/testfails (#2473) * add missing ir version to dictvectorizer-string.onnx * add missing ir version to relu.onnx * add missing ir version to zipmap*onnx * add IR version to manually generated models * remove an unnecessary ifdef dml * Brianma/windowsai fi (#2475) * update dockerfiles/README (#2336) * Make elementwise op run 4 items per thread (#2335) Description: Describe your changes. Make elementwise op run 4 items per thread unroll for loop to leverage ILP remove unnessary N==0 check inside elementwise GPU kernel Motivation and Context Why is this change required? What problem does it solve? It can improve the performance of GPU elementwise ops. ~2% performance gain on popular NLP bert model. If it fixes an open issue, please link to the issue here. * Add CUDA GatherElements kernel (#2310) * Updates * Update test * Update * Updates * nits * PR feedback * Update * Update * PR feedback * PR comments * Update * Fix build * Fix build * Nits * Fix * Layer Normalization Fusion (#2319) basic layer normalization transform * Add FastGelu Cuda Op for Gelu and Add bias fusion (#2293) * Add FastGelu cuda op * Add AddBiasGelu for experiment * Revert "Add AddBiasGelu for experiment" This reverts commit 5c1ee019858c657e6bb75887265cb85675626e5b. * Add bias * Add unit tests * update comment * update script * fix build error * update coding style * update for CR feedback Enable half2 optimization only when cuda arch >= 7.0 * move _Tanh to common.cuh * implement CPU contrib OP Attention (#2333) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanUnusedInitializers. (#2320) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanupUnusedInitializers. This means initializers that have been replaced during graph optimizations are not left in the GraphProto when we save an optimized model. * Handle edge case where a model has an unused initializer with matching graph input by also removing the graph input. * Use non-const iterators in std::find_if calls to make centos build happy. * Nuget pipeline changes (#2305) 1. refactor the pipeline, remove some duplicated code 2. Move Windows_py_GPU_Wheels job to Win-GPU-CUDA10. We'll deprecated the "Win-GPU" pool 3. Delete cpu-nocontribops-esrp-pipeline.yml and cpu-nocontribops-pipeline.yml 4. In Linux nuget jobs, run "make install" before creating the package. So that extra RPAH info will be removed * Cuda Reverse Sequence Op, maping types of same size using same template function. (#2281) * Set ElementType to String type of node metadata, instead of byte[] (#2348) * Set ElementType to String type of node metadata, instead of byte[] * Fix spacing * Introduce PrimitiveType into a Type System along with an integer constant (#2307) Improve perf by avoiding GetType<T>() calls. Introduce MLTypeCallDispatcher to switch on Input Type. Add Tensor IsType<T>() fast method. * Fix/test dim value of 0 handling in a couple of places (#2337) * Update the CUDA Where implementation broadcasting logic to handle a dim with value of 0. Add unit test Also add unit test for unary op with dim value of 0 * Exclude ngraph from Where test with 0 dim. * Openvino EP R3.1 onnxrt server (#2357) * onnxrt server with OVEP * onnxrt server with OVEP * Update Dockerfile.server.openvino * onnxrt server OVEP fix reviews * onnxrt server OVEP fix reviews * Implement cuda nonzero op. (#2056) Implement cuda nonzero op. * Direct use python numpy array's memory if already contiguous. (#2355) * Direct use python numpy array's memory if already contiguous. This could greatly improve performance for session with large input, like big image 1920x1080 fastrcnn, 30~40% speed up could be achieved. * Add test case enforce contiguous/non-contiguos numpy array as inputs. * Add helper to create output to minimize binary size. (#2365) Add ConstEigenTensorMap typedef so we don't unnecessarily const_cast the const input Tensor. * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS (#2369) * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS * update * Add Tracelogging for profiling (#1639) Enabled only if onnxruntime_ENABLE_INSTRUMENT is ON * test bidaf with nuphar for avx target (#2370) increase nuphar test coverage a bit * Fix a bug in TLS refcount that may destabilized CUDA CI (#2374) * update output size calculation for resize (#2366) * change how output size is calculated for resize op * add tests for ver 10 resize * Extend OneHot CPU kernel to support more types (#2311) * Extend OneHot CPU kernel to support input int64_t, depth int32_t, output float * Skip BERT before the test data fix is picked up * Fix bug with Slice. Need to pass in flattened input dimensions so the initial offset into the input is calculated correctly. (#2372) * Add opset 11 version of Split to CUDA ops (#2376) Organize the CUDA ops definitions so all the opset 10 and 11 parts are together (same setup used for CPU ops) * Layer Norm Fusion Fix (#2379) * layer norm fusion fix * Add input shape check in code and unit tests * Fuse Add + Gelu (#2360) Implement the transformer to fuse add + gelu Implement the accurate kernel * Skip layer norm transform (#2350) * skip layer normalization transformer * Another try to stabilize CUDA CI (#2383) The root cause seems to be failure in CUDA dealloc when tear down. cudaFree return code was ignored before, so should the debug check. * fix BUILD.md typo (#2375) build.py: error: argument --config: invalid choice: 'RelWithDebugInfo' (choose from 'Debug', 'MinSizeRel', 'Release', 'RelWithDebInfo') * Fixed compilation with ngraph (#2388) * Fix reuse logic in allocation planner. (#2393) * Fix reuse logic in allocation planner. * PR comments * Add helpful comments * Don't allow reuse across string tensors. * [NupharEP] Multiple optimizations (#2380) Fuse transpose into MatMul Implement Pow and constant scalar simplification Vectorize ReduceMean Improve symbolic shape inference Minor updates for better debugging in fused function name * Avoid using the default logger in the graph lib and optimizers (#2361) 1. Use the session logger if it is available. 2. Don't disable warning 4100 globally. We should fix the warnings instead of disabling it. * Change CUDA implementation of Transpose to support all fixed size tensor types (#2387) * Change CUDA implementation of Transpose to not use a typed kernel so we can support more types with minimum binary size. Add support for 8, 16, 32 and 64 bit types. Add unit tests. Add method so the implementation can be called directly (will be used by CUDA Scan very soon). * Disable TensorRT for MLFloat16 and int8 unit tests. * Address PR comment and add support for calling cublas implementation if type is mlfloat16. * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. (#2398) * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. * [NupharEP] force some low/zero cost ops to be inlined (#2409) * fix cross compile bug (#2415) * Minor optimization: if a node has already been placed, there's no need to find a kernel for it. (#2417) * Add Reshape Fusion (#2395) * Add reshape fusion * Add some comments * update comments * update comment format * update according to feedback * update for recent logger change * fix build error * (1) Support both input and output edges in find path in graphutils (2) Add a test case of only one constant initializer of Concat input. (3) Refactor ReshapeFusion class to allow add more subgraph fusion in the future. * fix error * (1) loose constraint on initializer: non constant is allowed for reshape fusion. (2) Change versions type to vector. (3) Add logging. (4) Return false when multiple output edges matched in FindPath. Add comments. * only allow one direction (input or output) in FindPath * [NupharEP] Update notebook and docker image (#2416) Add BERT squad in Nuphar tutorial Enhance speed comparsion readability * Fix the issue in matmul_add_fusion (#2407) Fix the issue in matmul_add_fusion If Muatmul + Add has shape [K] * [K, N], reset it to [1, K] * [K, N] will make the output shape to [1, N] will also requires a reshape on the output. Fix: just remove the shape reset to not fuse it. Add a negative test case for matmul+add fusion * feat(treeregressor): Update TreeEnsembleRegressor for type support (#2389) Updates the `TreeEnsembleRegressor` to allow for `double`, `float`, `int64`, and `int32` inputs to match the upstream specification. Signed-off-by: Nick Groszewski <nicholas.groszewski@capitalone.com> * onnxrt server documentation update (#2396) * Added support for Pad-2 operator in OpenVINO-EP (#2405) * Add CUDA If operator. (#2377) * Add CUDA If operator. Uses CPU operator for implementation. By adding a CUDA version the inputs/outputs (with the exception of the 'cond' input) stay on GPU, and no other logic is required to avoid a copy to CPU across the control flow node. * Improved documentation for onnxruntime::utils::SwapByteOrderCopy(), added precondition check. * Fix the type constraints on CUDA If operator to exclude strings. (#2431) * add Im2col<uint8_t> (#2438) * Adjust codegen vectorization width from target (#2439) * Adjust codegen vectorization width from target * Add CUDA Scan operator. (#2403) * Add Scan CUDA op. Uses CPU implementation for logic. Added some device specific functors for handling when data needs to be manipulated on a different device. Added ability to override the materialization logic in the OrtValue slicer so DML can plugin their handling. * Fix Windows GPU C API packaging pipeline failure (#2440) Fix Windows GPU C API packaging pipeline failure (#2440) * Correctly handle implicit inputs for fused nodes (#2390) * Correctly handle implicit inputs for fused nodes Previously, nuphar's partitioning function didn't include node's implicit inputs into the inputs list of MetaDef, and hence a crash was triggered in the onnx graph checker. This commit fixed the issue. Furthermore, it also fixed a related issue where we didn't add implicit inputs into graph_inputs_excluding_initializers_ in Graph::SetGraphInputsOutputs. the issue was that graph_inputs_including_initializers_ populated by SetInputs (e.g. called by FunctionImpl::FunctionImpl) may contain implicit inputs which were not of any node's initializers in the graph. Because they were not part of any initializers, these implicit inputs couldn't be visited by going through all nodes' inputs. Consequently, they would *not* be added into graph_inputs_excluding_initializers_. We fixed the issue by first copying the populated graph_inputs_including_initializers_ into graph_inputs_excluding_initalizers_, which then had both initializers and non-initializers as its initial content. Later, we erase initializers from the list. In this way, we can ensure all implicit inputs to remain in graph_inputs_excluding_initializers_. * refined comments and fixed duplicates Address CR by revisiting comments in terms of implicit inputs Also fixed an issue by skipping duplicates while copying inputs from graph_inputs_including_initializers_. * address CR explain why we need to collect nodes' implicit inputs * don't rely on pointer values for iterating std::set Previously, openvino relied on iterating a set of NodeArg pointers to construct inputs and outputs for a fused graph. It could cause non-determinism. The reason was that although iterating std::set by itself is stable, pointer values of NodeArgs may vary. Consequently, we could end up visiting the set's elements in different orders for different runs for the same test, which resulted in constructing inputs (and outputs) with different orders to the fused graph. For example, for the same test, we may have inputs [A, B] in some runs but inputs[B, A] in others. Let's use std::string as the key type to avoid such nondeterminism. This commit also added implicit inputs into meta->inputs while returning the capability from the openvino provider. * Fixed another latent issue in openvino's GetCapability function The issue was that we couldn't simply erase fused_inputs and fused_outputs while iterating the nodes. For example, an output NodeArg may have multiple uses, and it's wrong if we erase it from fused_outputs when we encounter only one of its uses as input. * Remove DeviceAllocatorRegistry class (#2451) Remove DeviceAllocatorRegistry class * CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library. * Parallel Gelu with ParallelFor (#2399) Parallel Gelu to get better performance for Gelu * Clean up build.py (#2446) * Pull the latest image before running docker build * Fuse SkipLayerNorm with Bias (#2453) Fuse SkipLayerNorm with Bias * Allow more than one invocation of CreateEnv in the same process. (#2467) * Allow more than one invocation of CreateEnv in the same process. * Fix centos build * Symbolic shape inference improvements: (#2460) * Symbolic shape inference improvements: - add a mode to guess unknown ops' output rank - add support for GatherND - add support for If - fix a bug in get_int_values when then tensor rank > 1D, by treating it as no sympy data - add symbol to literal merge when ONNX silently merges dims - fix a bug in Concat when input dim is 0 - fix a bug in ConstantOfShape that computed dim is not updated - add support for dynamic shape in ConstantOfShape - fix a bug in Loop output shape that loop iterator dim is not inserted at dim 0 - add support for dynamic padding in Pad - add support for dynamic shape in Reshape - add support for Resize with opset > 10, by treating output dims as dynamic - fix a bug in Slice when starts/ends are dynamic - restrict input model to opset 7 and above - make output model optional to avoid disk write when testing Run model tests for symbolic shape inference Reduce 2GB docker image size of nuphar * add additional test data set for nuget pipeline (#2448) * add SAS token to download internal test data for nuget pipeline * update azure endpoint * fix keyvault download step * fix variable declaration for secret group * fix indentation * fix yaml syntax for variables * fix setting secrets for script * fix env synctax * Fix macos pipeline * attempt to add secrets to windows download data * fix mac and win data download * fix windows data download * update test data set url and location * Revert "Brianma/windowsai fi (#2475)" This reverts commit 5780b864a15513fda4eadbfc2b5345fefe70b5ec. * Add scenario tests (#2457) * Add scenario tests * Remove TODO from model license * Add winml_api test dependency * fix model load test. fi from master changed the constructor (#2483) * make api tests all pass (#2486) * fix bad merge * fix bad model merge * Layer dev paulm (#2492) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * Rename ambiguous header (#2489) * fix one more missing IR version model (#2500) * add missing IR version to 4 more models used by scenario tests (#2501) * Add CLI parameters to test runner, build WinML in ARM and x86 CI (#2479) * Support test parameters through CLI arguments * Add WinML do Windows x86/ARM CI builds * Code style fixes * Update googletest Remove GPUTEST macros everywhere now that GTEST_SKIP is supported * Refactor main.cpp * Build scenario tests without DML * Link scenario tests to DML when it's enabled (#2502) * Layer dev release pipeline (#2488) Adds winml binaries to existing cpu nuget package, and creates new gpu dml nuget package with winml binaries and DML EP. * Layer dev paulm (#2506) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * Remove usage of IOBinding in WinML and use C_API Run method (#2504) * remove usage of iobinding * Change data structure to use vector of Ort::Values * Polish bind input / output * Use C APIrun method * Update providers on evaluate getresults * Remove run and IObinding interface from WinMLAdapter * Remove use of IObinding * bind unbound outputs code moved to learningmodelbinding * clean up unneeded istensor adapter function * Fix comment * Check if session is closed before binding and clearing * PR feedback * Layer dev paulm (#2507) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * Make tests dependend on winml_dll (#2509) * add dml binaries to DirectML package and be more explicit about condition variables (#2520) * re-enable warnings for winml builds and fix the warnings that were hiding (#2526) * turn devmode back on for winml builds * fix some warnings. include protobuf in a way that disables some warnings * undo protobufhelpers changes and just ignore 4100 errors in pb code * attempt to isolate protobufhelpers errors * add template specialization for getting tensor proto data * Layer dev paulm (#2533) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * Spawn child process to run DeviceLostRecovery scenario test (#2530) * Spawn child process to run DeviceLostRecovery scenario test * Layer dev paulm (#2536) ori said yes * add missing namespace to winml_trace_logging_provider in lotusenvironment.h (#2542) * Handle exception thrown from all apis in WinMLAdapter (#2539) * various changes to unblock windowsai ADO build * Fix custom ops scenario tests (#2562) * Do not shutdown protobuf after ort environment gets destroyed. Lazy load lotus environment first time it is needed * comment typo * pr comment about calling phoenix singleton * Make lotus_environment static in winmladapter * Layer dev paulm (#2567) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * making windowsAI azure dev ops work. * code review comments. * revert changes * Cmake and preprocessor fixes that where uncovered by building on agents without DML available via SDK * Layer dev dml delayload (#2580) * Brianma/cpu (#2583) * don't include dml stuff in cpu builds * tests that link the image lib also need the telemetry lib now * Throw Winml_err_invalid_binding if binding gpu resource on cpu device (#2589) * Throw Winml_err_invalid_binding if binding gpu resource on cpu device * PR comments. No need to query executionprovider for is gpu device * User/xianz/ortthrow (#2596) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * User/xianz/ortthrow (#2599) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * add status error message * Remove uwp onsuspending winrt call because logruntimeperf is getting removed (#2630) * User/xianz/dedup telemetry (#2631) * investigate duplication of telemetry in winml and ort * remove winml telemetry events * telemetry executionProviderEvent * remove unneccessary file and refactor code little bit * Revert back TelemetryEvent, which send up ETW event. * merge changes from layer_dev to windowsai (#2638) * Remove underscore from googletest names (#2616) * Fix leaking memory allocator Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24278761 and https://microsoft.visualstudio.com/OS/_workitems/edit/24330198 * Explicitly initialize Ort::Value with nullptr * Cache WinML adapter * bad merge * define private version of dxcore enum that is added in 19H1 SDK. (#2654) * add comment for explaning private definition of dxcore d3d feature level ennum value. (#2672) * do not package directml.pdb for redist packages. (#2676) * Fix leaking operator registry (#2645) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24354916 * User/orilevari/windowsai master merge (#2674) merge resolutions included pulling in telemetry logic that was merged to master and not windowsai and dereferencing InferenceSession::sessionstate now that it is a unique pointer * Delete Ort Allocator in LearningModelBinding (#2653) * Delete OrtAllocator in LearningModelBinding * PR comments to make Ort::Allocator a smart pointer * Small comment change * PR feedback to clean up code * PR feedback on move semantics * Clean up std::move * Fix memory leaks (#2679) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24356109, https://microsoft.visualstudio.com/OS/_workitems/edit/24388361 and https://microsoft.visualstudio.com/OS/_workitems/edit/24388596 * various changes to properly organize and skip GPU tests. For now for No DML builds we will not run GPU tests at all. In the future we should adapt the tests to expect the appropiate errors. (#2695) * Windowsai without fi (#2701) * Disable Attention fusion tests when DISABLE_CONTRIB_OPS is defined (#2529) * Setup java ci (#2528) * Add provision in ORT for session options to be parsed when available via model file (#2449) * Initial commit * Fix gitmodules * Nits * Nits * Updates * Update * More changes * Updates * Update * Some updates * More changes * Update * Update * Merge * Update * Updates * More changes * Update * Fix nits * Updates * Fix warning * Fix build * Add comment * PR feedback * PR feedback * Updates * Updates * Update * More changes * Fix build break * Comment test for now * Updates * Updates * PR feedback * Updates * Nits * Add tests * Fix build * Fix build * Fix build * Fix build break * Fix build * Nits * PR feedback * More change * Expose GetSessionOptions in pybind logic and add unit test for python * Fix build * PR feedback * PR feedback * Revert "Disable thread pool creation when enabled OpenMP (#2485)" (#2535) This reverts commit 7c7d5a149c9ed52eec67304bae5c4b132166a8a1. * Add dynamic shape support in TensorRT execution provider (#2450) * remove onnx-tensorrt submodule * add new onnx-tensorrt submodule (experiment) for trt6 * update engine build for trt6 * update compile and compute for tensorrt6.0 * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * switch to onnx-tensorrt master for TensorRT6' * Update tensorrt_execution_provider.cc * Handle dynamic batch size and add memcpy in TensorRT EP * update test cases * Update tensorrt_execution_provider.cc * update onnx-tensorrt submodule * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.ubuntu_tensorrt * Update run_dockerbuild.sh * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update concat_op_test.cc * Update tensorrt_execution_provider.cc * Upgrade TensorRT to version 6.0.1.5 * Update onnxruntime_providers.cmake * Update CMakeLists.txt * Update reduction_ops_test.cc * Update install_ubuntu.sh * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.tensorrt * Update BUILD.md * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update onnxruntime_providers.cmake * Update install_ubuntu.sh * Update install_ubuntu.sh * Update gemm_test.cc * Update gather_op_test.cc * Update CMakeLists.txt * Removed submodule * update onnx-tensorrt submodule * update header file * Removed submodule * add submodule onnx-tensorrt kevin's branch shape-test' * add debugging code * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * merge master * Removed submodule * update onnx-tensorrt submodule * add more changes for dynamic shapes * Update tensorrt_execution_provider.cc * update for dynamic shape * update dynamic shape processing * fix logger issue * remove submodule onnx-tensorrt * add submodule onnx-tensorrt * add env variable min_subgraph_size * remove redundency * update document * use onnxruntime::make_unique * fix multi-run issue * remove some tests to save CI build time * Add dynamic shape test * Update TensorRT-ExecutionProvider.md * Add example of running Faster R-CNN model on TensorRT EP * Add more details on env variables * update environment variables * Update tensorrt_basic_test.cc * Update model tests * Update tensor_op_test.cc * remove --use_full_protobuf * Update build.py * User/xianz/telemetry (#2458) * enabme telemetry * enable telemetry * set enable telemetry as default * for debugging * remove log and set disable telemetry as default back * delete private file while testing * resolve comment: mainly add license header, rename macro and update docs * rewording in privacy.md * Fix integer overflow in cuda NonMaxSuppression implementation (#2540) * add test case that should pass but fail * fix nms * extract int_max_output_boxes_per_class * Introduce container type runtime checks and other improvements (#2522) Rework TensorSeq in a manner consistent with Tensor and SparseTensor in terms of type system setup. Reduce templating. Introduce helpers to ensure the same data type. Make OrtValue __dtor not virtual. Introduce ContainerChecker * Fix C API tests for centos and mac (#2544) * change c++14 to c++11 * add ld lib path for centos * enable csharp tests on macos * fix C API test on MacOS + fix manylinux dotnet install * fix manylinux dotnet install * fix lib link * Add back executable bit to build.py * Fix a bug handling negative begin pad values in Pad op (#2550) * Fix bug in Pad op * Update * DNNL CMAKE update (#2548) * Fix android build (#2558) * Update win-x86-ci.yml (#2557) Fix build pipeline break * Re-enable Windows C# tests (#2564) * disable onnx_test_runner -x invocations for dnnl (#2568) * Allow sequence length to be symbolic (#2559) * setup java ci mac (#2570) * make layernorm fusion to support opset 11 (#2545) * Fix a warning found in the latest VS release * Add more check on SkipLayerNorm and BiasGelu fusion (#2574) * Fix file not found error during docker build. (#2569) * Add ConvTranspose1D (#2578) * Ryanunderhill/packagename test (#2582) * [Nuphar EP] fixes for some object detection models (#2581) Update notebook tutorial with multi-threaded int8 GEMM from #2517 * EmbedLayerNormalization Fusion Improvement (#2553) Embedding layer norm fusion improvements - add more checks * Update version (#2584) * Temporarily exclude vgg19 test from Python backend test 1. temporarily exclude vgg19 test which comsumes too much memory, run out of memory on Upsquared device. Single test pass for vgg19, need furture investigation (#2588) 2. Update docker file to decrease the docker image size * Update docs for Android NNAPI EP (#2586) * Fix lto bug for protobuf and ubuntu * add path to build dir before test run (#2590) * Add missig env variables for mac pipeline test (#2595) * Fixed an issue in updating realized dims (#2597) when we update realized dims for scan's output, the sliced axis also needs to be inclusive, i.e. we should check with "dim >= insert_inclusive_axis", because the offset in the symbols are based on Scan sugraph. Otherwise, we would end up with shape mismatch later. * Java API for onnxruntime (#2215) * Add support for opset 11 in reshape fusion (#2592) Support opset verion 11 in reshape fusion * Rename automl python tools folder to featurizer_ops. (#2593) * Support opset 11 subgraph of Squad model in Embed Layer Normalization (#2605) Support opset 11 Squad model that is exported from PyTorch nightly. The embed layer uses Range op which is missed in the transformer. * symbolic shape inference: fix warnings in GPT-2 model (#2608) And revise nuphar perf test on BERT squad * Dump subgraph ID and fused graph ID (#2607) * Dump subgraph ID and fused graph ID Dump subgraph ID and fused graph ID for better debugging * Remove local static fused_count added a field global_fused_count_ to NupharExecutionProvider class * EmbedLayerNormalization Fusion For Dynamic Squad Model Opset 10 (#2613) Support subgraph of SQuAD model exported from pytorch with dynamic input axes * Allow providers to be set for InferenceSession at construction (#2606) * Remove unnecessary parameter in some places in GatherElements implementation (#2612) * Remove unnecessary parameter in some places * Update * Update * Make sure fenced tensor could not reuse other tensor. (#2561) Fix random error caused by this. * Improve Embed Layer Norm Fusion for SQuAD with static input shape (#2621) * fix float16 comparison in initializer (#2629) * epsilon attribute for layernormalization fusion (#2639) * removed unnecessary batch file and fix path (#2640) * Add shape inference to ConvTransposeWithDynamicPads schema (#2632) * Improve cuda expand() opeator's performance. (#2624) * Cuda pad optimize when no padding is needed. (#2625) * Shortcut cuda Pad() when no padding is needed. * Optimize cuda scatter() on 2D compatible. (#2628) * Optimize cuda scatter() on 2D compatible. * Add some comments. * fix build error for ARM (#2648) * Improve performance of resize() in Nearest mode (#2626) Special treatment for 2D, check same size as input image. And in 2d kernel, template use_expolation. * Fix memory exception in Layer Norm Fusion (#2644) * Windows CI changes(#2650) * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Revert "Windowsai without fi (#2701)" This reverts commit 285d4c85ff5c4e265f963208170304ef3461e684. * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Deref unique pointer for session_state * send shutdown event when dll is unloaded and EvaluationStop, SessionC… (#2704) * send shutdown event when dll is unloaded and EvaluationStop, SessionCreationStart Events. * Add EvalutationStart Event * add comment * use correct type for for loop (#2755) * ARM CI (#2759) * Set ARM agent pool * Set CMake generator to VS 2019 in ARM * Use system-wide CMake instead of custom version Our custom version is too old for VS 2019 * Use DML and build shared lib in ARM CI * Restore nuget packages in ARM CI * Disable DML * Refactor ARM debug/release builds * Use system packaged Python version * Remove hardcoded Python path * Downgrade Python to 3.7 for build * Remove explicit CMake path * Fix invalid JSON in cgmanifest.json (#2760) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * Remove usage of WHOLEARCHIVE in WinML CMake and add WinMLAdapterFactory (#2726) * Remove usage of WHOLEARCHIVE in WinMLAdapter CMake and add WinMLAdapterFactory * PR feedback, no need for dll(export) since using def file * PR comments * Small comment in gen_def.py * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * Move winml_provider_factory.h to proper location (#2801) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Filter CPU case for IsFloat16Supported (#2802) * Merge fixes * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * Fix warnings that cause build to fail * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * Make DML include path global (#2882) * Make DML include path global * Add generated cppwinrt headers to winml_lib_common * Integrate changes to WindowsAI to make ADO Build (#2886) * Revert "CMake cross-generator fixes (#2790)" This reverts commit dbe7d97fa1ab155f1309bced87199527e8f35bd2. * add additional suppress warning in onnx_proto * ignore /wd4996 warning * DML execution provider fixes * Revert "Revert "CMake cross-generator fixes (#2790)"" This reverts commit 1ae7b4bcbc02edc881ad28685da98e095dfceb17. * Update func signature of custom op function overloads * common devicehelpers fixes * Add pch.h for winml_lib_common * re-add winml_lib_common_dir/inc to include path for winml_adapter * User/orilevari/dml redist shared folder (#2890) * move dml nuget package directory up one level to make it shared between build flavors * Merge conflict fix * Revert "Merge conflict fix" This reverts commit 142fa72cf9ce4344ad717b50b7ea2b8582aadc7c. * Revert "Merge remote-tracking branch 'origin/master' into windowsai" This reverts commit 6e2126d46e5e5f564d65da37dd4f70c93dd81165, reversing changes made to b3f5583dc9249834b947c8ea905f6a98060d5bd6. * Make winml_test_common free of test macros (#2902) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Don't generate files outside the build root (#2914) * Don't generate files outside the build root * Add onnxruntime_EXTERNAL_DEPENDENCIES to WinML * Add DML depedency on RESTORE_PACKAGES * User/orilevari/fix yaml merge bugs (#2918) * Add winml test source parameter into cmake function (#2919) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Add arguments to unittest cmake functions * remove comment * Revert "Revert "Merge remote-tracking branch 'origin/master' into windowsai"" This reverts commit ade5abe72a4234fdbc3623093c61c02c6b0bdc26. * Fix breaks from merge with ORT master * Brianma/linux (#2917) * don't include windows.h in cross-plat header * add default case for switch statement * signed/unsigned mismatch fix Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> * User/sheilk/winml adapter c api (#2891) * Create winml adapter c api * fix build * make it build * move adapter into onnxruntime core/session * entry point not exported * minor changes * make model metadata work * make tests pass * implement all the model reflection apis on the adapter c abi * update the new ort interface to create a lotus ennvironment with a logging sink * start adding ort env * move all winml code into adapter folder/lib to isolate it * ensure a single logging manager at a time * start refactoring session * refactor session creation interface * add cpu and dml session option methods to adapter * finish session init * stub out interfaces in ort lib to perform similar mechanics of iinference session * enable profiling, and enable schema override * update session register graph transformers * turn back on custom registry for custom ops * Add sync api * add last c api stubs * should build... but all feature values are broken since this is in flight to moving all implementation details into ivalue * remove ep adapter header * Implement DML execution provider functions from adapter (#2846) * Implement DML execution provider functions from adapter * Use functions in OnnxruntimeEngine.cpp * make map/sequence type_infos freeable, and start implementing ivalue * make it build again * implement value methods * implement remaining methods * remove com adapter abi * check dml session * cache the allocator on ivalue * check if resource is cpu/gpu when access its mutable data * update tensor * mismatched parentheses * fix tensor base and binding obj * it evaluates tensors! sometimes... * minor fixes * enable gpu evals * wrapper all existing winml adapter apis with API_IMPL to try catch (#2854) * update winml... tensor strings are broken, need to template tensorbase to do different things for strings * make tensor strings work with 2 copies in/2 copies out * Fix tensor string and allocator bug * make maps work again... needs some fixes still * Make it build! * enable map inputs * map outputs * unbound outputs for sequences and maps * User/xianz/merge windowsai (#2883) * Packaging pipeline changes for VS 2019 (#2711) * Tiny fix to codegen * Simplify cache implementation and avoid static variables that may carry over between models * Extend DML kernels (#2641) * Additional DML operators * Check unsupported attributes and inputs * Address PR comments * Add kernel capability function used for partitioning, and re-enable stride-based int64 support based on value range * Fix test failures * Build fix * PR comments * Update Nuphar tutorial notebook (#2721) 1. Reflect int8 GEMV improvements for multi-threading from #2696 2. Add notes on multi-threading control using OpenMP 3. Add samples of running multi-isa AOT, and show int8 GEMM differences between AVX and AVX2 4. Add rnn_benchmark example to resolve #1993 * Add schema for new Qops (#2611) * Add schema for new Qops * adding shape inference + qlinearaveragepool * plus review comments * plus review comments * updates per review comments * plus review comments * [server] Add supposed for model_name and model_version as cli parameter (#2708) * remove 64bit warning message from python validation. (#2727) * MLAS: ARM64 build fix (#2734) fix bad usage of vreinterpret to cast vector element types * Fix broken python docs links (#2740) * Fix build on Mac OS (#2731) mac os ld doesn't support --while-archive, correct option is -all_load * fix ngraph wheel (#2737) * fix ngraph wheel 1.1.0 onnxruntime_ngraph wheel doesn't work * remove libdnnl.so in nGraph Libs * make it easy to compare * Split onnxruntime server to a separated folder (#2744) * Fix build for Python 3.8 (#2747) * Fix build for Python 3.8 * Update protobuf to 3.11.2 (#1928) Update protobuf to 3.11.2 (#1928) * Change default optimization level to All (from Basic) (#2745) * change default optimization level to All (from Basic) * fix test * fix c# test * Update numpy to 1.18 (#2758) * Update numpy to 1.18 * Pipeline changes for python 3.8 (#2753) 1. Pipeline changes for python 3.8 2. Fix a regression in setup.py which was just introduced in the previous commit. Please notice, we still haven't made python 3.8 + Windows + CUDA work. * Add basic stacktrace output for posix debug builds. (#2749) * [NupharEP] fix a race condition when multiple sessions running different models concurrently (#2772) * Revert "Change default optimization level to All (from Basic) (#2745)" This reverts commit 56bb503c2f26474b6613bcb2a198691a11dcef38. * Fix typo in error message (#2736) * Rename MKL-DNN to DNNL to fix broken link (#2730) * Fix nightly build version number issue * Pass BUILD_BUILDNUMBER to linux docker * Disable featurizers in python packages * Import more featurizers (#2781) Make kernels non-template. Add input constraint for learnt data. Add min_max_scalar_transformer, robust_scalar_transformer, inputation_marker_transfomer, label_encoder_transformer, missing_dummies_transformer along with tests. Advance Featurizers library commit. * Implement a more stable softmax (#2715) * Implement a more stable SoftMax e^x is represented as infinity if x is large enough, like 100.f. Infinity divided by Infinity is a NAN. Thus, softmax gets a NAN if one or more item are large enough. A math transform as below is leveraged to get a stable softmax: e^xi/(e^x1 + ...e^xn) = e^(xi - max) / (e^(x1 - max) + ... + e^(xn - max)) And for convenience, force max to 0.f if all xi are negative * Contributing: Fix a typo (#2784) * ACL EP GEMM improvements (#2780) When it is posible we use a fully connected layer instead of the gemm implementation. This will let the library use the best implementation based on the input data. * ACL EP convolution improvements (#2774) Added the optimized implementation for depthwise convolution for both ACL v19.02 and ACL 19.05. Also the pointwise convolution seems to be more optimal in the CPU implementation so we opted for that instead. * Add script for release Nuget validation (#2719) * Initial commit * Nits * Disable a test temporarily * Change working directory * Test * Add download python step * Test update * More changes * Fix space issue * Fix * Verify nuget signing * Fix * Spaces * PR feedback * Nit * Fix * Fix * Remove temporary changes * add uint8 support to where op (#2792) * Improve bert optimization script: (#2712) (1) Move input int64=>int32 conversion to embed layer fusion. (2) Output epsilon attribute for LayerNormalization fusion. * add session creation time cost. (#2798) * ML.NET team needs featurizers within a package (#2789) Add auto ml featurizers to Windows, MacOS as well as to GPU packaging-pipelines. * Initialize max of softmax with lowest of float (#2786) * MLAS: update SGEMM threading parameters (#2808) * add interface to copy batch tensors. (#2807) * add interface to copy batch tensors. * onnxruntime * speed up Windows TRT CI (#2811) * don't run cuda tests if building with tensorrt * remove unnecessary build options for win trt ci * refactor win gpu tensorrt ci yml * --numpy_version=1.17 * update * update * azcopy and cuda path * Update test data (#2356) * Add timeseries imputer transformer featurizer kernel (#2813) Make kernels non-template. Add input constraint for learnt data. Fixup tests. Add two more featurizers along with tests. Tests fail. min_max_scalar_transformer robust_scalar_transformer Fix tests serialized stream by prepending version bytes. Add inputation_marker_transfomer and the test. Fix up float/double type designations. Added label_encoder_transformer along with a test. string_throw case is broken at the momement. Fix labelencodertransfomer_test.cc string_throw case Rename maxabsscalertransformer_test.cc Add MissingDummiesTransformer along with the test. Update manifest. Add TimeSeriesImputerTransformer definition, implementation and tests * Fix memory leak in TRT (#2815) * fix memory leak issue * revert EP_FAIL on enueueV2 * Add manifest missing comma * Run static code analyzer on most of our code (#2817) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * update quantization doc (#2783) * update documentation for quantization script * plus some spell corrections * Filter CPU case for IsFloat16Supported (#2802) * update default optimization level + fix gemm_activation fusion (#2791) * update defualt optimization level + fix gemm_activation fusion * fix typo * add unit test and incorporate review comments * fix test comment * Fix dnnl wheel package name (#2823) * Append '-dnnl' to whl package name when --use_dnnl * Update build.py * Update Ubuntu & TensorRT version in README (#2820) Dockerfile.tensorrt is using nvcr.io/nvidia/tensorrt:19.09-py3 as base Image, update Ubuntu and TensorRT version according to https://docs.nvidia.com/deeplearning/sdk/tensorrt-container-release-notes/rel_19-09.html#rel_19-09 * Merge fixes * Add OneHotEncoder and HashOneHotEncoder kernels. (#2830) Add defs and imlementation for OneHotEncoders, adjuist date_time_transformer kernel and test. Add OneHotEncoder kernel test. Add HashOneHotVectorizerTransformer unit test. This does not link due to multiple definitions of functions that are included into header from a CPP file. * Upgrade gtest to the latest version (#2827) WinML would like to update the googletest submodule. They want some newer features (namely GTEST_SKIP to skip tests programmatically and be able to skip entire fixtures easily) and would need to update the submodule version. However, because the new version of code hit a bug in gcc, even though the bug is already fixed in the latest gcc but we're using gcc 4.8.x and it won't get patched for the bug, so we have to do a compromise, change our code a little bit to make it work. The gcc bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=51213 * Add support for int64_t for topk CPU. Fixes github issue #2806. (#2833) * Ignore allocator type in ExecutionProviders allocator map. Make default initialization of OrtMemoryInfo more clearly invalid. (#2768) * Remove allocator type from the key comparison in ExecutionProviders. Remove usage of DummyArena as it's no longer necessary. * Fix x86 tests where arena allocator is disabled. Make initialization of OrtMemoryInfo clearer by adding Invalid enum value. * Make OrtValueNameIdxMap::MaxIdx more intuitive. * Convert ExternalProject Featurizers into git submodule (#2834) Add git submodule for Featurizer library. Update cmake to build for git submodule. * add domain check for nodes + update documentation (#2831) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Fix Linux CUDA nuget packaging pipeline break * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * update optimization doc for BERT related fusions (#2819) * Add bert related transformers to doc * Add execution provider and comment for bert optimizations * Add comment about accuracy impact of approximation * Fix warnings that cause build to fail * MLAS: enable threading for quantized GEMMs (#2844) * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * remove old winmladapter cpp Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * move sequence implementation into ort lib... still commented out... need to turn back on... * begin sequence implementation * make maps and sequences work * fix broken tests * remove dead code * misc cleanup * CR feedback * User/xianz/winml adapter c api (#2869) * wrapper all existing winml adapter apis with API_IMPL to try catch * Return HR or Throw for WinML adapter APIs if failed * undo macro wrapper for two places * Wrap error macros around ort apis, too. * address CR feedback #2 * add more api throw/return macros * Revert changes no longer needed * revert changes to cxx api * format winml lib.ort and winml adapter * remove static pheonix singleton Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * missing use_dml check in winml_adapter_session (#2930) * --use_dnnl flag was mangled in merge (#2931) * use dml macro not wrapping custom registry code (#2934) * Disable LNK4199 winml_dll to enable cuda builds (#2936) * Disable LNK4199 in winml_dll * linkler->linker * LearningModelSessionAPITestGpu.CreateSessionWithCastToFloat16InModel should return DXGI_ERROR_UNSUPPORTED when FP16 not supported (#2937) * Disable LNK4199 in winml_dll * linkler->linker * Need to return DXGI_ERROR_UNSUPPORTED when Model does not support fp16 * Publish build symbols (#2939) * Publish build symbols * Don't upload PDBs for .exe files * Make x86 build (#2943) * fix last remaining size_t/int64_t warnings->errors (#2948) * TensorString, Sequences and Maps use the first allocator, but should use the cpu default allocator. (#2952) * fix tensor string allcoator * clean up default allocator usage for strings in winml lib/api.ort Co-authored-by: Ryan Lai <ryalai96@gmail.com> * Handle tensor shape of zero (#2954) Co-authored-by: Ryan Lai <ryalai96@gmail.com> * CR feedback (#2970) * CR feedback * fix weird formatting on privacy readme * Add 'All rights reserved.' everywhere * readd all rights reserved to winml_provider_factory.h * remove extra space in comment * remove extra whitespace * fixes post master merge * remove winml from nuget gpu pipeline * set IR VERSION on generated_model in rnn_benchmark (#2972) * Fix slice conformance failures (#2908) Co-authored-by: Adrian Tsai <adtsai@microsoft.com> Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Paul McDaniel <paul_mcdaniel@hotmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Dwayne Robinson <fdwr@hotmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com>
2020-02-05 01:12:19 +00:00
[threading(both)]
[marshaling_behavior(agile)]
[dualapipartition(1)]
runtimeclass TensorUInt16Bit : ITensor, [contract(MachineLearningContract, 2)]Windows.Foundation.IMemoryBuffer
{
static TensorUInt16Bit Create();
static TensorUInt16Bit Create(Windows.Foundation.Collections.IIterable<Int64> shape);
static TensorUInt16Bit CreateFromArray(Windows.Foundation.Collections.IIterable<Int64> shape, UInt16[] data);
static TensorUInt16Bit CreateFromIterable(Windows.Foundation.Collections.IIterable<Int64> shape, Windows.Foundation.Collections.IIterable<UInt16> data);
Windows.Foundation.Collections.IVectorView <UInt16> GetAsVectorView();
[contract(MachineLearningContract, 2)]
{
//! Creates a tensor object, allocates a buffer of size 'shape', and copies all of 'data' into it.
static TensorUInt16Bit CreateFromShapeArrayAndDataArray(Int64[] shape, UInt16[] data);
//! Creates a tensor object, creates a tensor of size 'shape', and uses the data in 'buffer'.
static TensorUInt16Bit CreateFromBuffer(Int64[] shape, Windows.Storage.Streams.IBuffer buffer);
}
}
[contract(MachineLearningContract, 1)]
INBOX_ONLY([static_name("Windows.AI.MachineLearning.ITensorInt16BitStatics", 98646293-266e-4b1a-821f-e60d70898b91)])
INBOX_ONLY([interface_name("Windows.AI.MachineLearning.ITensorInt16Bit", 98a32d39-e6d6-44af-8afa-baebc44dc020)])
Merge windowsai (winml layering) into master (#2956) * Initial Commit * Merged PR 3985217: add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc (#2346) add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc and violating our OS layering requirements. We linked against onecoreuap_apiset.lib in VB so we will continue doing this, but I am still unsure why not to link against onecore instead since that is where we ship. However, since Sheil is the owner of this code we will wait to discuss with him before changing anything. * Initial changes for layering * more snipping to get core into ort * update build instructions to include --build_shared_lib (#2358) * update build instructions to include --build_shared_lib * fix line breaks * Task 23998197: add winml_lib_core into onnnxruntime.dll (#2368) * Task 23998197: add winml_lib_core into onnnxruntime.dll * PR feedback build break on perf_test * return proper error when the model path isn't found (#2391) * LearningModelSession is cleaned up to use the adapter, and parts of b… (#2382) this is a big PR. we are going to move it up to layer_dev , which is still a L3 so we are still safe to do work there agile. we are going to move this into the L3 so that ryan can start doing intergration testing. we will pause for a full code review and integration test result prior to going into the L2. >>>> raw comments from previous commits >>> * LearningModelSession is cleaned up to use the adapter, and parts of binding are. * moved everything in the winmladapter made it all nano-com using, WRL to construct objects in the ORT side. base interfaces for everythign for winml to call cleaned up a bunch of winml to use the base interfaces. * more pieces * GetData across the abi. * renamed some namepsace cleaned up OrtValue cleaned up Tensor cleaned up custom ops. everything *but* learnignmodel should be clean * make sure it's building. winml.dll is still a monolith. * model moved over. everything builds clean. step ! * weak ref comment * Layer dev paulm (#2408) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * Layer dev paulm (#2414) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * Layer dev paulm (#2423) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * Layer dev paulm (#2424) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * Layer dev paulm (#2425) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Layer dev paulm (#2426) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * fixed map and sequence when passing stl types across the ABI . found a leak in nvidia driver, but skipped it. all winmlapitests pass now * Moved SessionOptions over to the abi * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * Register WinML TraceLogging provider on Onnxruntime.dll (#2455) * Register WinML TraceLogging provider on Onnxruntime.dll * Add ifdef to make sure trace logging provider has telemetry option when LAYERING_DONE * No need for ifdef for TraceLoggingOptionMicrosoftTelemetry * PR feedback * Move etw registration into lotus environment constructor and deresgister in lotus environment destructor * Brianma/cpuwinml (#2466) * allow building winml cpu without dml. * Brianma/breaks (#2469) * fix some more breaks * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * move dml checks out of winml and into the adapter * better error handling * Brianma/fi (#2470) * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * fix bad merge * Checking in a staging checkpoint point so that Ryan can work with me in parrallel * build break. * Brianma/testfails (#2473) * add missing ir version to dictvectorizer-string.onnx * add missing ir version to relu.onnx * add missing ir version to zipmap*onnx * add IR version to manually generated models * remove an unnecessary ifdef dml * Brianma/windowsai fi (#2475) * update dockerfiles/README (#2336) * Make elementwise op run 4 items per thread (#2335) Description: Describe your changes. Make elementwise op run 4 items per thread unroll for loop to leverage ILP remove unnessary N==0 check inside elementwise GPU kernel Motivation and Context Why is this change required? What problem does it solve? It can improve the performance of GPU elementwise ops. ~2% performance gain on popular NLP bert model. If it fixes an open issue, please link to the issue here. * Add CUDA GatherElements kernel (#2310) * Updates * Update test * Update * Updates * nits * PR feedback * Update * Update * PR feedback * PR comments * Update * Fix build * Fix build * Nits * Fix * Layer Normalization Fusion (#2319) basic layer normalization transform * Add FastGelu Cuda Op for Gelu and Add bias fusion (#2293) * Add FastGelu cuda op * Add AddBiasGelu for experiment * Revert "Add AddBiasGelu for experiment" This reverts commit 5c1ee019858c657e6bb75887265cb85675626e5b. * Add bias * Add unit tests * update comment * update script * fix build error * update coding style * update for CR feedback Enable half2 optimization only when cuda arch >= 7.0 * move _Tanh to common.cuh * implement CPU contrib OP Attention (#2333) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanUnusedInitializers. (#2320) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanupUnusedInitializers. This means initializers that have been replaced during graph optimizations are not left in the GraphProto when we save an optimized model. * Handle edge case where a model has an unused initializer with matching graph input by also removing the graph input. * Use non-const iterators in std::find_if calls to make centos build happy. * Nuget pipeline changes (#2305) 1. refactor the pipeline, remove some duplicated code 2. Move Windows_py_GPU_Wheels job to Win-GPU-CUDA10. We'll deprecated the "Win-GPU" pool 3. Delete cpu-nocontribops-esrp-pipeline.yml and cpu-nocontribops-pipeline.yml 4. In Linux nuget jobs, run "make install" before creating the package. So that extra RPAH info will be removed * Cuda Reverse Sequence Op, maping types of same size using same template function. (#2281) * Set ElementType to String type of node metadata, instead of byte[] (#2348) * Set ElementType to String type of node metadata, instead of byte[] * Fix spacing * Introduce PrimitiveType into a Type System along with an integer constant (#2307) Improve perf by avoiding GetType<T>() calls. Introduce MLTypeCallDispatcher to switch on Input Type. Add Tensor IsType<T>() fast method. * Fix/test dim value of 0 handling in a couple of places (#2337) * Update the CUDA Where implementation broadcasting logic to handle a dim with value of 0. Add unit test Also add unit test for unary op with dim value of 0 * Exclude ngraph from Where test with 0 dim. * Openvino EP R3.1 onnxrt server (#2357) * onnxrt server with OVEP * onnxrt server with OVEP * Update Dockerfile.server.openvino * onnxrt server OVEP fix reviews * onnxrt server OVEP fix reviews * Implement cuda nonzero op. (#2056) Implement cuda nonzero op. * Direct use python numpy array's memory if already contiguous. (#2355) * Direct use python numpy array's memory if already contiguous. This could greatly improve performance for session with large input, like big image 1920x1080 fastrcnn, 30~40% speed up could be achieved. * Add test case enforce contiguous/non-contiguos numpy array as inputs. * Add helper to create output to minimize binary size. (#2365) Add ConstEigenTensorMap typedef so we don't unnecessarily const_cast the const input Tensor. * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS (#2369) * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS * update * Add Tracelogging for profiling (#1639) Enabled only if onnxruntime_ENABLE_INSTRUMENT is ON * test bidaf with nuphar for avx target (#2370) increase nuphar test coverage a bit * Fix a bug in TLS refcount that may destabilized CUDA CI (#2374) * update output size calculation for resize (#2366) * change how output size is calculated for resize op * add tests for ver 10 resize * Extend OneHot CPU kernel to support more types (#2311) * Extend OneHot CPU kernel to support input int64_t, depth int32_t, output float * Skip BERT before the test data fix is picked up * Fix bug with Slice. Need to pass in flattened input dimensions so the initial offset into the input is calculated correctly. (#2372) * Add opset 11 version of Split to CUDA ops (#2376) Organize the CUDA ops definitions so all the opset 10 and 11 parts are together (same setup used for CPU ops) * Layer Norm Fusion Fix (#2379) * layer norm fusion fix * Add input shape check in code and unit tests * Fuse Add + Gelu (#2360) Implement the transformer to fuse add + gelu Implement the accurate kernel * Skip layer norm transform (#2350) * skip layer normalization transformer * Another try to stabilize CUDA CI (#2383) The root cause seems to be failure in CUDA dealloc when tear down. cudaFree return code was ignored before, so should the debug check. * fix BUILD.md typo (#2375) build.py: error: argument --config: invalid choice: 'RelWithDebugInfo' (choose from 'Debug', 'MinSizeRel', 'Release', 'RelWithDebInfo') * Fixed compilation with ngraph (#2388) * Fix reuse logic in allocation planner. (#2393) * Fix reuse logic in allocation planner. * PR comments * Add helpful comments * Don't allow reuse across string tensors. * [NupharEP] Multiple optimizations (#2380) Fuse transpose into MatMul Implement Pow and constant scalar simplification Vectorize ReduceMean Improve symbolic shape inference Minor updates for better debugging in fused function name * Avoid using the default logger in the graph lib and optimizers (#2361) 1. Use the session logger if it is available. 2. Don't disable warning 4100 globally. We should fix the warnings instead of disabling it. * Change CUDA implementation of Transpose to support all fixed size tensor types (#2387) * Change CUDA implementation of Transpose to not use a typed kernel so we can support more types with minimum binary size. Add support for 8, 16, 32 and 64 bit types. Add unit tests. Add method so the implementation can be called directly (will be used by CUDA Scan very soon). * Disable TensorRT for MLFloat16 and int8 unit tests. * Address PR comment and add support for calling cublas implementation if type is mlfloat16. * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. (#2398) * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. * [NupharEP] force some low/zero cost ops to be inlined (#2409) * fix cross compile bug (#2415) * Minor optimization: if a node has already been placed, there's no need to find a kernel for it. (#2417) * Add Reshape Fusion (#2395) * Add reshape fusion * Add some comments * update comments * update comment format * update according to feedback * update for recent logger change * fix build error * (1) Support both input and output edges in find path in graphutils (2) Add a test case of only one constant initializer of Concat input. (3) Refactor ReshapeFusion class to allow add more subgraph fusion in the future. * fix error * (1) loose constraint on initializer: non constant is allowed for reshape fusion. (2) Change versions type to vector. (3) Add logging. (4) Return false when multiple output edges matched in FindPath. Add comments. * only allow one direction (input or output) in FindPath * [NupharEP] Update notebook and docker image (#2416) Add BERT squad in Nuphar tutorial Enhance speed comparsion readability * Fix the issue in matmul_add_fusion (#2407) Fix the issue in matmul_add_fusion If Muatmul + Add has shape [K] * [K, N], reset it to [1, K] * [K, N] will make the output shape to [1, N] will also requires a reshape on the output. Fix: just remove the shape reset to not fuse it. Add a negative test case for matmul+add fusion * feat(treeregressor): Update TreeEnsembleRegressor for type support (#2389) Updates the `TreeEnsembleRegressor` to allow for `double`, `float`, `int64`, and `int32` inputs to match the upstream specification. Signed-off-by: Nick Groszewski <nicholas.groszewski@capitalone.com> * onnxrt server documentation update (#2396) * Added support for Pad-2 operator in OpenVINO-EP (#2405) * Add CUDA If operator. (#2377) * Add CUDA If operator. Uses CPU operator for implementation. By adding a CUDA version the inputs/outputs (with the exception of the 'cond' input) stay on GPU, and no other logic is required to avoid a copy to CPU across the control flow node. * Improved documentation for onnxruntime::utils::SwapByteOrderCopy(), added precondition check. * Fix the type constraints on CUDA If operator to exclude strings. (#2431) * add Im2col<uint8_t> (#2438) * Adjust codegen vectorization width from target (#2439) * Adjust codegen vectorization width from target * Add CUDA Scan operator. (#2403) * Add Scan CUDA op. Uses CPU implementation for logic. Added some device specific functors for handling when data needs to be manipulated on a different device. Added ability to override the materialization logic in the OrtValue slicer so DML can plugin their handling. * Fix Windows GPU C API packaging pipeline failure (#2440) Fix Windows GPU C API packaging pipeline failure (#2440) * Correctly handle implicit inputs for fused nodes (#2390) * Correctly handle implicit inputs for fused nodes Previously, nuphar's partitioning function didn't include node's implicit inputs into the inputs list of MetaDef, and hence a crash was triggered in the onnx graph checker. This commit fixed the issue. Furthermore, it also fixed a related issue where we didn't add implicit inputs into graph_inputs_excluding_initializers_ in Graph::SetGraphInputsOutputs. the issue was that graph_inputs_including_initializers_ populated by SetInputs (e.g. called by FunctionImpl::FunctionImpl) may contain implicit inputs which were not of any node's initializers in the graph. Because they were not part of any initializers, these implicit inputs couldn't be visited by going through all nodes' inputs. Consequently, they would *not* be added into graph_inputs_excluding_initializers_. We fixed the issue by first copying the populated graph_inputs_including_initializers_ into graph_inputs_excluding_initalizers_, which then had both initializers and non-initializers as its initial content. Later, we erase initializers from the list. In this way, we can ensure all implicit inputs to remain in graph_inputs_excluding_initializers_. * refined comments and fixed duplicates Address CR by revisiting comments in terms of implicit inputs Also fixed an issue by skipping duplicates while copying inputs from graph_inputs_including_initializers_. * address CR explain why we need to collect nodes' implicit inputs * don't rely on pointer values for iterating std::set Previously, openvino relied on iterating a set of NodeArg pointers to construct inputs and outputs for a fused graph. It could cause non-determinism. The reason was that although iterating std::set by itself is stable, pointer values of NodeArgs may vary. Consequently, we could end up visiting the set's elements in different orders for different runs for the same test, which resulted in constructing inputs (and outputs) with different orders to the fused graph. For example, for the same test, we may have inputs [A, B] in some runs but inputs[B, A] in others. Let's use std::string as the key type to avoid such nondeterminism. This commit also added implicit inputs into meta->inputs while returning the capability from the openvino provider. * Fixed another latent issue in openvino's GetCapability function The issue was that we couldn't simply erase fused_inputs and fused_outputs while iterating the nodes. For example, an output NodeArg may have multiple uses, and it's wrong if we erase it from fused_outputs when we encounter only one of its uses as input. * Remove DeviceAllocatorRegistry class (#2451) Remove DeviceAllocatorRegistry class * CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library. * Parallel Gelu with ParallelFor (#2399) Parallel Gelu to get better performance for Gelu * Clean up build.py (#2446) * Pull the latest image before running docker build * Fuse SkipLayerNorm with Bias (#2453) Fuse SkipLayerNorm with Bias * Allow more than one invocation of CreateEnv in the same process. (#2467) * Allow more than one invocation of CreateEnv in the same process. * Fix centos build * Symbolic shape inference improvements: (#2460) * Symbolic shape inference improvements: - add a mode to guess unknown ops' output rank - add support for GatherND - add support for If - fix a bug in get_int_values when then tensor rank > 1D, by treating it as no sympy data - add symbol to literal merge when ONNX silently merges dims - fix a bug in Concat when input dim is 0 - fix a bug in ConstantOfShape that computed dim is not updated - add support for dynamic shape in ConstantOfShape - fix a bug in Loop output shape that loop iterator dim is not inserted at dim 0 - add support for dynamic padding in Pad - add support for dynamic shape in Reshape - add support for Resize with opset > 10, by treating output dims as dynamic - fix a bug in Slice when starts/ends are dynamic - restrict input model to opset 7 and above - make output model optional to avoid disk write when testing Run model tests for symbolic shape inference Reduce 2GB docker image size of nuphar * add additional test data set for nuget pipeline (#2448) * add SAS token to download internal test data for nuget pipeline * update azure endpoint * fix keyvault download step * fix variable declaration for secret group * fix indentation * fix yaml syntax for variables * fix setting secrets for script * fix env synctax * Fix macos pipeline * attempt to add secrets to windows download data * fix mac and win data download * fix windows data download * update test data set url and location * Revert "Brianma/windowsai fi (#2475)" This reverts commit 5780b864a15513fda4eadbfc2b5345fefe70b5ec. * Add scenario tests (#2457) * Add scenario tests * Remove TODO from model license * Add winml_api test dependency * fix model load test. fi from master changed the constructor (#2483) * make api tests all pass (#2486) * fix bad merge * fix bad model merge * Layer dev paulm (#2492) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * Rename ambiguous header (#2489) * fix one more missing IR version model (#2500) * add missing IR version to 4 more models used by scenario tests (#2501) * Add CLI parameters to test runner, build WinML in ARM and x86 CI (#2479) * Support test parameters through CLI arguments * Add WinML do Windows x86/ARM CI builds * Code style fixes * Update googletest Remove GPUTEST macros everywhere now that GTEST_SKIP is supported * Refactor main.cpp * Build scenario tests without DML * Link scenario tests to DML when it's enabled (#2502) * Layer dev release pipeline (#2488) Adds winml binaries to existing cpu nuget package, and creates new gpu dml nuget package with winml binaries and DML EP. * Layer dev paulm (#2506) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * Remove usage of IOBinding in WinML and use C_API Run method (#2504) * remove usage of iobinding * Change data structure to use vector of Ort::Values * Polish bind input / output * Use C APIrun method * Update providers on evaluate getresults * Remove run and IObinding interface from WinMLAdapter * Remove use of IObinding * bind unbound outputs code moved to learningmodelbinding * clean up unneeded istensor adapter function * Fix comment * Check if session is closed before binding and clearing * PR feedback * Layer dev paulm (#2507) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * Make tests dependend on winml_dll (#2509) * add dml binaries to DirectML package and be more explicit about condition variables (#2520) * re-enable warnings for winml builds and fix the warnings that were hiding (#2526) * turn devmode back on for winml builds * fix some warnings. include protobuf in a way that disables some warnings * undo protobufhelpers changes and just ignore 4100 errors in pb code * attempt to isolate protobufhelpers errors * add template specialization for getting tensor proto data * Layer dev paulm (#2533) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * Spawn child process to run DeviceLostRecovery scenario test (#2530) * Spawn child process to run DeviceLostRecovery scenario test * Layer dev paulm (#2536) ori said yes * add missing namespace to winml_trace_logging_provider in lotusenvironment.h (#2542) * Handle exception thrown from all apis in WinMLAdapter (#2539) * various changes to unblock windowsai ADO build * Fix custom ops scenario tests (#2562) * Do not shutdown protobuf after ort environment gets destroyed. Lazy load lotus environment first time it is needed * comment typo * pr comment about calling phoenix singleton * Make lotus_environment static in winmladapter * Layer dev paulm (#2567) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * making windowsAI azure dev ops work. * code review comments. * revert changes * Cmake and preprocessor fixes that where uncovered by building on agents without DML available via SDK * Layer dev dml delayload (#2580) * Brianma/cpu (#2583) * don't include dml stuff in cpu builds * tests that link the image lib also need the telemetry lib now * Throw Winml_err_invalid_binding if binding gpu resource on cpu device (#2589) * Throw Winml_err_invalid_binding if binding gpu resource on cpu device * PR comments. No need to query executionprovider for is gpu device * User/xianz/ortthrow (#2596) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * User/xianz/ortthrow (#2599) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * add status error message * Remove uwp onsuspending winrt call because logruntimeperf is getting removed (#2630) * User/xianz/dedup telemetry (#2631) * investigate duplication of telemetry in winml and ort * remove winml telemetry events * telemetry executionProviderEvent * remove unneccessary file and refactor code little bit * Revert back TelemetryEvent, which send up ETW event. * merge changes from layer_dev to windowsai (#2638) * Remove underscore from googletest names (#2616) * Fix leaking memory allocator Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24278761 and https://microsoft.visualstudio.com/OS/_workitems/edit/24330198 * Explicitly initialize Ort::Value with nullptr * Cache WinML adapter * bad merge * define private version of dxcore enum that is added in 19H1 SDK. (#2654) * add comment for explaning private definition of dxcore d3d feature level ennum value. (#2672) * do not package directml.pdb for redist packages. (#2676) * Fix leaking operator registry (#2645) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24354916 * User/orilevari/windowsai master merge (#2674) merge resolutions included pulling in telemetry logic that was merged to master and not windowsai and dereferencing InferenceSession::sessionstate now that it is a unique pointer * Delete Ort Allocator in LearningModelBinding (#2653) * Delete OrtAllocator in LearningModelBinding * PR comments to make Ort::Allocator a smart pointer * Small comment change * PR feedback to clean up code * PR feedback on move semantics * Clean up std::move * Fix memory leaks (#2679) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24356109, https://microsoft.visualstudio.com/OS/_workitems/edit/24388361 and https://microsoft.visualstudio.com/OS/_workitems/edit/24388596 * various changes to properly organize and skip GPU tests. For now for No DML builds we will not run GPU tests at all. In the future we should adapt the tests to expect the appropiate errors. (#2695) * Windowsai without fi (#2701) * Disable Attention fusion tests when DISABLE_CONTRIB_OPS is defined (#2529) * Setup java ci (#2528) * Add provision in ORT for session options to be parsed when available via model file (#2449) * Initial commit * Fix gitmodules * Nits * Nits * Updates * Update * More changes * Updates * Update * Some updates * More changes * Update * Update * Merge * Update * Updates * More changes * Update * Fix nits * Updates * Fix warning * Fix build * Add comment * PR feedback * PR feedback * Updates * Updates * Update * More changes * Fix build break * Comment test for now * Updates * Updates * PR feedback * Updates * Nits * Add tests * Fix build * Fix build * Fix build * Fix build break * Fix build * Nits * PR feedback * More change * Expose GetSessionOptions in pybind logic and add unit test for python * Fix build * PR feedback * PR feedback * Revert "Disable thread pool creation when enabled OpenMP (#2485)" (#2535) This reverts commit 7c7d5a149c9ed52eec67304bae5c4b132166a8a1. * Add dynamic shape support in TensorRT execution provider (#2450) * remove onnx-tensorrt submodule * add new onnx-tensorrt submodule (experiment) for trt6 * update engine build for trt6 * update compile and compute for tensorrt6.0 * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * switch to onnx-tensorrt master for TensorRT6' * Update tensorrt_execution_provider.cc * Handle dynamic batch size and add memcpy in TensorRT EP * update test cases * Update tensorrt_execution_provider.cc * update onnx-tensorrt submodule * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.ubuntu_tensorrt * Update run_dockerbuild.sh * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update concat_op_test.cc * Update tensorrt_execution_provider.cc * Upgrade TensorRT to version 6.0.1.5 * Update onnxruntime_providers.cmake * Update CMakeLists.txt * Update reduction_ops_test.cc * Update install_ubuntu.sh * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.tensorrt * Update BUILD.md * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update onnxruntime_providers.cmake * Update install_ubuntu.sh * Update install_ubuntu.sh * Update gemm_test.cc * Update gather_op_test.cc * Update CMakeLists.txt * Removed submodule * update onnx-tensorrt submodule * update header file * Removed submodule * add submodule onnx-tensorrt kevin's branch shape-test' * add debugging code * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * merge master * Removed submodule * update onnx-tensorrt submodule * add more changes for dynamic shapes * Update tensorrt_execution_provider.cc * update for dynamic shape * update dynamic shape processing * fix logger issue * remove submodule onnx-tensorrt * add submodule onnx-tensorrt * add env variable min_subgraph_size * remove redundency * update document * use onnxruntime::make_unique * fix multi-run issue * remove some tests to save CI build time * Add dynamic shape test * Update TensorRT-ExecutionProvider.md * Add example of running Faster R-CNN model on TensorRT EP * Add more details on env variables * update environment variables * Update tensorrt_basic_test.cc * Update model tests * Update tensor_op_test.cc * remove --use_full_protobuf * Update build.py * User/xianz/telemetry (#2458) * enabme telemetry * enable telemetry * set enable telemetry as default * for debugging * remove log and set disable telemetry as default back * delete private file while testing * resolve comment: mainly add license header, rename macro and update docs * rewording in privacy.md * Fix integer overflow in cuda NonMaxSuppression implementation (#2540) * add test case that should pass but fail * fix nms * extract int_max_output_boxes_per_class * Introduce container type runtime checks and other improvements (#2522) Rework TensorSeq in a manner consistent with Tensor and SparseTensor in terms of type system setup. Reduce templating. Introduce helpers to ensure the same data type. Make OrtValue __dtor not virtual. Introduce ContainerChecker * Fix C API tests for centos and mac (#2544) * change c++14 to c++11 * add ld lib path for centos * enable csharp tests on macos * fix C API test on MacOS + fix manylinux dotnet install * fix manylinux dotnet install * fix lib link * Add back executable bit to build.py * Fix a bug handling negative begin pad values in Pad op (#2550) * Fix bug in Pad op * Update * DNNL CMAKE update (#2548) * Fix android build (#2558) * Update win-x86-ci.yml (#2557) Fix build pipeline break * Re-enable Windows C# tests (#2564) * disable onnx_test_runner -x invocations for dnnl (#2568) * Allow sequence length to be symbolic (#2559) * setup java ci mac (#2570) * make layernorm fusion to support opset 11 (#2545) * Fix a warning found in the latest VS release * Add more check on SkipLayerNorm and BiasGelu fusion (#2574) * Fix file not found error during docker build. (#2569) * Add ConvTranspose1D (#2578) * Ryanunderhill/packagename test (#2582) * [Nuphar EP] fixes for some object detection models (#2581) Update notebook tutorial with multi-threaded int8 GEMM from #2517 * EmbedLayerNormalization Fusion Improvement (#2553) Embedding layer norm fusion improvements - add more checks * Update version (#2584) * Temporarily exclude vgg19 test from Python backend test 1. temporarily exclude vgg19 test which comsumes too much memory, run out of memory on Upsquared device. Single test pass for vgg19, need furture investigation (#2588) 2. Update docker file to decrease the docker image size * Update docs for Android NNAPI EP (#2586) * Fix lto bug for protobuf and ubuntu * add path to build dir before test run (#2590) * Add missig env variables for mac pipeline test (#2595) * Fixed an issue in updating realized dims (#2597) when we update realized dims for scan's output, the sliced axis also needs to be inclusive, i.e. we should check with "dim >= insert_inclusive_axis", because the offset in the symbols are based on Scan sugraph. Otherwise, we would end up with shape mismatch later. * Java API for onnxruntime (#2215) * Add support for opset 11 in reshape fusion (#2592) Support opset verion 11 in reshape fusion * Rename automl python tools folder to featurizer_ops. (#2593) * Support opset 11 subgraph of Squad model in Embed Layer Normalization (#2605) Support opset 11 Squad model that is exported from PyTorch nightly. The embed layer uses Range op which is missed in the transformer. * symbolic shape inference: fix warnings in GPT-2 model (#2608) And revise nuphar perf test on BERT squad * Dump subgraph ID and fused graph ID (#2607) * Dump subgraph ID and fused graph ID Dump subgraph ID and fused graph ID for better debugging * Remove local static fused_count added a field global_fused_count_ to NupharExecutionProvider class * EmbedLayerNormalization Fusion For Dynamic Squad Model Opset 10 (#2613) Support subgraph of SQuAD model exported from pytorch with dynamic input axes * Allow providers to be set for InferenceSession at construction (#2606) * Remove unnecessary parameter in some places in GatherElements implementation (#2612) * Remove unnecessary parameter in some places * Update * Update * Make sure fenced tensor could not reuse other tensor. (#2561) Fix random error caused by this. * Improve Embed Layer Norm Fusion for SQuAD with static input shape (#2621) * fix float16 comparison in initializer (#2629) * epsilon attribute for layernormalization fusion (#2639) * removed unnecessary batch file and fix path (#2640) * Add shape inference to ConvTransposeWithDynamicPads schema (#2632) * Improve cuda expand() opeator's performance. (#2624) * Cuda pad optimize when no padding is needed. (#2625) * Shortcut cuda Pad() when no padding is needed. * Optimize cuda scatter() on 2D compatible. (#2628) * Optimize cuda scatter() on 2D compatible. * Add some comments. * fix build error for ARM (#2648) * Improve performance of resize() in Nearest mode (#2626) Special treatment for 2D, check same size as input image. And in 2d kernel, template use_expolation. * Fix memory exception in Layer Norm Fusion (#2644) * Windows CI changes(#2650) * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Revert "Windowsai without fi (#2701)" This reverts commit 285d4c85ff5c4e265f963208170304ef3461e684. * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Deref unique pointer for session_state * send shutdown event when dll is unloaded and EvaluationStop, SessionC… (#2704) * send shutdown event when dll is unloaded and EvaluationStop, SessionCreationStart Events. * Add EvalutationStart Event * add comment * use correct type for for loop (#2755) * ARM CI (#2759) * Set ARM agent pool * Set CMake generator to VS 2019 in ARM * Use system-wide CMake instead of custom version Our custom version is too old for VS 2019 * Use DML and build shared lib in ARM CI * Restore nuget packages in ARM CI * Disable DML * Refactor ARM debug/release builds * Use system packaged Python version * Remove hardcoded Python path * Downgrade Python to 3.7 for build * Remove explicit CMake path * Fix invalid JSON in cgmanifest.json (#2760) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * Remove usage of WHOLEARCHIVE in WinML CMake and add WinMLAdapterFactory (#2726) * Remove usage of WHOLEARCHIVE in WinMLAdapter CMake and add WinMLAdapterFactory * PR feedback, no need for dll(export) since using def file * PR comments * Small comment in gen_def.py * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * Move winml_provider_factory.h to proper location (#2801) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Filter CPU case for IsFloat16Supported (#2802) * Merge fixes * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * Fix warnings that cause build to fail * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * Make DML include path global (#2882) * Make DML include path global * Add generated cppwinrt headers to winml_lib_common * Integrate changes to WindowsAI to make ADO Build (#2886) * Revert "CMake cross-generator fixes (#2790)" This reverts commit dbe7d97fa1ab155f1309bced87199527e8f35bd2. * add additional suppress warning in onnx_proto * ignore /wd4996 warning * DML execution provider fixes * Revert "Revert "CMake cross-generator fixes (#2790)"" This reverts commit 1ae7b4bcbc02edc881ad28685da98e095dfceb17. * Update func signature of custom op function overloads * common devicehelpers fixes * Add pch.h for winml_lib_common * re-add winml_lib_common_dir/inc to include path for winml_adapter * User/orilevari/dml redist shared folder (#2890) * move dml nuget package directory up one level to make it shared between build flavors * Merge conflict fix * Revert "Merge conflict fix" This reverts commit 142fa72cf9ce4344ad717b50b7ea2b8582aadc7c. * Revert "Merge remote-tracking branch 'origin/master' into windowsai" This reverts commit 6e2126d46e5e5f564d65da37dd4f70c93dd81165, reversing changes made to b3f5583dc9249834b947c8ea905f6a98060d5bd6. * Make winml_test_common free of test macros (#2902) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Don't generate files outside the build root (#2914) * Don't generate files outside the build root * Add onnxruntime_EXTERNAL_DEPENDENCIES to WinML * Add DML depedency on RESTORE_PACKAGES * User/orilevari/fix yaml merge bugs (#2918) * Add winml test source parameter into cmake function (#2919) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Add arguments to unittest cmake functions * remove comment * Revert "Revert "Merge remote-tracking branch 'origin/master' into windowsai"" This reverts commit ade5abe72a4234fdbc3623093c61c02c6b0bdc26. * Fix breaks from merge with ORT master * Brianma/linux (#2917) * don't include windows.h in cross-plat header * add default case for switch statement * signed/unsigned mismatch fix Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> * User/sheilk/winml adapter c api (#2891) * Create winml adapter c api * fix build * make it build * move adapter into onnxruntime core/session * entry point not exported * minor changes * make model metadata work * make tests pass * implement all the model reflection apis on the adapter c abi * update the new ort interface to create a lotus ennvironment with a logging sink * start adding ort env * move all winml code into adapter folder/lib to isolate it * ensure a single logging manager at a time * start refactoring session * refactor session creation interface * add cpu and dml session option methods to adapter * finish session init * stub out interfaces in ort lib to perform similar mechanics of iinference session * enable profiling, and enable schema override * update session register graph transformers * turn back on custom registry for custom ops * Add sync api * add last c api stubs * should build... but all feature values are broken since this is in flight to moving all implementation details into ivalue * remove ep adapter header * Implement DML execution provider functions from adapter (#2846) * Implement DML execution provider functions from adapter * Use functions in OnnxruntimeEngine.cpp * make map/sequence type_infos freeable, and start implementing ivalue * make it build again * implement value methods * implement remaining methods * remove com adapter abi * check dml session * cache the allocator on ivalue * check if resource is cpu/gpu when access its mutable data * update tensor * mismatched parentheses * fix tensor base and binding obj * it evaluates tensors! sometimes... * minor fixes * enable gpu evals * wrapper all existing winml adapter apis with API_IMPL to try catch (#2854) * update winml... tensor strings are broken, need to template tensorbase to do different things for strings * make tensor strings work with 2 copies in/2 copies out * Fix tensor string and allocator bug * make maps work again... needs some fixes still * Make it build! * enable map inputs * map outputs * unbound outputs for sequences and maps * User/xianz/merge windowsai (#2883) * Packaging pipeline changes for VS 2019 (#2711) * Tiny fix to codegen * Simplify cache implementation and avoid static variables that may carry over between models * Extend DML kernels (#2641) * Additional DML operators * Check unsupported attributes and inputs * Address PR comments * Add kernel capability function used for partitioning, and re-enable stride-based int64 support based on value range * Fix test failures * Build fix * PR comments * Update Nuphar tutorial notebook (#2721) 1. Reflect int8 GEMV improvements for multi-threading from #2696 2. Add notes on multi-threading control using OpenMP 3. Add samples of running multi-isa AOT, and show int8 GEMM differences between AVX and AVX2 4. Add rnn_benchmark example to resolve #1993 * Add schema for new Qops (#2611) * Add schema for new Qops * adding shape inference + qlinearaveragepool * plus review comments * plus review comments * updates per review comments * plus review comments * [server] Add supposed for model_name and model_version as cli parameter (#2708) * remove 64bit warning message from python validation. (#2727) * MLAS: ARM64 build fix (#2734) fix bad usage of vreinterpret to cast vector element types * Fix broken python docs links (#2740) * Fix build on Mac OS (#2731) mac os ld doesn't support --while-archive, correct option is -all_load * fix ngraph wheel (#2737) * fix ngraph wheel 1.1.0 onnxruntime_ngraph wheel doesn't work * remove libdnnl.so in nGraph Libs * make it easy to compare * Split onnxruntime server to a separated folder (#2744) * Fix build for Python 3.8 (#2747) * Fix build for Python 3.8 * Update protobuf to 3.11.2 (#1928) Update protobuf to 3.11.2 (#1928) * Change default optimization level to All (from Basic) (#2745) * change default optimization level to All (from Basic) * fix test * fix c# test * Update numpy to 1.18 (#2758) * Update numpy to 1.18 * Pipeline changes for python 3.8 (#2753) 1. Pipeline changes for python 3.8 2. Fix a regression in setup.py which was just introduced in the previous commit. Please notice, we still haven't made python 3.8 + Windows + CUDA work. * Add basic stacktrace output for posix debug builds. (#2749) * [NupharEP] fix a race condition when multiple sessions running different models concurrently (#2772) * Revert "Change default optimization level to All (from Basic) (#2745)" This reverts commit 56bb503c2f26474b6613bcb2a198691a11dcef38. * Fix typo in error message (#2736) * Rename MKL-DNN to DNNL to fix broken link (#2730) * Fix nightly build version number issue * Pass BUILD_BUILDNUMBER to linux docker * Disable featurizers in python packages * Import more featurizers (#2781) Make kernels non-template. Add input constraint for learnt data. Add min_max_scalar_transformer, robust_scalar_transformer, inputation_marker_transfomer, label_encoder_transformer, missing_dummies_transformer along with tests. Advance Featurizers library commit. * Implement a more stable softmax (#2715) * Implement a more stable SoftMax e^x is represented as infinity if x is large enough, like 100.f. Infinity divided by Infinity is a NAN. Thus, softmax gets a NAN if one or more item are large enough. A math transform as below is leveraged to get a stable softmax: e^xi/(e^x1 + ...e^xn) = e^(xi - max) / (e^(x1 - max) + ... + e^(xn - max)) And for convenience, force max to 0.f if all xi are negative * Contributing: Fix a typo (#2784) * ACL EP GEMM improvements (#2780) When it is posible we use a fully connected layer instead of the gemm implementation. This will let the library use the best implementation based on the input data. * ACL EP convolution improvements (#2774) Added the optimized implementation for depthwise convolution for both ACL v19.02 and ACL 19.05. Also the pointwise convolution seems to be more optimal in the CPU implementation so we opted for that instead. * Add script for release Nuget validation (#2719) * Initial commit * Nits * Disable a test temporarily * Change working directory * Test * Add download python step * Test update * More changes * Fix space issue * Fix * Verify nuget signing * Fix * Spaces * PR feedback * Nit * Fix * Fix * Remove temporary changes * add uint8 support to where op (#2792) * Improve bert optimization script: (#2712) (1) Move input int64=>int32 conversion to embed layer fusion. (2) Output epsilon attribute for LayerNormalization fusion. * add session creation time cost. (#2798) * ML.NET team needs featurizers within a package (#2789) Add auto ml featurizers to Windows, MacOS as well as to GPU packaging-pipelines. * Initialize max of softmax with lowest of float (#2786) * MLAS: update SGEMM threading parameters (#2808) * add interface to copy batch tensors. (#2807) * add interface to copy batch tensors. * onnxruntime * speed up Windows TRT CI (#2811) * don't run cuda tests if building with tensorrt * remove unnecessary build options for win trt ci * refactor win gpu tensorrt ci yml * --numpy_version=1.17 * update * update * azcopy and cuda path * Update test data (#2356) * Add timeseries imputer transformer featurizer kernel (#2813) Make kernels non-template. Add input constraint for learnt data. Fixup tests. Add two more featurizers along with tests. Tests fail. min_max_scalar_transformer robust_scalar_transformer Fix tests serialized stream by prepending version bytes. Add inputation_marker_transfomer and the test. Fix up float/double type designations. Added label_encoder_transformer along with a test. string_throw case is broken at the momement. Fix labelencodertransfomer_test.cc string_throw case Rename maxabsscalertransformer_test.cc Add MissingDummiesTransformer along with the test. Update manifest. Add TimeSeriesImputerTransformer definition, implementation and tests * Fix memory leak in TRT (#2815) * fix memory leak issue * revert EP_FAIL on enueueV2 * Add manifest missing comma * Run static code analyzer on most of our code (#2817) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * update quantization doc (#2783) * update documentation for quantization script * plus some spell corrections * Filter CPU case for IsFloat16Supported (#2802) * update default optimization level + fix gemm_activation fusion (#2791) * update defualt optimization level + fix gemm_activation fusion * fix typo * add unit test and incorporate review comments * fix test comment * Fix dnnl wheel package name (#2823) * Append '-dnnl' to whl package name when --use_dnnl * Update build.py * Update Ubuntu & TensorRT version in README (#2820) Dockerfile.tensorrt is using nvcr.io/nvidia/tensorrt:19.09-py3 as base Image, update Ubuntu and TensorRT version according to https://docs.nvidia.com/deeplearning/sdk/tensorrt-container-release-notes/rel_19-09.html#rel_19-09 * Merge fixes * Add OneHotEncoder and HashOneHotEncoder kernels. (#2830) Add defs and imlementation for OneHotEncoders, adjuist date_time_transformer kernel and test. Add OneHotEncoder kernel test. Add HashOneHotVectorizerTransformer unit test. This does not link due to multiple definitions of functions that are included into header from a CPP file. * Upgrade gtest to the latest version (#2827) WinML would like to update the googletest submodule. They want some newer features (namely GTEST_SKIP to skip tests programmatically and be able to skip entire fixtures easily) and would need to update the submodule version. However, because the new version of code hit a bug in gcc, even though the bug is already fixed in the latest gcc but we're using gcc 4.8.x and it won't get patched for the bug, so we have to do a compromise, change our code a little bit to make it work. The gcc bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=51213 * Add support for int64_t for topk CPU. Fixes github issue #2806. (#2833) * Ignore allocator type in ExecutionProviders allocator map. Make default initialization of OrtMemoryInfo more clearly invalid. (#2768) * Remove allocator type from the key comparison in ExecutionProviders. Remove usage of DummyArena as it's no longer necessary. * Fix x86 tests where arena allocator is disabled. Make initialization of OrtMemoryInfo clearer by adding Invalid enum value. * Make OrtValueNameIdxMap::MaxIdx more intuitive. * Convert ExternalProject Featurizers into git submodule (#2834) Add git submodule for Featurizer library. Update cmake to build for git submodule. * add domain check for nodes + update documentation (#2831) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Fix Linux CUDA nuget packaging pipeline break * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * update optimization doc for BERT related fusions (#2819) * Add bert related transformers to doc * Add execution provider and comment for bert optimizations * Add comment about accuracy impact of approximation * Fix warnings that cause build to fail * MLAS: enable threading for quantized GEMMs (#2844) * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * remove old winmladapter cpp Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * move sequence implementation into ort lib... still commented out... need to turn back on... * begin sequence implementation * make maps and sequences work * fix broken tests * remove dead code * misc cleanup * CR feedback * User/xianz/winml adapter c api (#2869) * wrapper all existing winml adapter apis with API_IMPL to try catch * Return HR or Throw for WinML adapter APIs if failed * undo macro wrapper for two places * Wrap error macros around ort apis, too. * address CR feedback #2 * add more api throw/return macros * Revert changes no longer needed * revert changes to cxx api * format winml lib.ort and winml adapter * remove static pheonix singleton Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * missing use_dml check in winml_adapter_session (#2930) * --use_dnnl flag was mangled in merge (#2931) * use dml macro not wrapping custom registry code (#2934) * Disable LNK4199 winml_dll to enable cuda builds (#2936) * Disable LNK4199 in winml_dll * linkler->linker * LearningModelSessionAPITestGpu.CreateSessionWithCastToFloat16InModel should return DXGI_ERROR_UNSUPPORTED when FP16 not supported (#2937) * Disable LNK4199 in winml_dll * linkler->linker * Need to return DXGI_ERROR_UNSUPPORTED when Model does not support fp16 * Publish build symbols (#2939) * Publish build symbols * Don't upload PDBs for .exe files * Make x86 build (#2943) * fix last remaining size_t/int64_t warnings->errors (#2948) * TensorString, Sequences and Maps use the first allocator, but should use the cpu default allocator. (#2952) * fix tensor string allcoator * clean up default allocator usage for strings in winml lib/api.ort Co-authored-by: Ryan Lai <ryalai96@gmail.com> * Handle tensor shape of zero (#2954) Co-authored-by: Ryan Lai <ryalai96@gmail.com> * CR feedback (#2970) * CR feedback * fix weird formatting on privacy readme * Add 'All rights reserved.' everywhere * readd all rights reserved to winml_provider_factory.h * remove extra space in comment * remove extra whitespace * fixes post master merge * remove winml from nuget gpu pipeline * set IR VERSION on generated_model in rnn_benchmark (#2972) * Fix slice conformance failures (#2908) Co-authored-by: Adrian Tsai <adtsai@microsoft.com> Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Paul McDaniel <paul_mcdaniel@hotmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Dwayne Robinson <fdwr@hotmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com>
2020-02-05 01:12:19 +00:00
[threading(both)]
[marshaling_behavior(agile)]
[dualapipartition(1)]
runtimeclass TensorInt16Bit : ITensor, [contract(MachineLearningContract, 2)]Windows.Foundation.IMemoryBuffer
{
static TensorInt16Bit Create();
static TensorInt16Bit Create(Windows.Foundation.Collections.IIterable<Int64> shape);
static TensorInt16Bit CreateFromArray(Windows.Foundation.Collections.IIterable<Int64> shape, Int16[] data);
static TensorInt16Bit CreateFromIterable(Windows.Foundation.Collections.IIterable<Int64> shape, Windows.Foundation.Collections.IIterable<Int16> data);
Windows.Foundation.Collections.IVectorView <Int16> GetAsVectorView();
[contract(MachineLearningContract, 2)]
{
//! Creates a tensor object, allocates a buffer of size 'shape', and copies all of 'data' into it.
static TensorInt16Bit CreateFromShapeArrayAndDataArray(Int64[] shape, Int16[] data);
//! Creates a tensor object, creates a tensor of size 'shape', and uses the data in 'buffer'.
static TensorInt16Bit CreateFromBuffer(Int64[] shape, Windows.Storage.Streams.IBuffer buffer);
}
}
[contract(MachineLearningContract, 1)]
INBOX_ONLY([static_name("Windows.AI.MachineLearning.ITensorUInt32BitStatics", 417c3837-e773-4378-8e7f-0cc33dbea697)])
INBOX_ONLY([interface_name("Windows.AI.MachineLearning.ITensorUInt32Bit", d8c9c2ff-7511-45a3-bfac-c38f370d2237)])
Merge windowsai (winml layering) into master (#2956) * Initial Commit * Merged PR 3985217: add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc (#2346) add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc and violating our OS layering requirements. We linked against onecoreuap_apiset.lib in VB so we will continue doing this, but I am still unsure why not to link against onecore instead since that is where we ship. However, since Sheil is the owner of this code we will wait to discuss with him before changing anything. * Initial changes for layering * more snipping to get core into ort * update build instructions to include --build_shared_lib (#2358) * update build instructions to include --build_shared_lib * fix line breaks * Task 23998197: add winml_lib_core into onnnxruntime.dll (#2368) * Task 23998197: add winml_lib_core into onnnxruntime.dll * PR feedback build break on perf_test * return proper error when the model path isn't found (#2391) * LearningModelSession is cleaned up to use the adapter, and parts of b… (#2382) this is a big PR. we are going to move it up to layer_dev , which is still a L3 so we are still safe to do work there agile. we are going to move this into the L3 so that ryan can start doing intergration testing. we will pause for a full code review and integration test result prior to going into the L2. >>>> raw comments from previous commits >>> * LearningModelSession is cleaned up to use the adapter, and parts of binding are. * moved everything in the winmladapter made it all nano-com using, WRL to construct objects in the ORT side. base interfaces for everythign for winml to call cleaned up a bunch of winml to use the base interfaces. * more pieces * GetData across the abi. * renamed some namepsace cleaned up OrtValue cleaned up Tensor cleaned up custom ops. everything *but* learnignmodel should be clean * make sure it's building. winml.dll is still a monolith. * model moved over. everything builds clean. step ! * weak ref comment * Layer dev paulm (#2408) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * Layer dev paulm (#2414) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * Layer dev paulm (#2423) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * Layer dev paulm (#2424) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * Layer dev paulm (#2425) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Layer dev paulm (#2426) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * fixed map and sequence when passing stl types across the ABI . found a leak in nvidia driver, but skipped it. all winmlapitests pass now * Moved SessionOptions over to the abi * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * Register WinML TraceLogging provider on Onnxruntime.dll (#2455) * Register WinML TraceLogging provider on Onnxruntime.dll * Add ifdef to make sure trace logging provider has telemetry option when LAYERING_DONE * No need for ifdef for TraceLoggingOptionMicrosoftTelemetry * PR feedback * Move etw registration into lotus environment constructor and deresgister in lotus environment destructor * Brianma/cpuwinml (#2466) * allow building winml cpu without dml. * Brianma/breaks (#2469) * fix some more breaks * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * move dml checks out of winml and into the adapter * better error handling * Brianma/fi (#2470) * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * fix bad merge * Checking in a staging checkpoint point so that Ryan can work with me in parrallel * build break. * Brianma/testfails (#2473) * add missing ir version to dictvectorizer-string.onnx * add missing ir version to relu.onnx * add missing ir version to zipmap*onnx * add IR version to manually generated models * remove an unnecessary ifdef dml * Brianma/windowsai fi (#2475) * update dockerfiles/README (#2336) * Make elementwise op run 4 items per thread (#2335) Description: Describe your changes. Make elementwise op run 4 items per thread unroll for loop to leverage ILP remove unnessary N==0 check inside elementwise GPU kernel Motivation and Context Why is this change required? What problem does it solve? It can improve the performance of GPU elementwise ops. ~2% performance gain on popular NLP bert model. If it fixes an open issue, please link to the issue here. * Add CUDA GatherElements kernel (#2310) * Updates * Update test * Update * Updates * nits * PR feedback * Update * Update * PR feedback * PR comments * Update * Fix build * Fix build * Nits * Fix * Layer Normalization Fusion (#2319) basic layer normalization transform * Add FastGelu Cuda Op for Gelu and Add bias fusion (#2293) * Add FastGelu cuda op * Add AddBiasGelu for experiment * Revert "Add AddBiasGelu for experiment" This reverts commit 5c1ee019858c657e6bb75887265cb85675626e5b. * Add bias * Add unit tests * update comment * update script * fix build error * update coding style * update for CR feedback Enable half2 optimization only when cuda arch >= 7.0 * move _Tanh to common.cuh * implement CPU contrib OP Attention (#2333) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanUnusedInitializers. (#2320) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanupUnusedInitializers. This means initializers that have been replaced during graph optimizations are not left in the GraphProto when we save an optimized model. * Handle edge case where a model has an unused initializer with matching graph input by also removing the graph input. * Use non-const iterators in std::find_if calls to make centos build happy. * Nuget pipeline changes (#2305) 1. refactor the pipeline, remove some duplicated code 2. Move Windows_py_GPU_Wheels job to Win-GPU-CUDA10. We'll deprecated the "Win-GPU" pool 3. Delete cpu-nocontribops-esrp-pipeline.yml and cpu-nocontribops-pipeline.yml 4. In Linux nuget jobs, run "make install" before creating the package. So that extra RPAH info will be removed * Cuda Reverse Sequence Op, maping types of same size using same template function. (#2281) * Set ElementType to String type of node metadata, instead of byte[] (#2348) * Set ElementType to String type of node metadata, instead of byte[] * Fix spacing * Introduce PrimitiveType into a Type System along with an integer constant (#2307) Improve perf by avoiding GetType<T>() calls. Introduce MLTypeCallDispatcher to switch on Input Type. Add Tensor IsType<T>() fast method. * Fix/test dim value of 0 handling in a couple of places (#2337) * Update the CUDA Where implementation broadcasting logic to handle a dim with value of 0. Add unit test Also add unit test for unary op with dim value of 0 * Exclude ngraph from Where test with 0 dim. * Openvino EP R3.1 onnxrt server (#2357) * onnxrt server with OVEP * onnxrt server with OVEP * Update Dockerfile.server.openvino * onnxrt server OVEP fix reviews * onnxrt server OVEP fix reviews * Implement cuda nonzero op. (#2056) Implement cuda nonzero op. * Direct use python numpy array's memory if already contiguous. (#2355) * Direct use python numpy array's memory if already contiguous. This could greatly improve performance for session with large input, like big image 1920x1080 fastrcnn, 30~40% speed up could be achieved. * Add test case enforce contiguous/non-contiguos numpy array as inputs. * Add helper to create output to minimize binary size. (#2365) Add ConstEigenTensorMap typedef so we don't unnecessarily const_cast the const input Tensor. * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS (#2369) * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS * update * Add Tracelogging for profiling (#1639) Enabled only if onnxruntime_ENABLE_INSTRUMENT is ON * test bidaf with nuphar for avx target (#2370) increase nuphar test coverage a bit * Fix a bug in TLS refcount that may destabilized CUDA CI (#2374) * update output size calculation for resize (#2366) * change how output size is calculated for resize op * add tests for ver 10 resize * Extend OneHot CPU kernel to support more types (#2311) * Extend OneHot CPU kernel to support input int64_t, depth int32_t, output float * Skip BERT before the test data fix is picked up * Fix bug with Slice. Need to pass in flattened input dimensions so the initial offset into the input is calculated correctly. (#2372) * Add opset 11 version of Split to CUDA ops (#2376) Organize the CUDA ops definitions so all the opset 10 and 11 parts are together (same setup used for CPU ops) * Layer Norm Fusion Fix (#2379) * layer norm fusion fix * Add input shape check in code and unit tests * Fuse Add + Gelu (#2360) Implement the transformer to fuse add + gelu Implement the accurate kernel * Skip layer norm transform (#2350) * skip layer normalization transformer * Another try to stabilize CUDA CI (#2383) The root cause seems to be failure in CUDA dealloc when tear down. cudaFree return code was ignored before, so should the debug check. * fix BUILD.md typo (#2375) build.py: error: argument --config: invalid choice: 'RelWithDebugInfo' (choose from 'Debug', 'MinSizeRel', 'Release', 'RelWithDebInfo') * Fixed compilation with ngraph (#2388) * Fix reuse logic in allocation planner. (#2393) * Fix reuse logic in allocation planner. * PR comments * Add helpful comments * Don't allow reuse across string tensors. * [NupharEP] Multiple optimizations (#2380) Fuse transpose into MatMul Implement Pow and constant scalar simplification Vectorize ReduceMean Improve symbolic shape inference Minor updates for better debugging in fused function name * Avoid using the default logger in the graph lib and optimizers (#2361) 1. Use the session logger if it is available. 2. Don't disable warning 4100 globally. We should fix the warnings instead of disabling it. * Change CUDA implementation of Transpose to support all fixed size tensor types (#2387) * Change CUDA implementation of Transpose to not use a typed kernel so we can support more types with minimum binary size. Add support for 8, 16, 32 and 64 bit types. Add unit tests. Add method so the implementation can be called directly (will be used by CUDA Scan very soon). * Disable TensorRT for MLFloat16 and int8 unit tests. * Address PR comment and add support for calling cublas implementation if type is mlfloat16. * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. (#2398) * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. * [NupharEP] force some low/zero cost ops to be inlined (#2409) * fix cross compile bug (#2415) * Minor optimization: if a node has already been placed, there's no need to find a kernel for it. (#2417) * Add Reshape Fusion (#2395) * Add reshape fusion * Add some comments * update comments * update comment format * update according to feedback * update for recent logger change * fix build error * (1) Support both input and output edges in find path in graphutils (2) Add a test case of only one constant initializer of Concat input. (3) Refactor ReshapeFusion class to allow add more subgraph fusion in the future. * fix error * (1) loose constraint on initializer: non constant is allowed for reshape fusion. (2) Change versions type to vector. (3) Add logging. (4) Return false when multiple output edges matched in FindPath. Add comments. * only allow one direction (input or output) in FindPath * [NupharEP] Update notebook and docker image (#2416) Add BERT squad in Nuphar tutorial Enhance speed comparsion readability * Fix the issue in matmul_add_fusion (#2407) Fix the issue in matmul_add_fusion If Muatmul + Add has shape [K] * [K, N], reset it to [1, K] * [K, N] will make the output shape to [1, N] will also requires a reshape on the output. Fix: just remove the shape reset to not fuse it. Add a negative test case for matmul+add fusion * feat(treeregressor): Update TreeEnsembleRegressor for type support (#2389) Updates the `TreeEnsembleRegressor` to allow for `double`, `float`, `int64`, and `int32` inputs to match the upstream specification. Signed-off-by: Nick Groszewski <nicholas.groszewski@capitalone.com> * onnxrt server documentation update (#2396) * Added support for Pad-2 operator in OpenVINO-EP (#2405) * Add CUDA If operator. (#2377) * Add CUDA If operator. Uses CPU operator for implementation. By adding a CUDA version the inputs/outputs (with the exception of the 'cond' input) stay on GPU, and no other logic is required to avoid a copy to CPU across the control flow node. * Improved documentation for onnxruntime::utils::SwapByteOrderCopy(), added precondition check. * Fix the type constraints on CUDA If operator to exclude strings. (#2431) * add Im2col<uint8_t> (#2438) * Adjust codegen vectorization width from target (#2439) * Adjust codegen vectorization width from target * Add CUDA Scan operator. (#2403) * Add Scan CUDA op. Uses CPU implementation for logic. Added some device specific functors for handling when data needs to be manipulated on a different device. Added ability to override the materialization logic in the OrtValue slicer so DML can plugin their handling. * Fix Windows GPU C API packaging pipeline failure (#2440) Fix Windows GPU C API packaging pipeline failure (#2440) * Correctly handle implicit inputs for fused nodes (#2390) * Correctly handle implicit inputs for fused nodes Previously, nuphar's partitioning function didn't include node's implicit inputs into the inputs list of MetaDef, and hence a crash was triggered in the onnx graph checker. This commit fixed the issue. Furthermore, it also fixed a related issue where we didn't add implicit inputs into graph_inputs_excluding_initializers_ in Graph::SetGraphInputsOutputs. the issue was that graph_inputs_including_initializers_ populated by SetInputs (e.g. called by FunctionImpl::FunctionImpl) may contain implicit inputs which were not of any node's initializers in the graph. Because they were not part of any initializers, these implicit inputs couldn't be visited by going through all nodes' inputs. Consequently, they would *not* be added into graph_inputs_excluding_initializers_. We fixed the issue by first copying the populated graph_inputs_including_initializers_ into graph_inputs_excluding_initalizers_, which then had both initializers and non-initializers as its initial content. Later, we erase initializers from the list. In this way, we can ensure all implicit inputs to remain in graph_inputs_excluding_initializers_. * refined comments and fixed duplicates Address CR by revisiting comments in terms of implicit inputs Also fixed an issue by skipping duplicates while copying inputs from graph_inputs_including_initializers_. * address CR explain why we need to collect nodes' implicit inputs * don't rely on pointer values for iterating std::set Previously, openvino relied on iterating a set of NodeArg pointers to construct inputs and outputs for a fused graph. It could cause non-determinism. The reason was that although iterating std::set by itself is stable, pointer values of NodeArgs may vary. Consequently, we could end up visiting the set's elements in different orders for different runs for the same test, which resulted in constructing inputs (and outputs) with different orders to the fused graph. For example, for the same test, we may have inputs [A, B] in some runs but inputs[B, A] in others. Let's use std::string as the key type to avoid such nondeterminism. This commit also added implicit inputs into meta->inputs while returning the capability from the openvino provider. * Fixed another latent issue in openvino's GetCapability function The issue was that we couldn't simply erase fused_inputs and fused_outputs while iterating the nodes. For example, an output NodeArg may have multiple uses, and it's wrong if we erase it from fused_outputs when we encounter only one of its uses as input. * Remove DeviceAllocatorRegistry class (#2451) Remove DeviceAllocatorRegistry class * CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library. * Parallel Gelu with ParallelFor (#2399) Parallel Gelu to get better performance for Gelu * Clean up build.py (#2446) * Pull the latest image before running docker build * Fuse SkipLayerNorm with Bias (#2453) Fuse SkipLayerNorm with Bias * Allow more than one invocation of CreateEnv in the same process. (#2467) * Allow more than one invocation of CreateEnv in the same process. * Fix centos build * Symbolic shape inference improvements: (#2460) * Symbolic shape inference improvements: - add a mode to guess unknown ops' output rank - add support for GatherND - add support for If - fix a bug in get_int_values when then tensor rank > 1D, by treating it as no sympy data - add symbol to literal merge when ONNX silently merges dims - fix a bug in Concat when input dim is 0 - fix a bug in ConstantOfShape that computed dim is not updated - add support for dynamic shape in ConstantOfShape - fix a bug in Loop output shape that loop iterator dim is not inserted at dim 0 - add support for dynamic padding in Pad - add support for dynamic shape in Reshape - add support for Resize with opset > 10, by treating output dims as dynamic - fix a bug in Slice when starts/ends are dynamic - restrict input model to opset 7 and above - make output model optional to avoid disk write when testing Run model tests for symbolic shape inference Reduce 2GB docker image size of nuphar * add additional test data set for nuget pipeline (#2448) * add SAS token to download internal test data for nuget pipeline * update azure endpoint * fix keyvault download step * fix variable declaration for secret group * fix indentation * fix yaml syntax for variables * fix setting secrets for script * fix env synctax * Fix macos pipeline * attempt to add secrets to windows download data * fix mac and win data download * fix windows data download * update test data set url and location * Revert "Brianma/windowsai fi (#2475)" This reverts commit 5780b864a15513fda4eadbfc2b5345fefe70b5ec. * Add scenario tests (#2457) * Add scenario tests * Remove TODO from model license * Add winml_api test dependency * fix model load test. fi from master changed the constructor (#2483) * make api tests all pass (#2486) * fix bad merge * fix bad model merge * Layer dev paulm (#2492) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * Rename ambiguous header (#2489) * fix one more missing IR version model (#2500) * add missing IR version to 4 more models used by scenario tests (#2501) * Add CLI parameters to test runner, build WinML in ARM and x86 CI (#2479) * Support test parameters through CLI arguments * Add WinML do Windows x86/ARM CI builds * Code style fixes * Update googletest Remove GPUTEST macros everywhere now that GTEST_SKIP is supported * Refactor main.cpp * Build scenario tests without DML * Link scenario tests to DML when it's enabled (#2502) * Layer dev release pipeline (#2488) Adds winml binaries to existing cpu nuget package, and creates new gpu dml nuget package with winml binaries and DML EP. * Layer dev paulm (#2506) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * Remove usage of IOBinding in WinML and use C_API Run method (#2504) * remove usage of iobinding * Change data structure to use vector of Ort::Values * Polish bind input / output * Use C APIrun method * Update providers on evaluate getresults * Remove run and IObinding interface from WinMLAdapter * Remove use of IObinding * bind unbound outputs code moved to learningmodelbinding * clean up unneeded istensor adapter function * Fix comment * Check if session is closed before binding and clearing * PR feedback * Layer dev paulm (#2507) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * Make tests dependend on winml_dll (#2509) * add dml binaries to DirectML package and be more explicit about condition variables (#2520) * re-enable warnings for winml builds and fix the warnings that were hiding (#2526) * turn devmode back on for winml builds * fix some warnings. include protobuf in a way that disables some warnings * undo protobufhelpers changes and just ignore 4100 errors in pb code * attempt to isolate protobufhelpers errors * add template specialization for getting tensor proto data * Layer dev paulm (#2533) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * Spawn child process to run DeviceLostRecovery scenario test (#2530) * Spawn child process to run DeviceLostRecovery scenario test * Layer dev paulm (#2536) ori said yes * add missing namespace to winml_trace_logging_provider in lotusenvironment.h (#2542) * Handle exception thrown from all apis in WinMLAdapter (#2539) * various changes to unblock windowsai ADO build * Fix custom ops scenario tests (#2562) * Do not shutdown protobuf after ort environment gets destroyed. Lazy load lotus environment first time it is needed * comment typo * pr comment about calling phoenix singleton * Make lotus_environment static in winmladapter * Layer dev paulm (#2567) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * making windowsAI azure dev ops work. * code review comments. * revert changes * Cmake and preprocessor fixes that where uncovered by building on agents without DML available via SDK * Layer dev dml delayload (#2580) * Brianma/cpu (#2583) * don't include dml stuff in cpu builds * tests that link the image lib also need the telemetry lib now * Throw Winml_err_invalid_binding if binding gpu resource on cpu device (#2589) * Throw Winml_err_invalid_binding if binding gpu resource on cpu device * PR comments. No need to query executionprovider for is gpu device * User/xianz/ortthrow (#2596) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * User/xianz/ortthrow (#2599) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * add status error message * Remove uwp onsuspending winrt call because logruntimeperf is getting removed (#2630) * User/xianz/dedup telemetry (#2631) * investigate duplication of telemetry in winml and ort * remove winml telemetry events * telemetry executionProviderEvent * remove unneccessary file and refactor code little bit * Revert back TelemetryEvent, which send up ETW event. * merge changes from layer_dev to windowsai (#2638) * Remove underscore from googletest names (#2616) * Fix leaking memory allocator Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24278761 and https://microsoft.visualstudio.com/OS/_workitems/edit/24330198 * Explicitly initialize Ort::Value with nullptr * Cache WinML adapter * bad merge * define private version of dxcore enum that is added in 19H1 SDK. (#2654) * add comment for explaning private definition of dxcore d3d feature level ennum value. (#2672) * do not package directml.pdb for redist packages. (#2676) * Fix leaking operator registry (#2645) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24354916 * User/orilevari/windowsai master merge (#2674) merge resolutions included pulling in telemetry logic that was merged to master and not windowsai and dereferencing InferenceSession::sessionstate now that it is a unique pointer * Delete Ort Allocator in LearningModelBinding (#2653) * Delete OrtAllocator in LearningModelBinding * PR comments to make Ort::Allocator a smart pointer * Small comment change * PR feedback to clean up code * PR feedback on move semantics * Clean up std::move * Fix memory leaks (#2679) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24356109, https://microsoft.visualstudio.com/OS/_workitems/edit/24388361 and https://microsoft.visualstudio.com/OS/_workitems/edit/24388596 * various changes to properly organize and skip GPU tests. For now for No DML builds we will not run GPU tests at all. In the future we should adapt the tests to expect the appropiate errors. (#2695) * Windowsai without fi (#2701) * Disable Attention fusion tests when DISABLE_CONTRIB_OPS is defined (#2529) * Setup java ci (#2528) * Add provision in ORT for session options to be parsed when available via model file (#2449) * Initial commit * Fix gitmodules * Nits * Nits * Updates * Update * More changes * Updates * Update * Some updates * More changes * Update * Update * Merge * Update * Updates * More changes * Update * Fix nits * Updates * Fix warning * Fix build * Add comment * PR feedback * PR feedback * Updates * Updates * Update * More changes * Fix build break * Comment test for now * Updates * Updates * PR feedback * Updates * Nits * Add tests * Fix build * Fix build * Fix build * Fix build break * Fix build * Nits * PR feedback * More change * Expose GetSessionOptions in pybind logic and add unit test for python * Fix build * PR feedback * PR feedback * Revert "Disable thread pool creation when enabled OpenMP (#2485)" (#2535) This reverts commit 7c7d5a149c9ed52eec67304bae5c4b132166a8a1. * Add dynamic shape support in TensorRT execution provider (#2450) * remove onnx-tensorrt submodule * add new onnx-tensorrt submodule (experiment) for trt6 * update engine build for trt6 * update compile and compute for tensorrt6.0 * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * switch to onnx-tensorrt master for TensorRT6' * Update tensorrt_execution_provider.cc * Handle dynamic batch size and add memcpy in TensorRT EP * update test cases * Update tensorrt_execution_provider.cc * update onnx-tensorrt submodule * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.ubuntu_tensorrt * Update run_dockerbuild.sh * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update concat_op_test.cc * Update tensorrt_execution_provider.cc * Upgrade TensorRT to version 6.0.1.5 * Update onnxruntime_providers.cmake * Update CMakeLists.txt * Update reduction_ops_test.cc * Update install_ubuntu.sh * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.tensorrt * Update BUILD.md * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update onnxruntime_providers.cmake * Update install_ubuntu.sh * Update install_ubuntu.sh * Update gemm_test.cc * Update gather_op_test.cc * Update CMakeLists.txt * Removed submodule * update onnx-tensorrt submodule * update header file * Removed submodule * add submodule onnx-tensorrt kevin's branch shape-test' * add debugging code * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * merge master * Removed submodule * update onnx-tensorrt submodule * add more changes for dynamic shapes * Update tensorrt_execution_provider.cc * update for dynamic shape * update dynamic shape processing * fix logger issue * remove submodule onnx-tensorrt * add submodule onnx-tensorrt * add env variable min_subgraph_size * remove redundency * update document * use onnxruntime::make_unique * fix multi-run issue * remove some tests to save CI build time * Add dynamic shape test * Update TensorRT-ExecutionProvider.md * Add example of running Faster R-CNN model on TensorRT EP * Add more details on env variables * update environment variables * Update tensorrt_basic_test.cc * Update model tests * Update tensor_op_test.cc * remove --use_full_protobuf * Update build.py * User/xianz/telemetry (#2458) * enabme telemetry * enable telemetry * set enable telemetry as default * for debugging * remove log and set disable telemetry as default back * delete private file while testing * resolve comment: mainly add license header, rename macro and update docs * rewording in privacy.md * Fix integer overflow in cuda NonMaxSuppression implementation (#2540) * add test case that should pass but fail * fix nms * extract int_max_output_boxes_per_class * Introduce container type runtime checks and other improvements (#2522) Rework TensorSeq in a manner consistent with Tensor and SparseTensor in terms of type system setup. Reduce templating. Introduce helpers to ensure the same data type. Make OrtValue __dtor not virtual. Introduce ContainerChecker * Fix C API tests for centos and mac (#2544) * change c++14 to c++11 * add ld lib path for centos * enable csharp tests on macos * fix C API test on MacOS + fix manylinux dotnet install * fix manylinux dotnet install * fix lib link * Add back executable bit to build.py * Fix a bug handling negative begin pad values in Pad op (#2550) * Fix bug in Pad op * Update * DNNL CMAKE update (#2548) * Fix android build (#2558) * Update win-x86-ci.yml (#2557) Fix build pipeline break * Re-enable Windows C# tests (#2564) * disable onnx_test_runner -x invocations for dnnl (#2568) * Allow sequence length to be symbolic (#2559) * setup java ci mac (#2570) * make layernorm fusion to support opset 11 (#2545) * Fix a warning found in the latest VS release * Add more check on SkipLayerNorm and BiasGelu fusion (#2574) * Fix file not found error during docker build. (#2569) * Add ConvTranspose1D (#2578) * Ryanunderhill/packagename test (#2582) * [Nuphar EP] fixes for some object detection models (#2581) Update notebook tutorial with multi-threaded int8 GEMM from #2517 * EmbedLayerNormalization Fusion Improvement (#2553) Embedding layer norm fusion improvements - add more checks * Update version (#2584) * Temporarily exclude vgg19 test from Python backend test 1. temporarily exclude vgg19 test which comsumes too much memory, run out of memory on Upsquared device. Single test pass for vgg19, need furture investigation (#2588) 2. Update docker file to decrease the docker image size * Update docs for Android NNAPI EP (#2586) * Fix lto bug for protobuf and ubuntu * add path to build dir before test run (#2590) * Add missig env variables for mac pipeline test (#2595) * Fixed an issue in updating realized dims (#2597) when we update realized dims for scan's output, the sliced axis also needs to be inclusive, i.e. we should check with "dim >= insert_inclusive_axis", because the offset in the symbols are based on Scan sugraph. Otherwise, we would end up with shape mismatch later. * Java API for onnxruntime (#2215) * Add support for opset 11 in reshape fusion (#2592) Support opset verion 11 in reshape fusion * Rename automl python tools folder to featurizer_ops. (#2593) * Support opset 11 subgraph of Squad model in Embed Layer Normalization (#2605) Support opset 11 Squad model that is exported from PyTorch nightly. The embed layer uses Range op which is missed in the transformer. * symbolic shape inference: fix warnings in GPT-2 model (#2608) And revise nuphar perf test on BERT squad * Dump subgraph ID and fused graph ID (#2607) * Dump subgraph ID and fused graph ID Dump subgraph ID and fused graph ID for better debugging * Remove local static fused_count added a field global_fused_count_ to NupharExecutionProvider class * EmbedLayerNormalization Fusion For Dynamic Squad Model Opset 10 (#2613) Support subgraph of SQuAD model exported from pytorch with dynamic input axes * Allow providers to be set for InferenceSession at construction (#2606) * Remove unnecessary parameter in some places in GatherElements implementation (#2612) * Remove unnecessary parameter in some places * Update * Update * Make sure fenced tensor could not reuse other tensor. (#2561) Fix random error caused by this. * Improve Embed Layer Norm Fusion for SQuAD with static input shape (#2621) * fix float16 comparison in initializer (#2629) * epsilon attribute for layernormalization fusion (#2639) * removed unnecessary batch file and fix path (#2640) * Add shape inference to ConvTransposeWithDynamicPads schema (#2632) * Improve cuda expand() opeator's performance. (#2624) * Cuda pad optimize when no padding is needed. (#2625) * Shortcut cuda Pad() when no padding is needed. * Optimize cuda scatter() on 2D compatible. (#2628) * Optimize cuda scatter() on 2D compatible. * Add some comments. * fix build error for ARM (#2648) * Improve performance of resize() in Nearest mode (#2626) Special treatment for 2D, check same size as input image. And in 2d kernel, template use_expolation. * Fix memory exception in Layer Norm Fusion (#2644) * Windows CI changes(#2650) * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Revert "Windowsai without fi (#2701)" This reverts commit 285d4c85ff5c4e265f963208170304ef3461e684. * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Deref unique pointer for session_state * send shutdown event when dll is unloaded and EvaluationStop, SessionC… (#2704) * send shutdown event when dll is unloaded and EvaluationStop, SessionCreationStart Events. * Add EvalutationStart Event * add comment * use correct type for for loop (#2755) * ARM CI (#2759) * Set ARM agent pool * Set CMake generator to VS 2019 in ARM * Use system-wide CMake instead of custom version Our custom version is too old for VS 2019 * Use DML and build shared lib in ARM CI * Restore nuget packages in ARM CI * Disable DML * Refactor ARM debug/release builds * Use system packaged Python version * Remove hardcoded Python path * Downgrade Python to 3.7 for build * Remove explicit CMake path * Fix invalid JSON in cgmanifest.json (#2760) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * Remove usage of WHOLEARCHIVE in WinML CMake and add WinMLAdapterFactory (#2726) * Remove usage of WHOLEARCHIVE in WinMLAdapter CMake and add WinMLAdapterFactory * PR feedback, no need for dll(export) since using def file * PR comments * Small comment in gen_def.py * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * Move winml_provider_factory.h to proper location (#2801) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Filter CPU case for IsFloat16Supported (#2802) * Merge fixes * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * Fix warnings that cause build to fail * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * Make DML include path global (#2882) * Make DML include path global * Add generated cppwinrt headers to winml_lib_common * Integrate changes to WindowsAI to make ADO Build (#2886) * Revert "CMake cross-generator fixes (#2790)" This reverts commit dbe7d97fa1ab155f1309bced87199527e8f35bd2. * add additional suppress warning in onnx_proto * ignore /wd4996 warning * DML execution provider fixes * Revert "Revert "CMake cross-generator fixes (#2790)"" This reverts commit 1ae7b4bcbc02edc881ad28685da98e095dfceb17. * Update func signature of custom op function overloads * common devicehelpers fixes * Add pch.h for winml_lib_common * re-add winml_lib_common_dir/inc to include path for winml_adapter * User/orilevari/dml redist shared folder (#2890) * move dml nuget package directory up one level to make it shared between build flavors * Merge conflict fix * Revert "Merge conflict fix" This reverts commit 142fa72cf9ce4344ad717b50b7ea2b8582aadc7c. * Revert "Merge remote-tracking branch 'origin/master' into windowsai" This reverts commit 6e2126d46e5e5f564d65da37dd4f70c93dd81165, reversing changes made to b3f5583dc9249834b947c8ea905f6a98060d5bd6. * Make winml_test_common free of test macros (#2902) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Don't generate files outside the build root (#2914) * Don't generate files outside the build root * Add onnxruntime_EXTERNAL_DEPENDENCIES to WinML * Add DML depedency on RESTORE_PACKAGES * User/orilevari/fix yaml merge bugs (#2918) * Add winml test source parameter into cmake function (#2919) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Add arguments to unittest cmake functions * remove comment * Revert "Revert "Merge remote-tracking branch 'origin/master' into windowsai"" This reverts commit ade5abe72a4234fdbc3623093c61c02c6b0bdc26. * Fix breaks from merge with ORT master * Brianma/linux (#2917) * don't include windows.h in cross-plat header * add default case for switch statement * signed/unsigned mismatch fix Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> * User/sheilk/winml adapter c api (#2891) * Create winml adapter c api * fix build * make it build * move adapter into onnxruntime core/session * entry point not exported * minor changes * make model metadata work * make tests pass * implement all the model reflection apis on the adapter c abi * update the new ort interface to create a lotus ennvironment with a logging sink * start adding ort env * move all winml code into adapter folder/lib to isolate it * ensure a single logging manager at a time * start refactoring session * refactor session creation interface * add cpu and dml session option methods to adapter * finish session init * stub out interfaces in ort lib to perform similar mechanics of iinference session * enable profiling, and enable schema override * update session register graph transformers * turn back on custom registry for custom ops * Add sync api * add last c api stubs * should build... but all feature values are broken since this is in flight to moving all implementation details into ivalue * remove ep adapter header * Implement DML execution provider functions from adapter (#2846) * Implement DML execution provider functions from adapter * Use functions in OnnxruntimeEngine.cpp * make map/sequence type_infos freeable, and start implementing ivalue * make it build again * implement value methods * implement remaining methods * remove com adapter abi * check dml session * cache the allocator on ivalue * check if resource is cpu/gpu when access its mutable data * update tensor * mismatched parentheses * fix tensor base and binding obj * it evaluates tensors! sometimes... * minor fixes * enable gpu evals * wrapper all existing winml adapter apis with API_IMPL to try catch (#2854) * update winml... tensor strings are broken, need to template tensorbase to do different things for strings * make tensor strings work with 2 copies in/2 copies out * Fix tensor string and allocator bug * make maps work again... needs some fixes still * Make it build! * enable map inputs * map outputs * unbound outputs for sequences and maps * User/xianz/merge windowsai (#2883) * Packaging pipeline changes for VS 2019 (#2711) * Tiny fix to codegen * Simplify cache implementation and avoid static variables that may carry over between models * Extend DML kernels (#2641) * Additional DML operators * Check unsupported attributes and inputs * Address PR comments * Add kernel capability function used for partitioning, and re-enable stride-based int64 support based on value range * Fix test failures * Build fix * PR comments * Update Nuphar tutorial notebook (#2721) 1. Reflect int8 GEMV improvements for multi-threading from #2696 2. Add notes on multi-threading control using OpenMP 3. Add samples of running multi-isa AOT, and show int8 GEMM differences between AVX and AVX2 4. Add rnn_benchmark example to resolve #1993 * Add schema for new Qops (#2611) * Add schema for new Qops * adding shape inference + qlinearaveragepool * plus review comments * plus review comments * updates per review comments * plus review comments * [server] Add supposed for model_name and model_version as cli parameter (#2708) * remove 64bit warning message from python validation. (#2727) * MLAS: ARM64 build fix (#2734) fix bad usage of vreinterpret to cast vector element types * Fix broken python docs links (#2740) * Fix build on Mac OS (#2731) mac os ld doesn't support --while-archive, correct option is -all_load * fix ngraph wheel (#2737) * fix ngraph wheel 1.1.0 onnxruntime_ngraph wheel doesn't work * remove libdnnl.so in nGraph Libs * make it easy to compare * Split onnxruntime server to a separated folder (#2744) * Fix build for Python 3.8 (#2747) * Fix build for Python 3.8 * Update protobuf to 3.11.2 (#1928) Update protobuf to 3.11.2 (#1928) * Change default optimization level to All (from Basic) (#2745) * change default optimization level to All (from Basic) * fix test * fix c# test * Update numpy to 1.18 (#2758) * Update numpy to 1.18 * Pipeline changes for python 3.8 (#2753) 1. Pipeline changes for python 3.8 2. Fix a regression in setup.py which was just introduced in the previous commit. Please notice, we still haven't made python 3.8 + Windows + CUDA work. * Add basic stacktrace output for posix debug builds. (#2749) * [NupharEP] fix a race condition when multiple sessions running different models concurrently (#2772) * Revert "Change default optimization level to All (from Basic) (#2745)" This reverts commit 56bb503c2f26474b6613bcb2a198691a11dcef38. * Fix typo in error message (#2736) * Rename MKL-DNN to DNNL to fix broken link (#2730) * Fix nightly build version number issue * Pass BUILD_BUILDNUMBER to linux docker * Disable featurizers in python packages * Import more featurizers (#2781) Make kernels non-template. Add input constraint for learnt data. Add min_max_scalar_transformer, robust_scalar_transformer, inputation_marker_transfomer, label_encoder_transformer, missing_dummies_transformer along with tests. Advance Featurizers library commit. * Implement a more stable softmax (#2715) * Implement a more stable SoftMax e^x is represented as infinity if x is large enough, like 100.f. Infinity divided by Infinity is a NAN. Thus, softmax gets a NAN if one or more item are large enough. A math transform as below is leveraged to get a stable softmax: e^xi/(e^x1 + ...e^xn) = e^(xi - max) / (e^(x1 - max) + ... + e^(xn - max)) And for convenience, force max to 0.f if all xi are negative * Contributing: Fix a typo (#2784) * ACL EP GEMM improvements (#2780) When it is posible we use a fully connected layer instead of the gemm implementation. This will let the library use the best implementation based on the input data. * ACL EP convolution improvements (#2774) Added the optimized implementation for depthwise convolution for both ACL v19.02 and ACL 19.05. Also the pointwise convolution seems to be more optimal in the CPU implementation so we opted for that instead. * Add script for release Nuget validation (#2719) * Initial commit * Nits * Disable a test temporarily * Change working directory * Test * Add download python step * Test update * More changes * Fix space issue * Fix * Verify nuget signing * Fix * Spaces * PR feedback * Nit * Fix * Fix * Remove temporary changes * add uint8 support to where op (#2792) * Improve bert optimization script: (#2712) (1) Move input int64=>int32 conversion to embed layer fusion. (2) Output epsilon attribute for LayerNormalization fusion. * add session creation time cost. (#2798) * ML.NET team needs featurizers within a package (#2789) Add auto ml featurizers to Windows, MacOS as well as to GPU packaging-pipelines. * Initialize max of softmax with lowest of float (#2786) * MLAS: update SGEMM threading parameters (#2808) * add interface to copy batch tensors. (#2807) * add interface to copy batch tensors. * onnxruntime * speed up Windows TRT CI (#2811) * don't run cuda tests if building with tensorrt * remove unnecessary build options for win trt ci * refactor win gpu tensorrt ci yml * --numpy_version=1.17 * update * update * azcopy and cuda path * Update test data (#2356) * Add timeseries imputer transformer featurizer kernel (#2813) Make kernels non-template. Add input constraint for learnt data. Fixup tests. Add two more featurizers along with tests. Tests fail. min_max_scalar_transformer robust_scalar_transformer Fix tests serialized stream by prepending version bytes. Add inputation_marker_transfomer and the test. Fix up float/double type designations. Added label_encoder_transformer along with a test. string_throw case is broken at the momement. Fix labelencodertransfomer_test.cc string_throw case Rename maxabsscalertransformer_test.cc Add MissingDummiesTransformer along with the test. Update manifest. Add TimeSeriesImputerTransformer definition, implementation and tests * Fix memory leak in TRT (#2815) * fix memory leak issue * revert EP_FAIL on enueueV2 * Add manifest missing comma * Run static code analyzer on most of our code (#2817) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * update quantization doc (#2783) * update documentation for quantization script * plus some spell corrections * Filter CPU case for IsFloat16Supported (#2802) * update default optimization level + fix gemm_activation fusion (#2791) * update defualt optimization level + fix gemm_activation fusion * fix typo * add unit test and incorporate review comments * fix test comment * Fix dnnl wheel package name (#2823) * Append '-dnnl' to whl package name when --use_dnnl * Update build.py * Update Ubuntu & TensorRT version in README (#2820) Dockerfile.tensorrt is using nvcr.io/nvidia/tensorrt:19.09-py3 as base Image, update Ubuntu and TensorRT version according to https://docs.nvidia.com/deeplearning/sdk/tensorrt-container-release-notes/rel_19-09.html#rel_19-09 * Merge fixes * Add OneHotEncoder and HashOneHotEncoder kernels. (#2830) Add defs and imlementation for OneHotEncoders, adjuist date_time_transformer kernel and test. Add OneHotEncoder kernel test. Add HashOneHotVectorizerTransformer unit test. This does not link due to multiple definitions of functions that are included into header from a CPP file. * Upgrade gtest to the latest version (#2827) WinML would like to update the googletest submodule. They want some newer features (namely GTEST_SKIP to skip tests programmatically and be able to skip entire fixtures easily) and would need to update the submodule version. However, because the new version of code hit a bug in gcc, even though the bug is already fixed in the latest gcc but we're using gcc 4.8.x and it won't get patched for the bug, so we have to do a compromise, change our code a little bit to make it work. The gcc bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=51213 * Add support for int64_t for topk CPU. Fixes github issue #2806. (#2833) * Ignore allocator type in ExecutionProviders allocator map. Make default initialization of OrtMemoryInfo more clearly invalid. (#2768) * Remove allocator type from the key comparison in ExecutionProviders. Remove usage of DummyArena as it's no longer necessary. * Fix x86 tests where arena allocator is disabled. Make initialization of OrtMemoryInfo clearer by adding Invalid enum value. * Make OrtValueNameIdxMap::MaxIdx more intuitive. * Convert ExternalProject Featurizers into git submodule (#2834) Add git submodule for Featurizer library. Update cmake to build for git submodule. * add domain check for nodes + update documentation (#2831) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Fix Linux CUDA nuget packaging pipeline break * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * update optimization doc for BERT related fusions (#2819) * Add bert related transformers to doc * Add execution provider and comment for bert optimizations * Add comment about accuracy impact of approximation * Fix warnings that cause build to fail * MLAS: enable threading for quantized GEMMs (#2844) * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * remove old winmladapter cpp Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * move sequence implementation into ort lib... still commented out... need to turn back on... * begin sequence implementation * make maps and sequences work * fix broken tests * remove dead code * misc cleanup * CR feedback * User/xianz/winml adapter c api (#2869) * wrapper all existing winml adapter apis with API_IMPL to try catch * Return HR or Throw for WinML adapter APIs if failed * undo macro wrapper for two places * Wrap error macros around ort apis, too. * address CR feedback #2 * add more api throw/return macros * Revert changes no longer needed * revert changes to cxx api * format winml lib.ort and winml adapter * remove static pheonix singleton Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * missing use_dml check in winml_adapter_session (#2930) * --use_dnnl flag was mangled in merge (#2931) * use dml macro not wrapping custom registry code (#2934) * Disable LNK4199 winml_dll to enable cuda builds (#2936) * Disable LNK4199 in winml_dll * linkler->linker * LearningModelSessionAPITestGpu.CreateSessionWithCastToFloat16InModel should return DXGI_ERROR_UNSUPPORTED when FP16 not supported (#2937) * Disable LNK4199 in winml_dll * linkler->linker * Need to return DXGI_ERROR_UNSUPPORTED when Model does not support fp16 * Publish build symbols (#2939) * Publish build symbols * Don't upload PDBs for .exe files * Make x86 build (#2943) * fix last remaining size_t/int64_t warnings->errors (#2948) * TensorString, Sequences and Maps use the first allocator, but should use the cpu default allocator. (#2952) * fix tensor string allcoator * clean up default allocator usage for strings in winml lib/api.ort Co-authored-by: Ryan Lai <ryalai96@gmail.com> * Handle tensor shape of zero (#2954) Co-authored-by: Ryan Lai <ryalai96@gmail.com> * CR feedback (#2970) * CR feedback * fix weird formatting on privacy readme * Add 'All rights reserved.' everywhere * readd all rights reserved to winml_provider_factory.h * remove extra space in comment * remove extra whitespace * fixes post master merge * remove winml from nuget gpu pipeline * set IR VERSION on generated_model in rnn_benchmark (#2972) * Fix slice conformance failures (#2908) Co-authored-by: Adrian Tsai <adtsai@microsoft.com> Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Paul McDaniel <paul_mcdaniel@hotmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Dwayne Robinson <fdwr@hotmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com>
2020-02-05 01:12:19 +00:00
[threading(both)]
[marshaling_behavior(agile)]
[dualapipartition(1)]
runtimeclass TensorUInt32Bit : ITensor, [contract(MachineLearningContract, 2)]Windows.Foundation.IMemoryBuffer
{
static TensorUInt32Bit Create();
static TensorUInt32Bit Create(Windows.Foundation.Collections.IIterable<Int64> shape);
static TensorUInt32Bit CreateFromArray(Windows.Foundation.Collections.IIterable<Int64> shape, UInt32[] data);
static TensorUInt32Bit CreateFromIterable(Windows.Foundation.Collections.IIterable<Int64> shape, Windows.Foundation.Collections.IIterable<UInt32> data);
Windows.Foundation.Collections.IVectorView <UInt32> GetAsVectorView();
[contract(MachineLearningContract, 2)]
{
//! Creates a tensor object, allocates a buffer of size 'shape', and copies all of 'data' into it.
static TensorUInt32Bit CreateFromShapeArrayAndDataArray(Int64[] shape, UInt32[] data);
//! Creates a tensor object, creates a tensor of size 'shape', and uses the data in 'buffer'.
static TensorUInt32Bit CreateFromBuffer(Int64[] shape, Windows.Storage.Streams.IBuffer buffer);
}
}
[contract(MachineLearningContract, 1)]
INBOX_ONLY([static_name("Windows.AI.MachineLearning.ITensorInt32BitStatics", 6539864b-52fa-4e35-907c-834cac417b50)])
INBOX_ONLY([interface_name("Windows.AI.MachineLearning.ITensorInt32Bit", 2c0c28d3-207c-4486-a7d2-884522c5e589)])
Merge windowsai (winml layering) into master (#2956) * Initial Commit * Merged PR 3985217: add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc (#2346) add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc and violating our OS layering requirements. We linked against onecoreuap_apiset.lib in VB so we will continue doing this, but I am still unsure why not to link against onecore instead since that is where we ship. However, since Sheil is the owner of this code we will wait to discuss with him before changing anything. * Initial changes for layering * more snipping to get core into ort * update build instructions to include --build_shared_lib (#2358) * update build instructions to include --build_shared_lib * fix line breaks * Task 23998197: add winml_lib_core into onnnxruntime.dll (#2368) * Task 23998197: add winml_lib_core into onnnxruntime.dll * PR feedback build break on perf_test * return proper error when the model path isn't found (#2391) * LearningModelSession is cleaned up to use the adapter, and parts of b… (#2382) this is a big PR. we are going to move it up to layer_dev , which is still a L3 so we are still safe to do work there agile. we are going to move this into the L3 so that ryan can start doing intergration testing. we will pause for a full code review and integration test result prior to going into the L2. >>>> raw comments from previous commits >>> * LearningModelSession is cleaned up to use the adapter, and parts of binding are. * moved everything in the winmladapter made it all nano-com using, WRL to construct objects in the ORT side. base interfaces for everythign for winml to call cleaned up a bunch of winml to use the base interfaces. * more pieces * GetData across the abi. * renamed some namepsace cleaned up OrtValue cleaned up Tensor cleaned up custom ops. everything *but* learnignmodel should be clean * make sure it's building. winml.dll is still a monolith. * model moved over. everything builds clean. step ! * weak ref comment * Layer dev paulm (#2408) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * Layer dev paulm (#2414) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * Layer dev paulm (#2423) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * Layer dev paulm (#2424) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * Layer dev paulm (#2425) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Layer dev paulm (#2426) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * fixed map and sequence when passing stl types across the ABI . found a leak in nvidia driver, but skipped it. all winmlapitests pass now * Moved SessionOptions over to the abi * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * Register WinML TraceLogging provider on Onnxruntime.dll (#2455) * Register WinML TraceLogging provider on Onnxruntime.dll * Add ifdef to make sure trace logging provider has telemetry option when LAYERING_DONE * No need for ifdef for TraceLoggingOptionMicrosoftTelemetry * PR feedback * Move etw registration into lotus environment constructor and deresgister in lotus environment destructor * Brianma/cpuwinml (#2466) * allow building winml cpu without dml. * Brianma/breaks (#2469) * fix some more breaks * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * move dml checks out of winml and into the adapter * better error handling * Brianma/fi (#2470) * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * fix bad merge * Checking in a staging checkpoint point so that Ryan can work with me in parrallel * build break. * Brianma/testfails (#2473) * add missing ir version to dictvectorizer-string.onnx * add missing ir version to relu.onnx * add missing ir version to zipmap*onnx * add IR version to manually generated models * remove an unnecessary ifdef dml * Brianma/windowsai fi (#2475) * update dockerfiles/README (#2336) * Make elementwise op run 4 items per thread (#2335) Description: Describe your changes. Make elementwise op run 4 items per thread unroll for loop to leverage ILP remove unnessary N==0 check inside elementwise GPU kernel Motivation and Context Why is this change required? What problem does it solve? It can improve the performance of GPU elementwise ops. ~2% performance gain on popular NLP bert model. If it fixes an open issue, please link to the issue here. * Add CUDA GatherElements kernel (#2310) * Updates * Update test * Update * Updates * nits * PR feedback * Update * Update * PR feedback * PR comments * Update * Fix build * Fix build * Nits * Fix * Layer Normalization Fusion (#2319) basic layer normalization transform * Add FastGelu Cuda Op for Gelu and Add bias fusion (#2293) * Add FastGelu cuda op * Add AddBiasGelu for experiment * Revert "Add AddBiasGelu for experiment" This reverts commit 5c1ee019858c657e6bb75887265cb85675626e5b. * Add bias * Add unit tests * update comment * update script * fix build error * update coding style * update for CR feedback Enable half2 optimization only when cuda arch >= 7.0 * move _Tanh to common.cuh * implement CPU contrib OP Attention (#2333) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanUnusedInitializers. (#2320) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanupUnusedInitializers. This means initializers that have been replaced during graph optimizations are not left in the GraphProto when we save an optimized model. * Handle edge case where a model has an unused initializer with matching graph input by also removing the graph input. * Use non-const iterators in std::find_if calls to make centos build happy. * Nuget pipeline changes (#2305) 1. refactor the pipeline, remove some duplicated code 2. Move Windows_py_GPU_Wheels job to Win-GPU-CUDA10. We'll deprecated the "Win-GPU" pool 3. Delete cpu-nocontribops-esrp-pipeline.yml and cpu-nocontribops-pipeline.yml 4. In Linux nuget jobs, run "make install" before creating the package. So that extra RPAH info will be removed * Cuda Reverse Sequence Op, maping types of same size using same template function. (#2281) * Set ElementType to String type of node metadata, instead of byte[] (#2348) * Set ElementType to String type of node metadata, instead of byte[] * Fix spacing * Introduce PrimitiveType into a Type System along with an integer constant (#2307) Improve perf by avoiding GetType<T>() calls. Introduce MLTypeCallDispatcher to switch on Input Type. Add Tensor IsType<T>() fast method. * Fix/test dim value of 0 handling in a couple of places (#2337) * Update the CUDA Where implementation broadcasting logic to handle a dim with value of 0. Add unit test Also add unit test for unary op with dim value of 0 * Exclude ngraph from Where test with 0 dim. * Openvino EP R3.1 onnxrt server (#2357) * onnxrt server with OVEP * onnxrt server with OVEP * Update Dockerfile.server.openvino * onnxrt server OVEP fix reviews * onnxrt server OVEP fix reviews * Implement cuda nonzero op. (#2056) Implement cuda nonzero op. * Direct use python numpy array's memory if already contiguous. (#2355) * Direct use python numpy array's memory if already contiguous. This could greatly improve performance for session with large input, like big image 1920x1080 fastrcnn, 30~40% speed up could be achieved. * Add test case enforce contiguous/non-contiguos numpy array as inputs. * Add helper to create output to minimize binary size. (#2365) Add ConstEigenTensorMap typedef so we don't unnecessarily const_cast the const input Tensor. * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS (#2369) * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS * update * Add Tracelogging for profiling (#1639) Enabled only if onnxruntime_ENABLE_INSTRUMENT is ON * test bidaf with nuphar for avx target (#2370) increase nuphar test coverage a bit * Fix a bug in TLS refcount that may destabilized CUDA CI (#2374) * update output size calculation for resize (#2366) * change how output size is calculated for resize op * add tests for ver 10 resize * Extend OneHot CPU kernel to support more types (#2311) * Extend OneHot CPU kernel to support input int64_t, depth int32_t, output float * Skip BERT before the test data fix is picked up * Fix bug with Slice. Need to pass in flattened input dimensions so the initial offset into the input is calculated correctly. (#2372) * Add opset 11 version of Split to CUDA ops (#2376) Organize the CUDA ops definitions so all the opset 10 and 11 parts are together (same setup used for CPU ops) * Layer Norm Fusion Fix (#2379) * layer norm fusion fix * Add input shape check in code and unit tests * Fuse Add + Gelu (#2360) Implement the transformer to fuse add + gelu Implement the accurate kernel * Skip layer norm transform (#2350) * skip layer normalization transformer * Another try to stabilize CUDA CI (#2383) The root cause seems to be failure in CUDA dealloc when tear down. cudaFree return code was ignored before, so should the debug check. * fix BUILD.md typo (#2375) build.py: error: argument --config: invalid choice: 'RelWithDebugInfo' (choose from 'Debug', 'MinSizeRel', 'Release', 'RelWithDebInfo') * Fixed compilation with ngraph (#2388) * Fix reuse logic in allocation planner. (#2393) * Fix reuse logic in allocation planner. * PR comments * Add helpful comments * Don't allow reuse across string tensors. * [NupharEP] Multiple optimizations (#2380) Fuse transpose into MatMul Implement Pow and constant scalar simplification Vectorize ReduceMean Improve symbolic shape inference Minor updates for better debugging in fused function name * Avoid using the default logger in the graph lib and optimizers (#2361) 1. Use the session logger if it is available. 2. Don't disable warning 4100 globally. We should fix the warnings instead of disabling it. * Change CUDA implementation of Transpose to support all fixed size tensor types (#2387) * Change CUDA implementation of Transpose to not use a typed kernel so we can support more types with minimum binary size. Add support for 8, 16, 32 and 64 bit types. Add unit tests. Add method so the implementation can be called directly (will be used by CUDA Scan very soon). * Disable TensorRT for MLFloat16 and int8 unit tests. * Address PR comment and add support for calling cublas implementation if type is mlfloat16. * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. (#2398) * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. * [NupharEP] force some low/zero cost ops to be inlined (#2409) * fix cross compile bug (#2415) * Minor optimization: if a node has already been placed, there's no need to find a kernel for it. (#2417) * Add Reshape Fusion (#2395) * Add reshape fusion * Add some comments * update comments * update comment format * update according to feedback * update for recent logger change * fix build error * (1) Support both input and output edges in find path in graphutils (2) Add a test case of only one constant initializer of Concat input. (3) Refactor ReshapeFusion class to allow add more subgraph fusion in the future. * fix error * (1) loose constraint on initializer: non constant is allowed for reshape fusion. (2) Change versions type to vector. (3) Add logging. (4) Return false when multiple output edges matched in FindPath. Add comments. * only allow one direction (input or output) in FindPath * [NupharEP] Update notebook and docker image (#2416) Add BERT squad in Nuphar tutorial Enhance speed comparsion readability * Fix the issue in matmul_add_fusion (#2407) Fix the issue in matmul_add_fusion If Muatmul + Add has shape [K] * [K, N], reset it to [1, K] * [K, N] will make the output shape to [1, N] will also requires a reshape on the output. Fix: just remove the shape reset to not fuse it. Add a negative test case for matmul+add fusion * feat(treeregressor): Update TreeEnsembleRegressor for type support (#2389) Updates the `TreeEnsembleRegressor` to allow for `double`, `float`, `int64`, and `int32` inputs to match the upstream specification. Signed-off-by: Nick Groszewski <nicholas.groszewski@capitalone.com> * onnxrt server documentation update (#2396) * Added support for Pad-2 operator in OpenVINO-EP (#2405) * Add CUDA If operator. (#2377) * Add CUDA If operator. Uses CPU operator for implementation. By adding a CUDA version the inputs/outputs (with the exception of the 'cond' input) stay on GPU, and no other logic is required to avoid a copy to CPU across the control flow node. * Improved documentation for onnxruntime::utils::SwapByteOrderCopy(), added precondition check. * Fix the type constraints on CUDA If operator to exclude strings. (#2431) * add Im2col<uint8_t> (#2438) * Adjust codegen vectorization width from target (#2439) * Adjust codegen vectorization width from target * Add CUDA Scan operator. (#2403) * Add Scan CUDA op. Uses CPU implementation for logic. Added some device specific functors for handling when data needs to be manipulated on a different device. Added ability to override the materialization logic in the OrtValue slicer so DML can plugin their handling. * Fix Windows GPU C API packaging pipeline failure (#2440) Fix Windows GPU C API packaging pipeline failure (#2440) * Correctly handle implicit inputs for fused nodes (#2390) * Correctly handle implicit inputs for fused nodes Previously, nuphar's partitioning function didn't include node's implicit inputs into the inputs list of MetaDef, and hence a crash was triggered in the onnx graph checker. This commit fixed the issue. Furthermore, it also fixed a related issue where we didn't add implicit inputs into graph_inputs_excluding_initializers_ in Graph::SetGraphInputsOutputs. the issue was that graph_inputs_including_initializers_ populated by SetInputs (e.g. called by FunctionImpl::FunctionImpl) may contain implicit inputs which were not of any node's initializers in the graph. Because they were not part of any initializers, these implicit inputs couldn't be visited by going through all nodes' inputs. Consequently, they would *not* be added into graph_inputs_excluding_initializers_. We fixed the issue by first copying the populated graph_inputs_including_initializers_ into graph_inputs_excluding_initalizers_, which then had both initializers and non-initializers as its initial content. Later, we erase initializers from the list. In this way, we can ensure all implicit inputs to remain in graph_inputs_excluding_initializers_. * refined comments and fixed duplicates Address CR by revisiting comments in terms of implicit inputs Also fixed an issue by skipping duplicates while copying inputs from graph_inputs_including_initializers_. * address CR explain why we need to collect nodes' implicit inputs * don't rely on pointer values for iterating std::set Previously, openvino relied on iterating a set of NodeArg pointers to construct inputs and outputs for a fused graph. It could cause non-determinism. The reason was that although iterating std::set by itself is stable, pointer values of NodeArgs may vary. Consequently, we could end up visiting the set's elements in different orders for different runs for the same test, which resulted in constructing inputs (and outputs) with different orders to the fused graph. For example, for the same test, we may have inputs [A, B] in some runs but inputs[B, A] in others. Let's use std::string as the key type to avoid such nondeterminism. This commit also added implicit inputs into meta->inputs while returning the capability from the openvino provider. * Fixed another latent issue in openvino's GetCapability function The issue was that we couldn't simply erase fused_inputs and fused_outputs while iterating the nodes. For example, an output NodeArg may have multiple uses, and it's wrong if we erase it from fused_outputs when we encounter only one of its uses as input. * Remove DeviceAllocatorRegistry class (#2451) Remove DeviceAllocatorRegistry class * CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library. * Parallel Gelu with ParallelFor (#2399) Parallel Gelu to get better performance for Gelu * Clean up build.py (#2446) * Pull the latest image before running docker build * Fuse SkipLayerNorm with Bias (#2453) Fuse SkipLayerNorm with Bias * Allow more than one invocation of CreateEnv in the same process. (#2467) * Allow more than one invocation of CreateEnv in the same process. * Fix centos build * Symbolic shape inference improvements: (#2460) * Symbolic shape inference improvements: - add a mode to guess unknown ops' output rank - add support for GatherND - add support for If - fix a bug in get_int_values when then tensor rank > 1D, by treating it as no sympy data - add symbol to literal merge when ONNX silently merges dims - fix a bug in Concat when input dim is 0 - fix a bug in ConstantOfShape that computed dim is not updated - add support for dynamic shape in ConstantOfShape - fix a bug in Loop output shape that loop iterator dim is not inserted at dim 0 - add support for dynamic padding in Pad - add support for dynamic shape in Reshape - add support for Resize with opset > 10, by treating output dims as dynamic - fix a bug in Slice when starts/ends are dynamic - restrict input model to opset 7 and above - make output model optional to avoid disk write when testing Run model tests for symbolic shape inference Reduce 2GB docker image size of nuphar * add additional test data set for nuget pipeline (#2448) * add SAS token to download internal test data for nuget pipeline * update azure endpoint * fix keyvault download step * fix variable declaration for secret group * fix indentation * fix yaml syntax for variables * fix setting secrets for script * fix env synctax * Fix macos pipeline * attempt to add secrets to windows download data * fix mac and win data download * fix windows data download * update test data set url and location * Revert "Brianma/windowsai fi (#2475)" This reverts commit 5780b864a15513fda4eadbfc2b5345fefe70b5ec. * Add scenario tests (#2457) * Add scenario tests * Remove TODO from model license * Add winml_api test dependency * fix model load test. fi from master changed the constructor (#2483) * make api tests all pass (#2486) * fix bad merge * fix bad model merge * Layer dev paulm (#2492) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * Rename ambiguous header (#2489) * fix one more missing IR version model (#2500) * add missing IR version to 4 more models used by scenario tests (#2501) * Add CLI parameters to test runner, build WinML in ARM and x86 CI (#2479) * Support test parameters through CLI arguments * Add WinML do Windows x86/ARM CI builds * Code style fixes * Update googletest Remove GPUTEST macros everywhere now that GTEST_SKIP is supported * Refactor main.cpp * Build scenario tests without DML * Link scenario tests to DML when it's enabled (#2502) * Layer dev release pipeline (#2488) Adds winml binaries to existing cpu nuget package, and creates new gpu dml nuget package with winml binaries and DML EP. * Layer dev paulm (#2506) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * Remove usage of IOBinding in WinML and use C_API Run method (#2504) * remove usage of iobinding * Change data structure to use vector of Ort::Values * Polish bind input / output * Use C APIrun method * Update providers on evaluate getresults * Remove run and IObinding interface from WinMLAdapter * Remove use of IObinding * bind unbound outputs code moved to learningmodelbinding * clean up unneeded istensor adapter function * Fix comment * Check if session is closed before binding and clearing * PR feedback * Layer dev paulm (#2507) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * Make tests dependend on winml_dll (#2509) * add dml binaries to DirectML package and be more explicit about condition variables (#2520) * re-enable warnings for winml builds and fix the warnings that were hiding (#2526) * turn devmode back on for winml builds * fix some warnings. include protobuf in a way that disables some warnings * undo protobufhelpers changes and just ignore 4100 errors in pb code * attempt to isolate protobufhelpers errors * add template specialization for getting tensor proto data * Layer dev paulm (#2533) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * Spawn child process to run DeviceLostRecovery scenario test (#2530) * Spawn child process to run DeviceLostRecovery scenario test * Layer dev paulm (#2536) ori said yes * add missing namespace to winml_trace_logging_provider in lotusenvironment.h (#2542) * Handle exception thrown from all apis in WinMLAdapter (#2539) * various changes to unblock windowsai ADO build * Fix custom ops scenario tests (#2562) * Do not shutdown protobuf after ort environment gets destroyed. Lazy load lotus environment first time it is needed * comment typo * pr comment about calling phoenix singleton * Make lotus_environment static in winmladapter * Layer dev paulm (#2567) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * making windowsAI azure dev ops work. * code review comments. * revert changes * Cmake and preprocessor fixes that where uncovered by building on agents without DML available via SDK * Layer dev dml delayload (#2580) * Brianma/cpu (#2583) * don't include dml stuff in cpu builds * tests that link the image lib also need the telemetry lib now * Throw Winml_err_invalid_binding if binding gpu resource on cpu device (#2589) * Throw Winml_err_invalid_binding if binding gpu resource on cpu device * PR comments. No need to query executionprovider for is gpu device * User/xianz/ortthrow (#2596) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * User/xianz/ortthrow (#2599) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * add status error message * Remove uwp onsuspending winrt call because logruntimeperf is getting removed (#2630) * User/xianz/dedup telemetry (#2631) * investigate duplication of telemetry in winml and ort * remove winml telemetry events * telemetry executionProviderEvent * remove unneccessary file and refactor code little bit * Revert back TelemetryEvent, which send up ETW event. * merge changes from layer_dev to windowsai (#2638) * Remove underscore from googletest names (#2616) * Fix leaking memory allocator Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24278761 and https://microsoft.visualstudio.com/OS/_workitems/edit/24330198 * Explicitly initialize Ort::Value with nullptr * Cache WinML adapter * bad merge * define private version of dxcore enum that is added in 19H1 SDK. (#2654) * add comment for explaning private definition of dxcore d3d feature level ennum value. (#2672) * do not package directml.pdb for redist packages. (#2676) * Fix leaking operator registry (#2645) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24354916 * User/orilevari/windowsai master merge (#2674) merge resolutions included pulling in telemetry logic that was merged to master and not windowsai and dereferencing InferenceSession::sessionstate now that it is a unique pointer * Delete Ort Allocator in LearningModelBinding (#2653) * Delete OrtAllocator in LearningModelBinding * PR comments to make Ort::Allocator a smart pointer * Small comment change * PR feedback to clean up code * PR feedback on move semantics * Clean up std::move * Fix memory leaks (#2679) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24356109, https://microsoft.visualstudio.com/OS/_workitems/edit/24388361 and https://microsoft.visualstudio.com/OS/_workitems/edit/24388596 * various changes to properly organize and skip GPU tests. For now for No DML builds we will not run GPU tests at all. In the future we should adapt the tests to expect the appropiate errors. (#2695) * Windowsai without fi (#2701) * Disable Attention fusion tests when DISABLE_CONTRIB_OPS is defined (#2529) * Setup java ci (#2528) * Add provision in ORT for session options to be parsed when available via model file (#2449) * Initial commit * Fix gitmodules * Nits * Nits * Updates * Update * More changes * Updates * Update * Some updates * More changes * Update * Update * Merge * Update * Updates * More changes * Update * Fix nits * Updates * Fix warning * Fix build * Add comment * PR feedback * PR feedback * Updates * Updates * Update * More changes * Fix build break * Comment test for now * Updates * Updates * PR feedback * Updates * Nits * Add tests * Fix build * Fix build * Fix build * Fix build break * Fix build * Nits * PR feedback * More change * Expose GetSessionOptions in pybind logic and add unit test for python * Fix build * PR feedback * PR feedback * Revert "Disable thread pool creation when enabled OpenMP (#2485)" (#2535) This reverts commit 7c7d5a149c9ed52eec67304bae5c4b132166a8a1. * Add dynamic shape support in TensorRT execution provider (#2450) * remove onnx-tensorrt submodule * add new onnx-tensorrt submodule (experiment) for trt6 * update engine build for trt6 * update compile and compute for tensorrt6.0 * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * switch to onnx-tensorrt master for TensorRT6' * Update tensorrt_execution_provider.cc * Handle dynamic batch size and add memcpy in TensorRT EP * update test cases * Update tensorrt_execution_provider.cc * update onnx-tensorrt submodule * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.ubuntu_tensorrt * Update run_dockerbuild.sh * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update concat_op_test.cc * Update tensorrt_execution_provider.cc * Upgrade TensorRT to version 6.0.1.5 * Update onnxruntime_providers.cmake * Update CMakeLists.txt * Update reduction_ops_test.cc * Update install_ubuntu.sh * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.tensorrt * Update BUILD.md * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update onnxruntime_providers.cmake * Update install_ubuntu.sh * Update install_ubuntu.sh * Update gemm_test.cc * Update gather_op_test.cc * Update CMakeLists.txt * Removed submodule * update onnx-tensorrt submodule * update header file * Removed submodule * add submodule onnx-tensorrt kevin's branch shape-test' * add debugging code * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * merge master * Removed submodule * update onnx-tensorrt submodule * add more changes for dynamic shapes * Update tensorrt_execution_provider.cc * update for dynamic shape * update dynamic shape processing * fix logger issue * remove submodule onnx-tensorrt * add submodule onnx-tensorrt * add env variable min_subgraph_size * remove redundency * update document * use onnxruntime::make_unique * fix multi-run issue * remove some tests to save CI build time * Add dynamic shape test * Update TensorRT-ExecutionProvider.md * Add example of running Faster R-CNN model on TensorRT EP * Add more details on env variables * update environment variables * Update tensorrt_basic_test.cc * Update model tests * Update tensor_op_test.cc * remove --use_full_protobuf * Update build.py * User/xianz/telemetry (#2458) * enabme telemetry * enable telemetry * set enable telemetry as default * for debugging * remove log and set disable telemetry as default back * delete private file while testing * resolve comment: mainly add license header, rename macro and update docs * rewording in privacy.md * Fix integer overflow in cuda NonMaxSuppression implementation (#2540) * add test case that should pass but fail * fix nms * extract int_max_output_boxes_per_class * Introduce container type runtime checks and other improvements (#2522) Rework TensorSeq in a manner consistent with Tensor and SparseTensor in terms of type system setup. Reduce templating. Introduce helpers to ensure the same data type. Make OrtValue __dtor not virtual. Introduce ContainerChecker * Fix C API tests for centos and mac (#2544) * change c++14 to c++11 * add ld lib path for centos * enable csharp tests on macos * fix C API test on MacOS + fix manylinux dotnet install * fix manylinux dotnet install * fix lib link * Add back executable bit to build.py * Fix a bug handling negative begin pad values in Pad op (#2550) * Fix bug in Pad op * Update * DNNL CMAKE update (#2548) * Fix android build (#2558) * Update win-x86-ci.yml (#2557) Fix build pipeline break * Re-enable Windows C# tests (#2564) * disable onnx_test_runner -x invocations for dnnl (#2568) * Allow sequence length to be symbolic (#2559) * setup java ci mac (#2570) * make layernorm fusion to support opset 11 (#2545) * Fix a warning found in the latest VS release * Add more check on SkipLayerNorm and BiasGelu fusion (#2574) * Fix file not found error during docker build. (#2569) * Add ConvTranspose1D (#2578) * Ryanunderhill/packagename test (#2582) * [Nuphar EP] fixes for some object detection models (#2581) Update notebook tutorial with multi-threaded int8 GEMM from #2517 * EmbedLayerNormalization Fusion Improvement (#2553) Embedding layer norm fusion improvements - add more checks * Update version (#2584) * Temporarily exclude vgg19 test from Python backend test 1. temporarily exclude vgg19 test which comsumes too much memory, run out of memory on Upsquared device. Single test pass for vgg19, need furture investigation (#2588) 2. Update docker file to decrease the docker image size * Update docs for Android NNAPI EP (#2586) * Fix lto bug for protobuf and ubuntu * add path to build dir before test run (#2590) * Add missig env variables for mac pipeline test (#2595) * Fixed an issue in updating realized dims (#2597) when we update realized dims for scan's output, the sliced axis also needs to be inclusive, i.e. we should check with "dim >= insert_inclusive_axis", because the offset in the symbols are based on Scan sugraph. Otherwise, we would end up with shape mismatch later. * Java API for onnxruntime (#2215) * Add support for opset 11 in reshape fusion (#2592) Support opset verion 11 in reshape fusion * Rename automl python tools folder to featurizer_ops. (#2593) * Support opset 11 subgraph of Squad model in Embed Layer Normalization (#2605) Support opset 11 Squad model that is exported from PyTorch nightly. The embed layer uses Range op which is missed in the transformer. * symbolic shape inference: fix warnings in GPT-2 model (#2608) And revise nuphar perf test on BERT squad * Dump subgraph ID and fused graph ID (#2607) * Dump subgraph ID and fused graph ID Dump subgraph ID and fused graph ID for better debugging * Remove local static fused_count added a field global_fused_count_ to NupharExecutionProvider class * EmbedLayerNormalization Fusion For Dynamic Squad Model Opset 10 (#2613) Support subgraph of SQuAD model exported from pytorch with dynamic input axes * Allow providers to be set for InferenceSession at construction (#2606) * Remove unnecessary parameter in some places in GatherElements implementation (#2612) * Remove unnecessary parameter in some places * Update * Update * Make sure fenced tensor could not reuse other tensor. (#2561) Fix random error caused by this. * Improve Embed Layer Norm Fusion for SQuAD with static input shape (#2621) * fix float16 comparison in initializer (#2629) * epsilon attribute for layernormalization fusion (#2639) * removed unnecessary batch file and fix path (#2640) * Add shape inference to ConvTransposeWithDynamicPads schema (#2632) * Improve cuda expand() opeator's performance. (#2624) * Cuda pad optimize when no padding is needed. (#2625) * Shortcut cuda Pad() when no padding is needed. * Optimize cuda scatter() on 2D compatible. (#2628) * Optimize cuda scatter() on 2D compatible. * Add some comments. * fix build error for ARM (#2648) * Improve performance of resize() in Nearest mode (#2626) Special treatment for 2D, check same size as input image. And in 2d kernel, template use_expolation. * Fix memory exception in Layer Norm Fusion (#2644) * Windows CI changes(#2650) * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Revert "Windowsai without fi (#2701)" This reverts commit 285d4c85ff5c4e265f963208170304ef3461e684. * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Deref unique pointer for session_state * send shutdown event when dll is unloaded and EvaluationStop, SessionC… (#2704) * send shutdown event when dll is unloaded and EvaluationStop, SessionCreationStart Events. * Add EvalutationStart Event * add comment * use correct type for for loop (#2755) * ARM CI (#2759) * Set ARM agent pool * Set CMake generator to VS 2019 in ARM * Use system-wide CMake instead of custom version Our custom version is too old for VS 2019 * Use DML and build shared lib in ARM CI * Restore nuget packages in ARM CI * Disable DML * Refactor ARM debug/release builds * Use system packaged Python version * Remove hardcoded Python path * Downgrade Python to 3.7 for build * Remove explicit CMake path * Fix invalid JSON in cgmanifest.json (#2760) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * Remove usage of WHOLEARCHIVE in WinML CMake and add WinMLAdapterFactory (#2726) * Remove usage of WHOLEARCHIVE in WinMLAdapter CMake and add WinMLAdapterFactory * PR feedback, no need for dll(export) since using def file * PR comments * Small comment in gen_def.py * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * Move winml_provider_factory.h to proper location (#2801) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Filter CPU case for IsFloat16Supported (#2802) * Merge fixes * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * Fix warnings that cause build to fail * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * Make DML include path global (#2882) * Make DML include path global * Add generated cppwinrt headers to winml_lib_common * Integrate changes to WindowsAI to make ADO Build (#2886) * Revert "CMake cross-generator fixes (#2790)" This reverts commit dbe7d97fa1ab155f1309bced87199527e8f35bd2. * add additional suppress warning in onnx_proto * ignore /wd4996 warning * DML execution provider fixes * Revert "Revert "CMake cross-generator fixes (#2790)"" This reverts commit 1ae7b4bcbc02edc881ad28685da98e095dfceb17. * Update func signature of custom op function overloads * common devicehelpers fixes * Add pch.h for winml_lib_common * re-add winml_lib_common_dir/inc to include path for winml_adapter * User/orilevari/dml redist shared folder (#2890) * move dml nuget package directory up one level to make it shared between build flavors * Merge conflict fix * Revert "Merge conflict fix" This reverts commit 142fa72cf9ce4344ad717b50b7ea2b8582aadc7c. * Revert "Merge remote-tracking branch 'origin/master' into windowsai" This reverts commit 6e2126d46e5e5f564d65da37dd4f70c93dd81165, reversing changes made to b3f5583dc9249834b947c8ea905f6a98060d5bd6. * Make winml_test_common free of test macros (#2902) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Don't generate files outside the build root (#2914) * Don't generate files outside the build root * Add onnxruntime_EXTERNAL_DEPENDENCIES to WinML * Add DML depedency on RESTORE_PACKAGES * User/orilevari/fix yaml merge bugs (#2918) * Add winml test source parameter into cmake function (#2919) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Add arguments to unittest cmake functions * remove comment * Revert "Revert "Merge remote-tracking branch 'origin/master' into windowsai"" This reverts commit ade5abe72a4234fdbc3623093c61c02c6b0bdc26. * Fix breaks from merge with ORT master * Brianma/linux (#2917) * don't include windows.h in cross-plat header * add default case for switch statement * signed/unsigned mismatch fix Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> * User/sheilk/winml adapter c api (#2891) * Create winml adapter c api * fix build * make it build * move adapter into onnxruntime core/session * entry point not exported * minor changes * make model metadata work * make tests pass * implement all the model reflection apis on the adapter c abi * update the new ort interface to create a lotus ennvironment with a logging sink * start adding ort env * move all winml code into adapter folder/lib to isolate it * ensure a single logging manager at a time * start refactoring session * refactor session creation interface * add cpu and dml session option methods to adapter * finish session init * stub out interfaces in ort lib to perform similar mechanics of iinference session * enable profiling, and enable schema override * update session register graph transformers * turn back on custom registry for custom ops * Add sync api * add last c api stubs * should build... but all feature values are broken since this is in flight to moving all implementation details into ivalue * remove ep adapter header * Implement DML execution provider functions from adapter (#2846) * Implement DML execution provider functions from adapter * Use functions in OnnxruntimeEngine.cpp * make map/sequence type_infos freeable, and start implementing ivalue * make it build again * implement value methods * implement remaining methods * remove com adapter abi * check dml session * cache the allocator on ivalue * check if resource is cpu/gpu when access its mutable data * update tensor * mismatched parentheses * fix tensor base and binding obj * it evaluates tensors! sometimes... * minor fixes * enable gpu evals * wrapper all existing winml adapter apis with API_IMPL to try catch (#2854) * update winml... tensor strings are broken, need to template tensorbase to do different things for strings * make tensor strings work with 2 copies in/2 copies out * Fix tensor string and allocator bug * make maps work again... needs some fixes still * Make it build! * enable map inputs * map outputs * unbound outputs for sequences and maps * User/xianz/merge windowsai (#2883) * Packaging pipeline changes for VS 2019 (#2711) * Tiny fix to codegen * Simplify cache implementation and avoid static variables that may carry over between models * Extend DML kernels (#2641) * Additional DML operators * Check unsupported attributes and inputs * Address PR comments * Add kernel capability function used for partitioning, and re-enable stride-based int64 support based on value range * Fix test failures * Build fix * PR comments * Update Nuphar tutorial notebook (#2721) 1. Reflect int8 GEMV improvements for multi-threading from #2696 2. Add notes on multi-threading control using OpenMP 3. Add samples of running multi-isa AOT, and show int8 GEMM differences between AVX and AVX2 4. Add rnn_benchmark example to resolve #1993 * Add schema for new Qops (#2611) * Add schema for new Qops * adding shape inference + qlinearaveragepool * plus review comments * plus review comments * updates per review comments * plus review comments * [server] Add supposed for model_name and model_version as cli parameter (#2708) * remove 64bit warning message from python validation. (#2727) * MLAS: ARM64 build fix (#2734) fix bad usage of vreinterpret to cast vector element types * Fix broken python docs links (#2740) * Fix build on Mac OS (#2731) mac os ld doesn't support --while-archive, correct option is -all_load * fix ngraph wheel (#2737) * fix ngraph wheel 1.1.0 onnxruntime_ngraph wheel doesn't work * remove libdnnl.so in nGraph Libs * make it easy to compare * Split onnxruntime server to a separated folder (#2744) * Fix build for Python 3.8 (#2747) * Fix build for Python 3.8 * Update protobuf to 3.11.2 (#1928) Update protobuf to 3.11.2 (#1928) * Change default optimization level to All (from Basic) (#2745) * change default optimization level to All (from Basic) * fix test * fix c# test * Update numpy to 1.18 (#2758) * Update numpy to 1.18 * Pipeline changes for python 3.8 (#2753) 1. Pipeline changes for python 3.8 2. Fix a regression in setup.py which was just introduced in the previous commit. Please notice, we still haven't made python 3.8 + Windows + CUDA work. * Add basic stacktrace output for posix debug builds. (#2749) * [NupharEP] fix a race condition when multiple sessions running different models concurrently (#2772) * Revert "Change default optimization level to All (from Basic) (#2745)" This reverts commit 56bb503c2f26474b6613bcb2a198691a11dcef38. * Fix typo in error message (#2736) * Rename MKL-DNN to DNNL to fix broken link (#2730) * Fix nightly build version number issue * Pass BUILD_BUILDNUMBER to linux docker * Disable featurizers in python packages * Import more featurizers (#2781) Make kernels non-template. Add input constraint for learnt data. Add min_max_scalar_transformer, robust_scalar_transformer, inputation_marker_transfomer, label_encoder_transformer, missing_dummies_transformer along with tests. Advance Featurizers library commit. * Implement a more stable softmax (#2715) * Implement a more stable SoftMax e^x is represented as infinity if x is large enough, like 100.f. Infinity divided by Infinity is a NAN. Thus, softmax gets a NAN if one or more item are large enough. A math transform as below is leveraged to get a stable softmax: e^xi/(e^x1 + ...e^xn) = e^(xi - max) / (e^(x1 - max) + ... + e^(xn - max)) And for convenience, force max to 0.f if all xi are negative * Contributing: Fix a typo (#2784) * ACL EP GEMM improvements (#2780) When it is posible we use a fully connected layer instead of the gemm implementation. This will let the library use the best implementation based on the input data. * ACL EP convolution improvements (#2774) Added the optimized implementation for depthwise convolution for both ACL v19.02 and ACL 19.05. Also the pointwise convolution seems to be more optimal in the CPU implementation so we opted for that instead. * Add script for release Nuget validation (#2719) * Initial commit * Nits * Disable a test temporarily * Change working directory * Test * Add download python step * Test update * More changes * Fix space issue * Fix * Verify nuget signing * Fix * Spaces * PR feedback * Nit * Fix * Fix * Remove temporary changes * add uint8 support to where op (#2792) * Improve bert optimization script: (#2712) (1) Move input int64=>int32 conversion to embed layer fusion. (2) Output epsilon attribute for LayerNormalization fusion. * add session creation time cost. (#2798) * ML.NET team needs featurizers within a package (#2789) Add auto ml featurizers to Windows, MacOS as well as to GPU packaging-pipelines. * Initialize max of softmax with lowest of float (#2786) * MLAS: update SGEMM threading parameters (#2808) * add interface to copy batch tensors. (#2807) * add interface to copy batch tensors. * onnxruntime * speed up Windows TRT CI (#2811) * don't run cuda tests if building with tensorrt * remove unnecessary build options for win trt ci * refactor win gpu tensorrt ci yml * --numpy_version=1.17 * update * update * azcopy and cuda path * Update test data (#2356) * Add timeseries imputer transformer featurizer kernel (#2813) Make kernels non-template. Add input constraint for learnt data. Fixup tests. Add two more featurizers along with tests. Tests fail. min_max_scalar_transformer robust_scalar_transformer Fix tests serialized stream by prepending version bytes. Add inputation_marker_transfomer and the test. Fix up float/double type designations. Added label_encoder_transformer along with a test. string_throw case is broken at the momement. Fix labelencodertransfomer_test.cc string_throw case Rename maxabsscalertransformer_test.cc Add MissingDummiesTransformer along with the test. Update manifest. Add TimeSeriesImputerTransformer definition, implementation and tests * Fix memory leak in TRT (#2815) * fix memory leak issue * revert EP_FAIL on enueueV2 * Add manifest missing comma * Run static code analyzer on most of our code (#2817) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * update quantization doc (#2783) * update documentation for quantization script * plus some spell corrections * Filter CPU case for IsFloat16Supported (#2802) * update default optimization level + fix gemm_activation fusion (#2791) * update defualt optimization level + fix gemm_activation fusion * fix typo * add unit test and incorporate review comments * fix test comment * Fix dnnl wheel package name (#2823) * Append '-dnnl' to whl package name when --use_dnnl * Update build.py * Update Ubuntu & TensorRT version in README (#2820) Dockerfile.tensorrt is using nvcr.io/nvidia/tensorrt:19.09-py3 as base Image, update Ubuntu and TensorRT version according to https://docs.nvidia.com/deeplearning/sdk/tensorrt-container-release-notes/rel_19-09.html#rel_19-09 * Merge fixes * Add OneHotEncoder and HashOneHotEncoder kernels. (#2830) Add defs and imlementation for OneHotEncoders, adjuist date_time_transformer kernel and test. Add OneHotEncoder kernel test. Add HashOneHotVectorizerTransformer unit test. This does not link due to multiple definitions of functions that are included into header from a CPP file. * Upgrade gtest to the latest version (#2827) WinML would like to update the googletest submodule. They want some newer features (namely GTEST_SKIP to skip tests programmatically and be able to skip entire fixtures easily) and would need to update the submodule version. However, because the new version of code hit a bug in gcc, even though the bug is already fixed in the latest gcc but we're using gcc 4.8.x and it won't get patched for the bug, so we have to do a compromise, change our code a little bit to make it work. The gcc bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=51213 * Add support for int64_t for topk CPU. Fixes github issue #2806. (#2833) * Ignore allocator type in ExecutionProviders allocator map. Make default initialization of OrtMemoryInfo more clearly invalid. (#2768) * Remove allocator type from the key comparison in ExecutionProviders. Remove usage of DummyArena as it's no longer necessary. * Fix x86 tests where arena allocator is disabled. Make initialization of OrtMemoryInfo clearer by adding Invalid enum value. * Make OrtValueNameIdxMap::MaxIdx more intuitive. * Convert ExternalProject Featurizers into git submodule (#2834) Add git submodule for Featurizer library. Update cmake to build for git submodule. * add domain check for nodes + update documentation (#2831) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Fix Linux CUDA nuget packaging pipeline break * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * update optimization doc for BERT related fusions (#2819) * Add bert related transformers to doc * Add execution provider and comment for bert optimizations * Add comment about accuracy impact of approximation * Fix warnings that cause build to fail * MLAS: enable threading for quantized GEMMs (#2844) * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * remove old winmladapter cpp Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * move sequence implementation into ort lib... still commented out... need to turn back on... * begin sequence implementation * make maps and sequences work * fix broken tests * remove dead code * misc cleanup * CR feedback * User/xianz/winml adapter c api (#2869) * wrapper all existing winml adapter apis with API_IMPL to try catch * Return HR or Throw for WinML adapter APIs if failed * undo macro wrapper for two places * Wrap error macros around ort apis, too. * address CR feedback #2 * add more api throw/return macros * Revert changes no longer needed * revert changes to cxx api * format winml lib.ort and winml adapter * remove static pheonix singleton Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * missing use_dml check in winml_adapter_session (#2930) * --use_dnnl flag was mangled in merge (#2931) * use dml macro not wrapping custom registry code (#2934) * Disable LNK4199 winml_dll to enable cuda builds (#2936) * Disable LNK4199 in winml_dll * linkler->linker * LearningModelSessionAPITestGpu.CreateSessionWithCastToFloat16InModel should return DXGI_ERROR_UNSUPPORTED when FP16 not supported (#2937) * Disable LNK4199 in winml_dll * linkler->linker * Need to return DXGI_ERROR_UNSUPPORTED when Model does not support fp16 * Publish build symbols (#2939) * Publish build symbols * Don't upload PDBs for .exe files * Make x86 build (#2943) * fix last remaining size_t/int64_t warnings->errors (#2948) * TensorString, Sequences and Maps use the first allocator, but should use the cpu default allocator. (#2952) * fix tensor string allcoator * clean up default allocator usage for strings in winml lib/api.ort Co-authored-by: Ryan Lai <ryalai96@gmail.com> * Handle tensor shape of zero (#2954) Co-authored-by: Ryan Lai <ryalai96@gmail.com> * CR feedback (#2970) * CR feedback * fix weird formatting on privacy readme * Add 'All rights reserved.' everywhere * readd all rights reserved to winml_provider_factory.h * remove extra space in comment * remove extra whitespace * fixes post master merge * remove winml from nuget gpu pipeline * set IR VERSION on generated_model in rnn_benchmark (#2972) * Fix slice conformance failures (#2908) Co-authored-by: Adrian Tsai <adtsai@microsoft.com> Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Paul McDaniel <paul_mcdaniel@hotmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Dwayne Robinson <fdwr@hotmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com>
2020-02-05 01:12:19 +00:00
[threading(both)]
[marshaling_behavior(agile)]
[dualapipartition(1)]
runtimeclass TensorInt32Bit : ITensor, [contract(MachineLearningContract, 2)]Windows.Foundation.IMemoryBuffer
{
static TensorInt32Bit Create();
static TensorInt32Bit Create(Windows.Foundation.Collections.IIterable<Int64> shape);
static TensorInt32Bit CreateFromArray(Windows.Foundation.Collections.IIterable<Int64> shape, Int32[] data);
static TensorInt32Bit CreateFromIterable(Windows.Foundation.Collections.IIterable<Int64> shape, Windows.Foundation.Collections.IIterable<Int32> data);
Windows.Foundation.Collections.IVectorView <Int32> GetAsVectorView();
[contract(MachineLearningContract, 2)]
{
//! Creates a tensor object, allocates a buffer of size 'shape', and copies all of 'data' into it.
static TensorInt32Bit CreateFromShapeArrayAndDataArray(Int64[] shape, Int32[] data);
//! Creates a tensor object, creates a tensor of size 'shape', and uses the data in 'buffer'.
static TensorInt32Bit CreateFromBuffer(Int64[] shape, Windows.Storage.Streams.IBuffer buffer);
}
}
[contract(MachineLearningContract, 1)]
INBOX_ONLY([static_name("Windows.AI.MachineLearning.ITensorUInt64BitStatics", 7a7e20eb-242f-47cb-a9c6-f602ecfbfee4)])
INBOX_ONLY([interface_name("Windows.AI.MachineLearning.ITensorUInt64Bit", 2e70ffad-04bf-4825-839a-82baef8c7886)])
Merge windowsai (winml layering) into master (#2956) * Initial Commit * Merged PR 3985217: add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc (#2346) add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc and violating our OS layering requirements. We linked against onecoreuap_apiset.lib in VB so we will continue doing this, but I am still unsure why not to link against onecore instead since that is where we ship. However, since Sheil is the owner of this code we will wait to discuss with him before changing anything. * Initial changes for layering * more snipping to get core into ort * update build instructions to include --build_shared_lib (#2358) * update build instructions to include --build_shared_lib * fix line breaks * Task 23998197: add winml_lib_core into onnnxruntime.dll (#2368) * Task 23998197: add winml_lib_core into onnnxruntime.dll * PR feedback build break on perf_test * return proper error when the model path isn't found (#2391) * LearningModelSession is cleaned up to use the adapter, and parts of b… (#2382) this is a big PR. we are going to move it up to layer_dev , which is still a L3 so we are still safe to do work there agile. we are going to move this into the L3 so that ryan can start doing intergration testing. we will pause for a full code review and integration test result prior to going into the L2. >>>> raw comments from previous commits >>> * LearningModelSession is cleaned up to use the adapter, and parts of binding are. * moved everything in the winmladapter made it all nano-com using, WRL to construct objects in the ORT side. base interfaces for everythign for winml to call cleaned up a bunch of winml to use the base interfaces. * more pieces * GetData across the abi. * renamed some namepsace cleaned up OrtValue cleaned up Tensor cleaned up custom ops. everything *but* learnignmodel should be clean * make sure it's building. winml.dll is still a monolith. * model moved over. everything builds clean. step ! * weak ref comment * Layer dev paulm (#2408) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * Layer dev paulm (#2414) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * Layer dev paulm (#2423) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * Layer dev paulm (#2424) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * Layer dev paulm (#2425) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Layer dev paulm (#2426) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * fixed map and sequence when passing stl types across the ABI . found a leak in nvidia driver, but skipped it. all winmlapitests pass now * Moved SessionOptions over to the abi * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * Register WinML TraceLogging provider on Onnxruntime.dll (#2455) * Register WinML TraceLogging provider on Onnxruntime.dll * Add ifdef to make sure trace logging provider has telemetry option when LAYERING_DONE * No need for ifdef for TraceLoggingOptionMicrosoftTelemetry * PR feedback * Move etw registration into lotus environment constructor and deresgister in lotus environment destructor * Brianma/cpuwinml (#2466) * allow building winml cpu without dml. * Brianma/breaks (#2469) * fix some more breaks * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * move dml checks out of winml and into the adapter * better error handling * Brianma/fi (#2470) * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * fix bad merge * Checking in a staging checkpoint point so that Ryan can work with me in parrallel * build break. * Brianma/testfails (#2473) * add missing ir version to dictvectorizer-string.onnx * add missing ir version to relu.onnx * add missing ir version to zipmap*onnx * add IR version to manually generated models * remove an unnecessary ifdef dml * Brianma/windowsai fi (#2475) * update dockerfiles/README (#2336) * Make elementwise op run 4 items per thread (#2335) Description: Describe your changes. Make elementwise op run 4 items per thread unroll for loop to leverage ILP remove unnessary N==0 check inside elementwise GPU kernel Motivation and Context Why is this change required? What problem does it solve? It can improve the performance of GPU elementwise ops. ~2% performance gain on popular NLP bert model. If it fixes an open issue, please link to the issue here. * Add CUDA GatherElements kernel (#2310) * Updates * Update test * Update * Updates * nits * PR feedback * Update * Update * PR feedback * PR comments * Update * Fix build * Fix build * Nits * Fix * Layer Normalization Fusion (#2319) basic layer normalization transform * Add FastGelu Cuda Op for Gelu and Add bias fusion (#2293) * Add FastGelu cuda op * Add AddBiasGelu for experiment * Revert "Add AddBiasGelu for experiment" This reverts commit 5c1ee019858c657e6bb75887265cb85675626e5b. * Add bias * Add unit tests * update comment * update script * fix build error * update coding style * update for CR feedback Enable half2 optimization only when cuda arch >= 7.0 * move _Tanh to common.cuh * implement CPU contrib OP Attention (#2333) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanUnusedInitializers. (#2320) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanupUnusedInitializers. This means initializers that have been replaced during graph optimizations are not left in the GraphProto when we save an optimized model. * Handle edge case where a model has an unused initializer with matching graph input by also removing the graph input. * Use non-const iterators in std::find_if calls to make centos build happy. * Nuget pipeline changes (#2305) 1. refactor the pipeline, remove some duplicated code 2. Move Windows_py_GPU_Wheels job to Win-GPU-CUDA10. We'll deprecated the "Win-GPU" pool 3. Delete cpu-nocontribops-esrp-pipeline.yml and cpu-nocontribops-pipeline.yml 4. In Linux nuget jobs, run "make install" before creating the package. So that extra RPAH info will be removed * Cuda Reverse Sequence Op, maping types of same size using same template function. (#2281) * Set ElementType to String type of node metadata, instead of byte[] (#2348) * Set ElementType to String type of node metadata, instead of byte[] * Fix spacing * Introduce PrimitiveType into a Type System along with an integer constant (#2307) Improve perf by avoiding GetType<T>() calls. Introduce MLTypeCallDispatcher to switch on Input Type. Add Tensor IsType<T>() fast method. * Fix/test dim value of 0 handling in a couple of places (#2337) * Update the CUDA Where implementation broadcasting logic to handle a dim with value of 0. Add unit test Also add unit test for unary op with dim value of 0 * Exclude ngraph from Where test with 0 dim. * Openvino EP R3.1 onnxrt server (#2357) * onnxrt server with OVEP * onnxrt server with OVEP * Update Dockerfile.server.openvino * onnxrt server OVEP fix reviews * onnxrt server OVEP fix reviews * Implement cuda nonzero op. (#2056) Implement cuda nonzero op. * Direct use python numpy array's memory if already contiguous. (#2355) * Direct use python numpy array's memory if already contiguous. This could greatly improve performance for session with large input, like big image 1920x1080 fastrcnn, 30~40% speed up could be achieved. * Add test case enforce contiguous/non-contiguos numpy array as inputs. * Add helper to create output to minimize binary size. (#2365) Add ConstEigenTensorMap typedef so we don't unnecessarily const_cast the const input Tensor. * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS (#2369) * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS * update * Add Tracelogging for profiling (#1639) Enabled only if onnxruntime_ENABLE_INSTRUMENT is ON * test bidaf with nuphar for avx target (#2370) increase nuphar test coverage a bit * Fix a bug in TLS refcount that may destabilized CUDA CI (#2374) * update output size calculation for resize (#2366) * change how output size is calculated for resize op * add tests for ver 10 resize * Extend OneHot CPU kernel to support more types (#2311) * Extend OneHot CPU kernel to support input int64_t, depth int32_t, output float * Skip BERT before the test data fix is picked up * Fix bug with Slice. Need to pass in flattened input dimensions so the initial offset into the input is calculated correctly. (#2372) * Add opset 11 version of Split to CUDA ops (#2376) Organize the CUDA ops definitions so all the opset 10 and 11 parts are together (same setup used for CPU ops) * Layer Norm Fusion Fix (#2379) * layer norm fusion fix * Add input shape check in code and unit tests * Fuse Add + Gelu (#2360) Implement the transformer to fuse add + gelu Implement the accurate kernel * Skip layer norm transform (#2350) * skip layer normalization transformer * Another try to stabilize CUDA CI (#2383) The root cause seems to be failure in CUDA dealloc when tear down. cudaFree return code was ignored before, so should the debug check. * fix BUILD.md typo (#2375) build.py: error: argument --config: invalid choice: 'RelWithDebugInfo' (choose from 'Debug', 'MinSizeRel', 'Release', 'RelWithDebInfo') * Fixed compilation with ngraph (#2388) * Fix reuse logic in allocation planner. (#2393) * Fix reuse logic in allocation planner. * PR comments * Add helpful comments * Don't allow reuse across string tensors. * [NupharEP] Multiple optimizations (#2380) Fuse transpose into MatMul Implement Pow and constant scalar simplification Vectorize ReduceMean Improve symbolic shape inference Minor updates for better debugging in fused function name * Avoid using the default logger in the graph lib and optimizers (#2361) 1. Use the session logger if it is available. 2. Don't disable warning 4100 globally. We should fix the warnings instead of disabling it. * Change CUDA implementation of Transpose to support all fixed size tensor types (#2387) * Change CUDA implementation of Transpose to not use a typed kernel so we can support more types with minimum binary size. Add support for 8, 16, 32 and 64 bit types. Add unit tests. Add method so the implementation can be called directly (will be used by CUDA Scan very soon). * Disable TensorRT for MLFloat16 and int8 unit tests. * Address PR comment and add support for calling cublas implementation if type is mlfloat16. * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. (#2398) * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. * [NupharEP] force some low/zero cost ops to be inlined (#2409) * fix cross compile bug (#2415) * Minor optimization: if a node has already been placed, there's no need to find a kernel for it. (#2417) * Add Reshape Fusion (#2395) * Add reshape fusion * Add some comments * update comments * update comment format * update according to feedback * update for recent logger change * fix build error * (1) Support both input and output edges in find path in graphutils (2) Add a test case of only one constant initializer of Concat input. (3) Refactor ReshapeFusion class to allow add more subgraph fusion in the future. * fix error * (1) loose constraint on initializer: non constant is allowed for reshape fusion. (2) Change versions type to vector. (3) Add logging. (4) Return false when multiple output edges matched in FindPath. Add comments. * only allow one direction (input or output) in FindPath * [NupharEP] Update notebook and docker image (#2416) Add BERT squad in Nuphar tutorial Enhance speed comparsion readability * Fix the issue in matmul_add_fusion (#2407) Fix the issue in matmul_add_fusion If Muatmul + Add has shape [K] * [K, N], reset it to [1, K] * [K, N] will make the output shape to [1, N] will also requires a reshape on the output. Fix: just remove the shape reset to not fuse it. Add a negative test case for matmul+add fusion * feat(treeregressor): Update TreeEnsembleRegressor for type support (#2389) Updates the `TreeEnsembleRegressor` to allow for `double`, `float`, `int64`, and `int32` inputs to match the upstream specification. Signed-off-by: Nick Groszewski <nicholas.groszewski@capitalone.com> * onnxrt server documentation update (#2396) * Added support for Pad-2 operator in OpenVINO-EP (#2405) * Add CUDA If operator. (#2377) * Add CUDA If operator. Uses CPU operator for implementation. By adding a CUDA version the inputs/outputs (with the exception of the 'cond' input) stay on GPU, and no other logic is required to avoid a copy to CPU across the control flow node. * Improved documentation for onnxruntime::utils::SwapByteOrderCopy(), added precondition check. * Fix the type constraints on CUDA If operator to exclude strings. (#2431) * add Im2col<uint8_t> (#2438) * Adjust codegen vectorization width from target (#2439) * Adjust codegen vectorization width from target * Add CUDA Scan operator. (#2403) * Add Scan CUDA op. Uses CPU implementation for logic. Added some device specific functors for handling when data needs to be manipulated on a different device. Added ability to override the materialization logic in the OrtValue slicer so DML can plugin their handling. * Fix Windows GPU C API packaging pipeline failure (#2440) Fix Windows GPU C API packaging pipeline failure (#2440) * Correctly handle implicit inputs for fused nodes (#2390) * Correctly handle implicit inputs for fused nodes Previously, nuphar's partitioning function didn't include node's implicit inputs into the inputs list of MetaDef, and hence a crash was triggered in the onnx graph checker. This commit fixed the issue. Furthermore, it also fixed a related issue where we didn't add implicit inputs into graph_inputs_excluding_initializers_ in Graph::SetGraphInputsOutputs. the issue was that graph_inputs_including_initializers_ populated by SetInputs (e.g. called by FunctionImpl::FunctionImpl) may contain implicit inputs which were not of any node's initializers in the graph. Because they were not part of any initializers, these implicit inputs couldn't be visited by going through all nodes' inputs. Consequently, they would *not* be added into graph_inputs_excluding_initializers_. We fixed the issue by first copying the populated graph_inputs_including_initializers_ into graph_inputs_excluding_initalizers_, which then had both initializers and non-initializers as its initial content. Later, we erase initializers from the list. In this way, we can ensure all implicit inputs to remain in graph_inputs_excluding_initializers_. * refined comments and fixed duplicates Address CR by revisiting comments in terms of implicit inputs Also fixed an issue by skipping duplicates while copying inputs from graph_inputs_including_initializers_. * address CR explain why we need to collect nodes' implicit inputs * don't rely on pointer values for iterating std::set Previously, openvino relied on iterating a set of NodeArg pointers to construct inputs and outputs for a fused graph. It could cause non-determinism. The reason was that although iterating std::set by itself is stable, pointer values of NodeArgs may vary. Consequently, we could end up visiting the set's elements in different orders for different runs for the same test, which resulted in constructing inputs (and outputs) with different orders to the fused graph. For example, for the same test, we may have inputs [A, B] in some runs but inputs[B, A] in others. Let's use std::string as the key type to avoid such nondeterminism. This commit also added implicit inputs into meta->inputs while returning the capability from the openvino provider. * Fixed another latent issue in openvino's GetCapability function The issue was that we couldn't simply erase fused_inputs and fused_outputs while iterating the nodes. For example, an output NodeArg may have multiple uses, and it's wrong if we erase it from fused_outputs when we encounter only one of its uses as input. * Remove DeviceAllocatorRegistry class (#2451) Remove DeviceAllocatorRegistry class * CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library. * Parallel Gelu with ParallelFor (#2399) Parallel Gelu to get better performance for Gelu * Clean up build.py (#2446) * Pull the latest image before running docker build * Fuse SkipLayerNorm with Bias (#2453) Fuse SkipLayerNorm with Bias * Allow more than one invocation of CreateEnv in the same process. (#2467) * Allow more than one invocation of CreateEnv in the same process. * Fix centos build * Symbolic shape inference improvements: (#2460) * Symbolic shape inference improvements: - add a mode to guess unknown ops' output rank - add support for GatherND - add support for If - fix a bug in get_int_values when then tensor rank > 1D, by treating it as no sympy data - add symbol to literal merge when ONNX silently merges dims - fix a bug in Concat when input dim is 0 - fix a bug in ConstantOfShape that computed dim is not updated - add support for dynamic shape in ConstantOfShape - fix a bug in Loop output shape that loop iterator dim is not inserted at dim 0 - add support for dynamic padding in Pad - add support for dynamic shape in Reshape - add support for Resize with opset > 10, by treating output dims as dynamic - fix a bug in Slice when starts/ends are dynamic - restrict input model to opset 7 and above - make output model optional to avoid disk write when testing Run model tests for symbolic shape inference Reduce 2GB docker image size of nuphar * add additional test data set for nuget pipeline (#2448) * add SAS token to download internal test data for nuget pipeline * update azure endpoint * fix keyvault download step * fix variable declaration for secret group * fix indentation * fix yaml syntax for variables * fix setting secrets for script * fix env synctax * Fix macos pipeline * attempt to add secrets to windows download data * fix mac and win data download * fix windows data download * update test data set url and location * Revert "Brianma/windowsai fi (#2475)" This reverts commit 5780b864a15513fda4eadbfc2b5345fefe70b5ec. * Add scenario tests (#2457) * Add scenario tests * Remove TODO from model license * Add winml_api test dependency * fix model load test. fi from master changed the constructor (#2483) * make api tests all pass (#2486) * fix bad merge * fix bad model merge * Layer dev paulm (#2492) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * Rename ambiguous header (#2489) * fix one more missing IR version model (#2500) * add missing IR version to 4 more models used by scenario tests (#2501) * Add CLI parameters to test runner, build WinML in ARM and x86 CI (#2479) * Support test parameters through CLI arguments * Add WinML do Windows x86/ARM CI builds * Code style fixes * Update googletest Remove GPUTEST macros everywhere now that GTEST_SKIP is supported * Refactor main.cpp * Build scenario tests without DML * Link scenario tests to DML when it's enabled (#2502) * Layer dev release pipeline (#2488) Adds winml binaries to existing cpu nuget package, and creates new gpu dml nuget package with winml binaries and DML EP. * Layer dev paulm (#2506) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * Remove usage of IOBinding in WinML and use C_API Run method (#2504) * remove usage of iobinding * Change data structure to use vector of Ort::Values * Polish bind input / output * Use C APIrun method * Update providers on evaluate getresults * Remove run and IObinding interface from WinMLAdapter * Remove use of IObinding * bind unbound outputs code moved to learningmodelbinding * clean up unneeded istensor adapter function * Fix comment * Check if session is closed before binding and clearing * PR feedback * Layer dev paulm (#2507) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * Make tests dependend on winml_dll (#2509) * add dml binaries to DirectML package and be more explicit about condition variables (#2520) * re-enable warnings for winml builds and fix the warnings that were hiding (#2526) * turn devmode back on for winml builds * fix some warnings. include protobuf in a way that disables some warnings * undo protobufhelpers changes and just ignore 4100 errors in pb code * attempt to isolate protobufhelpers errors * add template specialization for getting tensor proto data * Layer dev paulm (#2533) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * Spawn child process to run DeviceLostRecovery scenario test (#2530) * Spawn child process to run DeviceLostRecovery scenario test * Layer dev paulm (#2536) ori said yes * add missing namespace to winml_trace_logging_provider in lotusenvironment.h (#2542) * Handle exception thrown from all apis in WinMLAdapter (#2539) * various changes to unblock windowsai ADO build * Fix custom ops scenario tests (#2562) * Do not shutdown protobuf after ort environment gets destroyed. Lazy load lotus environment first time it is needed * comment typo * pr comment about calling phoenix singleton * Make lotus_environment static in winmladapter * Layer dev paulm (#2567) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * making windowsAI azure dev ops work. * code review comments. * revert changes * Cmake and preprocessor fixes that where uncovered by building on agents without DML available via SDK * Layer dev dml delayload (#2580) * Brianma/cpu (#2583) * don't include dml stuff in cpu builds * tests that link the image lib also need the telemetry lib now * Throw Winml_err_invalid_binding if binding gpu resource on cpu device (#2589) * Throw Winml_err_invalid_binding if binding gpu resource on cpu device * PR comments. No need to query executionprovider for is gpu device * User/xianz/ortthrow (#2596) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * User/xianz/ortthrow (#2599) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * add status error message * Remove uwp onsuspending winrt call because logruntimeperf is getting removed (#2630) * User/xianz/dedup telemetry (#2631) * investigate duplication of telemetry in winml and ort * remove winml telemetry events * telemetry executionProviderEvent * remove unneccessary file and refactor code little bit * Revert back TelemetryEvent, which send up ETW event. * merge changes from layer_dev to windowsai (#2638) * Remove underscore from googletest names (#2616) * Fix leaking memory allocator Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24278761 and https://microsoft.visualstudio.com/OS/_workitems/edit/24330198 * Explicitly initialize Ort::Value with nullptr * Cache WinML adapter * bad merge * define private version of dxcore enum that is added in 19H1 SDK. (#2654) * add comment for explaning private definition of dxcore d3d feature level ennum value. (#2672) * do not package directml.pdb for redist packages. (#2676) * Fix leaking operator registry (#2645) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24354916 * User/orilevari/windowsai master merge (#2674) merge resolutions included pulling in telemetry logic that was merged to master and not windowsai and dereferencing InferenceSession::sessionstate now that it is a unique pointer * Delete Ort Allocator in LearningModelBinding (#2653) * Delete OrtAllocator in LearningModelBinding * PR comments to make Ort::Allocator a smart pointer * Small comment change * PR feedback to clean up code * PR feedback on move semantics * Clean up std::move * Fix memory leaks (#2679) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24356109, https://microsoft.visualstudio.com/OS/_workitems/edit/24388361 and https://microsoft.visualstudio.com/OS/_workitems/edit/24388596 * various changes to properly organize and skip GPU tests. For now for No DML builds we will not run GPU tests at all. In the future we should adapt the tests to expect the appropiate errors. (#2695) * Windowsai without fi (#2701) * Disable Attention fusion tests when DISABLE_CONTRIB_OPS is defined (#2529) * Setup java ci (#2528) * Add provision in ORT for session options to be parsed when available via model file (#2449) * Initial commit * Fix gitmodules * Nits * Nits * Updates * Update * More changes * Updates * Update * Some updates * More changes * Update * Update * Merge * Update * Updates * More changes * Update * Fix nits * Updates * Fix warning * Fix build * Add comment * PR feedback * PR feedback * Updates * Updates * Update * More changes * Fix build break * Comment test for now * Updates * Updates * PR feedback * Updates * Nits * Add tests * Fix build * Fix build * Fix build * Fix build break * Fix build * Nits * PR feedback * More change * Expose GetSessionOptions in pybind logic and add unit test for python * Fix build * PR feedback * PR feedback * Revert "Disable thread pool creation when enabled OpenMP (#2485)" (#2535) This reverts commit 7c7d5a149c9ed52eec67304bae5c4b132166a8a1. * Add dynamic shape support in TensorRT execution provider (#2450) * remove onnx-tensorrt submodule * add new onnx-tensorrt submodule (experiment) for trt6 * update engine build for trt6 * update compile and compute for tensorrt6.0 * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * switch to onnx-tensorrt master for TensorRT6' * Update tensorrt_execution_provider.cc * Handle dynamic batch size and add memcpy in TensorRT EP * update test cases * Update tensorrt_execution_provider.cc * update onnx-tensorrt submodule * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.ubuntu_tensorrt * Update run_dockerbuild.sh * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update concat_op_test.cc * Update tensorrt_execution_provider.cc * Upgrade TensorRT to version 6.0.1.5 * Update onnxruntime_providers.cmake * Update CMakeLists.txt * Update reduction_ops_test.cc * Update install_ubuntu.sh * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.tensorrt * Update BUILD.md * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update onnxruntime_providers.cmake * Update install_ubuntu.sh * Update install_ubuntu.sh * Update gemm_test.cc * Update gather_op_test.cc * Update CMakeLists.txt * Removed submodule * update onnx-tensorrt submodule * update header file * Removed submodule * add submodule onnx-tensorrt kevin's branch shape-test' * add debugging code * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * merge master * Removed submodule * update onnx-tensorrt submodule * add more changes for dynamic shapes * Update tensorrt_execution_provider.cc * update for dynamic shape * update dynamic shape processing * fix logger issue * remove submodule onnx-tensorrt * add submodule onnx-tensorrt * add env variable min_subgraph_size * remove redundency * update document * use onnxruntime::make_unique * fix multi-run issue * remove some tests to save CI build time * Add dynamic shape test * Update TensorRT-ExecutionProvider.md * Add example of running Faster R-CNN model on TensorRT EP * Add more details on env variables * update environment variables * Update tensorrt_basic_test.cc * Update model tests * Update tensor_op_test.cc * remove --use_full_protobuf * Update build.py * User/xianz/telemetry (#2458) * enabme telemetry * enable telemetry * set enable telemetry as default * for debugging * remove log and set disable telemetry as default back * delete private file while testing * resolve comment: mainly add license header, rename macro and update docs * rewording in privacy.md * Fix integer overflow in cuda NonMaxSuppression implementation (#2540) * add test case that should pass but fail * fix nms * extract int_max_output_boxes_per_class * Introduce container type runtime checks and other improvements (#2522) Rework TensorSeq in a manner consistent with Tensor and SparseTensor in terms of type system setup. Reduce templating. Introduce helpers to ensure the same data type. Make OrtValue __dtor not virtual. Introduce ContainerChecker * Fix C API tests for centos and mac (#2544) * change c++14 to c++11 * add ld lib path for centos * enable csharp tests on macos * fix C API test on MacOS + fix manylinux dotnet install * fix manylinux dotnet install * fix lib link * Add back executable bit to build.py * Fix a bug handling negative begin pad values in Pad op (#2550) * Fix bug in Pad op * Update * DNNL CMAKE update (#2548) * Fix android build (#2558) * Update win-x86-ci.yml (#2557) Fix build pipeline break * Re-enable Windows C# tests (#2564) * disable onnx_test_runner -x invocations for dnnl (#2568) * Allow sequence length to be symbolic (#2559) * setup java ci mac (#2570) * make layernorm fusion to support opset 11 (#2545) * Fix a warning found in the latest VS release * Add more check on SkipLayerNorm and BiasGelu fusion (#2574) * Fix file not found error during docker build. (#2569) * Add ConvTranspose1D (#2578) * Ryanunderhill/packagename test (#2582) * [Nuphar EP] fixes for some object detection models (#2581) Update notebook tutorial with multi-threaded int8 GEMM from #2517 * EmbedLayerNormalization Fusion Improvement (#2553) Embedding layer norm fusion improvements - add more checks * Update version (#2584) * Temporarily exclude vgg19 test from Python backend test 1. temporarily exclude vgg19 test which comsumes too much memory, run out of memory on Upsquared device. Single test pass for vgg19, need furture investigation (#2588) 2. Update docker file to decrease the docker image size * Update docs for Android NNAPI EP (#2586) * Fix lto bug for protobuf and ubuntu * add path to build dir before test run (#2590) * Add missig env variables for mac pipeline test (#2595) * Fixed an issue in updating realized dims (#2597) when we update realized dims for scan's output, the sliced axis also needs to be inclusive, i.e. we should check with "dim >= insert_inclusive_axis", because the offset in the symbols are based on Scan sugraph. Otherwise, we would end up with shape mismatch later. * Java API for onnxruntime (#2215) * Add support for opset 11 in reshape fusion (#2592) Support opset verion 11 in reshape fusion * Rename automl python tools folder to featurizer_ops. (#2593) * Support opset 11 subgraph of Squad model in Embed Layer Normalization (#2605) Support opset 11 Squad model that is exported from PyTorch nightly. The embed layer uses Range op which is missed in the transformer. * symbolic shape inference: fix warnings in GPT-2 model (#2608) And revise nuphar perf test on BERT squad * Dump subgraph ID and fused graph ID (#2607) * Dump subgraph ID and fused graph ID Dump subgraph ID and fused graph ID for better debugging * Remove local static fused_count added a field global_fused_count_ to NupharExecutionProvider class * EmbedLayerNormalization Fusion For Dynamic Squad Model Opset 10 (#2613) Support subgraph of SQuAD model exported from pytorch with dynamic input axes * Allow providers to be set for InferenceSession at construction (#2606) * Remove unnecessary parameter in some places in GatherElements implementation (#2612) * Remove unnecessary parameter in some places * Update * Update * Make sure fenced tensor could not reuse other tensor. (#2561) Fix random error caused by this. * Improve Embed Layer Norm Fusion for SQuAD with static input shape (#2621) * fix float16 comparison in initializer (#2629) * epsilon attribute for layernormalization fusion (#2639) * removed unnecessary batch file and fix path (#2640) * Add shape inference to ConvTransposeWithDynamicPads schema (#2632) * Improve cuda expand() opeator's performance. (#2624) * Cuda pad optimize when no padding is needed. (#2625) * Shortcut cuda Pad() when no padding is needed. * Optimize cuda scatter() on 2D compatible. (#2628) * Optimize cuda scatter() on 2D compatible. * Add some comments. * fix build error for ARM (#2648) * Improve performance of resize() in Nearest mode (#2626) Special treatment for 2D, check same size as input image. And in 2d kernel, template use_expolation. * Fix memory exception in Layer Norm Fusion (#2644) * Windows CI changes(#2650) * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Revert "Windowsai without fi (#2701)" This reverts commit 285d4c85ff5c4e265f963208170304ef3461e684. * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Deref unique pointer for session_state * send shutdown event when dll is unloaded and EvaluationStop, SessionC… (#2704) * send shutdown event when dll is unloaded and EvaluationStop, SessionCreationStart Events. * Add EvalutationStart Event * add comment * use correct type for for loop (#2755) * ARM CI (#2759) * Set ARM agent pool * Set CMake generator to VS 2019 in ARM * Use system-wide CMake instead of custom version Our custom version is too old for VS 2019 * Use DML and build shared lib in ARM CI * Restore nuget packages in ARM CI * Disable DML * Refactor ARM debug/release builds * Use system packaged Python version * Remove hardcoded Python path * Downgrade Python to 3.7 for build * Remove explicit CMake path * Fix invalid JSON in cgmanifest.json (#2760) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * Remove usage of WHOLEARCHIVE in WinML CMake and add WinMLAdapterFactory (#2726) * Remove usage of WHOLEARCHIVE in WinMLAdapter CMake and add WinMLAdapterFactory * PR feedback, no need for dll(export) since using def file * PR comments * Small comment in gen_def.py * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * Move winml_provider_factory.h to proper location (#2801) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Filter CPU case for IsFloat16Supported (#2802) * Merge fixes * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * Fix warnings that cause build to fail * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * Make DML include path global (#2882) * Make DML include path global * Add generated cppwinrt headers to winml_lib_common * Integrate changes to WindowsAI to make ADO Build (#2886) * Revert "CMake cross-generator fixes (#2790)" This reverts commit dbe7d97fa1ab155f1309bced87199527e8f35bd2. * add additional suppress warning in onnx_proto * ignore /wd4996 warning * DML execution provider fixes * Revert "Revert "CMake cross-generator fixes (#2790)"" This reverts commit 1ae7b4bcbc02edc881ad28685da98e095dfceb17. * Update func signature of custom op function overloads * common devicehelpers fixes * Add pch.h for winml_lib_common * re-add winml_lib_common_dir/inc to include path for winml_adapter * User/orilevari/dml redist shared folder (#2890) * move dml nuget package directory up one level to make it shared between build flavors * Merge conflict fix * Revert "Merge conflict fix" This reverts commit 142fa72cf9ce4344ad717b50b7ea2b8582aadc7c. * Revert "Merge remote-tracking branch 'origin/master' into windowsai" This reverts commit 6e2126d46e5e5f564d65da37dd4f70c93dd81165, reversing changes made to b3f5583dc9249834b947c8ea905f6a98060d5bd6. * Make winml_test_common free of test macros (#2902) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Don't generate files outside the build root (#2914) * Don't generate files outside the build root * Add onnxruntime_EXTERNAL_DEPENDENCIES to WinML * Add DML depedency on RESTORE_PACKAGES * User/orilevari/fix yaml merge bugs (#2918) * Add winml test source parameter into cmake function (#2919) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Add arguments to unittest cmake functions * remove comment * Revert "Revert "Merge remote-tracking branch 'origin/master' into windowsai"" This reverts commit ade5abe72a4234fdbc3623093c61c02c6b0bdc26. * Fix breaks from merge with ORT master * Brianma/linux (#2917) * don't include windows.h in cross-plat header * add default case for switch statement * signed/unsigned mismatch fix Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> * User/sheilk/winml adapter c api (#2891) * Create winml adapter c api * fix build * make it build * move adapter into onnxruntime core/session * entry point not exported * minor changes * make model metadata work * make tests pass * implement all the model reflection apis on the adapter c abi * update the new ort interface to create a lotus ennvironment with a logging sink * start adding ort env * move all winml code into adapter folder/lib to isolate it * ensure a single logging manager at a time * start refactoring session * refactor session creation interface * add cpu and dml session option methods to adapter * finish session init * stub out interfaces in ort lib to perform similar mechanics of iinference session * enable profiling, and enable schema override * update session register graph transformers * turn back on custom registry for custom ops * Add sync api * add last c api stubs * should build... but all feature values are broken since this is in flight to moving all implementation details into ivalue * remove ep adapter header * Implement DML execution provider functions from adapter (#2846) * Implement DML execution provider functions from adapter * Use functions in OnnxruntimeEngine.cpp * make map/sequence type_infos freeable, and start implementing ivalue * make it build again * implement value methods * implement remaining methods * remove com adapter abi * check dml session * cache the allocator on ivalue * check if resource is cpu/gpu when access its mutable data * update tensor * mismatched parentheses * fix tensor base and binding obj * it evaluates tensors! sometimes... * minor fixes * enable gpu evals * wrapper all existing winml adapter apis with API_IMPL to try catch (#2854) * update winml... tensor strings are broken, need to template tensorbase to do different things for strings * make tensor strings work with 2 copies in/2 copies out * Fix tensor string and allocator bug * make maps work again... needs some fixes still * Make it build! * enable map inputs * map outputs * unbound outputs for sequences and maps * User/xianz/merge windowsai (#2883) * Packaging pipeline changes for VS 2019 (#2711) * Tiny fix to codegen * Simplify cache implementation and avoid static variables that may carry over between models * Extend DML kernels (#2641) * Additional DML operators * Check unsupported attributes and inputs * Address PR comments * Add kernel capability function used for partitioning, and re-enable stride-based int64 support based on value range * Fix test failures * Build fix * PR comments * Update Nuphar tutorial notebook (#2721) 1. Reflect int8 GEMV improvements for multi-threading from #2696 2. Add notes on multi-threading control using OpenMP 3. Add samples of running multi-isa AOT, and show int8 GEMM differences between AVX and AVX2 4. Add rnn_benchmark example to resolve #1993 * Add schema for new Qops (#2611) * Add schema for new Qops * adding shape inference + qlinearaveragepool * plus review comments * plus review comments * updates per review comments * plus review comments * [server] Add supposed for model_name and model_version as cli parameter (#2708) * remove 64bit warning message from python validation. (#2727) * MLAS: ARM64 build fix (#2734) fix bad usage of vreinterpret to cast vector element types * Fix broken python docs links (#2740) * Fix build on Mac OS (#2731) mac os ld doesn't support --while-archive, correct option is -all_load * fix ngraph wheel (#2737) * fix ngraph wheel 1.1.0 onnxruntime_ngraph wheel doesn't work * remove libdnnl.so in nGraph Libs * make it easy to compare * Split onnxruntime server to a separated folder (#2744) * Fix build for Python 3.8 (#2747) * Fix build for Python 3.8 * Update protobuf to 3.11.2 (#1928) Update protobuf to 3.11.2 (#1928) * Change default optimization level to All (from Basic) (#2745) * change default optimization level to All (from Basic) * fix test * fix c# test * Update numpy to 1.18 (#2758) * Update numpy to 1.18 * Pipeline changes for python 3.8 (#2753) 1. Pipeline changes for python 3.8 2. Fix a regression in setup.py which was just introduced in the previous commit. Please notice, we still haven't made python 3.8 + Windows + CUDA work. * Add basic stacktrace output for posix debug builds. (#2749) * [NupharEP] fix a race condition when multiple sessions running different models concurrently (#2772) * Revert "Change default optimization level to All (from Basic) (#2745)" This reverts commit 56bb503c2f26474b6613bcb2a198691a11dcef38. * Fix typo in error message (#2736) * Rename MKL-DNN to DNNL to fix broken link (#2730) * Fix nightly build version number issue * Pass BUILD_BUILDNUMBER to linux docker * Disable featurizers in python packages * Import more featurizers (#2781) Make kernels non-template. Add input constraint for learnt data. Add min_max_scalar_transformer, robust_scalar_transformer, inputation_marker_transfomer, label_encoder_transformer, missing_dummies_transformer along with tests. Advance Featurizers library commit. * Implement a more stable softmax (#2715) * Implement a more stable SoftMax e^x is represented as infinity if x is large enough, like 100.f. Infinity divided by Infinity is a NAN. Thus, softmax gets a NAN if one or more item are large enough. A math transform as below is leveraged to get a stable softmax: e^xi/(e^x1 + ...e^xn) = e^(xi - max) / (e^(x1 - max) + ... + e^(xn - max)) And for convenience, force max to 0.f if all xi are negative * Contributing: Fix a typo (#2784) * ACL EP GEMM improvements (#2780) When it is posible we use a fully connected layer instead of the gemm implementation. This will let the library use the best implementation based on the input data. * ACL EP convolution improvements (#2774) Added the optimized implementation for depthwise convolution for both ACL v19.02 and ACL 19.05. Also the pointwise convolution seems to be more optimal in the CPU implementation so we opted for that instead. * Add script for release Nuget validation (#2719) * Initial commit * Nits * Disable a test temporarily * Change working directory * Test * Add download python step * Test update * More changes * Fix space issue * Fix * Verify nuget signing * Fix * Spaces * PR feedback * Nit * Fix * Fix * Remove temporary changes * add uint8 support to where op (#2792) * Improve bert optimization script: (#2712) (1) Move input int64=>int32 conversion to embed layer fusion. (2) Output epsilon attribute for LayerNormalization fusion. * add session creation time cost. (#2798) * ML.NET team needs featurizers within a package (#2789) Add auto ml featurizers to Windows, MacOS as well as to GPU packaging-pipelines. * Initialize max of softmax with lowest of float (#2786) * MLAS: update SGEMM threading parameters (#2808) * add interface to copy batch tensors. (#2807) * add interface to copy batch tensors. * onnxruntime * speed up Windows TRT CI (#2811) * don't run cuda tests if building with tensorrt * remove unnecessary build options for win trt ci * refactor win gpu tensorrt ci yml * --numpy_version=1.17 * update * update * azcopy and cuda path * Update test data (#2356) * Add timeseries imputer transformer featurizer kernel (#2813) Make kernels non-template. Add input constraint for learnt data. Fixup tests. Add two more featurizers along with tests. Tests fail. min_max_scalar_transformer robust_scalar_transformer Fix tests serialized stream by prepending version bytes. Add inputation_marker_transfomer and the test. Fix up float/double type designations. Added label_encoder_transformer along with a test. string_throw case is broken at the momement. Fix labelencodertransfomer_test.cc string_throw case Rename maxabsscalertransformer_test.cc Add MissingDummiesTransformer along with the test. Update manifest. Add TimeSeriesImputerTransformer definition, implementation and tests * Fix memory leak in TRT (#2815) * fix memory leak issue * revert EP_FAIL on enueueV2 * Add manifest missing comma * Run static code analyzer on most of our code (#2817) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * update quantization doc (#2783) * update documentation for quantization script * plus some spell corrections * Filter CPU case for IsFloat16Supported (#2802) * update default optimization level + fix gemm_activation fusion (#2791) * update defualt optimization level + fix gemm_activation fusion * fix typo * add unit test and incorporate review comments * fix test comment * Fix dnnl wheel package name (#2823) * Append '-dnnl' to whl package name when --use_dnnl * Update build.py * Update Ubuntu & TensorRT version in README (#2820) Dockerfile.tensorrt is using nvcr.io/nvidia/tensorrt:19.09-py3 as base Image, update Ubuntu and TensorRT version according to https://docs.nvidia.com/deeplearning/sdk/tensorrt-container-release-notes/rel_19-09.html#rel_19-09 * Merge fixes * Add OneHotEncoder and HashOneHotEncoder kernels. (#2830) Add defs and imlementation for OneHotEncoders, adjuist date_time_transformer kernel and test. Add OneHotEncoder kernel test. Add HashOneHotVectorizerTransformer unit test. This does not link due to multiple definitions of functions that are included into header from a CPP file. * Upgrade gtest to the latest version (#2827) WinML would like to update the googletest submodule. They want some newer features (namely GTEST_SKIP to skip tests programmatically and be able to skip entire fixtures easily) and would need to update the submodule version. However, because the new version of code hit a bug in gcc, even though the bug is already fixed in the latest gcc but we're using gcc 4.8.x and it won't get patched for the bug, so we have to do a compromise, change our code a little bit to make it work. The gcc bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=51213 * Add support for int64_t for topk CPU. Fixes github issue #2806. (#2833) * Ignore allocator type in ExecutionProviders allocator map. Make default initialization of OrtMemoryInfo more clearly invalid. (#2768) * Remove allocator type from the key comparison in ExecutionProviders. Remove usage of DummyArena as it's no longer necessary. * Fix x86 tests where arena allocator is disabled. Make initialization of OrtMemoryInfo clearer by adding Invalid enum value. * Make OrtValueNameIdxMap::MaxIdx more intuitive. * Convert ExternalProject Featurizers into git submodule (#2834) Add git submodule for Featurizer library. Update cmake to build for git submodule. * add domain check for nodes + update documentation (#2831) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Fix Linux CUDA nuget packaging pipeline break * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * update optimization doc for BERT related fusions (#2819) * Add bert related transformers to doc * Add execution provider and comment for bert optimizations * Add comment about accuracy impact of approximation * Fix warnings that cause build to fail * MLAS: enable threading for quantized GEMMs (#2844) * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * remove old winmladapter cpp Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * move sequence implementation into ort lib... still commented out... need to turn back on... * begin sequence implementation * make maps and sequences work * fix broken tests * remove dead code * misc cleanup * CR feedback * User/xianz/winml adapter c api (#2869) * wrapper all existing winml adapter apis with API_IMPL to try catch * Return HR or Throw for WinML adapter APIs if failed * undo macro wrapper for two places * Wrap error macros around ort apis, too. * address CR feedback #2 * add more api throw/return macros * Revert changes no longer needed * revert changes to cxx api * format winml lib.ort and winml adapter * remove static pheonix singleton Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * missing use_dml check in winml_adapter_session (#2930) * --use_dnnl flag was mangled in merge (#2931) * use dml macro not wrapping custom registry code (#2934) * Disable LNK4199 winml_dll to enable cuda builds (#2936) * Disable LNK4199 in winml_dll * linkler->linker * LearningModelSessionAPITestGpu.CreateSessionWithCastToFloat16InModel should return DXGI_ERROR_UNSUPPORTED when FP16 not supported (#2937) * Disable LNK4199 in winml_dll * linkler->linker * Need to return DXGI_ERROR_UNSUPPORTED when Model does not support fp16 * Publish build symbols (#2939) * Publish build symbols * Don't upload PDBs for .exe files * Make x86 build (#2943) * fix last remaining size_t/int64_t warnings->errors (#2948) * TensorString, Sequences and Maps use the first allocator, but should use the cpu default allocator. (#2952) * fix tensor string allcoator * clean up default allocator usage for strings in winml lib/api.ort Co-authored-by: Ryan Lai <ryalai96@gmail.com> * Handle tensor shape of zero (#2954) Co-authored-by: Ryan Lai <ryalai96@gmail.com> * CR feedback (#2970) * CR feedback * fix weird formatting on privacy readme * Add 'All rights reserved.' everywhere * readd all rights reserved to winml_provider_factory.h * remove extra space in comment * remove extra whitespace * fixes post master merge * remove winml from nuget gpu pipeline * set IR VERSION on generated_model in rnn_benchmark (#2972) * Fix slice conformance failures (#2908) Co-authored-by: Adrian Tsai <adtsai@microsoft.com> Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Paul McDaniel <paul_mcdaniel@hotmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Dwayne Robinson <fdwr@hotmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com>
2020-02-05 01:12:19 +00:00
[threading(both)]
[marshaling_behavior(agile)]
[dualapipartition(1)]
runtimeclass TensorUInt64Bit : ITensor, [contract(MachineLearningContract, 2)]Windows.Foundation.IMemoryBuffer
{
static TensorUInt64Bit Create();
static TensorUInt64Bit Create(Windows.Foundation.Collections.IIterable<Int64> shape);
static TensorUInt64Bit CreateFromArray(Windows.Foundation.Collections.IIterable<Int64> shape, UInt64[] data);
static TensorUInt64Bit CreateFromIterable(Windows.Foundation.Collections.IIterable<Int64> shape, Windows.Foundation.Collections.IIterable<UInt64> data);
Windows.Foundation.Collections.IVectorView <UInt64> GetAsVectorView();
[contract(MachineLearningContract, 2)]
{
//! Creates a tensor object, allocates a buffer of size 'shape', and copies all of 'data' into it.
static TensorUInt64Bit CreateFromShapeArrayAndDataArray(Int64[] shape, UInt64[] data);
//! Creates a tensor object, creates a tensor of size 'shape', and uses the data in 'buffer'.
static TensorUInt64Bit CreateFromBuffer(Int64[] shape, Windows.Storage.Streams.IBuffer buffer);
}
}
[contract(MachineLearningContract, 1)]
INBOX_ONLY([static_name("Windows.AI.MachineLearning.ITensorInt64BitStatics", 9648ad9d-1198-4d74-9517-783ab62b9cc2)])
INBOX_ONLY([interface_name("Windows.AI.MachineLearning.ITensorInt64Bit", 499665ba-1fa2-45ad-af25-a0bd9bda4c87)])
Merge windowsai (winml layering) into master (#2956) * Initial Commit * Merged PR 3985217: add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc (#2346) add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc and violating our OS layering requirements. We linked against onecoreuap_apiset.lib in VB so we will continue doing this, but I am still unsure why not to link against onecore instead since that is where we ship. However, since Sheil is the owner of this code we will wait to discuss with him before changing anything. * Initial changes for layering * more snipping to get core into ort * update build instructions to include --build_shared_lib (#2358) * update build instructions to include --build_shared_lib * fix line breaks * Task 23998197: add winml_lib_core into onnnxruntime.dll (#2368) * Task 23998197: add winml_lib_core into onnnxruntime.dll * PR feedback build break on perf_test * return proper error when the model path isn't found (#2391) * LearningModelSession is cleaned up to use the adapter, and parts of b… (#2382) this is a big PR. we are going to move it up to layer_dev , which is still a L3 so we are still safe to do work there agile. we are going to move this into the L3 so that ryan can start doing intergration testing. we will pause for a full code review and integration test result prior to going into the L2. >>>> raw comments from previous commits >>> * LearningModelSession is cleaned up to use the adapter, and parts of binding are. * moved everything in the winmladapter made it all nano-com using, WRL to construct objects in the ORT side. base interfaces for everythign for winml to call cleaned up a bunch of winml to use the base interfaces. * more pieces * GetData across the abi. * renamed some namepsace cleaned up OrtValue cleaned up Tensor cleaned up custom ops. everything *but* learnignmodel should be clean * make sure it's building. winml.dll is still a monolith. * model moved over. everything builds clean. step ! * weak ref comment * Layer dev paulm (#2408) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * Layer dev paulm (#2414) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * Layer dev paulm (#2423) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * Layer dev paulm (#2424) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * Layer dev paulm (#2425) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Layer dev paulm (#2426) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * fixed map and sequence when passing stl types across the ABI . found a leak in nvidia driver, but skipped it. all winmlapitests pass now * Moved SessionOptions over to the abi * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * Register WinML TraceLogging provider on Onnxruntime.dll (#2455) * Register WinML TraceLogging provider on Onnxruntime.dll * Add ifdef to make sure trace logging provider has telemetry option when LAYERING_DONE * No need for ifdef for TraceLoggingOptionMicrosoftTelemetry * PR feedback * Move etw registration into lotus environment constructor and deresgister in lotus environment destructor * Brianma/cpuwinml (#2466) * allow building winml cpu without dml. * Brianma/breaks (#2469) * fix some more breaks * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * move dml checks out of winml and into the adapter * better error handling * Brianma/fi (#2470) * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * fix bad merge * Checking in a staging checkpoint point so that Ryan can work with me in parrallel * build break. * Brianma/testfails (#2473) * add missing ir version to dictvectorizer-string.onnx * add missing ir version to relu.onnx * add missing ir version to zipmap*onnx * add IR version to manually generated models * remove an unnecessary ifdef dml * Brianma/windowsai fi (#2475) * update dockerfiles/README (#2336) * Make elementwise op run 4 items per thread (#2335) Description: Describe your changes. Make elementwise op run 4 items per thread unroll for loop to leverage ILP remove unnessary N==0 check inside elementwise GPU kernel Motivation and Context Why is this change required? What problem does it solve? It can improve the performance of GPU elementwise ops. ~2% performance gain on popular NLP bert model. If it fixes an open issue, please link to the issue here. * Add CUDA GatherElements kernel (#2310) * Updates * Update test * Update * Updates * nits * PR feedback * Update * Update * PR feedback * PR comments * Update * Fix build * Fix build * Nits * Fix * Layer Normalization Fusion (#2319) basic layer normalization transform * Add FastGelu Cuda Op for Gelu and Add bias fusion (#2293) * Add FastGelu cuda op * Add AddBiasGelu for experiment * Revert "Add AddBiasGelu for experiment" This reverts commit 5c1ee019858c657e6bb75887265cb85675626e5b. * Add bias * Add unit tests * update comment * update script * fix build error * update coding style * update for CR feedback Enable half2 optimization only when cuda arch >= 7.0 * move _Tanh to common.cuh * implement CPU contrib OP Attention (#2333) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanUnusedInitializers. (#2320) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanupUnusedInitializers. This means initializers that have been replaced during graph optimizations are not left in the GraphProto when we save an optimized model. * Handle edge case where a model has an unused initializer with matching graph input by also removing the graph input. * Use non-const iterators in std::find_if calls to make centos build happy. * Nuget pipeline changes (#2305) 1. refactor the pipeline, remove some duplicated code 2. Move Windows_py_GPU_Wheels job to Win-GPU-CUDA10. We'll deprecated the "Win-GPU" pool 3. Delete cpu-nocontribops-esrp-pipeline.yml and cpu-nocontribops-pipeline.yml 4. In Linux nuget jobs, run "make install" before creating the package. So that extra RPAH info will be removed * Cuda Reverse Sequence Op, maping types of same size using same template function. (#2281) * Set ElementType to String type of node metadata, instead of byte[] (#2348) * Set ElementType to String type of node metadata, instead of byte[] * Fix spacing * Introduce PrimitiveType into a Type System along with an integer constant (#2307) Improve perf by avoiding GetType<T>() calls. Introduce MLTypeCallDispatcher to switch on Input Type. Add Tensor IsType<T>() fast method. * Fix/test dim value of 0 handling in a couple of places (#2337) * Update the CUDA Where implementation broadcasting logic to handle a dim with value of 0. Add unit test Also add unit test for unary op with dim value of 0 * Exclude ngraph from Where test with 0 dim. * Openvino EP R3.1 onnxrt server (#2357) * onnxrt server with OVEP * onnxrt server with OVEP * Update Dockerfile.server.openvino * onnxrt server OVEP fix reviews * onnxrt server OVEP fix reviews * Implement cuda nonzero op. (#2056) Implement cuda nonzero op. * Direct use python numpy array's memory if already contiguous. (#2355) * Direct use python numpy array's memory if already contiguous. This could greatly improve performance for session with large input, like big image 1920x1080 fastrcnn, 30~40% speed up could be achieved. * Add test case enforce contiguous/non-contiguos numpy array as inputs. * Add helper to create output to minimize binary size. (#2365) Add ConstEigenTensorMap typedef so we don't unnecessarily const_cast the const input Tensor. * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS (#2369) * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS * update * Add Tracelogging for profiling (#1639) Enabled only if onnxruntime_ENABLE_INSTRUMENT is ON * test bidaf with nuphar for avx target (#2370) increase nuphar test coverage a bit * Fix a bug in TLS refcount that may destabilized CUDA CI (#2374) * update output size calculation for resize (#2366) * change how output size is calculated for resize op * add tests for ver 10 resize * Extend OneHot CPU kernel to support more types (#2311) * Extend OneHot CPU kernel to support input int64_t, depth int32_t, output float * Skip BERT before the test data fix is picked up * Fix bug with Slice. Need to pass in flattened input dimensions so the initial offset into the input is calculated correctly. (#2372) * Add opset 11 version of Split to CUDA ops (#2376) Organize the CUDA ops definitions so all the opset 10 and 11 parts are together (same setup used for CPU ops) * Layer Norm Fusion Fix (#2379) * layer norm fusion fix * Add input shape check in code and unit tests * Fuse Add + Gelu (#2360) Implement the transformer to fuse add + gelu Implement the accurate kernel * Skip layer norm transform (#2350) * skip layer normalization transformer * Another try to stabilize CUDA CI (#2383) The root cause seems to be failure in CUDA dealloc when tear down. cudaFree return code was ignored before, so should the debug check. * fix BUILD.md typo (#2375) build.py: error: argument --config: invalid choice: 'RelWithDebugInfo' (choose from 'Debug', 'MinSizeRel', 'Release', 'RelWithDebInfo') * Fixed compilation with ngraph (#2388) * Fix reuse logic in allocation planner. (#2393) * Fix reuse logic in allocation planner. * PR comments * Add helpful comments * Don't allow reuse across string tensors. * [NupharEP] Multiple optimizations (#2380) Fuse transpose into MatMul Implement Pow and constant scalar simplification Vectorize ReduceMean Improve symbolic shape inference Minor updates for better debugging in fused function name * Avoid using the default logger in the graph lib and optimizers (#2361) 1. Use the session logger if it is available. 2. Don't disable warning 4100 globally. We should fix the warnings instead of disabling it. * Change CUDA implementation of Transpose to support all fixed size tensor types (#2387) * Change CUDA implementation of Transpose to not use a typed kernel so we can support more types with minimum binary size. Add support for 8, 16, 32 and 64 bit types. Add unit tests. Add method so the implementation can be called directly (will be used by CUDA Scan very soon). * Disable TensorRT for MLFloat16 and int8 unit tests. * Address PR comment and add support for calling cublas implementation if type is mlfloat16. * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. (#2398) * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. * [NupharEP] force some low/zero cost ops to be inlined (#2409) * fix cross compile bug (#2415) * Minor optimization: if a node has already been placed, there's no need to find a kernel for it. (#2417) * Add Reshape Fusion (#2395) * Add reshape fusion * Add some comments * update comments * update comment format * update according to feedback * update for recent logger change * fix build error * (1) Support both input and output edges in find path in graphutils (2) Add a test case of only one constant initializer of Concat input. (3) Refactor ReshapeFusion class to allow add more subgraph fusion in the future. * fix error * (1) loose constraint on initializer: non constant is allowed for reshape fusion. (2) Change versions type to vector. (3) Add logging. (4) Return false when multiple output edges matched in FindPath. Add comments. * only allow one direction (input or output) in FindPath * [NupharEP] Update notebook and docker image (#2416) Add BERT squad in Nuphar tutorial Enhance speed comparsion readability * Fix the issue in matmul_add_fusion (#2407) Fix the issue in matmul_add_fusion If Muatmul + Add has shape [K] * [K, N], reset it to [1, K] * [K, N] will make the output shape to [1, N] will also requires a reshape on the output. Fix: just remove the shape reset to not fuse it. Add a negative test case for matmul+add fusion * feat(treeregressor): Update TreeEnsembleRegressor for type support (#2389) Updates the `TreeEnsembleRegressor` to allow for `double`, `float`, `int64`, and `int32` inputs to match the upstream specification. Signed-off-by: Nick Groszewski <nicholas.groszewski@capitalone.com> * onnxrt server documentation update (#2396) * Added support for Pad-2 operator in OpenVINO-EP (#2405) * Add CUDA If operator. (#2377) * Add CUDA If operator. Uses CPU operator for implementation. By adding a CUDA version the inputs/outputs (with the exception of the 'cond' input) stay on GPU, and no other logic is required to avoid a copy to CPU across the control flow node. * Improved documentation for onnxruntime::utils::SwapByteOrderCopy(), added precondition check. * Fix the type constraints on CUDA If operator to exclude strings. (#2431) * add Im2col<uint8_t> (#2438) * Adjust codegen vectorization width from target (#2439) * Adjust codegen vectorization width from target * Add CUDA Scan operator. (#2403) * Add Scan CUDA op. Uses CPU implementation for logic. Added some device specific functors for handling when data needs to be manipulated on a different device. Added ability to override the materialization logic in the OrtValue slicer so DML can plugin their handling. * Fix Windows GPU C API packaging pipeline failure (#2440) Fix Windows GPU C API packaging pipeline failure (#2440) * Correctly handle implicit inputs for fused nodes (#2390) * Correctly handle implicit inputs for fused nodes Previously, nuphar's partitioning function didn't include node's implicit inputs into the inputs list of MetaDef, and hence a crash was triggered in the onnx graph checker. This commit fixed the issue. Furthermore, it also fixed a related issue where we didn't add implicit inputs into graph_inputs_excluding_initializers_ in Graph::SetGraphInputsOutputs. the issue was that graph_inputs_including_initializers_ populated by SetInputs (e.g. called by FunctionImpl::FunctionImpl) may contain implicit inputs which were not of any node's initializers in the graph. Because they were not part of any initializers, these implicit inputs couldn't be visited by going through all nodes' inputs. Consequently, they would *not* be added into graph_inputs_excluding_initializers_. We fixed the issue by first copying the populated graph_inputs_including_initializers_ into graph_inputs_excluding_initalizers_, which then had both initializers and non-initializers as its initial content. Later, we erase initializers from the list. In this way, we can ensure all implicit inputs to remain in graph_inputs_excluding_initializers_. * refined comments and fixed duplicates Address CR by revisiting comments in terms of implicit inputs Also fixed an issue by skipping duplicates while copying inputs from graph_inputs_including_initializers_. * address CR explain why we need to collect nodes' implicit inputs * don't rely on pointer values for iterating std::set Previously, openvino relied on iterating a set of NodeArg pointers to construct inputs and outputs for a fused graph. It could cause non-determinism. The reason was that although iterating std::set by itself is stable, pointer values of NodeArgs may vary. Consequently, we could end up visiting the set's elements in different orders for different runs for the same test, which resulted in constructing inputs (and outputs) with different orders to the fused graph. For example, for the same test, we may have inputs [A, B] in some runs but inputs[B, A] in others. Let's use std::string as the key type to avoid such nondeterminism. This commit also added implicit inputs into meta->inputs while returning the capability from the openvino provider. * Fixed another latent issue in openvino's GetCapability function The issue was that we couldn't simply erase fused_inputs and fused_outputs while iterating the nodes. For example, an output NodeArg may have multiple uses, and it's wrong if we erase it from fused_outputs when we encounter only one of its uses as input. * Remove DeviceAllocatorRegistry class (#2451) Remove DeviceAllocatorRegistry class * CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library. * Parallel Gelu with ParallelFor (#2399) Parallel Gelu to get better performance for Gelu * Clean up build.py (#2446) * Pull the latest image before running docker build * Fuse SkipLayerNorm with Bias (#2453) Fuse SkipLayerNorm with Bias * Allow more than one invocation of CreateEnv in the same process. (#2467) * Allow more than one invocation of CreateEnv in the same process. * Fix centos build * Symbolic shape inference improvements: (#2460) * Symbolic shape inference improvements: - add a mode to guess unknown ops' output rank - add support for GatherND - add support for If - fix a bug in get_int_values when then tensor rank > 1D, by treating it as no sympy data - add symbol to literal merge when ONNX silently merges dims - fix a bug in Concat when input dim is 0 - fix a bug in ConstantOfShape that computed dim is not updated - add support for dynamic shape in ConstantOfShape - fix a bug in Loop output shape that loop iterator dim is not inserted at dim 0 - add support for dynamic padding in Pad - add support for dynamic shape in Reshape - add support for Resize with opset > 10, by treating output dims as dynamic - fix a bug in Slice when starts/ends are dynamic - restrict input model to opset 7 and above - make output model optional to avoid disk write when testing Run model tests for symbolic shape inference Reduce 2GB docker image size of nuphar * add additional test data set for nuget pipeline (#2448) * add SAS token to download internal test data for nuget pipeline * update azure endpoint * fix keyvault download step * fix variable declaration for secret group * fix indentation * fix yaml syntax for variables * fix setting secrets for script * fix env synctax * Fix macos pipeline * attempt to add secrets to windows download data * fix mac and win data download * fix windows data download * update test data set url and location * Revert "Brianma/windowsai fi (#2475)" This reverts commit 5780b864a15513fda4eadbfc2b5345fefe70b5ec. * Add scenario tests (#2457) * Add scenario tests * Remove TODO from model license * Add winml_api test dependency * fix model load test. fi from master changed the constructor (#2483) * make api tests all pass (#2486) * fix bad merge * fix bad model merge * Layer dev paulm (#2492) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * Rename ambiguous header (#2489) * fix one more missing IR version model (#2500) * add missing IR version to 4 more models used by scenario tests (#2501) * Add CLI parameters to test runner, build WinML in ARM and x86 CI (#2479) * Support test parameters through CLI arguments * Add WinML do Windows x86/ARM CI builds * Code style fixes * Update googletest Remove GPUTEST macros everywhere now that GTEST_SKIP is supported * Refactor main.cpp * Build scenario tests without DML * Link scenario tests to DML when it's enabled (#2502) * Layer dev release pipeline (#2488) Adds winml binaries to existing cpu nuget package, and creates new gpu dml nuget package with winml binaries and DML EP. * Layer dev paulm (#2506) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * Remove usage of IOBinding in WinML and use C_API Run method (#2504) * remove usage of iobinding * Change data structure to use vector of Ort::Values * Polish bind input / output * Use C APIrun method * Update providers on evaluate getresults * Remove run and IObinding interface from WinMLAdapter * Remove use of IObinding * bind unbound outputs code moved to learningmodelbinding * clean up unneeded istensor adapter function * Fix comment * Check if session is closed before binding and clearing * PR feedback * Layer dev paulm (#2507) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * Make tests dependend on winml_dll (#2509) * add dml binaries to DirectML package and be more explicit about condition variables (#2520) * re-enable warnings for winml builds and fix the warnings that were hiding (#2526) * turn devmode back on for winml builds * fix some warnings. include protobuf in a way that disables some warnings * undo protobufhelpers changes and just ignore 4100 errors in pb code * attempt to isolate protobufhelpers errors * add template specialization for getting tensor proto data * Layer dev paulm (#2533) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * Spawn child process to run DeviceLostRecovery scenario test (#2530) * Spawn child process to run DeviceLostRecovery scenario test * Layer dev paulm (#2536) ori said yes * add missing namespace to winml_trace_logging_provider in lotusenvironment.h (#2542) * Handle exception thrown from all apis in WinMLAdapter (#2539) * various changes to unblock windowsai ADO build * Fix custom ops scenario tests (#2562) * Do not shutdown protobuf after ort environment gets destroyed. Lazy load lotus environment first time it is needed * comment typo * pr comment about calling phoenix singleton * Make lotus_environment static in winmladapter * Layer dev paulm (#2567) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * making windowsAI azure dev ops work. * code review comments. * revert changes * Cmake and preprocessor fixes that where uncovered by building on agents without DML available via SDK * Layer dev dml delayload (#2580) * Brianma/cpu (#2583) * don't include dml stuff in cpu builds * tests that link the image lib also need the telemetry lib now * Throw Winml_err_invalid_binding if binding gpu resource on cpu device (#2589) * Throw Winml_err_invalid_binding if binding gpu resource on cpu device * PR comments. No need to query executionprovider for is gpu device * User/xianz/ortthrow (#2596) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * User/xianz/ortthrow (#2599) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * add status error message * Remove uwp onsuspending winrt call because logruntimeperf is getting removed (#2630) * User/xianz/dedup telemetry (#2631) * investigate duplication of telemetry in winml and ort * remove winml telemetry events * telemetry executionProviderEvent * remove unneccessary file and refactor code little bit * Revert back TelemetryEvent, which send up ETW event. * merge changes from layer_dev to windowsai (#2638) * Remove underscore from googletest names (#2616) * Fix leaking memory allocator Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24278761 and https://microsoft.visualstudio.com/OS/_workitems/edit/24330198 * Explicitly initialize Ort::Value with nullptr * Cache WinML adapter * bad merge * define private version of dxcore enum that is added in 19H1 SDK. (#2654) * add comment for explaning private definition of dxcore d3d feature level ennum value. (#2672) * do not package directml.pdb for redist packages. (#2676) * Fix leaking operator registry (#2645) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24354916 * User/orilevari/windowsai master merge (#2674) merge resolutions included pulling in telemetry logic that was merged to master and not windowsai and dereferencing InferenceSession::sessionstate now that it is a unique pointer * Delete Ort Allocator in LearningModelBinding (#2653) * Delete OrtAllocator in LearningModelBinding * PR comments to make Ort::Allocator a smart pointer * Small comment change * PR feedback to clean up code * PR feedback on move semantics * Clean up std::move * Fix memory leaks (#2679) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24356109, https://microsoft.visualstudio.com/OS/_workitems/edit/24388361 and https://microsoft.visualstudio.com/OS/_workitems/edit/24388596 * various changes to properly organize and skip GPU tests. For now for No DML builds we will not run GPU tests at all. In the future we should adapt the tests to expect the appropiate errors. (#2695) * Windowsai without fi (#2701) * Disable Attention fusion tests when DISABLE_CONTRIB_OPS is defined (#2529) * Setup java ci (#2528) * Add provision in ORT for session options to be parsed when available via model file (#2449) * Initial commit * Fix gitmodules * Nits * Nits * Updates * Update * More changes * Updates * Update * Some updates * More changes * Update * Update * Merge * Update * Updates * More changes * Update * Fix nits * Updates * Fix warning * Fix build * Add comment * PR feedback * PR feedback * Updates * Updates * Update * More changes * Fix build break * Comment test for now * Updates * Updates * PR feedback * Updates * Nits * Add tests * Fix build * Fix build * Fix build * Fix build break * Fix build * Nits * PR feedback * More change * Expose GetSessionOptions in pybind logic and add unit test for python * Fix build * PR feedback * PR feedback * Revert "Disable thread pool creation when enabled OpenMP (#2485)" (#2535) This reverts commit 7c7d5a149c9ed52eec67304bae5c4b132166a8a1. * Add dynamic shape support in TensorRT execution provider (#2450) * remove onnx-tensorrt submodule * add new onnx-tensorrt submodule (experiment) for trt6 * update engine build for trt6 * update compile and compute for tensorrt6.0 * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * switch to onnx-tensorrt master for TensorRT6' * Update tensorrt_execution_provider.cc * Handle dynamic batch size and add memcpy in TensorRT EP * update test cases * Update tensorrt_execution_provider.cc * update onnx-tensorrt submodule * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.ubuntu_tensorrt * Update run_dockerbuild.sh * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update concat_op_test.cc * Update tensorrt_execution_provider.cc * Upgrade TensorRT to version 6.0.1.5 * Update onnxruntime_providers.cmake * Update CMakeLists.txt * Update reduction_ops_test.cc * Update install_ubuntu.sh * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.tensorrt * Update BUILD.md * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update onnxruntime_providers.cmake * Update install_ubuntu.sh * Update install_ubuntu.sh * Update gemm_test.cc * Update gather_op_test.cc * Update CMakeLists.txt * Removed submodule * update onnx-tensorrt submodule * update header file * Removed submodule * add submodule onnx-tensorrt kevin's branch shape-test' * add debugging code * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * merge master * Removed submodule * update onnx-tensorrt submodule * add more changes for dynamic shapes * Update tensorrt_execution_provider.cc * update for dynamic shape * update dynamic shape processing * fix logger issue * remove submodule onnx-tensorrt * add submodule onnx-tensorrt * add env variable min_subgraph_size * remove redundency * update document * use onnxruntime::make_unique * fix multi-run issue * remove some tests to save CI build time * Add dynamic shape test * Update TensorRT-ExecutionProvider.md * Add example of running Faster R-CNN model on TensorRT EP * Add more details on env variables * update environment variables * Update tensorrt_basic_test.cc * Update model tests * Update tensor_op_test.cc * remove --use_full_protobuf * Update build.py * User/xianz/telemetry (#2458) * enabme telemetry * enable telemetry * set enable telemetry as default * for debugging * remove log and set disable telemetry as default back * delete private file while testing * resolve comment: mainly add license header, rename macro and update docs * rewording in privacy.md * Fix integer overflow in cuda NonMaxSuppression implementation (#2540) * add test case that should pass but fail * fix nms * extract int_max_output_boxes_per_class * Introduce container type runtime checks and other improvements (#2522) Rework TensorSeq in a manner consistent with Tensor and SparseTensor in terms of type system setup. Reduce templating. Introduce helpers to ensure the same data type. Make OrtValue __dtor not virtual. Introduce ContainerChecker * Fix C API tests for centos and mac (#2544) * change c++14 to c++11 * add ld lib path for centos * enable csharp tests on macos * fix C API test on MacOS + fix manylinux dotnet install * fix manylinux dotnet install * fix lib link * Add back executable bit to build.py * Fix a bug handling negative begin pad values in Pad op (#2550) * Fix bug in Pad op * Update * DNNL CMAKE update (#2548) * Fix android build (#2558) * Update win-x86-ci.yml (#2557) Fix build pipeline break * Re-enable Windows C# tests (#2564) * disable onnx_test_runner -x invocations for dnnl (#2568) * Allow sequence length to be symbolic (#2559) * setup java ci mac (#2570) * make layernorm fusion to support opset 11 (#2545) * Fix a warning found in the latest VS release * Add more check on SkipLayerNorm and BiasGelu fusion (#2574) * Fix file not found error during docker build. (#2569) * Add ConvTranspose1D (#2578) * Ryanunderhill/packagename test (#2582) * [Nuphar EP] fixes for some object detection models (#2581) Update notebook tutorial with multi-threaded int8 GEMM from #2517 * EmbedLayerNormalization Fusion Improvement (#2553) Embedding layer norm fusion improvements - add more checks * Update version (#2584) * Temporarily exclude vgg19 test from Python backend test 1. temporarily exclude vgg19 test which comsumes too much memory, run out of memory on Upsquared device. Single test pass for vgg19, need furture investigation (#2588) 2. Update docker file to decrease the docker image size * Update docs for Android NNAPI EP (#2586) * Fix lto bug for protobuf and ubuntu * add path to build dir before test run (#2590) * Add missig env variables for mac pipeline test (#2595) * Fixed an issue in updating realized dims (#2597) when we update realized dims for scan's output, the sliced axis also needs to be inclusive, i.e. we should check with "dim >= insert_inclusive_axis", because the offset in the symbols are based on Scan sugraph. Otherwise, we would end up with shape mismatch later. * Java API for onnxruntime (#2215) * Add support for opset 11 in reshape fusion (#2592) Support opset verion 11 in reshape fusion * Rename automl python tools folder to featurizer_ops. (#2593) * Support opset 11 subgraph of Squad model in Embed Layer Normalization (#2605) Support opset 11 Squad model that is exported from PyTorch nightly. The embed layer uses Range op which is missed in the transformer. * symbolic shape inference: fix warnings in GPT-2 model (#2608) And revise nuphar perf test on BERT squad * Dump subgraph ID and fused graph ID (#2607) * Dump subgraph ID and fused graph ID Dump subgraph ID and fused graph ID for better debugging * Remove local static fused_count added a field global_fused_count_ to NupharExecutionProvider class * EmbedLayerNormalization Fusion For Dynamic Squad Model Opset 10 (#2613) Support subgraph of SQuAD model exported from pytorch with dynamic input axes * Allow providers to be set for InferenceSession at construction (#2606) * Remove unnecessary parameter in some places in GatherElements implementation (#2612) * Remove unnecessary parameter in some places * Update * Update * Make sure fenced tensor could not reuse other tensor. (#2561) Fix random error caused by this. * Improve Embed Layer Norm Fusion for SQuAD with static input shape (#2621) * fix float16 comparison in initializer (#2629) * epsilon attribute for layernormalization fusion (#2639) * removed unnecessary batch file and fix path (#2640) * Add shape inference to ConvTransposeWithDynamicPads schema (#2632) * Improve cuda expand() opeator's performance. (#2624) * Cuda pad optimize when no padding is needed. (#2625) * Shortcut cuda Pad() when no padding is needed. * Optimize cuda scatter() on 2D compatible. (#2628) * Optimize cuda scatter() on 2D compatible. * Add some comments. * fix build error for ARM (#2648) * Improve performance of resize() in Nearest mode (#2626) Special treatment for 2D, check same size as input image. And in 2d kernel, template use_expolation. * Fix memory exception in Layer Norm Fusion (#2644) * Windows CI changes(#2650) * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Revert "Windowsai without fi (#2701)" This reverts commit 285d4c85ff5c4e265f963208170304ef3461e684. * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Deref unique pointer for session_state * send shutdown event when dll is unloaded and EvaluationStop, SessionC… (#2704) * send shutdown event when dll is unloaded and EvaluationStop, SessionCreationStart Events. * Add EvalutationStart Event * add comment * use correct type for for loop (#2755) * ARM CI (#2759) * Set ARM agent pool * Set CMake generator to VS 2019 in ARM * Use system-wide CMake instead of custom version Our custom version is too old for VS 2019 * Use DML and build shared lib in ARM CI * Restore nuget packages in ARM CI * Disable DML * Refactor ARM debug/release builds * Use system packaged Python version * Remove hardcoded Python path * Downgrade Python to 3.7 for build * Remove explicit CMake path * Fix invalid JSON in cgmanifest.json (#2760) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * Remove usage of WHOLEARCHIVE in WinML CMake and add WinMLAdapterFactory (#2726) * Remove usage of WHOLEARCHIVE in WinMLAdapter CMake and add WinMLAdapterFactory * PR feedback, no need for dll(export) since using def file * PR comments * Small comment in gen_def.py * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * Move winml_provider_factory.h to proper location (#2801) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Filter CPU case for IsFloat16Supported (#2802) * Merge fixes * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * Fix warnings that cause build to fail * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * Make DML include path global (#2882) * Make DML include path global * Add generated cppwinrt headers to winml_lib_common * Integrate changes to WindowsAI to make ADO Build (#2886) * Revert "CMake cross-generator fixes (#2790)" This reverts commit dbe7d97fa1ab155f1309bced87199527e8f35bd2. * add additional suppress warning in onnx_proto * ignore /wd4996 warning * DML execution provider fixes * Revert "Revert "CMake cross-generator fixes (#2790)"" This reverts commit 1ae7b4bcbc02edc881ad28685da98e095dfceb17. * Update func signature of custom op function overloads * common devicehelpers fixes * Add pch.h for winml_lib_common * re-add winml_lib_common_dir/inc to include path for winml_adapter * User/orilevari/dml redist shared folder (#2890) * move dml nuget package directory up one level to make it shared between build flavors * Merge conflict fix * Revert "Merge conflict fix" This reverts commit 142fa72cf9ce4344ad717b50b7ea2b8582aadc7c. * Revert "Merge remote-tracking branch 'origin/master' into windowsai" This reverts commit 6e2126d46e5e5f564d65da37dd4f70c93dd81165, reversing changes made to b3f5583dc9249834b947c8ea905f6a98060d5bd6. * Make winml_test_common free of test macros (#2902) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Don't generate files outside the build root (#2914) * Don't generate files outside the build root * Add onnxruntime_EXTERNAL_DEPENDENCIES to WinML * Add DML depedency on RESTORE_PACKAGES * User/orilevari/fix yaml merge bugs (#2918) * Add winml test source parameter into cmake function (#2919) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Add arguments to unittest cmake functions * remove comment * Revert "Revert "Merge remote-tracking branch 'origin/master' into windowsai"" This reverts commit ade5abe72a4234fdbc3623093c61c02c6b0bdc26. * Fix breaks from merge with ORT master * Brianma/linux (#2917) * don't include windows.h in cross-plat header * add default case for switch statement * signed/unsigned mismatch fix Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> * User/sheilk/winml adapter c api (#2891) * Create winml adapter c api * fix build * make it build * move adapter into onnxruntime core/session * entry point not exported * minor changes * make model metadata work * make tests pass * implement all the model reflection apis on the adapter c abi * update the new ort interface to create a lotus ennvironment with a logging sink * start adding ort env * move all winml code into adapter folder/lib to isolate it * ensure a single logging manager at a time * start refactoring session * refactor session creation interface * add cpu and dml session option methods to adapter * finish session init * stub out interfaces in ort lib to perform similar mechanics of iinference session * enable profiling, and enable schema override * update session register graph transformers * turn back on custom registry for custom ops * Add sync api * add last c api stubs * should build... but all feature values are broken since this is in flight to moving all implementation details into ivalue * remove ep adapter header * Implement DML execution provider functions from adapter (#2846) * Implement DML execution provider functions from adapter * Use functions in OnnxruntimeEngine.cpp * make map/sequence type_infos freeable, and start implementing ivalue * make it build again * implement value methods * implement remaining methods * remove com adapter abi * check dml session * cache the allocator on ivalue * check if resource is cpu/gpu when access its mutable data * update tensor * mismatched parentheses * fix tensor base and binding obj * it evaluates tensors! sometimes... * minor fixes * enable gpu evals * wrapper all existing winml adapter apis with API_IMPL to try catch (#2854) * update winml... tensor strings are broken, need to template tensorbase to do different things for strings * make tensor strings work with 2 copies in/2 copies out * Fix tensor string and allocator bug * make maps work again... needs some fixes still * Make it build! * enable map inputs * map outputs * unbound outputs for sequences and maps * User/xianz/merge windowsai (#2883) * Packaging pipeline changes for VS 2019 (#2711) * Tiny fix to codegen * Simplify cache implementation and avoid static variables that may carry over between models * Extend DML kernels (#2641) * Additional DML operators * Check unsupported attributes and inputs * Address PR comments * Add kernel capability function used for partitioning, and re-enable stride-based int64 support based on value range * Fix test failures * Build fix * PR comments * Update Nuphar tutorial notebook (#2721) 1. Reflect int8 GEMV improvements for multi-threading from #2696 2. Add notes on multi-threading control using OpenMP 3. Add samples of running multi-isa AOT, and show int8 GEMM differences between AVX and AVX2 4. Add rnn_benchmark example to resolve #1993 * Add schema for new Qops (#2611) * Add schema for new Qops * adding shape inference + qlinearaveragepool * plus review comments * plus review comments * updates per review comments * plus review comments * [server] Add supposed for model_name and model_version as cli parameter (#2708) * remove 64bit warning message from python validation. (#2727) * MLAS: ARM64 build fix (#2734) fix bad usage of vreinterpret to cast vector element types * Fix broken python docs links (#2740) * Fix build on Mac OS (#2731) mac os ld doesn't support --while-archive, correct option is -all_load * fix ngraph wheel (#2737) * fix ngraph wheel 1.1.0 onnxruntime_ngraph wheel doesn't work * remove libdnnl.so in nGraph Libs * make it easy to compare * Split onnxruntime server to a separated folder (#2744) * Fix build for Python 3.8 (#2747) * Fix build for Python 3.8 * Update protobuf to 3.11.2 (#1928) Update protobuf to 3.11.2 (#1928) * Change default optimization level to All (from Basic) (#2745) * change default optimization level to All (from Basic) * fix test * fix c# test * Update numpy to 1.18 (#2758) * Update numpy to 1.18 * Pipeline changes for python 3.8 (#2753) 1. Pipeline changes for python 3.8 2. Fix a regression in setup.py which was just introduced in the previous commit. Please notice, we still haven't made python 3.8 + Windows + CUDA work. * Add basic stacktrace output for posix debug builds. (#2749) * [NupharEP] fix a race condition when multiple sessions running different models concurrently (#2772) * Revert "Change default optimization level to All (from Basic) (#2745)" This reverts commit 56bb503c2f26474b6613bcb2a198691a11dcef38. * Fix typo in error message (#2736) * Rename MKL-DNN to DNNL to fix broken link (#2730) * Fix nightly build version number issue * Pass BUILD_BUILDNUMBER to linux docker * Disable featurizers in python packages * Import more featurizers (#2781) Make kernels non-template. Add input constraint for learnt data. Add min_max_scalar_transformer, robust_scalar_transformer, inputation_marker_transfomer, label_encoder_transformer, missing_dummies_transformer along with tests. Advance Featurizers library commit. * Implement a more stable softmax (#2715) * Implement a more stable SoftMax e^x is represented as infinity if x is large enough, like 100.f. Infinity divided by Infinity is a NAN. Thus, softmax gets a NAN if one or more item are large enough. A math transform as below is leveraged to get a stable softmax: e^xi/(e^x1 + ...e^xn) = e^(xi - max) / (e^(x1 - max) + ... + e^(xn - max)) And for convenience, force max to 0.f if all xi are negative * Contributing: Fix a typo (#2784) * ACL EP GEMM improvements (#2780) When it is posible we use a fully connected layer instead of the gemm implementation. This will let the library use the best implementation based on the input data. * ACL EP convolution improvements (#2774) Added the optimized implementation for depthwise convolution for both ACL v19.02 and ACL 19.05. Also the pointwise convolution seems to be more optimal in the CPU implementation so we opted for that instead. * Add script for release Nuget validation (#2719) * Initial commit * Nits * Disable a test temporarily * Change working directory * Test * Add download python step * Test update * More changes * Fix space issue * Fix * Verify nuget signing * Fix * Spaces * PR feedback * Nit * Fix * Fix * Remove temporary changes * add uint8 support to where op (#2792) * Improve bert optimization script: (#2712) (1) Move input int64=>int32 conversion to embed layer fusion. (2) Output epsilon attribute for LayerNormalization fusion. * add session creation time cost. (#2798) * ML.NET team needs featurizers within a package (#2789) Add auto ml featurizers to Windows, MacOS as well as to GPU packaging-pipelines. * Initialize max of softmax with lowest of float (#2786) * MLAS: update SGEMM threading parameters (#2808) * add interface to copy batch tensors. (#2807) * add interface to copy batch tensors. * onnxruntime * speed up Windows TRT CI (#2811) * don't run cuda tests if building with tensorrt * remove unnecessary build options for win trt ci * refactor win gpu tensorrt ci yml * --numpy_version=1.17 * update * update * azcopy and cuda path * Update test data (#2356) * Add timeseries imputer transformer featurizer kernel (#2813) Make kernels non-template. Add input constraint for learnt data. Fixup tests. Add two more featurizers along with tests. Tests fail. min_max_scalar_transformer robust_scalar_transformer Fix tests serialized stream by prepending version bytes. Add inputation_marker_transfomer and the test. Fix up float/double type designations. Added label_encoder_transformer along with a test. string_throw case is broken at the momement. Fix labelencodertransfomer_test.cc string_throw case Rename maxabsscalertransformer_test.cc Add MissingDummiesTransformer along with the test. Update manifest. Add TimeSeriesImputerTransformer definition, implementation and tests * Fix memory leak in TRT (#2815) * fix memory leak issue * revert EP_FAIL on enueueV2 * Add manifest missing comma * Run static code analyzer on most of our code (#2817) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * update quantization doc (#2783) * update documentation for quantization script * plus some spell corrections * Filter CPU case for IsFloat16Supported (#2802) * update default optimization level + fix gemm_activation fusion (#2791) * update defualt optimization level + fix gemm_activation fusion * fix typo * add unit test and incorporate review comments * fix test comment * Fix dnnl wheel package name (#2823) * Append '-dnnl' to whl package name when --use_dnnl * Update build.py * Update Ubuntu & TensorRT version in README (#2820) Dockerfile.tensorrt is using nvcr.io/nvidia/tensorrt:19.09-py3 as base Image, update Ubuntu and TensorRT version according to https://docs.nvidia.com/deeplearning/sdk/tensorrt-container-release-notes/rel_19-09.html#rel_19-09 * Merge fixes * Add OneHotEncoder and HashOneHotEncoder kernels. (#2830) Add defs and imlementation for OneHotEncoders, adjuist date_time_transformer kernel and test. Add OneHotEncoder kernel test. Add HashOneHotVectorizerTransformer unit test. This does not link due to multiple definitions of functions that are included into header from a CPP file. * Upgrade gtest to the latest version (#2827) WinML would like to update the googletest submodule. They want some newer features (namely GTEST_SKIP to skip tests programmatically and be able to skip entire fixtures easily) and would need to update the submodule version. However, because the new version of code hit a bug in gcc, even though the bug is already fixed in the latest gcc but we're using gcc 4.8.x and it won't get patched for the bug, so we have to do a compromise, change our code a little bit to make it work. The gcc bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=51213 * Add support for int64_t for topk CPU. Fixes github issue #2806. (#2833) * Ignore allocator type in ExecutionProviders allocator map. Make default initialization of OrtMemoryInfo more clearly invalid. (#2768) * Remove allocator type from the key comparison in ExecutionProviders. Remove usage of DummyArena as it's no longer necessary. * Fix x86 tests where arena allocator is disabled. Make initialization of OrtMemoryInfo clearer by adding Invalid enum value. * Make OrtValueNameIdxMap::MaxIdx more intuitive. * Convert ExternalProject Featurizers into git submodule (#2834) Add git submodule for Featurizer library. Update cmake to build for git submodule. * add domain check for nodes + update documentation (#2831) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Fix Linux CUDA nuget packaging pipeline break * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * update optimization doc for BERT related fusions (#2819) * Add bert related transformers to doc * Add execution provider and comment for bert optimizations * Add comment about accuracy impact of approximation * Fix warnings that cause build to fail * MLAS: enable threading for quantized GEMMs (#2844) * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * remove old winmladapter cpp Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * move sequence implementation into ort lib... still commented out... need to turn back on... * begin sequence implementation * make maps and sequences work * fix broken tests * remove dead code * misc cleanup * CR feedback * User/xianz/winml adapter c api (#2869) * wrapper all existing winml adapter apis with API_IMPL to try catch * Return HR or Throw for WinML adapter APIs if failed * undo macro wrapper for two places * Wrap error macros around ort apis, too. * address CR feedback #2 * add more api throw/return macros * Revert changes no longer needed * revert changes to cxx api * format winml lib.ort and winml adapter * remove static pheonix singleton Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * missing use_dml check in winml_adapter_session (#2930) * --use_dnnl flag was mangled in merge (#2931) * use dml macro not wrapping custom registry code (#2934) * Disable LNK4199 winml_dll to enable cuda builds (#2936) * Disable LNK4199 in winml_dll * linkler->linker * LearningModelSessionAPITestGpu.CreateSessionWithCastToFloat16InModel should return DXGI_ERROR_UNSUPPORTED when FP16 not supported (#2937) * Disable LNK4199 in winml_dll * linkler->linker * Need to return DXGI_ERROR_UNSUPPORTED when Model does not support fp16 * Publish build symbols (#2939) * Publish build symbols * Don't upload PDBs for .exe files * Make x86 build (#2943) * fix last remaining size_t/int64_t warnings->errors (#2948) * TensorString, Sequences and Maps use the first allocator, but should use the cpu default allocator. (#2952) * fix tensor string allcoator * clean up default allocator usage for strings in winml lib/api.ort Co-authored-by: Ryan Lai <ryalai96@gmail.com> * Handle tensor shape of zero (#2954) Co-authored-by: Ryan Lai <ryalai96@gmail.com> * CR feedback (#2970) * CR feedback * fix weird formatting on privacy readme * Add 'All rights reserved.' everywhere * readd all rights reserved to winml_provider_factory.h * remove extra space in comment * remove extra whitespace * fixes post master merge * remove winml from nuget gpu pipeline * set IR VERSION on generated_model in rnn_benchmark (#2972) * Fix slice conformance failures (#2908) Co-authored-by: Adrian Tsai <adtsai@microsoft.com> Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Paul McDaniel <paul_mcdaniel@hotmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Dwayne Robinson <fdwr@hotmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com>
2020-02-05 01:12:19 +00:00
[threading(both)]
[marshaling_behavior(agile)]
[dualapipartition(1)]
runtimeclass TensorInt64Bit : ITensor, [contract(MachineLearningContract, 2)]Windows.Foundation.IMemoryBuffer
{
static TensorInt64Bit Create();
static TensorInt64Bit Create(Windows.Foundation.Collections.IIterable<Int64> shape);
static TensorInt64Bit CreateFromArray(Windows.Foundation.Collections.IIterable<Int64> shape, Int64[] data);
static TensorInt64Bit CreateFromIterable(Windows.Foundation.Collections.IIterable<Int64> shape, Windows.Foundation.Collections.IIterable<Int64> data);
Windows.Foundation.Collections.IVectorView <Int64> GetAsVectorView();
[contract(MachineLearningContract, 2)]
{
//! Creates a tensor object, allocates a buffer of size 'shape', and copies all of 'data' into it.
static TensorInt64Bit CreateFromShapeArrayAndDataArray(Int64[] shape, Int64[] data);
//! Creates a tensor object, creates a tensor of size 'shape', and uses the data in 'buffer'.
static TensorInt64Bit CreateFromBuffer(Int64[] shape, Windows.Storage.Streams.IBuffer buffer);
}
}
[contract(MachineLearningContract, 1)]
INBOX_ONLY([static_name("Windows.AI.MachineLearning.ITensorBooleanStatics", 2796862c-2357-49a7-b476-d0aa3dfe6866)])
INBOX_ONLY([interface_name("Windows.AI.MachineLearning.ITensorBoolean", 50f311ed-29e9-4a5c-a44d-8fc512584eed)])
Merge windowsai (winml layering) into master (#2956) * Initial Commit * Merged PR 3985217: add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc (#2346) add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc and violating our OS layering requirements. We linked against onecoreuap_apiset.lib in VB so we will continue doing this, but I am still unsure why not to link against onecore instead since that is where we ship. However, since Sheil is the owner of this code we will wait to discuss with him before changing anything. * Initial changes for layering * more snipping to get core into ort * update build instructions to include --build_shared_lib (#2358) * update build instructions to include --build_shared_lib * fix line breaks * Task 23998197: add winml_lib_core into onnnxruntime.dll (#2368) * Task 23998197: add winml_lib_core into onnnxruntime.dll * PR feedback build break on perf_test * return proper error when the model path isn't found (#2391) * LearningModelSession is cleaned up to use the adapter, and parts of b… (#2382) this is a big PR. we are going to move it up to layer_dev , which is still a L3 so we are still safe to do work there agile. we are going to move this into the L3 so that ryan can start doing intergration testing. we will pause for a full code review and integration test result prior to going into the L2. >>>> raw comments from previous commits >>> * LearningModelSession is cleaned up to use the adapter, and parts of binding are. * moved everything in the winmladapter made it all nano-com using, WRL to construct objects in the ORT side. base interfaces for everythign for winml to call cleaned up a bunch of winml to use the base interfaces. * more pieces * GetData across the abi. * renamed some namepsace cleaned up OrtValue cleaned up Tensor cleaned up custom ops. everything *but* learnignmodel should be clean * make sure it's building. winml.dll is still a monolith. * model moved over. everything builds clean. step ! * weak ref comment * Layer dev paulm (#2408) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * Layer dev paulm (#2414) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * Layer dev paulm (#2423) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * Layer dev paulm (#2424) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * Layer dev paulm (#2425) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Layer dev paulm (#2426) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * fixed map and sequence when passing stl types across the ABI . found a leak in nvidia driver, but skipped it. all winmlapitests pass now * Moved SessionOptions over to the abi * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * Register WinML TraceLogging provider on Onnxruntime.dll (#2455) * Register WinML TraceLogging provider on Onnxruntime.dll * Add ifdef to make sure trace logging provider has telemetry option when LAYERING_DONE * No need for ifdef for TraceLoggingOptionMicrosoftTelemetry * PR feedback * Move etw registration into lotus environment constructor and deresgister in lotus environment destructor * Brianma/cpuwinml (#2466) * allow building winml cpu without dml. * Brianma/breaks (#2469) * fix some more breaks * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * move dml checks out of winml and into the adapter * better error handling * Brianma/fi (#2470) * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * fix bad merge * Checking in a staging checkpoint point so that Ryan can work with me in parrallel * build break. * Brianma/testfails (#2473) * add missing ir version to dictvectorizer-string.onnx * add missing ir version to relu.onnx * add missing ir version to zipmap*onnx * add IR version to manually generated models * remove an unnecessary ifdef dml * Brianma/windowsai fi (#2475) * update dockerfiles/README (#2336) * Make elementwise op run 4 items per thread (#2335) Description: Describe your changes. Make elementwise op run 4 items per thread unroll for loop to leverage ILP remove unnessary N==0 check inside elementwise GPU kernel Motivation and Context Why is this change required? What problem does it solve? It can improve the performance of GPU elementwise ops. ~2% performance gain on popular NLP bert model. If it fixes an open issue, please link to the issue here. * Add CUDA GatherElements kernel (#2310) * Updates * Update test * Update * Updates * nits * PR feedback * Update * Update * PR feedback * PR comments * Update * Fix build * Fix build * Nits * Fix * Layer Normalization Fusion (#2319) basic layer normalization transform * Add FastGelu Cuda Op for Gelu and Add bias fusion (#2293) * Add FastGelu cuda op * Add AddBiasGelu for experiment * Revert "Add AddBiasGelu for experiment" This reverts commit 5c1ee019858c657e6bb75887265cb85675626e5b. * Add bias * Add unit tests * update comment * update script * fix build error * update coding style * update for CR feedback Enable half2 optimization only when cuda arch >= 7.0 * move _Tanh to common.cuh * implement CPU contrib OP Attention (#2333) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanUnusedInitializers. (#2320) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanupUnusedInitializers. This means initializers that have been replaced during graph optimizations are not left in the GraphProto when we save an optimized model. * Handle edge case where a model has an unused initializer with matching graph input by also removing the graph input. * Use non-const iterators in std::find_if calls to make centos build happy. * Nuget pipeline changes (#2305) 1. refactor the pipeline, remove some duplicated code 2. Move Windows_py_GPU_Wheels job to Win-GPU-CUDA10. We'll deprecated the "Win-GPU" pool 3. Delete cpu-nocontribops-esrp-pipeline.yml and cpu-nocontribops-pipeline.yml 4. In Linux nuget jobs, run "make install" before creating the package. So that extra RPAH info will be removed * Cuda Reverse Sequence Op, maping types of same size using same template function. (#2281) * Set ElementType to String type of node metadata, instead of byte[] (#2348) * Set ElementType to String type of node metadata, instead of byte[] * Fix spacing * Introduce PrimitiveType into a Type System along with an integer constant (#2307) Improve perf by avoiding GetType<T>() calls. Introduce MLTypeCallDispatcher to switch on Input Type. Add Tensor IsType<T>() fast method. * Fix/test dim value of 0 handling in a couple of places (#2337) * Update the CUDA Where implementation broadcasting logic to handle a dim with value of 0. Add unit test Also add unit test for unary op with dim value of 0 * Exclude ngraph from Where test with 0 dim. * Openvino EP R3.1 onnxrt server (#2357) * onnxrt server with OVEP * onnxrt server with OVEP * Update Dockerfile.server.openvino * onnxrt server OVEP fix reviews * onnxrt server OVEP fix reviews * Implement cuda nonzero op. (#2056) Implement cuda nonzero op. * Direct use python numpy array's memory if already contiguous. (#2355) * Direct use python numpy array's memory if already contiguous. This could greatly improve performance for session with large input, like big image 1920x1080 fastrcnn, 30~40% speed up could be achieved. * Add test case enforce contiguous/non-contiguos numpy array as inputs. * Add helper to create output to minimize binary size. (#2365) Add ConstEigenTensorMap typedef so we don't unnecessarily const_cast the const input Tensor. * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS (#2369) * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS * update * Add Tracelogging for profiling (#1639) Enabled only if onnxruntime_ENABLE_INSTRUMENT is ON * test bidaf with nuphar for avx target (#2370) increase nuphar test coverage a bit * Fix a bug in TLS refcount that may destabilized CUDA CI (#2374) * update output size calculation for resize (#2366) * change how output size is calculated for resize op * add tests for ver 10 resize * Extend OneHot CPU kernel to support more types (#2311) * Extend OneHot CPU kernel to support input int64_t, depth int32_t, output float * Skip BERT before the test data fix is picked up * Fix bug with Slice. Need to pass in flattened input dimensions so the initial offset into the input is calculated correctly. (#2372) * Add opset 11 version of Split to CUDA ops (#2376) Organize the CUDA ops definitions so all the opset 10 and 11 parts are together (same setup used for CPU ops) * Layer Norm Fusion Fix (#2379) * layer norm fusion fix * Add input shape check in code and unit tests * Fuse Add + Gelu (#2360) Implement the transformer to fuse add + gelu Implement the accurate kernel * Skip layer norm transform (#2350) * skip layer normalization transformer * Another try to stabilize CUDA CI (#2383) The root cause seems to be failure in CUDA dealloc when tear down. cudaFree return code was ignored before, so should the debug check. * fix BUILD.md typo (#2375) build.py: error: argument --config: invalid choice: 'RelWithDebugInfo' (choose from 'Debug', 'MinSizeRel', 'Release', 'RelWithDebInfo') * Fixed compilation with ngraph (#2388) * Fix reuse logic in allocation planner. (#2393) * Fix reuse logic in allocation planner. * PR comments * Add helpful comments * Don't allow reuse across string tensors. * [NupharEP] Multiple optimizations (#2380) Fuse transpose into MatMul Implement Pow and constant scalar simplification Vectorize ReduceMean Improve symbolic shape inference Minor updates for better debugging in fused function name * Avoid using the default logger in the graph lib and optimizers (#2361) 1. Use the session logger if it is available. 2. Don't disable warning 4100 globally. We should fix the warnings instead of disabling it. * Change CUDA implementation of Transpose to support all fixed size tensor types (#2387) * Change CUDA implementation of Transpose to not use a typed kernel so we can support more types with minimum binary size. Add support for 8, 16, 32 and 64 bit types. Add unit tests. Add method so the implementation can be called directly (will be used by CUDA Scan very soon). * Disable TensorRT for MLFloat16 and int8 unit tests. * Address PR comment and add support for calling cublas implementation if type is mlfloat16. * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. (#2398) * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. * [NupharEP] force some low/zero cost ops to be inlined (#2409) * fix cross compile bug (#2415) * Minor optimization: if a node has already been placed, there's no need to find a kernel for it. (#2417) * Add Reshape Fusion (#2395) * Add reshape fusion * Add some comments * update comments * update comment format * update according to feedback * update for recent logger change * fix build error * (1) Support both input and output edges in find path in graphutils (2) Add a test case of only one constant initializer of Concat input. (3) Refactor ReshapeFusion class to allow add more subgraph fusion in the future. * fix error * (1) loose constraint on initializer: non constant is allowed for reshape fusion. (2) Change versions type to vector. (3) Add logging. (4) Return false when multiple output edges matched in FindPath. Add comments. * only allow one direction (input or output) in FindPath * [NupharEP] Update notebook and docker image (#2416) Add BERT squad in Nuphar tutorial Enhance speed comparsion readability * Fix the issue in matmul_add_fusion (#2407) Fix the issue in matmul_add_fusion If Muatmul + Add has shape [K] * [K, N], reset it to [1, K] * [K, N] will make the output shape to [1, N] will also requires a reshape on the output. Fix: just remove the shape reset to not fuse it. Add a negative test case for matmul+add fusion * feat(treeregressor): Update TreeEnsembleRegressor for type support (#2389) Updates the `TreeEnsembleRegressor` to allow for `double`, `float`, `int64`, and `int32` inputs to match the upstream specification. Signed-off-by: Nick Groszewski <nicholas.groszewski@capitalone.com> * onnxrt server documentation update (#2396) * Added support for Pad-2 operator in OpenVINO-EP (#2405) * Add CUDA If operator. (#2377) * Add CUDA If operator. Uses CPU operator for implementation. By adding a CUDA version the inputs/outputs (with the exception of the 'cond' input) stay on GPU, and no other logic is required to avoid a copy to CPU across the control flow node. * Improved documentation for onnxruntime::utils::SwapByteOrderCopy(), added precondition check. * Fix the type constraints on CUDA If operator to exclude strings. (#2431) * add Im2col<uint8_t> (#2438) * Adjust codegen vectorization width from target (#2439) * Adjust codegen vectorization width from target * Add CUDA Scan operator. (#2403) * Add Scan CUDA op. Uses CPU implementation for logic. Added some device specific functors for handling when data needs to be manipulated on a different device. Added ability to override the materialization logic in the OrtValue slicer so DML can plugin their handling. * Fix Windows GPU C API packaging pipeline failure (#2440) Fix Windows GPU C API packaging pipeline failure (#2440) * Correctly handle implicit inputs for fused nodes (#2390) * Correctly handle implicit inputs for fused nodes Previously, nuphar's partitioning function didn't include node's implicit inputs into the inputs list of MetaDef, and hence a crash was triggered in the onnx graph checker. This commit fixed the issue. Furthermore, it also fixed a related issue where we didn't add implicit inputs into graph_inputs_excluding_initializers_ in Graph::SetGraphInputsOutputs. the issue was that graph_inputs_including_initializers_ populated by SetInputs (e.g. called by FunctionImpl::FunctionImpl) may contain implicit inputs which were not of any node's initializers in the graph. Because they were not part of any initializers, these implicit inputs couldn't be visited by going through all nodes' inputs. Consequently, they would *not* be added into graph_inputs_excluding_initializers_. We fixed the issue by first copying the populated graph_inputs_including_initializers_ into graph_inputs_excluding_initalizers_, which then had both initializers and non-initializers as its initial content. Later, we erase initializers from the list. In this way, we can ensure all implicit inputs to remain in graph_inputs_excluding_initializers_. * refined comments and fixed duplicates Address CR by revisiting comments in terms of implicit inputs Also fixed an issue by skipping duplicates while copying inputs from graph_inputs_including_initializers_. * address CR explain why we need to collect nodes' implicit inputs * don't rely on pointer values for iterating std::set Previously, openvino relied on iterating a set of NodeArg pointers to construct inputs and outputs for a fused graph. It could cause non-determinism. The reason was that although iterating std::set by itself is stable, pointer values of NodeArgs may vary. Consequently, we could end up visiting the set's elements in different orders for different runs for the same test, which resulted in constructing inputs (and outputs) with different orders to the fused graph. For example, for the same test, we may have inputs [A, B] in some runs but inputs[B, A] in others. Let's use std::string as the key type to avoid such nondeterminism. This commit also added implicit inputs into meta->inputs while returning the capability from the openvino provider. * Fixed another latent issue in openvino's GetCapability function The issue was that we couldn't simply erase fused_inputs and fused_outputs while iterating the nodes. For example, an output NodeArg may have multiple uses, and it's wrong if we erase it from fused_outputs when we encounter only one of its uses as input. * Remove DeviceAllocatorRegistry class (#2451) Remove DeviceAllocatorRegistry class * CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library. * Parallel Gelu with ParallelFor (#2399) Parallel Gelu to get better performance for Gelu * Clean up build.py (#2446) * Pull the latest image before running docker build * Fuse SkipLayerNorm with Bias (#2453) Fuse SkipLayerNorm with Bias * Allow more than one invocation of CreateEnv in the same process. (#2467) * Allow more than one invocation of CreateEnv in the same process. * Fix centos build * Symbolic shape inference improvements: (#2460) * Symbolic shape inference improvements: - add a mode to guess unknown ops' output rank - add support for GatherND - add support for If - fix a bug in get_int_values when then tensor rank > 1D, by treating it as no sympy data - add symbol to literal merge when ONNX silently merges dims - fix a bug in Concat when input dim is 0 - fix a bug in ConstantOfShape that computed dim is not updated - add support for dynamic shape in ConstantOfShape - fix a bug in Loop output shape that loop iterator dim is not inserted at dim 0 - add support for dynamic padding in Pad - add support for dynamic shape in Reshape - add support for Resize with opset > 10, by treating output dims as dynamic - fix a bug in Slice when starts/ends are dynamic - restrict input model to opset 7 and above - make output model optional to avoid disk write when testing Run model tests for symbolic shape inference Reduce 2GB docker image size of nuphar * add additional test data set for nuget pipeline (#2448) * add SAS token to download internal test data for nuget pipeline * update azure endpoint * fix keyvault download step * fix variable declaration for secret group * fix indentation * fix yaml syntax for variables * fix setting secrets for script * fix env synctax * Fix macos pipeline * attempt to add secrets to windows download data * fix mac and win data download * fix windows data download * update test data set url and location * Revert "Brianma/windowsai fi (#2475)" This reverts commit 5780b864a15513fda4eadbfc2b5345fefe70b5ec. * Add scenario tests (#2457) * Add scenario tests * Remove TODO from model license * Add winml_api test dependency * fix model load test. fi from master changed the constructor (#2483) * make api tests all pass (#2486) * fix bad merge * fix bad model merge * Layer dev paulm (#2492) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * Rename ambiguous header (#2489) * fix one more missing IR version model (#2500) * add missing IR version to 4 more models used by scenario tests (#2501) * Add CLI parameters to test runner, build WinML in ARM and x86 CI (#2479) * Support test parameters through CLI arguments * Add WinML do Windows x86/ARM CI builds * Code style fixes * Update googletest Remove GPUTEST macros everywhere now that GTEST_SKIP is supported * Refactor main.cpp * Build scenario tests without DML * Link scenario tests to DML when it's enabled (#2502) * Layer dev release pipeline (#2488) Adds winml binaries to existing cpu nuget package, and creates new gpu dml nuget package with winml binaries and DML EP. * Layer dev paulm (#2506) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * Remove usage of IOBinding in WinML and use C_API Run method (#2504) * remove usage of iobinding * Change data structure to use vector of Ort::Values * Polish bind input / output * Use C APIrun method * Update providers on evaluate getresults * Remove run and IObinding interface from WinMLAdapter * Remove use of IObinding * bind unbound outputs code moved to learningmodelbinding * clean up unneeded istensor adapter function * Fix comment * Check if session is closed before binding and clearing * PR feedback * Layer dev paulm (#2507) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * Make tests dependend on winml_dll (#2509) * add dml binaries to DirectML package and be more explicit about condition variables (#2520) * re-enable warnings for winml builds and fix the warnings that were hiding (#2526) * turn devmode back on for winml builds * fix some warnings. include protobuf in a way that disables some warnings * undo protobufhelpers changes and just ignore 4100 errors in pb code * attempt to isolate protobufhelpers errors * add template specialization for getting tensor proto data * Layer dev paulm (#2533) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * Spawn child process to run DeviceLostRecovery scenario test (#2530) * Spawn child process to run DeviceLostRecovery scenario test * Layer dev paulm (#2536) ori said yes * add missing namespace to winml_trace_logging_provider in lotusenvironment.h (#2542) * Handle exception thrown from all apis in WinMLAdapter (#2539) * various changes to unblock windowsai ADO build * Fix custom ops scenario tests (#2562) * Do not shutdown protobuf after ort environment gets destroyed. Lazy load lotus environment first time it is needed * comment typo * pr comment about calling phoenix singleton * Make lotus_environment static in winmladapter * Layer dev paulm (#2567) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * making windowsAI azure dev ops work. * code review comments. * revert changes * Cmake and preprocessor fixes that where uncovered by building on agents without DML available via SDK * Layer dev dml delayload (#2580) * Brianma/cpu (#2583) * don't include dml stuff in cpu builds * tests that link the image lib also need the telemetry lib now * Throw Winml_err_invalid_binding if binding gpu resource on cpu device (#2589) * Throw Winml_err_invalid_binding if binding gpu resource on cpu device * PR comments. No need to query executionprovider for is gpu device * User/xianz/ortthrow (#2596) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * User/xianz/ortthrow (#2599) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * add status error message * Remove uwp onsuspending winrt call because logruntimeperf is getting removed (#2630) * User/xianz/dedup telemetry (#2631) * investigate duplication of telemetry in winml and ort * remove winml telemetry events * telemetry executionProviderEvent * remove unneccessary file and refactor code little bit * Revert back TelemetryEvent, which send up ETW event. * merge changes from layer_dev to windowsai (#2638) * Remove underscore from googletest names (#2616) * Fix leaking memory allocator Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24278761 and https://microsoft.visualstudio.com/OS/_workitems/edit/24330198 * Explicitly initialize Ort::Value with nullptr * Cache WinML adapter * bad merge * define private version of dxcore enum that is added in 19H1 SDK. (#2654) * add comment for explaning private definition of dxcore d3d feature level ennum value. (#2672) * do not package directml.pdb for redist packages. (#2676) * Fix leaking operator registry (#2645) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24354916 * User/orilevari/windowsai master merge (#2674) merge resolutions included pulling in telemetry logic that was merged to master and not windowsai and dereferencing InferenceSession::sessionstate now that it is a unique pointer * Delete Ort Allocator in LearningModelBinding (#2653) * Delete OrtAllocator in LearningModelBinding * PR comments to make Ort::Allocator a smart pointer * Small comment change * PR feedback to clean up code * PR feedback on move semantics * Clean up std::move * Fix memory leaks (#2679) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24356109, https://microsoft.visualstudio.com/OS/_workitems/edit/24388361 and https://microsoft.visualstudio.com/OS/_workitems/edit/24388596 * various changes to properly organize and skip GPU tests. For now for No DML builds we will not run GPU tests at all. In the future we should adapt the tests to expect the appropiate errors. (#2695) * Windowsai without fi (#2701) * Disable Attention fusion tests when DISABLE_CONTRIB_OPS is defined (#2529) * Setup java ci (#2528) * Add provision in ORT for session options to be parsed when available via model file (#2449) * Initial commit * Fix gitmodules * Nits * Nits * Updates * Update * More changes * Updates * Update * Some updates * More changes * Update * Update * Merge * Update * Updates * More changes * Update * Fix nits * Updates * Fix warning * Fix build * Add comment * PR feedback * PR feedback * Updates * Updates * Update * More changes * Fix build break * Comment test for now * Updates * Updates * PR feedback * Updates * Nits * Add tests * Fix build * Fix build * Fix build * Fix build break * Fix build * Nits * PR feedback * More change * Expose GetSessionOptions in pybind logic and add unit test for python * Fix build * PR feedback * PR feedback * Revert "Disable thread pool creation when enabled OpenMP (#2485)" (#2535) This reverts commit 7c7d5a149c9ed52eec67304bae5c4b132166a8a1. * Add dynamic shape support in TensorRT execution provider (#2450) * remove onnx-tensorrt submodule * add new onnx-tensorrt submodule (experiment) for trt6 * update engine build for trt6 * update compile and compute for tensorrt6.0 * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * switch to onnx-tensorrt master for TensorRT6' * Update tensorrt_execution_provider.cc * Handle dynamic batch size and add memcpy in TensorRT EP * update test cases * Update tensorrt_execution_provider.cc * update onnx-tensorrt submodule * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.ubuntu_tensorrt * Update run_dockerbuild.sh * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update concat_op_test.cc * Update tensorrt_execution_provider.cc * Upgrade TensorRT to version 6.0.1.5 * Update onnxruntime_providers.cmake * Update CMakeLists.txt * Update reduction_ops_test.cc * Update install_ubuntu.sh * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.tensorrt * Update BUILD.md * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update onnxruntime_providers.cmake * Update install_ubuntu.sh * Update install_ubuntu.sh * Update gemm_test.cc * Update gather_op_test.cc * Update CMakeLists.txt * Removed submodule * update onnx-tensorrt submodule * update header file * Removed submodule * add submodule onnx-tensorrt kevin's branch shape-test' * add debugging code * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * merge master * Removed submodule * update onnx-tensorrt submodule * add more changes for dynamic shapes * Update tensorrt_execution_provider.cc * update for dynamic shape * update dynamic shape processing * fix logger issue * remove submodule onnx-tensorrt * add submodule onnx-tensorrt * add env variable min_subgraph_size * remove redundency * update document * use onnxruntime::make_unique * fix multi-run issue * remove some tests to save CI build time * Add dynamic shape test * Update TensorRT-ExecutionProvider.md * Add example of running Faster R-CNN model on TensorRT EP * Add more details on env variables * update environment variables * Update tensorrt_basic_test.cc * Update model tests * Update tensor_op_test.cc * remove --use_full_protobuf * Update build.py * User/xianz/telemetry (#2458) * enabme telemetry * enable telemetry * set enable telemetry as default * for debugging * remove log and set disable telemetry as default back * delete private file while testing * resolve comment: mainly add license header, rename macro and update docs * rewording in privacy.md * Fix integer overflow in cuda NonMaxSuppression implementation (#2540) * add test case that should pass but fail * fix nms * extract int_max_output_boxes_per_class * Introduce container type runtime checks and other improvements (#2522) Rework TensorSeq in a manner consistent with Tensor and SparseTensor in terms of type system setup. Reduce templating. Introduce helpers to ensure the same data type. Make OrtValue __dtor not virtual. Introduce ContainerChecker * Fix C API tests for centos and mac (#2544) * change c++14 to c++11 * add ld lib path for centos * enable csharp tests on macos * fix C API test on MacOS + fix manylinux dotnet install * fix manylinux dotnet install * fix lib link * Add back executable bit to build.py * Fix a bug handling negative begin pad values in Pad op (#2550) * Fix bug in Pad op * Update * DNNL CMAKE update (#2548) * Fix android build (#2558) * Update win-x86-ci.yml (#2557) Fix build pipeline break * Re-enable Windows C# tests (#2564) * disable onnx_test_runner -x invocations for dnnl (#2568) * Allow sequence length to be symbolic (#2559) * setup java ci mac (#2570) * make layernorm fusion to support opset 11 (#2545) * Fix a warning found in the latest VS release * Add more check on SkipLayerNorm and BiasGelu fusion (#2574) * Fix file not found error during docker build. (#2569) * Add ConvTranspose1D (#2578) * Ryanunderhill/packagename test (#2582) * [Nuphar EP] fixes for some object detection models (#2581) Update notebook tutorial with multi-threaded int8 GEMM from #2517 * EmbedLayerNormalization Fusion Improvement (#2553) Embedding layer norm fusion improvements - add more checks * Update version (#2584) * Temporarily exclude vgg19 test from Python backend test 1. temporarily exclude vgg19 test which comsumes too much memory, run out of memory on Upsquared device. Single test pass for vgg19, need furture investigation (#2588) 2. Update docker file to decrease the docker image size * Update docs for Android NNAPI EP (#2586) * Fix lto bug for protobuf and ubuntu * add path to build dir before test run (#2590) * Add missig env variables for mac pipeline test (#2595) * Fixed an issue in updating realized dims (#2597) when we update realized dims for scan's output, the sliced axis also needs to be inclusive, i.e. we should check with "dim >= insert_inclusive_axis", because the offset in the symbols are based on Scan sugraph. Otherwise, we would end up with shape mismatch later. * Java API for onnxruntime (#2215) * Add support for opset 11 in reshape fusion (#2592) Support opset verion 11 in reshape fusion * Rename automl python tools folder to featurizer_ops. (#2593) * Support opset 11 subgraph of Squad model in Embed Layer Normalization (#2605) Support opset 11 Squad model that is exported from PyTorch nightly. The embed layer uses Range op which is missed in the transformer. * symbolic shape inference: fix warnings in GPT-2 model (#2608) And revise nuphar perf test on BERT squad * Dump subgraph ID and fused graph ID (#2607) * Dump subgraph ID and fused graph ID Dump subgraph ID and fused graph ID for better debugging * Remove local static fused_count added a field global_fused_count_ to NupharExecutionProvider class * EmbedLayerNormalization Fusion For Dynamic Squad Model Opset 10 (#2613) Support subgraph of SQuAD model exported from pytorch with dynamic input axes * Allow providers to be set for InferenceSession at construction (#2606) * Remove unnecessary parameter in some places in GatherElements implementation (#2612) * Remove unnecessary parameter in some places * Update * Update * Make sure fenced tensor could not reuse other tensor. (#2561) Fix random error caused by this. * Improve Embed Layer Norm Fusion for SQuAD with static input shape (#2621) * fix float16 comparison in initializer (#2629) * epsilon attribute for layernormalization fusion (#2639) * removed unnecessary batch file and fix path (#2640) * Add shape inference to ConvTransposeWithDynamicPads schema (#2632) * Improve cuda expand() opeator's performance. (#2624) * Cuda pad optimize when no padding is needed. (#2625) * Shortcut cuda Pad() when no padding is needed. * Optimize cuda scatter() on 2D compatible. (#2628) * Optimize cuda scatter() on 2D compatible. * Add some comments. * fix build error for ARM (#2648) * Improve performance of resize() in Nearest mode (#2626) Special treatment for 2D, check same size as input image. And in 2d kernel, template use_expolation. * Fix memory exception in Layer Norm Fusion (#2644) * Windows CI changes(#2650) * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Revert "Windowsai without fi (#2701)" This reverts commit 285d4c85ff5c4e265f963208170304ef3461e684. * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Deref unique pointer for session_state * send shutdown event when dll is unloaded and EvaluationStop, SessionC… (#2704) * send shutdown event when dll is unloaded and EvaluationStop, SessionCreationStart Events. * Add EvalutationStart Event * add comment * use correct type for for loop (#2755) * ARM CI (#2759) * Set ARM agent pool * Set CMake generator to VS 2019 in ARM * Use system-wide CMake instead of custom version Our custom version is too old for VS 2019 * Use DML and build shared lib in ARM CI * Restore nuget packages in ARM CI * Disable DML * Refactor ARM debug/release builds * Use system packaged Python version * Remove hardcoded Python path * Downgrade Python to 3.7 for build * Remove explicit CMake path * Fix invalid JSON in cgmanifest.json (#2760) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * Remove usage of WHOLEARCHIVE in WinML CMake and add WinMLAdapterFactory (#2726) * Remove usage of WHOLEARCHIVE in WinMLAdapter CMake and add WinMLAdapterFactory * PR feedback, no need for dll(export) since using def file * PR comments * Small comment in gen_def.py * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * Move winml_provider_factory.h to proper location (#2801) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Filter CPU case for IsFloat16Supported (#2802) * Merge fixes * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * Fix warnings that cause build to fail * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * Make DML include path global (#2882) * Make DML include path global * Add generated cppwinrt headers to winml_lib_common * Integrate changes to WindowsAI to make ADO Build (#2886) * Revert "CMake cross-generator fixes (#2790)" This reverts commit dbe7d97fa1ab155f1309bced87199527e8f35bd2. * add additional suppress warning in onnx_proto * ignore /wd4996 warning * DML execution provider fixes * Revert "Revert "CMake cross-generator fixes (#2790)"" This reverts commit 1ae7b4bcbc02edc881ad28685da98e095dfceb17. * Update func signature of custom op function overloads * common devicehelpers fixes * Add pch.h for winml_lib_common * re-add winml_lib_common_dir/inc to include path for winml_adapter * User/orilevari/dml redist shared folder (#2890) * move dml nuget package directory up one level to make it shared between build flavors * Merge conflict fix * Revert "Merge conflict fix" This reverts commit 142fa72cf9ce4344ad717b50b7ea2b8582aadc7c. * Revert "Merge remote-tracking branch 'origin/master' into windowsai" This reverts commit 6e2126d46e5e5f564d65da37dd4f70c93dd81165, reversing changes made to b3f5583dc9249834b947c8ea905f6a98060d5bd6. * Make winml_test_common free of test macros (#2902) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Don't generate files outside the build root (#2914) * Don't generate files outside the build root * Add onnxruntime_EXTERNAL_DEPENDENCIES to WinML * Add DML depedency on RESTORE_PACKAGES * User/orilevari/fix yaml merge bugs (#2918) * Add winml test source parameter into cmake function (#2919) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Add arguments to unittest cmake functions * remove comment * Revert "Revert "Merge remote-tracking branch 'origin/master' into windowsai"" This reverts commit ade5abe72a4234fdbc3623093c61c02c6b0bdc26. * Fix breaks from merge with ORT master * Brianma/linux (#2917) * don't include windows.h in cross-plat header * add default case for switch statement * signed/unsigned mismatch fix Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> * User/sheilk/winml adapter c api (#2891) * Create winml adapter c api * fix build * make it build * move adapter into onnxruntime core/session * entry point not exported * minor changes * make model metadata work * make tests pass * implement all the model reflection apis on the adapter c abi * update the new ort interface to create a lotus ennvironment with a logging sink * start adding ort env * move all winml code into adapter folder/lib to isolate it * ensure a single logging manager at a time * start refactoring session * refactor session creation interface * add cpu and dml session option methods to adapter * finish session init * stub out interfaces in ort lib to perform similar mechanics of iinference session * enable profiling, and enable schema override * update session register graph transformers * turn back on custom registry for custom ops * Add sync api * add last c api stubs * should build... but all feature values are broken since this is in flight to moving all implementation details into ivalue * remove ep adapter header * Implement DML execution provider functions from adapter (#2846) * Implement DML execution provider functions from adapter * Use functions in OnnxruntimeEngine.cpp * make map/sequence type_infos freeable, and start implementing ivalue * make it build again * implement value methods * implement remaining methods * remove com adapter abi * check dml session * cache the allocator on ivalue * check if resource is cpu/gpu when access its mutable data * update tensor * mismatched parentheses * fix tensor base and binding obj * it evaluates tensors! sometimes... * minor fixes * enable gpu evals * wrapper all existing winml adapter apis with API_IMPL to try catch (#2854) * update winml... tensor strings are broken, need to template tensorbase to do different things for strings * make tensor strings work with 2 copies in/2 copies out * Fix tensor string and allocator bug * make maps work again... needs some fixes still * Make it build! * enable map inputs * map outputs * unbound outputs for sequences and maps * User/xianz/merge windowsai (#2883) * Packaging pipeline changes for VS 2019 (#2711) * Tiny fix to codegen * Simplify cache implementation and avoid static variables that may carry over between models * Extend DML kernels (#2641) * Additional DML operators * Check unsupported attributes and inputs * Address PR comments * Add kernel capability function used for partitioning, and re-enable stride-based int64 support based on value range * Fix test failures * Build fix * PR comments * Update Nuphar tutorial notebook (#2721) 1. Reflect int8 GEMV improvements for multi-threading from #2696 2. Add notes on multi-threading control using OpenMP 3. Add samples of running multi-isa AOT, and show int8 GEMM differences between AVX and AVX2 4. Add rnn_benchmark example to resolve #1993 * Add schema for new Qops (#2611) * Add schema for new Qops * adding shape inference + qlinearaveragepool * plus review comments * plus review comments * updates per review comments * plus review comments * [server] Add supposed for model_name and model_version as cli parameter (#2708) * remove 64bit warning message from python validation. (#2727) * MLAS: ARM64 build fix (#2734) fix bad usage of vreinterpret to cast vector element types * Fix broken python docs links (#2740) * Fix build on Mac OS (#2731) mac os ld doesn't support --while-archive, correct option is -all_load * fix ngraph wheel (#2737) * fix ngraph wheel 1.1.0 onnxruntime_ngraph wheel doesn't work * remove libdnnl.so in nGraph Libs * make it easy to compare * Split onnxruntime server to a separated folder (#2744) * Fix build for Python 3.8 (#2747) * Fix build for Python 3.8 * Update protobuf to 3.11.2 (#1928) Update protobuf to 3.11.2 (#1928) * Change default optimization level to All (from Basic) (#2745) * change default optimization level to All (from Basic) * fix test * fix c# test * Update numpy to 1.18 (#2758) * Update numpy to 1.18 * Pipeline changes for python 3.8 (#2753) 1. Pipeline changes for python 3.8 2. Fix a regression in setup.py which was just introduced in the previous commit. Please notice, we still haven't made python 3.8 + Windows + CUDA work. * Add basic stacktrace output for posix debug builds. (#2749) * [NupharEP] fix a race condition when multiple sessions running different models concurrently (#2772) * Revert "Change default optimization level to All (from Basic) (#2745)" This reverts commit 56bb503c2f26474b6613bcb2a198691a11dcef38. * Fix typo in error message (#2736) * Rename MKL-DNN to DNNL to fix broken link (#2730) * Fix nightly build version number issue * Pass BUILD_BUILDNUMBER to linux docker * Disable featurizers in python packages * Import more featurizers (#2781) Make kernels non-template. Add input constraint for learnt data. Add min_max_scalar_transformer, robust_scalar_transformer, inputation_marker_transfomer, label_encoder_transformer, missing_dummies_transformer along with tests. Advance Featurizers library commit. * Implement a more stable softmax (#2715) * Implement a more stable SoftMax e^x is represented as infinity if x is large enough, like 100.f. Infinity divided by Infinity is a NAN. Thus, softmax gets a NAN if one or more item are large enough. A math transform as below is leveraged to get a stable softmax: e^xi/(e^x1 + ...e^xn) = e^(xi - max) / (e^(x1 - max) + ... + e^(xn - max)) And for convenience, force max to 0.f if all xi are negative * Contributing: Fix a typo (#2784) * ACL EP GEMM improvements (#2780) When it is posible we use a fully connected layer instead of the gemm implementation. This will let the library use the best implementation based on the input data. * ACL EP convolution improvements (#2774) Added the optimized implementation for depthwise convolution for both ACL v19.02 and ACL 19.05. Also the pointwise convolution seems to be more optimal in the CPU implementation so we opted for that instead. * Add script for release Nuget validation (#2719) * Initial commit * Nits * Disable a test temporarily * Change working directory * Test * Add download python step * Test update * More changes * Fix space issue * Fix * Verify nuget signing * Fix * Spaces * PR feedback * Nit * Fix * Fix * Remove temporary changes * add uint8 support to where op (#2792) * Improve bert optimization script: (#2712) (1) Move input int64=>int32 conversion to embed layer fusion. (2) Output epsilon attribute for LayerNormalization fusion. * add session creation time cost. (#2798) * ML.NET team needs featurizers within a package (#2789) Add auto ml featurizers to Windows, MacOS as well as to GPU packaging-pipelines. * Initialize max of softmax with lowest of float (#2786) * MLAS: update SGEMM threading parameters (#2808) * add interface to copy batch tensors. (#2807) * add interface to copy batch tensors. * onnxruntime * speed up Windows TRT CI (#2811) * don't run cuda tests if building with tensorrt * remove unnecessary build options for win trt ci * refactor win gpu tensorrt ci yml * --numpy_version=1.17 * update * update * azcopy and cuda path * Update test data (#2356) * Add timeseries imputer transformer featurizer kernel (#2813) Make kernels non-template. Add input constraint for learnt data. Fixup tests. Add two more featurizers along with tests. Tests fail. min_max_scalar_transformer robust_scalar_transformer Fix tests serialized stream by prepending version bytes. Add inputation_marker_transfomer and the test. Fix up float/double type designations. Added label_encoder_transformer along with a test. string_throw case is broken at the momement. Fix labelencodertransfomer_test.cc string_throw case Rename maxabsscalertransformer_test.cc Add MissingDummiesTransformer along with the test. Update manifest. Add TimeSeriesImputerTransformer definition, implementation and tests * Fix memory leak in TRT (#2815) * fix memory leak issue * revert EP_FAIL on enueueV2 * Add manifest missing comma * Run static code analyzer on most of our code (#2817) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * update quantization doc (#2783) * update documentation for quantization script * plus some spell corrections * Filter CPU case for IsFloat16Supported (#2802) * update default optimization level + fix gemm_activation fusion (#2791) * update defualt optimization level + fix gemm_activation fusion * fix typo * add unit test and incorporate review comments * fix test comment * Fix dnnl wheel package name (#2823) * Append '-dnnl' to whl package name when --use_dnnl * Update build.py * Update Ubuntu & TensorRT version in README (#2820) Dockerfile.tensorrt is using nvcr.io/nvidia/tensorrt:19.09-py3 as base Image, update Ubuntu and TensorRT version according to https://docs.nvidia.com/deeplearning/sdk/tensorrt-container-release-notes/rel_19-09.html#rel_19-09 * Merge fixes * Add OneHotEncoder and HashOneHotEncoder kernels. (#2830) Add defs and imlementation for OneHotEncoders, adjuist date_time_transformer kernel and test. Add OneHotEncoder kernel test. Add HashOneHotVectorizerTransformer unit test. This does not link due to multiple definitions of functions that are included into header from a CPP file. * Upgrade gtest to the latest version (#2827) WinML would like to update the googletest submodule. They want some newer features (namely GTEST_SKIP to skip tests programmatically and be able to skip entire fixtures easily) and would need to update the submodule version. However, because the new version of code hit a bug in gcc, even though the bug is already fixed in the latest gcc but we're using gcc 4.8.x and it won't get patched for the bug, so we have to do a compromise, change our code a little bit to make it work. The gcc bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=51213 * Add support for int64_t for topk CPU. Fixes github issue #2806. (#2833) * Ignore allocator type in ExecutionProviders allocator map. Make default initialization of OrtMemoryInfo more clearly invalid. (#2768) * Remove allocator type from the key comparison in ExecutionProviders. Remove usage of DummyArena as it's no longer necessary. * Fix x86 tests where arena allocator is disabled. Make initialization of OrtMemoryInfo clearer by adding Invalid enum value. * Make OrtValueNameIdxMap::MaxIdx more intuitive. * Convert ExternalProject Featurizers into git submodule (#2834) Add git submodule for Featurizer library. Update cmake to build for git submodule. * add domain check for nodes + update documentation (#2831) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Fix Linux CUDA nuget packaging pipeline break * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * update optimization doc for BERT related fusions (#2819) * Add bert related transformers to doc * Add execution provider and comment for bert optimizations * Add comment about accuracy impact of approximation * Fix warnings that cause build to fail * MLAS: enable threading for quantized GEMMs (#2844) * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * remove old winmladapter cpp Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * move sequence implementation into ort lib... still commented out... need to turn back on... * begin sequence implementation * make maps and sequences work * fix broken tests * remove dead code * misc cleanup * CR feedback * User/xianz/winml adapter c api (#2869) * wrapper all existing winml adapter apis with API_IMPL to try catch * Return HR or Throw for WinML adapter APIs if failed * undo macro wrapper for two places * Wrap error macros around ort apis, too. * address CR feedback #2 * add more api throw/return macros * Revert changes no longer needed * revert changes to cxx api * format winml lib.ort and winml adapter * remove static pheonix singleton Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * missing use_dml check in winml_adapter_session (#2930) * --use_dnnl flag was mangled in merge (#2931) * use dml macro not wrapping custom registry code (#2934) * Disable LNK4199 winml_dll to enable cuda builds (#2936) * Disable LNK4199 in winml_dll * linkler->linker * LearningModelSessionAPITestGpu.CreateSessionWithCastToFloat16InModel should return DXGI_ERROR_UNSUPPORTED when FP16 not supported (#2937) * Disable LNK4199 in winml_dll * linkler->linker * Need to return DXGI_ERROR_UNSUPPORTED when Model does not support fp16 * Publish build symbols (#2939) * Publish build symbols * Don't upload PDBs for .exe files * Make x86 build (#2943) * fix last remaining size_t/int64_t warnings->errors (#2948) * TensorString, Sequences and Maps use the first allocator, but should use the cpu default allocator. (#2952) * fix tensor string allcoator * clean up default allocator usage for strings in winml lib/api.ort Co-authored-by: Ryan Lai <ryalai96@gmail.com> * Handle tensor shape of zero (#2954) Co-authored-by: Ryan Lai <ryalai96@gmail.com> * CR feedback (#2970) * CR feedback * fix weird formatting on privacy readme * Add 'All rights reserved.' everywhere * readd all rights reserved to winml_provider_factory.h * remove extra space in comment * remove extra whitespace * fixes post master merge * remove winml from nuget gpu pipeline * set IR VERSION on generated_model in rnn_benchmark (#2972) * Fix slice conformance failures (#2908) Co-authored-by: Adrian Tsai <adtsai@microsoft.com> Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Paul McDaniel <paul_mcdaniel@hotmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Dwayne Robinson <fdwr@hotmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com>
2020-02-05 01:12:19 +00:00
[threading(both)]
[marshaling_behavior(agile)]
[dualapipartition(1)]
runtimeclass TensorBoolean : ITensor, [contract(MachineLearningContract, 2)]Windows.Foundation.IMemoryBuffer
{
static TensorBoolean Create();
static TensorBoolean Create(Windows.Foundation.Collections.IIterable<Int64> shape);
static TensorBoolean CreateFromArray(Windows.Foundation.Collections.IIterable<Int64> shape, Boolean[] data);
static TensorBoolean CreateFromIterable(Windows.Foundation.Collections.IIterable<Int64> shape, Windows.Foundation.Collections.IIterable<Boolean> data);
Windows.Foundation.Collections.IVectorView <Boolean> GetAsVectorView();
[contract(MachineLearningContract, 2)]
{
//! Creates a tensor object, allocates a buffer of size 'shape', and copies all of 'data' into it.
static TensorBoolean CreateFromShapeArrayAndDataArray(Int64[] shape, Boolean[] data);
//! Creates a tensor object, creates a tensor of size 'shape', and uses the data in 'buffer'.
//! 'buffer' represents a byte packed array of boolean values.
static TensorBoolean CreateFromBuffer(Int64[] shape, Windows.Storage.Streams.IBuffer buffer);
}
}
[contract(MachineLearningContract, 1)]
INBOX_ONLY([static_name("Windows.AI.MachineLearning.ITensorDoubleStatics", a86693c5-9538-44e7-a3ca-5df374a5a70c)])
INBOX_ONLY([interface_name("Windows.AI.MachineLearning.ITensorDouble", 91e41252-7a8f-4f0e-a28f-9637ffc8a3d0)])
Merge windowsai (winml layering) into master (#2956) * Initial Commit * Merged PR 3985217: add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc (#2346) add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc and violating our OS layering requirements. We linked against onecoreuap_apiset.lib in VB so we will continue doing this, but I am still unsure why not to link against onecore instead since that is where we ship. However, since Sheil is the owner of this code we will wait to discuss with him before changing anything. * Initial changes for layering * more snipping to get core into ort * update build instructions to include --build_shared_lib (#2358) * update build instructions to include --build_shared_lib * fix line breaks * Task 23998197: add winml_lib_core into onnnxruntime.dll (#2368) * Task 23998197: add winml_lib_core into onnnxruntime.dll * PR feedback build break on perf_test * return proper error when the model path isn't found (#2391) * LearningModelSession is cleaned up to use the adapter, and parts of b… (#2382) this is a big PR. we are going to move it up to layer_dev , which is still a L3 so we are still safe to do work there agile. we are going to move this into the L3 so that ryan can start doing intergration testing. we will pause for a full code review and integration test result prior to going into the L2. >>>> raw comments from previous commits >>> * LearningModelSession is cleaned up to use the adapter, and parts of binding are. * moved everything in the winmladapter made it all nano-com using, WRL to construct objects in the ORT side. base interfaces for everythign for winml to call cleaned up a bunch of winml to use the base interfaces. * more pieces * GetData across the abi. * renamed some namepsace cleaned up OrtValue cleaned up Tensor cleaned up custom ops. everything *but* learnignmodel should be clean * make sure it's building. winml.dll is still a monolith. * model moved over. everything builds clean. step ! * weak ref comment * Layer dev paulm (#2408) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * Layer dev paulm (#2414) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * Layer dev paulm (#2423) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * Layer dev paulm (#2424) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * Layer dev paulm (#2425) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Layer dev paulm (#2426) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * fixed map and sequence when passing stl types across the ABI . found a leak in nvidia driver, but skipped it. all winmlapitests pass now * Moved SessionOptions over to the abi * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * Register WinML TraceLogging provider on Onnxruntime.dll (#2455) * Register WinML TraceLogging provider on Onnxruntime.dll * Add ifdef to make sure trace logging provider has telemetry option when LAYERING_DONE * No need for ifdef for TraceLoggingOptionMicrosoftTelemetry * PR feedback * Move etw registration into lotus environment constructor and deresgister in lotus environment destructor * Brianma/cpuwinml (#2466) * allow building winml cpu without dml. * Brianma/breaks (#2469) * fix some more breaks * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * move dml checks out of winml and into the adapter * better error handling * Brianma/fi (#2470) * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * fix bad merge * Checking in a staging checkpoint point so that Ryan can work with me in parrallel * build break. * Brianma/testfails (#2473) * add missing ir version to dictvectorizer-string.onnx * add missing ir version to relu.onnx * add missing ir version to zipmap*onnx * add IR version to manually generated models * remove an unnecessary ifdef dml * Brianma/windowsai fi (#2475) * update dockerfiles/README (#2336) * Make elementwise op run 4 items per thread (#2335) Description: Describe your changes. Make elementwise op run 4 items per thread unroll for loop to leverage ILP remove unnessary N==0 check inside elementwise GPU kernel Motivation and Context Why is this change required? What problem does it solve? It can improve the performance of GPU elementwise ops. ~2% performance gain on popular NLP bert model. If it fixes an open issue, please link to the issue here. * Add CUDA GatherElements kernel (#2310) * Updates * Update test * Update * Updates * nits * PR feedback * Update * Update * PR feedback * PR comments * Update * Fix build * Fix build * Nits * Fix * Layer Normalization Fusion (#2319) basic layer normalization transform * Add FastGelu Cuda Op for Gelu and Add bias fusion (#2293) * Add FastGelu cuda op * Add AddBiasGelu for experiment * Revert "Add AddBiasGelu for experiment" This reverts commit 5c1ee019858c657e6bb75887265cb85675626e5b. * Add bias * Add unit tests * update comment * update script * fix build error * update coding style * update for CR feedback Enable half2 optimization only when cuda arch >= 7.0 * move _Tanh to common.cuh * implement CPU contrib OP Attention (#2333) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanUnusedInitializers. (#2320) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanupUnusedInitializers. This means initializers that have been replaced during graph optimizations are not left in the GraphProto when we save an optimized model. * Handle edge case where a model has an unused initializer with matching graph input by also removing the graph input. * Use non-const iterators in std::find_if calls to make centos build happy. * Nuget pipeline changes (#2305) 1. refactor the pipeline, remove some duplicated code 2. Move Windows_py_GPU_Wheels job to Win-GPU-CUDA10. We'll deprecated the "Win-GPU" pool 3. Delete cpu-nocontribops-esrp-pipeline.yml and cpu-nocontribops-pipeline.yml 4. In Linux nuget jobs, run "make install" before creating the package. So that extra RPAH info will be removed * Cuda Reverse Sequence Op, maping types of same size using same template function. (#2281) * Set ElementType to String type of node metadata, instead of byte[] (#2348) * Set ElementType to String type of node metadata, instead of byte[] * Fix spacing * Introduce PrimitiveType into a Type System along with an integer constant (#2307) Improve perf by avoiding GetType<T>() calls. Introduce MLTypeCallDispatcher to switch on Input Type. Add Tensor IsType<T>() fast method. * Fix/test dim value of 0 handling in a couple of places (#2337) * Update the CUDA Where implementation broadcasting logic to handle a dim with value of 0. Add unit test Also add unit test for unary op with dim value of 0 * Exclude ngraph from Where test with 0 dim. * Openvino EP R3.1 onnxrt server (#2357) * onnxrt server with OVEP * onnxrt server with OVEP * Update Dockerfile.server.openvino * onnxrt server OVEP fix reviews * onnxrt server OVEP fix reviews * Implement cuda nonzero op. (#2056) Implement cuda nonzero op. * Direct use python numpy array's memory if already contiguous. (#2355) * Direct use python numpy array's memory if already contiguous. This could greatly improve performance for session with large input, like big image 1920x1080 fastrcnn, 30~40% speed up could be achieved. * Add test case enforce contiguous/non-contiguos numpy array as inputs. * Add helper to create output to minimize binary size. (#2365) Add ConstEigenTensorMap typedef so we don't unnecessarily const_cast the const input Tensor. * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS (#2369) * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS * update * Add Tracelogging for profiling (#1639) Enabled only if onnxruntime_ENABLE_INSTRUMENT is ON * test bidaf with nuphar for avx target (#2370) increase nuphar test coverage a bit * Fix a bug in TLS refcount that may destabilized CUDA CI (#2374) * update output size calculation for resize (#2366) * change how output size is calculated for resize op * add tests for ver 10 resize * Extend OneHot CPU kernel to support more types (#2311) * Extend OneHot CPU kernel to support input int64_t, depth int32_t, output float * Skip BERT before the test data fix is picked up * Fix bug with Slice. Need to pass in flattened input dimensions so the initial offset into the input is calculated correctly. (#2372) * Add opset 11 version of Split to CUDA ops (#2376) Organize the CUDA ops definitions so all the opset 10 and 11 parts are together (same setup used for CPU ops) * Layer Norm Fusion Fix (#2379) * layer norm fusion fix * Add input shape check in code and unit tests * Fuse Add + Gelu (#2360) Implement the transformer to fuse add + gelu Implement the accurate kernel * Skip layer norm transform (#2350) * skip layer normalization transformer * Another try to stabilize CUDA CI (#2383) The root cause seems to be failure in CUDA dealloc when tear down. cudaFree return code was ignored before, so should the debug check. * fix BUILD.md typo (#2375) build.py: error: argument --config: invalid choice: 'RelWithDebugInfo' (choose from 'Debug', 'MinSizeRel', 'Release', 'RelWithDebInfo') * Fixed compilation with ngraph (#2388) * Fix reuse logic in allocation planner. (#2393) * Fix reuse logic in allocation planner. * PR comments * Add helpful comments * Don't allow reuse across string tensors. * [NupharEP] Multiple optimizations (#2380) Fuse transpose into MatMul Implement Pow and constant scalar simplification Vectorize ReduceMean Improve symbolic shape inference Minor updates for better debugging in fused function name * Avoid using the default logger in the graph lib and optimizers (#2361) 1. Use the session logger if it is available. 2. Don't disable warning 4100 globally. We should fix the warnings instead of disabling it. * Change CUDA implementation of Transpose to support all fixed size tensor types (#2387) * Change CUDA implementation of Transpose to not use a typed kernel so we can support more types with minimum binary size. Add support for 8, 16, 32 and 64 bit types. Add unit tests. Add method so the implementation can be called directly (will be used by CUDA Scan very soon). * Disable TensorRT for MLFloat16 and int8 unit tests. * Address PR comment and add support for calling cublas implementation if type is mlfloat16. * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. (#2398) * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. * [NupharEP] force some low/zero cost ops to be inlined (#2409) * fix cross compile bug (#2415) * Minor optimization: if a node has already been placed, there's no need to find a kernel for it. (#2417) * Add Reshape Fusion (#2395) * Add reshape fusion * Add some comments * update comments * update comment format * update according to feedback * update for recent logger change * fix build error * (1) Support both input and output edges in find path in graphutils (2) Add a test case of only one constant initializer of Concat input. (3) Refactor ReshapeFusion class to allow add more subgraph fusion in the future. * fix error * (1) loose constraint on initializer: non constant is allowed for reshape fusion. (2) Change versions type to vector. (3) Add logging. (4) Return false when multiple output edges matched in FindPath. Add comments. * only allow one direction (input or output) in FindPath * [NupharEP] Update notebook and docker image (#2416) Add BERT squad in Nuphar tutorial Enhance speed comparsion readability * Fix the issue in matmul_add_fusion (#2407) Fix the issue in matmul_add_fusion If Muatmul + Add has shape [K] * [K, N], reset it to [1, K] * [K, N] will make the output shape to [1, N] will also requires a reshape on the output. Fix: just remove the shape reset to not fuse it. Add a negative test case for matmul+add fusion * feat(treeregressor): Update TreeEnsembleRegressor for type support (#2389) Updates the `TreeEnsembleRegressor` to allow for `double`, `float`, `int64`, and `int32` inputs to match the upstream specification. Signed-off-by: Nick Groszewski <nicholas.groszewski@capitalone.com> * onnxrt server documentation update (#2396) * Added support for Pad-2 operator in OpenVINO-EP (#2405) * Add CUDA If operator. (#2377) * Add CUDA If operator. Uses CPU operator for implementation. By adding a CUDA version the inputs/outputs (with the exception of the 'cond' input) stay on GPU, and no other logic is required to avoid a copy to CPU across the control flow node. * Improved documentation for onnxruntime::utils::SwapByteOrderCopy(), added precondition check. * Fix the type constraints on CUDA If operator to exclude strings. (#2431) * add Im2col<uint8_t> (#2438) * Adjust codegen vectorization width from target (#2439) * Adjust codegen vectorization width from target * Add CUDA Scan operator. (#2403) * Add Scan CUDA op. Uses CPU implementation for logic. Added some device specific functors for handling when data needs to be manipulated on a different device. Added ability to override the materialization logic in the OrtValue slicer so DML can plugin their handling. * Fix Windows GPU C API packaging pipeline failure (#2440) Fix Windows GPU C API packaging pipeline failure (#2440) * Correctly handle implicit inputs for fused nodes (#2390) * Correctly handle implicit inputs for fused nodes Previously, nuphar's partitioning function didn't include node's implicit inputs into the inputs list of MetaDef, and hence a crash was triggered in the onnx graph checker. This commit fixed the issue. Furthermore, it also fixed a related issue where we didn't add implicit inputs into graph_inputs_excluding_initializers_ in Graph::SetGraphInputsOutputs. the issue was that graph_inputs_including_initializers_ populated by SetInputs (e.g. called by FunctionImpl::FunctionImpl) may contain implicit inputs which were not of any node's initializers in the graph. Because they were not part of any initializers, these implicit inputs couldn't be visited by going through all nodes' inputs. Consequently, they would *not* be added into graph_inputs_excluding_initializers_. We fixed the issue by first copying the populated graph_inputs_including_initializers_ into graph_inputs_excluding_initalizers_, which then had both initializers and non-initializers as its initial content. Later, we erase initializers from the list. In this way, we can ensure all implicit inputs to remain in graph_inputs_excluding_initializers_. * refined comments and fixed duplicates Address CR by revisiting comments in terms of implicit inputs Also fixed an issue by skipping duplicates while copying inputs from graph_inputs_including_initializers_. * address CR explain why we need to collect nodes' implicit inputs * don't rely on pointer values for iterating std::set Previously, openvino relied on iterating a set of NodeArg pointers to construct inputs and outputs for a fused graph. It could cause non-determinism. The reason was that although iterating std::set by itself is stable, pointer values of NodeArgs may vary. Consequently, we could end up visiting the set's elements in different orders for different runs for the same test, which resulted in constructing inputs (and outputs) with different orders to the fused graph. For example, for the same test, we may have inputs [A, B] in some runs but inputs[B, A] in others. Let's use std::string as the key type to avoid such nondeterminism. This commit also added implicit inputs into meta->inputs while returning the capability from the openvino provider. * Fixed another latent issue in openvino's GetCapability function The issue was that we couldn't simply erase fused_inputs and fused_outputs while iterating the nodes. For example, an output NodeArg may have multiple uses, and it's wrong if we erase it from fused_outputs when we encounter only one of its uses as input. * Remove DeviceAllocatorRegistry class (#2451) Remove DeviceAllocatorRegistry class * CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library. * Parallel Gelu with ParallelFor (#2399) Parallel Gelu to get better performance for Gelu * Clean up build.py (#2446) * Pull the latest image before running docker build * Fuse SkipLayerNorm with Bias (#2453) Fuse SkipLayerNorm with Bias * Allow more than one invocation of CreateEnv in the same process. (#2467) * Allow more than one invocation of CreateEnv in the same process. * Fix centos build * Symbolic shape inference improvements: (#2460) * Symbolic shape inference improvements: - add a mode to guess unknown ops' output rank - add support for GatherND - add support for If - fix a bug in get_int_values when then tensor rank > 1D, by treating it as no sympy data - add symbol to literal merge when ONNX silently merges dims - fix a bug in Concat when input dim is 0 - fix a bug in ConstantOfShape that computed dim is not updated - add support for dynamic shape in ConstantOfShape - fix a bug in Loop output shape that loop iterator dim is not inserted at dim 0 - add support for dynamic padding in Pad - add support for dynamic shape in Reshape - add support for Resize with opset > 10, by treating output dims as dynamic - fix a bug in Slice when starts/ends are dynamic - restrict input model to opset 7 and above - make output model optional to avoid disk write when testing Run model tests for symbolic shape inference Reduce 2GB docker image size of nuphar * add additional test data set for nuget pipeline (#2448) * add SAS token to download internal test data for nuget pipeline * update azure endpoint * fix keyvault download step * fix variable declaration for secret group * fix indentation * fix yaml syntax for variables * fix setting secrets for script * fix env synctax * Fix macos pipeline * attempt to add secrets to windows download data * fix mac and win data download * fix windows data download * update test data set url and location * Revert "Brianma/windowsai fi (#2475)" This reverts commit 5780b864a15513fda4eadbfc2b5345fefe70b5ec. * Add scenario tests (#2457) * Add scenario tests * Remove TODO from model license * Add winml_api test dependency * fix model load test. fi from master changed the constructor (#2483) * make api tests all pass (#2486) * fix bad merge * fix bad model merge * Layer dev paulm (#2492) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * Rename ambiguous header (#2489) * fix one more missing IR version model (#2500) * add missing IR version to 4 more models used by scenario tests (#2501) * Add CLI parameters to test runner, build WinML in ARM and x86 CI (#2479) * Support test parameters through CLI arguments * Add WinML do Windows x86/ARM CI builds * Code style fixes * Update googletest Remove GPUTEST macros everywhere now that GTEST_SKIP is supported * Refactor main.cpp * Build scenario tests without DML * Link scenario tests to DML when it's enabled (#2502) * Layer dev release pipeline (#2488) Adds winml binaries to existing cpu nuget package, and creates new gpu dml nuget package with winml binaries and DML EP. * Layer dev paulm (#2506) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * Remove usage of IOBinding in WinML and use C_API Run method (#2504) * remove usage of iobinding * Change data structure to use vector of Ort::Values * Polish bind input / output * Use C APIrun method * Update providers on evaluate getresults * Remove run and IObinding interface from WinMLAdapter * Remove use of IObinding * bind unbound outputs code moved to learningmodelbinding * clean up unneeded istensor adapter function * Fix comment * Check if session is closed before binding and clearing * PR feedback * Layer dev paulm (#2507) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * Make tests dependend on winml_dll (#2509) * add dml binaries to DirectML package and be more explicit about condition variables (#2520) * re-enable warnings for winml builds and fix the warnings that were hiding (#2526) * turn devmode back on for winml builds * fix some warnings. include protobuf in a way that disables some warnings * undo protobufhelpers changes and just ignore 4100 errors in pb code * attempt to isolate protobufhelpers errors * add template specialization for getting tensor proto data * Layer dev paulm (#2533) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * Spawn child process to run DeviceLostRecovery scenario test (#2530) * Spawn child process to run DeviceLostRecovery scenario test * Layer dev paulm (#2536) ori said yes * add missing namespace to winml_trace_logging_provider in lotusenvironment.h (#2542) * Handle exception thrown from all apis in WinMLAdapter (#2539) * various changes to unblock windowsai ADO build * Fix custom ops scenario tests (#2562) * Do not shutdown protobuf after ort environment gets destroyed. Lazy load lotus environment first time it is needed * comment typo * pr comment about calling phoenix singleton * Make lotus_environment static in winmladapter * Layer dev paulm (#2567) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * making windowsAI azure dev ops work. * code review comments. * revert changes * Cmake and preprocessor fixes that where uncovered by building on agents without DML available via SDK * Layer dev dml delayload (#2580) * Brianma/cpu (#2583) * don't include dml stuff in cpu builds * tests that link the image lib also need the telemetry lib now * Throw Winml_err_invalid_binding if binding gpu resource on cpu device (#2589) * Throw Winml_err_invalid_binding if binding gpu resource on cpu device * PR comments. No need to query executionprovider for is gpu device * User/xianz/ortthrow (#2596) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * User/xianz/ortthrow (#2599) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * add status error message * Remove uwp onsuspending winrt call because logruntimeperf is getting removed (#2630) * User/xianz/dedup telemetry (#2631) * investigate duplication of telemetry in winml and ort * remove winml telemetry events * telemetry executionProviderEvent * remove unneccessary file and refactor code little bit * Revert back TelemetryEvent, which send up ETW event. * merge changes from layer_dev to windowsai (#2638) * Remove underscore from googletest names (#2616) * Fix leaking memory allocator Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24278761 and https://microsoft.visualstudio.com/OS/_workitems/edit/24330198 * Explicitly initialize Ort::Value with nullptr * Cache WinML adapter * bad merge * define private version of dxcore enum that is added in 19H1 SDK. (#2654) * add comment for explaning private definition of dxcore d3d feature level ennum value. (#2672) * do not package directml.pdb for redist packages. (#2676) * Fix leaking operator registry (#2645) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24354916 * User/orilevari/windowsai master merge (#2674) merge resolutions included pulling in telemetry logic that was merged to master and not windowsai and dereferencing InferenceSession::sessionstate now that it is a unique pointer * Delete Ort Allocator in LearningModelBinding (#2653) * Delete OrtAllocator in LearningModelBinding * PR comments to make Ort::Allocator a smart pointer * Small comment change * PR feedback to clean up code * PR feedback on move semantics * Clean up std::move * Fix memory leaks (#2679) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24356109, https://microsoft.visualstudio.com/OS/_workitems/edit/24388361 and https://microsoft.visualstudio.com/OS/_workitems/edit/24388596 * various changes to properly organize and skip GPU tests. For now for No DML builds we will not run GPU tests at all. In the future we should adapt the tests to expect the appropiate errors. (#2695) * Windowsai without fi (#2701) * Disable Attention fusion tests when DISABLE_CONTRIB_OPS is defined (#2529) * Setup java ci (#2528) * Add provision in ORT for session options to be parsed when available via model file (#2449) * Initial commit * Fix gitmodules * Nits * Nits * Updates * Update * More changes * Updates * Update * Some updates * More changes * Update * Update * Merge * Update * Updates * More changes * Update * Fix nits * Updates * Fix warning * Fix build * Add comment * PR feedback * PR feedback * Updates * Updates * Update * More changes * Fix build break * Comment test for now * Updates * Updates * PR feedback * Updates * Nits * Add tests * Fix build * Fix build * Fix build * Fix build break * Fix build * Nits * PR feedback * More change * Expose GetSessionOptions in pybind logic and add unit test for python * Fix build * PR feedback * PR feedback * Revert "Disable thread pool creation when enabled OpenMP (#2485)" (#2535) This reverts commit 7c7d5a149c9ed52eec67304bae5c4b132166a8a1. * Add dynamic shape support in TensorRT execution provider (#2450) * remove onnx-tensorrt submodule * add new onnx-tensorrt submodule (experiment) for trt6 * update engine build for trt6 * update compile and compute for tensorrt6.0 * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * switch to onnx-tensorrt master for TensorRT6' * Update tensorrt_execution_provider.cc * Handle dynamic batch size and add memcpy in TensorRT EP * update test cases * Update tensorrt_execution_provider.cc * update onnx-tensorrt submodule * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.ubuntu_tensorrt * Update run_dockerbuild.sh * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update concat_op_test.cc * Update tensorrt_execution_provider.cc * Upgrade TensorRT to version 6.0.1.5 * Update onnxruntime_providers.cmake * Update CMakeLists.txt * Update reduction_ops_test.cc * Update install_ubuntu.sh * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.tensorrt * Update BUILD.md * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update onnxruntime_providers.cmake * Update install_ubuntu.sh * Update install_ubuntu.sh * Update gemm_test.cc * Update gather_op_test.cc * Update CMakeLists.txt * Removed submodule * update onnx-tensorrt submodule * update header file * Removed submodule * add submodule onnx-tensorrt kevin's branch shape-test' * add debugging code * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * merge master * Removed submodule * update onnx-tensorrt submodule * add more changes for dynamic shapes * Update tensorrt_execution_provider.cc * update for dynamic shape * update dynamic shape processing * fix logger issue * remove submodule onnx-tensorrt * add submodule onnx-tensorrt * add env variable min_subgraph_size * remove redundency * update document * use onnxruntime::make_unique * fix multi-run issue * remove some tests to save CI build time * Add dynamic shape test * Update TensorRT-ExecutionProvider.md * Add example of running Faster R-CNN model on TensorRT EP * Add more details on env variables * update environment variables * Update tensorrt_basic_test.cc * Update model tests * Update tensor_op_test.cc * remove --use_full_protobuf * Update build.py * User/xianz/telemetry (#2458) * enabme telemetry * enable telemetry * set enable telemetry as default * for debugging * remove log and set disable telemetry as default back * delete private file while testing * resolve comment: mainly add license header, rename macro and update docs * rewording in privacy.md * Fix integer overflow in cuda NonMaxSuppression implementation (#2540) * add test case that should pass but fail * fix nms * extract int_max_output_boxes_per_class * Introduce container type runtime checks and other improvements (#2522) Rework TensorSeq in a manner consistent with Tensor and SparseTensor in terms of type system setup. Reduce templating. Introduce helpers to ensure the same data type. Make OrtValue __dtor not virtual. Introduce ContainerChecker * Fix C API tests for centos and mac (#2544) * change c++14 to c++11 * add ld lib path for centos * enable csharp tests on macos * fix C API test on MacOS + fix manylinux dotnet install * fix manylinux dotnet install * fix lib link * Add back executable bit to build.py * Fix a bug handling negative begin pad values in Pad op (#2550) * Fix bug in Pad op * Update * DNNL CMAKE update (#2548) * Fix android build (#2558) * Update win-x86-ci.yml (#2557) Fix build pipeline break * Re-enable Windows C# tests (#2564) * disable onnx_test_runner -x invocations for dnnl (#2568) * Allow sequence length to be symbolic (#2559) * setup java ci mac (#2570) * make layernorm fusion to support opset 11 (#2545) * Fix a warning found in the latest VS release * Add more check on SkipLayerNorm and BiasGelu fusion (#2574) * Fix file not found error during docker build. (#2569) * Add ConvTranspose1D (#2578) * Ryanunderhill/packagename test (#2582) * [Nuphar EP] fixes for some object detection models (#2581) Update notebook tutorial with multi-threaded int8 GEMM from #2517 * EmbedLayerNormalization Fusion Improvement (#2553) Embedding layer norm fusion improvements - add more checks * Update version (#2584) * Temporarily exclude vgg19 test from Python backend test 1. temporarily exclude vgg19 test which comsumes too much memory, run out of memory on Upsquared device. Single test pass for vgg19, need furture investigation (#2588) 2. Update docker file to decrease the docker image size * Update docs for Android NNAPI EP (#2586) * Fix lto bug for protobuf and ubuntu * add path to build dir before test run (#2590) * Add missig env variables for mac pipeline test (#2595) * Fixed an issue in updating realized dims (#2597) when we update realized dims for scan's output, the sliced axis also needs to be inclusive, i.e. we should check with "dim >= insert_inclusive_axis", because the offset in the symbols are based on Scan sugraph. Otherwise, we would end up with shape mismatch later. * Java API for onnxruntime (#2215) * Add support for opset 11 in reshape fusion (#2592) Support opset verion 11 in reshape fusion * Rename automl python tools folder to featurizer_ops. (#2593) * Support opset 11 subgraph of Squad model in Embed Layer Normalization (#2605) Support opset 11 Squad model that is exported from PyTorch nightly. The embed layer uses Range op which is missed in the transformer. * symbolic shape inference: fix warnings in GPT-2 model (#2608) And revise nuphar perf test on BERT squad * Dump subgraph ID and fused graph ID (#2607) * Dump subgraph ID and fused graph ID Dump subgraph ID and fused graph ID for better debugging * Remove local static fused_count added a field global_fused_count_ to NupharExecutionProvider class * EmbedLayerNormalization Fusion For Dynamic Squad Model Opset 10 (#2613) Support subgraph of SQuAD model exported from pytorch with dynamic input axes * Allow providers to be set for InferenceSession at construction (#2606) * Remove unnecessary parameter in some places in GatherElements implementation (#2612) * Remove unnecessary parameter in some places * Update * Update * Make sure fenced tensor could not reuse other tensor. (#2561) Fix random error caused by this. * Improve Embed Layer Norm Fusion for SQuAD with static input shape (#2621) * fix float16 comparison in initializer (#2629) * epsilon attribute for layernormalization fusion (#2639) * removed unnecessary batch file and fix path (#2640) * Add shape inference to ConvTransposeWithDynamicPads schema (#2632) * Improve cuda expand() opeator's performance. (#2624) * Cuda pad optimize when no padding is needed. (#2625) * Shortcut cuda Pad() when no padding is needed. * Optimize cuda scatter() on 2D compatible. (#2628) * Optimize cuda scatter() on 2D compatible. * Add some comments. * fix build error for ARM (#2648) * Improve performance of resize() in Nearest mode (#2626) Special treatment for 2D, check same size as input image. And in 2d kernel, template use_expolation. * Fix memory exception in Layer Norm Fusion (#2644) * Windows CI changes(#2650) * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Revert "Windowsai without fi (#2701)" This reverts commit 285d4c85ff5c4e265f963208170304ef3461e684. * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Deref unique pointer for session_state * send shutdown event when dll is unloaded and EvaluationStop, SessionC… (#2704) * send shutdown event when dll is unloaded and EvaluationStop, SessionCreationStart Events. * Add EvalutationStart Event * add comment * use correct type for for loop (#2755) * ARM CI (#2759) * Set ARM agent pool * Set CMake generator to VS 2019 in ARM * Use system-wide CMake instead of custom version Our custom version is too old for VS 2019 * Use DML and build shared lib in ARM CI * Restore nuget packages in ARM CI * Disable DML * Refactor ARM debug/release builds * Use system packaged Python version * Remove hardcoded Python path * Downgrade Python to 3.7 for build * Remove explicit CMake path * Fix invalid JSON in cgmanifest.json (#2760) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * Remove usage of WHOLEARCHIVE in WinML CMake and add WinMLAdapterFactory (#2726) * Remove usage of WHOLEARCHIVE in WinMLAdapter CMake and add WinMLAdapterFactory * PR feedback, no need for dll(export) since using def file * PR comments * Small comment in gen_def.py * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * Move winml_provider_factory.h to proper location (#2801) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Filter CPU case for IsFloat16Supported (#2802) * Merge fixes * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * Fix warnings that cause build to fail * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * Make DML include path global (#2882) * Make DML include path global * Add generated cppwinrt headers to winml_lib_common * Integrate changes to WindowsAI to make ADO Build (#2886) * Revert "CMake cross-generator fixes (#2790)" This reverts commit dbe7d97fa1ab155f1309bced87199527e8f35bd2. * add additional suppress warning in onnx_proto * ignore /wd4996 warning * DML execution provider fixes * Revert "Revert "CMake cross-generator fixes (#2790)"" This reverts commit 1ae7b4bcbc02edc881ad28685da98e095dfceb17. * Update func signature of custom op function overloads * common devicehelpers fixes * Add pch.h for winml_lib_common * re-add winml_lib_common_dir/inc to include path for winml_adapter * User/orilevari/dml redist shared folder (#2890) * move dml nuget package directory up one level to make it shared between build flavors * Merge conflict fix * Revert "Merge conflict fix" This reverts commit 142fa72cf9ce4344ad717b50b7ea2b8582aadc7c. * Revert "Merge remote-tracking branch 'origin/master' into windowsai" This reverts commit 6e2126d46e5e5f564d65da37dd4f70c93dd81165, reversing changes made to b3f5583dc9249834b947c8ea905f6a98060d5bd6. * Make winml_test_common free of test macros (#2902) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Don't generate files outside the build root (#2914) * Don't generate files outside the build root * Add onnxruntime_EXTERNAL_DEPENDENCIES to WinML * Add DML depedency on RESTORE_PACKAGES * User/orilevari/fix yaml merge bugs (#2918) * Add winml test source parameter into cmake function (#2919) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Add arguments to unittest cmake functions * remove comment * Revert "Revert "Merge remote-tracking branch 'origin/master' into windowsai"" This reverts commit ade5abe72a4234fdbc3623093c61c02c6b0bdc26. * Fix breaks from merge with ORT master * Brianma/linux (#2917) * don't include windows.h in cross-plat header * add default case for switch statement * signed/unsigned mismatch fix Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> * User/sheilk/winml adapter c api (#2891) * Create winml adapter c api * fix build * make it build * move adapter into onnxruntime core/session * entry point not exported * minor changes * make model metadata work * make tests pass * implement all the model reflection apis on the adapter c abi * update the new ort interface to create a lotus ennvironment with a logging sink * start adding ort env * move all winml code into adapter folder/lib to isolate it * ensure a single logging manager at a time * start refactoring session * refactor session creation interface * add cpu and dml session option methods to adapter * finish session init * stub out interfaces in ort lib to perform similar mechanics of iinference session * enable profiling, and enable schema override * update session register graph transformers * turn back on custom registry for custom ops * Add sync api * add last c api stubs * should build... but all feature values are broken since this is in flight to moving all implementation details into ivalue * remove ep adapter header * Implement DML execution provider functions from adapter (#2846) * Implement DML execution provider functions from adapter * Use functions in OnnxruntimeEngine.cpp * make map/sequence type_infos freeable, and start implementing ivalue * make it build again * implement value methods * implement remaining methods * remove com adapter abi * check dml session * cache the allocator on ivalue * check if resource is cpu/gpu when access its mutable data * update tensor * mismatched parentheses * fix tensor base and binding obj * it evaluates tensors! sometimes... * minor fixes * enable gpu evals * wrapper all existing winml adapter apis with API_IMPL to try catch (#2854) * update winml... tensor strings are broken, need to template tensorbase to do different things for strings * make tensor strings work with 2 copies in/2 copies out * Fix tensor string and allocator bug * make maps work again... needs some fixes still * Make it build! * enable map inputs * map outputs * unbound outputs for sequences and maps * User/xianz/merge windowsai (#2883) * Packaging pipeline changes for VS 2019 (#2711) * Tiny fix to codegen * Simplify cache implementation and avoid static variables that may carry over between models * Extend DML kernels (#2641) * Additional DML operators * Check unsupported attributes and inputs * Address PR comments * Add kernel capability function used for partitioning, and re-enable stride-based int64 support based on value range * Fix test failures * Build fix * PR comments * Update Nuphar tutorial notebook (#2721) 1. Reflect int8 GEMV improvements for multi-threading from #2696 2. Add notes on multi-threading control using OpenMP 3. Add samples of running multi-isa AOT, and show int8 GEMM differences between AVX and AVX2 4. Add rnn_benchmark example to resolve #1993 * Add schema for new Qops (#2611) * Add schema for new Qops * adding shape inference + qlinearaveragepool * plus review comments * plus review comments * updates per review comments * plus review comments * [server] Add supposed for model_name and model_version as cli parameter (#2708) * remove 64bit warning message from python validation. (#2727) * MLAS: ARM64 build fix (#2734) fix bad usage of vreinterpret to cast vector element types * Fix broken python docs links (#2740) * Fix build on Mac OS (#2731) mac os ld doesn't support --while-archive, correct option is -all_load * fix ngraph wheel (#2737) * fix ngraph wheel 1.1.0 onnxruntime_ngraph wheel doesn't work * remove libdnnl.so in nGraph Libs * make it easy to compare * Split onnxruntime server to a separated folder (#2744) * Fix build for Python 3.8 (#2747) * Fix build for Python 3.8 * Update protobuf to 3.11.2 (#1928) Update protobuf to 3.11.2 (#1928) * Change default optimization level to All (from Basic) (#2745) * change default optimization level to All (from Basic) * fix test * fix c# test * Update numpy to 1.18 (#2758) * Update numpy to 1.18 * Pipeline changes for python 3.8 (#2753) 1. Pipeline changes for python 3.8 2. Fix a regression in setup.py which was just introduced in the previous commit. Please notice, we still haven't made python 3.8 + Windows + CUDA work. * Add basic stacktrace output for posix debug builds. (#2749) * [NupharEP] fix a race condition when multiple sessions running different models concurrently (#2772) * Revert "Change default optimization level to All (from Basic) (#2745)" This reverts commit 56bb503c2f26474b6613bcb2a198691a11dcef38. * Fix typo in error message (#2736) * Rename MKL-DNN to DNNL to fix broken link (#2730) * Fix nightly build version number issue * Pass BUILD_BUILDNUMBER to linux docker * Disable featurizers in python packages * Import more featurizers (#2781) Make kernels non-template. Add input constraint for learnt data. Add min_max_scalar_transformer, robust_scalar_transformer, inputation_marker_transfomer, label_encoder_transformer, missing_dummies_transformer along with tests. Advance Featurizers library commit. * Implement a more stable softmax (#2715) * Implement a more stable SoftMax e^x is represented as infinity if x is large enough, like 100.f. Infinity divided by Infinity is a NAN. Thus, softmax gets a NAN if one or more item are large enough. A math transform as below is leveraged to get a stable softmax: e^xi/(e^x1 + ...e^xn) = e^(xi - max) / (e^(x1 - max) + ... + e^(xn - max)) And for convenience, force max to 0.f if all xi are negative * Contributing: Fix a typo (#2784) * ACL EP GEMM improvements (#2780) When it is posible we use a fully connected layer instead of the gemm implementation. This will let the library use the best implementation based on the input data. * ACL EP convolution improvements (#2774) Added the optimized implementation for depthwise convolution for both ACL v19.02 and ACL 19.05. Also the pointwise convolution seems to be more optimal in the CPU implementation so we opted for that instead. * Add script for release Nuget validation (#2719) * Initial commit * Nits * Disable a test temporarily * Change working directory * Test * Add download python step * Test update * More changes * Fix space issue * Fix * Verify nuget signing * Fix * Spaces * PR feedback * Nit * Fix * Fix * Remove temporary changes * add uint8 support to where op (#2792) * Improve bert optimization script: (#2712) (1) Move input int64=>int32 conversion to embed layer fusion. (2) Output epsilon attribute for LayerNormalization fusion. * add session creation time cost. (#2798) * ML.NET team needs featurizers within a package (#2789) Add auto ml featurizers to Windows, MacOS as well as to GPU packaging-pipelines. * Initialize max of softmax with lowest of float (#2786) * MLAS: update SGEMM threading parameters (#2808) * add interface to copy batch tensors. (#2807) * add interface to copy batch tensors. * onnxruntime * speed up Windows TRT CI (#2811) * don't run cuda tests if building with tensorrt * remove unnecessary build options for win trt ci * refactor win gpu tensorrt ci yml * --numpy_version=1.17 * update * update * azcopy and cuda path * Update test data (#2356) * Add timeseries imputer transformer featurizer kernel (#2813) Make kernels non-template. Add input constraint for learnt data. Fixup tests. Add two more featurizers along with tests. Tests fail. min_max_scalar_transformer robust_scalar_transformer Fix tests serialized stream by prepending version bytes. Add inputation_marker_transfomer and the test. Fix up float/double type designations. Added label_encoder_transformer along with a test. string_throw case is broken at the momement. Fix labelencodertransfomer_test.cc string_throw case Rename maxabsscalertransformer_test.cc Add MissingDummiesTransformer along with the test. Update manifest. Add TimeSeriesImputerTransformer definition, implementation and tests * Fix memory leak in TRT (#2815) * fix memory leak issue * revert EP_FAIL on enueueV2 * Add manifest missing comma * Run static code analyzer on most of our code (#2817) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * update quantization doc (#2783) * update documentation for quantization script * plus some spell corrections * Filter CPU case for IsFloat16Supported (#2802) * update default optimization level + fix gemm_activation fusion (#2791) * update defualt optimization level + fix gemm_activation fusion * fix typo * add unit test and incorporate review comments * fix test comment * Fix dnnl wheel package name (#2823) * Append '-dnnl' to whl package name when --use_dnnl * Update build.py * Update Ubuntu & TensorRT version in README (#2820) Dockerfile.tensorrt is using nvcr.io/nvidia/tensorrt:19.09-py3 as base Image, update Ubuntu and TensorRT version according to https://docs.nvidia.com/deeplearning/sdk/tensorrt-container-release-notes/rel_19-09.html#rel_19-09 * Merge fixes * Add OneHotEncoder and HashOneHotEncoder kernels. (#2830) Add defs and imlementation for OneHotEncoders, adjuist date_time_transformer kernel and test. Add OneHotEncoder kernel test. Add HashOneHotVectorizerTransformer unit test. This does not link due to multiple definitions of functions that are included into header from a CPP file. * Upgrade gtest to the latest version (#2827) WinML would like to update the googletest submodule. They want some newer features (namely GTEST_SKIP to skip tests programmatically and be able to skip entire fixtures easily) and would need to update the submodule version. However, because the new version of code hit a bug in gcc, even though the bug is already fixed in the latest gcc but we're using gcc 4.8.x and it won't get patched for the bug, so we have to do a compromise, change our code a little bit to make it work. The gcc bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=51213 * Add support for int64_t for topk CPU. Fixes github issue #2806. (#2833) * Ignore allocator type in ExecutionProviders allocator map. Make default initialization of OrtMemoryInfo more clearly invalid. (#2768) * Remove allocator type from the key comparison in ExecutionProviders. Remove usage of DummyArena as it's no longer necessary. * Fix x86 tests where arena allocator is disabled. Make initialization of OrtMemoryInfo clearer by adding Invalid enum value. * Make OrtValueNameIdxMap::MaxIdx more intuitive. * Convert ExternalProject Featurizers into git submodule (#2834) Add git submodule for Featurizer library. Update cmake to build for git submodule. * add domain check for nodes + update documentation (#2831) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Fix Linux CUDA nuget packaging pipeline break * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * update optimization doc for BERT related fusions (#2819) * Add bert related transformers to doc * Add execution provider and comment for bert optimizations * Add comment about accuracy impact of approximation * Fix warnings that cause build to fail * MLAS: enable threading for quantized GEMMs (#2844) * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * remove old winmladapter cpp Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * move sequence implementation into ort lib... still commented out... need to turn back on... * begin sequence implementation * make maps and sequences work * fix broken tests * remove dead code * misc cleanup * CR feedback * User/xianz/winml adapter c api (#2869) * wrapper all existing winml adapter apis with API_IMPL to try catch * Return HR or Throw for WinML adapter APIs if failed * undo macro wrapper for two places * Wrap error macros around ort apis, too. * address CR feedback #2 * add more api throw/return macros * Revert changes no longer needed * revert changes to cxx api * format winml lib.ort and winml adapter * remove static pheonix singleton Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * missing use_dml check in winml_adapter_session (#2930) * --use_dnnl flag was mangled in merge (#2931) * use dml macro not wrapping custom registry code (#2934) * Disable LNK4199 winml_dll to enable cuda builds (#2936) * Disable LNK4199 in winml_dll * linkler->linker * LearningModelSessionAPITestGpu.CreateSessionWithCastToFloat16InModel should return DXGI_ERROR_UNSUPPORTED when FP16 not supported (#2937) * Disable LNK4199 in winml_dll * linkler->linker * Need to return DXGI_ERROR_UNSUPPORTED when Model does not support fp16 * Publish build symbols (#2939) * Publish build symbols * Don't upload PDBs for .exe files * Make x86 build (#2943) * fix last remaining size_t/int64_t warnings->errors (#2948) * TensorString, Sequences and Maps use the first allocator, but should use the cpu default allocator. (#2952) * fix tensor string allcoator * clean up default allocator usage for strings in winml lib/api.ort Co-authored-by: Ryan Lai <ryalai96@gmail.com> * Handle tensor shape of zero (#2954) Co-authored-by: Ryan Lai <ryalai96@gmail.com> * CR feedback (#2970) * CR feedback * fix weird formatting on privacy readme * Add 'All rights reserved.' everywhere * readd all rights reserved to winml_provider_factory.h * remove extra space in comment * remove extra whitespace * fixes post master merge * remove winml from nuget gpu pipeline * set IR VERSION on generated_model in rnn_benchmark (#2972) * Fix slice conformance failures (#2908) Co-authored-by: Adrian Tsai <adtsai@microsoft.com> Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Paul McDaniel <paul_mcdaniel@hotmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Dwayne Robinson <fdwr@hotmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com>
2020-02-05 01:12:19 +00:00
[threading(both)]
[marshaling_behavior(agile)]
[dualapipartition(1)]
runtimeclass TensorDouble : ITensor, [contract(MachineLearningContract, 2)]Windows.Foundation.IMemoryBuffer
{
static TensorDouble Create();
static TensorDouble Create(Windows.Foundation.Collections.IIterable<Int64> shape);
static TensorDouble CreateFromArray(Windows.Foundation.Collections.IIterable<Int64> shape, Double[] data);
static TensorDouble CreateFromIterable(Windows.Foundation.Collections.IIterable<Int64> shape, Windows.Foundation.Collections.IIterable<Double> data);
Windows.Foundation.Collections.IVectorView <Double> GetAsVectorView();
[contract(MachineLearningContract, 2)]
{
//! Creates a tensor object, allocates a buffer of size 'shape', and copies all of 'data' into it.
static TensorDouble CreateFromShapeArrayAndDataArray(Int64[] shape, Double[] data);
//! Creates a tensor object, creates a tensor of size 'shape', and uses the data in 'buffer'.
static TensorDouble CreateFromBuffer(Int64[] shape, Windows.Storage.Streams.IBuffer buffer);
}
}
[contract(MachineLearningContract, 1)]
INBOX_ONLY([static_name("Windows.AI.MachineLearning.ITensorStringStatics", 83623324-cf26-4f17-a2d4-20ef8d097d53)])
INBOX_ONLY([interface_name("Windows.AI.MachineLearning.ITensorString", 582335c8-bdb1-4610-bc75-35e9cbf009b7)])
Merge windowsai (winml layering) into master (#2956) * Initial Commit * Merged PR 3985217: add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc (#2346) add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc and violating our OS layering requirements. We linked against onecoreuap_apiset.lib in VB so we will continue doing this, but I am still unsure why not to link against onecore instead since that is where we ship. However, since Sheil is the owner of this code we will wait to discuss with him before changing anything. * Initial changes for layering * more snipping to get core into ort * update build instructions to include --build_shared_lib (#2358) * update build instructions to include --build_shared_lib * fix line breaks * Task 23998197: add winml_lib_core into onnnxruntime.dll (#2368) * Task 23998197: add winml_lib_core into onnnxruntime.dll * PR feedback build break on perf_test * return proper error when the model path isn't found (#2391) * LearningModelSession is cleaned up to use the adapter, and parts of b… (#2382) this is a big PR. we are going to move it up to layer_dev , which is still a L3 so we are still safe to do work there agile. we are going to move this into the L3 so that ryan can start doing intergration testing. we will pause for a full code review and integration test result prior to going into the L2. >>>> raw comments from previous commits >>> * LearningModelSession is cleaned up to use the adapter, and parts of binding are. * moved everything in the winmladapter made it all nano-com using, WRL to construct objects in the ORT side. base interfaces for everythign for winml to call cleaned up a bunch of winml to use the base interfaces. * more pieces * GetData across the abi. * renamed some namepsace cleaned up OrtValue cleaned up Tensor cleaned up custom ops. everything *but* learnignmodel should be clean * make sure it's building. winml.dll is still a monolith. * model moved over. everything builds clean. step ! * weak ref comment * Layer dev paulm (#2408) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * Layer dev paulm (#2414) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * Layer dev paulm (#2423) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * Layer dev paulm (#2424) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * Layer dev paulm (#2425) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Layer dev paulm (#2426) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * fixed map and sequence when passing stl types across the ABI . found a leak in nvidia driver, but skipped it. all winmlapitests pass now * Moved SessionOptions over to the abi * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * Register WinML TraceLogging provider on Onnxruntime.dll (#2455) * Register WinML TraceLogging provider on Onnxruntime.dll * Add ifdef to make sure trace logging provider has telemetry option when LAYERING_DONE * No need for ifdef for TraceLoggingOptionMicrosoftTelemetry * PR feedback * Move etw registration into lotus environment constructor and deresgister in lotus environment destructor * Brianma/cpuwinml (#2466) * allow building winml cpu without dml. * Brianma/breaks (#2469) * fix some more breaks * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * move dml checks out of winml and into the adapter * better error handling * Brianma/fi (#2470) * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * fix bad merge * Checking in a staging checkpoint point so that Ryan can work with me in parrallel * build break. * Brianma/testfails (#2473) * add missing ir version to dictvectorizer-string.onnx * add missing ir version to relu.onnx * add missing ir version to zipmap*onnx * add IR version to manually generated models * remove an unnecessary ifdef dml * Brianma/windowsai fi (#2475) * update dockerfiles/README (#2336) * Make elementwise op run 4 items per thread (#2335) Description: Describe your changes. Make elementwise op run 4 items per thread unroll for loop to leverage ILP remove unnessary N==0 check inside elementwise GPU kernel Motivation and Context Why is this change required? What problem does it solve? It can improve the performance of GPU elementwise ops. ~2% performance gain on popular NLP bert model. If it fixes an open issue, please link to the issue here. * Add CUDA GatherElements kernel (#2310) * Updates * Update test * Update * Updates * nits * PR feedback * Update * Update * PR feedback * PR comments * Update * Fix build * Fix build * Nits * Fix * Layer Normalization Fusion (#2319) basic layer normalization transform * Add FastGelu Cuda Op for Gelu and Add bias fusion (#2293) * Add FastGelu cuda op * Add AddBiasGelu for experiment * Revert "Add AddBiasGelu for experiment" This reverts commit 5c1ee019858c657e6bb75887265cb85675626e5b. * Add bias * Add unit tests * update comment * update script * fix build error * update coding style * update for CR feedback Enable half2 optimization only when cuda arch >= 7.0 * move _Tanh to common.cuh * implement CPU contrib OP Attention (#2333) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanUnusedInitializers. (#2320) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanupUnusedInitializers. This means initializers that have been replaced during graph optimizations are not left in the GraphProto when we save an optimized model. * Handle edge case where a model has an unused initializer with matching graph input by also removing the graph input. * Use non-const iterators in std::find_if calls to make centos build happy. * Nuget pipeline changes (#2305) 1. refactor the pipeline, remove some duplicated code 2. Move Windows_py_GPU_Wheels job to Win-GPU-CUDA10. We'll deprecated the "Win-GPU" pool 3. Delete cpu-nocontribops-esrp-pipeline.yml and cpu-nocontribops-pipeline.yml 4. In Linux nuget jobs, run "make install" before creating the package. So that extra RPAH info will be removed * Cuda Reverse Sequence Op, maping types of same size using same template function. (#2281) * Set ElementType to String type of node metadata, instead of byte[] (#2348) * Set ElementType to String type of node metadata, instead of byte[] * Fix spacing * Introduce PrimitiveType into a Type System along with an integer constant (#2307) Improve perf by avoiding GetType<T>() calls. Introduce MLTypeCallDispatcher to switch on Input Type. Add Tensor IsType<T>() fast method. * Fix/test dim value of 0 handling in a couple of places (#2337) * Update the CUDA Where implementation broadcasting logic to handle a dim with value of 0. Add unit test Also add unit test for unary op with dim value of 0 * Exclude ngraph from Where test with 0 dim. * Openvino EP R3.1 onnxrt server (#2357) * onnxrt server with OVEP * onnxrt server with OVEP * Update Dockerfile.server.openvino * onnxrt server OVEP fix reviews * onnxrt server OVEP fix reviews * Implement cuda nonzero op. (#2056) Implement cuda nonzero op. * Direct use python numpy array's memory if already contiguous. (#2355) * Direct use python numpy array's memory if already contiguous. This could greatly improve performance for session with large input, like big image 1920x1080 fastrcnn, 30~40% speed up could be achieved. * Add test case enforce contiguous/non-contiguos numpy array as inputs. * Add helper to create output to minimize binary size. (#2365) Add ConstEigenTensorMap typedef so we don't unnecessarily const_cast the const input Tensor. * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS (#2369) * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS * update * Add Tracelogging for profiling (#1639) Enabled only if onnxruntime_ENABLE_INSTRUMENT is ON * test bidaf with nuphar for avx target (#2370) increase nuphar test coverage a bit * Fix a bug in TLS refcount that may destabilized CUDA CI (#2374) * update output size calculation for resize (#2366) * change how output size is calculated for resize op * add tests for ver 10 resize * Extend OneHot CPU kernel to support more types (#2311) * Extend OneHot CPU kernel to support input int64_t, depth int32_t, output float * Skip BERT before the test data fix is picked up * Fix bug with Slice. Need to pass in flattened input dimensions so the initial offset into the input is calculated correctly. (#2372) * Add opset 11 version of Split to CUDA ops (#2376) Organize the CUDA ops definitions so all the opset 10 and 11 parts are together (same setup used for CPU ops) * Layer Norm Fusion Fix (#2379) * layer norm fusion fix * Add input shape check in code and unit tests * Fuse Add + Gelu (#2360) Implement the transformer to fuse add + gelu Implement the accurate kernel * Skip layer norm transform (#2350) * skip layer normalization transformer * Another try to stabilize CUDA CI (#2383) The root cause seems to be failure in CUDA dealloc when tear down. cudaFree return code was ignored before, so should the debug check. * fix BUILD.md typo (#2375) build.py: error: argument --config: invalid choice: 'RelWithDebugInfo' (choose from 'Debug', 'MinSizeRel', 'Release', 'RelWithDebInfo') * Fixed compilation with ngraph (#2388) * Fix reuse logic in allocation planner. (#2393) * Fix reuse logic in allocation planner. * PR comments * Add helpful comments * Don't allow reuse across string tensors. * [NupharEP] Multiple optimizations (#2380) Fuse transpose into MatMul Implement Pow and constant scalar simplification Vectorize ReduceMean Improve symbolic shape inference Minor updates for better debugging in fused function name * Avoid using the default logger in the graph lib and optimizers (#2361) 1. Use the session logger if it is available. 2. Don't disable warning 4100 globally. We should fix the warnings instead of disabling it. * Change CUDA implementation of Transpose to support all fixed size tensor types (#2387) * Change CUDA implementation of Transpose to not use a typed kernel so we can support more types with minimum binary size. Add support for 8, 16, 32 and 64 bit types. Add unit tests. Add method so the implementation can be called directly (will be used by CUDA Scan very soon). * Disable TensorRT for MLFloat16 and int8 unit tests. * Address PR comment and add support for calling cublas implementation if type is mlfloat16. * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. (#2398) * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. * [NupharEP] force some low/zero cost ops to be inlined (#2409) * fix cross compile bug (#2415) * Minor optimization: if a node has already been placed, there's no need to find a kernel for it. (#2417) * Add Reshape Fusion (#2395) * Add reshape fusion * Add some comments * update comments * update comment format * update according to feedback * update for recent logger change * fix build error * (1) Support both input and output edges in find path in graphutils (2) Add a test case of only one constant initializer of Concat input. (3) Refactor ReshapeFusion class to allow add more subgraph fusion in the future. * fix error * (1) loose constraint on initializer: non constant is allowed for reshape fusion. (2) Change versions type to vector. (3) Add logging. (4) Return false when multiple output edges matched in FindPath. Add comments. * only allow one direction (input or output) in FindPath * [NupharEP] Update notebook and docker image (#2416) Add BERT squad in Nuphar tutorial Enhance speed comparsion readability * Fix the issue in matmul_add_fusion (#2407) Fix the issue in matmul_add_fusion If Muatmul + Add has shape [K] * [K, N], reset it to [1, K] * [K, N] will make the output shape to [1, N] will also requires a reshape on the output. Fix: just remove the shape reset to not fuse it. Add a negative test case for matmul+add fusion * feat(treeregressor): Update TreeEnsembleRegressor for type support (#2389) Updates the `TreeEnsembleRegressor` to allow for `double`, `float`, `int64`, and `int32` inputs to match the upstream specification. Signed-off-by: Nick Groszewski <nicholas.groszewski@capitalone.com> * onnxrt server documentation update (#2396) * Added support for Pad-2 operator in OpenVINO-EP (#2405) * Add CUDA If operator. (#2377) * Add CUDA If operator. Uses CPU operator for implementation. By adding a CUDA version the inputs/outputs (with the exception of the 'cond' input) stay on GPU, and no other logic is required to avoid a copy to CPU across the control flow node. * Improved documentation for onnxruntime::utils::SwapByteOrderCopy(), added precondition check. * Fix the type constraints on CUDA If operator to exclude strings. (#2431) * add Im2col<uint8_t> (#2438) * Adjust codegen vectorization width from target (#2439) * Adjust codegen vectorization width from target * Add CUDA Scan operator. (#2403) * Add Scan CUDA op. Uses CPU implementation for logic. Added some device specific functors for handling when data needs to be manipulated on a different device. Added ability to override the materialization logic in the OrtValue slicer so DML can plugin their handling. * Fix Windows GPU C API packaging pipeline failure (#2440) Fix Windows GPU C API packaging pipeline failure (#2440) * Correctly handle implicit inputs for fused nodes (#2390) * Correctly handle implicit inputs for fused nodes Previously, nuphar's partitioning function didn't include node's implicit inputs into the inputs list of MetaDef, and hence a crash was triggered in the onnx graph checker. This commit fixed the issue. Furthermore, it also fixed a related issue where we didn't add implicit inputs into graph_inputs_excluding_initializers_ in Graph::SetGraphInputsOutputs. the issue was that graph_inputs_including_initializers_ populated by SetInputs (e.g. called by FunctionImpl::FunctionImpl) may contain implicit inputs which were not of any node's initializers in the graph. Because they were not part of any initializers, these implicit inputs couldn't be visited by going through all nodes' inputs. Consequently, they would *not* be added into graph_inputs_excluding_initializers_. We fixed the issue by first copying the populated graph_inputs_including_initializers_ into graph_inputs_excluding_initalizers_, which then had both initializers and non-initializers as its initial content. Later, we erase initializers from the list. In this way, we can ensure all implicit inputs to remain in graph_inputs_excluding_initializers_. * refined comments and fixed duplicates Address CR by revisiting comments in terms of implicit inputs Also fixed an issue by skipping duplicates while copying inputs from graph_inputs_including_initializers_. * address CR explain why we need to collect nodes' implicit inputs * don't rely on pointer values for iterating std::set Previously, openvino relied on iterating a set of NodeArg pointers to construct inputs and outputs for a fused graph. It could cause non-determinism. The reason was that although iterating std::set by itself is stable, pointer values of NodeArgs may vary. Consequently, we could end up visiting the set's elements in different orders for different runs for the same test, which resulted in constructing inputs (and outputs) with different orders to the fused graph. For example, for the same test, we may have inputs [A, B] in some runs but inputs[B, A] in others. Let's use std::string as the key type to avoid such nondeterminism. This commit also added implicit inputs into meta->inputs while returning the capability from the openvino provider. * Fixed another latent issue in openvino's GetCapability function The issue was that we couldn't simply erase fused_inputs and fused_outputs while iterating the nodes. For example, an output NodeArg may have multiple uses, and it's wrong if we erase it from fused_outputs when we encounter only one of its uses as input. * Remove DeviceAllocatorRegistry class (#2451) Remove DeviceAllocatorRegistry class * CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library. * Parallel Gelu with ParallelFor (#2399) Parallel Gelu to get better performance for Gelu * Clean up build.py (#2446) * Pull the latest image before running docker build * Fuse SkipLayerNorm with Bias (#2453) Fuse SkipLayerNorm with Bias * Allow more than one invocation of CreateEnv in the same process. (#2467) * Allow more than one invocation of CreateEnv in the same process. * Fix centos build * Symbolic shape inference improvements: (#2460) * Symbolic shape inference improvements: - add a mode to guess unknown ops' output rank - add support for GatherND - add support for If - fix a bug in get_int_values when then tensor rank > 1D, by treating it as no sympy data - add symbol to literal merge when ONNX silently merges dims - fix a bug in Concat when input dim is 0 - fix a bug in ConstantOfShape that computed dim is not updated - add support for dynamic shape in ConstantOfShape - fix a bug in Loop output shape that loop iterator dim is not inserted at dim 0 - add support for dynamic padding in Pad - add support for dynamic shape in Reshape - add support for Resize with opset > 10, by treating output dims as dynamic - fix a bug in Slice when starts/ends are dynamic - restrict input model to opset 7 and above - make output model optional to avoid disk write when testing Run model tests for symbolic shape inference Reduce 2GB docker image size of nuphar * add additional test data set for nuget pipeline (#2448) * add SAS token to download internal test data for nuget pipeline * update azure endpoint * fix keyvault download step * fix variable declaration for secret group * fix indentation * fix yaml syntax for variables * fix setting secrets for script * fix env synctax * Fix macos pipeline * attempt to add secrets to windows download data * fix mac and win data download * fix windows data download * update test data set url and location * Revert "Brianma/windowsai fi (#2475)" This reverts commit 5780b864a15513fda4eadbfc2b5345fefe70b5ec. * Add scenario tests (#2457) * Add scenario tests * Remove TODO from model license * Add winml_api test dependency * fix model load test. fi from master changed the constructor (#2483) * make api tests all pass (#2486) * fix bad merge * fix bad model merge * Layer dev paulm (#2492) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * Rename ambiguous header (#2489) * fix one more missing IR version model (#2500) * add missing IR version to 4 more models used by scenario tests (#2501) * Add CLI parameters to test runner, build WinML in ARM and x86 CI (#2479) * Support test parameters through CLI arguments * Add WinML do Windows x86/ARM CI builds * Code style fixes * Update googletest Remove GPUTEST macros everywhere now that GTEST_SKIP is supported * Refactor main.cpp * Build scenario tests without DML * Link scenario tests to DML when it's enabled (#2502) * Layer dev release pipeline (#2488) Adds winml binaries to existing cpu nuget package, and creates new gpu dml nuget package with winml binaries and DML EP. * Layer dev paulm (#2506) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * Remove usage of IOBinding in WinML and use C_API Run method (#2504) * remove usage of iobinding * Change data structure to use vector of Ort::Values * Polish bind input / output * Use C APIrun method * Update providers on evaluate getresults * Remove run and IObinding interface from WinMLAdapter * Remove use of IObinding * bind unbound outputs code moved to learningmodelbinding * clean up unneeded istensor adapter function * Fix comment * Check if session is closed before binding and clearing * PR feedback * Layer dev paulm (#2507) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * Make tests dependend on winml_dll (#2509) * add dml binaries to DirectML package and be more explicit about condition variables (#2520) * re-enable warnings for winml builds and fix the warnings that were hiding (#2526) * turn devmode back on for winml builds * fix some warnings. include protobuf in a way that disables some warnings * undo protobufhelpers changes and just ignore 4100 errors in pb code * attempt to isolate protobufhelpers errors * add template specialization for getting tensor proto data * Layer dev paulm (#2533) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * Spawn child process to run DeviceLostRecovery scenario test (#2530) * Spawn child process to run DeviceLostRecovery scenario test * Layer dev paulm (#2536) ori said yes * add missing namespace to winml_trace_logging_provider in lotusenvironment.h (#2542) * Handle exception thrown from all apis in WinMLAdapter (#2539) * various changes to unblock windowsai ADO build * Fix custom ops scenario tests (#2562) * Do not shutdown protobuf after ort environment gets destroyed. Lazy load lotus environment first time it is needed * comment typo * pr comment about calling phoenix singleton * Make lotus_environment static in winmladapter * Layer dev paulm (#2567) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * making windowsAI azure dev ops work. * code review comments. * revert changes * Cmake and preprocessor fixes that where uncovered by building on agents without DML available via SDK * Layer dev dml delayload (#2580) * Brianma/cpu (#2583) * don't include dml stuff in cpu builds * tests that link the image lib also need the telemetry lib now * Throw Winml_err_invalid_binding if binding gpu resource on cpu device (#2589) * Throw Winml_err_invalid_binding if binding gpu resource on cpu device * PR comments. No need to query executionprovider for is gpu device * User/xianz/ortthrow (#2596) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * User/xianz/ortthrow (#2599) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * add status error message * Remove uwp onsuspending winrt call because logruntimeperf is getting removed (#2630) * User/xianz/dedup telemetry (#2631) * investigate duplication of telemetry in winml and ort * remove winml telemetry events * telemetry executionProviderEvent * remove unneccessary file and refactor code little bit * Revert back TelemetryEvent, which send up ETW event. * merge changes from layer_dev to windowsai (#2638) * Remove underscore from googletest names (#2616) * Fix leaking memory allocator Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24278761 and https://microsoft.visualstudio.com/OS/_workitems/edit/24330198 * Explicitly initialize Ort::Value with nullptr * Cache WinML adapter * bad merge * define private version of dxcore enum that is added in 19H1 SDK. (#2654) * add comment for explaning private definition of dxcore d3d feature level ennum value. (#2672) * do not package directml.pdb for redist packages. (#2676) * Fix leaking operator registry (#2645) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24354916 * User/orilevari/windowsai master merge (#2674) merge resolutions included pulling in telemetry logic that was merged to master and not windowsai and dereferencing InferenceSession::sessionstate now that it is a unique pointer * Delete Ort Allocator in LearningModelBinding (#2653) * Delete OrtAllocator in LearningModelBinding * PR comments to make Ort::Allocator a smart pointer * Small comment change * PR feedback to clean up code * PR feedback on move semantics * Clean up std::move * Fix memory leaks (#2679) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24356109, https://microsoft.visualstudio.com/OS/_workitems/edit/24388361 and https://microsoft.visualstudio.com/OS/_workitems/edit/24388596 * various changes to properly organize and skip GPU tests. For now for No DML builds we will not run GPU tests at all. In the future we should adapt the tests to expect the appropiate errors. (#2695) * Windowsai without fi (#2701) * Disable Attention fusion tests when DISABLE_CONTRIB_OPS is defined (#2529) * Setup java ci (#2528) * Add provision in ORT for session options to be parsed when available via model file (#2449) * Initial commit * Fix gitmodules * Nits * Nits * Updates * Update * More changes * Updates * Update * Some updates * More changes * Update * Update * Merge * Update * Updates * More changes * Update * Fix nits * Updates * Fix warning * Fix build * Add comment * PR feedback * PR feedback * Updates * Updates * Update * More changes * Fix build break * Comment test for now * Updates * Updates * PR feedback * Updates * Nits * Add tests * Fix build * Fix build * Fix build * Fix build break * Fix build * Nits * PR feedback * More change * Expose GetSessionOptions in pybind logic and add unit test for python * Fix build * PR feedback * PR feedback * Revert "Disable thread pool creation when enabled OpenMP (#2485)" (#2535) This reverts commit 7c7d5a149c9ed52eec67304bae5c4b132166a8a1. * Add dynamic shape support in TensorRT execution provider (#2450) * remove onnx-tensorrt submodule * add new onnx-tensorrt submodule (experiment) for trt6 * update engine build for trt6 * update compile and compute for tensorrt6.0 * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * switch to onnx-tensorrt master for TensorRT6' * Update tensorrt_execution_provider.cc * Handle dynamic batch size and add memcpy in TensorRT EP * update test cases * Update tensorrt_execution_provider.cc * update onnx-tensorrt submodule * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.ubuntu_tensorrt * Update run_dockerbuild.sh * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update concat_op_test.cc * Update tensorrt_execution_provider.cc * Upgrade TensorRT to version 6.0.1.5 * Update onnxruntime_providers.cmake * Update CMakeLists.txt * Update reduction_ops_test.cc * Update install_ubuntu.sh * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.tensorrt * Update BUILD.md * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update onnxruntime_providers.cmake * Update install_ubuntu.sh * Update install_ubuntu.sh * Update gemm_test.cc * Update gather_op_test.cc * Update CMakeLists.txt * Removed submodule * update onnx-tensorrt submodule * update header file * Removed submodule * add submodule onnx-tensorrt kevin's branch shape-test' * add debugging code * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * merge master * Removed submodule * update onnx-tensorrt submodule * add more changes for dynamic shapes * Update tensorrt_execution_provider.cc * update for dynamic shape * update dynamic shape processing * fix logger issue * remove submodule onnx-tensorrt * add submodule onnx-tensorrt * add env variable min_subgraph_size * remove redundency * update document * use onnxruntime::make_unique * fix multi-run issue * remove some tests to save CI build time * Add dynamic shape test * Update TensorRT-ExecutionProvider.md * Add example of running Faster R-CNN model on TensorRT EP * Add more details on env variables * update environment variables * Update tensorrt_basic_test.cc * Update model tests * Update tensor_op_test.cc * remove --use_full_protobuf * Update build.py * User/xianz/telemetry (#2458) * enabme telemetry * enable telemetry * set enable telemetry as default * for debugging * remove log and set disable telemetry as default back * delete private file while testing * resolve comment: mainly add license header, rename macro and update docs * rewording in privacy.md * Fix integer overflow in cuda NonMaxSuppression implementation (#2540) * add test case that should pass but fail * fix nms * extract int_max_output_boxes_per_class * Introduce container type runtime checks and other improvements (#2522) Rework TensorSeq in a manner consistent with Tensor and SparseTensor in terms of type system setup. Reduce templating. Introduce helpers to ensure the same data type. Make OrtValue __dtor not virtual. Introduce ContainerChecker * Fix C API tests for centos and mac (#2544) * change c++14 to c++11 * add ld lib path for centos * enable csharp tests on macos * fix C API test on MacOS + fix manylinux dotnet install * fix manylinux dotnet install * fix lib link * Add back executable bit to build.py * Fix a bug handling negative begin pad values in Pad op (#2550) * Fix bug in Pad op * Update * DNNL CMAKE update (#2548) * Fix android build (#2558) * Update win-x86-ci.yml (#2557) Fix build pipeline break * Re-enable Windows C# tests (#2564) * disable onnx_test_runner -x invocations for dnnl (#2568) * Allow sequence length to be symbolic (#2559) * setup java ci mac (#2570) * make layernorm fusion to support opset 11 (#2545) * Fix a warning found in the latest VS release * Add more check on SkipLayerNorm and BiasGelu fusion (#2574) * Fix file not found error during docker build. (#2569) * Add ConvTranspose1D (#2578) * Ryanunderhill/packagename test (#2582) * [Nuphar EP] fixes for some object detection models (#2581) Update notebook tutorial with multi-threaded int8 GEMM from #2517 * EmbedLayerNormalization Fusion Improvement (#2553) Embedding layer norm fusion improvements - add more checks * Update version (#2584) * Temporarily exclude vgg19 test from Python backend test 1. temporarily exclude vgg19 test which comsumes too much memory, run out of memory on Upsquared device. Single test pass for vgg19, need furture investigation (#2588) 2. Update docker file to decrease the docker image size * Update docs for Android NNAPI EP (#2586) * Fix lto bug for protobuf and ubuntu * add path to build dir before test run (#2590) * Add missig env variables for mac pipeline test (#2595) * Fixed an issue in updating realized dims (#2597) when we update realized dims for scan's output, the sliced axis also needs to be inclusive, i.e. we should check with "dim >= insert_inclusive_axis", because the offset in the symbols are based on Scan sugraph. Otherwise, we would end up with shape mismatch later. * Java API for onnxruntime (#2215) * Add support for opset 11 in reshape fusion (#2592) Support opset verion 11 in reshape fusion * Rename automl python tools folder to featurizer_ops. (#2593) * Support opset 11 subgraph of Squad model in Embed Layer Normalization (#2605) Support opset 11 Squad model that is exported from PyTorch nightly. The embed layer uses Range op which is missed in the transformer. * symbolic shape inference: fix warnings in GPT-2 model (#2608) And revise nuphar perf test on BERT squad * Dump subgraph ID and fused graph ID (#2607) * Dump subgraph ID and fused graph ID Dump subgraph ID and fused graph ID for better debugging * Remove local static fused_count added a field global_fused_count_ to NupharExecutionProvider class * EmbedLayerNormalization Fusion For Dynamic Squad Model Opset 10 (#2613) Support subgraph of SQuAD model exported from pytorch with dynamic input axes * Allow providers to be set for InferenceSession at construction (#2606) * Remove unnecessary parameter in some places in GatherElements implementation (#2612) * Remove unnecessary parameter in some places * Update * Update * Make sure fenced tensor could not reuse other tensor. (#2561) Fix random error caused by this. * Improve Embed Layer Norm Fusion for SQuAD with static input shape (#2621) * fix float16 comparison in initializer (#2629) * epsilon attribute for layernormalization fusion (#2639) * removed unnecessary batch file and fix path (#2640) * Add shape inference to ConvTransposeWithDynamicPads schema (#2632) * Improve cuda expand() opeator's performance. (#2624) * Cuda pad optimize when no padding is needed. (#2625) * Shortcut cuda Pad() when no padding is needed. * Optimize cuda scatter() on 2D compatible. (#2628) * Optimize cuda scatter() on 2D compatible. * Add some comments. * fix build error for ARM (#2648) * Improve performance of resize() in Nearest mode (#2626) Special treatment for 2D, check same size as input image. And in 2d kernel, template use_expolation. * Fix memory exception in Layer Norm Fusion (#2644) * Windows CI changes(#2650) * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Revert "Windowsai without fi (#2701)" This reverts commit 285d4c85ff5c4e265f963208170304ef3461e684. * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Deref unique pointer for session_state * send shutdown event when dll is unloaded and EvaluationStop, SessionC… (#2704) * send shutdown event when dll is unloaded and EvaluationStop, SessionCreationStart Events. * Add EvalutationStart Event * add comment * use correct type for for loop (#2755) * ARM CI (#2759) * Set ARM agent pool * Set CMake generator to VS 2019 in ARM * Use system-wide CMake instead of custom version Our custom version is too old for VS 2019 * Use DML and build shared lib in ARM CI * Restore nuget packages in ARM CI * Disable DML * Refactor ARM debug/release builds * Use system packaged Python version * Remove hardcoded Python path * Downgrade Python to 3.7 for build * Remove explicit CMake path * Fix invalid JSON in cgmanifest.json (#2760) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * Remove usage of WHOLEARCHIVE in WinML CMake and add WinMLAdapterFactory (#2726) * Remove usage of WHOLEARCHIVE in WinMLAdapter CMake and add WinMLAdapterFactory * PR feedback, no need for dll(export) since using def file * PR comments * Small comment in gen_def.py * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * Move winml_provider_factory.h to proper location (#2801) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Filter CPU case for IsFloat16Supported (#2802) * Merge fixes * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * Fix warnings that cause build to fail * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * Make DML include path global (#2882) * Make DML include path global * Add generated cppwinrt headers to winml_lib_common * Integrate changes to WindowsAI to make ADO Build (#2886) * Revert "CMake cross-generator fixes (#2790)" This reverts commit dbe7d97fa1ab155f1309bced87199527e8f35bd2. * add additional suppress warning in onnx_proto * ignore /wd4996 warning * DML execution provider fixes * Revert "Revert "CMake cross-generator fixes (#2790)"" This reverts commit 1ae7b4bcbc02edc881ad28685da98e095dfceb17. * Update func signature of custom op function overloads * common devicehelpers fixes * Add pch.h for winml_lib_common * re-add winml_lib_common_dir/inc to include path for winml_adapter * User/orilevari/dml redist shared folder (#2890) * move dml nuget package directory up one level to make it shared between build flavors * Merge conflict fix * Revert "Merge conflict fix" This reverts commit 142fa72cf9ce4344ad717b50b7ea2b8582aadc7c. * Revert "Merge remote-tracking branch 'origin/master' into windowsai" This reverts commit 6e2126d46e5e5f564d65da37dd4f70c93dd81165, reversing changes made to b3f5583dc9249834b947c8ea905f6a98060d5bd6. * Make winml_test_common free of test macros (#2902) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Don't generate files outside the build root (#2914) * Don't generate files outside the build root * Add onnxruntime_EXTERNAL_DEPENDENCIES to WinML * Add DML depedency on RESTORE_PACKAGES * User/orilevari/fix yaml merge bugs (#2918) * Add winml test source parameter into cmake function (#2919) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Add arguments to unittest cmake functions * remove comment * Revert "Revert "Merge remote-tracking branch 'origin/master' into windowsai"" This reverts commit ade5abe72a4234fdbc3623093c61c02c6b0bdc26. * Fix breaks from merge with ORT master * Brianma/linux (#2917) * don't include windows.h in cross-plat header * add default case for switch statement * signed/unsigned mismatch fix Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> * User/sheilk/winml adapter c api (#2891) * Create winml adapter c api * fix build * make it build * move adapter into onnxruntime core/session * entry point not exported * minor changes * make model metadata work * make tests pass * implement all the model reflection apis on the adapter c abi * update the new ort interface to create a lotus ennvironment with a logging sink * start adding ort env * move all winml code into adapter folder/lib to isolate it * ensure a single logging manager at a time * start refactoring session * refactor session creation interface * add cpu and dml session option methods to adapter * finish session init * stub out interfaces in ort lib to perform similar mechanics of iinference session * enable profiling, and enable schema override * update session register graph transformers * turn back on custom registry for custom ops * Add sync api * add last c api stubs * should build... but all feature values are broken since this is in flight to moving all implementation details into ivalue * remove ep adapter header * Implement DML execution provider functions from adapter (#2846) * Implement DML execution provider functions from adapter * Use functions in OnnxruntimeEngine.cpp * make map/sequence type_infos freeable, and start implementing ivalue * make it build again * implement value methods * implement remaining methods * remove com adapter abi * check dml session * cache the allocator on ivalue * check if resource is cpu/gpu when access its mutable data * update tensor * mismatched parentheses * fix tensor base and binding obj * it evaluates tensors! sometimes... * minor fixes * enable gpu evals * wrapper all existing winml adapter apis with API_IMPL to try catch (#2854) * update winml... tensor strings are broken, need to template tensorbase to do different things for strings * make tensor strings work with 2 copies in/2 copies out * Fix tensor string and allocator bug * make maps work again... needs some fixes still * Make it build! * enable map inputs * map outputs * unbound outputs for sequences and maps * User/xianz/merge windowsai (#2883) * Packaging pipeline changes for VS 2019 (#2711) * Tiny fix to codegen * Simplify cache implementation and avoid static variables that may carry over between models * Extend DML kernels (#2641) * Additional DML operators * Check unsupported attributes and inputs * Address PR comments * Add kernel capability function used for partitioning, and re-enable stride-based int64 support based on value range * Fix test failures * Build fix * PR comments * Update Nuphar tutorial notebook (#2721) 1. Reflect int8 GEMV improvements for multi-threading from #2696 2. Add notes on multi-threading control using OpenMP 3. Add samples of running multi-isa AOT, and show int8 GEMM differences between AVX and AVX2 4. Add rnn_benchmark example to resolve #1993 * Add schema for new Qops (#2611) * Add schema for new Qops * adding shape inference + qlinearaveragepool * plus review comments * plus review comments * updates per review comments * plus review comments * [server] Add supposed for model_name and model_version as cli parameter (#2708) * remove 64bit warning message from python validation. (#2727) * MLAS: ARM64 build fix (#2734) fix bad usage of vreinterpret to cast vector element types * Fix broken python docs links (#2740) * Fix build on Mac OS (#2731) mac os ld doesn't support --while-archive, correct option is -all_load * fix ngraph wheel (#2737) * fix ngraph wheel 1.1.0 onnxruntime_ngraph wheel doesn't work * remove libdnnl.so in nGraph Libs * make it easy to compare * Split onnxruntime server to a separated folder (#2744) * Fix build for Python 3.8 (#2747) * Fix build for Python 3.8 * Update protobuf to 3.11.2 (#1928) Update protobuf to 3.11.2 (#1928) * Change default optimization level to All (from Basic) (#2745) * change default optimization level to All (from Basic) * fix test * fix c# test * Update numpy to 1.18 (#2758) * Update numpy to 1.18 * Pipeline changes for python 3.8 (#2753) 1. Pipeline changes for python 3.8 2. Fix a regression in setup.py which was just introduced in the previous commit. Please notice, we still haven't made python 3.8 + Windows + CUDA work. * Add basic stacktrace output for posix debug builds. (#2749) * [NupharEP] fix a race condition when multiple sessions running different models concurrently (#2772) * Revert "Change default optimization level to All (from Basic) (#2745)" This reverts commit 56bb503c2f26474b6613bcb2a198691a11dcef38. * Fix typo in error message (#2736) * Rename MKL-DNN to DNNL to fix broken link (#2730) * Fix nightly build version number issue * Pass BUILD_BUILDNUMBER to linux docker * Disable featurizers in python packages * Import more featurizers (#2781) Make kernels non-template. Add input constraint for learnt data. Add min_max_scalar_transformer, robust_scalar_transformer, inputation_marker_transfomer, label_encoder_transformer, missing_dummies_transformer along with tests. Advance Featurizers library commit. * Implement a more stable softmax (#2715) * Implement a more stable SoftMax e^x is represented as infinity if x is large enough, like 100.f. Infinity divided by Infinity is a NAN. Thus, softmax gets a NAN if one or more item are large enough. A math transform as below is leveraged to get a stable softmax: e^xi/(e^x1 + ...e^xn) = e^(xi - max) / (e^(x1 - max) + ... + e^(xn - max)) And for convenience, force max to 0.f if all xi are negative * Contributing: Fix a typo (#2784) * ACL EP GEMM improvements (#2780) When it is posible we use a fully connected layer instead of the gemm implementation. This will let the library use the best implementation based on the input data. * ACL EP convolution improvements (#2774) Added the optimized implementation for depthwise convolution for both ACL v19.02 and ACL 19.05. Also the pointwise convolution seems to be more optimal in the CPU implementation so we opted for that instead. * Add script for release Nuget validation (#2719) * Initial commit * Nits * Disable a test temporarily * Change working directory * Test * Add download python step * Test update * More changes * Fix space issue * Fix * Verify nuget signing * Fix * Spaces * PR feedback * Nit * Fix * Fix * Remove temporary changes * add uint8 support to where op (#2792) * Improve bert optimization script: (#2712) (1) Move input int64=>int32 conversion to embed layer fusion. (2) Output epsilon attribute for LayerNormalization fusion. * add session creation time cost. (#2798) * ML.NET team needs featurizers within a package (#2789) Add auto ml featurizers to Windows, MacOS as well as to GPU packaging-pipelines. * Initialize max of softmax with lowest of float (#2786) * MLAS: update SGEMM threading parameters (#2808) * add interface to copy batch tensors. (#2807) * add interface to copy batch tensors. * onnxruntime * speed up Windows TRT CI (#2811) * don't run cuda tests if building with tensorrt * remove unnecessary build options for win trt ci * refactor win gpu tensorrt ci yml * --numpy_version=1.17 * update * update * azcopy and cuda path * Update test data (#2356) * Add timeseries imputer transformer featurizer kernel (#2813) Make kernels non-template. Add input constraint for learnt data. Fixup tests. Add two more featurizers along with tests. Tests fail. min_max_scalar_transformer robust_scalar_transformer Fix tests serialized stream by prepending version bytes. Add inputation_marker_transfomer and the test. Fix up float/double type designations. Added label_encoder_transformer along with a test. string_throw case is broken at the momement. Fix labelencodertransfomer_test.cc string_throw case Rename maxabsscalertransformer_test.cc Add MissingDummiesTransformer along with the test. Update manifest. Add TimeSeriesImputerTransformer definition, implementation and tests * Fix memory leak in TRT (#2815) * fix memory leak issue * revert EP_FAIL on enueueV2 * Add manifest missing comma * Run static code analyzer on most of our code (#2817) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * update quantization doc (#2783) * update documentation for quantization script * plus some spell corrections * Filter CPU case for IsFloat16Supported (#2802) * update default optimization level + fix gemm_activation fusion (#2791) * update defualt optimization level + fix gemm_activation fusion * fix typo * add unit test and incorporate review comments * fix test comment * Fix dnnl wheel package name (#2823) * Append '-dnnl' to whl package name when --use_dnnl * Update build.py * Update Ubuntu & TensorRT version in README (#2820) Dockerfile.tensorrt is using nvcr.io/nvidia/tensorrt:19.09-py3 as base Image, update Ubuntu and TensorRT version according to https://docs.nvidia.com/deeplearning/sdk/tensorrt-container-release-notes/rel_19-09.html#rel_19-09 * Merge fixes * Add OneHotEncoder and HashOneHotEncoder kernels. (#2830) Add defs and imlementation for OneHotEncoders, adjuist date_time_transformer kernel and test. Add OneHotEncoder kernel test. Add HashOneHotVectorizerTransformer unit test. This does not link due to multiple definitions of functions that are included into header from a CPP file. * Upgrade gtest to the latest version (#2827) WinML would like to update the googletest submodule. They want some newer features (namely GTEST_SKIP to skip tests programmatically and be able to skip entire fixtures easily) and would need to update the submodule version. However, because the new version of code hit a bug in gcc, even though the bug is already fixed in the latest gcc but we're using gcc 4.8.x and it won't get patched for the bug, so we have to do a compromise, change our code a little bit to make it work. The gcc bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=51213 * Add support for int64_t for topk CPU. Fixes github issue #2806. (#2833) * Ignore allocator type in ExecutionProviders allocator map. Make default initialization of OrtMemoryInfo more clearly invalid. (#2768) * Remove allocator type from the key comparison in ExecutionProviders. Remove usage of DummyArena as it's no longer necessary. * Fix x86 tests where arena allocator is disabled. Make initialization of OrtMemoryInfo clearer by adding Invalid enum value. * Make OrtValueNameIdxMap::MaxIdx more intuitive. * Convert ExternalProject Featurizers into git submodule (#2834) Add git submodule for Featurizer library. Update cmake to build for git submodule. * add domain check for nodes + update documentation (#2831) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Fix Linux CUDA nuget packaging pipeline break * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * update optimization doc for BERT related fusions (#2819) * Add bert related transformers to doc * Add execution provider and comment for bert optimizations * Add comment about accuracy impact of approximation * Fix warnings that cause build to fail * MLAS: enable threading for quantized GEMMs (#2844) * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * remove old winmladapter cpp Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * move sequence implementation into ort lib... still commented out... need to turn back on... * begin sequence implementation * make maps and sequences work * fix broken tests * remove dead code * misc cleanup * CR feedback * User/xianz/winml adapter c api (#2869) * wrapper all existing winml adapter apis with API_IMPL to try catch * Return HR or Throw for WinML adapter APIs if failed * undo macro wrapper for two places * Wrap error macros around ort apis, too. * address CR feedback #2 * add more api throw/return macros * Revert changes no longer needed * revert changes to cxx api * format winml lib.ort and winml adapter * remove static pheonix singleton Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * missing use_dml check in winml_adapter_session (#2930) * --use_dnnl flag was mangled in merge (#2931) * use dml macro not wrapping custom registry code (#2934) * Disable LNK4199 winml_dll to enable cuda builds (#2936) * Disable LNK4199 in winml_dll * linkler->linker * LearningModelSessionAPITestGpu.CreateSessionWithCastToFloat16InModel should return DXGI_ERROR_UNSUPPORTED when FP16 not supported (#2937) * Disable LNK4199 in winml_dll * linkler->linker * Need to return DXGI_ERROR_UNSUPPORTED when Model does not support fp16 * Publish build symbols (#2939) * Publish build symbols * Don't upload PDBs for .exe files * Make x86 build (#2943) * fix last remaining size_t/int64_t warnings->errors (#2948) * TensorString, Sequences and Maps use the first allocator, but should use the cpu default allocator. (#2952) * fix tensor string allcoator * clean up default allocator usage for strings in winml lib/api.ort Co-authored-by: Ryan Lai <ryalai96@gmail.com> * Handle tensor shape of zero (#2954) Co-authored-by: Ryan Lai <ryalai96@gmail.com> * CR feedback (#2970) * CR feedback * fix weird formatting on privacy readme * Add 'All rights reserved.' everywhere * readd all rights reserved to winml_provider_factory.h * remove extra space in comment * remove extra whitespace * fixes post master merge * remove winml from nuget gpu pipeline * set IR VERSION on generated_model in rnn_benchmark (#2972) * Fix slice conformance failures (#2908) Co-authored-by: Adrian Tsai <adtsai@microsoft.com> Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Paul McDaniel <paul_mcdaniel@hotmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Dwayne Robinson <fdwr@hotmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com>
2020-02-05 01:12:19 +00:00
[threading(both)]
[marshaling_behavior(agile)]
[dualapipartition(1)]
runtimeclass TensorString : ITensor, [contract(MachineLearningContract, 2)]Windows.Foundation.IMemoryBuffer
{
static TensorString Create();
static TensorString Create(Windows.Foundation.Collections.IIterable<Int64> shape);
static TensorString CreateFromArray(Windows.Foundation.Collections.IIterable<Int64> shape, String[] data);
static TensorString CreateFromIterable(Windows.Foundation.Collections.IIterable<Int64> shape, Windows.Foundation.Collections.IIterable<String> data);
Windows.Foundation.Collections.IVectorView <String> GetAsVectorView();
[contract(MachineLearningContract, 2)]
{
//! Creates a tensor object, allocates a buffer of size 'shape', and copies all of 'data' into it.
static TensorString CreateFromShapeArrayAndDataArray(Int64[] shape, String[] data);
}
}
[contract(MachineLearningContract, 1)]
INBOX_ONLY([static_name("Windows.AI.MachineLearning.IImageFeatureValueStatics", 1bc317fd-23cb-4610-b085-c8e1c87ebaa0)])
INBOX_ONLY([interface_name("Windows.AI.MachineLearning.IImageFeatureValue", f0414fd9-c9aa-4405-b7fb-94f87c8a3037)])
Merge windowsai (winml layering) into master (#2956) * Initial Commit * Merged PR 3985217: add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc (#2346) add onecoreuap_apiset.lib in order to avoid linking against kernel32.lib etc and violating our OS layering requirements. We linked against onecoreuap_apiset.lib in VB so we will continue doing this, but I am still unsure why not to link against onecore instead since that is where we ship. However, since Sheil is the owner of this code we will wait to discuss with him before changing anything. * Initial changes for layering * more snipping to get core into ort * update build instructions to include --build_shared_lib (#2358) * update build instructions to include --build_shared_lib * fix line breaks * Task 23998197: add winml_lib_core into onnnxruntime.dll (#2368) * Task 23998197: add winml_lib_core into onnnxruntime.dll * PR feedback build break on perf_test * return proper error when the model path isn't found (#2391) * LearningModelSession is cleaned up to use the adapter, and parts of b… (#2382) this is a big PR. we are going to move it up to layer_dev , which is still a L3 so we are still safe to do work there agile. we are going to move this into the L3 so that ryan can start doing intergration testing. we will pause for a full code review and integration test result prior to going into the L2. >>>> raw comments from previous commits >>> * LearningModelSession is cleaned up to use the adapter, and parts of binding are. * moved everything in the winmladapter made it all nano-com using, WRL to construct objects in the ORT side. base interfaces for everythign for winml to call cleaned up a bunch of winml to use the base interfaces. * more pieces * GetData across the abi. * renamed some namepsace cleaned up OrtValue cleaned up Tensor cleaned up custom ops. everything *but* learnignmodel should be clean * make sure it's building. winml.dll is still a monolith. * model moved over. everything builds clean. step ! * weak ref comment * Layer dev paulm (#2408) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * Layer dev paulm (#2414) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * Layer dev paulm (#2423) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * Layer dev paulm (#2424) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * Layer dev paulm (#2425) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Layer dev paulm (#2426) * model moved over. everything builds clean. step ! * weak ref comment * added a wrapper for RoGetActivationFactory to hook back into winml for creating winml objects. fixes model load. * fixed some lifetime management. fixed the debug build. squeezenet passes using winmlrunner for CPU and GPU * PR feedback. * couple of fixes and coded getmutabledata() * fixed 2 more heap corruptions * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * fixed map and sequence when passing stl types across the ABI . found a leak in nvidia driver, but skipped it. all winmlapitests pass now * Moved SessionOptions over to the abi * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * Register WinML TraceLogging provider on Onnxruntime.dll (#2455) * Register WinML TraceLogging provider on Onnxruntime.dll * Add ifdef to make sure trace logging provider has telemetry option when LAYERING_DONE * No need for ifdef for TraceLoggingOptionMicrosoftTelemetry * PR feedback * Move etw registration into lotus environment constructor and deresgister in lotus environment destructor * Brianma/cpuwinml (#2466) * allow building winml cpu without dml. * Brianma/breaks (#2469) * fix some more breaks * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * move dml checks out of winml and into the adapter * better error handling * Brianma/fi (#2470) * learning model doesn't need lotusEnvironment and CPU shouldn't include dmlEP headers * User/xianz/win ml telemetry (#2410) * add option to enable winml telemetry * add option to enable winml telemetry * clean logs while developping * clean the log of GUID * compile onnxruntime_common with winml telemetry * use option for use_telemetry * rename option winml_use_telemetry to onnxruntime_use_telemetry * little change * Add opset and IR check when loading model (#2413) * Add opset and IR check. * Add test case for future opsets. https://github.com/microsoft/onnxruntime/issues/2371 * WinML CI (#2412) * Pass flags to build/test WinML in CI * Add initial CMake config for unit tests in WinML * Set winml_unittests standard to C++17 * Add WinML API tests and port them to googletest * Install WinML test collateral * Add LearningModelSessionAPITests ported to googletest * Fix WinML test files encoding * Add GPU tests * Add parameterized test, skip GPU tests * Enable precompiled header * Remove unused code and collateral * Remove brand images * Add dllload.cpp * Remove images not used in API tests * Add LICENSE.md to image collaterals * Add models with licenses * Remove FNS Candy tests * Add API test models * Add ModelInSubdirectory * Install collaterals post-build with copy_if_different, split common lib * fix warnings * Link to gtest_main * fix bad merge * Checking in a staging checkpoint point so that Ryan can work with me in parrallel * build break. * Brianma/testfails (#2473) * add missing ir version to dictvectorizer-string.onnx * add missing ir version to relu.onnx * add missing ir version to zipmap*onnx * add IR version to manually generated models * remove an unnecessary ifdef dml * Brianma/windowsai fi (#2475) * update dockerfiles/README (#2336) * Make elementwise op run 4 items per thread (#2335) Description: Describe your changes. Make elementwise op run 4 items per thread unroll for loop to leverage ILP remove unnessary N==0 check inside elementwise GPU kernel Motivation and Context Why is this change required? What problem does it solve? It can improve the performance of GPU elementwise ops. ~2% performance gain on popular NLP bert model. If it fixes an open issue, please link to the issue here. * Add CUDA GatherElements kernel (#2310) * Updates * Update test * Update * Updates * nits * PR feedback * Update * Update * PR feedback * PR comments * Update * Fix build * Fix build * Nits * Fix * Layer Normalization Fusion (#2319) basic layer normalization transform * Add FastGelu Cuda Op for Gelu and Add bias fusion (#2293) * Add FastGelu cuda op * Add AddBiasGelu for experiment * Revert "Add AddBiasGelu for experiment" This reverts commit 5c1ee019858c657e6bb75887265cb85675626e5b. * Add bias * Add unit tests * update comment * update script * fix build error * update coding style * update for CR feedback Enable half2 optimization only when cuda arch >= 7.0 * move _Tanh to common.cuh * implement CPU contrib OP Attention (#2333) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanUnusedInitializers. (#2320) * Remove unused initializer from GraphProto as well as name_to_initial_tensor_ in CleanupUnusedInitializers. This means initializers that have been replaced during graph optimizations are not left in the GraphProto when we save an optimized model. * Handle edge case where a model has an unused initializer with matching graph input by also removing the graph input. * Use non-const iterators in std::find_if calls to make centos build happy. * Nuget pipeline changes (#2305) 1. refactor the pipeline, remove some duplicated code 2. Move Windows_py_GPU_Wheels job to Win-GPU-CUDA10. We'll deprecated the "Win-GPU" pool 3. Delete cpu-nocontribops-esrp-pipeline.yml and cpu-nocontribops-pipeline.yml 4. In Linux nuget jobs, run "make install" before creating the package. So that extra RPAH info will be removed * Cuda Reverse Sequence Op, maping types of same size using same template function. (#2281) * Set ElementType to String type of node metadata, instead of byte[] (#2348) * Set ElementType to String type of node metadata, instead of byte[] * Fix spacing * Introduce PrimitiveType into a Type System along with an integer constant (#2307) Improve perf by avoiding GetType<T>() calls. Introduce MLTypeCallDispatcher to switch on Input Type. Add Tensor IsType<T>() fast method. * Fix/test dim value of 0 handling in a couple of places (#2337) * Update the CUDA Where implementation broadcasting logic to handle a dim with value of 0. Add unit test Also add unit test for unary op with dim value of 0 * Exclude ngraph from Where test with 0 dim. * Openvino EP R3.1 onnxrt server (#2357) * onnxrt server with OVEP * onnxrt server with OVEP * Update Dockerfile.server.openvino * onnxrt server OVEP fix reviews * onnxrt server OVEP fix reviews * Implement cuda nonzero op. (#2056) Implement cuda nonzero op. * Direct use python numpy array's memory if already contiguous. (#2355) * Direct use python numpy array's memory if already contiguous. This could greatly improve performance for session with large input, like big image 1920x1080 fastrcnn, 30~40% speed up could be achieved. * Add test case enforce contiguous/non-contiguos numpy array as inputs. * Add helper to create output to minimize binary size. (#2365) Add ConstEigenTensorMap typedef so we don't unnecessarily const_cast the const input Tensor. * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS (#2369) * fix builds enabling onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS * update * Add Tracelogging for profiling (#1639) Enabled only if onnxruntime_ENABLE_INSTRUMENT is ON * test bidaf with nuphar for avx target (#2370) increase nuphar test coverage a bit * Fix a bug in TLS refcount that may destabilized CUDA CI (#2374) * update output size calculation for resize (#2366) * change how output size is calculated for resize op * add tests for ver 10 resize * Extend OneHot CPU kernel to support more types (#2311) * Extend OneHot CPU kernel to support input int64_t, depth int32_t, output float * Skip BERT before the test data fix is picked up * Fix bug with Slice. Need to pass in flattened input dimensions so the initial offset into the input is calculated correctly. (#2372) * Add opset 11 version of Split to CUDA ops (#2376) Organize the CUDA ops definitions so all the opset 10 and 11 parts are together (same setup used for CPU ops) * Layer Norm Fusion Fix (#2379) * layer norm fusion fix * Add input shape check in code and unit tests * Fuse Add + Gelu (#2360) Implement the transformer to fuse add + gelu Implement the accurate kernel * Skip layer norm transform (#2350) * skip layer normalization transformer * Another try to stabilize CUDA CI (#2383) The root cause seems to be failure in CUDA dealloc when tear down. cudaFree return code was ignored before, so should the debug check. * fix BUILD.md typo (#2375) build.py: error: argument --config: invalid choice: 'RelWithDebugInfo' (choose from 'Debug', 'MinSizeRel', 'Release', 'RelWithDebInfo') * Fixed compilation with ngraph (#2388) * Fix reuse logic in allocation planner. (#2393) * Fix reuse logic in allocation planner. * PR comments * Add helpful comments * Don't allow reuse across string tensors. * [NupharEP] Multiple optimizations (#2380) Fuse transpose into MatMul Implement Pow and constant scalar simplification Vectorize ReduceMean Improve symbolic shape inference Minor updates for better debugging in fused function name * Avoid using the default logger in the graph lib and optimizers (#2361) 1. Use the session logger if it is available. 2. Don't disable warning 4100 globally. We should fix the warnings instead of disabling it. * Change CUDA implementation of Transpose to support all fixed size tensor types (#2387) * Change CUDA implementation of Transpose to not use a typed kernel so we can support more types with minimum binary size. Add support for 8, 16, 32 and 64 bit types. Add unit tests. Add method so the implementation can be called directly (will be used by CUDA Scan very soon). * Disable TensorRT for MLFloat16 and int8 unit tests. * Address PR comment and add support for calling cublas implementation if type is mlfloat16. * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. (#2398) * Add opset 11 versions of the existing CUDA operators that had negative axis support explicitly added. * [NupharEP] force some low/zero cost ops to be inlined (#2409) * fix cross compile bug (#2415) * Minor optimization: if a node has already been placed, there's no need to find a kernel for it. (#2417) * Add Reshape Fusion (#2395) * Add reshape fusion * Add some comments * update comments * update comment format * update according to feedback * update for recent logger change * fix build error * (1) Support both input and output edges in find path in graphutils (2) Add a test case of only one constant initializer of Concat input. (3) Refactor ReshapeFusion class to allow add more subgraph fusion in the future. * fix error * (1) loose constraint on initializer: non constant is allowed for reshape fusion. (2) Change versions type to vector. (3) Add logging. (4) Return false when multiple output edges matched in FindPath. Add comments. * only allow one direction (input or output) in FindPath * [NupharEP] Update notebook and docker image (#2416) Add BERT squad in Nuphar tutorial Enhance speed comparsion readability * Fix the issue in matmul_add_fusion (#2407) Fix the issue in matmul_add_fusion If Muatmul + Add has shape [K] * [K, N], reset it to [1, K] * [K, N] will make the output shape to [1, N] will also requires a reshape on the output. Fix: just remove the shape reset to not fuse it. Add a negative test case for matmul+add fusion * feat(treeregressor): Update TreeEnsembleRegressor for type support (#2389) Updates the `TreeEnsembleRegressor` to allow for `double`, `float`, `int64`, and `int32` inputs to match the upstream specification. Signed-off-by: Nick Groszewski <nicholas.groszewski@capitalone.com> * onnxrt server documentation update (#2396) * Added support for Pad-2 operator in OpenVINO-EP (#2405) * Add CUDA If operator. (#2377) * Add CUDA If operator. Uses CPU operator for implementation. By adding a CUDA version the inputs/outputs (with the exception of the 'cond' input) stay on GPU, and no other logic is required to avoid a copy to CPU across the control flow node. * Improved documentation for onnxruntime::utils::SwapByteOrderCopy(), added precondition check. * Fix the type constraints on CUDA If operator to exclude strings. (#2431) * add Im2col<uint8_t> (#2438) * Adjust codegen vectorization width from target (#2439) * Adjust codegen vectorization width from target * Add CUDA Scan operator. (#2403) * Add Scan CUDA op. Uses CPU implementation for logic. Added some device specific functors for handling when data needs to be manipulated on a different device. Added ability to override the materialization logic in the OrtValue slicer so DML can plugin their handling. * Fix Windows GPU C API packaging pipeline failure (#2440) Fix Windows GPU C API packaging pipeline failure (#2440) * Correctly handle implicit inputs for fused nodes (#2390) * Correctly handle implicit inputs for fused nodes Previously, nuphar's partitioning function didn't include node's implicit inputs into the inputs list of MetaDef, and hence a crash was triggered in the onnx graph checker. This commit fixed the issue. Furthermore, it also fixed a related issue where we didn't add implicit inputs into graph_inputs_excluding_initializers_ in Graph::SetGraphInputsOutputs. the issue was that graph_inputs_including_initializers_ populated by SetInputs (e.g. called by FunctionImpl::FunctionImpl) may contain implicit inputs which were not of any node's initializers in the graph. Because they were not part of any initializers, these implicit inputs couldn't be visited by going through all nodes' inputs. Consequently, they would *not* be added into graph_inputs_excluding_initializers_. We fixed the issue by first copying the populated graph_inputs_including_initializers_ into graph_inputs_excluding_initalizers_, which then had both initializers and non-initializers as its initial content. Later, we erase initializers from the list. In this way, we can ensure all implicit inputs to remain in graph_inputs_excluding_initializers_. * refined comments and fixed duplicates Address CR by revisiting comments in terms of implicit inputs Also fixed an issue by skipping duplicates while copying inputs from graph_inputs_including_initializers_. * address CR explain why we need to collect nodes' implicit inputs * don't rely on pointer values for iterating std::set Previously, openvino relied on iterating a set of NodeArg pointers to construct inputs and outputs for a fused graph. It could cause non-determinism. The reason was that although iterating std::set by itself is stable, pointer values of NodeArgs may vary. Consequently, we could end up visiting the set's elements in different orders for different runs for the same test, which resulted in constructing inputs (and outputs) with different orders to the fused graph. For example, for the same test, we may have inputs [A, B] in some runs but inputs[B, A] in others. Let's use std::string as the key type to avoid such nondeterminism. This commit also added implicit inputs into meta->inputs while returning the capability from the openvino provider. * Fixed another latent issue in openvino's GetCapability function The issue was that we couldn't simply erase fused_inputs and fused_outputs while iterating the nodes. For example, an output NodeArg may have multiple uses, and it's wrong if we erase it from fused_outputs when we encounter only one of its uses as input. * Remove DeviceAllocatorRegistry class (#2451) Remove DeviceAllocatorRegistry class * CSharp api and test for loading custom op shared library (#2420) - Added C-API test for loading custom op shared lib. - Made some changes in C++ api header and C-api implementation to get it working. - Added C# API and corresponding test for loading custom op shared library. * Parallel Gelu with ParallelFor (#2399) Parallel Gelu to get better performance for Gelu * Clean up build.py (#2446) * Pull the latest image before running docker build * Fuse SkipLayerNorm with Bias (#2453) Fuse SkipLayerNorm with Bias * Allow more than one invocation of CreateEnv in the same process. (#2467) * Allow more than one invocation of CreateEnv in the same process. * Fix centos build * Symbolic shape inference improvements: (#2460) * Symbolic shape inference improvements: - add a mode to guess unknown ops' output rank - add support for GatherND - add support for If - fix a bug in get_int_values when then tensor rank > 1D, by treating it as no sympy data - add symbol to literal merge when ONNX silently merges dims - fix a bug in Concat when input dim is 0 - fix a bug in ConstantOfShape that computed dim is not updated - add support for dynamic shape in ConstantOfShape - fix a bug in Loop output shape that loop iterator dim is not inserted at dim 0 - add support for dynamic padding in Pad - add support for dynamic shape in Reshape - add support for Resize with opset > 10, by treating output dims as dynamic - fix a bug in Slice when starts/ends are dynamic - restrict input model to opset 7 and above - make output model optional to avoid disk write when testing Run model tests for symbolic shape inference Reduce 2GB docker image size of nuphar * add additional test data set for nuget pipeline (#2448) * add SAS token to download internal test data for nuget pipeline * update azure endpoint * fix keyvault download step * fix variable declaration for secret group * fix indentation * fix yaml syntax for variables * fix setting secrets for script * fix env synctax * Fix macos pipeline * attempt to add secrets to windows download data * fix mac and win data download * fix windows data download * update test data set url and location * Revert "Brianma/windowsai fi (#2475)" This reverts commit 5780b864a15513fda4eadbfc2b5345fefe70b5ec. * Add scenario tests (#2457) * Add scenario tests * Remove TODO from model license * Add winml_api test dependency * fix model load test. fi from master changed the constructor (#2483) * make api tests all pass (#2486) * fix bad merge * fix bad model merge * Layer dev paulm (#2492) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * Rename ambiguous header (#2489) * fix one more missing IR version model (#2500) * add missing IR version to 4 more models used by scenario tests (#2501) * Add CLI parameters to test runner, build WinML in ARM and x86 CI (#2479) * Support test parameters through CLI arguments * Add WinML do Windows x86/ARM CI builds * Code style fixes * Update googletest Remove GPUTEST macros everywhere now that GTEST_SKIP is supported * Refactor main.cpp * Build scenario tests without DML * Link scenario tests to DML when it's enabled (#2502) * Layer dev release pipeline (#2488) Adds winml binaries to existing cpu nuget package, and creates new gpu dml nuget package with winml binaries and DML EP. * Layer dev paulm (#2506) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * Remove usage of IOBinding in WinML and use C_API Run method (#2504) * remove usage of iobinding * Change data structure to use vector of Ort::Values * Polish bind input / output * Use C APIrun method * Update providers on evaluate getresults * Remove run and IObinding interface from WinMLAdapter * Remove use of IObinding * bind unbound outputs code moved to learningmodelbinding * clean up unneeded istensor adapter function * Fix comment * Check if session is closed before binding and clearing * PR feedback * Layer dev paulm (#2507) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * Make tests dependend on winml_dll (#2509) * add dml binaries to DirectML package and be more explicit about condition variables (#2520) * re-enable warnings for winml builds and fix the warnings that were hiding (#2526) * turn devmode back on for winml builds * fix some warnings. include protobuf in a way that disables some warnings * undo protobufhelpers changes and just ignore 4100 errors in pb code * attempt to isolate protobufhelpers errors * add template specialization for getting tensor proto data * Layer dev paulm (#2533) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * Spawn child process to run DeviceLostRecovery scenario test (#2530) * Spawn child process to run DeviceLostRecovery scenario test * Layer dev paulm (#2536) ori said yes * add missing namespace to winml_trace_logging_provider in lotusenvironment.h (#2542) * Handle exception thrown from all apis in WinMLAdapter (#2539) * various changes to unblock windowsai ADO build * Fix custom ops scenario tests (#2562) * Do not shutdown protobuf after ort environment gets destroyed. Lazy load lotus environment first time it is needed * comment typo * pr comment about calling phoenix singleton * Make lotus_environment static in winmladapter * Layer dev paulm (#2567) * commetns for dml graph transformer fixed ort value passing using the allocatir info * fixed and coded maps and sequences across the abi * cleaned up w4's cleaned up the model info ABI delayload directml.dll from winml * cleaned up namepsace aliases. renamed _winmla to winmla this was good PR feedback from tiago a while back. * moved files from inc to lib\api.core cleaned up some of the cmake * staged changes * making windowsAI azure dev ops work. * code review comments. * revert changes * Cmake and preprocessor fixes that where uncovered by building on agents without DML available via SDK * Layer dev dml delayload (#2580) * Brianma/cpu (#2583) * don't include dml stuff in cpu builds * tests that link the image lib also need the telemetry lib now * Throw Winml_err_invalid_binding if binding gpu resource on cpu device (#2589) * Throw Winml_err_invalid_binding if binding gpu resource on cpu device * PR comments. No need to query executionprovider for is gpu device * User/xianz/ortthrow (#2596) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * User/xianz/ortthrow (#2599) * thrown and handle onnxruntime exceptions * handle exception thrown from ort in winmladapter * undo changes in error.h * add message to HRESULT * add status error message * Remove uwp onsuspending winrt call because logruntimeperf is getting removed (#2630) * User/xianz/dedup telemetry (#2631) * investigate duplication of telemetry in winml and ort * remove winml telemetry events * telemetry executionProviderEvent * remove unneccessary file and refactor code little bit * Revert back TelemetryEvent, which send up ETW event. * merge changes from layer_dev to windowsai (#2638) * Remove underscore from googletest names (#2616) * Fix leaking memory allocator Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24278761 and https://microsoft.visualstudio.com/OS/_workitems/edit/24330198 * Explicitly initialize Ort::Value with nullptr * Cache WinML adapter * bad merge * define private version of dxcore enum that is added in 19H1 SDK. (#2654) * add comment for explaning private definition of dxcore d3d feature level ennum value. (#2672) * do not package directml.pdb for redist packages. (#2676) * Fix leaking operator registry (#2645) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24354916 * User/orilevari/windowsai master merge (#2674) merge resolutions included pulling in telemetry logic that was merged to master and not windowsai and dereferencing InferenceSession::sessionstate now that it is a unique pointer * Delete Ort Allocator in LearningModelBinding (#2653) * Delete OrtAllocator in LearningModelBinding * PR comments to make Ort::Allocator a smart pointer * Small comment change * PR feedback to clean up code * PR feedback on move semantics * Clean up std::move * Fix memory leaks (#2679) Fix https://microsoft.visualstudio.com/OS/_workitems/edit/24356109, https://microsoft.visualstudio.com/OS/_workitems/edit/24388361 and https://microsoft.visualstudio.com/OS/_workitems/edit/24388596 * various changes to properly organize and skip GPU tests. For now for No DML builds we will not run GPU tests at all. In the future we should adapt the tests to expect the appropiate errors. (#2695) * Windowsai without fi (#2701) * Disable Attention fusion tests when DISABLE_CONTRIB_OPS is defined (#2529) * Setup java ci (#2528) * Add provision in ORT for session options to be parsed when available via model file (#2449) * Initial commit * Fix gitmodules * Nits * Nits * Updates * Update * More changes * Updates * Update * Some updates * More changes * Update * Update * Merge * Update * Updates * More changes * Update * Fix nits * Updates * Fix warning * Fix build * Add comment * PR feedback * PR feedback * Updates * Updates * Update * More changes * Fix build break * Comment test for now * Updates * Updates * PR feedback * Updates * Nits * Add tests * Fix build * Fix build * Fix build * Fix build break * Fix build * Nits * PR feedback * More change * Expose GetSessionOptions in pybind logic and add unit test for python * Fix build * PR feedback * PR feedback * Revert "Disable thread pool creation when enabled OpenMP (#2485)" (#2535) This reverts commit 7c7d5a149c9ed52eec67304bae5c4b132166a8a1. * Add dynamic shape support in TensorRT execution provider (#2450) * remove onnx-tensorrt submodule * add new onnx-tensorrt submodule (experiment) for trt6 * update engine build for trt6 * update compile and compute for tensorrt6.0 * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * switch to onnx-tensorrt master for TensorRT6' * Update tensorrt_execution_provider.cc * Handle dynamic batch size and add memcpy in TensorRT EP * update test cases * Update tensorrt_execution_provider.cc * update onnx-tensorrt submodule * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.ubuntu_tensorrt * Update run_dockerbuild.sh * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update concat_op_test.cc * Update tensorrt_execution_provider.cc * Upgrade TensorRT to version 6.0.1.5 * Update onnxruntime_providers.cmake * Update CMakeLists.txt * Update reduction_ops_test.cc * Update install_ubuntu.sh * Update Dockerfile.ubuntu_tensorrt * Update Dockerfile.tensorrt * Update BUILD.md * Update run_dockerbuild.sh * Update install_ubuntu.sh * Update onnxruntime_providers.cmake * Update install_ubuntu.sh * Update install_ubuntu.sh * Update gemm_test.cc * Update gather_op_test.cc * Update CMakeLists.txt * Removed submodule * update onnx-tensorrt submodule * update header file * Removed submodule * add submodule onnx-tensorrt kevin's branch shape-test' * add debugging code * Update tensorrt_execution_provider.cc * Update tensorrt_execution_provider.cc * merge master * Removed submodule * update onnx-tensorrt submodule * add more changes for dynamic shapes * Update tensorrt_execution_provider.cc * update for dynamic shape * update dynamic shape processing * fix logger issue * remove submodule onnx-tensorrt * add submodule onnx-tensorrt * add env variable min_subgraph_size * remove redundency * update document * use onnxruntime::make_unique * fix multi-run issue * remove some tests to save CI build time * Add dynamic shape test * Update TensorRT-ExecutionProvider.md * Add example of running Faster R-CNN model on TensorRT EP * Add more details on env variables * update environment variables * Update tensorrt_basic_test.cc * Update model tests * Update tensor_op_test.cc * remove --use_full_protobuf * Update build.py * User/xianz/telemetry (#2458) * enabme telemetry * enable telemetry * set enable telemetry as default * for debugging * remove log and set disable telemetry as default back * delete private file while testing * resolve comment: mainly add license header, rename macro and update docs * rewording in privacy.md * Fix integer overflow in cuda NonMaxSuppression implementation (#2540) * add test case that should pass but fail * fix nms * extract int_max_output_boxes_per_class * Introduce container type runtime checks and other improvements (#2522) Rework TensorSeq in a manner consistent with Tensor and SparseTensor in terms of type system setup. Reduce templating. Introduce helpers to ensure the same data type. Make OrtValue __dtor not virtual. Introduce ContainerChecker * Fix C API tests for centos and mac (#2544) * change c++14 to c++11 * add ld lib path for centos * enable csharp tests on macos * fix C API test on MacOS + fix manylinux dotnet install * fix manylinux dotnet install * fix lib link * Add back executable bit to build.py * Fix a bug handling negative begin pad values in Pad op (#2550) * Fix bug in Pad op * Update * DNNL CMAKE update (#2548) * Fix android build (#2558) * Update win-x86-ci.yml (#2557) Fix build pipeline break * Re-enable Windows C# tests (#2564) * disable onnx_test_runner -x invocations for dnnl (#2568) * Allow sequence length to be symbolic (#2559) * setup java ci mac (#2570) * make layernorm fusion to support opset 11 (#2545) * Fix a warning found in the latest VS release * Add more check on SkipLayerNorm and BiasGelu fusion (#2574) * Fix file not found error during docker build. (#2569) * Add ConvTranspose1D (#2578) * Ryanunderhill/packagename test (#2582) * [Nuphar EP] fixes for some object detection models (#2581) Update notebook tutorial with multi-threaded int8 GEMM from #2517 * EmbedLayerNormalization Fusion Improvement (#2553) Embedding layer norm fusion improvements - add more checks * Update version (#2584) * Temporarily exclude vgg19 test from Python backend test 1. temporarily exclude vgg19 test which comsumes too much memory, run out of memory on Upsquared device. Single test pass for vgg19, need furture investigation (#2588) 2. Update docker file to decrease the docker image size * Update docs for Android NNAPI EP (#2586) * Fix lto bug for protobuf and ubuntu * add path to build dir before test run (#2590) * Add missig env variables for mac pipeline test (#2595) * Fixed an issue in updating realized dims (#2597) when we update realized dims for scan's output, the sliced axis also needs to be inclusive, i.e. we should check with "dim >= insert_inclusive_axis", because the offset in the symbols are based on Scan sugraph. Otherwise, we would end up with shape mismatch later. * Java API for onnxruntime (#2215) * Add support for opset 11 in reshape fusion (#2592) Support opset verion 11 in reshape fusion * Rename automl python tools folder to featurizer_ops. (#2593) * Support opset 11 subgraph of Squad model in Embed Layer Normalization (#2605) Support opset 11 Squad model that is exported from PyTorch nightly. The embed layer uses Range op which is missed in the transformer. * symbolic shape inference: fix warnings in GPT-2 model (#2608) And revise nuphar perf test on BERT squad * Dump subgraph ID and fused graph ID (#2607) * Dump subgraph ID and fused graph ID Dump subgraph ID and fused graph ID for better debugging * Remove local static fused_count added a field global_fused_count_ to NupharExecutionProvider class * EmbedLayerNormalization Fusion For Dynamic Squad Model Opset 10 (#2613) Support subgraph of SQuAD model exported from pytorch with dynamic input axes * Allow providers to be set for InferenceSession at construction (#2606) * Remove unnecessary parameter in some places in GatherElements implementation (#2612) * Remove unnecessary parameter in some places * Update * Update * Make sure fenced tensor could not reuse other tensor. (#2561) Fix random error caused by this. * Improve Embed Layer Norm Fusion for SQuAD with static input shape (#2621) * fix float16 comparison in initializer (#2629) * epsilon attribute for layernormalization fusion (#2639) * removed unnecessary batch file and fix path (#2640) * Add shape inference to ConvTransposeWithDynamicPads schema (#2632) * Improve cuda expand() opeator's performance. (#2624) * Cuda pad optimize when no padding is needed. (#2625) * Shortcut cuda Pad() when no padding is needed. * Optimize cuda scatter() on 2D compatible. (#2628) * Optimize cuda scatter() on 2D compatible. * Add some comments. * fix build error for ARM (#2648) * Improve performance of resize() in Nearest mode (#2626) Special treatment for 2D, check same size as input image. And in 2d kernel, template use_expolation. * Fix memory exception in Layer Norm Fusion (#2644) * Windows CI changes(#2650) * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Revert "Windowsai without fi (#2701)" This reverts commit 285d4c85ff5c4e265f963208170304ef3461e684. * Revert "User/orilevari/windowsai master merge (#2674)" This reverts commit fe261463112f0cf7cdef214c57eb7c70e816b616. * Deref unique pointer for session_state * send shutdown event when dll is unloaded and EvaluationStop, SessionC… (#2704) * send shutdown event when dll is unloaded and EvaluationStop, SessionCreationStart Events. * Add EvalutationStart Event * add comment * use correct type for for loop (#2755) * ARM CI (#2759) * Set ARM agent pool * Set CMake generator to VS 2019 in ARM * Use system-wide CMake instead of custom version Our custom version is too old for VS 2019 * Use DML and build shared lib in ARM CI * Restore nuget packages in ARM CI * Disable DML * Refactor ARM debug/release builds * Use system packaged Python version * Remove hardcoded Python path * Downgrade Python to 3.7 for build * Remove explicit CMake path * Fix invalid JSON in cgmanifest.json (#2760) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * Remove usage of WHOLEARCHIVE in WinML CMake and add WinMLAdapterFactory (#2726) * Remove usage of WHOLEARCHIVE in WinMLAdapter CMake and add WinMLAdapterFactory * PR feedback, no need for dll(export) since using def file * PR comments * Small comment in gen_def.py * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * Move winml_provider_factory.h to proper location (#2801) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Filter CPU case for IsFloat16Supported (#2802) * Merge fixes * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * Fix warnings that cause build to fail * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * Make DML include path global (#2882) * Make DML include path global * Add generated cppwinrt headers to winml_lib_common * Integrate changes to WindowsAI to make ADO Build (#2886) * Revert "CMake cross-generator fixes (#2790)" This reverts commit dbe7d97fa1ab155f1309bced87199527e8f35bd2. * add additional suppress warning in onnx_proto * ignore /wd4996 warning * DML execution provider fixes * Revert "Revert "CMake cross-generator fixes (#2790)"" This reverts commit 1ae7b4bcbc02edc881ad28685da98e095dfceb17. * Update func signature of custom op function overloads * common devicehelpers fixes * Add pch.h for winml_lib_common * re-add winml_lib_common_dir/inc to include path for winml_adapter * User/orilevari/dml redist shared folder (#2890) * move dml nuget package directory up one level to make it shared between build flavors * Merge conflict fix * Revert "Merge conflict fix" This reverts commit 142fa72cf9ce4344ad717b50b7ea2b8582aadc7c. * Revert "Merge remote-tracking branch 'origin/master' into windowsai" This reverts commit 6e2126d46e5e5f564d65da37dd4f70c93dd81165, reversing changes made to b3f5583dc9249834b947c8ea905f6a98060d5bd6. * Make winml_test_common free of test macros (#2902) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Don't generate files outside the build root (#2914) * Don't generate files outside the build root * Add onnxruntime_EXTERNAL_DEPENDENCIES to WinML * Add DML depedency on RESTORE_PACKAGES * User/orilevari/fix yaml merge bugs (#2918) * Add winml test source parameter into cmake function (#2919) * Add option to build winml_test_common without googletest specifics * remove test macros from squeezenet * comment change * Make cmake functions to get scenario and api source * PRcomments about hresult * Build errors fixed * Fix cmake variable * Make winml_google_test_lib to build main.cpp once * PRcomments * Add arguments to unittest cmake functions * remove comment * Revert "Revert "Merge remote-tracking branch 'origin/master' into windowsai"" This reverts commit ade5abe72a4234fdbc3623093c61c02c6b0bdc26. * Fix breaks from merge with ORT master * Brianma/linux (#2917) * don't include windows.h in cross-plat header * add default case for switch statement * signed/unsigned mismatch fix Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> * User/sheilk/winml adapter c api (#2891) * Create winml adapter c api * fix build * make it build * move adapter into onnxruntime core/session * entry point not exported * minor changes * make model metadata work * make tests pass * implement all the model reflection apis on the adapter c abi * update the new ort interface to create a lotus ennvironment with a logging sink * start adding ort env * move all winml code into adapter folder/lib to isolate it * ensure a single logging manager at a time * start refactoring session * refactor session creation interface * add cpu and dml session option methods to adapter * finish session init * stub out interfaces in ort lib to perform similar mechanics of iinference session * enable profiling, and enable schema override * update session register graph transformers * turn back on custom registry for custom ops * Add sync api * add last c api stubs * should build... but all feature values are broken since this is in flight to moving all implementation details into ivalue * remove ep adapter header * Implement DML execution provider functions from adapter (#2846) * Implement DML execution provider functions from adapter * Use functions in OnnxruntimeEngine.cpp * make map/sequence type_infos freeable, and start implementing ivalue * make it build again * implement value methods * implement remaining methods * remove com adapter abi * check dml session * cache the allocator on ivalue * check if resource is cpu/gpu when access its mutable data * update tensor * mismatched parentheses * fix tensor base and binding obj * it evaluates tensors! sometimes... * minor fixes * enable gpu evals * wrapper all existing winml adapter apis with API_IMPL to try catch (#2854) * update winml... tensor strings are broken, need to template tensorbase to do different things for strings * make tensor strings work with 2 copies in/2 copies out * Fix tensor string and allocator bug * make maps work again... needs some fixes still * Make it build! * enable map inputs * map outputs * unbound outputs for sequences and maps * User/xianz/merge windowsai (#2883) * Packaging pipeline changes for VS 2019 (#2711) * Tiny fix to codegen * Simplify cache implementation and avoid static variables that may carry over between models * Extend DML kernels (#2641) * Additional DML operators * Check unsupported attributes and inputs * Address PR comments * Add kernel capability function used for partitioning, and re-enable stride-based int64 support based on value range * Fix test failures * Build fix * PR comments * Update Nuphar tutorial notebook (#2721) 1. Reflect int8 GEMV improvements for multi-threading from #2696 2. Add notes on multi-threading control using OpenMP 3. Add samples of running multi-isa AOT, and show int8 GEMM differences between AVX and AVX2 4. Add rnn_benchmark example to resolve #1993 * Add schema for new Qops (#2611) * Add schema for new Qops * adding shape inference + qlinearaveragepool * plus review comments * plus review comments * updates per review comments * plus review comments * [server] Add supposed for model_name and model_version as cli parameter (#2708) * remove 64bit warning message from python validation. (#2727) * MLAS: ARM64 build fix (#2734) fix bad usage of vreinterpret to cast vector element types * Fix broken python docs links (#2740) * Fix build on Mac OS (#2731) mac os ld doesn't support --while-archive, correct option is -all_load * fix ngraph wheel (#2737) * fix ngraph wheel 1.1.0 onnxruntime_ngraph wheel doesn't work * remove libdnnl.so in nGraph Libs * make it easy to compare * Split onnxruntime server to a separated folder (#2744) * Fix build for Python 3.8 (#2747) * Fix build for Python 3.8 * Update protobuf to 3.11.2 (#1928) Update protobuf to 3.11.2 (#1928) * Change default optimization level to All (from Basic) (#2745) * change default optimization level to All (from Basic) * fix test * fix c# test * Update numpy to 1.18 (#2758) * Update numpy to 1.18 * Pipeline changes for python 3.8 (#2753) 1. Pipeline changes for python 3.8 2. Fix a regression in setup.py which was just introduced in the previous commit. Please notice, we still haven't made python 3.8 + Windows + CUDA work. * Add basic stacktrace output for posix debug builds. (#2749) * [NupharEP] fix a race condition when multiple sessions running different models concurrently (#2772) * Revert "Change default optimization level to All (from Basic) (#2745)" This reverts commit 56bb503c2f26474b6613bcb2a198691a11dcef38. * Fix typo in error message (#2736) * Rename MKL-DNN to DNNL to fix broken link (#2730) * Fix nightly build version number issue * Pass BUILD_BUILDNUMBER to linux docker * Disable featurizers in python packages * Import more featurizers (#2781) Make kernels non-template. Add input constraint for learnt data. Add min_max_scalar_transformer, robust_scalar_transformer, inputation_marker_transfomer, label_encoder_transformer, missing_dummies_transformer along with tests. Advance Featurizers library commit. * Implement a more stable softmax (#2715) * Implement a more stable SoftMax e^x is represented as infinity if x is large enough, like 100.f. Infinity divided by Infinity is a NAN. Thus, softmax gets a NAN if one or more item are large enough. A math transform as below is leveraged to get a stable softmax: e^xi/(e^x1 + ...e^xn) = e^(xi - max) / (e^(x1 - max) + ... + e^(xn - max)) And for convenience, force max to 0.f if all xi are negative * Contributing: Fix a typo (#2784) * ACL EP GEMM improvements (#2780) When it is posible we use a fully connected layer instead of the gemm implementation. This will let the library use the best implementation based on the input data. * ACL EP convolution improvements (#2774) Added the optimized implementation for depthwise convolution for both ACL v19.02 and ACL 19.05. Also the pointwise convolution seems to be more optimal in the CPU implementation so we opted for that instead. * Add script for release Nuget validation (#2719) * Initial commit * Nits * Disable a test temporarily * Change working directory * Test * Add download python step * Test update * More changes * Fix space issue * Fix * Verify nuget signing * Fix * Spaces * PR feedback * Nit * Fix * Fix * Remove temporary changes * add uint8 support to where op (#2792) * Improve bert optimization script: (#2712) (1) Move input int64=>int32 conversion to embed layer fusion. (2) Output epsilon attribute for LayerNormalization fusion. * add session creation time cost. (#2798) * ML.NET team needs featurizers within a package (#2789) Add auto ml featurizers to Windows, MacOS as well as to GPU packaging-pipelines. * Initialize max of softmax with lowest of float (#2786) * MLAS: update SGEMM threading parameters (#2808) * add interface to copy batch tensors. (#2807) * add interface to copy batch tensors. * onnxruntime * speed up Windows TRT CI (#2811) * don't run cuda tests if building with tensorrt * remove unnecessary build options for win trt ci * refactor win gpu tensorrt ci yml * --numpy_version=1.17 * update * update * azcopy and cuda path * Update test data (#2356) * Add timeseries imputer transformer featurizer kernel (#2813) Make kernels non-template. Add input constraint for learnt data. Fixup tests. Add two more featurizers along with tests. Tests fail. min_max_scalar_transformer robust_scalar_transformer Fix tests serialized stream by prepending version bytes. Add inputation_marker_transfomer and the test. Fix up float/double type designations. Added label_encoder_transformer along with a test. string_throw case is broken at the momement. Fix labelencodertransfomer_test.cc string_throw case Rename maxabsscalertransformer_test.cc Add MissingDummiesTransformer along with the test. Update manifest. Add TimeSeriesImputerTransformer definition, implementation and tests * Fix memory leak in TRT (#2815) * fix memory leak issue * revert EP_FAIL on enueueV2 * Add manifest missing comma * Run static code analyzer on most of our code (#2817) * Scneario Test : Build Google Test and Taef Test based on preprocessor definition (#2809) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * update quantization doc (#2783) * update documentation for quantization script * plus some spell corrections * Filter CPU case for IsFloat16Supported (#2802) * update default optimization level + fix gemm_activation fusion (#2791) * update defualt optimization level + fix gemm_activation fusion * fix typo * add unit test and incorporate review comments * fix test comment * Fix dnnl wheel package name (#2823) * Append '-dnnl' to whl package name when --use_dnnl * Update build.py * Update Ubuntu & TensorRT version in README (#2820) Dockerfile.tensorrt is using nvcr.io/nvidia/tensorrt:19.09-py3 as base Image, update Ubuntu and TensorRT version according to https://docs.nvidia.com/deeplearning/sdk/tensorrt-container-release-notes/rel_19-09.html#rel_19-09 * Merge fixes * Add OneHotEncoder and HashOneHotEncoder kernels. (#2830) Add defs and imlementation for OneHotEncoders, adjuist date_time_transformer kernel and test. Add OneHotEncoder kernel test. Add HashOneHotVectorizerTransformer unit test. This does not link due to multiple definitions of functions that are included into header from a CPP file. * Upgrade gtest to the latest version (#2827) WinML would like to update the googletest submodule. They want some newer features (namely GTEST_SKIP to skip tests programmatically and be able to skip entire fixtures easily) and would need to update the submodule version. However, because the new version of code hit a bug in gcc, even though the bug is already fixed in the latest gcc but we're using gcc 4.8.x and it won't get patched for the bug, so we have to do a compromise, change our code a little bit to make it work. The gcc bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=51213 * Add support for int64_t for topk CPU. Fixes github issue #2806. (#2833) * Ignore allocator type in ExecutionProviders allocator map. Make default initialization of OrtMemoryInfo more clearly invalid. (#2768) * Remove allocator type from the key comparison in ExecutionProviders. Remove usage of DummyArena as it's no longer necessary. * Fix x86 tests where arena allocator is disabled. Make initialization of OrtMemoryInfo clearer by adding Invalid enum value. * Make OrtValueNameIdxMap::MaxIdx more intuitive. * Convert ExternalProject Featurizers into git submodule (#2834) Add git submodule for Featurizer library. Update cmake to build for git submodule. * add domain check for nodes + update documentation (#2831) * Fix cgmanifest.json generating script (#2770) * Fix protobuf submodule name * Workaround pygit2 bug * User/orilevari/32bit comparison warning (#2800) * use correct type for for loop * explicitly specify void for parameters of OrtGetApiBase because the function is defined in c, so when the function is just (), it is interpreted as having an unknown number of parameters. This was causing compiler warning C4276. * CMake cross-generator fixes (#2790) * Fix compilation w/ non-VS CMake generators * Fix custom WINMD target in Ninja * Remove usage of msbuild .targets file * Fix linking using DML in Ninja * Automate SDK kit version choice * Cleanup DML package install * Fix SDK version detection * Fix comment * Revert unittest linkage changes * Fix latest SDK detection * Don't link to non-uapcore libraries * Remove MessageBoxA reference and unused link libs * Fix Linux CUDA nuget packaging pipeline break * Refactor WinMLAPI Tests to build both google and taef test based on preprocessor definition (#2829) * Add winml macro wrappers on top of google test macros * change test methods to disabled * Add custom winml macros for both taef and google tests * PR comments * Refactor winml api tests * Move additional gtest specific macro definition into googleTestMacros.h * Fix test build break since winml_lib_api needs to be statically linked to tests since winmlp::learningmodeldevice::iscpu() is being used in devicehelpers.cpp (#2837) * Enforce WINML_TEST_CLASS_BEGIN_* matches w/ a WINML_TEST_CLASS_END (#2841) * update optimization doc for BERT related fusions (#2819) * Add bert related transformers to doc * Add execution provider and comment for bert optimizations * Add comment about accuracy impact of approximation * Fix warnings that cause build to fail * MLAS: enable threading for quantized GEMMs (#2844) * Fix test warnings and delayload linking (#2843) * Ortmemoryinfo struct changed * mark the camera scenario test as edgecore because it uses d3d11 (#2852) * User/orilevari/pipeline fi breaks (#2853) * remove conflicting artifact names. Decided to stop using drop-nuget-cuda since this may have implications on other dependent pipelines. * change job name in gpu.yml back to Windows_CI_GPU_CUDA_Dev * Remove internal libs from tests (#2864) * Support custom DML in onnxruntime_providers.cmake (#2867) * remove old winmladapter cpp Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * move sequence implementation into ort lib... still commented out... need to turn back on... * begin sequence implementation * make maps and sequences work * fix broken tests * remove dead code * misc cleanup * CR feedback * User/xianz/winml adapter c api (#2869) * wrapper all existing winml adapter apis with API_IMPL to try catch * Return HR or Throw for WinML adapter APIs if failed * undo macro wrapper for two places * Wrap error macros around ort apis, too. * address CR feedback #2 * add more api throw/return macros * Revert changes no longer needed * revert changes to cxx api * format winml lib.ort and winml adapter * remove static pheonix singleton Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> * missing use_dml check in winml_adapter_session (#2930) * --use_dnnl flag was mangled in merge (#2931) * use dml macro not wrapping custom registry code (#2934) * Disable LNK4199 winml_dll to enable cuda builds (#2936) * Disable LNK4199 in winml_dll * linkler->linker * LearningModelSessionAPITestGpu.CreateSessionWithCastToFloat16InModel should return DXGI_ERROR_UNSUPPORTED when FP16 not supported (#2937) * Disable LNK4199 in winml_dll * linkler->linker * Need to return DXGI_ERROR_UNSUPPORTED when Model does not support fp16 * Publish build symbols (#2939) * Publish build symbols * Don't upload PDBs for .exe files * Make x86 build (#2943) * fix last remaining size_t/int64_t warnings->errors (#2948) * TensorString, Sequences and Maps use the first allocator, but should use the cpu default allocator. (#2952) * fix tensor string allcoator * clean up default allocator usage for strings in winml lib/api.ort Co-authored-by: Ryan Lai <ryalai96@gmail.com> * Handle tensor shape of zero (#2954) Co-authored-by: Ryan Lai <ryalai96@gmail.com> * CR feedback (#2970) * CR feedback * fix weird formatting on privacy readme * Add 'All rights reserved.' everywhere * readd all rights reserved to winml_provider_factory.h * remove extra space in comment * remove extra whitespace * fixes post master merge * remove winml from nuget gpu pipeline * set IR VERSION on generated_model in rnn_benchmark (#2972) * Fix slice conformance failures (#2908) Co-authored-by: Adrian Tsai <adtsai@microsoft.com> Co-authored-by: Brian Martin <42186431+martinb35@users.noreply.github.com> Co-authored-by: Ryan Lai <ryalai96@gmail.com> Co-authored-by: Paul McDaniel <paul_mcdaniel@hotmail.com> Co-authored-by: Xiang Zhang <xianz@microsoft.com> Co-authored-by: Dwayne Robinson <fdwr@hotmail.com> Co-authored-by: Tiago Koji Castro Shibata <tiago.shibata@gmail.com> Co-authored-by: Ori Levari <ori.levari@microsoft.com> Co-authored-by: Jeff <38966965+jeffbloo@users.noreply.github.com> Co-authored-by: Changming Sun <chasun@microsoft.com> Co-authored-by: KeDengMS <kedeng@microsoft.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Andrey <andrey.lompart@gmail.com> Co-authored-by: George Wu <jywu@microsoft.com> Co-authored-by: Tracy Sharpe <42477615+tracysh@users.noreply.github.com> Co-authored-by: Faith Xu <txsafx@gmail.com> Co-authored-by: zhanyi-ms <zhanyi@microsoft.com> Co-authored-by: Changyoung Koh <gkcy1019@gmail.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com> Co-authored-by: Takeshi Watanabe <take-cheeze@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Yufeng Li <liyufeng1987@gmail.com> Co-authored-by: Maher Jendoubi <maher.jendoubi@gmail.com> Co-authored-by: Andrews548 <32704142+Andrews548@users.noreply.github.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: Nathan <7902510+ybrnathan@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Ke Zhang <kezhan@microsoft.com> Co-authored-by: stevenlix <38092805+stevenlix@users.noreply.github.com> Co-authored-by: Yingge WAN <y-wan@users.noreply.github.com> Co-authored-by: Qing <cwq1913@gmail.com> Co-authored-by: Pranav Sharma <emailpranav@gmail.com>
2020-02-05 01:12:19 +00:00
[threading(both)]
[marshaling_behavior(agile)]
[dualapipartition(1)]
runtimeclass ImageFeatureValue : ILearningModelFeatureValue
{
static ImageFeatureValue CreateFromVideoFrame(Windows.Media.VideoFrame image);
Windows.Media.VideoFrame VideoFrame{ get; };
}
}