From c99aa3a3f3834adcbb888ce4b964f2695c524eae Mon Sep 17 00:00:00 2001 From: Ryan Hill <38674843+RyanUnderhill@users.noreply.github.com> Date: Thu, 20 May 2021 07:53:47 -0700 Subject: [PATCH 01/28] Ryanunderhill/cuda shared (#7626) * First iteration of making cuda a shared provider. Separated out shared OpKernel change, so doing this to merge with that change. * More cuda shared library refactoring * More cuda shared library refactoring * More build options tested, converted the training ops over. * Fix merge breaks * Fix submodules * Fix submodules * Fix submodules * Fix python * Fix compile errors * Duplicate symbol fix * Test fix for ROCM provider * Another ROCM test workaround * ROCM Build Test * ROCM build fix * ROCM * ROCM * ROCM * ROCM * ROCM * ROCM test * Reduce header dependencies * Remove redundant namespace * Test fix for linux * Fix linux build * Fix Eigen build error * Fix unused parameter warning * Test link error * Another linker test * Linker test * Linker test * Another test * Another build test * Fix linux link error * Build test * Fix control flow ops to use common base class with core code * Remove extra qualifiers * Fix template syntax for linux * Fix cuda memory leak * Fix pybind * Test disabling cast * Cleanup * Restore cuda in test * Remove more header dependencies * Test not adding cuda provider to session * Make GetProviderInfo_CUDA throw * No-op cuda provider creation * Fix some setup issues * Fix memory cleanup on unload * Diagnostics * Don't unload library * Add diagnostics * Fix deleting registry at right time. * Test disabling profiler * Fix merge break * Revert profiler change * Move unloading of shared providers into Environment * Free more global allocations before library unloads * Add more diagnostics * Move unloading back to the OrtEnv as there are multiple Environments created during a session. Remove some library dependencies for tests. * Fix more cmake files * ERROR -> WARNING * Fix python shutdown * Test not using dml in pipeline * Change python version and disable dml * Update python version * Test adding unload method for shared providers * Disable DLL test * Python test * Revert "Python test" This reverts commit c7ec2cfe98e4dcc30e1a54cad71adb89ab18adde. * Revert "Disable DLL test" This reverts commit e901cb93aafd537af088fa29629029a479926362. * Revert "Test adding unload method for shared providers" This reverts commit c427b7879989be93572b09c4662b1fa8b153eec9. * Point to RyanWinGPU * Revert python version * Fix id_to_allocator_map * Another python exit test * Remove extra debug messages Try a more clean python shutdown through DllMain * Revert DllMain idea, it didn't work * Merge conflicts * Fix merge with master issues. * Comments * Undo edit to file * Cleanup + new training ops * Revert yml changes * Fix another merge error * ROCM fix * ROCM fix v2 * Put back Linux hack, it is necessary * Stupid fixes * Fix submodule out of sync * ROCM fix 3 * ROCM 4 * Test java fix * Fix typos * Java test on my VM * Fix build error * Spotless fix * Leave temp file around to load properly * Fix cleanup on exit * Fix break * Java comments * Remove LongformerAttentionBase workaround * Spotless fix * Switch yml back to regular build pool * Revert "Switch yml back to regular build pool" This reverts commit be35fc2a5ab53868568791f1e54320e6c7fc75ce. * Code review feedback * Fix errors due to merge * Spotless fix * Fix minimal build * Java fix for non cuda case * Java fix for CPU build * Fix Nuphar? * Fix nuphar 2 * Fix formatting * Revert "Remove LongformerAttentionBase workaround" This reverts commit 648679b37093740bbefc7b41ff53d3239451333f. * Training fix * Another java fix * Formatting * Formatting * For orttraining * Last orttraining build fix... * training fixes * Fix test provider error * Missing pass command * Removed in wrong spot * Python typo * Python typos * Python crash on exit, possibly due to unloading of libraries. * Remove test_execution_provider from training build Only enable python atexit on windows Remove assert on provider library exit * Still can't unload providers in python, alas. * Disable Nvtx temporarily * MPI Kernels for Training * MPI Kernels part 2 * Patch through INcclService * Oops, wrong CMakeLists * Missing namespace * Fix missing () * Move INcclService::GetInstance around to link nicer * Missing } * Missing MPI libraries for Cuda * Add extra GetType functions used by MPI * Missing Nccl library * Remove LOGS statements as a test * Add in a couple more missing GetType methods * Update comments * Missed a logging reference in mpi_context.h * Convert aten_op to shared (due to marge with master) * Test moving DistributedRunContext instance into shared provider layer (with purpose error to verify it's being built properly) * Test passed, now with fix * Missing static * Oops, scope DistributedRunContext to just NCCL * Merge related issues and code review feedback. * Merge error * Bump to rel-1.9.1 (#7684) * Formatting * Code review feedback for Java build on non Windows * Remove cupti library dependency from core library * Test Java pipeline fix * Linux build fix * Revert "Linux build fix" This reverts commit a73a811516ad2acdbbea890a2ae2143341f915ad. * Revert "Remove cupti library dependency from core library" This reverts commit 6a889ee8bf72f34f44b8a4642728b7bd83f61255. * Packaging pipeline fixes to copy cuda shared provider for tensorrt & standard packages * Add cuda to Tensorrt nuget package * onnxruntime_common still has a cuda header dependency Co-authored-by: ashbhandare --- cmake/CMakeLists.txt | 4 +- cmake/onnxruntime.cmake | 1 - cmake/onnxruntime_common.cmake | 5 + cmake/onnxruntime_framework.cmake | 8 +- cmake/onnxruntime_java.cmake | 8 + cmake/onnxruntime_providers.cmake | 55 +- cmake/onnxruntime_python.cmake | 12 +- cmake/onnxruntime_training.cmake | 1 - cmake/onnxruntime_unittests.cmake | 16 +- .../core/framework/kernel_def_builder.h | 9 + include/onnxruntime/core/framework/ml_value.h | 2 + .../core/framework/op_node_proto_helper.h | 2 + include/onnxruntime/core/framework/tensor.h | 9 +- .../providers/cuda/cuda_provider_factory.h | 44 + .../main/java/ai/onnxruntime/OnnxRuntime.java | 38 +- onnxruntime/contrib_ops/cpu/bert/attention.cc | 8 - .../contrib_ops/cpu/bert/attention_base.h | 25 +- .../cpu/bert/embed_layer_norm_helper.cc | 21 +- .../cpu/bert/embed_layer_norm_helper.h | 2 + .../cpu/bert/longformer_attention_base.cc | 10 - .../cpu/bert/longformer_attention_base.h | 15 +- .../cuda/activation/activations.cc | 2 +- .../contrib_ops/cuda/bert/attention.cc | 5 +- onnxruntime/contrib_ops/cuda/bert/attention.h | 1 - .../contrib_ops/cuda/bert/embed_layer_norm.cc | 6 +- .../contrib_ops/cuda/bert/embed_layer_norm.h | 2 +- .../contrib_ops/cuda/bert/fast_gelu.cc | 5 +- .../cuda/bert/longformer_attention.cc | 6 +- .../contrib_ops/cuda/bert/skip_layer_norm.cc | 9 +- .../cuda/conv_transpose_with_dynamic_pads.cc | 4 +- .../contrib_ops/cuda/cuda_contrib_kernels.cc | 84 +- .../contrib_ops/cuda/cuda_contrib_kernels.h | 3 +- onnxruntime/contrib_ops/cuda/fused_conv.cc | 4 +- onnxruntime/contrib_ops/cuda/inverse.cc | 2 +- onnxruntime/contrib_ops/cuda/layer_norm.cc | 9 +- onnxruntime/contrib_ops/cuda/layer_norm.h | 1 - .../contrib_ops/cuda/math/bias_dropout.cc | 6 +- .../contrib_ops/cuda/math/bias_softmax.cc | 87 +- .../cuda/math/bias_softmax_impl.cu | 1 - .../cuda/math/binary_elementwise_ops.cc | 16 +- .../contrib_ops/cuda/math/complex_mul.cc | 5 +- .../contrib_ops/cuda/math/complex_mul.h | 2 +- onnxruntime/contrib_ops/cuda/math/fft_ops.cc | 4 +- onnxruntime/contrib_ops/cuda/math/fft_ops.h | 6 +- .../contrib_ops/cuda/math/fused_matmul.cc | 2 +- onnxruntime/contrib_ops/cuda/math/isfinite.cc | 2 +- .../quantization/attention_quantization.cc | 12 +- .../quantize_dequantize_linear.cc | 4 +- onnxruntime/contrib_ops/cuda/tensor/crop.cc | 2 +- .../contrib_ops/cuda/tensor/dynamicslice.cc | 22 +- .../contrib_ops/cuda/tensor/image_scaler.cc | 2 +- .../core/framework/feeds_fetches_manager.h | 2 + .../core/framework/ort_value_tensor_slicer.cc | 3 +- .../core/framework/ort_value_tensor_slicer.h | 2 + .../core/framework/provider_bridge_ort.cc | 687 ++++++-- .../core/framework/provider_bridge_ort.h | 1 + onnxruntime/core/framework/tensorprotoutils.h | 21 +- onnxruntime/core/platform/env_var_utils.h | 4 + onnxruntime/core/providers/common.h | 2 + .../core/providers/cpu/controlflow/if.cc | 43 +- .../core/providers/cpu/controlflow/if.h | 25 +- .../core/providers/cpu/controlflow/loop.cc | 84 +- .../core/providers/cpu/controlflow/loop.h | 29 +- .../core/providers/cpu/controlflow/scan.h | 42 +- .../core/providers/cpu/controlflow/scan_8.cc | 12 +- .../core/providers/cpu/controlflow/scan_9.cc | 12 +- .../providers/cpu/controlflow/scan_utils.h | 22 - .../core/providers/cpu/controlflow/utils.h | 6 +- .../cpu/generator/constant_of_shape_base.h | 16 +- onnxruntime/core/providers/cpu/math/clip.h | 9 +- onnxruntime/core/providers/cpu/math/einsum.h | 4 +- .../math/einsum_utils/einsum_auxiliary_ops.cc | 20 +- .../math/einsum_utils/einsum_auxiliary_ops.h | 18 +- .../einsum_compute_preprocessor.h | 3 +- .../core/providers/cpu/math/matmul_helper.h | 2 + .../core/providers/cpu/nn/batch_norm_helper.h | 2 + .../core/providers/cpu/nn/conv_attributes.h | 6 +- .../cpu/nn/conv_transpose_attributes.h | 2 +- .../providers/cpu/nn/instance_norm_helper.h | 2 + .../core/providers/cpu/nn/pool_attributes.h | 8 + onnxruntime/core/providers/cpu/nn/pool_base.h | 7 +- .../object_detection/non_max_suppression.h | 9 +- .../providers/cpu/reduction/reduction_ops.cc | 24 +- .../providers/cpu/reduction/reduction_ops.h | 10 +- .../core/providers/cpu/rnn/rnn_helpers.h | 3 - .../core/providers/cpu/tensor/concat.h | 42 +- .../core/providers/cpu/tensor/concatbase.h | 48 + .../core/providers/cpu/tensor/gather.h | 20 +- .../core/providers/cpu/tensor/gatherbase.h | 27 + onnxruntime/core/providers/cpu/tensor/pad.h | 64 +- .../core/providers/cpu/tensor/padbase.h | 71 + .../providers/cpu/tensor/reshape_helper.h | 4 - .../core/providers/cpu/tensor/scatter_nd.h | 9 +- .../core/providers/cpu/tensor/shape_op.h | 2 + onnxruntime/core/providers/cpu/tensor/size.h | 2 + onnxruntime/core/providers/cpu/tensor/slice.h | 2 + onnxruntime/core/providers/cpu/tensor/split.h | 19 +- .../core/providers/cpu/tensor/squeeze.h | 7 +- onnxruntime/core/providers/cpu/tensor/tile.h | 2 + .../core/providers/cpu/tensor/transpose.h | 5 +- .../core/providers/cpu/tensor/unsqueeze.h | 17 +- .../core/providers/cpu/tensor/upsample.h | 4 +- onnxruntime/core/providers/cpu/tensor/utils.h | 3 + .../providers/cuda/activation/activations.cc | 24 +- .../providers/cuda/activation/activations.h | 1 + .../core/providers/cuda/controlflow/if.cc | 13 +- .../core/providers/cuda/controlflow/loop.cc | 22 +- .../core/providers/cuda/controlflow/scan.cc | 13 +- .../core/providers/cuda/controlflow/scan.h | 1 - .../core/providers/cuda/cuda_allocator.cc | 2 - onnxruntime/core/providers/cuda/cuda_call.cc | 4 +- .../core/providers/cuda/cuda_common.cc | 17 - onnxruntime/core/providers/cuda/cuda_common.h | 24 +- .../providers/cuda/cuda_execution_provider.cc | 85 +- .../providers/cuda/cuda_execution_provider.h | 17 +- .../cuda/cuda_execution_provider_info.cc | 17 +- onnxruntime/core/providers/cuda/cuda_fence.cc | 4 +- onnxruntime/core/providers/cuda/cuda_fence.h | 4 +- onnxruntime/core/providers/cuda/cuda_fwd.h | 4 +- onnxruntime/core/providers/cuda/cuda_kernel.h | 3 - .../providers/cuda/cuda_provider_factory.cc | 196 ++- .../core/providers/cuda/cudnn_common.h | 4 +- .../core/providers/cuda/exported_symbols.lst | 1 + .../cuda/generator/constant_of_shape.cc | 17 +- .../cuda/generator/constant_of_shape.h | 2 - .../core/providers/cuda/generator/range.cc | 12 +- .../core/providers/cuda/gpu_data_transfer.cc | 1 + .../cuda/math/binary_elementwise_ops.cc | 70 +- onnxruntime/core/providers/cuda/math/clip.cc | 11 +- onnxruntime/core/providers/cuda/math/clip.h | 4 +- .../core/providers/cuda/math/clip_impl.h | 1 - .../core/providers/cuda/math/cumsum.cc | 10 +- .../core/providers/cuda/math/einsum.cc | 71 +- onnxruntime/core/providers/cuda/math/einsum.h | 5 +- .../math/einsum_utils/einsum_auxiliary_ops.cc | 24 +- .../math/einsum_utils/einsum_auxiliary_ops.h | 8 +- onnxruntime/core/providers/cuda/math/gemm.cc | 8 +- .../core/providers/cuda/math/matmul.cc | 10 +- .../providers/cuda/math/matmul_integer.cc | 6 +- .../core/providers/cuda/math/softmax.cc | 157 +- onnxruntime/core/providers/cuda/math/topk.cc | 6 +- .../core/providers/cuda/math/topk_impl.cu | 1 - .../cuda/math/unary_elementwise_ops.cc | 50 +- .../cuda/math/variadic_elementwise_ops.cc | 41 +- .../core/providers/cuda/nn/batch_norm.cc | 36 +- .../core/providers/cuda/nn/batch_norm.h | 1 - onnxruntime/core/providers/cuda/nn/conv.cc | 36 +- onnxruntime/core/providers/cuda/nn/conv.h | 1 - .../core/providers/cuda/nn/conv_transpose.cc | 32 +- .../core/providers/cuda/nn/conv_transpose.h | 1 + onnxruntime/core/providers/cuda/nn/dropout.cc | 12 +- .../core/providers/cuda/nn/instance_norm.cc | 2 +- onnxruntime/core/providers/cuda/nn/lrn.cc | 34 +- onnxruntime/core/providers/cuda/nn/pool.cc | 42 +- onnxruntime/core/providers/cuda/nn/pool.h | 5 +- onnxruntime/core/providers/cuda/nn/shrink.cc | 4 +- .../object_detection/non_max_suppression.cc | 16 +- .../non_max_suppression_impl.cu | 1 - .../non_max_suppression_impl.h | 3 - .../cuda/object_detection/roialign.cc | 11 +- .../cuda/object_detection/roialign_impl.h | 2 - .../cuda/reduction/reduction_functions.cc | 2 +- .../providers/cuda/reduction/reduction_ops.cc | 345 ++-- .../providers/cuda/reduction/reduction_ops.h | 9 +- .../core/providers/cuda/rnn/cudnn_rnn_base.cc | 3 +- onnxruntime/core/providers/cuda/rnn/gru.cc | 11 +- onnxruntime/core/providers/cuda/rnn/lstm.cc | 11 +- onnxruntime/core/providers/cuda/rnn/rnn.cc | 11 +- .../providers/cuda/shared_inc/cuda_utils.h | 1 - .../providers/cuda/shared_inc/fast_divmod.h | 5 +- onnxruntime/core/providers/cuda/symbols.def | 2 + .../core/providers/cuda/tensor/cast_op.cc | 45 +- .../core/providers/cuda/tensor/cast_op.h | 1 + .../core/providers/cuda/tensor/compress.cc | 6 +- .../core/providers/cuda/tensor/compress.h | 1 + .../core/providers/cuda/tensor/concat.cc | 6 +- .../core/providers/cuda/tensor/concat.h | 3 +- .../core/providers/cuda/tensor/expand.cc | 21 +- .../core/providers/cuda/tensor/expand.h | 2 +- .../core/providers/cuda/tensor/expand_impl.h | 3 - .../core/providers/cuda/tensor/eye_like.cc | 28 +- .../core/providers/cuda/tensor/eye_like.h | 2 +- .../core/providers/cuda/tensor/flatten.cc | 9 +- .../core/providers/cuda/tensor/flatten.h | 2 +- .../core/providers/cuda/tensor/gather.cc | 7 +- .../core/providers/cuda/tensor/gather.h | 4 +- .../providers/cuda/tensor/gather_elements.cc | 5 +- .../providers/cuda/tensor/gather_elements.h | 3 +- .../core/providers/cuda/tensor/gather_nd.cc | 4 +- .../core/providers/cuda/tensor/gather_nd.h | 4 +- .../core/providers/cuda/tensor/identity_op.cc | 12 +- .../core/providers/cuda/tensor/identity_op.h | 2 +- .../core/providers/cuda/tensor/nonzero_op.cc | 33 +- .../core/providers/cuda/tensor/nonzero_op.h | 6 +- .../core/providers/cuda/tensor/onehot.cc | 7 +- .../core/providers/cuda/tensor/onehot.h | 3 +- onnxruntime/core/providers/cuda/tensor/pad.cc | 9 +- onnxruntime/core/providers/cuda/tensor/pad.h | 4 +- .../providers/cuda/tensor/quantize_linear.cc | 6 +- .../providers/cuda/tensor/quantize_linear.cuh | 1 - .../providers/cuda/tensor/quantize_linear.h | 2 +- .../core/providers/cuda/tensor/reshape.cc | 14 +- .../core/providers/cuda/tensor/reshape.h | 2 +- .../core/providers/cuda/tensor/resize.cc | 20 +- .../core/providers/cuda/tensor/resize.h | 3 +- .../providers/cuda/tensor/reverse_sequence.cc | 4 +- .../providers/cuda/tensor/reverse_sequence.h | 2 +- .../providers/cuda/tensor/scatter_elements.cc | 7 +- .../providers/cuda/tensor/scatter_elements.h | 2 +- .../core/providers/cuda/tensor/scatter_nd.cc | 4 +- .../core/providers/cuda/tensor/scatter_nd.h | 4 +- .../core/providers/cuda/tensor/shape_op.cc | 9 +- .../core/providers/cuda/tensor/size.cc | 11 +- .../core/providers/cuda/tensor/slice.cc | 32 +- .../core/providers/cuda/tensor/slice.h | 2 +- .../core/providers/cuda/tensor/split.cc | 9 +- .../core/providers/cuda/tensor/split.h | 2 +- .../core/providers/cuda/tensor/squeeze.cc | 8 +- .../core/providers/cuda/tensor/squeeze.h | 2 +- .../core/providers/cuda/tensor/tile.cc | 8 +- onnxruntime/core/providers/cuda/tensor/tile.h | 2 +- .../core/providers/cuda/tensor/transpose.cc | 28 +- .../core/providers/cuda/tensor/transpose.h | 2 +- .../core/providers/cuda/tensor/unsqueeze.cc | 10 +- .../core/providers/cuda/tensor/unsqueeze.h | 2 +- .../core/providers/cuda/tensor/upsample.cc | 4 +- .../core/providers/cuda/tensor/upsample.h | 4 +- .../core/providers/cuda/tensor/where.cc | 2 +- .../core/providers/cuda/tensor/where.h | 2 +- .../core/providers/cuda/version_script.lds | 9 + .../openvino/openvino_provider_factory.cc | 2 +- .../openvino/ov_versions/data_ops.cc | 1522 ++++++++--------- .../providers/openvino/ov_versions/data_ops.h | 51 +- .../providers/openvino/ov_versions/utils.cc | 4 +- .../providers/openvino/ov_versions/utils.h | 2 +- .../providers/shared_library/provider_api.h | 130 +- .../provider_bridge_provider.cc | 264 ++- .../shared_library/provider_host_api.h | 2 +- .../shared_library/provider_interfaces.h | 870 ++++++++-- .../tensorrt/tensorrt_execution_provider.cc | 9 +- onnxruntime/core/session/onnxruntime_c_api.cc | 37 +- onnxruntime/core/util/math.h | 2 + onnxruntime/core/util/math_cpuonly.h | 2 + .../python/onnxruntime_pybind_state.cc | 59 +- .../test/framework/TestAllocatorManager.cc | 2 + .../framework/cuda/allocator_cuda_test.cc | 4 + .../test/framework/cuda/fence_cuda_test.cc | 3 +- .../test/framework/inference_session_test.cc | 2 + .../test/framework/memcpy_transformer_test.cc | 1 + .../test/framework/session_state_test.cc | 1 + onnxruntime/test/framework/test_utils.cc | 29 - onnxruntime/test/framework/test_utils.h | 17 +- .../test/providers/cuda/gemm_options_test.cc | 7 +- .../cuda/reduction_functions_test.cc | 3 +- .../test/providers/provider_test_utils.h | 1 + .../providers/tensorrt/tensorrt_basic_test.cc | 12 +- .../test/python/onnxruntime_test_python.py | 15 +- onnxruntime/test/util/default_providers.cc | 9 +- .../core/framework/adasum/adasum_interface.h | 33 +- .../communication/mpi/mpi_context.cc | 60 +- .../framework/communication/mpi/mpi_context.h | 100 +- .../communication/mpi/mpi_utilities.h | 5 +- .../core/framework/distributed_run_context.h | 10 +- .../orttraining/core/graph/optimizer_config.h | 4 +- .../orttraining/core/session/tensor_helper.cc | 47 +- .../core/session/training_session.cc | 11 +- orttraining/orttraining/models/bert/main.cc | 22 +- orttraining/orttraining/models/gpt2/main.cc | 42 +- orttraining/orttraining/models/mnist/main.cc | 22 +- .../orttraining/models/pipeline_poc/main.cc | 11 +- .../models/runner/training_runner.cc | 2 +- .../test/framework/slice_concatenate_test.cc | 14 +- .../test/gradient/allreduce_op_test.cc | 436 +++-- .../test/graph/gradient_graph_builder_test.cc | 177 +- .../test/model/training_runner_test.cc | 7 +- .../test/session/training_session_test.cc | 8 +- .../session/training_session_test_utils.cc | 4 +- .../session/training_session_test_utils.h | 16 +- .../cpu/reduction/reduction_ops_test.cc | 8 +- .../test/training_ops/cuda/cuda_utils_test.cc | 3 +- .../training_ops/communication_common.h | 4 +- .../training_ops/cpu/aten_ops/aten_op.cc | 10 +- .../training_ops/cpu/aten_ops/aten_op.h | 16 +- .../training_ops/cpu/loss/reduction_type.h | 2 + .../cuda/activation/activations_grad.cc | 2 +- .../cuda/activation/bias_gelu_grad.cc | 4 +- .../training_ops/cuda/aten_ops/aten_op.cc | 9 +- .../cuda/collective/adasum_kernels.cc | 20 +- .../cuda/collective/adasum_kernels.h | 24 +- .../training_ops/cuda/collective/megatron.cc | 5 +- .../cuda/collective/nccl_common.cc | 10 +- .../cuda/collective/nccl_kernels.cc | 6 +- .../cuda/communication/nccl_service.cc | 6 +- .../cuda/communication/nccl_service.h | 64 +- .../training_ops/cuda/communication/recv.cc | 20 +- .../training_ops/cuda/communication/send.cc | 14 +- .../training_ops/cuda/controlflow/group.cc | 7 +- .../training_ops/cuda/controlflow/record.cc | 6 +- .../training_ops/cuda/controlflow/wait.cc | 6 +- .../training_ops/cuda/controlflow/yield.cc | 3 +- .../cuda/cuda_training_kernels.cc | 2 +- .../training_ops/cuda/cuda_training_kernels.h | 5 +- .../training_ops/cuda/gist/gist.cc | 160 +- .../orttraining/training_ops/cuda/gist/gist.h | 2 - .../loss/softmax_cross_entropy_loss_impl.cc | 20 +- .../loss/softmax_cross_entropy_loss_impl.cu | 3 +- .../cuda/loss/softmaxcrossentropy_impl.cc | 18 +- .../cuda/loss/softmaxcrossentropy_impl.cu | 3 +- .../cuda/loss/softmaxcrossentropy_impl.h | 1 - .../training_ops/cuda/math/div_grad.cc | 16 +- .../training_ops/cuda/math/isfinite.cc | 2 +- .../cuda/math/mixed_precision_scale.cc | 31 +- .../training_ops/cuda/math/scale.cc | 30 +- .../training_ops/cuda/math/softmax_grad.cc | 78 +- .../training_ops/cuda/nn/batch_norm_grad.cc | 16 +- .../training_ops/cuda/nn/conv_grad.cc | 18 +- .../training_ops/cuda/nn/dropout_grad.cc | 22 +- .../training_ops/cuda/nn/layer_norm.cc | 22 +- .../training_ops/cuda/optimizer/adam.cc | 8 +- .../cuda/optimizer/gradient_control.cc | 6 +- .../training_ops/cuda/optimizer/lamb.cc | 8 +- .../training_ops/cuda/optimizer/sg.cc | 2 +- .../training_ops/cuda/reduction/all.cc | 16 +- .../cuda/reduction/reduction_all.cc | 21 +- .../cuda/reduction/reduction_ops.cc | 5 +- .../training_ops/cuda/tensor/concat.cc | 5 +- .../training_ops/cuda/tensor/concat.h | 3 +- .../cuda/tensor/gather_elements_grad.cc | 4 +- .../training_ops/cuda/tensor/gather_grad.cc | 13 +- .../cuda/tensor/gather_nd_grad.cc | 28 +- .../training_ops/cuda/tensor/slice_grad.cc | 12 +- .../training_ops/cuda/tensor/split.cc | 4 +- .../training_ops/cuda/tensor/view.cc | 4 +- setup.py | 3 + .../github/linux/copy_strip_binary.sh | 4 + .../github/linux/copy_strip_binary_trt.sh | 1 + .../github/linux/java_copy_strip_binary.sh | 3 + .../nuget/generate_nuspec_for_native_nuget.py | 15 +- 338 files changed, 5398 insertions(+), 3720 deletions(-) create mode 100644 onnxruntime/core/providers/cpu/tensor/concatbase.h create mode 100644 onnxruntime/core/providers/cpu/tensor/gatherbase.h create mode 100644 onnxruntime/core/providers/cpu/tensor/padbase.h create mode 100644 onnxruntime/core/providers/cuda/exported_symbols.lst create mode 100644 onnxruntime/core/providers/cuda/symbols.def create mode 100644 onnxruntime/core/providers/cuda/version_script.lds diff --git a/cmake/CMakeLists.txt b/cmake/CMakeLists.txt index 409e80cfee..f873019396 100644 --- a/cmake/CMakeLists.txt +++ b/cmake/CMakeLists.txt @@ -194,7 +194,9 @@ if(onnxruntime_USE_VALGRIND AND NOT WIN32) endif() if (onnxruntime_ENABLE_NVTX_PROFILE) - add_definitions(-DENABLE_NVTX_PROFILE=1) + message(WARNING "NTVX profile temporarily disabled, will be fixed soon") +# TODO: This doesn't work with the shared cuda provider. Disabling temporarily to do a clean fix later as it wasn't trivial +# add_definitions(-DENABLE_NVTX_PROFILE=1) endif() if (onnxruntime_ENABLE_BITCODE) diff --git a/cmake/onnxruntime.cmake b/cmake/onnxruntime.cmake index 0d2d58783d..256e07023a 100644 --- a/cmake/onnxruntime.cmake +++ b/cmake/onnxruntime.cmake @@ -135,7 +135,6 @@ target_link_libraries(onnxruntime PRIVATE ${PROVIDERS_ACL} ${PROVIDERS_ARMNN} ${PROVIDERS_COREML} - ${PROVIDERS_CUDA} ${PROVIDERS_DML} ${PROVIDERS_MIGRAPHX} ${PROVIDERS_NNAPI} diff --git a/cmake/onnxruntime_common.cmake b/cmake/onnxruntime_common.cmake index b572cc89d6..25571a4d7f 100644 --- a/cmake/onnxruntime_common.cmake +++ b/cmake/onnxruntime_common.cmake @@ -90,6 +90,11 @@ source_group(TREE ${REPO_ROOT} FILES ${onnxruntime_common_src}) onnxruntime_add_static_library(onnxruntime_common ${onnxruntime_common_src}) +if (onnxruntime_USE_CUDA) + # Some files, like the provider_bridge_ort include files that depend on cuda headers + target_include_directories(onnxruntime_common PUBLIC ${onnxruntime_CUDA_HOME}/include) +endif() + if (onnxruntime_USE_TELEMETRY) set_target_properties(onnxruntime_common PROPERTIES COMPILE_FLAGS "/FI${ONNXRUNTIME_INCLUDE_DIR}/core/platform/windows/TraceLoggingConfigPrivate.h") endif() diff --git a/cmake/onnxruntime_framework.cmake b/cmake/onnxruntime_framework.cmake index dbe083c9d5..89919d0dd1 100644 --- a/cmake/onnxruntime_framework.cmake +++ b/cmake/onnxruntime_framework.cmake @@ -33,9 +33,13 @@ if(onnxruntime_ENABLE_INSTRUMENT) endif() if(onnxruntime_USE_TENSORRT) # TODO: for now, core framework depends on CUDA. It should be moved to TensorRT EP -target_include_directories(onnxruntime_framework PRIVATE ${ONNXRUNTIME_ROOT} ${onnxruntime_CUDNN_HOME}/include PUBLIC ${CMAKE_CURRENT_BINARY_DIR} ${CMAKE_CUDA_TOOLKIT_INCLUDE_DIRECTORIES}) +target_include_directories(onnxruntime_framework PRIVATE ${ONNXRUNTIME_ROOT} ${eigen_INCLUDE_DIRS} ${onnxruntime_CUDNN_HOME}/include PUBLIC ${CMAKE_CURRENT_BINARY_DIR} ${CMAKE_CUDA_TOOLKIT_INCLUDE_DIRECTORIES}) else() -target_include_directories(onnxruntime_framework PRIVATE ${ONNXRUNTIME_ROOT} PUBLIC ${CMAKE_CURRENT_BINARY_DIR}) +target_include_directories(onnxruntime_framework PRIVATE ${ONNXRUNTIME_ROOT} ${eigen_INCLUDE_DIRS} PUBLIC ${CMAKE_CURRENT_BINARY_DIR}) +endif() +# Needed for the provider interface, as it includes training headers when training is enabled +if (onnxruntime_ENABLE_TRAINING OR onnxruntime_ENABLE_TRAINING_OPS) + target_include_directories(onnxruntime_framework PRIVATE ${ORTTRAINING_ROOT}) endif() onnxruntime_add_include_to_target(onnxruntime_framework onnxruntime_common onnx onnx_proto protobuf::libprotobuf flatbuffers) set_target_properties(onnxruntime_framework PROPERTIES FOLDER "ONNXRuntime") diff --git a/cmake/onnxruntime_java.cmake b/cmake/onnxruntime_java.cmake index 54fcdff5c1..b6f6d27caf 100644 --- a/cmake/onnxruntime_java.cmake +++ b/cmake/onnxruntime_java.cmake @@ -134,10 +134,18 @@ if (WIN32) if(NOT onnxruntime_ENABLE_STATIC_ANALYSIS) add_custom_command(TARGET onnxruntime4j_jni POST_BUILD COMMAND ${CMAKE_COMMAND} -E create_symlink $ ${JAVA_PACKAGE_LIB_DIR}/$) add_custom_command(TARGET onnxruntime4j_jni POST_BUILD COMMAND ${CMAKE_COMMAND} -E create_symlink $ ${JAVA_PACKAGE_JNI_DIR}/$) + if (onnxruntime_USE_CUDA) + add_custom_command(TARGET onnxruntime4j_jni POST_BUILD COMMAND ${CMAKE_COMMAND} -E create_symlink $ ${JAVA_PACKAGE_LIB_DIR}/$) + add_custom_command(TARGET onnxruntime4j_jni POST_BUILD COMMAND ${CMAKE_COMMAND} -E create_symlink $ ${JAVA_PACKAGE_LIB_DIR}/$) + endif() endif() else() add_custom_command(TARGET onnxruntime4j_jni POST_BUILD COMMAND ${CMAKE_COMMAND} -E create_symlink $ ${JAVA_PACKAGE_LIB_DIR}/$) add_custom_command(TARGET onnxruntime4j_jni POST_BUILD COMMAND ${CMAKE_COMMAND} -E create_symlink $ ${JAVA_PACKAGE_JNI_DIR}/$) + if (onnxruntime_USE_CUDA) + add_custom_command(TARGET onnxruntime4j_jni POST_BUILD COMMAND ${CMAKE_COMMAND} -E create_symlink $ ${JAVA_PACKAGE_LIB_DIR}/$) + add_custom_command(TARGET onnxruntime4j_jni POST_BUILD COMMAND ${CMAKE_COMMAND} -E create_symlink $ ${JAVA_PACKAGE_LIB_DIR}/$) + endif() endif() # run the build process (this copies the results back into CMAKE_CURRENT_BINARY_DIR) diff --git a/cmake/onnxruntime_providers.cmake b/cmake/onnxruntime_providers.cmake index 0707df50e9..a74633ae7d 100644 --- a/cmake/onnxruntime_providers.cmake +++ b/cmake/onnxruntime_providers.cmake @@ -226,13 +226,18 @@ if (onnxruntime_USE_CUDA) "${ONNXRUNTIME_ROOT}/core/providers/cuda/*.h" "${ONNXRUNTIME_ROOT}/core/providers/cuda/*.cc" ) + # The shared_library files are in a separate list since they use precompiled headers, and the above files have them disabled. + file(GLOB_RECURSE onnxruntime_providers_cuda_shared_srcs CONFIGURE_DEPENDS + "${ONNXRUNTIME_ROOT}/core/providers/shared_library/*.h" + "${ONNXRUNTIME_ROOT}/core/providers/shared_library/*.cc" + ) file(GLOB_RECURSE onnxruntime_providers_cuda_cu_srcs CONFIGURE_DEPENDS "${ONNXRUNTIME_ROOT}/core/providers/cuda/*.cu" "${ONNXRUNTIME_ROOT}/core/providers/cuda/*.cuh" ) - source_group(TREE ${ONNXRUNTIME_ROOT}/core FILES ${onnxruntime_providers_cuda_cc_srcs} ${onnxruntime_providers_cuda_cu_srcs}) - set(onnxruntime_providers_cuda_src ${onnxruntime_providers_cuda_cc_srcs} ${onnxruntime_providers_cuda_cu_srcs}) + source_group(TREE ${ONNXRUNTIME_ROOT}/core FILES ${onnxruntime_providers_cuda_cc_srcs} ${onnxruntime_providers_cuda_shared_srcs} ${onnxruntime_providers_cuda_cu_srcs}) + set(onnxruntime_providers_cuda_src ${onnxruntime_providers_cuda_cc_srcs} ${onnxruntime_providers_cuda_shared_srcs} ${onnxruntime_providers_cuda_cu_srcs}) # disable contrib ops conditionally if(NOT onnxruntime_DISABLE_CONTRIB_OPS) @@ -275,7 +280,7 @@ if (onnxruntime_USE_CUDA) list(APPEND onnxruntime_providers_cuda_src ${onnxruntime_cuda_training_ops_cc_srcs} ${onnxruntime_cuda_training_ops_cu_srcs}) endif() - onnxruntime_add_static_library(onnxruntime_providers_cuda ${onnxruntime_providers_cuda_src}) + onnxruntime_add_shared_library_module(onnxruntime_providers_cuda ${onnxruntime_providers_cuda_src}) #target_compile_options(onnxruntime_providers_cuda PRIVATE "$<$:SHELL:-Xcompiler \"/analyze:stacksize 131072\">") if (HAS_GUARD_CF) @@ -302,8 +307,11 @@ if (onnxruntime_USE_CUDA) onnxruntime_add_include_to_target(onnxruntime_providers_cuda onnxruntime_training) target_link_libraries(onnxruntime_providers_cuda PRIVATE onnxruntime_training) endif() - add_dependencies(onnxruntime_providers_cuda ${onnxruntime_EXTERNAL_DEPENDENCIES} ${onnxruntime_tvm_dependencies}) - target_include_directories(onnxruntime_providers_cuda PRIVATE ${ONNXRUNTIME_ROOT} ${onnxruntime_CUDNN_HOME}/include ${eigen_INCLUDE_DIRS} ${TVM_INCLUDES} PUBLIC ${CMAKE_CUDA_TOOLKIT_INCLUDE_DIRECTORIES}) + + add_dependencies(onnxruntime_providers_cuda onnxruntime_providers_shared ${onnxruntime_EXTERNAL_DEPENDENCIES} ${onnxruntime_tvm_dependencies}) + target_link_libraries(onnxruntime_providers_cuda PRIVATE cudart cublas cudnn curand cufft onnxruntime_providers_shared) + target_include_directories(onnxruntime_providers_cuda PRIVATE ${ONNXRUNTIME_ROOT} ${CMAKE_CURRENT_BINARY_DIR} ${onnxruntime_CUDNN_HOME}/include ${eigen_INCLUDE_DIRS} ${TVM_INCLUDES} PUBLIC ${CMAKE_CUDA_TOOLKIT_INCLUDE_DIRECTORIES}) + # ${CMAKE_CURRENT_BINARY_DIR} is so that #include "onnxruntime_config.h" inside tensor_shape.h is found install(DIRECTORY ${PROJECT_SOURCE_DIR}/../include/onnxruntime/core/providers/cuda DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/onnxruntime/core/providers) set_target_properties(onnxruntime_providers_cuda PROPERTIES LINKER_LANGUAGE CUDA) set_target_properties(onnxruntime_providers_cuda PROPERTIES FOLDER "ONNXRuntime") @@ -314,9 +322,13 @@ if (onnxruntime_USE_CUDA) if (onnxruntime_ENABLE_TRAINING OR onnxruntime_ENABLE_TRAINING_OPS) target_include_directories(onnxruntime_providers_cuda PRIVATE ${ORTTRAINING_ROOT} ${MPI_CXX_INCLUDE_DIRS}) + if(onnxruntime_USE_MPI) + target_link_libraries(onnxruntime_providers_cuda PRIVATE ${MPI_LIBRARIES} ${MPI_CXX_LINK_FLAGS}) + endif() if (onnxruntime_USE_NCCL) target_include_directories(onnxruntime_providers_cuda PRIVATE ${NCCL_INCLUDE_DIRS}) + target_link_libraries(onnxruntime_providers_cuda PRIVATE ${NCCL_LIBRARIES}) endif() endif() @@ -349,6 +361,24 @@ if (onnxruntime_USE_CUDA) set_target_properties(onnxruntime_providers_cuda PROPERTIES STATIC_LIBRARY_FLAGS "${onnxruntime_providers_cuda_static_library_flags}") endif() + + if(APPLE) + set_property(TARGET onnxruntime_providers_cuda APPEND_STRING PROPERTY LINK_FLAGS "-Xlinker -exported_symbols_list ${ONNXRUNTIME_ROOT}/core/providers/cuda/exported_symbols.lst") + target_link_libraries(onnxruntime_providers_cuda PRIVATE nsync_cpp) + elseif(UNIX) + set_property(TARGET onnxruntime_providers_cuda APPEND_STRING PROPERTY LINK_FLAGS "-Xlinker --version-script=${ONNXRUNTIME_ROOT}/core/providers/cuda/version_script.lds -Xlinker --gc-sections") + target_link_libraries(onnxruntime_providers_cuda PRIVATE nsync_cpp) + elseif(WIN32) + set_property(TARGET onnxruntime_providers_cuda APPEND_STRING PROPERTY LINK_FLAGS "-DEF:${ONNXRUNTIME_ROOT}/core/providers/cuda/symbols.def") + else() + message(FATAL_ERROR "onnxruntime_providers_cuda unknown platform, need to specify shared library exports for it") + endif() + + install(TARGETS onnxruntime_providers_cuda + ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR} + LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR} + RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR}) + endif() if (NOT onnxruntime_MINIMAL_BUILD AND NOT onnxruntime_EXTENDED_MINIMAL_BUILD @@ -401,6 +431,11 @@ if (onnxruntime_USE_DNNL) set_target_properties(onnxruntime_providers_dnnl PROPERTIES FOLDER "ONNXRuntime") set_target_properties(onnxruntime_providers_dnnl PROPERTIES LINKER_LANGUAGE CXX) + # Needed for the provider interface, as it includes training headers when training is enabled + if (onnxruntime_ENABLE_TRAINING OR onnxruntime_ENABLE_TRAINING_OPS) + target_include_directories(onnxruntime_providers_dnnl PRIVATE ${ORTTRAINING_ROOT}) + endif() + if(APPLE) set_property(TARGET onnxruntime_providers_dnnl APPEND_STRING PROPERTY LINK_FLAGS "-Xlinker -exported_symbols_list ${ONNXRUNTIME_ROOT}/core/providers/dnnl/exported_symbols.lst") set_target_properties(onnxruntime_providers_dnnl PROPERTIES @@ -487,6 +522,11 @@ if (onnxruntime_USE_TENSORRT) target_compile_options(onnxruntime_providers_tensorrt INTERFACE /wd4267 /wd4244 /wd4996 /wd4456) endif() + # Needed for the provider interface, as it includes training headers when training is enabled + if (onnxruntime_ENABLE_TRAINING OR onnxruntime_ENABLE_TRAINING_OPS) + target_include_directories(onnxruntime_providers_tensorrt PRIVATE ${ORTTRAINING_ROOT}) + endif() + if(APPLE) set_property(TARGET onnxruntime_providers_tensorrt APPEND_STRING PROPERTY LINK_FLAGS "-Xlinker -exported_symbols_list ${ONNXRUNTIME_ROOT}/core/providers/tensorrt/exported_symbols.lst") target_link_libraries(onnxruntime_providers_tensorrt PRIVATE nsync_cpp) @@ -603,6 +643,11 @@ if (onnxruntime_USE_OPENVINO) target_compile_options(onnxruntime_providers_openvino PUBLIC /wd4099 /wd4275 /wd4100 /wd4005 /wd4244 /wd4267) endif() + # Needed for the provider interface, as it includes training headers when training is enabled + if (onnxruntime_ENABLE_TRAINING OR onnxruntime_ENABLE_TRAINING_OPS) + target_include_directories(onnxruntime_providers_openvino PRIVATE ${ORTTRAINING_ROOT}) + endif() + if(APPLE) set_property(TARGET onnxruntime_providers_openvino APPEND_STRING PROPERTY LINK_FLAGS "-Xlinker -exported_symbols_list ${ONNXRUNTIME_ROOT}/core/providers/openvino/exported_symbols.lst") elseif(UNIX) diff --git a/cmake/onnxruntime_python.cmake b/cmake/onnxruntime_python.cmake index 7dd4076366..419fdca22c 100644 --- a/cmake/onnxruntime_python.cmake +++ b/cmake/onnxruntime_python.cmake @@ -92,7 +92,6 @@ endif() set(onnxruntime_pybind11_state_libs onnxruntime_session ${onnxruntime_libs} - ${PROVIDERS_CUDA} ${PROVIDERS_MIGRAPHX} ${PROVIDERS_NUPHAR} ${PROVIDERS_VITISAI} @@ -370,6 +369,7 @@ add_custom_command( ) if (NOT onnxruntime_MINIMAL_BUILD AND NOT onnxruntime_EXTENDED_MINIMAL_BUILD + AND NOT onnxruntime_ENABLE_TRAINING AND NOT ${CMAKE_SYSTEM_NAME} MATCHES "Darwin|iOS" AND NOT (CMAKE_SYSTEM_NAME STREQUAL "Android") AND NOT onnxruntime_BUILD_WEBASSEMBLY) @@ -457,6 +457,16 @@ if (onnxruntime_USE_OPENVINO) ) endif() +if (onnxruntime_USE_CUDA) + add_custom_command( + TARGET onnxruntime_pybind11_state POST_BUILD + COMMAND ${CMAKE_COMMAND} -E copy + $ + $ + $/onnxruntime/capi/ + ) +endif() + if (onnxruntime_USE_TVM) add_custom_command( TARGET onnxruntime_pybind11_state POST_BUILD diff --git a/cmake/onnxruntime_training.cmake b/cmake/onnxruntime_training.cmake index 69ff109478..b56b49d15b 100644 --- a/cmake/onnxruntime_training.cmake +++ b/cmake/onnxruntime_training.cmake @@ -106,7 +106,6 @@ target_include_directories(onnxruntime_training_mnist PUBLIC ${CMAKE_CURRENT_BIN set(ONNXRUNTIME_LIBS onnxruntime_session ${onnxruntime_libs} - ${PROVIDERS_CUDA} ${PROVIDERS_ROCM} ${PROVIDERS_MKLDNN} onnxruntime_optimizer diff --git a/cmake/onnxruntime_unittests.cmake b/cmake/onnxruntime_unittests.cmake index 00c52b3b56..a0d7ce76fd 100644 --- a/cmake/onnxruntime_unittests.cmake +++ b/cmake/onnxruntime_unittests.cmake @@ -396,18 +396,6 @@ endif() set (onnxruntime_test_providers_dependencies ${onnxruntime_EXTERNAL_DEPENDENCIES}) -if(onnxruntime_USE_CUDA) - list(APPEND onnxruntime_test_providers_dependencies onnxruntime_providers_cuda) -endif() - -if(onnxruntime_USE_DNNL) - list(APPEND onnxruntime_test_providers_dependencies onnxruntime_providers_dnnl onnxruntime_providers_shared) -endif() - -if(onnxruntime_USE_OPENVINO) - list(APPEND onnxruntime_test_providers_dependencies onnxruntime_providers_openvino onnxruntime_providers_shared) -endif() - if(onnxruntime_USE_NNAPI_BUILTIN) list(APPEND onnxruntime_test_providers_dependencies onnxruntime_providers_nnapi) endif() @@ -466,8 +454,7 @@ set(ONNXRUNTIME_TEST_LIBS onnxruntime_session ${ONNXRUNTIME_INTEROP_TEST_LIBS} ${onnxruntime_libs} - ${PROVIDERS_CUDA} - # TENSORRT, DNNL, and OpenVINO are explicitly linked at runtime + # CUDA, TENSORRT, DNNL, and OpenVINO are dynamically loaded at runtime ${PROVIDERS_MIGRAPHX} ${PROVIDERS_NUPHAR} ${PROVIDERS_NNAPI} @@ -1139,6 +1126,7 @@ endif() # limit to only test on windows first, due to a runtime path issue on linux if (NOT onnxruntime_MINIMAL_BUILD AND NOT onnxruntime_EXTENDED_MINIMAL_BUILD + AND NOT onnxruntime_ENABLE_TRAINING AND NOT ${CMAKE_SYSTEM_NAME} MATCHES "Darwin|iOS" AND NOT (CMAKE_SYSTEM_NAME STREQUAL "Android") AND NOT onnxruntime_BUILD_WEBASSEMBLY diff --git a/include/onnxruntime/core/framework/kernel_def_builder.h b/include/onnxruntime/core/framework/kernel_def_builder.h index 5d7c487c10..f01405e8d4 100644 --- a/include/onnxruntime/core/framework/kernel_def_builder.h +++ b/include/onnxruntime/core/framework/kernel_def_builder.h @@ -180,6 +180,8 @@ class KernelDef { class KernelDefBuilder { public: + static std::unique_ptr Create() { return std::make_unique(); } + explicit KernelDefBuilder() : kernel_def_(new KernelDef()) {} @@ -323,6 +325,13 @@ class KernelDefBuilder { return *this; } + KernelDefBuilder& InputMemoryType(OrtMemType type, const std::vector& input_indexes) { + for (auto input_index : input_indexes) { + kernel_def_->input_memory_type_args_.insert(std::make_pair(input_index, type)); + } + return *this; + } + /** Specify that this kernel provides an output arg in certain memory type (instead of the default, device memory). diff --git a/include/onnxruntime/core/framework/ml_value.h b/include/onnxruntime/core/framework/ml_value.h index 9a764a4b3b..96d6ee88a1 100644 --- a/include/onnxruntime/core/framework/ml_value.h +++ b/include/onnxruntime/core/framework/ml_value.h @@ -4,11 +4,13 @@ #pragma once #include +#ifndef SHARED_PROVIDER #include "core/common/common.h" #include "core/common/exceptions.h" #include "core/framework/allocator.h" #include "core/framework/data_types.h" #include "core/framework/tensor.h" +#endif namespace onnxruntime { class SparseTensor; diff --git a/include/onnxruntime/core/framework/op_node_proto_helper.h b/include/onnxruntime/core/framework/op_node_proto_helper.h index 7443459a15..84b1b7dfa7 100644 --- a/include/onnxruntime/core/framework/op_node_proto_helper.h +++ b/include/onnxruntime/core/framework/op_node_proto_helper.h @@ -3,9 +3,11 @@ #pragma once +#ifndef SHARED_PROVIDER #include "core/common/status.h" #include "core/graph/graph_viewer.h" #include "gsl/gsl" +#endif #ifdef __has_attribute #define ORT_HAVE_ATTRIBUTE(x) __has_attribute(x) diff --git a/include/onnxruntime/core/framework/tensor.h b/include/onnxruntime/core/framework/tensor.h index 1df19f40ce..4b09260856 100644 --- a/include/onnxruntime/core/framework/tensor.h +++ b/include/onnxruntime/core/framework/tensor.h @@ -34,6 +34,13 @@ namespace onnxruntime { */ class Tensor final { public: + static std::unique_ptr Create(MLDataType p_type, const TensorShape& shape, std::shared_ptr allocator) { + return std::make_unique(p_type, shape, allocator); + } + static std::unique_ptr Create(MLDataType p_type, const TensorShape& shape, void* p_data, const OrtMemoryInfo& alloc, ptrdiff_t offset = 0) { + return std::make_unique(p_type, shape, p_data, alloc, offset); + } + Tensor() = default; // to allow creating vector to support seq(tensor) /** @@ -235,4 +242,4 @@ class Tensor final { #ifdef __GNUC__ #pragma GCC diagnostic pop #endif -} // namespace onnxruntime \ No newline at end of file +} // namespace onnxruntime diff --git a/include/onnxruntime/core/providers/cuda/cuda_provider_factory.h b/include/onnxruntime/core/providers/cuda/cuda_provider_factory.h index 81b5477b3c..6fcf582684 100644 --- a/include/onnxruntime/core/providers/cuda/cuda_provider_factory.h +++ b/include/onnxruntime/core/providers/cuda/cuda_provider_factory.h @@ -4,6 +4,50 @@ #include "onnxruntime_c_api.h" #ifdef __cplusplus +#include "core/framework/provider_options.h" + +namespace onnxruntime { +class IAllocator; +class IDataTransfer; +struct IExecutionProviderFactory; +struct CUDAExecutionProviderInfo; +enum class ArenaExtendStrategy : int32_t; +struct CUDAExecutionProviderExternalAllocatorInfo; + +namespace cuda { +class INcclService; +} + +} // namespace onnxruntime + +struct ProviderInfo_CUDA { + virtual OrtStatus* SetCurrentGpuDeviceId(_In_ int device_id) = 0; + virtual OrtStatus* GetCurrentGpuDeviceId(_In_ int* device_id) = 0; + + virtual std::unique_ptr CreateCUDAAllocator(int16_t device_id, const char* name) = 0; + virtual std::unique_ptr CreateCUDAPinnedAllocator(int16_t device_id, const char* name) = 0; + virtual std::unique_ptr CreateGPUDataTransfer(void* stream) = 0; + + virtual void cuda__Impl_Cast(void* stream, const int64_t* input_data, int32_t* output_data, size_t count) = 0; + virtual void cuda__Impl_Cast(void* stream, const int32_t* input_data, int64_t* output_data, size_t count) = 0; + + virtual bool CudaCall_false(int retCode, const char* exprString, const char* libName, int successCode, const char* msg) = 0; + virtual bool CudaCall_true(int retCode, const char* exprString, const char* libName, int successCode, const char* msg) = 0; + + virtual void CopyGpuToCpu(void* dst_ptr, const void* src_ptr, const size_t size, const OrtMemoryInfo& dst_location, const OrtMemoryInfo& src_location) = 0; + virtual void cudaMemcpy_HostToDevice(void* dst, const void* src, size_t count) = 0; + virtual void cudaMemcpy_DeviceToHost(void* dst, const void* src, size_t count) = 0; + virtual int cudaGetDeviceCount() = 0; + virtual void CUDAExecutionProviderInfo__FromProviderOptions(const onnxruntime::ProviderOptions& options, onnxruntime::CUDAExecutionProviderInfo& info) = 0; + +#if defined(USE_CUDA) && defined(ORT_USE_NCCL) && defined(USE_NCCL_P2P) + virtual onnxruntime::cuda::INcclService& GetINcclService() = 0; +#endif + + virtual std::shared_ptr CreateExecutionProviderFactory(const onnxruntime::CUDAExecutionProviderInfo& info) = 0; + virtual std::shared_ptr CreateCudaAllocator(int16_t device_id, size_t gpu_mem_limit, onnxruntime::ArenaExtendStrategy arena_extend_strategy, onnxruntime::CUDAExecutionProviderExternalAllocatorInfo& external_allocator_info, OrtArenaCfg* default_memory_arena_cfg) = 0; +}; + extern "C" { #endif diff --git a/java/src/main/java/ai/onnxruntime/OnnxRuntime.java b/java/src/main/java/ai/onnxruntime/OnnxRuntime.java index bee63f566d..4e6db62ace 100644 --- a/java/src/main/java/ai/onnxruntime/OnnxRuntime.java +++ b/java/src/main/java/ai/onnxruntime/OnnxRuntime.java @@ -36,6 +36,11 @@ final class OnnxRuntime { /** The short name of the ONNX runtime JNI shared library */ static final String ONNXRUNTIME_JNI_LIBRARY_NAME = "onnxruntime4j_jni"; + /** The short name of the ONNX runtime shared provider library */ + static final String ONNXRUNTIME_LIBRARY_SHARED_NAME = "onnxruntime_providers_shared"; + /** The short name of the ONNX runtime cuda provider library */ + static final String ONNXRUNTIME_LIBRARY_CUDA_NAME = "onnxruntime_providers_cuda"; + private static final String OS_ARCH_STR = initOsArch(); private static boolean loaded = false; @@ -93,14 +98,19 @@ final class OnnxRuntime { } Path tempDirectory = isAndroid() ? null : Files.createTempDirectory("onnxruntime-java"); try { - load(tempDirectory, ONNXRUNTIME_LIBRARY_NAME); - load(tempDirectory, ONNXRUNTIME_JNI_LIBRARY_NAME); + // Extract and prepare the shared provider libraries but don't try to load them, Onnxruntime + // itself will load them + load(tempDirectory, ONNXRUNTIME_LIBRARY_SHARED_NAME, false); + load(tempDirectory, ONNXRUNTIME_LIBRARY_CUDA_NAME, false); + + load(tempDirectory, ONNXRUNTIME_LIBRARY_NAME, true); + load(tempDirectory, ONNXRUNTIME_JNI_LIBRARY_NAME, true); ortApiHandle = initialiseAPIBase(ORT_API_VERSION_7); providers = initialiseProviders(ortApiHandle); loaded = true; } finally { if (tempDirectory != null) { - cleanUp(tempDirectory.toFile()); + cleanUp(tempDirectory.toFile(), false); } } } @@ -110,13 +120,14 @@ final class OnnxRuntime { * in time. * * @param file The file to remove. + * @param onExitOnly Delete the file on exit only, vs trying to do it immediately */ - private static void cleanUp(File file) { + private static void cleanUp(File file, boolean onExitOnly) { if (!file.exists()) { return; } logger.log(Level.FINE, "Deleting " + file); - if (!file.delete()) { + if (onExitOnly || !file.delete()) { logger.log(Level.FINE, "Deleting " + file + " on exit"); file.deleteOnExit(); } @@ -136,9 +147,11 @@ final class OnnxRuntime { * * @param tempDirectory The temp directory to write the library resource to. * @param library The bare name of the library. + * @param systemLoad If system.Load(..) should be called on the library vs just preparing it * @throws IOException If the file failed to read or write. */ - private static void load(Path tempDirectory, String library) throws IOException { + private static void load(Path tempDirectory, String library, boolean systemLoad) + throws IOException { // On Android, we simply use System.loadLibrary if (isAndroid()) { System.loadLibrary("onnxruntime4j_jni"); @@ -180,6 +193,9 @@ final class OnnxRuntime { try (InputStream is = OnnxRuntime.class.getResourceAsStream(resourcePath)) { if (is == null) { // 3a) Not found in resources, load from library path + if (!systemLoad) { + return; // Failure is expected for optional components we don't need to load + } logger.log( Level.FINE, "Attempting to load native library '" + library + "' from library path"); System.loadLibrary(library); @@ -201,11 +217,15 @@ final class OnnxRuntime { os.write(buffer, 0, readBytes); } } - System.load(tempFile.getAbsolutePath()); - logger.log(Level.FINE, "Loaded native library '" + library + "' from resource path"); + if (systemLoad) { + System.load(tempFile.getAbsolutePath()); + logger.log(Level.FINE, "Loaded native library '" + library + "' from resource path"); + } else { + logger.log(Level.FINE, "Extracted native library '" + library + "' from resource path"); + } } } finally { - cleanUp(tempFile); + cleanUp(tempFile, !systemLoad); } } diff --git a/onnxruntime/contrib_ops/cpu/bert/attention.cc b/onnxruntime/contrib_ops/cpu/bert/attention.cc index d992eb6143..a1d6889c6c 100644 --- a/onnxruntime/contrib_ops/cpu/bert/attention.cc +++ b/onnxruntime/contrib_ops/cpu/bert/attention.cc @@ -47,14 +47,6 @@ ONNX_OPERATOR_TYPED_KERNEL_EX( .TypeConstraint("T", DataTypeImpl::GetTensorType()), Attention); -AttentionBase::AttentionBase(const OpKernelInfo& info) { - int64_t num_heads = 0; - ORT_ENFORCE(info.GetAttr("num_heads", &num_heads).IsOK() && num_heads > 0); - num_heads_ = static_cast(num_heads); - - is_unidirectional_ = info.GetAttrOrDefault("unidirectional", 0) == 1; -} - Status AttentionBase::CheckInputs(const TensorShape& input_shape, const TensorShape& weights_shape, const TensorShape& bias_shape, diff --git a/onnxruntime/contrib_ops/cpu/bert/attention_base.h b/onnxruntime/contrib_ops/cpu/bert/attention_base.h index 3421f14999..f14268af3e 100644 --- a/onnxruntime/contrib_ops/cpu/bert/attention_base.h +++ b/onnxruntime/contrib_ops/cpu/bert/attention_base.h @@ -10,15 +10,7 @@ namespace onnxruntime { namespace contrib { class AttentionBase { - protected: - AttentionBase(const OpKernelInfo& info); - - Status CheckInputs(const TensorShape& input_shape, - const TensorShape& weights_shape, - const TensorShape& bias_shape, - const Tensor*& mask_index, // For dummy mask with shape (1, 1) or (batch_size, 1), it will be updated to nullptr. - const Tensor* past) const; - + public: // This check function is specifically used in cuda Status CheckInputs(const TensorShape& input_shape, const TensorShape& weights_shape, @@ -34,6 +26,21 @@ class AttentionBase { int sequence_length, int& past_sequence_length) const; + protected: + AttentionBase(const OpKernelInfo& info) { + int64_t num_heads = 0; + ORT_ENFORCE(info.GetAttr("num_heads", &num_heads).IsOK() && num_heads > 0); + num_heads_ = static_cast(num_heads); + + is_unidirectional_ = info.GetAttrOrDefault("unidirectional", 0) == 1; + } + + Status CheckInputs(const TensorShape& input_shape, + const TensorShape& weights_shape, + const TensorShape& bias_shape, + const Tensor*& mask_index, // For dummy mask with shape (1, 1) or (batch_size, 1), it will be updated to nullptr. + const Tensor* past) const; + int num_heads_; // number of attention heads bool is_unidirectional_; // whether every token can only attend to previous tokens. }; diff --git a/onnxruntime/contrib_ops/cpu/bert/embed_layer_norm_helper.cc b/onnxruntime/contrib_ops/cpu/bert/embed_layer_norm_helper.cc index 6dd13a056c..e991502745 100644 --- a/onnxruntime/contrib_ops/cpu/bert/embed_layer_norm_helper.cc +++ b/onnxruntime/contrib_ops/cpu/bert/embed_layer_norm_helper.cc @@ -5,19 +5,33 @@ #include "core/framework/tensorprotoutils.h" #include "onnx/defs/tensor_proto_util.h" +#include "longformer_attention_base.h" + namespace onnxruntime { namespace contrib { + +Status LongformerAttentionBase__CheckInputs(const LongformerAttentionBase* p, + const TensorShape& input_shape, + const TensorShape& weights_shape, + const TensorShape& bias_shape, + const TensorShape& mask_shape, + const TensorShape& global_weights_shape, + const TensorShape& global_bias_shape, + const TensorShape& global_shape) { + return p->CheckInputs(input_shape, weights_shape, bias_shape, mask_shape, global_weights_shape, global_bias_shape, global_shape); +} + namespace embed_layer_norm { Status CheckInputs(const OpKernelContext* context) { const Tensor* input_ids = context->Input(0); - const Tensor* segment_ids = context->Input(1); // optional. nullptr if it's distill-bert + const Tensor* segment_ids = context->Input(1); // optional. nullptr if it's distill-bert const Tensor* word_embedding = context->Input(2); const Tensor* position_embedding = context->Input(3); - const Tensor* segment_embedding = context->Input(4); // optional. nullptr if it's distill-bert + const Tensor* segment_embedding = context->Input(4); // optional. nullptr if it's distill-bert const Tensor* gamma = context->Input(5); const Tensor* beta = context->Input(6); - const Tensor* mask = context->Input(7); // optional. nullptr if not provided + const Tensor* mask = context->Input(7); // optional. nullptr if not provided if (nullptr != segment_ids && input_ids->Shape() != segment_ids->Shape()) { return ORT_MAKE_STATUS(ONNXRUNTIME, INVALID_ARGUMENT, @@ -29,7 +43,6 @@ Status CheckInputs(const OpKernelContext* context) { "Input 0 and 7 (mask) shall have same shape"); } - const auto& input_dims = input_ids->Shape().GetDims(); if (input_dims.size() != 2) { return ORT_MAKE_STATUS(ONNXRUNTIME, INVALID_ARGUMENT, diff --git a/onnxruntime/contrib_ops/cpu/bert/embed_layer_norm_helper.h b/onnxruntime/contrib_ops/cpu/bert/embed_layer_norm_helper.h index 09eb6b4ddf..067cbfafab 100644 --- a/onnxruntime/contrib_ops/cpu/bert/embed_layer_norm_helper.h +++ b/onnxruntime/contrib_ops/cpu/bert/embed_layer_norm_helper.h @@ -3,8 +3,10 @@ #pragma once +#ifndef SHARED_PROVIDER #include "core/common/common.h" #include "core/framework/op_kernel.h" +#endif namespace onnxruntime { namespace contrib { diff --git a/onnxruntime/contrib_ops/cpu/bert/longformer_attention_base.cc b/onnxruntime/contrib_ops/cpu/bert/longformer_attention_base.cc index 385653991f..6a6685ac3f 100644 --- a/onnxruntime/contrib_ops/cpu/bert/longformer_attention_base.cc +++ b/onnxruntime/contrib_ops/cpu/bert/longformer_attention_base.cc @@ -6,16 +6,6 @@ namespace onnxruntime { namespace contrib { -LongformerAttentionBase::LongformerAttentionBase(const OpKernelInfo& info) { - int64_t num_heads = 0; - ORT_ENFORCE(info.GetAttr("num_heads", &num_heads).IsOK() && num_heads > 0); - num_heads_ = static_cast(num_heads); - - int64_t window = 0; - ORT_ENFORCE(info.GetAttr("window", &window).IsOK() && window > 0); - window_ = static_cast(window); -} - Status LongformerAttentionBase::CheckInputs(const TensorShape& input_shape, const TensorShape& weights_shape, const TensorShape& bias_shape, diff --git a/onnxruntime/contrib_ops/cpu/bert/longformer_attention_base.h b/onnxruntime/contrib_ops/cpu/bert/longformer_attention_base.h index 00cc451073..1f974db5cd 100644 --- a/onnxruntime/contrib_ops/cpu/bert/longformer_attention_base.h +++ b/onnxruntime/contrib_ops/cpu/bert/longformer_attention_base.h @@ -10,9 +10,7 @@ namespace onnxruntime { namespace contrib { class LongformerAttentionBase { - protected: - LongformerAttentionBase(const OpKernelInfo& info); - + public: Status CheckInputs(const TensorShape& input_shape, const TensorShape& weights_shape, const TensorShape& bias_shape, @@ -21,6 +19,17 @@ class LongformerAttentionBase { const TensorShape& global_bias_shape, const TensorShape& global_shape) const; + protected: + LongformerAttentionBase(const OpKernelInfo& info) { + int64_t num_heads = 0; + ORT_ENFORCE(info.GetAttr("num_heads", &num_heads).IsOK() && num_heads > 0); + num_heads_ = static_cast(num_heads); + + int64_t window = 0; + ORT_ENFORCE(info.GetAttr("window", &window).IsOK() && window > 0); + window_ = static_cast(window); + } + int num_heads_; // Number of attention heads int window_; // Attention windows length (W). It is half (one-sided) of total window size. }; diff --git a/onnxruntime/contrib_ops/cuda/activation/activations.cc b/onnxruntime/contrib_ops/cuda/activation/activations.cc index 6a26e0f6c3..e018af8340 100644 --- a/onnxruntime/contrib_ops/cuda/activation/activations.cc +++ b/onnxruntime/contrib_ops/cuda/activation/activations.cc @@ -17,7 +17,7 @@ namespace cuda { ver, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()) \ .MayInplace(0, 0), \ x); diff --git a/onnxruntime/contrib_ops/cuda/bert/attention.cc b/onnxruntime/contrib_ops/cuda/bert/attention.cc index e9f73d3fe0..733b98ea1c 100644 --- a/onnxruntime/contrib_ops/cuda/bert/attention.cc +++ b/onnxruntime/contrib_ops/cuda/bert/attention.cc @@ -2,10 +2,9 @@ // Licensed under the MIT License. #include "attention.h" -#include "core/framework/tensorprotoutils.h" +#include "attention_impl.h" #include "core/providers/cuda/cuda_common.h" #include "core/providers/cuda/shared_inc/fpgeneric.h" -#include "attention_impl.h" using namespace onnxruntime::cuda; using namespace ::onnxruntime::common; @@ -22,7 +21,7 @@ namespace cuda { 1, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()), \ Attention); diff --git a/onnxruntime/contrib_ops/cuda/bert/attention.h b/onnxruntime/contrib_ops/cuda/bert/attention.h index 97eb60d291..2aec22cc87 100644 --- a/onnxruntime/contrib_ops/cuda/bert/attention.h +++ b/onnxruntime/contrib_ops/cuda/bert/attention.h @@ -3,7 +3,6 @@ #pragma once -#include "core/common/common.h" #include "core/providers/cuda/cuda_kernel.h" #include "contrib_ops/cpu/bert/attention_base.h" diff --git a/onnxruntime/contrib_ops/cuda/bert/embed_layer_norm.cc b/onnxruntime/contrib_ops/cuda/bert/embed_layer_norm.cc index e975181d29..0fd880b87b 100644 --- a/onnxruntime/contrib_ops/cuda/bert/embed_layer_norm.cc +++ b/onnxruntime/contrib_ops/cuda/bert/embed_layer_norm.cc @@ -1,9 +1,7 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. -#include "core/providers/common.h" -#include "core/framework/tensorprotoutils.h" -#include "onnx/defs/tensor_proto_util.h" +#include "core/providers/cuda/cuda_common.h" #include "contrib_ops/cpu/bert/embed_layer_norm_helper.h" #include "embed_layer_norm.h" #include "embed_layer_norm_impl.h" @@ -19,7 +17,7 @@ namespace cuda { 1, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()), \ EmbedLayerNorm); diff --git a/onnxruntime/contrib_ops/cuda/bert/embed_layer_norm.h b/onnxruntime/contrib_ops/cuda/bert/embed_layer_norm.h index 36865640aa..8749dc925e 100644 --- a/onnxruntime/contrib_ops/cuda/bert/embed_layer_norm.h +++ b/onnxruntime/contrib_ops/cuda/bert/embed_layer_norm.h @@ -2,7 +2,6 @@ // Licensed under the MIT License. #pragma once -#include "core/common/common.h" #include "core/providers/cuda/cuda_kernel.h" namespace onnxruntime { @@ -16,6 +15,7 @@ class EmbedLayerNorm final : public CudaKernel { public: EmbedLayerNorm(const OpKernelInfo& op_kernel_info); Status ComputeInternal(OpKernelContext* ctx) const override; + private: float epsilon_; }; diff --git a/onnxruntime/contrib_ops/cuda/bert/fast_gelu.cc b/onnxruntime/contrib_ops/cuda/bert/fast_gelu.cc index 8e4bfb1c84..a45b5509b5 100644 --- a/onnxruntime/contrib_ops/cuda/bert/fast_gelu.cc +++ b/onnxruntime/contrib_ops/cuda/bert/fast_gelu.cc @@ -1,9 +1,8 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. -#include "core/providers/common.h" +#include "core/providers/cuda/cuda_common.h" #include "core/providers/cuda/cudnn_common.h" -#include "core/framework/tensorprotoutils.h" #include "fast_gelu.h" #include "fast_gelu_impl.h" #include "contrib_ops/cpu/bert/bias_gelu_helper.h" @@ -19,7 +18,7 @@ namespace cuda { 1, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()), \ FastGelu); diff --git a/onnxruntime/contrib_ops/cuda/bert/longformer_attention.cc b/onnxruntime/contrib_ops/cuda/bert/longformer_attention.cc index 19e838971a..c21adcfc72 100644 --- a/onnxruntime/contrib_ops/cuda/bert/longformer_attention.cc +++ b/onnxruntime/contrib_ops/cuda/bert/longformer_attention.cc @@ -1,11 +1,10 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. -#include "longformer_attention.h" -#include "core/framework/tensorprotoutils.h" #include "core/providers/cuda/cuda_common.h" #include "core/providers/cuda/shared_inc/fpgeneric.h" #include "core/platform/env_var_utils.h" +#include "longformer_attention.h" #include "longformer_global_impl.h" #include "longformer_attention_impl.h" @@ -24,7 +23,7 @@ namespace cuda { 1, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()), \ LongformerAttention); @@ -45,6 +44,7 @@ class AutoDestoryCudaEvent { cudaEvent_t& Get() { return cuda_event_; } + private: cudaEvent_t cuda_event_; }; diff --git a/onnxruntime/contrib_ops/cuda/bert/skip_layer_norm.cc b/onnxruntime/contrib_ops/cuda/bert/skip_layer_norm.cc index da2e23b103..dd975ca90f 100644 --- a/onnxruntime/contrib_ops/cuda/bert/skip_layer_norm.cc +++ b/onnxruntime/contrib_ops/cuda/bert/skip_layer_norm.cc @@ -1,12 +1,9 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. -#include "core/providers/common.h" #include "core/providers/cuda/cuda_common.h" -#include "core/framework/tensorprotoutils.h" -#include "onnx/defs/tensor_proto_util.h" -#include "contrib_ops/cuda/bert/skip_layer_norm.h" -#include "contrib_ops/cuda/bert/skip_layer_norm_impl.h" +#include "skip_layer_norm.h" +#include "skip_layer_norm_impl.h" namespace onnxruntime { namespace contrib { @@ -19,7 +16,7 @@ namespace cuda { 1, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()), \ SkipLayerNorm); diff --git a/onnxruntime/contrib_ops/cuda/conv_transpose_with_dynamic_pads.cc b/onnxruntime/contrib_ops/cuda/conv_transpose_with_dynamic_pads.cc index e526c23e64..c2116e1d8e 100644 --- a/onnxruntime/contrib_ops/cuda/conv_transpose_with_dynamic_pads.cc +++ b/onnxruntime/contrib_ops/cuda/conv_transpose_with_dynamic_pads.cc @@ -12,7 +12,9 @@ ONNX_OPERATOR_TYPED_KERNEL_EX( 1, float, kCudaExecutionProvider, - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()).InputMemoryType(2), + (*KernelDefBuilder::Create()) + .TypeConstraint("T", DataTypeImpl::GetTensorType()) + .InputMemoryType(OrtMemTypeCPUInput, 2), ConvTransposeWithDynamicPads); } // namespace cuda } // namespace contrib diff --git a/onnxruntime/contrib_ops/cuda/cuda_contrib_kernels.cc b/onnxruntime/contrib_ops/cuda/cuda_contrib_kernels.cc index 26fd54bc8c..64288cab64 100644 --- a/onnxruntime/contrib_ops/cuda/cuda_contrib_kernels.cc +++ b/onnxruntime/contrib_ops/cuda/cuda_contrib_kernels.cc @@ -1,8 +1,8 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. +#include "core/providers/shared_library/provider_api.h" #include "core/providers/cuda/cuda_common.h" -#include "core/framework/kernel_registry.h" using namespace onnxruntime::common; @@ -17,9 +17,9 @@ class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kMSDomain, 1 class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kMSDomain, 1, float, BiasGelu); class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kMSDomain, 1, double, BiasGelu); class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kMSDomain, 1, MLFloat16, BiasGelu); -class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kMSDomain, 1, float, TransposeMatMul); // backward compatibility -class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kMSDomain, 1, double, TransposeMatMul); // backward compatibility -class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kMSDomain, 1, MLFloat16, TransposeMatMul); // backward compatibility +class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kMSDomain, 1, float, TransposeMatMul); // backward compatibility +class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kMSDomain, 1, double, TransposeMatMul); // backward compatibility +class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kMSDomain, 1, MLFloat16, TransposeMatMul); // backward compatibility class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kMSDomain, 1, float, FusedMatMul); class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kMSDomain, 1, double, FusedMatMul); class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kMSDomain, 1, MLFloat16, FusedMatMul); @@ -84,7 +84,7 @@ class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kMSDomain, 1 #if defined(CUDA_VERSION) && CUDA_VERSION >= 11000 class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kMSDomain, 1, BFloat16, FastGelu); -class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kMSDomain, 1, BFloat16, TransposeMatMul); // backward compatibility +class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kMSDomain, 1, BFloat16, TransposeMatMul); // backward compatibility class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kMSDomain, 1, BFloat16, FusedMatMul); class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, 1, BFloat16_float, LayerNormalization); #endif @@ -97,31 +97,31 @@ KernelCreateInfo BuildKernelCreateInfo() { Status RegisterCudaContribKernels(KernelRegistry& kernel_registry) { static const BuildKernelCreateInfoFn function_table[] = { - BuildKernelCreateInfo, //default entry to avoid the list become empty after ops-reducing - BuildKernelCreateInfo, - BuildKernelCreateInfo, - BuildKernelCreateInfo, - BuildKernelCreateInfo, - BuildKernelCreateInfo, - BuildKernelCreateInfo, - BuildKernelCreateInfo, - BuildKernelCreateInfo, - BuildKernelCreateInfo, // backward compatibility - BuildKernelCreateInfo, // backward compatibility - BuildKernelCreateInfo, // backward compatibility - BuildKernelCreateInfo, - BuildKernelCreateInfo, - BuildKernelCreateInfo, - BuildKernelCreateInfo, - BuildKernelCreateInfo, - BuildKernelCreateInfo, - BuildKernelCreateInfo, - BuildKernelCreateInfo, - BuildKernelCreateInfo, - BuildKernelCreateInfo, - BuildKernelCreateInfo, - BuildKernelCreateInfo, - BuildKernelCreateInfo, + BuildKernelCreateInfo, //default entry to avoid the list become empty after ops-reducing + BuildKernelCreateInfo, + BuildKernelCreateInfo, + BuildKernelCreateInfo, + BuildKernelCreateInfo, + BuildKernelCreateInfo, + BuildKernelCreateInfo, + BuildKernelCreateInfo, + BuildKernelCreateInfo, + BuildKernelCreateInfo, // backward compatibility + BuildKernelCreateInfo, // backward compatibility + BuildKernelCreateInfo, // backward compatibility + BuildKernelCreateInfo, + BuildKernelCreateInfo, + BuildKernelCreateInfo, + BuildKernelCreateInfo, + BuildKernelCreateInfo, + BuildKernelCreateInfo, + BuildKernelCreateInfo, + BuildKernelCreateInfo, + BuildKernelCreateInfo, + BuildKernelCreateInfo, + BuildKernelCreateInfo, + BuildKernelCreateInfo, + BuildKernelCreateInfo, // These ops were experimental ops in onnx domain which have been removed now. We add them here as // contrib ops to maintain backward compatibility @@ -164,21 +164,21 @@ Status RegisterCudaContribKernels(KernelRegistry& kernel_registry) { BuildKernelCreateInfo, BuildKernelCreateInfo, - BuildKernelCreateInfo, - BuildKernelCreateInfo, - BuildKernelCreateInfo, - BuildKernelCreateInfo, - BuildKernelCreateInfo, - BuildKernelCreateInfo, + BuildKernelCreateInfo, + BuildKernelCreateInfo, + BuildKernelCreateInfo, + BuildKernelCreateInfo, + BuildKernelCreateInfo, + BuildKernelCreateInfo, #if defined(CUDA_VERSION) && CUDA_VERSION >= 11000 - BuildKernelCreateInfo, - // TransposedMatMul is still here for backward compatibility - BuildKernelCreateInfo, // backward compatibility - BuildKernelCreateInfo, - BuildKernelCreateInfo, + BuildKernelCreateInfo, + // TransposedMatMul is still here for backward compatibility + BuildKernelCreateInfo, // backward compatibility + BuildKernelCreateInfo, + BuildKernelCreateInfo, #endif - BuildKernelCreateInfo, + BuildKernelCreateInfo, }; for (auto& function_table_entry : function_table) { diff --git a/onnxruntime/contrib_ops/cuda/cuda_contrib_kernels.h b/onnxruntime/contrib_ops/cuda/cuda_contrib_kernels.h index ef2f5866ab..d047e8fc78 100644 --- a/onnxruntime/contrib_ops/cuda/cuda_contrib_kernels.h +++ b/onnxruntime/contrib_ops/cuda/cuda_contrib_kernels.h @@ -2,7 +2,6 @@ // Licensed under the MIT License. #pragma once -#include "core/framework/kernel_registry.h" namespace onnxruntime { namespace contrib { @@ -12,4 +11,4 @@ Status RegisterCudaContribKernels(KernelRegistry& kernel_registry); } // namespace cuda } // namespace contrib -} // namespace onnxruntime \ No newline at end of file +} // namespace onnxruntime diff --git a/onnxruntime/contrib_ops/cuda/fused_conv.cc b/onnxruntime/contrib_ops/cuda/fused_conv.cc index a45e784d8d..02fd74061a 100644 --- a/onnxruntime/contrib_ops/cuda/fused_conv.cc +++ b/onnxruntime/contrib_ops/cuda/fused_conv.cc @@ -119,9 +119,9 @@ ONNX_OPERATOR_TYPED_KERNEL_EX( 1, float, kCudaExecutionProvider, - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), FusedConv); } // namespace cuda } // namespace contrib -} // namespace onnxruntime \ No newline at end of file +} // namespace onnxruntime diff --git a/onnxruntime/contrib_ops/cuda/inverse.cc b/onnxruntime/contrib_ops/cuda/inverse.cc index d9682c9b67..b0560c874a 100644 --- a/onnxruntime/contrib_ops/cuda/inverse.cc +++ b/onnxruntime/contrib_ops/cuda/inverse.cc @@ -28,7 +28,7 @@ ONNX_OPERATOR_KERNEL_EX( kMSDomain, 1, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .TypeConstraint("T", BuildKernelDefConstraints()), Inverse); diff --git a/onnxruntime/contrib_ops/cuda/layer_norm.cc b/onnxruntime/contrib_ops/cuda/layer_norm.cc index bbd292ffa9..50cb1c735f 100644 --- a/onnxruntime/contrib_ops/cuda/layer_norm.cc +++ b/onnxruntime/contrib_ops/cuda/layer_norm.cc @@ -1,10 +1,9 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. +#include "core/providers/shared_library/provider_api.h" #include "layer_norm.h" #include "layer_norm_impl.h" - -#include "core/providers/common.h" #include "core/providers/cuda/cuda_common.h" namespace onnxruntime { @@ -18,7 +17,7 @@ namespace cuda { 1, \ T##_##U, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()) \ .TypeConstraint("U", DataTypeImpl::GetTensorType()), \ LayerNorm); \ @@ -28,7 +27,7 @@ namespace cuda { 1, \ T##_##U, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()) \ .TypeConstraint("U", DataTypeImpl::GetTensorType()), \ LayerNorm); @@ -59,7 +58,7 @@ Status LayerNorm::ComputeInternal(OpKernelContext* ctx) const auto X_data = reinterpret_cast(X->template Data()); auto scale_data = reinterpret_cast(scale->template Data()); - auto bias_data = (simplified || (nullptr == bias)) ? nullptr: reinterpret_cast(bias->template Data()); + auto bias_data = (simplified || (nullptr == bias)) ? nullptr : reinterpret_cast(bias->template Data()); const TensorShape& x_shape = X->Shape(); const int64_t axis = HandleNegativeAxis(axis_, x_shape.NumDimensions()); diff --git a/onnxruntime/contrib_ops/cuda/layer_norm.h b/onnxruntime/contrib_ops/cuda/layer_norm.h index 4f4d95fc99..5d6dd6ff1f 100644 --- a/onnxruntime/contrib_ops/cuda/layer_norm.h +++ b/onnxruntime/contrib_ops/cuda/layer_norm.h @@ -2,7 +2,6 @@ // Licensed under the MIT License. #pragma once -#include "core/common/common.h" #include "core/providers/cuda/cuda_kernel.h" namespace onnxruntime { diff --git a/onnxruntime/contrib_ops/cuda/math/bias_dropout.cc b/onnxruntime/contrib_ops/cuda/math/bias_dropout.cc index a13974ad99..950106d879 100644 --- a/onnxruntime/contrib_ops/cuda/math/bias_dropout.cc +++ b/onnxruntime/contrib_ops/cuda/math/bias_dropout.cc @@ -16,12 +16,12 @@ ONNX_OPERATOR_KERNEL_EX( kMSDomain, 1, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .TypeConstraint("T", ALL_IEEE_FLOAT_TENSOR_TYPES) .TypeConstraint("T1", ALL_IEEE_FLOAT_TENSOR_TYPES) .TypeConstraint("T2", DataTypeImpl::GetTensorType()) - .InputMemoryType(3) - .InputMemoryType(4), + .InputMemoryType(OrtMemTypeCPUInput, 3) + .InputMemoryType(OrtMemTypeCPUInput, 4), BiasDropout); template diff --git a/onnxruntime/contrib_ops/cuda/math/bias_softmax.cc b/onnxruntime/contrib_ops/cuda/math/bias_softmax.cc index 9a09761a03..05bbd57d9f 100644 --- a/onnxruntime/contrib_ops/cuda/math/bias_softmax.cc +++ b/onnxruntime/contrib_ops/cuda/math/bias_softmax.cc @@ -1,10 +1,9 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. +#include "core/providers/cuda/cuda_common.h" #include "contrib_ops/cuda/math/bias_softmax.h" -#include "core/providers/common.h" - using namespace onnxruntime; using namespace onnxruntime::cuda; using namespace onnxruntime::contrib::cuda; @@ -43,7 +42,7 @@ ONNX_OPERATOR_KERNEL_EX( kMSDomain, 1, kCudaExecutionProvider, - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::AllIEEEFloatTensorTypes()), + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::AllIEEEFloatTensorTypes()), BiasSoftmax); Status BiasSoftmax::ComputeInternal(OpKernelContext* ctx) const { @@ -77,52 +76,52 @@ Status BiasSoftmax::ComputeInternal(OpKernelContext* ctx) const { template void DispatchBiasSoftmaxForward::operator()( - cudaStream_t stream, - Tensor* output, - const Tensor* input, - const Tensor* input_bias, - int element_count, - int batch_count, - int batch_stride, - int bias_broadcast_size_per_batch) { - DispatchBiasSoftmaxForwardImpl( - stream, - output, - input, - input_bias, - element_count, - batch_count, - batch_stride, - bias_broadcast_size_per_batch); + cudaStream_t stream, + Tensor* output, + const Tensor* input, + const Tensor* input_bias, + int element_count, + int batch_count, + int batch_stride, + int bias_broadcast_size_per_batch) { + DispatchBiasSoftmaxForwardImpl( + stream, + output, + input, + input_bias, + element_count, + batch_count, + batch_stride, + bias_broadcast_size_per_batch); } template void DispatchBiasSoftMaxForwardViaDnnLibrary::operator()( - cudaStream_t stream, - cudnnHandle_t cudaDnnHandle, - int element_count, - int batch_count, - int broadcast_axis, - int softmax_axis, - const onnxruntime::TensorShape& X_shape, - const onnxruntime::Tensor* X, - const onnxruntime::TensorShape& B_shape, - const onnxruntime::Tensor* B, - onnxruntime::Tensor* Y) { - DispatchBiasSoftMaxForwardViaDnnLibraryImpl( - stream, - cudaDnnHandle, - element_count, - batch_count, - broadcast_axis, - softmax_axis, - X_shape, - X, - B_shape, - B, - Y); + cudaStream_t stream, + cudnnHandle_t cudaDnnHandle, + int element_count, + int batch_count, + int broadcast_axis, + int softmax_axis, + const onnxruntime::TensorShape& X_shape, + const onnxruntime::Tensor* X, + const onnxruntime::TensorShape& B_shape, + const onnxruntime::Tensor* B, + onnxruntime::Tensor* Y) { + DispatchBiasSoftMaxForwardViaDnnLibraryImpl( + stream, + cudaDnnHandle, + element_count, + batch_count, + broadcast_axis, + softmax_axis, + X_shape, + X, + B_shape, + B, + Y); } } // namespace cuda } // namespace contrib -} // namespace onnxruntime \ No newline at end of file +} // namespace onnxruntime diff --git a/onnxruntime/contrib_ops/cuda/math/bias_softmax_impl.cu b/onnxruntime/contrib_ops/cuda/math/bias_softmax_impl.cu index 27b2363219..a3e0189a2b 100644 --- a/onnxruntime/contrib_ops/cuda/math/bias_softmax_impl.cu +++ b/onnxruntime/contrib_ops/cuda/math/bias_softmax_impl.cu @@ -6,7 +6,6 @@ #include #include -#include "core/providers/common.h" #include "core/providers/cuda/cuda_common.h" #include "core/providers/cuda/cudnn_common.h" #include "core/providers/cuda/cu_inc/binary_elementwise_impl.cuh" diff --git a/onnxruntime/contrib_ops/cuda/math/binary_elementwise_ops.cc b/onnxruntime/contrib_ops/cuda/math/binary_elementwise_ops.cc index 5f85223a6b..0df46fb763 100644 --- a/onnxruntime/contrib_ops/cuda/math/binary_elementwise_ops.cc +++ b/onnxruntime/contrib_ops/cuda/math/binary_elementwise_ops.cc @@ -9,14 +9,14 @@ namespace onnxruntime { namespace contrib { namespace cuda { -#define CONTRIB_BINARY_ELEMENTWISE_REGISTER_KERNEL_TYPED(x, ver, T) \ - ONNX_OPERATOR_TYPED_KERNEL_EX( \ - x, \ - kMSDomain, \ - ver, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ +#define CONTRIB_BINARY_ELEMENTWISE_REGISTER_KERNEL_TYPED(x, ver, T) \ + ONNX_OPERATOR_TYPED_KERNEL_EX( \ + x, \ + kMSDomain, \ + ver, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ x); #define CONTRIB_BINARY_ELEMENTWISE_COMPUTE(x, T) \ diff --git a/onnxruntime/contrib_ops/cuda/math/complex_mul.cc b/onnxruntime/contrib_ops/cuda/math/complex_mul.cc index 9584e8de3c..9f8ec8fbba 100644 --- a/onnxruntime/contrib_ops/cuda/math/complex_mul.cc +++ b/onnxruntime/contrib_ops/cuda/math/complex_mul.cc @@ -1,7 +1,6 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. -#include "complex_mul.h" #include "complex_mul_impl.h" namespace onnxruntime { @@ -14,7 +13,7 @@ namespace cuda { 1, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()), \ ComplexMul); \ ONNX_OPERATOR_TYPED_KERNEL_EX( \ @@ -23,7 +22,7 @@ namespace cuda { 1, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()), \ ComplexMul); diff --git a/onnxruntime/contrib_ops/cuda/math/complex_mul.h b/onnxruntime/contrib_ops/cuda/math/complex_mul.h index b8cefdd4db..d8677a0045 100644 --- a/onnxruntime/contrib_ops/cuda/math/complex_mul.h +++ b/onnxruntime/contrib_ops/cuda/math/complex_mul.h @@ -13,7 +13,7 @@ using namespace ::onnxruntime::cuda; template class ComplexMul : public BinaryElementwise { public: - ComplexMul(const OpKernelInfo info) : BinaryElementwise{info} {} + ComplexMul(const OpKernelInfo& info) : BinaryElementwise{info} {} Status ComputeInternal(OpKernelContext* context) const override; }; diff --git a/onnxruntime/contrib_ops/cuda/math/fft_ops.cc b/onnxruntime/contrib_ops/cuda/math/fft_ops.cc index c685882e92..d808c00bd5 100644 --- a/onnxruntime/contrib_ops/cuda/math/fft_ops.cc +++ b/onnxruntime/contrib_ops/cuda/math/fft_ops.cc @@ -35,7 +35,7 @@ void SetFFTState(FFTState* state, 1, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()), \ Rfft); \ ONNX_OPERATOR_TYPED_KERNEL_EX( \ @@ -44,7 +44,7 @@ void SetFFTState(FFTState* state, 1, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()), \ Irfft); diff --git a/onnxruntime/contrib_ops/cuda/math/fft_ops.h b/onnxruntime/contrib_ops/cuda/math/fft_ops.h index bd97a15f8f..aeb2379f25 100644 --- a/onnxruntime/contrib_ops/cuda/math/fft_ops.h +++ b/onnxruntime/contrib_ops/cuda/math/fft_ops.h @@ -16,7 +16,7 @@ namespace cuda { template class FFTBase : public ::onnxruntime::cuda::CudaKernel { public: - FFTBase(const OpKernelInfo info) : ::onnxruntime::cuda::CudaKernel{info}, normalized_{0}, onesided_{1} { + FFTBase(const OpKernelInfo& info) : ::onnxruntime::cuda::CudaKernel{info}, normalized_{0}, onesided_{1} { ORT_ENFORCE((info.GetAttr("signal_ndim", &signal_ndim_)).IsOK(), "Attribute signal_ndim is missing in Node ", info.node().Name()); ORT_ENFORCE(signal_ndim_ >= 1 && signal_ndim_ <= 3, @@ -36,14 +36,14 @@ class FFTBase : public ::onnxruntime::cuda::CudaKernel { template class Rfft final : public FFTBase { public: - Rfft(const OpKernelInfo info) : FFTBase{info} {} + Rfft(const OpKernelInfo& info) : FFTBase{info} {} Status ComputeInternal(OpKernelContext* context) const override; }; template class Irfft final : public FFTBase { public: - Irfft(const OpKernelInfo info) : FFTBase{info} {} + Irfft(const OpKernelInfo& info) : FFTBase{info} {} Status ComputeInternal(OpKernelContext* context) const override; }; diff --git a/onnxruntime/contrib_ops/cuda/math/fused_matmul.cc b/onnxruntime/contrib_ops/cuda/math/fused_matmul.cc index d9edac84eb..86ce8d3efa 100644 --- a/onnxruntime/contrib_ops/cuda/math/fused_matmul.cc +++ b/onnxruntime/contrib_ops/cuda/math/fused_matmul.cc @@ -14,7 +14,7 @@ namespace cuda { 1, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()), \ onnxruntime::cuda::MatMul); diff --git a/onnxruntime/contrib_ops/cuda/math/isfinite.cc b/onnxruntime/contrib_ops/cuda/math/isfinite.cc index 121daa1880..0ebe21b23d 100644 --- a/onnxruntime/contrib_ops/cuda/math/isfinite.cc +++ b/onnxruntime/contrib_ops/cuda/math/isfinite.cc @@ -15,7 +15,7 @@ namespace cuda { 1, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("V", DataTypeImpl::GetTensorType()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()), \ IsAllFiniteOp); diff --git a/onnxruntime/contrib_ops/cuda/quantization/attention_quantization.cc b/onnxruntime/contrib_ops/cuda/quantization/attention_quantization.cc index a083849fb3..383f69ebd1 100644 --- a/onnxruntime/contrib_ops/cuda/quantization/attention_quantization.cc +++ b/onnxruntime/contrib_ops/cuda/quantization/attention_quantization.cc @@ -4,8 +4,6 @@ #include "attention_quantization.h" #include "attention_quantization_impl.cuh" #include "contrib_ops/cuda/bert/attention_impl.h" -#include "core/framework/tensorprotoutils.h" -#include "core/providers/common.h" #include "core/providers/cuda/cuda_common.h" #include "core/providers/cuda/shared_inc/fpgeneric.h" #include "core/providers/cuda/shared_inc/integer_gemm.h" @@ -25,11 +23,11 @@ namespace cuda { 1, \ T##_##TQuant, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ - .InputMemoryType(3) \ - .InputMemoryType(4) \ - .InputMemoryType(6) \ - .InputMemoryType(7) \ + (*KernelDefBuilder::Create()) \ + .InputMemoryType(OrtMemTypeCPUInput, 3) \ + .InputMemoryType(OrtMemTypeCPUInput, 4) \ + .InputMemoryType(OrtMemTypeCPUInput, 6) \ + .InputMemoryType(OrtMemTypeCPUInput, 7) \ .TypeConstraint("T1", DataTypeImpl::GetTensorType()) \ .TypeConstraint("T2", DataTypeImpl::GetTensorType()) \ .TypeConstraint("T3", DataTypeImpl::GetTensorType()) \ diff --git a/onnxruntime/contrib_ops/cuda/quantization/quantize_dequantize_linear.cc b/onnxruntime/contrib_ops/cuda/quantization/quantize_dequantize_linear.cc index f0afad58a5..f612a9b6a6 100644 --- a/onnxruntime/contrib_ops/cuda/quantization/quantize_dequantize_linear.cc +++ b/onnxruntime/contrib_ops/cuda/quantization/quantize_dequantize_linear.cc @@ -17,7 +17,7 @@ using namespace onnxruntime::cuda; 1, \ T##_##U, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T1", DataTypeImpl::GetTensorType()) \ .TypeConstraint("T2", DataTypeImpl::GetTensorType()), \ QuantizeLinear); @@ -32,7 +32,7 @@ REGISTER_Q_KERNEL_TYPED(uint8_t, MLFloat16) 1, \ T##_##U, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T1", DataTypeImpl::GetTensorType()) \ .TypeConstraint("T2", DataTypeImpl::GetTensorType()), \ DequantizeLinear); diff --git a/onnxruntime/contrib_ops/cuda/tensor/crop.cc b/onnxruntime/contrib_ops/cuda/tensor/crop.cc index 76495c8b23..3731c6b8ea 100644 --- a/onnxruntime/contrib_ops/cuda/tensor/crop.cc +++ b/onnxruntime/contrib_ops/cuda/tensor/crop.cc @@ -15,7 +15,7 @@ namespace cuda { 1, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()), \ Crop); diff --git a/onnxruntime/contrib_ops/cuda/tensor/dynamicslice.cc b/onnxruntime/contrib_ops/cuda/tensor/dynamicslice.cc index 5cd3e94fb7..a0c4cfb92e 100644 --- a/onnxruntime/contrib_ops/cuda/tensor/dynamicslice.cc +++ b/onnxruntime/contrib_ops/cuda/tensor/dynamicslice.cc @@ -4,21 +4,25 @@ #include "core/providers/cuda/tensor/slice.h" #include "core/providers/cuda/tensor/slice_impl.h" - namespace onnxruntime { namespace contrib { namespace cuda { using namespace onnxruntime::cuda; -#define REGISTER_TYPED_DYNAMICSLICE(TIND) \ - ONNX_OPERATOR_TYPED_KERNEL_EX( \ - DynamicSlice, \ - kOnnxDomain, \ - 1, \ - TIND, \ - kCudaExecutionProvider, \ - KernelDefBuilder().InputMemoryType(1).InputMemoryType(2).InputMemoryType(3).TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()).TypeConstraint("Tind", DataTypeImpl::GetTensorType()), \ +#define REGISTER_TYPED_DYNAMICSLICE(TIND) \ + ONNX_OPERATOR_TYPED_KERNEL_EX( \ + DynamicSlice, \ + kOnnxDomain, \ + 1, \ + TIND, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()) \ + .InputMemoryType(OrtMemTypeCPUInput, 1) \ + .InputMemoryType(OrtMemTypeCPUInput, 2) \ + .InputMemoryType(OrtMemTypeCPUInput, 3) \ + .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()) \ + .TypeConstraint("Tind", DataTypeImpl::GetTensorType()), \ cuda::Slice); REGISTER_TYPED_DYNAMICSLICE(int32_t) diff --git a/onnxruntime/contrib_ops/cuda/tensor/image_scaler.cc b/onnxruntime/contrib_ops/cuda/tensor/image_scaler.cc index fb592a5cf3..6fffd2dfa3 100644 --- a/onnxruntime/contrib_ops/cuda/tensor/image_scaler.cc +++ b/onnxruntime/contrib_ops/cuda/tensor/image_scaler.cc @@ -16,7 +16,7 @@ namespace cuda { 1, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()), \ ImageScaler); diff --git a/onnxruntime/core/framework/feeds_fetches_manager.h b/onnxruntime/core/framework/feeds_fetches_manager.h index 2110b8b5a5..889e3594c5 100644 --- a/onnxruntime/core/framework/feeds_fetches_manager.h +++ b/onnxruntime/core/framework/feeds_fetches_manager.h @@ -6,7 +6,9 @@ #include #include +#ifndef SHARED_PROVIDER #include "core/framework/ml_value.h" +#endif namespace onnxruntime { class ExecutionProviders; diff --git a/onnxruntime/core/framework/ort_value_tensor_slicer.cc b/onnxruntime/core/framework/ort_value_tensor_slicer.cc index fc982b02ad..b916aa90ba 100644 --- a/onnxruntime/core/framework/ort_value_tensor_slicer.cc +++ b/onnxruntime/core/framework/ort_value_tensor_slicer.cc @@ -80,8 +80,7 @@ void OrtValueTensorSlicer::Iterator::MaterializeMLValue() const { // // TODO: Ideally we could avoid the overhead of creating a new Tensor (mainly cost of copying type and shape info) // and would simply update Tensor::p_data_ given all other info remains constant for each slice. - auto sub_tensor = std::make_unique(tensor_data_type_, per_iteration_shape_, - const_cast(tensor_slice_data_raw), *tensor_location_); + auto sub_tensor = Tensor::Create(tensor_data_type_, per_iteration_shape_, const_cast(tensor_slice_data_raw), *tensor_location_); auto ml_tensor = DataTypeImpl::GetType(); current_ = OrtValue{sub_tensor.release(), ml_tensor, ml_tensor->GetDeleteFunc()}; } diff --git a/onnxruntime/core/framework/ort_value_tensor_slicer.h b/onnxruntime/core/framework/ort_value_tensor_slicer.h index a7a9c90462..3069e79e9a 100644 --- a/onnxruntime/core/framework/ort_value_tensor_slicer.h +++ b/onnxruntime/core/framework/ort_value_tensor_slicer.h @@ -7,9 +7,11 @@ #include #include +#ifndef SHARED_PROVIDER #include "core/common/common.h" #include "core/framework/ml_value.h" #include "core/framework/tensor.h" +#endif namespace onnxruntime { diff --git a/onnxruntime/core/framework/provider_bridge_ort.cc b/onnxruntime/core/framework/provider_bridge_ort.cc index 53dc664757..e462064477 100644 --- a/onnxruntime/core/framework/provider_bridge_ort.cc +++ b/onnxruntime/core/framework/provider_bridge_ort.cc @@ -3,7 +3,6 @@ // This is the Onnxruntime side of the bridge to allow providers to be built as a DLL // It implements onnxruntime::ProviderHost - #include "core/framework/allocatormgr.h" #include "core/framework/compute_capability.h" #include "core/framework/data_types.h" @@ -18,13 +17,55 @@ #include "core/session/inference_session.h" #include "core/session/abi_session_options_impl.h" #include "core/session/ort_apis.h" +#include "core/util/math.h" +#include "core/framework/tensorprotoutils.h" +#include "core/framework/fallback_cpu_capability.h" +#include "core/framework/random_generator.h" +#include "core/providers/cpu/tensor/gatherbase.h" +#include "core/providers/cpu/tensor/gather_elements.h" +#include "core/providers/cpu/tensor/unsqueeze.h" +#include "core/providers/cpu/tensor/slice.h" +#include "core/providers/cpu/tensor/split.h" +#include "core/providers/cpu/tensor/size.h" +#include "core/providers/cpu/tensor/scatter_nd.h" +#include "core/providers/cpu/tensor/padbase.h" +#include "core/providers/cpu/tensor/concatbase.h" +#include "core/providers/cpu/tensor/gatherbase.h" +#include "core/providers/cpu/tensor/onehot.h" +#include "core/providers/cpu/tensor/tile.h" +#include "core/providers/cpu/controlflow/if.h" +#include "core/providers/cpu/controlflow/loop.h" +#include "core/providers/cpu/controlflow/scan.h" +#include "core/providers/cpu/math/einsum.h" +#include "core/providers/cpu/math/cumsum.h" +#include "core/providers/cpu/object_detection/non_max_suppression.h" +#include "core/providers/cpu/object_detection/roialign.h" -#ifdef USE_TENSORRT -#include "core/providers/cuda/cuda_allocator.h" -#include "core/providers/cuda/gpu_data_transfer.h" -#include "core/providers/cuda/math/unary_elementwise_ops_impl.h" -#include "core/providers/cuda/cuda_common.h" +#ifndef DISABLE_CONTRIB_OPS +#include "contrib_ops/cpu/bert/bias_gelu_helper.h" +#include "contrib_ops/cpu/bert/embed_layer_norm_helper.h" +#include "contrib_ops/cpu/bert/longformer_attention_base.h" +namespace onnxruntime { +namespace contrib { +Status LongformerAttentionBase__CheckInputs(const LongformerAttentionBase* p, const TensorShape& input_shape, const TensorShape& weights_shape, const TensorShape& bias_shape, const TensorShape& mask_shape, const TensorShape& global_weights_shape, const TensorShape& global_bias_shape, const TensorShape& global_shape); +} +} // namespace onnxruntime +#include "contrib_ops/cpu/bert/attention_base.h" +#endif + +#ifdef ENABLE_TRAINING +#include "orttraining/training_ops/cpu/aten_ops/aten_op.h" +#include "orttraining/training_ops/cpu/controlflow/group.h" +#include "orttraining/training_ops/cpu/controlflow/record.h" +#include "orttraining/training_ops/cpu/controlflow/wait.h" +#include "orttraining/training_ops/cpu/controlflow/yield.h" +#include "orttraining/training_ops/cpu/loss/softmax_cross_entropy_loss.h" +#include "orttraining/training_ops/cpu/tensor/split.h" +#endif +#if defined(USE_CUDA) && defined(ORT_USE_NCCL) && defined(USE_NCCL_P2P) +#include "orttraining/training_ops/cuda/communication/nccl_service.h" +#include "orttraining/core/framework/distributed_run_context.h" #endif namespace ONNX_NAMESPACE { @@ -44,6 +85,7 @@ using IndexedSubGraph_MetaDef = IndexedSubGraph::MetaDef; #include "core/common/logging/logging.h" #include "core/providers/shared_library/provider_interfaces.h" +#include "core/providers/cuda/cuda_provider_factory.h" #include "core/providers/dnnl/dnnl_provider_factory.h" #include "core/providers/tensorrt/tensorrt_provider_factory.h" #include "core/providers/openvino/openvino_provider_factory.h" @@ -66,7 +108,9 @@ using IndexedSubGraph_MetaDef = IndexedSubGraph::MetaDef; namespace onnxruntime { -ProviderHost* g_host{}; +//ProviderHost* g_host{}; + +ProviderInfo_CUDA* GetProviderInfo_CUDA(); struct TensorShapeProto_Dimension_Iterator_Impl : TensorShapeProto_Dimension_Iterator { TensorShapeProto_Dimension_Iterator_Impl(google::protobuf::internal::RepeatedPtrIterator&& v) : v_{std::move(v)} {} @@ -85,8 +129,10 @@ struct NodeAttributes_Iterator_Impl : NodeAttributes_Iterator { bool operator!=(const NodeAttributes_Iterator& p) const override { return v_ != static_cast(&p)->v_; } void operator++() override { v_.operator++(); } + const std::pair& operator*() const override { return v_.operator*(); } + const std::string& first() const override { return v_->first; } - const ONNX_NAMESPACE::AttributeProto& second() override { return v_->second; } + const ONNX_NAMESPACE::AttributeProto& second() const override { return v_->second; } NodeAttributes::const_iterator v_; }; @@ -115,77 +161,71 @@ struct Node__EdgeIterator_Impl : Node__EdgeIterator { Node::EdgeConstIterator v_; }; +// wrapped = The internal object is exposed as an opaque pointer, so we wrap it in a class that forwards every call to the real calls. No members are ever directly accessed +// direct = Same implementation is used for shared providers & core code, but some of the methods need to be routed through here to make the linker happy struct ProviderHostImpl : ProviderHost { - ProviderHostImpl() { - DataTypeImpl_GetType_Tensor = &DataTypeImpl::GetType; - DataTypeImpl_GetType_float = &DataTypeImpl::GetType; - DataTypeImpl_GetTensorType_float = &DataTypeImpl::GetTensorType; - } - - AllocatorPtr CreateAllocator(const AllocatorCreationInfo& info) override { - return onnxruntime::CreateAllocator(info); - } - - std::unique_ptr CreateCPUAllocator(const OrtMemoryInfo& memory_info) override { - return std::make_unique(memory_info); - }; - -#ifdef USE_TENSORRT - std::unique_ptr CreateCUDAAllocator(int16_t device_id, const char* name) override { - return std::make_unique(device_id, name); - } - - std::unique_ptr CreateCUDAPinnedAllocator(int16_t device_id, const char* name) override { - return std::make_unique(device_id, name); - } - - std::unique_ptr CreateGPUDataTransfer(void* stream) override { - return std::make_unique(static_cast(stream)); - } - - void cuda__Impl_Cast(void* stream, const int64_t* input_data, int32_t* output_data, size_t count) override { - return cuda::Impl_Cast(static_cast(stream), input_data, output_data, count); - } - - void cuda__Impl_Cast(void* stream, const int32_t* input_data, int64_t* output_data, size_t count) override { - return cuda::Impl_Cast(static_cast(stream), input_data, output_data, count); - } - - bool CudaCall_false(int retCode, const char* exprString, const char* libName, int successCode, const char* msg) override { return CudaCall(cudaError(retCode), exprString, libName, cudaError(successCode), msg); } - bool CudaCall_true(int retCode, const char* exprString, const char* libName, int successCode, const char* msg) override { return CudaCall(cudaError(retCode), exprString, libName, cudaError(successCode), msg); } -#endif - - std::string GetEnvironmentVar(const std::string& var_name) override { - return Env::Default().GetEnvironmentVar(var_name); - } + void* HeapAllocate(size_t size) override { return new uint8_t[size]; } + void HeapFree(void* p) override { delete[] reinterpret_cast(p); } logging::Logger* LoggingManager_GetDefaultLogger() override { return const_cast(&logging::LoggingManager::DefaultLogger()); } - int32_t PrimitiveDataTypeBase__GetDataType(const PrimitiveDataTypeBase* p) override { return p->GetDataType(); } - - const char* DataTypeImpl__ToString(MLDataType type) override { return DataTypeImpl::ToString(type); } - const std::vector& DataTypeImpl__AllFixedSizeTensorTypes() override { return DataTypeImpl::AllFixedSizeTensorTypes(); } - const std::vector& DataTypeImpl__AllTensorTypes() override { return DataTypeImpl::AllTensorTypes(); } - size_t DataTypeImpl__Size(const DataTypeImpl* p) override { return p->Size(); } - const PrimitiveDataTypeBase* DataTypeImpl__AsPrimitiveDataType(const DataTypeImpl* p) override { return p->AsPrimitiveDataType(); } - - void* HeapAllocate(size_t size) override { return new uint8_t[size]; } - void HeapFree(void* p) override { delete[] reinterpret_cast(p); } - - std::vector GetStackTrace() override { return onnxruntime::GetStackTrace(); } - - AutoPadType StringToAutoPadType(const std::string& str) override { return onnxruntime::StringToAutoPadType(str); } - void LogRuntimeError(uint32_t session_id, const common::Status& status, const char* file, const char* function, uint32_t line) override { return ::onnxruntime::LogRuntimeError(session_id, status, file, function, line); } - // IAllocator + std::vector GetStackTrace() override { return onnxruntime::GetStackTrace(); } + + OrtStatus* CreateStatus(OrtErrorCode code, _In_ const char* msg) noexcept override { return OrtApis::CreateStatus(code, msg); } + + AllocatorPtr CreateAllocator(const AllocatorCreationInfo& info) override { return onnxruntime::CreateAllocator(info); } + std::unique_ptr CreateCPUAllocator(const OrtMemoryInfo& memory_info) override { return std::make_unique(memory_info); }; + + void* CPUAllocator__Alloc(CPUAllocator* p, size_t size) override { return p->CPUAllocator::Alloc(size); } + void CPUAllocator__Free(CPUAllocator* p, void* allocation) override { return p->CPUAllocator::Free(allocation); } + +#ifdef USE_CUDA + std::unique_ptr CreateCUDAAllocator(int16_t device_id, const char* name) override { return GetProviderInfo_CUDA()->CreateCUDAAllocator(device_id, name); } + std::unique_ptr CreateCUDAPinnedAllocator(int16_t device_id, const char* name) override { return GetProviderInfo_CUDA()->CreateCUDAPinnedAllocator(device_id, name); } + std::unique_ptr CreateGPUDataTransfer(void* stream) override { return GetProviderInfo_CUDA()->CreateGPUDataTransfer(stream); } + + void cuda__Impl_Cast(void* stream, const int64_t* input_data, int32_t* output_data, size_t count) override { return GetProviderInfo_CUDA()->cuda__Impl_Cast(stream, input_data, output_data, count); } + void cuda__Impl_Cast(void* stream, const int32_t* input_data, int64_t* output_data, size_t count) override { return GetProviderInfo_CUDA()->cuda__Impl_Cast(stream, input_data, output_data, count); } + + bool CudaCall_false(int retCode, const char* exprString, const char* libName, int successCode, const char* msg) override { return GetProviderInfo_CUDA()->CudaCall_false(retCode, exprString, libName, successCode, msg); } + bool CudaCall_true(int retCode, const char* exprString, const char* libName, int successCode, const char* msg) override { return GetProviderInfo_CUDA()->CudaCall_true(retCode, exprString, libName, successCode, msg); } +#endif + + std::string GetEnvironmentVar(const std::string& var_name) override { return Env::Default().GetEnvironmentVar(var_name); } + + std::unordered_set GetCpuPreferredNodes(const onnxruntime::GraphViewer& graph, + const std::string& provider_type, + const std::vector& kernel_registries, + const std::vector& tentative_nodes) override { + return onnxruntime::GetCpuPreferredNodes(graph, provider_type, kernel_registries, tentative_nodes); + } + + Status UnpackTensor(const ONNX_NAMESPACE::TensorProto& tensor, const void* raw_data, size_t raw_data_len, /*out*/ bool* p_data, size_t expected_size) override { return utils::UnpackTensor(tensor, raw_data, raw_data_len, p_data, expected_size); } + Status UnpackTensor(const ONNX_NAMESPACE::TensorProto& tensor, const void* raw_data, size_t raw_data_len, /*out*/ float* p_data, size_t expected_size) override { return utils::UnpackTensor(tensor, raw_data, raw_data_len, p_data, expected_size); } + Status UnpackTensor(const ONNX_NAMESPACE::TensorProto& tensor, const void* raw_data, size_t raw_data_len, /*out*/ double* p_data, size_t expected_size) override { return utils::UnpackTensor(tensor, raw_data, raw_data_len, p_data, expected_size); } + Status UnpackTensor(const ONNX_NAMESPACE::TensorProto& tensor, const void* raw_data, size_t raw_data_len, /*out*/ MLFloat16* p_data, size_t expected_size) override { return utils::UnpackTensor(tensor, raw_data, raw_data_len, p_data, expected_size); } + Status UnpackTensor(const ONNX_NAMESPACE::TensorProto& tensor, const void* raw_data, size_t raw_data_len, /*out*/ int8_t* p_data, size_t expected_size) override { return utils::UnpackTensor(tensor, raw_data, raw_data_len, p_data, expected_size); } + Status UnpackTensor(const ONNX_NAMESPACE::TensorProto& tensor, const void* raw_data, size_t raw_data_len, /*out*/ uint8_t* p_data, size_t expected_size) override { return utils::UnpackTensor(tensor, raw_data, raw_data_len, p_data, expected_size); } + Status UnpackTensor(const ONNX_NAMESPACE::TensorProto& tensor, const void* raw_data, size_t raw_data_len, /*out*/ int16_t* p_data, size_t expected_size) override { return utils::UnpackTensor(tensor, raw_data, raw_data_len, p_data, expected_size); } + Status UnpackTensor(const ONNX_NAMESPACE::TensorProto& tensor, const void* raw_data, size_t raw_data_len, /*out*/ uint16_t* p_data, size_t expected_size) override { return utils::UnpackTensor(tensor, raw_data, raw_data_len, p_data, expected_size); } + Status UnpackTensor(const ONNX_NAMESPACE::TensorProto& tensor, const void* raw_data, size_t raw_data_len, /*out*/ int32_t* p_data, size_t expected_size) override { return utils::UnpackTensor(tensor, raw_data, raw_data_len, p_data, expected_size); } + Status UnpackTensor(const ONNX_NAMESPACE::TensorProto& tensor, const void* raw_data, size_t raw_data_len, /*out*/ uint32_t* p_data, size_t expected_size) override { return utils::UnpackTensor(tensor, raw_data, raw_data_len, p_data, expected_size); } + Status UnpackTensor(const ONNX_NAMESPACE::TensorProto& tensor, const void* raw_data, size_t raw_data_len, /*out*/ int64_t* p_data, size_t expected_size) override { return utils::UnpackTensor(tensor, raw_data, raw_data_len, p_data, expected_size); } + Status UnpackTensor(const ONNX_NAMESPACE::TensorProto& tensor, const void* raw_data, size_t raw_data_len, /*out*/ uint64_t* p_data, size_t expected_size) override { return utils::UnpackTensor(tensor, raw_data, raw_data_len, p_data, expected_size); } + + uint16_t math__floatToHalf(float f) override { return math::floatToHalf(f); } + float math__halfToFloat(uint16_t h) override { return math::halfToFloat(h); } + + // IAllocator (direct) bool IAllocator__CalcMemSizeForArrayWithAlignment(size_t nmemb, size_t size, size_t alignment, size_t* out) override { return IAllocator::CalcMemSizeForArrayWithAlignment(nmemb, size, alignment, out); } - // IExecutionProvider + // IExecutionProvider (direct) AllocatorPtr IExecutionProvider__GetAllocator(const IExecutionProvider* p, int id, OrtMemType mem_type) override { return p->IExecutionProvider::GetAllocator(id, mem_type); } void IExecutionProvider__InsertAllocator(IExecutionProvider* p, AllocatorPtr allocator) override { return p->IExecutionProvider::InsertAllocator(allocator); } void IExecutionProvider__TryInsertAllocator(IExecutionProvider* p, AllocatorPtr allocator) override { return p->IExecutionProvider::TryInsertAllocator(allocator); } @@ -211,48 +251,53 @@ struct ProviderHostImpl : ProviderHost { return p->IExecutionProvider::RegisterAllocator(allocator_manager); } - // Status - std::string Status__ToString(const Status* p) override { return p->ToString(); } + // Status (direct) + std::string Status__ToString(const Status* p) override { return p->Status::ToString(); } - // TensorShape - int64_t TensorShape__SizeHelper(const TensorShape* p, size_t start, size_t end) override { return p->SizeHelper(start, end); } - std::string TensorShape__ToString(const TensorShape* p) override { return p->ToString(); } + // TensorShape (direct) + int64_t TensorShape__SizeHelper(const TensorShape* p, size_t start, size_t end) override { return p->TensorShape::SizeHelper(start, end); } + std::string TensorShape__ToString(const TensorShape* p) override { return p->TensorShape::ToString(); } + int64_t TensorShape__SizeToDimension(const TensorShape* p, size_t dimension) override { return p->TensorShape::SizeToDimension(dimension); } + int64_t TensorShape__SizeFromDimension(const TensorShape* p, size_t dimension) override { return p->TensorShape::SizeFromDimension(dimension); } + std::ostream& operator_left_shift(std::ostream& out, const TensorShape& shape) override { return out << shape; } - // CPUIDInfo + // CPUIDInfo (wrapped) const CPUIDInfo& CPUIDInfo__GetCPUIDInfo() override { return CPUIDInfo::GetCPUIDInfo(); } bool CPUIDInfo__HasAVX2(const CPUIDInfo* p) override { return p->HasAVX2(); } bool CPUIDInfo__HasAVX512f(const CPUIDInfo* p) override { return p->HasAVX512f(); } - // logging::Logger + // logging::Logger (wrapped) bool logging__Logger__OutputIsEnabled(const logging::Logger* p, logging::Severity severity, logging::DataType data_type) override { return p->OutputIsEnabled(severity, data_type); } - // logging::LoggingManager + // logging::LoggingManager (wrapped) const logging::Logger& logging__LoggingManager__DefaultLogger() override { return logging::LoggingManager::DefaultLogger(); } - // logging::Capture + // logging::Capture (wrapped) std::unique_ptr logging__Capture__construct(const logging::Logger& logger, logging::Severity severity, const char* category, logging::DataType dataType, const CodeLocation& location) override { return std::make_unique(logger, severity, category, dataType, location); } void logging__Capture__operator_delete(logging::Capture* p) noexcept override { delete p; } std::ostream& logging__Capture__Stream(logging::Capture* p) noexcept override { return p->Stream(); } - // Utils::DataTypeUtils + // Utils::DataTypeUtils (wrapped) const std::string* Utils__DataTypeUtils__ToType(const ONNX_NAMESPACE::TypeProto& type_proto) override { return ONNX_NAMESPACE::Utils::DataTypeUtils::ToType(type_proto); } - // int64s + // int64s (wrapped) int int64s__size(const ONNX_NAMESPACE::int64s* p) override { return p->size(); } const int64_t& int64s__Get(const ONNX_NAMESPACE::int64s* p, int index) override { return p->Get(index); } - // TypeProto_Tensor + // TypeProto_Tensor (wrapped) + bool TypeProto_Tensor__has_shape(const ONNX_NAMESPACE::TypeProto_Tensor* p) override { return p->has_shape(); } const ONNX_NAMESPACE::TensorShapeProto& TypeProto_Tensor__shape(const ONNX_NAMESPACE::TypeProto_Tensor* p) override { return p->shape(); } ONNX_NAMESPACE::TensorShapeProto* TypeProto_Tensor__mutable_shape(ONNX_NAMESPACE::TypeProto_Tensor* p) override { return p->mutable_shape(); } int32_t TypeProto_Tensor__elem_type(const ONNX_NAMESPACE::TypeProto_Tensor* p) override { return p->elem_type(); } - // TypeProto + // TypeProto (wrapped) const ONNX_NAMESPACE::TypeProto_Tensor& TypeProto__tensor_type(const ONNX_NAMESPACE::TypeProto* p) override { return p->tensor_type(); } ONNX_NAMESPACE::TypeProto_Tensor* TypeProto__mutable_tensor_type(ONNX_NAMESPACE::TypeProto* p) override { return p->mutable_tensor_type(); } + int TypeProto__value_case(const ONNX_NAMESPACE::TypeProto* p) override { return p->value_case(); } - // AttributeProto + // AttributeProto (wrapped) std::unique_ptr AttributeProto__construct() override { return std::make_unique(); } void AttributeProto__operator_delete(ONNX_NAMESPACE::AttributeProto* p) override { delete p; } void AttributeProto__operator_assign(ONNX_NAMESPACE::AttributeProto* p, const ONNX_NAMESPACE::AttributeProto& v) override { *p = v; } @@ -260,8 +305,10 @@ struct ProviderHostImpl : ProviderHost { ONNX_NAMESPACE::AttributeProto_AttributeType AttributeProto__type(const ONNX_NAMESPACE::AttributeProto* p) override { return p->type(); } int AttributeProto__ints_size(const ONNX_NAMESPACE::AttributeProto* p) override { return p->ints_size(); } int AttributeProto__floats_size(const ONNX_NAMESPACE::AttributeProto* p) override { return p->floats_size(); } + int AttributeProto__strings_size(const ONNX_NAMESPACE::AttributeProto* p) override { return p->strings_size(); } int64_t AttributeProto__ints(const ONNX_NAMESPACE::AttributeProto* p, int i) override { return p->ints(i); } float AttributeProto__floats(const ONNX_NAMESPACE::AttributeProto* p, int i) override { return p->floats(i); } + const std::string& AttributeProto__strings(const ONNX_NAMESPACE::AttributeProto* p, int i) override { return p->strings(i); } const ONNX_NAMESPACE::int64s& AttributeProto__ints(const ONNX_NAMESPACE::AttributeProto* p) override { return p->ints(); } int64_t AttributeProto__i(const ONNX_NAMESPACE::AttributeProto* p) override { return p->i(); } float AttributeProto__f(const ONNX_NAMESPACE::AttributeProto* p) override { return p->f(); } @@ -271,7 +318,7 @@ struct ProviderHostImpl : ProviderHost { void AttributeProto__set_type(ONNX_NAMESPACE::AttributeProto* p, ONNX_NAMESPACE::AttributeProto_AttributeType value) override { return p->set_type(value); } ONNX_NAMESPACE::TensorProto* AttributeProto__add_tensors(ONNX_NAMESPACE::AttributeProto* p) override { return p->add_tensors(); } - // GraphProto + // GraphProto (wrapped) void GraphProto__operator_delete(ONNX_NAMESPACE::GraphProto* p) override { delete p; } const ONNX_NAMESPACE::ValueInfoProto& GraphProto__input(const ONNX_NAMESPACE::GraphProto* p, int index) override { return p->input(index); } @@ -289,7 +336,7 @@ struct ProviderHostImpl : ProviderHost { void GraphProto__operator_assign(ONNX_NAMESPACE::GraphProto* p, const ONNX_NAMESPACE::GraphProto& v) override { *p = v; } - // ModelProto + // ModelProto (wrapped) std::unique_ptr ModelProto__construct() override { return std::make_unique(); } void ModelProto__operator_delete(ONNX_NAMESPACE::ModelProto* p) override { delete p; } @@ -303,16 +350,25 @@ struct ProviderHostImpl : ProviderHost { void ModelProto__set_ir_version(ONNX_NAMESPACE::ModelProto* p, int64_t value) override { p->set_ir_version(value); } - // TensorProto + // TensorProto (wrapped) + std::unique_ptr TensorProto__construct() override { return std::make_unique(); } void TensorProto__operator_delete(ONNX_NAMESPACE::TensorProto* p) override { delete p; } void TensorProto__operator_assign(ONNX_NAMESPACE::TensorProto* p, const ONNX_NAMESPACE::TensorProto& v) override { *p = v; } + bool TensorProto__has_name(const ONNX_NAMESPACE::TensorProto* p) override { return p->has_name(); } + int TensorProto__dims_size(const ONNX_NAMESPACE::TensorProto* p) override { return p->dims_size(); } + const ONNX_NAMESPACE::int64s& TensorProto__dims(const ONNX_NAMESPACE::TensorProto* p) override { return p->dims(); } bool TensorProto__has_data_location(const ONNX_NAMESPACE::TensorProto* p) override { return p->has_data_location(); } int TensorProto__data_location(const ONNX_NAMESPACE::TensorProto* p) override { return p->data_location(); } + bool TensorProto__has_raw_data(const ONNX_NAMESPACE::TensorProto* p) override { return p->has_raw_data(); } + const std::string& TensorProto__raw_data(const ONNX_NAMESPACE::TensorProto* p) override { return p->raw_data(); } + int32_t TensorProto__data_type(const ONNX_NAMESPACE::TensorProto* p) override { return p->data_type(); } - // TensorProtos + bool TensorProto_DataType_IsValid(int value) override { return ONNX_NAMESPACE::TensorProto::DataType_IsValid(value); } + + // TensorProtos (wrapped) ONNX_NAMESPACE::TensorProto* TensorProtos__Add(ONNX_NAMESPACE::TensorProtos* p) override { return p->Add(); } - // TensorShapeProto_Dimension + // TensorShapeProto_Dimension (wrapped) int TensorShapeProto_Dimension__value_case(const ONNX_NAMESPACE::TensorShapeProto_Dimension* p) override { return p->value_case(); } const std::string& TensorShapeProto_Dimension__dim_param(const ONNX_NAMESPACE::TensorShapeProto_Dimension* p) override { return p->dim_param(); } int64_t TensorShapeProto_Dimension__dim_value(const ONNX_NAMESPACE::TensorShapeProto_Dimension* p) override { return p->dim_value(); } @@ -321,7 +377,7 @@ struct ProviderHostImpl : ProviderHost { bool TensorShapeProto_Dimension__has_dim_value(const ONNX_NAMESPACE::TensorShapeProto_Dimension* p) override { return p->has_dim_value(); } bool TensorShapeProto_Dimension__has_dim_param(const ONNX_NAMESPACE::TensorShapeProto_Dimension* p) override { return p->has_dim_param(); } - // TensorShapeProto_Dimensions + // TensorShapeProto_Dimensions (wrapped) std::unique_ptr TensorShapeProto_Dimensions__begin(const ONNX_NAMESPACE::TensorShapeProto_Dimensions* p) override { return std::make_unique(p->begin()); } @@ -330,7 +386,7 @@ struct ProviderHostImpl : ProviderHost { return std::make_unique(p->end()); } - // TensorShapeProto + // TensorShapeProto (wrapped) int TensorShapeProto__dim_size(const ONNX_NAMESPACE::TensorShapeProto* p) override { return p->dim_size(); } const ONNX_NAMESPACE::TensorShapeProto_Dimensions& TensorShapeProto__dim(const ONNX_NAMESPACE::TensorShapeProto* p) override { return p->dim(); } const ONNX_NAMESPACE::TensorShapeProto_Dimension& TensorShapeProto__dim(const ONNX_NAMESPACE::TensorShapeProto* p, int index) override { return p->dim(index); } @@ -338,29 +394,31 @@ struct ProviderHostImpl : ProviderHost { void TensorShapeProto__clear_dim(ONNX_NAMESPACE::TensorShapeProto* p) override { return p->clear_dim(); } ONNX_NAMESPACE::TensorShapeProto_Dimension* TensorShapeProto__add_dim(ONNX_NAMESPACE::TensorShapeProto* p) override { return p->add_dim(); } - // ValueInfoProto + // ValueInfoProto (wrapped) const ONNX_NAMESPACE::TypeProto& ValueInfoProto__type(const ONNX_NAMESPACE::ValueInfoProto* p) override { return p->type(); } ONNX_NAMESPACE::TypeProto* ValueInfoProto__mutable_type(ONNX_NAMESPACE::ValueInfoProto* p) override { return p->mutable_type(); } virtual void ValueInfoProto__operator_assign(ONNX_NAMESPACE::ValueInfoProto* p, const ONNX_NAMESPACE::ValueInfoProto& v) override { *p = v; } - // ValueInfoProtos + // ValueInfoProtos (wrapped) ONNX_NAMESPACE::ValueInfoProto* ValueInfoProtos__Add(ONNX_NAMESPACE::ValueInfoProtos* p) override { return p->Add(); } const ONNX_NAMESPACE::ValueInfoProto& ValueInfoProtos__operator_array(const ONNX_NAMESPACE::ValueInfoProtos* p, int index) override { return (*p)[index]; } - // ComputeCapability + // ComputeCapability (wrapped) std::unique_ptr ComputeCapability__construct(std::unique_ptr t_sub_graph) override { return std::make_unique(std::move(t_sub_graph)); } void ComputeCapability__operator_delete(ComputeCapability* p) override { delete p; } std::unique_ptr& ComputeCapability__SubGraph(ComputeCapability* p) override { return p->sub_graph; } - // DataTransferManager + // DataTransferManager (wrapped) Status DataTransferManager__CopyTensor(const DataTransferManager* p, const Tensor& src, Tensor& dst, int exec_queue_id) override { return p->CopyTensor(src, dst, exec_queue_id); } + Status DataTransferManager__CopyTensor(const DataTransferManager* p, const Tensor& src, Tensor& dst) override { return p->CopyTensor(src, dst); } + const IDataTransfer* DataTransferManager__GetDataTransfer(const DataTransferManager* p, const OrtDevice& src_device, const OrtDevice& dst_device) override { return p->GetDataTransfer(src_device, dst_device); } - // IDataTransfer + // IDataTransfer (direct) Status IDataTransfer__CopyTensor(const IDataTransfer* p, const Tensor& src, Tensor& dst) override { return p->IDataTransfer::CopyTensor(src, dst); } Status IDataTransfer__CopyTensors(const IDataTransfer* p, const std::vector& src_dst_pairs) override { return p->IDataTransfer::CopyTensors(src_dst_pairs); } - // IndexedSubGraph_MetaDef + // IndexedSubGraph_MetaDef (wrapped) std::unique_ptr IndexedSubGraph_MetaDef__construct() override { return std::make_unique(); } void IndexedSubGraph_MetaDef__operator_delete(IndexedSubGraph_MetaDef* p) override { delete p; } @@ -373,7 +431,7 @@ struct ProviderHostImpl : ProviderHost { NodeAttributes& IndexedSubGraph_MetaDef__attributes(IndexedSubGraph_MetaDef* p) override { return p->attributes; } std::string& IndexedSubGraph_MetaDef__doc_string(IndexedSubGraph_MetaDef* p) override { return p->doc_string; } - // IndexedSubGraph + // IndexedSubGraph (wrapped) std::unique_ptr IndexedSubGraph__construct() override { return std::make_unique(); } void IndexedSubGraph__operator_delete(IndexedSubGraph* p) override { delete p; } @@ -382,46 +440,107 @@ struct ProviderHostImpl : ProviderHost { void IndexedSubGraph__SetMetaDef(IndexedSubGraph* p, std::unique_ptr&& meta_def_) override { return p->SetMetaDef(std::move(meta_def_)); } const IndexedSubGraph_MetaDef* IndexedSubGraph__GetMetaDef(const IndexedSubGraph* p) override { return p->GetMetaDef(); } - // KernelDef + // KernelDef (wrapped) void KernelDef__operator_delete(KernelDef* p) override { delete p; } + void KernelDef__SinceVersion(const KernelDef* p, int* start, int* end) override { return p->SinceVersion(start, end); } + const std::string& KernelDef__Domain(const KernelDef* p) override { return p->Domain(); } + const std::string& KernelDef__OpName(const KernelDef* p) override { return p->OpName(); } int KernelDef__ExecQueueId(const KernelDef* p) override { return p->ExecQueueId(); } - // KernelDefBuilder + // KernelDefBuilder (wrapped) std::unique_ptr KernelDefBuilder__construct() override { return std::make_unique(); } void KernelDefBuilder__operator_delete(KernelDefBuilder* p) override { delete p; } void KernelDefBuilder__SetName(KernelDefBuilder* p, const char* op_name) override { p->SetName(op_name); } void KernelDefBuilder__SetDomain(KernelDefBuilder* p, const char* domain) override { p->SetDomain(domain); } void KernelDefBuilder__SinceVersion(KernelDefBuilder* p, int since_version) override { p->SinceVersion(since_version); } + void KernelDefBuilder__SinceVersion(KernelDefBuilder* p, int since_version_start, int since_version_end) override { p->SinceVersion(since_version_start, since_version_end); } void KernelDefBuilder__Provider(KernelDefBuilder* p, const char* provider_type) override { p->Provider(provider_type); } void KernelDefBuilder__TypeConstraint(KernelDefBuilder* p, const char* arg_name, MLDataType supported_type) override { p->TypeConstraint(arg_name, supported_type); } void KernelDefBuilder__TypeConstraint(KernelDefBuilder* p, const char* arg_name, const std::vector& supported_types) override { p->TypeConstraint(arg_name, supported_types); } void KernelDefBuilder__InputMemoryType(KernelDefBuilder* p, OrtMemType type, int input_index) override { p->InputMemoryType(type, input_index); } + void KernelDefBuilder__InputMemoryType(KernelDefBuilder* p, OrtMemType type, const std::vector& input_indexes) override { p->InputMemoryType(type, input_indexes); } void KernelDefBuilder__OutputMemoryType(KernelDefBuilder* p, OrtMemType type, int input_index) override { p->OutputMemoryType(type, input_index); } void KernelDefBuilder__ExecQueueId(KernelDefBuilder* p, int queue_id) override { p->ExecQueueId(queue_id); } + void KernelDefBuilder__MayInplace(KernelDefBuilder* p, int input_index, int output_index) override { p->MayInplace(input_index, output_index); } + void KernelDefBuilder__Alias(KernelDefBuilder* p, int input_index, int output_index) override { p->Alias(input_index, output_index); } + void KernelDefBuilder__Alias(KernelDefBuilder* p, const std::vector>& aliases) override { p->Alias(aliases); } + void KernelDefBuilder__VariadicAlias(KernelDefBuilder* p, int input_offset, int output_offset) override { p->VariadicAlias(input_offset, output_offset); } + void KernelDefBuilder__ExternalOutputs(KernelDefBuilder* p) override { p->ExternalOutputs(); } + void KernelDefBuilder__AllocateInputsContiguously(KernelDefBuilder* p) override { p->AllocateInputsContiguously(); } std::unique_ptr KernelDefBuilder__Build(KernelDefBuilder* p) override { return p->Build(); } - // KernelRegistry + // KernelRegistry (wrapped) std::shared_ptr KernelRegistry__construct() override { return std::make_shared(); } void KernelRegistry__operator_delete(KernelRegistry* p) override { delete p; } Status KernelRegistry__Register(KernelRegistry* p, KernelCreateInfo&& create_info) override { return p->Register(std::move(create_info)); } - // Function + Status KernelRegistry__TryFindKernel(const KernelRegistry* p, const Node& node, ProviderType exec_provider, const KernelCreateInfo** out) override { + return p->TryFindKernel(node, exec_provider, out); + } + + // PrimitiveDataTypeBase (wrapped) + int32_t PrimitiveDataTypeBase__GetDataType(const PrimitiveDataTypeBase* p) override { return p->GetDataType(); } + + // DataTypeImpl (wrapped) + MLDataType DataTypeImpl__GetType_Tensor() override { return DataTypeImpl::GetType(); } + MLDataType DataTypeImpl__GetType_bool() override { return DataTypeImpl::GetType(); } + MLDataType DataTypeImpl__GetType_int8() override { return DataTypeImpl::GetType(); } + MLDataType DataTypeImpl__GetType_uint8() override { return DataTypeImpl::GetType(); } + MLDataType DataTypeImpl__GetType_int16() override { return DataTypeImpl::GetType(); } + MLDataType DataTypeImpl__GetType_uint16() override { return DataTypeImpl::GetType(); } + MLDataType DataTypeImpl__GetType_int32() override { return DataTypeImpl::GetType(); } + MLDataType DataTypeImpl__GetType_uint32() override { return DataTypeImpl::GetType(); } + MLDataType DataTypeImpl__GetType_int64() override { return DataTypeImpl::GetType(); } + MLDataType DataTypeImpl__GetType_uint64() override { return DataTypeImpl::GetType(); } + MLDataType DataTypeImpl__GetType_float() override { return DataTypeImpl::GetType(); } + MLDataType DataTypeImpl__GetType_double() override { return DataTypeImpl::GetType(); } + MLDataType DataTypeImpl__GetType_BFloat16() override { return DataTypeImpl::GetType(); } + MLDataType DataTypeImpl__GetType_MLFloat16() override { return DataTypeImpl::GetType(); } + MLDataType DataTypeImpl__GetTensorType_bool() override { return DataTypeImpl::GetTensorType(); } + MLDataType DataTypeImpl__GetTensorType_int8() override { return DataTypeImpl::GetTensorType(); } + MLDataType DataTypeImpl__GetTensorType_uint8() override { return DataTypeImpl::GetTensorType(); } + MLDataType DataTypeImpl__GetTensorType_int16() override { return DataTypeImpl::GetTensorType(); } + MLDataType DataTypeImpl__GetTensorType_uint16() override { return DataTypeImpl::GetTensorType(); } + MLDataType DataTypeImpl__GetTensorType_int32() override { return DataTypeImpl::GetTensorType(); } + MLDataType DataTypeImpl__GetTensorType_uint32() override { return DataTypeImpl::GetTensorType(); } + MLDataType DataTypeImpl__GetTensorType_int64() override { return DataTypeImpl::GetTensorType(); } + MLDataType DataTypeImpl__GetTensorType_uint64() override { return DataTypeImpl::GetTensorType(); } + MLDataType DataTypeImpl__GetTensorType_float() override { return DataTypeImpl::GetTensorType(); } + MLDataType DataTypeImpl__GetTensorType_double() override { return DataTypeImpl::GetTensorType(); } + MLDataType DataTypeImpl__GetTensorType_BFloat16() override { return DataTypeImpl::GetTensorType(); } + MLDataType DataTypeImpl__GetTensorType_MLFloat16() override { return DataTypeImpl::GetTensorType(); } + + const char* DataTypeImpl__ToString(MLDataType type) override { return DataTypeImpl::ToString(type); } + bool DataTypeImpl__IsTensorType(const DataTypeImpl* p) override { return p->IsTensorType(); } + bool DataTypeImpl__IsTensorSequenceType(const DataTypeImpl* p) override { return p->IsTensorSequenceType(); } + bool DataTypeImpl__IsSparseTensorType(const DataTypeImpl* p) override { return p->IsSparseTensorType(); } + DeleteFunc DataTypeImpl__GetDeleteFunc(const DataTypeImpl* p) override { return p->GetDeleteFunc(); } + const std::vector& DataTypeImpl__AllFixedSizeTensorTypes() override { return DataTypeImpl::AllFixedSizeTensorTypes(); } + const std::vector& DataTypeImpl__AllTensorTypes() override { return DataTypeImpl::AllTensorTypes(); } + const std::vector& DataTypeImpl__AllIEEEFloatTensorTypes() override { return DataTypeImpl::AllIEEEFloatTensorTypes(); } + const std::vector& DataTypeImpl__AllTensorAndSequenceTensorTypes() override { return DataTypeImpl::AllTensorAndSequenceTensorTypes(); } + size_t DataTypeImpl__Size(const DataTypeImpl* p) override { return p->Size(); } + const PrimitiveDataTypeBase* DataTypeImpl__AsPrimitiveDataType(const DataTypeImpl* p) override { return p->AsPrimitiveDataType(); } + + // Function (wrapped) const Graph& Function__Body(const Function* p) override { return p->Body(); } - // Node + // Node (wrapped) const std::string& Node__Name(const Node* p) noexcept override { return p->Name(); } const std::string& Node__Description(const Node* p) noexcept override { return p->Description(); } const std::string& Node__Domain(const Node* p) noexcept override { return p->Domain(); } const std::string& Node__OpType(const Node* p) noexcept override { return p->OpType(); } + int Node__SinceVersion(const Node* p) override { return p->SinceVersion(); } const Function* Node__GetFunctionBody(const Node* p) noexcept override { return p->GetFunctionBody(); } + ProviderType Node__GetExecutionProviderType(const Node* p) const noexcept override { return p->GetExecutionProviderType(); } + const std::vector& Node__InputArgCount(const Node* p) override { return p->InputArgCount(); } ConstPointerContainer> Node__ImplicitInputDefs(const Node* p) noexcept override { return p->ImplicitInputDefs(); } ConstPointerContainer> Node__InputDefs(const Node* p) noexcept override { return p->InputDefs(); } ConstPointerContainer> Node__OutputDefs(const Node* p) noexcept override { return p->OutputDefs(); } - NodeIndex Node__Index(const Node* p) noexcept override { return p->Index(); } void Node__ToProto(const Node* p, ONNX_NAMESPACE::NodeProto& proto, bool update_subgraphs = false) override { p->ToProto(proto, update_subgraphs); } @@ -441,7 +560,7 @@ struct ProviderHostImpl : ProviderHost { void Node__ForEachDef(const Node* p, std::function func, bool include_missing_optional_defs) override { p->ForEachDef(func, std::move(include_missing_optional_defs)); } - // NodeArg + // NodeArg (wrapped) const std::string& NodeArg__Name(const NodeArg* p) noexcept override { return p->Name(); } const ONNX_NAMESPACE::TensorShapeProto* NodeArg__Shape(const NodeArg* p) override { return p->Shape(); } ONNX_NAMESPACE::DataType NodeArg__Type(const NodeArg* p) noexcept override { return p->Type(); } @@ -449,7 +568,7 @@ struct ProviderHostImpl : ProviderHost { bool NodeArg__Exists(const NodeArg* p) const noexcept override { return p->Exists(); } const ONNX_NAMESPACE::TypeProto* NodeArg__TypeAsProto(const NodeArg* p) noexcept override { return p->TypeAsProto(); } - // NodeAttributes + // NodeAttributes (wrapped) std::unique_ptr NodeAttributes__construct() override { return std::make_unique(); } void NodeAttributes__operator_delete(NodeAttributes* p) noexcept override { delete p; } size_t NodeAttributes__size(const NodeAttributes* p) override { return p->size(); } @@ -470,12 +589,12 @@ struct ProviderHostImpl : ProviderHost { } void NodeAttributes__insert(NodeAttributes* p, const NodeAttributes& v) override { return p->insert(v.begin(), v.end()); } - // Model + // Model (wrapped) void Model__operator_delete(Model* p) override { delete p; } Graph& Model__MainGraph(Model* p) override { return p->MainGraph(); } std::unique_ptr Model__ToProto(Model* p) override { return std::make_unique(p->ToProto()); } - // Graph + // Graph (wrapped) std::unique_ptr Graph__CreateGraphViewer(const Graph* p) override { return std::make_unique(*p); } std::unique_ptr Graph__ToGraphProto(const Graph* p) override { return std::make_unique(p->ToGraphProto()); } @@ -493,12 +612,12 @@ struct ProviderHostImpl : ProviderHost { const std::vector& Graph__GetInputs(const Graph* p) noexcept override { return p->GetInputs(); } bool Graph__GetInitializedTensor(const Graph* p, const std::string& tensor_name, const ONNX_NAMESPACE::TensorProto*& value) override { return p->GetInitializedTensor(tensor_name, value); } - // GraphViewer + // GraphViewer (wrapped) void GraphViewer__operator_delete(GraphViewer* p) override { delete p; } std::unique_ptr GraphViewer__CreateModel(const GraphViewer* graph_viewer, const logging::Logger& logger) override { return std::make_unique(graph_viewer->Name(), true, ModelMetaData(), PathString(), - IOnnxRuntimeOpSchemaRegistryList(), graph_viewer->DomainToVersionMap(), - std::vector(), logger); + IOnnxRuntimeOpSchemaRegistryList(), graph_viewer->DomainToVersionMap(), + std::vector(), logger); } const std::string& GraphViewer__Name(const GraphViewer* p) noexcept override { return p->Name(); } @@ -516,7 +635,7 @@ struct ProviderHostImpl : ProviderHost { const std::vector& GraphViewer__GetOutputs(const GraphViewer* p) noexcept override { return p->GetOutputs(); } const std::vector& GraphViewer__GetValueInfo(const GraphViewer* p) noexcept override { return p->GetValueInfo(); } - const Provider_InitializedTensorSet& GraphViewer__GetAllInitializedTensors(const GraphViewer* p) override { return p->GetAllInitializedTensors(); } + const InitializedTensorSet& GraphViewer__GetAllInitializedTensors(const GraphViewer* p) override { return p->GetAllInitializedTensors(); } bool GraphViewer__GetInitializedTensor(const GraphViewer* p, const std::string& tensor_name, const ONNX_NAMESPACE::TensorProto*& value) override { return p->GetInitializedTensor(tensor_name, value); } const std::unordered_map& GraphViewer__DomainToVersionMap(const GraphViewer* p) override { return p->DomainToVersionMap(); } @@ -524,37 +643,251 @@ struct ProviderHostImpl : ProviderHost { const std::vector& GraphViewer__GetNodesInTopologicalOrder(const GraphViewer* p) override { return p->GetNodesInTopologicalOrder(); } const std::vector& GraphViewer__GetInputsIncludingInitializers(const GraphViewer* p) noexcept override { return p->GetInputsIncludingInitializers(); } - // Path + // Path (wrapped) PathString Path__ToPathString(const Path* p) noexcept override { return p->ToPathString(); } - // OpKernelContext - const Tensor* OpKernelContext__Input_Tensor(const OpKernelContext* p, int index) override { return p->Input(index); } - Tensor* OpKernelContext__Output(OpKernelContext* p, int index, const TensorShape& shape) override { return p->Output(index, shape); } + // OpKernel (direct) + const Node& OpKernel__Node(const OpKernel* p) override { return p->OpKernel::Node(); } - // OpKernelInfo + // OpKernelContext (wrapped) + const Tensor* OpKernelContext__Input_Tensor(const OpKernelContext* p, int index) override { return p->Input(index); } + const Tensor& OpKernelContext__RequiredInput_Tensor(const OpKernelContext* p, int index) override { return p->RequiredInput(index); } + Tensor* OpKernelContext__Output_Tensor(OpKernelContext* p, int index) override { return p->Output(index); } + Tensor* OpKernelContext__Output(OpKernelContext* p, int index, const TensorShape& shape) override { return p->Output(index, shape); } + Tensor& OpKernelContext__RequiredOutput(OpKernelContext* p, int index, const TensorShape& shape) override { return p->RequiredOutput(index, shape); } + int OpKernelContext__InputCount(const OpKernelContext* p) override { return p->InputCount(); } + int OpKernelContext__OutputCount(const OpKernelContext* p) override { return p->OutputCount(); } + Status OpKernelContext__GetTempSpaceAllocator(const OpKernelContext* p, AllocatorPtr* output) override { return p->GetTempSpaceAllocator(output); } + bool OpKernelContext__GetUseDeterministicCompute(const OpKernelContext* p) override { return p->GetUseDeterministicCompute(); } + bool OpKernelContext__TryGetInferredOutputShape(const OpKernelContext* p, int index, TensorShape& shape) override { return p->TryGetInferredOutputShape(index, shape); } + bool OpKernelContext__TryGetInferredInputShape(const OpKernelContext* p, int index, TensorShape& shape) override { return p->TryGetInferredInputShape(index, shape); } + + // OpKernelInfo (wrapped) std::unique_ptr CopyOpKernelInfo(const OpKernelInfo& info) override { return onnxruntime::CopyOpKernelInfo(info); } void OpKernelInfo__operator_delete(OpKernelInfo* p) override { delete p; } + AllocatorPtr OpKernelInfo__GetAllocator(const OpKernelInfo* p, int device_id, OrtMemType mem_type) override { return p->GetAllocator(device_id, mem_type); } + const IExecutionProvider* OpKernelInfo__GetExecutionProvider(const OpKernelInfo* p) override { return p->GetExecutionProvider(); } Status OpKernelInfo__GetAttr_int64(const OpKernelInfo* p, const std::string& name, int64_t* value) override { return p->GetAttr(name, value); } Status OpKernelInfo__GetAttr_float(const OpKernelInfo* p, const std::string& name, float* value) override { return p->GetAttr(name, value); } + Status OpKernelInfo__GetAttr_string(const OpKernelInfo* p, const std::string& name, std::string* value) override { return p->GetAttr(name, value); } + Status OpKernelInfo__GetAttr_TensorProto(const OpKernelInfo* p, const std::string& name, ONNX_NAMESPACE::TensorProto* value) override { return p->GetAttr(name, value); } + Status OpKernelInfo__GetAttrs(const OpKernelInfo* p, const std::string& name, std::vector& values) override { return p->GetAttrs(name, values); } + Status OpKernelInfo__GetAttrs(const OpKernelInfo* p, const std::string& name, std::vector& values) override { return p->GetAttrs(name, values); } + Status OpKernelInfo__GetAttrs(const OpKernelInfo* p, const std::string& name, std::vector& values) override { return p->GetAttrs(name, values); } const DataTransferManager& OpKernelInfo__GetDataTransferManager(const OpKernelInfo* p) noexcept override { return p->GetDataTransferManager(); } const KernelDef& OpKernelInfo__GetKernelDef(const OpKernelInfo* p) override { return p->GetKernelDef(); } + bool OpKernelInfo__TryGetConstantInput(const OpKernelInfo* p, int input_index, const Tensor** constant_input_value) override { return p->TryGetConstantInput(input_index, constant_input_value); } - // Tensor + uint32_t OpKernelInfo__GetInputCount(const OpKernelInfo* p) override { return p->GetInputCount(); } + uint32_t OpKernelInfo__GetOutputCount(const OpKernelInfo* p) override { return p->GetOutputCount(); } + const Node& OpKernelInfo__node(const OpKernelInfo* p) override { return p->node(); } + + // SessionState (wrapped) + const DataTransferManager& SessionState__GetDataTransferMgr(const SessionState* p) override { return p->GetDataTransferMgr(); } + + // Tensor (wrapped) + std::unique_ptr Tensor__construct(MLDataType p_type, const TensorShape& shape, std::shared_ptr allocator) override { return std::make_unique(p_type, shape, allocator); } + std::unique_ptr Tensor__construct(MLDataType p_type, const TensorShape& shape, void* p_data, const OrtMemoryInfo& alloc, ptrdiff_t offset) override { return std::make_unique(p_type, shape, p_data, alloc, offset); } + void Tensor__operator_delete(Tensor* p) override { delete p; } + + bool* Tensor__MutableData_bool(Tensor* p) override { return p->MutableData(); } + int8_t* Tensor__MutableData_int8(Tensor* p) override { return p->MutableData(); } + uint8_t* Tensor__MutableData_uint8(Tensor* p) override { return p->MutableData(); } + int16_t* Tensor__MutableData_int16(Tensor* p) override { return p->MutableData(); } + uint16_t* Tensor__MutableData_uint16(Tensor* p) override { return p->MutableData(); } + int32_t* Tensor__MutableData_int32(Tensor* p) override { return p->MutableData(); } + uint32_t* Tensor__MutableData_uint32(Tensor* p) override { return p->MutableData(); } + int64_t* Tensor__MutableData_int64(Tensor* p) override { return p->MutableData(); } + uint64_t* Tensor__MutableData_uint64(Tensor* p) override { return p->MutableData(); } float* Tensor__MutableData_float(Tensor* p) override { return p->MutableData(); } - const float* Tensor__Data_float(const Tensor* p) override { return p->Data(); } + double* Tensor__MutableData_double(Tensor* p) override { return p->MutableData(); } + BFloat16* Tensor__MutableData_BFloat16(Tensor* p) override { return p->MutableData(); } + MLFloat16* Tensor__MutableData_MLFloat16(Tensor* p) override { return p->MutableData(); } + const bool* Tensor__Data_bool(const Tensor* p) override { return p->Data(); } + const int8_t* Tensor__Data_int8(const Tensor* p) override { return p->Data(); } + const uint8_t* Tensor__Data_uint8(const Tensor* p) override { return p->Data(); } + const int16_t* Tensor__Data_int16(const Tensor* p) override { return p->Data(); } + const uint16_t* Tensor__Data_uint16(const Tensor* p) override { return p->Data(); } + const int32_t* Tensor__Data_int32(const Tensor* p) override { return p->Data(); } + const uint32_t* Tensor__Data_uint32(const Tensor* p) override { return p->Data(); } + const int64_t* Tensor__Data_int64(const Tensor* p) override { return p->Data(); } + const uint64_t* Tensor__Data_uint64(const Tensor* p) override { return p->Data(); } + const float* Tensor__Data_float(const Tensor* p) override { return p->Data(); } + const double* Tensor__Data_double(const Tensor* p) override { return p->Data(); } + const BFloat16* Tensor__Data_BFloat16(const Tensor* p) override { return p->Data(); } + const MLFloat16* Tensor__Data_MLFloat16(const Tensor* p) override { return p->Data(); } + + gsl::span Tensor__DataAsSpan_int64(const Tensor* p) override { return p->DataAsSpan(); } + + void* Tensor__MutableDataRaw(Tensor* p, MLDataType type) override { return p->MutableDataRaw(type); } + const void* Tensor__DataRaw(const Tensor* p, MLDataType type) override { return p->DataRaw(type); } void* Tensor__MutableDataRaw(Tensor* p) noexcept override { return p->MutableDataRaw(); } - const void* Tensor__DataRaw(const Tensor* p) const noexcept override { return p->DataRaw(); } + const void* Tensor__DataRaw(const Tensor* p) noexcept override { return p->DataRaw(); } + + bool Tensor__IsDataType_bool(const Tensor* p) noexcept override { return p->IsDataType(); } + bool Tensor__IsDataType_int8(const Tensor* p) noexcept override { return p->IsDataType(); } + bool Tensor__IsDataType_uint8(const Tensor* p) noexcept override { return p->IsDataType(); } + bool Tensor__IsDataType_int16(const Tensor* p) noexcept override { return p->IsDataType(); } + bool Tensor__IsDataType_uint16(const Tensor* p) noexcept override { return p->IsDataType(); } + bool Tensor__IsDataType_int32(const Tensor* p) noexcept override { return p->IsDataType(); } + bool Tensor__IsDataType_uint32(const Tensor* p) noexcept override { return p->IsDataType(); } + bool Tensor__IsDataType_int64(const Tensor* p) noexcept override { return p->IsDataType(); } + bool Tensor__IsDataType_uint64(const Tensor* p) noexcept override { return p->IsDataType(); } + bool Tensor__IsDataType_float(const Tensor* p) noexcept override { return p->IsDataType(); } + bool Tensor__IsDataType_double(const Tensor* p) noexcept override { return p->IsDataType(); } + bool Tensor__IsDataType_MLFloat16(const Tensor* p) noexcept override { return p->IsDataType(); } + bool Tensor__IsDataTypeString(const Tensor* p) noexcept override { return p->IsDataTypeString(); } const TensorShape& Tensor__Shape(const Tensor* p) override { return p->Shape(); } + void Tensor__Reshape(Tensor* p, const TensorShape& new_shape) override { return p->Reshape(new_shape); } + void Tensor__SetByteOffset(Tensor* p, ptrdiff_t byte_offset) override { p->SetByteOffset(byte_offset); } + ptrdiff_t Tensor__ByteOffset(const Tensor* p) override { return p->ByteOffset(); } size_t Tensor__SizeInBytes(const Tensor* p) override { return p->SizeInBytes(); } const OrtMemoryInfo& Tensor__Location(const Tensor* p) override { return p->Location(); } + int32_t Tensor__GetElementType(const Tensor* p) override { return p->GetElementType(); } + MLDataType Tensor__DataType(const Tensor* p) override { return p->DataType(); } - // AllocatorManager - void AllocatorManager__InsertAllocator(AllocatorManager* p, AllocatorPtr allocator) override { p->InsertAllocator(allocator); } - AllocatorPtr AllocatorManager__GetAllocator(AllocatorManager* p, int id, OrtMemType mem_type) override { return p->GetAllocator(id, mem_type); }; + // AllocatorManager (direct) + void AllocatorManager__InsertAllocator(AllocatorManager* p, AllocatorPtr allocator) override { p->AllocatorManager::InsertAllocator(allocator); } + AllocatorPtr AllocatorManager__GetAllocator(const AllocatorManager* p, int id, OrtMemType mem_type) override { return p->AllocatorManager::GetAllocator(id, mem_type); }; +#ifdef USE_CUDA + // GatherElements (direct) + Status GatherElements__ValidateInputShapes(const TensorShape& input_data_shape, const TensorShape& indices_shape, int64_t axis) override { return GatherElements::ValidateInputShapes(input_data_shape, indices_shape, axis); } + + // cumsum (direct) + Status cumsum_op__GetAxis(const Tensor* axis_tensor, int64_t input_rank, int64_t& axis_out) override { return cumsum_op::GetAxis(axis_tensor, input_rank, axis_out); } + + // TileOp (direct) + bool TileOp__IsTileMemcpy(const TensorShape& input_shape, const int64_t* repeats, size_t rank, bool& is_batched_memcpy, size_t& num_of_elements_per_batch, size_t& num_of_copies_per_batch, size_t& num_of_batch_copies) override { return TileOp::IsTileMemcpy(input_shape, repeats, rank, is_batched_memcpy, num_of_elements_per_batch, num_of_copies_per_batch, num_of_batch_copies); } + + // ROI (direct) + Status CheckROIAlignValidInput(const Tensor* X_ptr, const Tensor* rois_ptr, const Tensor* batch_indices_ptr) override { return onnxruntime::CheckROIAlignValidInput(X_ptr, rois_ptr, batch_indices_ptr); } + + // NonMaxSuppressionBase (direct) + Status NonMaxSuppressionBase__PrepareCompute(OpKernelContext* ctx, PrepareContext& pc) override { return NonMaxSuppressionBase::PrepareCompute(ctx, pc); } + Status NonMaxSuppressionBase__GetThresholdsFromInputs(const PrepareContext& pc, int64_t& max_output_boxes_per_class, float& iou_threshold, float& score_threshold) override { return NonMaxSuppressionBase::GetThresholdsFromInputs(pc, max_output_boxes_per_class, iou_threshold, score_threshold); } + + // From onehot.h (direct) + Status ValidateInputs(const Tensor* depth, const Tensor* values) override { return onnxruntime::ValidateInputs(depth, values); } + Status PrepareOutputShape(const Tensor* indices, const int64_t depth_val, const int64_t axis, int64_t& prefix_dim_size, int64_t& suffix_dim_size, std::vector& output_shape) override { return onnxruntime::PrepareOutputShape(indices, depth_val, axis, prefix_dim_size, suffix_dim_size, output_shape); } + + // From cpu/tensor/unsqueeze.h (direct) + Status UnsqueezeBase__PrepareCompute(const UnsqueezeBase* p, OpKernelContext* ctx, UnsqueezeBase__Prepare& prepare) override { return p->UnsqueezeBase::PrepareCompute(ctx, reinterpret_cast(prepare)); } + // From cpu/tensor/slice.h (direct) + Status SliceBase__PrepareForCompute(const std::vector& raw_starts, + const std::vector& raw_ends, + const std::vector& raw_axes, + SliceOp__PrepareForComputeMetadata& compute_metadata) override { return SliceBase::PrepareForCompute(raw_starts, raw_ends, raw_axes, reinterpret_cast(compute_metadata)); } + + Status SliceBase__PrepareForCompute(const std::vector& raw_starts, + const std::vector& raw_ends, + const std::vector& raw_axes, + const std::vector& raw_steps, + SliceOp__PrepareForComputeMetadata& compute_metadata) override { return SliceBase::PrepareForCompute(raw_starts, raw_ends, raw_axes, raw_steps, reinterpret_cast(compute_metadata)); } + + Status SliceBase__FillVectorsFromInput(const Tensor& start_tensor, + const Tensor& ends_tensor, + const Tensor* axes_tensor, + const Tensor* steps_tensor, + std::vector& input_starts, + std::vector& input_ends, + std::vector& input_axes, + std::vector& input_steps) override { return SliceBase::FillVectorsFromInput(start_tensor, ends_tensor, axes_tensor, steps_tensor, input_starts, input_ends, input_axes, input_steps); } + // From cpu/tensor/size.h (direct) + Status Size__Compute(const Size* p, OpKernelContext* context) override { return p->Size::Compute(context); } + // From cpu/tensor/scatter_nd.h (direct) + Status ScatterNDBase__ValidateShapes(const TensorShape& input_shape, + const TensorShape& indice_shape, + const TensorShape& update_shape) override { return ScatterNDBase::ValidateShapes(input_shape, indice_shape, update_shape); } + // From cpu/tensor/padbase.h (direct) + Status PadBase__HandleDimValueZero(const Mode& mode, const TensorShape& input_shape, TensorShape& output_shape) override { return PadBase::HandleDimValueZero(mode, input_shape, output_shape); } + // From cpu/tensor/split.h (direct) + Status SplitBase__PrepareForCompute(const SplitBase* p, const TensorShape& input_shape, int num_outputs, int64_t& axis, int& before_dims, + int& after_dims_including_split_axis, int& after_dims_excluding_split, + std::vector& split_sizes) override { return p->SplitBase::PrepareForCompute(input_shape, num_outputs, axis, before_dims, after_dims_including_split_axis, after_dims_excluding_split, split_sizes); } + // From cpu/tensor/concatbase.h (direct) + Status ConcatBase__PrepareForCompute(const ConcatBase* p, OpKernelContext* ctx, const std::vector& input_tensors, Prepare& prepare) override { return p->ConcatBase::PrepareForCompute(ctx, input_tensors, prepare); } + // From cpu/tensor/gatherbase.h (direct) + Status GatherBase__PrepareForCompute(const GatherBase* p, OpKernelContext* context, GatherBase__Prepare& prepare) override { return p->GatherBase::PrepareForCompute(context, reinterpret_cast(prepare)); } + + PhiloxGenerator& PhiloxGenerator__Default() override { return PhiloxGenerator::Default(); } + + Status Einsum__Compute(const Einsum* p, OpKernelContext* context) override { return p->Einsum::Compute(context); } + + // EinsumComputePreprocessor (wrapped) + void EinsumComputePreprocessor__operator_delete(EinsumComputePreprocessor* p) override { delete p; } + virtual std::unique_ptr EinsumComputePreprocessor__Create(EinsumEquationPreprocessor& equation_preprocessor, + const std::vector& inputs, + AllocatorPtr allocator, + void* einsum_cuda_assets) override { return std::make_unique(equation_preprocessor, inputs, allocator, einsum_cuda_assets); } + + virtual Status EinsumComputePreprocessor__Run(EinsumComputePreprocessor* p) override { return p->Run(); } + virtual void EinsumComputePreprocessor__SetDeviceHelpers(EinsumComputePreprocessor* p, const EinsumOp::DeviceHelpers::Diagonal& diagonal_func, const EinsumOp::DeviceHelpers::Transpose& transpose_func) override { return p->SetDeviceHelpers(diagonal_func, transpose_func); } + + // EinsumTypedComputeProcessor (wrapped) + void EinsumTypedComputeProcessor__operator_delete(EinsumTypedComputeProcessor* p) override { delete p; } + void EinsumTypedComputeProcessor__operator_delete(EinsumTypedComputeProcessor* p) override { delete p; } + void EinsumTypedComputeProcessor__operator_delete(EinsumTypedComputeProcessor* p) override { delete p; } + std::unique_ptr> EinsumTypedComputeProcessor_float__Create(OpKernelContext* context, AllocatorPtr allocator, concurrency::ThreadPool* tp, EinsumComputePreprocessor& einsum_compute_preprocessor, void* einsum_cuda_assets) override { return std::make_unique>(context, allocator, tp, einsum_compute_preprocessor, einsum_cuda_assets); } + std::unique_ptr> EinsumTypedComputeProcessor_double__Create(OpKernelContext* context, AllocatorPtr allocator, concurrency::ThreadPool* tp, EinsumComputePreprocessor& einsum_compute_preprocessor, void* einsum_cuda_assets) override { return std::make_unique>(context, allocator, tp, einsum_compute_preprocessor, einsum_cuda_assets); } + std::unique_ptr> EinsumTypedComputeProcessor_MLFloat16__Create(OpKernelContext* context, AllocatorPtr allocator, concurrency::ThreadPool* tp, EinsumComputePreprocessor& einsum_compute_preprocessor, void* einsum_cuda_assets) override { return std::make_unique>(context, allocator, tp, einsum_compute_preprocessor, einsum_cuda_assets); } + void EinsumTypedComputeProcessor__SetDeviceHelpers(EinsumTypedComputeProcessor* p, const EinsumOp::DeviceHelpers::Transpose& device_transpose_func, const EinsumOp::DeviceHelpers::MatMul& device_matmul_func, const EinsumOp::DeviceHelpers::ReduceSum& device_reduce_sum_func, const EinsumOp::DeviceHelpers::DataCopy& device_data_copy_func) override { return p->SetDeviceHelpers(device_transpose_func, device_matmul_func, device_reduce_sum_func, device_data_copy_func); } + void EinsumTypedComputeProcessor__SetDeviceHelpers(EinsumTypedComputeProcessor* p, const EinsumOp::DeviceHelpers::Transpose& device_transpose_func, const EinsumOp::DeviceHelpers::MatMul& device_matmul_func, const EinsumOp::DeviceHelpers::ReduceSum& device_reduce_sum_func, const EinsumOp::DeviceHelpers::DataCopy& device_data_copy_func) override { return p->SetDeviceHelpers(device_transpose_func, device_matmul_func, device_reduce_sum_func, device_data_copy_func); } + void EinsumTypedComputeProcessor__SetDeviceHelpers(EinsumTypedComputeProcessor* p, const EinsumOp::DeviceHelpers::Transpose& device_transpose_func, const EinsumOp::DeviceHelpers::MatMul& device_matmul_func, const EinsumOp::DeviceHelpers::ReduceSum& device_reduce_sum_func, const EinsumOp::DeviceHelpers::DataCopy& device_data_copy_func) override { return p->SetDeviceHelpers(device_transpose_func, device_matmul_func, device_reduce_sum_func, device_data_copy_func); } + Status EinsumTypedComputeProcessor__Run(EinsumTypedComputeProcessor* p) override { return p->Run(); } + Status EinsumTypedComputeProcessor__Run(EinsumTypedComputeProcessor* p) override { return p->Run(); } + Status EinsumTypedComputeProcessor__Run(EinsumTypedComputeProcessor* p) override { return p->Run(); } + + // If (direct) + void If__Init(If* p, const OpKernelInfo& info) override { p->If::Init(info); } + Status If__Compute(const If* p, OpKernelContext* ctx) override { return p->If::Compute(ctx); } + Status If__SetupSubgraphExecutionInfo(If* p, const SessionState& session_state, const std::string& attribute_name, const SessionState& subgraph_session_state) override { return p->If::SetupSubgraphExecutionInfo(session_state, attribute_name, subgraph_session_state); } + + // Loop (direct) + void Loop__Init(Loop* p, const OpKernelInfo& info) override { p->Loop::Init(info); } + Status Loop__Compute(const Loop* p, OpKernelContext* ctx) override { return p->Loop::Compute(ctx); } + Status Loop__SetupSubgraphExecutionInfo(Loop* p, const SessionState& session_state, const std::string& attribute_name, const SessionState& subgraph_session_state) override { return p->Loop::SetupSubgraphExecutionInfo(session_state, attribute_name, subgraph_session_state); } + + // Scan (direct) + void Scan__Init(Scan<8>* p, const OpKernelInfo& info) override { p->Scan::Init(info); } + void Scan__Init(Scan<9>* p, const OpKernelInfo& info) override { p->Scan::Init(info); } + Status Scan__Compute(const Scan<8>* p, OpKernelContext* ctx) override { return p->Scan<8>::Compute(ctx); } + Status Scan__Compute(const Scan<9>* p, OpKernelContext* ctx) override { return p->Scan<9>::Compute(ctx); } + Status Scan__SetupSubgraphExecutionInfo(Scan<8>* p, const SessionState& session_state, const std::string& attribute_name, const SessionState& subgraph_session_state) override { return p->Scan<8>::SetupSubgraphExecutionInfo(session_state, attribute_name, subgraph_session_state); } + Status Scan__SetupSubgraphExecutionInfo(Scan<9>* p, const SessionState& session_state, const std::string& attribute_name, const SessionState& subgraph_session_state) override { return p->Scan<9>::SetupSubgraphExecutionInfo(session_state, attribute_name, subgraph_session_state); } + + // ContribOps (direct) +#ifndef DISABLE_CONTRIB_OPS + Status embed_layer_norm__CheckInputs(const OpKernelContext* context) override { return contrib::embed_layer_norm::CheckInputs(context); } + Status bias_gelu_helper__CheckInputs(const OpKernelContext* context) override { return contrib::bias_gelu_helper::CheckInputs(context); } + Status LongformerAttentionBase__CheckInputs(const contrib::LongformerAttentionBase* p, const TensorShape& input_shape, const TensorShape& weights_shape, const TensorShape& bias_shape, const TensorShape& mask_shape, const TensorShape& global_weights_shape, const TensorShape& global_bias_shape, const TensorShape& global_shape) override { + return contrib::LongformerAttentionBase__CheckInputs(p, input_shape, weights_shape, bias_shape, mask_shape, global_weights_shape, global_bias_shape, global_shape); + } + Status AttentionBase__CheckInputs(const contrib::AttentionBase* p, const TensorShape& input_shape, const TensorShape& weights_shape, const TensorShape& bias_shape, const Tensor*& mask_index, const Tensor* past, const int max_threads_per_block) override { return p->contrib::AttentionBase::CheckInputs(input_shape, weights_shape, bias_shape, mask_index, past, max_threads_per_block); } + Tensor* AttentionBase__GetPresent(const contrib::AttentionBase* p, OpKernelContext* context, const Tensor* past, int batch_size, int head_size, int sequence_length, int& past_sequence_length) override { return p->contrib::AttentionBase::GetPresent(context, past, batch_size, head_size, sequence_length, past_sequence_length); } +#endif + +#ifdef ENABLE_TRAINING + void ATenOpBase__Init(contrib::ATenOpBase* p, const OpKernelInfo& info, bool is_backward) override { return p->ATenOpBase::Init(info, is_backward); } + Status ATenOpBase__Compute(const contrib::ATenOpBase* p, OpKernelContext* p_ctx) override { return p->ATenOpBase::Compute(p_ctx); } + void contrib__record_event_in_tensor(const Tensor& event_id_tensor) override { return contrib::record_event_in_tensor(event_id_tensor); } + void contrib__wait_event_in_tensor(const Tensor& event_id_tensor) override { return contrib::wait_event_in_tensor(event_id_tensor); } + Status contrib__Group__Compute(const contrib::Group* p, OpKernelContext* context) override { return p->Group::Compute(context); } + Status contrib__PassThrough__Compute(const contrib::PassThrough* p, OpKernelContext* context) override { return p->PassThrough::Compute(context); } + void contrib__VerifyLogitWeightAndLabelShape(const TensorShape& logit_shape, const TensorShape& label_shape, const TensorShape* weight_shape) override { contrib::VerifyLogitWeightAndLabelShape(logit_shape, label_shape, weight_shape); } + void contrib__GetNDCFromLogitAndLabelShape(const TensorShape& logit_shape, const TensorShape& label_shape, int64_t& N_D, int64_t& C) override { contrib::GetNDCFromLogitAndLabelShape(logit_shape, label_shape, N_D, C); } + void contrib__GetPermutationAndShape(bool ncd_to_ndc, const TensorShape& tensor_shape, std::vector& new_shape, std::vector& permutations) override { contrib::GetPermutationAndShape(ncd_to_ndc, tensor_shape, new_shape, permutations); } + Status contrib__PrepareForTrainingCompute(const TensorShape& input_shape, int num_outputs, int64_t& axis, int& before_dims, int& after_dims_including_split_axis, int& after_dims_excluding_split, std::vector& split_sizes) override { return contrib::PrepareForTrainingCompute(input_shape, num_outputs, axis, before_dims, after_dims_including_split_axis, after_dims_excluding_split, split_sizes); } + Status contrib__YieldOp__Compute(const contrib::YieldOp* p, OpKernelContext* context) override { return p->YieldOp::Compute(context); } + +#if defined(ORT_USE_NCCL) && defined(USE_NCCL_P2P) + training::DistributedRunContext& GetDistributedRunContextInstance() override { return training::DistributedRunContext::GetInstance(); } +#endif +#endif +#endif } provider_host_; struct ProviderSharedLibrary { @@ -584,8 +917,9 @@ struct ProviderSharedLibrary { } ProviderSharedLibrary() = default; - ~ProviderSharedLibrary() { /*assert(!handle_);*/ - } // We should already be unloaded at this point (disabled until Python shuts down deterministically) + ~ProviderSharedLibrary() { + // assert(!handle_); // We should already be unloaded at this point (disabled until Python shuts down deterministically) + } private: void* handle_{}; @@ -595,14 +929,15 @@ struct ProviderSharedLibrary { static ProviderSharedLibrary s_library_shared; -bool InitProvidersSharedLibrary(){ +bool InitProvidersSharedLibrary() { return s_library_shared.Ensure(); } struct ProviderLibrary { - ProviderLibrary(const char* filename) : filename_{filename} {} - ~ProviderLibrary() { /*assert(!handle_);*/ - } // We should already be unloaded at this point (disabled until Python shuts down deterministically) + ProviderLibrary(const char* filename, bool unload = true) : filename_{filename}, unload_{unload} {} + ~ProviderLibrary() { + // assert(!handle_); // We should already be unloaded at this point (disabled until Python shuts down deterministically) + } Provider* Get() { if (provider_) @@ -630,7 +965,9 @@ struct ProviderLibrary { if (provider_) provider_->Shutdown(); - Env::Default().UnloadDynamicLibrary(handle_); + if (unload_) + Env::Default().UnloadDynamicLibrary(handle_); + handle_ = nullptr; provider_ = nullptr; } @@ -638,12 +975,19 @@ struct ProviderLibrary { private: const char* filename_; + bool unload_; Provider* provider_{}; void* handle_{}; ORT_DISALLOW_COPY_AND_ASSIGNMENT(ProviderLibrary); }; +static ProviderLibrary s_library_cuda(LIBRARY_PREFIX "onnxruntime_providers_cuda" LIBRARY_EXTENSION +#ifndef _WIN32 + , + false /* unload - On Linux if we unload the cuda shared provider we crash */ +#endif +); static ProviderLibrary s_library_dnnl(LIBRARY_PREFIX "onnxruntime_providers_dnnl" LIBRARY_EXTENSION); static ProviderLibrary s_library_openvino(LIBRARY_PREFIX "onnxruntime_providers_openvino" LIBRARY_EXTENSION); static ProviderLibrary s_library_tensorrt(LIBRARY_PREFIX "onnxruntime_providers_tensorrt" LIBRARY_EXTENSION); @@ -652,43 +996,91 @@ void UnloadSharedProviders() { s_library_dnnl.Unload(); s_library_openvino.Unload(); s_library_tensorrt.Unload(); + s_library_cuda.Unload(); s_library_shared.Unload(); } +// Used by test code +std::unique_ptr CreateCUDAPinnedAllocator(int16_t device_id, const char* name) { + if (auto* info = onnxruntime::GetProviderInfo_CUDA()) + return info->CreateCUDAPinnedAllocator(device_id, name); + + return nullptr; +} + +std::shared_ptr CreateExecutionProviderFactory_Cuda(const OrtCUDAProviderOptions* provider_options) { + if (auto* provider = s_library_cuda.Get()) + return provider->CreateExecutionProviderFactory(provider_options); + + return nullptr; +} + std::shared_ptr CreateExecutionProviderFactory_Dnnl(int use_arena) { - if (auto provider = s_library_dnnl.Get()) + if (auto* provider = s_library_dnnl.Get()) return provider->CreateExecutionProviderFactory(use_arena); return nullptr; } std::shared_ptr CreateExecutionProviderFactory_Tensorrt(int device_id) { - if (auto provider = s_library_tensorrt.Get()) + if (auto* provider = s_library_tensorrt.Get()) return provider->CreateExecutionProviderFactory(device_id); return nullptr; } std::shared_ptr CreateExecutionProviderFactory_Tensorrt(const OrtTensorRTProviderOptions* provider_options) { - if (auto provider = s_library_tensorrt.Get()) + if (auto* provider = s_library_tensorrt.Get()) return provider->CreateExecutionProviderFactory(provider_options); return nullptr; } std::shared_ptr CreateExecutionProviderFactory_OpenVINO(const OrtOpenVINOProviderOptions* provider_options) { - if (auto provider = s_library_openvino.Get()) + if (auto* provider = s_library_openvino.Get()) return provider->CreateExecutionProviderFactory(provider_options); return nullptr; } -const ProviderInfo_OpenVINO* GetProviderInfo_OpenVINO() { - if (auto provider = s_library_openvino.Get()) - return reinterpret_cast(provider->GetInfo()); +ProviderInfo_OpenVINO* GetProviderInfo_OpenVINO() { + if (auto* provider = s_library_openvino.Get()) + return reinterpret_cast(provider->GetInfo()); return nullptr; } +ProviderInfo_CUDA* GetProviderInfo_CUDA() { + if (auto* provider = s_library_cuda.Get()) + return reinterpret_cast(provider->GetInfo()); + LOGS_DEFAULT(WARNING) << "GetProviderInfo_CUDA called, returning nullptr"; + ORT_THROW("CUDA Provider not available, can't get interface for it"); +} + +void CopyGpuToCpu( + void* dst_ptr, + const void* src_ptr, + const size_t size, + const OrtMemoryInfo& dst_location, + const OrtMemoryInfo& src_location) { + if (auto* info = onnxruntime::GetProviderInfo_CUDA()) + return info->CopyGpuToCpu(dst_ptr, src_ptr, size, dst_location, src_location); + ORT_THROW("GPU-to-CPU copy is not implemented."); +} + +void cudaMemcpy_HostToDevice(void* dst, const void* src, size_t count) { + if (auto* info = onnxruntime::GetProviderInfo_CUDA()) + return info->cudaMemcpy_HostToDevice(dst, src, count); + ORT_THROW("cudaMemcpy_HostToDevice is not implemented."); +} + +#if defined(USE_CUDA) && defined(ORT_USE_NCCL) && defined(USE_NCCL_P2P) +namespace cuda { +INcclService& INcclService::GetInstance() { + return GetProviderInfo_CUDA()->GetINcclService(); +} +} // namespace cuda +#endif + } // namespace onnxruntime ORT_API_STATUS_IMPL(OrtSessionOptionsAppendExecutionProvider_Dnnl, _In_ OrtSessionOptions* options, int use_arena) { @@ -734,9 +1126,32 @@ ORT_API_STATUS_IMPL(OrtApis::SessionOptionsAppendExecutionProvider_OpenVINO, _In ORT_API_STATUS_IMPL(OrtSessionOptionsAppendExecutionProvider_OpenVINO, _In_ OrtSessionOptions* options, _In_ const char* device_type) { OrtOpenVINOProviderOptions provider_options; provider_options.device_type = device_type; - auto factory = onnxruntime::CreateExecutionProviderFactory_OpenVINO(&provider_options); + return OrtApis::SessionOptionsAppendExecutionProvider_OpenVINO(options, &provider_options); +} + +ORT_API_STATUS_IMPL(OrtSessionOptionsAppendExecutionProvider_CUDA, _In_ OrtSessionOptions* options, int device_id) { + OrtCUDAProviderOptions provider_options{}; + provider_options.device_id = device_id; + + return OrtApis::SessionOptionsAppendExecutionProvider_CUDA(options, &provider_options); +} + +ORT_API_STATUS_IMPL(OrtApis::SetCurrentGpuDeviceId, _In_ int device_id) { + if (auto* info = onnxruntime::GetProviderInfo_CUDA()) + return info->SetCurrentGpuDeviceId(device_id); + return CreateStatus(ORT_FAIL, "CUDA execution provider is not enabled."); +} + +ORT_API_STATUS_IMPL(OrtApis::GetCurrentGpuDeviceId, _In_ int* device_id) { + if (auto* info = onnxruntime::GetProviderInfo_CUDA()) + return info->GetCurrentGpuDeviceId(device_id); + return CreateStatus(ORT_FAIL, "CUDA execution provider is not enabled."); +} + +ORT_API_STATUS_IMPL(OrtApis::SessionOptionsAppendExecutionProvider_CUDA, _In_ OrtSessionOptions* options, _In_ const OrtCUDAProviderOptions* cuda_options) { + auto factory = onnxruntime::CreateExecutionProviderFactory_Cuda(cuda_options); if (!factory) { - return OrtApis::CreateStatus(ORT_FAIL, "OrtSessionOptionsAppendExecutionProvider_OpenVINO: Failed to load shared library"); + return OrtApis::CreateStatus(ORT_FAIL, "OrtSessionOptionsAppendExecutionProvider_Cuda: Failed to load shared library"); } options->provider_factories.push_back(factory); diff --git a/onnxruntime/core/framework/provider_bridge_ort.h b/onnxruntime/core/framework/provider_bridge_ort.h index 0d96a19d70..2ea1bb896b 100644 --- a/onnxruntime/core/framework/provider_bridge_ort.h +++ b/onnxruntime/core/framework/provider_bridge_ort.h @@ -6,5 +6,6 @@ namespace onnxruntime { bool InitProvidersSharedLibrary(); +void UnloadSharedProviders(); } // namespace onnxruntime diff --git a/onnxruntime/core/framework/tensorprotoutils.h b/onnxruntime/core/framework/tensorprotoutils.h index f429349c5f..9f79968fdd 100644 --- a/onnxruntime/core/framework/tensorprotoutils.h +++ b/onnxruntime/core/framework/tensorprotoutils.h @@ -6,6 +6,7 @@ #include #include +#ifndef SHARED_PROVIDER #include "core/common/common.h" #include "core/common/path.h" #include "core/common/status.h" @@ -26,15 +27,17 @@ bool operator==(const TensorShapeProto_Dimension& l, const TensorShapeProto_Dime bool operator!=(const TensorShapeProto_Dimension& l, const TensorShapeProto_Dimension& r); } // namespace ONNX_NAMESPACE +#endif namespace onnxruntime { -class Tensor; namespace utils { +#ifndef SHARED_PROVIDER + TensorShape GetTensorShapeFromTensorShapeProto(const ONNX_NAMESPACE::TensorShapeProto& tensor_shape_proto); std::vector GetTensorShapeFromTensorProto(const ONNX_NAMESPACE::TensorProto& tensor_proto); - /** +/** * deserialize a TensorProto into a preallocated memory buffer. * \param tensor_proto_path A local file path of where the 'input' was loaded from. Can be NULL if the tensor proto doesn't * have any external data or it was loaded from current working dir. This path could be either a @@ -51,8 +54,8 @@ common::Status TensorProtoToMLValue(const Env& env, const ORTCHAR_T* tensor_prot * @return */ common::Status TensorProtoToTensor(const Env& env, const ORTCHAR_T* model_path, - const ONNX_NAMESPACE::TensorProto& tensor_proto, - Tensor& tensor); + const ONNX_NAMESPACE::TensorProto& tensor_proto, + Tensor& tensor); /** Creates a TensorProto from a Tensor. @param[in] tensor the Tensor whose data and shape will be used to create the TensorProto. @@ -99,6 +102,7 @@ common::Status DenseTensorToSparseTensorProto(const ONNX_NAMESPACE::TensorProto& const Path& model_path, ONNX_NAMESPACE::SparseTensorProto& sparse); #endif // !ORT_MINIMAL_BUILD +#endif inline bool HasDimValue(const ONNX_NAMESPACE::TensorShapeProto_Dimension& dim) { return dim.value_case() == ONNX_NAMESPACE::TensorShapeProto_Dimension::kDimValue; @@ -122,10 +126,12 @@ inline bool HasShape(const ONNX_NAMESPACE::TypeProto_Tensor& ten_proto) { return ten_proto.has_shape(); } +#ifndef SHARED_PROVIDER inline bool HasShape(const ONNX_NAMESPACE::TypeProto_SparseTensor& ten_proto) { // XXX: Figure out how do in proto3 return ten_proto.has_shape(); } +#endif inline bool HasRawData(const ONNX_NAMESPACE::TensorProto& ten_proto) { // Can not be UNDEFINED and can not be STRING but test for STRING is usually performed separately @@ -145,6 +151,7 @@ inline bool HasDataType(const ONNX_NAMESPACE::TensorProto& ten_proto) { return ten_proto.data_type() != ONNX_NAMESPACE::TensorProto::UNDEFINED; } +#ifndef SHARED_PROVIDER inline bool HasName(const ONNX_NAMESPACE::TensorProto& ten_proto) { return ten_proto.has_name(); // XXX } @@ -168,11 +175,13 @@ inline bool HasKeyType(const ONNX_NAMESPACE::TypeProto_Map& map_proto) { inline bool HasValueType(const ONNX_NAMESPACE::TypeProto_Map& map_proto) { return map_proto.value_type().value_case() != ONNX_NAMESPACE::TypeProto::VALUE_NOT_SET; } +#endif inline bool HasType(const ONNX_NAMESPACE::ValueInfoProto& vi_proto) { return vi_proto.type().value_case() != ONNX_NAMESPACE::TypeProto::VALUE_NOT_SET; } +#ifndef SHARED_PROVIDER inline bool HasName(const ONNX_NAMESPACE::ValueInfoProto& vi_proto) { return vi_proto.has_name(); // XXX: Figure out proto3 way } @@ -184,6 +193,7 @@ inline bool HasDomain(const ONNX_NAMESPACE::TypeProto_Opaque& op_proto) { inline bool HasName(const ONNX_NAMESPACE::TypeProto_Opaque& op_proto) { return !op_proto.name().empty(); } +#endif inline bool HasType(const ONNX_NAMESPACE::AttributeProto& at_proto) { return at_proto.type() != ONNX_NAMESPACE::AttributeProto::AttributeType::AttributeProto_AttributeType_UNDEFINED; @@ -229,6 +239,7 @@ inline bool HasGraphs(const ONNX_NAMESPACE::AttributeProto& at_proto) { return at_proto.type() == ONNX_NAMESPACE::AttributeProto::AttributeType::AttributeProto_AttributeType_GRAPHS; } +#ifndef SHARED_PROVIDER inline bool HasName(const ONNX_NAMESPACE::AttributeProto& at_proto) { return at_proto.has_name(); // XXX: Fugure out proto3 } @@ -249,6 +260,7 @@ inline bool HasName(const ONNX_NAMESPACE::NodeProto& node_proto) { //XXX: Figure out proto3 style return node_proto.has_name(); } +#endif // UnpackTensor from raw data or the type specific data field. Does not handle external data. // If the tensor does not contain raw data then raw_data should be nullptr and raw_data_len should be 0. @@ -276,6 +288,5 @@ common::Status UnpackInitializerData(const ONNX_NAMESPACE::TensorProto& initiali const Path& model_path, std::unique_ptr& unpacked_tensor, size_t& tensor_byte_size) ORT_MUST_USE_RESULT; - } // namespace utils } // namespace onnxruntime diff --git a/onnxruntime/core/platform/env_var_utils.h b/onnxruntime/core/platform/env_var_utils.h index 3c8fba76bf..61d086a28a 100644 --- a/onnxruntime/core/platform/env_var_utils.h +++ b/onnxruntime/core/platform/env_var_utils.h @@ -14,7 +14,11 @@ namespace onnxruntime { */ template optional ParseEnvironmentVariable(const std::string& name) { +#ifndef SHARED_PROVIDER const std::string value_str = Env::Default().GetEnvironmentVar(name); +#else + const std::string value_str = GetEnvironmentVar(name); +#endif if (value_str.empty()) { return {}; } diff --git a/onnxruntime/core/providers/common.h b/onnxruntime/core/providers/common.h index 3688133c6d..fca366b295 100644 --- a/onnxruntime/core/providers/common.h +++ b/onnxruntime/core/providers/common.h @@ -6,8 +6,10 @@ #include #include +#ifndef SHARED_PROVIDER #include "core/common/common.h" #include "core/framework/tensor.h" +#endif namespace onnxruntime { diff --git a/onnxruntime/core/providers/cpu/controlflow/if.cc b/onnxruntime/core/providers/cpu/controlflow/if.cc index 5caae9bb9d..df34f67234 100644 --- a/onnxruntime/core/providers/cpu/controlflow/if.cc +++ b/onnxruntime/core/providers/cpu/controlflow/if.cc @@ -94,34 +94,24 @@ ONNX_CPU_OPERATOR_KERNEL(If, .TypeConstraint("V", DataTypeImpl::AllTensorAndSequenceTensorTypes()), If); -struct If::Info { - Info(const onnxruntime::Node& node, const GraphViewer& subgraph_in) : subgraph(subgraph_in) { - num_implicit_inputs = static_cast(node.ImplicitInputDefs().size()); - used_implicit_inputs = std::vector(num_implicit_inputs, true); - num_outputs = static_cast(node.OutputDefs().size()); +If::Info::Info(const onnxruntime::Node& node, const GraphViewer& subgraph_in) : subgraph(subgraph_in) { + num_implicit_inputs = static_cast(node.ImplicitInputDefs().size()); + used_implicit_inputs = std::vector(num_implicit_inputs, true); + num_outputs = static_cast(node.OutputDefs().size()); - auto& subgraph_outputs = subgraph.GetOutputs(); - auto num_subgraph_outputs = subgraph_outputs.size(); + auto& subgraph_outputs = subgraph.GetOutputs(); + auto num_subgraph_outputs = subgraph_outputs.size(); - ORT_ENFORCE(num_subgraph_outputs == static_cast(num_outputs), - "'If' node has ", num_outputs, " outputs which doesn't match the subgraph's ", - num_subgraph_outputs, " outputs."); + ORT_ENFORCE(num_subgraph_outputs == static_cast(num_outputs), + "'If' node has ", num_outputs, " outputs which doesn't match the subgraph's ", + num_subgraph_outputs, " outputs."); - subgraph_output_names.reserve(num_subgraph_outputs); - for (size_t i = 0; i < num_subgraph_outputs; ++i) { - auto& output = subgraph_outputs[i]; - subgraph_output_names.push_back(output->Name()); - } + subgraph_output_names.reserve(num_subgraph_outputs); + for (size_t i = 0; i < num_subgraph_outputs; ++i) { + auto& output = subgraph_outputs[i]; + subgraph_output_names.push_back(output->Name()); } - - const GraphViewer& subgraph; - - std::vector used_implicit_inputs; - int num_implicit_inputs; - int num_outputs; - - std::vector subgraph_output_names; -}; +} class IfImpl { public: @@ -154,7 +144,7 @@ class IfImpl { std::vector> outputs_; }; -If::If(const OpKernelInfo& info) : IControlFlowKernel(info) { +void If::Init(const OpKernelInfo& info) { // make sure the required attributes are present even though we don't need it here. // The GraphProto attributes are loaded as a Graph instance by main Graph::Resolve, // and a SessionState instance for executing the subgraph is created by InferenceSession. @@ -165,9 +155,6 @@ If::If(const OpKernelInfo& info) : IControlFlowKernel(info) { ORT_IGNORE_RETURN_VALUE(proto); } -// we need this to be in the .cc so 'unique_ptr info_' can be handled -If::~If() = default; - common::Status If::SetupSubgraphExecutionInfo(const SessionState& session_state, const std::string& attribute_name, const SessionState& subgraph_session_state) { diff --git a/onnxruntime/core/providers/cpu/controlflow/if.h b/onnxruntime/core/providers/cpu/controlflow/if.h index 5832af3818..ffbbc3185c 100644 --- a/onnxruntime/core/providers/cpu/controlflow/if.h +++ b/onnxruntime/core/providers/cpu/controlflow/if.h @@ -5,9 +5,6 @@ #include #include "gsl/gsl" -#include "core/common/common.h" -#include "core/framework/feeds_fetches_manager.h" -#include "core/framework/op_kernel.h" #include "core/providers/cpu/controlflow/utils.h" namespace onnxruntime { @@ -15,17 +12,25 @@ class SessionState; class If : public controlflow::IControlFlowKernel { public: - If(const OpKernelInfo& info); + If(const OpKernelInfo& info) : IControlFlowKernel(info) { Init(info); } + void Init(const OpKernelInfo& info); Status Compute(OpKernelContext* ctx) const override; - common::Status SetupSubgraphExecutionInfo(const SessionState& session_state, - const std::string& attribute_name, - const SessionState& subgraph_session_state) override; + Status SetupSubgraphExecutionInfo(const SessionState& session_state, + const std::string& attribute_name, + const SessionState& subgraph_session_state) override; - // hide internal implementation details via forward declaration. - struct Info; - ~If(); + struct Info { + Info(const onnxruntime::Node& node, const GraphViewer& subgraph_in); + const GraphViewer& subgraph; + + std::vector used_implicit_inputs; + int num_implicit_inputs; + int num_outputs; + + std::vector subgraph_output_names; + }; private: // Info and FeedsFetchesManager re-used for each subgraph execution. diff --git a/onnxruntime/core/providers/cpu/controlflow/loop.cc b/onnxruntime/core/providers/cpu/controlflow/loop.cc index c3801bc8d1..46b38a88f3 100644 --- a/onnxruntime/core/providers/cpu/controlflow/loop.cc +++ b/onnxruntime/core/providers/cpu/controlflow/loop.cc @@ -122,56 +122,42 @@ ONNX_CPU_OPERATOR_KERNEL(Loop, .TypeConstraint("V", DataTypeImpl::AllTensorAndSequenceTensorTypes()), Loop); -struct Loop::Info { - Info(const onnxruntime::Node& node, const GraphViewer& subgraph_in) - : subgraph(subgraph_in) { - num_loop_carried_vars = static_cast(node.InputDefs().size()) - 2; // skip 'M' and 'cond' - num_implicit_inputs = static_cast(node.ImplicitInputDefs().size()); - num_subgraph_inputs = 2 + num_loop_carried_vars; // iter_num, cond, loop carried vars - num_outputs = static_cast(node.OutputDefs().size()); +Loop::Info::Info(const onnxruntime::Node& node, const GraphViewer& subgraph_in) + : subgraph(subgraph_in) { + num_loop_carried_vars = static_cast(node.InputDefs().size()) - 2; // skip 'M' and 'cond' + num_implicit_inputs = static_cast(node.ImplicitInputDefs().size()); + num_subgraph_inputs = 2 + num_loop_carried_vars; // iter_num, cond, loop carried vars + num_outputs = static_cast(node.OutputDefs().size()); - auto& subgraph_inputs = subgraph.GetInputs(); - auto& subgraph_outputs = subgraph.GetOutputs(); + auto& subgraph_inputs = subgraph.GetInputs(); + auto& subgraph_outputs = subgraph.GetOutputs(); - // we know how many inputs we are going to call the subgraph with based on the Loop inputs, - // and that value is in num_subgraph_inputs. - // validate that the subgraph has that many inputs. - ORT_ENFORCE(static_cast(num_subgraph_inputs) == subgraph_inputs.size(), - "Graph in 'body' attribute of Loop should have ", num_subgraph_inputs, " inputs. Found:", - subgraph_inputs.size()); + // we know how many inputs we are going to call the subgraph with based on the Loop inputs, + // and that value is in num_subgraph_inputs. + // validate that the subgraph has that many inputs. + ORT_ENFORCE(static_cast(num_subgraph_inputs) == subgraph_inputs.size(), + "Graph in 'body' attribute of Loop should have ", num_subgraph_inputs, " inputs. Found:", + subgraph_inputs.size()); - // check num outputs are correct. the 'cond' output from the subgraph is not a Loop output, so diff is 1 - num_subgraph_outputs = static_cast(subgraph_outputs.size()); - ORT_ENFORCE(num_subgraph_outputs - 1 == num_outputs, - "'Loop' node has ", num_outputs, " outputs so the subgraph requires ", num_outputs + 1, - " but has ", num_subgraph_outputs); + // check num outputs are correct. the 'cond' output from the subgraph is not a Loop output, so diff is 1 + num_subgraph_outputs = static_cast(subgraph_outputs.size()); + ORT_ENFORCE(num_subgraph_outputs - 1 == num_outputs, + "'Loop' node has ", num_outputs, " outputs so the subgraph requires ", num_outputs + 1, + " but has ", num_subgraph_outputs); - subgraph_input_names.reserve(num_subgraph_inputs); - for (int i = 0; i < num_subgraph_inputs; ++i) { - subgraph_input_names.push_back(subgraph_inputs[i]->Name()); - } - - // save list of subgraph output names in their provided order to use when fetching the results - // from each subgraph execution. the Loop outputs will match this order. - subgraph_output_names.reserve(num_subgraph_outputs); - for (int i = 0; i < num_subgraph_outputs; ++i) { - auto& output = subgraph_outputs[i]; - subgraph_output_names.push_back(output->Name()); - } + subgraph_input_names.reserve(num_subgraph_inputs); + for (int i = 0; i < num_subgraph_inputs; ++i) { + subgraph_input_names.push_back(subgraph_inputs[i]->Name()); } - const GraphViewer& subgraph; - - int num_loop_carried_vars; - int num_implicit_inputs; - int num_outputs; - - int num_subgraph_inputs; - int num_subgraph_outputs; - - std::vector subgraph_input_names; - std::vector subgraph_output_names; -}; + // save list of subgraph output names in their provided order to use when fetching the results + // from each subgraph execution. the Loop outputs will match this order. + subgraph_output_names.reserve(num_subgraph_outputs); + for (int i = 0; i < num_subgraph_outputs; ++i) { + auto& output = subgraph_outputs[i]; + subgraph_output_names.push_back(output->Name()); + } +} class LoopImpl { public: @@ -246,7 +232,7 @@ static Status ConcatenateCpuOutput(void* /*stream*/, return Status::OK(); } -Loop::Loop(const OpKernelInfo& info) : IControlFlowKernel(info) { +void Loop::Init(const OpKernelInfo& info) { // make sure the attribute was present even though we don't need it here. // The GraphProto is loaded as a Graph instance by main Graph::Resolve, // and a SessionState instance for executing the subgraph is created by InferenceSession. @@ -259,8 +245,12 @@ Loop::Loop(const OpKernelInfo& info) : IControlFlowKernel(info) { stream_ = nullptr; } -// we need this to be in the .cc so 'unique_ptr info_' can be handled -Loop::~Loop() = default; +std::unique_ptr Loop::Create(const OpKernelInfo& info, const ConcatOutput& concat_output_func, void* stream) { + auto result = std::make_unique(info); + result->SetConcatOutputFunc(concat_output_func); + result->SetComputeStream(stream); + return result; +} common::Status Loop::SetupSubgraphExecutionInfo(const SessionState& session_state, const std::string& attribute_name, diff --git a/onnxruntime/core/providers/cpu/controlflow/loop.h b/onnxruntime/core/providers/cpu/controlflow/loop.h index 73f863b22d..5bb495cb40 100644 --- a/onnxruntime/core/providers/cpu/controlflow/loop.h +++ b/onnxruntime/core/providers/cpu/controlflow/loop.h @@ -14,17 +14,30 @@ namespace onnxruntime { class Loop : public controlflow::IControlFlowKernel { public: - Loop(const OpKernelInfo& info); + Loop(const OpKernelInfo& info) : IControlFlowKernel(info) { Init(info); } + void Init(const OpKernelInfo& info); Status Compute(OpKernelContext* ctx) const override; - common::Status SetupSubgraphExecutionInfo(const SessionState& session_state, - const std::string& attribute_name, - const SessionState& subgraph_session_state) override; + Status SetupSubgraphExecutionInfo(const SessionState& session_state, + const std::string& attribute_name, + const SessionState& subgraph_session_state) override; - // hide internal implementation details via forward declaration. - struct Info; - ~Loop(); + struct Info { + Info(const onnxruntime::Node& node, const GraphViewer& subgraph_in); + + const GraphViewer& subgraph; + + int num_loop_carried_vars; + int num_implicit_inputs; + int num_outputs; + + int num_subgraph_inputs; + int num_subgraph_outputs; + + std::vector subgraph_input_names; + std::vector subgraph_output_names; + }; // function to concatenate the OrtValue instances from each Loop iteration into a single output buffer. // @param per_iteration_output OrtValue instances from each iteration. Never empty. All should have the same shape. @@ -32,6 +45,8 @@ class Loop : public controlflow::IControlFlowKernel { using ConcatOutput = std::function& per_iteration_output, void* output, size_t output_size_in_bytes)>; + static std::unique_ptr Create(const OpKernelInfo& info, const ConcatOutput& concat_output_func, void* stream); + protected: // derived class can provide implementation for handling concatenation of Loop output on a different device void SetConcatOutputFunc(const ConcatOutput& concat_output_func) { concat_output_func_ = concat_output_func; } diff --git a/onnxruntime/core/providers/cpu/controlflow/scan.h b/onnxruntime/core/providers/cpu/controlflow/scan.h index 01bcfcd564..1130e9ee26 100644 --- a/onnxruntime/core/providers/cpu/controlflow/scan.h +++ b/onnxruntime/core/providers/cpu/controlflow/scan.h @@ -5,8 +5,11 @@ #include #include "gsl/gsl" +#ifndef SHARED_PROVIDER #include "core/common/common.h" #include "core/framework/op_kernel.h" +#endif + #include "core/framework/feeds_fetches_manager.h" #include "core/providers/cpu/controlflow/utils.h" #include "core/framework/ort_value_tensor_slicer.h" @@ -14,6 +17,28 @@ namespace onnxruntime { namespace scan { namespace detail { +/** +Helper struct for keeping static information about the Scan node and its subgraph. +Used to create the FeedsFetchesManager needed for efficient subgraph execution. +*/ +struct Info { + Info(const Node& node, const GraphViewer& subgraph_in, int num_scan_inputs_in, bool is_v8); + + const GraphViewer& subgraph; + + int num_inputs; + int num_variadic_inputs; + int num_outputs; + int num_loop_state_variables; + int num_scan_inputs; + int num_scan_outputs; + + int num_implicit_inputs; + + std::vector subgraph_input_names; + std::vector subgraph_output_names; +}; + // helpers for handling data on a non-CPU device. // Provide as needed when Scan is being run by a non-CPU based ExecutionProvider struct DeviceHelpers { @@ -44,19 +69,20 @@ struct DeviceHelpers { template class Scan : public controlflow::IControlFlowKernel { public: - Scan(const OpKernelInfo& info); + Scan(const OpKernelInfo& info) : IControlFlowKernel(info) { Init(info); } + void Init(const OpKernelInfo& info); Status Compute(OpKernelContext* ctx) const override; - common::Status SetupSubgraphExecutionInfo(const SessionState& session_state, - const std::string& attribute_name, - const SessionState& subgraph_session_state) override; + Status SetupSubgraphExecutionInfo(const SessionState& session_state, + const std::string& attribute_name, + const SessionState& subgraph_session_state) override; - // hide internal implementation details via forward declaration. - struct Info; - ~Scan(); + struct Info : scan::detail::Info { + Info(const onnxruntime::Node& node, const GraphViewer& subgraph_in, int num_scan_inputs_in) + : scan::detail::Info(node, subgraph_in, num_scan_inputs_in, /* is_v8 */ OpSet == 8) {} + }; - protected: void SetDeviceHelpers(const scan::detail::DeviceHelpers& device_helpers) { device_helpers_ = device_helpers; // copy } diff --git a/onnxruntime/core/providers/cpu/controlflow/scan_8.cc b/onnxruntime/core/providers/cpu/controlflow/scan_8.cc index ad6d098a03..854f3ba2d2 100644 --- a/onnxruntime/core/providers/cpu/controlflow/scan_8.cc +++ b/onnxruntime/core/providers/cpu/controlflow/scan_8.cc @@ -83,12 +83,6 @@ ONNX_OPERATOR_SET_SCHEMA( .TypeConstraint("V", OpSchema::all_tensor_types(), "All Tensor types")); */ -template <> -struct Scan<8>::Info : public scan::detail::Info { - Info(const onnxruntime::Node& node, const GraphViewer& subgraph_in, int num_scan_inputs_in) - : scan::detail::Info(node, subgraph_in, num_scan_inputs_in, /* is_v8 */ true) {} -}; - class Scan8Impl { public: Scan8Impl(OpKernelContextInternal& context, @@ -135,7 +129,7 @@ class Scan8Impl { }; template <> -Scan<8>::Scan(const OpKernelInfo& info) : IControlFlowKernel(info) { +void Scan<8>::Init(const OpKernelInfo& info) { // make sure the attribute was present even though we don't need it here. // The GraphProto is loaded as a Graph instance by main Graph::Resolve, // and a SessionState instance for executing the subgraph is created by InferenceSession. @@ -175,10 +169,6 @@ Status Scan<8>::SetupSubgraphExecutionInfo(const SessionState& session_state, return status; } -// we need this to be in the .cc so 'unique_ptr info_' can be handled -template <> -Scan<8>::~Scan() = default; - template <> Status Scan<8>::Compute(OpKernelContext* ctx) const { ORT_ENFORCE(feeds_fetches_manager_ && info_, diff --git a/onnxruntime/core/providers/cpu/controlflow/scan_9.cc b/onnxruntime/core/providers/cpu/controlflow/scan_9.cc index 918203a256..001c2d9ac8 100644 --- a/onnxruntime/core/providers/cpu/controlflow/scan_9.cc +++ b/onnxruntime/core/providers/cpu/controlflow/scan_9.cc @@ -97,12 +97,6 @@ ONNX_OPERATOR_SET_SCHEMA( } */ -template <> -struct Scan<9>::Info : public scan::detail::Info { - Info(const onnxruntime::Node& node, const GraphViewer& subgraph_in, int num_scan_inputs_in) - : scan::detail::Info(node, subgraph_in, num_scan_inputs_in, /* is_v8 */ false) {} -}; - class ScanImpl { public: ScanImpl(OpKernelContextInternal& context, @@ -159,7 +153,7 @@ class ScanImpl { }; template <> -Scan<9>::Scan(const OpKernelInfo& info) : IControlFlowKernel(info) { +void Scan<9>::Init(const OpKernelInfo& info) { // make sure the attribute was present even though we don't need it here. // The GraphProto is loaded as a Graph instance by main Graph::Resolve, // and a SessionState instance for executing the subgraph is created by InferenceSession. @@ -202,10 +196,6 @@ Scan<9>::Scan(const OpKernelInfo& info) : IControlFlowKernel(info) { }; } -// we need this to be in the .cc so 'unique_ptr info_' can be handled -template <> -Scan<9>::~Scan() = default; - template <> Status Scan<9>::SetupSubgraphExecutionInfo(const SessionState& session_state, const std::string& attribute_name, diff --git a/onnxruntime/core/providers/cpu/controlflow/scan_utils.h b/onnxruntime/core/providers/cpu/controlflow/scan_utils.h index 7fb44fff78..0d17ede9d0 100644 --- a/onnxruntime/core/providers/cpu/controlflow/scan_utils.h +++ b/onnxruntime/core/providers/cpu/controlflow/scan_utils.h @@ -26,28 +26,6 @@ namespace detail { enum class ScanDirection { kForward = 0, kReverse = 1 }; -/** -Helper struct for keeping static information about the Scan node and its subgraph. -Used to create the FeedsFetchesManager needed for efficient subgraph execution. -*/ -struct Info { - Info(const Node& node, const GraphViewer& subgraph_in, int num_scan_inputs_in, bool is_v8); - - const GraphViewer& subgraph; - - int num_inputs; - int num_variadic_inputs; - int num_outputs; - int num_loop_state_variables; - int num_scan_inputs; - int num_scan_outputs; - - int num_implicit_inputs; - - std::vector subgraph_input_names; - std::vector subgraph_output_names; -}; - /** Class to provide input/output OrtValue instances for a loop state variable. The OrtValue flips between two internal temporary buffers to minimize copies. diff --git a/onnxruntime/core/providers/cpu/controlflow/utils.h b/onnxruntime/core/providers/cpu/controlflow/utils.h index c6fc4a4704..1c5feb2a93 100644 --- a/onnxruntime/core/providers/cpu/controlflow/utils.h +++ b/onnxruntime/core/providers/cpu/controlflow/utils.h @@ -7,12 +7,14 @@ #include #include -#include "core/common/common.h" #include "core/framework/feeds_fetches_manager.h" +#ifdef SHARED_PROVIDER +#include "core/framework/ml_value.h" +#else #include "core/framework/op_kernel.h" +#endif namespace onnxruntime { -class Graph; // Creates a scalar MLValue based on given value and allocator. template diff --git a/onnxruntime/core/providers/cpu/generator/constant_of_shape_base.h b/onnxruntime/core/providers/cpu/generator/constant_of_shape_base.h index 2e72f97e69..f105e2762f 100644 --- a/onnxruntime/core/providers/cpu/generator/constant_of_shape_base.h +++ b/onnxruntime/core/providers/cpu/generator/constant_of_shape_base.h @@ -3,6 +3,7 @@ #pragma once +#ifndef SHARED_PROVIDER #include "core/common/common.h" #include "core/common/type_list.h" #include "core/framework/data_types.h" @@ -10,6 +11,7 @@ #include "core/framework/op_kernel.h" #include "core/framework/tensorprotoutils.h" #include "core/providers/op_kernel_type_control_utils.h" +#endif namespace onnxruntime { @@ -25,11 +27,17 @@ template class ConstantOfShapeBase { protected: ConstantOfShapeBase(const OpKernelInfo& info) { +#ifndef SHARED_PROVIDER ONNX_NAMESPACE::TensorProto t_proto; - if (info.GetAttr("value", &t_proto).IsOK()) { - ORT_ENFORCE(t_proto.dims_size() == 1, "Must have a single dimension"); - ORT_ENFORCE(t_proto.dims()[0] == 1, "Must have a single dimension of 1"); - SetValueFromTensorProto(t_proto); + auto* t_proto_p = &t_proto; +#else + auto t_proto = ONNX_NAMESPACE::TensorProto::Create(); + auto* t_proto_p = t_proto.get(); +#endif + if (info.GetAttr("value", t_proto_p).IsOK()) { + ORT_ENFORCE(t_proto_p->dims_size() == 1, "Must have a single dimension"); + ORT_ENFORCE(t_proto_p->dims()[0] == 1, "Must have a single dimension of 1"); + SetValueFromTensorProto(*t_proto_p); } else { float f_value = 0.f; SetValue(sizeof(float), reinterpret_cast(&f_value)); diff --git a/onnxruntime/core/providers/cpu/math/clip.h b/onnxruntime/core/providers/cpu/math/clip.h index b7c6032c6b..cdbe86ee66 100644 --- a/onnxruntime/core/providers/cpu/math/clip.h +++ b/onnxruntime/core/providers/cpu/math/clip.h @@ -3,14 +3,16 @@ #pragma once +#ifndef SHARED_PROVIDER #include "core/common/common.h" #include "core/framework/op_kernel.h" +#endif namespace onnxruntime { namespace clip_internal { -template +template class Clip_6Base { public: explicit Clip_6Base(const OpKernelInfo& info) { @@ -20,11 +22,12 @@ class Clip_6Base { info.GetAttrOrDefault("max", &max_, max_val); ORT_ENFORCE(min_ <= max_); } + protected: T max_; T min_; }; -} // namespace clip_internal +} // namespace clip_internal template class Clip_6 final : public clip_internal::Clip_6Base, public OpKernel { @@ -45,7 +48,7 @@ class Clip final : public OpKernel { Status Compute(OpKernelContext* ctx) const override; private: - template + template struct ComputeImpl; }; diff --git a/onnxruntime/core/providers/cpu/math/einsum.h b/onnxruntime/core/providers/cpu/math/einsum.h index 29719134f9..74a62b8f7d 100644 --- a/onnxruntime/core/providers/cpu/math/einsum.h +++ b/onnxruntime/core/providers/cpu/math/einsum.h @@ -3,10 +3,12 @@ #pragma once +#ifndef SHARED_PROVIDER #include "core/common/common.h" #include "core/framework/op_kernel.h" -#include "einsum_utils/einsum_compute_preprocessor.h" #include "einsum_utils/einsum_typed_compute_processor.h" +#endif +#include "einsum_utils/einsum_compute_preprocessor.h" namespace onnxruntime { diff --git a/onnxruntime/core/providers/cpu/math/einsum_utils/einsum_auxiliary_ops.cc b/onnxruntime/core/providers/cpu/math/einsum_utils/einsum_auxiliary_ops.cc index 4367463baa..b87aa1733b 100644 --- a/onnxruntime/core/providers/cpu/math/einsum_utils/einsum_auxiliary_ops.cc +++ b/onnxruntime/core/providers/cpu/math/einsum_utils/einsum_auxiliary_ops.cc @@ -49,10 +49,10 @@ Status MatMul(const T* input_1_data, const T* input_2_data, T* output_data, // CPU specific ReduceSum helper template -Tensor ReduceSum(const Tensor& input, const std::vector& reduce_axes, - bool keep_dims, AllocatorPtr allocator, - const TensorShape* input_shape_override, - concurrency::ThreadPool* tp, void* /*einsum_cuda_assets*/) { +std::unique_ptr ReduceSum(const Tensor& input, const std::vector& reduce_axes, + bool keep_dims, AllocatorPtr allocator, + const TensorShape* input_shape_override, + concurrency::ThreadPool* tp, void* /*einsum_cuda_assets*/) { return onnxruntime::ReduceSum::Impl(input, reduce_axes, allocator, tp, keep_dims, input_shape_override); @@ -351,8 +351,7 @@ std::unique_ptr ReduceSum(const Tensor& input, const std::vector& device_reduce_sum_func) { TensorShape overriden_shape(input_shape_override); - auto output = device_reduce_sum_func(input, reduce_axes, true, allocator, &overriden_shape, tp, einsum_cuda_assets); - return std::make_unique(std::move(output)); + return device_reduce_sum_func(input, reduce_axes, true, allocator, &overriden_shape, tp, einsum_cuda_assets); } // Explicit template instantiations of functions @@ -370,7 +369,7 @@ template std::unique_ptr MatMul( AllocatorPtr allocator, concurrency::ThreadPool* tp, void* einsum_cuda_assets, const DeviceHelpers::MatMul& device_matmul_func); -template Tensor DeviceHelpers::CpuDeviceHelpers::ReduceSum( +template std::unique_ptr DeviceHelpers::CpuDeviceHelpers::ReduceSum( const Tensor& input, const std::vector& reduce_axes, bool keep_dims, AllocatorPtr allocator, const TensorShape* input_shape_override, @@ -394,7 +393,7 @@ template std::unique_ptr MatMul( AllocatorPtr allocator, concurrency::ThreadPool* tp, void* einsum_cuda_assets, const DeviceHelpers::MatMul& device_matmul_func); -template Tensor DeviceHelpers::CpuDeviceHelpers::ReduceSum( +template std::unique_ptr DeviceHelpers::CpuDeviceHelpers::ReduceSum( const Tensor& input, const std::vector& reduce_axes, bool keep_dims, AllocatorPtr allocator, const TensorShape* input_shape_override, @@ -419,7 +418,7 @@ template std::unique_ptr MatMul( AllocatorPtr allocator, concurrency::ThreadPool* tp, void* einsum_cuda_assets, const DeviceHelpers::MatMul& device_matmul_func); -template Tensor DeviceHelpers::CpuDeviceHelpers::ReduceSum( +template std::unique_ptr DeviceHelpers::CpuDeviceHelpers::ReduceSum( const Tensor& input, const std::vector& reduce_axes, bool keep_dims, AllocatorPtr allocator, const TensorShape* input_shape_override, @@ -438,7 +437,7 @@ template Status DeviceHelpers::CpuDeviceHelpers::MatMul( size_t num_batches, size_t M, size_t K, size_t N, concurrency::ThreadPool* tp, void* einsum_cuda_assets); -template Tensor DeviceHelpers::CpuDeviceHelpers::ReduceSum( +template std::unique_ptr DeviceHelpers::CpuDeviceHelpers::ReduceSum( const Tensor& input, const std::vector& reduce_axes, bool keep_dims, AllocatorPtr allocator, const TensorShape* input_shape_override, @@ -470,3 +469,4 @@ template std::unique_ptr ReduceSum( } // namespace EinsumOp } // namespace onnxruntime + diff --git a/onnxruntime/core/providers/cpu/math/einsum_utils/einsum_auxiliary_ops.h b/onnxruntime/core/providers/cpu/math/einsum_utils/einsum_auxiliary_ops.h index d5f96a79b9..dd21654c4a 100644 --- a/onnxruntime/core/providers/cpu/math/einsum_utils/einsum_auxiliary_ops.h +++ b/onnxruntime/core/providers/cpu/math/einsum_utils/einsum_auxiliary_ops.h @@ -6,9 +6,11 @@ #pragma once +#ifndef SHARED_PROVIDER #include "core/util/math.h" #include "core/providers/cpu/tensor/transpose.h" #include "core/providers/cpu/reduction/reduction_ops.h" +#endif #include @@ -39,10 +41,10 @@ using MatMul = std::function -using ReduceSum = std::function& reduce_axes, - bool keep_dims, AllocatorPtr allocator, - const TensorShape* input_shape_override, - concurrency::ThreadPool* tp, void* einsum_cuda_assets)>; +using ReduceSum = std::function(const Tensor& input, const std::vector& reduce_axes, + bool keep_dims, AllocatorPtr allocator, + const TensorShape* input_shape_override, + concurrency::ThreadPool* tp, void* einsum_cuda_assets)>; // Diagonal op // Diagonal - A specialized implementation somewhat similar to Torch's Diagonal op @@ -71,10 +73,10 @@ Status MatMul(const T* input_1_data, const T* input_2_data, T* output_data, void* einsum_cuda_assets); template -Tensor ReduceSum(const Tensor& input, const std::vector& reduce_axes, - bool keep_dims, AllocatorPtr allocator, - const TensorShape* input_shape_override, - concurrency::ThreadPool* tp, void* einsum_cuda_assets); +std::unique_ptr ReduceSum(const Tensor& input, const std::vector& reduce_axes, + bool keep_dims, AllocatorPtr allocator, + const TensorShape* input_shape_override, + concurrency::ThreadPool* tp, void* einsum_cuda_assets); std::unique_ptr Diagonal(const Tensor& input, int64_t dim_1, int64_t dim_2, AllocatorPtr allocator, void* einsum_cuda_assets); diff --git a/onnxruntime/core/providers/cpu/math/einsum_utils/einsum_compute_preprocessor.h b/onnxruntime/core/providers/cpu/math/einsum_utils/einsum_compute_preprocessor.h index 7c7b25f2a7..955ff4a920 100644 --- a/onnxruntime/core/providers/cpu/math/einsum_utils/einsum_compute_preprocessor.h +++ b/onnxruntime/core/providers/cpu/math/einsum_utils/einsum_compute_preprocessor.h @@ -104,6 +104,7 @@ struct EinsumEquationPreprocessor { // Subscript labels (letter) and subcript indices (a unique id to the letter) are interchangeable // This is a pre-processor class that maps subscript labels to a dimension value, etc. +#ifndef SHARED_PROVIDER class EinsumComputePreprocessor final { public: explicit EinsumComputePreprocessor(EinsumEquationPreprocessor& equation_preprocessor, @@ -224,5 +225,5 @@ class EinsumComputePreprocessor final { // Holds EP-specific assets required for (auxiliary) ops that need to be executed on non-CPU EPs void* einsum_ep_assets_; }; - +#endif } // namespace onnxruntime diff --git a/onnxruntime/core/providers/cpu/math/matmul_helper.h b/onnxruntime/core/providers/cpu/math/matmul_helper.h index 05e5281c92..0a889a3814 100644 --- a/onnxruntime/core/providers/cpu/math/matmul_helper.h +++ b/onnxruntime/core/providers/cpu/math/matmul_helper.h @@ -2,8 +2,10 @@ // Licensed under the MIT License. #pragma once +#ifndef SHARED_PROVIDER #include "core/common/common.h" #include "core/framework/tensor.h" +#endif namespace onnxruntime { template diff --git a/onnxruntime/core/providers/cpu/nn/batch_norm_helper.h b/onnxruntime/core/providers/cpu/nn/batch_norm_helper.h index 9add18710c..92efe04472 100644 --- a/onnxruntime/core/providers/cpu/nn/batch_norm_helper.h +++ b/onnxruntime/core/providers/cpu/nn/batch_norm_helper.h @@ -3,8 +3,10 @@ #pragma once +#ifndef SHARED_PROVIDER #include "core/common/status.h" #include "core/framework/tensor.h" +#endif #include namespace onnxruntime { diff --git a/onnxruntime/core/providers/cpu/nn/conv_attributes.h b/onnxruntime/core/providers/cpu/nn/conv_attributes.h index b0d0075b1d..d7041d4145 100644 --- a/onnxruntime/core/providers/cpu/nn/conv_attributes.h +++ b/onnxruntime/core/providers/cpu/nn/conv_attributes.h @@ -3,16 +3,18 @@ #pragma once +#ifndef SHARED_PROVIDER #include "core/common/common.h" -#include "core/framework/op_node_proto_helper.h" #include "core/providers/common.h" #include "core/util/math.h" +#endif +#include "core/framework/op_node_proto_helper.h" namespace onnxruntime { // A helper struct holding attributes for Conv-family ops struct ConvAttributes { - explicit ConvAttributes(const OpNodeProtoHelper& info) { + explicit ConvAttributes(const OpKernelInfo& info) { std::string auto_pad_str; auto status = info.GetAttr("auto_pad", &auto_pad_str); if (status.IsOK()) { diff --git a/onnxruntime/core/providers/cpu/nn/conv_transpose_attributes.h b/onnxruntime/core/providers/cpu/nn/conv_transpose_attributes.h index 23d34d4675..9ded92912f 100644 --- a/onnxruntime/core/providers/cpu/nn/conv_transpose_attributes.h +++ b/onnxruntime/core/providers/cpu/nn/conv_transpose_attributes.h @@ -22,7 +22,7 @@ namespace onnxruntime { struct ConvTransposeAttributes : public ConvAttributes { - explicit ConvTransposeAttributes(const OpNodeProtoHelper& info) + explicit ConvTransposeAttributes(const OpKernelInfo& info) : ConvAttributes(info), output_padding(info.GetAttrsOrDefault("output_padding")), output_shape(info.GetAttrsOrDefault("output_shape")) { diff --git a/onnxruntime/core/providers/cpu/nn/instance_norm_helper.h b/onnxruntime/core/providers/cpu/nn/instance_norm_helper.h index bdf6f90b38..48e54ac7ee 100644 --- a/onnxruntime/core/providers/cpu/nn/instance_norm_helper.h +++ b/onnxruntime/core/providers/cpu/nn/instance_norm_helper.h @@ -3,8 +3,10 @@ #pragma once +#ifndef SHARED_PROVIDER #include "core/common/status.h" #include "core/framework/tensor.h" +#endif #include namespace onnxruntime { diff --git a/onnxruntime/core/providers/cpu/nn/pool_attributes.h b/onnxruntime/core/providers/cpu/nn/pool_attributes.h index 8f14f16d88..452fe52831 100644 --- a/onnxruntime/core/providers/cpu/nn/pool_attributes.h +++ b/onnxruntime/core/providers/cpu/nn/pool_attributes.h @@ -4,10 +4,12 @@ #pragma once #include +#ifndef SHARED_PROVIDER #include "core/common/common.h" #include "core/framework/op_node_proto_helper.h" #include "core/framework/tensor_shape.h" #include "core/providers/common.h" +#endif namespace onnxruntime { @@ -17,7 +19,13 @@ struct PoolAttributes { return op_name == "GlobalAveragePool" || op_name == "GlobalMaxPool" || op_name == "GlobalLpPool"; } +#ifdef SHARED_PROVIDER + // Shared providers don't know about OpNodeProtoHelper + PoolAttributes(const OpKernelInfo& info, +#else + // Providers like Nuphar don't know about OpKernelInfo PoolAttributes(const OpNodeProtoHelper& info, +#endif const std::string& op_name, int start_version) : global_pooling(IsGlobalPooling(op_name)) { if (global_pooling) { diff --git a/onnxruntime/core/providers/cpu/nn/pool_base.h b/onnxruntime/core/providers/cpu/nn/pool_base.h index 4e2c9910b9..d03754eec9 100644 --- a/onnxruntime/core/providers/cpu/nn/pool_base.h +++ b/onnxruntime/core/providers/cpu/nn/pool_base.h @@ -4,10 +4,12 @@ #pragma once #include +#ifndef SHARED_PROVIDER #include "core/common/common.h" #include "core/framework/op_kernel.h" -#include "core/providers/cpu/nn/pool_attributes.h" #include "core/util/math.h" +#endif +#include "core/providers/cpu/nn/pool_attributes.h" #include "core/mlas/inc/mlas.h" namespace onnxruntime { @@ -106,8 +108,7 @@ class PoolBase { protected: PoolBase(const OpKernelInfo& info) - : op_name_(info.GetKernelDef().OpName().rfind("QLinear", 0) != 0 ? - info.GetKernelDef().OpName() : info.GetKernelDef().OpName().substr(7)), + : op_name_(info.GetKernelDef().OpName().rfind("QLinear", 0) != 0 ? info.GetKernelDef().OpName() : info.GetKernelDef().OpName().substr(7)), pool_attrs_(info, op_name_, GetStartVersion(info)) { } diff --git a/onnxruntime/core/providers/cpu/object_detection/non_max_suppression.h b/onnxruntime/core/providers/cpu/object_detection/non_max_suppression.h index f6465ea3ee..e349a53cbd 100644 --- a/onnxruntime/core/providers/cpu/object_detection/non_max_suppression.h +++ b/onnxruntime/core/providers/cpu/object_detection/non_max_suppression.h @@ -17,16 +17,17 @@ class NonMaxSuppressionBase { ORT_ENFORCE(0 == center_point_box_ || 1 == center_point_box_, "center_point_box only support 0 or 1"); } + int64_t GetCenterPointBox() const { + return center_point_box_; + } + + public: static Status PrepareCompute(OpKernelContext* ctx, PrepareContext& pc); static Status GetThresholdsFromInputs(const PrepareContext& pc, int64_t& max_output_boxes_per_class, float& iou_threshold, float& score_threshold); - int64_t GetCenterPointBox() const { - return center_point_box_; - } - private: int64_t center_point_box_; }; diff --git a/onnxruntime/core/providers/cpu/reduction/reduction_ops.cc b/onnxruntime/core/providers/cpu/reduction/reduction_ops.cc index d55a13a857..ac83870ff6 100644 --- a/onnxruntime/core/providers/cpu/reduction/reduction_ops.cc +++ b/onnxruntime/core/providers/cpu/reduction/reduction_ops.cc @@ -838,9 +838,9 @@ Status ReduceSum::Compute(OpKernelContext* ctx) const { } template -Tensor ReduceSum::Impl(const Tensor& input, const std::vector& reduce_axes, - AllocatorPtr allocator, concurrency::ThreadPool* tp, bool keep_dims, - const TensorShape* input_shape_override) { +std::unique_ptr ReduceSum::Impl(const Tensor& input, const std::vector& reduce_axes, + AllocatorPtr allocator, concurrency::ThreadPool* tp, bool keep_dims, + const TensorShape* input_shape_override) { std::vector axes; std::vector output_shape, fast_shape, fast_axes; TensorShape new_input_shape = input_shape_override == nullptr ? input.Shape() : *input_shape_override; @@ -849,12 +849,12 @@ Tensor ReduceSum::Impl(const Tensor& input, const std::vector& reduc FastReduceKind fast_kind = OptimizeShapeForFastReduce( reduced_dims, reduce_axes, fast_shape, output_shape, fast_axes, keep_dims, false); - Tensor output(input.DataType(), keep_dims ? output_shape : std::vector(), allocator); + auto output = make_unique(input.DataType(), keep_dims ? output_shape : std::vector(), allocator); if (fast_kind == FastReduceKind::kEmpty) { if (new_input_shape.Size() == 1) { const T* from_data = input.template Data(); - T* to_data = output.template MutableData(); + T* to_data = output->template MutableData(); *to_data = *from_data; } else { ValidateKeepDims(new_input_shape, keep_dims); @@ -865,18 +865,18 @@ Tensor ReduceSum::Impl(const Tensor& input, const std::vector& reduc if (IsFastReduceKindAvailable(fast_kind, ReduceAggregatorSum::WhichFastReduce())) { switch (fast_kind) { case FastReduceKind::kKR: { - ValidateFastReduceKR(fast_shape, output); - ReduceAggregatorSum::FastReduceKR(input, fast_shape, output, tp); + ValidateFastReduceKR(fast_shape, *output); + ReduceAggregatorSum::FastReduceKR(input, fast_shape, *output, tp); return output; } case FastReduceKind::kRK: { - ValidateFastReduceRK(fast_shape, output); - ReduceAggregatorSum::FastReduceRK(input, fast_shape, output, tp); + ValidateFastReduceRK(fast_shape, *output); + ReduceAggregatorSum::FastReduceRK(input, fast_shape, *output, tp); return output; } case FastReduceKind::kKRK: { - ValidateFastReduceKRK(fast_shape, output); - ReduceAggregatorSum::FastReduceKRK(input, fast_shape, output, tp); + ValidateFastReduceKRK(fast_shape, *output); + ReduceAggregatorSum::FastReduceKRK(input, fast_shape, *output, tp); return output; } case FastReduceKind::kR: @@ -889,7 +889,7 @@ Tensor ReduceSum::Impl(const Tensor& input, const std::vector& reduc } ResultsNoTransposePrepareForReduce last_results; - NoTransposeReduce1Loop>(&output, fast_shape, input, fast_axes, tp, last_results); + NoTransposeReduce1Loop>(output.get(), fast_shape, input, fast_axes, tp, last_results); return output; } diff --git a/onnxruntime/core/providers/cpu/reduction/reduction_ops.h b/onnxruntime/core/providers/cpu/reduction/reduction_ops.h index e3ddfac4c4..08f1c37863 100644 --- a/onnxruntime/core/providers/cpu/reduction/reduction_ops.h +++ b/onnxruntime/core/providers/cpu/reduction/reduction_ops.h @@ -4,12 +4,14 @@ #ifndef CORE_PROVIDERS_CPU_REDUCTION_OPS_H #define CORE_PROVIDERS_CPU_REDUCTION_OPS_H +#ifndef SHARED_PROVIDER #include "core/common/common.h" -#include "core/common/optional.h" #include "core/framework/op_kernel.h" #include "core/providers/cpu/containers.h" #include "core/util/math.h" +#endif #include "core/util/math_cpuonly.h" +#include "core/common/optional.h" #include "core/platform/threadpool.h" #include "core/common/safeint.h" #include @@ -733,9 +735,9 @@ class ReduceSum final : public ReduceKernel { // For external calls requiring ReduceSum implementation - will return the reduced output. //`input_shape_override` overrides the shape of `input` for compute purposes. - static Tensor Impl(const Tensor& input, const std::vector& reduce_axes, - AllocatorPtr allocator, concurrency::ThreadPool* tp, bool keep_dims, - const TensorShape* input_shape_override = nullptr); + static std::unique_ptr Impl(const Tensor& input, const std::vector& reduce_axes, + AllocatorPtr allocator, concurrency::ThreadPool* tp, bool keep_dims, + const TensorShape* input_shape_override = nullptr); }; template diff --git a/onnxruntime/core/providers/cpu/rnn/rnn_helpers.h b/onnxruntime/core/providers/cpu/rnn/rnn_helpers.h index 711afd9fff..2b534c7295 100644 --- a/onnxruntime/core/providers/cpu/rnn/rnn_helpers.h +++ b/onnxruntime/core/providers/cpu/rnn/rnn_helpers.h @@ -20,9 +20,6 @@ #include "gsl/gsl" namespace onnxruntime { -class Tensor; -class OpKernelContext; - namespace rnn { namespace detail { diff --git a/onnxruntime/core/providers/cpu/tensor/concat.h b/onnxruntime/core/providers/cpu/tensor/concat.h index 7752f042a6..227f98bb4f 100644 --- a/onnxruntime/core/providers/cpu/tensor/concat.h +++ b/onnxruntime/core/providers/cpu/tensor/concat.h @@ -7,50 +7,10 @@ #include "core/framework/op_kernel.h" #include "core/util/math_cpuonly.h" #include "core/framework/tensor.h" +#include "concatbase.h" namespace onnxruntime { -// structure to hold some inputs and some metadata to be used during Compute() -struct Prepare { - struct InputInfo { - const Tensor* tensor; - int64_t axis_pitch; - int64_t num_elements; - }; - std::vector inputs; - int64_t output_num_elements; - int64_t output_axis_pitch; - Tensor* output_tensor; - uint64_t axis; - bool is_string_type; -}; - -class ConcatBase { - protected: - ConcatBase(const OpKernelInfo& info, bool is_sequence_op = false) { - if (!info.GetAttr("axis", &axis_).IsOK()) { - ORT_ENFORCE(false, "Must have valid 'axis' attribute"); - } - - is_sequence_op_ = is_sequence_op; - - if (is_sequence_op) { // Only ConcatFromSequence supports stacking - is_stack_ = info.GetAttrOrDefault("new_axis", 0) == 0 ? false : true; - } - } - - // the core method that will be invoked by the 'Concat' (CPU and GPU) - // and 'ConcatFromSequence' kernels - Status PrepareForCompute(OpKernelContext* ctx, const std::vector& input_tensors, - Prepare& p) const; - - Status ComputeImpl(Prepare& p) const; - - int64_t axis_; - bool is_stack_ = false; - bool is_sequence_op_; -}; - class Concat : public OpKernel, public ConcatBase { public: Concat(const OpKernelInfo& info) : OpKernel(info), ConcatBase(info) {} diff --git a/onnxruntime/core/providers/cpu/tensor/concatbase.h b/onnxruntime/core/providers/cpu/tensor/concatbase.h new file mode 100644 index 0000000000..fb8ebd8fdf --- /dev/null +++ b/onnxruntime/core/providers/cpu/tensor/concatbase.h @@ -0,0 +1,48 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. +#pragma once + +namespace onnxruntime { + +// structure to hold some inputs and some metadata to be used during Compute() +struct Prepare { + struct InputInfo { + const Tensor* tensor; + int64_t axis_pitch; + int64_t num_elements; + }; + std::vector inputs; + int64_t output_num_elements; + int64_t output_axis_pitch; + Tensor* output_tensor; + uint64_t axis; + bool is_string_type; +}; + +class ConcatBase { + public: + // the core method that will be invoked by the 'Concat' (CPU and GPU) + // and 'ConcatFromSequence' kernels + Status PrepareForCompute(OpKernelContext* ctx, const std::vector& input_tensors, + Prepare& p) const; + + protected: + ConcatBase(const OpKernelInfo& info, bool is_sequence_op = false) { + if (!info.GetAttr("axis", &axis_).IsOK()) { + ORT_ENFORCE(false, "Must have valid 'axis' attribute"); + } + + is_sequence_op_ = is_sequence_op; + + if (is_sequence_op) { // Only ConcatFromSequence supports stacking + is_stack_ = info.GetAttrOrDefault("new_axis", 0) == 0 ? false : true; + } + } + Status ComputeImpl(Prepare& p) const; + + int64_t axis_; + bool is_stack_ = false; + bool is_sequence_op_; +}; + +} // namespace onnxruntime diff --git a/onnxruntime/core/providers/cpu/tensor/gather.h b/onnxruntime/core/providers/cpu/tensor/gather.h index 0637b78e61..3508ff981d 100644 --- a/onnxruntime/core/providers/cpu/tensor/gather.h +++ b/onnxruntime/core/providers/cpu/tensor/gather.h @@ -6,28 +6,10 @@ #include "core/common/common.h" #include "core/framework/op_kernel.h" #include "core/providers/common.h" +#include "gatherbase.h" namespace onnxruntime { -class GatherBase { - protected: - GatherBase(const OpKernelInfo& info) { - ORT_ENFORCE(info.GetAttr("axis", &axis_).IsOK(), "Missing/Invalid 'axis' attribute value"); - } - - struct Prepare { - const Tensor* input_tensor; - const Tensor* indices_tensor; - Tensor* output_tensor; - int64_t axis; - }; - - Status PrepareForCompute(OpKernelContext* context, Prepare& p) const; - - private: - int64_t axis_; -}; - class Gather final : public OpKernel, public GatherBase { public: Gather(const OpKernelInfo& info) : OpKernel(info), GatherBase(info) {} diff --git a/onnxruntime/core/providers/cpu/tensor/gatherbase.h b/onnxruntime/core/providers/cpu/tensor/gatherbase.h new file mode 100644 index 0000000000..195b67553a --- /dev/null +++ b/onnxruntime/core/providers/cpu/tensor/gatherbase.h @@ -0,0 +1,27 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. +#pragma once + +namespace onnxruntime { + +class GatherBase { + public: + struct Prepare { + const Tensor* input_tensor; + const Tensor* indices_tensor; + Tensor* output_tensor; + int64_t axis; + }; + + Status PrepareForCompute(OpKernelContext* context, Prepare& p) const; + + protected: + GatherBase(const OpKernelInfo& info) { + ORT_ENFORCE(info.GetAttr("axis", &axis_).IsOK(), "Missing/Invalid 'axis' attribute value"); + } + + private: + int64_t axis_; +}; + +} // namespace onnxruntime diff --git a/onnxruntime/core/providers/cpu/tensor/pad.h b/onnxruntime/core/providers/cpu/tensor/pad.h index 13d18257a5..3e6ace7997 100644 --- a/onnxruntime/core/providers/cpu/tensor/pad.h +++ b/onnxruntime/core/providers/cpu/tensor/pad.h @@ -5,72 +5,10 @@ #include "core/common/common.h" #include "core/framework/op_kernel.h" +#include "padbase.h" namespace onnxruntime { -enum class Mode : int { - Constant = 0, - Reflect, - Edge -}; - -class PadBase { - public: - // Update the output_shape to make it consistent with numpy handling where there are one or more dimensions - // in the input_shape with a value of zero. - static Status HandleDimValueZero(const Mode& mode, const TensorShape& input_shape, TensorShape& output_shape); - - protected: - PadBase(const OpKernelInfo& info) : value_(info.GetAttrOrDefault("value", 0.f)) { - std::string mode; - if (info.GetAttr("mode", &mode).IsOK()) { - if (mode == "constant") - mode_ = Mode::Constant; - else if (mode == "reflect") - mode_ = Mode::Reflect; - else if (mode == "edge") - mode_ = Mode::Edge; - else - ORT_THROW("Invalid 'mode' attribute value"); - } - - const auto& kernel_def = info.GetKernelDef(); - - int start_ver, end_ver; - kernel_def.SinceVersion(&start_ver, &end_ver); - - // kMSDomain contrib kernel AND OnnxDomain start version >= 11 => DynamicPad - if (start_ver >= 11 || kernel_def.Domain() == kMSDomain) { - is_dynamic_ = true; - } - - if (!is_dynamic_) { - if (!info.GetAttrs("pads", pads_).IsOK()) - ORT_THROW("Invalid 'pads' attribute value"); - - // Separate out any negative pads_ into the slices_ array - slices_.resize(pads_.size(), 0); - for (size_t index = 0; index < pads_.size(); index++) { - if (pads_[index] < 0) { - slices_[index] = pads_[index]; - pads_[index] = 0; - } - } - } - } - - ~PadBase() = default; - - Mode mode_{Mode::Constant}; - std::vector pads_; // After construction, only >=0 values are in here - std::vector slices_; // All of the negative padding values are separated out into slices_ - const float value_; // will always be float (when 'value' parsed from attribute - opset 10 and below) - - // flag used to differentiate the cases where some input values to the op are - // to be obtained from (is_dynamic_ = false) attributes vs (is_dynamic_ = true) inputs - bool is_dynamic_ = false; -}; - struct Pad final : public OpKernel, public PadBase { explicit Pad(const OpKernelInfo& info) : OpKernel(info), PadBase(info) {} diff --git a/onnxruntime/core/providers/cpu/tensor/padbase.h b/onnxruntime/core/providers/cpu/tensor/padbase.h new file mode 100644 index 0000000000..6da4a8fa5c --- /dev/null +++ b/onnxruntime/core/providers/cpu/tensor/padbase.h @@ -0,0 +1,71 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +#pragma once + +namespace onnxruntime { + +enum class Mode : int { + Constant = 0, + Reflect, + Edge +}; + +class PadBase { + public: + // Update the output_shape to make it consistent with numpy handling where there are one or more dimensions + // in the input_shape with a value of zero. + static Status HandleDimValueZero(const Mode& mode, const TensorShape& input_shape, TensorShape& output_shape); + + protected: + PadBase(const OpKernelInfo& info) : value_(info.GetAttrOrDefault("value", 0.f)) { + std::string mode; + if (info.GetAttr("mode", &mode).IsOK()) { + if (mode == "constant") + mode_ = Mode::Constant; + else if (mode == "reflect") + mode_ = Mode::Reflect; + else if (mode == "edge") + mode_ = Mode::Edge; + else + ORT_THROW("Invalid 'mode' attribute value"); + } + + const auto& kernel_def = info.GetKernelDef(); + + int start_ver, end_ver; + kernel_def.SinceVersion(&start_ver, &end_ver); + + // kMSDomain contrib kernel AND OnnxDomain start version >= 11 => DynamicPad + if (start_ver >= 11 || kernel_def.Domain() == kMSDomain) { + is_dynamic_ = true; + } + + if (!is_dynamic_) { + if (!info.GetAttrs("pads", pads_).IsOK()) + ORT_THROW("Invalid 'pads' attribute value"); + + // Separate out any negative pads_ into the slices_ array + slices_.resize(pads_.size(), 0); + for (size_t index = 0; index < pads_.size(); index++) { + if (pads_[index] < 0) { + slices_[index] = pads_[index]; + pads_[index] = 0; + } + } + } + } + + ~PadBase() = default; + + Mode mode_{Mode::Constant}; + std::vector pads_; // After construction, only >=0 values are in here + std::vector slices_; // All of the negative padding values are separated out into slices_ + const float value_; // will always be float (when 'value' parsed from attribute - opset 10 and below) + + // flag used to differentiate the cases where some input values to the op are + // to be obtained from (is_dynamic_ = false) attributes vs (is_dynamic_ = true) inputs + bool is_dynamic_ = false; +}; + +} // namespace onnxruntime diff --git a/onnxruntime/core/providers/cpu/tensor/reshape_helper.h b/onnxruntime/core/providers/cpu/tensor/reshape_helper.h index 655df9455a..b8fd8ea14f 100644 --- a/onnxruntime/core/providers/cpu/tensor/reshape_helper.h +++ b/onnxruntime/core/providers/cpu/tensor/reshape_helper.h @@ -3,10 +3,6 @@ #pragma once -#include "core/common/common.h" -#include "core/framework/tensor.h" -#include "gsl/gsl" - namespace onnxruntime { // Verify and convert unknown dim during reshape diff --git a/onnxruntime/core/providers/cpu/tensor/scatter_nd.h b/onnxruntime/core/providers/cpu/tensor/scatter_nd.h index 308c542722..da405b3203 100644 --- a/onnxruntime/core/providers/cpu/tensor/scatter_nd.h +++ b/onnxruntime/core/providers/cpu/tensor/scatter_nd.h @@ -3,8 +3,10 @@ #pragma once +#ifndef SHARED_PROVIDER #include "core/common/common.h" #include "core/framework/op_kernel.h" +#endif namespace onnxruntime { namespace concurrency { @@ -32,12 +34,13 @@ class ScatterNDBase { element_offsets(0) {} }; // struct Prepare + Status PrepareForCompute(OpKernelContext* context, Prepare& p) const; + + public: // Shared between the CPU and CUDA implementation static Status ValidateShapes(const TensorShape& input_shape, const TensorShape& indice_shape, const TensorShape& update_shape); - - Status PrepareForCompute(OpKernelContext* context, Prepare& p) const; }; // class ScatterNDBase class ScatterND final : public OpKernel, protected ScatterNDBase { @@ -50,4 +53,4 @@ class ScatterND final : public OpKernel, protected ScatterNDBase { Status ScatterString(const Prepare& p, concurrency::ThreadPool* tp) const; }; -} // namespace onnxruntime \ No newline at end of file +} // namespace onnxruntime diff --git a/onnxruntime/core/providers/cpu/tensor/shape_op.h b/onnxruntime/core/providers/cpu/tensor/shape_op.h index fe8346c503..6588109261 100644 --- a/onnxruntime/core/providers/cpu/tensor/shape_op.h +++ b/onnxruntime/core/providers/cpu/tensor/shape_op.h @@ -3,8 +3,10 @@ #pragma once +#ifndef SHARED_PROVIDER #include "core/common/common.h" #include "core/framework/op_kernel.h" +#endif #include "gsl/gsl" diff --git a/onnxruntime/core/providers/cpu/tensor/size.h b/onnxruntime/core/providers/cpu/tensor/size.h index 64e6b94356..3622e6cc4f 100644 --- a/onnxruntime/core/providers/cpu/tensor/size.h +++ b/onnxruntime/core/providers/cpu/tensor/size.h @@ -3,8 +3,10 @@ #pragma once +#ifndef SHARED_PROVIDER #include "core/common/common.h" #include "core/framework/op_kernel.h" +#endif namespace onnxruntime { diff --git a/onnxruntime/core/providers/cpu/tensor/slice.h b/onnxruntime/core/providers/cpu/tensor/slice.h index be69df1b38..4e52777fd1 100644 --- a/onnxruntime/core/providers/cpu/tensor/slice.h +++ b/onnxruntime/core/providers/cpu/tensor/slice.h @@ -1,9 +1,11 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. +#ifndef SHARED_PROVIDER #include "core/common/common.h" #include "core/framework/op_kernel.h" #include "core/util/math_cpuonly.h" +#endif namespace onnxruntime { diff --git a/onnxruntime/core/providers/cpu/tensor/split.h b/onnxruntime/core/providers/cpu/tensor/split.h index ff6dcacdb3..e455cb9157 100644 --- a/onnxruntime/core/providers/cpu/tensor/split.h +++ b/onnxruntime/core/providers/cpu/tensor/split.h @@ -5,18 +5,28 @@ #include +#ifndef SHARED_PROVIDER #include "core/common/common.h" #include "core/framework/op_kernel.h" +#endif namespace onnxruntime { class SplitBase { + public: + /* + * \param num_outputs must >=0 + */ + Status PrepareForCompute(const TensorShape& input_shape, int num_outputs, int64_t& axis, int& before_dims, + int& after_dims_including_split_axis, int& after_dims_excluding_split, + std::vector& split_sizes) const; + protected: SplitBase(const OpKernelInfo& info) { axis_ = info.GetAttrOrDefault("axis", 0); size_t numInputs = info.GetInputCount(); - if(numInputs == 1){ + if (numInputs == 1) { // optional if (info.GetAttrs("split", split_sizes_).IsOK()) { split_size_sum_ = std::accumulate(split_sizes_.cbegin(), split_sizes_.cend(), 0LL); @@ -26,13 +36,6 @@ class SplitBase { } } - /* - * \param num_outputs must >=0 - */ - Status PrepareForCompute(const TensorShape& input_shape, int num_outputs, int64_t& axis, int& before_dims, - int& after_dims_including_split_axis, int& after_dims_excluding_split, - std::vector& split_sizes) const; - int64_t axis_; std::vector split_sizes_; int64_t split_size_sum_ = -1; diff --git a/onnxruntime/core/providers/cpu/tensor/squeeze.h b/onnxruntime/core/providers/cpu/tensor/squeeze.h index c3d2544c40..31189d5f4e 100644 --- a/onnxruntime/core/providers/cpu/tensor/squeeze.h +++ b/onnxruntime/core/providers/cpu/tensor/squeeze.h @@ -3,10 +3,13 @@ #pragma once +#ifndef SHARED_PROVIDER #include "core/common/common.h" #include "core/framework/op_kernel.h" -#include "utils.h" #include "core/providers/common.h" +#endif + +#include "utils.h" namespace onnxruntime { @@ -91,4 +94,4 @@ class Squeeze final : public OpKernel, public SqueezeBase { } }; -} // namespace onnxruntime \ No newline at end of file +} // namespace onnxruntime diff --git a/onnxruntime/core/providers/cpu/tensor/tile.h b/onnxruntime/core/providers/cpu/tensor/tile.h index 093ee74969..c9479c01ad 100644 --- a/onnxruntime/core/providers/cpu/tensor/tile.h +++ b/onnxruntime/core/providers/cpu/tensor/tile.h @@ -1,9 +1,11 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. +#ifndef SHARED_PROVIDER #include "core/common/common.h" #include "core/framework/op_kernel.h" #include "core/util/math_cpuonly.h" +#endif namespace onnxruntime { diff --git a/onnxruntime/core/providers/cpu/tensor/transpose.h b/onnxruntime/core/providers/cpu/tensor/transpose.h index c003b2e8f2..aa335dcf55 100644 --- a/onnxruntime/core/providers/cpu/tensor/transpose.h +++ b/onnxruntime/core/providers/cpu/tensor/transpose.h @@ -3,9 +3,12 @@ #pragma once -#include "gsl/gsl" +#ifndef SHARED_PROVIDER #include "core/common/common.h" #include "core/framework/op_kernel.h" +#endif + +#include "gsl/gsl" #include namespace onnxruntime { diff --git a/onnxruntime/core/providers/cpu/tensor/unsqueeze.h b/onnxruntime/core/providers/cpu/tensor/unsqueeze.h index 4e9cfad137..9324edcdb5 100644 --- a/onnxruntime/core/providers/cpu/tensor/unsqueeze.h +++ b/onnxruntime/core/providers/cpu/tensor/unsqueeze.h @@ -1,14 +1,24 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. +#ifndef SHARED_PROVIDER #include "core/common/common.h" #include "core/framework/op_kernel.h" #include "core/util/math_cpuonly.h" #include "core/framework/tensor.h" +#endif namespace onnxruntime { class UnsqueezeBase { + public: + struct Prepare { + const Tensor* input_tensor = nullptr; + Tensor* output_tensor = nullptr; + }; + + Status PrepareCompute(OpKernelContext* context, Prepare& p) const; + protected: UnsqueezeBase(const OpKernelInfo& info) { size_t num_inputs = info.GetInputCount(); @@ -17,13 +27,6 @@ class UnsqueezeBase { } } - struct Prepare { - const Tensor* input_tensor = nullptr; - Tensor* output_tensor = nullptr; - }; - - Status PrepareCompute(OpKernelContext* context, Prepare& p) const; - private: std::vector axes_; }; diff --git a/onnxruntime/core/providers/cpu/tensor/upsample.h b/onnxruntime/core/providers/cpu/tensor/upsample.h index a7b3a77214..c2aea9374b 100644 --- a/onnxruntime/core/providers/cpu/tensor/upsample.h +++ b/onnxruntime/core/providers/cpu/tensor/upsample.h @@ -3,7 +3,9 @@ #pragma once +#ifndef SHARED_PROVIDER #include "core/framework/op_kernel.h" +#endif #include namespace onnxruntime { @@ -46,7 +48,7 @@ enum ResizeNearestMode { class UpsampleBase { protected: - UpsampleBase(OpKernelInfo info) : scales_cached_(false), roi_cached_(false), use_extrapolation_(false) { + UpsampleBase(const OpKernelInfo& info) : scales_cached_(false), roi_cached_(false), use_extrapolation_(false) { const auto& node = info.node(); auto opset = node.SinceVersion(); is_resize_ = (opset >= 10); diff --git a/onnxruntime/core/providers/cpu/tensor/utils.h b/onnxruntime/core/providers/cpu/tensor/utils.h index ddbad0a515..17beb1714c 100644 --- a/onnxruntime/core/providers/cpu/tensor/utils.h +++ b/onnxruntime/core/providers/cpu/tensor/utils.h @@ -3,7 +3,10 @@ #pragma once #include "gsl/gsl" + +#ifndef SHARED_PROVIDER #include "core/framework/utils.h" +#endif #include "core/common/safeint.h" namespace onnxruntime { diff --git a/onnxruntime/core/providers/cuda/activation/activations.cc b/onnxruntime/core/providers/cuda/activation/activations.cc index 4fb6b90655..08e0f86106 100644 --- a/onnxruntime/core/providers/cuda/activation/activations.cc +++ b/onnxruntime/core/providers/cuda/activation/activations.cc @@ -6,17 +6,17 @@ namespace onnxruntime { namespace cuda { -#define REGISTER_ACTIVATION_VERSIONED_KERNEL(x, startver, endver, T) \ - ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ - x, \ - kOnnxDomain, \ - startver, \ - endver, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder() \ - .TypeConstraint("T", DataTypeImpl::GetTensorType()) \ - .MayInplace(0, 0), \ +#define REGISTER_ACTIVATION_VERSIONED_KERNEL(x, startver, endver, T) \ + ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ + x, \ + kOnnxDomain, \ + startver, \ + endver, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()) \ + .TypeConstraint("T", DataTypeImpl::GetTensorType()) \ + .MayInplace(0, 0), \ x); #define REGISTER_ACTIVATION_KERNEL(x, ver, T) \ @@ -26,7 +26,7 @@ namespace cuda { ver, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()) \ .MayInplace(0, 0), \ x); diff --git a/onnxruntime/core/providers/cuda/activation/activations.h b/onnxruntime/core/providers/cuda/activation/activations.h index c0fdc1afdf..3a3656b12d 100644 --- a/onnxruntime/core/providers/cuda/activation/activations.h +++ b/onnxruntime/core/providers/cuda/activation/activations.h @@ -3,6 +3,7 @@ #pragma once +#include "core/providers/shared_library/provider_api.h" #include "core/providers/cuda/cuda_common.h" #include "core/providers/cuda/math/unary_elementwise_ops.h" #include "core/providers/cuda/math/binary_elementwise_ops.h" diff --git a/onnxruntime/core/providers/cuda/controlflow/if.cc b/onnxruntime/core/providers/cuda/controlflow/if.cc index de7c927af2..24d84dd520 100644 --- a/onnxruntime/core/providers/cuda/controlflow/if.cc +++ b/onnxruntime/core/providers/cuda/controlflow/if.cc @@ -1,6 +1,7 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. +#include "core/providers/shared_library/provider_api.h" #include "core/providers/cuda/controlflow/if.h" #include "core/providers/cuda/cuda_fwd.h" @@ -15,8 +16,8 @@ ONNX_OPERATOR_VERSIONED_KERNEL_EX(If, kOnnxDomain, 1, 10, kCudaExecutionProvider, - KernelDefBuilder() - .InputMemoryType(0) // 'cond' needs to be on CPU + (*KernelDefBuilder::Create()) + .InputMemoryType(OrtMemTypeCPUInput, 0) // 'cond' needs to be on CPU .TypeConstraint("B", DataTypeImpl::GetTensorType()) .TypeConstraint("V", DataTypeImpl::AllFixedSizeTensorTypes()), If); @@ -27,8 +28,8 @@ ONNX_OPERATOR_VERSIONED_KERNEL_EX(If, kOnnxDomain, 11, 12, kCudaExecutionProvider, - KernelDefBuilder() - .InputMemoryType(0) // 'cond' needs to be on CPU + (*KernelDefBuilder::Create()) + .InputMemoryType(OrtMemTypeCPUInput, 0) // 'cond' needs to be on CPU .TypeConstraint("B", DataTypeImpl::GetTensorType()) .TypeConstraint("V", DataTypeImpl::AllFixedSizeTensorTypes()), If); @@ -40,8 +41,8 @@ ONNX_OPERATOR_KERNEL_EX(If, kOnnxDomain, 13, kCudaExecutionProvider, - KernelDefBuilder() - .InputMemoryType(0) // 'cond' needs to be on CPU + (*KernelDefBuilder::Create()) + .InputMemoryType(OrtMemTypeCPUInput, 0) // 'cond' needs to be on CPU .TypeConstraint("B", DataTypeImpl::GetTensorType()) .TypeConstraint("V", DataTypeImpl::AllFixedSizeTensorTypes()), If); diff --git a/onnxruntime/core/providers/cuda/controlflow/loop.cc b/onnxruntime/core/providers/cuda/controlflow/loop.cc index 5d430ecfa6..cb5a38bb6c 100644 --- a/onnxruntime/core/providers/cuda/controlflow/loop.cc +++ b/onnxruntime/core/providers/cuda/controlflow/loop.cc @@ -1,9 +1,11 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. +#include "core/providers/shared_library/provider_api.h" #include "core/providers/cuda/controlflow/loop.h" #include "core/providers/cuda/cuda_common.h" #include "core/providers/cuda/cuda_fwd.h" +#include "core/framework/ml_value.h" using namespace ONNX_NAMESPACE; using namespace onnxruntime::common; @@ -15,9 +17,9 @@ ONNX_OPERATOR_VERSIONED_KERNEL_EX(Loop, kOnnxDomain, 1, 10, kCudaExecutionProvider, - KernelDefBuilder() - .InputMemoryType(0) // 'M' needs to be on CPU - .InputMemoryType(1) // 'cond' needs to be on CPU + (*KernelDefBuilder::Create()) + .InputMemoryType(OrtMemTypeCPUInput, 0) // 'M' needs to be on CPU + .InputMemoryType(OrtMemTypeCPUInput, 1) // 'cond' needs to be on CPU .TypeConstraint("I", DataTypeImpl::GetTensorType()) .TypeConstraint("B", DataTypeImpl::GetTensorType()) .TypeConstraint("V", DataTypeImpl::AllFixedSizeTensorTypes()), @@ -28,9 +30,9 @@ ONNX_OPERATOR_VERSIONED_KERNEL_EX(Loop, kOnnxDomain, 11, 12, kCudaExecutionProvider, - KernelDefBuilder() - .InputMemoryType(0) // 'M' needs to be on CPU - .InputMemoryType(1) // 'cond' needs to be on CPU + (*KernelDefBuilder::Create()) + .InputMemoryType(OrtMemTypeCPUInput, 0) // 'M' needs to be on CPU + .InputMemoryType(OrtMemTypeCPUInput, 1) // 'cond' needs to be on CPU .TypeConstraint("I", DataTypeImpl::GetTensorType()) .TypeConstraint("B", DataTypeImpl::GetTensorType()) .TypeConstraint("V", DataTypeImpl::AllFixedSizeTensorTypes()), @@ -43,9 +45,9 @@ ONNX_OPERATOR_KERNEL_EX(Loop, kOnnxDomain, 13, kCudaExecutionProvider, - KernelDefBuilder() - .InputMemoryType(0) // 'M' needs to be on CPU - .InputMemoryType(1) // 'cond' needs to be on CPU + (*KernelDefBuilder::Create()) + .InputMemoryType(OrtMemTypeCPUInput, 0) // 'M' needs to be on CPU + .InputMemoryType(OrtMemTypeCPUInput, 1) // 'cond' needs to be on CPU .TypeConstraint("I", DataTypeImpl::GetTensorType()) .TypeConstraint("B", DataTypeImpl::GetTensorType()) .TypeConstraint("V", DataTypeImpl::AllFixedSizeTensorTypes()), @@ -69,7 +71,7 @@ static Status ConcatenateGpuOutput(void* stream, std::vector& per_iter } CUDA_RETURN_IF_ERROR(cudaMemcpyAsync(cur_output, iteration_data.DataRaw(), bytes_per_iteration, - cudaMemcpyDeviceToDevice, static_cast(stream))); + cudaMemcpyDeviceToDevice, static_cast(stream))); cur_output = static_cast((static_cast(cur_output) + bytes_per_iteration)); } diff --git a/onnxruntime/core/providers/cuda/controlflow/scan.cc b/onnxruntime/core/providers/cuda/controlflow/scan.cc index 5bb83b5f7f..71caf71611 100644 --- a/onnxruntime/core/providers/cuda/controlflow/scan.cc +++ b/onnxruntime/core/providers/cuda/controlflow/scan.cc @@ -1,10 +1,15 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. +#include "core/providers/shared_library/provider_api.h" #include "core/providers/cuda/controlflow/scan.h" #include "core/providers/cuda/cuda_common.h" #include "core/providers/cuda/tensor/transpose.h" +#include "core/framework/ml_value.h" + +// TODO: It's ugly to include a .cc file but this .cc file defines the implementation of some templates which we need. +#include "core/framework/ort_value_tensor_slicer.cc" using namespace ONNX_NAMESPACE; using namespace onnxruntime::common; @@ -66,8 +71,8 @@ ONNX_OPERATOR_VERSIONED_KERNEL_EX(Scan, kOnnxDomain, 8, 8, kCudaExecutionProvider, - KernelDefBuilder() - .InputMemoryType(0) // 'sequence_lens' needs to be on CPU + (*KernelDefBuilder::Create()) + .InputMemoryType(OrtMemTypeCPUInput, 0) // 'sequence_lens' needs to be on CPU .TypeConstraint("I", DataTypeImpl::GetTensorType()) .TypeConstraint("V", DataTypeImpl::AllTensorTypes()), Scan<8>); @@ -76,7 +81,7 @@ ONNX_OPERATOR_VERSIONED_KERNEL_EX(Scan, kOnnxDomain, 9, 10, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .TypeConstraint("I", DataTypeImpl::GetTensorType()) .TypeConstraint("V", DataTypeImpl::AllFixedSizeTensorTypes()), Scan<9>); @@ -86,7 +91,7 @@ ONNX_OPERATOR_KERNEL_EX(Scan, kOnnxDomain, 11, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .TypeConstraint("I", DataTypeImpl::GetTensorType()) .TypeConstraint("V", DataTypeImpl::AllFixedSizeTensorTypes()), Scan<9>); diff --git a/onnxruntime/core/providers/cuda/controlflow/scan.h b/onnxruntime/core/providers/cuda/controlflow/scan.h index 577f3d8fc2..5b8fbb3ce8 100644 --- a/onnxruntime/core/providers/cuda/controlflow/scan.h +++ b/onnxruntime/core/providers/cuda/controlflow/scan.h @@ -5,7 +5,6 @@ #include #include "gsl/gsl" -#include "core/common/common.h" #include "core/providers/cpu/controlflow/scan.h" namespace onnxruntime { diff --git a/onnxruntime/core/providers/cuda/cuda_allocator.cc b/onnxruntime/core/providers/cuda/cuda_allocator.cc index 4adcd3e4c5..9dad792efc 100644 --- a/onnxruntime/core/providers/cuda/cuda_allocator.cc +++ b/onnxruntime/core/providers/cuda/cuda_allocator.cc @@ -4,7 +4,6 @@ #include "cuda_allocator.h" #include "cuda_common.h" #include "core/framework/allocatormgr.h" -#include "core/framework/session_state.h" #include "cuda_fence.h" #include "gpu_data_transfer.h" @@ -71,7 +70,6 @@ void* CUDAExternalAllocator::Alloc(size_t size) { // review(codemzs): ORT_ENFORCE does not seem appropiate. ORT_ENFORCE(p != nullptr); - } return p; diff --git a/onnxruntime/core/providers/cuda/cuda_call.cc b/onnxruntime/core/providers/cuda/cuda_call.cc index 6d94570a7e..227d1a19f1 100644 --- a/onnxruntime/core/providers/cuda/cuda_call.cc +++ b/onnxruntime/core/providers/cuda/cuda_call.cc @@ -1,10 +1,8 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. +#include "core/providers/shared_library/provider_api.h" #include "shared_inc/cuda_call.h" -#include "core/common/common.h" -#include "core/common/status.h" -#include "core/common/logging/logging.h" #ifdef _WIN32 #else // POSIX diff --git a/onnxruntime/core/providers/cuda/cuda_common.cc b/onnxruntime/core/providers/cuda/cuda_common.cc index 771c0338f1..57477f167c 100644 --- a/onnxruntime/core/providers/cuda/cuda_common.cc +++ b/onnxruntime/core/providers/cuda/cuda_common.cc @@ -2,8 +2,6 @@ // Licensed under the MIT License. #include "core/providers/cuda/cuda_common.h" -#include "core/common/logging/logging.h" -#include "core/common/logging/severity.h" #include "core/platform/env_var_utils.h" namespace onnxruntime { @@ -29,20 +27,5 @@ const HalfGemmOptions* HalfGemmOptions::GetInstance() { return &instance; } -void HalfGemmOptions::Initialize(int value) -{ - compute_16f_ = (value & 0x01) > 0; -#if defined(CUDA_VERSION) && CUDA_VERSION >= 11000 - disallow_reduced_precision_reduction_ = (value & 0x02) > 0; - pedantic_ = (value & 0x04) > 0; - LOGS_DEFAULT(INFO) << "ORT_CUDA_GEMM_OPTIONS: compute_16f=" << instance.compute_16f_ - << " disallow_reduced_precision_reduction=" << instance.disallow_reduced_precision_reduction_ - << " pedantic=" << instance.pedantic_; -#else - LOGS_DEFAULT(INFO) << "ORT_CUDA_GEMM_OPTIONS: compute_16f=" << instance.compute_16f_; -#endif - initialized_ = true; -} - } // namespace cuda } // namespace onnxruntime diff --git a/onnxruntime/core/providers/cuda/cuda_common.h b/onnxruntime/core/providers/cuda/cuda_common.h index 3835da81c8..94f56a346c 100644 --- a/onnxruntime/core/providers/cuda/cuda_common.h +++ b/onnxruntime/core/providers/cuda/cuda_common.h @@ -3,11 +3,13 @@ #pragma once +#include "core/providers/shared_library/provider_api.h" #include "core/common/status.h" +#include "core/framework/float16.h" #include "core/providers/cuda/cuda_pch.h" #include "core/providers/cuda/shared_inc/cuda_call.h" #include "core/providers/cuda/shared_inc/fast_divmod.h" -#include "core/util/math.h" +#include "gsl/gsl" namespace onnxruntime { namespace cuda { @@ -96,12 +98,12 @@ inline bool CalculateFdmStrides(gsl::span p, const std::vector= 7 : true ); +#if defined(CUDA_VERSION) && CUDA_VERSION < 11000 + enable_ = (mode == CUBLAS_TENSOR_OP_MATH ? prop.major >= 7 : true); #else enable_ = (mode == CUBLAS_TF32_TENSOR_OP_MATH ? prop.major >= 8 : true); #endif - + if (enable_) { cublasGetMathMode(handle, &mode_); enable_ = (mode_ != mode); @@ -156,7 +158,19 @@ class HalfGemmOptions { bool IsCompute16F() const { return compute_16f_; } - void Initialize(int value); + void Initialize(int value) { + compute_16f_ = (value & 0x01) > 0; +#if defined(CUDA_VERSION) && CUDA_VERSION >= 11000 + disallow_reduced_precision_reduction_ = (value & 0x02) > 0; + pedantic_ = (value & 0x04) > 0; + LOGS_DEFAULT(INFO) << "ORT_CUDA_GEMM_OPTIONS: compute_16f=" << instance.compute_16f_ + << " disallow_reduced_precision_reduction=" << instance.disallow_reduced_precision_reduction_ + << " pedantic=" << instance.pedantic_; +#else + LOGS_DEFAULT(INFO) << "ORT_CUDA_GEMM_OPTIONS: compute_16f=" << instance.compute_16f_; +#endif + initialized_ = true; + } private: // Default is FP32. Aggregate in FP16 might be faster but the cost is loss in precision. diff --git a/onnxruntime/core/providers/cuda/cuda_execution_provider.cc b/onnxruntime/core/providers/cuda/cuda_execution_provider.cc index ca2a59dd68..f3d5b7e8de 100644 --- a/onnxruntime/core/providers/cuda/cuda_execution_provider.cc +++ b/onnxruntime/core/providers/cuda/cuda_execution_provider.cc @@ -1,14 +1,9 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. +#include "core/providers/shared_library/provider_api.h" #include "core/providers/cuda/cuda_execution_provider.h" - -#include "core/framework/compute_capability.h" -#include "core/framework/fallback_cpu_capability.h" -#include "core/framework/kernel_registry.h" -#include "core/framework/memcpy.h" -#include "core/framework/provider_options_utils.h" -#include "core/graph/graph_utils.h" +#include "core/providers/cuda/cuda_common.h" #include "core/providers/cuda/cuda_allocator.h" #include "core/providers/cuda/cuda_fence.h" #include "core/providers/cuda/cuda_fwd.h" @@ -24,24 +19,28 @@ using namespace onnxruntime::common; -namespace { -struct KernelRegistryAndStatus { - std::shared_ptr kernel_registry = std::make_shared(); - Status st; -}; -} // namespace - namespace onnxruntime { -namespace cuda { +class Memcpy final : public OpKernel { + public: + Memcpy(const OpKernelInfo& info) : OpKernel{info} {} + Status Compute(OpKernelContext* ctx) const override { + const auto* X = ctx->Input(0); + Tensor* Y = ctx->Output(0, X->Shape()); + Status retval = Info().GetDataTransferManager().CopyTensor(*X, *Y, Info().GetKernelDef().ExecQueueId()); + return retval; + } +}; + +namespace cuda { ONNX_OPERATOR_KERNEL_EX( MemcpyFromHost, kOnnxDomain, 1, kCudaExecutionProvider, - KernelDefBuilder() - .InputMemoryType(0) + (*KernelDefBuilder::Create()) + .InputMemoryType(OrtMemTypeCPUInput, 0) .ExecQueueId(kCudaStreamCopyIn) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()), Memcpy); @@ -51,8 +50,8 @@ ONNX_OPERATOR_KERNEL_EX( kOnnxDomain, 1, kCudaExecutionProvider, - KernelDefBuilder() - .OutputMemoryType(0) + (*KernelDefBuilder::Create()) + .OutputMemoryType(OrtMemTypeCPUOutput, 0) .ExecQueueId(kCudaStreamCopyOut) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()), Memcpy); @@ -1123,8 +1122,7 @@ class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, template <> KernelCreateInfo BuildKernelCreateInfo() { - KernelCreateInfo info; - return info; + return {}; } static Status RegisterCudaKernels(KernelRegistry& kernel_registry) { @@ -1948,19 +1946,24 @@ static Status RegisterCudaKernels(KernelRegistry& kernel_registry) { return Status::OK(); } -KernelRegistryAndStatus GetCudaKernelRegistry() { - KernelRegistryAndStatus ret; - ret.st = RegisterCudaKernels(*ret.kernel_registry); - return ret; -} - } // namespace cuda +static std::shared_ptr s_kernel_registry; + +void Shutdown_DeleteRegistry() { + s_kernel_registry.reset(); +} + std::shared_ptr CUDAExecutionProvider::GetKernelRegistry() const { - static KernelRegistryAndStatus k = onnxruntime::cuda::GetCudaKernelRegistry(); - // throw if the registry failed to initialize - ORT_THROW_IF_ERROR(k.st); - return k.kernel_registry; + if (!s_kernel_registry) { + s_kernel_registry = KernelRegistry::Create(); + auto status = cuda::RegisterCudaKernels(*s_kernel_registry); + if (!status.IsOK()) + s_kernel_registry.reset(); + ORT_THROW_IF_ERROR(status); + } + + return s_kernel_registry; } static bool RNNNeedFallbackToCPU(const onnxruntime::Node& node, @@ -1969,8 +1972,8 @@ static bool RNNNeedFallbackToCPU(const onnxruntime::Node& node, const auto& node_attributes = node.GetAttributes(); // Check attributes for (auto& attr : node_attributes) { - auto attr_name = attr.first; - auto attr_value = attr.second; + auto& attr_name = attr.first; + auto& attr_value = attr.second; if ("activation_alpha" == attr_name || "activation_beta" == attr_name || "clip" == attr_name) { return true; @@ -2023,8 +2026,8 @@ static bool ConvTransposeNeedFallbackToCPU(const onnxruntime::Node& node) { const auto& node_attributes = node.GetAttributes(); // Check attributes for (auto& attr : node_attributes) { - auto attr_name = attr.first; - auto attr_value = attr.second; + auto& attr_name = attr.first; + auto& attr_value = attr.second; // cudnn only supports symmetric padding, so drop the node down to CPU if the padding provided is asymmetric // TODO: Check if we can adopt a similar approach to deal with asymmetric pads in 'ConvTranspose' @@ -2074,8 +2077,8 @@ static bool CastNeedFallbackToCPU(const onnxruntime::Node& node) { const auto& node_attributes = node.GetAttributes(); // Check attributes for (auto& attr : node_attributes) { - auto attr_name = attr.first; - auto attr_value = attr.second; + auto& attr_name = attr.first; + auto& attr_value = attr.second; // string is not supported if ("to" == attr_name && ::ONNX_NAMESPACE::AttributeProto_AttributeType::AttributeProto_AttributeType_INT == attr_value.type()) { @@ -2163,9 +2166,9 @@ CUDAExecutionProvider::GetCapability(const onnxruntime::GraphViewer& graph, if (cpu_nodes.count(node_index) > 0) continue; - std::unique_ptr sub_graph = std::make_unique(); - sub_graph->nodes.push_back(node_index); - result.push_back(std::make_unique(std::move(sub_graph))); + auto sub_graph = IndexedSubGraph::Create(); + sub_graph->Nodes().push_back(node_index); + result.push_back(ComputeCapability::Create(std::move(sub_graph))); } return result; } @@ -2218,4 +2221,4 @@ void CUDAExecutionProvider::RegisterAllocator(std::shared_ptr TryInsertAllocator(cuda_cpu_alloc); } -} // namespace onnxruntime \ No newline at end of file +} // namespace onnxruntime diff --git a/onnxruntime/core/providers/cuda/cuda_execution_provider.h b/onnxruntime/core/providers/cuda/cuda_execution_provider.h index 192af61ffb..c1c35284ee 100644 --- a/onnxruntime/core/providers/cuda/cuda_execution_provider.h +++ b/onnxruntime/core/providers/cuda/cuda_execution_provider.h @@ -6,7 +6,6 @@ #include #include -#include "core/graph/constants.h" #include "core/framework/allocatormgr.h" #include "core/framework/arena_extend_strategy.h" #include "core/framework/execution_provider.h" @@ -170,9 +169,21 @@ class CUDAExecutionProvider : public IExecutionProvider { using PerThreadContextMap = std::unordered_map>; // thread local PerThreadContext cache + + struct ContextCacheHolder { + ContextCacheHolder() { + // Keep a weak pointer to the object, if the weak pointer can be locked, then the shared pointer is still around, so we can reset it + RunOnUnload([&, weak_p_ = std::weak_ptr(p)] { + if (auto lock = weak_p_.lock()) + p.reset(); + }); + } + std::shared_ptr p{std::make_shared()}; + }; + static const std::shared_ptr& PerThreadContextCache() { - thread_local const auto per_thread_context_cache = std::make_shared(); - return per_thread_context_cache; + thread_local const ContextCacheHolder per_thread_context_cache; + return per_thread_context_cache.p; } struct PerThreadContextState { diff --git a/onnxruntime/core/providers/cuda/cuda_execution_provider_info.cc b/onnxruntime/core/providers/cuda/cuda_execution_provider_info.cc index d91572cd03..02dfc5af69 100644 --- a/onnxruntime/core/providers/cuda/cuda_execution_provider_info.cc +++ b/onnxruntime/core/providers/cuda/cuda_execution_provider_info.cc @@ -1,6 +1,7 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. +#include "core/providers/shared_library/provider_api.h" #include "core/providers/cuda/cuda_execution_provider_info.h" #include "core/common/make_string.h" @@ -22,13 +23,13 @@ constexpr const char* kGpuExternalFree = "gpu_external_free"; } // namespace cuda namespace { -const EnumNameMapping ort_cudnn_conv_algo_search_mapping{ +const DeleteOnUnloadPtr> ort_cudnn_conv_algo_search_mapping = new EnumNameMapping{ {OrtCudnnConvAlgoSearch::EXHAUSTIVE, "EXHAUSTIVE"}, {OrtCudnnConvAlgoSearch::HEURISTIC, "HEURISTIC"}, {OrtCudnnConvAlgoSearch::DEFAULT, "DEFAULT"}, }; -const EnumNameMapping arena_extend_strategy_mapping{ +const DeleteOnUnloadPtr> arena_extend_strategy_mapping = new EnumNameMapping{ {ArenaExtendStrategy::kNextPowerOfTwo, "kNextPowerOfTwo"}, {ArenaExtendStrategy::kSameAsRequested, "kSameAsRequested"}, }; @@ -59,7 +60,7 @@ CUDAExecutionProviderInfo CUDAExecutionProviderInfo::FromProviderOptions(const P [&alloc](const std::string& value_str) -> Status { size_t address; ORT_RETURN_IF_ERROR(ParseStringWithClassicLocale(value_str, address)); - alloc = reinterpret_cast(address); + alloc = reinterpret_cast(address); return Status::OK(); }) .AddValueParser( @@ -67,16 +68,16 @@ CUDAExecutionProviderInfo CUDAExecutionProviderInfo::FromProviderOptions(const P [&free](const std::string& value_str) -> Status { size_t address; ORT_RETURN_IF_ERROR(ParseStringWithClassicLocale(value_str, address)); - free = reinterpret_cast(address); + free = reinterpret_cast(address); return Status::OK(); }) .AddAssignmentToReference(cuda::provider_option_names::kMemLimit, info.gpu_mem_limit) .AddAssignmentToEnumReference( cuda::provider_option_names::kArenaExtendStrategy, - arena_extend_strategy_mapping, info.arena_extend_strategy) + *arena_extend_strategy_mapping, info.arena_extend_strategy) .AddAssignmentToEnumReference( cuda::provider_option_names::kCudnnConvAlgoSearch, - ort_cudnn_conv_algo_search_mapping, info.cudnn_conv_algo_search) + *ort_cudnn_conv_algo_search_mapping, info.cudnn_conv_algo_search) .AddAssignmentToReference(cuda::provider_option_names::kDoCopyInDefaultStream, info.do_copy_in_default_stream) .Parse(options)); @@ -92,9 +93,9 @@ ProviderOptions CUDAExecutionProviderInfo::ToProviderOptions(const CUDAExecution {cuda::provider_option_names::kGpuExternalAlloc, MakeStringWithClassicLocale(reinterpret_cast(info.external_allocator_info.alloc))}, {cuda::provider_option_names::kGpuExternalFree, MakeStringWithClassicLocale(reinterpret_cast(info.external_allocator_info.free))}, {cuda::provider_option_names::kArenaExtendStrategy, - EnumToName(arena_extend_strategy_mapping, info.arena_extend_strategy)}, + EnumToName(*arena_extend_strategy_mapping, info.arena_extend_strategy)}, {cuda::provider_option_names::kCudnnConvAlgoSearch, - EnumToName(ort_cudnn_conv_algo_search_mapping, info.cudnn_conv_algo_search)}, + EnumToName(*ort_cudnn_conv_algo_search_mapping, info.cudnn_conv_algo_search)}, {cuda::provider_option_names::kDoCopyInDefaultStream, MakeStringWithClassicLocale(info.do_copy_in_default_stream)}, }; diff --git a/onnxruntime/core/providers/cuda/cuda_fence.cc b/onnxruntime/core/providers/cuda/cuda_fence.cc index 3dbb3464dd..74462650a5 100644 --- a/onnxruntime/core/providers/cuda/cuda_fence.cc +++ b/onnxruntime/core/providers/cuda/cuda_fence.cc @@ -1,11 +1,9 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. -#include "core/providers/cuda/cuda_fence.h" - -#include "core/graph/constants.h" #include "core/providers/cuda/cuda_common.h" #include "core/providers/cuda/gpu_data_transfer.h" +#include "core/providers/cuda/cuda_fence.h" namespace onnxruntime { diff --git a/onnxruntime/core/providers/cuda/cuda_fence.h b/onnxruntime/core/providers/cuda/cuda_fence.h index 462ea4f6b4..1d0c5b3512 100644 --- a/onnxruntime/core/providers/cuda/cuda_fence.h +++ b/onnxruntime/core/providers/cuda/cuda_fence.h @@ -3,9 +3,7 @@ #pragma once -#include "core/framework/tensor.h" -#include "core/graph/basic_types.h" -#include "core/providers/cuda/cuda_common.h" +#include "core/framework/fence.h" namespace onnxruntime { class GPUDataTransfer; diff --git a/onnxruntime/core/providers/cuda/cuda_fwd.h b/onnxruntime/core/providers/cuda/cuda_fwd.h index dde99ab0f4..4f2cc556c0 100644 --- a/onnxruntime/core/providers/cuda/cuda_fwd.h +++ b/onnxruntime/core/providers/cuda/cuda_fwd.h @@ -3,11 +3,9 @@ #pragma once -#include "core/framework/op_kernel.h" - namespace onnxruntime { namespace cuda { template KernelCreateInfo BuildKernelCreateInfo(); } -} +} // namespace onnxruntime diff --git a/onnxruntime/core/providers/cuda/cuda_kernel.h b/onnxruntime/core/providers/cuda/cuda_kernel.h index 8dd0a5b781..f6fe4807b7 100644 --- a/onnxruntime/core/providers/cuda/cuda_kernel.h +++ b/onnxruntime/core/providers/cuda/cuda_kernel.h @@ -3,9 +3,6 @@ #pragma once -#include "core/common/status.h" -#include "core/framework/data_transfer_manager.h" -#include "core/framework/op_kernel.h" #include "core/providers/cuda/cuda_common.h" #include "core/providers/cuda/cuda_execution_provider.h" #include "core/providers/cuda/cuda_fwd.h" diff --git a/onnxruntime/core/providers/cuda/cuda_provider_factory.cc b/onnxruntime/core/providers/cuda/cuda_provider_factory.cc index d87988ab25..20906f349c 100644 --- a/onnxruntime/core/providers/cuda/cuda_provider_factory.cc +++ b/onnxruntime/core/providers/cuda/cuda_provider_factory.cc @@ -1,6 +1,7 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. +#include "core/providers/shared_library/provider_api.h" #include "core/providers/cuda/cuda_provider_factory_creator.h" #include "core/providers/cuda/cuda_provider_factory.h" @@ -9,15 +10,22 @@ #include "gsl/gsl" #include "core/providers/cuda/cuda_execution_provider.h" -#include "core/providers/cuda/cuda_execution_provider_info.h" -#include "core/session/abi_session_options_impl.h" -#include "core/session/onnxruntime_c_api.h" -#include "core/session/ort_apis.h" +#include "core/providers/cuda/cuda_allocator.h" +#include "core/providers/cuda/gpu_data_transfer.h" +#include "core/providers/cuda/math/unary_elementwise_ops_impl.h" using namespace onnxruntime; namespace onnxruntime { +#if defined(USE_CUDA) && defined(ORT_USE_NCCL) && defined(USE_NCCL_P2P) +namespace cuda { +cuda::INcclService& GetINcclService(); +} +#endif + +void Shutdown_DeleteRegistry(); + struct CUDAProviderFactory : IExecutionProviderFactory { CUDAProviderFactory(const CUDAExecutionProviderInfo& info) : info_{info} {} @@ -37,57 +45,139 @@ std::shared_ptr CreateExecutionProviderFactory_CUDA(c return std::make_shared(info); } +struct ProviderInfo_CUDA_Impl : ProviderInfo_CUDA { + OrtStatus* SetCurrentGpuDeviceId(_In_ int device_id) override { + int num_devices; + auto cuda_err = ::cudaGetDeviceCount(&num_devices); + if (cuda_err != cudaSuccess) { + return CreateStatus(ORT_FAIL, "Failed to set device id since cudaGetDeviceCount failed."); + } + + if (device_id >= num_devices) { + std::ostringstream ostr; + ostr << "Invalid device id. Device id should be less than total number of devices (" << num_devices << ")"; + return CreateStatus(ORT_INVALID_ARGUMENT, ostr.str().c_str()); + } + + cuda_err = cudaSetDevice(device_id); + if (cuda_err != cudaSuccess) { + return CreateStatus(ORT_FAIL, "Failed to set device id."); + } + return nullptr; + } + + OrtStatus* GetCurrentGpuDeviceId(_In_ int* device_id) override { + auto cuda_err = cudaGetDevice(device_id); + if (cuda_err != cudaSuccess) { + return CreateStatus(ORT_FAIL, "Failed to get device id."); + } + return nullptr; + } + + std::unique_ptr CreateCUDAAllocator(int16_t device_id, const char* name) override { + return std::make_unique(device_id, name); + } + + std::unique_ptr CreateCUDAPinnedAllocator(int16_t device_id, const char* name) override { + return std::make_unique(device_id, name); + } + + std::unique_ptr CreateGPUDataTransfer(void* stream) override { + return std::make_unique(static_cast(stream)); + } + + void cuda__Impl_Cast(void* stream, const int64_t* input_data, int32_t* output_data, size_t count) override { + return cuda::Impl_Cast(static_cast(stream), input_data, output_data, count); + } + + void cuda__Impl_Cast(void* stream, const int32_t* input_data, int64_t* output_data, size_t count) override { + return cuda::Impl_Cast(static_cast(stream), input_data, output_data, count); + } + + bool CudaCall_false(int retCode, const char* exprString, const char* libName, int successCode, const char* msg) override { return CudaCall(cudaError(retCode), exprString, libName, cudaError(successCode), msg); } + bool CudaCall_true(int retCode, const char* exprString, const char* libName, int successCode, const char* msg) override { return CudaCall(cudaError(retCode), exprString, libName, cudaError(successCode), msg); } + + void CopyGpuToCpu(void* dst_ptr, const void* src_ptr, const size_t size, const OrtMemoryInfo& dst_location, const OrtMemoryInfo& src_location) override { + ORT_ENFORCE(dst_location.device.Type() == OrtDevice::CPU); + + // Current CUDA device. + int device; + CUDA_CALL(cudaGetDevice(&device)); + + if (device != src_location.id) { + // Need to switch to the allocating device. + CUDA_CALL(cudaSetDevice(src_location.id)); + // Copy from GPU to CPU. + CUDA_CALL(cudaMemcpy(dst_ptr, src_ptr, size, cudaMemcpyDeviceToHost)); + // Switch back to current device. + CUDA_CALL(cudaSetDevice(device)); + } else { + // Copy from GPU to CPU. + CUDA_CALL(cudaMemcpy(dst_ptr, src_ptr, size, cudaMemcpyDeviceToHost)); + } + } + + // Used by slice_concatenate_test.cc and onnxruntime_pybind_state.cc + void cudaMemcpy_HostToDevice(void* dst, const void* src, size_t count) override { CUDA_CALL_THROW(cudaMemcpy(dst, src, count, cudaMemcpyHostToDevice)); } + // Used by onnxruntime_pybind_state.cc + void cudaMemcpy_DeviceToHost(void* dst, const void* src, size_t count) override { CUDA_CALL_THROW(cudaMemcpy(dst, src, count, cudaMemcpyDeviceToHost)); } + + int cudaGetDeviceCount() override { + int num_devices = 0; + CUDA_CALL_THROW(::cudaGetDeviceCount(&num_devices)); + return num_devices; + } + + void CUDAExecutionProviderInfo__FromProviderOptions(const ProviderOptions& options, CUDAExecutionProviderInfo& info) { + info = CUDAExecutionProviderInfo::FromProviderOptions(options); + } + +#if defined(USE_CUDA) && defined(ORT_USE_NCCL) && defined(USE_NCCL_P2P) + cuda::INcclService& GetINcclService() override { + return cuda::GetINcclService(); + } +#endif + + std::shared_ptr CreateExecutionProviderFactory(const CUDAExecutionProviderInfo& info) override { + return std::make_shared(info); + } + + std::shared_ptr CreateCudaAllocator(int16_t device_id, size_t gpu_mem_limit, onnxruntime::ArenaExtendStrategy arena_extend_strategy, onnxruntime::CUDAExecutionProviderExternalAllocatorInfo& external_allocator_info, OrtArenaCfg* default_memory_arena_cfg) override { + return CUDAExecutionProvider::CreateCudaAllocator(device_id, gpu_mem_limit, arena_extend_strategy, external_allocator_info, default_memory_arena_cfg); + } + +} g_info; + +struct CUDA_Provider : Provider { + void* GetInfo() override { return &g_info; } + + std::shared_ptr CreateExecutionProviderFactory(const void* void_params) override { + auto params = reinterpret_cast(void_params); + + CUDAExecutionProviderInfo info{}; + info.device_id = gsl::narrow(params->device_id); + info.gpu_mem_limit = params->gpu_mem_limit; + info.arena_extend_strategy = static_cast(params->arena_extend_strategy); + info.cudnn_conv_algo_search = params->cudnn_conv_algo_search; + info.do_copy_in_default_stream = params->do_copy_in_default_stream; + info.has_user_compute_stream = params->has_user_compute_stream; + info.user_compute_stream = params->user_compute_stream; + info.default_memory_arena_cfg = params->default_memory_arena_cfg; + + return std::make_shared(info); + } + + void Shutdown() override { + Shutdown_DeleteRegistry(); + } + +} g_provider; + } // namespace onnxruntime -ORT_API_STATUS_IMPL(OrtSessionOptionsAppendExecutionProvider_CUDA, _In_ OrtSessionOptions* options, int device_id) { - CUDAExecutionProviderInfo info{}; - info.device_id = gsl::narrow(device_id); +extern "C" { - options->provider_factories.push_back(onnxruntime::CreateExecutionProviderFactory_CUDA(info)); - - return nullptr; +ORT_API(onnxruntime::Provider*, GetProvider) { + return &onnxruntime::g_provider; } - -ORT_API_STATUS_IMPL(OrtApis::SessionOptionsAppendExecutionProvider_CUDA, - _In_ OrtSessionOptions* options, _In_ const OrtCUDAProviderOptions* cuda_options) { - CUDAExecutionProviderInfo info{}; - info.device_id = gsl::narrow(cuda_options->device_id); - info.gpu_mem_limit = cuda_options->gpu_mem_limit; - info.arena_extend_strategy = static_cast(cuda_options->arena_extend_strategy); - info.cudnn_conv_algo_search = cuda_options->cudnn_conv_algo_search; - info.do_copy_in_default_stream = cuda_options->do_copy_in_default_stream; - info.has_user_compute_stream = cuda_options->has_user_compute_stream; - info.user_compute_stream = cuda_options->user_compute_stream; - info.default_memory_arena_cfg = cuda_options->default_memory_arena_cfg; - options->provider_factories.push_back(onnxruntime::CreateExecutionProviderFactory_CUDA(info)); - - return nullptr; -} - -ORT_API_STATUS_IMPL(OrtApis::SetCurrentGpuDeviceId, _In_ int device_id) { - int num_devices; - auto cuda_err = cudaGetDeviceCount(&num_devices); - if (cuda_err != cudaSuccess) { - return CreateStatus(ORT_FAIL, "Failed to set device id since cudaGetDeviceCount failed."); - } - - if (device_id >= num_devices) { - std::ostringstream ostr; - ostr << "Invalid device id. Device id should be less than total number of devices (" << num_devices << ")"; - return CreateStatus(ORT_INVALID_ARGUMENT, ostr.str().c_str()); - } - - cuda_err = cudaSetDevice(device_id); - if (cuda_err != cudaSuccess) { - return CreateStatus(ORT_FAIL, "Failed to set device id."); - } - return nullptr; -} - -ORT_API_STATUS_IMPL(OrtApis::GetCurrentGpuDeviceId, _In_ int* device_id) { - auto cuda_err = cudaGetDevice(device_id); - if (cuda_err != cudaSuccess) { - return CreateStatus(ORT_FAIL, "Failed to get device id."); - } - return nullptr; } diff --git a/onnxruntime/core/providers/cuda/cudnn_common.h b/onnxruntime/core/providers/cuda/cudnn_common.h index 39c1b74405..b9337d9127 100644 --- a/onnxruntime/core/providers/cuda/cudnn_common.h +++ b/onnxruntime/core/providers/cuda/cudnn_common.h @@ -5,8 +5,6 @@ #include -#include "core/common/logging/logging.h" -#include "core/framework/tensor.h" #include "core/providers/cuda/cuda_common.h" namespace onnxruntime { @@ -127,7 +125,7 @@ struct Consts { }; #if defined(CUDA_VERSION) && CUDA_VERSION >= 11000 -template<> +template <> struct Consts { static const float Zero; static const float One; diff --git a/onnxruntime/core/providers/cuda/exported_symbols.lst b/onnxruntime/core/providers/cuda/exported_symbols.lst new file mode 100644 index 0000000000..f4c4141259 --- /dev/null +++ b/onnxruntime/core/providers/cuda/exported_symbols.lst @@ -0,0 +1 @@ +_GetProvider diff --git a/onnxruntime/core/providers/cuda/generator/constant_of_shape.cc b/onnxruntime/core/providers/cuda/generator/constant_of_shape.cc index 052fdf5dfe..bdcd3ff48e 100644 --- a/onnxruntime/core/providers/cuda/generator/constant_of_shape.cc +++ b/onnxruntime/core/providers/cuda/generator/constant_of_shape.cc @@ -1,9 +1,6 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. - #include "constant_of_shape.h" -#include "core/providers/common.h" -#include "gsl/gsl" using namespace ::onnxruntime::common; using namespace ONNX_NAMESPACE; @@ -15,8 +12,8 @@ ONNX_OPERATOR_KERNEL_EX( kOnnxDomain, 9, kCudaExecutionProvider, - KernelDefBuilder() - .InputMemoryType(0) + (*KernelDefBuilder::Create()) + .InputMemoryType(OrtMemTypeCPUInput, 0) .TypeConstraint("T1", DataTypeImpl::GetTensorType()) .TypeConstraint("T2", DataTypeImpl::AllFixedSizeTensorTypes()), ConstantOfShape); @@ -29,11 +26,11 @@ Status ConstantOfShape::ComputeInternal(OpKernelContext* ctx) const { const void* value_ptr = GetValuePtr(); const auto element_size = output_tensor->DataType()->Size(); -#define CASE(TYPE) \ - case sizeof(TYPE): \ - if (size > 0) { \ - cuda::Fill(Stream(), reinterpret_cast(output_data), *(reinterpret_cast(value_ptr)), size); \ - } \ +#define CASE(TYPE) \ + case sizeof(TYPE): \ + if (size > 0) { \ + cuda::Fill(Stream(), reinterpret_cast(output_data), *(reinterpret_cast(value_ptr)), size); \ + } \ break; switch (element_size) { diff --git a/onnxruntime/core/providers/cuda/generator/constant_of_shape.h b/onnxruntime/core/providers/cuda/generator/constant_of_shape.h index a0870d915a..99c5da0615 100644 --- a/onnxruntime/core/providers/cuda/generator/constant_of_shape.h +++ b/onnxruntime/core/providers/cuda/generator/constant_of_shape.h @@ -3,8 +3,6 @@ #pragma once -#include "core/common/common.h" -#include "core/framework/data_types.h" #include "core/providers/cuda/cuda_kernel.h" #include "core/providers/cpu/generator/constant_of_shape_base.h" #include "core/providers/cuda/shared_inc/cuda_utils.h" diff --git a/onnxruntime/core/providers/cuda/generator/range.cc b/onnxruntime/core/providers/cuda/generator/range.cc index df1fda338e..5b2afa01fc 100644 --- a/onnxruntime/core/providers/cuda/generator/range.cc +++ b/onnxruntime/core/providers/cuda/generator/range.cc @@ -1,7 +1,7 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. -#include "core/framework/tensorprotoutils.h" +#include "core/providers/shared_library/provider_api.h" #include "core/providers/cuda/cuda_common.h" #include "range.h" #include "range_impl.h" @@ -18,10 +18,10 @@ ONNX_OPERATOR_KERNEL_EX( kOnnxDomain, 11, kCudaExecutionProvider, - KernelDefBuilder() - .InputMemoryType(0) // start - .InputMemoryType(1) // limit - .InputMemoryType(2) // delta + (*KernelDefBuilder::Create()) + .InputMemoryType(OrtMemTypeCPUInput, 0) // start + .InputMemoryType(OrtMemTypeCPUInput, 1) // limit + .InputMemoryType(OrtMemTypeCPUInput, 2) // delta .TypeConstraint("T", {DataTypeImpl::GetTensorType(), DataTypeImpl::GetTensorType(), DataTypeImpl::GetTensorType(), @@ -51,7 +51,7 @@ static Status ComputeRange(cudaStream_t stream, OpKernelContext* ctx) { delta_tensor_ptr->Shape()); } - // Start, Limit and Delta are stored in CPU. + // Start, Limit and Delta are stored in CPU. T start = *(start_tensor.template Data()); T limit = *(limit_tensor.template Data()); diff --git a/onnxruntime/core/providers/cuda/gpu_data_transfer.cc b/onnxruntime/core/providers/cuda/gpu_data_transfer.cc index cd83b3d612..a8a7d2eb60 100644 --- a/onnxruntime/core/providers/cuda/gpu_data_transfer.cc +++ b/onnxruntime/core/providers/cuda/gpu_data_transfer.cc @@ -1,6 +1,7 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. +#include "core/providers/shared_library/provider_api.h" #include "core/providers/cuda/gpu_data_transfer.h" #include "cuda_common.h" diff --git a/onnxruntime/core/providers/cuda/math/binary_elementwise_ops.cc b/onnxruntime/core/providers/cuda/math/binary_elementwise_ops.cc index e1764a49b2..a20e8e15f3 100644 --- a/onnxruntime/core/providers/cuda/math/binary_elementwise_ops.cc +++ b/onnxruntime/core/providers/cuda/math/binary_elementwise_ops.cc @@ -85,47 +85,47 @@ Status BinaryElementwise::Prepare(OpKernelContext* context, Bin return Status::OK(); } -#define BINARY_ELEMENTWISE_REGISTER_KERNEL_TYPED_V(x, class_name, ver, T) \ - ONNX_OPERATOR_TYPED_KERNEL_EX( \ - x, \ - kOnnxDomain, \ - ver, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ +#define BINARY_ELEMENTWISE_REGISTER_KERNEL_TYPED_V(x, class_name, ver, T) \ + ONNX_OPERATOR_TYPED_KERNEL_EX( \ + x, \ + kOnnxDomain, \ + ver, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ class_name); #define BINARY_ELEMENTWISE_REGISTER_KERNEL_TYPED(x, ver, T) \ BINARY_ELEMENTWISE_REGISTER_KERNEL_TYPED_V(x, x, ver, T) -#define BINARY_ELEMENTWISE_REGISTER_KERNEL_NONTEMP(x, class_name, ver, ...) \ - ONNX_OPERATOR_TYPED_KERNEL_EX( \ - x, \ - kOnnxDomain, \ - ver, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", BuildKernelDefConstraints<>(__VAR_ARGS__)), \ +#define BINARY_ELEMENTWISE_REGISTER_KERNEL_NONTEMP(x, class_name, ver, ...) \ + ONNX_OPERATOR_TYPED_KERNEL_EX( \ + x, \ + kOnnxDomain, \ + ver, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", BuildKernelDefConstraints<>(__VAR_ARGS__)), \ class_name); -#define BINARY_ELEMENTWISE_LOGICALOP_REGISTER_KERNEL_TYPED(x, ver, T) \ - ONNX_OPERATOR_TYPED_KERNEL_EX( \ - x, \ - kOnnxDomain, \ - ver, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()).TypeConstraint("T1", DataTypeImpl::GetTensorType()), \ +#define BINARY_ELEMENTWISE_LOGICALOP_REGISTER_KERNEL_TYPED(x, ver, T) \ + ONNX_OPERATOR_TYPED_KERNEL_EX( \ + x, \ + kOnnxDomain, \ + ver, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()).TypeConstraint("T1", DataTypeImpl::GetTensorType()), \ x); -#define BINARY_ELEMENTWISE_REGISTER_KERNEL_VERSIONED_TYPED(x, startver, endver, T) \ - ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ - x, \ - kOnnxDomain, \ - startver, \ - endver, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ +#define BINARY_ELEMENTWISE_REGISTER_KERNEL_VERSIONED_TYPED(x, startver, endver, T) \ + ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ + x, \ + kOnnxDomain, \ + startver, \ + endver, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ x); #define BINARY_ELEMENTWISE_REGISTER_KERNEL_VERSIONED_TYPED_CLASS(x, class_name, startver, endver, T) \ @@ -136,7 +136,7 @@ Status BinaryElementwise::Prepare(OpKernelContext* context, Bin endver, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ class_name); #define BINARY_ELEMENTWISE_COMPUTE(x, T) \ @@ -318,7 +318,7 @@ ONNX_OPERATOR_VERSIONED_KERNEL_EX( kOnnxDomain, 12, 12, kCudaExecutionProvider, - KernelDefBuilder().TypeConstraint("T", BuildKernelDefConstraints()).TypeConstraint("T1", BuildKernelDefConstraints()), + (*KernelDefBuilder::Create()).TypeConstraint("T", BuildKernelDefConstraints()).TypeConstraint("T1", BuildKernelDefConstraints()), Pow); ONNX_OPERATOR_KERNEL_EX( @@ -326,7 +326,7 @@ ONNX_OPERATOR_KERNEL_EX( kOnnxDomain, 13, kCudaExecutionProvider, - KernelDefBuilder().TypeConstraint("T", BuildKernelDefConstraints()).TypeConstraint("T1", BuildKernelDefConstraints()), + (*KernelDefBuilder::Create()).TypeConstraint("T", BuildKernelDefConstraints()).TypeConstraint("T1", BuildKernelDefConstraints()), Pow); namespace pow12_internal { diff --git a/onnxruntime/core/providers/cuda/math/clip.cc b/onnxruntime/core/providers/cuda/math/clip.cc index 02d986c393..1189d1e698 100644 --- a/onnxruntime/core/providers/cuda/math/clip.cc +++ b/onnxruntime/core/providers/cuda/math/clip.cc @@ -1,7 +1,6 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. -#include "core/providers/common.h" #include "core/providers/cuda/math/clip.h" #include "core/providers/cuda/math/clip_impl.h" @@ -15,7 +14,7 @@ ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( 10, float, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .TypeConstraint("T", DataTypeImpl::GetTensorType()), Clip_6); @@ -24,7 +23,7 @@ ONNX_OPERATOR_VERSIONED_KERNEL_EX( kOnnxDomain, 11, 11, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .TypeConstraint("T", DataTypeImpl::GetTensorType()), Clip); @@ -33,7 +32,7 @@ ONNX_OPERATOR_VERSIONED_KERNEL_EX( kOnnxDomain, 12, 12, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .TypeConstraint("T", BuildKernelDefConstraints()), Clip); @@ -42,11 +41,10 @@ ONNX_OPERATOR_KERNEL_EX( kOnnxDomain, 13, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .TypeConstraint("T", BuildKernelDefConstraints()), Clip); - template Status Clip_6::ComputeInternal(OpKernelContext* ctx) const { const Tensor& X = *ctx->Input(0); @@ -127,3 +125,4 @@ Status Clip::ComputeInternal(OpKernelContext* ctx) const { } // namespace cuda } // namespace onnxruntime + diff --git a/onnxruntime/core/providers/cuda/math/clip.h b/onnxruntime/core/providers/cuda/math/clip.h index b156deaecf..bfa4b42aa5 100644 --- a/onnxruntime/core/providers/cuda/math/clip.h +++ b/onnxruntime/core/providers/cuda/math/clip.h @@ -9,7 +9,7 @@ namespace onnxruntime { namespace cuda { template -class Clip_6 final : public onnxruntime::clip_internal::Clip_6Base, public CudaKernel { +class Clip_6 final : public onnxruntime::clip_internal::Clip_6Base, public CudaKernel { public: explicit Clip_6(const OpKernelInfo& info) : onnxruntime::clip_internal::Clip_6Base(info), CudaKernel{info} { } @@ -27,7 +27,7 @@ class Clip final : public CudaKernel { Status ComputeInternal(OpKernelContext* context) const override; private: - template + template struct ComputeImpl; }; diff --git a/onnxruntime/core/providers/cuda/math/clip_impl.h b/onnxruntime/core/providers/cuda/math/clip_impl.h index 9f537110aa..365615830f 100644 --- a/onnxruntime/core/providers/cuda/math/clip_impl.h +++ b/onnxruntime/core/providers/cuda/math/clip_impl.h @@ -2,7 +2,6 @@ // Licensed under the MIT License. #pragma once - #include "core/providers/cuda/math/clip.h" #include "core/providers/cuda/cuda_common.h" #include "core/providers/cuda/shared_inc/cuda_utils.h" diff --git a/onnxruntime/core/providers/cuda/math/cumsum.cc b/onnxruntime/core/providers/cuda/math/cumsum.cc index 9541eec6da..7e496ef928 100644 --- a/onnxruntime/core/providers/cuda/math/cumsum.cc +++ b/onnxruntime/core/providers/cuda/math/cumsum.cc @@ -14,8 +14,8 @@ ONNX_OPERATOR_VERSIONED_KERNEL_EX( kOnnxDomain, 11, 13, kCudaExecutionProvider, - KernelDefBuilder() - .InputMemoryType(1) // 'axis' needs to be on CPU + (*KernelDefBuilder::Create()) + .InputMemoryType(OrtMemTypeCPUInput, 1) // 'axis' needs to be on CPU .TypeConstraint("T", std::vector{ DataTypeImpl::GetTensorType(), DataTypeImpl::GetTensorType(), @@ -31,8 +31,8 @@ ONNX_OPERATOR_KERNEL_EX( kOnnxDomain, 14, kCudaExecutionProvider, - KernelDefBuilder() - .InputMemoryType(1) // 'axis' needs to be on CPU + (*KernelDefBuilder::Create()) + .InputMemoryType(OrtMemTypeCPUInput, 1) // 'axis' needs to be on CPU .TypeConstraint("T", std::vector{ DataTypeImpl::GetTensorType(), DataTypeImpl::GetTensorType(), @@ -40,7 +40,7 @@ ONNX_OPERATOR_KERNEL_EX( DataTypeImpl::GetTensorType(), DataTypeImpl::GetTensorType(), DataTypeImpl::GetTensorType(), - DataTypeImpl::GetTensorType()}) // MLFloat16 is added in opset 14 + DataTypeImpl::GetTensorType()}) // MLFloat16 is added in opset 14 .TypeConstraint("T2", std::vector{DataTypeImpl::GetTensorType(), DataTypeImpl::GetTensorType()}), CumSum); diff --git a/onnxruntime/core/providers/cuda/math/einsum.cc b/onnxruntime/core/providers/cuda/math/einsum.cc index 8333561fac..8c07a8190e 100644 --- a/onnxruntime/core/providers/cuda/math/einsum.cc +++ b/onnxruntime/core/providers/cuda/math/einsum.cc @@ -5,6 +5,13 @@ namespace onnxruntime { +// This function must exist due to the C++ base class constructor needing this to be defined for the vtable, but it is never called. +Status Einsum::DeviceCompute(OpKernelContext* /*context*/, const std::vector& /*inputs*/, + AllocatorPtr /*allocator*/, concurrency::ThreadPool* /*tp*/) const { + assert(false); + return Status::OK(); +} + namespace cuda { ONNX_OPERATOR_KERNEL_EX( @@ -12,11 +19,7 @@ ONNX_OPERATOR_KERNEL_EX( kOnnxDomain, 12, kCudaExecutionProvider, - KernelDefBuilder().TypeConstraint("T", - std::vector{ - DataTypeImpl::GetTensorType(), - DataTypeImpl::GetTensorType(), - DataTypeImpl::GetTensorType()}), + (*KernelDefBuilder::Create()).TypeConstraint("T", std::vector{DataTypeImpl::GetTensorType(), DataTypeImpl::GetTensorType(), DataTypeImpl::GetTensorType()}), Einsum); Status Einsum::Compute(OpKernelContext* context) const { @@ -30,46 +33,46 @@ Status Einsum::DeviceCompute(OpKernelContext* context, const std::vectorSetDeviceHelpers(EinsumOp::DeviceHelpers::CudaDeviceHelpers::Diagonal, + EinsumOp::DeviceHelpers::CudaDeviceHelpers::Transpose); // Compute all required metadata to be used at Einsum compute time and return error status code if one was generated - ORT_RETURN_IF_ERROR(einsum_compute_preprocessor.Run()); + ORT_RETURN_IF_ERROR(einsum_compute_preprocessor->Run()); // EinsumComputeProcessor section - if (inputs[0]->IsDataType()) { - auto einsum_compute_processor = EinsumTypedComputeProcessor(context, allocator, tp, - einsum_compute_preprocessor, - &einsum_cuda_assets); + auto einsum_compute_processor = EinsumTypedComputeProcessor::Create(context, allocator, tp, + *einsum_compute_preprocessor, + &einsum_cuda_assets); - einsum_compute_processor.SetDeviceHelpers(EinsumOp::DeviceHelpers::CudaDeviceHelpers::Transpose, - EinsumOp::DeviceHelpers::CudaDeviceHelpers::MatMul, - EinsumOp::DeviceHelpers::CudaDeviceHelpers::ReduceSum, - EinsumOp::DeviceHelpers::CudaDeviceHelpers::DataCopy); - return einsum_compute_processor.Run(); + einsum_compute_processor->SetDeviceHelpers(EinsumOp::DeviceHelpers::CudaDeviceHelpers::Transpose, + EinsumOp::DeviceHelpers::CudaDeviceHelpers::MatMul, + EinsumOp::DeviceHelpers::CudaDeviceHelpers::ReduceSum, + EinsumOp::DeviceHelpers::CudaDeviceHelpers::DataCopy); + return einsum_compute_processor->Run(); } else if (inputs[0]->IsDataType()) { - auto einsum_compute_processor = EinsumTypedComputeProcessor(context, allocator, tp, - einsum_compute_preprocessor, - &einsum_cuda_assets); + auto einsum_compute_processor = EinsumTypedComputeProcessor::Create(context, allocator, tp, + *einsum_compute_preprocessor, + &einsum_cuda_assets); // Set device specific methods (CPU methods) to be used during processing - einsum_compute_processor.SetDeviceHelpers(EinsumOp::DeviceHelpers::CudaDeviceHelpers::Transpose, - EinsumOp::DeviceHelpers::CudaDeviceHelpers::MatMul, - EinsumOp::DeviceHelpers::CudaDeviceHelpers::ReduceSum, - EinsumOp::DeviceHelpers::CudaDeviceHelpers::DataCopy); - return einsum_compute_processor.Run(); + einsum_compute_processor->SetDeviceHelpers(EinsumOp::DeviceHelpers::CudaDeviceHelpers::Transpose, + EinsumOp::DeviceHelpers::CudaDeviceHelpers::MatMul, + EinsumOp::DeviceHelpers::CudaDeviceHelpers::ReduceSum, + EinsumOp::DeviceHelpers::CudaDeviceHelpers::DataCopy); + return einsum_compute_processor->Run(); } else if (inputs[0]->IsDataType()) { - auto einsum_compute_processor = EinsumTypedComputeProcessor(context, allocator, tp, - einsum_compute_preprocessor, - &einsum_cuda_assets); + auto einsum_compute_processor = EinsumTypedComputeProcessor::Create(context, allocator, tp, + *einsum_compute_preprocessor, + &einsum_cuda_assets); - einsum_compute_processor.SetDeviceHelpers(EinsumOp::DeviceHelpers::CudaDeviceHelpers::Transpose, - EinsumOp::DeviceHelpers::CudaDeviceHelpers::MatMul, - EinsumOp::DeviceHelpers::CudaDeviceHelpers::ReduceSum, - EinsumOp::DeviceHelpers::CudaDeviceHelpers::DataCopy); - return einsum_compute_processor.Run(); + einsum_compute_processor->SetDeviceHelpers(EinsumOp::DeviceHelpers::CudaDeviceHelpers::Transpose, + EinsumOp::DeviceHelpers::CudaDeviceHelpers::MatMul, + EinsumOp::DeviceHelpers::CudaDeviceHelpers::ReduceSum, + EinsumOp::DeviceHelpers::CudaDeviceHelpers::DataCopy); + return einsum_compute_processor->Run(); } return ORT_MAKE_STATUS(ONNXRUNTIME, NOT_IMPLEMENTED, diff --git a/onnxruntime/core/providers/cuda/math/einsum.h b/onnxruntime/core/providers/cuda/math/einsum.h index ca7563d367..7ca541b0ce 100644 --- a/onnxruntime/core/providers/cuda/math/einsum.h +++ b/onnxruntime/core/providers/cuda/math/einsum.h @@ -3,10 +3,9 @@ #pragma once -#include "core/providers/cpu/math/einsum.h" -#include "core/common/common.h" -#include "core/framework/op_kernel.h" +#include "core/platform/threadpool.h" #include "core/providers/cuda/cuda_common.h" +#include "core/providers/cpu/math/einsum.h" #include "einsum_utils/einsum_auxiliary_ops.h" #include "core/providers/cuda/cuda_execution_provider.h" diff --git a/onnxruntime/core/providers/cuda/math/einsum_utils/einsum_auxiliary_ops.cc b/onnxruntime/core/providers/cuda/math/einsum_utils/einsum_auxiliary_ops.cc index d9dd97919a..5868bf225b 100644 --- a/onnxruntime/core/providers/cuda/math/einsum_utils/einsum_auxiliary_ops.cc +++ b/onnxruntime/core/providers/cuda/math/einsum_utils/einsum_auxiliary_ops.cc @@ -1,6 +1,14 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. +#include "core/providers/shared_library/provider_api.h" + +namespace onnxruntime { +namespace concurrency { +class ThreadPool; +} +} // namespace onnxruntime + #include "einsum_auxiliary_ops.h" namespace onnxruntime { @@ -70,10 +78,10 @@ Status MatMul(const T* input_1_data, const T* input_2_data, T* output_data, // CUDA EP specific ReduceSum helper template -Tensor ReduceSum(const Tensor& input, const std::vector& reduce_axes, - bool keep_dims, AllocatorPtr allocator, - const TensorShape* input_shape_override, - concurrency::ThreadPool* /*tp*/, void* einsum_cuda_assets) { +std::unique_ptr ReduceSum(const Tensor& input, const std::vector& reduce_axes, + bool keep_dims, AllocatorPtr allocator, + const TensorShape* input_shape_override, + concurrency::ThreadPool* /*tp*/, void* einsum_cuda_assets) { return cuda::ReductionOps::ReduceCompute(*static_cast(einsum_cuda_assets)->cuda_ep_, CUDNN_REDUCE_TENSOR_ADD, allocator, input, reduce_axes, keep_dims, false, false, false, @@ -106,7 +114,7 @@ std::unique_ptr Diagonal(const Tensor& input, int64_t dim_1, int64_t dim // The diagonal values are stored along `first_dim` output_dims.erase(output_dims.begin() + second_dim); - std::unique_ptr output = std::make_unique(input.DataType(), output_dims, allocator); + auto output = Tensor::Create(input.DataType(), output_dims, allocator); TensorPitches input_strides(input.Shape().GetDims()); cuda::TArray gpu_input_strides(input_strides); @@ -146,7 +154,7 @@ template Status DeviceHelpers::CudaDeviceHelpers::MatMul( size_t num_batches, size_t M, size_t K, size_t N, concurrency::ThreadPool* tp, void* einsum_cuda_assets); -template Tensor DeviceHelpers::CudaDeviceHelpers::ReduceSum( +template std::unique_ptr DeviceHelpers::CudaDeviceHelpers::ReduceSum( const Tensor& input, const std::vector& reduce_axes, bool keep_dims, AllocatorPtr allocator, const TensorShape* input_shape_override, @@ -159,7 +167,7 @@ template Status DeviceHelpers::CudaDeviceHelpers::MatMul( size_t num_batches, size_t M, size_t K, size_t N, concurrency::ThreadPool* tp, void* einsum_cuda_assets); -template Tensor DeviceHelpers::CudaDeviceHelpers::ReduceSum( +template std::unique_ptr DeviceHelpers::CudaDeviceHelpers::ReduceSum( const Tensor& input, const std::vector& reduce_axes, bool keep_dims, AllocatorPtr allocator, const TensorShape* input_shape_override, @@ -172,7 +180,7 @@ template Status DeviceHelpers::CudaDeviceHelpers::MatMul( size_t num_batches, size_t M, size_t K, size_t N, concurrency::ThreadPool* tp, void* einsum_cuda_assets); -template Tensor DeviceHelpers::CudaDeviceHelpers::ReduceSum( +template std::unique_ptr DeviceHelpers::CudaDeviceHelpers::ReduceSum( const Tensor& input, const std::vector& reduce_axes, bool keep_dims, AllocatorPtr allocator, const TensorShape* input_shape_override, diff --git a/onnxruntime/core/providers/cuda/math/einsum_utils/einsum_auxiliary_ops.h b/onnxruntime/core/providers/cuda/math/einsum_utils/einsum_auxiliary_ops.h index 797f8f301e..6ba139f2da 100644 --- a/onnxruntime/core/providers/cuda/math/einsum_utils/einsum_auxiliary_ops.h +++ b/onnxruntime/core/providers/cuda/math/einsum_utils/einsum_auxiliary_ops.h @@ -47,10 +47,10 @@ Status MatMul(const T* input_1_data, const T* input_2_data, T* output_data, void* einsum_cuda_assets); template -Tensor ReduceSum(const Tensor& input, const std::vector& reduce_axes, - bool keep_dims, AllocatorPtr allocator, - const TensorShape* input_shape_override, - concurrency::ThreadPool* /*tp*/, void* einsum_cuda_assets); +std::unique_ptr ReduceSum(const Tensor& input, const std::vector& reduce_axes, + bool keep_dims, AllocatorPtr allocator, + const TensorShape* input_shape_override, + concurrency::ThreadPool* /*tp*/, void* einsum_cuda_assets); std::unique_ptr Diagonal(const Tensor& input, int64_t dim_1, int64_t dim_2, AllocatorPtr allocator, void* einsum_cuda_assets); diff --git a/onnxruntime/core/providers/cuda/math/gemm.cc b/onnxruntime/core/providers/cuda/math/gemm.cc index 79eeb5eaf2..def63e4db2 100644 --- a/onnxruntime/core/providers/cuda/math/gemm.cc +++ b/onnxruntime/core/providers/cuda/math/gemm.cc @@ -17,7 +17,7 @@ namespace cuda { 8, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()), \ Gemm); \ ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ @@ -27,7 +27,7 @@ namespace cuda { 10, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()), \ Gemm); \ ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ @@ -37,7 +37,7 @@ namespace cuda { 12, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()), \ Gemm); \ ONNX_OPERATOR_TYPED_KERNEL_EX( \ @@ -46,7 +46,7 @@ namespace cuda { 13, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()), \ Gemm); diff --git a/onnxruntime/core/providers/cuda/math/matmul.cc b/onnxruntime/core/providers/cuda/math/matmul.cc index e0ec185bdb..1a233aec96 100644 --- a/onnxruntime/core/providers/cuda/math/matmul.cc +++ b/onnxruntime/core/providers/cuda/math/matmul.cc @@ -16,25 +16,25 @@ namespace cuda { 1, 8, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()), \ MatMul); \ ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ MatMul, \ kOnnxDomain, \ - 9, 12, \ + 9, 12, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()), \ MatMul); \ ONNX_OPERATOR_TYPED_KERNEL_EX( \ MatMul, \ kOnnxDomain, \ - 13, \ + 13, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()), \ MatMul); diff --git a/onnxruntime/core/providers/cuda/math/matmul_integer.cc b/onnxruntime/core/providers/cuda/math/matmul_integer.cc index 89e738fca0..57e835da20 100644 --- a/onnxruntime/core/providers/cuda/math/matmul_integer.cc +++ b/onnxruntime/core/providers/cuda/math/matmul_integer.cc @@ -18,9 +18,9 @@ ONNX_OPERATOR_TYPED_KERNEL_EX( 10, int8_t, kCudaExecutionProvider, - KernelDefBuilder() - .InputMemoryType(2) - .InputMemoryType(3) + (*KernelDefBuilder::Create()) + .InputMemoryType(OrtMemTypeCPUInput, 2) + .InputMemoryType(OrtMemTypeCPUInput, 3) .TypeConstraint("T1", DataTypeImpl::GetTensorType()) .TypeConstraint("T2", DataTypeImpl::GetTensorType()) .TypeConstraint("T3", DataTypeImpl::GetTensorType()), diff --git a/onnxruntime/core/providers/cuda/math/softmax.cc b/onnxruntime/core/providers/cuda/math/softmax.cc index 7efeb6425e..a9f2f61f63 100644 --- a/onnxruntime/core/providers/cuda/math/softmax.cc +++ b/onnxruntime/core/providers/cuda/math/softmax.cc @@ -50,7 +50,7 @@ Status SoftMaxComputeHelper( return Status::OK(); } -#define SPECIALIZED_SOFTMAX_HELPER_IMPL(T) \ +#define SPECIALIZED_SOFTMAX_HELPER_IMPL(T) \ template Status SoftMaxComputeHelper(cudaStream_t stream, const T* input, const TensorShape& shape, T* Y, cudnnHandle_t handle, int64_t axis); \ template Status SoftMaxComputeHelper(cudaStream_t stream, const T* input, const TensorShape& shape, T* Y, cudnnHandle_t handle, int64_t axis); @@ -60,81 +60,81 @@ SPECIALIZED_SOFTMAX_HELPER_IMPL(MLFloat16) #if defined(CUDA_VERSION) && CUDA_VERSION >= 11000 // cudnnSoftmaxForward/Backward doesn't support BFloat16. -#define SPECIALIZED_SOFTMAX_HELPER_IMPL_BFloat16(is_log_softmax) \ - template <> \ - Status SoftMaxComputeHelper( \ - cudaStream_t stream, \ - const BFloat16* X, \ - const TensorShape& input_shape, \ - BFloat16* Y, \ - cudnnHandle_t, \ - int64_t axis) { \ - typedef typename ToCudaType::MappedType CudaT; \ - int64_t N = input_shape.SizeToDimension(axis); \ - int64_t D = input_shape.SizeFromDimension(axis); \ - auto Y_data = reinterpret_cast(Y); \ - auto X_data = reinterpret_cast(X); \ - dispatch_softmax_forward, is_log_softmax>( \ - stream, Y_data, X_data, gsl::narrow_cast(D), gsl::narrow_cast(D), gsl::narrow_cast(N)); \ - return Status::OK(); \ +#define SPECIALIZED_SOFTMAX_HELPER_IMPL_BFloat16(is_log_softmax) \ + template <> \ + Status SoftMaxComputeHelper( \ + cudaStream_t stream, \ + const BFloat16* X, \ + const TensorShape& input_shape, \ + BFloat16* Y, \ + cudnnHandle_t, \ + int64_t axis) { \ + typedef typename ToCudaType::MappedType CudaT; \ + int64_t N = input_shape.SizeToDimension(axis); \ + int64_t D = input_shape.SizeFromDimension(axis); \ + auto Y_data = reinterpret_cast(Y); \ + auto X_data = reinterpret_cast(X); \ + dispatch_softmax_forward, is_log_softmax>( \ + stream, Y_data, X_data, gsl::narrow_cast(D), gsl::narrow_cast(D), gsl::narrow_cast(N)); \ + return Status::OK(); \ } SPECIALIZED_SOFTMAX_HELPER_IMPL_BFloat16(true) -SPECIALIZED_SOFTMAX_HELPER_IMPL_BFloat16(false) + SPECIALIZED_SOFTMAX_HELPER_IMPL_BFloat16(false) #endif -#define REGISTER_KERNEL_TYPED(T) \ - ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ - Softmax, \ - kOnnxDomain, \ - 1, 10, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ - Softmax); \ - ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ - Softmax, \ - kOnnxDomain, \ - 11, 12, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ - Softmax); \ - ONNX_OPERATOR_TYPED_KERNEL_EX( \ - Softmax, \ - kOnnxDomain, \ - 13, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ - Softmax); \ - ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ - LogSoftmax, \ - kOnnxDomain, \ - 1, 10, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ - Softmax); \ - ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ - LogSoftmax, \ - kOnnxDomain, \ - 11, 12, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ - Softmax); \ - ONNX_OPERATOR_TYPED_KERNEL_EX( \ - LogSoftmax, \ - kOnnxDomain, \ - 13, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ +#define REGISTER_KERNEL_TYPED(T) \ + ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ + Softmax, \ + kOnnxDomain, \ + 1, 10, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ + Softmax); \ + ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ + Softmax, \ + kOnnxDomain, \ + 11, 12, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ + Softmax); \ + ONNX_OPERATOR_TYPED_KERNEL_EX( \ + Softmax, \ + kOnnxDomain, \ + 13, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ + Softmax); \ + ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ + LogSoftmax, \ + kOnnxDomain, \ + 1, 10, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ + Softmax); \ + ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ + LogSoftmax, \ + kOnnxDomain, \ + 11, 12, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ + Softmax); \ + ONNX_OPERATOR_TYPED_KERNEL_EX( \ + LogSoftmax, \ + kOnnxDomain, \ + 13, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ Softmax); -template -Status Softmax::ComputeInternal(OpKernelContext* ctx) const { + template + Status Softmax::ComputeInternal(OpKernelContext* ctx) const { const Tensor* X = ctx->Input(0); const TensorShape& input_shape{X->Shape()}; size_t rank = input_shape.NumDimensions(); @@ -148,9 +148,9 @@ Status Softmax::ComputeInternal(OpKernelContext* ctx) const { const size_t axis = static_cast(HandleNegativeAxis(axis_, rank)); bool is_transpose_required = false; - Tensor transposed_input; + std::unique_ptr transposed_input; std::vector transposed_input_dims; - Tensor intermediate_output; // output that the softmax implementation will write into while using transposed input + std::unique_ptr intermediate_output; // output that the softmax implementation will write into while using transposed input std::vector permutation(rank); // The "semantic" meaning of axis has changed in opset-13. @@ -181,18 +181,17 @@ Status Softmax::ComputeInternal(OpKernelContext* ctx) const { } // Allocate a temporary tensor to hold transposed input - Tensor temp_input(X->DataType(), TensorShape(transposed_input_dims), alloc); + auto temp_input = Tensor::Create(X->DataType(), TensorShape(transposed_input_dims), alloc); // Perform the transpose ORT_RETURN_IF_ERROR(Transpose::DoTranspose(cuda_ep_->GetDeviceProp(), Stream(), CublasHandle(), - permutation, *X, temp_input)); + permutation, *X, *temp_input)); transposed_input = std::move(temp_input); // Allocate memory for the intermediate output - Tensor temp_output(Y->DataType(), TensorShape(transposed_input_dims), alloc); - intermediate_output = std::move(temp_output); + intermediate_output = Tensor::Create(Y->DataType(), TensorShape(transposed_input_dims), alloc); } const T* X_data = nullptr; @@ -200,9 +199,9 @@ Status Softmax::ComputeInternal(OpKernelContext* ctx) const { const TensorShape* compute_input_shape = nullptr; if (is_transpose_required) { // use intermediate buffers to compute the softmax values - X_data = transposed_input.template Data(); - Y_data = intermediate_output.template MutableData(); - compute_input_shape = &transposed_input.Shape(); + X_data = transposed_input->template Data(); + Y_data = intermediate_output->template MutableData(); + compute_input_shape = &transposed_input->Shape(); } else { // use the node input/output directly X_data = X->template Data(); Y_data = Y->template MutableData(); @@ -232,7 +231,7 @@ Status Softmax::ComputeInternal(OpKernelContext* ctx) const { ORT_RETURN_IF_ERROR(Transpose::DoTranspose(cuda_ep_->GetDeviceProp(), Stream(), CublasHandle(), - reverse_permutation, intermediate_output, *Y)); + reverse_permutation, *intermediate_output, *Y)); } return Status::OK(); diff --git a/onnxruntime/core/providers/cuda/math/topk.cc b/onnxruntime/core/providers/cuda/math/topk.cc index 8903437ead..6baaaf2a6d 100644 --- a/onnxruntime/core/providers/cuda/math/topk.cc +++ b/onnxruntime/core/providers/cuda/math/topk.cc @@ -12,7 +12,7 @@ ONNX_OPERATOR_VERSIONED_KERNEL_EX( kOnnxDomain, 1, 9, kCudaExecutionProvider, - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()), + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()), TopK); ONNX_OPERATOR_VERSIONED_KERNEL_EX( @@ -20,7 +20,7 @@ ONNX_OPERATOR_VERSIONED_KERNEL_EX( kOnnxDomain, 10, 10, kCudaExecutionProvider, - KernelDefBuilder().InputMemoryType(1).TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()).TypeConstraint("I", DataTypeImpl::GetTensorType()), + (*KernelDefBuilder::Create()).InputMemoryType(OrtMemTypeCPUInput, 1).TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()).TypeConstraint("I", DataTypeImpl::GetTensorType()), TopK); ONNX_OPERATOR_KERNEL_EX( @@ -28,7 +28,7 @@ ONNX_OPERATOR_KERNEL_EX( kOnnxDomain, 11, kCudaExecutionProvider, - KernelDefBuilder().InputMemoryType(1).TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()).TypeConstraint("I", DataTypeImpl::GetTensorType()), + (*KernelDefBuilder::Create()).InputMemoryType(OrtMemTypeCPUInput, 1).TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()).TypeConstraint("I", DataTypeImpl::GetTensorType()), TopK); template diff --git a/onnxruntime/core/providers/cuda/math/topk_impl.cu b/onnxruntime/core/providers/cuda/math/topk_impl.cu index db609b9aa9..9f81f428dd 100644 --- a/onnxruntime/core/providers/cuda/math/topk_impl.cu +++ b/onnxruntime/core/providers/cuda/math/topk_impl.cu @@ -2,7 +2,6 @@ // Licensed under the MIT License. #include "topk_impl.h" -#include "core/framework/data_types.h" #include "core/providers/cuda/cu_inc/common.cuh" #include "device_atomic_functions.h" #include "cub/cub.cuh" diff --git a/onnxruntime/core/providers/cuda/math/unary_elementwise_ops.cc b/onnxruntime/core/providers/cuda/math/unary_elementwise_ops.cc index 5a235749a9..a2780f7b51 100644 --- a/onnxruntime/core/providers/cuda/math/unary_elementwise_ops.cc +++ b/onnxruntime/core/providers/cuda/math/unary_elementwise_ops.cc @@ -13,35 +13,35 @@ Status UnaryElementwise::Prepare(OpKernelContext* context, UnaryElementwisePrepa return Status::OK(); } -#define UNARY_ELEMENTWISE_REGISTER_VERSIONED_KERNEL(x, startver, endver, T) \ - ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ - x, \ - kOnnxDomain, \ - startver, \ - endver, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ +#define UNARY_ELEMENTWISE_REGISTER_VERSIONED_KERNEL(x, startver, endver, T) \ + ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ + x, \ + kOnnxDomain, \ + startver, \ + endver, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ x); -#define UNARY_ELEMENTWISE_REGISTER_KERNEL(x, ver, T) \ - ONNX_OPERATOR_TYPED_KERNEL_EX( \ - x, \ - kOnnxDomain, \ - ver, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ +#define UNARY_ELEMENTWISE_REGISTER_KERNEL(x, ver, T) \ + ONNX_OPERATOR_TYPED_KERNEL_EX( \ + x, \ + kOnnxDomain, \ + ver, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ x); -#define UNARY_ELEMENTWISE_LOGICALOP_REGISTER_KERNEL_TYPED(x, ver, T) \ - ONNX_OPERATOR_TYPED_KERNEL_EX( \ - x, \ - kOnnxDomain, \ - ver, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()).TypeConstraint("T1", DataTypeImpl::GetTensorType()), \ +#define UNARY_ELEMENTWISE_LOGICALOP_REGISTER_KERNEL_TYPED(x, ver, T) \ + ONNX_OPERATOR_TYPED_KERNEL_EX( \ + x, \ + kOnnxDomain, \ + ver, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()).TypeConstraint("T1", DataTypeImpl::GetTensorType()), \ x); #define UNARY_ELEMENTWISE_COMPUTE(x, T) \ diff --git a/onnxruntime/core/providers/cuda/math/variadic_elementwise_ops.cc b/onnxruntime/core/providers/cuda/math/variadic_elementwise_ops.cc index 96223da2f9..24e066d5d8 100644 --- a/onnxruntime/core/providers/cuda/math/variadic_elementwise_ops.cc +++ b/onnxruntime/core/providers/cuda/math/variadic_elementwise_ops.cc @@ -1,6 +1,7 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. +#include "core/providers/shared_library/provider_api.h" #include "core/providers/cuda/math/variadic_elementwise_ops.h" #include @@ -204,22 +205,20 @@ using MaxOp = VariadicElementwiseOp< variadic_elementwise_ops::Max, uint32_t, uint64_t, int32_t, int64_t, ALL_IEEE_FLOAT_DATA_TYPES>; -const auto k_uzilhfd_datatypes = - BuildKernelDefConstraints(); -const auto k_hfd_datatypes = - BuildKernelDefConstraints(); +const DeleteOnUnloadPtr> k_uzilhfd_datatypes = new std::vector(BuildKernelDefConstraints()); +const DeleteOnUnloadPtr> k_hfd_datatypes = new std::vector(BuildKernelDefConstraints()); } // namespace // kernel registration -#define REGISTER_KERNEL(name, impl_class, version, datatypes) \ - ONNX_OPERATOR_KERNEL_EX( \ - name, \ - kOnnxDomain, \ - version, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", datatypes), \ +#define REGISTER_KERNEL(name, impl_class, version, datatypes) \ + ONNX_OPERATOR_KERNEL_EX( \ + name, \ + kOnnxDomain, \ + version, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", datatypes), \ impl_class) #define REGISTER_VERSIONED_KERNEL(name, impl_class, start_version, end_version, datatypes) \ @@ -228,20 +227,20 @@ const auto k_hfd_datatypes = kOnnxDomain, \ start_version, end_version, \ kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", datatypes), \ + (*KernelDefBuilder::Create()).TypeConstraint("T", datatypes), \ impl_class) -REGISTER_KERNEL(Sum, SumOp, 13, k_hfd_datatypes) -REGISTER_VERSIONED_KERNEL(Sum, SumOp, 8, 12, k_hfd_datatypes) -REGISTER_VERSIONED_KERNEL(Sum, SumOp, 6, 7, k_hfd_datatypes) +REGISTER_KERNEL(Sum, SumOp, 13, *k_hfd_datatypes) +REGISTER_VERSIONED_KERNEL(Sum, SumOp, 8, 12, *k_hfd_datatypes) +REGISTER_VERSIONED_KERNEL(Sum, SumOp, 6, 7, *k_hfd_datatypes) -REGISTER_KERNEL(Min, MinOp, 13, k_uzilhfd_datatypes) -REGISTER_VERSIONED_KERNEL(Min, MinOp, 12, 12, k_uzilhfd_datatypes) -REGISTER_VERSIONED_KERNEL(Min, MinOp, 6, 11, k_hfd_datatypes) +REGISTER_KERNEL(Min, MinOp, 13, *k_uzilhfd_datatypes) +REGISTER_VERSIONED_KERNEL(Min, MinOp, 12, 12, *k_uzilhfd_datatypes) +REGISTER_VERSIONED_KERNEL(Min, MinOp, 6, 11, *k_hfd_datatypes) -REGISTER_KERNEL(Max, MaxOp, 13, k_uzilhfd_datatypes) -REGISTER_VERSIONED_KERNEL(Max, MaxOp, 12, 12, k_uzilhfd_datatypes) -REGISTER_VERSIONED_KERNEL(Max, MaxOp, 6, 11, k_hfd_datatypes) +REGISTER_KERNEL(Max, MaxOp, 13, *k_uzilhfd_datatypes) +REGISTER_VERSIONED_KERNEL(Max, MaxOp, 12, 12, *k_uzilhfd_datatypes) +REGISTER_VERSIONED_KERNEL(Max, MaxOp, 6, 11, *k_hfd_datatypes) #undef REGISTER_VERSIONED_KERNEL #undef REGISTER_KERNEL diff --git a/onnxruntime/core/providers/cuda/nn/batch_norm.cc b/onnxruntime/core/providers/cuda/nn/batch_norm.cc index db312fdd2c..56d297429b 100644 --- a/onnxruntime/core/providers/cuda/nn/batch_norm.cc +++ b/onnxruntime/core/providers/cuda/nn/batch_norm.cc @@ -11,24 +11,24 @@ using namespace std; namespace onnxruntime { namespace cuda { -#define REGISTER_KERNEL_TYPED(T) \ - ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ - BatchNormalization, \ - kOnnxDomain, \ - 7, 8, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder() \ - .TypeConstraint("T", DataTypeImpl::GetTensorType()), \ - BatchNorm); \ - ONNX_OPERATOR_TYPED_KERNEL_EX( \ - BatchNormalization, \ - kOnnxDomain, \ - 9, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder() \ - .TypeConstraint("T", DataTypeImpl::GetTensorType()), \ +#define REGISTER_KERNEL_TYPED(T) \ + ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ + BatchNormalization, \ + kOnnxDomain, \ + 7, 8, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()) \ + .TypeConstraint("T", DataTypeImpl::GetTensorType()), \ + BatchNorm); \ + ONNX_OPERATOR_TYPED_KERNEL_EX( \ + BatchNormalization, \ + kOnnxDomain, \ + 9, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()) \ + .TypeConstraint("T", DataTypeImpl::GetTensorType()), \ BatchNorm); template diff --git a/onnxruntime/core/providers/cuda/nn/batch_norm.h b/onnxruntime/core/providers/cuda/nn/batch_norm.h index 57baae17a6..62241ef52d 100644 --- a/onnxruntime/core/providers/cuda/nn/batch_norm.h +++ b/onnxruntime/core/providers/cuda/nn/batch_norm.h @@ -3,7 +3,6 @@ #pragma once -#include "gsl/gsl" #include "core/providers/cuda/cuda_kernel.h" #include "core/providers/cuda/cudnn_common.h" diff --git a/onnxruntime/core/providers/cuda/nn/conv.cc b/onnxruntime/core/providers/cuda/nn/conv.cc index d159bd339a..644bbe4d2d 100644 --- a/onnxruntime/core/providers/cuda/nn/conv.cc +++ b/onnxruntime/core/providers/cuda/nn/conv.cc @@ -1,7 +1,6 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. -#include "core/providers/common.h" #include "core/providers/cuda/cuda_common.h" #include "core/providers/cuda/nn/conv.h" #include "core/providers/cuda/shared_inc/fpgeneric.h" @@ -12,22 +11,22 @@ namespace cuda { // Op Set 11 for Conv only update document to clearify default dilations and strides value. // which are already convered by op set 11 cpu versoin, so simply add declaration. -#define REGISTER_KERNEL_TYPED(T) \ - ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ - Conv, \ - kOnnxDomain, \ - 1, 10, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ - Conv); \ - ONNX_OPERATOR_TYPED_KERNEL_EX( \ - Conv, \ - kOnnxDomain, \ - 11, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ +#define REGISTER_KERNEL_TYPED(T) \ + ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ + Conv, \ + kOnnxDomain, \ + 1, 10, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ + Conv); \ + ONNX_OPERATOR_TYPED_KERNEL_EX( \ + Conv, \ + kOnnxDomain, \ + 11, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ Conv); REGISTER_KERNEL_TYPED(float) @@ -82,8 +81,7 @@ size_t getMaxWorkspaceSize(const CudnnConvState& } Status SliceOutUnwantedOutputSection(cudaStream_t stream, - const void* input_data, - const std::vector& input_dims, + const void* input_data, const std::vector& input_dims, void* output_data, const std::vector& output_dims, std::vector starts, diff --git a/onnxruntime/core/providers/cuda/nn/conv.h b/onnxruntime/core/providers/cuda/nn/conv.h index 8e54c1fde2..ab6647941b 100644 --- a/onnxruntime/core/providers/cuda/nn/conv.h +++ b/onnxruntime/core/providers/cuda/nn/conv.h @@ -3,7 +3,6 @@ #pragma once -#include "core/common/common.h" #include "core/platform/ort_mutex.h" #include "core/providers/cuda/cuda_kernel.h" #include "core/providers/cuda/cudnn_common.h" diff --git a/onnxruntime/core/providers/cuda/nn/conv_transpose.cc b/onnxruntime/core/providers/cuda/nn/conv_transpose.cc index 574fb0acd9..ad8f9a1362 100644 --- a/onnxruntime/core/providers/cuda/nn/conv_transpose.cc +++ b/onnxruntime/core/providers/cuda/nn/conv_transpose.cc @@ -8,22 +8,22 @@ namespace cuda { // Op Set 11 for ConvTranspose only update document to clearify default dilations and strides value. // which are already covered by op set 11 cpu version, so simply add declaration. -#define REGISTER_KERNEL_TYPED(T) \ - ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ - ConvTranspose, \ - kOnnxDomain, \ - 1, 10, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ - ConvTranspose); \ - ONNX_OPERATOR_TYPED_KERNEL_EX( \ - ConvTranspose, \ - kOnnxDomain, \ - 11, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ +#define REGISTER_KERNEL_TYPED(T) \ + ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ + ConvTranspose, \ + kOnnxDomain, \ + 1, 10, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ + ConvTranspose); \ + ONNX_OPERATOR_TYPED_KERNEL_EX( \ + ConvTranspose, \ + kOnnxDomain, \ + 11, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ ConvTranspose); REGISTER_KERNEL_TYPED(float) diff --git a/onnxruntime/core/providers/cuda/nn/conv_transpose.h b/onnxruntime/core/providers/cuda/nn/conv_transpose.h index 480b0c1678..165d548d27 100644 --- a/onnxruntime/core/providers/cuda/nn/conv_transpose.h +++ b/onnxruntime/core/providers/cuda/nn/conv_transpose.h @@ -3,6 +3,7 @@ #pragma once +#include "core/providers/cuda/cuda_common.h" #include "core/providers/cuda/cuda_kernel.h" #include "core/providers/cuda/cudnn_common.h" #include "core/providers/cuda/nn/conv.h" diff --git a/onnxruntime/core/providers/cuda/nn/dropout.cc b/onnxruntime/core/providers/cuda/nn/dropout.cc index 42ec6597ac..91d9560582 100644 --- a/onnxruntime/core/providers/cuda/nn/dropout.cc +++ b/onnxruntime/core/providers/cuda/nn/dropout.cc @@ -11,12 +11,12 @@ ONNX_OPERATOR_VERSIONED_KERNEL_EX( kOnnxDomain, 12, 12, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .TypeConstraint("T", DataTypeImpl::AllIEEEFloatTensorTypes()) .TypeConstraint("T1", DataTypeImpl::AllIEEEFloatTensorTypes()) .TypeConstraint("T2", DataTypeImpl::GetTensorType()) - .InputMemoryType(1) - .InputMemoryType(2), + .InputMemoryType(OrtMemTypeCPUInput, 1) + .InputMemoryType(OrtMemTypeCPUInput, 2), Dropout); ONNX_OPERATOR_KERNEL_EX( @@ -24,12 +24,12 @@ ONNX_OPERATOR_KERNEL_EX( kOnnxDomain, 13, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .TypeConstraint("T", ALL_IEEE_FLOAT_TENSOR_TYPES) .TypeConstraint("T1", ALL_IEEE_FLOAT_TENSOR_TYPES) .TypeConstraint("T2", DataTypeImpl::GetTensorType()) - .InputMemoryType(1) - .InputMemoryType(2), + .InputMemoryType(OrtMemTypeCPUInput, 1) + .InputMemoryType(OrtMemTypeCPUInput, 2), Dropout); Status Dropout::ComputeInternal(OpKernelContext* context) const { diff --git a/onnxruntime/core/providers/cuda/nn/instance_norm.cc b/onnxruntime/core/providers/cuda/nn/instance_norm.cc index 8945d639fc..3360db8f25 100644 --- a/onnxruntime/core/providers/cuda/nn/instance_norm.cc +++ b/onnxruntime/core/providers/cuda/nn/instance_norm.cc @@ -16,7 +16,7 @@ namespace cuda { 6, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()), \ InstanceNorm); diff --git a/onnxruntime/core/providers/cuda/nn/lrn.cc b/onnxruntime/core/providers/cuda/nn/lrn.cc index a481357dfe..33e1e15e6a 100644 --- a/onnxruntime/core/providers/cuda/nn/lrn.cc +++ b/onnxruntime/core/providers/cuda/nn/lrn.cc @@ -6,25 +6,25 @@ namespace onnxruntime { namespace cuda { -#define REGISTER_KERNEL_VERSIONED_TYPED(START_VER, END_VER, T) \ - ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ - LRN, \ - kOnnxDomain, \ - START_VER, \ - END_VER, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ +#define REGISTER_KERNEL_VERSIONED_TYPED(START_VER, END_VER, T) \ + ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ + LRN, \ + kOnnxDomain, \ + START_VER, \ + END_VER, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ LRN); -#define REGISTER_KERNEL_TYPED(VER, T) \ - ONNX_OPERATOR_TYPED_KERNEL_EX( \ - LRN, \ - kOnnxDomain, \ - VER, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ +#define REGISTER_KERNEL_TYPED(VER, T) \ + ONNX_OPERATOR_TYPED_KERNEL_EX( \ + LRN, \ + kOnnxDomain, \ + VER, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ LRN); REGISTER_KERNEL_VERSIONED_TYPED(1, 12, float) diff --git a/onnxruntime/core/providers/cuda/nn/pool.cc b/onnxruntime/core/providers/cuda/nn/pool.cc index af9eeb5381..b9ee3975e1 100644 --- a/onnxruntime/core/providers/cuda/nn/pool.cc +++ b/onnxruntime/core/providers/cuda/nn/pool.cc @@ -1,8 +1,8 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. +#include "core/providers/shared_library/provider_api.h" #include "core/providers/cuda/nn/pool.h" -#include "core/providers/common.h" #include "core/providers/cuda/cudnn_common.h" #include "core/providers/cuda/nn/max_pool_with_index.h" #include "core/providers/cuda/math/unary_elementwise_ops_impl.h" @@ -11,28 +11,27 @@ using namespace onnxruntime::common; namespace onnxruntime { namespace cuda { -#define POOLING_KERNEL(op_name, data_type, pool_type, since_version) \ - ONNX_OPERATOR_TYPED_KERNEL_EX( \ - op_name, \ - kOnnxDomain, \ - since_version, \ - data_type, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ +#define POOLING_KERNEL(op_name, data_type, pool_type, since_version) \ + ONNX_OPERATOR_TYPED_KERNEL_EX( \ + op_name, \ + kOnnxDomain, \ + since_version, \ + data_type, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ Pool); -#define POOLING_KERNEL_VERSIONED(op_name, data_type, pool_type, since_version, end_version) \ - ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ - op_name, \ - kOnnxDomain, \ - since_version, \ - end_version, \ - data_type, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()).TypeConstraint("I", DataTypeImpl::GetTensorType()), \ +#define POOLING_KERNEL_VERSIONED(op_name, data_type, pool_type, since_version, end_version) \ + ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ + op_name, \ + kOnnxDomain, \ + since_version, \ + end_version, \ + data_type, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()).TypeConstraint("I", DataTypeImpl::GetTensorType()), \ Pool); - POOLING_KERNEL_VERSIONED(AveragePool, float, AveragePool, 7, 9) POOLING_KERNEL_VERSIONED(AveragePool, double, AveragePool, 7, 9) POOLING_KERNEL_VERSIONED(AveragePool, MLFloat16, AveragePool, 7, 9) @@ -64,7 +63,6 @@ POOLING_KERNEL(MaxPool, MLFloat16, MaxPool<8>, 12) POOLING_KERNEL(MaxPool, int8_t, MaxPool<8>, 12) POOLING_KERNEL(MaxPool, uint8_t, MaxPool<8>, 12) - POOLING_KERNEL(GlobalMaxPool, float, MaxPool<1>, 1) POOLING_KERNEL(GlobalMaxPool, double, MaxPool<1>, 1) POOLING_KERNEL(GlobalMaxPool, MLFloat16, MaxPool<1>, 1) @@ -167,8 +165,8 @@ Status Pool::ComputeInternal(OpKernelContext* context) const { cudnnPoolingMode_t mode = CUDNN_POOLING_MAX; if (PoolType::type == onnxruntime::PoolType::kAveragePool) { - mode = pool_attrs_.count_include_pad ? CUDNN_POOLING_AVERAGE_COUNT_INCLUDE_PADDING - : CUDNN_POOLING_AVERAGE_COUNT_EXCLUDE_PADDING; + mode = pool_attrs_.count_include_pad ? CUDNN_POOLING_AVERAGE_COUNT_INCLUDE_PADDING + : CUDNN_POOLING_AVERAGE_COUNT_EXCLUDE_PADDING; } CudnnPoolingDescriptor pooling_desc; ORT_RETURN_IF_ERROR(pooling_desc.Set(mode, kernel_shape, pads, strides)); diff --git a/onnxruntime/core/providers/cuda/nn/pool.h b/onnxruntime/core/providers/cuda/nn/pool.h index 32969e8520..fb223c18d2 100644 --- a/onnxruntime/core/providers/cuda/nn/pool.h +++ b/onnxruntime/core/providers/cuda/nn/pool.h @@ -3,7 +3,6 @@ #pragma once -#include "core/common/common.h" #include "core/providers/cuda/cuda_kernel.h" #include "core/providers/cuda/cudnn_common.h" #include "core/providers/cpu/nn/pool_base.h" @@ -14,7 +13,7 @@ namespace cuda { template class Pool : public CudaKernel, public PoolBase { public: - Pool(OpKernelInfo info) : CudaKernel(info), PoolBase(info) {} + Pool(const OpKernelInfo& info) : CudaKernel(info), PoolBase(info) {} Status ComputeInternal(OpKernelContext* context) const override; }; @@ -22,7 +21,7 @@ class Pool : public CudaKernel, public PoolBase { template class Pool> final : public Pool> { public: - Pool(OpKernelInfo info) : Pool>(info) {} + Pool(const OpKernelInfo& info) : Pool>(info) {} Status ComputeInternal(OpKernelContext* context) const override; }; diff --git a/onnxruntime/core/providers/cuda/nn/shrink.cc b/onnxruntime/core/providers/cuda/nn/shrink.cc index cd8d9e2cf3..e526f58a46 100644 --- a/onnxruntime/core/providers/cuda/nn/shrink.cc +++ b/onnxruntime/core/providers/cuda/nn/shrink.cc @@ -16,7 +16,7 @@ namespace cuda { 9, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .MayInplace(0, 0) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()), \ Shrink); @@ -51,4 +51,4 @@ SHRINK_REGISTER_KERNEL(uint64_t) SHRINK_REGISTER_KERNEL(int64_t) } // namespace cuda -} // namespace onnxruntime \ No newline at end of file +} // namespace onnxruntime diff --git a/onnxruntime/core/providers/cuda/object_detection/non_max_suppression.cc b/onnxruntime/core/providers/cuda/object_detection/non_max_suppression.cc index 5c7545399c..9bf91e2e2b 100644 --- a/onnxruntime/core/providers/cuda/object_detection/non_max_suppression.cc +++ b/onnxruntime/core/providers/cuda/object_detection/non_max_suppression.cc @@ -14,10 +14,10 @@ ONNX_OPERATOR_VERSIONED_KERNEL_EX( kOnnxDomain, 10, 10, kCudaExecutionProvider, - KernelDefBuilder() - .InputMemoryType(2) - .InputMemoryType(3) - .InputMemoryType(4), + (*KernelDefBuilder::Create()) + .InputMemoryType(OrtMemTypeCPUInput, 2) + .InputMemoryType(OrtMemTypeCPUInput, 3) + .InputMemoryType(OrtMemTypeCPUInput, 4), NonMaxSuppression); ONNX_OPERATOR_KERNEL_EX( @@ -25,10 +25,10 @@ ONNX_OPERATOR_KERNEL_EX( kOnnxDomain, 11, kCudaExecutionProvider, - KernelDefBuilder() - .InputMemoryType(2) - .InputMemoryType(3) - .InputMemoryType(4), + (*KernelDefBuilder::Create()) + .InputMemoryType(OrtMemTypeCPUInput, 2) + .InputMemoryType(OrtMemTypeCPUInput, 3) + .InputMemoryType(OrtMemTypeCPUInput, 4), NonMaxSuppression); Status NonMaxSuppression::ComputeInternal(OpKernelContext* ctx) const { diff --git a/onnxruntime/core/providers/cuda/object_detection/non_max_suppression_impl.cu b/onnxruntime/core/providers/cuda/object_detection/non_max_suppression_impl.cu index ec98e00413..4aa98f03bd 100644 --- a/onnxruntime/core/providers/cuda/object_detection/non_max_suppression_impl.cu +++ b/onnxruntime/core/providers/cuda/object_detection/non_max_suppression_impl.cu @@ -22,7 +22,6 @@ limitations under the License. #include "core/providers/cuda/cu_inc/common.cuh" #include "core/providers/cuda/cuda_common.h" -#include "core/framework/tensor.h" #include //TODO:fix the warnings diff --git a/onnxruntime/core/providers/cuda/object_detection/non_max_suppression_impl.h b/onnxruntime/core/providers/cuda/object_detection/non_max_suppression_impl.h index 648420125c..aa1b78e53d 100644 --- a/onnxruntime/core/providers/cuda/object_detection/non_max_suppression_impl.h +++ b/onnxruntime/core/providers/cuda/object_detection/non_max_suppression_impl.h @@ -4,9 +4,6 @@ #pragma once #include #include "core/providers/cuda/shared_inc/cuda_utils.h" -#include "core/common/common.h" -#include "core/framework/allocator.h" -#include "core/framework/data_types.h" #include #include "core/providers/cpu/object_detection/non_max_suppression_helper.h" diff --git a/onnxruntime/core/providers/cuda/object_detection/roialign.cc b/onnxruntime/core/providers/cuda/object_detection/roialign.cc index 513b82f6aa..7bf97dc7b5 100644 --- a/onnxruntime/core/providers/cuda/object_detection/roialign.cc +++ b/onnxruntime/core/providers/cuda/object_detection/roialign.cc @@ -1,5 +1,5 @@ -// Copyright (c) Microsoft Corporation. All rights reserved. -// Licensed under the MIT License. +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. #include "roialign.h" #include "roialign_impl.h" @@ -14,8 +14,8 @@ namespace cuda { 10, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ - .TypeConstraint("T", DataTypeImpl::GetTensorType()) \ + (*KernelDefBuilder::Create()) \ + .TypeConstraint("T", DataTypeImpl::GetTensorType()) \ .TypeConstraint("T2", DataTypeImpl::GetTensorType()), \ RoiAlign); @@ -59,8 +59,7 @@ Status RoiAlign::ComputeInternal(OpKernelContext* context) const { num_roi_cols, reinterpret_cast::MappedType*>(Y.template MutableData()), this->mode_ == RoiAlignMode::avg, - batch_indices_ptr->template Data() - ); + batch_indices_ptr->template Data()); } return Status::OK(); diff --git a/onnxruntime/core/providers/cuda/object_detection/roialign_impl.h b/onnxruntime/core/providers/cuda/object_detection/roialign_impl.h index 312c35a93a..4ff9d4cb1d 100644 --- a/onnxruntime/core/providers/cuda/object_detection/roialign_impl.h +++ b/onnxruntime/core/providers/cuda/object_detection/roialign_impl.h @@ -4,8 +4,6 @@ #pragma once #include #include "core/providers/cuda/shared_inc/cuda_utils.h" -#include "core/framework/data_types.h" -#include "core/common/common.h" namespace onnxruntime { namespace cuda { diff --git a/onnxruntime/core/providers/cuda/reduction/reduction_functions.cc b/onnxruntime/core/providers/cuda/reduction/reduction_functions.cc index 6889fcfbb1..955df6d9a1 100644 --- a/onnxruntime/core/providers/cuda/reduction/reduction_functions.cc +++ b/onnxruntime/core/providers/cuda/reduction/reduction_functions.cc @@ -1,6 +1,7 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. +#include "core/providers/shared_library/provider_api.h" #include "core/providers/cuda/reduction/reduction_functions.h" #include @@ -10,7 +11,6 @@ #include "core/common/optional.h" #include "core/framework/tensor_shape.h" -#include "core/providers/common.h" namespace onnxruntime { namespace cuda { diff --git a/onnxruntime/core/providers/cuda/reduction/reduction_ops.cc b/onnxruntime/core/providers/cuda/reduction/reduction_ops.cc index fbb52db0f4..338442b887 100644 --- a/onnxruntime/core/providers/cuda/reduction/reduction_ops.cc +++ b/onnxruntime/core/providers/cuda/reduction/reduction_ops.cc @@ -1,12 +1,9 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. -#include "core/providers/cuda/reduction/reduction_ops.h" - -#include "core/framework/data_types_internal.h" -#include "core/framework/op_kernel_context_internal.h" -#include "core/providers/common.h" +#include "core/providers/shared_library/provider_api.h" #include "core/providers/cpu/tensor/utils.h" +#include "core/providers/cuda/reduction/reduction_ops.h" #include "core/providers/cuda/cudnn_common.h" #include "core/providers/cuda/math/binary_elementwise_ops_impl.h" #include "core/providers/cuda/math/binary_elementwise_ops.h" @@ -17,113 +14,113 @@ namespace onnxruntime { namespace cuda { // opset 11 explicitly added support for negative axis. implementation already allowed it. -#define REGISTER_KERNEL_TYPED(name, T) \ - ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ - name, \ - kOnnxDomain, \ - 1, 10, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ - name); \ - ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ - name, \ - kOnnxDomain, \ - 11, 12, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ - name); \ - ONNX_OPERATOR_TYPED_KERNEL_EX( \ - name, \ - kOnnxDomain, \ - 13, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ +#define REGISTER_KERNEL_TYPED(name, T) \ + ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ + name, \ + kOnnxDomain, \ + 1, 10, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ + name); \ + ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ + name, \ + kOnnxDomain, \ + 11, 12, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ + name); \ + ONNX_OPERATOR_TYPED_KERNEL_EX( \ + name, \ + kOnnxDomain, \ + 13, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ name); // Register those with changes in OpSet12. -#define REGISTER_KERNEL_TYPED_12(name, T) \ - ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ - name, \ - kOnnxDomain, \ - 1, 10, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ - name); \ - ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ - name, \ - kOnnxDomain, \ - 11, 11, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ - name); \ - ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ - name, \ - kOnnxDomain, \ - 12, 12, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ - name); \ - ONNX_OPERATOR_TYPED_KERNEL_EX( \ - name, \ - kOnnxDomain, \ - 13, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ +#define REGISTER_KERNEL_TYPED_12(name, T) \ + ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ + name, \ + kOnnxDomain, \ + 1, 10, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ + name); \ + ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ + name, \ + kOnnxDomain, \ + 11, 11, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ + name); \ + ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ + name, \ + kOnnxDomain, \ + 12, 12, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ + name); \ + ONNX_OPERATOR_TYPED_KERNEL_EX( \ + name, \ + kOnnxDomain, \ + 13, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ name); // CUDA ArgMax/ArgMin doesn't have OpSet12 implementation (with select_last_index attr), keep it in OpSet11 for now. -#define REGISTER_KERNEL_TYPED_11(name, T) \ - ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ - name, \ - kOnnxDomain, \ - 1, 10, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ - name); \ - ONNX_OPERATOR_TYPED_KERNEL_EX( \ - name, \ - kOnnxDomain, \ - 11, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ +#define REGISTER_KERNEL_TYPED_11(name, T) \ + ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ + name, \ + kOnnxDomain, \ + 1, 10, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ + name); \ + ONNX_OPERATOR_TYPED_KERNEL_EX( \ + name, \ + kOnnxDomain, \ + 11, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ name); // Register with the latest version 13 -#define REGISTER_KERNEL_TYPED_13(name, T) \ - ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ - name, \ - kOnnxDomain, \ - 1, 10, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ - name); \ - ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ - name, \ - kOnnxDomain, \ - 11, 12, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ - name); \ - ONNX_OPERATOR_TYPED_KERNEL_EX( \ - name, \ - kOnnxDomain, \ - 13, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder() \ - .InputMemoryType(1) \ - .TypeConstraint("T", DataTypeImpl::GetTensorType()), \ +#define REGISTER_KERNEL_TYPED_13(name, T) \ + ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ + name, \ + kOnnxDomain, \ + 1, 10, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ + name); \ + ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ + name, \ + kOnnxDomain, \ + 11, 12, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ + name); \ + ONNX_OPERATOR_TYPED_KERNEL_EX( \ + name, \ + kOnnxDomain, \ + 13, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()) \ + .InputMemoryType(OrtMemTypeCPUInput, 1) \ + .TypeConstraint("T", DataTypeImpl::GetTensorType()), \ name); // TODO ReduceKernel::ReduceKernelShared() is still used by some other training classes though it's not used here - this should be refactored. @@ -686,66 +683,66 @@ Status ReduceKernel::ComputeImpl(OpKernelContext* ctx, cudnnRe calculate_log_, calculate_sqt_, log_sum_exp_, fast_reduction); } -#define SPECIALIZED_REDUCEKERNEL_COMPUTEIMPL(T) \ - template <> \ - template <> \ - Status ReduceKernel::ComputeImpl( \ - OpKernelContext * ctx, cudnnReduceTensorOp_t cudnn_reduce_op) const { \ - typedef typename ToCudaType::MappedType CudaT; \ - const Tensor* X = ctx->Input(0); \ - std::vector axes; \ - size_t num_inputs = ctx->InputCount(); \ - if (num_inputs == 2) { \ - const Tensor* axes_tensor = ctx->Input(1); \ - ORT_ENFORCE(axes_tensor != nullptr, "Axes input is null"); \ - ORT_ENFORCE(axes_tensor->Shape().NumDimensions() == 1, "An axes tensor must be a vector tensor."); \ - auto nDims = static_cast(axes_tensor->Shape()[0]); \ - const auto* data = axes_tensor->template Data(); \ - axes.assign(data, data + nDims); \ - } else { \ - axes.assign(axes_.begin(), axes_.end()); \ - } \ - \ - if (axes.empty() && noop_with_empty_axes_) { \ - auto* Y = ctx->Output(0, X->Shape()); \ - CUDA_RETURN_IF_ERROR(cudaMemcpyAsync(Y->template MutableData(), X->template Data(), X->SizeInBytes(), \ - cudaMemcpyDeviceToDevice, Stream())); \ - return Status::OK(); \ - } \ - \ - PrepareReduceMetadata prepare_reduce_metadata; \ - ORT_RETURN_IF_ERROR(PrepareForReduce(X, keepdims_, axes, prepare_reduce_metadata)); \ - \ - Tensor* Y = ctx->Output(0, prepare_reduce_metadata.squeezed_output_dims); \ - \ - int64_t input_count = prepare_reduce_metadata.input_count; \ - int64_t output_count = prepare_reduce_metadata.output_count; \ - std::vector& input_dims_cudnn = prepare_reduce_metadata.input_dims_cudnn; \ - std::vector& output_dims_cudnn = prepare_reduce_metadata.output_dims_cudnn; \ - \ - if (input_count == 0) { \ - assert(Y->Shape().Size() == 0); \ - return Status::OK(); \ - } \ - \ - if (input_count == output_count) { \ - if (Y->template MutableData() != X->template Data()) { \ - CUDA_RETURN_IF_ERROR(cudaMemcpyAsync(Y->template MutableData(), X->template Data(), \ - input_count * sizeof(T), cudaMemcpyDeviceToDevice, Stream())); \ - } \ - return Status::OK(); \ - } \ - \ - CUDA_RETURN_IF_ERROR(cudaMemsetAsync(Y->MutableDataRaw(), 0, Y->SizeInBytes(), Stream())); \ - \ - size_t indices_bytes = 0; \ - size_t workspace_bytes = 0; \ - CudnnTensor input_tensor; \ - CudnnTensor output_tensor; \ - CudnnReduceDescriptor reduce_desc; \ - \ - cudnnDataType_t cudnn_type_X = CUDNN_DATA_FLOAT; \ - IAllocatorUniquePtr temp_X = GetScratchBuffer(input_count); \ +#define SPECIALIZED_REDUCEKERNEL_COMPUTEIMPL(T) \ + template <> \ + template <> \ + Status ReduceKernel::ComputeImpl( \ + OpKernelContext * ctx, cudnnReduceTensorOp_t cudnn_reduce_op) const { \ + typedef typename ToCudaType::MappedType CudaT; \ + const Tensor* X = ctx->Input(0); \ + std::vector axes; \ + size_t num_inputs = ctx->InputCount(); \ + if (num_inputs == 2) { \ + const Tensor* axes_tensor = ctx->Input(1); \ + ORT_ENFORCE(axes_tensor != nullptr, "Axes input is null"); \ + ORT_ENFORCE(axes_tensor->Shape().NumDimensions() == 1, "An axes tensor must be a vector tensor."); \ + auto nDims = static_cast(axes_tensor->Shape()[0]); \ + const auto* data = axes_tensor->template Data(); \ + axes.assign(data, data + nDims); \ + } else { \ + axes.assign(axes_.begin(), axes_.end()); \ + } \ + \ + if (axes.empty() && noop_with_empty_axes_) { \ + auto* Y = ctx->Output(0, X->Shape()); \ + CUDA_RETURN_IF_ERROR(cudaMemcpyAsync(Y->template MutableData(), X->template Data(), X->SizeInBytes(), \ + cudaMemcpyDeviceToDevice, Stream())); \ + return Status::OK(); \ + } \ + \ + PrepareReduceMetadata prepare_reduce_metadata; \ + ORT_RETURN_IF_ERROR(PrepareForReduce(X, keepdims_, axes, prepare_reduce_metadata)); \ + \ + Tensor* Y = ctx->Output(0, prepare_reduce_metadata.squeezed_output_dims); \ + \ + int64_t input_count = prepare_reduce_metadata.input_count; \ + int64_t output_count = prepare_reduce_metadata.output_count; \ + std::vector& input_dims_cudnn = prepare_reduce_metadata.input_dims_cudnn; \ + std::vector& output_dims_cudnn = prepare_reduce_metadata.output_dims_cudnn; \ + \ + if (input_count == 0) { \ + assert(Y->Shape().Size() == 0); \ + return Status::OK(); \ + } \ + \ + if (input_count == output_count) { \ + if (Y->template MutableData() != X->template Data()) { \ + CUDA_RETURN_IF_ERROR(cudaMemcpyAsync(Y->template MutableData(), X->template Data(), \ + input_count * sizeof(T), cudaMemcpyDeviceToDevice, Stream())); \ + } \ + return Status::OK(); \ + } \ + \ + CUDA_RETURN_IF_ERROR(cudaMemsetAsync(Y->MutableDataRaw(), 0, Y->SizeInBytes(), Stream())); \ + \ + size_t indices_bytes = 0; \ + size_t workspace_bytes = 0; \ + CudnnTensor input_tensor; \ + CudnnTensor output_tensor; \ + CudnnReduceDescriptor reduce_desc; \ + \ + cudnnDataType_t cudnn_type_X = CUDNN_DATA_FLOAT; \ + IAllocatorUniquePtr temp_X = GetScratchBuffer(input_count); \ Impl_Cast(Stream(), reinterpret_cast(X->template Data()), temp_X.get(), X->Shape().Size()); \ \ ORT_RETURN_IF_ERROR(reduce_desc.Set(cudnn_reduce_op, cudnn_type_X, CUDNN_REDUCE_TENSOR_NO_INDICES)); \ @@ -766,8 +763,8 @@ Status ReduceKernel::ComputeImpl(OpKernelContext* ctx, cudnnRe &zero, output_tensor, temp_Y.get())); \ \ Impl_Cast(Stream(), temp_Y.get(), reinterpret_cast(Y->template MutableData()), output_count); \ - \ - return Status::OK(); \ + \ + return Status::OK(); \ } SPECIALIZED_REDUCEKERNEL_COMPUTEIMPL(int32_t) @@ -885,10 +882,10 @@ Status ReduceKernel::ComputeImpl namespace ReductionOps { template -Tensor ReduceCompute(CUDAExecutionProvider& cuda_ep, cudnnReduceTensorOp_t cudnn_reduce_op, AllocatorPtr allocator, - const Tensor& input, const std::vector& axes, - bool keep_dims, bool calculate_log, bool calculate_sqt, bool log_sum_exp, - bool fast_reduction, const TensorShape* input_shape_override) { +std::unique_ptr ReduceCompute(CUDAExecutionProvider& cuda_ep, cudnnReduceTensorOp_t cudnn_reduce_op, AllocatorPtr allocator, + const Tensor& input, const std::vector& axes, + bool keep_dims, bool calculate_log, bool calculate_sqt, bool log_sum_exp, + bool fast_reduction, const TensorShape* input_shape_override) { PrepareReduceMetadata prepare_reduce_metadata; auto status = PrepareForReduce(&input, keep_dims, @@ -900,9 +897,9 @@ Tensor ReduceCompute(CUDAExecutionProvider& cuda_ep, cudnnReduceTensorOp_t cudnn ORT_THROW(ONNXRUNTIME, FAIL, "Failed to perform reduce op: ", status.ErrorMessage()); } - Tensor output(input.DataType(), prepare_reduce_metadata.squeezed_output_dims, allocator); + auto output = Tensor::Create(input.DataType(), prepare_reduce_metadata.squeezed_output_dims, allocator); - status = ReduceComputeCore(cuda_ep, input, prepare_reduce_metadata, output, cudnn_reduce_op, axes, + status = ReduceComputeCore(cuda_ep, input, prepare_reduce_metadata, *output, cudnn_reduce_op, axes, calculate_log, calculate_sqt, log_sum_exp, fast_reduction, input_shape_override); if (!status.IsOK()) { @@ -914,21 +911,21 @@ Tensor ReduceCompute(CUDAExecutionProvider& cuda_ep, cudnnReduceTensorOp_t cudnn // Explicit template instantiation (needed to be used in einsum_auxiliary_ops.cc) -template Tensor ReduceCompute( +template std::unique_ptr ReduceCompute( CUDAExecutionProvider& cuda_ep, cudnnReduceTensorOp_t cudnn_reduce_op, AllocatorPtr allocator, const Tensor& input, const std::vector& axes, bool keep_dims, bool calculate_log, bool calculate_sqt, bool log_sum_exp, bool fast_reduction, const TensorShape* input_shape_override); -template Tensor ReduceCompute( +template std::unique_ptr ReduceCompute( CUDAExecutionProvider& cuda_ep, cudnnReduceTensorOp_t cudnn_reduce_op, AllocatorPtr allocator, const Tensor& input, const std::vector& axes, bool keep_dims, bool calculate_log, bool calculate_sqt, bool log_sum_exp, bool fast_reduction, const TensorShape* input_shape_override); -template Tensor ReduceCompute( +template std::unique_ptr ReduceCompute( CUDAExecutionProvider& cuda_ep, cudnnReduceTensorOp_t cudnn_reduce_op, AllocatorPtr allocator, const Tensor& input, const std::vector& axes, diff --git a/onnxruntime/core/providers/cuda/reduction/reduction_ops.h b/onnxruntime/core/providers/cuda/reduction/reduction_ops.h index 64e501660f..61a71f5e07 100644 --- a/onnxruntime/core/providers/cuda/reduction/reduction_ops.h +++ b/onnxruntime/core/providers/cuda/reduction/reduction_ops.h @@ -2,7 +2,6 @@ // Licensed under the MIT License. #pragma once -#include "core/common/optional.h" #include "core/providers/cuda/cuda_kernel.h" #include "core/providers/cpu/reduction/reduction_ops.h" #include "core/providers/cuda/reduction/reduction_functions.h" @@ -16,10 +15,10 @@ namespace ReductionOps { // `input_shape_override` is the input shape for compute purposes (if provided) template -Tensor ReduceCompute(CUDAExecutionProvider& cuda_ep, cudnnReduceTensorOp_t cudnn_reduce_op, AllocatorPtr allocator, - const Tensor& input, const std::vector& axes, - bool keep_dims, bool calculate_log, bool calculate_sqt, bool log_sum_exp, - bool fast_reduction, const TensorShape* input_shape_override = nullptr); +std::unique_ptr ReduceCompute(CUDAExecutionProvider& cuda_ep, cudnnReduceTensorOp_t cudnn_reduce_op, AllocatorPtr allocator, + const Tensor& input, const std::vector& axes, + bool keep_dims, bool calculate_log, bool calculate_sqt, bool log_sum_exp, + bool fast_reduction, const TensorShape* input_shape_override = nullptr); } // namespace ReductionOps diff --git a/onnxruntime/core/providers/cuda/rnn/cudnn_rnn_base.cc b/onnxruntime/core/providers/cuda/rnn/cudnn_rnn_base.cc index 7f73a142f3..1dcbebf695 100644 --- a/onnxruntime/core/providers/cuda/rnn/cudnn_rnn_base.cc +++ b/onnxruntime/core/providers/cuda/rnn/cudnn_rnn_base.cc @@ -1,9 +1,9 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. +#include "core/providers/shared_library/provider_api.h" #include "cudnn_rnn_base.h" #include "rnn_impl.h" -#include "core/providers/cpu/rnn/rnn_helpers.h" namespace onnxruntime { namespace cuda { @@ -102,7 +102,6 @@ Status CudnnRnnBase::ReorganizeWeights(const Tensor* W, const Tensor* R, cons // TODO! refine allocation size for each case. cudaMemset(reorganized_w_data.get(), 0, w_size * sizeof(T)); - const T* W_data = W->template Data(); const T* R_data = R->template Data(); const T* B_data = B == nullptr ? nullptr : B->template Data(); diff --git a/onnxruntime/core/providers/cuda/rnn/gru.cc b/onnxruntime/core/providers/cuda/rnn/gru.cc index e75aa99a87..964aebf560 100644 --- a/onnxruntime/core/providers/cuda/rnn/gru.cc +++ b/onnxruntime/core/providers/cuda/rnn/gru.cc @@ -1,11 +1,10 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. +#include "core/providers/shared_library/provider_api.h" #include "gru.h" #include "rnn_impl.h" -#include "core/providers/common.h" #include "core/providers/cuda/cudnn_common.h" -#include "core/providers/cpu/math/gemm_helper.h" namespace onnxruntime { namespace cuda { @@ -18,10 +17,10 @@ namespace cuda { 13, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()) \ .TypeConstraint("T1", DataTypeImpl::GetTensorType()) \ - .InputMemoryType(RNN_Input_Index::sequence_lens), \ + .InputMemoryType(OrtMemTypeCPUInput, RNN_Input_Index::sequence_lens), \ GRU); #define REGISTER_KERNEL_TYPED(T) \ @@ -31,10 +30,10 @@ namespace cuda { 14, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()) \ .TypeConstraint("T1", DataTypeImpl::GetTensorType()) \ - .InputMemoryType(RNN_Input_Index::sequence_lens), \ + .InputMemoryType(OrtMemTypeCPUInput, RNN_Input_Index::sequence_lens), \ GRU); REGISTER_KERNEL_VERSIONED_TYPED(float); diff --git a/onnxruntime/core/providers/cuda/rnn/lstm.cc b/onnxruntime/core/providers/cuda/rnn/lstm.cc index 87c8f13a7c..890d15cef6 100644 --- a/onnxruntime/core/providers/cuda/rnn/lstm.cc +++ b/onnxruntime/core/providers/cuda/rnn/lstm.cc @@ -1,9 +1,8 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. +#include "core/providers/shared_library/provider_api.h" #include "lstm.h" -#include "core/providers/cpu/rnn/rnn_helpers.h" -#include "core/providers/common.h" namespace onnxruntime { namespace cuda { @@ -16,10 +15,10 @@ namespace cuda { 13, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()) \ .TypeConstraint("T1", DataTypeImpl::GetTensorType()) \ - .InputMemoryType(RNN_Input_Index::sequence_lens), \ + .InputMemoryType(OrtMemTypeCPUInput, RNN_Input_Index::sequence_lens), \ LSTM); #define REGISTER_KERNEL_TYPED(T) \ @@ -29,10 +28,10 @@ namespace cuda { 14, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()) \ .TypeConstraint("T1", DataTypeImpl::GetTensorType()) \ - .InputMemoryType(RNN_Input_Index::sequence_lens), \ + .InputMemoryType(OrtMemTypeCPUInput, RNN_Input_Index::sequence_lens), \ LSTM); REGISTER_KERNEL_VERSIONED_TYPED(float); diff --git a/onnxruntime/core/providers/cuda/rnn/rnn.cc b/onnxruntime/core/providers/cuda/rnn/rnn.cc index 21d9a3b219..4bd22340ef 100644 --- a/onnxruntime/core/providers/cuda/rnn/rnn.cc +++ b/onnxruntime/core/providers/cuda/rnn/rnn.cc @@ -1,11 +1,10 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. +#include "core/providers/shared_library/provider_api.h" #include "rnn.h" #include "rnn_impl.h" -#include "core/providers/common.h" #include "core/providers/cuda/cudnn_common.h" -#include "core/providers/cpu/math/gemm_helper.h" namespace onnxruntime { namespace cuda { @@ -18,10 +17,10 @@ namespace cuda { 13, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()) \ .TypeConstraint("T1", DataTypeImpl::GetTensorType()) \ - .InputMemoryType(RNN_Input_Index::sequence_lens), \ + .InputMemoryType(OrtMemTypeCPUInput, RNN_Input_Index::sequence_lens), \ RNN); #define REGISTER_KERNEL_TYPED(T) \ @@ -31,10 +30,10 @@ namespace cuda { 14, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()) \ .TypeConstraint("T1", DataTypeImpl::GetTensorType()) \ - .InputMemoryType(RNN_Input_Index::sequence_lens), \ + .InputMemoryType(OrtMemTypeCPUInput, RNN_Input_Index::sequence_lens), \ RNN); REGISTER_KERNEL_VERSIONED_TYPED(float); diff --git a/onnxruntime/core/providers/cuda/shared_inc/cuda_utils.h b/onnxruntime/core/providers/cuda/shared_inc/cuda_utils.h index 244fe0a5c7..1c4b497e88 100644 --- a/onnxruntime/core/providers/cuda/shared_inc/cuda_utils.h +++ b/onnxruntime/core/providers/cuda/shared_inc/cuda_utils.h @@ -10,7 +10,6 @@ #include #include -#include "core/common/common.h" #include "core/providers/cuda/shared_inc/fast_divmod.h" namespace onnxruntime { diff --git a/onnxruntime/core/providers/cuda/shared_inc/fast_divmod.h b/onnxruntime/core/providers/cuda/shared_inc/fast_divmod.h index 95e269743d..55a4daa66b 100644 --- a/onnxruntime/core/providers/cuda/shared_inc/fast_divmod.h +++ b/onnxruntime/core/providers/cuda/shared_inc/fast_divmod.h @@ -23,7 +23,8 @@ struct fast_divmod { d_ = d == 0 ? 1 : d; ORT_ENFORCE(d_ >= 1 && d_ <= static_cast(std::numeric_limits::max())); - for (l_ = 0; l_ < 32; l_++) if ((1U << l_) >= d_) break; + for (l_ = 0; l_ < 32; l_++) + if ((1U << l_) >= d_) break; uint64_t one = 1; uint64_t m = ((one << 32) * ((one << l_) - d_)) / d_ + 1; @@ -38,7 +39,7 @@ struct fast_divmod { return (t + n) >> l_; #else // Using uint64_t for t, then t + n won't overflow. - uint64_t t = ((uint64_t) M_ * n) >> 32; + uint64_t t = ((uint64_t)M_ * n) >> 32; return static_cast((t + n) >> l_); #endif } diff --git a/onnxruntime/core/providers/cuda/symbols.def b/onnxruntime/core/providers/cuda/symbols.def new file mode 100644 index 0000000000..4ec2f7914c --- /dev/null +++ b/onnxruntime/core/providers/cuda/symbols.def @@ -0,0 +1,2 @@ +EXPORTS + GetProvider diff --git a/onnxruntime/core/providers/cuda/tensor/cast_op.cc b/onnxruntime/core/providers/cuda/tensor/cast_op.cc index fe5146c17c..27f1363f77 100644 --- a/onnxruntime/core/providers/cuda/tensor/cast_op.cc +++ b/onnxruntime/core/providers/cuda/tensor/cast_op.cc @@ -9,22 +9,23 @@ using namespace onnxruntime::common; namespace onnxruntime { namespace cuda { -const std::vector castOpTypeConstraints{ - DataTypeImpl::GetTensorType(), +const DeleteOnUnloadPtr> castOpTypeConstraints = new std::vector { + DataTypeImpl::GetTensorType(), #if defined(CUDA_VERSION) && CUDA_VERSION >= 11000 - DataTypeImpl::GetTensorType(), + DataTypeImpl::GetTensorType(), #endif - DataTypeImpl::GetTensorType(), - DataTypeImpl::GetTensorType(), - DataTypeImpl::GetTensorType(), - DataTypeImpl::GetTensorType(), - DataTypeImpl::GetTensorType(), - DataTypeImpl::GetTensorType(), - DataTypeImpl::GetTensorType(), - DataTypeImpl::GetTensorType(), - DataTypeImpl::GetTensorType(), - DataTypeImpl::GetTensorType(), - DataTypeImpl::GetTensorType()}; + DataTypeImpl::GetTensorType(), + DataTypeImpl::GetTensorType(), + DataTypeImpl::GetTensorType(), + DataTypeImpl::GetTensorType(), + DataTypeImpl::GetTensorType(), + DataTypeImpl::GetTensorType(), + DataTypeImpl::GetTensorType(), + DataTypeImpl::GetTensorType(), + DataTypeImpl::GetTensorType(), + DataTypeImpl::GetTensorType(), + DataTypeImpl::GetTensorType() +}; #define REGISTER_KERNEL_TYPED(T) \ ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ @@ -33,29 +34,29 @@ const std::vector castOpTypeConstraints{ 6, 8, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T1", DataTypeImpl::GetTensorType()) \ - .TypeConstraint("T2", castOpTypeConstraints), \ + .TypeConstraint("T2", *castOpTypeConstraints), \ Cast); \ ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ Cast, \ kOnnxDomain, \ - 9, 12, \ + 9, 12, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T1", DataTypeImpl::GetTensorType()) \ - .TypeConstraint("T2", castOpTypeConstraints), \ + .TypeConstraint("T2", *castOpTypeConstraints), \ Cast); \ ONNX_OPERATOR_TYPED_KERNEL_EX( \ Cast, \ kOnnxDomain, \ - 13, \ + 13, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T1", DataTypeImpl::GetTensorType()) \ - .TypeConstraint("T2", castOpTypeConstraints), \ + .TypeConstraint("T2", *castOpTypeConstraints), \ Cast); template diff --git a/onnxruntime/core/providers/cuda/tensor/cast_op.h b/onnxruntime/core/providers/cuda/tensor/cast_op.h index 9b22b20643..1d2d723da8 100644 --- a/onnxruntime/core/providers/cuda/tensor/cast_op.h +++ b/onnxruntime/core/providers/cuda/tensor/cast_op.h @@ -3,6 +3,7 @@ #pragma once +#include "core/providers/shared_library/provider_api.h" #include "core/providers/cuda/cuda_kernel.h" namespace onnxruntime { diff --git a/onnxruntime/core/providers/cuda/tensor/compress.cc b/onnxruntime/core/providers/cuda/tensor/compress.cc index 91dd3f5222..5c9537fdce 100644 --- a/onnxruntime/core/providers/cuda/tensor/compress.cc +++ b/onnxruntime/core/providers/cuda/tensor/compress.cc @@ -2,8 +2,6 @@ // Licensed under the MIT License. #include "compress.h" -#include "core/providers/cpu/tensor/utils.h" -#include "core/providers/common.h" #include "compress_impl.h" namespace onnxruntime { @@ -14,7 +12,7 @@ ONNX_OPERATOR_VERSIONED_KERNEL_EX( kOnnxDomain, 9, 10, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()) .TypeConstraint("T1", DataTypeImpl::GetTensorType()), Compress); @@ -25,7 +23,7 @@ ONNX_OPERATOR_KERNEL_EX( kOnnxDomain, 11, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()) .TypeConstraint("T1", DataTypeImpl::GetTensorType()), Compress); diff --git a/onnxruntime/core/providers/cuda/tensor/compress.h b/onnxruntime/core/providers/cuda/tensor/compress.h index 9a0e104c26..aa50836afc 100644 --- a/onnxruntime/core/providers/cuda/tensor/compress.h +++ b/onnxruntime/core/providers/cuda/tensor/compress.h @@ -1,6 +1,7 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. +#include "core/providers/shared_library/provider_api.h" #include "core/common/common.h" #include "core/providers/cuda/cuda_kernel.h" diff --git a/onnxruntime/core/providers/cuda/tensor/concat.cc b/onnxruntime/core/providers/cuda/tensor/concat.cc index dfeace4b7e..486c97710b 100644 --- a/onnxruntime/core/providers/cuda/tensor/concat.cc +++ b/onnxruntime/core/providers/cuda/tensor/concat.cc @@ -10,7 +10,7 @@ ONNX_OPERATOR_VERSIONED_KERNEL_EX(Concat, kOnnxDomain, 4, 10, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()), Concat); @@ -19,7 +19,7 @@ ONNX_OPERATOR_VERSIONED_KERNEL_EX(Concat, kOnnxDomain, 11, 12, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()), Concat); @@ -27,7 +27,7 @@ ONNX_OPERATOR_KERNEL_EX(Concat, kOnnxDomain, 13, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()), Concat); diff --git a/onnxruntime/core/providers/cuda/tensor/concat.h b/onnxruntime/core/providers/cuda/tensor/concat.h index 3d9aa70113..d880b54ee0 100644 --- a/onnxruntime/core/providers/cuda/tensor/concat.h +++ b/onnxruntime/core/providers/cuda/tensor/concat.h @@ -1,9 +1,10 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. +#include "core/providers/shared_library/provider_api.h" #include "core/common/common.h" #include "core/providers/cuda/cuda_kernel.h" -#include "core/providers/cpu/tensor/concat.h" +#include "core/providers/cpu/tensor/concatbase.h" namespace onnxruntime { namespace cuda { diff --git a/onnxruntime/core/providers/cuda/tensor/expand.cc b/onnxruntime/core/providers/cuda/tensor/expand.cc index 8ee8e3df91..4d58e4fb40 100644 --- a/onnxruntime/core/providers/cuda/tensor/expand.cc +++ b/onnxruntime/core/providers/cuda/tensor/expand.cc @@ -22,8 +22,7 @@ static void CalcEffectiveDims(vector& x_dims, vector& y_dims) if (xdim == ydim || xdim == 1) { x_reverse.push_back(xdim); y_reverse.push_back(ydim); - } - else { // xdim < ydim && xdim > 1, split + } else { // xdim < ydim && xdim > 1, split ydim /= xdim; x_reverse.push_back(xdim); y_reverse.push_back(xdim); @@ -44,18 +43,15 @@ static void CalcEffectiveDims(vector& x_dims, vector& y_dims) } if (x_dims.back() == 1) { y_dims.back() *= y_reverse[i]; - } - else { + } else { x_dims.push_back(1); y_dims.push_back(y_reverse[i]); } - } - else { // x_reverse[i] == y_reverse[i] + } else { // x_reverse[i] == y_reverse[i] if (x_dims.back() == y_dims.back()) { x_dims.back() *= x_reverse[i]; y_dims.back() *= y_reverse[i]; - } - else { + } else { x_dims.push_back(x_reverse[i]); y_dims.push_back(y_reverse[i]); } @@ -108,15 +104,14 @@ Status Expand::ComputeInternal(OpKernelContext* ctx) const { input_strides); } - ONNX_OPERATOR_VERSIONED_KERNEL_EX( Expand, kOnnxDomain, 8, 12, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()) - .InputMemoryType(1), + .InputMemoryType(OrtMemTypeCPUInput, 1), Expand); ONNX_OPERATOR_KERNEL_EX( @@ -124,9 +119,9 @@ ONNX_OPERATOR_KERNEL_EX( kOnnxDomain, 13, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()) - .InputMemoryType(1), + .InputMemoryType(OrtMemTypeCPUInput, 1), Expand); } // namespace cuda diff --git a/onnxruntime/core/providers/cuda/tensor/expand.h b/onnxruntime/core/providers/cuda/tensor/expand.h index 81bc7c3d43..4cf4c14e61 100644 --- a/onnxruntime/core/providers/cuda/tensor/expand.h +++ b/onnxruntime/core/providers/cuda/tensor/expand.h @@ -1,7 +1,7 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. -#include "core/common/common.h" +#include "core/providers/shared_library/provider_api.h" #include "core/providers/cuda/cuda_kernel.h" namespace onnxruntime { diff --git a/onnxruntime/core/providers/cuda/tensor/expand_impl.h b/onnxruntime/core/providers/cuda/tensor/expand_impl.h index e64c601323..f5bae955a9 100644 --- a/onnxruntime/core/providers/cuda/tensor/expand_impl.h +++ b/onnxruntime/core/providers/cuda/tensor/expand_impl.h @@ -4,8 +4,6 @@ #pragma once #include #include "core/providers/cuda/shared_inc/cuda_utils.h" -#include "core/framework/data_types.h" -#include "core/common/common.h" #include "core/providers/cuda/cuda_common.h" namespace onnxruntime { @@ -21,6 +19,5 @@ Status ExpandImpl( const TArray& output_strides, const TArray& input_strides); - } // namespace cuda } // namespace onnxruntime diff --git a/onnxruntime/core/providers/cuda/tensor/eye_like.cc b/onnxruntime/core/providers/cuda/tensor/eye_like.cc index 82ea145da9..95fed4c601 100644 --- a/onnxruntime/core/providers/cuda/tensor/eye_like.cc +++ b/onnxruntime/core/providers/cuda/tensor/eye_like.cc @@ -4,7 +4,6 @@ #include "eye_like.h" #include "eye_like_impl.h" #include "core/providers/cpu/tensor/utils.h" -#include "core/framework/tensorprotoutils.h" #include "core/providers/cuda/shared_inc/fast_divmod.h" using namespace onnxruntime::common; @@ -17,20 +16,17 @@ ONNX_OPERATOR_KERNEL_EX( kOnnxDomain, 9, kCudaExecutionProvider, - KernelDefBuilder().TypeConstraint("T1", - std::vector{ - DataTypeImpl::GetTensorType(), - DataTypeImpl::GetTensorType(), - DataTypeImpl::GetTensorType(), - DataTypeImpl::GetTensorType(), - DataTypeImpl::GetTensorType()}) - .TypeConstraint("T2", - std::vector{ - DataTypeImpl::GetTensorType(), - DataTypeImpl::GetTensorType(), - DataTypeImpl::GetTensorType(), - DataTypeImpl::GetTensorType(), - DataTypeImpl::GetTensorType()}), + (*KernelDefBuilder::Create()) + .TypeConstraint("T1", std::vector{DataTypeImpl::GetTensorType(), + DataTypeImpl::GetTensorType(), + DataTypeImpl::GetTensorType(), + DataTypeImpl::GetTensorType(), + DataTypeImpl::GetTensorType()}) + .TypeConstraint("T2", std::vector{DataTypeImpl::GetTensorType(), + DataTypeImpl::GetTensorType(), + DataTypeImpl::GetTensorType(), + DataTypeImpl::GetTensorType(), + DataTypeImpl::GetTensorType()}), EyeLike); #define TYPED_FUNCTION_CALL(T) \ @@ -71,7 +67,7 @@ Status EyeLike::ComputeInternal(OpKernelContext* context) const { diag_count = std::min(dim0 + k_, dim1); } - auto output_tensor_dtype = has_dtype_ ? static_cast(dtype_) : T1->GetElementType(); + auto output_tensor_dtype = has_dtype_ ? static_cast(dtype_) : T1->GetElementType(); switch (output_tensor_dtype) { case ONNX_NAMESPACE::TensorProto_DataType_FLOAT: TYPED_FUNCTION_CALL(float) diff --git a/onnxruntime/core/providers/cuda/tensor/eye_like.h b/onnxruntime/core/providers/cuda/tensor/eye_like.h index da0a5a5eac..7abb1cb889 100644 --- a/onnxruntime/core/providers/cuda/tensor/eye_like.h +++ b/onnxruntime/core/providers/cuda/tensor/eye_like.h @@ -1,7 +1,7 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. -#include "core/common/common.h" +#include "core/providers/shared_library/provider_api.h" #include "core/providers/cuda/cuda_kernel.h" namespace onnxruntime { diff --git a/onnxruntime/core/providers/cuda/tensor/flatten.cc b/onnxruntime/core/providers/cuda/tensor/flatten.cc index 7f36a5fdb0..1d19a7fdf9 100644 --- a/onnxruntime/core/providers/cuda/tensor/flatten.cc +++ b/onnxruntime/core/providers/cuda/tensor/flatten.cc @@ -2,7 +2,6 @@ // Licensed under the MIT License. #include "flatten.h" -#include "core/providers/common.h" namespace onnxruntime { namespace cuda { @@ -12,7 +11,7 @@ ONNX_OPERATOR_VERSIONED_KERNEL_EX( kOnnxDomain, 1, 8, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .Alias(0, 0) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()), Flatten); @@ -22,7 +21,7 @@ ONNX_OPERATOR_VERSIONED_KERNEL_EX( kOnnxDomain, 9, 10, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .Alias(0, 0) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()), Flatten); @@ -33,7 +32,7 @@ ONNX_OPERATOR_VERSIONED_KERNEL_EX( kOnnxDomain, 11, 12, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .Alias(0, 0) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()), Flatten); @@ -43,7 +42,7 @@ ONNX_OPERATOR_KERNEL_EX( kOnnxDomain, 13, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .Alias(0, 0) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()), Flatten); diff --git a/onnxruntime/core/providers/cuda/tensor/flatten.h b/onnxruntime/core/providers/cuda/tensor/flatten.h index 2b66347e28..d83929ed7c 100644 --- a/onnxruntime/core/providers/cuda/tensor/flatten.h +++ b/onnxruntime/core/providers/cuda/tensor/flatten.h @@ -1,7 +1,7 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. -#include "core/common/common.h" +#include "core/providers/shared_library/provider_api.h" #include "core/providers/cuda/cuda_kernel.h" namespace onnxruntime { diff --git a/onnxruntime/core/providers/cuda/tensor/gather.cc b/onnxruntime/core/providers/cuda/tensor/gather.cc index f648a59e06..189055cab7 100644 --- a/onnxruntime/core/providers/cuda/tensor/gather.cc +++ b/onnxruntime/core/providers/cuda/tensor/gather.cc @@ -4,7 +4,6 @@ #include "core/providers/cuda/tensor/gather_impl.h" #include "core/providers/cuda/tensor/gather.h" #include "core/providers/cpu/tensor/utils.h" -#include "core/providers/common.h" namespace onnxruntime { namespace cuda { @@ -13,7 +12,7 @@ ONNX_OPERATOR_VERSIONED_KERNEL_EX( kOnnxDomain, 1, 10, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()) .TypeConstraint("Tind", std::vector{ DataTypeImpl::GetTensorType(), @@ -25,7 +24,7 @@ ONNX_OPERATOR_VERSIONED_KERNEL_EX( kOnnxDomain, 11, 12, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()) .TypeConstraint("Tind", std::vector{ DataTypeImpl::GetTensorType(), @@ -38,7 +37,7 @@ ONNX_OPERATOR_KERNEL_EX( kOnnxDomain, 13, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()) .TypeConstraint("Tind", std::vector{ DataTypeImpl::GetTensorType(), diff --git a/onnxruntime/core/providers/cuda/tensor/gather.h b/onnxruntime/core/providers/cuda/tensor/gather.h index 39c2fba55c..c4a2a71f5a 100644 --- a/onnxruntime/core/providers/cuda/tensor/gather.h +++ b/onnxruntime/core/providers/cuda/tensor/gather.h @@ -1,9 +1,9 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. -#include "core/common/common.h" +#include "core/providers/shared_library/provider_api.h" #include "core/providers/cuda/cuda_kernel.h" -#include "core/providers/cpu/tensor/gather.h" +#include "core/providers/cpu/tensor/gatherbase.h" namespace onnxruntime { namespace cuda { diff --git a/onnxruntime/core/providers/cuda/tensor/gather_elements.cc b/onnxruntime/core/providers/cuda/tensor/gather_elements.cc index 6ada2248f7..08312899de 100644 --- a/onnxruntime/core/providers/cuda/tensor/gather_elements.cc +++ b/onnxruntime/core/providers/cuda/tensor/gather_elements.cc @@ -4,7 +4,6 @@ #include "gather_elements.h" #include "gather_elements_impl.h" #include "core/providers/cpu/tensor/utils.h" -#include "core/providers/common.h" namespace onnxruntime { namespace cuda { @@ -14,7 +13,7 @@ ONNX_OPERATOR_KERNEL_EX( kOnnxDomain, 13, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()) .TypeConstraint("Tind", std::vector{ DataTypeImpl::GetTensorType(), @@ -26,7 +25,7 @@ ONNX_OPERATOR_VERSIONED_KERNEL_EX( kOnnxDomain, 11, 12, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()) .TypeConstraint("Tind", std::vector{ DataTypeImpl::GetTensorType(), diff --git a/onnxruntime/core/providers/cuda/tensor/gather_elements.h b/onnxruntime/core/providers/cuda/tensor/gather_elements.h index e155858335..9038c35d8d 100644 --- a/onnxruntime/core/providers/cuda/tensor/gather_elements.h +++ b/onnxruntime/core/providers/cuda/tensor/gather_elements.h @@ -2,8 +2,7 @@ // Licensed under the MIT License. #pragma once -#include "core/common/common.h" -#include "core/providers/cpu/tensor/gather_elements.h" +#include "core/providers/shared_library/provider_api.h" #include "core/providers/cuda/cuda_kernel.h" namespace onnxruntime { diff --git a/onnxruntime/core/providers/cuda/tensor/gather_nd.cc b/onnxruntime/core/providers/cuda/tensor/gather_nd.cc index 03b65164ff..3cd7b7dc80 100644 --- a/onnxruntime/core/providers/cuda/tensor/gather_nd.cc +++ b/onnxruntime/core/providers/cuda/tensor/gather_nd.cc @@ -100,7 +100,7 @@ Status GatherNDBase::PrepareCompute( endver, \ TIndex, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T", \ std::vector{ \ DataTypeImpl::GetTensorType(), \ @@ -135,7 +135,7 @@ Status GatherNDBase::PrepareCompute( ver, \ TIndex, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T", GATHER_ND_T_TENSOR_TYPES) \ .TypeConstraint("Tind", DataTypeImpl::GetTensorType()), \ GatherND); diff --git a/onnxruntime/core/providers/cuda/tensor/gather_nd.h b/onnxruntime/core/providers/cuda/tensor/gather_nd.h index 527a4b8c54..fb619cd1fd 100644 --- a/onnxruntime/core/providers/cuda/tensor/gather_nd.h +++ b/onnxruntime/core/providers/cuda/tensor/gather_nd.h @@ -3,7 +3,7 @@ #pragma once -#include "core/common/common.h" +#include "core/providers/shared_library/provider_api.h" #include "core/providers/cuda/cuda_kernel.h" namespace onnxruntime { @@ -43,4 +43,4 @@ class GatherND final : public GatherNDBase { }; } // namespace cuda -} // namespace onnxruntime \ No newline at end of file +} // namespace onnxruntime diff --git a/onnxruntime/core/providers/cuda/tensor/identity_op.cc b/onnxruntime/core/providers/cuda/tensor/identity_op.cc index bc3b6e4dd7..9281e3a42b 100644 --- a/onnxruntime/core/providers/cuda/tensor/identity_op.cc +++ b/onnxruntime/core/providers/cuda/tensor/identity_op.cc @@ -10,7 +10,7 @@ ONNX_OPERATOR_VERSIONED_KERNEL_EX( kOnnxDomain, 7, 9, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .TypeConstraint("T", {DataTypeImpl::GetTensorType(), DataTypeImpl::GetTensorType(), DataTypeImpl::GetTensorType()}) @@ -23,7 +23,7 @@ ONNX_OPERATOR_VERSIONED_KERNEL_EX( 10, 11, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .TypeConstraint("T", {DataTypeImpl::GetTensorType(), DataTypeImpl::GetTensorType(), DataTypeImpl::GetTensorType()}) @@ -35,8 +35,8 @@ ONNX_OPERATOR_VERSIONED_KERNEL_EX( Identity, kOnnxDomain, 1, 12, - kCudaExecutionProvider, - KernelDefBuilder() + kCudaExecutionProvider, + (*KernelDefBuilder::Create()) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()) .Alias(0, 0), IdentityOp); @@ -46,7 +46,7 @@ ONNX_OPERATOR_VERSIONED_KERNEL_EX( kOnnxDomain, 13, 13, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()) .Alias(0, 0), IdentityOp); @@ -56,7 +56,7 @@ ONNX_OPERATOR_KERNEL_EX( kOnnxDomain, 14, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .TypeConstraint("V", DataTypeImpl::AllFixedSizeTensorTypes()) .Alias(0, 0), IdentityOp); diff --git a/onnxruntime/core/providers/cuda/tensor/identity_op.h b/onnxruntime/core/providers/cuda/tensor/identity_op.h index f00bb6414d..4bac78d042 100644 --- a/onnxruntime/core/providers/cuda/tensor/identity_op.h +++ b/onnxruntime/core/providers/cuda/tensor/identity_op.h @@ -2,7 +2,7 @@ // Licensed under the MIT License. #pragma once -#include "core/common/common.h" +#include "core/providers/shared_library/provider_api.h" #include "core/providers/cuda/cuda_kernel.h" namespace onnxruntime { diff --git a/onnxruntime/core/providers/cuda/tensor/nonzero_op.cc b/onnxruntime/core/providers/cuda/tensor/nonzero_op.cc index 992fe5dfab..f4f4004986 100644 --- a/onnxruntime/core/providers/cuda/tensor/nonzero_op.cc +++ b/onnxruntime/core/providers/cuda/tensor/nonzero_op.cc @@ -4,28 +4,27 @@ #include "nonzero_op.h" #include "nonzero_impl.h" #include "core/providers/cpu/tensor/utils.h" -#include "core/providers/common.h" namespace onnxruntime { namespace cuda { // kernel builder functions -#define NONZERO_TYPED_KERNEL_WITH_TYPE_NAME(type, type_name) \ - ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ - NonZero, \ - kOnnxDomain, \ - 9, 12, \ - type_name, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ - NonZero) \ - ONNX_OPERATOR_TYPED_KERNEL_EX( \ - NonZero, \ - kOnnxDomain, \ - 13, \ - type_name, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ +#define NONZERO_TYPED_KERNEL_WITH_TYPE_NAME(type, type_name) \ + ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ + NonZero, \ + kOnnxDomain, \ + 9, 12, \ + type_name, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ + NonZero) \ + ONNX_OPERATOR_TYPED_KERNEL_EX( \ + NonZero, \ + kOnnxDomain, \ + 13, \ + type_name, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ NonZero) #define NONZERO_TYPED_KERNEL(type) \ diff --git a/onnxruntime/core/providers/cuda/tensor/nonzero_op.h b/onnxruntime/core/providers/cuda/tensor/nonzero_op.h index a267d56f7d..cac48e0ee8 100644 --- a/onnxruntime/core/providers/cuda/tensor/nonzero_op.h +++ b/onnxruntime/core/providers/cuda/tensor/nonzero_op.h @@ -2,15 +2,15 @@ // Licensed under the MIT License. #pragma once -#include "core/common/common.h" +#include "core/providers/shared_library/provider_api.h" #include "core/providers/cuda/cuda_kernel.h" namespace onnxruntime { namespace cuda { template -class NonZero final: public CudaKernel { -public: +class NonZero final : public CudaKernel { + public: NonZero(const OpKernelInfo& info) : CudaKernel(info) {} Status ComputeInternal(OpKernelContext* context) const override; diff --git a/onnxruntime/core/providers/cuda/tensor/onehot.cc b/onnxruntime/core/providers/cuda/tensor/onehot.cc index 7847a2309b..1ca9c3216f 100644 --- a/onnxruntime/core/providers/cuda/tensor/onehot.cc +++ b/onnxruntime/core/providers/cuda/tensor/onehot.cc @@ -2,7 +2,6 @@ // Licensed under the MIT License. #include "core/providers/cuda/tensor/onehot.h" -#include "core/providers/cpu/tensor/onehot.h" using namespace onnxruntime::common; @@ -17,9 +16,9 @@ namespace cuda { 11, \ in_type##_##out_type##_##depth_type, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ - .InputMemoryType(1) /* Keep depth in CPU */ \ - .InputMemoryType(2) /* Keep values in CPU */ \ + (*KernelDefBuilder::Create()) \ + .InputMemoryType(OrtMemTypeCPUInput, 1) /* Keep depth in CPU */ \ + .InputMemoryType(OrtMemTypeCPUInput, 2) /* Keep values in CPU */ \ .TypeConstraint("T1", DataTypeImpl::GetTensorType()) \ .TypeConstraint("T2", DataTypeImpl::GetTensorType()) \ .TypeConstraint("T3", DataTypeImpl::GetTensorType()), \ diff --git a/onnxruntime/core/providers/cuda/tensor/onehot.h b/onnxruntime/core/providers/cuda/tensor/onehot.h index fff0acd1f1..6d13524055 100644 --- a/onnxruntime/core/providers/cuda/tensor/onehot.h +++ b/onnxruntime/core/providers/cuda/tensor/onehot.h @@ -2,8 +2,7 @@ // Licensed under the MIT License. #pragma once - -#include "core/common/common.h" +#include "core/providers/shared_library/provider_api.h" #include "core/providers/cuda/cuda_kernel.h" namespace onnxruntime { diff --git a/onnxruntime/core/providers/cuda/tensor/pad.cc b/onnxruntime/core/providers/cuda/tensor/pad.cc index 2e344ebb7e..f76f209d21 100644 --- a/onnxruntime/core/providers/cuda/tensor/pad.cc +++ b/onnxruntime/core/providers/cuda/tensor/pad.cc @@ -4,7 +4,6 @@ #include "pad.h" #include "pad_impl.h" #include "core/providers/cpu/tensor/utils.h" -#include "core/providers/cpu/tensor/pad.h" namespace onnxruntime { namespace cuda { @@ -16,7 +15,7 @@ namespace cuda { 2, 10, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()), \ Pad); \ ONNX_OPERATOR_TYPED_KERNEL_EX( \ @@ -25,9 +24,9 @@ namespace cuda { 11, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ - .InputMemoryType(1) \ - .InputMemoryType(2) \ + (*KernelDefBuilder::Create()) \ + .InputMemoryType(OrtMemTypeCPUInput, 1) \ + .InputMemoryType(OrtMemTypeCPUInput, 2) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()), \ Pad); diff --git a/onnxruntime/core/providers/cuda/tensor/pad.h b/onnxruntime/core/providers/cuda/tensor/pad.h index d011a14cab..b206a35995 100644 --- a/onnxruntime/core/providers/cuda/tensor/pad.h +++ b/onnxruntime/core/providers/cuda/tensor/pad.h @@ -1,9 +1,9 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. -#include "core/common/common.h" +#include "core/providers/shared_library/provider_api.h" #include "core/providers/cuda/cuda_kernel.h" -#include "core/providers/cpu/tensor/pad.h" +#include "core/providers/cpu/tensor/padbase.h" using onnxruntime::PadBase; diff --git a/onnxruntime/core/providers/cuda/tensor/quantize_linear.cc b/onnxruntime/core/providers/cuda/tensor/quantize_linear.cc index 17fafa0af5..92c1c7d98c 100644 --- a/onnxruntime/core/providers/cuda/tensor/quantize_linear.cc +++ b/onnxruntime/core/providers/cuda/tensor/quantize_linear.cc @@ -4,8 +4,6 @@ #include "quantize_linear.h" #include "quantize_linear.cuh" -#include "core/providers/common.h" - namespace onnxruntime { namespace cuda { @@ -72,7 +70,7 @@ Status DequantizeLinear::ComputeInternal(OpKernelContext* ctx) const { 10, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T1", DataTypeImpl::GetTensorType()) \ .TypeConstraint("T2", DataTypeImpl::GetTensorType()), \ QuantizeLinear); @@ -88,7 +86,7 @@ REGISTER_Q_KERNEL_TYPED(uint8_t) 10, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()), \ DequantizeLinear); diff --git a/onnxruntime/core/providers/cuda/tensor/quantize_linear.cuh b/onnxruntime/core/providers/cuda/tensor/quantize_linear.cuh index b6773de316..3c11e3b9a5 100644 --- a/onnxruntime/core/providers/cuda/tensor/quantize_linear.cuh +++ b/onnxruntime/core/providers/cuda/tensor/quantize_linear.cuh @@ -4,7 +4,6 @@ #pragma once #include "quantize_linear.h" -#include "core/providers/cpu/math/matmul_helper.h" #include "core/providers/cuda/cuda_common.h" #include "core/providers/cuda/shared_inc/cuda_utils.h" diff --git a/onnxruntime/core/providers/cuda/tensor/quantize_linear.h b/onnxruntime/core/providers/cuda/tensor/quantize_linear.h index 1a3f1a33c1..fe50e9f273 100644 --- a/onnxruntime/core/providers/cuda/tensor/quantize_linear.h +++ b/onnxruntime/core/providers/cuda/tensor/quantize_linear.h @@ -3,7 +3,7 @@ #pragma once -#include "core/common/common.h" +#include "core/providers/shared_library/provider_api.h" #include "core/providers/cuda/cuda_kernel.h" #include "gsl/gsl" diff --git a/onnxruntime/core/providers/cuda/tensor/reshape.cc b/onnxruntime/core/providers/cuda/tensor/reshape.cc index 6037fa5e9b..61bca5bfe7 100644 --- a/onnxruntime/core/providers/cuda/tensor/reshape.cc +++ b/onnxruntime/core/providers/cuda/tensor/reshape.cc @@ -11,11 +11,11 @@ ONNX_OPERATOR_KERNEL_EX( kOnnxDomain, 14, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()) .TypeConstraint("shape", DataTypeImpl::GetTensorType()) .Alias(0, 0) - .InputMemoryType(1), + .InputMemoryType(OrtMemTypeCPUInput, 1), Reshape); ONNX_OPERATOR_VERSIONED_KERNEL_EX( @@ -23,11 +23,11 @@ ONNX_OPERATOR_VERSIONED_KERNEL_EX( kOnnxDomain, 13, 13, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()) .TypeConstraint("shape", DataTypeImpl::GetTensorType()) .Alias(0, 0) - .InputMemoryType(1), + .InputMemoryType(OrtMemTypeCPUInput, 1), Reshape); ONNX_OPERATOR_VERSIONED_KERNEL_EX( @@ -35,11 +35,11 @@ ONNX_OPERATOR_VERSIONED_KERNEL_EX( kOnnxDomain, 5, 12, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()) .TypeConstraint("shape", DataTypeImpl::GetTensorType()) .Alias(0, 0) - .InputMemoryType(1), + .InputMemoryType(OrtMemTypeCPUInput, 1), Reshape); ONNX_OPERATOR_VERSIONED_KERNEL_EX( @@ -48,7 +48,7 @@ ONNX_OPERATOR_VERSIONED_KERNEL_EX( 1, 4, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .Alias(0, 0) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()), Reshape_1); diff --git a/onnxruntime/core/providers/cuda/tensor/reshape.h b/onnxruntime/core/providers/cuda/tensor/reshape.h index cc55ecd186..ba57d6e602 100644 --- a/onnxruntime/core/providers/cuda/tensor/reshape.h +++ b/onnxruntime/core/providers/cuda/tensor/reshape.h @@ -3,7 +3,7 @@ #pragma once -#include "core/common/common.h" +#include "core/providers/shared_library/provider_api.h" #include "core/providers/cuda/cuda_kernel.h" #include "gsl/gsl" #include "core/providers/cpu/tensor/reshape_helper.h" diff --git a/onnxruntime/core/providers/cuda/tensor/resize.cc b/onnxruntime/core/providers/cuda/tensor/resize.cc index 842599c0c5..764172a8d1 100644 --- a/onnxruntime/core/providers/cuda/tensor/resize.cc +++ b/onnxruntime/core/providers/cuda/tensor/resize.cc @@ -12,8 +12,8 @@ namespace cuda { 10, 10, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ - .InputMemoryType(1) \ + (*KernelDefBuilder::Create()) \ + .InputMemoryType(OrtMemTypeCPUInput, 1) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()), \ Resize); \ ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ @@ -22,10 +22,10 @@ namespace cuda { 11, 12, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ - .InputMemoryType(1) \ - .InputMemoryType(2) \ - .InputMemoryType(3) \ + (*KernelDefBuilder::Create()) \ + .InputMemoryType(OrtMemTypeCPUInput, 1) \ + .InputMemoryType(OrtMemTypeCPUInput, 2) \ + .InputMemoryType(OrtMemTypeCPUInput, 3) \ .TypeConstraint("T1", DataTypeImpl::GetTensorType()), \ Resize); \ ONNX_OPERATOR_TYPED_KERNEL_EX( \ @@ -34,10 +34,10 @@ namespace cuda { 13, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ - .InputMemoryType(1) \ - .InputMemoryType(2) \ - .InputMemoryType(3) \ + (*KernelDefBuilder::Create()) \ + .InputMemoryType(OrtMemTypeCPUInput, 1) \ + .InputMemoryType(OrtMemTypeCPUInput, 2) \ + .InputMemoryType(OrtMemTypeCPUInput, 3) \ .TypeConstraint("T1", DataTypeImpl::GetTensorType()), \ Resize); diff --git a/onnxruntime/core/providers/cuda/tensor/resize.h b/onnxruntime/core/providers/cuda/tensor/resize.h index 0ddbd9a01e..89abd9456e 100644 --- a/onnxruntime/core/providers/cuda/tensor/resize.h +++ b/onnxruntime/core/providers/cuda/tensor/resize.h @@ -3,6 +3,7 @@ #pragma once +#include "core/providers/shared_library/provider_api.h" #include "core/providers/cuda/tensor/upsample.h" namespace onnxruntime { @@ -11,7 +12,7 @@ namespace cuda { template class Resize : public Upsample { public: - Resize(OpKernelInfo info) : Upsample(info) { + Resize(const OpKernelInfo& info) : Upsample(info) { } Status ComputeInternal(OpKernelContext* context) const override { diff --git a/onnxruntime/core/providers/cuda/tensor/reverse_sequence.cc b/onnxruntime/core/providers/cuda/tensor/reverse_sequence.cc index 7f70c57503..7739ce18f9 100644 --- a/onnxruntime/core/providers/cuda/tensor/reverse_sequence.cc +++ b/onnxruntime/core/providers/cuda/tensor/reverse_sequence.cc @@ -5,8 +5,6 @@ #include "reverse_sequence_impl.h" #include "core/providers/cpu/tensor/utils.h" -#include "core/providers/common.h" -#include "core/framework/utils.h" namespace onnxruntime { namespace cuda { @@ -16,7 +14,7 @@ ONNX_OPERATOR_KERNEL_EX( kOnnxDomain, 10, kCudaExecutionProvider, - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()), + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()), ReverseSequenceOp); #define ReverseSequenceCallCudaImplTypeAs(T, TEqual) \ diff --git a/onnxruntime/core/providers/cuda/tensor/reverse_sequence.h b/onnxruntime/core/providers/cuda/tensor/reverse_sequence.h index 0e5ef66551..84730c3bd8 100644 --- a/onnxruntime/core/providers/cuda/tensor/reverse_sequence.h +++ b/onnxruntime/core/providers/cuda/tensor/reverse_sequence.h @@ -3,7 +3,7 @@ #pragma once -#include "core/common/common.h" +#include "core/providers/shared_library/provider_api.h" #include "core/providers/cuda/cuda_kernel.h" namespace onnxruntime { diff --git a/onnxruntime/core/providers/cuda/tensor/scatter_elements.cc b/onnxruntime/core/providers/cuda/tensor/scatter_elements.cc index 8142888402..112ae5bbfd 100755 --- a/onnxruntime/core/providers/cuda/tensor/scatter_elements.cc +++ b/onnxruntime/core/providers/cuda/tensor/scatter_elements.cc @@ -4,7 +4,6 @@ #include "scatter_elements.h" #include "scatter_elements_impl.h" #include "core/providers/cpu/tensor/utils.h" -#include "core/providers/common.h" namespace onnxruntime { namespace cuda { @@ -15,7 +14,7 @@ ONNX_OPERATOR_VERSIONED_KERNEL_EX( 9, 10, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()) .TypeConstraint("Tind", std::vector{ DataTypeImpl::GetTensorType(), @@ -28,7 +27,7 @@ ONNX_OPERATOR_VERSIONED_KERNEL_EX( 11, 12, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()) .TypeConstraint("Tind", std::vector{ DataTypeImpl::GetTensorType(), @@ -40,7 +39,7 @@ ONNX_OPERATOR_KERNEL_EX( kOnnxDomain, 13, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()) .TypeConstraint("Tind", std::vector{ DataTypeImpl::GetTensorType(), diff --git a/onnxruntime/core/providers/cuda/tensor/scatter_elements.h b/onnxruntime/core/providers/cuda/tensor/scatter_elements.h index 341fd861fd..bbf33b84a1 100755 --- a/onnxruntime/core/providers/cuda/tensor/scatter_elements.h +++ b/onnxruntime/core/providers/cuda/tensor/scatter_elements.h @@ -2,7 +2,7 @@ // Licensed under the MIT License. #pragma once -#include "core/common/common.h" +#include "core/providers/shared_library/provider_api.h" #include "core/providers/cuda/cuda_kernel.h" namespace onnxruntime { diff --git a/onnxruntime/core/providers/cuda/tensor/scatter_nd.cc b/onnxruntime/core/providers/cuda/tensor/scatter_nd.cc index d5f632a1bf..6a78150ce1 100644 --- a/onnxruntime/core/providers/cuda/tensor/scatter_nd.cc +++ b/onnxruntime/core/providers/cuda/tensor/scatter_nd.cc @@ -13,7 +13,7 @@ ONNX_OPERATOR_VERSIONED_KERNEL_EX(ScatterND, kOnnxDomain, 11, 12, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()) .MayInplace(0, 0), ScatterND); @@ -22,7 +22,7 @@ ONNX_OPERATOR_KERNEL_EX(ScatterND, kOnnxDomain, 13, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()) .MayInplace(0, 0), ScatterND); diff --git a/onnxruntime/core/providers/cuda/tensor/scatter_nd.h b/onnxruntime/core/providers/cuda/tensor/scatter_nd.h index 89bd6104a5..bececcf29b 100644 --- a/onnxruntime/core/providers/cuda/tensor/scatter_nd.h +++ b/onnxruntime/core/providers/cuda/tensor/scatter_nd.h @@ -3,7 +3,7 @@ #pragma once -#include "core/common/common.h" +#include "core/providers/shared_library/provider_api.h" #include "core/providers/cuda/cuda_kernel.h" #include "core/providers/cpu/tensor/scatter_nd.h" @@ -17,4 +17,4 @@ class ScatterND final : public CudaKernel, protected ScatterNDBase { }; } // namespace cuda -} // namespace onnxruntime \ No newline at end of file +} // namespace onnxruntime diff --git a/onnxruntime/core/providers/cuda/tensor/shape_op.cc b/onnxruntime/core/providers/cuda/tensor/shape_op.cc index 4781c90182..5680831f5d 100644 --- a/onnxruntime/core/providers/cuda/tensor/shape_op.cc +++ b/onnxruntime/core/providers/cuda/tensor/shape_op.cc @@ -1,6 +1,7 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. +#include "core/providers/shared_library/provider_api.h" #include "core/providers/cpu/tensor/shape_op.h" #include "core/providers/cuda/cuda_fwd.h" @@ -12,9 +13,9 @@ ONNX_OPERATOR_VERSIONED_KERNEL_EX( kOnnxDomain, 1, 12, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) // properly force CPU/GPU synch inside the kernel - .OutputMemoryType(0) + .OutputMemoryType(OrtMemTypeCPUInput, 0) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()) .TypeConstraint("T1", DataTypeImpl::GetTensorType()), Shape); @@ -24,9 +25,9 @@ ONNX_OPERATOR_KERNEL_EX( kOnnxDomain, 13, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) // properly force CPU/GPU synch inside the kernel - .OutputMemoryType(0) + .OutputMemoryType(OrtMemTypeCPUInput, 0) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()) .TypeConstraint("T1", DataTypeImpl::GetTensorType()), Shape); diff --git a/onnxruntime/core/providers/cuda/tensor/size.cc b/onnxruntime/core/providers/cuda/tensor/size.cc index 9c36d6aafe..48ca32a548 100644 --- a/onnxruntime/core/providers/cuda/tensor/size.cc +++ b/onnxruntime/core/providers/cuda/tensor/size.cc @@ -1,6 +1,7 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. +#include "core/providers/shared_library/provider_api.h" #include "core/providers/cpu/tensor/size.h" #include "core/providers/cuda/cuda_fwd.h" @@ -12,9 +13,9 @@ ONNX_OPERATOR_VERSIONED_KERNEL_EX( kOnnxDomain, 1, 12, kCudaExecutionProvider, - KernelDefBuilder() - .OutputMemoryType(0) - .TypeConstraint("T", DataTypeImpl::AllTensorTypes()) + (*KernelDefBuilder::Create()) + .OutputMemoryType(OrtMemTypeCPUInput, 0) + .TypeConstraint("T", DataTypeImpl::AllTensorTypes()) .TypeConstraint("T1", DataTypeImpl::GetTensorType()), Size); @@ -23,9 +24,9 @@ ONNX_OPERATOR_KERNEL_EX( kOnnxDomain, 13, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) // properly force CPU/GPU synch inside the kernel - .OutputMemoryType(0) + .OutputMemoryType(OrtMemTypeCPUInput, 0) .TypeConstraint("T", DataTypeImpl::AllTensorTypes()) .TypeConstraint("T1", DataTypeImpl::GetTensorType()), Size); diff --git a/onnxruntime/core/providers/cuda/tensor/slice.cc b/onnxruntime/core/providers/cuda/tensor/slice.cc index cf4d7ad75c..7ced12e581 100644 --- a/onnxruntime/core/providers/cuda/tensor/slice.cc +++ b/onnxruntime/core/providers/cuda/tensor/slice.cc @@ -14,7 +14,7 @@ namespace cuda { 1, 9, \ TIND, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()) \ .TypeConstraint("Tind", DataTypeImpl::GetTensorType()), \ Slice); @@ -30,11 +30,11 @@ REGISTER_VERSIONED_TYPED_SLICE(float) 10, 10, \ TIND, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ - .InputMemoryType(1) \ - .InputMemoryType(2) \ - .InputMemoryType(3) \ - .InputMemoryType(4) \ + (*KernelDefBuilder::Create()) \ + .InputMemoryType(OrtMemTypeCPUInput, 1) \ + .InputMemoryType(OrtMemTypeCPUInput, 2) \ + .InputMemoryType(OrtMemTypeCPUInput, 3) \ + .InputMemoryType(OrtMemTypeCPUInput, 4) \ .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()) \ .TypeConstraint("Tind", DataTypeImpl::GetTensorType()), \ Slice); @@ -50,11 +50,11 @@ REGISTER_V10_TYPED_SLICE(float) 11, 12, \ TIND, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ - .InputMemoryType(1) \ - .InputMemoryType(2) \ - .InputMemoryType(3) \ - .InputMemoryType(4) \ + (*KernelDefBuilder::Create()) \ + .InputMemoryType(OrtMemTypeCPUInput, 1) \ + .InputMemoryType(OrtMemTypeCPUInput, 2) \ + .InputMemoryType(OrtMemTypeCPUInput, 3) \ + .InputMemoryType(OrtMemTypeCPUInput, 4) \ .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()) \ .TypeConstraint("Tind", DataTypeImpl::GetTensorType()), \ Slice); @@ -70,11 +70,11 @@ REGISTER_V12_TYPED_SLICE(float) 13, \ TIND, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ - .InputMemoryType(1) \ - .InputMemoryType(2) \ - .InputMemoryType(3) \ - .InputMemoryType(4) \ + (*KernelDefBuilder::Create()) \ + .InputMemoryType(OrtMemTypeCPUInput, 1) \ + .InputMemoryType(OrtMemTypeCPUInput, 2) \ + .InputMemoryType(OrtMemTypeCPUInput, 3) \ + .InputMemoryType(OrtMemTypeCPUInput, 4) \ .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()) \ .TypeConstraint("Tind", DataTypeImpl::GetTensorType()), \ Slice); diff --git a/onnxruntime/core/providers/cuda/tensor/slice.h b/onnxruntime/core/providers/cuda/tensor/slice.h index b43cbfee78..87607bd4cc 100644 --- a/onnxruntime/core/providers/cuda/tensor/slice.h +++ b/onnxruntime/core/providers/cuda/tensor/slice.h @@ -1,7 +1,7 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. -#include "core/common/common.h" +#include "core/providers/shared_library/provider_api.h" #include "core/providers/cuda/cuda_kernel.h" #include "core/providers/cpu/tensor/slice.h" #include "core/providers/cpu/tensor/utils.h" diff --git a/onnxruntime/core/providers/cuda/tensor/split.cc b/onnxruntime/core/providers/cuda/tensor/split.cc index 4b130be232..0e6805ac1a 100644 --- a/onnxruntime/core/providers/cuda/tensor/split.cc +++ b/onnxruntime/core/providers/cuda/tensor/split.cc @@ -4,7 +4,6 @@ #include "core/providers/cuda/tensor/split.h" #include "core/providers/cuda/tensor/split_impl.h" #include "core/providers/cpu/tensor/utils.h" -#include "core/providers/common.h" namespace onnxruntime { namespace cuda { @@ -12,7 +11,7 @@ ONNX_OPERATOR_VERSIONED_KERNEL_EX(Split, kOnnxDomain, 2, 10, kCudaExecutionProvider, - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()), + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()), Split); // explicitly supports negative axis @@ -20,7 +19,7 @@ ONNX_OPERATOR_VERSIONED_KERNEL_EX(Split, kOnnxDomain, 11, 12, kCudaExecutionProvider, - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()), + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()), Split); // explicitly supports 'split' as optional input @@ -28,8 +27,8 @@ ONNX_OPERATOR_KERNEL_EX(Split, kOnnxDomain, 13, kCudaExecutionProvider, - KernelDefBuilder() - .InputMemoryType(1) + (*KernelDefBuilder::Create()) + .InputMemoryType(OrtMemTypeCPUInput, 1) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()), Split); diff --git a/onnxruntime/core/providers/cuda/tensor/split.h b/onnxruntime/core/providers/cuda/tensor/split.h index 1f7558aa89..bc01d11c72 100644 --- a/onnxruntime/core/providers/cuda/tensor/split.h +++ b/onnxruntime/core/providers/cuda/tensor/split.h @@ -1,7 +1,7 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. -#include "core/common/common.h" +#include "core/providers/shared_library/provider_api.h" #include "core/providers/cuda/cuda_kernel.h" #include "core/providers/cpu/tensor/split.h" diff --git a/onnxruntime/core/providers/cuda/tensor/squeeze.cc b/onnxruntime/core/providers/cuda/tensor/squeeze.cc index 3cd5eab410..94ee38cdb3 100644 --- a/onnxruntime/core/providers/cuda/tensor/squeeze.cc +++ b/onnxruntime/core/providers/cuda/tensor/squeeze.cc @@ -11,7 +11,7 @@ ONNX_OPERATOR_VERSIONED_KERNEL_EX( kOnnxDomain, 1, 10, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .Alias(0, 0) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()), Squeeze); @@ -22,7 +22,7 @@ ONNX_OPERATOR_VERSIONED_KERNEL_EX( kOnnxDomain, 11, 12, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .Alias(0, 0) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()), Squeeze); @@ -33,10 +33,10 @@ ONNX_OPERATOR_KERNEL_EX( kOnnxDomain, 13, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .Alias(0, 0) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()) - .InputMemoryType(1), + .InputMemoryType(OrtMemTypeCPUInput, 1), Squeeze); Status Squeeze::ComputeInternal(OpKernelContext* ctx) const { diff --git a/onnxruntime/core/providers/cuda/tensor/squeeze.h b/onnxruntime/core/providers/cuda/tensor/squeeze.h index 45b4974407..da4cdc5d3e 100644 --- a/onnxruntime/core/providers/cuda/tensor/squeeze.h +++ b/onnxruntime/core/providers/cuda/tensor/squeeze.h @@ -1,7 +1,7 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. -#include "core/common/common.h" +#include "core/providers/shared_library/provider_api.h" #include "core/providers/cuda/cuda_kernel.h" #include "core/providers/cpu/tensor/squeeze.h" diff --git a/onnxruntime/core/providers/cuda/tensor/tile.cc b/onnxruntime/core/providers/cuda/tensor/tile.cc index 7d70baa3a9..23a7e9dd1e 100644 --- a/onnxruntime/core/providers/cuda/tensor/tile.cc +++ b/onnxruntime/core/providers/cuda/tensor/tile.cc @@ -15,8 +15,8 @@ ONNX_OPERATOR_VERSIONED_KERNEL_EX( 6, 12, kCudaExecutionProvider, - KernelDefBuilder() - .InputMemoryType(1) + (*KernelDefBuilder::Create()) + .InputMemoryType(OrtMemTypeCPUInput, 1) .TypeConstraint("T", {DataTypeImpl::GetTensorType(), DataTypeImpl::GetTensorType(), DataTypeImpl::GetTensorType(), @@ -30,8 +30,8 @@ ONNX_OPERATOR_KERNEL_EX( kOnnxDomain, 13, kCudaExecutionProvider, - KernelDefBuilder() - .InputMemoryType(1) + (*KernelDefBuilder::Create()) + .InputMemoryType(OrtMemTypeCPUInput, 1) .TypeConstraint("T", {DataTypeImpl::GetTensorType(), DataTypeImpl::GetTensorType(), DataTypeImpl::GetTensorType(), diff --git a/onnxruntime/core/providers/cuda/tensor/tile.h b/onnxruntime/core/providers/cuda/tensor/tile.h index 5aea457452..a1deb53a00 100644 --- a/onnxruntime/core/providers/cuda/tensor/tile.h +++ b/onnxruntime/core/providers/cuda/tensor/tile.h @@ -1,7 +1,7 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. -#include "core/common/common.h" +#include "core/providers/shared_library/provider_api.h" #include "core/providers/cuda/cuda_kernel.h" #include "core/providers/cpu/tensor/tile.h" diff --git a/onnxruntime/core/providers/cuda/tensor/transpose.cc b/onnxruntime/core/providers/cuda/tensor/transpose.cc index f5fb7c0147..33bf3ce05e 100644 --- a/onnxruntime/core/providers/cuda/tensor/transpose.cc +++ b/onnxruntime/core/providers/cuda/tensor/transpose.cc @@ -14,7 +14,7 @@ ONNX_OPERATOR_VERSIONED_KERNEL_EX( kOnnxDomain, 1, 12, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()), Transpose); @@ -23,7 +23,7 @@ ONNX_OPERATOR_KERNEL_EX( kOnnxDomain, 13, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()), Transpose); @@ -90,9 +90,9 @@ Status Transpose::DoTranspose(const cudaDeviceProp& prop, return Status::OK(); auto element_type = input.GetElementType(); - if (element_type == utils::GetONNXTensorElementDataType() || - element_type == utils::GetONNXTensorElementDataType() || - element_type == utils::GetONNXTensorElementDataType()) { + if (element_type == ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT || + element_type == ONNX_TENSOR_ELEMENT_DATA_TYPE_DOUBLE || + element_type == ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT16) { auto mn = TryTransposeWithCublas(permutations, input_shape_override ? *input_shape_override : input.Shape()); int M = std::get<0>(mn); int N = std::get<1>(mn); @@ -128,25 +128,25 @@ Status Transpose::DoTranspose(const cudaDeviceProp& prop, new_permutations[j] -= 1; } } - for (auto j = i+1; j < new_rank; j++) { - new_permutations[j-1] = new_permutations[j]; + for (auto j = i + 1; j < new_rank; j++) { + new_permutations[j - 1] = new_permutations[j]; } // update input dims new_input_dims[prev] *= new_input_dims[curr]; new_input_dims[curr] = 1; - for (auto j = static_cast(curr+1); j < new_rank; j++) { - new_input_dims[j-1] = new_input_dims[j]; + for (auto j = static_cast(curr + 1); j < new_rank; j++) { + new_input_dims[j - 1] = new_input_dims[j]; } - new_input_dims[new_rank-1] = 1; + new_input_dims[new_rank - 1] = 1; // update output dims - new_output_dims[i-1] *= new_output_dims[i]; + new_output_dims[i - 1] *= new_output_dims[i]; new_output_dims[i] = 1; - for (auto j = i+1; j < new_rank; j++) { - new_output_dims[j-1] = new_output_dims[j]; + for (auto j = i + 1; j < new_rank; j++) { + new_output_dims[j - 1] = new_output_dims[j]; } - new_output_dims[new_rank-1] = 1; + new_output_dims[new_rank - 1] = 1; new_rank--; } diff --git a/onnxruntime/core/providers/cuda/tensor/transpose.h b/onnxruntime/core/providers/cuda/tensor/transpose.h index c9b41e9102..ab996ba151 100644 --- a/onnxruntime/core/providers/cuda/tensor/transpose.h +++ b/onnxruntime/core/providers/cuda/tensor/transpose.h @@ -3,8 +3,8 @@ #pragma once +#include "core/providers/shared_library/provider_api.h" #include "gsl/gsl" -#include "core/common/common.h" #include "core/providers/cuda/cuda_kernel.h" #include "core/providers/cpu/tensor/transpose.h" diff --git a/onnxruntime/core/providers/cuda/tensor/unsqueeze.cc b/onnxruntime/core/providers/cuda/tensor/unsqueeze.cc index 9d293b8821..282876c8cc 100644 --- a/onnxruntime/core/providers/cuda/tensor/unsqueeze.cc +++ b/onnxruntime/core/providers/cuda/tensor/unsqueeze.cc @@ -11,7 +11,7 @@ ONNX_OPERATOR_VERSIONED_KERNEL_EX( kOnnxDomain, 1, 10, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .Alias(0, 0) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()), Unsqueeze); @@ -22,7 +22,7 @@ ONNX_OPERATOR_VERSIONED_KERNEL_EX( kOnnxDomain, 11, 12, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .Alias(0, 0) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()), Unsqueeze); @@ -33,11 +33,11 @@ ONNX_OPERATOR_KERNEL_EX( kOnnxDomain, 13, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .Alias(0, 0) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()) - .InputMemoryType(1), - Unsqueeze); + .InputMemoryType(OrtMemTypeCPUInput, 1), + Unsqueeze); Status Unsqueeze::ComputeInternal(OpKernelContext* ctx) const { Prepare p; diff --git a/onnxruntime/core/providers/cuda/tensor/unsqueeze.h b/onnxruntime/core/providers/cuda/tensor/unsqueeze.h index 87e2b94010..a6bc6e9508 100644 --- a/onnxruntime/core/providers/cuda/tensor/unsqueeze.h +++ b/onnxruntime/core/providers/cuda/tensor/unsqueeze.h @@ -1,7 +1,7 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. -#include "core/common/common.h" +#include "core/providers/shared_library/provider_api.h" #include "core/providers/cuda/cuda_kernel.h" #include "core/providers/cpu/tensor/unsqueeze.h" diff --git a/onnxruntime/core/providers/cuda/tensor/upsample.cc b/onnxruntime/core/providers/cuda/tensor/upsample.cc index 2fec80ab1f..150787f92e 100644 --- a/onnxruntime/core/providers/cuda/tensor/upsample.cc +++ b/onnxruntime/core/providers/cuda/tensor/upsample.cc @@ -19,8 +19,8 @@ namespace cuda { end, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ - .InputMemoryType(1) \ + (*KernelDefBuilder::Create()) \ + .InputMemoryType(OrtMemTypeCPUInput, 1) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()), \ Upsample) diff --git a/onnxruntime/core/providers/cuda/tensor/upsample.h b/onnxruntime/core/providers/cuda/tensor/upsample.h index 85d14a292e..06f98e821d 100644 --- a/onnxruntime/core/providers/cuda/tensor/upsample.h +++ b/onnxruntime/core/providers/cuda/tensor/upsample.h @@ -3,7 +3,7 @@ #pragma once -#include "core/common/common.h" +#include "core/providers/shared_library/provider_api.h" #include "core/providers/cuda/cuda_kernel.h" #include "core/providers/cpu/tensor/upsample.h" @@ -13,7 +13,7 @@ namespace cuda { template class Upsample : public UpsampleBase, public CudaKernel { public: - Upsample(OpKernelInfo info) : UpsampleBase(info), CudaKernel(info) { + Upsample(const OpKernelInfo& info) : UpsampleBase(info), CudaKernel(info) { } Status ComputeInternal(OpKernelContext* context) const override; diff --git a/onnxruntime/core/providers/cuda/tensor/where.cc b/onnxruntime/core/providers/cuda/tensor/where.cc index 5dd3d6a098..e718ba0876 100644 --- a/onnxruntime/core/providers/cuda/tensor/where.cc +++ b/onnxruntime/core/providers/cuda/tensor/where.cc @@ -16,7 +16,7 @@ namespace cuda { 9, \ TName, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("B", DataTypeImpl::GetTensorType()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()), \ Where); diff --git a/onnxruntime/core/providers/cuda/tensor/where.h b/onnxruntime/core/providers/cuda/tensor/where.h index ec4dd6e47e..0ca0681852 100644 --- a/onnxruntime/core/providers/cuda/tensor/where.h +++ b/onnxruntime/core/providers/cuda/tensor/where.h @@ -1,7 +1,7 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. -#include "core/common/common.h" +#include "core/providers/shared_library/provider_api.h" #include "core/providers/cuda/cuda_kernel.h" namespace onnxruntime { diff --git a/onnxruntime/core/providers/cuda/version_script.lds b/onnxruntime/core/providers/cuda/version_script.lds new file mode 100644 index 0000000000..094abb3329 --- /dev/null +++ b/onnxruntime/core/providers/cuda/version_script.lds @@ -0,0 +1,9 @@ +#_init and _fini should be local +VERS_1.0 { + global: + GetProvider; + + # Hide everything else. + local: + *; +}; diff --git a/onnxruntime/core/providers/openvino/openvino_provider_factory.cc b/onnxruntime/core/providers/openvino/openvino_provider_factory.cc index 6b2b990615..4bb6509b14 100644 --- a/onnxruntime/core/providers/openvino/openvino_provider_factory.cc +++ b/onnxruntime/core/providers/openvino/openvino_provider_factory.cc @@ -50,7 +50,7 @@ struct ProviderInfo_OpenVINO_Impl : ProviderInfo_OpenVINO { } g_info; struct OpenVINO_Provider : Provider { - const void* GetInfo() override { return &g_info; } + void* GetInfo() override { return &g_info; } std::shared_ptr CreateExecutionProviderFactory(const void* void_params) override { auto& params = *reinterpret_cast(void_params); diff --git a/onnxruntime/core/providers/openvino/ov_versions/data_ops.cc b/onnxruntime/core/providers/openvino/ov_versions/data_ops.cc index fb421ac97c..c5dd205da0 100644 --- a/onnxruntime/core/providers/openvino/ov_versions/data_ops.cc +++ b/onnxruntime/core/providers/openvino/ov_versions/data_ops.cc @@ -5,8 +5,8 @@ #include "core/providers/shared_library/provider_api.h" #include "../backend_utils.h" #include "../backend_manager.h" -#include -#include +#include +#include #include "data_ops.h" #include "capabilities.h" #include "utils.h" @@ -29,836 +29,817 @@ namespace onnxruntime { namespace openvino_ep { std::set ops_supported_only_in_model = { - "Cast", - "Concat", - "ConstantOfShape", - "Dropout", - "Expand", - "EyeLike", - "Exp", - "GatherND", - "Identity", - "Loop", - "NonMaxSuppression", - "NonZero", - "Not", - "OneHot", - "Pad", - "Range", - "ReduceMin", - "Resize", - "Round", - "Shape", - "Split", - "Tile", - "TopK" -}; + "Cast", + "Concat", + "ConstantOfShape", + "Dropout", + "Expand", + "EyeLike", + "Exp", + "GatherND", + "Identity", + "Loop", + "NonMaxSuppression", + "NonZero", + "Not", + "OneHot", + "Pad", + "Range", + "ReduceMin", + "Resize", + "Round", + "Shape", + "Split", + "Tile", + "TopK"}; std::vector supported_op_mode = { - {"Abs", V_2020_4,{"CPU", "GPU"}}, - {"Acos", V_2020_4,{"CPU"}}, - {"Acosh", V_2020_4,{"CPU"}}, - {"Add", V_2020_4,{"All"}}, - {"And", V_2020_4,{"All"}}, - {"ArgMax", V_2020_4,{"CPU"}}, - {"ArgMax", V_2021_1,{"All"}}, - {"ArgMin", V_2020_4,{"CPU"}}, - {"ArgMin", V_2021_2,{"CPU","MYRIAD"}}, - {"Asin", V_2020_4,{"CPU", "GPU"}}, - {"Asinh", V_2020_4,{"CPU", "GPU"}}, - {"Atan", V_2020_4,{"CPU", "GPU"}}, - {"Atanh", V_2020_4,{"CPU"}}, - {"AveragePool", V_2020_4,{"All"}}, - {"BatchNormalization", V_2020_4,{"All"}}, - {"Cast", V_2020_4,{"All"}}, - {"Ceil", V_2020_4,{"GPU"}}, - {"Ceil", V_2021_3,{"MYRIAD"}}, - {"Ceil", V_2021_2,{"GPU","MYRIAD"}}, - {"Clip", V_2020_4,{"All"}}, - {"Concat", V_2020_4,{"All"}}, - {"Constant", V_2020_4,{"All"}}, - {"ConstantOfShape", V_2020_4,{"All"}}, - {"Conv", V_2020_4,{"All"}}, - {"ConvTranspose", V_2020_4,{"All"}}, - {"Cos", V_2020_4,{"CPU"}}, - {"Cosh", V_2020_4,{"CPU"}}, - {"DepthToSpace", V_2020_4,{"All"}}, - {"Div", V_2020_4,{"All"}}, - {"Dropout", V_2020_4,{"All"}}, - {"Elu", V_2020_4,{"All"}}, - {"Equal", V_2020_4,{"All"}}, - {"Erf", V_2020_4,{"All"}}, - {"Exp", V_2020_4,{"All"}}, - {"Expand", V_2021_1,{"MYRIAD"}}, - {"Flatten", V_2020_4,{"All"}}, - {"Floor", V_2020_4,{"All"}}, - {"Gather", V_2020_4,{"All"}}, - {"GatherElements", V_2021_3,{"MYRIAD"}}, - {"GatherND", V_2021_2,{"MYRIAD"}}, - {"Gemm", V_2020_4,{"All"}}, - {"GlobalAveragePool", V_2020_4,{"All"}}, - {"GlobalLpPool", V_2020_4,{"CPU", "GPU"}}, - {"Greater", V_2020_4,{"All"}}, - {"Identity", V_2020_4,{"All"}}, - {"InstanceNormalization", V_2020_4,{"All"}}, - {"HardSigmoid", V_2020_4,{"CPU", "GPU"}}, - {"LeakyRelu", V_2020_4,{"All"}}, - {"Less", V_2020_4,{"All"}}, - {"Log", V_2020_4,{"All"}}, - {"Loop", V_2021_3,{"MYRIAD"}}, - {"LRN", V_2020_4,{"All"}}, - {"LSTM", V_2020_4,{"All"}}, - {"MatMul", V_2020_4,{"All"}}, - {"Max", V_2020_4,{"All"}}, - {"MaxPool", V_2020_4,{"All"}}, - {"Mean", V_2020_4,{"All"}}, - {"Min", V_2020_4,{"All"}}, - {"Mul", V_2020_4,{"All"}}, - {"Neg", V_2020_4,{"All"}}, - {"NonMaxSuppression", V_2021_1,{"All"}}, - {"NonZero", V_2021_1,{"CPU", "MYRIAD"}}, - {"Not", V_2021_1,{"All"}}, - {"Not", V_2020_4,{"CPU", "GPU"}}, - {"OneHot", V_2020_4,{"All"}}, - {"Pad", V_2020_4,{"All"}}, - {"Pow", V_2020_4,{"All"}}, - {"PRelu", V_2020_4,{"All"}}, - {"Range", V_2021_2,{"MYRIAD"}}, - {"Reciprocal", V_2020_4,{"All"}}, - {"ReduceLogSum", V_2020_4,{"CPU", "MYRIAD"}}, - {"ReduceMax", V_2020_4 ,{"All"}}, - {"ReduceMean", V_2020_4,{"All"}}, - {"ReduceMin", V_2020_4,{"All"}}, - {"ReduceProd", V_2020_4,{"CPU"}}, - {"ReduceSum", V_2020_4,{"All"}}, - {"ReduceSumSquare", V_2020_4,{"CPU", "MYRIAD"}}, - {"Relu", V_2020_4,{"All"}}, - {"Resize", V_2020_4,{"CPU"}}, - {"Resize", V_2021_3,{"MYRIAD"}}, - {"Reshape", V_2020_4,{"All"}}, - {"RoiAlign", V_2021_1,{"All"}}, - {"Round", V_2021_2,{"MYRIAD"}}, - {"Scatter", V_2021_1,{"MYRIAD"}}, - {"ScatterElements", V_2021_2,{"MYRIAD"}}, - {"Selu", V_2020_4,{"CPU", "GPU"}}, - {"Shape", V_2020_4,{"All"}}, - {"Sigmoid", V_2020_4,{"All"}}, - {"Sign", V_2020_4,{"CPU"}}, - {"Sign", V_2020_4,{"CPU"}}, - {"Sinh", V_2020_4,{"CPU"}}, - {"SinFloat", V_2020_4,{"MYRIAD"}}, - {"Slice", V_2020_4,{"All"}}, - {"Softmax", V_2020_4,{"All"}}, - {"SpaceToDepth", V_2020_4,{"All"}}, - {"Split", V_2020_4,{"All"}}, - {"Sqrt", V_2020_4,{"All"}}, - {"Squeeze", V_2020_4,{"All"}}, - {"Softsign", V_2020_4,{"CPU"}}, - {"Sub", V_2020_4,{"All"}}, - {"Sum", V_2020_4,{"All"}}, - {"Tan", V_2020_4,{"CPU", "GPU"}}, - {"Tanh", V_2020_4,{"All"}}, - {"Tile", V_2021_3,{"MYRIAD"}}, - {"Transpose", V_2020_4,{"All"}}, - {"TopK", V_2020_4,{"All"}}, - {"Unsqueeze", V_2020_4,{"All"}}, - {"Upsample", V_2021_1,{"CPU"}}, - {"Where", V_2021_2,{"MYRIAD"}}, + {"Abs", V_2020_4, {"CPU", "GPU"}}, + {"Acos", V_2020_4, {"CPU"}}, + {"Acosh", V_2020_4, {"CPU"}}, + {"Add", V_2020_4, {"All"}}, + {"And", V_2020_4, {"All"}}, + {"ArgMax", V_2020_4, {"CPU"}}, + {"ArgMax", V_2021_1, {"All"}}, + {"ArgMin", V_2020_4, {"CPU"}}, + {"ArgMin", V_2021_2, {"CPU", "MYRIAD"}}, + {"Asin", V_2020_4, {"CPU", "GPU"}}, + {"Asinh", V_2020_4, {"CPU", "GPU"}}, + {"Atan", V_2020_4, {"CPU", "GPU"}}, + {"Atanh", V_2020_4, {"CPU"}}, + {"AveragePool", V_2020_4, {"All"}}, + {"BatchNormalization", V_2020_4, {"All"}}, + {"Cast", V_2020_4, {"All"}}, + {"Ceil", V_2020_4, {"GPU"}}, + {"Ceil", V_2021_3, {"MYRIAD"}}, + {"Ceil", V_2021_2, {"GPU", "MYRIAD"}}, + {"Clip", V_2020_4, {"All"}}, + {"Concat", V_2020_4, {"All"}}, + {"Constant", V_2020_4, {"All"}}, + {"ConstantOfShape", V_2020_4, {"All"}}, + {"Conv", V_2020_4, {"All"}}, + {"ConvTranspose", V_2020_4, {"All"}}, + {"Cos", V_2020_4, {"CPU"}}, + {"Cosh", V_2020_4, {"CPU"}}, + {"DepthToSpace", V_2020_4, {"All"}}, + {"Div", V_2020_4, {"All"}}, + {"Dropout", V_2020_4, {"All"}}, + {"Elu", V_2020_4, {"All"}}, + {"Equal", V_2020_4, {"All"}}, + {"Erf", V_2020_4, {"All"}}, + {"Exp", V_2020_4, {"All"}}, + {"Expand", V_2021_1, {"MYRIAD"}}, + {"Flatten", V_2020_4, {"All"}}, + {"Floor", V_2020_4, {"All"}}, + {"Gather", V_2020_4, {"All"}}, + {"GatherElements", V_2021_3, {"MYRIAD"}}, + {"GatherND", V_2021_2, {"MYRIAD"}}, + {"Gemm", V_2020_4, {"All"}}, + {"GlobalAveragePool", V_2020_4, {"All"}}, + {"GlobalLpPool", V_2020_4, {"CPU", "GPU"}}, + {"Greater", V_2020_4, {"All"}}, + {"Identity", V_2020_4, {"All"}}, + {"InstanceNormalization", V_2020_4, {"All"}}, + {"HardSigmoid", V_2020_4, {"CPU", "GPU"}}, + {"LeakyRelu", V_2020_4, {"All"}}, + {"Less", V_2020_4, {"All"}}, + {"Log", V_2020_4, {"All"}}, + {"Loop", V_2021_3, {"MYRIAD"}}, + {"LRN", V_2020_4, {"All"}}, + {"LSTM", V_2020_4, {"All"}}, + {"MatMul", V_2020_4, {"All"}}, + {"Max", V_2020_4, {"All"}}, + {"MaxPool", V_2020_4, {"All"}}, + {"Mean", V_2020_4, {"All"}}, + {"Min", V_2020_4, {"All"}}, + {"Mul", V_2020_4, {"All"}}, + {"Neg", V_2020_4, {"All"}}, + {"NonMaxSuppression", V_2021_1, {"All"}}, + {"NonZero", V_2021_1, {"CPU", "MYRIAD"}}, + {"Not", V_2021_1, {"All"}}, + {"Not", V_2020_4, {"CPU", "GPU"}}, + {"OneHot", V_2020_4, {"All"}}, + {"Pad", V_2020_4, {"All"}}, + {"Pow", V_2020_4, {"All"}}, + {"PRelu", V_2020_4, {"All"}}, + {"Range", V_2021_2, {"MYRIAD"}}, + {"Reciprocal", V_2020_4, {"All"}}, + {"ReduceLogSum", V_2020_4, {"CPU", "MYRIAD"}}, + {"ReduceMax", V_2020_4, {"All"}}, + {"ReduceMean", V_2020_4, {"All"}}, + {"ReduceMin", V_2020_4, {"All"}}, + {"ReduceProd", V_2020_4, {"CPU"}}, + {"ReduceSum", V_2020_4, {"All"}}, + {"ReduceSumSquare", V_2020_4, {"CPU", "MYRIAD"}}, + {"Relu", V_2020_4, {"All"}}, + {"Resize", V_2020_4, {"CPU"}}, + {"Resize", V_2021_3, {"MYRIAD"}}, + {"Reshape", V_2020_4, {"All"}}, + {"RoiAlign", V_2021_1, {"All"}}, + {"Round", V_2021_2, {"MYRIAD"}}, + {"Scatter", V_2021_1, {"MYRIAD"}}, + {"ScatterElements", V_2021_2, {"MYRIAD"}}, + {"Selu", V_2020_4, {"CPU", "GPU"}}, + {"Shape", V_2020_4, {"All"}}, + {"Sigmoid", V_2020_4, {"All"}}, + {"Sign", V_2020_4, {"CPU"}}, + {"Sign", V_2020_4, {"CPU"}}, + {"Sinh", V_2020_4, {"CPU"}}, + {"SinFloat", V_2020_4, {"MYRIAD"}}, + {"Slice", V_2020_4, {"All"}}, + {"Softmax", V_2020_4, {"All"}}, + {"SpaceToDepth", V_2020_4, {"All"}}, + {"Split", V_2020_4, {"All"}}, + {"Sqrt", V_2020_4, {"All"}}, + {"Squeeze", V_2020_4, {"All"}}, + {"Softsign", V_2020_4, {"CPU"}}, + {"Sub", V_2020_4, {"All"}}, + {"Sum", V_2020_4, {"All"}}, + {"Tan", V_2020_4, {"CPU", "GPU"}}, + {"Tanh", V_2020_4, {"All"}}, + {"Tile", V_2021_3, {"MYRIAD"}}, + {"Transpose", V_2020_4, {"All"}}, + {"TopK", V_2020_4, {"All"}}, + {"Unsqueeze", V_2020_4, {"All"}}, + {"Upsample", V_2021_1, {"CPU"}}, + {"Where", V_2021_2, {"MYRIAD"}}, }; void DataOps::populate_types_supported() { + supported_types_initializer_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_BOOL)); + supported_types_initializer_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_FLOAT)); + supported_types_initializer_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_INT32)); + supported_types_initializer_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_INT64)); + supported_types_initializer_.insert(std::make_pair(V_2021_1, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_FLOAT16)); - supported_types_initializer_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_BOOL)); - supported_types_initializer_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_FLOAT)); - supported_types_initializer_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_INT32)); - supported_types_initializer_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_INT64)); - supported_types_initializer_.insert(std::make_pair(V_2021_1, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_FLOAT16)); + supported_types_vpu_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_BOOL)); + supported_types_vpu_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_FLOAT)); + supported_types_vpu_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_UINT8)); + supported_types_vpu_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_INT8)); + supported_types_vpu_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_INT16)); + supported_types_vpu_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_INT32)); + supported_types_vpu_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_INT64)); + supported_types_vpu_.insert(std::make_pair(V_2021_1, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_FLOAT16)); - supported_types_vpu_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_BOOL)); - supported_types_vpu_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_FLOAT)); - supported_types_vpu_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_UINT8)); - supported_types_vpu_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_INT8)); - supported_types_vpu_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_INT16)); - supported_types_vpu_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_INT32)); - supported_types_vpu_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_INT64)); - supported_types_vpu_.insert(std::make_pair(V_2021_1, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_FLOAT16)); + supported_types_cpu_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_BOOL)); + supported_types_cpu_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_FLOAT)); + supported_types_cpu_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_INT32)); + supported_types_cpu_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_INT16)); + supported_types_cpu_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_INT8)); + supported_types_cpu_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_UINT8)); + supported_types_cpu_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_INT64)); + supported_types_cpu_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_UINT8)); + supported_types_cpu_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_INT64)); - supported_types_cpu_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_BOOL)); - supported_types_cpu_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_FLOAT)); - supported_types_cpu_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_INT32)); - supported_types_cpu_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_INT16)); - supported_types_cpu_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_INT8)); - supported_types_cpu_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_UINT8)); - supported_types_cpu_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_INT64)); - supported_types_cpu_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_UINT8)); - supported_types_cpu_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_INT64)); - - supported_types_gpu_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_FLOAT)); - supported_types_gpu_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_INT32)); - supported_types_gpu_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_INT64)); - supported_types_gpu_.insert(std::make_pair(V_2021_1, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_FLOAT16)); + supported_types_gpu_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_FLOAT)); + supported_types_gpu_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_INT32)); + supported_types_gpu_.insert(std::make_pair(V_2020_4, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_INT64)); + supported_types_gpu_.insert(std::make_pair(V_2021_1, ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_FLOAT16)); } void DataOps::populate_op_mode_supported() { + no_dimension_supported_.push_back({"Unsqueeze", V_2020_4, {"All"}}); + no_dimension_supported_.push_back({"Squeeze", V_2020_4, {"All"}}); + no_dimension_supported_.push_back({"Cast", V_2020_4, {"All"}}); + no_dimension_supported_.push_back({"Gather", V_2020_4, {"All"}}); + no_dimension_supported_.push_back({"Mul", V_2020_4, {"All"}}); + no_dimension_supported_.push_back({"Sub", V_2020_4, {"All"}}); + no_dimension_supported_.push_back({"Min", V_2020_4, {"All"}}); + no_dimension_supported_.push_back({"Div", V_2020_4, {"All"}}); + no_dimension_supported_.push_back({"Floor", V_2020_4, {"All"}}); + no_dimension_supported_.push_back({"Where", V_2021_2, {"All"}}); + no_dimension_supported_.push_back({"Range", V_2021_2, {"All"}}); + no_dimension_supported_.push_back({"ArgMin", V_2021_2, {"Myriad"}}); + no_dimension_supported_.push_back({"Max", V_2021_2, {"Myriad"}}); + no_dimension_supported_.push_back({"Add", V_2021_2, {"Myriad"}}); + no_dimension_supported_.push_back({"Less", V_2021_2, {"Myriad"}}); + no_dimension_supported_.push_back({"Greater", V_2021_2, {"Myriad"}}); + no_dimension_supported_.push_back({"Clip", V_2021_2, {"Myriad"}}); + no_dimension_supported_.push_back({"Resize", V_2021_2, {"Myriad"}}); + no_dimension_supported_.push_back({"Equal", V_2021_2, {"Myriad"}}); - no_dimension_supported_.push_back({"Unsqueeze", V_2020_4,{"All"}}); - no_dimension_supported_.push_back({"Squeeze", V_2020_4,{"All"}}); - no_dimension_supported_.push_back({"Cast", V_2020_4,{"All"}}); - no_dimension_supported_.push_back({"Gather", V_2020_4,{"All"}}); - no_dimension_supported_.push_back({"Mul", V_2020_4,{"All"}}); - no_dimension_supported_.push_back({"Sub", V_2020_4,{"All"}}); - no_dimension_supported_.push_back({"Min", V_2020_4,{"All"}}); - no_dimension_supported_.push_back({"Div", V_2020_4,{"All"}}); - no_dimension_supported_.push_back({"Floor", V_2020_4,{"All"}}); - no_dimension_supported_.push_back({"Where", V_2021_2,{"All"}}); - no_dimension_supported_.push_back({"Range", V_2021_2,{"All"}}); - no_dimension_supported_.push_back({"ArgMin", V_2021_2,{"MYRIAD"}}); - no_dimension_supported_.push_back({"Max", V_2021_2,{"MYRIAD"}}); - no_dimension_supported_.push_back({"Add", V_2021_2,{"MYRIAD"}}); - no_dimension_supported_.push_back({"Less", V_2021_2,{"MYRIAD"}}); - no_dimension_supported_.push_back({"Greater", V_2021_2,{"MYRIAD"}}); - no_dimension_supported_.push_back({"Clip", V_2021_2,{"MYRIAD"}}); - no_dimension_supported_.push_back({"Resize", V_2021_2,{"MYRIAD"}}); - no_dimension_supported_.push_back({"Equal", V_2021_2,{"MYRIAD"}}); - no_dimension_supported_.push_back({"Reshape", V_2021_3,{"MYRIAD"}}); - no_dimension_supported_.push_back({"Ceil", V_2021_3,{"MYRIAD"}}); - no_dimension_supported_.push_back({"Loop", V_2021_3,{"MYRIAD"}}); - no_dimension_supported_.push_back({"ReduceMin", V_2021_3,{"MYRIAD"}}); + no_dimension_supported_.push_back({"Unsqueeze", V_2020_4, {"All"}}); + no_dimension_supported_.push_back({"Squeeze", V_2020_4, {"All"}}); + no_dimension_supported_.push_back({"Cast", V_2020_4, {"All"}}); + no_dimension_supported_.push_back({"Gather", V_2020_4, {"All"}}); + no_dimension_supported_.push_back({"Mul", V_2020_4, {"All"}}); + no_dimension_supported_.push_back({"Sub", V_2020_4, {"All"}}); + no_dimension_supported_.push_back({"Min", V_2020_4, {"All"}}); + no_dimension_supported_.push_back({"Div", V_2020_4, {"All"}}); + no_dimension_supported_.push_back({"Floor", V_2020_4, {"All"}}); + no_dimension_supported_.push_back({"Where", V_2021_2, {"All"}}); + no_dimension_supported_.push_back({"Range", V_2021_2, {"All"}}); + no_dimension_supported_.push_back({"ArgMin", V_2021_2, {"MYRIAD"}}); + no_dimension_supported_.push_back({"Max", V_2021_2, {"MYRIAD"}}); + no_dimension_supported_.push_back({"Add", V_2021_2, {"MYRIAD"}}); + no_dimension_supported_.push_back({"Less", V_2021_2, {"MYRIAD"}}); + no_dimension_supported_.push_back({"Greater", V_2021_2, {"MYRIAD"}}); + no_dimension_supported_.push_back({"Clip", V_2021_2, {"MYRIAD"}}); + no_dimension_supported_.push_back({"Resize", V_2021_2, {"MYRIAD"}}); + no_dimension_supported_.push_back({"Equal", V_2021_2, {"MYRIAD"}}); + no_dimension_supported_.push_back({"Reshape", V_2021_3, {"MYRIAD"}}); + no_dimension_supported_.push_back({"Ceil", V_2021_3, {"MYRIAD"}}); + no_dimension_supported_.push_back({"Loop", V_2021_3, {"MYRIAD"}}); + no_dimension_supported_.push_back({"ReduceMin", V_2021_3, {"MYRIAD"}}); + subgraph_supported_.push_back({"Mul", V_2020_4, {"All"}}); + subgraph_supported_.push_back({"Transpose", V_2020_4, {"All"}}); + subgraph_supported_.push_back({"Unsqueeze", V_2020_4, {"All"}}); + subgraph_supported_.push_back({"Cast", V_2020_4, {"All"}}); + subgraph_supported_.push_back({"Concat", V_2020_4, {"All"}}); + subgraph_supported_.push_back({"Gather", V_2020_4, {"All"}}); + subgraph_supported_.push_back({"Div", V_2020_4, {"MYRIAD"}}); + subgraph_supported_.push_back({"Sub", V_2020_4, {"MYRIAD"}}); + subgraph_supported_.push_back({"Identity", V_2021_1, {"CPU"}}); + subgraph_supported_.push_back({"Div", V_2021_1, {"CPU"}}); + subgraph_supported_.push_back({"Sub", V_2021_1, {"CPU"}}); - - subgraph_supported_.push_back({"Mul", V_2020_4,{"All"}}); - subgraph_supported_.push_back({"Transpose", V_2020_4,{"All"}}); - subgraph_supported_.push_back({"Unsqueeze", V_2020_4,{"All"}}); - subgraph_supported_.push_back({"Cast", V_2020_4,{"All"}}); - subgraph_supported_.push_back({"Concat", V_2020_4,{"All"}}); - subgraph_supported_.push_back({"Gather", V_2020_4,{"All"}}); - subgraph_supported_.push_back({"Div", V_2020_4,{"MYRIAD"}}); - subgraph_supported_.push_back({"Sub", V_2020_4,{"MYRIAD"}}); - subgraph_supported_.push_back({"Identity", V_2021_1,{"CPU"}}); - subgraph_supported_.push_back({"Div", V_2021_1,{"CPU"}}); - subgraph_supported_.push_back({"Sub", V_2021_1,{"CPU"}}); - - //populate unsupportedmode_t { - UnsupportedOpMode obj = {{V_2020_4,V_2021_1,V_2021_2, V_2021_3}, - [this](const Node* node, const Provider_InitializedTensorSet&) { - for (size_t i = 0; i < node->InputDefs().size(); i++) { - if (node->InputDefs()[i]->TypeAsProto()->tensor_type().elem_type() != ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_FLOAT) - return true; - } - return false; - } - }; + UnsupportedOpMode obj = {{V_2020_4, V_2021_1, V_2021_2, V_2021_3}, + [this](const Node* node, const InitializedTensorSet&) { + for (size_t i = 0; i < node->InputDefs().size(); i++) { + if (node->InputDefs()[i]->TypeAsProto()->tensor_type().elem_type() != ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_FLOAT) + return true; + } + return false; + }}; op_list_.insert({"Abs", obj}); } { - UnsupportedOpMode obj = {{V_2020_4,V_2021_1,V_2021_2, V_2021_3}, - [this](const Node* node, const Provider_InitializedTensorSet&) { - //tensor type does not support select last index - auto& attributes = node->GetAttributes(); - auto last_index_arg = attributes.count("select_last_index") > 0 ? attributes.at("select_last_index").i() : 0; - if (last_index_arg != 0) - return true; - // tensor type supports float as input for argmax and argmin - if (node->InputDefs()[0]->TypeAsProto()->tensor_type().elem_type() != ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_FLOAT) - return true; - return false; - } - }; + UnsupportedOpMode obj = {{V_2020_4, V_2021_1, V_2021_2, V_2021_3}, + [this](const Node* node, const InitializedTensorSet&) { + //tensor type does not support select last index + auto& attributes = node->GetAttributes(); + auto last_index_arg = attributes.count("select_last_index") > 0 ? attributes.at("select_last_index").i() : 0; + if (last_index_arg != 0) + return true; + // tensor type supports float as input for argmax and argmin + if (node->InputDefs()[0]->TypeAsProto()->tensor_type().elem_type() != ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_FLOAT) + return true; + return false; + }}; op_list_.insert({"ArgMax", obj}); op_list_.insert({"ArgMin", obj}); } { UnsupportedOpMode obj = {{V_2020_4}, - [this](const Node* node, const Provider_InitializedTensorSet&) { - // ceil_mode attribute is not supported in nGraph - const auto& attributes = node->GetAttributes(); - auto ceil_attr = attributes.find("ceil_mode"); - // default value of ceil_mode (0) is supported. - if (ceil_attr != attributes.end() && ceil_attr->second().i() != 0) return true; - return (!dimension_unsupported(node)); - } - }; + [this](const Node* node, const InitializedTensorSet&) { + // ceil_mode attribute is not supported in nGraph + const auto& attributes = node->GetAttributes(); + auto ceil_attr = attributes.find("ceil_mode"); + // default value of ceil_mode (0) is supported. + if (ceil_attr != attributes.end() && ceil_attr->second().i() != 0) return true; + return (!dimension_unsupported(node)); + }}; op_list_.insert({"AveragePool", obj}); } { - UnsupportedOpMode obj = {{V_2021_1,V_2021_2,V_2021_3}, - [this](const Node* node, const Provider_InitializedTensorSet&) { - //auto pad null value is not supported - const auto& attributes = node->GetAttributes(); - auto auto_attr = attributes.find("auto_pad"); - if (auto_attr->second().s() == "") { - return true; - } - // default value of ceil_mode (0) is supported. - auto ceil_attr = attributes.find("ceil_mode"); - if (ceil_attr != attributes.end() && ceil_attr->second().i() != 0) return true; - return (!dimension_unsupported(node)); - } - }; + UnsupportedOpMode obj = {{V_2021_1, V_2021_2, V_2021_3}, + [this](const Node* node, const InitializedTensorSet&) { + //auto pad null value is not supported + const auto& attributes = node->GetAttributes(); + auto auto_attr = attributes.find("auto_pad"); + if (auto_attr->second().s() == "") { + return true; + } + // default value of ceil_mode (0) is supported. + auto ceil_attr = attributes.find("ceil_mode"); + if (ceil_attr != attributes.end() && ceil_attr->second().i() != 0) return true; + return (!dimension_unsupported(node)); + }}; op_list_.insert({"AveragePool", obj}); - } + } { - UnsupportedOpMode obj = {{V_2020_4,V_2021_1,V_2021_2, V_2021_3}, - [this](const Node* node, const Provider_InitializedTensorSet&) { - //Only float 16, float and double data types are supported - const bool data_is_float = node->InputDefs()[0]->Type()->find("float") != std::string::npos; - const bool data_is_float16 = node->InputDefs()[0]->Type()->find("float16") != std::string::npos; - const bool data_is_double = node->InputDefs()[0]->Type()->find("double") != std::string::npos; - return !(data_is_float || data_is_float16 || data_is_double); - } - }; + UnsupportedOpMode obj = {{V_2020_4, V_2021_1, V_2021_2, V_2021_3}, + [this](const Node* node, const InitializedTensorSet&) { + //Only float 16, float and double data types are supported + const bool data_is_float = node->InputDefs()[0]->Type()->find("float") != std::string::npos; + const bool data_is_float16 = node->InputDefs()[0]->Type()->find("float16") != std::string::npos; + const bool data_is_double = node->InputDefs()[0]->Type()->find("double") != std::string::npos; + return !(data_is_float || data_is_float16 || data_is_double); + }}; op_list_.insert({"Clip", obj}); } - { - UnsupportedOpMode obj = {{V_2020_4,V_2021_1,V_2021_2, V_2021_3}, - [this](const Node* node, const Provider_InitializedTensorSet& initializers) { - if (GetInputCount(node, initializers) > 1) - return true; - return false; - } }; + { + UnsupportedOpMode obj = {{V_2020_4, V_2021_1, V_2021_2, V_2021_3}, + [this](const Node* node, const InitializedTensorSet& initializers) { + if (GetInputCount(node, initializers) > 1) + return true; + return false; + }}; op_list_.insert({"Conv", obj}); } - { - UnsupportedOpMode obj = {{V_2020_4,V_2021_1,V_2021_2, V_2021_3}, - [this](const Node* node, const Provider_InitializedTensorSet& initializers) { - if (GetInputCount(node, initializers) > 1) - return true; - return false; - } }; + { + UnsupportedOpMode obj = {{V_2020_4, V_2021_1, V_2021_2, V_2021_3}, + [this](const Node* node, const InitializedTensorSet& initializers) { + if (GetInputCount(node, initializers) > 1) + return true; + return false; + }}; op_list_.insert({"ConvTranspose", obj}); } - { - UnsupportedOpMode obj = {{V_2021_1,V_2021_2, V_2021_3}, - [this](const Node* node, const Provider_InitializedTensorSet& ) { - auto& attributes = node->GetAttributes(); - if (attributes.count("auto_pad") == 0 || attributes.at("auto_pad").s() == "") { - return true; - } - return false; - } }; + { + UnsupportedOpMode obj = {{V_2021_1, V_2021_2, V_2021_3}, + [this](const Node* node, const InitializedTensorSet&) { + auto& attributes = node->GetAttributes(); + if (attributes.count("auto_pad") == 0 || attributes.at("auto_pad").s() == "") { + return true; + } + return false; + }}; op_list_.insert({"Conv", obj}); } - { - UnsupportedOpMode obj = {{V_2021_1,V_2021_2, V_2021_3}, - [this](const Node* node, const Provider_InitializedTensorSet& ) { - auto& attributes = node->GetAttributes(); - if (attributes.count("auto_pad") == 0 || attributes.at("auto_pad").s() == "") { - return true; - } - return false; - } }; - op_list_.insert({"ConvTranspose", obj}); - } - { - UnsupportedOpMode obj = {{V_2021_2}, - [this](const Node* node, const Provider_InitializedTensorSet& ) { - if (device_id_.find("MYRIAD") != std::string::npos) { - const auto& input_arg = node->InputDefs()[0]; - auto shape = input_arg->Shape(); - if ((shape != nullptr) && (shape->dim(0).value_case() != shape->dim(0).kDimValue)) { - return true; - } - } - return false; - } }; + { + UnsupportedOpMode obj = {{V_2021_1, V_2021_2, V_2021_3}, + [this](const Node* node, const InitializedTensorSet&) { + auto& attributes = node->GetAttributes(); + if (attributes.count("auto_pad") == 0 || attributes.at("auto_pad").s() == "") { + return true; + } + return false; + }}; op_list_.insert({"ConvTranspose", obj}); } { - UnsupportedOpMode obj = {{V_2020_4,V_2021_1,V_2021_2,V_2021_3}, - [this](const Node* node, const Provider_InitializedTensorSet& initializers) { - // all ConvInteger zero points need to be constants - if (node->InputDefs().size() == 3) { - return (initializers.find(node->InputDefs()[2]->Name()) == initializers.end()); - } else if (node->InputDefs().size() == 4) { - return initializers.find(node->InputDefs()[2]->Name()) == initializers.end() || - initializers.find(node->InputDefs()[3]->Name()) == initializers.end(); - } - return false; - } - }; - op_list_.insert({"ConvInteger", obj}); + UnsupportedOpMode obj = {{V_2021_2}, + [this](const Node* node, const InitializedTensorSet&) { + if (device_id_.find("MYRIAD") != std::string::npos) { + const auto& input_arg = node->InputDefs()[0]; + auto shape = input_arg->Shape(); + if ((shape != nullptr) && (shape->dim(0).value_case() != shape->dim(0).kDimValue)) { + return true; + } + } + return false; + }}; + op_list_.insert({"ConvTranspose", obj}); } { - UnsupportedOpMode obj = {{V_2020_4,V_2021_1}, - [this](const Node* node, const Provider_InitializedTensorSet&) { - using onnx_dtype = ONNX_NAMESPACE::TensorProto_DataType; - auto supportedOps = std::set>{ - {onnx_dtype::TensorProto_DataType_FLOAT, onnx_dtype::TensorProto_DataType_FLOAT, onnx_dtype::TensorProto_DataType_FLOAT}, - {onnx_dtype::TensorProto_DataType_FLOAT, onnx_dtype::TensorProto_DataType_INT8, onnx_dtype::TensorProto_DataType_FLOAT}, - {onnx_dtype::TensorProto_DataType_FLOAT, onnx_dtype::TensorProto_DataType_FLOAT, onnx_dtype::TensorProto_DataType_INT8}, - {onnx_dtype::TensorProto_DataType_FLOAT, onnx_dtype::TensorProto_DataType_UINT8, onnx_dtype::TensorProto_DataType_FLOAT}, - {onnx_dtype::TensorProto_DataType_FLOAT, onnx_dtype::TensorProto_DataType_FLOAT, onnx_dtype::TensorProto_DataType_UINT8}, - {onnx_dtype::TensorProto_DataType_INT8, onnx_dtype::TensorProto_DataType_INT8, onnx_dtype::TensorProto_DataType_INT8}, - {onnx_dtype::TensorProto_DataType_INT8, onnx_dtype::TensorProto_DataType_INT8, onnx_dtype::TensorProto_DataType_UINT8}, - {onnx_dtype::TensorProto_DataType_INT8, onnx_dtype::TensorProto_DataType_UINT8, onnx_dtype::TensorProto_DataType_INT8}, - {onnx_dtype::TensorProto_DataType_INT32, onnx_dtype::TensorProto_DataType_INT32, onnx_dtype::TensorProto_DataType_INT32}, - {onnx_dtype::TensorProto_DataType_FLOAT, onnx_dtype::TensorProto_DataType_UINT8, onnx_dtype::TensorProto_DataType_FLOAT}, - {onnx_dtype::TensorProto_DataType_FLOAT, onnx_dtype::TensorProto_DataType_FLOAT, onnx_dtype::TensorProto_DataType_UINT8}}; - - if (node->OpType() == "Equal") { - supportedOps.insert(std::vector{onnx_dtype::TensorProto_DataType_UINT8, onnx_dtype::TensorProto_DataType_INT32, onnx_dtype::TensorProto_DataType_INT32}), - supportedOps.insert(std::vector{onnx_dtype::TensorProto_DataType_UINT8, onnx_dtype::TensorProto_DataType_FLOAT, onnx_dtype::TensorProto_DataType_FLOAT}); - } - - onnx_dtype input_0_data_type = (ONNX_NAMESPACE::TensorProto_DataType)node->InputDefs()[0]->TypeAsProto()->tensor_type().elem_type(); - onnx_dtype input_1_data_type = (ONNX_NAMESPACE::TensorProto_DataType)node->InputDefs()[1]->TypeAsProto()->tensor_type().elem_type(); - onnx_dtype output_data_type = (ONNX_NAMESPACE::TensorProto_DataType)node->OutputDefs()[0]->TypeAsProto()->tensor_type().elem_type(); - const std::vector typePair{output_data_type, input_0_data_type, input_1_data_type}; - const auto match = supportedOps.find(typePair); - if (match == supportedOps.end()) - return true; - else - return false; - } - }; - op_list_.insert({"Equal", obj}); - op_list_.insert({"And", obj}); + UnsupportedOpMode obj = {{V_2020_4, V_2021_1, V_2021_2, V_2021_3}, + [this](const Node* node, const InitializedTensorSet& initializers) { + // all ConvInteger zero points need to be constants + if (node->InputDefs().size() == 3) { + return (initializers.find(node->InputDefs()[2]->Name()) == initializers.end()); + } else if (node->InputDefs().size() == 4) { + return initializers.find(node->InputDefs()[2]->Name()) == initializers.end() || + initializers.find(node->InputDefs()[3]->Name()) == initializers.end(); + } + return false; + }}; + op_list_.insert({"ConvInteger", obj}); } { - UnsupportedOpMode obj = {{V_2021_1,V_2021_2, V_2021_3}, - [this](const Node* node, const Provider_InitializedTensorSet& ) { - if (device_id_.find("GPU") != std::string::npos) { - const auto& input = node->InputDefs()[0]; - auto graph_inputs = graph_viewer_.GetInputs(); - auto it = find(graph_inputs.begin(), graph_inputs.end(), input); - if (it != graph_inputs.end()) { - const auto& indices_arg = node->InputDefs()[1]; - if (indices_arg->TypeAsProto()->tensor_type().elem_type() == ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_INT64) - return true; - } - } - return false; - } - }; + UnsupportedOpMode obj = {{V_2020_4, V_2021_1}, + [this](const Node* node, const InitializedTensorSet&) { + using onnx_dtype = ONNX_NAMESPACE::TensorProto_DataType; + auto supportedOps = std::set>{ + {onnx_dtype::TensorProto_DataType_FLOAT, onnx_dtype::TensorProto_DataType_FLOAT, onnx_dtype::TensorProto_DataType_FLOAT}, + {onnx_dtype::TensorProto_DataType_FLOAT, onnx_dtype::TensorProto_DataType_INT8, onnx_dtype::TensorProto_DataType_FLOAT}, + {onnx_dtype::TensorProto_DataType_FLOAT, onnx_dtype::TensorProto_DataType_FLOAT, onnx_dtype::TensorProto_DataType_INT8}, + {onnx_dtype::TensorProto_DataType_FLOAT, onnx_dtype::TensorProto_DataType_UINT8, onnx_dtype::TensorProto_DataType_FLOAT}, + {onnx_dtype::TensorProto_DataType_FLOAT, onnx_dtype::TensorProto_DataType_FLOAT, onnx_dtype::TensorProto_DataType_UINT8}, + {onnx_dtype::TensorProto_DataType_INT8, onnx_dtype::TensorProto_DataType_INT8, onnx_dtype::TensorProto_DataType_INT8}, + {onnx_dtype::TensorProto_DataType_INT8, onnx_dtype::TensorProto_DataType_INT8, onnx_dtype::TensorProto_DataType_UINT8}, + {onnx_dtype::TensorProto_DataType_INT8, onnx_dtype::TensorProto_DataType_UINT8, onnx_dtype::TensorProto_DataType_INT8}, + {onnx_dtype::TensorProto_DataType_INT32, onnx_dtype::TensorProto_DataType_INT32, onnx_dtype::TensorProto_DataType_INT32}, + {onnx_dtype::TensorProto_DataType_FLOAT, onnx_dtype::TensorProto_DataType_UINT8, onnx_dtype::TensorProto_DataType_FLOAT}, + {onnx_dtype::TensorProto_DataType_FLOAT, onnx_dtype::TensorProto_DataType_FLOAT, onnx_dtype::TensorProto_DataType_UINT8}}; + + if (node->OpType() == "Equal") { + supportedOps.insert(std::vector{onnx_dtype::TensorProto_DataType_UINT8, onnx_dtype::TensorProto_DataType_INT32, onnx_dtype::TensorProto_DataType_INT32}), + supportedOps.insert(std::vector{onnx_dtype::TensorProto_DataType_UINT8, onnx_dtype::TensorProto_DataType_FLOAT, onnx_dtype::TensorProto_DataType_FLOAT}); + } + + onnx_dtype input_0_data_type = (ONNX_NAMESPACE::TensorProto_DataType)node->InputDefs()[0]->TypeAsProto()->tensor_type().elem_type(); + onnx_dtype input_1_data_type = (ONNX_NAMESPACE::TensorProto_DataType)node->InputDefs()[1]->TypeAsProto()->tensor_type().elem_type(); + onnx_dtype output_data_type = (ONNX_NAMESPACE::TensorProto_DataType)node->OutputDefs()[0]->TypeAsProto()->tensor_type().elem_type(); + const std::vector typePair{output_data_type, input_0_data_type, input_1_data_type}; + const auto match = supportedOps.find(typePair); + if (match == supportedOps.end()) + return true; + else + return false; + }}; + op_list_.insert({"Equal", obj}); + op_list_.insert({"And", obj}); + } + { + UnsupportedOpMode obj = {{V_2021_1, V_2021_2, V_2021_3}, + [this](const Node* node, const InitializedTensorSet&) { + if (device_id_.find("GPU") != std::string::npos) { + const auto& input = node->InputDefs()[0]; + auto graph_inputs = graph_viewer_.GetInputs(); + auto it = find(graph_inputs.begin(), graph_inputs.end(), input); + if (it != graph_inputs.end()) { + const auto& indices_arg = node->InputDefs()[1]; + if (indices_arg->TypeAsProto()->tensor_type().elem_type() == ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_INT64) + return true; + } + } + return false; + }}; op_list_.insert({"Gather", obj}); } { UnsupportedOpMode obj = {{V_2021_3}, - [this](const Node* node, const Provider_InitializedTensorSet& ) { - const auto& indices_arg = node->InputDefs()[0]; - const auto& output_arg = node->OutputDefs()[0]; - if (indices_arg->TypeAsProto()->tensor_type().elem_type() != output_arg->TypeAsProto()->tensor_type().elem_type()) - return true; - if ((indices_arg->TypeAsProto()->tensor_type().elem_type() == ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_FLOAT16) || - (indices_arg->TypeAsProto()->tensor_type().elem_type() == ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_UINT8) || - (indices_arg->TypeAsProto()->tensor_type().elem_type() == ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_FLOAT)) { - return false; - } - return true; - }}; + [this](const Node* node, const InitializedTensorSet&) { + const auto& indices_arg = node->InputDefs()[0]; + const auto& output_arg = node->OutputDefs()[0]; + if (indices_arg->TypeAsProto()->tensor_type().elem_type() != output_arg->TypeAsProto()->tensor_type().elem_type()) + return true; + if ((indices_arg->TypeAsProto()->tensor_type().elem_type() == ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_FLOAT16) || + (indices_arg->TypeAsProto()->tensor_type().elem_type() == ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_UINT8) || + (indices_arg->TypeAsProto()->tensor_type().elem_type() == ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_FLOAT)) { + return false; + } + return true; + }}; op_list_.insert({"GatherElements", obj}); } { - UnsupportedOpMode obj = {{V_2020_4,V_2021_1,V_2021_2, V_2021_3}, - [this](const Node* node, const Provider_InitializedTensorSet&) { - const auto& input = node->InputDefs()[0]; - const auto& output = node->OutputDefs()[0]; - auto graph_inputs = this->graph_viewer_.GetInputs(); - auto graph_outputs = this->graph_viewer_.GetOutputs(); - auto input_it = find(graph_inputs.begin(), graph_inputs.end(), input); - auto output_it = find(graph_outputs.begin(), graph_outputs.end(), output); - if (input_it != graph_inputs.end() && output_it != graph_outputs.end()) - return true; - return false; - } - }; + UnsupportedOpMode obj = {{V_2020_4, V_2021_1, V_2021_2, V_2021_3}, + [this](const Node* node, const InitializedTensorSet&) { + const auto& input = node->InputDefs()[0]; + const auto& output = node->OutputDefs()[0]; + auto graph_inputs = this->graph_viewer_.GetInputs(); + auto graph_outputs = this->graph_viewer_.GetOutputs(); + auto input_it = find(graph_inputs.begin(), graph_inputs.end(), input); + auto output_it = find(graph_outputs.begin(), graph_outputs.end(), output); + if (input_it != graph_inputs.end() && output_it != graph_outputs.end()) + return true; + return false; + }}; op_list_.insert({"Identity", obj}); } - { + { UnsupportedOpMode obj = {{V_2021_3}, - [this](const Node* node, const Provider_InitializedTensorSet& initializers) { - //Loop has to be initializer - const auto& cond = node->InputDefs()[1]; - return (initializers.find(cond->Name()) == initializers.end()); - } - }; + [this](const Node* node, const InitializedTensorSet& initializers) { + //Loop has to be initializer + const auto& cond = node->InputDefs()[1]; + return (initializers.find(cond->Name()) == initializers.end()); + }}; op_list_.insert({"Loop", obj}); } { - UnsupportedOpMode obj = {{V_2021_3}, - [this](const Node* node, const Provider_InitializedTensorSet&) { - //MaxPool "indices" output is not currently supported. - //if (node->OutputDefs().size() > 1) - // return true; - const auto& attributes = node->GetAttributes(); - /* default value of ceil_mode (0) is supported. + UnsupportedOpMode obj = {{V_2021_3}, + [this](const Node* node, const InitializedTensorSet&) { + //MaxPool "indices" output is not currently supported. + //if (node->OutputDefs().size() > 1) + // return true; + const auto& attributes = node->GetAttributes(); + /* default value of ceil_mode (0) is supported. auto ceil_attr = attributes.find("ceil_mode"); if (ceil_attr != attributes.end() && ceil_attr->second().i() != 0) return true;*/ - auto auto_attr = attributes.find("auto_pad"); - //auto pad null value is not supported - if (auto_attr->second().s() == "") - return true; - // dilations attrs are not supported in nGraph - if (attributes.find("dilations") != attributes.end()) - return true; - return(!this->dimension_unsupported(node)); - } - }; + auto auto_attr = attributes.find("auto_pad"); + //auto pad null value is not supported + if (auto_attr->second().s() == "") + return true; + // dilations attrs are not supported in nGraph + if (attributes.find("dilations") != attributes.end()) + return true; + return (!this->dimension_unsupported(node)); + }}; op_list_.insert({"MaxPool", obj}); } { - UnsupportedOpMode obj = {{V_2020_4,V_2021_1,V_2021_2}, - [this](const Node* node, const Provider_InitializedTensorSet&) { - //MaxPool "indices" output is not currently supported. - if (node->OutputDefs().size() > 1) - return true; - const auto& attributes = node->GetAttributes(); - // default value of ceil_mode (0) is supported. - auto ceil_attr = attributes.find("ceil_mode"); - if (ceil_attr != attributes.end() && ceil_attr->second().i() != 0) - return true; - auto auto_attr = attributes.find("auto_pad"); - //auto pad null value is not supported - if (auto_attr->second().s() == "") - return true; - // dilations attrs are not supported in nGraph - if (attributes.find("dilations") != attributes.end()) - return true; - return(!this->dimension_unsupported(node)); - } - }; + UnsupportedOpMode obj = {{V_2020_4, V_2021_1, V_2021_2}, + [this](const Node* node, const InitializedTensorSet&) { + //MaxPool "indices" output is not currently supported. + if (node->OutputDefs().size() > 1) + return true; + const auto& attributes = node->GetAttributes(); + // default value of ceil_mode (0) is supported. + auto ceil_attr = attributes.find("ceil_mode"); + if (ceil_attr != attributes.end() && ceil_attr->second().i() != 0) + return true; + auto auto_attr = attributes.find("auto_pad"); + //auto pad null value is not supported + if (auto_attr->second().s() == "") + return true; + // dilations attrs are not supported in nGraph + if (attributes.find("dilations") != attributes.end()) + return true; + return (!this->dimension_unsupported(node)); + }}; op_list_.insert({"MaxPool", obj}); } { - UnsupportedOpMode obj = {{V_2021_2, V_2021_3}, - [this](const Node* node, const Provider_InitializedTensorSet& initializers) { - if (device_id_.find("MYRIAD") == std::string::npos) { - if (GetInputCount(node, initializers) == 1) - return true; - } - return false; - } - }; + UnsupportedOpMode obj = {{V_2021_2, V_2021_3}, + [this](const Node* node, const InitializedTensorSet& initializers) { + if (device_id_.find("MYRIAD") == std::string::npos) { + if (GetInputCount(node, initializers) == 1) + return true; + } + return false; + }}; op_list_.insert({"Max", obj}); op_list_.insert({"Min", obj}); op_list_.insert({"Mean", obj}); op_list_.insert({"Sum", obj}); } { - UnsupportedOpMode obj = {{V_2020_4,V_2021_1}, - [this](const Node* node, const Provider_InitializedTensorSet& initializers) { - if (GetInputCount(node, initializers) == 1) - return true; - return false; - } - }; + UnsupportedOpMode obj = {{V_2020_4, V_2021_1}, + [this](const Node* node, const InitializedTensorSet& initializers) { + if (GetInputCount(node, initializers) == 1) + return true; + return false; + }}; op_list_.insert({"Mean", obj}); op_list_.insert({"Sum", obj}); } { - UnsupportedOpMode obj = {{V_2020_4,V_2021_1}, - [this](const Node* node, const Provider_InitializedTensorSet& initializers) { - if (GetInputCount(node, initializers) == 1) - return true; - for (size_t i = 0; i < node->InputDefs().size(); i++) { - auto dtype = node->InputDefs()[i]->TypeAsProto()->tensor_type().elem_type(); - if (dtype == ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_UINT8 || - dtype == ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_INT16) - return true; - } - return false; - } - }; + UnsupportedOpMode obj = {{V_2020_4, V_2021_1}, + [this](const Node* node, const InitializedTensorSet& initializers) { + if (GetInputCount(node, initializers) == 1) + return true; + for (size_t i = 0; i < node->InputDefs().size(); i++) { + auto dtype = node->InputDefs()[i]->TypeAsProto()->tensor_type().elem_type(); + if (dtype == ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_UINT8 || + dtype == ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_INT16) + return true; + } + return false; + }}; op_list_.insert({"Max", obj}); op_list_.insert({"Min", obj}); - } - { - UnsupportedOpMode obj = {{V_2020_4,V_2021_1,V_2021_2, V_2021_3}, - [this](const Node* node, const Provider_InitializedTensorSet&) { - //All matmuls except float have computation missmatch - const bool A_is_float = node->InputDefs()[0]->Type()->find("float") != std::string::npos; - const bool B_is_float = node->InputDefs()[1]->Type()->find("float") != std::string::npos; - return (A_is_float && B_is_float) ? false : true; - } - }; - op_list_.insert({"MatMul", obj}); } - { - UnsupportedOpMode obj = {{V_2020_4,V_2021_1,V_2021_2, V_2021_3}, - [this](const Node* node, const Provider_InitializedTensorSet& initializers) { - // all MatMulInteger zero points need to be constants - if (node->InputDefs().size() == 3) { - // not found in initializers -> not const - return initializers.find(node->InputDefs()[2]->Name()) == initializers.end(); - } else if (node->InputDefs().size() == 4) { - // not found in initializers -> not const - return ((initializers.find(node->InputDefs()[2]->Name()) == initializers.end()) || - (initializers.find(node->InputDefs()[2]->Name()) == initializers.end())) ; - } - return false; - } - }; + { + UnsupportedOpMode obj = {{V_2020_4, V_2021_1, V_2021_2, V_2021_3}, + [this](const Node* node, const InitializedTensorSet&) { + //All matmuls except float have computation missmatch + const bool A_is_float = node->InputDefs()[0]->Type()->find("float") != std::string::npos; + const bool B_is_float = node->InputDefs()[1]->Type()->find("float") != std::string::npos; + return (A_is_float && B_is_float) ? false : true; + }}; + op_list_.insert({"MatMul", obj}); + } + { + UnsupportedOpMode obj = {{V_2020_4, V_2021_1, V_2021_2, V_2021_3}, + [this](const Node* node, const InitializedTensorSet& initializers) { + // all MatMulInteger zero points need to be constants + if (node->InputDefs().size() == 3) { + // not found in initializers -> not const + return initializers.find(node->InputDefs()[2]->Name()) == initializers.end(); + } else if (node->InputDefs().size() == 4) { + // not found in initializers -> not const + return ((initializers.find(node->InputDefs()[2]->Name()) == initializers.end()) || + (initializers.find(node->InputDefs()[2]->Name()) == initializers.end())); + } + return false; + }}; op_list_.insert({"MatMulInteger", obj}); } - { - UnsupportedOpMode obj = {{V_2020_4,V_2021_2}, - [this](const Node* node, const Provider_InitializedTensorSet&) { - //Only FP32 data type is allowed - auto& attributes = node->GetAttributes(); - auto fmod = attributes.count("fmod") > 0 ? attributes.at("fmod").i() : 0; - if (fmod != 1) return true; - //Only FP32 data type is allowed - for (const auto& input : node->InputDefs()) { - if (input->Type()->find("float") == std::string::npos) - return true; - } - return false; - } - }; + { + UnsupportedOpMode obj = {{V_2020_4, V_2021_2}, + [this](const Node* node, const InitializedTensorSet&) { + //Only FP32 data type is allowed + auto& attributes = node->GetAttributes(); + auto fmod = attributes.count("fmod") > 0 ? attributes.at("fmod").i() : 0; + if (fmod != 1) return true; + //Only FP32 data type is allowed + for (const auto& input : node->InputDefs()) { + if (input->Type()->find("float") == std::string::npos) + return true; + } + return false; + }}; op_list_.insert({"Mod", obj}); } { UnsupportedOpMode obj = {{V_2021_2, V_2021_3}, - [this](const Node* node, const Provider_InitializedTensorSet&) { - auto graph_outputs = graph_viewer_.GetOutputs(); - const auto& output = node->OutputDefs()[0]; - auto output_it = find(graph_outputs.begin(), graph_outputs.end(), output); - if (output_it != graph_outputs.end()) - return true; - return false; - } - }; + [this](const Node* node, const InitializedTensorSet&) { + auto graph_outputs = graph_viewer_.GetOutputs(); + const auto& output = node->OutputDefs()[0]; + auto output_it = find(graph_outputs.begin(), graph_outputs.end(), output); + if (output_it != graph_outputs.end()) + return true; + return false; + }}; op_list_.insert({"NonMaxSuppression", obj}); } { - UnsupportedOpMode obj = {{V_2020_4,V_2021_1}, - [this](const Node* node, const Provider_InitializedTensorSet&) { - //Only supported if the data type of both inputs is same - auto x_data_type = node->InputDefs()[0]->TypeAsProto()->tensor_type().elem_type(); - auto y_data_type = node->InputDefs()[1]->TypeAsProto()->tensor_type().elem_type(); - return x_data_type != y_data_type; - } - }; + UnsupportedOpMode obj = {{V_2020_4, V_2021_1}, + [this](const Node* node, const InitializedTensorSet&) { + //Only supported if the data type of both inputs is same + auto x_data_type = node->InputDefs()[0]->TypeAsProto()->tensor_type().elem_type(); + auto y_data_type = node->InputDefs()[1]->TypeAsProto()->tensor_type().elem_type(); + return x_data_type != y_data_type; + }}; op_list_.insert({"Pow", obj}); } { UnsupportedOpMode obj = {{V_2021_2, V_2021_3}, - [this](const Node* node, const Provider_InitializedTensorSet&) { - if (device_id_.find("GPU") != std::string::npos) { - auto x_data_type = node->InputDefs()[0]->TypeAsProto()->tensor_type().elem_type(); - auto y_data_type = node->InputDefs()[1]->TypeAsProto()->tensor_type().elem_type(); - return x_data_type != y_data_type; - } - //currently both inputs with int32 or int64 datatype are not supported - const bool A_is_int32 = node->InputDefs()[0]->Type()->find("int32") != std::string::npos; - const bool B_is_int32 = node->InputDefs()[1]->Type()->find("int32") != std::string::npos; - const bool A_is_int64 = node->InputDefs()[0]->Type()->find("int64") != std::string::npos; - const bool B_is_int64 = node->InputDefs()[1]->Type()->find("int64") != std::string::npos; - if((A_is_int32 && B_is_int32) || (A_is_int64 && B_is_int64)) - return true; - return false; - } - }; + [this](const Node* node, const InitializedTensorSet&) { + if (device_id_.find("GPU") != std::string::npos) { + auto x_data_type = node->InputDefs()[0]->TypeAsProto()->tensor_type().elem_type(); + auto y_data_type = node->InputDefs()[1]->TypeAsProto()->tensor_type().elem_type(); + return x_data_type != y_data_type; + } + //currently both inputs with int32 or int64 datatype are not supported + const bool A_is_int32 = node->InputDefs()[0]->Type()->find("int32") != std::string::npos; + const bool B_is_int32 = node->InputDefs()[1]->Type()->find("int32") != std::string::npos; + const bool A_is_int64 = node->InputDefs()[0]->Type()->find("int64") != std::string::npos; + const bool B_is_int64 = node->InputDefs()[1]->Type()->find("int64") != std::string::npos; + if ((A_is_int32 && B_is_int32) || (A_is_int64 && B_is_int64)) + return true; + return false; + }}; op_list_.insert({"Pow", obj}); } { - UnsupportedOpMode obj = {{V_2020_4,V_2021_1,V_2021_2, V_2021_3}, - [this](const Node* node, const Provider_InitializedTensorSet& initializers) { - auto slope = node->InputDefs()[1]; - //PRelu slope has to be an initializer or needs to come from a constant node - if (initializers.count(slope->Name())) - return false; - else { - for (auto input_node = node->InputNodesBegin(); input_node != node->InputNodesEnd(); ++input_node) { - if (GetInputCount(this->graph_viewer_.GetNode((*input_node).Index()), initializers) == 0) - return false; - } } - return true; - } - }; + UnsupportedOpMode obj = {{V_2020_4, V_2021_1, V_2021_2, V_2021_3}, + [this](const Node* node, const InitializedTensorSet& initializers) { + auto slope = node->InputDefs()[1]; + //PRelu slope has to be an initializer or needs to come from a constant node + if (initializers.count(slope->Name())) + return false; + else { + for (auto input_node = node->InputNodesBegin(); input_node != node->InputNodesEnd(); ++input_node) { + if (GetInputCount(this->graph_viewer_.GetNode((*input_node).Index()), initializers) == 0) + return false; + } + } + return true; + }}; op_list_.insert({"PRelu", obj}); } { - UnsupportedOpMode obj = {{V_2020_4,V_2021_1,V_2021_2, V_2021_3}, - [this](const Node* node, const Provider_InitializedTensorSet& initializers) { - bool non_const_zero_point = false; - // check if any of the zero points is NOT in the initializers list - non_const_zero_point |= initializers.find(node->InputDefs()[2]->Name()) == initializers.end(); - non_const_zero_point |= initializers.find(node->InputDefs()[5]->Name()) == initializers.end(); - non_const_zero_point |= initializers.find(node->InputDefs()[7]->Name()) == initializers.end(); - // QLinearMatMul is not supported if any of the zero points is a dynamic input - return non_const_zero_point; - } - }; + UnsupportedOpMode obj = {{V_2020_4, V_2021_1, V_2021_2, V_2021_3}, + [this](const Node* node, const InitializedTensorSet& initializers) { + bool non_const_zero_point = false; + // check if any of the zero points is NOT in the initializers list + non_const_zero_point |= initializers.find(node->InputDefs()[2]->Name()) == initializers.end(); + non_const_zero_point |= initializers.find(node->InputDefs()[5]->Name()) == initializers.end(); + non_const_zero_point |= initializers.find(node->InputDefs()[7]->Name()) == initializers.end(); + // QLinearMatMul is not supported if any of the zero points is a dynamic input + return non_const_zero_point; + }}; op_list_.insert({"QLinearMatMul", obj}); } { - UnsupportedOpMode obj = {{V_2020_4,V_2021_1}, - [this](const Node* node, const Provider_InitializedTensorSet&) { - //Only FP32, INT32 and U8 data types are supported - const bool data_is_float = node->InputDefs()[0]->Type()->find("float") != std::string::npos; - const bool data_is_int32 = node->InputDefs()[0]->Type()->find("int32") != std::string::npos; - const bool data_is_u8 = node->InputDefs()[0]->Type()->find("uint8") != std::string::npos; - return !(data_is_float || data_is_int32 || data_is_u8); - } - }; - op_list_.insert({"ReduceMin", obj}); + UnsupportedOpMode obj = {{V_2020_4, V_2021_1}, + [this](const Node* node, const InitializedTensorSet&) { + //Only FP32, INT32 and U8 data types are supported + const bool data_is_float = node->InputDefs()[0]->Type()->find("float") != std::string::npos; + const bool data_is_int32 = node->InputDefs()[0]->Type()->find("int32") != std::string::npos; + const bool data_is_u8 = node->InputDefs()[0]->Type()->find("uint8") != std::string::npos; + return !(data_is_float || data_is_int32 || data_is_u8); + }}; + op_list_.insert({"ReduceMin", obj}); } { - UnsupportedOpMode obj = {{V_2020_4,V_2021_1}, - [this](const Node* node, const Provider_InitializedTensorSet&) { - //Resize opset 11 is not supported - if (node->InputDefs().size() > 2) - return true; - return false; - } - }; + UnsupportedOpMode obj = {{V_2020_4, V_2021_1}, + [this](const Node* node, const InitializedTensorSet&) { + //Resize opset 11 is not supported + if (node->InputDefs().size() > 2) + return true; + return false; + }}; op_list_.insert({"Resize", obj}); } { - UnsupportedOpMode obj = {{V_2021_2, V_2021_3}, - [this](const Node* node, const Provider_InitializedTensorSet&) { - const auto& attributes = node->GetAttributes(); - auto axis_attr = attributes.find("axis"); - //Negative axis is not supported - if (axis_attr->second().i() < 0) - return true; - return false; - } - }; + UnsupportedOpMode obj = {{V_2021_2, V_2021_3}, + [this](const Node* node, const InitializedTensorSet&) { + const auto& attributes = node->GetAttributes(); + auto axis_attr = attributes.find("axis"); + //Negative axis is not supported + if (axis_attr->second().i() < 0) + return true; + return false; + }}; op_list_.insert({"Scatter", obj}); - op_list_.insert({"ScatterElements", obj}); + op_list_.insert({"ScatterElements", obj}); } { - UnsupportedOpMode obj = {{V_2020_4,V_2021_1, V_2021_2, V_2021_3}, - [this](const Node* node, const Provider_InitializedTensorSet& initializers) { - //start, end, axes need to be a initializer - bool cond_for_slice = false; - const auto& data_arg = node->InputDefs()[0]; - auto graph_inputs = graph_viewer_.GetInputs(); + UnsupportedOpMode obj = {{V_2020_4, V_2021_1, V_2021_2, V_2021_3}, + [this](const Node* node, const InitializedTensorSet& initializers) { + //start, end, axes need to be a initializer + bool cond_for_slice = false; + const auto& data_arg = node->InputDefs()[0]; + auto graph_inputs = graph_viewer_.GetInputs(); - auto it = find(graph_inputs.begin(), graph_inputs.end(), data_arg); - if (it != graph_inputs.end()) { - if (node->InputDefs().size() > 1) { - const auto& start_arg = node->InputDefs()[1]; - const auto& end_arg = node->InputDefs()[2]; - cond_for_slice |= initializers.find(start_arg->Name()) == initializers.end(); - cond_for_slice |= initializers.find(end_arg->Name()) == initializers.end(); - } - if (node->InputDefs().size() > 3) { - const auto& axes_arg = node->InputDefs()[3]; - cond_for_slice |= initializers.find(axes_arg->Name()) == initializers.end(); - } - } + auto it = find(graph_inputs.begin(), graph_inputs.end(), data_arg); + if (it != graph_inputs.end()) { + if (node->InputDefs().size() > 1) { + const auto& start_arg = node->InputDefs()[1]; + const auto& end_arg = node->InputDefs()[2]; + cond_for_slice |= initializers.find(start_arg->Name()) == initializers.end(); + cond_for_slice |= initializers.find(end_arg->Name()) == initializers.end(); + } + if (node->InputDefs().size() > 3) { + const auto& axes_arg = node->InputDefs()[3]; + cond_for_slice |= initializers.find(axes_arg->Name()) == initializers.end(); + } + } - return cond_for_slice; - } - }; + return cond_for_slice; + }}; op_list_.insert({"Slice", obj}); } { - UnsupportedOpMode obj = {{V_2020_4,V_2021_1,V_2021_2, V_2021_3}, - [this](const Node* node, const Provider_InitializedTensorSet&) { - //Shape can't have empty axes attribute - const auto& attributes = node->GetAttributes(); - if (attributes.count("axes") == 0) - return true; - return false; - } - }; + UnsupportedOpMode obj = {{V_2020_4, V_2021_1, V_2021_2, V_2021_3}, + [this](const Node* node, const InitializedTensorSet&) { + //Shape can't have empty axes attribute + const auto& attributes = node->GetAttributes(); + if (attributes.count("axes") == 0) + return true; + return false; + }}; op_list_.insert({"Squeeze", obj}); } { - UnsupportedOpMode obj = {{V_2020_4}, - [this](const Node* node, const Provider_InitializedTensorSet&) { - return node->InputDefs().size() > 1; - } - }; + UnsupportedOpMode obj = {{V_2020_4}, + [this](const Node* node, const InitializedTensorSet&) { + return node->InputDefs().size() > 1; + }}; op_list_.insert({"TopK", obj}); } { - UnsupportedOpMode obj = {{V_2020_4,V_2021_2, V_2021_3}, - [this](const Node* node, const Provider_InitializedTensorSet&) { - return (!this->dimension_unsupported(node)); - } - }; + UnsupportedOpMode obj = {{V_2020_4, V_2021_2, V_2021_3}, + [this](const Node* node, const InitializedTensorSet&) { + return (!this->dimension_unsupported(node)); + }}; op_list_.insert({"Unsqueeze", obj}); } { - UnsupportedOpMode obj = {{V_2021_1,V_2021_2, V_2021_3}, - [this](const Node* node, const Provider_InitializedTensorSet& ) { - //check for attributes - auto& upsample_attr = node->GetAttributes(); - if (upsample_attr.count("scales") > 0) { - auto& upsample_arg = upsample_attr.at("scales"); - auto float_size = upsample_arg.floats_size(); - if (float_size > 2 && (upsample_arg.floats(0) != 1.f || upsample_arg.floats(1) != 1.f)) { - return true; - } - } + UnsupportedOpMode obj = {{V_2021_1, V_2021_2, V_2021_3}, + [this](const Node* node, const InitializedTensorSet&) { + //check for attributes + auto& upsample_attr = node->GetAttributes(); + if (upsample_attr.count("scales") > 0) { + auto& upsample_arg = upsample_attr.at("scales"); + auto float_size = upsample_arg.floats_size(); + if (float_size > 2 && (upsample_arg.floats(0) != 1.f || upsample_arg.floats(1) != 1.f)) { + return true; + } + } - //check for input dimensions - const auto& x_arg = node->InputDefs()[0]; - auto shape = x_arg->Shape(); - if (shape != nullptr) { - //input tensor rank cannot be of one dimension - if (shape->dim_size() == 1) { - return true; - } - } - // x_arg supports only float, int8 and float16 type - if ((x_arg->TypeAsProto()->tensor_type().elem_type() == ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_FLOAT) || - (x_arg->TypeAsProto()->tensor_type().elem_type() == ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_INT8) || - (x_arg->TypeAsProto()->tensor_type().elem_type() == ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_FLOAT16)) { - return false; - } else { - return true; - } - } - }; + //check for input dimensions + const auto& x_arg = node->InputDefs()[0]; + auto shape = x_arg->Shape(); + if (shape != nullptr) { + //input tensor rank cannot be of one dimension + if (shape->dim_size() == 1) { + return true; + } + } + // x_arg supports only float, int8 and float16 type + if ((x_arg->TypeAsProto()->tensor_type().elem_type() == ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_FLOAT) || + (x_arg->TypeAsProto()->tensor_type().elem_type() == ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_INT8) || + (x_arg->TypeAsProto()->tensor_type().elem_type() == ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_FLOAT16)) { + return false; + } else { + return true; + } + }}; op_list_.insert({"Upsample", obj}); } { UnsupportedOpMode obj = {{V_2021_2}, - [this](const Node* node, const Provider_InitializedTensorSet& ) { - //float data type is not supported - const bool data_is_float = node->InputDefs()[1]->Type()->find("float") != std::string::npos; - return data_is_float; - } }; + [this](const Node* node, const InitializedTensorSet&) { + //float data type is not supported + const bool data_is_float = node->InputDefs()[1]->Type()->find("float") != std::string::npos; + return data_is_float; + }}; op_list_.insert({"Where", obj}); } { UnsupportedOpMode obj = {{V_2021_3}, - [this](const Node* node, const Provider_InitializedTensorSet&) { - return (!this->dimension_unsupported(node)); - } - }; + [this](const Node* node, const InitializedTensorSet&) { + return (!this->dimension_unsupported(node)); + }}; op_list_.insert({"ReduceSum", obj}); } -} +} bool DataOps::op_is_supported(std::string name, std::vector& op_list) { - for (size_t i=0; i < op_list.size(); i++) { + for (size_t i = 0; i < op_list.size(); i++) { if (op_list[i].optype == name) { if (op_list[i].version <= version_id_) { auto it = op_list[i].device_type.begin(); while (it != op_list[i].device_type.end()) { - - //if device supported is all then we support it - if(*it == "All") { + //if device supported is all then we support it + if (*it == "All") { return true; - } - + } + //check for device supported if (device_id_.find(*it) != std::string::npos) { return true; } - + it++; - } + } } } } - + return false; } bool DataOps::type_is_supported(const NodeArg* node_arg, bool is_initializer) { - const auto* type_proto = node_arg->TypeAsProto(); if (!type_proto) { return false; @@ -866,13 +847,13 @@ bool DataOps::type_is_supported(const NodeArg* node_arg, bool is_initializer) { if (is_initializer) { auto dtype = type_proto->tensor_type().elem_type(); - for (auto const &var : supported_types_initializer_) { - if ((var.first <= version_id_) && + for (auto const& var : supported_types_initializer_) { + if ((var.first <= version_id_) && (var.second == dtype)) { return true; } } - + #ifndef NDEBUG if (openvino_ep::backend_utils::IsDebugEnabled()) { std::cout << "Initializer Data Type is not supported" << std::endl; @@ -880,55 +861,53 @@ bool DataOps::type_is_supported(const NodeArg* node_arg, bool is_initializer) { #endif return false; } else { - auto dtype = type_proto->tensor_type().elem_type(); if (device_id_ == "MYRIAD" || device_id_ == "HDDL" || device_id_.find("HETERO") != std::string::npos || device_id_.find("MULTI") != std::string::npos) { - for (auto const &var : supported_types_vpu_) { - if ((var.first <= version_id_) && + for (auto const& var : supported_types_vpu_) { + if ((var.first <= version_id_) && (var.second == dtype)) { return true; } } - + #ifndef NDEBUG - if (openvino_ep::backend_utils::IsDebugEnabled()) { - std::cout << "I/O data type is not supported" << std::endl; - } + if (openvino_ep::backend_utils::IsDebugEnabled()) { + std::cout << "I/O data type is not supported" << std::endl; + } #endif - return false; - + return false; + } else if (device_id_ == "CPU") { - for (auto const &var : supported_types_cpu_) { - if ((var.first <= version_id_) && + for (auto const& var : supported_types_cpu_) { + if ((var.first <= version_id_) && (var.second == dtype)) { return true; } } - #ifndef NDEBUG - if (openvino_ep::backend_utils::IsDebugEnabled()) { - std::cout << "I/O data type is not supported" << std::endl; - } - #endif - return false; - +#ifndef NDEBUG + if (openvino_ep::backend_utils::IsDebugEnabled()) { + std::cout << "I/O data type is not supported" << std::endl; + } +#endif + return false; + } else if (device_id_ == "GPU") { auto prec_str = openvino_ep::BackendManager::GetGlobalContext().precision_str; if (prec_str == "FP32" && dtype == ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_FLOAT16) return false; - for (auto const &var : supported_types_gpu_) { - if ((var.first <= version_id_) && + for (auto const& var : supported_types_gpu_) { + if ((var.first <= version_id_) && (var.second == dtype)) { return true; } } - #ifndef NDEBUG - if (openvino_ep::backend_utils::IsDebugEnabled()) { - std::cout << "I/O data type is not supported" << std::endl; - } - #endif - return false; - +#ifndef NDEBUG + if (openvino_ep::backend_utils::IsDebugEnabled()) { + std::cout << "I/O data type is not supported" << std::endl; + } +#endif + return false; } return true; } @@ -940,7 +919,7 @@ bool DataOps::unsupported_op_mode(const Node* node) { const auto& initializers = graph_viewer_.GetAllInitializedTensors(); auto iter = op_list_.equal_range(optype); - for (auto it=iter.first; it!=iter.second; ++it) { + for (auto it = iter.first; it != iter.second; ++it) { auto ob = it->second; if (std::find(ob.ver.begin(), ob.ver.end(), version_id_) != ob.ver.end()) { return ob.func(node, initializers); @@ -980,7 +959,6 @@ bool DataOps::dimension_unsupported(const Node* node) { bool DataOps::node_is_supported(const std::map>& op_map, const NodeIndex node_idx) { - const auto& node = graph_viewer_.GetNode(node_idx); const auto& optype = node->OpType(); @@ -1085,27 +1063,25 @@ bool DataOps::node_is_supported(const std::map DataOps::GetUnsupportedNodeIndices(std::unordered_set& ng_required_initializers) { +std::vector DataOps::GetUnsupportedNodeIndices(std::unordered_set& ng_required_initializers) { + const auto ng_supported_ops = GetNgSupportedOps(GetOnnxOpSet(graph_viewer_)); - const auto ng_supported_ops = GetNgSupportedOps(GetOnnxOpSet(graph_viewer_)); + std::vector unsupported_nodes_idx; - std::vector unsupported_nodes_idx; - - for (const auto& node_idx : graph_viewer_.GetNodesInTopologicalOrder()) { - if (node_is_supported(ng_supported_ops, node_idx)) { - // Collect inputs that are initializers - graph_viewer_.GetNode(node_idx)->ForEachDef([&ng_required_initializers, this](const NodeArg& node_arg, bool is_input) { + for (const auto& node_idx : graph_viewer_.GetNodesInTopologicalOrder()) { + if (node_is_supported(ng_supported_ops, node_idx)) { + // Collect inputs that are initializers + graph_viewer_.GetNode(node_idx)->ForEachDef([&ng_required_initializers, this](const NodeArg& node_arg, bool is_input) { if(is_input && this->graph_viewer_.GetAllInitializedTensors().count(node_arg.Name())) { ng_required_initializers.insert(node_arg.Name()); } }, true); - } else { - unsupported_nodes_idx.push_back(node_idx); - } + } else { + unsupported_nodes_idx.push_back(node_idx); } - return unsupported_nodes_idx; + } + return unsupported_nodes_idx; } bool DataOps::IsOpSupportedOnlyInModel(std::string name) { @@ -1113,65 +1089,65 @@ bool DataOps::IsOpSupportedOnlyInModel(std::string name) { } bool DataOps::SpecialConditionForClusterSizeOne(std::unordered_set& ng_required_initializers, const Node* node) { - if (node->OpType() == "Where") { - if (device_id_.find("MYRIAD") != std::string::npos) { - if (node->InputDefs()[1]->TypeAsProto()->tensor_type().elem_type() == ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_FLOAT) - return true; + if (node->OpType() == "Where") { + if (device_id_.find("MYRIAD") != std::string::npos) { + if (node->InputDefs()[1]->TypeAsProto()->tensor_type().elem_type() == ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_FLOAT) + return true; + } + } else if (node->OpType() == "Reshape") { + const auto& shape_arg = node->InputDefs()[1]; + if (ng_required_initializers.find(shape_arg->Name()) == ng_required_initializers.end()) { + return true; + } + } else if (node->OpType() == "Expand") { + // nGraph only supports constant shape input values + const auto& output = node->OutputDefs()[0]; + if (output->TypeAsProto()->tensor_type().elem_type() != ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_FLOAT16) + return true; + } else if (node->OpType() == "RoiAlign") { + using onnx_dtype = ONNX_NAMESPACE::TensorProto_DataType; + + onnx_dtype input_0_data_type = (ONNX_NAMESPACE::TensorProto_DataType)node->InputDefs()[0]->TypeAsProto()->tensor_type().elem_type(); + onnx_dtype input_1_data_type = (ONNX_NAMESPACE::TensorProto_DataType)node->InputDefs()[1]->TypeAsProto()->tensor_type().elem_type(); + onnx_dtype input_2_data_type = (ONNX_NAMESPACE::TensorProto_DataType)node->InputDefs()[2]->TypeAsProto()->tensor_type().elem_type(); + onnx_dtype output_data_type = (ONNX_NAMESPACE::TensorProto_DataType)node->OutputDefs()[0]->TypeAsProto()->tensor_type().elem_type(); + + if ((input_0_data_type != onnx_dtype::TensorProto_DataType_FLOAT16) || + (input_1_data_type != onnx_dtype::TensorProto_DataType_FLOAT16) || + (input_2_data_type != onnx_dtype::TensorProto_DataType_FLOAT) || + (output_data_type != onnx_dtype::TensorProto_DataType_FLOAT16)) + return true; + } else if ((node->OpType() == "Greater") || (node->OpType() == "Less")) { + if (device_id_.find("MYRIAD") != std::string::npos) { + auto input_0_data_type = (ONNX_NAMESPACE::TensorProto_DataType)node->InputDefs()[0]->TypeAsProto()->tensor_type().elem_type(); + auto input_1_data_type = (ONNX_NAMESPACE::TensorProto_DataType)node->InputDefs()[1]->TypeAsProto()->tensor_type().elem_type(); + auto output_data_type = (ONNX_NAMESPACE::TensorProto_DataType)node->OutputDefs()[0]->TypeAsProto()->tensor_type().elem_type(); + + if (!((output_data_type == ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_FLOAT) || + (output_data_type == ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_FLOAT16))) { + return true; } - } else if (node->OpType() == "Reshape") { - const auto& shape_arg = node->InputDefs()[1]; - if (ng_required_initializers.find(shape_arg->Name()) == ng_required_initializers.end()) { - return true; - } - } else if (node->OpType() == "Expand") { - // nGraph only supports constant shape input values - const auto& output = node->OutputDefs()[0]; - if (output->TypeAsProto()->tensor_type().elem_type() != ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_FLOAT16) - return true; - } else if (node->OpType() == "RoiAlign") { - using onnx_dtype = ONNX_NAMESPACE::TensorProto_DataType; - onnx_dtype input_0_data_type = (ONNX_NAMESPACE::TensorProto_DataType)node->InputDefs()[0]->TypeAsProto()->tensor_type().elem_type(); - onnx_dtype input_1_data_type = (ONNX_NAMESPACE::TensorProto_DataType)node->InputDefs()[1]->TypeAsProto()->tensor_type().elem_type(); - onnx_dtype input_2_data_type = (ONNX_NAMESPACE::TensorProto_DataType)node->InputDefs()[2]->TypeAsProto()->tensor_type().elem_type(); - onnx_dtype output_data_type = (ONNX_NAMESPACE::TensorProto_DataType)node->OutputDefs()[0]->TypeAsProto()->tensor_type().elem_type(); - - if ((input_0_data_type != onnx_dtype::TensorProto_DataType_FLOAT16) || - (input_1_data_type != onnx_dtype::TensorProto_DataType_FLOAT16) || - (input_2_data_type != onnx_dtype::TensorProto_DataType_FLOAT) || - (output_data_type != onnx_dtype::TensorProto_DataType_FLOAT16)) - return true; - } else if ((node->OpType() == "Greater") || (node->OpType() == "Less")) { - if (device_id_.find("MYRIAD") != std::string::npos) { - auto input_0_data_type = (ONNX_NAMESPACE::TensorProto_DataType)node->InputDefs()[0]->TypeAsProto()->tensor_type().elem_type(); - auto input_1_data_type = (ONNX_NAMESPACE::TensorProto_DataType)node->InputDefs()[1]->TypeAsProto()->tensor_type().elem_type(); - auto output_data_type = (ONNX_NAMESPACE::TensorProto_DataType)node->OutputDefs()[0]->TypeAsProto()->tensor_type().elem_type(); - - if (!((output_data_type == ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_FLOAT) || - (output_data_type == ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_FLOAT16))) { - return true; - } - - if ((input_0_data_type != ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_FLOAT16) || - (input_1_data_type != ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_FLOAT16)) { - return true; - } - } - } else if (node->OpType() == "MaxPool" && device_id_.find("MYRIAD") != std::string::npos) { - auto output_data_type = node->OutputDefs()[0]->TypeAsProto()->tensor_type().elem_type(); - if (output_data_type != ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_FLOAT || - output_data_type != ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_FLOAT16) { + if ((input_0_data_type != ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_FLOAT16) || + (input_1_data_type != ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_FLOAT16)) { return true; } } - return false; + } else if (node->OpType() == "MaxPool" && device_id_.find("MYRIAD") != std::string::npos) { + auto output_data_type = node->OutputDefs()[0]->TypeAsProto()->tensor_type().elem_type(); + if (output_data_type != ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_FLOAT || + output_data_type != ONNX_NAMESPACE::TensorProto_DataType::TensorProto_DataType_FLOAT16) { + return true; + } + } + return false; } bool DataOps::DoNotOmitSubGraph(const std::string& name) { return op_is_supported(name, subgraph_supported_); } -bool DataOps::InsertNode(const Node *node, const std::string& optype) { +bool DataOps::InsertNode(const Node* node, const std::string& optype) { if (optype == "TopK" || optype == "NonZero") { return true; } @@ -1187,4 +1163,4 @@ bool DataOps::InsertNode(const Node *node, const std::string& optype) { } } // namespace openvino_ep -} // namespace onnxruntime \ No newline at end of file +} // namespace onnxruntime diff --git a/onnxruntime/core/providers/openvino/ov_versions/data_ops.h b/onnxruntime/core/providers/openvino/ov_versions/data_ops.h index f2acfe64fd..90f1e1e9de 100644 --- a/onnxruntime/core/providers/openvino/ov_versions/data_ops.h +++ b/onnxruntime/core/providers/openvino/ov_versions/data_ops.h @@ -4,9 +4,9 @@ namespace onnxruntime { namespace openvino_ep { -using VarianceFunc = std::function; +using VarianceFunc = std::function; -enum versionNum{ +enum versionNum { V_2020_4, V_2021_1, V_2021_2, @@ -18,46 +18,44 @@ using VersionNum = enum versionNum; struct supportedOp { std::string optype; VersionNum version; - std::vector device_type; + std::vector device_type; }; -struct unsupportedOpMode{ +struct unsupportedOpMode { std::vector ver; - VarianceFunc func; + VarianceFunc func; }; using SupportedOp = struct supportedOp; using UnsupportedOpMode = struct unsupportedOpMode; -using Pairs = std::pair; +using Pairs = std::pair; -class DataOps{ +class DataOps { + private: + const GraphViewer& graph_viewer_; + VersionNum version_id_; + std::string device_id_; + std::multimap op_list_; + std::vector subgraph_supported_; + std::vector no_dimension_supported_; + std::set supported_types_vpu_; + std::set supported_types_cpu_; + std::set supported_types_gpu_; + std::set supported_types_initializer_; -private: -const GraphViewer& graph_viewer_; -VersionNum version_id_; -std::string device_id_; -std::multimap op_list_; -std::vector subgraph_supported_; -std::vector no_dimension_supported_; -std::set supported_types_vpu_; -std::set supported_types_cpu_; -std::set supported_types_gpu_; -std::set supported_types_initializer_; - -protected: + protected: virtual void populate_op_mode_supported(); virtual void populate_types_supported(); bool op_is_supported(std::string name, std::vector& list); bool dimension_unsupported(const Node* node); bool unsupported_op_mode(const Node* node); bool type_is_supported(const NodeArg* node_arg, bool is_initializer); - bool node_is_supported(const std::map>& op_map, + bool node_is_supported(const std::map>& op_map, const NodeIndex node_idx); - -public: - DataOps(const GraphViewer& graph_viewer_param, VersionNum ver, std::string dev_id): - graph_viewer_(graph_viewer_param), version_id_(ver), device_id_(dev_id) { + + public: + DataOps(const GraphViewer& graph_viewer_param, VersionNum ver, std::string dev_id) : graph_viewer_(graph_viewer_param), version_id_(ver), device_id_(dev_id) { populate_op_mode_supported(); populate_types_supported(); } @@ -68,7 +66,6 @@ public: virtual bool DoNotOmitSubGraph(const std::string& name); virtual bool InsertNode(const Node* node, const std::string& name); VersionNum GetVersion() const { return version_id_; } - }; } //namespace openvino_ep diff --git a/onnxruntime/core/providers/openvino/ov_versions/utils.cc b/onnxruntime/core/providers/openvino/ov_versions/utils.cc index bc252db08f..2ca9f03817 100644 --- a/onnxruntime/core/providers/openvino/ov_versions/utils.cc +++ b/onnxruntime/core/providers/openvino/ov_versions/utils.cc @@ -21,7 +21,7 @@ namespace onnxruntime { namespace openvino_ep { //Gets the input count of given node -int GetInputCount(const Node* node, const Provider_InitializedTensorSet& initializer_set) { +int GetInputCount(const Node* node, const InitializedTensorSet& initializer_set) { int count = 0; for (const auto& input : node->InputDefs()) { auto name = input->Name(); @@ -175,7 +175,7 @@ void GetInputsOutputsOfCluster(const GraphViewer& graph_viewer, // Collect all inputs and outputs node->ForEachDef( [&input_args, &ordered_input_args, &output_args](const NodeArg& node_arg, bool is_input) { - if(node_arg.Name() != ""){ + if (node_arg.Name() != "") { if (is_input) { if (!input_args.count(node_arg.Name())) { ordered_input_args.push_back(node_arg.Name()); diff --git a/onnxruntime/core/providers/openvino/ov_versions/utils.h b/onnxruntime/core/providers/openvino/ov_versions/utils.h index 51b957aed7..0d923fe39b 100644 --- a/onnxruntime/core/providers/openvino/ov_versions/utils.h +++ b/onnxruntime/core/providers/openvino/ov_versions/utils.h @@ -4,7 +4,7 @@ namespace onnxruntime { namespace openvino_ep { -int GetInputCount(const Node* node, const Provider_InitializedTensorSet& initializer_set); +int GetInputCount(const Node* node, const InitializedTensorSet& initializer_set); bool IsOpSupportedOnlyInModel(std::string name); diff --git a/onnxruntime/core/providers/shared_library/provider_api.h b/onnxruntime/core/providers/shared_library/provider_api.h index 25702b7d12..3c58150e9f 100644 --- a/onnxruntime/core/providers/shared_library/provider_api.h +++ b/onnxruntime/core/providers/shared_library/provider_api.h @@ -7,12 +7,39 @@ // switching providers to be runnable as shared libraries. The interfaces will become more tightly integrated into the core code. #pragma once +// ROCM uses the CUDA provider's files, which are shared provider files. This 'fakes them out' and makes them be non shared provider files if they're being built as part of ROCM. +#ifdef USE_ROCM +#include "core/providers/common.h" +#include "core/providers/cpu/tensor/onehot.h" +#include "core/providers/cpu/tensor/gather_elements.h" + +namespace onnxruntime { +// The ROCM version of this just deletes on destruction, but is drop in compatible with the regular DeleteOnUnloadPtr +template +struct DeleteOnUnloadPtr { + DeleteOnUnloadPtr(T* p) : p_(p) {} + ~DeleteOnUnloadPtr() { delete p_; } + + T& operator*() { return *p_; } + const T& operator*() const { return *p_; } + + operator T*() { + return p_; + } + + private: + T* p_; +}; +} // namespace onnxruntime +#else #define SHARED_PROVIDER 1 #include #include #include +#include #include +#include #include "onnx/common/stl_backports.h" #include "core/common/common.h" #include "core/common/const_pointer_container.h" @@ -91,12 +118,14 @@ struct AttributeProto; struct GraphProto; struct ModelProto; struct NodeProto; +struct SparseTensorProto; struct TensorProto; struct TensorProtos; // RepeatedPtrField struct TensorShapeProto_Dimension; struct TensorShapeProto_Dimensions; // RepeatedPtrField struct TensorShapeProto; struct TypeProto_Tensor; +struct TypeProto_SparseTensor; struct TypeProto; struct ValueInfoProto; struct ValueInfoProtos; // RepeatedPtrField @@ -112,13 +141,6 @@ enum class DataType { } // namespace logging -enum class AutoPadType { - NOTSET = 0, - VALID = 1, - SAME_UPPER = 2, - SAME_LOWER = 3, -}; - // OnnxRuntime Types (these are the internal types) struct CPUIDInfo; namespace logging { @@ -141,21 +163,46 @@ struct Path; struct Node; struct NodeArg; struct NodeAttributes; +class OpKernel; struct OpKernelContext; struct OpKernelInfo; struct PrimitiveDataTypeBase; struct Tensor; +class UnsqueezeBase; +class SliceBase; +class SplitBase; +class Size; +class ScatterNDBase; +enum class Mode : int; +class GatherBase; +class ConcatBase; +template +class Scan; +struct EinsumComputePreprocessor; +template +struct EinsumTypedComputeProcessor; + class DataTypeImpl; using MLDataType = const DataTypeImpl*; +// be used with class MLValue +using DeleteFunc = void (*)(void*); using NodeArgInfo = ONNX_NAMESPACE::ValueInfoProto; + +using NameMLValMap = std::unordered_map; } // namespace onnxruntime +#include "core/platform/threadpool.h" +#include "core/providers/cpu/math/einsum_utils/einsum_compute_preprocessor.h" #include "core/framework/data_transfer.h" #include "core/framework/execution_provider.h" #include "provider_interfaces.h" #include "core/framework/op_kernel.h" #include "core/framework/data_types_internal.h" +#include "core/framework/tensorprotoutils.h" +#include "core/providers/common.h" +#include "core/providers/op_kernel_type_control_utils.h" +#include "core/util/math.h" namespace onnxruntime { @@ -172,6 +219,9 @@ struct DeleteOnUnloadPtr { }); } + T& operator*() { return *p_; } + const T& operator*() const { return *p_; } + operator T*() { return p_; } @@ -183,19 +233,15 @@ struct DeleteOnUnloadPtr { constexpr const char* kOnnxDomain = ""; constexpr const char* kMSDomain = "com.microsoft"; constexpr const char* kNGraphDomain = "com.intel.ai"; +constexpr const char* kCudaExecutionProvider = "CUDAExecutionProvider"; constexpr const char* kDnnlExecutionProvider = "DnnlExecutionProvider"; constexpr const char* kOpenVINOExecutionProvider = "OpenVINOExecutionProvider"; constexpr const char* kTensorrtExecutionProvider = "TensorrtExecutionProvider"; -enum CUDAStreamType : int { - kCudaStreamDefault = 0, - kCudaStreamCopyIn, - kCudaStreamCopyOut, - kTotalCudaStreams, -}; - template -using IAllocatorUniquePtr = std::unique_ptr>; +using IAllocatorUniquePtr = std::unique_ptr >; + +inline OrtStatus* CreateStatus(OrtErrorCode code, _In_ const char* msg) noexcept { return g_host->CreateStatus(code, msg); } std::unique_ptr CreateCPUAllocator(const OrtMemoryInfo& memory_info); std::unique_ptr CreateCUDAAllocator(int16_t device_id, const char* name); @@ -203,13 +249,13 @@ std::unique_ptr CreateCUDAPinnedAllocator(int16_t device_id, const c std::unique_ptr CreateGPUDataTransfer(void* stream); +std::unordered_set GetCpuPreferredNodes(const onnxruntime::GraphViewer& graph, + const std::string& provider_type, + const std::vector& kernel_registries, + const std::vector& tentative_nodes); + std::string GetEnvironmentVar(const std::string& var_name); -inline AutoPadType StringToAutoPadType(const std::string& str) { return g_host->StringToAutoPadType(str); } - -void AllocatorManager__InsertAllocator(AllocatorManager* p, AllocatorPtr allocator); -AllocatorPtr AllocatorManager__GetAllocator(AllocatorManager* p, int id, OrtMemType mem_type); - namespace logging { struct Category { @@ -218,16 +264,40 @@ struct Category { } // namespace logging -namespace math { +namespace utils { -// Rounds a up to the next highest multiple of b, which is power-of-2. User must be careful -// to ensure that there is no overflow or underflow in the calculation -// of divUp. -template -constexpr T roundUpPow2(T a) { - return (a + (b - 1)) & (~(b - 1)); -} -} // namespace math +template +constexpr ONNXTensorElementDataType GetONNXTensorElementDataType() { return ONNX_TENSOR_ELEMENT_DATA_TYPE_UNDEFINED; } +template <> +constexpr ONNXTensorElementDataType GetONNXTensorElementDataType() { return ONNX_TENSOR_ELEMENT_DATA_TYPE_BOOL; } +template <> +constexpr ONNXTensorElementDataType GetONNXTensorElementDataType() { return ONNX_TENSOR_ELEMENT_DATA_TYPE_STRING; } +template <> +constexpr ONNXTensorElementDataType GetONNXTensorElementDataType() { return ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT; } +template <> +constexpr ONNXTensorElementDataType GetONNXTensorElementDataType() { return ONNX_TENSOR_ELEMENT_DATA_TYPE_DOUBLE; } +template <> +constexpr ONNXTensorElementDataType GetONNXTensorElementDataType() { return ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT16; } +template <> +constexpr ONNXTensorElementDataType GetONNXTensorElementDataType() { return ONNX_TENSOR_ELEMENT_DATA_TYPE_BFLOAT16; } +template <> +constexpr ONNXTensorElementDataType GetONNXTensorElementDataType() { return ONNX_TENSOR_ELEMENT_DATA_TYPE_INT8; } +template <> +constexpr ONNXTensorElementDataType GetONNXTensorElementDataType() { return ONNX_TENSOR_ELEMENT_DATA_TYPE_UINT8; } +template <> +constexpr ONNXTensorElementDataType GetONNXTensorElementDataType() { return ONNX_TENSOR_ELEMENT_DATA_TYPE_INT16; } +template <> +constexpr ONNXTensorElementDataType GetONNXTensorElementDataType() { return ONNX_TENSOR_ELEMENT_DATA_TYPE_UINT16; } +template <> +constexpr ONNXTensorElementDataType GetONNXTensorElementDataType() { return ONNX_TENSOR_ELEMENT_DATA_TYPE_INT32; } +template <> +constexpr ONNXTensorElementDataType GetONNXTensorElementDataType() { return ONNX_TENSOR_ELEMENT_DATA_TYPE_UINT32; } +template <> +constexpr ONNXTensorElementDataType GetONNXTensorElementDataType() { return ONNX_TENSOR_ELEMENT_DATA_TYPE_INT64; } +template <> +constexpr ONNXTensorElementDataType GetONNXTensorElementDataType() { return ONNX_TENSOR_ELEMENT_DATA_TYPE_UINT64; } + +} // namespace utils } // namespace onnxruntime @@ -244,3 +314,5 @@ constexpr T roundUpPow2(T a) { #define LOGS_DEFAULT(severity) \ LOGS_DEFAULT_CATEGORY(severity, ::onnxruntime::logging::Category::onnxruntime) + +#endif diff --git a/onnxruntime/core/providers/shared_library/provider_bridge_provider.cc b/onnxruntime/core/providers/shared_library/provider_bridge_provider.cc index d6945d2f81..058211379d 100644 --- a/onnxruntime/core/providers/shared_library/provider_bridge_provider.cc +++ b/onnxruntime/core/providers/shared_library/provider_bridge_provider.cc @@ -8,19 +8,49 @@ #include #include "core/providers/shared/common.h" +#include "core/framework/random_generator.h" +#include "core/providers/cpu/controlflow/if.h" +#include "core/providers/cpu/controlflow/loop.h" +#include "core/providers/cpu/controlflow/scan.h" +#include "core/providers/cpu/math/einsum.h" +#include "core/providers/cpu/object_detection/non_max_suppression.h" +#include "core/providers/cpu/tensor/concatbase.h" +#include "core/providers/cpu/tensor/padbase.h" +#include "core/providers/cpu/tensor/gatherbase.h" +#include "core/providers/cpu/tensor/slice.h" +#include "core/providers/cpu/tensor/split.h" +#include "core/providers/cpu/tensor/size.h" +#include "core/providers/cpu/tensor/scatter_nd.h" +#include "core/providers/cpu/tensor/unsqueeze.h" +#include "core/providers/cpu/tensor/tile.h" + +#ifndef DISABLE_CONTRIB_OPS +#include "contrib_ops/cpu/bert/attention_base.h" +#include "contrib_ops/cpu/bert/bias_gelu_helper.h" +#include "contrib_ops/cpu/bert/embed_layer_norm_helper.h" +#include "contrib_ops/cpu/bert/longformer_attention_base.h" +#endif + +#ifdef ENABLE_TRAINING +#include "orttraining/training_ops/cpu/aten_ops/aten_op.h" +#include "orttraining/training_ops/cpu/controlflow/group.h" +#include "orttraining/training_ops/cpu/controlflow/yield.h" +#endif + #ifndef _Ret_notnull_ #define _Ret_notnull_ #endif - #ifndef _Post_writable_byte_size_ #define _Post_writable_byte_size_(n) #endif +#ifdef _WIN32 // Override default new/delete so that we match the host's allocator _Ret_notnull_ _Post_writable_byte_size_(n) void* operator new(size_t n) { return Provider_GetHost()->HeapAllocate(n); } void operator delete(void* p) noexcept { return Provider_GetHost()->HeapFree(p); } -void operator delete(void* p, size_t /*size*/) { return Provider_GetHost()->HeapFree(p); } +void operator delete(void* p, size_t /*size*/) noexcept { return Provider_GetHost()->HeapFree(p); } +#endif namespace onnxruntime { @@ -50,27 +80,75 @@ struct OnUnload { } g_on_unload; +void* CPUAllocator::Alloc(size_t size) { return g_host->CPUAllocator__Alloc(this, size); } +void CPUAllocator::Free(void* p) { g_host->CPUAllocator__Free(this, p); } + AllocatorPtr CreateAllocator(const AllocatorCreationInfo& info) { return g_host->CreateAllocator(info); } -void AllocatorManager__InsertAllocator(AllocatorManager* p, AllocatorPtr allocator) { - return g_host->AllocatorManager__InsertAllocator(p, allocator); +void AllocatorManager::InsertAllocator(AllocatorPtr allocator) { + return g_host->AllocatorManager__InsertAllocator(this, allocator); } -AllocatorPtr AllocatorManager__GetAllocator(AllocatorManager* p, int id, OrtMemType mem_type) { - return g_host->AllocatorManager__GetAllocator(p, id, mem_type); +AllocatorPtr AllocatorManager::GetAllocator(int id, OrtMemType mem_type) const { + return g_host->AllocatorManager__GetAllocator(this, id, mem_type); } template <> -MLDataType DataTypeImpl::GetType() { - return g_host->DataTypeImpl_GetType_float(); -} - +MLDataType DataTypeImpl::GetType() { return Provider_GetHost()->DataTypeImpl__GetType_Tensor(); } template <> -MLDataType DataTypeImpl::GetTensorType() { - return g_host->DataTypeImpl_GetTensorType_float(); -} +MLDataType DataTypeImpl::GetType() { return Provider_GetHost()->DataTypeImpl__GetType_bool(); } +template <> +MLDataType DataTypeImpl::GetType() { return Provider_GetHost()->DataTypeImpl__GetType_int8(); } +template <> +MLDataType DataTypeImpl::GetType() { return Provider_GetHost()->DataTypeImpl__GetType_uint8(); } +template <> +MLDataType DataTypeImpl::GetType() { return Provider_GetHost()->DataTypeImpl__GetType_int16(); } +template <> +MLDataType DataTypeImpl::GetType() { return Provider_GetHost()->DataTypeImpl__GetType_uint16(); } +template <> +MLDataType DataTypeImpl::GetType() { return Provider_GetHost()->DataTypeImpl__GetType_int32(); } +template <> +MLDataType DataTypeImpl::GetType() { return Provider_GetHost()->DataTypeImpl__GetType_uint32(); } +template <> +MLDataType DataTypeImpl::GetType() { return Provider_GetHost()->DataTypeImpl__GetType_int64(); } +template <> +MLDataType DataTypeImpl::GetType() { return Provider_GetHost()->DataTypeImpl__GetType_uint64(); } +template <> +MLDataType DataTypeImpl::GetType() { return Provider_GetHost()->DataTypeImpl__GetType_float(); } +template <> +MLDataType DataTypeImpl::GetType() { return Provider_GetHost()->DataTypeImpl__GetType_double(); } +template <> +MLDataType DataTypeImpl::GetType() { return Provider_GetHost()->DataTypeImpl__GetType_BFloat16(); } +template <> +MLDataType DataTypeImpl::GetType() { return Provider_GetHost()->DataTypeImpl__GetType_MLFloat16(); } +template <> +MLDataType DataTypeImpl::GetTensorType() { return Provider_GetHost()->DataTypeImpl__GetTensorType_bool(); } +template <> +MLDataType DataTypeImpl::GetTensorType() { return Provider_GetHost()->DataTypeImpl__GetTensorType_int8(); } +template <> +MLDataType DataTypeImpl::GetTensorType() { return Provider_GetHost()->DataTypeImpl__GetTensorType_uint8(); } +template <> +MLDataType DataTypeImpl::GetTensorType() { return Provider_GetHost()->DataTypeImpl__GetTensorType_int16(); } +template <> +MLDataType DataTypeImpl::GetTensorType() { return Provider_GetHost()->DataTypeImpl__GetTensorType_uint16(); } +template <> +MLDataType DataTypeImpl::GetTensorType() { return Provider_GetHost()->DataTypeImpl__GetTensorType_int32(); } +template <> +MLDataType DataTypeImpl::GetTensorType() { return Provider_GetHost()->DataTypeImpl__GetTensorType_uint32(); } +template <> +MLDataType DataTypeImpl::GetTensorType() { return Provider_GetHost()->DataTypeImpl__GetTensorType_int64(); } +template <> +MLDataType DataTypeImpl::GetTensorType() { return Provider_GetHost()->DataTypeImpl__GetTensorType_uint64(); } +template <> +MLDataType DataTypeImpl::GetTensorType() { return Provider_GetHost()->DataTypeImpl__GetTensorType_float(); } +template <> +MLDataType DataTypeImpl::GetTensorType() { return Provider_GetHost()->DataTypeImpl__GetTensorType_double(); } +template <> +MLDataType DataTypeImpl::GetTensorType() { return Provider_GetHost()->DataTypeImpl__GetTensorType_BFloat16(); } +template <> +MLDataType DataTypeImpl::GetTensorType() { return Provider_GetHost()->DataTypeImpl__GetTensorType_MLFloat16(); } Status IDataTransfer::CopyTensor(const Tensor& src, Tensor& dst) const { return g_host->IDataTransfer__CopyTensor(this, src, dst); @@ -80,6 +158,8 @@ Status IDataTransfer::CopyTensors(const std::vector& src_dst_pairs) return g_host->IDataTransfer__CopyTensors(this, src_dst_pairs); } +const Node& OpKernel::Node() const { return g_host->OpKernel__Node(this); } + TensorShape::TensorShape(const int64_t* dimension_sizes, size_t dimension_count) : std::vector(dimension_count) { for (size_t i = 0; i < dimension_count; ++i) { @@ -115,6 +195,11 @@ std::string TensorShape::ToString() const { return g_host->TensorShape__ToString(this); } +int64_t TensorShape::SizeToDimension(size_t dimension) const { return g_host->TensorShape__SizeToDimension(this, dimension); } +int64_t TensorShape::SizeFromDimension(size_t dimension) const { return g_host->TensorShape__SizeFromDimension(this, dimension); } + +std::ostream& operator<<(std::ostream& out, const TensorShape& shape) { return g_host->operator_left_shift(out, shape); } + AllocatorPtr CreateAllocator(AllocatorCreationInfo info) { return g_host->CreateAllocator(info); } @@ -185,6 +270,13 @@ std::string GetEnvironmentVar(const std::string& var_name) { return g_host->GetEnvironmentVar(var_name); } +std::unordered_set GetCpuPreferredNodes(const onnxruntime::GraphViewer& graph, + const std::string& provider_type, + const std::vector& kernel_registries, + const std::vector& tentative_nodes) { + return g_host->GetCpuPreferredNodes(graph, provider_type, kernel_registries, tentative_nodes); +} + namespace logging { const char* Category::onnxruntime = "onnxruntime"; @@ -227,6 +319,16 @@ const std::string& Status::EmptyString() noexcept { } // namespace common +namespace math { +uint16_t floatToHalf(float f) { return g_host->math__floatToHalf(f); } +float halfToFloat(uint16_t h) { return g_host->math__halfToFloat(h); } + +} // namespace math + +float MLFloat16::ToFloat() const { + return math::halfToFloat(val); +} + std::vector GetStackTrace() { return g_host->GetStackTrace(); } void LogRuntimeError(uint32_t session_id, const common::Status& status, @@ -238,4 +340,140 @@ std::unique_ptr CopyOpKernelInfo(const OpKernelInfo& info) { return g_host->CopyOpKernelInfo(info); } +namespace utils { +template <> +Status UnpackTensor(const ONNX_NAMESPACE::TensorProto& tensor, const void* raw_data, size_t raw_data_len, /*out*/ bool* p_data, size_t expected_size) { return g_host->UnpackTensor(tensor, raw_data, raw_data_len, p_data, expected_size); } +template <> +Status UnpackTensor(const ONNX_NAMESPACE::TensorProto& tensor, const void* raw_data, size_t raw_data_len, /*out*/ float* p_data, size_t expected_size) { return g_host->UnpackTensor(tensor, raw_data, raw_data_len, p_data, expected_size); } +template <> +Status UnpackTensor(const ONNX_NAMESPACE::TensorProto& tensor, const void* raw_data, size_t raw_data_len, /*out*/ double* p_data, size_t expected_size) { return g_host->UnpackTensor(tensor, raw_data, raw_data_len, p_data, expected_size); } +template <> +Status UnpackTensor(const ONNX_NAMESPACE::TensorProto& tensor, const void* raw_data, size_t raw_data_len, /*out*/ MLFloat16* p_data, size_t expected_size) { return g_host->UnpackTensor(tensor, raw_data, raw_data_len, p_data, expected_size); } +template <> +Status UnpackTensor(const ONNX_NAMESPACE::TensorProto& tensor, const void* raw_data, size_t raw_data_len, /*out*/ int8_t* p_data, size_t expected_size) { return g_host->UnpackTensor(tensor, raw_data, raw_data_len, p_data, expected_size); } +template <> +Status UnpackTensor(const ONNX_NAMESPACE::TensorProto& tensor, const void* raw_data, size_t raw_data_len, /*out*/ uint8_t* p_data, size_t expected_size) { return g_host->UnpackTensor(tensor, raw_data, raw_data_len, p_data, expected_size); } +template <> +Status UnpackTensor(const ONNX_NAMESPACE::TensorProto& tensor, const void* raw_data, size_t raw_data_len, /*out*/ int16_t* p_data, size_t expected_size) { return g_host->UnpackTensor(tensor, raw_data, raw_data_len, p_data, expected_size); } +template <> +Status UnpackTensor(const ONNX_NAMESPACE::TensorProto& tensor, const void* raw_data, size_t raw_data_len, /*out*/ uint16_t* p_data, size_t expected_size) { return g_host->UnpackTensor(tensor, raw_data, raw_data_len, p_data, expected_size); } +template <> +Status UnpackTensor(const ONNX_NAMESPACE::TensorProto& tensor, const void* raw_data, size_t raw_data_len, /*out*/ int32_t* p_data, size_t expected_size) { return g_host->UnpackTensor(tensor, raw_data, raw_data_len, p_data, expected_size); } +template <> +Status UnpackTensor(const ONNX_NAMESPACE::TensorProto& tensor, const void* raw_data, size_t raw_data_len, /*out*/ uint32_t* p_data, size_t expected_size) { return g_host->UnpackTensor(tensor, raw_data, raw_data_len, p_data, expected_size); } +template <> +Status UnpackTensor(const ONNX_NAMESPACE::TensorProto& tensor, const void* raw_data, size_t raw_data_len, /*out*/ int64_t* p_data, size_t expected_size) { return g_host->UnpackTensor(tensor, raw_data, raw_data_len, p_data, expected_size); } +template <> +Status UnpackTensor(const ONNX_NAMESPACE::TensorProto& tensor, const void* raw_data, size_t raw_data_len, /*out*/ uint64_t* p_data, size_t expected_size) { return g_host->UnpackTensor(tensor, raw_data, raw_data_len, p_data, expected_size); } + +} // namespace utils + +#ifdef USE_CUDA +bool TileOp::IsTileMemcpy(const TensorShape& input_shape, const int64_t* repeats, size_t rank, bool& is_batched_memcpy, size_t& num_of_elements_per_batch, size_t& num_of_copies_per_batch, size_t& num_of_batch_copies) { + return g_host->TileOp__IsTileMemcpy(input_shape, repeats, rank, is_batched_memcpy, num_of_elements_per_batch, num_of_copies_per_batch, num_of_batch_copies); +} + +Status NonMaxSuppressionBase::PrepareCompute(OpKernelContext* ctx, PrepareContext& pc) { return g_host->NonMaxSuppressionBase__PrepareCompute(ctx, pc); } +Status NonMaxSuppressionBase::GetThresholdsFromInputs(const PrepareContext& pc, int64_t& max_output_boxes_per_class, float& iou_threshold, float& score_threshold) { return g_host->NonMaxSuppressionBase__GetThresholdsFromInputs(pc, max_output_boxes_per_class, iou_threshold, score_threshold); } + +Status UnsqueezeBase::PrepareCompute(OpKernelContext* ctx, UnsqueezeBase::Prepare& p) const { return g_host->UnsqueezeBase__PrepareCompute(this, ctx, reinterpret_cast(p)); } + +Status SliceBase::PrepareForCompute(const std::vector& raw_starts, + const std::vector& raw_ends, + const std::vector& raw_axes, + SliceOp::PrepareForComputeMetadata& compute_metadata) { return g_host->SliceBase__PrepareForCompute(raw_starts, raw_ends, raw_axes, reinterpret_cast(compute_metadata)); } + +Status SliceBase::PrepareForCompute(const std::vector& raw_starts, + const std::vector& raw_ends, + const std::vector& raw_axes, + const std::vector& raw_steps, + SliceOp::PrepareForComputeMetadata& compute_metadata) { return g_host->SliceBase__PrepareForCompute(raw_starts, raw_ends, raw_axes, raw_steps, reinterpret_cast(compute_metadata)); } + +Status SliceBase::FillVectorsFromInput(const Tensor& start_tensor, + const Tensor& ends_tensor, + const Tensor* axes_tensor, + const Tensor* steps_tensor, + std::vector& input_starts, + std::vector& input_ends, + std::vector& input_axes, + std::vector& input_steps) { return g_host->SliceBase__FillVectorsFromInput(start_tensor, ends_tensor, axes_tensor, steps_tensor, input_starts, input_ends, input_axes, input_steps); } + +Status SplitBase::PrepareForCompute(const TensorShape& input_shape, int num_outputs, int64_t& axis, int& before_dims, + int& after_dims_including_split_axis, int& after_dims_excluding_split, + std::vector& split_sizes) const { return g_host->SplitBase__PrepareForCompute(this, input_shape, num_outputs, axis, before_dims, after_dims_including_split_axis, after_dims_excluding_split, split_sizes); } + +Status Size::Compute(OpKernelContext* context) const { return g_host->Size__Compute(this, context); } + +Status ScatterNDBase::ValidateShapes(const TensorShape& input_shape, + const TensorShape& indice_shape, + const TensorShape& update_shape) { return g_host->ScatterNDBase__ValidateShapes(input_shape, indice_shape, update_shape); } + +Status PadBase::HandleDimValueZero(const Mode& mode, const TensorShape& input_shape, TensorShape& output_shape) { return g_host->PadBase__HandleDimValueZero(mode, input_shape, output_shape); } + +Status ConcatBase::PrepareForCompute(OpKernelContext* ctx, const std::vector& input_tensors, + Prepare& p) const { return g_host->ConcatBase__PrepareForCompute(this, ctx, input_tensors, p); } + +Status GatherBase::PrepareForCompute(OpKernelContext* context, GatherBase::Prepare& p) const { return g_host->GatherBase__PrepareForCompute(this, context, reinterpret_cast(p)); } + +PhiloxGenerator& PhiloxGenerator::Default() { return g_host->PhiloxGenerator__Default(); } + +Status Einsum::Compute(OpKernelContext* context) const { return g_host->Einsum__Compute(this, context); } + +template <> +std::unique_ptr> EinsumTypedComputeProcessor::Create(OpKernelContext* context, AllocatorPtr allocator, concurrency::ThreadPool* tp, EinsumComputePreprocessor& einsum_compute_preprocessor, void* einsum_cuda_assets) { return g_host->EinsumTypedComputeProcessor_float__Create(context, allocator, tp, einsum_compute_preprocessor, einsum_cuda_assets); } +template <> +std::unique_ptr> EinsumTypedComputeProcessor::Create(OpKernelContext* context, AllocatorPtr allocator, concurrency::ThreadPool* tp, EinsumComputePreprocessor& einsum_compute_preprocessor, void* einsum_cuda_assets) { return g_host->EinsumTypedComputeProcessor_double__Create(context, allocator, tp, einsum_compute_preprocessor, einsum_cuda_assets); } +template <> +std::unique_ptr> EinsumTypedComputeProcessor::Create(OpKernelContext* context, AllocatorPtr allocator, concurrency::ThreadPool* tp, EinsumComputePreprocessor& einsum_compute_preprocessor, void* einsum_cuda_assets) { return g_host->EinsumTypedComputeProcessor_MLFloat16__Create(context, allocator, tp, einsum_compute_preprocessor, einsum_cuda_assets); } + +#ifndef DISABLE_CONTRIB_OPS +namespace contrib { +Status embed_layer_norm::CheckInputs(const OpKernelContext* context) { return g_host->embed_layer_norm__CheckInputs(context); } +Status bias_gelu_helper::CheckInputs(const OpKernelContext* context) { return g_host->bias_gelu_helper__CheckInputs(context); } +Status LongformerAttentionBase::CheckInputs(const TensorShape& input_shape, const TensorShape& weights_shape, const TensorShape& bias_shape, const TensorShape& mask_shape, const TensorShape& global_weights_shape, const TensorShape& global_bias_shape, const TensorShape& global_shape) const { + return g_host->LongformerAttentionBase__CheckInputs(this, input_shape, weights_shape, bias_shape, mask_shape, global_weights_shape, global_bias_shape, global_shape); +} + +Status AttentionBase::CheckInputs(const TensorShape& input_shape, const TensorShape& weights_shape, const TensorShape& bias_shape, const Tensor*& mask_index, const Tensor* past, const int max_threads_per_block) const { + return g_host->AttentionBase__CheckInputs(this, input_shape, weights_shape, bias_shape, mask_index, past, max_threads_per_block); +} +Tensor* AttentionBase::GetPresent(OpKernelContext* context, const Tensor* past, int batch_size, int head_size, int sequence_length, int& past_sequence_length) const { + return g_host->AttentionBase__GetPresent(this, context, past, batch_size, head_size, sequence_length, past_sequence_length); +} + +} // namespace contrib +#endif + +void If::Init(const OpKernelInfo& info) { g_host->If__Init(this, info); } +Status If::Compute(OpKernelContext* ctx) const { return g_host->If__Compute(this, ctx); } +Status If::SetupSubgraphExecutionInfo(const SessionState& session_state, const std::string& attribute_name, const SessionState& subgraph_session_state) { return g_host->If__SetupSubgraphExecutionInfo(this, session_state, attribute_name, subgraph_session_state); } + +void Loop::Init(const OpKernelInfo& info) { g_host->Loop__Init(this, info); } +Status Loop::Compute(OpKernelContext* ctx) const { return g_host->Loop__Compute(this, ctx); } +Status Loop::SetupSubgraphExecutionInfo(const SessionState& session_state, const std::string& attribute_name, const SessionState& subgraph_session_state) { return g_host->Loop__SetupSubgraphExecutionInfo(this, session_state, attribute_name, subgraph_session_state); } + +template <> +void Scan<8>::Init(const OpKernelInfo& info) { g_host->Scan__Init(this, info); } +template <> +void Scan<9>::Init(const OpKernelInfo& info) { g_host->Scan__Init(this, info); } +template <> +Status Scan<8>::Compute(OpKernelContext* ctx) const { return g_host->Scan__Compute(this, ctx); } +template <> +Status Scan<9>::Compute(OpKernelContext* ctx) const { return g_host->Scan__Compute(this, ctx); } +template <> +Status Scan<8>::SetupSubgraphExecutionInfo(const SessionState& session_state, const std::string& attribute_name, const SessionState& subgraph_session_state) { return g_host->Scan__SetupSubgraphExecutionInfo(this, session_state, attribute_name, subgraph_session_state); } +template <> +Status Scan<9>::SetupSubgraphExecutionInfo(const SessionState& session_state, const std::string& attribute_name, const SessionState& subgraph_session_state) { return g_host->Scan__SetupSubgraphExecutionInfo(this, session_state, attribute_name, subgraph_session_state); } + +#ifdef ENABLE_TRAINING +namespace contrib { +void ATenOpBase::Init(const OpKernelInfo& info, bool is_backward) { return g_host->ATenOpBase__Init(this, info, is_backward); } +Status ATenOpBase::Compute(OpKernelContext* p_ctx) const { return g_host->ATenOpBase__Compute(this, p_ctx); } + +Status Group::Compute(OpKernelContext* context) const { return g_host->contrib__Group__Compute(this, context); } +Status PassThrough::Compute(OpKernelContext* context) const { return g_host->contrib__PassThrough__Compute(this, context); } +Status YieldOp::Compute(OpKernelContext* context) const { return g_host->contrib__YieldOp__Compute(this, context); } +} // namespace contrib +#endif +#endif } // namespace onnxruntime diff --git a/onnxruntime/core/providers/shared_library/provider_host_api.h b/onnxruntime/core/providers/shared_library/provider_host_api.h index 8259c1f0a6..ab8d4e1da7 100644 --- a/onnxruntime/core/providers/shared_library/provider_host_api.h +++ b/onnxruntime/core/providers/shared_library/provider_host_api.h @@ -10,7 +10,7 @@ struct Provider { // Old simple device_id API to create provider factories, currently used by DNNL And TensorRT virtual std::shared_ptr CreateExecutionProviderFactory(int /*device_id*/) { return nullptr; } - virtual const void* GetInfo() { return nullptr; } // Returns a provider specific information interface if it exists + virtual void* GetInfo() { return nullptr; } // Returns a provider specific information interface if it exists virtual void Shutdown() = 0; }; diff --git a/onnxruntime/core/providers/shared_library/provider_interfaces.h b/onnxruntime/core/providers/shared_library/provider_interfaces.h index a2e5448b54..d2ad57e925 100644 --- a/onnxruntime/core/providers/shared_library/provider_interfaces.h +++ b/onnxruntime/core/providers/shared_library/provider_interfaces.h @@ -2,9 +2,10 @@ // Licensed under the MIT License. // Public wrappers around internal ort interfaces (currently) -// In the future the internal implementations could derive from these to remove the need for the wrapper implementations #include "core/providers/shared_library/provider_host_api.h" +#include "core/providers/shared/common.h" + #define PROVIDER_DISALLOW_ALL(TypeName) \ TypeName() = delete; \ TypeName(const TypeName&) = delete; \ @@ -26,7 +27,28 @@ namespace onnxruntime { // These types don't directly map to internal types struct ProviderHost; -class TensorShape; +class If; +class Loop; +class UnsqueezeBase__Prepare; // Directly maps to UnsqueezeBase::Prepare +class SliceOp__PrepareForComputeMetadata; // Directly maps to SliceOp::PrepareForComputeMetadata +struct Prepare; // ConcatBase, TODO: Scope to ConcatBase +struct PrepareContext; +class GatherBase__Prepare; +class PhiloxGenerator; +class Einsum; + +namespace contrib { +class ATenOpBase; +class LongformerAttentionBase; +class AttentionBase; +class Group; +class PassThrough; +class YieldOp; +} // namespace contrib + +namespace training { +class DistributedRunContext; +} template struct IteratorHolder { @@ -35,7 +57,7 @@ struct IteratorHolder { bool operator!=(const IteratorHolder& p) const { return p_->operator!=(*p.p_); } void operator++() { p_->operator++(); } - TResult& operator*() { return p_->operator*(); } + const TResult& operator*() { return p_->operator*(); } T* operator->() { return p_.get(); } private: @@ -48,8 +70,10 @@ struct NodeAttributes_Iterator { virtual bool operator!=(const NodeAttributes_Iterator& p) const = 0; virtual void operator++() = 0; + virtual const std::pair& operator*() const = 0; + virtual const std::string& first() const = 0; - virtual const ONNX_NAMESPACE::AttributeProto& second() = 0; + virtual const ONNX_NAMESPACE::AttributeProto& second() const = 0; }; struct TensorShapeProto_Dimension_Iterator { @@ -65,7 +89,7 @@ using NodeIndex = size_t; // We can't just reinterpret_cast this one, since it's an unordered_map of object BY VALUE (can't do anything by value on the real types) // using NodeAttributes = std::unordered_map; -using Provider_InitializedTensorSet = std::unordered_map; +using InitializedTensorSet = std::unordered_map; struct Node__NodeIterator { virtual ~Node__NodeIterator() {} @@ -92,13 +116,28 @@ struct Node__EdgeIterator { // calls the virtual function (which will lead to infinite recursion in the bridge). There is no known way to get the non virtual member // function pointer implementation in this case. struct ProviderHost { - virtual AllocatorPtr CreateAllocator(const AllocatorCreationInfo& info) = 0; + virtual void* HeapAllocate(size_t size) = 0; + virtual void HeapFree(void*) = 0; virtual logging::Logger* LoggingManager_GetDefaultLogger() = 0; + virtual std::string GetEnvironmentVar(const std::string& var_name) = 0; + + virtual void LogRuntimeError(uint32_t session_id, const common::Status& status, + const char* file, const char* function, uint32_t line) = 0; + + virtual std::vector GetStackTrace() = 0; + + virtual OrtStatus* CreateStatus(OrtErrorCode code, _In_ const char* msg) noexcept = 0; + + virtual AllocatorPtr CreateAllocator(const AllocatorCreationInfo& info) = 0; + virtual std::unique_ptr CreateCPUAllocator(const OrtMemoryInfo& memory_info) = 0; -#ifdef USE_TENSORRT + virtual void* CPUAllocator__Alloc(CPUAllocator* p, size_t size) = 0; + virtual void CPUAllocator__Free(CPUAllocator* p, void* allocation) = 0; + +#ifdef USE_CUDA virtual std::unique_ptr CreateCUDAAllocator(int16_t device_id, const char* name) = 0; virtual std::unique_ptr CreateCUDAPinnedAllocator(int16_t device_id, const char* name) = 0; virtual std::unique_ptr CreateGPUDataTransfer(void* stream) = 0; @@ -110,31 +149,26 @@ struct ProviderHost { virtual bool CudaCall_true(int retCode, const char* exprString, const char* libName, int successCode, const char* msg) = 0; #endif - virtual std::string GetEnvironmentVar(const std::string& var_name) = 0; + virtual std::unordered_set GetCpuPreferredNodes(const onnxruntime::GraphViewer& graph, + const std::string& provider_type, + const std::vector& kernel_registries, + const std::vector& tentative_nodes) = 0; - // PrimitiveDataTypeBase - virtual int32_t PrimitiveDataTypeBase__GetDataType(const PrimitiveDataTypeBase* p) = 0; + virtual Status UnpackTensor(const ONNX_NAMESPACE::TensorProto& tensor, const void* raw_data, size_t raw_data_len, /*out*/ bool* p_data, size_t expected_size) = 0; + virtual Status UnpackTensor(const ONNX_NAMESPACE::TensorProto& tensor, const void* raw_data, size_t raw_data_len, /*out*/ float* p_data, size_t expected_size) = 0; + virtual Status UnpackTensor(const ONNX_NAMESPACE::TensorProto& tensor, const void* raw_data, size_t raw_data_len, /*out*/ double* p_data, size_t expected_size) = 0; + virtual Status UnpackTensor(const ONNX_NAMESPACE::TensorProto& tensor, const void* raw_data, size_t raw_data_len, /*out*/ MLFloat16* p_data, size_t expected_size) = 0; + virtual Status UnpackTensor(const ONNX_NAMESPACE::TensorProto& tensor, const void* raw_data, size_t raw_data_len, /*out*/ int8_t* p_data, size_t expected_size) = 0; + virtual Status UnpackTensor(const ONNX_NAMESPACE::TensorProto& tensor, const void* raw_data, size_t raw_data_len, /*out*/ uint8_t* p_data, size_t expected_size) = 0; + virtual Status UnpackTensor(const ONNX_NAMESPACE::TensorProto& tensor, const void* raw_data, size_t raw_data_len, /*out*/ int16_t* p_data, size_t expected_size) = 0; + virtual Status UnpackTensor(const ONNX_NAMESPACE::TensorProto& tensor, const void* raw_data, size_t raw_data_len, /*out*/ uint16_t* p_data, size_t expected_size) = 0; + virtual Status UnpackTensor(const ONNX_NAMESPACE::TensorProto& tensor, const void* raw_data, size_t raw_data_len, /*out*/ int32_t* p_data, size_t expected_size) = 0; + virtual Status UnpackTensor(const ONNX_NAMESPACE::TensorProto& tensor, const void* raw_data, size_t raw_data_len, /*out*/ uint32_t* p_data, size_t expected_size) = 0; + virtual Status UnpackTensor(const ONNX_NAMESPACE::TensorProto& tensor, const void* raw_data, size_t raw_data_len, /*out*/ int64_t* p_data, size_t expected_size) = 0; + virtual Status UnpackTensor(const ONNX_NAMESPACE::TensorProto& tensor, const void* raw_data, size_t raw_data_len, /*out*/ uint64_t* p_data, size_t expected_size) = 0; - // DataTypeImpl - MLDataType (*DataTypeImpl_GetType_Tensor)(); - MLDataType (*DataTypeImpl_GetType_float)(); - MLDataType (*DataTypeImpl_GetTensorType_float)(); - - virtual const char* DataTypeImpl__ToString(MLDataType type) = 0; - virtual const std::vector& DataTypeImpl__AllFixedSizeTensorTypes() = 0; - virtual const std::vector& DataTypeImpl__AllTensorTypes() = 0; - virtual size_t DataTypeImpl__Size(const DataTypeImpl* p) = 0; - virtual const PrimitiveDataTypeBase* DataTypeImpl__AsPrimitiveDataType(const DataTypeImpl* p) = 0; - - virtual void* HeapAllocate(size_t size) = 0; - virtual void HeapFree(void*) = 0; - - virtual AutoPadType StringToAutoPadType(const std::string& str) = 0; - - virtual void LogRuntimeError(uint32_t session_id, const common::Status& status, - const char* file, const char* function, uint32_t line) = 0; - - virtual std::vector GetStackTrace() = 0; + virtual uint16_t math__floatToHalf(float f) = 0; + virtual float math__halfToFloat(uint16_t h) = 0; // IAllocator virtual bool IAllocator__CalcMemSizeForArrayWithAlignment(size_t nmemb, size_t size, size_t alignment, size_t* out) = 0; @@ -158,6 +192,9 @@ struct ProviderHost { // TensorShape virtual int64_t TensorShape__SizeHelper(const TensorShape* p, size_t start, size_t end) = 0; virtual std::string TensorShape__ToString(const TensorShape* p) = 0; + virtual int64_t TensorShape__SizeToDimension(const TensorShape* p, size_t dimension) = 0; + virtual int64_t TensorShape__SizeFromDimension(const TensorShape* p, size_t dimension) = 0; + virtual std::ostream& operator_left_shift(std::ostream& out, const TensorShape& shape) = 0; // CPUIDInfo virtual const CPUIDInfo& CPUIDInfo__GetCPUIDInfo() = 0; @@ -183,6 +220,7 @@ struct ProviderHost { virtual const int64_t& int64s__Get(const ONNX_NAMESPACE::int64s* p, int index) = 0; // TypeProto_Tensor + virtual bool TypeProto_Tensor__has_shape(const ONNX_NAMESPACE::TypeProto_Tensor* p) = 0; virtual const ONNX_NAMESPACE::TensorShapeProto& TypeProto_Tensor__shape(const ONNX_NAMESPACE::TypeProto_Tensor* p) = 0; virtual ONNX_NAMESPACE::TensorShapeProto* TypeProto_Tensor__mutable_shape(ONNX_NAMESPACE::TypeProto_Tensor* p) = 0; virtual int32_t TypeProto_Tensor__elem_type(const ONNX_NAMESPACE::TypeProto_Tensor* p) = 0; @@ -190,6 +228,7 @@ struct ProviderHost { // TypeProto virtual const ONNX_NAMESPACE::TypeProto_Tensor& TypeProto__tensor_type(const ONNX_NAMESPACE::TypeProto* p) = 0; virtual ONNX_NAMESPACE::TypeProto_Tensor* TypeProto__mutable_tensor_type(ONNX_NAMESPACE::TypeProto* p) = 0; + virtual int TypeProto__value_case(const ONNX_NAMESPACE::TypeProto* p) = 0; // AttributeProto virtual std::unique_ptr AttributeProto__construct() = 0; @@ -199,8 +238,10 @@ struct ProviderHost { virtual ONNX_NAMESPACE::AttributeProto_AttributeType AttributeProto__type(const ONNX_NAMESPACE::AttributeProto* p) = 0; virtual int AttributeProto__ints_size(const ONNX_NAMESPACE::AttributeProto* p) = 0; virtual int AttributeProto__floats_size(const ONNX_NAMESPACE::AttributeProto* p) = 0; + virtual int AttributeProto__strings_size(const ONNX_NAMESPACE::AttributeProto* p) = 0; virtual int64_t AttributeProto__ints(const ONNX_NAMESPACE::AttributeProto* p, int i) = 0; virtual float AttributeProto__floats(const ONNX_NAMESPACE::AttributeProto* p, int i) = 0; + virtual const ::std::string& AttributeProto__strings(const ONNX_NAMESPACE::AttributeProto* p, int i) = 0; virtual const ONNX_NAMESPACE::int64s& AttributeProto__ints(const ONNX_NAMESPACE::AttributeProto* p) = 0; virtual int64_t AttributeProto__i(const ONNX_NAMESPACE::AttributeProto* p) = 0; virtual float AttributeProto__f(const ONNX_NAMESPACE::AttributeProto* p) = 0; @@ -242,10 +283,19 @@ struct ProviderHost { virtual void ModelProto__set_ir_version(ONNX_NAMESPACE::ModelProto* p, int64_t value) = 0; // TensorProto + virtual std::unique_ptr TensorProto__construct() = 0; virtual void TensorProto__operator_delete(ONNX_NAMESPACE::TensorProto* p) = 0; virtual void TensorProto__operator_assign(ONNX_NAMESPACE::TensorProto* p, const ONNX_NAMESPACE::TensorProto& v) = 0; + virtual bool TensorProto__has_name(const ONNX_NAMESPACE::TensorProto* p) = 0; + virtual int TensorProto__dims_size(const ONNX_NAMESPACE::TensorProto* p) = 0; + virtual const ONNX_NAMESPACE::int64s& TensorProto__dims(const ONNX_NAMESPACE::TensorProto* p) = 0; virtual bool TensorProto__has_data_location(const ONNX_NAMESPACE::TensorProto* p) = 0; virtual int TensorProto__data_location(const ONNX_NAMESPACE::TensorProto* p) = 0; + virtual bool TensorProto__has_raw_data(const ONNX_NAMESPACE::TensorProto* p) = 0; + virtual const std::string& TensorProto__raw_data(const ONNX_NAMESPACE::TensorProto* p) = 0; + virtual int32_t TensorProto__data_type(const ONNX_NAMESPACE::TensorProto* p) = 0; + + virtual bool TensorProto_DataType_IsValid(int value) = 0; // TensorProtos virtual ONNX_NAMESPACE::TensorProto* TensorProtos__Add(ONNX_NAMESPACE::TensorProtos* p) = 0; @@ -288,6 +338,8 @@ struct ProviderHost { // DataTransferManager virtual Status DataTransferManager__CopyTensor(const DataTransferManager* p, const Tensor& src, Tensor& dst, int exec_queue_id) = 0; + virtual Status DataTransferManager__CopyTensor(const DataTransferManager* p, const Tensor& src, Tensor& dst) = 0; + virtual const IDataTransfer* DataTransferManager__GetDataTransfer(const DataTransferManager* p, const OrtDevice& src_device, const OrtDevice& dst_device) = 0; // IDataTransfer virtual Status IDataTransfer__CopyTensor(const IDataTransfer* p, const Tensor& src, Tensor& dst) = 0; @@ -318,6 +370,9 @@ struct ProviderHost { // KernelDef virtual void KernelDef__operator_delete(KernelDef* p) = 0; virtual int KernelDef__ExecQueueId(const KernelDef* p) = 0; + virtual void KernelDef__SinceVersion(const KernelDef* p, int* start, int* end) = 0; + virtual const std::string& KernelDef__Domain(const KernelDef* p) = 0; + virtual const std::string& KernelDef__OpName(const KernelDef* p) = 0; // KernelDefBuilder virtual std::unique_ptr KernelDefBuilder__construct() = 0; @@ -326,12 +381,20 @@ struct ProviderHost { virtual void KernelDefBuilder__SetName(KernelDefBuilder* p, const char* op_name) = 0; virtual void KernelDefBuilder__SetDomain(KernelDefBuilder* p, const char* domain) = 0; virtual void KernelDefBuilder__SinceVersion(KernelDefBuilder* p, int since_version) = 0; + virtual void KernelDefBuilder__SinceVersion(KernelDefBuilder* p, int since_version_start, int since_version_end) = 0; virtual void KernelDefBuilder__Provider(KernelDefBuilder* p, const char* provider_type) = 0; virtual void KernelDefBuilder__TypeConstraint(KernelDefBuilder* p, const char* arg_name, MLDataType supported_type) = 0; virtual void KernelDefBuilder__TypeConstraint(KernelDefBuilder* p, const char* arg_name, const std::vector& supported_types) = 0; virtual void KernelDefBuilder__InputMemoryType(KernelDefBuilder* p, OrtMemType type, int input_index) = 0; + virtual void KernelDefBuilder__InputMemoryType(KernelDefBuilder* p, OrtMemType type, const std::vector& input_indexes) = 0; virtual void KernelDefBuilder__OutputMemoryType(KernelDefBuilder* p, OrtMemType type, int input_index) = 0; virtual void KernelDefBuilder__ExecQueueId(KernelDefBuilder* p, int queue_id) = 0; + virtual void KernelDefBuilder__MayInplace(KernelDefBuilder* p, int input_index, int output_index) = 0; + virtual void KernelDefBuilder__Alias(KernelDefBuilder* p, int input_index, int output_index) = 0; + virtual void KernelDefBuilder__Alias(KernelDefBuilder* p, const std::vector>& aliases) = 0; + virtual void KernelDefBuilder__VariadicAlias(KernelDefBuilder* p, int input_offset, int output_offset) = 0; + virtual void KernelDefBuilder__ExternalOutputs(KernelDefBuilder* p) = 0; + virtual void KernelDefBuilder__AllocateInputsContiguously(KernelDefBuilder* p) = 0; virtual std::unique_ptr KernelDefBuilder__Build(KernelDefBuilder* p) = 0; @@ -339,6 +402,50 @@ struct ProviderHost { virtual std::shared_ptr KernelRegistry__construct() = 0; virtual void KernelRegistry__operator_delete(KernelRegistry* p) = 0; virtual Status KernelRegistry__Register(KernelRegistry* p, KernelCreateInfo&& create_info) = 0; + virtual Status KernelRegistry__TryFindKernel(const KernelRegistry* p, const Node& node, ProviderType exec_provider, const KernelCreateInfo** out) = 0; + + // PrimitiveDataTypeBase + virtual int32_t PrimitiveDataTypeBase__GetDataType(const PrimitiveDataTypeBase* p) = 0; + + // DataTypeImpl + virtual MLDataType DataTypeImpl__GetType_Tensor() = 0; + virtual MLDataType DataTypeImpl__GetType_bool() = 0; + virtual MLDataType DataTypeImpl__GetType_int8() = 0; + virtual MLDataType DataTypeImpl__GetType_uint8() = 0; + virtual MLDataType DataTypeImpl__GetType_int16() = 0; + virtual MLDataType DataTypeImpl__GetType_uint16() = 0; + virtual MLDataType DataTypeImpl__GetType_int32() = 0; + virtual MLDataType DataTypeImpl__GetType_uint32() = 0; + virtual MLDataType DataTypeImpl__GetType_int64() = 0; + virtual MLDataType DataTypeImpl__GetType_uint64() = 0; + virtual MLDataType DataTypeImpl__GetType_float() = 0; + virtual MLDataType DataTypeImpl__GetType_double() = 0; + virtual MLDataType DataTypeImpl__GetType_BFloat16() = 0; + virtual MLDataType DataTypeImpl__GetType_MLFloat16() = 0; + virtual MLDataType DataTypeImpl__GetTensorType_bool() = 0; + virtual MLDataType DataTypeImpl__GetTensorType_int8() = 0; + virtual MLDataType DataTypeImpl__GetTensorType_uint8() = 0; + virtual MLDataType DataTypeImpl__GetTensorType_int16() = 0; + virtual MLDataType DataTypeImpl__GetTensorType_uint16() = 0; + virtual MLDataType DataTypeImpl__GetTensorType_int32() = 0; + virtual MLDataType DataTypeImpl__GetTensorType_uint32() = 0; + virtual MLDataType DataTypeImpl__GetTensorType_int64() = 0; + virtual MLDataType DataTypeImpl__GetTensorType_uint64() = 0; + virtual MLDataType DataTypeImpl__GetTensorType_float() = 0; + virtual MLDataType DataTypeImpl__GetTensorType_double() = 0; + virtual MLDataType DataTypeImpl__GetTensorType_BFloat16() = 0; + virtual MLDataType DataTypeImpl__GetTensorType_MLFloat16() = 0; + virtual const char* DataTypeImpl__ToString(MLDataType type) = 0; + virtual bool DataTypeImpl__IsTensorType(const DataTypeImpl* p) = 0; + virtual bool DataTypeImpl__IsTensorSequenceType(const DataTypeImpl* p) = 0; + virtual bool DataTypeImpl__IsSparseTensorType(const DataTypeImpl* p) = 0; + virtual DeleteFunc DataTypeImpl__GetDeleteFunc(const DataTypeImpl* p) = 0; + virtual const std::vector& DataTypeImpl__AllFixedSizeTensorTypes() = 0; + virtual const std::vector& DataTypeImpl__AllTensorTypes() = 0; + virtual const std::vector& DataTypeImpl__AllIEEEFloatTensorTypes() = 0; + virtual const std::vector& DataTypeImpl__AllTensorAndSequenceTensorTypes() = 0; + virtual size_t DataTypeImpl__Size(const DataTypeImpl* p) = 0; + virtual const PrimitiveDataTypeBase* DataTypeImpl__AsPrimitiveDataType(const DataTypeImpl* p) = 0; // Function virtual const Graph& Function__Body(const Function* p) = 0; @@ -348,11 +455,13 @@ struct ProviderHost { virtual const std::string& Node__Description(const Node* p) noexcept = 0; virtual const std::string& Node__Domain(const Node* p) noexcept = 0; virtual const std::string& Node__OpType(const Node* p) noexcept = 0; + virtual int Node__SinceVersion(const Node* p) = 0; virtual const Function* Node__GetFunctionBody(const Node* p) noexcept = 0; + virtual ProviderType Node__GetExecutionProviderType(const Node* p) const noexcept = 0; + virtual const std::vector& Node__InputArgCount(const Node* p) = 0; virtual ConstPointerContainer> Node__ImplicitInputDefs(const Node* p) noexcept = 0; - virtual ConstPointerContainer> Node__InputDefs(const Node* p) noexcept = 0; virtual ConstPointerContainer> Node__OutputDefs(const Node* p) noexcept = 0; virtual NodeIndex Node__Index(const Node* p) noexcept = 0; @@ -438,7 +547,7 @@ struct ProviderHost { virtual const std::vector& GraphViewer__GetOutputs(const GraphViewer* p) noexcept = 0; virtual const std::vector& GraphViewer__GetValueInfo(const GraphViewer* p) noexcept = 0; - virtual const Provider_InitializedTensorSet& GraphViewer__GetAllInitializedTensors(const GraphViewer* p) = 0; + virtual const InitializedTensorSet& GraphViewer__GetAllInitializedTensors(const GraphViewer* p) = 0; virtual bool GraphViewer__GetInitializedTensor(const GraphViewer* p, const std::string& tensor_name, const ONNX_NAMESPACE::TensorProto*& value) = 0; virtual const std::unordered_map& GraphViewer__DomainToVersionMap(const GraphViewer* p) = 0; @@ -448,40 +557,252 @@ struct ProviderHost { // Path virtual PathString Path__ToPathString(const Path* p) noexcept = 0; + // OpKernel + virtual const Node& OpKernel__Node(const OpKernel* p) = 0; + // OpKernelContext virtual const Tensor* OpKernelContext__Input_Tensor(const OpKernelContext* p, int index) = 0; + virtual const Tensor& OpKernelContext__RequiredInput_Tensor(const OpKernelContext* p, int index) = 0; + virtual Tensor* OpKernelContext__Output_Tensor(OpKernelContext* p, int index) = 0; virtual Tensor* OpKernelContext__Output(OpKernelContext* p, int index, const TensorShape& shape) = 0; + virtual Tensor& OpKernelContext__RequiredOutput(OpKernelContext* p, int index, const TensorShape& shape) = 0; + virtual int OpKernelContext__InputCount(const OpKernelContext* p) = 0; + virtual int OpKernelContext__OutputCount(const OpKernelContext* p) = 0; + virtual Status OpKernelContext__GetTempSpaceAllocator(const OpKernelContext* p, AllocatorPtr* output) = 0; + virtual bool OpKernelContext__GetUseDeterministicCompute(const OpKernelContext* p) = 0; + virtual bool OpKernelContext__TryGetInferredOutputShape(const OpKernelContext* p, int index, TensorShape& shape) = 0; + virtual bool OpKernelContext__TryGetInferredInputShape(const OpKernelContext* p, int index, TensorShape& shape) = 0; // OpKernelInfo virtual std::unique_ptr CopyOpKernelInfo(const OpKernelInfo& info) = 0; virtual void OpKernelInfo__operator_delete(OpKernelInfo* p) = 0; + virtual AllocatorPtr OpKernelInfo__GetAllocator(const OpKernelInfo* p, int device_id, OrtMemType mem_type) = 0; + virtual const IExecutionProvider* OpKernelInfo__GetExecutionProvider(const OpKernelInfo* p) = 0; virtual Status OpKernelInfo__GetAttr_int64(const OpKernelInfo* p, const std::string& name, int64_t* value) = 0; virtual Status OpKernelInfo__GetAttr_float(const OpKernelInfo* p, const std::string& name, float* value) = 0; + virtual Status OpKernelInfo__GetAttr_string(const OpKernelInfo* p, const std::string& name, std::string* value) = 0; + virtual Status OpKernelInfo__GetAttr_TensorProto(const OpKernelInfo* p, const std::string& name, ONNX_NAMESPACE::TensorProto* value) = 0; + virtual Status OpKernelInfo__GetAttrs(const OpKernelInfo* p, const std::string& name, std::vector& values) = 0; + virtual Status OpKernelInfo__GetAttrs(const OpKernelInfo* p, const std::string& name, std::vector& values) = 0; + virtual Status OpKernelInfo__GetAttrs(const OpKernelInfo* p, const std::string& name, std::vector& values) = 0; virtual const DataTransferManager& OpKernelInfo__GetDataTransferManager(const OpKernelInfo* p) noexcept = 0; virtual const KernelDef& OpKernelInfo__GetKernelDef(const OpKernelInfo* p) = 0; + virtual bool OpKernelInfo__TryGetConstantInput(const OpKernelInfo* p, int input_index, const Tensor** constant_input_value) = 0; + + virtual uint32_t OpKernelInfo__GetInputCount(const OpKernelInfo* p) = 0; + virtual uint32_t OpKernelInfo__GetOutputCount(const OpKernelInfo* p) = 0; + virtual const Node& OpKernelInfo__node(const OpKernelInfo* p) = 0; + + // SessionState + virtual const DataTransferManager& SessionState__GetDataTransferMgr(const SessionState* p) = 0; // Tensor - virtual float* Tensor__MutableData_float(Tensor* p) = 0; - virtual const float* Tensor__Data_float(const Tensor* p) = 0; + virtual std::unique_ptr Tensor__construct(MLDataType p_type, const TensorShape& shape, std::shared_ptr allocator) = 0; + virtual std::unique_ptr Tensor__construct(MLDataType p_type, const TensorShape& shape, void* p_data, const OrtMemoryInfo& alloc, ptrdiff_t offset) = 0; + virtual void Tensor__operator_delete(Tensor* p) = 0; + virtual bool* Tensor__MutableData_bool(Tensor* p) = 0; + virtual int8_t* Tensor__MutableData_int8(Tensor* p) = 0; + virtual uint8_t* Tensor__MutableData_uint8(Tensor* p) = 0; + virtual int16_t* Tensor__MutableData_int16(Tensor* p) = 0; + virtual uint16_t* Tensor__MutableData_uint16(Tensor* p) = 0; + virtual int32_t* Tensor__MutableData_int32(Tensor* p) = 0; + virtual uint32_t* Tensor__MutableData_uint32(Tensor* p) = 0; + virtual int64_t* Tensor__MutableData_int64(Tensor* p) = 0; + virtual uint64_t* Tensor__MutableData_uint64(Tensor* p) = 0; + virtual float* Tensor__MutableData_float(Tensor* p) = 0; + virtual double* Tensor__MutableData_double(Tensor* p) = 0; + virtual BFloat16* Tensor__MutableData_BFloat16(Tensor* p) = 0; + virtual MLFloat16* Tensor__MutableData_MLFloat16(Tensor* p) = 0; + + virtual const bool* Tensor__Data_bool(const Tensor* p) = 0; + virtual const int8_t* Tensor__Data_int8(const Tensor* p) = 0; + virtual const uint8_t* Tensor__Data_uint8(const Tensor* p) = 0; + virtual const int16_t* Tensor__Data_int16(const Tensor* p) = 0; + virtual const uint16_t* Tensor__Data_uint16(const Tensor* p) = 0; + virtual const int32_t* Tensor__Data_int32(const Tensor* p) = 0; + virtual const uint32_t* Tensor__Data_uint32(const Tensor* p) = 0; + virtual const int64_t* Tensor__Data_int64(const Tensor* p) = 0; + virtual const uint64_t* Tensor__Data_uint64(const Tensor* p) = 0; + virtual const float* Tensor__Data_float(const Tensor* p) = 0; + virtual const double* Tensor__Data_double(const Tensor* p) = 0; + virtual const BFloat16* Tensor__Data_BFloat16(const Tensor* p) = 0; + virtual const MLFloat16* Tensor__Data_MLFloat16(const Tensor* p) = 0; + + virtual gsl::span Tensor__DataAsSpan_int64(const Tensor* p) = 0; + + virtual void* Tensor__MutableDataRaw(Tensor* p, MLDataType type) = 0; + virtual const void* Tensor__DataRaw(const Tensor* p, MLDataType type) = 0; virtual void* Tensor__MutableDataRaw(Tensor* p) noexcept = 0; - virtual const void* Tensor__DataRaw(const Tensor* p) const noexcept = 0; + virtual const void* Tensor__DataRaw(const Tensor* p) noexcept = 0; + + virtual bool Tensor__IsDataType_bool(const Tensor* p) noexcept = 0; + virtual bool Tensor__IsDataType_int8(const Tensor* p) noexcept = 0; + virtual bool Tensor__IsDataType_uint8(const Tensor* p) noexcept = 0; + virtual bool Tensor__IsDataType_int16(const Tensor* p) noexcept = 0; + virtual bool Tensor__IsDataType_uint16(const Tensor* p) noexcept = 0; + virtual bool Tensor__IsDataType_int32(const Tensor* p) noexcept = 0; + virtual bool Tensor__IsDataType_uint32(const Tensor* p) noexcept = 0; + virtual bool Tensor__IsDataType_int64(const Tensor* p) noexcept = 0; + virtual bool Tensor__IsDataType_uint64(const Tensor* p) noexcept = 0; + virtual bool Tensor__IsDataType_float(const Tensor* p) noexcept = 0; + virtual bool Tensor__IsDataType_double(const Tensor* p) noexcept = 0; + virtual bool Tensor__IsDataType_MLFloat16(const Tensor* p) noexcept = 0; + virtual bool Tensor__IsDataTypeString(const Tensor* p) noexcept = 0; virtual const TensorShape& Tensor__Shape(const Tensor* p) = 0; + virtual void Tensor__Reshape(Tensor* p, const TensorShape& new_shape) = 0; + virtual void Tensor__SetByteOffset(Tensor* p, ptrdiff_t byte_offset) = 0; + virtual ptrdiff_t Tensor__ByteOffset(const Tensor* p) = 0; virtual size_t Tensor__SizeInBytes(const Tensor* p) = 0; virtual const OrtMemoryInfo& Tensor__Location(const Tensor* p) = 0; + virtual int32_t Tensor__GetElementType(const Tensor* p) = 0; + virtual MLDataType Tensor__DataType(const Tensor* p) = 0; // AllocatorManager virtual void AllocatorManager__InsertAllocator(AllocatorManager* p, AllocatorPtr allocator) = 0; - virtual AllocatorPtr AllocatorManager__GetAllocator(AllocatorManager* p, int id, OrtMemType mem_type) = 0; -}; + virtual AllocatorPtr AllocatorManager__GetAllocator(const AllocatorManager* p, int id, OrtMemType mem_type) = 0; -extern ProviderHost* g_host; +#ifdef USE_CUDA + // GatherElements + virtual Status GatherElements__ValidateInputShapes(const TensorShape& input_data_shape, const TensorShape& indices_shape, int64_t axis) = 0; + + // cumsum.cc + virtual Status cumsum_op__GetAxis(const Tensor* axis_tensor, int64_t input_rank, int64_t& axis_out) = 0; + + // TileOp + virtual bool TileOp__IsTileMemcpy(const TensorShape& input_shape, const int64_t* repeats, size_t rank, bool& is_batched_memcpy, size_t& num_of_elements_per_batch, size_t& num_of_copies_per_batch, size_t& num_of_batch_copies) = 0; + + // ROI + virtual Status CheckROIAlignValidInput(const Tensor* X_ptr, const Tensor* rois_ptr, const Tensor* batch_indices_ptr) = 0; + + // NonMaxSuppresionBase + virtual Status NonMaxSuppressionBase__PrepareCompute(OpKernelContext* ctx, PrepareContext& pc) = 0; + virtual Status NonMaxSuppressionBase__GetThresholdsFromInputs(const PrepareContext& pc, int64_t& max_output_boxes_per_class, float& iou_threshold, float& score_threshold) = 0; + + // From onehot.h + virtual Status ValidateInputs(const Tensor* depth, const Tensor* values) = 0; + virtual Status PrepareOutputShape(const Tensor* indices, const int64_t depth_val, const int64_t axis, int64_t& prefix_dim_size, int64_t& suffix_dim_size, std::vector& output_shape) = 0; + + // From cpu/tensor/unsqueeze.h + virtual Status UnsqueezeBase__PrepareCompute(const UnsqueezeBase* p, OpKernelContext* ctx, UnsqueezeBase__Prepare& prepare) = 0; + // From cpu/tensor/slice.h + virtual Status SliceBase__PrepareForCompute(const std::vector& raw_starts, + const std::vector& raw_ends, + const std::vector& raw_axes, + SliceOp__PrepareForComputeMetadata& compute_metadata) = 0; + + virtual Status SliceBase__PrepareForCompute(const std::vector& raw_starts, + const std::vector& raw_ends, + const std::vector& raw_axes, + const std::vector& raw_steps, + SliceOp__PrepareForComputeMetadata& compute_metadata) = 0; + virtual Status SliceBase__FillVectorsFromInput(const Tensor& start_tensor, + const Tensor& ends_tensor, + const Tensor* axes_tensor, + const Tensor* steps_tensor, + std::vector& input_starts, + std::vector& input_ends, + std::vector& input_axes, + std::vector& input_steps) = 0; + // From cpu/tensor/size.h + virtual Status Size__Compute(const Size* p, OpKernelContext* context) = 0; + // From cpu/tensor/scatter_nd.h + virtual Status ScatterNDBase__ValidateShapes(const TensorShape& input_shape, + const TensorShape& indice_shape, + const TensorShape& update_shape) = 0; + // From cpu/tensor/padbase.h + virtual Status PadBase__HandleDimValueZero(const Mode& mode, const TensorShape& input_shape, TensorShape& output_shape) = 0; + // From cpu/tensor/split.h + virtual Status SplitBase__PrepareForCompute(const SplitBase* p, const TensorShape& input_shape, int num_outputs, int64_t& axis, int& before_dims, + int& after_dims_including_split_axis, int& after_dims_excluding_split, + std::vector& split_sizes) = 0; + // From cpu/tensor/concatbase.h + virtual Status ConcatBase__PrepareForCompute(const ConcatBase* p, OpKernelContext* ctx, const std::vector& input_tensors, Prepare& prepare) = 0; + // From cpu/tensor/gatherbase.h + virtual Status GatherBase__PrepareForCompute(const GatherBase* p, OpKernelContext* context, GatherBase__Prepare& prepare) = 0; + + virtual PhiloxGenerator& PhiloxGenerator__Default() = 0; + + virtual Status Einsum__Compute(const Einsum* p, OpKernelContext* context) = 0; + + // EinsumComputePreprocessor + virtual void EinsumComputePreprocessor__operator_delete(EinsumComputePreprocessor* p) = 0; + virtual std::unique_ptr EinsumComputePreprocessor__Create(EinsumEquationPreprocessor& equation_preprocessor, + const std::vector& inputs, + AllocatorPtr allocator, + void* einsum_cuda_assets) = 0; + + virtual Status EinsumComputePreprocessor__Run(EinsumComputePreprocessor* p) = 0; + virtual void EinsumComputePreprocessor__SetDeviceHelpers(EinsumComputePreprocessor* p, const EinsumOp::DeviceHelpers::Diagonal& diagonal_func, const EinsumOp::DeviceHelpers::Transpose& transpose_func) = 0; + + // EinsumTypedComputeProcessor + virtual void EinsumTypedComputeProcessor__operator_delete(EinsumTypedComputeProcessor* p) = 0; + virtual void EinsumTypedComputeProcessor__operator_delete(EinsumTypedComputeProcessor* p) = 0; + virtual void EinsumTypedComputeProcessor__operator_delete(EinsumTypedComputeProcessor* p) = 0; + virtual std::unique_ptr> EinsumTypedComputeProcessor_float__Create(OpKernelContext* context, AllocatorPtr allocator, concurrency::ThreadPool* tp, EinsumComputePreprocessor& einsum_compute_preprocessor, void* einsum_cuda_assets) = 0; + virtual std::unique_ptr> EinsumTypedComputeProcessor_double__Create(OpKernelContext* context, AllocatorPtr allocator, concurrency::ThreadPool* tp, EinsumComputePreprocessor& einsum_compute_preprocessor, void* einsum_cuda_assets) = 0; + virtual std::unique_ptr> EinsumTypedComputeProcessor_MLFloat16__Create(OpKernelContext* context, AllocatorPtr allocator, concurrency::ThreadPool* tp, EinsumComputePreprocessor& einsum_compute_preprocessor, void* einsum_cuda_assets) = 0; + virtual void EinsumTypedComputeProcessor__SetDeviceHelpers(EinsumTypedComputeProcessor* p, const EinsumOp::DeviceHelpers::Transpose& device_transpose_func, const EinsumOp::DeviceHelpers::MatMul& device_matmul_func, const EinsumOp::DeviceHelpers::ReduceSum& device_reduce_sum_func, const EinsumOp::DeviceHelpers::DataCopy& device_data_copy_func) = 0; + virtual void EinsumTypedComputeProcessor__SetDeviceHelpers(EinsumTypedComputeProcessor* p, const EinsumOp::DeviceHelpers::Transpose& device_transpose_func, const EinsumOp::DeviceHelpers::MatMul& device_matmul_func, const EinsumOp::DeviceHelpers::ReduceSum& device_reduce_sum_func, const EinsumOp::DeviceHelpers::DataCopy& device_data_copy_func) = 0; + virtual void EinsumTypedComputeProcessor__SetDeviceHelpers(EinsumTypedComputeProcessor* p, const EinsumOp::DeviceHelpers::Transpose& device_transpose_func, const EinsumOp::DeviceHelpers::MatMul& device_matmul_func, const EinsumOp::DeviceHelpers::ReduceSum& device_reduce_sum_func, const EinsumOp::DeviceHelpers::DataCopy& device_data_copy_func) = 0; + virtual Status EinsumTypedComputeProcessor__Run(EinsumTypedComputeProcessor* p) = 0; + virtual Status EinsumTypedComputeProcessor__Run(EinsumTypedComputeProcessor* p) = 0; + virtual Status EinsumTypedComputeProcessor__Run(EinsumTypedComputeProcessor* p) = 0; + + // If + virtual void If__Init(If* p, const OpKernelInfo& info) = 0; + virtual Status If__Compute(const If* p, OpKernelContext* ctx) = 0; + virtual Status If__SetupSubgraphExecutionInfo(If* p, const SessionState& session_state, const std::string& attribute_name, const SessionState& subgraph_session_state) = 0; + + // Loop + virtual void Loop__Init(Loop* p, const OpKernelInfo& info) = 0; + virtual Status Loop__Compute(const Loop* p, OpKernelContext* ctx) = 0; + virtual Status Loop__SetupSubgraphExecutionInfo(Loop* p, const SessionState& session_state, const std::string& attribute_name, const SessionState& subgraph_session_state) = 0; + + // Scan + virtual void Scan__Init(Scan<8>* p, const OpKernelInfo& info) = 0; + virtual void Scan__Init(Scan<9>* p, const OpKernelInfo& info) = 0; + virtual Status Scan__Compute(const Scan<8>* p, OpKernelContext* ctx) = 0; + virtual Status Scan__Compute(const Scan<9>* p, OpKernelContext* ctx) = 0; + virtual Status Scan__SetupSubgraphExecutionInfo(Scan<8>* p, const SessionState& session_state, const std::string& attribute_name, const SessionState& subgraph_session_state) = 0; + virtual Status Scan__SetupSubgraphExecutionInfo(Scan<9>* p, const SessionState& session_state, const std::string& attribute_name, const SessionState& subgraph_session_state) = 0; + + // ContribOps +#ifndef DISABLE_CONTRIB_OPS + virtual Status embed_layer_norm__CheckInputs(const OpKernelContext* context) = 0; + virtual Status bias_gelu_helper__CheckInputs(const OpKernelContext* context) = 0; + virtual Status LongformerAttentionBase__CheckInputs(const contrib::LongformerAttentionBase* p, const TensorShape& input_shape, const TensorShape& weights_shape, const TensorShape& bias_shape, const TensorShape& mask_shape, const TensorShape& global_weights_shape, const TensorShape& global_bias_shape, const TensorShape& global_shape) = 0; + virtual Status AttentionBase__CheckInputs(const contrib::AttentionBase* p, const TensorShape& input_shape, const TensorShape& weights_shape, const TensorShape& bias_shape, const Tensor*& mask_index, const Tensor* past, const int max_threads_per_block) = 0; + virtual Tensor* AttentionBase__GetPresent(const contrib::AttentionBase* p, OpKernelContext* context, const Tensor* past, int batch_size, int head_size, int sequence_length, int& past_sequence_length) = 0; +#endif + +#ifdef ENABLE_TRAINING + virtual void ATenOpBase__Init(contrib::ATenOpBase* p, const OpKernelInfo& info, bool is_backward) = 0; + virtual Status ATenOpBase__Compute(const contrib::ATenOpBase* p, OpKernelContext* p_ctx) = 0; + virtual void contrib__record_event_in_tensor(const Tensor& event_id_tensor) = 0; + virtual void contrib__wait_event_in_tensor(const Tensor& event_id_tensor) = 0; + virtual Status contrib__Group__Compute(const contrib::Group* p, OpKernelContext* context) = 0; + virtual Status contrib__PassThrough__Compute(const contrib::PassThrough* p, OpKernelContext* context) = 0; + virtual void contrib__VerifyLogitWeightAndLabelShape(const TensorShape& logit_shape, const TensorShape& label_shape, const TensorShape* weight_shape) = 0; + virtual void contrib__GetNDCFromLogitAndLabelShape(const TensorShape& logit_shape, const TensorShape& label_shape, int64_t& N_D, int64_t& C) = 0; + virtual void contrib__GetPermutationAndShape(bool ncd_to_ndc, const TensorShape& tensor_shape, std::vector& new_shape, std::vector& permutations) = 0; + virtual Status contrib__PrepareForTrainingCompute(const TensorShape& input_shape, int num_outputs, int64_t& axis, int& before_dims, int& after_dims_including_split_axis, int& after_dims_excluding_split, std::vector& split_sizes) = 0; + virtual Status contrib__YieldOp__Compute(const contrib::YieldOp* p, OpKernelContext* context) = 0; + +#if defined(ORT_USE_NCCL) && defined(USE_NCCL_P2P) + virtual training::DistributedRunContext& GetDistributedRunContextInstance() = 0; +#endif +#endif +#endif +}; #ifdef SHARED_PROVIDER -struct CPUIDInfo { +extern ProviderHost* g_host; + +struct CPUIDInfo final { static const CPUIDInfo& GetCPUIDInfo() { return g_host->CPUIDInfo__GetCPUIDInfo(); } bool HasAVX2() const { return g_host->CPUIDInfo__HasAVX2(this); } @@ -492,19 +813,19 @@ struct CPUIDInfo { namespace logging { -struct Logger { +struct Logger final { bool OutputIsEnabled(Severity severity, DataType data_type) const noexcept { return g_host->logging__Logger__OutputIsEnabled(this, severity, data_type); } PROVIDER_DISALLOW_ALL(Logger) }; -struct LoggingManager { +struct LoggingManager final { static const Logger& DefaultLogger() { return g_host->logging__LoggingManager__DefaultLogger(); } PROVIDER_DISALLOW_ALL(LoggingManager) }; -struct Capture { +struct Capture final { static std::unique_ptr Create(const Logger& logger, logging::Severity severity, const char* category, logging::DataType dataType, const CodeLocation& location) { return g_host->logging__Capture__construct(logger, severity, category, dataType, location); } static void operator delete(void* p) { g_host->logging__Capture__operator_delete(reinterpret_cast(p)); } @@ -516,28 +837,19 @@ struct Capture { void operator=(const Capture&) = delete; }; } // namespace logging - -namespace Utils { - -struct DataTypeUtils { - static const std::string* ToType(const ONNX_NAMESPACE::TypeProto& type_proto) { return g_host->Utils__DataTypeUtils__ToType(type_proto); } - - PROVIDER_DISALLOW_ALL(DataTypeUtils) -}; - -} // namespace Utils } namespace ONNX_NAMESPACE { -struct int64s { +struct int64s final { int size() const { return g_host->int64s__size(this); } const int64_t& Get(int index) const { return g_host->int64s__Get(this, index); } + const int64_t& operator[](int index) const { return Get(index); } PROVIDER_DISALLOW_ALL(int64s) }; -struct AttributeProto { +struct AttributeProto final { static std::unique_ptr Create() { return g_host->AttributeProto__construct(); } void operator=(const AttributeProto& v) { g_host->AttributeProto__operator_assign(this, v); } static void operator delete(void* p) { g_host->AttributeProto__operator_delete(reinterpret_cast(p)); } @@ -545,8 +857,10 @@ struct AttributeProto { AttributeProto_AttributeType type() const { return g_host->AttributeProto__type(this); } int ints_size() const { return g_host->AttributeProto__ints_size(this); } int floats_size() const { return g_host->AttributeProto__floats_size(this); } + int strings_size() const { return g_host->AttributeProto__strings_size(this); } int64_t ints(int i) const { return g_host->AttributeProto__ints(this, i); } float floats(int i) const { return g_host->AttributeProto__floats(this, i); } + const std::string& strings(int i) const { return g_host->AttributeProto__strings(this, i); } const int64s& ints() const { return g_host->AttributeProto__ints(this); } int64_t i() const { return g_host->AttributeProto__i(this); } float f() const { return g_host->AttributeProto__f(this); } @@ -556,11 +870,26 @@ struct AttributeProto { void set_type(AttributeProto_AttributeType value) { return g_host->AttributeProto__set_type(this, value); } TensorProto* add_tensors() { return g_host->AttributeProto__add_tensors(this); } + typedef AttributeProto_AttributeType AttributeType; + static constexpr AttributeType UNDEFINED = AttributeProto_AttributeType_UNDEFINED; + static constexpr AttributeType FLOAT = AttributeProto_AttributeType_FLOAT; + static constexpr AttributeType INT = AttributeProto_AttributeType_INT; + static constexpr AttributeType STRING = AttributeProto_AttributeType_STRING; + static constexpr AttributeType TENSOR = AttributeProto_AttributeType_TENSOR; + static constexpr AttributeType GRAPH = AttributeProto_AttributeType_GRAPH; + static constexpr AttributeType SPARSE_TENSOR = AttributeProto_AttributeType_SPARSE_TENSOR; + static constexpr AttributeType FLOATS = AttributeProto_AttributeType_FLOATS; + static constexpr AttributeType INTS = AttributeProto_AttributeType_INTS; + static constexpr AttributeType STRINGS = AttributeProto_AttributeType_STRINGS; + static constexpr AttributeType TENSORS = AttributeProto_AttributeType_TENSORS; + static constexpr AttributeType GRAPHS = AttributeProto_AttributeType_GRAPHS; + static constexpr AttributeType SPARSE_TENSORS = AttributeProto_AttributeType_SPARSE_TENSORS; + AttributeProto() = delete; AttributeProto(const AttributeProto&) = delete; }; -struct GraphProto { +struct GraphProto final { static void operator delete(void* p) { g_host->GraphProto__operator_delete(reinterpret_cast(p)); } void operator=(const GraphProto& v) { return g_host->GraphProto__operator_assign(this, v); } @@ -581,7 +910,7 @@ struct GraphProto { GraphProto(const GraphProto&) = delete; }; -struct ModelProto { +struct ModelProto final { static std::unique_ptr Create() { return g_host->ModelProto__construct(); } static void operator delete(void* p) { g_host->ModelProto__operator_delete(reinterpret_cast(p)); } @@ -600,24 +929,40 @@ struct ModelProto { void operator=(const ModelProto&) = delete; }; -struct TensorProto { +struct TensorProto final { + static std::unique_ptr Create() { return g_host->TensorProto__construct(); } static void operator delete(void* p) { g_host->TensorProto__operator_delete(reinterpret_cast(p)); } void operator=(const TensorProto& v) { g_host->TensorProto__operator_assign(this, v); } + bool has_name() const { return g_host->TensorProto__has_name(this); } + + int dims_size() const { return g_host->TensorProto__dims_size(this); } + const int64s& dims() const { return g_host->TensorProto__dims(this); } + bool has_data_location() const { return g_host->TensorProto__has_data_location(this); } TensorProto_DataLocation data_location() const { return TensorProto_DataLocation(g_host->TensorProto__data_location(this)); } + bool has_raw_data() const { return g_host->TensorProto__has_raw_data(this); } + const std::string& raw_data() const { return g_host->TensorProto__raw_data(this); } + + int32_t data_type() const { return g_host->TensorProto__data_type(this); } + + typedef TensorProto_DataType DataType; + static constexpr DataType UNDEFINED = TensorProto_DataType_UNDEFINED; + + static bool DataType_IsValid(int value) { return g_host->TensorProto_DataType_IsValid(value); } + TensorProto() = delete; TensorProto(const TensorProto&) = delete; }; -struct TensorProtos { +struct TensorProtos final { TensorProto* Add() { return g_host->TensorProtos__Add(this); } PROVIDER_DISALLOW_ALL(TensorProtos) }; -struct TensorShapeProto_Dimension { +struct TensorShapeProto_Dimension final { enum ValueCase { kDimValue = 1, kDimParam = 2, @@ -635,14 +980,14 @@ struct TensorShapeProto_Dimension { PROVIDER_DISALLOW_ALL(TensorShapeProto_Dimension) }; -struct TensorShapeProto_Dimensions { +struct TensorShapeProto_Dimensions final { IteratorHolder begin() const { return g_host->TensorShapeProto_Dimensions__begin(this); } IteratorHolder end() const { return g_host->TensorShapeProto_Dimensions__end(this); } PROVIDER_DISALLOW_ALL(TensorShapeProto_Dimensions) }; -struct TensorShapeProto { +struct TensorShapeProto final { int dim_size() const { return g_host->TensorShapeProto__dim_size(this); } const TensorShapeProto_Dimensions& dim() const { return g_host->TensorShapeProto__dim(this); } const TensorShapeProto_Dimension& dim(int index) const { return g_host->TensorShapeProto__dim(this, index); } @@ -653,7 +998,8 @@ struct TensorShapeProto { PROVIDER_DISALLOW_ALL(TensorShapeProto) }; -struct TypeProto_Tensor { +struct TypeProto_Tensor final { + bool has_shape() const { return g_host->TypeProto_Tensor__has_shape(this); } const TensorShapeProto& shape() const { return g_host->TypeProto_Tensor__shape(this); } TensorShapeProto* mutable_shape() { return g_host->TypeProto_Tensor__mutable_shape(this); } int32_t elem_type() const { return g_host->TypeProto_Tensor__elem_type(this); } @@ -661,14 +1007,25 @@ struct TypeProto_Tensor { PROVIDER_DISALLOW_ALL(TypeProto_Tensor) }; -struct TypeProto { +struct TypeProto final { const TypeProto_Tensor& tensor_type() const { return g_host->TypeProto__tensor_type(this); } TypeProto_Tensor* mutable_tensor_type() { return g_host->TypeProto__mutable_tensor_type(this); } + enum ValueCase { + kTensorType = 1, + kSequenceType = 4, + kMapType = 5, + kSparseTensorType = 8, + kOpaqueType = 7, + VALUE_NOT_SET = 0, + }; + + ValueCase value_case() const { return ValueCase(g_host->TypeProto__value_case(this)); } + PROVIDER_DISALLOW_ALL(TypeProto) }; -struct ValueInfoProto { +struct ValueInfoProto final { const TypeProto& type() const { return g_host->ValueInfoProto__type(this); } TypeProto* mutable_type() { return g_host->ValueInfoProto__mutable_type(this); } @@ -679,7 +1036,7 @@ struct ValueInfoProto { static void operator delete(void*) = delete; }; -struct ValueInfoProtos { +struct ValueInfoProtos final { ValueInfoProto* Add() { return g_host->ValueInfoProtos__Add(this); } const ValueInfoProto& operator[](int index) const { return g_host->ValueInfoProtos__operator_array(this, index); } @@ -690,7 +1047,17 @@ struct ValueInfoProtos { namespace onnxruntime { -struct ComputeCapability { +namespace Utils { + +struct DataTypeUtils final { + static const std::string* ToType(const ONNX_NAMESPACE::TypeProto& type_proto) { return g_host->Utils__DataTypeUtils__ToType(type_proto); } + + PROVIDER_DISALLOW_ALL(DataTypeUtils) +}; + +} // namespace Utils + +struct ComputeCapability final { static std::unique_ptr Create(std::unique_ptr t_sub_graph) { return g_host->ComputeCapability__construct(std::move(t_sub_graph)); } static void operator delete(void* p) { g_host->ComputeCapability__operator_delete(reinterpret_cast(p)); } @@ -701,13 +1068,16 @@ struct ComputeCapability { void operator=(const ComputeCapability&) = delete; }; -struct DataTransferManager { +struct DataTransferManager final { Status CopyTensor(const Tensor& src, Tensor& dst, int exec_queue_id) const { return g_host->DataTransferManager__CopyTensor(this, src, dst, exec_queue_id); } + Status CopyTensor(const Tensor& src, Tensor& dst) const { return g_host->DataTransferManager__CopyTensor(this, src, dst); } + + const IDataTransfer* GetDataTransfer(const OrtDevice& src_device, const OrtDevice& dst_device) const { return g_host->DataTransferManager__GetDataTransfer(this, src_device, dst_device); } PROVIDER_DISALLOW_ALL(DataTransferManager) }; -struct IndexedSubGraph_MetaDef { +struct IndexedSubGraph_MetaDef final { static std::unique_ptr Create() { return g_host->IndexedSubGraph_MetaDef__construct(); } static void operator delete(void* p) { g_host->IndexedSubGraph_MetaDef__operator_delete(reinterpret_cast(p)); } @@ -733,7 +1103,7 @@ struct IndexedSubGraph_MetaDef { void operator=(const IndexedSubGraph_MetaDef&) = delete; }; -struct IndexedSubGraph { +struct IndexedSubGraph final { static std::unique_ptr Create() { return g_host->IndexedSubGraph__construct(); } static void operator delete(void* p) { g_host->IndexedSubGraph__operator_delete(reinterpret_cast(p)); } @@ -747,21 +1117,23 @@ struct IndexedSubGraph { void operator=(const IndexedSubGraph&) = delete; }; -struct KernelDef { +struct KernelDef final { static void operator delete(void* p) { g_host->KernelDef__operator_delete(reinterpret_cast(p)); } int ExecQueueId() const { return g_host->KernelDef__ExecQueueId(this); } + void SinceVersion(/*out*/ int* start, /*out*/ int* end) const { g_host->KernelDef__SinceVersion(this, start, end); } + const std::string& Domain() const { return g_host->KernelDef__Domain(this); } + const std::string& OpName() const { return g_host->KernelDef__OpName(this); } + KernelDef() = delete; KernelDef(const KernelDef*) = delete; void operator=(const KernelDef&) = delete; }; -#endif using BuildKernelCreateInfoFn = KernelCreateInfo (*)(); -#ifdef SHARED_PROVIDER -struct KernelDefBuilder { +struct KernelDefBuilder final { static std::unique_ptr Create() { return g_host->KernelDefBuilder__construct(); } static void operator delete(void* p) { g_host->KernelDefBuilder__operator_delete(reinterpret_cast(p)); } @@ -777,6 +1149,10 @@ struct KernelDefBuilder { g_host->KernelDefBuilder__SinceVersion(this, since_version); return *this; } + KernelDefBuilder& SinceVersion(int since_version_start, int since_version_end) { + g_host->KernelDefBuilder__SinceVersion(this, since_version_start, since_version_end); + return *this; + } KernelDefBuilder& Provider(const char* provider_type) { g_host->KernelDefBuilder__Provider(this, provider_type); return *this; @@ -793,6 +1169,10 @@ struct KernelDefBuilder { g_host->KernelDefBuilder__InputMemoryType(this, type, input_index); return *this; } + KernelDefBuilder& InputMemoryType(OrtMemType type, const std::vector& input_indexes) { + g_host->KernelDefBuilder__InputMemoryType(this, type, input_indexes); + return *this; + } KernelDefBuilder& OutputMemoryType(OrtMemType type, int input_index) { g_host->KernelDefBuilder__OutputMemoryType(this, type, input_index); return *this; @@ -801,32 +1181,62 @@ struct KernelDefBuilder { g_host->KernelDefBuilder__ExecQueueId(this, queue_id); return *this; } + KernelDefBuilder& MayInplace(int input_index, int output_index) { + g_host->KernelDefBuilder__MayInplace(this, input_index, output_index); + return *this; + } + KernelDefBuilder& Alias(const std::vector>& aliases) { + g_host->KernelDefBuilder__Alias(this, aliases); + return *this; + } + KernelDefBuilder& Alias(int input_index, int output_index) { + g_host->KernelDefBuilder__Alias(this, input_index, output_index); + return *this; + } + KernelDefBuilder& VariadicAlias(int input_offset, int output_offset) { + g_host->KernelDefBuilder__VariadicAlias(this, input_offset, output_offset); + return *this; + } - std::unique_ptr Build() { return g_host->KernelDefBuilder__Build(this); } + KernelDefBuilder& ExternalOutputs() { + g_host->KernelDefBuilder__ExternalOutputs(this); + return *this; + } + + KernelDefBuilder& AllocateInputsContiguously() { + g_host->KernelDefBuilder__AllocateInputsContiguously(this); + return *this; + } + + std::unique_ptr Build() { + return g_host->KernelDefBuilder__Build(this); + } KernelDefBuilder() = delete; KernelDefBuilder(const KernelDefBuilder&) = delete; void operator=(const KernelDefBuilder&) = delete; }; -struct KernelRegistry { +struct KernelRegistry final { static std::shared_ptr Create() { return g_host->KernelRegistry__construct(); } static void operator delete(void* p) { g_host->KernelRegistry__operator_delete(reinterpret_cast(p)); } Status Register(KernelCreateInfo&& create_info) { return g_host->KernelRegistry__Register(this, std::move(create_info)); } + Status TryFindKernel(const Node& node, ProviderType exec_provider, const KernelCreateInfo** out) const { return g_host->KernelRegistry__TryFindKernel(this, node, exec_provider, out); } + KernelRegistry() = delete; KernelRegistry(const KernelRegistry&) = delete; void operator=(const KernelRegistry&) = delete; }; -struct PrimitiveDataTypeBase { +struct PrimitiveDataTypeBase final { int32_t GetDataType() const { return g_host->PrimitiveDataTypeBase__GetDataType(this); } PROVIDER_DISALLOW_ALL(PrimitiveDataTypeBase) }; -class DataTypeImpl { +class DataTypeImpl final { public: size_t Size() const { return g_host->DataTypeImpl__Size(this); } @@ -835,8 +1245,15 @@ class DataTypeImpl { template static MLDataType GetTensorType(); + bool IsTensorType() const { return g_host->DataTypeImpl__IsTensorType(this); } + bool IsTensorSequenceType() const { return g_host->DataTypeImpl__IsTensorSequenceType(this); } + bool IsSparseTensorType() const { return g_host->DataTypeImpl__IsSparseTensorType(this); } + DeleteFunc GetDeleteFunc() const { return g_host->DataTypeImpl__GetDeleteFunc(this); } + static const std::vector& AllFixedSizeTensorTypes() { return g_host->DataTypeImpl__AllFixedSizeTensorTypes(); } static const std::vector& AllTensorTypes() { return g_host->DataTypeImpl__AllTensorTypes(); } + static const std::vector& AllIEEEFloatTensorTypes() { return g_host->DataTypeImpl__AllIEEEFloatTensorTypes(); } + static const std::vector& AllTensorAndSequenceTensorTypes() { return g_host->DataTypeImpl__AllTensorAndSequenceTensorTypes(); } const PrimitiveDataTypeBase* AsPrimitiveDataType() const { return g_host->DataTypeImpl__AsPrimitiveDataType(this); } @@ -845,22 +1262,27 @@ class DataTypeImpl { PROVIDER_DISALLOW_ALL(DataTypeImpl) }; -struct Function { +struct Function final { const Graph& Body() const { return g_host->Function__Body(this); } PROVIDER_DISALLOW_ALL(Function) }; -struct Node { +struct Node final { const std::string& Name() const noexcept { return g_host->Node__Name(this); } const std::string& Description() const noexcept { return g_host->Node__Description(this); } const std::string& Domain() const noexcept { return g_host->Node__Domain(this); } const std::string& OpType() const noexcept { return g_host->Node__OpType(this); } + int SinceVersion() const noexcept { return g_host->Node__SinceVersion(this); } + const Function* GetFunctionBody() const noexcept { return g_host->Node__GetFunctionBody(this); } + ProviderType GetExecutionProviderType() const noexcept { return g_host->Node__GetExecutionProviderType(this); } ConstPointerContainer> ImplicitInputDefs() const noexcept { return g_host->Node__ImplicitInputDefs(this); } + const std::vector& InputArgCount() const noexcept { return g_host->Node__InputArgCount(this); } + ConstPointerContainer> InputDefs() const noexcept { return g_host->Node__InputDefs(this); } ConstPointerContainer> OutputDefs() const noexcept { return g_host->Node__OutputDefs(this); } NodeIndex Index() const noexcept { return g_host->Node__Index(this); } @@ -911,7 +1333,7 @@ struct Node { PROVIDER_DISALLOW_ALL(Node) }; -struct NodeArg { +struct NodeArg final { const std::string& Name() const noexcept { return g_host->NodeArg__Name(this); } const ONNX_NAMESPACE::TensorShapeProto* Shape() const { return g_host->NodeArg__Shape(this); } ONNX_NAMESPACE::DataType Type() const noexcept { return g_host->NodeArg__Type(this); } @@ -922,7 +1344,7 @@ struct NodeArg { PROVIDER_DISALLOW_ALL(NodeArg) }; -struct NodeAttributes { +struct NodeAttributes final { static std::unique_ptr Create() { return g_host->NodeAttributes__construct(); } void operator=(const NodeAttributes& v) { return g_host->NodeAttributes__operator_assign(this, v); } static void operator delete(void* p) { g_host->NodeAttributes__operator_delete(reinterpret_cast(p)); } @@ -933,16 +1355,16 @@ struct NodeAttributes { ONNX_NAMESPACE::AttributeProto& operator[](const std::string& string) { return g_host->NodeAttributes__operator_array(this, string); } const ONNX_NAMESPACE::AttributeProto& at(const std::string& string) const { return g_host->NodeAttributes__at(this, string); } - IteratorHolder> begin() const { return g_host->NodeAttributes__begin(this); } - IteratorHolder> end() const { return g_host->NodeAttributes__end(this); } - IteratorHolder> find(const std::string& key) const { return g_host->NodeAttributes__find(this, key); } + IteratorHolder> begin() const { return g_host->NodeAttributes__begin(this); } + IteratorHolder> end() const { return g_host->NodeAttributes__end(this); } + IteratorHolder> find(const std::string& key) const { return g_host->NodeAttributes__find(this, key); } void insert(const NodeAttributes& v) { return g_host->NodeAttributes__insert(this, v); } NodeAttributes() = delete; NodeAttributes(const NodeAttributes&) = delete; }; -struct Model { +struct Model final { static void operator delete(void* p) { g_host->Model__operator_delete(reinterpret_cast(p)); } Graph& MainGraph() { return g_host->Model__MainGraph(this); } @@ -954,7 +1376,7 @@ struct Model { void operator=(const Model&) = delete; }; -struct Graph { +struct Graph final { std::unique_ptr CreateGraphViewer() const { return g_host->Graph__CreateGraphViewer(this); } std::unique_ptr ToGraphProto() const { return g_host->Graph__ToGraphProto(this); } @@ -974,7 +1396,7 @@ struct Graph { PROVIDER_DISALLOW_ALL(Graph) }; -struct GraphViewer { +struct GraphViewer final { static void operator delete(void* p) { g_host->GraphViewer__operator_delete(reinterpret_cast(p)); } std::unique_ptr CreateModel(const logging::Logger& logger) const { return g_host->GraphViewer__CreateModel(this, logger); } @@ -995,7 +1417,7 @@ struct GraphViewer { const std::vector& GetOutputs() const noexcept { return g_host->GraphViewer__GetOutputs(this); } const std::vector& GetValueInfo() const noexcept { return g_host->GraphViewer__GetValueInfo(this); } - const Provider_InitializedTensorSet& GetAllInitializedTensors() const noexcept { return g_host->GraphViewer__GetAllInitializedTensors(this); } + const InitializedTensorSet& GetAllInitializedTensors() const noexcept { return g_host->GraphViewer__GetAllInitializedTensors(this); } bool GetInitializedTensor(const std::string& tensor_name, const ONNX_NAMESPACE::TensorProto*& value) const { return g_host->GraphViewer__GetInitializedTensor(this, tensor_name, value); } const std::unordered_map& DomainToVersionMap() const noexcept { return g_host->GraphViewer__DomainToVersionMap(this); } @@ -1008,61 +1430,125 @@ struct GraphViewer { void operator=(const GraphViewer&) = delete; }; -struct Path { +struct Path final { PathString ToPathString() const noexcept { return g_host->Path__ToPathString(this); } PROVIDER_DISALLOW_ALL(Path) }; -#endif - -#ifdef SHARED_PROVIDER -struct OpKernelContext { - const Tensor* Input_Tensor(int index) const { return g_host->OpKernelContext__Input_Tensor(this, index); } +struct OpKernelContext final { + template + const T& RequiredInput(int index) const; + Tensor& RequiredOutput(int index, const TensorShape& shape) { return g_host->OpKernelContext__RequiredOutput(this, index, shape); } template const T* Input(int index) const; + int InputCount() const { return g_host->OpKernelContext__InputCount(this); } + + template + T* Output(int index); Tensor* Output(int index, const TensorShape& shape) { return g_host->OpKernelContext__Output(this, index, shape); } + int OutputCount() const { return g_host->OpKernelContext__OutputCount(this); } + + Status GetTempSpaceAllocator(AllocatorPtr* output) const { return g_host->OpKernelContext__GetTempSpaceAllocator(this, output); } + + bool GetUseDeterministicCompute() const { return g_host->OpKernelContext__GetUseDeterministicCompute(this); } + + bool TryGetInferredOutputShape(int index, TensorShape& shape) const { return g_host->OpKernelContext__TryGetInferredOutputShape(this, index, shape); } + bool TryGetInferredInputShape(int index, TensorShape& shape) const { return g_host->OpKernelContext__TryGetInferredInputShape(this, index, shape); } PROVIDER_DISALLOW_ALL(OpKernelContext) }; template <> inline const Tensor* OpKernelContext::Input(int index) const { - return Input_Tensor(index); + return g_host->OpKernelContext__Input_Tensor(this, index); } -struct OpKernelInfo { +template <> +inline Tensor* OpKernelContext::Output(int index) { + return g_host->OpKernelContext__Output_Tensor(this, index); +} + +template <> +inline const Tensor& OpKernelContext::RequiredInput(int index) const { + return g_host->OpKernelContext__RequiredInput_Tensor(this, index); +} + +struct OpKernelInfo final { static void operator delete(void* p) { g_host->OpKernelInfo__operator_delete(reinterpret_cast(p)); } + AllocatorPtr GetAllocator(int device_id, OrtMemType mem_type) const { return g_host->OpKernelInfo__GetAllocator(this, device_id, mem_type); } + + const IExecutionProvider* GetExecutionProvider() const noexcept { return g_host->OpKernelInfo__GetExecutionProvider(this); } + template Status GetAttr(const std::string& name, T* value) const; - Status GetAttr(const std::string& name, int64_t* value) const { return g_host->OpKernelInfo__GetAttr_int64(this, name, value); } - Status GetAttr(const std::string& name, float* value) const { return g_host->OpKernelInfo__GetAttr_float(this, name, value); } + template + Status GetAttrs(const std::string& name, std::vector& values) const; + + template + T GetAttrOrDefault(const std::string& name, const T& default_value) const { + T tmp; + return GetAttr(name, &tmp).IsOK() ? tmp : default_value; + } + + template + void GetAttrOrDefault(const std::string& name, T* value, const T& default_value) const { + if (!GetAttr(name, value).IsOK()) + *value = default_value; + } + + template + std::vector GetAttrsOrDefault(const std::string& name, const std::vector& default_value = std::vector{}) const { + std::vector tmp; + return GetAttrs(name, tmp).IsOK() ? tmp : default_value; + } + + bool TryGetConstantInput(int input_index, const Tensor** constant_input_value) const { return g_host->OpKernelInfo__TryGetConstantInput(this, input_index, constant_input_value); } const DataTransferManager& GetDataTransferManager() const noexcept { return g_host->OpKernelInfo__GetDataTransferManager(this); } const KernelDef& GetKernelDef() const { return g_host->OpKernelInfo__GetKernelDef(this); } + uint32_t GetInputCount() const { return g_host->OpKernelInfo__GetInputCount(this); } + uint32_t GetOutputCount() const { return g_host->OpKernelInfo__GetOutputCount(this); } + + const Node& node() const noexcept { return g_host->OpKernelInfo__node(this); } + OpKernelInfo() = delete; OpKernelInfo(const OpKernelInfo&) = delete; void operator=(const OpKernelInfo&) = delete; }; template <> -inline Status OpKernelInfo::GetAttr(const std::string& name, int64_t* value) const { - return GetAttr(name, value); -} - +inline Status OpKernelInfo::GetAttr(const std::string& name, int64_t* value) const { return g_host->OpKernelInfo__GetAttr_int64(this, name, value); } template <> -inline Status OpKernelInfo::GetAttr(const std::string& name, float* value) const { - return GetAttr(name, value); -} +inline Status OpKernelInfo::GetAttr(const std::string& name, float* value) const { return g_host->OpKernelInfo__GetAttr_float(this, name, value); } +template <> +inline Status OpKernelInfo::GetAttr(const std::string& name, std::string* value) const { return g_host->OpKernelInfo__GetAttr_string(this, name, value); } +template <> +inline Status OpKernelInfo::GetAttr(const std::string& name, ONNX_NAMESPACE::TensorProto* value) const { return g_host->OpKernelInfo__GetAttr_TensorProto(this, name, value); } +template <> +inline Status OpKernelInfo::GetAttrs(const std::string& name, std::vector& values) const { return g_host->OpKernelInfo__GetAttrs(this, name, values); } +template <> +inline Status OpKernelInfo::GetAttrs(const std::string& name, std::vector& values) const { return g_host->OpKernelInfo__GetAttrs(this, name, values); } +template <> +inline Status OpKernelInfo::GetAttrs(const std::string& name, std::vector& values) const { return g_host->OpKernelInfo__GetAttrs(this, name, values); } -struct Tensor { - float* MutableData_float() { return g_host->Tensor__MutableData_float(this); } - const float* Data_float() const { return g_host->Tensor__Data_float(this); } +class SessionState { + public: + const DataTransferManager& GetDataTransferMgr() const noexcept { return g_host->SessionState__GetDataTransferMgr(this); } + + PROVIDER_DISALLOW_ALL(SessionState) +}; + +struct Tensor final { + static std::unique_ptr Create(MLDataType p_type, const TensorShape& shape, std::shared_ptr allocator) { return g_host->Tensor__construct(p_type, shape, allocator); } + static std::unique_ptr Create(MLDataType p_type, const TensorShape& shape, void* p_data, const OrtMemoryInfo& alloc, ptrdiff_t offset = 0) { return g_host->Tensor__construct(p_type, shape, p_data, alloc, offset); } + + static void operator delete(void* p) { g_host->Tensor__operator_delete(reinterpret_cast(p)); } template T* MutableData(); @@ -1070,30 +1556,182 @@ struct Tensor { template const T* Data() const; + template + gsl::span DataAsSpan() const; + + void* MutableDataRaw(MLDataType type) { return g_host->Tensor__MutableDataRaw(this, type); } + const void* DataRaw(MLDataType type) const { return g_host->Tensor__DataRaw(this, type); } + void* MutableDataRaw() noexcept { return g_host->Tensor__MutableDataRaw(this); } const void* DataRaw() const noexcept { return g_host->Tensor__DataRaw(this); } const TensorShape& Shape() const { return g_host->Tensor__Shape(this); } + void Reshape(const TensorShape& new_shape) { g_host->Tensor__Reshape(this, new_shape); } + void SetByteOffset(ptrdiff_t byte_offset) { return g_host->Tensor__SetByteOffset(this, byte_offset); } + ptrdiff_t ByteOffset() const { return g_host->Tensor__ByteOffset(this); } size_t SizeInBytes() const { return g_host->Tensor__SizeInBytes(this); } const OrtMemoryInfo& Location() const { return g_host->Tensor__Location(this); } - PROVIDER_DISALLOW_ALL(Tensor) + int32_t GetElementType() const { return g_host->Tensor__GetElementType(this); } + MLDataType DataType() const { return g_host->Tensor__DataType(this); } + bool IsDataTypeString() const { return g_host->Tensor__IsDataTypeString(this); } + + template + bool IsDataType() const; + + Tensor() = delete; + Tensor(const Tensor&) = delete; + void operator=(const Tensor&) = delete; }; template <> -inline float* Tensor::MutableData() { return MutableData_float(); } +inline bool Tensor::IsDataType() const { return g_host->Tensor__IsDataType_bool(this); } +template <> +inline bool Tensor::IsDataType() const { return g_host->Tensor__IsDataType_int8(this); } +template <> +inline bool Tensor::IsDataType() const { return g_host->Tensor__IsDataType_uint8(this); } +template <> +inline bool Tensor::IsDataType() const { return g_host->Tensor__IsDataType_int16(this); } +template <> +inline bool Tensor::IsDataType() const { return g_host->Tensor__IsDataType_uint16(this); } +template <> +inline bool Tensor::IsDataType() const { return g_host->Tensor__IsDataType_int32(this); } +template <> +inline bool Tensor::IsDataType() const { return g_host->Tensor__IsDataType_uint32(this); } +template <> +inline bool Tensor::IsDataType() const { return g_host->Tensor__IsDataType_int64(this); } +template <> +inline bool Tensor::IsDataType() const { return g_host->Tensor__IsDataType_uint64(this); } +template <> +inline bool Tensor::IsDataType() const { return g_host->Tensor__IsDataType_float(this); } +template <> +inline bool Tensor::IsDataType() const { return g_host->Tensor__IsDataType_double(this); } +template <> +inline bool Tensor::IsDataType() const { return g_host->Tensor__IsDataType_MLFloat16(this); } template <> -inline const float* Tensor::Data() const { return Data_float(); } +inline bool* Tensor::MutableData() { return g_host->Tensor__MutableData_bool(this); } +template <> +inline int8_t* Tensor::MutableData() { return g_host->Tensor__MutableData_int8(this); } +template <> +inline uint8_t* Tensor::MutableData() { return g_host->Tensor__MutableData_uint8(this); } +template <> +inline int16_t* Tensor::MutableData() { return g_host->Tensor__MutableData_int16(this); } +template <> +inline uint16_t* Tensor::MutableData() { return g_host->Tensor__MutableData_uint16(this); } +template <> +inline int32_t* Tensor::MutableData() { return g_host->Tensor__MutableData_int32(this); } +template <> +inline uint32_t* Tensor::MutableData() { return g_host->Tensor__MutableData_uint32(this); } +template <> +inline int64_t* Tensor::MutableData() { return g_host->Tensor__MutableData_int64(this); } +template <> +inline uint64_t* Tensor::MutableData() { return g_host->Tensor__MutableData_uint64(this); } +template <> +inline float* Tensor::MutableData() { return g_host->Tensor__MutableData_float(this); } +template <> +inline double* Tensor::MutableData() { return g_host->Tensor__MutableData_double(this); } +template <> +inline BFloat16* Tensor::MutableData() { return g_host->Tensor__MutableData_BFloat16(this); } +template <> +inline MLFloat16* Tensor::MutableData() { return g_host->Tensor__MutableData_MLFloat16(this); } + +template <> +inline const bool* Tensor::Data() const { return g_host->Tensor__Data_bool(this); } +template <> +inline const int8_t* Tensor::Data() const { return g_host->Tensor__Data_int8(this); } +template <> +inline const uint8_t* Tensor::Data() const { return g_host->Tensor__Data_uint8(this); } +template <> +inline const int16_t* Tensor::Data() const { return g_host->Tensor__Data_int16(this); } +template <> +inline const uint16_t* Tensor::Data() const { return g_host->Tensor__Data_uint16(this); } +template <> +inline const int32_t* Tensor::Data() const { return g_host->Tensor__Data_int32(this); } +template <> +inline const uint32_t* Tensor::Data() const { return g_host->Tensor__Data_uint32(this); } +template <> +inline const int64_t* Tensor::Data() const { return g_host->Tensor__Data_int64(this); } +template <> +inline const uint64_t* Tensor::Data() const { return g_host->Tensor__Data_uint64(this); } +template <> +inline const float* Tensor::Data() const { return g_host->Tensor__Data_float(this); } +template <> +inline const double* Tensor::Data() const { return g_host->Tensor__Data_double(this); } +template <> +inline const BFloat16* Tensor::Data() const { return g_host->Tensor__Data_BFloat16(this); } +template <> +inline const MLFloat16* Tensor::Data() const { return g_host->Tensor__Data_MLFloat16(this); } + +template <> +inline gsl::span Tensor::DataAsSpan() const { return g_host->Tensor__DataAsSpan_int64(this); } namespace utils { - -inline bool HasDimValue(const ONNX_NAMESPACE::TensorShapeProto_Dimension& dim) { - return dim.value_case() == ONNX_NAMESPACE::TensorShapeProto_Dimension::kDimValue; -} +bool IsDataTypeString(MLDataType dt_type); } // namespace utils -#endif +#ifdef USE_CUDA +namespace GatherElements { +inline Status ValidateInputShapes(const TensorShape& input_data_shape, + const TensorShape& indices_shape, + int64_t axis) { return g_host->GatherElements__ValidateInputShapes(input_data_shape, indices_shape, axis); } +} // namespace GatherElements + +namespace cumsum_op { +inline Status GetAxis(const Tensor* axis_tensor, int64_t input_rank, int64_t& axis_out) { return g_host->cumsum_op__GetAxis(axis_tensor, input_rank, axis_out); } +} // namespace cumsum_op + +inline Status CheckROIAlignValidInput(const Tensor* X_ptr, const Tensor* rois_ptr, const Tensor* batch_indices_ptr) { return g_host->CheckROIAlignValidInput(X_ptr, rois_ptr, batch_indices_ptr); } + +// From onehot.h +inline Status ValidateInputs(const Tensor* depth, const Tensor* values) { return g_host->ValidateInputs(depth, values); } +inline Status PrepareOutputShape(const Tensor* indices, const int64_t depth_val, const int64_t axis, + int64_t& prefix_dim_size, int64_t& suffix_dim_size, + std::vector& output_shape) { return g_host->PrepareOutputShape(indices, depth_val, axis, prefix_dim_size, suffix_dim_size, output_shape); } + +struct EinsumComputePreprocessor { + static void operator delete(void* p) { g_host->EinsumComputePreprocessor__operator_delete(reinterpret_cast(p)); } + static std::unique_ptr Create(EinsumEquationPreprocessor& equation_preprocessor, + const std::vector& inputs, + AllocatorPtr allocator, + void* einsum_cuda_assets) { return g_host->EinsumComputePreprocessor__Create(equation_preprocessor, inputs, allocator, einsum_cuda_assets); } + + Status Run() { return g_host->EinsumComputePreprocessor__Run(this); } + + void SetDeviceHelpers(const EinsumOp::DeviceHelpers::Diagonal& diagonal_func, const EinsumOp::DeviceHelpers::Transpose& transpose_func) { return g_host->EinsumComputePreprocessor__SetDeviceHelpers(this, diagonal_func, transpose_func); } +}; + +template +struct EinsumTypedComputeProcessor { + static void operator delete(void* p) { g_host->EinsumTypedComputeProcessor__operator_delete(reinterpret_cast(p)); } + static std::unique_ptr Create(OpKernelContext* context, AllocatorPtr allocator, + concurrency::ThreadPool* tp, + EinsumComputePreprocessor& einsum_compute_preprocessor, + void* einsum_cuda_assets); + + void SetDeviceHelpers(const EinsumOp::DeviceHelpers::Transpose& device_transpose_func, + const EinsumOp::DeviceHelpers::MatMul& device_matmul_func, + const EinsumOp::DeviceHelpers::ReduceSum& device_reduce_sum_func, + const EinsumOp::DeviceHelpers::DataCopy& device_data_copy_func) { + g_host->EinsumTypedComputeProcessor__SetDeviceHelpers(this, device_transpose_func, device_matmul_func, device_reduce_sum_func, device_data_copy_func); + } + + Status Run() { return g_host->EinsumTypedComputeProcessor__Run(this); } +}; + +#ifdef ENABLE_TRAINING +namespace contrib { +inline void record_event_in_tensor(const Tensor& event_id_tensor) { return g_host->contrib__record_event_in_tensor(event_id_tensor); } +inline void wait_event_in_tensor(const Tensor& event_id_tensor) { return g_host->contrib__wait_event_in_tensor(event_id_tensor); } + +inline void VerifyLogitWeightAndLabelShape(const TensorShape& logit_shape, const TensorShape& label_shape, const TensorShape* weight_shape) { g_host->contrib__VerifyLogitWeightAndLabelShape(logit_shape, label_shape, weight_shape); } +inline void GetNDCFromLogitAndLabelShape(const TensorShape& logit_shape, const TensorShape& label_shape, int64_t& N_D, int64_t& C) { g_host->contrib__GetNDCFromLogitAndLabelShape(logit_shape, label_shape, N_D, C); } +inline void GetPermutationAndShape(bool ncd_to_ndc, const TensorShape& tensor_shape, std::vector& new_shape, std::vector& permutations) { g_host->contrib__GetPermutationAndShape(ncd_to_ndc, tensor_shape, new_shape, permutations); } +inline Status PrepareForTrainingCompute(const TensorShape& input_shape, int num_outputs, int64_t& axis, int& before_dims, int& after_dims_including_split_axis, int& after_dims_excluding_split, std::vector& split_sizes) { return g_host->contrib__PrepareForTrainingCompute(input_shape, num_outputs, axis, before_dims, after_dims_including_split_axis, after_dims_excluding_split, split_sizes); } +} // namespace contrib +#endif // ENABLE_TRAINING +#endif // USE_CUDA +#endif // SHARED_PROVIDER } // namespace onnxruntime diff --git a/onnxruntime/core/providers/tensorrt/tensorrt_execution_provider.cc b/onnxruntime/core/providers/tensorrt/tensorrt_execution_provider.cc index b9c6529e28..1a8753de3c 100644 --- a/onnxruntime/core/providers/tensorrt/tensorrt_execution_provider.cc +++ b/onnxruntime/core/providers/tensorrt/tensorrt_execution_provider.cc @@ -10,6 +10,7 @@ #include "tensorrt_execution_provider.h" #include "core/providers/cuda/shared_inc/cuda_call.h" #include "core/providers/cuda/math/unary_elementwise_ops_impl.h" +#include "core/providers/cuda/gpu_data_transfer.h" #include "cuda_runtime_api.h" #include "gsl/gsl" #include @@ -573,16 +574,16 @@ AllocatorPtr TensorrtExecutionProvider::GetAllocator(int id, OrtMemType mem_type } void TensorrtExecutionProvider::RegisterAllocator(std::shared_ptr allocator_manager) { - allocator_ = AllocatorManager__GetAllocator(allocator_manager.get(), device_id_, OrtMemTypeDefault); + allocator_ = allocator_manager->GetAllocator(device_id_, OrtMemTypeDefault); if (nullptr == allocator_) { AllocatorCreationInfo default_memory_info( [](OrtDevice::DeviceId device_id) { return CreateCUDAAllocator(device_id, onnxruntime::CUDA); }, device_id_); allocator_ = CreateAllocator(default_memory_info); - AllocatorManager__InsertAllocator(allocator_manager.get(), allocator_); + allocator_manager->InsertAllocator(allocator_); } TryInsertAllocator(allocator_); - auto cuda_pinned_alloc = AllocatorManager__GetAllocator(allocator_manager.get(), DEFAULT_CPU_ALLOCATOR_DEVICE_ID, OrtMemTypeCPUOutput); + auto cuda_pinned_alloc = allocator_manager->GetAllocator(DEFAULT_CPU_ALLOCATOR_DEVICE_ID, OrtMemTypeCPUOutput); if (nullptr == cuda_pinned_alloc) { AllocatorCreationInfo pinned_allocator_info( [](OrtDevice::DeviceId device_id) { @@ -590,7 +591,7 @@ void TensorrtExecutionProvider::RegisterAllocator(std::shared_ptrInsertAllocator(cuda_pinned_alloc); } TryInsertAllocator(cuda_pinned_alloc); } diff --git a/onnxruntime/core/session/onnxruntime_c_api.cc b/onnxruntime/core/session/onnxruntime_c_api.cc index be818df5ef..5c2b1fe9ac 100644 --- a/onnxruntime/core/session/onnxruntime_c_api.cc +++ b/onnxruntime/core/session/onnxruntime_c_api.cc @@ -1826,25 +1826,6 @@ ORT_API_STATUS_IMPL(OrtApis::SessionGetProfilingStartTimeNs, _In_ const OrtSessi // End support for non-tensor types -#ifndef USE_CUDA -ORT_API_STATUS_IMPL(OrtApis::SessionOptionsAppendExecutionProvider_CUDA, - _In_ OrtSessionOptions* options, _In_ const OrtCUDAProviderOptions* cuda_options) { - ORT_UNUSED_PARAMETER(options); - ORT_UNUSED_PARAMETER(cuda_options); - return CreateStatus(ORT_FAIL, "CUDA execution provider is not enabled."); -} - -ORT_API_STATUS_IMPL(OrtApis::SetCurrentGpuDeviceId, _In_ int device_id) { - ORT_UNUSED_PARAMETER(device_id); - return CreateStatus(ORT_FAIL, "CUDA execution provider is not enabled."); -} - -ORT_API_STATUS_IMPL(OrtApis::GetCurrentGpuDeviceId, _In_ int* device_id) { - ORT_UNUSED_PARAMETER(device_id); - return CreateStatus(ORT_FAIL, "CUDA execution provider is not enabled."); -} -#endif - #ifndef USE_ROCM ORT_API_STATUS_IMPL(OrtApis::SessionOptionsAppendExecutionProvider_ROCM, _In_ OrtSessionOptions* options, _In_ const OrtROCMProviderOptions* rocm_options) { @@ -1863,6 +1844,24 @@ ORT_API_STATUS_IMPL(OrtApis::SessionOptionsAppendExecutionProvider_OpenVINO, } #endif +#if defined(ORT_MINIMAL_BUILD) +ORT_API_STATUS_IMPL(OrtApis::SessionOptionsAppendExecutionProvider_CUDA, _In_ OrtSessionOptions* options, _In_ const OrtCUDAProviderOptions* provider_options) { + ORT_UNUSED_PARAMETER(options); + ORT_UNUSED_PARAMETER(provider_options); + return CreateStatus(ORT_FAIL, "CUDA execution provider is not enabled."); +} + +ORT_API_STATUS_IMPL(OrtApis::GetCurrentGpuDeviceId, _In_ int* device_id) { + ORT_UNUSED_PARAMETER(device_id); + return CreateStatus(ORT_FAIL, "CUDA execution provider is not enabled."); +} + +ORT_API_STATUS_IMPL(OrtApis::SetCurrentGpuDeviceId, _In_ int device_id) { + ORT_UNUSED_PARAMETER(device_id); + return CreateStatus(ORT_FAIL, "CUDA execution provider is not enabled."); +} +#endif + ORT_API_STATUS_IMPL(OrtApis::CreateArenaCfg, _In_ size_t max_mem, int arena_extend_strategy, int initial_chunk_size_bytes, int max_dead_bytes_per_chunk, _Outptr_ OrtArenaCfg** out) { API_IMPL_BEGIN diff --git a/onnxruntime/core/util/math.h b/onnxruntime/core/util/math.h index 1bacbf2d3a..16e7c477da 100644 --- a/onnxruntime/core/util/math.h +++ b/onnxruntime/core/util/math.h @@ -16,8 +16,10 @@ #pragma once +#ifndef SHARED_PROVIDER #include "core/common/common.h" #include "core/framework/tensor.h" +#endif #ifndef CBLAS_ENUM_DEFINED_H #define CBLAS_ENUM_DEFINED_H diff --git a/onnxruntime/core/util/math_cpuonly.h b/onnxruntime/core/util/math_cpuonly.h index 7a8030b034..6f17552e0a 100644 --- a/onnxruntime/core/util/math_cpuonly.h +++ b/onnxruntime/core/util/math_cpuonly.h @@ -57,7 +57,9 @@ #pragma warning(pop) #endif +#ifndef SHARED_PROVIDER #include "core/framework/tensor.h" +#endif namespace onnxruntime { // common Eigen types that we will often use diff --git a/onnxruntime/python/onnxruntime_pybind_state.cc b/onnxruntime/python/onnxruntime_pybind_state.cc index 129d3e1a55..233be1504d 100644 --- a/onnxruntime/python/onnxruntime_pybind_state.cc +++ b/onnxruntime/python/onnxruntime_pybind_state.cc @@ -34,9 +34,6 @@ // execution provider factory creator headers #include "core/providers/cpu/cpu_provider_factory_creator.h" -#ifdef USE_CUDA -#include "core/providers/cuda/cuda_provider_factory_creator.h" -#endif #ifdef USE_ROCM #include "core/providers/rocm/rocm_provider_factory_creator.h" #endif @@ -146,9 +143,7 @@ struct OrtStatus { #if defined(USE_CUDA) || defined(USE_ROCM) #ifdef USE_CUDA -#include "core/providers/cuda/shared_inc/cuda_call.h" -#include "core/providers/cuda/cuda_execution_provider.h" -#include "core/providers/cuda/cuda_allocator.h" +#include "core/providers/cuda/cuda_execution_provider_info.h" // TODO remove deprecated global config OrtCudnnConvAlgoSearch cudnn_conv_algo_search = OrtCudnnConvAlgoSearch::EXHAUSTIVE; // TODO remove deprecated global config @@ -170,6 +165,9 @@ size_t gpu_mem_limit = std::numeric_limits::max(); onnxruntime::ArenaExtendStrategy arena_extend_strategy = onnxruntime::ArenaExtendStrategy::kNextPowerOfTwo; #endif +#ifdef USE_CUDA +#include "core/providers/cuda/cuda_provider_factory.h" +#endif #ifdef USE_TENSORRT #include "core/providers/tensorrt/tensorrt_provider_factory.h" #endif @@ -210,10 +208,14 @@ namespace onnxruntime { std::shared_ptr CreateExecutionProviderFactory_Tensorrt(const OrtTensorRTProviderOptions* params); std::shared_ptr CreateExecutionProviderFactory_Tensorrt(int device_id); std::shared_ptr CreateExecutionProviderFactory_MIGraphX(int device_id); +std::shared_ptr CreateExecutionProviderFactory_Cuda(const OrtCUDAProviderOptions* params); std::shared_ptr CreateExecutionProviderFactory_Dnnl(int use_arena); std::shared_ptr CreateExecutionProviderFactory_OpenVINO(const OrtOpenVINOProviderOptions* params); +#ifdef USE_CUDA +ProviderInfo_CUDA* GetProviderInfo_CUDA(); +#endif #ifdef USE_OPENVINO -const ProviderInfo_OpenVINO* GetProviderInfo_OpenVINO(); +ProviderInfo_OpenVINO* GetProviderInfo_OpenVINO(); #endif std::shared_ptr CreateExecutionProviderFactory_Nuphar(bool, const char*); std::shared_ptr CreateExecutionProviderFactory_VITISAI(const char* backend_type, int device_id); @@ -469,8 +471,7 @@ static std::unique_ptr LoadExecutionProvider( #ifdef USE_CUDA static bool IsCudaDeviceIdValid(const onnxruntime::logging::Logger& logger, int id) { - int num_devices = 0; - CUDA_CALL_THROW(cudaGetDeviceCount(&num_devices)); + int num_devices = GetProviderInfo_CUDA()->cudaGetDeviceCount(); if (0 == num_devices) { LOGS(logger, WARNING) << "your system does not have a CUDA capable device."; @@ -488,22 +489,23 @@ static bool IsCudaDeviceIdValid(const onnxruntime::logging::Logger& logger, int static AllocatorPtr GetCudaAllocator(OrtDevice::DeviceId id) { // Current approach is not thread-safe, but there are some bigger infra pieces to put together in order to make // multi-threaded CUDA allocation work we need to maintain a per-thread CUDA allocator - static std::unordered_map id_to_allocator_map; - if (id_to_allocator_map.find(id) == id_to_allocator_map.end()) { + static auto* id_to_allocator_map = new std::unordered_map(); + + if (id_to_allocator_map->find(id) == id_to_allocator_map->end()) { // TODO: Expose knobs so that users can set fields associated with OrtArenaCfg so that we can pass it to the following method - id_to_allocator_map.insert({id, CUDAExecutionProvider::CreateCudaAllocator(id, gpu_mem_limit, arena_extend_strategy, external_allocator_info, nullptr)}); + id_to_allocator_map->insert({id, GetProviderInfo_CUDA()->CreateCudaAllocator(id, gpu_mem_limit, arena_extend_strategy, external_allocator_info, nullptr)}); } - return id_to_allocator_map[id]; + return (*id_to_allocator_map)[id]; } static void CpuToCudaMemCpy(void* dst, const void* src, size_t num_bytes) { - CUDA_CALL_THROW(cudaMemcpy(dst, src, num_bytes, cudaMemcpyHostToDevice)); + GetProviderInfo_CUDA()->cudaMemcpy_HostToDevice(dst, src, num_bytes); } static void CudaToCpuMemCpy(void* dst, const void* src, size_t num_bytes) { - CUDA_CALL_THROW(cudaMemcpy(dst, src, num_bytes, cudaMemcpyDeviceToHost)); + GetProviderInfo_CUDA()->cudaMemcpy_DeviceToHost(dst, src, num_bytes); } static const std::unordered_map* GetCudaToHostMemCpyFunction() { @@ -730,26 +732,23 @@ static void RegisterExecutionProviders(InferenceSession* sess, const std::vector } else if (type == kCudaExecutionProvider) { #ifdef USE_CUDA const auto it = provider_options_map.find(type); - const CUDAExecutionProviderInfo info = - it != provider_options_map.end() - ? CUDAExecutionProviderInfo::FromProviderOptions(it->second) - : [&]() { - CUDAExecutionProviderInfo info{}; - info.device_id = cuda_device_id; - info.gpu_mem_limit = gpu_mem_limit; - info.arena_extend_strategy = arena_extend_strategy; - info.cudnn_conv_algo_search = cudnn_conv_algo_search; - info.do_copy_in_default_stream = do_copy_in_default_stream; - info.external_allocator_info = external_allocator_info; - return info; - }(); + CUDAExecutionProviderInfo info{}; + if (it != provider_options_map.end()) + GetProviderInfo_CUDA()->CUDAExecutionProviderInfo__FromProviderOptions(it->second, info); + else { + info.device_id = cuda_device_id; + info.gpu_mem_limit = gpu_mem_limit; + info.arena_extend_strategy = arena_extend_strategy; + info.cudnn_conv_algo_search = cudnn_conv_algo_search; + info.do_copy_in_default_stream = do_copy_in_default_stream; + info.external_allocator_info = external_allocator_info; + } // This variable is never initialized because the APIs by which is it should be initialized are deprecated, however they still // exist are are in-use. Neverthless, it is used to return CUDAAllocator, hence we must try to initialize it here if we can // since FromProviderOptions might contain external CUDA allocator. external_allocator_info = info.external_allocator_info; - RegisterExecutionProvider( - sess, *onnxruntime::CreateExecutionProviderFactory_CUDA(info)); + RegisterExecutionProvider(sess, *GetProviderInfo_CUDA()->CreateExecutionProviderFactory(info)); #endif } else if (type == kRocmExecutionProvider) { #ifdef USE_ROCM diff --git a/onnxruntime/test/framework/TestAllocatorManager.cc b/onnxruntime/test/framework/TestAllocatorManager.cc index 5ad38b8753..a4b9058938 100644 --- a/onnxruntime/test/framework/TestAllocatorManager.cc +++ b/onnxruntime/test/framework/TestAllocatorManager.cc @@ -1,6 +1,8 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. +#undef USE_CUDA // TODO: Cuda is a shared library, so can't call any Cuda provider methods directly from here + #include "test/framework/TestAllocatorManager.h" #include "core/framework/allocatormgr.h" #ifdef USE_CUDA diff --git a/onnxruntime/test/framework/cuda/allocator_cuda_test.cc b/onnxruntime/test/framework/cuda/allocator_cuda_test.cc index ac9f0d0605..c96b68b8ba 100644 --- a/onnxruntime/test/framework/cuda/allocator_cuda_test.cc +++ b/onnxruntime/test/framework/cuda/allocator_cuda_test.cc @@ -1,6 +1,8 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. +#if 0 // TODO: Can't call these directly from external code as Cuda is now a shared library + #include "core/framework/allocatormgr.h" #include "test/framework/test_utils.h" #include "gtest/gtest.h" @@ -112,3 +114,5 @@ TEST(AllocatorTest, CUDAAllocatorFallbackTest) { } } // namespace test } // namespace onnxruntime + +#endif diff --git a/onnxruntime/test/framework/cuda/fence_cuda_test.cc b/onnxruntime/test/framework/cuda/fence_cuda_test.cc index 4ed0aba3fb..e04b101c65 100644 --- a/onnxruntime/test/framework/cuda/fence_cuda_test.cc +++ b/onnxruntime/test/framework/cuda/fence_cuda_test.cc @@ -1,5 +1,6 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. +#if 0 // TODO: Can't call these directly from external code as Cuda is now a shared library #include "core/graph/onnx_protobuf.h" #include "core/session/inference_session.h" @@ -32,7 +33,6 @@ using namespace std; using namespace ONNX_NAMESPACE; using namespace onnxruntime::logging; -namespace onnxruntime { namespace test { typedef std::vector ArgMap; @@ -263,3 +263,4 @@ TEST(CUDAFenceTests, TileWithComputedInput) { } // namespace test } // namespace onnxruntime +#endif diff --git a/onnxruntime/test/framework/inference_session_test.cc b/onnxruntime/test/framework/inference_session_test.cc index 9fb9f23b4a..25a0e0a272 100644 --- a/onnxruntime/test/framework/inference_session_test.cc +++ b/onnxruntime/test/framework/inference_session_test.cc @@ -1,5 +1,7 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. +#undef USE_CUDA // TODO: Cuda is a shared library, so can't call any Cuda provider methods directly from here + #include "core/graph/onnx_protobuf.h" #include "core/session/inference_session.h" diff --git a/onnxruntime/test/framework/memcpy_transformer_test.cc b/onnxruntime/test/framework/memcpy_transformer_test.cc index 305de9977b..7b08f9df2c 100644 --- a/onnxruntime/test/framework/memcpy_transformer_test.cc +++ b/onnxruntime/test/framework/memcpy_transformer_test.cc @@ -1,5 +1,6 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. +#undef USE_CUDA // TODO: Cuda is a shared library, so can't call any Cuda provider methods directly from here #include diff --git a/onnxruntime/test/framework/session_state_test.cc b/onnxruntime/test/framework/session_state_test.cc index 2c7647f4fe..736b2c1094 100644 --- a/onnxruntime/test/framework/session_state_test.cc +++ b/onnxruntime/test/framework/session_state_test.cc @@ -19,6 +19,7 @@ #include "core/util/thread_utils.h" #include "gtest/gtest.h" #include "test/test_environment.h" +#include "test/util/include/default_providers.h" using namespace ONNX_NAMESPACE; using namespace std; diff --git a/onnxruntime/test/framework/test_utils.cc b/onnxruntime/test/framework/test_utils.cc index 228c565ab5..8c5414a4a8 100644 --- a/onnxruntime/test/framework/test_utils.cc +++ b/onnxruntime/test/framework/test_utils.cc @@ -1,6 +1,5 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. - #include "test_utils.h" #include "core/graph/graph.h" @@ -12,14 +11,6 @@ IExecutionProvider* TestCPUExecutionProvider() { return &cpu_provider; } -#ifdef USE_CUDA -IExecutionProvider* TestCudaExecutionProvider() { - static CUDAExecutionProviderInfo info; - static CUDAExecutionProvider cuda_provider(info); - return &cuda_provider; -} -#endif - #ifdef USE_ROCM IExecutionProvider* TestRocmExecutionProvider() { static ROCMExecutionProviderInfo info; @@ -28,26 +19,6 @@ IExecutionProvider* TestRocmExecutionProvider() { } #endif -#ifdef USE_TENSORRT -#if 0 // TODO: TensorRT is shared, can't access these directly anymore -IExecutionProvider* TestTensorrtExecutionProvider() { - static TensorrtExecutionProviderInfo info; - static TensorrtExecutionProvider trt_provider(info); - return &trt_provider; -} -#endif -#endif - -#ifdef USE_OPENVINO -#if 0 // TODO: OpenVINO is shared, can't access these directly anymore -IExecutionProvider* TestOpenVINOExecutionProvider() { - static OpenVINOExecutionProviderInfo info; - static OpenVINOExecutionProvider openvino_provider(info); - return &openvino_provider; -} -#endif -#endif - #ifdef USE_NNAPI IExecutionProvider* TestNnapiExecutionProvider() { static NnapiExecutionProvider nnapi_provider(0); diff --git a/onnxruntime/test/framework/test_utils.h b/onnxruntime/test/framework/test_utils.h index 7ca43840f3..d9edf64073 100644 --- a/onnxruntime/test/framework/test_utils.h +++ b/onnxruntime/test/framework/test_utils.h @@ -13,7 +13,7 @@ #include "gsl/gsl" #ifdef USE_CUDA -#include "core/providers/cuda/cuda_execution_provider.h" +#include "core/providers/providers.h" #endif #ifdef USE_ROCM #include "core/providers/rocm/rocm_execution_provider.h" @@ -35,24 +35,9 @@ namespace test { // Doesn't work with ExecutionProviders class and KernelRegistryManager IExecutionProvider* TestCPUExecutionProvider(); -#ifdef USE_CUDA -// Doesn't work with ExecutionProviders class and KernelRegistryManager -IExecutionProvider* TestCudaExecutionProvider(); -#endif - #ifdef USE_ROCM IExecutionProvider* TestRocmExecutionProvider(); #endif - -#ifdef USE_TENSORRT -// Doesn't work with ExecutionProviders class and KernelRegistryManager -IExecutionProvider* TestTensorrtExecutionProvider(); -#endif - -#ifdef USE_OPENVINO -IExecutionProvider* TestOpenVINOExecutionProvider(); -#endif - #ifdef USE_NNAPI IExecutionProvider* TestNnapiExecutionProvider(); #endif diff --git a/onnxruntime/test/providers/cuda/gemm_options_test.cc b/onnxruntime/test/providers/cuda/gemm_options_test.cc index 3dec06da25..2ec64551de 100644 --- a/onnxruntime/test/providers/cuda/gemm_options_test.cc +++ b/onnxruntime/test/providers/cuda/gemm_options_test.cc @@ -1,13 +1,18 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. -#ifdef USE_CUDA +#if 0 // TODO: Can't call these directly from external code as Cuda is now a shared library +//#ifdef USE_CUDA #include "gtest/gtest.h" #include "core/providers/cuda/cuda_common.h" namespace onnxruntime { namespace cuda { + +// Initialize the singleton instance +HalfGemmOptions HalfGemmOptions::instance; + namespace test { TEST(CudaGemmOptionsTest, DefaultOptions) { diff --git a/onnxruntime/test/providers/cuda/reduction_functions_test.cc b/onnxruntime/test/providers/cuda/reduction_functions_test.cc index 23a37a5e80..7a677b6ed2 100644 --- a/onnxruntime/test/providers/cuda/reduction_functions_test.cc +++ b/onnxruntime/test/providers/cuda/reduction_functions_test.cc @@ -1,7 +1,8 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. -#ifdef USE_CUDA +#if 0 // TODO: Can't call these directly from external code as Cuda is now a shared library +//#ifdef USE_CUDA #include diff --git a/onnxruntime/test/providers/provider_test_utils.h b/onnxruntime/test/providers/provider_test_utils.h index f14941a95d..e4c12f69f8 100644 --- a/onnxruntime/test/providers/provider_test_utils.h +++ b/onnxruntime/test/providers/provider_test_utils.h @@ -20,6 +20,7 @@ #include "test/framework/TestAllocatorManager.h" #include "core/framework/TensorSeq.h" #include "core/framework/session_options.h" +#include "core/providers/providers.h" #include "test/util/include/asserts.h" #include "gmock/gmock.h" diff --git a/onnxruntime/test/providers/tensorrt/tensorrt_basic_test.cc b/onnxruntime/test/providers/tensorrt/tensorrt_basic_test.cc index b605428cab..c24401a1b8 100644 --- a/onnxruntime/test/providers/tensorrt/tensorrt_basic_test.cc +++ b/onnxruntime/test/providers/tensorrt/tensorrt_basic_test.cc @@ -27,7 +27,9 @@ void VerifyOutputs(const std::vector& fetches, const std::vector inputs; @@ -68,7 +70,7 @@ TEST(TensorrtExecutionProviderTest, EngineCachingTest) { run_options.run_tag = so.session_logid; InferenceSession session_object{so, GetEnvironment()}; auto allocator_manager = session_object.GetAllocatorManager(); - auto cuda_provider = TestCudaExecutionProvider(); + auto cuda_provider = DefaultCudaExecutionProvider(); cuda_provider->RegisterAllocator(allocator_manager); auto cpu_allocator = cuda_provider->GetAllocator(0, OrtMemTypeCPU); // First run with input shape {1, 3, 2} @@ -180,7 +182,7 @@ TEST(TensorrtExecutionProviderTest, FunctionTest) { InferenceSession session_object{so, GetEnvironment()}; auto allocator_manager = session_object.GetAllocatorManager(); - auto cuda_provider = TestCudaExecutionProvider(); + auto cuda_provider = DefaultCudaExecutionProvider(); cuda_provider->RegisterAllocator(allocator_manager); auto cpu_allocator = cuda_provider->GetAllocator(0, OrtMemTypeCPU); @@ -292,7 +294,7 @@ TEST(TensorrtExecutionProviderTest, NodeIndexMappingTest) { InferenceSession session_object{so, GetEnvironment()}; auto allocator_manager = session_object.GetAllocatorManager(); - auto cuda_provider = TestCudaExecutionProvider(); + auto cuda_provider = DefaultCudaExecutionProvider(); cuda_provider->RegisterAllocator(allocator_manager); auto cpu_allocator = cuda_provider->GetAllocator(0, OrtMemTypeCPU); @@ -413,7 +415,7 @@ TEST(TensorrtExecutionProviderTest, RemoveCycleTest) { InferenceSession session_object{so, GetEnvironment()}; auto allocator_manager = session_object.GetAllocatorManager(); - auto cuda_provider = TestCudaExecutionProvider(); + auto cuda_provider = DefaultCudaExecutionProvider(); cuda_provider->RegisterAllocator(allocator_manager); auto cpu_allocator = cuda_provider->GetAllocator(0, OrtMemTypeCPU); diff --git a/onnxruntime/test/python/onnxruntime_test_python.py b/onnxruntime/test/python/onnxruntime_test_python.py index 15751d3166..2b7186ea76 100644 --- a/onnxruntime/test/python/onnxruntime_test_python.py +++ b/onnxruntime/test/python/onnxruntime_test_python.py @@ -962,18 +962,27 @@ class TestInferenceSession(unittest.TestCase): # exclude for macos and linux if not sys.platform.startswith("win"): return - + + # Exclude for training + training_enabled = False + try: + from onnxruntime.capi.ort_trainer import ORTTrainer + training_enabled = True + except: + pass + + if training_enabled: + return + shared_library = 'test_execution_provider.dll' if not os.path.exists(shared_library): raise FileNotFoundError("Unable to find '{0}'".format(shared_library)) - this = os.path.dirname(__file__) custom_op_model = os.path.join(this, "testdata", "custom_execution_provider_library", "test_model.onnx") if not os.path.exists(custom_op_model): raise FileNotFoundError("Unable to find '{0}'".format(custom_op_model)) - from onnxruntime.capi import _pybind_state as C session_options = C.get_default_session_options() sess = C.InferenceSession(session_options, custom_op_model, True, True) diff --git a/onnxruntime/test/util/default_providers.cc b/onnxruntime/test/util/default_providers.cc index 72c70f9728..9418bda827 100644 --- a/onnxruntime/test/util/default_providers.cc +++ b/onnxruntime/test/util/default_providers.cc @@ -20,6 +20,7 @@ namespace onnxruntime { std::shared_ptr CreateExecutionProviderFactory_OpenVINO( const char* device_type, bool enable_vpu_fast_compile, const char* device_id, size_t num_of_threads, bool use_compiled_network, const char* blob_dump_path); +std::shared_ptr CreateExecutionProviderFactory_Cuda(const OrtCUDAProviderOptions* provider_options); std::shared_ptr CreateExecutionProviderFactory_Dnnl(int use_arena); std::shared_ptr CreateExecutionProviderFactory_OpenVINO(const OrtOpenVINOProviderOptions* params); std::shared_ptr CreateExecutionProviderFactory_Nuphar(bool, const char*); @@ -87,10 +88,12 @@ std::unique_ptr DefaultOpenVINOExecutionProvider() { std::unique_ptr DefaultCudaExecutionProvider() { #ifdef USE_CUDA - return CreateExecutionProviderFactory_CUDA(CUDAExecutionProviderInfo{})->CreateProvider(); -#else - return nullptr; + OrtCUDAProviderOptions provider_options{}; + provider_options.do_copy_in_default_stream = true; + if (auto factory = CreateExecutionProviderFactory_Cuda(&provider_options)) + return factory->CreateProvider(); #endif + return nullptr; } std::unique_ptr DefaultDnnlExecutionProvider(bool enable_arena) { diff --git a/orttraining/orttraining/core/framework/adasum/adasum_interface.h b/orttraining/orttraining/core/framework/adasum/adasum_interface.h index 631167c0ed..0cd164b31f 100644 --- a/orttraining/orttraining/core/framework/adasum/adasum_interface.h +++ b/orttraining/orttraining/core/framework/adasum/adasum_interface.h @@ -7,11 +7,12 @@ #include #include +#ifndef SHARED_PROVIDER #include "core/framework/tensor.h" #include "core/framework/op_kernel.h" +#endif #include "orttraining/core/graph/optimizer_config.h" - #include "orttraining/core/framework/distributed_run_context.h" #ifdef ENABLE_CPU_FP16_TRAINING_OPS @@ -21,16 +22,14 @@ namespace onnxruntime { namespace training { -static inline bool IsPowerOfTwo(ulong x) -{ +static inline bool IsPowerOfTwo(unsigned x) { return (x != 0) && ((x & (x - 1)) == 0); } // Interface for Adasum algorithm template class AdasumInterface { -public: - + public: Status DispatchFusedAllreduce(void* grad_buffer, void* recv_buffer, std::vector& tensor_counts, int start_level, Communicator_type communicator, int tag, @@ -44,7 +43,7 @@ public: FusedAllreduce((float*)grad_buffer, (float*)recv_buffer, data_type, tensor_counts, start_level, communicator, tag, reduction_comms); - } else if(data_type == DataTypeImpl::GetType()) { + } else if (data_type == DataTypeImpl::GetType()) { FusedAllreduce((double*)grad_buffer, (double*)recv_buffer, data_type, tensor_counts, start_level, communicator, tag, reduction_comms); @@ -61,7 +60,7 @@ public: virtual const Communicator_type* GetReductionComms() = 0; -protected: + protected: // Communication primitives required for Adasum algorithm virtual void PointToPointSendRecv(void* input_data_buffer, int64_t input_buffer_bytes, @@ -113,7 +112,7 @@ protected: } } -private: + private: // Allocator for temporary buffer allocations AllocatorPtr allocator_ = nullptr; @@ -161,7 +160,7 @@ private: if (IsPowerOfTwo(size) == false) { ORT_THROW( - "Adasum doesn't currently support reduction among non-power-of-2 number of ranks."); + "Adasum doesn't currently support reduction among non-power-of-2 number of ranks."); } std::vector> nghrCountVec; @@ -213,10 +212,10 @@ private: tensor_counts[i] = 0; } else { nghrCountVec[nghrCountVec_index][i] = - nghrCount - nghrCountSoFar; // should not be negative + nghrCount - nghrCountSoFar; // should not be negative tensor_counts[i] = tensor_counts[i] - - (nghrCount - nghrCountSoFar); // should not be negative + (nghrCount - nghrCountSoFar); // should not be negative } } else { tensor_counts[i] = tensor_counts[i]; @@ -240,9 +239,9 @@ private: assert((myCount - myCountSoFar) >= 0); nghrCountVec[nghrCountVec_index][i] = tensor_counts[i] - - (myCount - myCountSoFar); // should not be negative + (myCount - myCountSoFar); // should not be negative tensor_counts[i] = - myCount - myCountSoFar; // should not be negative + myCount - myCountSoFar; // should not be negative } } else { nghrCountVec[nghrCountVec_index][i] = tensor_counts[i]; @@ -264,8 +263,8 @@ private: recv_buffer = &recv_buffer[nghrCount]; } FusedPairwiseReduceWithComm((uint8_t*)grad_buffer, (uint8_t*)recv_buffer, - data_type, tensor_counts, reduction_comms[comm_index], - (rank & level) == 0, normAndDots); + data_type, tensor_counts, reduction_comms[comm_index], + (rank & level) == 0, normAndDots); } for (level = (size >> 1); level > 0; level = (level >> 1)) { @@ -383,5 +382,5 @@ private: } }; -} // namespace training -} // namespace onnxruntime +} // namespace training +} // namespace onnxruntime diff --git a/orttraining/orttraining/core/framework/communication/mpi/mpi_context.cc b/orttraining/orttraining/core/framework/communication/mpi/mpi_context.cc index 2140e60bfc..ef54afee7a 100644 --- a/orttraining/orttraining/core/framework/communication/mpi/mpi_context.cc +++ b/orttraining/orttraining/core/framework/communication/mpi/mpi_context.cc @@ -1,6 +1,8 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. +#define SHARED_PROVIDER_TODO 0 + #include "orttraining/core/framework/communication/mpi/mpi_context.h" #ifndef _WIN32 #include @@ -17,17 +19,19 @@ MPIContext::~MPIContext() { #ifdef USE_MPI #ifndef _WIN32 // Assume ungraceful shutdown. - std::atomic perform_graceful_exit {false}; + std::atomic perform_graceful_exit{false}; auto release_func_executor_thread = std::thread([this, &perform_graceful_exit]() { - ReleaseComms(); - perform_graceful_exit = true; - }); + ReleaseComms(); + perform_graceful_exit = true; + }); // Wait MPI_TIMEOUT_IN_SECONDS seconds and check the flag again, if it's still false, // that means some process has crashed, not responding or unable to complete pending communications // due to unknown reasons, we then proceed to an ungraceful exit. std::this_thread::sleep_for(std::chrono::seconds(MPIContext::MPI_TIMEOUT_IN_SECONDS)); if (!perform_graceful_exit) { +#if SHARED_PROVIDER_TODO LOGS(logger_, INFO) << "MPI is not able to gracefully shut down. Aborting MPI."; +#endif // Request to cancel the thread since it's not responsive. pthread_t native_handle = release_func_executor_thread.native_handle(); pthread_cancel(native_handle); @@ -38,8 +42,8 @@ MPIContext::~MPIContext() { shutdown_mpi(perform_graceful_exit); #else ReleaseComms(); -#endif // _WIN32 -#endif // USE_MPI +#endif // _WIN32 +#endif // USE_MPI } MPIContext& MPIContext::GetInstance() { @@ -47,13 +51,12 @@ MPIContext& MPIContext::GetInstance() { return context; } -// Default constructor of MPIContext only creates global parallel group i.e. MPI_WORLD +// Default constructor of MPIContext only creates global parallel group i.e. MPI_WORLD // and nodel local parallel group. -MPIContext::MPIContext() : -world_rank_(0), -local_rank_(0), -world_size_(1), -local_size_(1) { +MPIContext::MPIContext() : world_rank_(0), + local_rank_(0), + world_size_(1), + local_size_(1) { #ifdef USE_MPI // setup MPI int is_mpi_initialized = 0; @@ -79,7 +82,7 @@ local_size_(1) { MPI_Comm shmcomm; MPI_CHECK(MPI_Comm_split_type(MPI_COMM_WORLD, MPI_COMM_TYPE_SHARED, 0, - MPI_INFO_NULL, &shmcomm)); + MPI_INFO_NULL, &shmcomm)); MPI_CHECK(MPI_Comm_rank(shmcomm, &local_rank)); MPI_CHECK(MPI_Comm_size(shmcomm, &local_size)); @@ -88,10 +91,12 @@ local_size_(1) { char version[MPI_MAX_LIBRARY_VERSION_STRING]; MPI_Get_library_version(version, &len); +#if SHARED_PROVIDER_TODO LOGS(logger_, INFO) << "MPI context initialized. World size: " << world_size - << ". World rank: " << world_rank - << ". Local size: " << local_size - << ". Local rank: " << local_rank; + << ". World rank: " << world_rank + << ". Local size: " << local_size + << ". Local rank: " << local_rank; +#endif mpi_groups_.resize(WorkerGroupType::WorkerGroupTypeCount); // Create global parallel group @@ -124,17 +129,17 @@ void MPIContext::ReleaseComms() { // If MPI is finalized, return right away. int is_mpi_finalized = 0; MPI_Finalized(&is_mpi_finalized); - if(is_mpi_finalized) + if (is_mpi_finalized) return; for (auto group : mpi_groups_) { - if (group.is_group_initialized) { - MPI_CHECK(MPI_Group_free(&group.mpi_group)); + if (group.is_group_initialized) { + MPI_CHECK(MPI_Group_free(&group.mpi_group)); #ifndef _WIN32 - MPI_CHECK(MPI_Comm_disconnect(&group.communicator)); + MPI_CHECK(MPI_Comm_disconnect(&group.communicator)); #else - MPI_CHECK(MPI_Comm_free(&group.communicator)); + MPI_CHECK(MPI_Comm_free(&group.communicator)); #endif - } + } } } #endif @@ -142,8 +147,10 @@ void MPIContext::ReleaseComms() { void MPIContext::AddMPIGroup(WorkerGroupType group_type, WorkerGroup& group) { #ifdef USE_MPI auto group_name = DistributedRunContext::GetInstance().GetWorkerGroupName(group_type); - if(this->mpi_groups_[group_type].is_group_initialized) { - LOGS(logger_, INFO) << "Group "<mpi_groups_[group_type].is_group_initialized) { +#if SHARED_PROVIDER_TODO + LOGS(logger_, INFO) << "Group " << group_name << " already exists. Re-initializing with different ranks."; +#endif MPI_CHECK(MPI_Group_free(&this->mpi_groups_[group_type].mpi_group)); MPI_CHECK(MPI_Comm_free(&this->mpi_groups_[group_type].communicator)); } @@ -163,13 +170,12 @@ void MPIContext::AddMPIGroup(WorkerGroupType group_type, WorkerGroup& group) { "Failed to add new MPI group for worker group: ", DistributedRunContext::GetInstance().GetWorkerGroupName(group_type)); #else - ORT_THROW("ORT must be built with MPI to add ", DistributedRunContext::GetInstance().GetWorkerGroupName(group_type) - , " with group id: ", group.group_id); + ORT_THROW("ORT must be built with MPI to add ", DistributedRunContext::GetInstance().GetWorkerGroupName(group_type), " with group id: ", group.group_id); #endif } #ifdef USE_MPI -void MPIContext::shutdown_mpi(bool perform_graceful_exit/*default=true*/) { +void MPIContext::shutdown_mpi(bool perform_graceful_exit /*default=true*/) { int is_mpi_initialized = 0; MPI_CHECK(MPI_Initialized(&is_mpi_initialized)); if (!is_mpi_initialized) diff --git a/orttraining/orttraining/core/framework/communication/mpi/mpi_context.h b/orttraining/orttraining/core/framework/communication/mpi/mpi_context.h index b1ac54940e..750077c56f 100644 --- a/orttraining/orttraining/core/framework/communication/mpi/mpi_context.h +++ b/orttraining/orttraining/core/framework/communication/mpi/mpi_context.h @@ -2,9 +2,11 @@ // Licensed under the MIT License. #pragma once +#ifndef SHARED_PROVIDER #include "core/common/common.h" -#include "orttraining/core/framework/distributed_run_context.h" #include "core/common/logging/logging.h" +#endif +#include "orttraining/core/framework/distributed_run_context.h" #if defined(USE_MPI) #include @@ -14,76 +16,78 @@ namespace onnxruntime { namespace training { #if defined(USE_MPI) -#define MPI_CHECK(condition) \ - do { \ - int error = (condition); \ - ORT_ENFORCE( \ - error == MPI_SUCCESS, \ - "MPI Error at: ", \ - __FILE__, \ - ":", \ - __LINE__, \ - ": ", \ - error); \ +#define MPI_CHECK(condition) \ + do { \ + int error = (condition); \ + ORT_ENFORCE( \ + error == MPI_SUCCESS, \ + "MPI Error at: ", \ + __FILE__, \ + ":", \ + __LINE__, \ + ": ", \ + error); \ } while (0) #endif struct MPIGroup { #if defined(USE_MPI) - MPI_Group mpi_group {MPI_GROUP_EMPTY}; // MPI group - MPI_Comm communicator {MPI_COMM_NULL}; // MPI communicator of this group + MPI_Group mpi_group{MPI_GROUP_EMPTY}; // MPI group + MPI_Comm communicator{MPI_COMM_NULL}; // MPI communicator of this group #endif - bool is_group_initialized {false}; // Whether it's initialized + bool is_group_initialized{false}; // Whether it's initialized }; class MPIContext { // https://stackoverflow.com/questions/1008019/c-singleton-design-pattern - public: - static MPIContext& GetInstance(); + public: + static MPIContext& GetInstance(); - MPIContext(MPIContext const&) = delete; - void operator=(MPIContext const&) = delete; + MPIContext(MPIContext const&) = delete; + void operator=(MPIContext const&) = delete; - // within ~MPIContext() we need to check for _WIN32 before calling shutdown_mpi(). - ~MPIContext(); + // within ~MPIContext() we need to check for _WIN32 before calling shutdown_mpi(). + ~MPIContext(); - void AddMPIGroup(WorkerGroupType group_type, WorkerGroup& group); + void AddMPIGroup(WorkerGroupType group_type, WorkerGroup& group); - const std::vector& GetAllMPIGroups() const { return mpi_groups_; } - - const MPIGroup& GetMPIGroup(WorkerGroupType group_type) const { return mpi_groups_[group_type]; } + const std::vector& GetAllMPIGroups() const { return mpi_groups_; } - int GetWorldRank() const { return world_rank_; } - int GetLocalRank() const { return local_rank_; } - int GetWorldSize() const { return world_size_; } - int GetLocalSize() const { return local_size_; } - - const static int MPI_TIMEOUT_IN_SECONDS = 10; + const MPIGroup& GetMPIGroup(WorkerGroupType group_type) const { return mpi_groups_[group_type]; } + + int GetWorldRank() const { return world_rank_; } + int GetLocalRank() const { return local_rank_; } + int GetWorldSize() const { return world_size_; } + int GetLocalSize() const { return local_size_; } + + const static int MPI_TIMEOUT_IN_SECONDS = 10; #if defined(USE_MPI) - // https://docs.microsoft.com/en-us/windows/win32/dlls/dynamic-link-library-best-practices - // in case of _WIN32 we cannot call shutdown_mpi() in MPIContext destructor because of DllMain's restriction - // shutdown_mpi shall be called specifically in user code. - static void shutdown_mpi(bool perform_graceful_exit = true); + // https://docs.microsoft.com/en-us/windows/win32/dlls/dynamic-link-library-best-practices + // in case of _WIN32 we cannot call shutdown_mpi() in MPIContext destructor because of DllMain's restriction + // shutdown_mpi shall be called specifically in user code. + static void shutdown_mpi(bool perform_graceful_exit = true); #endif - private: - MPIContext(); + private: + MPIContext(); - // Groups containing mpi communicator for any worker group. - std::vector mpi_groups_; + // Groups containing mpi communicator for any worker group. + std::vector mpi_groups_; #if defined(USE_MPI) - // Global counter for MPI groups - int mpi_group_id_ = 0; - void setup_mpi(); - void ReleaseComms(); + // Global counter for MPI groups + int mpi_group_id_ = 0; + void setup_mpi(); + void ReleaseComms(); #endif - int world_rank_; - int local_rank_; - int world_size_; - int local_size_; + int world_rank_; + int local_rank_; + int world_size_; + int local_size_; - const logging::Logger& logger_ = logging::LoggingManager::DefaultLogger(); +#if SHARED_PROVIDER_TODO + const logging::Logger& logger_ = logging::LoggingManager::DefaultLogger(); +#endif }; } // namespace training diff --git a/orttraining/orttraining/core/framework/communication/mpi/mpi_utilities.h b/orttraining/orttraining/core/framework/communication/mpi/mpi_utilities.h index e9910f2c46..ee966427a4 100644 --- a/orttraining/orttraining/core/framework/communication/mpi/mpi_utilities.h +++ b/orttraining/orttraining/core/framework/communication/mpi/mpi_utilities.h @@ -3,15 +3,18 @@ #pragma once +#ifndef SHARED_PROVIDER #include "core/framework/tensor.h" #include "core/framework/op_kernel.h" +#endif + #ifdef USE_MPI #include #endif namespace onnxruntime { namespace training { #ifdef USE_MPI -MPI_Datatype GetMPIDataType (MLDataType data_type); +MPI_Datatype GetMPIDataType(MLDataType data_type); int GetMPIRank(MPI_Comm comm); diff --git a/orttraining/orttraining/core/framework/distributed_run_context.h b/orttraining/orttraining/core/framework/distributed_run_context.h index 366c15523f..617c7c962f 100644 --- a/orttraining/orttraining/core/framework/distributed_run_context.h +++ b/orttraining/orttraining/core/framework/distributed_run_context.h @@ -33,8 +33,7 @@ struct WorkerGroup { std::string ToString() const { std::stringstream msg; - msg << "group_type: " << group_type << ", group_id: " << group_id << - ", rank in group:" << rank_in_group << ", world-rank:" << ranks.at(rank_in_group); + msg << "group_type: " << group_type << ", group_id: " << group_id << ", rank in group:" << rank_in_group << ", world-rank:" << ranks.at(rank_in_group); msg << ", ranks: ["; for (size_t i = 0; i < ranks.size(); ++i) { msg << ranks.at(i); @@ -81,10 +80,13 @@ class DistributedRunContext { config.pipeline_stage_size); } +#ifndef SHARED_PROVIDER static DistributedRunContext& GetInstance() { return DistributedRunContext::GetOrCreateInstance(); } - +#else + static DistributedRunContext& GetInstance() { return Provider_GetHost()->GetDistributedRunContextInstance(); } +#endif /* SHORTCUT FUNCTIONS START */ static DistributedRunConfig& RunConfig() { @@ -121,7 +123,7 @@ class DistributedRunContext { return DistributedRunContext::GetInstance().GetWorkerGroup(group_type).group_id; } - static std::vector GetRanks(WorkerGroupType group_type){ + static std::vector GetRanks(WorkerGroupType group_type) { return DistributedRunContext::GetInstance().GetWorkerGroup(group_type).ranks; } diff --git a/orttraining/orttraining/core/graph/optimizer_config.h b/orttraining/orttraining/core/graph/optimizer_config.h index d6b31809c5..934c431e13 100644 --- a/orttraining/orttraining/core/graph/optimizer_config.h +++ b/orttraining/orttraining/core/graph/optimizer_config.h @@ -5,10 +5,12 @@ #include #include +#ifndef SHARED_PROVIDER #include "core/common/logging/logging.h" #include "core/framework/framework_common.h" -#include "core/framework/ml_value.h" #include "core/graph/node_arg.h" +#endif +#include "core/framework/ml_value.h" namespace onnxruntime { namespace training { diff --git a/orttraining/orttraining/core/session/tensor_helper.cc b/orttraining/orttraining/core/session/tensor_helper.cc index ec7380e251..a9e7b39269 100644 --- a/orttraining/orttraining/core/session/tensor_helper.cc +++ b/orttraining/orttraining/core/session/tensor_helper.cc @@ -3,11 +3,11 @@ #include "orttraining/core/session/tensor_helper.h" -#ifdef USE_CUDA -#include "core/providers/cuda/cuda_common.h" -#endif - namespace onnxruntime { + +// Copy a chunk of memory to CPU from GPU. +void CopyGpuToCpu(void* dst_ptr, const void* src_ptr, const size_t size, const OrtMemoryInfo& dst_location, const OrtMemoryInfo& src_location); + namespace training { // Return the shape of a tensor slice. @@ -55,41 +55,6 @@ OrtValue CreateCpuTensorValue( return cpu_value; } -// Copy a chunk of memory to CPU from GPU. -void CopyGpuToCpu( - void* dst_ptr, - const void* src_ptr, - const size_t size, - const OrtMemoryInfo& dst_location, - const OrtMemoryInfo& src_location) { - ORT_ENFORCE(dst_location.device.Type() == OrtDevice::CPU); - -#ifdef USE_CUDA - // Current CUDA device. - int device; - CUDA_CALL(cudaGetDevice(&device)); - - if (device != src_location.id) { - // Need to switch to the allocating device. - CUDA_CALL(cudaSetDevice(src_location.id)); - // Copy from GPU to CPU. - CUDA_CALL(cudaMemcpy(dst_ptr, src_ptr, size, cudaMemcpyDeviceToHost)); - // Switch back to current device. - CUDA_CALL(cudaSetDevice(device)); - } else { - // Copy from GPU to CPU. - CUDA_CALL(cudaMemcpy(dst_ptr, src_ptr, size, cudaMemcpyDeviceToHost)); - } -#else - ORT_UNUSED_PARAMETER(dst_ptr); - ORT_UNUSED_PARAMETER(src_ptr); - ORT_UNUSED_PARAMETER(size); - ORT_UNUSED_PARAMETER(dst_location); - ORT_UNUSED_PARAMETER(src_location); - ORT_THROW("CPU-to-CPU copy is not implemented."); -#endif -} - // Copy a chunk of memory to CPU from CPU. void CopyCpuToCpu( void* dst_ptr, @@ -270,7 +235,7 @@ void CopyConcat( auto chunk_size = src_shape[concat_axis] * segment_size; // Bias of the i_seg-th segment in input tensor. Its unit is the number of tensor elements. - // chunk_size * + // chunk_size * auto src_bias = UpdateLinearIndex(i_seg, 0, src_shape[concat_axis]) * segment_size; auto dst_bias = UpdateLinearIndex(i_seg, anchor_bias, dst_shape[concat_axis]) * segment_size; @@ -333,4 +298,4 @@ OrtValue ConcatenateTensors( return result_value; } } // namespace training -} // namespace onnxruntime \ No newline at end of file +} // namespace onnxruntime diff --git a/orttraining/orttraining/core/session/training_session.cc b/orttraining/orttraining/core/session/training_session.cc index eda1042434..57f8816aba 100644 --- a/orttraining/orttraining/core/session/training_session.cc +++ b/orttraining/orttraining/core/session/training_session.cc @@ -28,11 +28,6 @@ //Gist Encoding #include "orttraining/core/optimizer/gist_encode_decode.h" -#ifdef USE_CUDA -#include "core/providers/cuda/cuda_common.h" -#include "core/providers/cuda/cuda_allocator.h" -#endif - #include "orttraining/training_ops/cpu/controlflow/event_pool.h" #if defined(USE_CUDA) && defined(ORT_USE_NCCL) && defined(USE_NCCL_P2P) #include "orttraining/training_ops/cuda/communication/nccl_service.h" @@ -1425,7 +1420,7 @@ std::unordered_set TrainingSession::GetTrainableModelInitializers( // Send and Recv call SubmitSendAndWait and SubmitRecvAndWait, respectively. void PipelineTrainingSession::LaunchNcclService(const int pipeline_stage_id) { ORT_ENFORCE(pipeline_stage_id >= 0, "Pipeline stage ID cannot be negative."); - auto& nccl_service = cuda::NcclService::GetInstance(); + auto& nccl_service = cuda::INcclService::GetInstance(); // Create NCCL communication plan. The plan is a vector of communication task group. // Each communication task group contains tasks which should be done in parallel. @@ -1902,7 +1897,7 @@ common::Status PipelineTrainingSession::RunWithPipeline(const RunOptions& run_op pipeline_worker_pool_.JoinAll(); onnxruntime::contrib::OrtEventPool::GetInstance().ResetAllEvents(); #if defined(USE_CUDA) && defined(ORT_USE_NCCL) && defined(USE_NCCL_P2P) - auto& nccl_service = cuda::NcclService::GetInstance(); + auto& nccl_service = cuda::INcclService::GetInstance(); nccl_service.Reset(); #endif @@ -1911,7 +1906,7 @@ common::Status PipelineTrainingSession::RunWithPipeline(const RunOptions& run_op PipelineTrainingSession::~PipelineTrainingSession() { #if defined(USE_CUDA) && defined(ORT_USE_NCCL) && defined(USE_NCCL_P2P) - auto& nccl_service = cuda::NcclService::GetInstance(); + auto& nccl_service = cuda::INcclService::GetInstance(); nccl_service.Terminate(); #endif } diff --git a/orttraining/orttraining/models/bert/main.cc b/orttraining/orttraining/models/bert/main.cc index efd0327a66..1c6b02cdc5 100644 --- a/orttraining/orttraining/models/bert/main.cc +++ b/orttraining/orttraining/models/bert/main.cc @@ -12,8 +12,10 @@ #include "core/framework/random_seed.h" #include "core/providers/cpu/cpu_provider_factory_creator.h" #ifdef USE_CUDA -#include "core/providers/cuda/cuda_allocator.h" -#include "core/providers/cuda/cuda_provider_factory_creator.h" +namespace onnxruntime { +std::shared_ptr CreateExecutionProviderFactory_Cuda(const OrtCUDAProviderOptions* provider_options); +std::unique_ptr CreateCUDAPinnedAllocator(int16_t device_id, const char* name); +} // namespace onnxruntime #endif #ifdef USE_ROCM #include "core/providers/rocm/rocm_allocator.h" @@ -607,15 +609,23 @@ void setup_training_params(BertParameters& params) { #ifdef USE_CUDA { - CUDAExecutionProviderInfo info{}; - info.device_id = gsl::narrow(MPIContext::GetInstance().GetLocalRank()); + OrtCUDAProviderOptions info{ + gsl::narrow(MPIContext::GetInstance().GetLocalRank()), + OrtCudnnConvAlgoSearch::EXHAUSTIVE, + std::numeric_limits::max(), + 0, + true, + 0, + nullptr, + nullptr}; + if (params.gpu_mem_limit_in_gb > 0) { info.gpu_mem_limit = gsl::narrow(params.gpu_mem_limit_in_gb * 1024 * 1024 * 1024); } info.cudnn_conv_algo_search = OrtCudnnConvAlgoSearch::EXHAUSTIVE; - params.providers.emplace(kCudaExecutionProvider, CreateExecutionProviderFactory_CUDA(info)); - params.input_allocator = std::make_shared(info.device_id, CUDA_PINNED); + params.providers.emplace(kCudaExecutionProvider, CreateExecutionProviderFactory_Cuda(&info)); + params.input_allocator = CreateCUDAPinnedAllocator(info.device_id, CUDA_PINNED); } #endif diff --git a/orttraining/orttraining/models/gpt2/main.cc b/orttraining/orttraining/models/gpt2/main.cc index 50e8bf59bc..773fbac07d 100644 --- a/orttraining/orttraining/models/gpt2/main.cc +++ b/orttraining/orttraining/models/gpt2/main.cc @@ -9,9 +9,12 @@ #include "core/session/environment.h" #include "core/framework/random_seed.h" #include "core/framework/bfc_arena.h" +#include "core/providers/providers.h" #ifdef USE_CUDA -#include "core/providers/cuda/cuda_allocator.h" -#include "core/providers/cuda/cuda_provider_factory_creator.h" +namespace onnxruntime { +std::shared_ptr CreateExecutionProviderFactory_Cuda(const OrtCUDAProviderOptions* provider_options); +std::unique_ptr CreateCUDAPinnedAllocator(int16_t device_id, const char* name); +} // namespace onnxruntime #endif #include "orttraining/core/framework/communication/mpi/mpi_context.h" #include "orttraining/core/framework/tensorboard/event_writer.h" @@ -243,7 +246,7 @@ Status ParseArguments(int argc, char* argv[], GPT2Parameters& params, OrtParamet int64_t seed = flags["seed"].as(); if (params.horizontal_parallel_size > 1 && seed <= 0) { - seed = 8211; // Megatron needs a random seed. + seed = 8211; // Megatron needs a random seed. } if (seed > 0) { utils::SetRandomSeed(seed); @@ -283,7 +286,7 @@ float GetLossValue(const Tensor& loss_tensor) { // mapping to define what to be stored in mapped_dimensions // see GetTensorDimensionsFromInputs() in training_util.h and training_runner.cc for more details const std::map> input_to_dimension_mapping = { - {"input_ids", {"SeqLen", 0}}, // int64[batch,seqlen] "seqlen" -> "SeqLen", 0 + {"input_ids", {"SeqLen", 0}}, // int64[batch,seqlen] "seqlen" -> "SeqLen", 0 }; // generic properties for storing perf metrics @@ -349,11 +352,18 @@ void setup_training_params(GPT2Parameters& params) { #ifdef USE_CUDA { - CUDAExecutionProviderInfo info{}; - info.device_id = gsl::narrow(MPIContext::GetInstance().GetLocalRank()); + OrtCUDAProviderOptions info{ + gsl::narrow(MPIContext::GetInstance().GetLocalRank()), + OrtCudnnConvAlgoSearch::EXHAUSTIVE, + std::numeric_limits::max(), + 0, + true, + 0, + nullptr, + nullptr}; - params.providers.emplace(kCudaExecutionProvider, CreateExecutionProviderFactory_CUDA(info)); - params.input_allocator = std::make_shared(info.device_id, CUDA_PINNED); + params.providers.emplace(kCudaExecutionProvider, CreateExecutionProviderFactory_Cuda(&info)); + params.input_allocator = CreateCUDAPinnedAllocator(info.device_id, CUDA_PINNED); } #endif @@ -430,17 +440,17 @@ static Status RunTraining(const GPT2Parameters& params, const Environment& env) auto rank_in_data_parallel_group = MPIContext::GetInstance().GetWorldRank() / params.horizontal_parallel_size; auto training_data_loader = std::make_unique(params.input_name_map, - params.train_data_dir, - max_num_files_preload, - rank_in_data_parallel_group, - params.data_parallel_size); + params.train_data_dir, + max_num_files_preload, + rank_in_data_parallel_group, + params.data_parallel_size); std::unique_ptr test_data_loader; // Evaluation is only done in device #0 if (MPIContext::GetInstance().GetWorldRank() == 0) { test_data_loader = std::make_unique(params.input_name_map, - params.test_data_dir, - max_num_files_preload); + params.test_data_dir, + max_num_files_preload); } if (!params.perf_output_dir.empty()) { @@ -454,8 +464,8 @@ static Status RunTraining(const GPT2Parameters& params, const Environment& env) // only test and save trained model on device #0 if (MPIContext::GetInstance().GetWorldRank() == 0) { test_data_loader = std::make_unique(params.input_name_map, - params.test_data_dir, - max_num_files_preload); + params.test_data_dir, + max_num_files_preload); ORT_RETURN_IF_ERROR(runner->EndTraining(test_data_loader.get())); } diff --git a/orttraining/orttraining/models/mnist/main.cc b/orttraining/orttraining/models/mnist/main.cc index 287231bd98..a503994896 100644 --- a/orttraining/orttraining/models/mnist/main.cc +++ b/orttraining/orttraining/models/mnist/main.cc @@ -6,8 +6,11 @@ #include "core/common/logging/sinks/clog_sink.h" #include "core/framework/bfc_arena.h" #include "core/platform/env.h" +#include "core/providers/providers.h" #ifdef USE_CUDA -#include "core/providers/cuda/cuda_provider_factory_creator.h" +namespace onnxruntime { +std::shared_ptr CreateExecutionProviderFactory_Cuda(const OrtCUDAProviderOptions* provider_options); +} // namespace onnxruntime #endif #include "core/session/environment.h" #include "orttraining/core/session/training_session.h" @@ -22,11 +25,6 @@ #include namespace onnxruntime { -std::shared_ptr CreateExecutionProviderFactory_CUDA(OrtDevice::DeviceId device_id, - OrtCudnnConvAlgoSearch cudnn_conv_algo_search = OrtCudnnConvAlgoSearch::EXHAUSTIVE, - size_t gpu_mem_limit = std::numeric_limits::max(), - onnxruntime::ArenaExtendStrategy arena_extend_strategy = ArenaExtendStrategy::kNextPowerOfTwo, - bool do_copy_in_default_stream = true); std::shared_ptr CreateExecutionProviderFactory_Dnnl(int use_arena); } @@ -171,7 +169,17 @@ Status ParseArguments(int argc, char* argv[], MnistParameters& params) { #ifdef USE_CUDA bool use_cuda = flags.count("use_cuda") > 0; if (use_cuda) { - params.providers.emplace(kCudaExecutionProvider, CreateExecutionProviderFactory_CUDA(CUDAExecutionProviderInfo{})); + OrtCUDAProviderOptions info{ + 0, + OrtCudnnConvAlgoSearch::EXHAUSTIVE, + std::numeric_limits::max(), + 0, + true, + 0, + nullptr, + nullptr}; + + params.providers.emplace(kCudaExecutionProvider, CreateExecutionProviderFactory_Cuda(&info)); } #endif diff --git a/orttraining/orttraining/models/pipeline_poc/main.cc b/orttraining/orttraining/models/pipeline_poc/main.cc index 6d44578671..0e4fe1716d 100644 --- a/orttraining/orttraining/models/pipeline_poc/main.cc +++ b/orttraining/orttraining/models/pipeline_poc/main.cc @@ -16,7 +16,10 @@ #include "orttraining/models/runner/training_util.h" #include "orttraining/models/runner/data_loader.h" -#include "core/providers/cuda/cuda_execution_provider.h" +#include "core/providers/cuda/cuda_provider_factory_creator.h" +namespace onnxruntime { +std::shared_ptr CreateExecutionProviderFactory_Cuda(const OrtCUDAProviderOptions* provider_options); +} #include #include @@ -111,8 +114,10 @@ int main(int argc, char* argv[]) { InferenceSession session_object{so, *env}; Status st; - CUDAExecutionProviderInfo xp_info{static_cast(world_rank)}; - st = session_object.RegisterExecutionProvider(std::make_unique(xp_info)); + OrtCUDAProviderOptions xp_info{}; + xp_info.device_id = static_cast(world_rank); + auto cuda_factory = CreateExecutionProviderFactory_Cuda(&xp_info); + st = session_object.RegisterExecutionProvider(cuda_factory->CreateProvider()); ORT_ENFORCE(st == Status::OK(), "MPI rank ", world_rank, ": ", st.ErrorMessage()); std::string model_at_rank; diff --git a/orttraining/orttraining/models/runner/training_runner.cc b/orttraining/orttraining/models/runner/training_runner.cc index d85f4ec27b..2f0301417c 100644 --- a/orttraining/orttraining/models/runner/training_runner.cc +++ b/orttraining/orttraining/models/runner/training_runner.cc @@ -764,7 +764,7 @@ Status TrainingRunner::TrainingLoop(IDataLoader& training_data_loader, IDataLoad #if defined(USE_CUDA) && defined(ORT_USE_NCCL) && defined(USE_NCCL_P2P) // Create communication plan. - auto& nccl_service = cuda::NcclService::GetInstance(); + auto& nccl_service = cuda::INcclService::GetInstance(); nccl_service.PlanStart(); for (auto& slot : pipeline_schedule_.GetSchedule(pipeline_context_.pipeline_stage_id)) { diff --git a/orttraining/orttraining/test/framework/slice_concatenate_test.cc b/orttraining/orttraining/test/framework/slice_concatenate_test.cc index fe7e1963e3..21e1a450bf 100644 --- a/orttraining/orttraining/test/framework/slice_concatenate_test.cc +++ b/orttraining/orttraining/test/framework/slice_concatenate_test.cc @@ -7,10 +7,15 @@ #include "core/session/inference_session.h" #include "gtest/gtest.h" #include "orttraining/core/session/tensor_helper.h" +#include "test/util/include/default_providers.h" #include "test/framework/test_utils.h" #include "test/test_environment.h" namespace onnxruntime { +#ifdef USE_CUDA +void cudaMemcpy_HostToDevice(void* dst, const void* src, size_t count); +#endif + namespace test { typedef std::vector ArgMap; @@ -18,7 +23,7 @@ typedef std::vector ArgMap; // Create ML value. OrtValue CreateTensorValue(const std::vector& shape, const std::vector& initializer, const bool allocate_on_gpu) { #ifdef USE_CUDA - auto cpu_allocator = allocate_on_gpu ? TestCudaExecutionProvider()->GetAllocator(0, OrtMemTypeDefault) : TestCPUExecutionProvider()->GetAllocator(0, OrtMemTypeDefault); + auto cpu_allocator = allocate_on_gpu ? DefaultCudaExecutionProvider()->GetAllocator(0, OrtMemTypeDefault) : TestCPUExecutionProvider()->GetAllocator(0, OrtMemTypeDefault); #else ORT_ENFORCE(allocate_on_gpu != true); auto cpu_allocator = TestCPUExecutionProvider()->GetAllocator(0, OrtMemTypeDefault); @@ -34,7 +39,7 @@ OrtValue CreateTensorValue(const std::vector& shape, const std::vector< memcpy(p_tensor->MutableData(), initializer.data(), initializer.size() * sizeof(float)); } else { #ifdef USE_CUDA - cudaMemcpy(p_tensor->MutableData(), initializer.data(), initializer.size() * sizeof(float), cudaMemcpyHostToDevice); + cudaMemcpy_HostToDevice(p_tensor->MutableData(), initializer.data(), initializer.size() * sizeof(float)); #else ORT_THROW("Cannot use CUDA function when ORT is not built with CUDA."); #endif @@ -79,8 +84,7 @@ void InitializeSession(onnxruntime::InferenceSession& session, onnxruntime::Mode // Initialize the session. #if defined(USE_CUDA) - onnxruntime::CUDAExecutionProviderInfo xp_info; - ASSERT_STATUS_OK(session.RegisterExecutionProvider(std::make_unique(xp_info))); + ASSERT_STATUS_OK(session.RegisterExecutionProvider(DefaultCudaExecutionProvider())); #endif ASSERT_STATUS_OK(session.Initialize()); } @@ -378,4 +382,4 @@ TEST(PipelineParallel, FloatTensorConcat1dGpu) { #endif } // namespace test -} // namespace onnxruntime \ No newline at end of file +} // namespace onnxruntime diff --git a/orttraining/orttraining/test/gradient/allreduce_op_test.cc b/orttraining/orttraining/test/gradient/allreduce_op_test.cc index 6f1136398b..a68b54a159 100644 --- a/orttraining/orttraining/test/gradient/allreduce_op_test.cc +++ b/orttraining/orttraining/test/gradient/allreduce_op_test.cc @@ -14,7 +14,9 @@ #include "test/test_environment.h" #ifdef USE_CUDA -#include "core/providers/cuda/cuda_execution_provider.h" +namespace onnxruntime { +std::shared_ptr CreateExecutionProviderFactory_Cuda(const OrtCUDAProviderOptions* provider_options); +} #endif namespace onnxruntime { @@ -29,13 +31,12 @@ TEST(AllreduceTest, CPUAdasumAllreduceTestReduceTwoTensors) { // Alternating inputs to test symmetry std::vector grad_1 = {4.0f, 5.0f, 6.0f}; std::vector grad_2 = {7.0f, 8.0f, 9.0f}; - if (training::MPIContext::GetInstance().GetWorldRank() == 0){ - allreduce_test.AddInput("G1", {3}, grad_1); - allreduce_test.AddInput("G2", {3}, grad_2); - } - else if(training::MPIContext::GetInstance().GetWorldRank() == 1) { - allreduce_test.AddInput("G1", {3}, grad_2); - allreduce_test.AddInput("G2", {3}, grad_1); + if (training::MPIContext::GetInstance().GetWorldRank() == 0) { + allreduce_test.AddInput("G1", {3}, grad_1); + allreduce_test.AddInput("G2", {3}, grad_2); + } else if (training::MPIContext::GetInstance().GetWorldRank() == 1) { + allreduce_test.AddInput("G1", {3}, grad_2); + allreduce_test.AddInput("G2", {3}, grad_1); } std::vector output_grad = {5.6301f, 6.5235f, 7.4169f}; @@ -47,10 +48,10 @@ TEST(AllreduceTest, CPUAdasumAllreduceTestReduceTwoTensors) { std::vector> providers; providers.push_back(DefaultCpuExecutionProvider()); - allreduce_test.Run(OpTester::ExpectResult::kExpectSuccess/*expect_result*/, ""/*expected_failure_string*/, - {}/*excluded_provider_types*/, nullptr/*run_options*/, &providers/*execution_providers*/, - ExecutionMode::ORT_SEQUENTIAL/*execution_mode*/, - {}/*resolve_options*/); + allreduce_test.Run(OpTester::ExpectResult::kExpectSuccess /*expect_result*/, "" /*expected_failure_string*/, + {} /*excluded_provider_types*/, nullptr /*run_options*/, &providers /*execution_providers*/, + ExecutionMode::ORT_SEQUENTIAL /*execution_mode*/, + {} /*resolve_options*/); } TEST(AllreduceTest, CPUAdasumAllreduceTestReduceTwoTensorsFP16) { @@ -68,13 +69,12 @@ TEST(AllreduceTest, CPUAdasumAllreduceTestReduceTwoTensorsFP16) { ConvertFloatToMLFloat16(grad_1.data(), grad_1_half.data(), 3); ConvertFloatToMLFloat16(grad_2.data(), grad_2_half.data(), 3); - if (training::MPIContext::GetInstance().GetWorldRank() == 0){ - allreduce_test.AddInput("G1", {3}, grad_1_half); - allreduce_test.AddInput("G2", {3}, grad_2_half); - } - else if(training::MPIContext::GetInstance().GetWorldRank() == 1) { - allreduce_test.AddInput("G1", {3}, grad_2_half); - allreduce_test.AddInput("G2", {3}, grad_1_half); + if (training::MPIContext::GetInstance().GetWorldRank() == 0) { + allreduce_test.AddInput("G1", {3}, grad_1_half); + allreduce_test.AddInput("G2", {3}, grad_2_half); + } else if (training::MPIContext::GetInstance().GetWorldRank() == 1) { + allreduce_test.AddInput("G1", {3}, grad_2_half); + allreduce_test.AddInput("G2", {3}, grad_1_half); } std::vector output_grad = {6.32478f, 7.2628f, 8.2009f}; @@ -91,10 +91,10 @@ TEST(AllreduceTest, CPUAdasumAllreduceTestReduceTwoTensorsFP16) { std::vector> providers; providers.push_back(DefaultCpuExecutionProvider()); - allreduce_test.Run(OpTester::ExpectResult::kExpectSuccess/*expect_result*/, ""/*expected_failure_string*/, - {}/*excluded_provider_types*/, nullptr/*run_options*/, &providers/*execution_providers*/, - ExecutionMode::ORT_SEQUENTIAL/*execution_mode*/, - {}/*resolve_options*/); + allreduce_test.Run(OpTester::ExpectResult::kExpectSuccess /*expect_result*/, "" /*expected_failure_string*/, + {} /*excluded_provider_types*/, nullptr /*run_options*/, &providers /*execution_providers*/, + ExecutionMode::ORT_SEQUENTIAL /*execution_mode*/, + {} /*resolve_options*/); } TEST(AllreduceTest, CPUAdasumAllreduceTestFailTensorCountMismatch) { @@ -103,12 +103,11 @@ TEST(AllreduceTest, CPUAdasumAllreduceTestFailTensorCountMismatch) { } OpTester allreduce_test("AdasumAllReduce", 1, onnxruntime::kMSDomain); - if (training::MPIContext::GetInstance().GetWorldRank() == 0){ - allreduce_test.AddInput("G1", {3}, {4, 5, 6}); - } - else if(training::MPIContext::GetInstance().GetWorldRank() == 1) { - allreduce_test.AddInput("G1", {3}, {7, 8, 9}); - allreduce_test.AddInput("G2", {3}, {4, 5, 6}); + if (training::MPIContext::GetInstance().GetWorldRank() == 0) { + allreduce_test.AddInput("G1", {3}, {4, 5, 6}); + } else if (training::MPIContext::GetInstance().GetWorldRank() == 1) { + allreduce_test.AddInput("G1", {3}, {7, 8, 9}); + allreduce_test.AddInput("G2", {3}, {4, 5, 6}); } allreduce_test.AddOutput("G_new1", {3}, {5.6301f, 6.5235f, 7.4169f}); @@ -118,10 +117,10 @@ TEST(AllreduceTest, CPUAdasumAllreduceTestFailTensorCountMismatch) { std::vector> providers; providers.push_back(DefaultCpuExecutionProvider()); - allreduce_test.Run(OpTester::ExpectResult::kExpectFailure/*expect_result*/, ""/*expected_failure_string*/, - {}/*excluded_provider_types*/, nullptr/*run_options*/, &providers/*execution_providers*/, - ExecutionMode::ORT_SEQUENTIAL/*execution_mode*/, - {}/*resolve_options*/); + allreduce_test.Run(OpTester::ExpectResult::kExpectFailure /*expect_result*/, "" /*expected_failure_string*/, + {} /*excluded_provider_types*/, nullptr /*run_options*/, &providers /*execution_providers*/, + ExecutionMode::ORT_SEQUENTIAL /*execution_mode*/, + {} /*resolve_options*/); } const std::string lr_string = "ETA"; @@ -212,8 +211,7 @@ void build_optimizer_node(Graph& graph, auto& gradient_out_arg = graph.GetOrCreateNodeArg("", nullptr); optimizer_outputs.push_back(&gradient_out_arg); - } - else { + } else { auto& weight_out_arg = graph.GetOrCreateNodeArg("", nullptr); optimizer_outputs.push_back(&weight_out_arg); ONNX_NAMESPACE::TypeProto gradient_out_tensor; @@ -223,8 +221,8 @@ void build_optimizer_node(Graph& graph, optimizer_outputs.push_back(&gradient_out_arg); } - auto& optimizer_node = graph.AddNode(input_gradient->Name() + "_adam_optimizer", "AdamOptimizer", "Adam optimizer.", optimizer_inputs, optimizer_outputs, - nullptr/*attributes*/, kMSDomain); + auto& optimizer_node = graph.AddNode(input_gradient->Name() + "_adam_optimizer", "AdamOptimizer", "Adam optimizer.", optimizer_inputs, optimizer_outputs, + nullptr /*attributes*/, kMSDomain); ONNX_NAMESPACE::AttributeProto bias_correction_attribute, weight_decay_mode_attribute; @@ -240,20 +238,18 @@ void build_optimizer_node(Graph& graph, optimizer_node.AddAttribute("weight_decay_mode", weight_decay_mode_attribute); } -using AllreduceGraphConfigVector = std::vector>; +using AllreduceGraphConfigVector = std::vector>; void build_allreduce_graph(Graph& graph, AllreduceGraphConfigVector& config, training::AdasumReductionType adasum_reduce_type = training::AdasumReductionType::None, - bool build_optimizer=false, - bool half_precision=false) { - + bool build_optimizer = false, + bool half_precision = false) { std::vector inputs; std::vector outputs; - auto element_type = half_precision ? ONNX_NAMESPACE::TensorProto_DataType_FLOAT16 : - ONNX_NAMESPACE::TensorProto_DataType_FLOAT; + auto element_type = half_precision ? ONNX_NAMESPACE::TensorProto_DataType_FLOAT16 : ONNX_NAMESPACE::TensorProto_DataType_FLOAT; // Tensor proto. ONNX_NAMESPACE::TypeProto float_tensor; @@ -261,22 +257,20 @@ void build_allreduce_graph(Graph& graph, AllreduceGraphConfigVector& config, for (size_t i = 0; i < config.size(); i++) { float_tensor.mutable_tensor_type()->set_elem_type(element_type); float_tensor.mutable_tensor_type()->mutable_shape()->add_dim()->set_dim_value(std::get<2>(config[i])); - + // Input gradient tensor defined tests auto& allreduce_input_arg = graph.GetOrCreateNodeArg(std::get<0>(config[i]), &float_tensor); inputs.push_back(&allreduce_input_arg); // Output tensor defined by tests ONNX_NAMESPACE::TypeProto output_float_tensor; - auto output_type = build_optimizer ? ONNX_NAMESPACE::TensorProto_DataType_FLOAT : - ONNX_NAMESPACE::TensorProto_DataType_FLOAT16; + auto output_type = build_optimizer ? ONNX_NAMESPACE::TensorProto_DataType_FLOAT : ONNX_NAMESPACE::TensorProto_DataType_FLOAT16; output_float_tensor.mutable_tensor_type()->set_elem_type(output_type); output_float_tensor.mutable_tensor_type()->mutable_shape()->add_dim()->set_dim_value(std::get<2>(config[i])); auto& output_arg_1 = graph.GetOrCreateNodeArg(std::get<1>(config[i]), &output_float_tensor); outputs.push_back(&output_arg_1); } - std::string allreduce_op_name = adasum_reduce_type == training::AdasumReductionType::None ? - "NcclAllReduce" : "AdasumAllReduce"; + std::string allreduce_op_name = adasum_reduce_type == training::AdasumReductionType::None ? "NcclAllReduce" : "AdasumAllReduce"; // If using hierarchical reduction, nccl allreduce will be used before adasum to get sum on local ranks. if (adasum_reduce_type == training::AdasumReductionType::GpuHierarchicalReduction) { @@ -288,17 +282,17 @@ void build_allreduce_graph(Graph& graph, AllreduceGraphConfigVector& config, level_1_inputs.push_back(inputs[i]); // Output tensor auto& level_1_output_arg = graph.GetOrCreateNodeArg(std::get<0>(config[i]) + "_level_1_out", &float_tensor); - + level_1_outputs.push_back(&level_1_output_arg); } - auto& level_1_allreduce_node = graph.AddNode("node_level_1", level_1_allreduce, - "level 1 allreduce.", level_1_inputs, level_1_outputs, - nullptr/*attributes*/, kMSDomain); + auto& level_1_allreduce_node = graph.AddNode("node_level_1", level_1_allreduce, + "level 1 allreduce.", level_1_inputs, level_1_outputs, + nullptr /*attributes*/, kMSDomain); ONNX_NAMESPACE::AttributeProto level_1_group_type_attribute; level_1_group_type_attribute.set_name("group_type"); level_1_group_type_attribute.set_type(ONNX_NAMESPACE::AttributeProto_AttributeType::AttributeProto_AttributeType_INT); - level_1_group_type_attribute.set_i(2/*node local data parallel*/); + level_1_group_type_attribute.set_i(2 /*node local data parallel*/); level_1_allreduce_node.AddAttribute("group_type", level_1_group_type_attribute); // Set inputs of next node to be outputs of level 1 reduction node. inputs.clear(); @@ -313,16 +307,16 @@ void build_allreduce_graph(Graph& graph, AllreduceGraphConfigVector& config, scale_tensor.mutable_tensor_type()->mutable_shape()->add_dim()->set_dim_value(1); auto& scale_input_arg = graph.GetOrCreateNodeArg(grad_divisor_string, &scale_tensor); - for (size_t i = 0; i < config.size(); i ++) { + for (size_t i = 0; i < config.size(); i++) { scale_grad_inputs.clear(); scale_grad_inputs.push_back(&scale_input_arg); scale_grad_inputs.push_back(inputs[i]); // Output tensor auto& scale_grad_output_arg = graph.GetOrCreateNodeArg(inputs[i]->Name() + "_scaled", &float_tensor); scale_grad_outputs.push_back(&scale_grad_output_arg); - auto& scaled_grad_node = graph.AddNode(std::get<0>(config[i]) + "_scaled_grad", "MixedPrecisionScale", - "scale grad", scale_grad_inputs, {&scale_grad_output_arg}, - nullptr/*attributes*/, kMSDomain); + auto& scaled_grad_node = graph.AddNode(std::get<0>(config[i]) + "_scaled_grad", "MixedPrecisionScale", + "scale grad", scale_grad_inputs, {&scale_grad_output_arg}, + nullptr /*attributes*/, kMSDomain); ONNX_NAMESPACE::AttributeProto scale_attribute; scale_attribute.set_name("to"); scale_attribute.set_type(ONNX_NAMESPACE::AttributeProto_AttributeType::AttributeProto_AttributeType_INT); @@ -336,7 +330,7 @@ void build_allreduce_graph(Graph& graph, AllreduceGraphConfigVector& config, } // Build optimizer before reduction for Adasum - if(build_optimizer && adasum_reduce_type != training::AdasumReductionType::None) { + if (build_optimizer && adasum_reduce_type != training::AdasumReductionType::None) { std::vector intermediate_output; for (size_t i = 0; i < config.size(); i++) { // Adasum computes on the updated gradients of optimizer, so suffix tensor name with gradient out. @@ -357,14 +351,13 @@ void build_allreduce_graph(Graph& graph, AllreduceGraphConfigVector& config, auto& allreduce_output_arg = graph.GetOrCreateNodeArg(std::get<0>(config[i]) + allreduce_output_suffix_string, &float_tensor); allreduce_outputs.push_back(&allreduce_output_arg); } - } - else { + } else { // If not build_optimizer, outputs of allreduce are graph outputs. allreduce_outputs = std::move(outputs); } - auto& allreduce_node = graph.AddNode("node_allreduce", allreduce_op_name, "node allreduce.", inputs, allreduce_outputs, - nullptr/*attributes*/, kMSDomain); + auto& allreduce_node = graph.AddNode("node_allreduce", allreduce_op_name, "node allreduce.", inputs, allreduce_outputs, + nullptr /*attributes*/, kMSDomain); if (adasum_reduce_type != training::AdasumReductionType::None) { // Attribute ONNX_NAMESPACE::AttributeProto adasum_reduction_type_attribute; @@ -372,28 +365,26 @@ void build_allreduce_graph(Graph& graph, AllreduceGraphConfigVector& config, adasum_reduction_type_attribute.set_type(ONNX_NAMESPACE::AttributeProto_AttributeType::AttributeProto_AttributeType_INT); adasum_reduction_type_attribute.set_i(static_cast(adasum_reduce_type)); allreduce_node.AddAttribute("reduce_algo", adasum_reduction_type_attribute); - } - else { + } else { // Attribute ONNX_NAMESPACE::AttributeProto group_type_attribute; group_type_attribute.set_name("group_type"); group_type_attribute.set_type(ONNX_NAMESPACE::AttributeProto_AttributeType::AttributeProto_AttributeType_INT); - group_type_attribute.set_i(0/*data parallel*/); + group_type_attribute.set_i(0 /*data parallel*/); allreduce_node.AddAttribute("group_type", group_type_attribute); } - if(build_optimizer) { + if (build_optimizer) { if (adasum_reduce_type == training::AdasumReductionType::None) { - // If build_optimizer and regular allreduce, inputs are outputs of previous allreduce node. + // If build_optimizer and regular allreduce, inputs are outputs of previous allreduce node. inputs.clear(); inputs = std::move(allreduce_outputs); - for (size_t i = 0; i< config.size(); i++) { + for (size_t i = 0; i < config.size(); i++) { build_optimizer_node(graph, element_type, std::get<2>(config[i]), std::get<0>(config[i]), inputs[i], outputs[i], adasum_reduce_type); } - } - else { - // If build_optimizer and Adasum allreduce, outputs of Adasum nodes need to be added back to original weights. + } else { + // If build_optimizer and Adasum allreduce, outputs of Adasum nodes need to be added back to original weights. std::vector weight_update_input_args; std::vector weight_update_output_args; std::string accumulator_op = "InPlaceAccumulator"; @@ -419,28 +410,28 @@ void build_allreduce_graph(Graph& graph, AllreduceGraphConfigVector& config, graph.AddNode(std::get<0>(config[i]) + "_node_update_weight", accumulator_op, "node update weight.", weight_update_input_args, {&updated_weight_out_arg}, - nullptr/*attributes*/, kMSDomain); + nullptr /*attributes*/, kMSDomain); } } } auto status = graph.Resolve(); if (!status.IsOK()) { - std::cout<<"Status not OK. Error: "< create_cuda_execution_provider() { - CUDAExecutionProviderInfo info; OrtDevice::DeviceId device_id = static_cast(training::MPIContext::GetInstance().GetLocalRank()); size_t gpu_mem_limit = std::numeric_limits::max(); gpu_mem_limit = static_cast(1 * 1024 * 1024 * 1024); - info.device_id = device_id; - info.gpu_mem_limit = gpu_mem_limit; - info.arena_extend_strategy = ArenaExtendStrategy::kNextPowerOfTwo; - return std::make_unique(info); + OrtCUDAProviderOptions options{}; + options.device_id = device_id; + options.gpu_mem_limit = gpu_mem_limit; + auto factory = CreateExecutionProviderFactory_Cuda(&options); + return factory->CreateProvider(); } TEST(AllreduceTest, GPUHierarchicalAdasumAllreduceOptimizerTest) { @@ -448,12 +439,13 @@ TEST(AllreduceTest, GPUHierarchicalAdasumAllreduceOptimizerTest) { return; } - training::DistributedRunConfig config = {training::MPIContext::GetInstance().GetWorldRank(),// world rank - training::MPIContext::GetInstance().GetWorldSize(),// world size - training::MPIContext::GetInstance().GetLocalRank(),// local rank - training::MPIContext::GetInstance().GetLocalSize(),// local size - training::MPIContext::GetInstance().GetWorldSize(),// data parallel group - }; + training::DistributedRunConfig config = { + training::MPIContext::GetInstance().GetWorldRank(), // world rank + training::MPIContext::GetInstance().GetWorldSize(), // world size + training::MPIContext::GetInstance().GetLocalRank(), // local rank + training::MPIContext::GetInstance().GetLocalSize(), // local size + training::MPIContext::GetInstance().GetWorldSize(), // data parallel group + }; training::DistributedRunContext::CreateInstance(config); std::string input_gradient_string = "input_t"; @@ -473,16 +465,14 @@ TEST(AllreduceTest, GPUHierarchicalAdasumAllreduceOptimizerTest) { std::vector dims_allreduce_input = {3}; std::vector values_allreduce_input; - if(training::MPIContext::GetInstance().GetWorldRank() == 0) { + if (training::MPIContext::GetInstance().GetWorldRank() == 0) { values_allreduce_input.push_back(3.f); values_allreduce_input.push_back(4.f); values_allreduce_input.push_back(5.f); - } - else { + } else { values_allreduce_input.push_back(5.f); values_allreduce_input.push_back(6.f); values_allreduce_input.push_back(7.f); - } // Weights @@ -506,25 +496,25 @@ TEST(AllreduceTest, GPUHierarchicalAdasumAllreduceOptimizerTest) { onnxruntime::Model model("adasum_optimizer_graph", false, DefaultLoggingManager().DefaultLogger()); auto& graph = model.MainGraph(); auto adasum_graph_config = std::tuple(input_gradient_string, - output_gradient_string, - dims_allreduce_input[0]); - adasum_graph_configs.push_back(adasum_graph_config); + output_gradient_string, + dims_allreduce_input[0]); + adasum_graph_configs.push_back(adasum_graph_config); + + build_allreduce_graph(graph, adasum_graph_configs, training::AdasumReductionType::GpuHierarchicalReduction, true /*build_optimizer*/, + false /*half_precision*/); - build_allreduce_graph(graph, adasum_graph_configs, training::AdasumReductionType::GpuHierarchicalReduction, true/*build_optimizer*/, - false/*half_precision*/); - std::string model_file_name = "GPUHierarchicalAdasumAllreduceOptimizerTest.onnx"; auto status = onnxruntime::Model::Save(model, model_file_name); SessionOptions so; so.session_logid = "AllreduceTest.GPUHierarchicalAdasumAllreduceOptimizerTest"; - + onnxruntime::InferenceSession session_object{so, GetEnvironment()}; RunOptions run_options; run_options.run_tag = so.session_logid; - + auto test_cuda_ep = create_cuda_execution_provider(); - + CPUExecutionProviderInfo epi; auto testCPUExecutionProvider = std::make_unique<::onnxruntime::CPUExecutionProvider>(epi); @@ -593,12 +583,12 @@ TEST(AllreduceTest, GPUHierarchicalAdasumAllreduceOptimizerTest) { // Now run status = session_object.Run(run_options, feeds, output_names, &fetches); if (!status.IsOK()) { - std::cout<<"Status not OK. Error: "<(); TensorShape expected_shape(expected_dims_allreduce); @@ -606,12 +596,12 @@ TEST(AllreduceTest, GPUHierarchicalAdasumAllreduceOptimizerTest) { *reinterpret_cast*>(&actual_output_tensor.Shape())); const std::vector found(actual_output_tensor.template Data(), - actual_output_tensor.template Data() + expected_values_allreduce.size()); + actual_output_tensor.template Data() + expected_values_allreduce.size()); for (size_t i = 0; i < found.size(); i++) ASSERT_NEAR((double)expected_values_allreduce[i], (double)found[i], 1e-4f); - if(training::MPIContext::GetInstance().GetWorldRank() == 0) - std::remove(model_file_name.c_str()); + if (training::MPIContext::GetInstance().GetWorldRank() == 0) + std::remove(model_file_name.c_str()); } TEST(AllreduceTest, GPUHierarchicalAdasumAllreduceOptimizerFP16Test) { @@ -619,12 +609,13 @@ TEST(AllreduceTest, GPUHierarchicalAdasumAllreduceOptimizerFP16Test) { return; } - training::DistributedRunConfig config = {training::MPIContext::GetInstance().GetWorldRank(),// world rank - training::MPIContext::GetInstance().GetWorldSize(),// world size - training::MPIContext::GetInstance().GetLocalRank(),// local rank - training::MPIContext::GetInstance().GetLocalSize(),// local size - training::MPIContext::GetInstance().GetWorldSize(),// data parallel group - }; + training::DistributedRunConfig config = { + training::MPIContext::GetInstance().GetWorldRank(), // world rank + training::MPIContext::GetInstance().GetWorldSize(), // world size + training::MPIContext::GetInstance().GetLocalRank(), // local rank + training::MPIContext::GetInstance().GetLocalSize(), // local size + training::MPIContext::GetInstance().GetWorldSize(), // data parallel group + }; training::DistributedRunContext::CreateInstance(config); std::string input_gradient_string = "input_t"; @@ -644,16 +635,14 @@ TEST(AllreduceTest, GPUHierarchicalAdasumAllreduceOptimizerFP16Test) { std::vector dims_allreduce_input = {3}; std::vector values_allreduce_input; - if(training::MPIContext::GetInstance().GetWorldRank() == 0) { + if (training::MPIContext::GetInstance().GetWorldRank() == 0) { values_allreduce_input.push_back(3.f); values_allreduce_input.push_back(4.f); values_allreduce_input.push_back(5.f); - } - else { + } else { values_allreduce_input.push_back(5.f); values_allreduce_input.push_back(6.f); values_allreduce_input.push_back(7.f); - } std::vector values_allreduce_input_half(dims_allreduce_input[0]); @@ -685,25 +674,25 @@ TEST(AllreduceTest, GPUHierarchicalAdasumAllreduceOptimizerFP16Test) { onnxruntime::Model model("adasum_optimizer_graph", false, DefaultLoggingManager().DefaultLogger()); auto& graph = model.MainGraph(); auto adasum_graph_config = std::tuple(input_gradient_string, - output_gradient_string, - dims_allreduce_input[0]); - adasum_graph_configs.push_back(adasum_graph_config); + output_gradient_string, + dims_allreduce_input[0]); + adasum_graph_configs.push_back(adasum_graph_config); + + build_allreduce_graph(graph, adasum_graph_configs, training::AdasumReductionType::GpuHierarchicalReduction, true /*build_optimizer*/, + true /*half_precision*/); - build_allreduce_graph(graph, adasum_graph_configs, training::AdasumReductionType::GpuHierarchicalReduction, true/*build_optimizer*/, - true/*half_precision*/); - std::string model_file_name = "GPUHierarchicalAdasumAllreduceOptimizerFP16Test.onnx"; auto status = onnxruntime::Model::Save(model, model_file_name); SessionOptions so; so.session_logid = "AllreduceTest.GPUHierarchicalAdasumAllreduceOptimizerFP16Test"; - + onnxruntime::InferenceSession session_object{so, GetEnvironment()}; RunOptions run_options; run_options.run_tag = so.session_logid; - + auto test_cuda_ep = create_cuda_execution_provider(); - + CPUExecutionProviderInfo epi; auto testCPUExecutionProvider = std::make_unique<::onnxruntime::CPUExecutionProvider>(epi); @@ -775,12 +764,12 @@ TEST(AllreduceTest, GPUHierarchicalAdasumAllreduceOptimizerFP16Test) { // Now run status = session_object.Run(run_options, feeds, output_names, &fetches); if (!status.IsOK()) { - std::cout<<"Status not OK. Error: "<(); TensorShape expected_shape(expected_dims_allreduce); @@ -788,13 +777,13 @@ TEST(AllreduceTest, GPUHierarchicalAdasumAllreduceOptimizerFP16Test) { *reinterpret_cast*>(&actual_output_tensor.Shape())); const std::vector found(actual_output_tensor.template Data(), - actual_output_tensor.template Data() + expected_values_allreduce.size()); + actual_output_tensor.template Data() + expected_values_allreduce.size()); for (size_t i = 0; i < found.size(); i++) ASSERT_NEAR((double)expected_values_allreduce[i], (double)found[i], 1e-4f); - if(training::MPIContext::GetInstance().GetWorldRank() == 0) - std::remove(model_file_name.c_str()); + if (training::MPIContext::GetInstance().GetWorldRank() == 0) + std::remove(model_file_name.c_str()); } TEST(AllreduceTest, GPUHierarchicalAdasumAllreduceTest) { @@ -802,12 +791,13 @@ TEST(AllreduceTest, GPUHierarchicalAdasumAllreduceTest) { return; } - training::DistributedRunConfig config = {training::MPIContext::GetInstance().GetWorldRank(),// world rank - training::MPIContext::GetInstance().GetWorldSize(),// world size - training::MPIContext::GetInstance().GetLocalRank(),// local rank - training::MPIContext::GetInstance().GetLocalSize(),// local size - training::MPIContext::GetInstance().GetWorldSize(),// data parallel group - }; + training::DistributedRunConfig config = { + training::MPIContext::GetInstance().GetWorldRank(), // world rank + training::MPIContext::GetInstance().GetWorldSize(), // world size + training::MPIContext::GetInstance().GetLocalRank(), // local rank + training::MPIContext::GetInstance().GetLocalSize(), // local size + training::MPIContext::GetInstance().GetWorldSize(), // data parallel group + }; training::DistributedRunContext::CreateInstance(config); std::vector dims_allreduce_input = {3}; @@ -816,38 +806,36 @@ TEST(AllreduceTest, GPUHierarchicalAdasumAllreduceTest) { std::string output_gradient_string = "output_t"; AllreduceGraphConfigVector adasum_graph_configs; - if(training::MPIContext::GetInstance().GetWorldRank() == 0) { + if (training::MPIContext::GetInstance().GetWorldRank() == 0) { values_allreduce_input.push_back(4.f); values_allreduce_input.push_back(5.f); values_allreduce_input.push_back(6.f); - } - else { + } else { values_allreduce_input.push_back(7.f); values_allreduce_input.push_back(8.f); values_allreduce_input.push_back(9.f); - } onnxruntime::Model model("adasum_graph", false, DefaultLoggingManager().DefaultLogger()); auto& graph = model.MainGraph(); auto adasum_graph_config = std::tuple(input_gradient_string, - output_gradient_string, - dims_allreduce_input[0]); - adasum_graph_configs.push_back(adasum_graph_config); + output_gradient_string, + dims_allreduce_input[0]); + adasum_graph_configs.push_back(adasum_graph_config); build_allreduce_graph(graph, adasum_graph_configs, training::AdasumReductionType::GpuHierarchicalReduction); - + std::string model_file_name = "GPUHierarchicalAdasumAllreduceTest.onnx"; auto status = onnxruntime::Model::Save(model, model_file_name); SessionOptions so; so.session_logid = "AllreduceTest.GPUHierarchicalAdasumAllreduceTest"; - + onnxruntime::InferenceSession session_object{so, GetEnvironment()}; RunOptions run_options; run_options.run_tag = so.session_logid; - + auto test_cuda_ep = create_cuda_execution_provider(); - + CPUExecutionProviderInfo epi; auto testCPUExecutionProvider = std::make_unique<::onnxruntime::CPUExecutionProvider>(epi); @@ -859,7 +847,7 @@ TEST(AllreduceTest, GPUHierarchicalAdasumAllreduceTest) { ASSERT_TRUE(status.IsOK()); OrtValue ml_value_input_t; CreateMLValue(testCPUExecutionProvider->GetAllocator(0, OrtMemTypeDefault), dims_allreduce_input, values_allreduce_input, &ml_value_input_t); - + NameMLValMap feeds; feeds.insert(std::make_pair(input_gradient_string, ml_value_input_t)); @@ -875,12 +863,12 @@ TEST(AllreduceTest, GPUHierarchicalAdasumAllreduceTest) { // Now run status = session_object.Run(run_options, feeds, output_names, &fetches); if (!status.IsOK()) { - std::cout<<"Status not OK. Error: "<(); TensorShape expected_shape(expected_dims_allreduce); @@ -888,12 +876,12 @@ TEST(AllreduceTest, GPUHierarchicalAdasumAllreduceTest) { *reinterpret_cast*>(&actual_output_tensor.Shape())); const std::vector found(actual_output_tensor.template Data(), - actual_output_tensor.template Data() + expected_values_allreduce.size()); + actual_output_tensor.template Data() + expected_values_allreduce.size()); for (size_t i = 0; i < found.size(); i++) ASSERT_NEAR((double)expected_values_allreduce[i], (double)found[i], 1e-4f); - if(training::MPIContext::GetInstance().GetWorldRank() == 0) - std::remove(model_file_name.c_str()); + if (training::MPIContext::GetInstance().GetWorldRank() == 0) + std::remove(model_file_name.c_str()); } TEST(AllreduceTest, GPUHierarchicalAdasumFP16AllreduceTest) { @@ -901,12 +889,13 @@ TEST(AllreduceTest, GPUHierarchicalAdasumFP16AllreduceTest) { return; } - training::DistributedRunConfig config = {training::MPIContext::GetInstance().GetWorldRank(),// world rank - training::MPIContext::GetInstance().GetWorldSize(),// world size - training::MPIContext::GetInstance().GetLocalRank(),// local rank - training::MPIContext::GetInstance().GetLocalSize(),// local size - training::MPIContext::GetInstance().GetWorldSize(),// data parallel group - }; + training::DistributedRunConfig config = { + training::MPIContext::GetInstance().GetWorldRank(), // world rank + training::MPIContext::GetInstance().GetWorldSize(), // world size + training::MPIContext::GetInstance().GetLocalRank(), // local rank + training::MPIContext::GetInstance().GetLocalSize(), // local size + training::MPIContext::GetInstance().GetWorldSize(), // data parallel group + }; training::DistributedRunContext::CreateInstance(config); std::vector dims_allreduce_input = {4}; @@ -915,13 +904,12 @@ TEST(AllreduceTest, GPUHierarchicalAdasumFP16AllreduceTest) { std::string output_gradient_string = "output_t"; AllreduceGraphConfigVector adasum_graph_configs; - if(training::MPIContext::GetInstance().GetWorldRank() == 0) { + if (training::MPIContext::GetInstance().GetWorldRank() == 0) { values_allreduce_input.push_back(4.f); values_allreduce_input.push_back(5.f); values_allreduce_input.push_back(6.f); values_allreduce_input.push_back(7.f); - } - else { + } else { values_allreduce_input.push_back(8.f); values_allreduce_input.push_back(9.f); values_allreduce_input.push_back(10.f); @@ -937,23 +925,23 @@ TEST(AllreduceTest, GPUHierarchicalAdasumFP16AllreduceTest) { auto adasum_graph_config = std::tuple(input_gradient_string, output_gradient_string, dims_allreduce_input[0]); - adasum_graph_configs.push_back(adasum_graph_config); + adasum_graph_configs.push_back(adasum_graph_config); build_allreduce_graph(graph, adasum_graph_configs, training::AdasumReductionType::GpuHierarchicalReduction, - false/*build_optimizer*/, - true/*half_precision*/); - + false /*build_optimizer*/, + true /*half_precision*/); + std::string model_file_name = "GPUHierarchicalAdasumFP16AllreduceTest.onnx"; auto status = onnxruntime::Model::Save(model, model_file_name); SessionOptions so; so.session_logid = "AllreduceTest.GPUHierarchicalAdasumFP16AllreduceTest"; - + onnxruntime::InferenceSession session_object{so, GetEnvironment()}; RunOptions run_options; run_options.run_tag = so.session_logid; - + auto test_cuda_ep = create_cuda_execution_provider(); - + CPUExecutionProviderInfo epi; auto testCPUExecutionProvider = std::make_unique<::onnxruntime::CPUExecutionProvider>(epi); @@ -966,7 +954,7 @@ TEST(AllreduceTest, GPUHierarchicalAdasumFP16AllreduceTest) { OrtValue ml_value_input_t; CreateMLValue(testCPUExecutionProvider->GetAllocator(0, OrtMemTypeDefault), dims_allreduce_input, values_allreduce_input_half, &ml_value_input_t); - + NameMLValMap feeds; feeds.insert(std::make_pair(input_gradient_string, ml_value_input_t)); @@ -986,12 +974,12 @@ TEST(AllreduceTest, GPUHierarchicalAdasumFP16AllreduceTest) { // Now run status = session_object.Run(run_options, feeds, output_names, &fetches); if (!status.IsOK()) { - std::cout<<"Status not OK. Error: "<(); TensorShape expected_shape(expected_dims_allreduce); @@ -999,14 +987,14 @@ TEST(AllreduceTest, GPUHierarchicalAdasumFP16AllreduceTest) { *reinterpret_cast*>(&actual_output_tensor.Shape())); const std::vector found_half(actual_output_tensor.template Data(), - actual_output_tensor.template Data() + expected_values_allreduce_half.size()); + actual_output_tensor.template Data() + expected_values_allreduce_half.size()); std::vector found(found_half.size()); ConvertMLFloat16ToFloat(found_half.data(), found.data(), found.size()); for (size_t i = 0; i < found.size(); i++) ASSERT_NEAR((double)expected_values_allreduce[i], (double)found[i], 1e-3f); - - if(training::MPIContext::GetInstance().GetWorldRank() == 0) - std::remove(model_file_name.c_str()); + + if (training::MPIContext::GetInstance().GetWorldRank() == 0) + std::remove(model_file_name.c_str()); } TEST(AllreduceTest, GPUAdasumAllreduceTest) { @@ -1014,12 +1002,13 @@ TEST(AllreduceTest, GPUAdasumAllreduceTest) { return; } - training::DistributedRunConfig config = {training::MPIContext::GetInstance().GetWorldRank(),// world rank - training::MPIContext::GetInstance().GetWorldSize(),// world size - training::MPIContext::GetInstance().GetLocalRank(),// local rank - training::MPIContext::GetInstance().GetLocalSize(),// local size - training::MPIContext::GetInstance().GetWorldSize(),// data parallel group - }; + training::DistributedRunConfig config = { + training::MPIContext::GetInstance().GetWorldRank(), // world rank + training::MPIContext::GetInstance().GetWorldSize(), // world size + training::MPIContext::GetInstance().GetLocalRank(), // local rank + training::MPIContext::GetInstance().GetLocalSize(), // local size + training::MPIContext::GetInstance().GetWorldSize(), // data parallel group + }; training::DistributedRunContext::CreateInstance(config); std::vector dims_allreduce_input = {4}; @@ -1028,13 +1017,12 @@ TEST(AllreduceTest, GPUAdasumAllreduceTest) { std::string output_gradient_string = "output_t"; AllreduceGraphConfigVector adasum_graph_configs; - if(training::MPIContext::GetInstance().GetWorldRank() == 0) { + if (training::MPIContext::GetInstance().GetWorldRank() == 0) { values_allreduce_input.push_back(4.f); values_allreduce_input.push_back(5.f); values_allreduce_input.push_back(6.f); values_allreduce_input.push_back(7.f); - } - else { + } else { values_allreduce_input.push_back(8.f); values_allreduce_input.push_back(9.f); values_allreduce_input.push_back(10.f); @@ -1046,22 +1034,22 @@ TEST(AllreduceTest, GPUAdasumAllreduceTest) { auto adasum_graph_config = std::tuple(input_gradient_string, output_gradient_string, dims_allreduce_input[0]); - adasum_graph_configs.push_back(adasum_graph_config); + adasum_graph_configs.push_back(adasum_graph_config); build_allreduce_graph(graph, adasum_graph_configs, training::AdasumReductionType::CpuReduction); - + std::string model_file_name = "GPUAdasumAllreduceTest.onnx"; auto status = onnxruntime::Model::Save(model, model_file_name); SessionOptions so; so.session_logid = "AllreduceTest.GPUAdasumAllreduceTest"; - + onnxruntime::InferenceSession session_object{so, GetEnvironment()}; RunOptions run_options; run_options.run_tag = so.session_logid; - + auto test_cuda_ep = create_cuda_execution_provider(); - + CPUExecutionProviderInfo epi; auto testCPUExecutionProvider = std::make_unique<::onnxruntime::CPUExecutionProvider>(epi); @@ -1076,7 +1064,7 @@ TEST(AllreduceTest, GPUAdasumAllreduceTest) { dims_allreduce_input, values_allreduce_input, &ml_value_input_t); - + NameMLValMap feeds; feeds.insert(std::make_pair(input_gradient_string, ml_value_input_t)); @@ -1092,12 +1080,12 @@ TEST(AllreduceTest, GPUAdasumAllreduceTest) { // Now run status = session_object.Run(run_options, feeds, output_names, &fetches); if (!status.IsOK()) { - std::cout<<"Status not OK. Error: "<(); TensorShape expected_shape(expected_dims_allreduce); @@ -1105,12 +1093,12 @@ TEST(AllreduceTest, GPUAdasumAllreduceTest) { *reinterpret_cast*>(&actual_output_tensor.Shape())); const std::vector found(actual_output_tensor.template Data(), - actual_output_tensor.template Data() + expected_values_allreduce.size()); + actual_output_tensor.template Data() + expected_values_allreduce.size()); for (size_t i = 0; i < found.size(); i++) ASSERT_NEAR((double)expected_values_allreduce[i], (double)found[i], 1e-4f); - - if(training::MPIContext::GetInstance().GetWorldRank() == 0) - std::remove(model_file_name.c_str()); + + if (training::MPIContext::GetInstance().GetWorldRank() == 0) + std::remove(model_file_name.c_str()); } TEST(AllreduceTest, GPUAdasumFP16AllreduceTest) { @@ -1118,12 +1106,13 @@ TEST(AllreduceTest, GPUAdasumFP16AllreduceTest) { return; } - training::DistributedRunConfig config = {training::MPIContext::GetInstance().GetWorldRank(),// world rank - training::MPIContext::GetInstance().GetWorldSize(),// world size - training::MPIContext::GetInstance().GetLocalRank(),// local rank - training::MPIContext::GetInstance().GetLocalSize(),// local size - training::MPIContext::GetInstance().GetWorldSize(),// data parallel group - }; + training::DistributedRunConfig config = { + training::MPIContext::GetInstance().GetWorldRank(), // world rank + training::MPIContext::GetInstance().GetWorldSize(), // world size + training::MPIContext::GetInstance().GetLocalRank(), // local rank + training::MPIContext::GetInstance().GetLocalSize(), // local size + training::MPIContext::GetInstance().GetWorldSize(), // data parallel group + }; training::DistributedRunContext::CreateInstance(config); std::vector dims_allreduce_input = {4}; @@ -1132,13 +1121,12 @@ TEST(AllreduceTest, GPUAdasumFP16AllreduceTest) { std::string output_gradient_string = "output_t"; AllreduceGraphConfigVector adasum_graph_configs; - if(training::MPIContext::GetInstance().GetWorldRank() == 0) { + if (training::MPIContext::GetInstance().GetWorldRank() == 0) { values_allreduce_input.push_back(4.f); values_allreduce_input.push_back(5.f); values_allreduce_input.push_back(6.f); values_allreduce_input.push_back(7.f); - } - else { + } else { values_allreduce_input.push_back(8.f); values_allreduce_input.push_back(9.f); values_allreduce_input.push_back(10.f); @@ -1154,22 +1142,22 @@ TEST(AllreduceTest, GPUAdasumFP16AllreduceTest) { auto adasum_graph_config = std::tuple(input_gradient_string, output_gradient_string, dims_allreduce_input[0]); - adasum_graph_configs.push_back(adasum_graph_config); + adasum_graph_configs.push_back(adasum_graph_config); + + build_allreduce_graph(graph, adasum_graph_configs, training::AdasumReductionType::CpuReduction, true /*half_precision*/); - build_allreduce_graph(graph, adasum_graph_configs, training::AdasumReductionType::CpuReduction, true/*half_precision*/); - std::string model_file_name = "GPUAdasumFP16AllreduceTest.onnx"; auto status = onnxruntime::Model::Save(model, model_file_name); SessionOptions so; so.session_logid = "AllreduceTest.GPUAdasumFP16AllreduceTest"; - + onnxruntime::InferenceSession session_object{so, GetEnvironment()}; RunOptions run_options; run_options.run_tag = so.session_logid; - + auto test_cuda_ep = create_cuda_execution_provider(); - + CPUExecutionProviderInfo epi; auto testCPUExecutionProvider = std::make_unique<::onnxruntime::CPUExecutionProvider>(epi); @@ -1182,7 +1170,7 @@ TEST(AllreduceTest, GPUAdasumFP16AllreduceTest) { OrtValue ml_value_input_t; CreateMLValue(testCPUExecutionProvider->GetAllocator(0, OrtMemTypeDefault), dims_allreduce_input, values_allreduce_input_half, &ml_value_input_t); - + NameMLValMap feeds; feeds.insert(std::make_pair(input_gradient_string, ml_value_input_t)); @@ -1202,12 +1190,12 @@ TEST(AllreduceTest, GPUAdasumFP16AllreduceTest) { // Now run status = session_object.Run(run_options, feeds, output_names, &fetches); if (!status.IsOK()) { - std::cout<<"Status not OK. Error: "<(); TensorShape expected_shape(expected_dims_allreduce); @@ -1215,17 +1203,17 @@ TEST(AllreduceTest, GPUAdasumFP16AllreduceTest) { *reinterpret_cast*>(&actual_output_tensor.Shape())); const std::vector found_half(actual_output_tensor.template Data(), - actual_output_tensor.template Data() + expected_values_allreduce_half.size()); + actual_output_tensor.template Data() + expected_values_allreduce_half.size()); std::vector found(found_half.size()); ConvertMLFloat16ToFloat(found_half.data(), found.data(), found.size()); for (size_t i = 0; i < found.size(); i++) ASSERT_NEAR((double)expected_values_allreduce[i], (double)found[i], 5e-3); - - if(training::MPIContext::GetInstance().GetWorldRank() == 0) - std::remove(model_file_name.c_str()); + + if (training::MPIContext::GetInstance().GetWorldRank() == 0) + std::remove(model_file_name.c_str()); } #endif } // namespace test } // namespace onnxruntime -#endif // USE_MPI \ No newline at end of file +#endif // USE_MPI diff --git a/orttraining/orttraining/test/graph/gradient_graph_builder_test.cc b/orttraining/orttraining/test/graph/gradient_graph_builder_test.cc index 359facf89e..e87e5443ff 100644 --- a/orttraining/orttraining/test/graph/gradient_graph_builder_test.cc +++ b/orttraining/orttraining/test/graph/gradient_graph_builder_test.cc @@ -6,6 +6,7 @@ #include "orttraining/core/optimizer/gist_encode_decode.h" #include "test/providers/provider_test_utils.h" #include "test/framework/test_utils.h" +#include "test/util/include/default_providers.h" #include "core/common/path_utils.h" #include "core/providers/cpu/cpu_execution_provider.h" #include "core/session/environment.h" @@ -17,9 +18,7 @@ #if defined(USE_CUDA) || defined(USE_ROCM) #include "bert_toy_fetches.h" -#ifdef USE_CUDA -#include "core/providers/cuda/cuda_execution_provider.h" -#elif USE_ROCM +#ifdef USE_ROCM #include "core/providers/rocm/rocm_execution_provider.h" #endif #endif @@ -263,8 +262,7 @@ TEST(GradientGraphBuilderTest, TrainingSession_WithGist) { std::cout << "Loaded " << model_metadata->graph_name << '\n'; // Add cuda execution provider for gist encode/decode nodes - CUDAExecutionProviderInfo xp_info; - ASSERT_STATUS_OK(training_session.RegisterExecutionProvider(std::make_unique(xp_info))); + ASSERT_STATUS_OK(training_session.RegisterExecutionProvider(DefaultCudaExecutionProvider())); ORT_THROW_IF_ERROR(training_session.Initialize()); @@ -406,8 +404,7 @@ static void RunBertTrainingWithChecks( std::cout << "Loaded " << model_metadata->graph_name << '\n'; #ifdef USE_CUDA - CUDAExecutionProviderInfo xp_info; - ASSERT_STATUS_OK(training_session->RegisterExecutionProvider(std::make_unique(xp_info))); + ASSERT_STATUS_OK(training_session->RegisterExecutionProvider(DefaultCudaExecutionProvider())); #elif USE_ROCM ROCMExecutionProviderInfo xp_info; ASSERT_STATUS_OK(training_session->RegisterExecutionProvider(std::make_unique(xp_info))); @@ -1095,21 +1092,21 @@ class PipelineBatchPlanner { }; void RetrieveEventOperators( - Graph& graph, - const int stage_index, - const int num_stages, - Node** forward_recv_wait, - Node** forward_recv_record, - Node** forward_compute_wait, - Node** forward_compute_record, - Node** forward_send_wait, - Node** forward_send_record, - Node** backward_recv_wait, - Node** backward_recv_record, - Node** backward_compute_wait, - Node** backward_compute_record, - Node** backward_send_wait, - Node** backward_send_record) { + Graph& graph, + const int stage_index, + const int num_stages, + Node** forward_recv_wait, + Node** forward_recv_record, + Node** forward_compute_wait, + Node** forward_compute_record, + Node** forward_send_wait, + Node** forward_send_record, + Node** backward_recv_wait, + Node** backward_recv_record, + Node** backward_compute_wait, + Node** backward_compute_record, + Node** backward_send_wait, + Node** backward_send_record) { // Initialize retrieved nodes. // Non-existing nodes may hold NULL forever. // Existing nodes may get valid pointers below. @@ -1195,16 +1192,16 @@ void RetrieveEventOperators( *backward_send_record = records[2]; } else { ORT_THROW("Wrong number of WaitEvent operators: ", - waits.size(), " allowed value range is [0, ", num_stages - 1, ")."); + waits.size(), " allowed value range is [0, ", num_stages - 1, ")."); } } void RetrieveSendRecvOperators( - Graph& graph, - Node** forward_recv, - Node** forward_send, - Node** backward_recv, - Node** backward_send) { + Graph& graph, + Node** forward_recv, + Node** forward_send, + Node** backward_recv, + Node** backward_send) { // Initialize retrieved nodes. // Non-existing nodes may hold NULL forever. // Existing nodes may get valid pointers below. @@ -1256,32 +1253,30 @@ PathString GenerateFileNameWithIndex(const std::string& base_str, int index, con // DistributedRunTestContext provides a method to override existing DistributedRunTestContext instance. // This is for test purpose only. Please don't use it for other scenarios. -class DistributedRunTestContext : public DistributedRunContext -{ -public: - DistributedRunTestContext(const TrainingSession::TrainingConfiguration &config) - : DistributedRunContext(config.distributed_config.world_rank, - config.distributed_config.world_size, - config.distributed_config.local_rank, - config.distributed_config.local_size, - config.distributed_config.data_parallel_size, - config.distributed_config.horizontal_parallel_size, - config.distributed_config.pipeline_parallel_size) - { - } +class DistributedRunTestContext : public DistributedRunContext { + public: + DistributedRunTestContext(const TrainingSession::TrainingConfiguration& config) + : DistributedRunContext(config.distributed_config.world_rank, + config.distributed_config.world_size, + config.distributed_config.local_rank, + config.distributed_config.local_size, + config.distributed_config.data_parallel_size, + config.distributed_config.horizontal_parallel_size, + config.distributed_config.pipeline_parallel_size) { + } - // Reset the static DistributedRunContext object with new value. - void ResetDistributedRunContext(){ - DistributedRunContext::GetRunConfig() = params_; - auto& dp_group = DistributedRunContext::GetWorkerGroup(WorkerGroupType::DataParallel); - dp_group = groups_[WorkerGroupType::DataParallel]; + // Reset the static DistributedRunContext object with new value. + void ResetDistributedRunContext() { + DistributedRunContext::GetRunConfig() = params_; + auto& dp_group = DistributedRunContext::GetWorkerGroup(WorkerGroupType::DataParallel); + dp_group = groups_[WorkerGroupType::DataParallel]; - auto& hp_group = DistributedRunContext::GetWorkerGroup(WorkerGroupType::HorizontalParallel); - hp_group = groups_[WorkerGroupType::HorizontalParallel]; + auto& hp_group = DistributedRunContext::GetWorkerGroup(WorkerGroupType::HorizontalParallel); + hp_group = groups_[WorkerGroupType::HorizontalParallel]; - auto& mp_group = DistributedRunContext::GetInstance().GetWorkerGroup(WorkerGroupType::PipelineParallel); - mp_group = groups_[WorkerGroupType::PipelineParallel]; - } + auto& mp_group = DistributedRunContext::GetInstance().GetWorkerGroup(WorkerGroupType::PipelineParallel); + mp_group = groups_[WorkerGroupType::PipelineParallel]; + } }; void OverwritePipelineRank(const TrainingSession::TrainingConfiguration& config, const int pipeline_rank) { @@ -1424,7 +1419,7 @@ TEST(GradientGraphBuilderTest, PipelineOnlinePartition_MLP) { // 2 test variations - full precision and mixed precision const std::vector test_with_fp32{true, false}; - for(auto is_fp32 : test_with_fp32) { + for (auto is_fp32 : test_with_fp32) { // graph is partitioned into 3 parts. for (int i = 0; i < 3; ++i) { PathString output_file = GenerateFileNameWithIndex("pipeline_partition_", i, "_back.onnx"); @@ -1449,7 +1444,7 @@ TEST(GradientGraphBuilderTest, PipelineOnlinePartition_MLP) { PathString backprop_model_file; Status status = BuildBackPropGraph(model_uri, config, backprop_model_file); - ASSERT_TRUE(status.IsOK()) << status<<" (is_fp32 = " << is_fp32 << ", stage = " << i << ").\n"; + ASSERT_TRUE(status.IsOK()) << status << " (is_fp32 = " << is_fp32 << ", stage = " << i << ").\n"; // Skip the re-load for mixed-precision case. This model contains grad op that has function body, // which takes a const tensor input. Const cast for input in function body won't be saved in the output @@ -1460,7 +1455,7 @@ TEST(GradientGraphBuilderTest, PipelineOnlinePartition_MLP) { std::shared_ptr model; // Ensure the partitioned model load. status = Model::Load(backprop_model_file, model, nullptr, DefaultLoggingManager().DefaultLogger()); - ASSERT_TRUE(status.IsOK()) << status<<" (is_fp32 = " << is_fp32 << ", stage = " << i << ").\n"; + ASSERT_TRUE(status.IsOK()) << status << " (is_fp32 = " << is_fp32 << ", stage = " << i << ").\n"; } } } @@ -1519,7 +1514,7 @@ TEST(GradientGraphBuilderTest, PipelineOnlinePartition_Invalid_Input) { // verify pipeline config can load and gradient graph can construct. TEST(GradientGraphBuilderTest, TrainingSession_PipelineTransform_base) { - std::string filename_base = "testdata/test_training_model_"; + std::string filename_base = "testdata/test_training_model_"; auto load_and_check_gradient_graph = [](int stageIdx, PathString& input_file, PathString& output_file) { auto config = MakeBasicTrainingConfig(); @@ -1555,21 +1550,21 @@ TEST(GradientGraphBuilderTest, TrainingSession_PipelineTransform_base) { // Find event nodes. RetrieveEventOperators( - graph, - stageIdx, - 3, - &forward_recv_wait, - &forward_recv_record, - &forward_compute_wait, - &forward_compute_record, - &forward_send_wait, - &forward_send_record, - &backward_recv_wait, - &backward_recv_record, - &backward_compute_wait, - &backward_compute_record, - &backward_send_wait, - &backward_send_record); + graph, + stageIdx, + 3, + &forward_recv_wait, + &forward_recv_record, + &forward_compute_wait, + &forward_compute_record, + &forward_send_wait, + &forward_send_record, + &backward_recv_wait, + &backward_recv_record, + &backward_compute_wait, + &backward_compute_record, + &backward_send_wait, + &backward_send_record); // Check event nodes. if (stageIdx == 2) { @@ -1631,11 +1626,11 @@ TEST(GradientGraphBuilderTest, TrainingSession_PipelineTransform_base) { Node* backward_send{nullptr}; RetrieveSendRecvOperators( - graph, - &forward_recv, - &forward_send, - &backward_recv, - &backward_send); + graph, + &forward_recv, + &forward_send, + &backward_recv, + &backward_send); // Except the last partion, each partition should have send forward and recv backward. if (stageIdx == 0 || stageIdx == 1) { @@ -1700,23 +1695,21 @@ TEST(GradientGraphBuilderTest, TrainingSession_WithPipeline) { {}, {}, {}}, - {{ - "MeanSquaredError_reduce_mean_Grad/Sized_X", - "MeanSquaredError_reduce_mean_Grad/Sized_Grad", - "MeanSquaredError_reduce_mean_Grad/Scale", - "MeanSquaredError_reduce_mean_Grad/Scaled_Grad", - "MeanSquaredError_reduce_mean_Grad/Shaped_X", - "MeanSquaredError_diff_square_grad", - "MeanSquaredError_pow_Grad/Sub_I1", - "MeanSquaredError_pow_Grad/Pow_I0", - "MeanSquaredError_pow_Grad/Mul_Pow_I0_I1", - "MeanSquaredError_diff_grad", - "predictions_grad", - "B3_grad", - "T7_grad", - "W3_grad", - "T6_grad" - }, + {{"MeanSquaredError_reduce_mean_Grad/Sized_X", + "MeanSquaredError_reduce_mean_Grad/Sized_Grad", + "MeanSquaredError_reduce_mean_Grad/Scale", + "MeanSquaredError_reduce_mean_Grad/Scaled_Grad", + "MeanSquaredError_reduce_mean_Grad/Shaped_X", + "MeanSquaredError_diff_square_grad", + "MeanSquaredError_pow_Grad/Sub_I1", + "MeanSquaredError_pow_Grad/Pow_I0", + "MeanSquaredError_pow_Grad/Mul_Pow_I0_I1", + "MeanSquaredError_diff_grad", + "predictions_grad", + "B3_grad", + "T7_grad", + "W3_grad", + "T6_grad"}, {}, {"T6_grad"}, {}, diff --git a/orttraining/orttraining/test/model/training_runner_test.cc b/orttraining/orttraining/test/model/training_runner_test.cc index 2fed7cfa60..b8be4dfd20 100644 --- a/orttraining/orttraining/test/model/training_runner_test.cc +++ b/orttraining/orttraining/test/model/training_runner_test.cc @@ -11,9 +11,9 @@ #include "core/common/path_string.h" #include "core/platform/path_lib.h" #include "core/session/environment.h" -#include "core/providers/cuda/cuda_execution_provider.h" #include "orttraining/models/runner/data_loader.h" #include "orttraining/models/runner/training_util.h" +#include "test/util/include/default_providers.h" namespace onnxruntime { namespace training { @@ -45,8 +45,7 @@ TEST(TrainingRunnerTest, Basic) { TrainingRunner runner{params, *env}; - CUDAExecutionProviderInfo xp_info; - ASSERT_TRUE(runner.GetSession().RegisterExecutionProvider(std::make_unique(xp_info)).IsOK()); + ASSERT_TRUE(runner.GetSession().RegisterExecutionProvider(onnxruntime::test::DefaultCudaExecutionProvider()).IsOK()); auto status = runner.Initialize(); ASSERT_TRUE(status.IsOK()) << status.ErrorMessage(); @@ -122,4 +121,4 @@ TEST(TrainingRunnerTest, DISABLED_PipelineRun) { } // namespace training } // namespace onnxruntime -#endif \ No newline at end of file +#endif diff --git a/orttraining/orttraining/test/session/training_session_test.cc b/orttraining/orttraining/test/session/training_session_test.cc index 578d4ce1ce..ff2935c075 100644 --- a/orttraining/orttraining/test/session/training_session_test.cc +++ b/orttraining/orttraining/test/session/training_session_test.cc @@ -15,12 +15,6 @@ #include "orttraining/training_ops/cpu/controlflow/event_pool.h" // TODO: move with PipelineBatchPlanner -#ifdef USE_CUDA -#include "core/providers/cuda/cuda_execution_provider.h" -#elif USE_ROCM -#include "core/providers/rocm/rocm_execution_provider.h" -#endif - using namespace onnxruntime::logging; using namespace onnxruntime::training; using namespace google::protobuf::util; @@ -34,7 +28,7 @@ static void RunTrainingSessionLoadOptimTests(std::string optim_name, bool mixed_ auto config = MakeBasicTrainingConfig(); if (mixed_precision) { TrainingSession::TrainingConfiguration::MixedPrecisionConfiguration mp{}; - mp.use_mixed_precision_initializers=true; + mp.use_mixed_precision_initializers = true; config.mixed_precision_config = mp; } GenerateOptimizerConfig(optim_name, mixed_precision_moments, config); diff --git a/orttraining/orttraining/test/session/training_session_test_utils.cc b/orttraining/orttraining/test/session/training_session_test_utils.cc index bd3150f735..098cd13cf1 100644 --- a/orttraining/orttraining/test/session/training_session_test_utils.cc +++ b/orttraining/orttraining/test/session/training_session_test_utils.cc @@ -3,6 +3,7 @@ #include "orttraining/test/session/training_session_test_utils.h" #include "orttraining/core/graph/optimizer_builder.h" +#include "test/util/include/default_providers.h" using namespace onnxruntime::logging; using namespace onnxruntime::training; @@ -180,8 +181,7 @@ std::unique_ptr BuildAndRunTrainingSessionWithChecks( std::cout << "Loaded " << model_metadata->graph_name << '\n'; #ifdef USE_CUDA - CUDAExecutionProviderInfo xp_info; - ORT_THROW_IF_ERROR(training_session->RegisterExecutionProvider(std::make_unique(xp_info))); + ORT_THROW_IF_ERROR(training_session->RegisterExecutionProvider(DefaultCudaExecutionProvider())); #elif USE_ROCM ROCMExecutionProviderInfo xp_info; ORT_THROW_IF_ERROR(training_session->RegisterExecutionProvider(std::make_unique(xp_info))); diff --git a/orttraining/orttraining/test/session/training_session_test_utils.h b/orttraining/orttraining/test/session/training_session_test_utils.h index 008947ddf9..ae84d8dc4e 100644 --- a/orttraining/orttraining/test/session/training_session_test_utils.h +++ b/orttraining/orttraining/test/session/training_session_test_utils.h @@ -17,7 +17,7 @@ #include "orttraining/training_ops/cpu/controlflow/event_pool.h" // TODO: move with PipelineBatchPlanner #ifdef USE_CUDA -#include "core/providers/cuda/cuda_execution_provider.h" +#include "core/providers/cuda/cuda_execution_provider_info.h" #elif USE_ROCM #include "core/providers/rocm/rocm_execution_provider.h" #endif @@ -44,18 +44,18 @@ void GenerateOptimizerConfig(const std::string optimizer_name, training::TrainingSession::TrainingConfiguration& config); template -void GenerateOptimizerInitialState(const std::string& optimizer_op_name, - const T init_moment_value, - training::TrainingSession::OptimizerState& optimizer_state); +void GenerateOptimizerInitialState(const std::string& optimizer_op_name, + const T init_moment_value, + training::TrainingSession::OptimizerState& optimizer_state); -void SeparateStateTensors(const NameMLValMap& training_state, - NameMLValMap& model_state, +void SeparateStateTensors(const NameMLValMap& training_state, + NameMLValMap& model_state, training::TrainingSession::OptimizerState& optimizer_state); void VerifyState(const DataTransferManager& data_transfer_mgr, const NameMLValMap& expected_state, const NameMLValMap& actual_state); -void VerifyOptimizerState(const DataTransferManager& data_transfer_manager, - const training::TrainingSession::OptimizerState& expected_state, +void VerifyOptimizerState(const DataTransferManager& data_transfer_manager, + const training::TrainingSession::OptimizerState& expected_state, const training::TrainingSession::OptimizerState& actual_state); std::unordered_set GetModelOutputNames(const InferenceSession& session); diff --git a/orttraining/orttraining/test/training_ops/cpu/reduction/reduction_ops_test.cc b/orttraining/orttraining/test/training_ops/cpu/reduction/reduction_ops_test.cc index b5af3b54e2..eff6cd567c 100644 --- a/orttraining/orttraining/test/training_ops/cpu/reduction/reduction_ops_test.cc +++ b/orttraining/orttraining/test/training_ops/cpu/reduction/reduction_ops_test.cc @@ -6,10 +6,6 @@ #include #include "gtest/gtest.h" #include "test/providers/provider_test_utils.h" -//#include "test/providers/cpu/reduction/reduction_test_cases.h" -#ifdef USE_CUDA -#include "core/providers/cuda/reduction/reduction_functions.h" -#endif namespace onnxruntime { namespace test { @@ -87,8 +83,8 @@ TEST(AllOpTest, All_1d_large) { #endif class ReductionOpTest : public ::testing::TestWithParam { - protected: - bool use_determinism; + protected: + bool use_determinism; }; TEST_P(ReductionOpTest, ReduceAllL2) { diff --git a/orttraining/orttraining/test/training_ops/cuda/cuda_utils_test.cc b/orttraining/orttraining/test/training_ops/cuda/cuda_utils_test.cc index 964822966d..112273de2e 100644 --- a/orttraining/orttraining/test/training_ops/cuda/cuda_utils_test.cc +++ b/orttraining/orttraining/test/training_ops/cuda/cuda_utils_test.cc @@ -1,7 +1,8 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. -#ifdef USE_CUDA +#if 0 // TODO: Can't call these directly from external code as Cuda is now a shared library +//#ifdef USE_CUDA #include #include diff --git a/orttraining/orttraining/training_ops/communication_common.h b/orttraining/orttraining/training_ops/communication_common.h index 957ca03de9..74825f9867 100644 --- a/orttraining/orttraining/training_ops/communication_common.h +++ b/orttraining/orttraining/training_ops/communication_common.h @@ -1,7 +1,9 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. +#ifndef SHARED_PROVIDER #include "core/common/common.h" +#endif #pragma once #include "orttraining/core/framework/communication/mpi/mpi_context.h" @@ -191,7 +193,7 @@ inline void ReceiveShapeInfo( info_shapes.buffer, info_shapes.size, MPI_CHAR, info_shapes.rank, info_shapes.tag, MPI_COMM_WORLD, MPI_STATUS_IGNORE)); } -#endif // USE_MPI +#endif // USE_MPI inline void ComputeTensorSizeAndBufferLength(OpKernelContext* context, std::vector& tensor_element_counts, diff --git a/orttraining/orttraining/training_ops/cpu/aten_ops/aten_op.cc b/orttraining/orttraining/training_ops/cpu/aten_ops/aten_op.cc index f40500e88d..532628efe0 100644 --- a/orttraining/orttraining/training_ops/cpu/aten_ops/aten_op.cc +++ b/orttraining/orttraining/training_ops/cpu/aten_ops/aten_op.cc @@ -13,14 +13,13 @@ namespace contrib { ONNX_OPERATOR_KERNEL_EX(ATenOp, kMSDomain, 1, kCpuExecutionProvider, KernelDefBuilder().TypeConstraint("T", DataTypeImpl::AllTensorAndSequenceTensorTypes()), - ATenOpBase); + ATenOpForward); ONNX_OPERATOR_KERNEL_EX(ATenOpGrad, kMSDomain, 1, kCpuExecutionProvider, KernelDefBuilder().TypeConstraint("T", DataTypeImpl::AllTensorAndSequenceTensorTypes()), - ATenOpBase); + ATenOpBackward); -template -ATenOpBase::ATenOpBase(const OpKernelInfo& info) : OpKernel(info) { +void ATenOpBase::Init(const OpKernelInfo& info, bool is_backward) { std::string op_name; ORT_THROW_IF_ERROR(info.GetAttr("name", &op_name)); const auto* op_config_ptr = aten_ops::GetATenOperatorConfig(op_name); @@ -58,8 +57,7 @@ ATenOpBase::ATenOpBase(const OpKernelInfo& info) : OpKernel(info) { } } -template -Status ATenOpBase::Compute(OpKernelContext* p_ctx) const { +Status ATenOpBase::Compute(OpKernelContext* p_ctx) const { auto* p_ctx_internal = static_cast(p_ctx); std::vector> tensor_arguments; for (size_t i = 0; i < tensor_argument_indices_.size(); i++) { diff --git a/orttraining/orttraining/training_ops/cpu/aten_ops/aten_op.h b/orttraining/orttraining/training_ops/cpu/aten_ops/aten_op.h index 1bc0ac5059..903e4d8381 100644 --- a/orttraining/orttraining/training_ops/cpu/aten_ops/aten_op.h +++ b/orttraining/orttraining/training_ops/cpu/aten_ops/aten_op.h @@ -8,10 +8,10 @@ namespace onnxruntime { namespace contrib { -template -class ATenOpBase final : public OpKernel { +class ATenOpBase : public OpKernel { public: - ATenOpBase(const OpKernelInfo& info); + ATenOpBase(const OpKernelInfo& info, bool is_backward) : OpKernel(info) { Init(info, is_backward); } + void Init(const OpKernelInfo& info, bool is_backward); // Separated into a regular member for shared provider access Status Compute(OpKernelContext* p_ctx) const override; private: @@ -26,5 +26,15 @@ class ATenOpBase final : public OpKernel { std::vector> bool_arguments_; }; +class ATenOpForward final : public ATenOpBase { + public: + ATenOpForward(const OpKernelInfo& info) : ATenOpBase(info, false) {} +}; + +class ATenOpBackward final : public ATenOpBase { + public: + ATenOpBackward(const OpKernelInfo& info) : ATenOpBase(info, true) {} +}; + } // namespace contrib } // namespace onnxruntime diff --git a/orttraining/orttraining/training_ops/cpu/loss/reduction_type.h b/orttraining/orttraining/training_ops/cpu/loss/reduction_type.h index 97c4a8362c..96681fffad 100644 --- a/orttraining/orttraining/training_ops/cpu/loss/reduction_type.h +++ b/orttraining/orttraining/training_ops/cpu/loss/reduction_type.h @@ -3,8 +3,10 @@ #pragma once +#ifndef SHARED_PROVIDER #include "core/common/common.h" #include "core/common/exceptions.h" +#endif namespace onnxruntime { diff --git a/orttraining/orttraining/training_ops/cuda/activation/activations_grad.cc b/orttraining/orttraining/training_ops/cuda/activation/activations_grad.cc index 1049079082..8490d6d6b7 100644 --- a/orttraining/orttraining/training_ops/cuda/activation/activations_grad.cc +++ b/orttraining/orttraining/training_ops/cuda/activation/activations_grad.cc @@ -14,7 +14,7 @@ namespace cuda { ver, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()) \ .MayInplace(0, 0), \ x); diff --git a/orttraining/orttraining/training_ops/cuda/activation/bias_gelu_grad.cc b/orttraining/orttraining/training_ops/cuda/activation/bias_gelu_grad.cc index 703623f16b..30c174dd77 100644 --- a/orttraining/orttraining/training_ops/cuda/activation/bias_gelu_grad.cc +++ b/orttraining/orttraining/training_ops/cuda/activation/bias_gelu_grad.cc @@ -21,7 +21,7 @@ ONNX_OPERATOR_KERNEL_EX( kMSDomain, 1, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .TypeConstraint("T", BuildKernelDefConstraints()) .MayInplace(0, 0), BiasGeluGrad_dX); @@ -31,7 +31,7 @@ ONNX_OPERATOR_KERNEL_EX( kMSDomain, 1, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .TypeConstraint("T", BuildKernelDefConstraints()) .MayInplace(0, 0), BiasGeluGrad_dX); diff --git a/orttraining/orttraining/training_ops/cuda/aten_ops/aten_op.cc b/orttraining/orttraining/training_ops/cuda/aten_ops/aten_op.cc index 32560f59fd..abb2b3a4bb 100644 --- a/orttraining/orttraining/training_ops/cuda/aten_ops/aten_op.cc +++ b/orttraining/orttraining/training_ops/cuda/aten_ops/aten_op.cc @@ -1,6 +1,7 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. +#include "core/providers/shared_library/provider_api.h" #include "orttraining/training_ops/cpu/aten_ops/aten_op.h" #include "core/providers/cuda/cuda_fwd.h" @@ -8,12 +9,12 @@ namespace onnxruntime { namespace cuda { ONNX_OPERATOR_KERNEL_EX(ATenOp, kMSDomain, 1, kCudaExecutionProvider, - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::AllTensorAndSequenceTensorTypes()), - onnxruntime::contrib::ATenOpBase); + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::AllTensorAndSequenceTensorTypes()), + onnxruntime::contrib::ATenOpForward); ONNX_OPERATOR_KERNEL_EX(ATenOpGrad, kMSDomain, 1, kCudaExecutionProvider, - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::AllTensorAndSequenceTensorTypes()), - onnxruntime::contrib::ATenOpBase); + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::AllTensorAndSequenceTensorTypes()), + onnxruntime::contrib::ATenOpBackward); } // namespace cuda } // namespace onnxruntime diff --git a/orttraining/orttraining/training_ops/cuda/collective/adasum_kernels.cc b/orttraining/orttraining/training_ops/cuda/collective/adasum_kernels.cc index 053d5fee1a..99e9eccfda 100644 --- a/orttraining/orttraining/training_ops/cuda/collective/adasum_kernels.cc +++ b/orttraining/orttraining/training_ops/cuda/collective/adasum_kernels.cc @@ -1,6 +1,7 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. #ifdef USE_MPI +#include "core/providers/shared_library/provider_api.h" #include "orttraining/training_ops/cuda/collective/adasum_kernels.h" #include "orttraining/training_ops/communication_common.h" #include "orttraining/core/framework/communication/mpi/mpi_context.h" @@ -9,7 +10,6 @@ namespace onnxruntime { namespace cuda { Status AdasumAllReduce::ComputeInternal(OpKernelContext* context) const { - int vhdd_start_level = 1; if (adasum_reduce_algo_ == training::AdasumReductionType::GpuHierarchicalReduction) { vhdd_start_level = training::DistributedRunContext::GetInstance().GroupSize(training::WorkerGroupType::NodeLocalDataParallel); @@ -37,23 +37,23 @@ Status AdasumAllReduce::ComputeInternal(OpKernelContext* context) const { for (int i = 0; i < num_tensors; ++i) { const Tensor* x_tensor = context->Input(i); CUDA_CALL(cudaMemcpyAsync((uint8_t*)data_buffer_ptr.get() + tensor_offsets[i], x_tensor->DataRaw(), - tensor_sizes[i], cudaMemcpyDeviceToHost, Stream())); + tensor_sizes[i], cudaMemcpyDeviceToHost, Stream())); } auto recv_buffer = allocator->Alloc(total_recv_buffer_len); BufferUniquePtr recv_buffer_ptr(recv_buffer, BufferDeleter(allocator)); ORT_RETURN_IF_ERROR(adasum_reducer_->DispatchFusedAllreduce((void*)data_buffer, recv_buffer, tensor_element_counts, - vhdd_start_level, // start level - training::MPIContext::GetInstance().GetMPIGroup(training::WorkerGroupType::GlobalParallel).communicator, // communicator - 0, // tag - adasum_reducer_->GetReductionComms(), // reduction_comms - context->Input(0)->DataType())); + vhdd_start_level, // start level + training::MPIContext::GetInstance().GetMPIGroup(training::WorkerGroupType::GlobalParallel).communicator, // communicator + 0, // tag + adasum_reducer_->GetReductionComms(), // reduction_comms + context->Input(0)->DataType())); for (int i = 0; i < num_tensors; i++) { Tensor* y_tensor = context->Output(i, context->Input(i)->Shape()); CUDA_CALL(cudaMemcpyAsync(y_tensor->MutableDataRaw(), (uint8_t*)data_buffer + tensor_offsets[i], - tensor_sizes[i], cudaMemcpyHostToDevice, Stream())); + tensor_sizes[i], cudaMemcpyHostToDevice, Stream())); } return Status::OK(); } @@ -63,11 +63,11 @@ ONNX_OPERATOR_KERNEL_EX( kMSDomain, 1, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .VariadicAlias(0, 0) // outputs and inputs are mapped one to one .TypeConstraint("T", DataTypeImpl::AllIEEEFloatTensorTypes()), AdasumAllReduce); } // namespace cuda } // namespace onnxruntime -#endif // USE_MPI +#endif // USE_MPI diff --git a/orttraining/orttraining/training_ops/cuda/collective/adasum_kernels.h b/orttraining/orttraining/training_ops/cuda/collective/adasum_kernels.h index 3138771e93..693dcafe13 100644 --- a/orttraining/orttraining/training_ops/cuda/collective/adasum_kernels.h +++ b/orttraining/orttraining/training_ops/cuda/collective/adasum_kernels.h @@ -4,25 +4,25 @@ #pragma once #include "orttraining/training_ops/cuda/collective/nccl_common.h" - #include "orttraining/core/framework/adasum/adasum_interface.h" #include "orttraining/core/framework/adasum/adasum_mpi.h" + namespace onnxruntime { namespace cuda { class AdasumAllReduce final : public NcclKernel { public: explicit AdasumAllReduce(const OpKernelInfo& info) : NcclKernel(info) { - int64_t adasum_reduce_algo; - info.GetAttrOrDefault("reduce_algo", &adasum_reduce_algo, static_cast(0)); - adasum_reduce_algo_ = static_cast(adasum_reduce_algo); - if (adasum_reduce_algo_ == training::AdasumReductionType::GpuHierarchicalReduction || - adasum_reduce_algo_ == training::AdasumReductionType::CpuReduction) { - adasum_reducer_ = std::make_unique(); - } - if(!adasum_reducer_->IsAdasumInitialized()) { - adasum_reducer_->InitializeVHDDReductionComms(); - } + int64_t adasum_reduce_algo; + info.GetAttrOrDefault("reduce_algo", &adasum_reduce_algo, static_cast(0)); + adasum_reduce_algo_ = static_cast(adasum_reduce_algo); + if (adasum_reduce_algo_ == training::AdasumReductionType::GpuHierarchicalReduction || + adasum_reduce_algo_ == training::AdasumReductionType::CpuReduction) { + adasum_reducer_ = std::make_unique(); + } + if (!adasum_reducer_->IsAdasumInitialized()) { + adasum_reducer_->InitializeVHDDReductionComms(); + } } Status ComputeInternal(OpKernelContext* context) const override; @@ -33,4 +33,4 @@ class AdasumAllReduce final : public NcclKernel { }; } // namespace cuda } // namespace onnxruntime -#endif // USE_MPI +#endif // USE_MPI diff --git a/orttraining/orttraining/training_ops/cuda/collective/megatron.cc b/orttraining/orttraining/training_ops/cuda/collective/megatron.cc index 55ba247966..27bf09c3eb 100644 --- a/orttraining/orttraining/training_ops/cuda/collective/megatron.cc +++ b/orttraining/orttraining/training_ops/cuda/collective/megatron.cc @@ -1,6 +1,5 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. - #include "nccl_kernels.h" #include "core/providers/cuda/tensor/identity_op.h" @@ -12,7 +11,7 @@ ONNX_OPERATOR_KERNEL_EX( kMSDomain, 1, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .Alias(0, 0) .TypeConstraint("T", DataTypeImpl::AllIEEEFloatTensorTypes()), IdentityOp); @@ -22,7 +21,7 @@ ONNX_OPERATOR_KERNEL_EX( kMSDomain, 1, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .Alias(0, 0) .TypeConstraint("T", DataTypeImpl::AllIEEEFloatTensorTypes()), NcclAllReduce); diff --git a/orttraining/orttraining/training_ops/cuda/collective/nccl_common.cc b/orttraining/orttraining/training_ops/cuda/collective/nccl_common.cc index aec65686c9..930a1a6937 100644 --- a/orttraining/orttraining/training_ops/cuda/collective/nccl_common.cc +++ b/orttraining/orttraining/training_ops/cuda/collective/nccl_common.cc @@ -2,11 +2,9 @@ // Licensed under the MIT License. #include "orttraining/training_ops/cuda/collective/nccl_common.h" -#include "core/common/logging/logging.h" #include "orttraining/core/framework/communication/mpi/mpi_context.h" #include - namespace onnxruntime { namespace cuda { @@ -80,7 +78,7 @@ NcclContext::NcclContext() { // Initialize Data Parallel Group NCCL Communicator ret = CreateNcclCommunicator(&mpi_world_group, training::WorkerGroupType::DataParallel, - &data_group_comm_); + &data_group_comm_); ORT_ENFORCE(ret.IsOK()); // Initialize Horizontal Model Parallel Group NCCL Communicator @@ -111,11 +109,9 @@ ncclComm_t NcclContext::Comm(training::WorkerGroupType group_type) { return data_group_comm_; } else if (training::WorkerGroupType::HorizontalParallel == group_type) { return horizontal_group_comm_; - } - else if (training::WorkerGroupType::NodeLocalDataParallel == group_type) { + } else if (training::WorkerGroupType::NodeLocalDataParallel == group_type) { return node_local_comm_; - } - else if (training::WorkerGroupType::CrossNodeDataParallel == group_type) { + } else if (training::WorkerGroupType::CrossNodeDataParallel == group_type) { return cross_node_comm_; } diff --git a/orttraining/orttraining/training_ops/cuda/collective/nccl_kernels.cc b/orttraining/orttraining/training_ops/cuda/collective/nccl_kernels.cc index 7bd7dabdbc..0db6629f45 100644 --- a/orttraining/orttraining/training_ops/cuda/collective/nccl_kernels.cc +++ b/orttraining/orttraining/training_ops/cuda/collective/nccl_kernels.cc @@ -217,7 +217,7 @@ ONNX_OPERATOR_KERNEL_EX( kMSDomain, 1, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .VariadicAlias(0, 0) // outputs and inputs are mapped one to one .AllocateInputsContiguously() .TypeConstraint("T", DataTypeImpl::AllIEEEFloatTensorTypes()), @@ -228,7 +228,7 @@ ONNX_OPERATOR_KERNEL_EX( kMSDomain, 1, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .VariadicAlias(0, 0) // outputs and inputs are mapped one to one .AllocateInputsContiguously() .TypeConstraint("T", DataTypeImpl::AllIEEEFloatTensorTypes()), @@ -239,7 +239,7 @@ ONNX_OPERATOR_KERNEL_EX( kMSDomain, 1, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .VariadicAlias(0, 0) // outputs and inputs are mapped one to one .AllocateInputsContiguously() .TypeConstraint("T", DataTypeImpl::AllIEEEFloatTensorTypes()), diff --git a/orttraining/orttraining/training_ops/cuda/communication/nccl_service.cc b/orttraining/orttraining/training_ops/cuda/communication/nccl_service.cc index 7118affb1e..a591b7bd3a 100644 --- a/orttraining/orttraining/training_ops/cuda/communication/nccl_service.cc +++ b/orttraining/orttraining/training_ops/cuda/communication/nccl_service.cc @@ -13,6 +13,10 @@ namespace onnxruntime { namespace cuda { +INcclService& GetINcclService() { + return NcclService::GetInstance(); +} + bool NcclTask::Compare(const NcclTask& other) const { if (type != other.type) { return false; @@ -375,4 +379,4 @@ void NcclService::Terminate() { } // namespace cuda } // namespace onnxruntime -#endif \ No newline at end of file +#endif diff --git a/orttraining/orttraining/training_ops/cuda/communication/nccl_service.h b/orttraining/orttraining/training_ops/cuda/communication/nccl_service.h index b108357cce..48a48fde60 100644 --- a/orttraining/orttraining/training_ops/cuda/communication/nccl_service.h +++ b/orttraining/orttraining/training_ops/cuda/communication/nccl_service.h @@ -81,7 +81,7 @@ struct NcclTaskGroup final { // Below is an example of planning tasks. Notice that the communication operations in the same group are // called in random order, so those operations cannot have mutual dependency. // -// auto& nccl_service = cuda::NcclService::GetInstance(); +// auto& nccl_service = cuda::INcclService::GetInstance(); // // nccl_service.PlanStart(); // Signal the begin of communication planning. // @@ -96,7 +96,43 @@ struct NcclTaskGroup final { // nccl_service.PlanEndNewGroup(); // Mark the end of the second time slot. // // nccl_service.EndPlan(); // Signal the end of communication planning. -class NcclService final { +class INcclService { + public: + static INcclService& GetInstance(); + + // Planning APIs. They are not thread-safe. + + // Mark the start of entire plan. + virtual void PlanStart() = 0; + // Mark the end of entire plan. + virtual void PlanEnd() = 0; + // Mark the begin of a new communication group. It uses the latest time slot. + // Operations in a group can happen in random order. + virtual void PlanNewGroupStart() = 0; + // Mark the end of the current communication group. + virtual void PlanNewGroupEnd() = 0; + // Add Send to the current communication group. + virtual void PlanSend(const int dst) = 0; + // Add Recv to the current communication group. + virtual void PlanRecv(const int src) = 0; + + // Runtime APIs. They are thread-safe. + + // Launch NCCL service. It's an infinite loop which repeatedly calls corresponding NCCL + // when planned operators (e.g., Send and Recv) arrive. + virtual void Launch() = 0; + // Submit a Send request with needed information such as tensor's address and number bytes to send. + virtual void SubmitSendAndWait(void* buffer, size_t count, int peer) = 0; + // Submit a Recv request with needed information such as tensor's address and number bytes to recv. + virtual void SubmitRecvAndWait(void* buffer, size_t count, int peer) = 0; + // Reset communication plan's status so that we can reuse the same communication plan for multiple + // model update steps. + virtual void Reset() = 0; + // Terminate NCCL service. + virtual void Terminate() = 0; +}; + +class NcclService final : public INcclService { public: // Get the singleton of this class. static NcclService& GetInstance() { @@ -107,33 +143,33 @@ class NcclService final { // Planning APIs. They are not thread-safe. // Mark the start of entire plan. - void PlanStart(); + void PlanStart() override; // Mark the end of entire plan. - void PlanEnd(); + void PlanEnd() override; // Mark the begin of a new communication group. It uses the latest time slot. // Operations in a group can happen in random order. - void PlanNewGroupStart(); + void PlanNewGroupStart() override; // Mark the end of the current communication group. - void PlanNewGroupEnd(); + void PlanNewGroupEnd() override; // Add Send to the current communication group. - void PlanSend(const int dst); + void PlanSend(const int dst) override; // Add Recv to the current communication group. - void PlanRecv(const int src); + void PlanRecv(const int src) override; // Runtime APIs. They are thread-safe. // Launch NCCL service. It's an infinite loop which repeatedly calls corresponding NCCL // when planned operators (e.g., Send and Recv) arrive. - void Launch(); + void Launch() override; // Submit a Send request with needed information such as tensor's address and number bytes to send. - void SubmitSendAndWait(void* buffer, size_t count, int peer); + void SubmitSendAndWait(void* buffer, size_t count, int peer) override; // Submit a Recv request with needed information such as tensor's address and number bytes to recv. - void SubmitRecvAndWait(void* buffer, size_t count, int peer); + void SubmitRecvAndWait(void* buffer, size_t count, int peer) override; // Reset communication plan's status so that we can reuse the same communication plan for multiple // model update steps. - void Reset(); + void Reset() override; // Terminate NCCL service. - void Terminate(); + void Terminate() override; // Print debug string. friend std::ostream& operator<<(std::ostream& stream, const NcclService& service); @@ -184,4 +220,4 @@ class NcclService final { } // namespace cuda } // namespace onnxruntime -#endif \ No newline at end of file +#endif diff --git a/orttraining/orttraining/training_ops/cuda/communication/recv.cc b/orttraining/orttraining/training_ops/cuda/communication/recv.cc index fb9b383cf0..46d682066b 100644 --- a/orttraining/orttraining/training_ops/cuda/communication/recv.cc +++ b/orttraining/orttraining/training_ops/cuda/communication/recv.cc @@ -90,16 +90,16 @@ void Recv::ReceiveData( // Copy data out from buffer. #if defined(ORT_USE_NCCL) && defined(USE_NCCL_P2P) CUDA_CALL(cudaMemcpyAsync(tensor->MutableDataRaw(), buffer.get() + tensor_offset_in_bytes, - tensor->SizeInBytes(), cudaMemcpyDeviceToDevice, Stream())); + tensor->SizeInBytes(), cudaMemcpyDeviceToDevice, Stream())); #else CUDA_CALL(cudaMemcpyAsync(tensor->MutableDataRaw(), buffer.get() + tensor_offset_in_bytes, - tensor->SizeInBytes(), cudaMemcpyHostToDevice, Stream())); + tensor->SizeInBytes(), cudaMemcpyHostToDevice, Stream())); #endif #ifndef NDEBUG - // In addition to the first output, other tensors are allocated on GPU. - // We check if the allocated memory is on the current CUDA device. - CheckIfMemoryOnCurrentGpuDevice(tensor->DataRaw()); + // In addition to the first output, other tensors are allocated on GPU. + // We check if the allocated memory is on the current CUDA device. + CheckIfMemoryOnCurrentGpuDevice(tensor->DataRaw()); #endif tensor_offset_in_bytes += tensor->SizeInBytes(); } @@ -121,10 +121,10 @@ ONNX_OPERATOR_KERNEL_EX( kMSDomain, 1, kCudaExecutionProvider, - KernelDefBuilder() - .InputMemoryType(0) /* CPU variable */ - .InputMemoryType(1) /* CPU variable */ - .OutputMemoryType(0) /* CPU variable */ + (*KernelDefBuilder::Create()) + .InputMemoryType(OrtMemTypeCPUInput, 0) /* CPU variable */ + .InputMemoryType(OrtMemTypeCPUInput, 1) /* CPU variable */ + .OutputMemoryType(OrtMemTypeCPUOutput, 0) /* CPU variable */ .TypeConstraint("TBool", DataTypeImpl::GetTensorType()) .TypeConstraint("TInt64", DataTypeImpl::GetTensorType()) .TypeConstraint("V", DataTypeImpl::AllFixedSizeTensorTypes()), @@ -274,4 +274,4 @@ Status Recv::ComputeInternal(OpKernelContext* ctx) const { } // namespace cuda } // namespace onnxruntime -#endif \ No newline at end of file +#endif diff --git a/orttraining/orttraining/training_ops/cuda/communication/send.cc b/orttraining/orttraining/training_ops/cuda/communication/send.cc index 6a5bc71fd3..7a3a21a63c 100644 --- a/orttraining/orttraining/training_ops/cuda/communication/send.cc +++ b/orttraining/orttraining/training_ops/cuda/communication/send.cc @@ -22,10 +22,10 @@ ONNX_OPERATOR_KERNEL_EX( kMSDomain, 1, kCudaExecutionProvider, - KernelDefBuilder() - .InputMemoryType(0) /* CPU variable */ - .InputMemoryType(1) /* CPU variable */ - .OutputMemoryType(0) /* CPU variable */ + (*KernelDefBuilder::Create()) + .InputMemoryType(OrtMemTypeCPUInput, 0) /* CPU variable */ + .InputMemoryType(OrtMemTypeCPUInput, 1) /* CPU variable */ + .OutputMemoryType(OrtMemTypeCPUOutput, 0) /* CPU variable */ .TypeConstraint("TBool", DataTypeImpl::GetTensorType()) .TypeConstraint("TInt64", DataTypeImpl::GetTensorType()) .TypeConstraint("V", DataTypeImpl::AllFixedSizeTensorTypes()), @@ -67,10 +67,10 @@ void Send::SendData( #if defined(ORT_USE_NCCL) && defined(USE_NCCL_P2P) CUDA_CALL(cudaMemcpyAsync(buffer.get() + tensor_offsets_in_bytes[i], tensor->DataRaw(), - tensor_sizes_in_bytes[i], cudaMemcpyDeviceToDevice, Stream())); + tensor_sizes_in_bytes[i], cudaMemcpyDeviceToDevice, Stream())); #else CUDA_CALL(cudaMemcpyAsync(buffer.get() + tensor_offsets_in_bytes[i], tensor->DataRaw(), - tensor_sizes_in_bytes[i], cudaMemcpyDeviceToHost, Stream())); + tensor_sizes_in_bytes[i], cudaMemcpyDeviceToHost, Stream())); #endif } @@ -226,4 +226,4 @@ Status Send::ComputeInternal(OpKernelContext* ctx) const { } // namespace cuda } // namespace onnxruntime -#endif \ No newline at end of file +#endif diff --git a/orttraining/orttraining/training_ops/cuda/controlflow/group.cc b/orttraining/orttraining/training_ops/cuda/controlflow/group.cc index 38456b4866..128407c97c 100644 --- a/orttraining/orttraining/training_ops/cuda/controlflow/group.cc +++ b/orttraining/orttraining/training_ops/cuda/controlflow/group.cc @@ -1,6 +1,7 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. +#include "core/providers/shared_library/provider_api.h" #include "orttraining/training_ops/cpu/controlflow/group.h" #include "core/providers/cuda/cuda_fwd.h" @@ -12,8 +13,8 @@ ONNX_OPERATOR_KERNEL_EX( kMSDomain, 1, kCudaExecutionProvider, - KernelDefBuilder() - .OutputMemoryType(0) + (*KernelDefBuilder::Create()) + .OutputMemoryType(OrtMemTypeCPUOutput, 0) .TypeConstraint("B", DataTypeImpl::GetTensorType()) .TypeConstraint("T", DataTypeImpl::AllTensorTypes()), onnxruntime::contrib::Group); @@ -23,7 +24,7 @@ ONNX_OPERATOR_KERNEL_EX( kMSDomain, 1, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .TypeConstraint("T", DataTypeImpl::AllTensorTypes()) .VariadicAlias(0, 0), // outputs and inputs are mapped one to one onnxruntime::contrib::PassThrough); diff --git a/orttraining/orttraining/training_ops/cuda/controlflow/record.cc b/orttraining/orttraining/training_ops/cuda/controlflow/record.cc index ef47938785..91a98966bd 100644 --- a/orttraining/orttraining/training_ops/cuda/controlflow/record.cc +++ b/orttraining/orttraining/training_ops/cuda/controlflow/record.cc @@ -17,8 +17,8 @@ ONNX_OPERATOR_KERNEL_EX( kMSDomain, 1, kCudaExecutionProvider, - KernelDefBuilder() - .InputMemoryType(0) /* Keep EventIdentifier in CPU */ + (*KernelDefBuilder::Create()) + .InputMemoryType(OrtMemTypeCPUInput, 0) /* Keep EventIdentifier in CPU */ .TypeConstraint("TInt64", DataTypeImpl::GetTensorType()) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()) .VariadicAlias(1, 0), // outputs and inputs are mapped one to one, with input offset by 1 @@ -32,7 +32,7 @@ Status RecordEvent::ComputeInternal(OpKernelContext* ctx) const { auto& profile_context = profile::Context::GetInstance(); const auto tag = profile_context.GetThreadTagOrDefault(std::this_thread::get_id()); profile::NvtxRangeCreator range( - "Batch-" + tag + " Record-" + std::to_string(event_id), profile::Color::Magenta); + "Batch-" + tag + " Record-" + std::to_string(event_id), profile::Color::Magenta); range.Begin(); #endif diff --git a/orttraining/orttraining/training_ops/cuda/controlflow/wait.cc b/orttraining/orttraining/training_ops/cuda/controlflow/wait.cc index 47dc0c15b7..13b32fa7af 100644 --- a/orttraining/orttraining/training_ops/cuda/controlflow/wait.cc +++ b/orttraining/orttraining/training_ops/cuda/controlflow/wait.cc @@ -17,8 +17,8 @@ ONNX_OPERATOR_KERNEL_EX( kMSDomain, 1, kCudaExecutionProvider, - KernelDefBuilder() - .InputMemoryType(0) /* CPU variable */ + (*KernelDefBuilder::Create()) + .InputMemoryType(OrtMemTypeCPUInput, 0) /* CPU variable */ .TypeConstraint("TInt64", DataTypeImpl::GetTensorType()) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()) .VariadicAlias(1, 0), // outputs and inputs are mapped one to one, with input offset by 1 @@ -32,7 +32,7 @@ Status WaitEvent::ComputeInternal(OpKernelContext* ctx) const { auto& profile_context = profile::Context::GetInstance(); const auto tag = profile_context.GetThreadTagOrDefault(std::this_thread::get_id()); profile::NvtxRangeCreator range( - "Batch-" + tag + " Wait-" + std::to_string(event_id), profile::Color::Blue); + "Batch-" + tag + " Wait-" + std::to_string(event_id), profile::Color::Blue); range.Begin(); #endif diff --git a/orttraining/orttraining/training_ops/cuda/controlflow/yield.cc b/orttraining/orttraining/training_ops/cuda/controlflow/yield.cc index 32756afa8a..5ebbf0d1d6 100644 --- a/orttraining/orttraining/training_ops/cuda/controlflow/yield.cc +++ b/orttraining/orttraining/training_ops/cuda/controlflow/yield.cc @@ -1,6 +1,7 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. +#include "core/providers/shared_library/provider_api.h" #include "orttraining/training_ops/cpu/controlflow/yield.h" #include "core/providers/cuda/cuda_fwd.h" @@ -12,7 +13,7 @@ ONNX_OPERATOR_KERNEL_EX( kMSDomain, 1, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()) .ExternalOutputs(), onnxruntime::contrib::YieldOp); diff --git a/orttraining/orttraining/training_ops/cuda/cuda_training_kernels.cc b/orttraining/orttraining/training_ops/cuda/cuda_training_kernels.cc index f5e7e62c09..326b88a494 100644 --- a/orttraining/orttraining/training_ops/cuda/cuda_training_kernels.cc +++ b/orttraining/orttraining/training_ops/cuda/cuda_training_kernels.cc @@ -1,9 +1,9 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. +#include "core/providers/shared_library/provider_api.h" #include "core/providers/cuda/cuda_fwd.h" #include "core/providers/cuda/cuda_pch.h" -#include "core/framework/kernel_registry.h" using namespace onnxruntime::common; diff --git a/orttraining/orttraining/training_ops/cuda/cuda_training_kernels.h b/orttraining/orttraining/training_ops/cuda/cuda_training_kernels.h index 1304d53e06..87e5a2afdd 100644 --- a/orttraining/orttraining/training_ops/cuda/cuda_training_kernels.h +++ b/orttraining/orttraining/training_ops/cuda/cuda_training_kernels.h @@ -3,13 +3,10 @@ #pragma once -#include "core/common/common.h" -#include "core/framework/kernel_registry.h" - namespace onnxruntime { namespace cuda { Status RegisterCudaTrainingKernels(KernelRegistry& kernel_registry); } // namespace cuda -} // namespace onnxruntime \ No newline at end of file +} // namespace onnxruntime diff --git a/orttraining/orttraining/training_ops/cuda/gist/gist.cc b/orttraining/orttraining/training_ops/cuda/gist/gist.cc index 1b53081792..1591bb8f58 100644 --- a/orttraining/orttraining/training_ops/cuda/gist/gist.cc +++ b/orttraining/orttraining/training_ops/cuda/gist/gist.cc @@ -8,14 +8,14 @@ namespace onnxruntime { namespace cuda { // Pack Binary -#define REGISTER_KERNEL_TYPED_BIN_ENC(T) \ - ONNX_OPERATOR_TYPED_KERNEL_EX( \ - GistBinarizeEncoder, \ - kMSDomain, \ - 1, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ +#define REGISTER_KERNEL_TYPED_BIN_ENC(T) \ + ONNX_OPERATOR_TYPED_KERNEL_EX( \ + GistBinarizeEncoder, \ + kMSDomain, \ + 1, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ GistBinarizeEncoderOp); REGISTER_KERNEL_TYPED_BIN_ENC(float) @@ -38,14 +38,14 @@ Status GistBinarizeEncoderOp::ComputeInternal(OpKernelContext* context) const return Status::OK(); } -#define REGISTER_KERNEL_TYPED_BIN_DEC(T) \ - ONNX_OPERATOR_TYPED_KERNEL_EX( \ - GistBinarizeDecoder, \ - kMSDomain, \ - 1, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ +#define REGISTER_KERNEL_TYPED_BIN_DEC(T) \ + ONNX_OPERATOR_TYPED_KERNEL_EX( \ + GistBinarizeDecoder, \ + kMSDomain, \ + 1, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ GistBinarizeDecoderOp); REGISTER_KERNEL_TYPED_BIN_DEC(float) @@ -69,14 +69,14 @@ Status GistBinarizeDecoderOp::ComputeInternal(OpKernelContext* context) const } // Pack1 -#define REGISTER_KERNEL_TYPED_PACK1_ENC(T) \ - ONNX_OPERATOR_TYPED_KERNEL_EX( \ - GistPack1Encoder, \ - kMSDomain, \ - 1, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ +#define REGISTER_KERNEL_TYPED_PACK1_ENC(T) \ + ONNX_OPERATOR_TYPED_KERNEL_EX( \ + GistPack1Encoder, \ + kMSDomain, \ + 1, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ GistPack1EncoderOp); REGISTER_KERNEL_TYPED_PACK1_ENC(bool) @@ -99,14 +99,14 @@ Status GistPack1EncoderOp::ComputeInternal(OpKernelContext* context) const { return Status::OK(); } -#define REGISTER_KERNEL_TYPED_PACK1_DEC(T) \ - ONNX_OPERATOR_TYPED_KERNEL_EX( \ - GistPack1Decoder, \ - kMSDomain, \ - 1, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ +#define REGISTER_KERNEL_TYPED_PACK1_DEC(T) \ + ONNX_OPERATOR_TYPED_KERNEL_EX( \ + GistPack1Decoder, \ + kMSDomain, \ + 1, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ GistPack1DecoderOp); REGISTER_KERNEL_TYPED_PACK1_DEC(bool) @@ -129,14 +129,14 @@ Status GistPack1DecoderOp::ComputeInternal(OpKernelContext* context) const { } // Pack 8 -#define REGISTER_KERNEL_TYPED_PACK8_ENC(T) \ - ONNX_OPERATOR_TYPED_KERNEL_EX( \ - GistPack8Encoder, \ - kMSDomain, \ - 1, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ +#define REGISTER_KERNEL_TYPED_PACK8_ENC(T) \ + ONNX_OPERATOR_TYPED_KERNEL_EX( \ + GistPack8Encoder, \ + kMSDomain, \ + 1, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ GistPack8EncoderOp); REGISTER_KERNEL_TYPED_PACK8_ENC(float) @@ -160,14 +160,14 @@ Status GistPack8EncoderOp::ComputeInternal(OpKernelContext* context) const { return Status::OK(); } -#define REGISTER_KERNEL_TYPED_PACK8_DEC(T) \ - ONNX_OPERATOR_TYPED_KERNEL_EX( \ - GistPack8Decoder, \ - kMSDomain, \ - 1, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ +#define REGISTER_KERNEL_TYPED_PACK8_DEC(T) \ + ONNX_OPERATOR_TYPED_KERNEL_EX( \ + GistPack8Decoder, \ + kMSDomain, \ + 1, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ GistPack8DecoderOp); REGISTER_KERNEL_TYPED_PACK8_DEC(float) @@ -191,14 +191,14 @@ Status GistPack8DecoderOp::ComputeInternal(OpKernelContext* context) const { } // Pack 16 -#define REGISTER_KERNEL_TYPED_PACK16_ENC(T) \ - ONNX_OPERATOR_TYPED_KERNEL_EX( \ - GistPack16Encoder, \ - kMSDomain, \ - 1, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ +#define REGISTER_KERNEL_TYPED_PACK16_ENC(T) \ + ONNX_OPERATOR_TYPED_KERNEL_EX( \ + GistPack16Encoder, \ + kMSDomain, \ + 1, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ GistPack16EncoderOp); REGISTER_KERNEL_TYPED_PACK16_ENC(float) @@ -221,14 +221,14 @@ Status GistPack16EncoderOp::ComputeInternal(OpKernelContext* context) const { return Status::OK(); } -#define REGISTER_KERNEL_TYPED_PACK16_DEC(T) \ - ONNX_OPERATOR_TYPED_KERNEL_EX( \ - GistPack16Decoder, \ - kMSDomain, \ - 1, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ +#define REGISTER_KERNEL_TYPED_PACK16_DEC(T) \ + ONNX_OPERATOR_TYPED_KERNEL_EX( \ + GistPack16Decoder, \ + kMSDomain, \ + 1, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ GistPack16DecoderOp); REGISTER_KERNEL_TYPED_PACK16_DEC(float) @@ -251,14 +251,14 @@ Status GistPack16DecoderOp::ComputeInternal(OpKernelContext* context) const { } // Pack MSFP15 -#define REGISTER_KERNEL_TYPED_PACKMSFP15_ENC(T) \ - ONNX_OPERATOR_TYPED_KERNEL_EX( \ - GistPackMsfp15Encoder, \ - kMSDomain, \ - 1, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ +#define REGISTER_KERNEL_TYPED_PACKMSFP15_ENC(T) \ + ONNX_OPERATOR_TYPED_KERNEL_EX( \ + GistPackMsfp15Encoder, \ + kMSDomain, \ + 1, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ GistPackMsfp15EncoderOp); REGISTER_KERNEL_TYPED_PACKMSFP15_ENC(float) @@ -291,14 +291,14 @@ Status GistPackMsfp15EncoderOp::ComputeInternal(OpKernelContext* context) con return Status::OK(); } -#define REGISTER_KERNEL_TYPED_PACKMSFP15_DEC(T) \ - ONNX_OPERATOR_TYPED_KERNEL_EX( \ - GistPackMsfp15Decoder, \ - kMSDomain, \ - 1, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ +#define REGISTER_KERNEL_TYPED_PACKMSFP15_DEC(T) \ + ONNX_OPERATOR_TYPED_KERNEL_EX( \ + GistPackMsfp15Decoder, \ + kMSDomain, \ + 1, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ GistPackMsfp15DecoderOp); REGISTER_KERNEL_TYPED_PACKMSFP15_DEC(float) diff --git a/orttraining/orttraining/training_ops/cuda/gist/gist.h b/orttraining/orttraining/training_ops/cuda/gist/gist.h index c125444e8d..55d55499d9 100644 --- a/orttraining/orttraining/training_ops/cuda/gist/gist.h +++ b/orttraining/orttraining/training_ops/cuda/gist/gist.h @@ -3,8 +3,6 @@ #pragma once -#include "core/common/common.h" -#include "core/framework/op_kernel.h" #include "core/providers/cuda/cuda_common.h" #include "core/providers/cuda/cuda_kernel.h" diff --git a/orttraining/orttraining/training_ops/cuda/loss/softmax_cross_entropy_loss_impl.cc b/orttraining/orttraining/training_ops/cuda/loss/softmax_cross_entropy_loss_impl.cc index a860497fdd..f2da682b67 100644 --- a/orttraining/orttraining/training_ops/cuda/loss/softmax_cross_entropy_loss_impl.cc +++ b/orttraining/orttraining/training_ops/cuda/loss/softmax_cross_entropy_loss_impl.cc @@ -4,14 +4,21 @@ #include "core/providers/cuda/math/softmax.h" #include "core/providers/cuda/reduction/reduction_functions.h" #include "core/providers/cuda/tensor/transpose.h" -#include "core/providers/cuda/math/unary_elementwise_ops_impl.h" -#include "core/providers/cpu/controlflow/scan_utils.h" +#include "core/framework/ml_value.h" #include "orttraining/training_ops/cpu/loss/softmax_cross_entropy_loss.h" #include "orttraining/training_ops/cuda/loss/softmax_cross_entropy_loss_impl.h" namespace onnxruntime { namespace cuda { +OrtValue AllocateTensorInMLValue(const MLDataType data_type, const TensorShape& shape, AllocatorPtr& allocator) { + auto new_tensor = Tensor::Create(data_type, shape, allocator); + + auto ml_tensor = DataTypeImpl::GetType(); + return OrtValue{new_tensor.release(), ml_tensor, + ml_tensor->GetDeleteFunc()}; +}; + #define REGISTER_KERNEL_VERSIONED_TYPED_TWO_TYPES(Class, T, Tin, domain, startver, endver) \ ONNX_OPERATOR_VERSIONED_TWO_TYPED_KERNEL_EX( \ Class, \ @@ -19,7 +26,7 @@ namespace cuda { startver, endver, \ T, Tin, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()) \ .TypeConstraint("Tin", DataTypeImpl::GetTensorType()), \ Class); @@ -31,7 +38,7 @@ namespace cuda { version, \ T, Tin, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()) \ .TypeConstraint("Tin", DataTypeImpl::GetTensorType()), \ Class); @@ -77,7 +84,6 @@ Status SoftmaxCrossEntropyLoss::ComputeInternal(OpKernelContext* ctx) co } OrtValue transpose_output; - Tensor transpose_tensor; std::vector new_shape; std::vector permutations; AllocatorPtr alloc; @@ -87,7 +93,7 @@ Status SoftmaxCrossEntropyLoss::ComputeInternal(OpKernelContext* ctx) co if (logit_shape.NumDimensions() > 2) { ORT_RETURN_IF_ERROR(ctx->GetTempSpaceAllocator(&alloc)); onnxruntime::contrib::GetPermutationAndShape(true, logit_shape, new_shape, permutations); - transpose_output = scan::detail::AllocateTensorInMLValue(logit.DataType(), new_shape, alloc); + transpose_output = AllocateTensorInMLValue(logit.DataType(), new_shape, alloc); ORT_RETURN_IF_ERROR(cuda::Transpose::DoTranspose(cuda::Transpose(info), permutations, logit, *transpose_output.GetMutable())); logit_data = (*transpose_output.GetMutable()).template Data(); } @@ -208,7 +214,7 @@ Status SoftmaxCrossEntropyLossGrad::ComputeInternal(OpKernelContext* ctx if (probability_shape.NumDimensions() > 2) { ORT_RETURN_IF_ERROR(ctx->GetTempSpaceAllocator(&alloc)); onnxruntime::contrib::GetPermutationAndShape(true, probability_shape, new_shape, permutations); - transpose_output = scan::detail::AllocateTensorInMLValue(log_prob.DataType(), new_shape, alloc); + transpose_output = AllocateTensorInMLValue(log_prob.DataType(), new_shape, alloc); ORT_RETURN_IF_ERROR(cuda::Transpose::DoTranspose(cuda::Transpose(info), permutations, log_prob, *transpose_output.GetMutable())); log_prob_data = (*transpose_output.GetMutable()).template Data(); } diff --git a/orttraining/orttraining/training_ops/cuda/loss/softmax_cross_entropy_loss_impl.cu b/orttraining/orttraining/training_ops/cuda/loss/softmax_cross_entropy_loss_impl.cu index 82565b8ab9..12b2e0f66e 100644 --- a/orttraining/orttraining/training_ops/cuda/loss/softmax_cross_entropy_loss_impl.cu +++ b/orttraining/orttraining/training_ops/cuda/loss/softmax_cross_entropy_loss_impl.cu @@ -1,9 +1,8 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. -#include "core/providers/cuda/cu_inc/common.cuh" -#include "orttraining/training_ops/cuda/loss/softmax_cross_entropy_loss_impl.h" #include "core/providers/cuda/cuda_common.h" +#include "core/providers/cuda/cu_inc/common.cuh" namespace onnxruntime { namespace cuda { diff --git a/orttraining/orttraining/training_ops/cuda/loss/softmaxcrossentropy_impl.cc b/orttraining/orttraining/training_ops/cuda/loss/softmaxcrossentropy_impl.cc index a4b1b55330..5f677fa62a 100644 --- a/orttraining/orttraining/training_ops/cuda/loss/softmaxcrossentropy_impl.cc +++ b/orttraining/orttraining/training_ops/cuda/loss/softmaxcrossentropy_impl.cc @@ -7,14 +7,14 @@ namespace onnxruntime { namespace cuda { -#define REGISTER_KERNEL_TYPED(Class, T, domain, version) \ - ONNX_OPERATOR_TYPED_KERNEL_EX( \ - Class, \ - domain, \ - version, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ +#define REGISTER_KERNEL_TYPED(Class, T, domain, version) \ + ONNX_OPERATOR_TYPED_KERNEL_EX( \ + Class, \ + domain, \ + version, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ Class); #define REGISTER_KERNEL_TYPED_TWO_TYPES(Class, T, Tin, domain, version) \ @@ -24,7 +24,7 @@ namespace cuda { version, \ T, Tin, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()) \ .TypeConstraint("Tin", DataTypeImpl::GetTensorType()), \ Class); diff --git a/orttraining/orttraining/training_ops/cuda/loss/softmaxcrossentropy_impl.cu b/orttraining/orttraining/training_ops/cuda/loss/softmaxcrossentropy_impl.cu index a9165fd9ed..9fd01720b5 100644 --- a/orttraining/orttraining/training_ops/cuda/loss/softmaxcrossentropy_impl.cu +++ b/orttraining/orttraining/training_ops/cuda/loss/softmaxcrossentropy_impl.cu @@ -1,9 +1,8 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. -#include "core/providers/cuda/cu_inc/common.cuh" -#include "softmaxcrossentropy_impl.h" #include "core/providers/cuda/cuda_common.h" +#include "core/providers/cuda/cu_inc/common.cuh" namespace onnxruntime { namespace cuda { diff --git a/orttraining/orttraining/training_ops/cuda/loss/softmaxcrossentropy_impl.h b/orttraining/orttraining/training_ops/cuda/loss/softmaxcrossentropy_impl.h index d41718d276..094f61bd01 100644 --- a/orttraining/orttraining/training_ops/cuda/loss/softmaxcrossentropy_impl.h +++ b/orttraining/orttraining/training_ops/cuda/loss/softmaxcrossentropy_impl.h @@ -108,6 +108,5 @@ class SparseSoftmaxCrossEntropyGrad final : public LossBase { Status ComputeInternal(OpKernelContext* context) const override; }; - } // namespace cuda } // namespace onnxruntime diff --git a/orttraining/orttraining/training_ops/cuda/math/div_grad.cc b/orttraining/orttraining/training_ops/cuda/math/div_grad.cc index 21b3dd9c8d..a9ad00c199 100644 --- a/orttraining/orttraining/training_ops/cuda/math/div_grad.cc +++ b/orttraining/orttraining/training_ops/cuda/math/div_grad.cc @@ -9,14 +9,14 @@ using namespace onnxruntime::common; namespace onnxruntime { namespace cuda { -#define DIVGRAD_REGISTER_KERNEL_TYPED(T) \ - ONNX_OPERATOR_TYPED_KERNEL_EX( \ - DivGrad, \ - kMSDomain, \ - 1, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ +#define DIVGRAD_REGISTER_KERNEL_TYPED(T) \ + ONNX_OPERATOR_TYPED_KERNEL_EX( \ + DivGrad, \ + kMSDomain, \ + 1, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ DivGrad); DIVGRAD_REGISTER_KERNEL_TYPED(MLFloat16) diff --git a/orttraining/orttraining/training_ops/cuda/math/isfinite.cc b/orttraining/orttraining/training_ops/cuda/math/isfinite.cc index fb11abd04e..9741ad2d54 100644 --- a/orttraining/orttraining/training_ops/cuda/math/isfinite.cc +++ b/orttraining/orttraining/training_ops/cuda/math/isfinite.cc @@ -15,7 +15,7 @@ namespace cuda { 1, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()) \ .TypeConstraint("T1", DataTypeImpl::GetTensorType()), \ IsFiniteOp); diff --git a/orttraining/orttraining/training_ops/cuda/math/mixed_precision_scale.cc b/orttraining/orttraining/training_ops/cuda/math/mixed_precision_scale.cc index 1ab0a00307..af9dd95ed1 100644 --- a/orttraining/orttraining/training_ops/cuda/math/mixed_precision_scale.cc +++ b/orttraining/orttraining/training_ops/cuda/math/mixed_precision_scale.cc @@ -9,25 +9,26 @@ namespace onnxruntime { namespace cuda { #if defined(CUDA_VERSION) && CUDA_VERSION >= 11000 -#define ALL_IEEE_FLOAT_TENSOR_TYPES {DataTypeImpl::GetTensorType(), \ - DataTypeImpl::GetTensorType(), \ - DataTypeImpl::GetTensorType(), \ - DataTypeImpl::GetTensorType()} +#define ALL_IEEE_FLOAT_TENSOR_TYPES \ + { DataTypeImpl::GetTensorType(), \ + DataTypeImpl::GetTensorType(), \ + DataTypeImpl::GetTensorType(), \ + DataTypeImpl::GetTensorType() } #else #define ALL_IEEE_FLOAT_TENSOR_TYPES DataTypeImpl::AllIEEEFloatTensorTypes() #endif -#define REGISTER_MIXEDPRECISIONSCALE_KERNEL_TYPED(SrcT) \ - ONNX_OPERATOR_TYPED_KERNEL_EX( \ - MixedPrecisionScale, \ - kMSDomain, \ - 1, \ - SrcT, \ - kCudaExecutionProvider, \ - KernelDefBuilder() \ - .TypeConstraint("SrcT", DataTypeImpl::GetTensorType()) \ - .TypeConstraint("ScaleT", DataTypeImpl::GetTensorType()) \ - .TypeConstraint("DstT", ALL_IEEE_FLOAT_TENSOR_TYPES), \ +#define REGISTER_MIXEDPRECISIONSCALE_KERNEL_TYPED(SrcT) \ + ONNX_OPERATOR_TYPED_KERNEL_EX( \ + MixedPrecisionScale, \ + kMSDomain, \ + 1, \ + SrcT, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()) \ + .TypeConstraint("SrcT", DataTypeImpl::GetTensorType()) \ + .TypeConstraint("ScaleT", DataTypeImpl::GetTensorType()) \ + .TypeConstraint("DstT", ALL_IEEE_FLOAT_TENSOR_TYPES), \ MixedPrecisionScale); Status BytesPerElement(ONNX_NAMESPACE::TensorProto_DataType to, size_t& bytes_per_elem) { diff --git a/orttraining/orttraining/training_ops/cuda/math/scale.cc b/orttraining/orttraining/training_ops/cuda/math/scale.cc index 2345213d6b..8b2ebb45a3 100644 --- a/orttraining/orttraining/training_ops/cuda/math/scale.cc +++ b/orttraining/orttraining/training_ops/cuda/math/scale.cc @@ -8,21 +8,21 @@ using namespace onnxruntime::common; namespace onnxruntime { namespace cuda { -#define REGISTER_SCALE_KERNEL_TYPED(T) \ - ONNX_OPERATOR_TYPED_KERNEL_EX( \ - Scale, \ - kMSDomain, \ - 1, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder() \ - .TypeConstraint("T", DataTypeImpl::GetTensorType()) \ - .TypeConstraint("ScaleT", {DataTypeImpl::GetTensorType(), \ - DataTypeImpl::GetTensorType(), \ - DataTypeImpl::GetTensorType(), \ - DataTypeImpl::GetTensorType(), \ - DataTypeImpl::GetTensorType()}) \ - .InputMemoryType(1), \ +#define REGISTER_SCALE_KERNEL_TYPED(T) \ + ONNX_OPERATOR_TYPED_KERNEL_EX( \ + Scale, \ + kMSDomain, \ + 1, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()) \ + .TypeConstraint("T", DataTypeImpl::GetTensorType()) \ + .TypeConstraint("ScaleT", {DataTypeImpl::GetTensorType(), \ + DataTypeImpl::GetTensorType(), \ + DataTypeImpl::GetTensorType(), \ + DataTypeImpl::GetTensorType(), \ + DataTypeImpl::GetTensorType()}) \ + .InputMemoryType(OrtMemTypeCPUInput, 1), \ Scale); template diff --git a/orttraining/orttraining/training_ops/cuda/math/softmax_grad.cc b/orttraining/orttraining/training_ops/cuda/math/softmax_grad.cc index 74976d0639..9e00f9ede1 100644 --- a/orttraining/orttraining/training_ops/cuda/math/softmax_grad.cc +++ b/orttraining/orttraining/training_ops/cuda/math/softmax_grad.cc @@ -63,52 +63,52 @@ Status SoftMaxGradComputeHelper( #if defined(CUDA_VERSION) && CUDA_VERSION >= 11000 // cudnnSoftmaxForward/Backward doesn't support BFloat16. -#define SPECIALIZED_SOFTMAXGRAD_HELPER_IMPL_BFloat16(is_log_softmax) \ - template <> \ - Status SoftMaxGradComputeHelper( \ - cudaStream_t stream, \ - const BFloat16* dY, \ - const TensorShape& input_shape, \ - const BFloat16* Y, \ - BFloat16* dX, \ - cudnnHandle_t, \ - int64_t axis) { \ - typedef typename ToCudaType::MappedType CudaT; \ - const int64_t normalized_axis = HandleNegativeAxis(axis, input_shape.NumDimensions()); \ - int64_t N = input_shape.SizeToDimension(normalized_axis); \ - int64_t D = input_shape.SizeFromDimension(normalized_axis); \ - auto dY_data = reinterpret_cast(dY); \ - auto Y_data = reinterpret_cast(Y); \ - auto dX_data = reinterpret_cast(dX); \ - dispatch_softmax_backward, is_log_softmax>( \ - stream, dX_data, dY_data, Y_data, gsl::narrow_cast(D), gsl::narrow_cast(D), gsl::narrow_cast(N)); \ - return Status::OK(); \ +#define SPECIALIZED_SOFTMAXGRAD_HELPER_IMPL_BFloat16(is_log_softmax) \ + template <> \ + Status SoftMaxGradComputeHelper( \ + cudaStream_t stream, \ + const BFloat16* dY, \ + const TensorShape& input_shape, \ + const BFloat16* Y, \ + BFloat16* dX, \ + cudnnHandle_t, \ + int64_t axis) { \ + typedef typename ToCudaType::MappedType CudaT; \ + const int64_t normalized_axis = HandleNegativeAxis(axis, input_shape.NumDimensions()); \ + int64_t N = input_shape.SizeToDimension(normalized_axis); \ + int64_t D = input_shape.SizeFromDimension(normalized_axis); \ + auto dY_data = reinterpret_cast(dY); \ + auto Y_data = reinterpret_cast(Y); \ + auto dX_data = reinterpret_cast(dX); \ + dispatch_softmax_backward, is_log_softmax>( \ + stream, dX_data, dY_data, Y_data, gsl::narrow_cast(D), gsl::narrow_cast(D), gsl::narrow_cast(N)); \ + return Status::OK(); \ } SPECIALIZED_SOFTMAXGRAD_HELPER_IMPL_BFloat16(true) -SPECIALIZED_SOFTMAXGRAD_HELPER_IMPL_BFloat16(false) + SPECIALIZED_SOFTMAXGRAD_HELPER_IMPL_BFloat16(false) #endif -#define REGISTER_GRADIENT_KERNEL_TYPED(T) \ - ONNX_OPERATOR_TYPED_KERNEL_EX( \ - SoftmaxGrad, \ - kMSDomain, \ - 1, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ - SoftmaxGrad); \ - ONNX_OPERATOR_TYPED_KERNEL_EX( \ - LogSoftmaxGrad, \ - kMSDomain, \ - 1, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ +#define REGISTER_GRADIENT_KERNEL_TYPED(T) \ + ONNX_OPERATOR_TYPED_KERNEL_EX( \ + SoftmaxGrad, \ + kMSDomain, \ + 1, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ + SoftmaxGrad); \ + ONNX_OPERATOR_TYPED_KERNEL_EX( \ + LogSoftmaxGrad, \ + kMSDomain, \ + 1, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ SoftmaxGrad); -template -Status SoftmaxGrad::ComputeInternal(OpKernelContext* ctx) const { + template + Status SoftmaxGrad::ComputeInternal(OpKernelContext* ctx) const { const Tensor* dY = ctx->Input(0); const TensorShape& input_shape{dY->Shape()}; const Tensor* Y = ctx->Input(1); diff --git a/orttraining/orttraining/training_ops/cuda/nn/batch_norm_grad.cc b/orttraining/orttraining/training_ops/cuda/nn/batch_norm_grad.cc index 7267c46013..4f59eba9ee 100644 --- a/orttraining/orttraining/training_ops/cuda/nn/batch_norm_grad.cc +++ b/orttraining/orttraining/training_ops/cuda/nn/batch_norm_grad.cc @@ -10,14 +10,14 @@ using namespace std; namespace onnxruntime { namespace cuda { -#define REGISTER_GRADIENT_KERNEL_TYPED(T) \ - ONNX_OPERATOR_TYPED_KERNEL_EX( \ - BatchNormalizationGrad, \ - kMSDomain, \ - 1, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ +#define REGISTER_GRADIENT_KERNEL_TYPED(T) \ + ONNX_OPERATOR_TYPED_KERNEL_EX( \ + BatchNormalizationGrad, \ + kMSDomain, \ + 1, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ BatchNormalizationGrad); template diff --git a/orttraining/orttraining/training_ops/cuda/nn/conv_grad.cc b/orttraining/orttraining/training_ops/cuda/nn/conv_grad.cc index ca742ad897..bff551e548 100644 --- a/orttraining/orttraining/training_ops/cuda/nn/conv_grad.cc +++ b/orttraining/orttraining/training_ops/cuda/nn/conv_grad.cc @@ -10,14 +10,14 @@ namespace onnxruntime { namespace cuda { -#define REGISTER_GRADIENT_KERNEL_TYPED(T) \ - ONNX_OPERATOR_TYPED_KERNEL_EX( \ - ConvGrad, \ - kMSDomain, \ - 1, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ +#define REGISTER_GRADIENT_KERNEL_TYPED(T) \ + ONNX_OPERATOR_TYPED_KERNEL_EX( \ + ConvGrad, \ + kMSDomain, \ + 1, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ ConvGrad); REGISTER_GRADIENT_KERNEL_TYPED(float) @@ -226,4 +226,4 @@ Status ConvGrad::ComputeBiasGradient(Tensor* dB, const Tensor* dY) const { } } // namespace cuda -} // namespace onnxruntime \ No newline at end of file +} // namespace onnxruntime diff --git a/orttraining/orttraining/training_ops/cuda/nn/dropout_grad.cc b/orttraining/orttraining/training_ops/cuda/nn/dropout_grad.cc index 5bbfefb80e..3ce65f6a4d 100644 --- a/orttraining/orttraining/training_ops/cuda/nn/dropout_grad.cc +++ b/orttraining/orttraining/training_ops/cuda/nn/dropout_grad.cc @@ -9,17 +9,17 @@ namespace onnxruntime { namespace cuda { -#define REGISTER_GRADIENT_KERNEL(OpName) \ - ONNX_OPERATOR_KERNEL_EX( \ - OpName, \ - kMSDomain, \ - 1, \ - kCudaExecutionProvider, \ - KernelDefBuilder() \ - .TypeConstraint("T", ALL_IEEE_FLOAT_TENSOR_TYPES) \ - .TypeConstraint("T1", ALL_IEEE_FLOAT_TENSOR_TYPES) \ - .TypeConstraint("T2", DataTypeImpl::GetTensorType()) \ - .InputMemoryType(2), \ +#define REGISTER_GRADIENT_KERNEL(OpName) \ + ONNX_OPERATOR_KERNEL_EX( \ + OpName, \ + kMSDomain, \ + 1, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()) \ + .TypeConstraint("T", ALL_IEEE_FLOAT_TENSOR_TYPES) \ + .TypeConstraint("T1", ALL_IEEE_FLOAT_TENSOR_TYPES) \ + .TypeConstraint("T2", DataTypeImpl::GetTensorType()) \ + .InputMemoryType(OrtMemTypeCPUInput, 2), \ DropoutGrad); REGISTER_GRADIENT_KERNEL(DropoutGrad) diff --git a/orttraining/orttraining/training_ops/cuda/nn/layer_norm.cc b/orttraining/orttraining/training_ops/cuda/nn/layer_norm.cc index 64262a1ed1..5b0bcb684f 100644 --- a/orttraining/orttraining/training_ops/cuda/nn/layer_norm.cc +++ b/orttraining/orttraining/training_ops/cuda/nn/layer_norm.cc @@ -16,7 +16,7 @@ namespace cuda { 1, \ T##_##U, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()) \ .TypeConstraint("U", DataTypeImpl::GetTensorType()), \ LayerNormGrad); \ @@ -26,7 +26,7 @@ namespace cuda { 1, \ T##_##U, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()) \ .TypeConstraint("U", DataTypeImpl::GetTensorType()), \ InvertibleLayerNormGrad); \ @@ -36,11 +36,11 @@ namespace cuda { 1, \ T##_##U, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()) \ .TypeConstraint("U", DataTypeImpl::GetTensorType()), \ LayerNormGrad); - + REGISTER_GRADIENT_KERNEL_TYPED(float, float) REGISTER_GRADIENT_KERNEL_TYPED(double, double) REGISTER_GRADIENT_KERNEL_TYPED(MLFloat16, float) @@ -71,7 +71,7 @@ Status LayerNormGrad::ComputeInternal(OpKernelContext* p_op_ke auto Y_grad_data = reinterpret_cast(Y_grad->template Data()); auto X_data = reinterpret_cast(X->template Data()); auto scale_data = reinterpret_cast(scale->template Data()); - auto mean_data = simplified ? nullptr: reinterpret_cast(mean->template Data()); + auto mean_data = simplified ? nullptr : reinterpret_cast(mean->template Data()); auto inv_std_var_data = reinterpret_cast(inv_std_var->template Data()); const TensorShape& x_shape = X->Shape(); @@ -96,9 +96,9 @@ Status LayerNormGrad::ComputeInternal(OpKernelContext* p_op_ke auto part_grad_beta = GetScratchBuffer(part_size * n2); HostLayerNormGradient(GetDeviceProp(), Stream(), Y_grad_data, X_data, reinterpret_cast(NULL), - scale_data, reinterpret_cast(NULL), mean_data, inv_std_var_data, n1, n2, - X_grad_data, scale_grad_data, bias_grad_data, - part_grad_gamma.get(), part_grad_beta.get(), part_size); + scale_data, reinterpret_cast(NULL), mean_data, inv_std_var_data, n1, n2, + X_grad_data, scale_grad_data, bias_grad_data, + part_grad_gamma.get(), part_grad_beta.get(), part_size); return Status::OK(); } @@ -145,9 +145,9 @@ Status InvertibleLayerNormGrad::ComputeInternal(OpKernelContext* p_op_kern auto part_grad_beta = GetScratchBuffer(part_size * n2); HostLayerNormGradient(GetDeviceProp(), Stream(), Y_grad_data, reinterpret_cast(NULL), Y_data, - scale_data, bias_data, reinterpret_cast(NULL), inv_std_var_data, n1, n2, - X_grad_data, scale_grad_data, bias_grad_data, - part_grad_gamma.get(), part_grad_beta.get(), part_size); + scale_data, bias_data, reinterpret_cast(NULL), inv_std_var_data, n1, n2, + X_grad_data, scale_grad_data, bias_grad_data, + part_grad_gamma.get(), part_grad_beta.get(), part_size); return Status::OK(); } diff --git a/orttraining/orttraining/training_ops/cuda/optimizer/adam.cc b/orttraining/orttraining/training_ops/cuda/optimizer/adam.cc index 56b7997f13..cc96e8bdfd 100644 --- a/orttraining/orttraining/training_ops/cuda/optimizer/adam.cc +++ b/orttraining/orttraining/training_ops/cuda/optimizer/adam.cc @@ -18,16 +18,16 @@ namespace cuda { 1, \ T1##_##T2##_##T3##_##T4##_##T_GRAD##_##T_GRAD_NORM##_##T_MIXED_PRECISION_FP, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .Alias(1, 0) /* Update step count in-place */ \ .Alias(2, 3) /* Update weights in-place */ \ .Alias(3, 4) /* Update gradients in-place */ \ .Alias(4, 1) /* Update moment-1 in-place */ \ .Alias(5, 2) /* Update moment-2 in-place */ \ .Alias(6, 5) /* Update mixed_precision weights in-place */ \ - .InputMemoryType(1) /* Keep step count in CPU */ \ - .InputMemoryType(9) /* Keep do_update in CPU */ \ - .OutputMemoryType(0) /* Keep step count in CPU */ \ + .InputMemoryType(OrtMemTypeCPUInput, 1) /* Keep step count in CPU */ \ + .InputMemoryType(OrtMemTypeCPUInput, 9) /* Keep do_update in CPU */ \ + .OutputMemoryType(OrtMemTypeCPUOutput, 0) /* Keep step count in CPU */ \ .TypeConstraint("T1", DataTypeImpl::GetTensorType()) \ .TypeConstraint("T2", DataTypeImpl::GetTensorType()) \ .TypeConstraint("T3", DataTypeImpl::GetTensorType()) \ diff --git a/orttraining/orttraining/training_ops/cuda/optimizer/gradient_control.cc b/orttraining/orttraining/training_ops/cuda/optimizer/gradient_control.cc index a8c4de6f5f..ab116b20d7 100644 --- a/orttraining/orttraining/training_ops/cuda/optimizer/gradient_control.cc +++ b/orttraining/orttraining/training_ops/cuda/optimizer/gradient_control.cc @@ -17,9 +17,9 @@ namespace cuda { 1, \ T##_##T_GRAD, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .Alias(0, 0) /* Accumulate tensors in-place */ \ - .InputMemoryType(2) /* Keep do_update in CPU */ \ + .InputMemoryType(OrtMemTypeCPUInput, 2) /* Keep do_update in CPU */ \ .TypeConstraint("T", DataTypeImpl::GetTensorType()) \ .TypeConstraint("T_GRAD", DataTypeImpl::GetTensorType()), \ InPlaceAccumulator); @@ -54,7 +54,7 @@ Status ZeroGradient::ComputeInternal(OpKernelContext* ctx) const { 1, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .Alias(0, 0) /* Zero out gradients in-place */ \ .TypeConstraint("T1", DataTypeImpl::GetTensorType()) \ .TypeConstraint("T2", DataTypeImpl::AllTensorTypes()), \ diff --git a/orttraining/orttraining/training_ops/cuda/optimizer/lamb.cc b/orttraining/orttraining/training_ops/cuda/optimizer/lamb.cc index f7ab646650..59395bc623 100644 --- a/orttraining/orttraining/training_ops/cuda/optimizer/lamb.cc +++ b/orttraining/orttraining/training_ops/cuda/optimizer/lamb.cc @@ -53,11 +53,11 @@ std::vector> GenerateLambExtraAliasMapping() { 1, \ T1##_##T2##_##T3##_##T4##_##T_GRAD_NORM##_##T_MIXED_PRECISION_FP, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ + (*KernelDefBuilder::Create()) \ .Alias(GenerateLambExtraAliasMapping()) \ - .InputMemoryType(0) /* Keep do_update in CPU */ \ - .InputMemoryType(4) /* Keep iteration_count in CPU */ \ - .OutputMemoryType(0) /* Keep iteration_count in CPU */ \ + .InputMemoryType(OrtMemTypeCPUInput, 0) /* Keep do_update in CPU */ \ + .InputMemoryType(OrtMemTypeCPUInput, 4) /* Keep iteration_count in CPU */ \ + .OutputMemoryType(OrtMemTypeCPUOutput, 0) /* Keep iteration_count in CPU */ \ .TypeConstraint("T1", DataTypeImpl::GetTensorType()) \ .TypeConstraint("T2", DataTypeImpl::GetTensorType()) \ .TypeConstraint("T3", DataTypeImpl::GetTensorType()) \ diff --git a/orttraining/orttraining/training_ops/cuda/optimizer/sg.cc b/orttraining/orttraining/training_ops/cuda/optimizer/sg.cc index 048f1da8e7..e2133af960 100644 --- a/orttraining/orttraining/training_ops/cuda/optimizer/sg.cc +++ b/orttraining/orttraining/training_ops/cuda/optimizer/sg.cc @@ -14,7 +14,7 @@ ONNX_OPERATOR_KERNEL_EX( kMSDomain, 1, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .Alias(1, 0) // Update weights in-place .Alias(2, 1) // Update gradients in-place .TypeConstraint("T", DataTypeImpl::GetTensorType()), diff --git a/orttraining/orttraining/training_ops/cuda/reduction/all.cc b/orttraining/orttraining/training_ops/cuda/reduction/all.cc index 53b80154b2..4178d2dd59 100644 --- a/orttraining/orttraining/training_ops/cuda/reduction/all.cc +++ b/orttraining/orttraining/training_ops/cuda/reduction/all.cc @@ -6,14 +6,14 @@ namespace onnxruntime { namespace cuda { -#define REGISTER_ALL_KERNEL_TYPED(T) \ - ONNX_OPERATOR_TYPED_KERNEL_EX( \ - All, \ - kMSDomain, \ - 1, \ - T, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", DataTypeImpl::GetTensorType()), \ +#define REGISTER_ALL_KERNEL_TYPED(T) \ + ONNX_OPERATOR_TYPED_KERNEL_EX( \ + All, \ + kMSDomain, \ + 1, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", DataTypeImpl::GetTensorType()), \ All); template diff --git a/orttraining/orttraining/training_ops/cuda/reduction/reduction_all.cc b/orttraining/orttraining/training_ops/cuda/reduction/reduction_all.cc index 654b915ffe..71d04393e6 100644 --- a/orttraining/orttraining/training_ops/cuda/reduction/reduction_all.cc +++ b/orttraining/orttraining/training_ops/cuda/reduction/reduction_all.cc @@ -3,21 +3,20 @@ #include "orttraining/training_ops/cuda/reduction/reduction_all.h" -#include "core/framework/op_kernel_context_internal.h" #include "core/providers/cuda/reduction/reduction_functions.h" #include "core/providers/cuda/shared_inc/accumulation_type.h" namespace onnxruntime { namespace cuda { -#define REGISTER_REDUCE_ALL_KERNEL_TYPED(Name, TIn, TOut) \ - ONNX_OPERATOR_TYPED_KERNEL_EX( \ - Name, \ - kMSDomain, \ - 1, \ - TIn##_##TOut, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("TIn", DataTypeImpl::GetTensorType()).TypeConstraint("TOut", DataTypeImpl::GetTensorType()), \ +#define REGISTER_REDUCE_ALL_KERNEL_TYPED(Name, TIn, TOut) \ + ONNX_OPERATOR_TYPED_KERNEL_EX( \ + Name, \ + kMSDomain, \ + 1, \ + TIn##_##TOut, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("TIn", DataTypeImpl::GetTensorType()).TypeConstraint("TOut", DataTypeImpl::GetTensorType()), \ Name); template @@ -55,7 +54,7 @@ Status ReduceAllL2::ComputeInternal(OpKernelContext* ctx) const { // Check if all values are finite and write true to deviceOutput. // Otherwise, false will be written. launch_multi_tensor_functor<1, TFunctor>(Stream(), - 2048 * 32, tensor_sizes, grouped_tensor_pointers, functor, p_output); + 2048 * 32, tensor_sizes, grouped_tensor_pointers, functor, p_output); // *p_output is the squared sum of all elements. // Let's take a sqrt to get the actual L2-norm. @@ -111,4 +110,4 @@ REGISTER_REDUCE_ALL_KERNEL_TYPED(ReduceAllL2, BFloat16, BFloat16) #endif } // namespace cuda -} // namespace onnxruntime \ No newline at end of file +} // namespace onnxruntime diff --git a/orttraining/orttraining/training_ops/cuda/reduction/reduction_ops.cc b/orttraining/orttraining/training_ops/cuda/reduction/reduction_ops.cc index adfaeff30f..a01152140d 100644 --- a/orttraining/orttraining/training_ops/cuda/reduction/reduction_ops.cc +++ b/orttraining/orttraining/training_ops/cuda/reduction/reduction_ops.cc @@ -8,7 +8,6 @@ #include "core/providers/cuda/math/binary_elementwise_ops_impl.h" #include "core/providers/cuda/math/binary_elementwise_ops.h" #include "core/providers/cpu/tensor/utils.h" -#include "core/framework/op_kernel_context_internal.h" using namespace onnxruntime::common; namespace onnxruntime { @@ -21,8 +20,8 @@ namespace cuda { 1, \ T, \ kCudaExecutionProvider, \ - KernelDefBuilder() \ - .InputMemoryType(1) \ + (*KernelDefBuilder::Create()) \ + .InputMemoryType(OrtMemTypeCPUInput, 1) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()), \ name); diff --git a/orttraining/orttraining/training_ops/cuda/tensor/concat.cc b/orttraining/orttraining/training_ops/cuda/tensor/concat.cc index 736b8d9c52..103cc91892 100644 --- a/orttraining/orttraining/training_ops/cuda/tensor/concat.cc +++ b/orttraining/orttraining/training_ops/cuda/tensor/concat.cc @@ -3,15 +3,14 @@ #include "orttraining/training_ops/cuda/tensor/concat.h" #include "core/providers/cuda/tensor/concat_impl.h" - namespace onnxruntime { namespace cuda { ONNX_OPERATOR_KERNEL_EX(ConcatTraining, kMSDomain, 1, kCudaExecutionProvider, - KernelDefBuilder() - .OutputMemoryType(1) + (*KernelDefBuilder::Create()) + .OutputMemoryType(OrtMemTypeCPUInput, 1) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()), ConcatTraining); diff --git a/orttraining/orttraining/training_ops/cuda/tensor/concat.h b/orttraining/orttraining/training_ops/cuda/tensor/concat.h index 3d401433b6..702c363b4c 100644 --- a/orttraining/orttraining/training_ops/cuda/tensor/concat.h +++ b/orttraining/orttraining/training_ops/cuda/tensor/concat.h @@ -1,9 +1,8 @@ // Copyright (c) Microsoft Corporation. All rights reserved. // Licensed under the MIT License. #pragma once -#include "core/common/common.h" #include "core/providers/cuda/cuda_kernel.h" -#include "core/providers/cpu/tensor/concat.h" +#include "core/providers/cpu/tensor/concatbase.h" namespace onnxruntime { namespace cuda { diff --git a/orttraining/orttraining/training_ops/cuda/tensor/gather_elements_grad.cc b/orttraining/orttraining/training_ops/cuda/tensor/gather_elements_grad.cc index 2a7c32ea1d..f5590ff0c6 100644 --- a/orttraining/orttraining/training_ops/cuda/tensor/gather_elements_grad.cc +++ b/orttraining/orttraining/training_ops/cuda/tensor/gather_elements_grad.cc @@ -14,8 +14,8 @@ ONNX_OPERATOR_KERNEL_EX( kMSDomain, 1, kCudaExecutionProvider, - KernelDefBuilder() - .InputMemoryType(1) // 'GatherElements' data shape needs to be on CPU + (*KernelDefBuilder::Create()) + .InputMemoryType(OrtMemTypeCPUInput, 1) // 'GatherElements' data shape needs to be on CPU .TypeConstraint("T", DataTypeImpl::AllIEEEFloatTensorTypes()) .TypeConstraint("I", DataTypeImpl::GetTensorType()) .TypeConstraint("Tind", std::vector{DataTypeImpl::GetTensorType(), diff --git a/orttraining/orttraining/training_ops/cuda/tensor/gather_grad.cc b/orttraining/orttraining/training_ops/cuda/tensor/gather_grad.cc index 9a89cda8c1..865443d207 100644 --- a/orttraining/orttraining/training_ops/cuda/tensor/gather_grad.cc +++ b/orttraining/orttraining/training_ops/cuda/tensor/gather_grad.cc @@ -11,10 +11,11 @@ namespace onnxruntime { namespace cuda { #if defined(CUDA_VERSION) && CUDA_VERSION >= 11000 -#define ALL_IEEE_FLOAT_TENSOR_TYPES {DataTypeImpl::GetTensorType(), \ - DataTypeImpl::GetTensorType(), \ - DataTypeImpl::GetTensorType(), \ - DataTypeImpl::GetTensorType()} +#define ALL_IEEE_FLOAT_TENSOR_TYPES \ + { DataTypeImpl::GetTensorType(), \ + DataTypeImpl::GetTensorType(), \ + DataTypeImpl::GetTensorType(), \ + DataTypeImpl::GetTensorType() } #else #define ALL_IEEE_FLOAT_TENSOR_TYPES DataTypeImpl::AllIEEEFloatTensorTypes() #endif @@ -24,8 +25,8 @@ ONNX_OPERATOR_KERNEL_EX( kMSDomain, 1, kCudaExecutionProvider, - KernelDefBuilder() - .InputMemoryType(0) + (*KernelDefBuilder::Create()) + .InputMemoryType(OrtMemTypeCPUInput, 0) .TypeConstraint("I", DataTypeImpl::GetTensorType()) .TypeConstraint("T", ALL_IEEE_FLOAT_TENSOR_TYPES) .TypeConstraint("Tind", std::vector{ diff --git a/orttraining/orttraining/training_ops/cuda/tensor/gather_nd_grad.cc b/orttraining/orttraining/training_ops/cuda/tensor/gather_nd_grad.cc index 6bfc0eb68f..285c7e66f1 100644 --- a/orttraining/orttraining/training_ops/cuda/tensor/gather_nd_grad.cc +++ b/orttraining/orttraining/training_ops/cuda/tensor/gather_nd_grad.cc @@ -9,27 +9,25 @@ namespace onnxruntime { namespace cuda { #if defined(CUDA_VERSION) && CUDA_VERSION >= 11000 -#define ALL_IEEE_FLOAT_TENSOR_TYPES {DataTypeImpl::GetTensorType(), \ - DataTypeImpl::GetTensorType(), \ - DataTypeImpl::GetTensorType(), \ - DataTypeImpl::GetTensorType()} +#define ALL_IEEE_FLOAT_TENSOR_TYPES \ + { DataTypeImpl::GetTensorType(), \ + DataTypeImpl::GetTensorType(), \ + DataTypeImpl::GetTensorType(), \ + DataTypeImpl::GetTensorType() } #define ALL_IEEE_FLOAT_DATA_TYPES float, MLFloat16, double, BFloat16 #else #define ALL_IEEE_FLOAT_TENSOR_TYPES DataTypeImpl::AllIEEEFloatTensorTypes() #define ALL_IEEE_FLOAT_DATA_TYPES float, MLFloat16, double #endif -#define REGISTER_KERNEL_TYPED_GATHER_ND_GRAD(TIndex) \ - ONNX_OPERATOR_TYPED_KERNEL_EX( \ - GatherNDGrad, \ - kMSDomain, \ - 1, \ - TIndex, \ - kCudaExecutionProvider, \ - KernelDefBuilder().TypeConstraint("T", ALL_IEEE_FLOAT_TENSOR_TYPES) \ - .TypeConstraint("Tind", DataTypeImpl::GetTensorType()) \ - .TypeConstraint("T1", DataTypeImpl::GetTensorType()) \ - .InputMemoryType(0), \ +#define REGISTER_KERNEL_TYPED_GATHER_ND_GRAD(TIndex) \ + ONNX_OPERATOR_TYPED_KERNEL_EX( \ + GatherNDGrad, \ + kMSDomain, \ + 1, \ + TIndex, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()).TypeConstraint("T", ALL_IEEE_FLOAT_TENSOR_TYPES).TypeConstraint("Tind", DataTypeImpl::GetTensorType()).TypeConstraint("T1", DataTypeImpl::GetTensorType()).InputMemoryType(OrtMemTypeCPUInput, 0), \ GatherNDGrad); REGISTER_KERNEL_TYPED_GATHER_ND_GRAD(int64_t) diff --git a/orttraining/orttraining/training_ops/cuda/tensor/slice_grad.cc b/orttraining/orttraining/training_ops/cuda/tensor/slice_grad.cc index 7004cbfb70..f76d968687 100644 --- a/orttraining/orttraining/training_ops/cuda/tensor/slice_grad.cc +++ b/orttraining/orttraining/training_ops/cuda/tensor/slice_grad.cc @@ -13,12 +13,12 @@ ONNX_OPERATOR_KERNEL_EX( kMSDomain, 1, kCudaExecutionProvider, - KernelDefBuilder() - .InputMemoryType(1) - .InputMemoryType(2) - .InputMemoryType(3) - .InputMemoryType(4) - .InputMemoryType(5) + (*KernelDefBuilder::Create()) + .InputMemoryType(OrtMemTypeCPUInput, 1) + .InputMemoryType(OrtMemTypeCPUInput, 2) + .InputMemoryType(OrtMemTypeCPUInput, 3) + .InputMemoryType(OrtMemTypeCPUInput, 4) + .InputMemoryType(OrtMemTypeCPUInput, 5) .TypeConstraint("I", DataTypeImpl::GetTensorType()) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()) .TypeConstraint("Tind", std::vector{ diff --git a/orttraining/orttraining/training_ops/cuda/tensor/split.cc b/orttraining/orttraining/training_ops/cuda/tensor/split.cc index 37a18c603f..956c423255 100644 --- a/orttraining/orttraining/training_ops/cuda/tensor/split.cc +++ b/orttraining/orttraining/training_ops/cuda/tensor/split.cc @@ -12,8 +12,8 @@ ONNX_OPERATOR_KERNEL_EX(SplitTraining, kMSDomain, 1, kCudaExecutionProvider, - KernelDefBuilder() - .InputMemoryType(1) + (*KernelDefBuilder::Create()) + .InputMemoryType(OrtMemTypeCPUInput, 1) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()), SplitTraining); diff --git a/orttraining/orttraining/training_ops/cuda/tensor/view.cc b/orttraining/orttraining/training_ops/cuda/tensor/view.cc index 6d5d9da000..47de07ae1f 100644 --- a/orttraining/orttraining/training_ops/cuda/tensor/view.cc +++ b/orttraining/orttraining/training_ops/cuda/tensor/view.cc @@ -32,10 +32,10 @@ ONNX_OPERATOR_KERNEL_EX( kMSDomain, 1, kCudaExecutionProvider, - KernelDefBuilder() + (*KernelDefBuilder::Create()) .TypeConstraint("T", DataTypeImpl::AllFixedSizeTensorTypes()) .TypeConstraint("Shape", DataTypeImpl::GetTensorType()) - .InputMemoryType(GenerateInputMemoryType()) // all shape inputs are in CPU + .InputMemoryType(OrtMemTypeCPUInput, GenerateInputMemoryType()) // all shape inputs are in CPU .Alias(GenerateAliasMapping()), // all output tensors are sharing the same bffer as input[0], // execept that the byte_offset is different View); diff --git a/setup.py b/setup.py index a04c9c72d8..d6c3f2eff3 100644 --- a/setup.py +++ b/setup.py @@ -170,6 +170,7 @@ if platform.system() == 'Linux': libs.extend(['libonnxruntime_providers_dnnl.so']) libs.extend(['libonnxruntime_providers_tensorrt.so']) libs.extend(['libonnxruntime_providers_openvino.so']) + libs.extend(['libonnxruntime_providers_cuda.so']) # Nuphar Libs libs.extend(['libtvm.so.0.5.1']) if nightly_build: @@ -180,6 +181,7 @@ elif platform.system() == "Darwin": libs.extend(['libonnxruntime_providers_shared.dylib']) libs.extend(['libonnxruntime_providers_dnnl.dylib']) libs.extend(['libonnxruntime_providers_tensorrt.dylib']) + libs.extend(['libonnxruntime_providers_cuda.dylib']) if nightly_build: libs.extend(['libonnxruntime_pywrapper.dylib']) else: @@ -189,6 +191,7 @@ else: libs.extend(['onnxruntime_providers_dnnl.dll']) libs.extend(['onnxruntime_providers_tensorrt.dll']) libs.extend(['onnxruntime_providers_openvino.dll']) + libs.extend(['onnxruntime_providers_cuda.dll']) # DirectML Libs libs.extend(['DirectML.dll']) # Nuphar Libs diff --git a/tools/ci_build/github/linux/copy_strip_binary.sh b/tools/ci_build/github/linux/copy_strip_binary.sh index 3e3e2911ed..5962677eab 100755 --- a/tools/ci_build/github/linux/copy_strip_binary.sh +++ b/tools/ci_build/github/linux/copy_strip_binary.sh @@ -21,6 +21,10 @@ mkdir $BINARY_DIR/$ARTIFACT_NAME/lib mkdir $BINARY_DIR/$ARTIFACT_NAME/include echo "Directories created" cp $BINARY_DIR/$BUILD_CONFIG/$LIB_NAME $BINARY_DIR/$ARTIFACT_NAME/lib +if [[ -f "$BINARY_DIR/$BUILD_CONFIG/libonnxruntime_providers_cuda.so" ]]; then + cp $BINARY_DIR/$BUILD_CONFIG/libonnxruntime_providers_shared.so $BINARY_DIR/$ARTIFACT_NAME/lib + cp $BINARY_DIR/$BUILD_CONFIG/libonnxruntime_providers_cuda.so $BINARY_DIR/$ARTIFACT_NAME/lib +fi echo "Copy debug symbols in a separate file and strip the original binary." if [[ $LIB_NAME == *.dylib ]] then diff --git a/tools/ci_build/github/linux/copy_strip_binary_trt.sh b/tools/ci_build/github/linux/copy_strip_binary_trt.sh index 0c187bd9db..f033d6b114 100644 --- a/tools/ci_build/github/linux/copy_strip_binary_trt.sh +++ b/tools/ci_build/github/linux/copy_strip_binary_trt.sh @@ -23,6 +23,7 @@ echo "Directories created" cp $BINARY_DIR/$BUILD_CONFIG/$LIB_NAME $BINARY_DIR/$ARTIFACT_NAME/lib cp $BINARY_DIR/$BUILD_CONFIG/libonnxruntime_providers_shared.so $BINARY_DIR/$ARTIFACT_NAME/lib cp $BINARY_DIR/$BUILD_CONFIG/libonnxruntime_providers_tensorrt.so $BINARY_DIR/$ARTIFACT_NAME/lib +cp $BINARY_DIR/$BUILD_CONFIG/libonnxruntime_providers_cuda.so $BINARY_DIR/$ARTIFACT_NAME/lib echo "Copy debug symbols in a separate file and strip the original binary." if [[ $LIB_NAME == *.dylib ]] then diff --git a/tools/ci_build/github/linux/java_copy_strip_binary.sh b/tools/ci_build/github/linux/java_copy_strip_binary.sh index 568a11bf96..11f097d14e 100755 --- a/tools/ci_build/github/linux/java_copy_strip_binary.sh +++ b/tools/ci_build/github/linux/java_copy_strip_binary.sh @@ -45,6 +45,9 @@ then cp $BINARY_DIR/$BUILD_CONFIG/$NATIVE_LIB_NAME $BINARY_DIR/$ARTIFACT_NAME/$NATIVE_FOLDER/libonnxruntime4j_jni.so # Add custom lib cp $BINARY_DIR/$BUILD_CONFIG/libcustom_op_library.so $BINARY_DIR/$ARTIFACT_NAME + # Add cuda provider if it exists + cp $BINARY_DIR/$BUILD_CONFIG/$LIB_NAME $BINARY_DIR/$ARTIFACT_NAME/$NATIVE_FOLDER/libonnxruntime_providers_shared.so + cp $BINARY_DIR/$BUILD_CONFIG/$LIB_NAME $BINARY_DIR/$ARTIFACT_NAME/$NATIVE_FOLDER/libonnxruntime_providers_cuda.so fi find $BINARY_DIR/$ARTIFACT_NAME -ls diff --git a/tools/nuget/generate_nuspec_for_native_nuget.py b/tools/nuget/generate_nuspec_for_native_nuget.py index b240670d2d..60e600d129 100644 --- a/tools/nuget/generate_nuspec_for_native_nuget.py +++ b/tools/nuget/generate_nuspec_for_native_nuget.py @@ -195,6 +195,7 @@ def generate_files(list, args): 'dnnl_ep_shared_lib': 'onnxruntime_providers_dnnl.dll', 'tensorrt_ep_shared_lib': 'onnxruntime_providers_tensorrt.dll', 'openvino_ep_shared_lib': 'onnxruntime_providers_openvino.dll', + 'cuda_ep_shared_lib': 'onnxruntime_providers_cuda.dll', 'onnxruntime_perf_test': 'onnxruntime_perf_test.exe', 'onnx_test_runner': 'onnx_test_runner.exe'} @@ -210,6 +211,7 @@ def generate_files(list, args): 'dnnl_ep_shared_lib': 'libonnxruntime_providers_dnnl.so', 'tensorrt_ep_shared_lib': 'libonnxruntime_providers_tensorrt.so', 'openvino_ep_shared_lib': 'libonnxruntime_providers_openvino.so', + 'cuda_ep_shared_lib': 'libonnxruntime_providers_cuda.so', 'onnxruntime_perf_test': 'onnxruntime_perf_test', 'onnx_test_runner': 'onnx_test_runner'} @@ -334,6 +336,9 @@ def generate_files(list, args): files_list.append('') + files_list.append('') files_list.append('') @@ -354,6 +359,14 @@ def generate_files(list, args): nuget_dependencies['openvino_ep_shared_lib']) + runtimes_target + args.target_architecture + '\\native" />') + if args.execution_provider == "cuda": + files_list.append('') + files_list.append('') + # process all other library dependencies if is_cpu_package or is_cuda_gpu_package or is_dml_package or is_mklml_package: # Process dnnl dependency @@ -468,7 +481,7 @@ def validate_platform(): def validate_execution_provider(execution_provider): if is_linux(): - if not (execution_provider == 'None' or execution_provider == 'dnnl' + if not (execution_provider == 'None' or execution_provider == 'dnnl' or execution_provider == 'cuda' or execution_provider == 'tensorrt' or execution_provider == 'openvino'): raise Exception('On Linux platform nuget generation is supported only ' 'for cpu|cuda|dnnl|tensorrt|openvino execution providers.') From f6eb0f76ae5b6f6a5cedf22ee8b2994eb9503361 Mon Sep 17 00:00:00 2001 From: liqunfu Date: Thu, 20 May 2021 09:18:41 -0700 Subject: [PATCH 02/28] to used cudnn7 to build onnxruntime-training wheel with Cuda 10.2 support (#7760) --- ...raining-py-packaging-pipeline-cuda102.yml} | 3 +- ...training-py-packaging-pipeline-cuda111.yml | 15 ++ ...orttraining-py-packaging-pipeline-rocm.yml | 3 +- .../templates/py-packaging-stage.yml | 200 ++++++++++++++++-- ...Dockerfile.manylinux2014_training_cuda10_2 | 2 +- 5 files changed, 199 insertions(+), 24 deletions(-) rename tools/ci_build/github/azure-pipelines/{orttraining-py-packaging-pipeline.yml => orttraining-py-packaging-pipeline-cuda102.yml} (80%) create mode 100644 tools/ci_build/github/azure-pipelines/orttraining-py-packaging-pipeline-cuda111.yml diff --git a/tools/ci_build/github/azure-pipelines/orttraining-py-packaging-pipeline.yml b/tools/ci_build/github/azure-pipelines/orttraining-py-packaging-pipeline-cuda102.yml similarity index 80% rename from tools/ci_build/github/azure-pipelines/orttraining-py-packaging-pipeline.yml rename to tools/ci_build/github/azure-pipelines/orttraining-py-packaging-pipeline-cuda102.yml index e1b358f8ad..7a75a93245 100644 --- a/tools/ci_build/github/azure-pipelines/orttraining-py-packaging-pipeline.yml +++ b/tools/ci_build/github/azure-pipelines/orttraining-py-packaging-pipeline-cuda102.yml @@ -6,7 +6,8 @@ stages: build_py_parameters: --enable_training --update --build enable_linux_cpu: false enable_linux_gpu: false - enable_linux_gpu_training: true + enable_linux_gpu_training_cu102: true + enable_linux_gpu_training_cu111: false enable_linux_rocm_training: false enable_windows_cpu: false enable_windows_gpu: false diff --git a/tools/ci_build/github/azure-pipelines/orttraining-py-packaging-pipeline-cuda111.yml b/tools/ci_build/github/azure-pipelines/orttraining-py-packaging-pipeline-cuda111.yml new file mode 100644 index 0000000000..1684625233 --- /dev/null +++ b/tools/ci_build/github/azure-pipelines/orttraining-py-packaging-pipeline-cuda111.yml @@ -0,0 +1,15 @@ +trigger: none + +stages: +- template: templates/py-packaging-stage.yml + parameters: + build_py_parameters: --enable_training --update --build + enable_linux_cpu: false + enable_linux_gpu: false + enable_linux_gpu_training_cu102: false + enable_linux_gpu_training_cu111: true + enable_linux_rocm_training: false + enable_windows_cpu: false + enable_windows_gpu: false + enable_mac_cpu: false + enable_linux_arm: false diff --git a/tools/ci_build/github/azure-pipelines/orttraining-py-packaging-pipeline-rocm.yml b/tools/ci_build/github/azure-pipelines/orttraining-py-packaging-pipeline-rocm.yml index cbd4ac3354..784e916028 100644 --- a/tools/ci_build/github/azure-pipelines/orttraining-py-packaging-pipeline-rocm.yml +++ b/tools/ci_build/github/azure-pipelines/orttraining-py-packaging-pipeline-rocm.yml @@ -6,7 +6,8 @@ stages: build_py_parameters: --enable_training enable_linux_cpu: false enable_linux_gpu: false - enable_linux_gpu_training: false + enable_linux_gpu_training_cu102: false + enable_linux_gpu_training_cu111: false enable_linux_rocm_training: true enable_windows_cpu: false enable_windows_gpu: false diff --git a/tools/ci_build/github/azure-pipelines/templates/py-packaging-stage.yml b/tools/ci_build/github/azure-pipelines/templates/py-packaging-stage.yml index e6670efd08..4c17c152a2 100644 --- a/tools/ci_build/github/azure-pipelines/templates/py-packaging-stage.yml +++ b/tools/ci_build/github/azure-pipelines/templates/py-packaging-stage.yml @@ -15,8 +15,13 @@ parameters: type: boolean default: true -- name: enable_linux_gpu_training - displayName: 'Whether Linux GPU package is built.' +- name: enable_linux_gpu_training_cu102 + displayName: 'Whether Linux GPU Cuda 10.2 package is built.' + type: boolean + default: false + +- name: enable_linux_gpu_training_cu111 + displayName: 'Whether Linux GPU Cuda 11.1 package is built.' type: boolean default: false @@ -472,42 +477,199 @@ stages: - template: clean-agent-build-directory-step.yml - - ${{ if eq(parameters.enable_linux_gpu_training, true) }}: - - job: Linux_py_GPU_Wheels + - ${{ if eq(parameters.enable_linux_gpu_training_cu102, true) }}: + - job: Linux_py_Cuda102_Wheels timeoutInMinutes: 180 workspace: clean: all - pool: Onnxruntime-Linux-GPU + pool: Onnxruntime-Linux-GPU-NV6 strategy: matrix: Python36 Cuda10.2: PythonVersion: '3.6' CudaVersion: '10.2' DockerFile: 'Dockerfile.manylinux2014_training_cuda10_2' - Python36 Cuda11.1: - PythonVersion: '3.6' - CudaVersion: '11.1' - DockerFile: 'Dockerfile.manylinux2014_training_cuda11_1' Python37 Cuda10.2: PythonVersion: '3.7' CudaVersion: '10.2' DockerFile: 'Dockerfile.manylinux2014_training_cuda10_2' - Python37 Cuda11.1: - PythonVersion: '3.7' - CudaVersion: '11.1' - DockerFile: 'Dockerfile.manylinux2014_training_cuda11_1' Python38 Cuda10.2: PythonVersion: '3.8' CudaVersion: '10.2' DockerFile: 'Dockerfile.manylinux2014_training_cuda10_2' - Python38 Cuda11.1: - PythonVersion: '3.8' - CudaVersion: '11.1' - DockerFile: 'Dockerfile.manylinux2014_training_cuda11_1' Python39 Cuda10.2: PythonVersion: '3.9' CudaVersion: '10.2' DockerFile: 'Dockerfile.manylinux2014_training_cuda10_2' + steps: + + - checkout: self + clean: true + submodules: recursive + + - template: set-python-manylinux-variables-step.yml + + - template: get-docker-image-steps.yml + parameters: + Dockerfile: tools/ci_build/github/linux/docker/$(DockerFile) + Context: tools/ci_build/github/linux/docker + DockerBuildArgs: >- + --build-arg PYTHON_VERSION=$(PythonVersion) + --build-arg CUDA_VERSION=$(CudaVersion) + --build-arg INSTALL_DEPS_EXTRA_ARGS=-tu + --build-arg BUILD_UID=$(id -u) + Repository: onnxruntimetraininggpubuild + + - bash: tools/ci_build/github/linux/docker/scripts/training/azure_scale_set_vm_mount_test_data.sh -p $(orttrainingtestdata-storage-key) -s "//orttrainingtestdata.file.core.windows.net/mnist" -d "/mnist" + displayName: 'Mount MNIST' + condition: succeededOrFailed() + + - bash: tools/ci_build/github/linux/docker/scripts/training/azure_scale_set_vm_mount_test_data.sh -p $(orttrainingtestdata-storage-key) -s "//orttrainingtestdata.file.core.windows.net/bert-data" -d "/bert_data" + displayName: 'Mount bert-data' + condition: succeededOrFailed() + + - bash: tools/ci_build/github/linux/docker/scripts/training/azure_scale_set_vm_mount_test_data.sh -p $(orttrainingtestdata-storage-key) -s "//orttrainingtestdata.file.core.windows.net/hf-models-cache" -d "/hf_models_cache" + displayName: 'Mount hf-models-cache' + condition: succeededOrFailed() + + - task: CmdLine@2 + displayName: 'build onnxruntime' + inputs: + script: | + mkdir -p $HOME/.onnx + docker run --rm --gpus all -e CC=/opt/rh/devtoolset-8/root/usr/bin/cc -e CXX=/opt/rh/devtoolset-8/root/usr/bin/c++ -e CFLAGS="-Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -fstack-clash-protection -fcf-protection -O3 -Wl,--strip-all" -e CXXFLAGS="-Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -fstack-clash-protection -fcf-protection -O3 -Wl,--strip-all" \ + --volume /data/onnx:/data/onnx:ro \ + --volume $(Build.SourcesDirectory):/onnxruntime_src \ + --volume $(Build.BinariesDirectory):/build \ + --volume /data/models:/build/models:ro \ + --volume $HOME/.onnx:/home/onnxruntimedev/.onnx \ + -e NVIDIA_VISIBLE_DEVICES=all \ + -e NIGHTLY_BUILD \ + -e BUILD_BUILDNUMBER \ + onnxruntimetraininggpubuild \ + $(PythonManylinuxDir)/bin/python3 /onnxruntime_src/tools/ci_build/build.py \ + --build_dir /build \ + --config Release \ + --skip_submodule_sync \ + --parallel \ + --build_wheel \ + --enable_onnx_tests \ + ${{ parameters.build_py_parameters }} \ + --cmake_extra_defines CMAKE_CUDA_HOST_COMPILER=/opt/rh/devtoolset-8/root/usr/bin/cc PYTHON_INCLUDE_DIR=$(PythonManylinuxIncludeDir) PYTHON_LIBRARY=/usr/lib64/librt.so \ + --use_cuda --cuda_version=$(CudaVersion) --cuda_home=/usr/local/cuda-$(CudaVersion) --cudnn_home=/usr/local/cuda-$(CudaVersion) ; + workingDirectory: $(Build.SourcesDirectory) + + - task: CmdLine@2 + displayName: 'test ortmodule' + inputs: + script: | + rm -rf $(Build.BinariesDirectory)/Release/onnxruntime/ && \ + files=($(Build.BinariesDirectory)/Release/dist/*.whl) && \ + echo ${files[0]} && \ + whlfilename=$(basename ${files[0]}) && \ + echo $whlfilename && \ + docker run --rm \ + --gpus all \ + -e NVIDIA_VISIBLE_DEVICES=all \ + --volume $(Build.BinariesDirectory):/build \ + --volume /mnist:/mnist \ + --volume /bert_data:/bert_data \ + --volume /hf_models_cache:/hf_models_cache \ + onnxruntimetraininggpubuild \ + bash -c " $(PythonManylinuxDir)/bin/python3 -m pip install /build/Release/dist/$whlfilename ; $(PythonManylinuxDir)/bin/python3 /build/Release/launch_test.py --cmd_line_with_args 'python orttraining_ortmodule_tests.py --mnist /mnist --bert_data /bert_data/hf_data/glue_data/CoLA/original/raw --transformers_cache /hf_models_cache/huggingface/transformers' --cwd /build/Release " ; + workingDirectory: $(Build.SourcesDirectory) + + - task: CopyFiles@2 + displayName: 'Copy Python Wheel to: $(Build.ArtifactStagingDirectory)' + inputs: + SourceFolder: '$(Build.BinariesDirectory)' + Contents: 'Release/dist/*.whl' + TargetFolder: '$(Build.ArtifactStagingDirectory)' + + - task: CmdLine@2 + displayName: 'Build Python Documentation' + condition: ne(variables['PythonVersion'], '3.9') # tensorflow not available on python 3.9 + inputs: + script: | + mkdir -p $HOME/.onnx + docker run --rm \ + --volume /data/onnx:/data/onnx:ro \ + --volume $(Build.SourcesDirectory):/onnxruntime_src \ + --volume $(Build.BinariesDirectory):/build \ + --volume /data/models:/build/models:ro \ + --volume $HOME/.onnx:/home/onnxruntimedev/.onnx \ + -e NIGHTLY_BUILD \ + -e BUILD_BUILDNUMBER \ + onnxruntimetraininggpubuild \ + bash /onnxruntime_src/tools/doc/builddoc.sh $(PythonManylinuxDir)/bin/ /onnxruntime_src /build Release + workingDirectory: $(Build.SourcesDirectory) + + - task: CopyFiles@2 + displayName: 'Copy Python Documentation to: $(Build.ArtifactStagingDirectory)' + condition: ne(variables['PythonVersion'], '3.9') # tensorflow not available on python 3.9 + inputs: + SourceFolder: '$(Build.BinariesDirectory)/docs/training/html' + Contents: '**' + TargetFolder: '$(Build.ArtifactStagingDirectory)/training_html_doc' + + - task: PublishBuildArtifacts@1 + displayName: 'Publish Artifact: ONNXRuntime python wheel and documentation' + inputs: + ArtifactName: onnxruntime_gpu + + # - script: | + # sudo apt-get update + # sudo apt-get install python3-pip python-dev + # displayName: 'sudo apt-get install python3-pip python-dev' + + # - script: | + # python3 -m pip install azure-storage-blob==2.1.0 + # displayName: 'python3 -m pip install azure-storage-blob==2.1.0' + # timeoutInMinutes: 20 + + - task: AzureCLI@2 + inputs: + azureSubscription: 'AIInfraBuildOnnxRuntimeOSS' + scriptType: 'bash' + scriptLocation: 'inlineScript' + inlineScript: | + python3 -m pip install azure-storage-blob==2.1.0 + files=($(Build.ArtifactStagingDirectory)/Release/dist/*.whl) && \ + echo ${files[0]} && \ + python3 tools/ci_build/upload_python_package_to_azure_storage.py \ + --python_wheel_path ${files[0]} \ + --account_name onnxruntimepackages \ + --account_key $(orttrainingpackagestorageaccountkey) \ + --container_name '$web' + condition: succeededOrFailed() + displayName: + + - template: component-governance-component-detection-steps.yml + parameters: + condition: 'succeeded' + + - template: clean-agent-build-directory-step.yml + + - ${{ if eq(parameters.enable_linux_gpu_training_cu111, true) }}: + - job: Linux_py_Cuda111_Wheels + timeoutInMinutes: 180 + workspace: + clean: all + pool: Onnxruntime-Linux-GPU + strategy: + matrix: + Python36 Cuda11.1: + PythonVersion: '3.6' + CudaVersion: '11.1' + DockerFile: 'Dockerfile.manylinux2014_training_cuda11_1' + Python37 Cuda11.1: + PythonVersion: '3.7' + CudaVersion: '11.1' + DockerFile: 'Dockerfile.manylinux2014_training_cuda11_1' + Python38 Cuda11.1: + PythonVersion: '3.8' + CudaVersion: '11.1' + DockerFile: 'Dockerfile.manylinux2014_training_cuda11_1' Python39 Cuda11.1: PythonVersion: '3.9' CudaVersion: '11.1' @@ -570,12 +732,8 @@ stages: --use_cuda --cuda_version=$(CudaVersion) --cuda_home=/usr/local/cuda-$(CudaVersion) --cudnn_home=/usr/local/cuda-$(CudaVersion) ; workingDirectory: $(Build.SourcesDirectory) - # with Cuda 11.1: - # test_bert_inputs_with_dynamic_shape: RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)` - # test_gpu_reserved_memory_with_torch_no_grad: RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemmStridedBatched( handle, opa, opb, m, n, k, &alpha, a, lda, stridea, b, ldb, strideb, &beta, c, ldc, stridec, num_batches)` - task: CmdLine@2 displayName: 'test ortmodule' - condition: ne(variables['CudaVersion'], '11.1') inputs: script: | rm -rf $(Build.BinariesDirectory)/Release/onnxruntime/ && \ diff --git a/tools/ci_build/github/linux/docker/Dockerfile.manylinux2014_training_cuda10_2 b/tools/ci_build/github/linux/docker/Dockerfile.manylinux2014_training_cuda10_2 index d0c9cd9a49..6e584ddfd9 100644 --- a/tools/ci_build/github/linux/docker/Dockerfile.manylinux2014_training_cuda10_2 +++ b/tools/ci_build/github/linux/docker/Dockerfile.manylinux2014_training_cuda10_2 @@ -1,6 +1,6 @@ # TODO unify this with Dockerfile.manylinux2014_cuda10_2 -FROM nvcr.io/nvidia/cuda:10.2-cudnn8-devel-centos7 +FROM nvcr.io/nvidia/cuda:10.2-cudnn7-devel-centos7 #We need both CUDA and manylinux. But the CUDA Toolkit End User License Agreement says NVIDIA CUDA Driver Libraries(libcuda.so, libnvidia-ptxjitcompiler.so) are only distributable in applications that meet this criteria: #1. The application was developed starting from a NVIDIA CUDA container obtained from Docker Hub or the NVIDIA GPU Cloud, and From 6c252a0bea807aa748b8cfc5dbaad52af3a9e11e Mon Sep 17 00:00:00 2001 From: Hariharan Seshadri Date: Thu, 20 May 2021 09:19:04 -0700 Subject: [PATCH 03/28] Fix builds that use gcc 5 (#7765) --- onnxruntime/core/framework/prepacked_weights_container.cc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/onnxruntime/core/framework/prepacked_weights_container.cc b/onnxruntime/core/framework/prepacked_weights_container.cc index cc6b45e00f..42cdd3debb 100644 --- a/onnxruntime/core/framework/prepacked_weights_container.cc +++ b/onnxruntime/core/framework/prepacked_weights_container.cc @@ -31,12 +31,12 @@ AllocatorPtr PrepackedWeightsContainer::GetOrCreateAllocator(const std::string& } const PrePackedWeights& PrepackedWeightsContainer::GetWeight(const std::string& key) const { - // .at() will throw if th key doesn't exist + // .at() will throw if the key doesn't exist return prepacked_weights_map_.at(key); } bool PrepackedWeightsContainer::WriteWeight(const std::string& key, PrePackedWeights&& packed_weight) { - auto ret = prepacked_weights_map_.insert({key, std::move(packed_weight)}); + auto ret = prepacked_weights_map_.insert(std::make_pair(key, std::move(packed_weight))); return ret.second; } From c91602070d1fd3c159240b62ea247686f41c9f0a Mon Sep 17 00:00:00 2001 From: Yulong Wang Date: Thu, 20 May 2021 09:52:34 -0700 Subject: [PATCH 04/28] [js] update version of package "onnxruntime-web" and "onnxruntime-react-native" (#7769) --- js/react_native/package.json | 2 +- js/web/package-lock.json | 2 +- js/web/package.json | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/js/react_native/package.json b/js/react_native/package.json index 4fd2b1de30..c98618f3c5 100644 --- a/js/react_native/package.json +++ b/js/react_native/package.json @@ -1,6 +1,6 @@ { "name": "onnxruntime-react-native", - "version": "1.7.0", + "version": "1.8.0", "description": "Onnxruntime bridge for react native", "main": "dist/commonjs/index", "module": "dist/module/index", diff --git a/js/web/package-lock.json b/js/web/package-lock.json index 92bb37b6c8..755aa33a89 100644 --- a/js/web/package-lock.json +++ b/js/web/package-lock.json @@ -1,6 +1,6 @@ { "name": "onnxruntime-web", - "version": "1.7.0", + "version": "1.8.0", "lockfileVersion": 1, "requires": true, "dependencies": { diff --git a/js/web/package.json b/js/web/package.json index 053d627305..ff2fa0e391 100644 --- a/js/web/package.json +++ b/js/web/package.json @@ -7,7 +7,7 @@ "type": "git" }, "author": "fs-eire", - "version": "1.7.0", + "version": "1.8.0", "keywords": [ "ONNX", "ONNXRuntime", From c2435d24ecbeededf1dc50187ab3bd11ad4a6994 Mon Sep 17 00:00:00 2001 From: Peng Date: Thu, 20 May 2021 12:04:49 -0500 Subject: [PATCH 05/28] Clean up ROCm4.1 Dockerfile build directory (#7732) * Clean up ROCm4.1 Dockerfile build directory * remove the UCX and OMPI build directories after installation --- orttraining/tools/amdgpu/Dockerfile.rocm4.1.pytorch | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/orttraining/tools/amdgpu/Dockerfile.rocm4.1.pytorch b/orttraining/tools/amdgpu/Dockerfile.rocm4.1.pytorch index 4920fb3e10..0a10760274 100644 --- a/orttraining/tools/amdgpu/Dockerfile.rocm4.1.pytorch +++ b/orttraining/tools/amdgpu/Dockerfile.rocm4.1.pytorch @@ -73,7 +73,9 @@ RUN git clone https://github.com/openucx/ucx.git \ && cd build \ && ../contrib/configure-opt --prefix=$UCX_DIR --without-rocm --without-knem --without-cuda \ && make -j"$(nproc)" \ - && make install + && make install \ + && cd .. \ + && rm -rf build # OpenMPI # note: require --enable-orterun-prefix-by-default for Azure machine learning compute @@ -93,6 +95,8 @@ RUN git clone --recursive https://github.com/open-mpi/ompi.git \ --enable-mca-no-build=btl-uct --disable-mpi-fortran \ && make -j"$(nproc)" \ && make install \ + && cd .. \ + && rm -rf build \ && ldconfig \ && test -f ${OPENMPI_DIR}/bin/mpic++ @@ -128,6 +132,7 @@ RUN git clone --recursive https://github.com/microsoft/onnxruntime.git \ --enable_training \ && test -f $ORT_DIR/build/RelWithDebInfo/onnxruntime_training_bert \ && pip install $ORT_DIR/build/RelWithDebInfo/dist/*.whl \ + && rm -rf $ORT_DIR/build \ && ldconfig # ONNX Runtime Training Examples From 2a028711571ba0086b3a264409a34d40056fc0b8 Mon Sep 17 00:00:00 2001 From: Sherlock Date: Thu, 20 May 2021 21:39:36 -0700 Subject: [PATCH 06/28] Disable reuse for YieldOp's inputs (FW partial graph's output) (#7767) * Disable reuse for YieldOp's input Co-authored-by: Sherlock Huang --- onnxruntime/core/framework/allocation_planner.cc | 14 ++++++++++++++ .../test/python/orttraining_test_ortmodule_api.py | 11 +++++------ 2 files changed, 19 insertions(+), 6 deletions(-) diff --git a/onnxruntime/core/framework/allocation_planner.cc b/onnxruntime/core/framework/allocation_planner.cc index 257db07fd8..4f6f21ba45 100644 --- a/onnxruntime/core/framework/allocation_planner.cc +++ b/onnxruntime/core/framework/allocation_planner.cc @@ -257,6 +257,20 @@ class PlannerImpl { // Find if there exists some input tensor that we can use in-place for output_arg_num-th input in the node. bool FindReusableInput(const onnxruntime::Node& node, int output_arg_num, OrtValueIndex* reusable_input) { +#ifdef ENABLE_TRAINING + // Inputs of Yields are essentially the outputs for FW partial subgraph + // Thses tensors will be pass back to pytorch, thus cannot share the buffer with other tensors + + // Unhandled corner case: + // If FW output tensor is consumed by BW graph, and pytorch performs an inplace operation on th returned tensor, + // we will run into a buffer corruption problem. + // One potential fix is returning a copy of output tensor, if it has downstream dependency + auto p_next_node = node.OutputNodesBegin(); + if (p_next_node != node.OutputNodesEnd() && p_next_node->OpType() == "YieldOp") { + return false; + } +#endif //ENABLE_TRAINING + auto p_output_arg = node.OutputDefs()[output_arg_num]; const KernelCreateInfo& ci = GetKernelCreateInfo(kernel_create_info_map_, node.Index()); diff --git a/orttraining/orttraining/test/python/orttraining_test_ortmodule_api.py b/orttraining/orttraining/test/python/orttraining_test_ortmodule_api.py index 20963c38b9..8ebb9c1c61 100644 --- a/orttraining/orttraining/test/python/orttraining_test_ortmodule_api.py +++ b/orttraining/orttraining/test/python/orttraining_test_ortmodule_api.py @@ -46,6 +46,8 @@ class NeuralNetMultiplePositionalArgumentsMultiOutputsWithoutDependency(torch.nn self.fc2 = torch.nn.Linear(input_size, hidden_size) self.softmax1 = torch.nn.Softmax(dim=1) self.softmax2 = torch.nn.Softmax(dim=1) + self.relu1 = torch.nn.ReLU() + self.relu2 = torch.nn.ReLU() def forward(self, input1, input2): model_input = input1 + input2 @@ -53,11 +55,8 @@ class NeuralNetMultiplePositionalArgumentsMultiOutputsWithoutDependency(torch.nn out2 = self.fc2(model_input) out1 = self.softmax1(out1) out2 = self.softmax2(out2) - # TODO: Using relu here will cause the forward prediction error - # ORT's Relu output is sharing the same buffer as input, - # and this buffer is returned as ORTModule's output to Pytorch - # out1 = self.relu1(out1) - # out2 = self.relu2(out2) + out1 = self.relu1(out1) + out2 = self.relu2(out2) return out1, out2 class NeuralNetMultiplePositionalArgumentsMultiOutputsWithDependency(torch.nn.Module): @@ -593,7 +592,7 @@ def test_gradient_correctness_conv1d(use_fp16, input_requires_grad): if use_fp16: _test_helpers.assert_values_are_close(ort_prediction, pt_prediction, atol=1e-3, rtol=1e-3) - _test_helpers.assert_gradients_match_and_reset_gradient(ort_model, pt_model, rtol=1e-2, atol=1e-2) + _test_helpers.assert_gradients_match_and_reset_gradient(ort_model, pt_model, rtol=1e-2, atol=1.1e-2) else: _test_helpers.assert_values_are_close(ort_prediction, pt_prediction, atol=1e-5) _test_helpers.assert_gradients_match_and_reset_gradient(ort_model, pt_model, rtol=5e-3, atol=4e-3) From 7c4a5faef57b2255544f6ca0f5b34cfb89792513 Mon Sep 17 00:00:00 2001 From: Yulong Wang Date: Fri, 21 May 2021 01:32:00 -0700 Subject: [PATCH 07/28] [wasm] enable DWARF format debug info for ORT WASM (#7777) * [wasm] enable DWARF format debug info for ORT WASM * resolve comments --- cmake/CMakeLists.txt | 13 +++++++------ tools/ci_build/build.py | 15 ++++++++------- 2 files changed, 15 insertions(+), 13 deletions(-) diff --git a/cmake/CMakeLists.txt b/cmake/CMakeLists.txt index f873019396..6f5db8cc40 100644 --- a/cmake/CMakeLists.txt +++ b/cmake/CMakeLists.txt @@ -151,7 +151,7 @@ option(onnxruntime_USE_MPI "Build with MPI support" OFF) option(onnxruntime_BUILD_WEBASSEMBLY "Enable this option to create WebAssembly byte codes" OFF) option(onnxruntime_ENABLE_WEBASSEMBLY_THREADS "Enable this option to create WebAssembly byte codes with multi-threads support" OFF) option(onnxruntime_ENABLE_WEBASSEMBLY_EXCEPTION_CATCHING "Enable this option to turn on exception catching" OFF) -option(onnxruntime_ENABLE_WEBASSEMBLY_SOURCEMAP "Enable this option to turn on sourcemap" OFF) +option(onnxruntime_ENABLE_WEBASSEMBLY_DEBUG_INFO "Enable this option to turn on DWARF format debug info" OFF) # Enable bitcode for iOS option(onnxruntime_ENABLE_BITCODE "Enable bitcode for iOS only" OFF) @@ -285,12 +285,13 @@ if (onnxruntime_BUILD_WEBASSEMBLY) string(APPEND CMAKE_CXX_FLAGS " -flto") endif() - if (onnxruntime_ENABLE_WEBASSEMBLY_SOURCEMAP) - # Build with sourcemap support - set(CMAKE_CXX_FLAGS_DEBUG "-g4 --source-map-base ${onnxruntime_WEBASSEMBLY_SOURCEMAP_BASE}") + if (onnxruntime_ENABLE_WEBASSEMBLY_DEBUG_INFO) + # "-g3" generates DWARF format debug info. + # NOTE: With debug info enabled, web assembly artifacts will be very huge (>1GB). So we offer an option to build without debug info. + set(CMAKE_CXX_FLAGS_DEBUG "-g3") else() - # override default "-g3" as it generates super huge (1GB+) WASM targets which fails to load - set(CMAKE_CXX_FLAGS_DEBUG "-g2") + # do not generate any debug info. This helps to accelerate building process and reduce binary size. + set(CMAKE_CXX_FLAGS_DEBUG "-g0") endif() # Build WebAssembly with multi-threads support. diff --git a/tools/ci_build/build.py b/tools/ci_build/build.py index e49502ec7f..f601a779e3 100644 --- a/tools/ci_build/build.py +++ b/tools/ci_build/build.py @@ -343,11 +343,8 @@ def parse_arguments(): "--enable_wasm_threads", action='store_true', help="Enable WebAssembly multi-threads support") parser.add_argument( - "--enable_wasm_sourcemap", action='store_true', - help="Build WebAssembly with source map") - parser.add_argument( - "--wasm_sourcemap_base", default="http://localhost:9876/onnxruntime/", - help="Set base URL of the source map") + "--enable_wasm_debug_info", action='store_true', + help="Build WebAssembly with DWARF format debug info") # Arguments needed by CI parser.add_argument( @@ -746,9 +743,8 @@ def generate_build_tree(cmake_path, source_dir, build_dir, cuda_home, cudnn_home "-Donnxruntime_ENABLE_WEBASSEMBLY_EXCEPTION_CATCHING=" + ("OFF" if args.disable_wasm_exception_catching else "ON"), "-Donnxruntime_ENABLE_WEBASSEMBLY_THREADS=" + ("ON" if args.enable_wasm_threads else "OFF"), + "-Donnxruntime_ENABLE_WEBASSEMBLY_DEBUG_INFO=" + ("ON" if args.enable_wasm_debug_info else "OFF"), "-Donnxruntime_ENABLE_EAGER_MODE=" + ("ON" if args.build_eager_mode else "OFF"), - "-Donnxruntime_ENABLE_WEBASSEMBLY_SOURCEMAP=" + ("ON" if args.enable_wasm_sourcemap else "OFF"), - "-Donnxruntime_WEBASSEMBLY_SOURCEMAP_BASE=" + (args.wasm_sourcemap_base if args.enable_wasm_sourcemap else ""), ] if acl_home and os.path.exists(acl_home): @@ -1870,6 +1866,11 @@ def main(): args.disable_wasm_exception_catching = True if args.test and args.disable_wasm_exception_catching and not args.minimal_build: raise BuildError("WebAssembly tests need exception catching enabled to run if it's not minimal build") + if args.test and args.enable_wasm_debug_info: + # With flag --enable_wasm_debug_info, onnxruntime_test_all.wasm will be very huge (>1GB). This will fail + # Node.js when trying to load the .wasm file. + # To debug ONNX Runtime WebAssembly, use ONNX Runtime Web to debug ort-wasm.wasm in browsers. + raise BuildError("WebAssembly tests cannot be enabled with flag --enable_wasm_debug_info") if args.code_coverage and not args.android: raise BuildError("Using --code_coverage requires --android") From db0d608ff02b6a89922f778c5c7596c3f48e8f73 Mon Sep 17 00:00:00 2001 From: ashbhandare Date: Fri, 21 May 2021 09:45:49 -0700 Subject: [PATCH 08/28] Fix build errors on Dev machines after PR #7626 merge (#7781) * two fixes * more Fixes * Disable mpi by default * Revert "Disable mpi by default" This reverts commit 46c774ad9c6fcb0f3c1a81cd08b7d5e0ba09a985. --- cmake/onnxruntime_framework.cmake | 3 +++ onnxruntime/core/framework/provider_bridge_ort.cc | 4 ++-- onnxruntime/core/providers/shared_library/provider_api.h | 1 + .../core/providers/shared_library/provider_interfaces.h | 2 +- 4 files changed, 7 insertions(+), 3 deletions(-) diff --git a/cmake/onnxruntime_framework.cmake b/cmake/onnxruntime_framework.cmake index 89919d0dd1..803a95501d 100644 --- a/cmake/onnxruntime_framework.cmake +++ b/cmake/onnxruntime_framework.cmake @@ -40,6 +40,9 @@ endif() # Needed for the provider interface, as it includes training headers when training is enabled if (onnxruntime_ENABLE_TRAINING OR onnxruntime_ENABLE_TRAINING_OPS) target_include_directories(onnxruntime_framework PRIVATE ${ORTTRAINING_ROOT}) + if (onnxruntime_USE_NCCL OR onnxruntime_USE_MPI) + target_include_directories(onnxruntime_framework PUBLIC ${MPI_CXX_INCLUDE_DIRS}) + endif() endif() onnxruntime_add_include_to_target(onnxruntime_framework onnxruntime_common onnx onnx_proto protobuf::libprotobuf flatbuffers) set_target_properties(onnxruntime_framework PROPERTIES FOLDER "ONNXRuntime") diff --git a/onnxruntime/core/framework/provider_bridge_ort.cc b/onnxruntime/core/framework/provider_bridge_ort.cc index e462064477..554b3eea37 100644 --- a/onnxruntime/core/framework/provider_bridge_ort.cc +++ b/onnxruntime/core/framework/provider_bridge_ort.cc @@ -63,7 +63,7 @@ Status LongformerAttentionBase__CheckInputs(const LongformerAttentionBase* p, co #include "orttraining/training_ops/cpu/loss/softmax_cross_entropy_loss.h" #include "orttraining/training_ops/cpu/tensor/split.h" #endif -#if defined(USE_CUDA) && defined(ORT_USE_NCCL) && defined(USE_NCCL_P2P) +#if defined(USE_CUDA) && defined(ORT_USE_NCCL) #include "orttraining/training_ops/cuda/communication/nccl_service.h" #include "orttraining/core/framework/distributed_run_context.h" #endif @@ -883,7 +883,7 @@ struct ProviderHostImpl : ProviderHost { Status contrib__PrepareForTrainingCompute(const TensorShape& input_shape, int num_outputs, int64_t& axis, int& before_dims, int& after_dims_including_split_axis, int& after_dims_excluding_split, std::vector& split_sizes) override { return contrib::PrepareForTrainingCompute(input_shape, num_outputs, axis, before_dims, after_dims_including_split_axis, after_dims_excluding_split, split_sizes); } Status contrib__YieldOp__Compute(const contrib::YieldOp* p, OpKernelContext* context) override { return p->YieldOp::Compute(context); } -#if defined(ORT_USE_NCCL) && defined(USE_NCCL_P2P) +#if defined(ORT_USE_NCCL) training::DistributedRunContext& GetDistributedRunContextInstance() override { return training::DistributedRunContext::GetInstance(); } #endif #endif diff --git a/onnxruntime/core/providers/shared_library/provider_api.h b/onnxruntime/core/providers/shared_library/provider_api.h index 3c58150e9f..aa3304718f 100644 --- a/onnxruntime/core/providers/shared_library/provider_api.h +++ b/onnxruntime/core/providers/shared_library/provider_api.h @@ -40,6 +40,7 @@ struct DeleteOnUnloadPtr { #include #include #include +#include #include "onnx/common/stl_backports.h" #include "core/common/common.h" #include "core/common/const_pointer_container.h" diff --git a/onnxruntime/core/providers/shared_library/provider_interfaces.h b/onnxruntime/core/providers/shared_library/provider_interfaces.h index d2ad57e925..7261b5f1ea 100644 --- a/onnxruntime/core/providers/shared_library/provider_interfaces.h +++ b/onnxruntime/core/providers/shared_library/provider_interfaces.h @@ -791,7 +791,7 @@ struct ProviderHost { virtual Status contrib__PrepareForTrainingCompute(const TensorShape& input_shape, int num_outputs, int64_t& axis, int& before_dims, int& after_dims_including_split_axis, int& after_dims_excluding_split, std::vector& split_sizes) = 0; virtual Status contrib__YieldOp__Compute(const contrib::YieldOp* p, OpKernelContext* context) = 0; -#if defined(ORT_USE_NCCL) && defined(USE_NCCL_P2P) +#if defined(ORT_USE_NCCL) virtual training::DistributedRunContext& GetDistributedRunContextInstance() = 0; #endif #endif From a6ca9f0a402636bb25ce47d5349ac6c489ad3888 Mon Sep 17 00:00:00 2001 From: baijumeswani Date: Fri, 21 May 2021 10:28:09 -0700 Subject: [PATCH 09/28] Use list comprehensions instead of list appends where possible (#7753) * Use list comprehensions instead of list appends where possible * Add OrtValueVector class as an opaque object in pybind * Add dlpack methods to the OrtValueVector pybind class --- .../python/onnxruntime_pybind_state.cc | 24 +---------- .../python/onnxruntime_pybind_state_common.cc | 30 ++++++++++++++ .../python/onnxruntime_pybind_state_common.h | 16 ++++++++ .../python/orttraining_pybind_state.cc | 22 +++++++++- .../python/training/ortmodule/_io.py | 7 ++-- .../training/ortmodule/_training_manager.py | 41 +++++++++++-------- .../python/training/ortmodule/_utils.py | 5 +++ 7 files changed, 101 insertions(+), 44 deletions(-) diff --git a/onnxruntime/python/onnxruntime_pybind_state.cc b/onnxruntime/python/onnxruntime_pybind_state.cc index 233be1504d..74bf596d48 100644 --- a/onnxruntime/python/onnxruntime_pybind_state.cc +++ b/onnxruntime/python/onnxruntime_pybind_state.cc @@ -1336,20 +1336,6 @@ void addOpSchemaSubmodule(py::module& m) { #endif //onnxruntime_PYBIND_EXPORT_OPSCHEMA -#ifdef ENABLE_TRAINING -void DlpackCapsuleDestructor(PyObject* data) { - DLManagedTensor* dlmanged_tensor = (DLManagedTensor*)PyCapsule_GetPointer(data, "dltensor"); - if (dlmanged_tensor) { - // the dlmanged_tensor has not been consumed, call deleter ourselves. - dlmanged_tensor->deleter(const_cast(dlmanged_tensor)); - } else { - // the dlmanged_tensor has been consumed, - // PyCapsule_GetPointer has set an error indicator. - PyErr_Clear(); - } -} -#endif - void addObjectMethods(py::module& m, Environment& env) { py::enum_(m, "GraphOptimizationLevel") .value("ORT_DISABLE_ALL", GraphOptimizationLevel::ORT_DISABLE_ALL) @@ -1577,16 +1563,10 @@ void addObjectMethods(py::module& m, Environment& env) { }) #ifdef ENABLE_TRAINING .def("to_dlpack", [](OrtValue* ort_value) -> py::object { - DLManagedTensor* dlmanaged_tensor = dlpack::OrtValueToDlpack(*ort_value); - return py::reinterpret_steal( - PyCapsule_New(dlmanaged_tensor, "dltensor", DlpackCapsuleDestructor)); + return ToDlpack(*ort_value); }) .def_static("from_dlpack", [](py::object data, bool is_bool_tensor = false) { - DLManagedTensor* dlmanaged_tensor = (DLManagedTensor*)PyCapsule_GetPointer(data.ptr(), "dltensor"); - OrtValue ort_value = dlpack::DlpackToOrtValue(dlmanaged_tensor, is_bool_tensor); - // Make sure this capsule will never be used again. - PyCapsule_SetName(data.ptr(), "used_dltensor"); - return ort_value; + return FromDlpack(data, is_bool_tensor); }) #endif ; diff --git a/onnxruntime/python/onnxruntime_pybind_state_common.cc b/onnxruntime/python/onnxruntime_pybind_state_common.cc index 0f85dbef15..78f80aea72 100644 --- a/onnxruntime/python/onnxruntime_pybind_state_common.cc +++ b/onnxruntime/python/onnxruntime_pybind_state_common.cc @@ -23,5 +23,35 @@ void ThrowIfPyErrOccured() { } } +#ifdef ENABLE_TRAINING + +void DlpackCapsuleDestructor(PyObject* data) { + DLManagedTensor* dlmanged_tensor = (DLManagedTensor*)PyCapsule_GetPointer(data, "dltensor"); + if (dlmanged_tensor) { + // the dlmanged_tensor has not been consumed, call deleter ourselves. + dlmanged_tensor->deleter(const_cast(dlmanged_tensor)); + } else { + // the dlmanged_tensor has been consumed, + // PyCapsule_GetPointer has set an error indicator. + PyErr_Clear(); + } +} + +py::object ToDlpack(OrtValue& ort_value) { + DLManagedTensor* dlmanaged_tensor = dlpack::OrtValueToDlpack(ort_value); + return py::reinterpret_steal( + PyCapsule_New(dlmanaged_tensor, "dltensor", DlpackCapsuleDestructor)); +} + +OrtValue FromDlpack(py::object dlpack_tensor, const bool is_bool_tensor) { + DLManagedTensor* dlmanaged_tensor = (DLManagedTensor*)PyCapsule_GetPointer(dlpack_tensor.ptr(), "dltensor"); + OrtValue ort_value = dlpack::DlpackToOrtValue(dlmanaged_tensor, is_bool_tensor); + // Make sure this capsule will never be used again. + PyCapsule_SetName(dlpack_tensor.ptr(), "used_dltensor"); + return ort_value; +} + +#endif + } // namespace python } // namespace onnxruntime diff --git a/onnxruntime/python/onnxruntime_pybind_state_common.h b/onnxruntime/python/onnxruntime_pybind_state_common.h index a9e0a22792..72100bd780 100644 --- a/onnxruntime/python/onnxruntime_pybind_state_common.h +++ b/onnxruntime/python/onnxruntime_pybind_state_common.h @@ -8,6 +8,10 @@ #include "core/session/environment.h" #include "core/session/inference_session.h" +#ifdef ENABLE_TRAINING +#include "core/dlpack/dlpack_converter.h" +#endif + namespace onnxruntime { namespace python { @@ -135,5 +139,17 @@ void InitializeSession(InferenceSession* sess, // Checks if PyErrOccured, fetches status and throws. void ThrowIfPyErrOccured(); +#ifdef ENABLE_TRAINING + +namespace py = pybind11; + +void DlpackCapsuleDestructor(PyObject* data); + +py::object ToDlpack(OrtValue& ort_value); + +OrtValue FromDlpack(py::object dlpack_tensor, const bool is_bool_tensor); + +#endif + } // namespace python } // namespace onnxruntime diff --git a/orttraining/orttraining/python/orttraining_pybind_state.cc b/orttraining/orttraining/python/orttraining_pybind_state.cc index cebbbd5ab7..c28a4aada8 100644 --- a/orttraining/orttraining/python/orttraining_pybind_state.cc +++ b/orttraining/orttraining/python/orttraining_pybind_state.cc @@ -8,6 +8,7 @@ #include #include +#include "core/dlpack/dlpack_converter.h" #include "core/session/environment.h" #include "orttraining/core/session/training_session.h" #include "orttraining/core/agent/training_agent.h" @@ -304,7 +305,26 @@ std::unordered_map> Con } void addObjectMethodsForTraining(py::module& m) { - py::bind_vector>(m, "OrtValueVector"); + py::class_>(m, "OrtValueVector") + .def(py::init<>()) + .def("push_back", [](std::vector* v, const OrtValue& ortvalue) { + v->push_back(ortvalue); + }) + .def("push_back", [](std::vector* v, py::object dlpack_tensor, const bool is_bool_tensor) { + v->push_back(FromDlpack(dlpack_tensor, is_bool_tensor)); + }) + .def("reserve", [](std::vector* v, const size_t len) { v->reserve(len); }) + .def("shrink_to_fit", [](std::vector* v) { v->shrink_to_fit(); }) + .def("__len__", [](const std::vector &v) { return v.size(); }) + .def("__iter__", [](const std::vector &v) { + return py::make_iterator(v.cbegin(), v.cend()); + }, py::keep_alive<0, 1>()) + .def("__getitem__", [](const std::vector &v, const size_t idx) { + return v.at(idx); + }) + .def("dlpack_at", [](std::vector* v, const size_t idx) { + return ToDlpack(v->at(idx)); + }); py::class_ parameters(m, "TrainingParameters", R"pbdoc(Configuration information for training.)pbdoc"); parameters.def(py::init()) diff --git a/orttraining/orttraining/python/training/ortmodule/_io.py b/orttraining/orttraining/python/training/ortmodule/_io.py index 1b88c3996d..b7cb3e79a8 100644 --- a/orttraining/orttraining/python/training/ortmodule/_io.py +++ b/orttraining/orttraining/python/training/ortmodule/_io.py @@ -175,9 +175,8 @@ def _combine_input_buffers_initializers(params, onnx_input_names, input_info, bu def deepcopy_model_input(*inputs, **kwargs): - sample_inputs_copy = [] - for model_input in inputs: - sample_inputs_copy.append(model_input.data if isinstance(model_input, torch.Tensor) else model_input) + sample_inputs_copy = [model_input.data if isinstance(model_input, torch.Tensor) else model_input + for model_input in inputs] sample_inputs_copy = copy.deepcopy(tuple(sample_inputs_copy)) sample_kwargs_copy = {} @@ -406,7 +405,7 @@ def parse_inputs_for_onnx_export(all_input_parameters, onnx_graph, inputs, kwarg # ONNX exporter may remove unused inputs onnx_graph_input_names = [] if onnx_graph is not None: - onnx_graph_input_names = set([inp.name for inp in onnx_graph.graph.input]) + onnx_graph_input_names = {inp.name for inp in onnx_graph.graph.input} input_names = [] dynamic_axes = {} diff --git a/orttraining/orttraining/python/training/ortmodule/_training_manager.py b/orttraining/orttraining/python/training/ortmodule/_training_manager.py index 21faf6d6ea..97d9802467 100644 --- a/orttraining/orttraining/python/training/ortmodule/_training_manager.py +++ b/orttraining/orttraining/python/training/ortmodule/_training_manager.py @@ -12,6 +12,7 @@ from onnxruntime.capi.onnxruntime_inference_collection import get_ort_device_typ import onnx import torch +from torch.utils.dlpack import from_dlpack, to_dlpack class TrainingManager(GraphExecutionManager): @@ -37,8 +38,9 @@ class TrainingManager(GraphExecutionManager): # have the need for passing IOBinding. state = C.PartialGraphExecutionState() forward_inputs = C.OrtValueVector() + forward_inputs.reserve(len(inputs)) for input in inputs: - forward_inputs.append(_utils._ortvalue_from_torch_tensor(input)) + forward_inputs.push_back(to_dlpack(input), input.dtype == torch.bool) forward_outputs = C.OrtValueVector() # Run and return module outputs. @@ -127,7 +129,9 @@ class TrainingManager(GraphExecutionManager): # Use IO binding # Push user output grads to ONNX backend. - contiguous_grad_outputs = [] + backward_inputs = C.OrtValueVector() + # Preallocate length of the vector. And then delete as required towards the end. + backward_inputs.reserve(len(grad_outputs)) for idx, grad_output in enumerate(grad_outputs): if idx in self._graph_info.output_grad_indices_non_differentiable: assert grad_output is None, "ORT found the {}-th module output '{}' is " \ @@ -145,13 +149,10 @@ class TrainingManager(GraphExecutionManager): grad_output = torch.tensor(0., device=device, dtype=dtype) elif not grad_output.is_contiguous(): grad_output = grad_output.contiguous() - contiguous_grad_outputs.append(grad_output) + backward_inputs.push_back(to_dlpack(grad_output), grad_output.dtype == torch.bool) + backward_inputs.shrink_to_fit() # Run and get results - backward_inputs = C.OrtValueVector() - for input in contiguous_grad_outputs: - backward_inputs.append(_utils._ortvalue_from_torch_tensor(input)) - backward_outputs = C.OrtValueVector() self._execution_agent.run_backward(backward_inputs, backward_outputs, ctx.run_info.state) # Destroy the state immediately (as opposed to be at the mercy of garbage collector) so it does not @@ -165,7 +166,9 @@ class TrainingManager(GraphExecutionManager): for input_name in self._graph_info.user_input_names: # Append to the results the backward output for each input that required grad if input_name in require_grad_names_set: - results.append(_utils._ortvalue_to_torch_tensor(backward_outputs[require_grad_names_index])) + results.append(_utils._torch_tensor_from_dl_pack( + backward_outputs.dlpack_at(require_grad_names_index), + backward_outputs[require_grad_names_index])) require_grad_names_index += 1 else: # input_name is not found in the self._input_info.require_grad_names list @@ -177,7 +180,9 @@ class TrainingManager(GraphExecutionManager): initializer_index = num_user_input_grads for initializer_name in self._graph_info.initializer_names: if initializer_name in self._graph_initializer_names_to_train: - results.append(_utils._ortvalue_to_torch_tensor(backward_outputs[initializer_index])) + results.append(_utils._torch_tensor_from_dl_pack( + backward_outputs.dlpack_at(initializer_index), + backward_outputs[initializer_index])) initializer_index += 1 else: results.append(None) @@ -211,16 +216,18 @@ class TrainingManager(GraphExecutionManager): session_options, providers, provider_options = self._get_session_config() fw_feed_names = [input.name for input in self._optimized_onnx_model.graph.input] - fw_outputs_device_info = [] - for idx in range(len(self._graph_info.user_output_names)): - fw_outputs_device_info.append(C.OrtDevice(get_ort_device_type(self._device.type), - C.OrtDevice.default_memory(), _utils.get_device_index(self._device))) + fw_outputs_device_info = [ + C.OrtDevice(get_ort_device_type(self._device.type), + C.OrtDevice.default_memory(), + _utils.get_device_index(self._device) + )] * len(self._graph_info.user_output_names) bw_fetches_names = [output.name for output in self._optimized_onnx_model.graph.output] - bw_outputs_device_info = [] - for idx in range(len(bw_fetches_names)): - bw_outputs_device_info.append(C.OrtDevice(get_ort_device_type(self._device.type), - C.OrtDevice.default_memory(), _utils.get_device_index(self._device))) + bw_outputs_device_info = [ + C.OrtDevice(get_ort_device_type(self._device.type), + C.OrtDevice.default_memory(), + _utils.get_device_index(self._device) + )] * len(bw_fetches_names) self._execution_agent = TrainingAgent(self._optimized_onnx_model.SerializeToString(), fw_feed_names, diff --git a/orttraining/orttraining/python/training/ortmodule/_utils.py b/orttraining/orttraining/python/training/ortmodule/_utils.py index 6c28942069..98d553bcac 100644 --- a/orttraining/orttraining/python/training/ortmodule/_utils.py +++ b/orttraining/orttraining/python/training/ortmodule/_utils.py @@ -22,6 +22,11 @@ def _ortvalue_from_torch_tensor(torch_tensor): return C.OrtValue.from_dlpack(to_dlpack(torch_tensor), torch_tensor.dtype == torch.bool) +def _torch_tensor_from_dl_pack(dlpack, ortvalue): + torch_tensor = from_dlpack(dlpack) + return torch_tensor.to(torch.bool) if ortvalue.data_type() == 'tensor(bool)' else torch_tensor + + def _check_same_device(device, argument_str, *args): '''Check that all tensor arguments in *args reside on the same device as the input device''' From b852b73e84ff1f2d4e0462e682aac026b492b2e0 Mon Sep 17 00:00:00 2001 From: Yulong Wang Date: Fri, 21 May 2021 11:42:36 -0700 Subject: [PATCH 10/28] [js][doc] update some part of documents. (#7768) * [js][doc] update some part of documents. * resolve comments --- js/README.md | 2 +- js/node/README.md | 2 +- js/web/README.md | 96 ++--------------------------------------------- 3 files changed, 6 insertions(+), 94 deletions(-) diff --git a/js/README.md b/js/README.md index 488b025cc6..cbc7fb588f 100644 --- a/js/README.md +++ b/js/README.md @@ -138,7 +138,7 @@ Node.js v12+ (recommended v14+) ./build.sh --config Release --build_wasm --skip_tests --disable_wasm_exception_catching --disable_rtti ``` - To build with multi-thread support, append flag ` --enable_wasm_threads` to the command. + To build with multi-thread support, append flag ` --enable_wasm_threads` to the command. Make sure to build both single-thread and multi-thread before next step. 3. Copy following files from build output folder to `/js/web/dist/`: diff --git a/js/node/README.md b/js/node/README.md index c5bec8043f..d2f616696c 100644 --- a/js/node/README.md +++ b/js/node/README.md @@ -10,7 +10,7 @@ Install the latest stable version: npm install onnxruntime-node ``` -Refer to [Node.js samples](../../samples/nodejs/README.md) for samples and tutorials. +Refer to [ONNX Runtime JavaScript examples](https://github.com/microsoft/onnxruntime-inference-examples/tree/main/js) for samples and tutorials. ## Requirements diff --git a/js/web/README.md b/js/web/README.md index 40d8fab5df..a59c84db11 100644 --- a/js/web/README.md +++ b/js/web/README.md @@ -16,105 +16,17 @@ ONNX Runtime Web can run on both CPU and GPU. For running on CPU, [WebAssembly]( See [Compatibility](#Compatibility) and [Operators Supported](#Operators) for a list of platforms and operators ONNX Runtime Web currently supports. -## Getting Started +## Usage -There are multiple ways to use ONNX Runtime Web in a project: - -### Using ` - - - - -``` - - - -### Using NPM and bundling tools - -Modern browser based applications are usually built by frameworks like [Angular](https://angular.io/), [React](https://reactjs.org/), [Vue.js](https://vuejs.org/) and so on. This solution usually builds the source code into one or more bundle file(s). The following TypeScript example shows how to use ONNX Runtime Web in an async context: - -1. Import `Tensor` and `InferenceSession`. - -```ts -import { Tensor, InferenceSession } from "onnxruntime-web"; -``` - -2. Create an instance of `InferenceSession` and load ONNX model. - -```ts -// use the following in an async method -const url = "./data/models/resnet/model.onnx"; -const session = await InferenceSession.create(url); -``` - -3. Create your input Tensor(s) similar to the example below. You need to do any pre-processing required by - your model at this stage. For that refer to the documentation of the model you have: - -```javascript -// creating an array of input Tensors is the easiest way. For other options see the API documentation -const input0 = new Tensor(new Float32Array([1.0, 2.0, 3.0, 4.0]), [2, 2]); -``` - -4. Run the model with the input Tensors. The output Tensor(s) are available once the run operation is complete: - -```javascript -// run this in an async method: -// assume model's input name is 'input_0' and output name is 'output_0' -const outputs = await session.run({ input_0: input0 }); -const outputTensor = outputs.output_0; -``` - -5. Bundle your code. All web application frameworks offer bundling tools and instructions. Specifically, you can specify onnxruntime-web as an external dependency: - -```js - // a webpack example - externals: { - 'onnxruntime-web': 'ort', // add this line in your webpack.config.js - // ... - } -``` - -so that you can consume the file `ort.min.js` from a CDN provider demonstrated as above. - - +Refer to [ONNX Runtime JavaScript examples](https://github.com/microsoft/onnxruntime-inference-examples/tree/main/js) for samples and tutorials. ## Documents ### Developers - +Refer to [Using VSCode](../README.md#Using-VSCode) for setting up development environment. -For information on ONNX.js development, please check [Development](./docs/development.md) - -For API reference, please check [API](./docs/api.md). +For information about building ONNX Runtime Web development, please check [Build](../README.md#build-2). ### Getting ONNX models From 1fbc04d691142370af3ece415146ccf424ef361f Mon Sep 17 00:00:00 2001 From: Sunghoon <35605090+hanbitmyths@users.noreply.github.com> Date: Fri, 21 May 2021 13:06:14 -0700 Subject: [PATCH 11/28] Enable training ops in inference (#7783) * Enable training ops in inference * fix a build error * relu test name is the same as trainig test --- cmake/onnxruntime_graph.cmake | 12 +++- cmake/onnxruntime_session.cmake | 2 +- onnxruntime/core/session/environment.cc | 15 +++-- .../cpu/activation/activation_op_test.cc | 67 +++++++++++++++++++ 4 files changed, 87 insertions(+), 9 deletions(-) diff --git a/cmake/onnxruntime_graph.cmake b/cmake/onnxruntime_graph.cmake index 25ee83d15f..0e4a7e004d 100644 --- a/cmake/onnxruntime_graph.cmake +++ b/cmake/onnxruntime_graph.cmake @@ -60,6 +60,12 @@ file(GLOB_RECURSE onnxruntime_ir_defs_src CONFIGURE_DEPENDS "${ONNXRUNTIME_ROOT}/core/defs/*.cc" ) +if (onnxruntime_ENABLE_TRAINING_OPS AND NOT onnxruntime_ENABLE_TRAINING) + set(orttraining_graph_src + "${ORTTRAINING_SOURCE_DIR}/core/graph/training_op_defs.cc" + "${ORTTRAINING_SOURCE_DIR}/core/graph/training_op_defs.h" + ) +endif() if (onnxruntime_ENABLE_TRAINING) file(GLOB_RECURSE orttraining_graph_src CONFIGURE_DEPENDS "${ORTTRAINING_SOURCE_DIR}/core/graph/*.h" @@ -68,7 +74,7 @@ if (onnxruntime_ENABLE_TRAINING) endif() set(onnxruntime_graph_lib_src ${onnxruntime_graph_src} ${onnxruntime_ir_defs_src}) -if (onnxruntime_ENABLE_TRAINING) +if (onnxruntime_ENABLE_TRAINING OR onnxruntime_ENABLE_TRAINING_OPS) list(APPEND onnxruntime_graph_lib_src ${orttraining_graph_src}) endif() @@ -83,7 +89,7 @@ endif() target_include_directories(onnxruntime_graph PRIVATE ${ONNXRUNTIME_ROOT}) -if (onnxruntime_ENABLE_TRAINING) +if (onnxruntime_ENABLE_TRAINING OR onnxruntime_ENABLE_TRAINING_OPS) target_include_directories(onnxruntime_graph PRIVATE ${ORTTRAINING_ROOT}) if (onnxruntime_USE_NCCL) @@ -95,7 +101,7 @@ set_target_properties(onnxruntime_graph PROPERTIES FOLDER "ONNXRuntime") set_target_properties(onnxruntime_graph PROPERTIES LINKER_LANGUAGE CXX) install(DIRECTORY ${PROJECT_SOURCE_DIR}/../include/onnxruntime/core/graph DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/onnxruntime/core) source_group(TREE ${REPO_ROOT} FILES ${onnxruntime_graph_src} ${onnxruntime_ir_defs_src}) -if (onnxruntime_ENABLE_TRAINING) +if (onnxruntime_ENABLE_TRAINING OR onnxruntime_ENABLE_TRAINING_OPS) source_group(TREE ${ORTTRAINING_ROOT} FILES ${orttraining_graph_src}) endif() diff --git a/cmake/onnxruntime_session.cmake b/cmake/onnxruntime_session.cmake index 564dc53dfe..4850d48c0a 100644 --- a/cmake/onnxruntime_session.cmake +++ b/cmake/onnxruntime_session.cmake @@ -22,6 +22,6 @@ set_target_properties(onnxruntime_session PROPERTIES FOLDER "ONNXRuntime") if (onnxruntime_USE_CUDA) target_include_directories(onnxruntime_session PRIVATE ${onnxruntime_CUDNN_HOME}/include ${CMAKE_CUDA_TOOLKIT_INCLUDE_DIRECTORIES}) endif() -if (onnxruntime_ENABLE_TRAINING) +if (onnxruntime_ENABLE_TRAINING OR onnxruntime_ENABLE_TRAINING_OPS) target_include_directories(onnxruntime_session PRIVATE ${ORTTRAINING_ROOT}) endif() diff --git a/onnxruntime/core/session/environment.cc b/onnxruntime/core/session/environment.cc index 3bd28d630a..07636bbc15 100644 --- a/onnxruntime/core/session/environment.cc +++ b/onnxruntime/core/session/environment.cc @@ -8,7 +8,7 @@ #if !defined(ORT_MINIMAL_BUILD) #include "onnx/defs/operator_sets.h" #include "onnx/defs/operator_sets_ml.h" -#if defined(ENABLE_TRAINING) +#if defined(ENABLE_TRAINING) || defined(ENABLE_TRAINING_OPS) #include "onnx/defs/operator_sets_training.h" #endif #endif @@ -30,8 +30,10 @@ #include "core/platform/tracing.h" #endif -#ifdef ENABLE_TRAINING +#if defined(ENABLE_TRAINING) || defined(ENABLE_TRAINING_OPS) #include "orttraining/core/graph/training_op_defs.h" +#endif +#ifdef ENABLE_TRAINING #include "orttraining/core/graph/gradient_builder_registry.h" #include "orttraining/core/graph/loss_function_registry.h" #include "orttraining/core/graph/optimizer_builder.h" @@ -180,17 +182,20 @@ Status Environment::Initialize(std::unique_ptr logging_ RegisterOnnxMLOperatorSetSchema(); #endif -#ifdef ENABLE_TRAINING +#if defined(ENABLE_TRAINING) || defined(ENABLE_TRAINING_OPS) RegisterOnnxTrainingOperatorSetSchema(); #endif -#ifdef ENABLE_TRAINING - // preserve this order: this depends on operatorsetschema registration. +#if defined(ENABLE_TRAINING) || defined(ENABLE_TRAINING_OPS) + // preserve this order until : this depends on operatorsetschema registration. training::RegisterTrainingOpSchemas(); +#endif +#ifdef ENABLE_TRAINING training::GradientBuilderRegistry::GetInstance().RegisterGradientBuilders(); training::LossFunctionRegistry::GetInstance().RegisterNonOperatorLossFunctions(); training::OptimizerBuilderRegistry::GetInstance().RegisterBuilders(); training::OptimizerGraphBuilderRegistry::GetInstance().RegisterGraphBuilders(); + // #endif }); diff --git a/onnxruntime/test/providers/cpu/activation/activation_op_test.cc b/onnxruntime/test/providers/cpu/activation/activation_op_test.cc index 104e62e62e..d49726619b 100644 --- a/onnxruntime/test/providers/cpu/activation/activation_op_test.cc +++ b/onnxruntime/test/providers/cpu/activation/activation_op_test.cc @@ -7,6 +7,55 @@ namespace onnxruntime { namespace test { +#if defined(ENABLE_TRAINING_OPS) +namespace { +void TestElementwiseGradientOp( + const char* op, + const std::vector>>& inputs, + std::function&)> expected_func, + const std::unordered_map attrs = {}, + int opset_version = 7, const char* domain = kOnnxDomain) { + const auto first_input = inputs.begin(); + ASSERT_NE(first_input, inputs.end()); + for (auto input = first_input; input != inputs.end(); ++input) { + if (input == first_input) continue; + ASSERT_EQ(first_input->second.size(), input->second.size()); + } + + OpTester test(op, opset_version, domain); + + for (auto attr : attrs) { + test.AddAttribute(attr.first, attr.second); + } + + const auto input_size = first_input->second.size(); + std::vector dims{static_cast(input_size)}; + + std::vector expected_vals; + for (size_t i = 0; i < input_size; i++) { + std::vector params(inputs.size()); + std::transform( + inputs.begin(), inputs.end(), params.begin(), + [i](const std::pair>& input) { + return input.second[i]; + }); + expected_vals.push_back(expected_func(params)); + } + + for (const auto& input : inputs) { + test.AddInput(input.first.c_str(), dims, input.second); + } + test.AddOutput("dX", dims, expected_vals); + + test.Run(OpTester::ExpectResult::kExpectSuccess, "", {}); +} + +float ReluGrad(float dy, float x) { + return x > 0 ? dy : 0; +} +} +#endif + TEST_F(ActivationOpTest, Sigmoid) { TestActivationOp("Sigmoid", input_values, @@ -210,5 +259,23 @@ TEST_F(ActivationOpNoInfTest, Softsign) { {}, false); // Disable TensorRT because result mismatches } +#if defined(ENABLE_TRAINING_OPS) +TEST(ReluGradInferenceTest, Basic) { + const std::vector x_vals = {-1.0f, 0, 1.0f, 100.0f, -100.0f, 1000.0f, -1000.0f}; + const std::vector dY(7, 1.0f); + + TestElementwiseGradientOp( + "ReluGrad", + {{"dY", dY}, {"X", x_vals}}, + [](const std::vector& params) { + ORT_ENFORCE(params.size() == 2); + const auto dy = params[0], x = params[1]; + + return ReluGrad(dy, x); + }, + {}, 1, kMSDomain); +} +#endif + } // namespace test } // namespace onnxruntime From b5c5e8c1cadf078a1fa1f163589ae4f3aded19f5 Mon Sep 17 00:00:00 2001 From: Edward Chen <18449977+edgchen1@users.noreply.github.com> Date: Fri, 21 May 2021 13:12:13 -0700 Subject: [PATCH 12/28] Update C++ API comment to resolve warning. (#7776) --- include/onnxruntime/core/session/onnxruntime_cxx_api.h | 1 - 1 file changed, 1 deletion(-) diff --git a/include/onnxruntime/core/session/onnxruntime_cxx_api.h b/include/onnxruntime/core/session/onnxruntime_cxx_api.h index ffd4a9d6c1..4bd07b47c6 100644 --- a/include/onnxruntime/core/session/onnxruntime_cxx_api.h +++ b/include/onnxruntime/core/session/onnxruntime_cxx_api.h @@ -569,7 +569,6 @@ struct ArenaCfg : Base { * \param arena_extend_strategy - use -1 to allow ORT to choose the default, 0 = kNextPowerOfTwo, 1 = kSameAsRequested * \param initial_chunk_size_bytes - use -1 to allow ORT to choose the default * \param max_dead_bytes_per_chunk - use -1 to allow ORT to choose the default - * \return an instance of ArenaCfg * See docs/C_API.md for details on what the following parameters mean and how to choose these values */ ArenaCfg(size_t max_mem, int arena_extend_strategy, int initial_chunk_size_bytes, int max_dead_bytes_per_chunk); From ee29330cabf6075f190776d35d70096dde39905f Mon Sep 17 00:00:00 2001 From: Changming Sun Date: Fri, 21 May 2021 17:05:35 -0700 Subject: [PATCH 13/28] Delete unused file: Dockerfile.ubuntu_gpu (#7797) --- .../github/linux/docker/Dockerfile.ubuntu_gpu | 29 ------------------- .../ci_build/github/linux/run_dockerbuild.sh | 14 +++------ 2 files changed, 4 insertions(+), 39 deletions(-) delete mode 100644 tools/ci_build/github/linux/docker/Dockerfile.ubuntu_gpu diff --git a/tools/ci_build/github/linux/docker/Dockerfile.ubuntu_gpu b/tools/ci_build/github/linux/docker/Dockerfile.ubuntu_gpu deleted file mode 100644 index 8f748bba31..0000000000 --- a/tools/ci_build/github/linux/docker/Dockerfile.ubuntu_gpu +++ /dev/null @@ -1,29 +0,0 @@ -FROM nvidia/cuda:10.1-cudnn7-devel-ubuntu18.04 - -ARG PYTHON_VERSION=3.6 -ARG INSTALL_DEPS_EXTRA_ARGS - -ADD scripts /tmp/scripts -RUN /tmp/scripts/install_ubuntu.sh -p $PYTHON_VERSION && \ - /tmp/scripts/install_os_deps.sh -d gpu $INSTALL_DEPS_EXTRA_ARGS && \ - /tmp/scripts/install_python_deps.sh -p $PYTHON_VERSION -d gpu $INSTALL_DEPS_EXTRA_ARGS && \ - rm -rf /tmp/scripts - -WORKDIR /root - -# Allow configure to pick up GDK and CuDNN where it expects it. -# (Note: $CUDNN_VERSION is defined by NVidia's base image) -RUN _CUDNN_VERSION=$(echo $CUDNN_VERSION | cut -d. -f1-2) && \ - mkdir -p /usr/local/cudnn-$_CUDNN_VERSION/cuda/include && \ - ln -s /usr/include/cudnn.h /usr/local/cudnn-$_CUDNN_VERSION/cuda/include/cudnn.h && \ - mkdir -p /usr/local/cudnn-$_CUDNN_VERSION/cuda/lib64 && \ - ln -s /etc/alternatives/libcudnn_so /usr/local/cudnn-$_CUDNN_VERSION/cuda/lib64/libcudnn.so && \ - ln -s /usr/local/cudnn{-$_CUDNN_VERSION,} - -ENV LD_LIBRARY_PATH /usr/local/openblas/lib:$LD_LIBRARY_PATH - -ARG BUILD_USER=onnxruntimedev -ARG BUILD_UID=1000 -RUN adduser --gecos 'onnxruntime Build User' --disabled-password $BUILD_USER --uid $BUILD_UID -WORKDIR /home/$BUILD_USER -USER $BUILD_USER diff --git a/tools/ci_build/github/linux/run_dockerbuild.sh b/tools/ci_build/github/linux/run_dockerbuild.sh index 012d8a945c..ab8482c56c 100755 --- a/tools/ci_build/github/linux/run_dockerbuild.sh +++ b/tools/ci_build/github/linux/run_dockerbuild.sh @@ -3,7 +3,6 @@ set -e -o -x id SCRIPT_DIR="$( dirname "${BASH_SOURCE[0]}" )" SOURCE_ROOT=$(realpath $SCRIPT_DIR/../../../../) -CUDA_VER=cuda10.1-cudnn7.6 YOCTO_VERSION="4.19" INSTALL_DEPS_DISTRIBUTED_SETUP=false ORTMODULE_BUILD=false @@ -23,8 +22,6 @@ r) BUILD_DIR=${OPTARG};; p) PYTHON_VER=${OPTARG};; # "--build_wheel --use_openblas" x) BUILD_EXTR_PAR=${OPTARG};; -# "cuda10.0-cudnn7.3, cuda9.1-cudnn7.1" -c) CUDA_VER=${OPTARG};; # x86 or other, only for ubuntu16.04 os a) BUILD_ARCH=${OPTARG};; # openvino version tag: 2020.3 (OpenVINO EP 2.0 supports version starting 2020.3) @@ -81,12 +78,9 @@ elif [ $BUILD_OS = "yocto" ]; then --dockerfile $DOCKER_FILE --context . else if [ $BUILD_DEVICE = "gpu" ]; then - IMAGE="$BUILD_OS-$CUDA_VER" - DOCKER_FILE=Dockerfile.ubuntu_gpu - if [[ $BUILD_EXTR_PAR = *--enable_training* ]]; then - INSTALL_DEPS_EXTRA_ARGS="${INSTALL_DEPS_EXTRA_ARGS} -t" - DOCKER_FILE=Dockerfile.ubuntu_gpu_training - fi + #This code path is only for training. Inferecing pipeline uses CentOS + IMAGE="ubuntu_gpu_training" + INSTALL_DEPS_EXTRA_ARGS="${INSTALL_DEPS_EXTRA_ARGS} -t" if [[ $INSTALL_DEPS_DISTRIBUTED_SETUP = true ]]; then INSTALL_DEPS_EXTRA_ARGS="${INSTALL_DEPS_EXTRA_ARGS} -m" fi @@ -95,7 +89,7 @@ else fi $GET_DOCKER_IMAGE_CMD --repository "onnxruntime-$IMAGE" \ --docker-build-args="--build-arg BUILD_USER=onnxruntimedev --build-arg BUILD_UID=$(id -u) --build-arg PYTHON_VERSION=${PYTHON_VER} --build-arg INSTALL_DEPS_EXTRA_ARGS=\"${INSTALL_DEPS_EXTRA_ARGS}\" --build-arg USE_CONDA=${USE_CONDA}" \ - --dockerfile $DOCKER_FILE --context . + --dockerfile Dockerfile.ubuntu_gpu_training --context . elif [ $BUILD_DEVICE = "tensorrt" ]; then # TensorRT container release 20.12 IMAGE="$BUILD_OS-cuda11.1-cudnn8.0-tensorrt7.2" From 21ff8fabe933524e53fb5c2f22befa182671b61c Mon Sep 17 00:00:00 2001 From: Yulong Wang Date: Fri, 21 May 2021 19:18:22 -0700 Subject: [PATCH 14/28] [js/web] fix webpack config for onnxruntime-web (#7785) * fix webpack config for onnxruntime-web * fix regex of .worker.js file * fix regex * fix for testrunner * fix require('..') in test-main --- js/web/package.json | 9 +-------- js/web/webpack.config.js | 33 +++++++++++++++++++++++---------- 2 files changed, 24 insertions(+), 18 deletions(-) diff --git a/js/web/package.json b/js/web/package.json index ff2fa0e391..662261261e 100644 --- a/js/web/package.json +++ b/js/web/package.json @@ -73,12 +73,5 @@ "jsdelivr": "dist/ort.min.js", "unpkg": "dist/ort.min.js", "module": "./lib/index.js", - "browser": { - "fs": false, - "path": false, - "util": false, - "worker_threads": false, - "perf_hooks": false, - "os": false - } + "browser": "dist/ort-web.min.js" } diff --git a/js/web/webpack.config.js b/js/web/webpack.config.js index 345d9f056c..452246029f 100644 --- a/js/web/webpack.config.js +++ b/js/web/webpack.config.js @@ -43,7 +43,20 @@ function buildConfig({ filename, format, target, mode, devtool }) { type: format } }, - resolve: { extensions: ['.ts', '.js'] }, + resolve: { + extensions: ['.ts', '.js'], + alias: { + "util": false, + }, + fallback: { + "fs": false, + "path": false, + "util": false, + "os": false, + "worker_threads": false, + "perf_hooks": false, + } + }, plugins: [new webpack.WatchIgnorePlugin({ paths: [/\.js$/, /\.d\.ts$/] })], module: { rules: [{ @@ -57,7 +70,7 @@ function buildConfig({ filename, format, target, mode, devtool }) { } ] }, { - test: /\.worker.js$/, + test: /ort-wasm.*\.worker\.js$/, type: 'asset/source' }] }, @@ -66,10 +79,9 @@ function buildConfig({ filename, format, target, mode, devtool }) { }; if (mode === 'production') { - config.resolve.alias = { - './binding/ort-wasm-threaded.js': './binding/ort-wasm-threaded.min.js', - './binding/ort-wasm-threaded.worker.js': './binding/ort-wasm-threaded.min.worker.js' - }; + config.resolve.alias['./binding/ort-wasm-threaded.js'] = './binding/ort-wasm-threaded.min.js'; + config.resolve.alias['./binding/ort-wasm-threaded.worker.js'] = './binding/ort-wasm-threaded.min.worker.js'; + const options = defaultTerserPluginOptions(); options.terserOptions.format.preamble = COPYRIGHT_BANNER; config.plugins.push(new TerserPlugin(options)); @@ -90,8 +102,6 @@ function buildOrtConfig({ const config = buildConfig({ filename: `ort${suffix}.js`, format: 'umd', target, mode, devtool }); // set global name 'ort' config.output.library.name = 'ort'; - // do not use those node builtin modules in browser - config.resolve.fallback = { path: false, fs: false, util: false }; return config; } @@ -150,8 +160,11 @@ function buildTestRunnerConfig({ '../../node': '../../node' }, resolve: { + alias: { + // make sure to refer to original source files instead of generated bundle in test-main. + '..$': '../lib/index' + }, extensions: ['.ts', '.js'], - aliasFields: [], fallback: { './binding/ort-wasm.js': false, './binding/ort-wasm-threaded.js': false, @@ -174,7 +187,7 @@ function buildTestRunnerConfig({ } ] }, { - test: /\.worker\.js$/, + test: /ort-wasm.*\.worker\.js$/, type: 'asset/source' }] }, From ae14cedd6336bc3114d66d26ca45fa48baab82ee Mon Sep 17 00:00:00 2001 From: Guoyu Wang <62914304+gwang-msft@users.noreply.github.com> Date: Sat, 22 May 2021 01:23:39 -0700 Subject: [PATCH 15/28] Fix c_api warning (#7803) --- .../core/session/onnxruntime_c_api.h | 24 +++++++++---------- 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/include/onnxruntime/core/session/onnxruntime_c_api.h b/include/onnxruntime/core/session/onnxruntime_c_api.h index 746a344827..23309b7d5d 100644 --- a/include/onnxruntime/core/session/onnxruntime_c_api.h +++ b/include/onnxruntime/core/session/onnxruntime_c_api.h @@ -338,7 +338,7 @@ struct OrtApiBase { const OrtApi*(ORT_API_CALL* GetApi)(uint32_t version)NO_EXCEPTION; // Pass in ORT_API_VERSION // nullptr will be returned if the version is unsupported, for example when using a runtime older than this header file - const char*(ORT_API_CALL* GetVersionString)() NO_EXCEPTION; + const char*(ORT_API_CALL* GetVersionString)(void)NO_EXCEPTION; }; typedef struct OrtApiBase OrtApiBase; @@ -1301,16 +1301,16 @@ struct OrtApi { * \param arena_config_values - values to configure the arena * \param num_keys - number of keys passed in * Supported keys are (See docs/C_API.md for details on what the following parameters mean and how to choose these values.): - * "max_mem": Maximum memory that can be allocated by the arena based allocator. + * "max_mem": Maximum memory that can be allocated by the arena based allocator. Use 0 for ORT to pick the best value. Default is 0. - * "arena_extend_strategy": 0 = kNextPowerOfTwo, 1 = kSameAsRequested. + * "arena_extend_strategy": 0 = kNextPowerOfTwo, 1 = kSameAsRequested. Use -1 to allow ORT to choose the default. - * "initial_chunk_size_bytes": (Possible) Size of the first allocation in the arena. + * "initial_chunk_size_bytes": (Possible) Size of the first allocation in the arena. Only relevant if arena strategy is `kNextPowerOfTwo`. Use -1 to allow ORT to choose the default. - Ultimately, the first allocation size is determined by the allocation memory request. - * "max_dead_bytes_per_chunk": Threshold of unused memory in an allocated chunk of arena memory after + Ultimately, the first allocation size is determined by the allocation memory request. + * "max_dead_bytes_per_chunk": Threshold of unused memory in an allocated chunk of arena memory after crossing which the current chunk is chunked into 2. - * "initial_growth_chunk_size_bytes": (Possible) Size of the second allocation in the arena. + * "initial_growth_chunk_size_bytes": (Possible) Size of the second allocation in the arena. Only relevant if arena strategy is `kNextPowerOfTwo`. Use -1 to allow ORT to choose the default. Ultimately, the allocation size is determined by the allocation memory request. Further allocation sizes are governed by the arena extend strategy. @@ -1332,10 +1332,10 @@ struct OrtApi { /* * Creates an OrtPrepackedWeightsContainer instance. * This container will hold pre-packed buffers of shared initializers for sharing between sessions - * (i.e.) if there are shared initializers that can be shared between sessions, the pre-packed buffers + * (i.e.) if there are shared initializers that can be shared between sessions, the pre-packed buffers * of these (if any) may possibly be shared to provide memory footprint savings. Pass this container - * to sessions that you would like to share pre-packed buffers of shared initializers at session - * creation time. + * to sessions that you would like to share pre-packed buffers of shared initializers at session + * creation time. * \out - created OrtPrepackedWeightsContainer instance */ ORT_API2_STATUS(CreatePrepackedWeightsContainer, _Outptr_ OrtPrepackedWeightsContainer** out); @@ -1350,7 +1350,7 @@ struct OrtApi { * Same functionality offered by CreateSession() API except that a container that contains pre-packed weights' buffers is written into/read from by the created session. This is useful when used in conjunction with the AddInitializer() API which injects - shared initializer info into sessions. Wherever possible, the pre-packed versions of these + shared initializer info into sessions. Wherever possible, the pre-packed versions of these shared initializers are cached in this container so that multiple sessions can just re-use these instead of duplicating these in memory. * \env - OrtEnv instance instance @@ -1367,7 +1367,7 @@ struct OrtApi { * Same functionality offered by CreateSessionFromArray() API except that a container that contains pre-packed weights' buffers is written into/read from by the created session. This is useful when used in conjunction with the AddInitializer() API which injects - shared initializer info into sessions. Wherever possible, the pre-packed versions of these + shared initializer info into sessions. Wherever possible, the pre-packed versions of these shared initializers are cached in this container so that multiple sessions can just re-use these instead of duplicating these in memory. * \env - OrtEnv instance instance From de4c2217123718a8975007557e350af86c13ad82 Mon Sep 17 00:00:00 2001 From: stevenlix <38092805+stevenlix@users.noreply.github.com> Date: Sat, 22 May 2021 09:45:13 -0700 Subject: [PATCH 16/28] set max profile before opt profile (#7801) --- .../core/providers/tensorrt/tensorrt_execution_provider.cc | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/onnxruntime/core/providers/tensorrt/tensorrt_execution_provider.cc b/onnxruntime/core/providers/tensorrt/tensorrt_execution_provider.cc index 1a8753de3c..cc21cf99c6 100644 --- a/onnxruntime/core/providers/tensorrt/tensorrt_execution_provider.cc +++ b/onnxruntime/core/providers/tensorrt/tensorrt_execution_provider.cc @@ -1544,9 +1544,8 @@ common::Status TensorrtExecutionProvider::Compile(const std::vector& fuse *trt_profile = trt_builder->createOptimizationProfile(); } (*trt_profile)->setShapeValues(input_name.c_str(), nvinfer1::OptProfileSelector::kMIN, &shapes_min[0], shape_size); - (*trt_profile)->setShapeValues(input_name.c_str(), nvinfer1::OptProfileSelector::kOPT, &shapes_opt[0], shape_size); (*trt_profile)->setShapeValues(input_name.c_str(), nvinfer1::OptProfileSelector::kMAX, &shapes_max[0], shape_size); - + (*trt_profile)->setShapeValues(input_name.c_str(), nvinfer1::OptProfileSelector::kOPT, &shapes_opt[0], shape_size); } else { // Execution tensor nvinfer1::Dims dims_min(dims), dims_opt(dims), dims_max(dims); for (int j = 0, end = nb_dims; j < end; ++j) { @@ -1576,8 +1575,8 @@ common::Status TensorrtExecutionProvider::Compile(const std::vector& fuse *trt_profile = trt_builder->createOptimizationProfile(); } (*trt_profile)->setDimensions(input_name.c_str(), nvinfer1::OptProfileSelector::kMIN, dims_min); - (*trt_profile)->setDimensions(input_name.c_str(), nvinfer1::OptProfileSelector::kOPT, dims_opt); (*trt_profile)->setDimensions(input_name.c_str(), nvinfer1::OptProfileSelector::kMAX, dims_max); + (*trt_profile)->setDimensions(input_name.c_str(), nvinfer1::OptProfileSelector::kOPT, dims_opt); } ort.ReleaseTensorTypeAndShapeInfo(tensor_info); } From c4f515d38083548db94791f2bb383b50416672d4 Mon Sep 17 00:00:00 2001 From: Scott McKay Date: Sun, 23 May 2021 09:53:15 +1000 Subject: [PATCH 17/28] - Fix training cmake file so it builds if `--cmake_extra_defines onnxruntime_BUILD_UNIT_TESTS=OFF` is specified. (#7789) - Fix check on cudart_versions when building on Windows to handle None being returned --- cmake/onnxruntime_training.cmake | 309 ++++++++++++++++--------------- setup.py | 6 +- 2 files changed, 159 insertions(+), 156 deletions(-) diff --git a/cmake/onnxruntime_training.cmake b/cmake/onnxruntime_training.cmake index b56b49d15b..edaebee6e7 100644 --- a/cmake/onnxruntime_training.cmake +++ b/cmake/onnxruntime_training.cmake @@ -36,172 +36,173 @@ if (onnxruntime_USE_NCCL) target_include_directories(onnxruntime_training PRIVATE ${NCCL_INCLUDE_DIRS}) endif() -set_target_properties(onnxruntime_training PROPERTIES FOLDER "ONNXRuntime") -source_group(TREE ${ORTTRAINING_ROOT} FILES ${onnxruntime_training_srcs}) +if (onnxruntime_BUILD_UNIT_TESTS) + set_target_properties(onnxruntime_training PROPERTIES FOLDER "ONNXRuntime") + source_group(TREE ${ORTTRAINING_ROOT} FILES ${onnxruntime_training_srcs}) -# training runner lib -file(GLOB_RECURSE onnxruntime_training_runner_srcs - "${ORTTRAINING_SOURCE_DIR}/models/runner/*.h" - "${ORTTRAINING_SOURCE_DIR}/models/runner/*.cc" -) + # training runner lib + file(GLOB_RECURSE onnxruntime_training_runner_srcs + "${ORTTRAINING_SOURCE_DIR}/models/runner/*.h" + "${ORTTRAINING_SOURCE_DIR}/models/runner/*.cc" + ) -# perf test utils -set(onnxruntime_perf_test_src_dir ${TEST_SRC_DIR}/perftest) -set(onnxruntime_perf_test_src -"${onnxruntime_perf_test_src_dir}/utils.h") + # perf test utils + set(onnxruntime_perf_test_src_dir ${TEST_SRC_DIR}/perftest) + set(onnxruntime_perf_test_src + "${onnxruntime_perf_test_src_dir}/utils.h") -if(WIN32) - list(APPEND onnxruntime_perf_test_src - "${onnxruntime_perf_test_src_dir}/windows/utils.cc") -else () - list(APPEND onnxruntime_perf_test_src - "${onnxruntime_perf_test_src_dir}/posix/utils.cc") -endif() - -onnxruntime_add_static_library(onnxruntime_training_runner ${onnxruntime_training_runner_srcs} ${onnxruntime_perf_test_src}) -add_dependencies(onnxruntime_training_runner ${onnxruntime_EXTERNAL_DEPENDENCIES} onnx onnxruntime_providers) - -onnxruntime_add_include_to_target(onnxruntime_training_runner onnxruntime_training onnxruntime_framework onnxruntime_common onnx onnx_proto protobuf::libprotobuf onnxruntime_training flatbuffers) - -target_include_directories(onnxruntime_training_runner PRIVATE ${CMAKE_CURRENT_BINARY_DIR} ${ONNXRUNTIME_ROOT} ${ORTTRAINING_ROOT} ${eigen_INCLUDE_DIRS} PUBLIC ${onnxruntime_graph_header}) -target_link_libraries(onnxruntime_training_runner PRIVATE nlohmann_json::nlohmann_json) -if (onnxruntime_USE_CUDA) - target_include_directories(onnxruntime_training_runner PUBLIC ${onnxruntime_CUDNN_HOME}/include ${CMAKE_CUDA_TOOLKIT_INCLUDE_DIRECTORIES}) -endif() - -if (onnxruntime_USE_NCCL) - target_include_directories(onnxruntime_training_runner PRIVATE ${NCCL_INCLUDE_DIRS}) -endif() - -if (onnxruntime_USE_ROCM) - add_definitions(-DUSE_ROCM=1) - target_include_directories(onnxruntime_training_runner PUBLIC ${onnxruntime_ROCM_HOME}/include) -endif() - -check_cxx_compiler_flag(-Wno-maybe-uninitialized HAS_NO_MAYBE_UNINITIALIZED) -if(UNIX AND NOT APPLE) - if (HAS_NO_MAYBE_UNINITIALIZED) - target_compile_options(onnxruntime_training_runner PUBLIC "-Wno-maybe-uninitialized") + if(WIN32) + list(APPEND onnxruntime_perf_test_src + "${onnxruntime_perf_test_src_dir}/windows/utils.cc") + else () + list(APPEND onnxruntime_perf_test_src + "${onnxruntime_perf_test_src_dir}/posix/utils.cc") endif() -endif() -if (onnxruntime_USE_ROCM) - target_compile_options(onnxruntime_training_runner PUBLIC -D__HIP_PLATFORM_HCC__=1) -endif() + onnxruntime_add_static_library(onnxruntime_training_runner ${onnxruntime_training_runner_srcs} ${onnxruntime_perf_test_src}) + add_dependencies(onnxruntime_training_runner ${onnxruntime_EXTERNAL_DEPENDENCIES} onnx onnxruntime_providers) -set_target_properties(onnxruntime_training_runner PROPERTIES FOLDER "ONNXRuntimeTest") -source_group(TREE ${REPO_ROOT} FILES ${onnxruntime_training_runner_srcs} ${onnxruntime_perf_test_src}) + onnxruntime_add_include_to_target(onnxruntime_training_runner onnxruntime_training onnxruntime_framework onnxruntime_common onnx onnx_proto protobuf::libprotobuf onnxruntime_training flatbuffers) - -# MNIST -file(GLOB_RECURSE training_mnist_src - "${ORTTRAINING_SOURCE_DIR}/models/mnist/*.h" - "${ORTTRAINING_SOURCE_DIR}/models/mnist/mnist_data_provider.cc" - "${ORTTRAINING_SOURCE_DIR}/models/mnist/main.cc" -) -onnxruntime_add_executable(onnxruntime_training_mnist ${training_mnist_src}) -onnxruntime_add_include_to_target(onnxruntime_training_mnist onnxruntime_common onnx onnx_proto protobuf::libprotobuf onnxruntime_training flatbuffers) -target_include_directories(onnxruntime_training_mnist PUBLIC ${CMAKE_CURRENT_BINARY_DIR} ${ONNXRUNTIME_ROOT} ${ORTTRAINING_ROOT} ${eigen_INCLUDE_DIRS} ${CXXOPTS} ${extra_includes} ${onnxruntime_graph_header} ${onnxruntime_exec_src_dir} ${CMAKE_CURRENT_BINARY_DIR} ${CMAKE_CURRENT_BINARY_DIR}/onnx onnxruntime_training_runner) - -set(ONNXRUNTIME_LIBS - onnxruntime_session - ${onnxruntime_libs} - ${PROVIDERS_ROCM} - ${PROVIDERS_MKLDNN} - onnxruntime_optimizer - onnxruntime_providers - onnxruntime_util - onnxruntime_framework - onnxruntime_graph - onnxruntime_common - onnxruntime_mlas - onnxruntime_flatbuffers -) - -if (onnxruntime_ENABLE_LANGUAGE_INTEROP_OPS) - list(APPEND ONNXRUNTIME_LIBS onnxruntime_language_interop onnxruntime_pyop) -endif() - -if(UNIX AND NOT APPLE) - if (HAS_NO_MAYBE_UNINITIALIZED) - target_compile_options(onnxruntime_training_mnist PUBLIC "-Wno-maybe-uninitialized") + target_include_directories(onnxruntime_training_runner PRIVATE ${CMAKE_CURRENT_BINARY_DIR} ${ONNXRUNTIME_ROOT} ${ORTTRAINING_ROOT} ${eigen_INCLUDE_DIRS} PUBLIC ${onnxruntime_graph_header}) + target_link_libraries(onnxruntime_training_runner PRIVATE nlohmann_json::nlohmann_json) + if (onnxruntime_USE_CUDA) + target_include_directories(onnxruntime_training_runner PUBLIC ${onnxruntime_CUDNN_HOME}/include ${CMAKE_CUDA_TOOLKIT_INCLUDE_DIRECTORIES}) endif() -endif() -target_link_libraries(onnxruntime_training_mnist PRIVATE onnxruntime_training_runner onnxruntime_training ${ONNXRUNTIME_LIBS} ${onnxruntime_EXTERNAL_LIBRARIES}) -set_target_properties(onnxruntime_training_mnist PROPERTIES FOLDER "ONNXRuntimeTest") - -# squeezenet -# Disabling build for squeezenet, as no one is using this -#[[ -file(GLOB_RECURSE training_squeezene_src - "${ORTTRAINING_SOURCE_DIR}/models/squeezenet/*.h" - "${ORTTRAINING_SOURCE_DIR}/models/squeezenet/*.cc" -) -onnxruntime_add_executable(onnxruntime_training_squeezenet ${training_squeezene_src}) -onnxruntime_add_include_to_target(onnxruntime_training_squeezenet onnxruntime_common onnx onnx_proto protobuf::libprotobuf onnxruntime_training flatbuffers) -target_include_directories(onnxruntime_training_squeezenet PUBLIC ${ONNXRUNTIME_ROOT} ${ORTTRAINING_ROOT} ${eigen_INCLUDE_DIRS} ${extra_includes} ${onnxruntime_graph_header} ${onnxruntime_exec_src_dir} ${CMAKE_CURRENT_BINARY_DIR} ${CMAKE_CURRENT_BINARY_DIR}/onnx onnxruntime_training_runner) -if(UNIX AND NOT APPLE) - target_compile_options(onnxruntime_training_squeezenet PUBLIC "-Wno-maybe-uninitialized") -endif() -target_link_libraries(onnxruntime_training_squeezenet PRIVATE onnxruntime_training_runner onnxruntime_training ${ONNXRUNTIME_LIBS} ${onnxruntime_EXTERNAL_LIBRARIES}) -set_target_properties(onnxruntime_training_squeezenet PROPERTIES FOLDER "ONNXRuntimeTest") -]] - -# BERT -file(GLOB_RECURSE training_bert_src - "${ORTTRAINING_SOURCE_DIR}/models/bert/*.h" - "${ORTTRAINING_SOURCE_DIR}/models/bert/*.cc" -) -onnxruntime_add_executable(onnxruntime_training_bert ${training_bert_src}) - -if(UNIX AND NOT APPLE) - if (HAS_NO_MAYBE_UNINITIALIZED) - target_compile_options(onnxruntime_training_bert PUBLIC "-Wno-maybe-uninitialized") + if (onnxruntime_USE_NCCL) + target_include_directories(onnxruntime_training_runner PRIVATE ${NCCL_INCLUDE_DIRS}) endif() -endif() -onnxruntime_add_include_to_target(onnxruntime_training_bert onnxruntime_common onnx onnx_proto protobuf::libprotobuf onnxruntime_training flatbuffers) -target_include_directories(onnxruntime_training_bert PUBLIC ${CMAKE_CURRENT_BINARY_DIR} ${ONNXRUNTIME_ROOT} ${ORTTRAINING_ROOT} ${MPI_CXX_INCLUDE_DIRS} ${eigen_INCLUDE_DIRS} ${CXXOPTS} ${extra_includes} ${onnxruntime_graph_header} ${onnxruntime_exec_src_dir} ${CMAKE_CURRENT_BINARY_DIR} ${CMAKE_CURRENT_BINARY_DIR}/onnx onnxruntime_training_runner) - -target_link_libraries(onnxruntime_training_bert PRIVATE onnxruntime_training_runner onnxruntime_training ${ONNXRUNTIME_LIBS} ${onnxruntime_EXTERNAL_LIBRARIES}) -set_target_properties(onnxruntime_training_bert PROPERTIES FOLDER "ONNXRuntimeTest") - -# Pipeline -file(GLOB_RECURSE training_pipeline_poc_src - "${ORTTRAINING_SOURCE_DIR}/models/pipeline_poc/*.h" - "${ORTTRAINING_SOURCE_DIR}/models/pipeline_poc/*.cc" -) -onnxruntime_add_executable(onnxruntime_training_pipeline_poc ${training_pipeline_poc_src}) - -if(UNIX AND NOT APPLE) - if (HAS_NO_MAYBE_UNINITIALIZED) - target_compile_options(onnxruntime_training_pipeline_poc PUBLIC "-Wno-maybe-uninitialized") + if (onnxruntime_USE_ROCM) + add_definitions(-DUSE_ROCM=1) + target_include_directories(onnxruntime_training_runner PUBLIC ${onnxruntime_ROCM_HOME}/include) endif() -endif() -onnxruntime_add_include_to_target(onnxruntime_training_pipeline_poc onnxruntime_common onnx onnx_proto protobuf::libprotobuf onnxruntime_training flatbuffers) -target_include_directories(onnxruntime_training_pipeline_poc PUBLIC ${CMAKE_CURRENT_BINARY_DIR} ${ONNXRUNTIME_ROOT} ${ORTTRAINING_ROOT} ${MPI_CXX_INCLUDE_DIRS} ${eigen_INCLUDE_DIRS} ${CXXOPTS} ${extra_includes} ${onnxruntime_graph_header} ${onnxruntime_exec_src_dir} ${CMAKE_CURRENT_BINARY_DIR} ${CMAKE_CURRENT_BINARY_DIR}/onnx onnxruntime_training_runner) -if (onnxruntime_USE_NCCL) - target_include_directories(onnxruntime_training_pipeline_poc PRIVATE ${NCCL_INCLUDE_DIRS}) -endif() - -target_link_libraries(onnxruntime_training_pipeline_poc PRIVATE onnxruntime_training_runner onnxruntime_training ${ONNXRUNTIME_LIBS} ${onnxruntime_EXTERNAL_LIBRARIES}) -set_target_properties(onnxruntime_training_pipeline_poc PROPERTIES FOLDER "ONNXRuntimeTest") - -# GPT-2 -file(GLOB_RECURSE training_gpt2_src - "${ORTTRAINING_SOURCE_DIR}/models/gpt2/*.h" - "${ORTTRAINING_SOURCE_DIR}/models/gpt2/*.cc" -) -onnxruntime_add_executable(onnxruntime_training_gpt2 ${training_gpt2_src}) -if(UNIX AND NOT APPLE) - if (HAS_NO_MAYBE_UNINITIALIZED) - target_compile_options(onnxruntime_training_gpt2 PUBLIC "-Wno-maybe-uninitialized") + check_cxx_compiler_flag(-Wno-maybe-uninitialized HAS_NO_MAYBE_UNINITIALIZED) + if(UNIX AND NOT APPLE) + if (HAS_NO_MAYBE_UNINITIALIZED) + target_compile_options(onnxruntime_training_runner PUBLIC "-Wno-maybe-uninitialized") + endif() endif() -endif() -onnxruntime_add_include_to_target(onnxruntime_training_gpt2 onnxruntime_common onnx onnx_proto protobuf::libprotobuf onnxruntime_training flatbuffers) -target_include_directories(onnxruntime_training_gpt2 PUBLIC ${CMAKE_CURRENT_BINARY_DIR} ${ONNXRUNTIME_ROOT} ${ORTTRAINING_ROOT} ${MPI_CXX_INCLUDE_DIRS} ${eigen_INCLUDE_DIRS} ${CXXOPTS} ${extra_includes} ${onnxruntime_graph_header} ${onnxruntime_exec_src_dir} ${CMAKE_CURRENT_BINARY_DIR} ${CMAKE_CURRENT_BINARY_DIR}/onnx onnxruntime_training_runner) -target_link_libraries(onnxruntime_training_gpt2 PRIVATE onnxruntime_training_runner onnxruntime_training ${ONNXRUNTIME_LIBS} ${onnxruntime_EXTERNAL_LIBRARIES}) -set_target_properties(onnxruntime_training_gpt2 PROPERTIES FOLDER "ONNXRuntimeTest") + if (onnxruntime_USE_ROCM) + target_compile_options(onnxruntime_training_runner PUBLIC -D__HIP_PLATFORM_HCC__=1) + endif() + + set_target_properties(onnxruntime_training_runner PROPERTIES FOLDER "ONNXRuntimeTest") + source_group(TREE ${REPO_ROOT} FILES ${onnxruntime_training_runner_srcs} ${onnxruntime_perf_test_src}) + + # MNIST + file(GLOB_RECURSE training_mnist_src + "${ORTTRAINING_SOURCE_DIR}/models/mnist/*.h" + "${ORTTRAINING_SOURCE_DIR}/models/mnist/mnist_data_provider.cc" + "${ORTTRAINING_SOURCE_DIR}/models/mnist/main.cc" + ) + onnxruntime_add_executable(onnxruntime_training_mnist ${training_mnist_src}) + onnxruntime_add_include_to_target(onnxruntime_training_mnist onnxruntime_common onnx onnx_proto protobuf::libprotobuf onnxruntime_training flatbuffers) + target_include_directories(onnxruntime_training_mnist PUBLIC ${CMAKE_CURRENT_BINARY_DIR} ${ONNXRUNTIME_ROOT} ${ORTTRAINING_ROOT} ${eigen_INCLUDE_DIRS} ${CXXOPTS} ${extra_includes} ${onnxruntime_graph_header} ${onnxruntime_exec_src_dir} ${CMAKE_CURRENT_BINARY_DIR} ${CMAKE_CURRENT_BINARY_DIR}/onnx onnxruntime_training_runner) + + set(ONNXRUNTIME_LIBS + onnxruntime_session + ${onnxruntime_libs} + ${PROVIDERS_ROCM} + ${PROVIDERS_MKLDNN} + onnxruntime_optimizer + onnxruntime_providers + onnxruntime_util + onnxruntime_framework + onnxruntime_graph + onnxruntime_common + onnxruntime_mlas + onnxruntime_flatbuffers + ) + + if (onnxruntime_ENABLE_LANGUAGE_INTEROP_OPS) + list(APPEND ONNXRUNTIME_LIBS onnxruntime_language_interop onnxruntime_pyop) + endif() + + if(UNIX AND NOT APPLE) + if (HAS_NO_MAYBE_UNINITIALIZED) + target_compile_options(onnxruntime_training_mnist PUBLIC "-Wno-maybe-uninitialized") + endif() + endif() + target_link_libraries(onnxruntime_training_mnist PRIVATE onnxruntime_training_runner onnxruntime_training ${ONNXRUNTIME_LIBS} ${onnxruntime_EXTERNAL_LIBRARIES}) + set_target_properties(onnxruntime_training_mnist PROPERTIES FOLDER "ONNXRuntimeTest") + + # squeezenet + # Disabling build for squeezenet, as no one is using this + #[[ + file(GLOB_RECURSE training_squeezene_src + "${ORTTRAINING_SOURCE_DIR}/models/squeezenet/*.h" + "${ORTTRAINING_SOURCE_DIR}/models/squeezenet/*.cc" + ) + onnxruntime_add_executable(onnxruntime_training_squeezenet ${training_squeezene_src}) + onnxruntime_add_include_to_target(onnxruntime_training_squeezenet onnxruntime_common onnx onnx_proto protobuf::libprotobuf onnxruntime_training flatbuffers) + target_include_directories(onnxruntime_training_squeezenet PUBLIC ${ONNXRUNTIME_ROOT} ${ORTTRAINING_ROOT} ${eigen_INCLUDE_DIRS} ${extra_includes} ${onnxruntime_graph_header} ${onnxruntime_exec_src_dir} ${CMAKE_CURRENT_BINARY_DIR} ${CMAKE_CURRENT_BINARY_DIR}/onnx onnxruntime_training_runner) + if(UNIX AND NOT APPLE) + target_compile_options(onnxruntime_training_squeezenet PUBLIC "-Wno-maybe-uninitialized") + endif() + target_link_libraries(onnxruntime_training_squeezenet PRIVATE onnxruntime_training_runner onnxruntime_training ${ONNXRUNTIME_LIBS} ${onnxruntime_EXTERNAL_LIBRARIES}) + set_target_properties(onnxruntime_training_squeezenet PROPERTIES FOLDER "ONNXRuntimeTest") + ]] + + # BERT + file(GLOB_RECURSE training_bert_src + "${ORTTRAINING_SOURCE_DIR}/models/bert/*.h" + "${ORTTRAINING_SOURCE_DIR}/models/bert/*.cc" + ) + onnxruntime_add_executable(onnxruntime_training_bert ${training_bert_src}) + + if(UNIX AND NOT APPLE) + if (HAS_NO_MAYBE_UNINITIALIZED) + target_compile_options(onnxruntime_training_bert PUBLIC "-Wno-maybe-uninitialized") + endif() + endif() + + onnxruntime_add_include_to_target(onnxruntime_training_bert onnxruntime_common onnx onnx_proto protobuf::libprotobuf onnxruntime_training flatbuffers) + target_include_directories(onnxruntime_training_bert PUBLIC ${CMAKE_CURRENT_BINARY_DIR} ${ONNXRUNTIME_ROOT} ${ORTTRAINING_ROOT} ${MPI_CXX_INCLUDE_DIRS} ${eigen_INCLUDE_DIRS} ${CXXOPTS} ${extra_includes} ${onnxruntime_graph_header} ${onnxruntime_exec_src_dir} ${CMAKE_CURRENT_BINARY_DIR} ${CMAKE_CURRENT_BINARY_DIR}/onnx onnxruntime_training_runner) + + target_link_libraries(onnxruntime_training_bert PRIVATE onnxruntime_training_runner onnxruntime_training ${ONNXRUNTIME_LIBS} ${onnxruntime_EXTERNAL_LIBRARIES}) + set_target_properties(onnxruntime_training_bert PROPERTIES FOLDER "ONNXRuntimeTest") + + # Pipeline + file(GLOB_RECURSE training_pipeline_poc_src + "${ORTTRAINING_SOURCE_DIR}/models/pipeline_poc/*.h" + "${ORTTRAINING_SOURCE_DIR}/models/pipeline_poc/*.cc" + ) + onnxruntime_add_executable(onnxruntime_training_pipeline_poc ${training_pipeline_poc_src}) + + if(UNIX AND NOT APPLE) + if (HAS_NO_MAYBE_UNINITIALIZED) + target_compile_options(onnxruntime_training_pipeline_poc PUBLIC "-Wno-maybe-uninitialized") + endif() + endif() + + onnxruntime_add_include_to_target(onnxruntime_training_pipeline_poc onnxruntime_common onnx onnx_proto protobuf::libprotobuf onnxruntime_training flatbuffers) + target_include_directories(onnxruntime_training_pipeline_poc PUBLIC ${CMAKE_CURRENT_BINARY_DIR} ${ONNXRUNTIME_ROOT} ${ORTTRAINING_ROOT} ${MPI_CXX_INCLUDE_DIRS} ${eigen_INCLUDE_DIRS} ${CXXOPTS} ${extra_includes} ${onnxruntime_graph_header} ${onnxruntime_exec_src_dir} ${CMAKE_CURRENT_BINARY_DIR} ${CMAKE_CURRENT_BINARY_DIR}/onnx onnxruntime_training_runner) + if (onnxruntime_USE_NCCL) + target_include_directories(onnxruntime_training_pipeline_poc PRIVATE ${NCCL_INCLUDE_DIRS}) + endif() + + target_link_libraries(onnxruntime_training_pipeline_poc PRIVATE onnxruntime_training_runner onnxruntime_training ${ONNXRUNTIME_LIBS} ${onnxruntime_EXTERNAL_LIBRARIES}) + set_target_properties(onnxruntime_training_pipeline_poc PROPERTIES FOLDER "ONNXRuntimeTest") + + # GPT-2 + file(GLOB_RECURSE training_gpt2_src + "${ORTTRAINING_SOURCE_DIR}/models/gpt2/*.h" + "${ORTTRAINING_SOURCE_DIR}/models/gpt2/*.cc" + ) + onnxruntime_add_executable(onnxruntime_training_gpt2 ${training_gpt2_src}) + if(UNIX AND NOT APPLE) + if (HAS_NO_MAYBE_UNINITIALIZED) + target_compile_options(onnxruntime_training_gpt2 PUBLIC "-Wno-maybe-uninitialized") + endif() + endif() + onnxruntime_add_include_to_target(onnxruntime_training_gpt2 onnxruntime_common onnx onnx_proto protobuf::libprotobuf onnxruntime_training flatbuffers) + target_include_directories(onnxruntime_training_gpt2 PUBLIC ${CMAKE_CURRENT_BINARY_DIR} ${ONNXRUNTIME_ROOT} ${ORTTRAINING_ROOT} ${MPI_CXX_INCLUDE_DIRS} ${eigen_INCLUDE_DIRS} ${CXXOPTS} ${extra_includes} ${onnxruntime_graph_header} ${onnxruntime_exec_src_dir} ${CMAKE_CURRENT_BINARY_DIR} ${CMAKE_CURRENT_BINARY_DIR}/onnx onnxruntime_training_runner) + + target_link_libraries(onnxruntime_training_gpt2 PRIVATE onnxruntime_training_runner onnxruntime_training ${ONNXRUNTIME_LIBS} ${onnxruntime_EXTERNAL_LIBRARIES}) + set_target_properties(onnxruntime_training_gpt2 PROPERTIES FOLDER "ONNXRuntimeTest") + +endif() diff --git a/setup.py b/setup.py index d6c3f2eff3..074d8ed6f8 100644 --- a/setup.py +++ b/setup.py @@ -387,12 +387,14 @@ if enable_training: # cudart_versions are integers cudart_versions = find_cudart_versions(build_env=True) - if len(cudart_versions) == 1: + if cudart_versions and len(cudart_versions) == 1: f.write("cudart_version = {}\n".format(cudart_versions[0])) else: print( "Error getting cudart version. ", - "did not find any cudart library" if len(cudart_versions) == 0 else "found multiple cudart libraries") + "did not find any cudart library" + if not cudart_versions or len(cudart_versions) == 0 + else "found multiple cudart libraries") else: # TODO: rocm pass From 13a129054f00562b08784d0a8d9ff318bd8b8063 Mon Sep 17 00:00:00 2001 From: baijumeswani Date: Sat, 22 May 2021 20:52:26 -0700 Subject: [PATCH 18/28] Prevent unnecessary re-initialization of the graph when model has unused parameters (#7799) --- .../ortmodule/_graph_execution_manager.py | 16 +++++++++---- .../training/ortmodule/_training_manager.py | 6 ++++- .../python/orttraining_test_ortmodule_api.py | 24 ++++++++++++++++++- 3 files changed, 39 insertions(+), 7 deletions(-) diff --git a/orttraining/orttraining/python/training/ortmodule/_graph_execution_manager.py b/orttraining/orttraining/python/training/ortmodule/_graph_execution_manager.py index e726a62105..8c54e93ddb 100644 --- a/orttraining/orttraining/python/training/ortmodule/_graph_execution_manager.py +++ b/orttraining/orttraining/python/training/ortmodule/_graph_execution_manager.py @@ -158,11 +158,6 @@ class GraphExecutionManager(ABC): self._optimized_onnx_model = onnx.load_model_from_string(self._graph_builder.get_model()) self._graph_info = self._graph_builder.get_graph_info() - # TODO: Explore ways to make self._graph_info.initializer_names and self._graph_info.initializer_names_to_train - # a set (unordered_set in the backend) that does not require a copy on each reference. - self._graph_initializer_names = set(self._graph_info.initializer_names) - self._graph_initializer_names_to_train = set(self._graph_info.initializer_names_to_train) - def _get_session_config(self): """Creates and returns the session configuration to be used for the ExecutionAgent""" providers = None @@ -202,6 +197,12 @@ class GraphExecutionManager(ABC): # 3. Export the user model under self._export_training_flag mode # Return True if the model needed to be exported, False if no export was required. + # Note: Model is only exported when: + # 1. Model has never been exported before. + # 2. Model input schema has changed (changes in inputs requiring gradient, shape, boolean inputs values change, etc) + # Model is not re-exported when the model parameters change. This can happen when the model is a stateful model, + # or the user explicitly changed model parameters after the onnx export. + schema = _io._extract_schema({'args': copy.copy(inputs), 'kwargs': copy.copy(kwargs)}) if self._onnx_model and schema == self._input_info.schema: # All required models have already been exported previously @@ -314,3 +315,8 @@ class GraphExecutionManager(ABC): # It is assumed here that the order and names of the inputs and outputs are not modified by the backend in any way # and are kept as they appear in the exported onnx model. self._graph_builder.initialize(self._onnx_model.SerializeToString(), grad_builder_config) + + # TODO: Explore ways to make self._graph_info.initializer_names and self._graph_info.initializer_names_to_train + # a set (unordered_set in the backend) that does not require a copy on each reference. + self._graph_initializer_names = set(initializer_names) + self._graph_initializer_names_to_train = set(initializer_names_to_train) diff --git a/orttraining/orttraining/python/training/ortmodule/_training_manager.py b/orttraining/orttraining/python/training/ortmodule/_training_manager.py index 97d9802467..96c634105a 100644 --- a/orttraining/orttraining/python/training/ortmodule/_training_manager.py +++ b/orttraining/orttraining/python/training/ortmodule/_training_manager.py @@ -241,8 +241,12 @@ class TrainingManager(GraphExecutionManager): def _reinitialize_graph_builder(self, input_info): """Return true if the module graph builder was reinitialized""" + # Model could have unused parameters which are dropped after export and so not a part of self._graph_initializer_names_to_train. + # To see if any trainable initializers changed, compare self._graph_initializer_names_to_train + # with initializers in module named_parameters that are known to the onnx graph. initializer_names_to_train_set_user_model = {name for name, param in - self._flattened_module.named_parameters() if param.requires_grad} + self._flattened_module.named_parameters() + if param.requires_grad and name in self._graph_initializer_names} # If inputs requiring gradient change from forward to the next, the module_gradient_graph_builder # needs to be reinitialized so it can compute the backward output for the new inputs that require_grad diff --git a/orttraining/orttraining/test/python/orttraining_test_ortmodule_api.py b/orttraining/orttraining/test/python/orttraining_test_ortmodule_api.py index 8ebb9c1c61..3a6607895d 100644 --- a/orttraining/orttraining/test/python/orttraining_test_ortmodule_api.py +++ b/orttraining/orttraining/test/python/orttraining_test_ortmodule_api.py @@ -16,7 +16,7 @@ from collections import OrderedDict from collections import namedtuple from inspect import signature -from onnxruntime.training.ortmodule import ORTModule, _utils +from onnxruntime.training.ortmodule import ORTModule, _utils, _io import _test_helpers # Import autocasting libs @@ -2597,3 +2597,25 @@ def test_stateless_model_unspecified_device(): ort_y = ort_model(ort_x) _test_helpers.assert_values_are_close(pt_y, ort_y) + +@pytest.mark.parametrize("model", + [(UnusedBeginParameterNet(784, 500, 400, 10)), + (UnusedMiddleParameterNet(784, 500, 400, 10)), + (UnusedEndParameterNet(784, 500, 400, 10))]) +def test_unused_parameters_does_not_unnecssarily_reinitilize(model): + device = 'cuda' + + N, D_in, H1, H2, D_out = 64, 784, 500, 400, 10 + model = model.to(device) + ort_model = ORTModule(copy.deepcopy(model)) + training_manager = ort_model._execution_manager(ort_model._is_training()) + + x = torch.randn(N, D_in, device=device) + _ = ort_model(x) + + input_info = _io.parse_inputs_for_onnx_export(training_manager._module_parameters, + training_manager._onnx_model, + x, + {}) + + assert not training_manager._reinitialize_graph_builder(input_info) From bd5067a2ff0769b5f11ea157fe565998a5f3e349 Mon Sep 17 00:00:00 2001 From: Sheil Kumar Date: Mon, 24 May 2021 08:00:24 -0700 Subject: [PATCH 19/28] Cannot upgrade SDK version because winml_lib_telemetry pulls in SDK cppwinrt version (#7795) * fix telemetry includes * add dependencies Co-authored-by: Sheil Kumar --- cmake/winml.cmake | 12 +++++++++++- winml/lib/Common/inc/winrt_headers.h | 2 +- winml/lib/Telemetry/pch.h | 2 +- 3 files changed, 13 insertions(+), 3 deletions(-) diff --git a/cmake/winml.cmake b/cmake/winml.cmake index c2ee74e799..4207c55b3f 100644 --- a/cmake/winml.cmake +++ b/cmake/winml.cmake @@ -169,6 +169,7 @@ if (onnxruntime_USE_TELEMETRY) endif() # Compiler flags +target_compile_definitions(winml_lib_telemetry PRIVATE WINML_ROOT_NS=${winml_root_ns}) target_compile_definitions(winml_lib_telemetry PRIVATE PLATFORM_WINDOWS) target_compile_definitions(winml_lib_telemetry PRIVATE _SCL_SECURE_NO_WARNINGS) # remove warnings about unchecked iterators target_compile_definitions(winml_lib_telemetry PRIVATE BINARY_NAME=\"${BINARY_NAME}\") @@ -177,7 +178,10 @@ target_compile_definitions(winml_lib_telemetry PRIVATE BINARY_NAME=\"${BINARY_NA target_precompiled_header(winml_lib_telemetry pch.h) # Includes -target_include_directories(winml_lib_telemetry PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/winml/sdk/cppwinrt/include) +target_include_directories(winml_lib_telemetry PRIVATE ${CMAKE_CURRENT_BINARY_DIR}) # windows machine learning generated component headers +target_include_directories(winml_lib_telemetry PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/winml_api) # windows machine learning generated component headers +target_include_directories(winml_lib_telemetry PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/winml_api/comp_generated) # windows machine learning generated component headers +target_include_directories(winml_lib_telemetry PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/winml/sdk/cppwinrt/include) # sdk cppwinrt headers target_include_directories(winml_lib_telemetry PRIVATE ${CMAKE_SOURCE_DIR}/common/inc) target_include_directories(winml_lib_telemetry PRIVATE ${winml_lib_telemetry_dir}) target_include_directories(winml_lib_telemetry PRIVATE ${winml_lib_common_dir}/inc) @@ -189,6 +193,12 @@ set_target_properties(winml_lib_telemetry FOLDER ${target_folder}) +# Add deps +add_dependencies(winml_lib_telemetry winml_sdk_cppwinrt) +add_dependencies(winml_lib_telemetry winml_api) +add_dependencies(winml_lib_telemetry winml_api_native) +add_dependencies(winml_lib_telemetry winml_api_native_internal) + # Link libraries target_link_libraries(winml_lib_telemetry PRIVATE wil) diff --git a/winml/lib/Common/inc/winrt_headers.h b/winml/lib/Common/inc/winrt_headers.h index a1f57871f2..2791016be7 100644 --- a/winml/lib/Common/inc/winrt_headers.h +++ b/winml/lib/Common/inc/winrt_headers.h @@ -5,7 +5,7 @@ #include "common.h" -#include +#include "winrt/windows.media.h" #include "winrt/windows.graphics.imaging.h" #include "winrt/windows.foundation.h" #include "winrt/windows.foundation.collections.h" diff --git a/winml/lib/Telemetry/pch.h b/winml/lib/Telemetry/pch.h index 5ec2dd9e47..976f62c7f9 100644 --- a/winml/lib/Telemetry/pch.h +++ b/winml/lib/Telemetry/pch.h @@ -3,6 +3,6 @@ #pragma once -#include "common.h" +#include "winrt_headers.h" #include "TraceLoggingConfig.h" #include "NamespaceAliases.h" \ No newline at end of file From da5ab325eeecf70b08614e27ceb5a3e99cac291c Mon Sep 17 00:00:00 2001 From: Ashwini Khade Date: Mon, 24 May 2021 11:02:45 -0700 Subject: [PATCH 20/28] register batchnorm for opset14 cuda provider (#7806) --- .../providers/cuda/cuda_execution_provider.cc | 18 ++++++++++++------ .../core/providers/cuda/nn/batch_norm.cc | 11 ++++++++++- .../core/providers/cuda/nn/batch_norm.h | 8 ++++++++ 3 files changed, 30 insertions(+), 7 deletions(-) diff --git a/onnxruntime/core/providers/cuda/cuda_execution_provider.cc b/onnxruntime/core/providers/cuda/cuda_execution_provider.cc index f3d5b7e8de..0ad7015519 100644 --- a/onnxruntime/core/providers/cuda/cuda_execution_provider.cc +++ b/onnxruntime/core/providers/cuda/cuda_execution_provider.cc @@ -498,9 +498,9 @@ class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, class ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, 7, 8, float, BatchNormalization); class ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, 7, 8, double, BatchNormalization); class ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, 7, 8, MLFloat16, BatchNormalization); -class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, 9, float, BatchNormalization); -class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, 9, double, BatchNormalization); -class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, 9, MLFloat16, BatchNormalization); +class ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, 9, 13, float, BatchNormalization); +class ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, 9, 13, double, BatchNormalization); +class ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, 9, 13, MLFloat16, BatchNormalization); class ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, 1, 12, float, LRN); class ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, 1, 12, double, LRN); class ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, 1, 12, MLFloat16, LRN); @@ -1112,6 +1112,9 @@ class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, 14, float, LSTM); class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, 14, double, LSTM); class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, 14, MLFloat16, LSTM); +class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, 14, float, BatchNormalization); +class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, 14, double, BatchNormalization); +class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, 14, MLFloat16, BatchNormalization); #if defined(CUDA_VERSION) && CUDA_VERSION >= 11000 class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, 14, BFloat16, Add); class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, 14, BFloat16, Sub); @@ -1307,9 +1310,9 @@ static Status RegisterCudaKernels(KernelRegistry& kernel_registry) { BuildKernelCreateInfo, BuildKernelCreateInfo, BuildKernelCreateInfo, - BuildKernelCreateInfo, - BuildKernelCreateInfo, - BuildKernelCreateInfo, + BuildKernelCreateInfo, + BuildKernelCreateInfo, + BuildKernelCreateInfo, BuildKernelCreateInfo, BuildKernelCreateInfo, BuildKernelCreateInfo, @@ -1919,6 +1922,9 @@ static Status RegisterCudaKernels(KernelRegistry& kernel_registry) { BuildKernelCreateInfo, BuildKernelCreateInfo, BuildKernelCreateInfo, + BuildKernelCreateInfo, + BuildKernelCreateInfo, + BuildKernelCreateInfo, #if defined(CUDA_VERSION) && CUDA_VERSION >= 11000 BuildKernelCreateInfo, BuildKernelCreateInfo, diff --git a/onnxruntime/core/providers/cuda/nn/batch_norm.cc b/onnxruntime/core/providers/cuda/nn/batch_norm.cc index 56d297429b..1a09ec8f45 100644 --- a/onnxruntime/core/providers/cuda/nn/batch_norm.cc +++ b/onnxruntime/core/providers/cuda/nn/batch_norm.cc @@ -21,10 +21,19 @@ namespace cuda { (*KernelDefBuilder::Create()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()), \ BatchNorm); \ + ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ + BatchNormalization, \ + kOnnxDomain, \ + 9, 13, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()) \ + .TypeConstraint("T", DataTypeImpl::GetTensorType()), \ + BatchNorm); \ ONNX_OPERATOR_TYPED_KERNEL_EX( \ BatchNormalization, \ kOnnxDomain, \ - 9, \ + 14, \ T, \ kCudaExecutionProvider, \ (*KernelDefBuilder::Create()) \ diff --git a/onnxruntime/core/providers/cuda/nn/batch_norm.h b/onnxruntime/core/providers/cuda/nn/batch_norm.h index 62241ef52d..e8ded6a571 100644 --- a/onnxruntime/core/providers/cuda/nn/batch_norm.h +++ b/onnxruntime/core/providers/cuda/nn/batch_norm.h @@ -34,6 +34,13 @@ class BatchNorm final : public CudaKernel { if (op_kernel_info.GetAttr("momentum", &tmp_momentum).IsOK()) { momentum_ = static_cast(tmp_momentum); } + + is_training_mode_ = (op_kernel_info.GetAttrOrDefault("training_mode", 0) == 1); + const auto& node = op_kernel_info.node(); + auto opset = node.SinceVersion(); + + // batch norm opset 14 is not implemented for training mode + ORT_ENFORCE(!(is_training_mode_ && opset==14), "Training mode does not support BN opset 14 yet."); } Status ComputeInternal(OpKernelContext* context) const override; @@ -43,6 +50,7 @@ class BatchNorm final : public CudaKernel { int64_t spatial_ = 1; // default as per spec cudnnBatchNormMode_t cudnn_batch_norm_mode_; double momentum_; + bool is_training_mode_ = 0; //default as per spec }; } // namespace cuda From 02c78a8aa8ec79653ca6c78ed351a6bc0e29cc64 Mon Sep 17 00:00:00 2001 From: Suffian Khan Date: Mon, 24 May 2021 11:48:44 -0700 Subject: [PATCH 21/28] test migration to rocm4.2 (#7800) --- .../github/azure-pipelines/templates/py-packaging-stage.yml | 2 +- .../ci_build/github/linux/docker/Dockerfile.manylinux2014_rocm | 2 +- .../ci_build/github/linux/docker/scripts/install_python_deps.sh | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/tools/ci_build/github/azure-pipelines/templates/py-packaging-stage.yml b/tools/ci_build/github/azure-pipelines/templates/py-packaging-stage.yml index 4c17c152a2..4b6db23260 100644 --- a/tools/ci_build/github/azure-pipelines/templates/py-packaging-stage.yml +++ b/tools/ci_build/github/azure-pipelines/templates/py-packaging-stage.yml @@ -285,7 +285,7 @@ stages: /onnxruntime_src/tools/ci_build/build.py \ --config Release \ --use_rocm \ - --rocm_version=4.1 \ + --rocm_version=4.2 \ --rocm_home=/opt/rocm \ --nccl_home=/opt/rocm \ --update \ diff --git a/tools/ci_build/github/linux/docker/Dockerfile.manylinux2014_rocm b/tools/ci_build/github/linux/docker/Dockerfile.manylinux2014_rocm index eb5cd9b3ce..00b38f7af0 100644 --- a/tools/ci_build/github/linux/docker/Dockerfile.manylinux2014_rocm +++ b/tools/ci_build/github/linux/docker/Dockerfile.manylinux2014_rocm @@ -1,4 +1,4 @@ -FROM rocm/pytorch:rocm4.1.1_centos7_py3.6_pytorch +FROM rocm/pytorch:rocm4.2_centos7_py3.6_pytorch #Build manylinux2014 docker image begin ENV AUDITWHEEL_ARCH x86_64 diff --git a/tools/ci_build/github/linux/docker/scripts/install_python_deps.sh b/tools/ci_build/github/linux/docker/scripts/install_python_deps.sh index 9c6bc7188d..149982de5d 100755 --- a/tools/ci_build/github/linux/docker/scripts/install_python_deps.sh +++ b/tools/ci_build/github/linux/docker/scripts/install_python_deps.sh @@ -58,7 +58,7 @@ if [ $DEVICE_TYPE = "gpu" ]; then ${PYTHON_EXE} -m pip install -r ${0/%install_python_deps.sh/training\/ortmodule\/stage2\/requirements.txt} else ${PYTHON_EXE} -m pip install \ - --pre -f https://download.pytorch.org/whl/nightly/rocm4.1/torch_nightly.html \ + --pre -f https://download.pytorch.org/whl/nightly/rocm4.2/torch_nightly.html \ torch torchvision torchtext ${PYTHON_EXE} -m pip install -r ${0/%install_python_deps.sh/training\/ortmodule\/stage1\/requirements-rocm.txt} ${PYTHON_EXE} -m pip install fairscale From 98007f0be6b9961bc8703b0ee5109fe71e0d17be Mon Sep 17 00:00:00 2001 From: Guoyu Wang <62914304+gwang-msft@users.noreply.github.com> Date: Mon, 24 May 2021 11:57:13 -0700 Subject: [PATCH 22/28] Fix typo in the ios packaging script (#7802) --- tools/ci_build/github/apple/build_ios_framework.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tools/ci_build/github/apple/build_ios_framework.py b/tools/ci_build/github/apple/build_ios_framework.py index e521025e50..cf0c8a1196 100644 --- a/tools/ci_build/github/apple/build_ios_framework.py +++ b/tools/ci_build/github/apple/build_ios_framework.py @@ -75,16 +75,16 @@ def _build_package(args): build_dir_current_arch, build_config, build_config + "-" + sysroot, 'onnxruntime.framework') ort_libs.append(os.path.join(framework_dir, 'onnxruntime')) - # We actually only need to define the info.plist and headers once since they are all the same + # We actually only need to define the Info.plist and headers once since they are all the same if not info_plist_path: - info_plist_path = os.path.join(build_dir_current_arch, build_config, 'info.plist') + info_plist_path = os.path.join(build_dir_current_arch, build_config, 'Info.plist') headers = glob.glob(os.path.join(framework_dir, 'Headers', '*.h')) # manually create the fat framework framework_dir = os.path.join(build_dir, 'framework_out', 'onnxruntime.framework') pathlib.Path(framework_dir).mkdir(parents=True, exist_ok=True) - # copy the header files and info.plist + # copy the header files and Info.plist shutil.copy(info_plist_path, framework_dir) header_dir = os.path.join(framework_dir, 'Headers') pathlib.Path(header_dir).mkdir(parents=True, exist_ok=True) From f487f6be2577d8b69a1378ff79f71b87c270f3eb Mon Sep 17 00:00:00 2001 From: Pranav Prakash Date: Mon, 24 May 2021 13:19:14 -0700 Subject: [PATCH 23/28] Add int8/int32 Relu for Opset 14 (#7536) * Add int8 Relu for Opset 14 * update kernel def hashes and exclude TensorRT for int8 relu Co-authored-by: Hari Vallabhaneni --- .../core/providers/cpu/activation/activations.cc | 2 ++ .../core/providers/cpu/cpu_execution_provider.cc | 6 ++++++ .../providers/cpu/activation/activation_op_test.cc | 6 ++++++ .../test/providers/cpu/activation/activation_op_test.h | 2 ++ .../test/testdata/kernel_def_hashes/onnx.cpu.json | 10 +++++++++- 5 files changed, 25 insertions(+), 1 deletion(-) diff --git a/onnxruntime/core/providers/cpu/activation/activations.cc b/onnxruntime/core/providers/cpu/activation/activations.cc index db14b7c677..2aef903066 100644 --- a/onnxruntime/core/providers/cpu/activation/activations.cc +++ b/onnxruntime/core/providers/cpu/activation/activations.cc @@ -49,6 +49,8 @@ REGISTER_VERSIONED_UNARY_ELEMENTWISE_TYPED_KERNEL(Relu, 13, 13, float); REGISTER_VERSIONED_UNARY_ELEMENTWISE_TYPED_KERNEL(Relu, 13, 13, double); REGISTER_UNARY_ELEMENTWISE_TYPED_KERNEL(Relu, 14, float); REGISTER_UNARY_ELEMENTWISE_TYPED_KERNEL(Relu, 14, double); +REGISTER_UNARY_ELEMENTWISE_TYPED_KERNEL(Relu, 14, int8_t); +REGISTER_UNARY_ELEMENTWISE_TYPED_KERNEL(Relu, 14, int32_t); REGISTER_UNARY_ELEMENTWISE_KERNEL(Selu, 6); REGISTER_VERSIONED_UNARY_ELEMENTWISE_TYPED_KERNEL(Sigmoid, 6, 12, float); REGISTER_VERSIONED_UNARY_ELEMENTWISE_TYPED_KERNEL(Sigmoid, 6, 12, double); diff --git a/onnxruntime/core/providers/cpu/cpu_execution_provider.cc b/onnxruntime/core/providers/cpu/cpu_execution_provider.cc index 11ec495b17..2186d7c2f2 100644 --- a/onnxruntime/core/providers/cpu/cpu_execution_provider.cc +++ b/onnxruntime/core/providers/cpu/cpu_execution_provider.cc @@ -665,6 +665,8 @@ class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCpuExecutionProvider, kOnnxDomain, class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCpuExecutionProvider, kOnnxDomain, 14, int64_t, CumSum); class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCpuExecutionProvider, kOnnxDomain, 14, float, Relu); class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCpuExecutionProvider, kOnnxDomain, 14, double, Relu); +class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCpuExecutionProvider, kOnnxDomain, 14, int8_t, Relu); +class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCpuExecutionProvider, kOnnxDomain, 14, int32_t, Relu); class ONNX_OPERATOR_KERNEL_CLASS_NAME(kCpuExecutionProvider, kOnnxDomain, 14, Trilu); class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCpuExecutionProvider, kOnnxDomain, 14, float, Add); class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCpuExecutionProvider, kOnnxDomain, 14, double, Add); @@ -1777,6 +1779,10 @@ Status RegisterOnnxOperatorKernels(KernelRegistry& kernel_registry) { Relu)>, BuildKernelCreateInfo, + BuildKernelCreateInfo, + BuildKernelCreateInfo, BuildKernelCreateInfo, BuildKernelCreateInfo, BuildKernelCreateInfo, diff --git a/onnxruntime/test/providers/cpu/activation/activation_op_test.cc b/onnxruntime/test/providers/cpu/activation/activation_op_test.cc index d49726619b..6167905916 100644 --- a/onnxruntime/test/providers/cpu/activation/activation_op_test.cc +++ b/onnxruntime/test/providers/cpu/activation/activation_op_test.cc @@ -100,6 +100,12 @@ TEST_F(ActivationOpTest, Relu) { TestActivationOp("Relu", input_values_double, [](double x) { return std::max(x, 0.0); }); + TestActivationOp("Relu", + input_values_int8, + [](int8_t x) { return std::max(x, static_cast(0)); }, + {}, + /*is_tensorrt_supported=*/ false, + /*opset_version= */ 14); } TEST_F(ActivationOpTest, Elu) { diff --git a/onnxruntime/test/providers/cpu/activation/activation_op_test.h b/onnxruntime/test/providers/cpu/activation/activation_op_test.h index 5cedc61708..e1078a3946 100644 --- a/onnxruntime/test/providers/cpu/activation/activation_op_test.h +++ b/onnxruntime/test/providers/cpu/activation/activation_op_test.h @@ -82,6 +82,8 @@ class ActivationOpTest : public ::testing::Test { 100.0, -100.0, 1000.0, -1000.0, // input values that leads to exp() overflow DBL_MIN, DBL_MIN / 10, -DBL_MIN / 10, // min, denorm, -denorm DBL_MAX, -DBL_MAX, std::numeric_limits::infinity()}}; // max, -max, inf + std::vector> input_values_int8{{-1, -5, 0, 1, 5, 100, -100, // normal input values for activation + std::numeric_limits::min(), std::numeric_limits::max()}}; // min, max void SetUp() override { float low = -1.0f, high = 1.0f; diff --git a/onnxruntime/test/testdata/kernel_def_hashes/onnx.cpu.json b/onnxruntime/test/testdata/kernel_def_hashes/onnx.cpu.json index 2aa13288f8..7a42831764 100644 --- a/onnxruntime/test/testdata/kernel_def_hashes/onnx.cpu.json +++ b/onnxruntime/test/testdata/kernel_def_hashes/onnx.cpu.json @@ -1999,6 +1999,14 @@ "Relu ai.onnx CPUExecutionProvider", 17992752176767437224 ], + [ + "Relu ai.onnx CPUExecutionProvider", + 800813007613791272 + ], + [ + "Relu ai.onnx CPUExecutionProvider", + 8895401642537447048 + ], [ "Reshape ai.onnx CPUExecutionProvider", 15585679549439997664 @@ -2523,4 +2531,4 @@ "Xor ai.onnx CPUExecutionProvider", 14631049987911195736 ] -] \ No newline at end of file +] From a41255c280b06a7de748f211af42abccc724a225 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Xavier=20Dupr=C3=A9?= Date: Tue, 25 May 2021 01:41:12 +0200 Subject: [PATCH 24/28] Fix performance regression in Reduce operators introduced by PR #7206 (#7719) * improves ArgMin implementation * update parallelization cost * choose former implementation for KRK case, when K=1 * improves unit test --- .../providers/cpu/reduction/reduction_ops.cc | 189 +++++++++++------- .../providers/cpu/reduction/reduction_ops.h | 20 +- .../cpu/reduction/reduction_ops_test.cc | 59 +++--- 3 files changed, 161 insertions(+), 107 deletions(-) diff --git a/onnxruntime/core/providers/cpu/reduction/reduction_ops.cc b/onnxruntime/core/providers/cpu/reduction/reduction_ops.cc index ac83870ff6..d8426c3f78 100644 --- a/onnxruntime/core/providers/cpu/reduction/reduction_ops.cc +++ b/onnxruntime/core/providers/cpu/reduction/reduction_ops.cc @@ -277,10 +277,10 @@ void ReduceAggregatorBase::FastReduceKRK(const Tensor&, const std::vector(n_col * n_row * element_size), +TensorOpCost ParallelReduceFastCost(int64_t n_row, int64_t n_col, int64_t element_size, int n_ops) { + return TensorOpCost{static_cast(n_row * n_col * element_size), static_cast(n_row * element_size), - static_cast(n_col * n_row * element_size * 2)}; + static_cast(n_row * n_col * element_size * n_ops)}; } void NoTransposePrepareForReduce(const TensorShape& new_input_shape, @@ -384,6 +384,15 @@ void ValidateNoTransposeReduce(int64_t count) { ORT_ENFORCE(count == 1, "Reduction on all axes, output size should be 1."); } +template +struct ParallelizedData { + int64_t denominator; + int64_t loop_size; + ResultsNoTransposePrepareForReduce* last_results; + const typename AGG::input_type* from_data; + typename AGG::value_type* to_data; +}; + template void NoTransposeReduce1Loop(Tensor* output, const TensorShape& new_input_shape, const Tensor& input, const std::vector& reduced_axes, concurrency::ThreadPool* tp, @@ -406,35 +415,47 @@ void NoTransposeReduce1Loop(Tensor* output, const TensorShape& new_input_shape, return; } last_results.ValidateNotEmpty(); - int64_t denominator = last_results.last_loop_red_size * last_results.projected_index.size(); - auto fn = [&](std::ptrdiff_t first, std::ptrdiff_t end) { - int64_t loop; + ParallelizedData data; + data.denominator = last_results.last_loop_red_size * last_results.projected_index.size(); + data.loop_size = last_results.last_loop_red_size * last_results.last_loop_red_inc; + data.last_results = &last_results; + data.from_data = from_data; + data.to_data = to_data; + + auto fn = [&data](std::ptrdiff_t first, std::ptrdiff_t end) { const typename AGG::input_type* loop_red_ptr; - const typename AGG::input_type* loop_red_ptr_end; - int64_t current_index = first * last_results.last_loop_size; - for (int64_t main_index = first; main_index < end; ++main_index) { - for (loop = 0; loop < last_results.last_loop_size; ++loop, ++current_index) { - int64_t origin = last_results.unprojected_index[main_index] + loop * last_results.last_loop_inc; - AGG accumulator(denominator, from_data[origin + last_results.projected_index[0]]); - for (auto it = last_results.projected_index.begin(); it != last_results.projected_index.end(); ++it) { - loop_red_ptr = from_data + (origin + *it); - loop_red_ptr_end = loop_red_ptr + last_results.last_loop_red_size * last_results.last_loop_red_inc; - for (; loop_red_ptr != loop_red_ptr_end; loop_red_ptr += last_results.last_loop_red_inc) { - accumulator.update(*loop_red_ptr); - } + const ResultsNoTransposePrepareForReduce& last_results = *data.last_results; + int64_t main_index = first / last_results.last_loop_size; + int64_t loop = first % last_results.last_loop_size; + int64_t origin = last_results.unprojected_index[main_index] + loop * last_results.last_loop_inc; + for (int64_t main_index_last_loop = first; main_index_last_loop < end; ++main_index_last_loop) { + AGG accumulator(data.denominator, data.from_data[origin + last_results.projected_index[0]]); + for (auto it = last_results.projected_index.begin(); it != last_results.projected_index.end(); ++it) { + loop_red_ptr = data.from_data + (origin + *it); + for (int64_t red = 0; red < data.loop_size; red += last_results.last_loop_red_inc) { + accumulator.update(loop_red_ptr[red]); } - to_data[current_index] = accumulator.get_value(); + } + data.to_data[main_index_last_loop] = accumulator.get_value(); + + ++loop; + if (loop >= last_results.last_loop_size) { + loop = 0; + ++main_index; + if (main_index < static_cast(last_results.unprojected_index.size())) { + origin = last_results.unprojected_index[main_index]; + } + } else { + origin += last_results.last_loop_inc; } } }; - auto cost = TensorOpCost{(double)(last_results.projected_index.size() * sizeof(typename AGG::input_type) * - last_results.last_loop_size * last_results.last_loop_red_size), - (double)last_results.last_loop_size * last_results.last_loop_red_size, - (double)last_results.projected_index.size() * last_results.last_loop_size * - last_results.last_loop_red_size}; - concurrency::ThreadPool::TryParallelFor(tp, count / last_results.last_loop_size, cost, fn); + auto cost = ParallelReduceFastCost(1, + last_results.projected_index.size() * last_results.last_loop_red_size, + sizeof(typename AGG::input_type), 6); + concurrency::ThreadPool::TryParallelFor(tp, count, cost, fn); } template @@ -459,42 +480,54 @@ void NoTransposeReduce2Loops(Tensor* output, const TensorShape& new_input_shape, return; } last_results.ValidateNotEmpty(); - int64_t denominator = last_results.last_loop_red_size * last_results.projected_index.size(); + + ParallelizedData data; + data.denominator = last_results.last_loop_red_size * last_results.projected_index.size(); + data.loop_size = last_results.last_loop_red_size * last_results.last_loop_red_inc; + data.last_results = &last_results; + data.from_data = from_data; + data.to_data = to_data; auto fn = [&](std::ptrdiff_t first, std::ptrdiff_t end) { - int64_t loop; const typename AGG::input_type* loop_red_ptr; - const typename AGG::input_type* loop_red_ptr_end; - int64_t current_index = first * last_results.last_loop_size; - for (int64_t main_index = first; main_index < end; ++main_index) { - for (loop = 0; loop < last_results.last_loop_size; ++loop, ++current_index) { - int64_t origin = last_results.unprojected_index[main_index] + loop * last_results.last_loop_inc; - AGG accumulator(denominator, from_data[origin + last_results.projected_index[0]]); - for (auto it = last_results.projected_index.begin(); it != last_results.projected_index.end(); ++it) { - loop_red_ptr = from_data + (origin + *it); - loop_red_ptr_end = loop_red_ptr + last_results.last_loop_red_size * last_results.last_loop_red_inc; - for (; loop_red_ptr != loop_red_ptr_end; loop_red_ptr += last_results.last_loop_red_inc) { - accumulator.update0(*loop_red_ptr); - } + const ResultsNoTransposePrepareForReduce& last_results = *data.last_results; + int64_t main_index = first / last_results.last_loop_size; + int64_t loop = first % last_results.last_loop_size; + int64_t origin = last_results.unprojected_index[main_index] + loop * last_results.last_loop_inc; + for (int64_t main_index_last_loop = first; main_index_last_loop < end; ++main_index_last_loop) { + AGG accumulator(data.denominator, data.from_data[origin + last_results.projected_index[0]]); + for (auto it = last_results.projected_index.begin(); it != last_results.projected_index.end(); ++it) { + loop_red_ptr = data.from_data + (origin + *it); + for (int64_t red = 0; red < data.loop_size; red += last_results.last_loop_red_inc) { + accumulator.update0(loop_red_ptr[red]); } - for (auto it = last_results.projected_index.begin(); it != last_results.projected_index.end(); ++it) { - loop_red_ptr = from_data + (origin + *it); - loop_red_ptr_end = loop_red_ptr + last_results.last_loop_red_size * last_results.last_loop_red_inc; - for (; loop_red_ptr != loop_red_ptr_end; loop_red_ptr += last_results.last_loop_red_inc) { - accumulator.update(*loop_red_ptr); - } + } + + for (auto it = last_results.projected_index.begin(); it != last_results.projected_index.end(); ++it) { + loop_red_ptr = data.from_data + (origin + *it); + for (int64_t red = 0; red < data.loop_size; red += last_results.last_loop_red_inc) { + accumulator.update(loop_red_ptr[red]); } - to_data[current_index] = accumulator.get_value(); + } + data.to_data[main_index_last_loop] = accumulator.get_value(); + + ++loop; + if (loop >= last_results.last_loop_size) { + loop = 0; + ++main_index; + if (main_index < static_cast(last_results.unprojected_index.size())) { + origin = last_results.unprojected_index[main_index]; + } + } else { + origin += last_results.last_loop_inc; } } }; - auto cost = TensorOpCost{(double)(last_results.projected_index.size() * sizeof(typename AGG::input_type) * - last_results.last_loop_size * last_results.last_loop_red_size), - (double)last_results.last_loop_size * last_results.last_loop_red_size, - (double)last_results.projected_index.size() * last_results.last_loop_size * - last_results.last_loop_red_size * 2}; - concurrency::ThreadPool::TryParallelFor(tp, count / last_results.last_loop_size, cost, fn); + auto cost = ParallelReduceFastCost(1, + last_results.projected_index.size() * last_results.last_loop_red_size, + sizeof(typename AGG::input_type), 8); + concurrency::ThreadPool::TryParallelFor(tp, count, cost, fn); } void DropDimensions(const std::vector& input_shape, @@ -660,6 +693,7 @@ bool CommonFastReduceSwitch(OpKernelContext* ctx, fast_kind = OptimizeShapeForFastReduce( reduced_dims, input_axes.empty() ? axes_ : input_axes, fast_shape, output_shape, fast_axes, keepdims_, noop_with_empty_axes); + if (which_fast_reduce != FastReduceKind::kNone) { if (IsFastReduceKindAvailable(fast_kind, which_fast_reduce)) { Tensor* output = ctx->Output(0, output_shape); @@ -671,14 +705,25 @@ bool CommonFastReduceSwitch(OpKernelContext* ctx, } case FastReduceKind::kRK: { ValidateFastReduceRK(fast_shape, *output); - case_rk(*input, fast_shape, *output, ctx->GetOperatorThreadPool()); - return true; + if ((fast_shape[0] > concurrency::ThreadPool::DegreeOfParallelism(ctx->GetOperatorThreadPool()) * 16) && + (std::max(fast_shape[0], fast_shape[1]) > + concurrency::ThreadPool::DegreeOfParallelism(ctx->GetOperatorThreadPool()) * 256)) { + // See benchmarks in PR #7719. + case_rk(*input, fast_shape, *output, ctx->GetOperatorThreadPool()); + return true; + } else { + break; + } } - case FastReduceKind::kKRK: { + case FastReduceKind::kKRK: ValidateFastReduceKRK(fast_shape, *output); - case_krk(*input, fast_shape, *output, ctx->GetOperatorThreadPool()); - return true; - } + if (fast_shape[0] >= std::max(2, concurrency::ThreadPool::DegreeOfParallelism(ctx->GetOperatorThreadPool()))) { + // See benchmarks in PR #7719. + case_krk(*input, fast_shape, *output, ctx->GetOperatorThreadPool()); + return true; + } else { + break; + } case FastReduceKind::kR: case FastReduceKind::kK: case FastReduceKind::kNone: @@ -700,7 +745,8 @@ bool CommonFastReduce(OpKernelContext* ctx, std::vector& fast_shape, std::vector& output_shape, std::vector& fast_axes) { - return CommonFastReduceSwitch(ctx, axes_, keepdims_, noop_with_empty_axes, fast_kind, fast_shape, output_shape, fast_axes, + return CommonFastReduceSwitch(ctx, axes_, keepdims_, noop_with_empty_axes, + fast_kind, fast_shape, output_shape, fast_axes, AGG::WhichFastReduce(), &AGG::FastReduceKR, &AGG::FastReduceRK, &AGG::FastReduceKRK); } @@ -869,16 +915,25 @@ std::unique_ptr ReduceSum::Impl(const Tensor& input, const std::vecto ReduceAggregatorSum::FastReduceKR(input, fast_shape, *output, tp); return output; } - case FastReduceKind::kRK: { + case FastReduceKind::kRK: ValidateFastReduceRK(fast_shape, *output); - ReduceAggregatorSum::FastReduceRK(input, fast_shape, *output, tp); - return output; - } - case FastReduceKind::kKRK: { + if (std::max(fast_shape[0], fast_shape[1]) > + concurrency::ThreadPool::DegreeOfParallelism(tp) * 256) { + // See benchmarks in PR #7719. + ReduceAggregatorSum::FastReduceRK(input, fast_shape, *output, tp); + return output; + } else { + break; + } + case FastReduceKind::kKRK: ValidateFastReduceKRK(fast_shape, *output); - ReduceAggregatorSum::FastReduceKRK(input, fast_shape, *output, tp); - return output; - } + if (fast_shape[0] >= std::max(2, concurrency::ThreadPool::DegreeOfParallelism(tp))) { + // See benchmarks in PR #7719. + ReduceAggregatorSum::FastReduceKRK(input, fast_shape, *output, tp); + return output; + } else { + break; + } case FastReduceKind::kR: case FastReduceKind::kK: case FastReduceKind::kNone: diff --git a/onnxruntime/core/providers/cpu/reduction/reduction_ops.h b/onnxruntime/core/providers/cpu/reduction/reduction_ops.h index 08f1c37863..a809311af6 100644 --- a/onnxruntime/core/providers/cpu/reduction/reduction_ops.h +++ b/onnxruntime/core/providers/cpu/reduction/reduction_ops.h @@ -37,7 +37,7 @@ bool operator!=(FastReduceKind a, FastReduceKind b); bool IsFastReduceKindAvailable(FastReduceKind scenario, FastReduceKind available); /* Evaluate the cost of parallelized FastReduce implementations. */ -TensorOpCost ParallelReduceFastCost(int64_t n_row, int64_t n_col, int64_t element_size); +TensorOpCost ParallelReduceFastCost(int64_t n_row, int64_t n_col, int64_t element_size, int n_ops); /** This only improves reduce function when reduced axes are contiguous: @@ -192,7 +192,7 @@ class ReduceAggregatorSum : public ReduceAggregator { T* out = output.MutableData(); int64_t stridei = fast_shape[1]; concurrency::ThreadPool::TryParallelFor( - tp, fast_shape[0], ParallelReduceFastCost(1, stridei, sizeof(T)), + tp, fast_shape[0], ParallelReduceFastCost(1, stridei, sizeof(T), 6), [data, stridei, out](ptrdiff_t first, ptrdiff_t last) { for (ptrdiff_t d = first; d < last; ++d) { out[d] = ConstEigenVectorArrayMap(data + d * stridei, stridei).sum(); @@ -209,7 +209,7 @@ class ReduceAggregatorSum : public ReduceAggregator { int64_t n_rows = fast_shape[0]; memcpy(out, data, N * sizeof(T)); concurrency::ThreadPool::TryParallelFor( - tp, N, ParallelReduceFastCost(1, n_rows, sizeof(T)), + tp, N, ParallelReduceFastCost(1, n_rows, sizeof(T), 6), [data, out, N, n_rows](ptrdiff_t begin, ptrdiff_t end) { for (int64_t row = 1; row < n_rows; ++row) { EigenVectorArrayMap(out + begin, end - begin) += ConstEigenVectorArrayMap( @@ -227,7 +227,7 @@ class ReduceAggregatorSum : public ReduceAggregator { T* out = output.MutableData(); std::vector one(fast_shape[1], 1); concurrency::ThreadPool::TryParallelFor( - tp, fast_shape[0], ParallelReduceFastCost(fast_shape[1], fast_shape[2], sizeof(T)), + tp, fast_shape[0], ParallelReduceFastCost(fast_shape[1], fast_shape[2], sizeof(T), 6), [one, data, fast_shape, stridei, strideo, out, N](ptrdiff_t begin, ptrdiff_t last) { for (ptrdiff_t d = begin; d < last; ++d) { math::MatMul(1, N, fast_shape[1], one.data(), data + stridei * d, out + strideo * d, nullptr); @@ -318,7 +318,7 @@ class ReduceAggregatorMax : public ReduceAggregator { T* out = output.MutableData(); int64_t stridei = fast_shape[1]; concurrency::ThreadPool::TryParallelFor( - tp, fast_shape[0], ParallelReduceFastCost(1, stridei, sizeof(T)), + tp, fast_shape[0], ParallelReduceFastCost(1, stridei, sizeof(T), 6), [data, stridei, out](std::ptrdiff_t first, std::ptrdiff_t last) { EigenVectorMap(out + first, last - first) = ConstEigenMatrixMap( data + first * stridei, stridei, last - first) @@ -336,7 +336,7 @@ class ReduceAggregatorMax : public ReduceAggregator { memcpy(out, data, N * sizeof(T)); concurrency::ThreadPool::TryParallelFor( - tp, N, ParallelReduceFastCost(1, n_rows, sizeof(T)), + tp, N, ParallelReduceFastCost(1, n_rows, sizeof(T), 6), [data, out, N, n_rows](ptrdiff_t begin, ptrdiff_t end) { const T* p; for (int64_t row = 1; row < n_rows; ++row) { @@ -355,7 +355,7 @@ class ReduceAggregatorMax : public ReduceAggregator { int64_t stridei = fast_shape[1] * fast_shape[2]; int64_t strideo = fast_shape[2]; concurrency::ThreadPool::TryParallelFor( - tp, fast_shape[0], ParallelReduceFastCost(fast_shape[1], fast_shape[2], sizeof(T)), + tp, fast_shape[0], ParallelReduceFastCost(fast_shape[1], fast_shape[2], sizeof(T), 6), [data, fast_shape, stridei, strideo, out](ptrdiff_t begin, ptrdiff_t end) { for (ptrdiff_t j = begin; j < end; ++j) { EigenVectorMap(out + j * strideo, strideo) = @@ -477,7 +477,7 @@ class ReduceAggregatorMin : public ReduceAggregator { T* out = output.MutableData(); int64_t stridei = fast_shape[1]; concurrency::ThreadPool::TryParallelFor( - tp, fast_shape[0], ParallelReduceFastCost(1, stridei, sizeof(T)), + tp, fast_shape[0], ParallelReduceFastCost(1, stridei, sizeof(T), 6), [data, stridei, out](std::ptrdiff_t first, std::ptrdiff_t last) { EigenVectorMap(out + first, last - first) = ConstEigenMatrixMap( data + first * stridei, stridei, last - first) @@ -495,7 +495,7 @@ class ReduceAggregatorMin : public ReduceAggregator { memcpy(out, data, N * sizeof(T)); concurrency::ThreadPool::TryParallelFor( - tp, N, ParallelReduceFastCost(1, n_rows, sizeof(T)), + tp, N, ParallelReduceFastCost(1, n_rows, sizeof(T), 6), [data, out, N, n_rows](ptrdiff_t begin, ptrdiff_t end) { const T* p; for (int64_t row = 1; row < n_rows; ++row) { @@ -514,7 +514,7 @@ class ReduceAggregatorMin : public ReduceAggregator { int64_t stridei = fast_shape[1] * fast_shape[2]; int64_t strideo = fast_shape[2]; concurrency::ThreadPool::TryParallelFor( - tp, fast_shape[0], ParallelReduceFastCost(fast_shape[1], fast_shape[2], sizeof(T)), + tp, fast_shape[0], ParallelReduceFastCost(fast_shape[1], fast_shape[2], sizeof(T), 6), [data, fast_shape, stridei, strideo, out](ptrdiff_t begin, ptrdiff_t end) { for (ptrdiff_t j = begin; j < end; ++j) { EigenVectorMap(out + j * strideo, strideo) = diff --git a/onnxruntime/test/providers/cpu/reduction/reduction_ops_test.cc b/onnxruntime/test/providers/cpu/reduction/reduction_ops_test.cc index 14785ed2e5..799412b42a 100644 --- a/onnxruntime/test/providers/cpu/reduction/reduction_ops_test.cc +++ b/onnxruntime/test/providers/cpu/reduction/reduction_ops_test.cc @@ -3137,16 +3137,15 @@ TEST(ReductionOpTest, ReduceMax_RK_parallel) { OpTester test("ReduceMax"); test.AddAttribute("axes", std::vector{0}); test.AddAttribute("keepdims", (int64_t)0); - std::vector in_data(128); + std::vector in_data(65536); for (size_t i = 0; i < in_data.size(); ++i) - in_data[i] = (float)i; - test.AddInput("data", {4, 32}, in_data); + in_data[i] = 1.f + (float)(i % 17) / 17.f; + test.AddInput("data", {2048, 32}, in_data); std::vector expected(32); for (size_t i = 0; i < expected.size(); ++i) { expected[i] = 0; - for (size_t j = 0; j < 4; ++j) { - if (in_data[i + j * expected.size()] > expected[i]) - expected[i] = in_data[i + j * expected.size()]; + for (size_t j = 0; j < 2048; ++j) { + expected[i] = std::max(expected[i], in_data[i + j * expected.size()]); } } test.AddOutput("reduced", {32}, expected); @@ -3465,16 +3464,15 @@ TEST(ReductionOpTest, ReduceMin_RK_parallel) { OpTester test("ReduceMin"); test.AddAttribute("axes", std::vector{0}); test.AddAttribute("keepdims", (int64_t)0); - std::vector in_data(128); + std::vector in_data(65536); for (size_t i = 0; i < in_data.size(); ++i) - in_data[i] = (float)i; - test.AddInput("data", {4, 32}, in_data); + in_data[i] = 1.f + (float)(i % 17) / 17.f; + test.AddInput("data", {2048, 32}, in_data); std::vector expected(32); for (size_t i = 0; i < expected.size(); ++i) { - expected[i] = 1000000000; - for (size_t j = 0; j < 4; ++j) { - if (in_data[i + j * expected.size()] < expected[i]) - expected[i] = in_data[i + j * expected.size()]; + expected[i] = 10.f; + for (size_t j = 0; j < 2048; ++j) { + expected[i] = std::min(expected[i], in_data[i + j * expected.size()]); } } test.AddOutput("reduced", {32}, expected); @@ -3690,14 +3688,14 @@ TEST(ReductionOpTest, ReduceSum_RK_parallel) { OpTester test("ReduceSum"); test.AddAttribute("axes", std::vector{0}); test.AddAttribute("keepdims", (int64_t)0); - std::vector in_data(128); + std::vector in_data(65536); for (size_t i = 0; i < in_data.size(); ++i) - in_data[i] = (float)i; - test.AddInput("data", {4, 32}, in_data); + in_data[i] = 1.f + (float)(i % 17) / 17.f; + test.AddInput("data", {2048, 32}, in_data); std::vector expected(32); for (size_t i = 0; i < expected.size(); ++i) { expected[i] = 0; - for (size_t j = 0; j < 4; ++j) { + for (size_t j = 0; j < 2048; ++j) { expected[i] += in_data[i + j * expected.size()]; } } @@ -3777,19 +3775,20 @@ TEST(ReductionOpTest, ReduceSum_KRK_parallel) { OpTester test("ReduceSum"); test.AddAttribute("axes", std::vector{1}); test.AddAttribute("keepdims", (int64_t)0); - test.AddInput("data", {4, 2, 2}, - {1.0f, 2.0f, - 3.0f, 4.0f, - - 5.0f, 6.0f, - 7.0f, 8.0f, - - 9.0f, 10.0f, - 11.0f, 12.0f, - - 13.0f, 14.0f, - 15.0f, 16.0f}); - test.AddOutput("reduced", {4, 2}, {4.f, 6.f, 12.f, 14.f, 20.f, 22.f, 28.f, 30.f}); + std::vector in_data(512); + for (size_t i = 0; i < in_data.size(); ++i) + in_data[i] = (float)i; + test.AddInput("data", {128, 2, 2}, in_data); + std::vector expected(256); + for (size_t i = 0; i < 128; ++i) { + for (size_t j = 0; j < 2; ++j) { + expected[i * 2 + j] = 0; + for (size_t k = 0; k < 2; ++k) { + expected[i * 2 + j] += in_data[i * 4 + k * 2 + j]; + } + } + } + test.AddOutput("reduced", {128, 2}, expected); test.Run(); } From cdd2129fed4c7583011338485143e641084366a0 Mon Sep 17 00:00:00 2001 From: Hariharan Seshadri Date: Mon, 24 May 2021 21:11:19 -0700 Subject: [PATCH 25/28] Fix CUDA Pad kernel registrations (#7813) --- .../providers/cuda/cuda_execution_provider.cc | 20 ++++++++++++------- onnxruntime/core/providers/cuda/tensor/pad.cc | 13 +++++++++++- .../providers/rocm/rocm_execution_provider.cc | 18 +++++++++++------ 3 files changed, 37 insertions(+), 14 deletions(-) diff --git a/onnxruntime/core/providers/cuda/cuda_execution_provider.cc b/onnxruntime/core/providers/cuda/cuda_execution_provider.cc index 0ad7015519..19a18d0dc9 100644 --- a/onnxruntime/core/providers/cuda/cuda_execution_provider.cc +++ b/onnxruntime/core/providers/cuda/cuda_execution_provider.cc @@ -789,9 +789,9 @@ class ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kO class ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, 11, 12, int32_t, Resize); class ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, 11, 12, uint8_t, Resize); class ONNX_OPERATOR_VERSIONED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, 11, 11, Clip); -class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, 11, float, Pad); -class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, 11, double, Pad); -class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, 11, MLFloat16, Pad); +class ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, 11, 12, float, Pad); +class ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, 11, 12, double, Pad); +class ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, 11, 12, MLFloat16, Pad); class ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, 11, 12, bool, Equal); class ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, 11, 12, int32_t, Equal); class ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, 11, 12, int64_t, Equal); @@ -1052,6 +1052,9 @@ class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, 13, MLFloat16, LRN); class ONNX_OPERATOR_VERSIONED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, 13, 13, Identity); class ONNX_OPERATOR_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, 13, ScatterND); +class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, 13, float, Pad); +class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, 13, double, Pad); +class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, 13, MLFloat16, Pad); #if defined(CUDA_VERSION) && CUDA_VERSION >= 11000 class ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_CLASS_NAME(kCudaExecutionProvider, kOnnxDomain, 13, 13, BFloat16, Add); @@ -1605,9 +1608,9 @@ static Status RegisterCudaKernels(KernelRegistry& kernel_registry) { BuildKernelCreateInfo, BuildKernelCreateInfo, BuildKernelCreateInfo, - BuildKernelCreateInfo, - BuildKernelCreateInfo, - BuildKernelCreateInfo, + BuildKernelCreateInfo, + BuildKernelCreateInfo, + BuildKernelCreateInfo, BuildKernelCreateInfo, BuildKernelCreateInfo, BuildKernelCreateInfo, @@ -1864,6 +1867,9 @@ static Status RegisterCudaKernels(KernelRegistry& kernel_registry) { BuildKernelCreateInfo, BuildKernelCreateInfo, BuildKernelCreateInfo, + BuildKernelCreateInfo, + BuildKernelCreateInfo, + BuildKernelCreateInfo, #if defined(CUDA_VERSION) && CUDA_VERSION >= 11000 BuildKernelCreateInfo, BuildKernelCreateInfo, @@ -1925,7 +1931,7 @@ static Status RegisterCudaKernels(KernelRegistry& kernel_registry) { BuildKernelCreateInfo, BuildKernelCreateInfo, BuildKernelCreateInfo, - #if defined(CUDA_VERSION) && CUDA_VERSION >= 11000 +#if defined(CUDA_VERSION) && CUDA_VERSION >= 11000 BuildKernelCreateInfo, BuildKernelCreateInfo, BuildKernelCreateInfo, diff --git a/onnxruntime/core/providers/cuda/tensor/pad.cc b/onnxruntime/core/providers/cuda/tensor/pad.cc index f76f209d21..26f5f975ce 100644 --- a/onnxruntime/core/providers/cuda/tensor/pad.cc +++ b/onnxruntime/core/providers/cuda/tensor/pad.cc @@ -18,10 +18,21 @@ namespace cuda { (*KernelDefBuilder::Create()) \ .TypeConstraint("T", DataTypeImpl::GetTensorType()), \ Pad); \ + ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_EX( \ + Pad, \ + kOnnxDomain, \ + 11, 12, \ + T, \ + kCudaExecutionProvider, \ + (*KernelDefBuilder::Create()) \ + .InputMemoryType(OrtMemTypeCPUInput, 1) \ + .InputMemoryType(OrtMemTypeCPUInput, 2) \ + .TypeConstraint("T", DataTypeImpl::GetTensorType()), \ + Pad); \ ONNX_OPERATOR_TYPED_KERNEL_EX( \ Pad, \ kOnnxDomain, \ - 11, \ + 13, \ T, \ kCudaExecutionProvider, \ (*KernelDefBuilder::Create()) \ diff --git a/onnxruntime/core/providers/rocm/rocm_execution_provider.cc b/onnxruntime/core/providers/rocm/rocm_execution_provider.cc index 2805cb4d4e..ba74470fbe 100644 --- a/onnxruntime/core/providers/rocm/rocm_execution_provider.cc +++ b/onnxruntime/core/providers/rocm/rocm_execution_provider.cc @@ -767,9 +767,9 @@ class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kRocmExecutionProvider, kOnnxDomain, class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kRocmExecutionProvider, kOnnxDomain, 11, int32_t, Resize); class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kRocmExecutionProvider, kOnnxDomain, 11, uint8_t, Resize); class ONNX_OPERATOR_VERSIONED_KERNEL_CLASS_NAME(kRocmExecutionProvider, kOnnxDomain, 11, 11, Clip); -class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kRocmExecutionProvider, kOnnxDomain, 11, float, Pad); -class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kRocmExecutionProvider, kOnnxDomain, 11, double, Pad); -class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kRocmExecutionProvider, kOnnxDomain, 11, MLFloat16, Pad); +class ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_CLASS_NAME(kRocmExecutionProvider, kOnnxDomain, 11, 12, float, Pad); +class ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_CLASS_NAME(kRocmExecutionProvider, kOnnxDomain, 11, 12, double, Pad); +class ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_CLASS_NAME(kRocmExecutionProvider, kOnnxDomain, 11, 12, MLFloat16, Pad); class ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_CLASS_NAME(kRocmExecutionProvider, kOnnxDomain, 11, 12, bool, Equal); class ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_CLASS_NAME(kRocmExecutionProvider, kOnnxDomain, 11, 12, int32_t, Equal); class ONNX_OPERATOR_VERSIONED_TYPED_KERNEL_CLASS_NAME(kRocmExecutionProvider, kOnnxDomain, 11, 12, int64_t, Equal); @@ -1017,6 +1017,9 @@ class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kRocmExecutionProvider, kOnnxDomain, class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kRocmExecutionProvider, kOnnxDomain, 13, MLFloat16, LRN); class ONNX_OPERATOR_VERSIONED_KERNEL_CLASS_NAME(kRocmExecutionProvider, kOnnxDomain, 13, 13, Identity); class ONNX_OPERATOR_KERNEL_CLASS_NAME(kRocmExecutionProvider, kOnnxDomain, 13, ScatterND); +class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kRocmExecutionProvider, kOnnxDomain, 13, float, Pad); +class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kRocmExecutionProvider, kOnnxDomain, 13, double, Pad); +class ONNX_OPERATOR_TYPED_KERNEL_CLASS_NAME(kRocmExecutionProvider, kOnnxDomain, 13, MLFloat16, Pad); template <> KernelCreateInfo BuildKernelCreateInfo() { @@ -1496,9 +1499,9 @@ static Status RegisterRocmKernels(KernelRegistry& kernel_registry) { // BuildKernelCreateInfo, // BuildKernelCreateInfo, BuildKernelCreateInfo, - BuildKernelCreateInfo, - BuildKernelCreateInfo, - BuildKernelCreateInfo, + BuildKernelCreateInfo, + BuildKernelCreateInfo, + BuildKernelCreateInfo, BuildKernelCreateInfo, BuildKernelCreateInfo, BuildKernelCreateInfo, @@ -1742,6 +1745,9 @@ static Status RegisterRocmKernels(KernelRegistry& kernel_registry) { // BuildKernelCreateInfo, BuildKernelCreateInfo, BuildKernelCreateInfo, + BuildKernelCreateInfo, + BuildKernelCreateInfo, + BuildKernelCreateInfo, }; for (auto& function_table_entry : function_table) { From 93c8e29782138b71a9e0605745192e92c0613f2e Mon Sep 17 00:00:00 2001 From: Changming Sun Date: Tue, 25 May 2021 08:26:01 -0700 Subject: [PATCH 26/28] Improve code coverage report (#7770) --- ...ndroid-x86_64-crosscompile-ci-pipeline.yml | 2 +- .../azure-pipelines/post-merge-jobs.yml | 16 +++++--- .../post_code_coverage_to_dashboard.py | 41 ++++++++++--------- 3 files changed, 33 insertions(+), 26 deletions(-) diff --git a/tools/ci_build/github/azure-pipelines/android-x86_64-crosscompile-ci-pipeline.yml b/tools/ci_build/github/azure-pipelines/android-x86_64-crosscompile-ci-pipeline.yml index e593931cd0..247c37b44c 100644 --- a/tools/ci_build/github/azure-pipelines/android-x86_64-crosscompile-ci-pipeline.yml +++ b/tools/ci_build/github/azure-pipelines/android-x86_64-crosscompile-ci-pipeline.yml @@ -101,6 +101,6 @@ jobs: --commit_hash=$(Build.SourceVersion) --report_url="https://dev.azure.com/onnxruntime/onnxruntime/_build/results?buildId=$(Build.BuildId)" --report_file="$(Build.BinariesDirectory)/coverage_rpt.txt" - --build_config="{\"os\":\"android\", \"arch\":\"x86_64\", \"config\":\"nnapi\"}" + --branch $(Build.SourceBranchName) --arch $(Agent.OSArchitecture) --os "android" --build_config nnapi env: DASHBOARD_MYSQL_ORT_PASSWORD: $(dashboard-mysql-ort-password) diff --git a/tools/ci_build/github/azure-pipelines/post-merge-jobs.yml b/tools/ci_build/github/azure-pipelines/post-merge-jobs.yml index 3442e174c6..0c4c0aaee6 100644 --- a/tools/ci_build/github/azure-pipelines/post-merge-jobs.yml +++ b/tools/ci_build/github/azure-pipelines/post-merge-jobs.yml @@ -24,22 +24,26 @@ jobs: #Merge the mulitple prof data into a single indexed profile data file llvm-profdata merge -sparse -o ort.profdata *.profraw #Create coverage report, output the result to 'report.json' - llvm-cov export -summary-only -instr-profile=ort.profdata onnxruntime_test_all -object onnxruntime_mlas_test $(Build.SourcesDirectory)/include/onnxruntime $(Build.SourcesDirectory)/onnxruntime/core $(Build.SourcesDirectory)/onnxruntime/contrib_ops > $(Build.BinariesDirectory)/report.json - llvm-cov show -instr-profile=ort.profdata onnxruntime_test_all -object onnxruntime_mlas_test $(Build.SourcesDirectory)/include/onnxruntime $(Build.SourcesDirectory)/onnxruntime/core $(Build.SourcesDirectory)/onnxruntime/contrib_ops --format=html -output-dir=$(Build.ArtifactStagingDirectory) + llvm-cov export -summary-only -instr-profile=ort.profdata onnxruntime_test_all -object onnxruntime_mlas_test -object onnxruntime_api_tests_without_env -object onnx_test_runner -object onnxruntime_shared_lib_test -object onnxruntime_global_thread_pools_test -object onnxruntime_api_tests_without_env $(Build.SourcesDirectory)/include/onnxruntime $(Build.SourcesDirectory)/onnxruntime/core $(Build.SourcesDirectory)/onnxruntime/contrib_ops > $(Build.BinariesDirectory)/report.json + + llvm-cov show -instr-profile=ort.profdata onnxruntime_test_all -object onnxruntime_mlas_test -object onnxruntime_api_tests_without_env -object onnx_test_runner -object onnxruntime_shared_lib_test -object onnxruntime_global_thread_pools_test -object onnxruntime_api_tests_without_env $(Build.SourcesDirectory)/include/onnxruntime $(Build.SourcesDirectory)/onnxruntime/core $(Build.SourcesDirectory)/onnxruntime/contrib_ops --format=html -output-dir=$(Build.ArtifactStagingDirectory) workingDirectory: $(Build.BinariesDirectory) + + - task: PublishPipelineArtifact@1 displayName: 'Publish Pipeline Artifact' inputs: targetPath: '$(Build.ArtifactStagingDirectory)' artifact: html - - task: CmdLine@2 + - task: PythonScript@0 displayName: 'Post code coverage data to the Dashboard' inputs: - script: | - python3 $(Build.SourcesDirectory)/tools/ci_build/github/windows/post_code_coverage_to_dashboard.py --commit_hash=$(OnnxRuntimeGitCommitHash) --report_file="$(Build.BinariesDirectory)/report.json" --report_url="https://aiinfra.visualstudio.com/Lotus/_build/results?buildId=$(Build.BuildId)" + scriptPath: $(Build.SourcesDirectory)/tools/ci_build/github/windows/post_code_coverage_to_dashboard.py + arguments: --commit_hash=$(OnnxRuntimeGitCommitHash) --report_file="$(Build.BinariesDirectory)/report.json" --report_url="https://aiinfra.visualstudio.com/Lotus/_build/results?buildId=$(Build.BuildId)" --branch $(Build.SourceBranchName) --arch $(Agent.OSArchitecture) --os $(Agent.OS) --build_config default workingDirectory: $(Build.BinariesDirectory) + pythonInterpreter: '/usr/bin/python3' env: DASHBOARD_MYSQL_ORT_PASSWORD: $(dashboard-mysql-ort-password) @@ -69,4 +73,4 @@ jobs: azureSubscription: AIInfraBuildOnnxRuntimeOSS Destination: AzureBlob storage: onnxruntime - ContainerName: '$web' \ No newline at end of file + ContainerName: '$web' diff --git a/tools/ci_build/github/windows/post_code_coverage_to_dashboard.py b/tools/ci_build/github/windows/post_code_coverage_to_dashboard.py index 0c587dac1b..bd458dca33 100644 --- a/tools/ci_build/github/windows/post_code_coverage_to_dashboard.py +++ b/tools/ci_build/github/windows/post_code_coverage_to_dashboard.py @@ -7,7 +7,6 @@ # --report_url= # --report_file= # --commit_hash= -# --build_config= import argparse import mysql.connector @@ -19,11 +18,14 @@ import os def parse_arguments(): parser = argparse.ArgumentParser( description="ONNXRuntime test coverge report uploader for dashboard") - parser.add_argument("--report_url", help="URL to the LLVM json report") + parser.add_argument("--report_url", type=str, help="URL to the LLVM json report") parser.add_argument( - "--report_file", help="Path to the local JSON/TXT report", required=True) - parser.add_argument("--commit_hash", help="Full Git commit hash", required=True) - parser.add_argument("--build_config", help="Build configuration, os, arch and config, in JSON format") + "--report_file", type=str, help="Path to the local JSON/TXT report", required=True) + parser.add_argument("--commit_hash", type=str, help="Full Git commit hash", required=True) + parser.add_argument("--branch", type=str, help="Source code branch") + parser.add_argument("--os", type=str, help="Build configuration:os") + parser.add_argument("--arch", type=str, help="Build configuration:arch") + parser.add_argument("--build_config", type=str, help="Build configuration: build variants") return parser.parse_args() @@ -52,7 +54,7 @@ def parse_json_report(report_file): return result -def write_to_db(coverage_data, build_config, args): +def write_to_db(coverage_data, args): # connect to database cnx = mysql.connector.connect( @@ -74,26 +76,28 @@ def write_to_db(coverage_data, build_config, args): # insert current record insert_query = ('INSERT INTO onnxruntime.test_coverage ' '''(UploadTime, CommitId, Coverage, LinesCovered, TotalLines, OS, - Arch, BuildConfig, ReportURL) ''' - 'VALUES (Now(), "%s", %f, %d, %d, "%s", "%s", "%s", "%s") ' + Arch, BuildConfig, ReportURL, Branch) ''' + 'VALUES (Now(), "%s", %f, %d, %d, "%s", "%s", "%s", "%s", "%s") ' 'ON DUPLICATE KEY UPDATE ' '''UploadTime=Now(), Coverage=%f, LinesCovered=%d, TotalLines=%d, - OS="%s", Arch="%s", BuildConfig="%s", ReportURL="%s"; ''' + OS="%s", Arch="%s", BuildConfig="%s", ReportURL="%s", Branch="%s"; ''' ) % (args.commit_hash, coverage_data['coverage'], coverage_data['lines_covered'], coverage_data['lines_valid'], - build_config.get('os', 'win'), - build_config.get('arch', 'x64'), - build_config.get('config', 'default'), - args.report_url, + args.os.lower(), + args.arch.lower(), + args.build_config.lower(), + args.report_url.lower(), + args.branch.lower(), coverage_data['coverage'], coverage_data['lines_covered'], coverage_data['lines_valid'], - build_config.get('os', 'win'), - build_config.get('arch', 'x64'), - build_config.get('config', 'default'), - args.report_url + args.os.lower(), + args.arch.lower(), + args.build_config.lower(), + args.report_url.lower(), + args.branch.lower() ) cursor.execute(insert_query) cnx.commit() @@ -120,8 +124,7 @@ if __name__ == "__main__": else: raise ValueError("Only report extensions txt or json are accepted") - build_config = json.loads(args.build_config) if args.build_config else {} - write_to_db(coverage_data, build_config, args) + write_to_db(coverage_data, args) except BaseException as e: print(str(e)) sys.exit(1) From 9241d7639656cdf3e3eac84b1aab2165f8b3e1ba Mon Sep 17 00:00:00 2001 From: Ryan Hill <38674843+RyanUnderhill@users.noreply.github.com> Date: Tue, 25 May 2021 10:01:15 -0700 Subject: [PATCH 27/28] Remove unnecessary cuda libraries refernced in cmake (#7824) --- cmake/CMakeLists.txt | 1 - 1 file changed, 1 deletion(-) diff --git a/cmake/CMakeLists.txt b/cmake/CMakeLists.txt index 6f5db8cc40..35822e2107 100644 --- a/cmake/CMakeLists.txt +++ b/cmake/CMakeLists.txt @@ -1402,7 +1402,6 @@ if (onnxruntime_USE_CUDA) list(APPEND ONNXRUNTIME_CUDA_LIBRARIES cublas cudnn curand cufft) endif() - list(APPEND onnxruntime_EXTERNAL_LIBRARIES ${ONNXRUNTIME_CUDA_LIBRARIES}) if(NOT CMAKE_CUDA_ARCHITECTURES) if(CMAKE_LIBRARY_ARCHITECTURE STREQUAL "aarch64-linux-gnu") # Support for Jetson/Tegra ARM devices From ff655175ff2ecb32f24cdc43870a6c34667d73d6 Mon Sep 17 00:00:00 2001 From: ytaous <4484531+ytaous@users.noreply.github.com> Date: Tue, 25 May 2021 13:01:34 -0700 Subject: [PATCH 28/28] Eliminate no op node - add 0 (#7798) * eliminate add 0 * typo * rank check * fix build Co-authored-by: Ethan Tao --- .../core/optimizer/graph_transformer_utils.cc | 2 + .../core/optimizer/noop_elimination.cc | 96 ++++++++++++++++++ onnxruntime/core/optimizer/noop_elimination.h | 31 ++++++ .../test/optimizer/graph_transform_test.cc | 19 ++++ .../test/testdata/transform/noop-add.onnx | Bin 0 -> 688 bytes .../test/testdata/transform/noop-add.py | 71 +++++++++++++ .../core/optimizer/graph_transformer_utils.cc | 2 + 7 files changed, 221 insertions(+) create mode 100644 onnxruntime/core/optimizer/noop_elimination.cc create mode 100644 onnxruntime/core/optimizer/noop_elimination.h create mode 100644 onnxruntime/test/testdata/transform/noop-add.onnx create mode 100644 onnxruntime/test/testdata/transform/noop-add.py diff --git a/onnxruntime/core/optimizer/graph_transformer_utils.cc b/onnxruntime/core/optimizer/graph_transformer_utils.cc index 48be11c496..0330c9ef5e 100644 --- a/onnxruntime/core/optimizer/graph_transformer_utils.cc +++ b/onnxruntime/core/optimizer/graph_transformer_utils.cc @@ -32,6 +32,7 @@ #include "core/optimizer/matmul_scale_fusion.h" #include "core/optimizer/nchwc_transformer.h" #include "core/optimizer/nhwc_transformer.h" +#include "core/optimizer/noop_elimination.h" #include "core/optimizer/not_where_fusion.h" #include "core/optimizer/relu_clip_fusion.h" #include "core/optimizer/reshape_fusion.h" @@ -69,6 +70,7 @@ std::vector> GenerateRewriteRules( rules.push_back(std::make_unique()); rules.push_back(std::make_unique()); rules.push_back(std::make_unique()); + rules.push_back(std::make_unique()); rules.push_back(std::make_unique()); rules.push_back(std::make_unique()); rules.push_back(std::make_unique()); diff --git a/onnxruntime/core/optimizer/noop_elimination.cc b/onnxruntime/core/optimizer/noop_elimination.cc new file mode 100644 index 0000000000..1421ea7416 --- /dev/null +++ b/onnxruntime/core/optimizer/noop_elimination.cc @@ -0,0 +1,96 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +#include "core/optimizer/noop_elimination.h" + +#include "core/common/logging/logging.h" +#include "core/graph/graph_viewer.h" +#include "core/graph/op.h" +#include "core/graph/graph_utils.h" +#include "core/optimizer/initializer.h" +#include "core/optimizer/rewrite_rule.h" + +namespace onnxruntime { + +/** + Eliminate no op node - handling Add op for now + Add example: + + X 0 + \ / + Add + | + Y + */ +Status NoopElimination::Apply(Graph& graph, Node& node, RewriteRuleEffect& rule_effect, const logging::Logger&) const { + if (graph_utils::RemoveNode(graph, node)) { + rule_effect = RewriteRuleEffect::kRemovedCurrentNode; + } + + return Status::OK(); +} + +bool NoopElimination::SatisfyCondition(const Graph& graph, const Node& node, const logging::Logger& logger) const { + + bool input0_is_initializer = graph_utils::IsConstantInitializer(graph, node.InputDefs()[0]->Name()); + bool input1_is_initializer = graph_utils::IsConstantInitializer(graph, node.InputDefs()[1]->Name()); + + // reject if both or neither inputs are initializers for now + if (input0_is_initializer == input1_is_initializer) { + return false; + } + + const auto* initializer = graph_utils::GetConstantInitializer(graph, node.InputDefs()[input0_is_initializer ? 0 : 1]->Name()); + + // if initializer_rank is bigger, the output is expected to be initializer_rank per broadcasting rule, + // but it won't happen if the case is accepted, thus reject it + auto initializer_rank = initializer->dims().size(); + const auto* other_input_shape = node.InputDefs()[input0_is_initializer ? 1 : 0]->Shape(); + if (other_input_shape == nullptr || initializer_rank > other_input_shape->dim_size()) { + return false; + } + + int32_t data_type = initializer->data_type(); + Initializer add_init(*initializer, graph.ModelPath()); + if (add_init.size() > 1) { + return false; + } + switch (data_type) { + case ONNX_NAMESPACE::TensorProto_DataType_FLOAT: + if (*add_init.data() != 0.f) { + return false; + } + break; + case ONNX_NAMESPACE::TensorProto_DataType_FLOAT16: + if (math::halfToFloat(add_init.data()->val) != 0.f) { + return false; + } + break; + case ONNX_NAMESPACE::TensorProto_DataType_DOUBLE: + if (*add_init.data() != static_cast(0.f)) { + return false; + } + break; + case ONNX_NAMESPACE::TensorProto_DataType_INT32: + if (*add_init.data() != static_cast(0)) { + return false; + } + break; + case ONNX_NAMESPACE::TensorProto_DataType_INT64: + if (*add_init.data() != static_cast(0)) { + return false; + } + break; + default: + return false; + } + + // reject node output is graph output for now + if (!graph_utils::CanRemoveNode(graph, node, logger)) { + return false; + } + + return true; +} + +} // namespace onnxruntime diff --git a/onnxruntime/core/optimizer/noop_elimination.h b/onnxruntime/core/optimizer/noop_elimination.h new file mode 100644 index 0000000000..7a11046277 --- /dev/null +++ b/onnxruntime/core/optimizer/noop_elimination.h @@ -0,0 +1,31 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +#pragma once + +#include "core/optimizer/rewrite_rule.h" + +namespace onnxruntime { + +/** +@Class NoopElimination + +Rewrite rule that eliminates the no op node. +So far only Add node with 0 as one of its inputs is eliminated. +But this class could be the placeholder for other no op nodes in future. +*/ +class NoopElimination : public RewriteRule { + public: + NoopElimination() noexcept : RewriteRule("NoopElimination") {} + + std::vector TargetOpTypes() const noexcept override { + return {"Add"}; + } + + private: + bool SatisfyCondition(const Graph& graph, const Node& node, const logging::Logger& logger) const override; + + Status Apply(Graph& graph, Node& node, RewriteRuleEffect& rule_effect, const logging::Logger& logger) const override; +}; // namespace onnxruntime + +} // namespace onnxruntime diff --git a/onnxruntime/test/optimizer/graph_transform_test.cc b/onnxruntime/test/optimizer/graph_transform_test.cc index d0b0a47e58..934ddf0982 100644 --- a/onnxruntime/test/optimizer/graph_transform_test.cc +++ b/onnxruntime/test/optimizer/graph_transform_test.cc @@ -48,6 +48,7 @@ #include "core/optimizer/matmul_integer_to_float.h" #include "core/optimizer/matmul_scale_fusion.h" #include "core/optimizer/matmul_transpose_fusion.h" +#include "core/optimizer/noop_elimination.h" #include "core/optimizer/not_where_fusion.h" #include "core/optimizer/relu_clip_fusion.h" #include "core/optimizer/reshape_fusion.h" @@ -166,6 +167,24 @@ TEST_F(GraphTransformationTests, IdentityInputIsGraphOutputNotEliminated) { ASSERT_TRUE(op_to_count["Identity"] == 1); } +TEST_F(GraphTransformationTests, NoopElimination) { + auto model_uri = MODEL_FOLDER "noop-add.onnx"; + std::shared_ptr model; + ASSERT_STATUS_OK(Model::Load(model_uri, model, nullptr, *logger_)); + Graph& graph = model->MainGraph(); + std::map op_to_count = CountOpsInGraph(graph); + ASSERT_TRUE(op_to_count["Add"] == 4); + + auto rule_transformer_L1 = std::make_unique("RuleTransformer1"); + rule_transformer_L1->Register(std::make_unique()); + onnxruntime::GraphTransformerManager graph_transformation_mgr{5}; + graph_transformation_mgr.Register(std::move(rule_transformer_L1), TransformerLevel::Level1); + ASSERT_STATUS_OK(graph_transformation_mgr.ApplyTransformers(graph, TransformerLevel::Level1, *logger_)); + + op_to_count = CountOpsInGraph(graph); + ASSERT_TRUE(op_to_count["Add"] == 1); +} + TEST_F(GraphTransformationTests, DropoutElimination) { auto model_uri = MODEL_FOLDER "dropout.onnx"; std::shared_ptr model; diff --git a/onnxruntime/test/testdata/transform/noop-add.onnx b/onnxruntime/test/testdata/transform/noop-add.onnx new file mode 100644 index 0000000000000000000000000000000000000000..e5793f5fda7c9369fee47503468d775900f02380 GIT binary patch literal 688 zcmZwEK~IA)6bEq1SXy6{U|l@S8}Y=!gtiQ`o7{}ivv?ufa0!VUM!EPk{KU4e5=t0N zO8S4zFOLVK>)$yn!GBG8(9~7B#64z@_4~1Fx*v#v?mya1SEYa@KHt>lo_)o$hq{Iv zuqW=977z|#aDli)TspN|I72*L{qxfD!t!Fo3(X58ltUO^BA;O`b80cRoB?f)-AYxd z#ihmN2$veS{jD6p;0o~^@!Y9dJnwN~abhUF4j6@_3JpB~? zuoJFBNQhSq6H3B}u65FXPO@OctWhf7uJLb15Hi96W}G~m?FnVooQyn&w;W ymnr5WAM+DF<|q8E9&V}l0yv{Q9Qp*Gks<&uIxjBaV%Hw-4$bbX{ciVNME(H#;fSaJ literal 0 HcmV?d00001 diff --git a/onnxruntime/test/testdata/transform/noop-add.py b/onnxruntime/test/testdata/transform/noop-add.py new file mode 100644 index 0000000000..0d1bb9801b --- /dev/null +++ b/onnxruntime/test/testdata/transform/noop-add.py @@ -0,0 +1,71 @@ +import onnx +from onnx import helper +from onnx import TensorProto, OperatorSetIdProto + +opsets = [] +onnxdomain = OperatorSetIdProto() +onnxdomain.version = 12 +onnxdomain.domain = "" # The empty string ("") or absence of this field implies the operator set that is defined as part of the ONNX specification. +opsets.append(onnxdomain) + +msdomain = OperatorSetIdProto() +msdomain.version = 1 +msdomain.domain = 'com.microsoft' + +opsets.append(msdomain) +kwargs={} +kwargs['opset_imports'] = opsets + +def GenerateModel(model_name): + nodes = [ # subgraph + # float + helper.make_node("Identity", ["X1"], ["id_1"], "id_1"), + helper.make_node("Add", ["float_1", "id_1"], ["add_1"], "add_1"), + helper.make_node("Identity", ["add_1"], ["Y1"], "id_2"), + # float_16 + helper.make_node("Identity", ["X2"], ["id_3"], "id_3"), + helper.make_node("Add", ["float16_1", "id_3"], ["add_2"], "add_2"), + helper.make_node("Identity", ["add_2"], ["Y2"], "id_4"), + # int64 - flip the input 0 and 1 + helper.make_node("Identity", ["X3"], ["id_5"], "id_5"), + helper.make_node("Add", ["id_5", "int64_1"], ["add_3"], "add_3"), + helper.make_node("Identity", ["add_3"], ["Y3"], "id_6"), + # int64 + helper.make_node("Identity", ["X4"], ["id_7"], "id_7"), + helper.make_node("Add", ["id_7", "int64_2"], ["add_4"], "add_4"), + helper.make_node("Identity", ["add_4"], ["Y4"], "id_8"), + ] + + inputs = [ # inputs + helper.make_tensor_value_info('X1', TensorProto.FLOAT, ['M', 'K']), + helper.make_tensor_value_info('X2', TensorProto.FLOAT16, ['M', 'K']), + helper.make_tensor_value_info('X3', TensorProto.INT64, ['M', 'K']), + helper.make_tensor_value_info('X4', TensorProto.INT64, ['M', 'K']), + ] + + initializers = [ + helper.make_tensor('float_1', TensorProto.FLOAT, [1], [0.0]), + helper.make_tensor('float16_1', TensorProto.FLOAT16, [1], [0]), + # int64 - set tensor size to 0 + helper.make_tensor('int64_1', TensorProto.INT64, (), [0]), + # higher rank + helper.make_tensor('int64_2', TensorProto.INT64, [1,1,1], [0]), + ] + + graph = helper.make_graph( + nodes, + "NoopAdd", #name + inputs, + [ # outputs + helper.make_tensor_value_info('Y1', TensorProto.FLOAT, ['M', 'K']), + helper.make_tensor_value_info('Y2', TensorProto.FLOAT16, ['M', 'K']), + helper.make_tensor_value_info('Y3', TensorProto.INT64, ['M', 'K']), + helper.make_tensor_value_info('Y4', TensorProto.INT64, ['M', 'K', 1]), + ], + initializers) + + model = helper.make_model(graph, **kwargs) + onnx.save(model, model_name) + +if __name__ == "__main__": + GenerateModel('noop-add.onnx') \ No newline at end of file diff --git a/orttraining/orttraining/core/optimizer/graph_transformer_utils.cc b/orttraining/orttraining/core/optimizer/graph_transformer_utils.cc index 9eb7b764c4..de393c762e 100644 --- a/orttraining/orttraining/core/optimizer/graph_transformer_utils.cc +++ b/orttraining/orttraining/core/optimizer/graph_transformer_utils.cc @@ -33,6 +33,7 @@ #include "core/optimizer/matmul_scale_fusion.h" #include "core/optimizer/matmul_transpose_fusion.h" #include "core/optimizer/nchwc_transformer.h" +#include "core/optimizer/noop_elimination.h" #include "core/optimizer/not_where_fusion.h" #include "core/optimizer/relu_clip_fusion.h" #include "core/optimizer/reshape_fusion.h" @@ -79,6 +80,7 @@ std::vector> GeneratePreTrainingTransformers( rule_transformer->Register(std::make_unique()); rule_transformer->Register(std::make_unique()); rule_transformer->Register(std::make_unique()); + rule_transformer->Register(std::make_unique()); rule_transformer->Register(std::make_unique()); rule_transformer->Register(std::make_unique()); rule_transformer->Register(std::make_unique());