saymrwulf/onnxruntime
1
0
Fork 0
mirror of https://github.com/saymrwulf/onnxruntime.git synced 2026-05-18 21:21:17 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
onnxruntime/cmake/onnxruntime_optimizer.cmake

140 lines
8 KiB
CMake
Raw Normal View History

Create a separate component for graph optimization. (#421) * Create a project for graph optimizer. Move optimizer related code to the folder optimizer. * Fix build failures. * rebase and fix build failures. * fix build failure. * fix build failure with cuda path. * fix python build failure. * Move two transformers(memcpy and insert_cast) from framework to optimizer. * rebase. * SessionState should not depend on optimizer.
2019-02-04 23:45:12 +00:00
# Copyright (c) Microsoft Corporation. All rights reserved.
# Licensed under the MIT License.
Enable selector action transformer infrastructure in minimal build. (#8804)
2021-08-27 07:16:05 +00:00
set(onnxruntime_optimizer_src_patterns)
Initial changes to disable code in a minimal build (#4872) * Initial set of changes to start disabling code in the minimal build. Breaking changes into multiple PRs so they're more easily reviewed. Focus on InferenceSession, Model and Graph here. SessionState will be next. Needs to be integrated with de/serialization code before being testable so changes are all off by default. Changes are limited to - #ifdef'ing out code - moving some things around so there are fewer #ifdef statements - moving definition of some one-line methods into the header so we don't need to #ifdef out in a .cc as well - exclude some things in the cmake setup * Update session state and a few other places. The core code builds if ORT_MINIMAL_BUILD is specified.
2020-08-21 21:14:53 +00:00
if (onnxruntime_MINIMAL_BUILD)
# we include a couple of files so a library is produced and we minimize other changes to the build setup.
Enable selector action transformer infrastructure in minimal build. (#8804)
2021-08-27 07:16:05 +00:00
# if the transformer base class is unused it will be excluded from the final binary size
list(APPEND onnxruntime_optimizer_src_patterns
Initial changes to disable code in a minimal build (#4872) * Initial set of changes to start disabling code in the minimal build. Breaking changes into multiple PRs so they're more easily reviewed. Focus on InferenceSession, Model and Graph here. SessionState will be next. Needs to be integrated with de/serialization code before being testable so changes are all off by default. Changes are limited to - #ifdef'ing out code - moving some things around so there are fewer #ifdef statements - moving definition of some one-line methods into the header so we don't need to #ifdef out in a .cc as well - exclude some things in the cmake setup * Update session state and a few other places. The core code builds if ORT_MINIMAL_BUILD is specified.
2020-08-21 21:14:53 +00:00
"${ONNXRUNTIME_INCLUDE_DIR}/core/optimizer/graph_transformer.h"
"${ONNXRUNTIME_ROOT}/core/optimizer/graph_transformer.cc"
)
Enable selector action transformer infrastructure in minimal build. (#8804)
2021-08-27 07:16:05 +00:00
Remove ORT_ENABLE_RUNTIME_OPTIMIZATION_IN_MINIMAL_BUILD. (#10778) Remove ORT_ENABLE_RUNTIME_OPTIMIZATION_IN_MINIMAL_BUILD as it is now implied by ORT_EXTENDED_MINIMAL_BUILD. Remove related CMake option.
2022-03-09 00:18:49 +00:00
if (onnxruntime_EXTENDED_MINIMAL_BUILD)
[QDQ] Hookup NNAPI GetCapability/Compile with shared QDQ selectors (#10347) * add qdqgroup as input for NodeUnit * minor update * hookup nnapi_ep * minor update * update compiler setting * Add a simple UT * Pipeline change to add build minimal extended with NNAPI for Android * move GetAllNodeUnits to node_unit.h, add UT for NodeUnits, minor updates * minor updates * address CR comments Co-authored-by: gwang0000 <62914304+gwang0000@users.noreply.github.com>
2022-01-26 01:13:46 +00:00
list(APPEND onnxruntime_optimizer_src_patterns
Enable loading of ORT format model graph runtime optimizations (#9901) Initial implementation of load/replay of runtime optimizations in an ORT format model.
2022-01-04 20:09:07 +00:00
"${ONNXRUNTIME_INCLUDE_DIR}/core/optimizer/graph_transformer_utils.h"
Remove ORT_ENABLE_RUNTIME_OPTIMIZATION_IN_MINIMAL_BUILD. (#10778) Remove ORT_ENABLE_RUNTIME_OPTIMIZATION_IN_MINIMAL_BUILD as it is now implied by ORT_EXTENDED_MINIMAL_BUILD. Remove related CMake option.
2022-03-09 00:18:49 +00:00
"${ONNXRUNTIME_ROOT}/core/optimizer/conv_activation_fusion.cc"
"${ONNXRUNTIME_ROOT}/core/optimizer/conv_activation_fusion.h"
Enable loading of ORT format model graph runtime optimizations (#9901) Initial implementation of load/replay of runtime optimizations in an ORT format model.
2022-01-04 20:09:07 +00:00
"${ONNXRUNTIME_ROOT}/core/optimizer/graph_transformer_utils.cc"
Convert ConvActivationFusion transformer to a selector action transformer. (#10687)
2022-03-02 03:47:55 +00:00
"${ONNXRUNTIME_ROOT}/core/optimizer/initializer.cc"
Remove ORT_ENABLE_RUNTIME_OPTIMIZATION_IN_MINIMAL_BUILD. (#10778) Remove ORT_ENABLE_RUNTIME_OPTIMIZATION_IN_MINIMAL_BUILD as it is now implied by ORT_EXTENDED_MINIMAL_BUILD. Remove related CMake option.
2022-03-09 00:18:49 +00:00
"${ONNXRUNTIME_ROOT}/core/optimizer/initializer.h"
MatMulNBits + Add fusion (#20587) - Add MatMulNBits Bias input - Add graph transformer to fuse MatMulNBits + Add
2024-05-16 18:00:59 +00:00
"${ONNXRUNTIME_ROOT}/core/optimizer/matmul_nbits_fusion.cc"
"${ONNXRUNTIME_ROOT}/core/optimizer/matmul_nbits_fusion.h"
Enable QDQ cleanup and NHWC optimizers in an extended minimal build. (#10729) * Enable QDQ cleanup and NHWC optimizers in an extended minimal build.
2022-03-04 05:45:42 +00:00
"${ONNXRUNTIME_ROOT}/core/optimizer/nhwc_transformer.cc"
Remove ORT_ENABLE_RUNTIME_OPTIMIZATION_IN_MINIMAL_BUILD. (#10778) Remove ORT_ENABLE_RUNTIME_OPTIMIZATION_IN_MINIMAL_BUILD as it is now implied by ORT_EXTENDED_MINIMAL_BUILD. Remove related CMake option.
2022-03-09 00:18:49 +00:00
"${ONNXRUNTIME_ROOT}/core/optimizer/nhwc_transformer.h"
Enable QDQ cleanup and NHWC optimizers in an extended minimal build. (#10729) * Enable QDQ cleanup and NHWC optimizers in an extended minimal build.
2022-03-04 05:45:42 +00:00
"${ONNXRUNTIME_ROOT}/core/optimizer/qdq_transformer/qdq_final_cleanup.cc"
Remove ORT_ENABLE_RUNTIME_OPTIMIZATION_IN_MINIMAL_BUILD. (#10778) Remove ORT_ENABLE_RUNTIME_OPTIMIZATION_IN_MINIMAL_BUILD as it is now implied by ORT_EXTENDED_MINIMAL_BUILD. Remove related CMake option.
2022-03-09 00:18:49 +00:00
"${ONNXRUNTIME_ROOT}/core/optimizer/qdq_transformer/qdq_final_cleanup.h"
Enable selector action transformer infrastructure in minimal build. (#8804)
2021-08-27 07:16:05 +00:00
"${ONNXRUNTIME_ROOT}/core/optimizer/qdq_transformer/qdq_util.cc"
Remove ORT_ENABLE_RUNTIME_OPTIMIZATION_IN_MINIMAL_BUILD. (#10778) Remove ORT_ENABLE_RUNTIME_OPTIMIZATION_IN_MINIMAL_BUILD as it is now implied by ORT_EXTENDED_MINIMAL_BUILD. Remove related CMake option.
2022-03-09 00:18:49 +00:00
"${ONNXRUNTIME_ROOT}/core/optimizer/qdq_transformer/qdq_util.h"
"${ONNXRUNTIME_ROOT}/core/optimizer/qdq_transformer/selectors_actions/qdq_actions.cc"
"${ONNXRUNTIME_ROOT}/core/optimizer/qdq_transformer/selectors_actions/qdq_actions.h"
"${ONNXRUNTIME_ROOT}/core/optimizer/qdq_transformer/selectors_actions/qdq_selector_action_transformer.cc"
"${ONNXRUNTIME_ROOT}/core/optimizer/qdq_transformer/selectors_actions/qdq_selector_action_transformer.h"
"${ONNXRUNTIME_ROOT}/core/optimizer/qdq_transformer/selectors_actions/qdq_selectors.cc"
"${ONNXRUNTIME_ROOT}/core/optimizer/qdq_transformer/selectors_actions/qdq_selectors.h"
"${ONNXRUNTIME_ROOT}/core/optimizer/qdq_transformer/selectors_actions/shared/utils.cc"
"${ONNXRUNTIME_ROOT}/core/optimizer/qdq_transformer/selectors_actions/shared/utils.h"
"${ONNXRUNTIME_ROOT}/core/optimizer/selectors_actions/actions.cc"
"${ONNXRUNTIME_ROOT}/core/optimizer/selectors_actions/actions.h"
"${ONNXRUNTIME_ROOT}/core/optimizer/selectors_actions/helpers.cc"
"${ONNXRUNTIME_ROOT}/core/optimizer/selectors_actions/helpers.cc"
"${ONNXRUNTIME_ROOT}/core/optimizer/selectors_actions/helpers.h"
"${ONNXRUNTIME_ROOT}/core/optimizer/selectors_actions/helpers.h"
"${ONNXRUNTIME_ROOT}/core/optimizer/selectors_actions/selector_action_transformer_apply_contexts.h"
"${ONNXRUNTIME_ROOT}/core/optimizer/selectors_actions/selector_action_transformer.cc"
"${ONNXRUNTIME_ROOT}/core/optimizer/selectors_actions/selector_action_transformer.h"
Re-organize the transpose optimization and layout transformation files. (#16246) ### Description <!-- Describe your changes. --> Split out the more basic changes from #15552 for easier review. Re-organize to clarify the structure - Separate out generic base functionality from ORT specific components - pass in handlers for internal ORT ops to Optimize - Split out layout transformation from transpose optimization - Separate out level 1 transpose optimizer - Cleanup some naming to try and clarify things like an optimizer vs. general optimization code Most of the changes are from this movement of code. Two implementation changes: - the extended handlers are queried first in GetHandler - allows the extended handlers to override the default behaviour for an ONNX operator - simplify the Optimize function to remove OptimizerMode. - `can_modify_node` is used instead of `mode` and `ignore_assigned_nodes` and a long description of the current usage is added. I don't _think_ that changes the current behavior and hopefully clarifies what happens and when, and makes the base transpose optimizer implementation more generic. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Create a cleaner separation to support adding EP specific logic next to cleanly handle where an EP has additional layout sensitive behaviour required (e.g. it's Resize implementation only handles one layout).
2023-07-06 22:24:47 +00:00
# files required for layout transformation
"${ONNXRUNTIME_ROOT}/core/optimizer/layout_transformation/layout_transformation.h"
"${ONNXRUNTIME_ROOT}/core/optimizer/layout_transformation/layout_transformation.cc"
"${ONNXRUNTIME_ROOT}/core/optimizer/layout_transformation/layout_transformation_potentially_added_ops.h"
# files required for transpose optimization post-layout transformation
"${ONNXRUNTIME_ROOT}/core/optimizer/transpose_optimization/optimizer_api.h"
"${ONNXRUNTIME_ROOT}/core/optimizer/transpose_optimization/onnx_transpose_optimization.h"
"${ONNXRUNTIME_ROOT}/core/optimizer/transpose_optimization/onnx_transpose_optimization.cc"
"${ONNXRUNTIME_ROOT}/core/optimizer/transpose_optimization/ort_optimizer_api_impl.cc"
"${ONNXRUNTIME_ROOT}/core/optimizer/transpose_optimization/ort_optimizer_utils.h"
"${ONNXRUNTIME_ROOT}/core/optimizer/transpose_optimization/ort_transpose_optimization.h"
"${ONNXRUNTIME_ROOT}/core/optimizer/transpose_optimization/ort_transpose_optimization.cc"
Convert ConvActivationFusion transformer to a selector action transformer. (#10687)
2022-03-02 03:47:55 +00:00
"${ONNXRUNTIME_ROOT}/core/optimizer/utils.cc"
Remove ORT_ENABLE_RUNTIME_OPTIMIZATION_IN_MINIMAL_BUILD. (#10778) Remove ORT_ENABLE_RUNTIME_OPTIMIZATION_IN_MINIMAL_BUILD as it is now implied by ORT_EXTENDED_MINIMAL_BUILD. Remove related CMake option.
2022-03-09 00:18:49 +00:00
"${ONNXRUNTIME_ROOT}/core/optimizer/utils.h"
Enable selector action transformer infrastructure in minimal build. (#8804)
2021-08-27 07:16:05 +00:00
)
endif()
Initial changes to disable code in a minimal build (#4872) * Initial set of changes to start disabling code in the minimal build. Breaking changes into multiple PRs so they're more easily reviewed. Focus on InferenceSession, Model and Graph here. SessionState will be next. Needs to be integrated with de/serialization code before being testable so changes are all off by default. Changes are limited to - #ifdef'ing out code - moving some things around so there are fewer #ifdef statements - moving definition of some one-line methods into the header so we don't need to #ifdef out in a .cc as well - exclude some things in the cmake setup * Update session state and a few other places. The core code builds if ORT_MINIMAL_BUILD is specified.
2020-08-21 21:14:53 +00:00
else()
Enable selector action transformer infrastructure in minimal build. (#8804)
2021-08-27 07:16:05 +00:00
list(APPEND onnxruntime_optimizer_src_patterns
Cleanup some aspects of the Initializer class used by optimizers (#2005) * Move check on data type outside of the Initializer class as it's specific to Conv processing. Use references for arguments that can't be null.
2019-10-09 00:37:44 +00:00
"${ONNXRUNTIME_INCLUDE_DIR}/core/optimizer/*.h"
Create a separate component for graph optimization. (#421) * Create a project for graph optimizer. Move optimizer related code to the folder optimizer. * Fix build failures. * rebase and fix build failures. * fix build failure. * fix build failure with cuda path. * fix python build failure. * Move two transformers(memcpy and insert_cast) from framework to optimizer. * rebase. * SessionState should not depend on optimizer.
2019-02-04 23:45:12 +00:00
"${ONNXRUNTIME_ROOT}/core/optimizer/*.h"
"${ONNXRUNTIME_ROOT}/core/optimizer/*.cc"
Optimize computation orders (#13672) ### Optimize computation orders In `Roberta/Electra`, when `ClassificationHead` is used, there is slicing operation on features on sequence_length dimensions, then loss calculations only depend on this sliced data. This is a slicing at axis 1. Before slicing the shape is [batch, sequence_length, hidden], after slicing, it becomes [batch , hidden_stage] We had opportunities to bring this slicing earlier as much as possible, by passing through simple elementwise ops (like Add/Div), or Layernorm/Softmax(if their reduce axis is after the slicing axis), or even MatMul's the left operand (if only it did not affect the last dims). For operators like Reshape/Transpose, it is special since they have either data specified (after slicing we need update), or they have perm specified, which requires the input rank remain unchanged. So for those kinds of operators, we can remain the original rank, but just leave the sliced dim to be 1, after the compute completed, we do a Squeeze. ``` class RobertaClassificationHead(nn.Module): """Head for sentence-level classification tasks.""" def __init__(self, config): super().__init__() self.dense = nn.Linear(config.hidden_size, config.hidden_size) classifier_dropout = ( config.classifier_dropout if config.classifier_dropout is not None else config.hidden_dropout_prob ) self.dropout = nn.Dropout(classifier_dropout) self.out_proj = nn.Linear(config.hidden_size, config.num_labels) def forward(self, features, **kwargs): x = features[:, 0, :] # take <s> token (equiv. to [CLS]) x = self.dropout(x) x = self.dense(x) x = torch.tanh(x) x = self.dropout(x) x = self.out_proj(x) return x ``` src\transformers\models\roberta\modeling_roberta.py src\transformers\models\electra\modeling_electra.py #### Benchmark A simple benchmark shows Robeta training latency dropped from 208ms ~ 199ms. 4.5+% reduction. More comprehensive tests are on the way. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. -->
2022-12-22 07:12:52 +00:00
"${ONNXRUNTIME_ROOT}/core/optimizer/compute_optimizer/*.h"
"${ONNXRUNTIME_ROOT}/core/optimizer/compute_optimizer/*.cc"
Re-organize the transpose optimization and layout transformation files. (#16246) ### Description <!-- Describe your changes. --> Split out the more basic changes from #15552 for easier review. Re-organize to clarify the structure - Separate out generic base functionality from ORT specific components - pass in handlers for internal ORT ops to Optimize - Split out layout transformation from transpose optimization - Separate out level 1 transpose optimizer - Cleanup some naming to try and clarify things like an optimizer vs. general optimization code Most of the changes are from this movement of code. Two implementation changes: - the extended handlers are queried first in GetHandler - allows the extended handlers to override the default behaviour for an ONNX operator - simplify the Optimize function to remove OptimizerMode. - `can_modify_node` is used instead of `mode` and `ignore_assigned_nodes` and a long description of the current usage is added. I don't _think_ that changes the current behavior and hopefully clarifies what happens and when, and makes the base transpose optimizer implementation more generic. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Create a cleaner separation to support adding EP specific logic next to cleanly handle where an EP has additional layout sensitive behaviour required (e.g. it's Resize implementation only handles one layout).
2023-07-06 22:24:47 +00:00
"${ONNXRUNTIME_ROOT}/core/optimizer/layout_transformation/*.h"
"${ONNXRUNTIME_ROOT}/core/optimizer/layout_transformation/*.cc"
QDQ implementation (#7033) * Add QDQ basic implementation
2021-03-25 16:17:23 +00:00
"${ONNXRUNTIME_ROOT}/core/optimizer/qdq_transformer/*.h"
"${ONNXRUNTIME_ROOT}/core/optimizer/qdq_transformer/*.cc"
Enable selector action transformer infrastructure in minimal build. (#8804)
2021-08-27 07:16:05 +00:00
"${ONNXRUNTIME_ROOT}/core/optimizer/qdq_transformer/selectors_actions/*.h"
"${ONNXRUNTIME_ROOT}/core/optimizer/qdq_transformer/selectors_actions/*.cc"
[QDQ] Add shared qdq selectors (#10178) * wip * wip * wip * wip * wip * save * minor changes * update test graph name * address pr comments * update * address pr comments * address pr comments * fix warning * minor include fix * update to nodegroupselectors * delete unnecessary includes Co-authored-by: rachguo <rachguo@rachguos-Mini.attlocal.net>
2022-01-12 03:41:45 +00:00
"${ONNXRUNTIME_ROOT}/core/optimizer/qdq_transformer/selectors_actions/shared/utils.h"
"${ONNXRUNTIME_ROOT}/core/optimizer/qdq_transformer/selectors_actions/shared/utils.cc"
Refactor QDQ optimizers to enable future usage in minimal build (#8191) * Add new transformer that can split node selection from node modification to allow just the modifications to be applied at runtime in a minimal build. This is the first step of a few to enable a QDQ model to be optimized for the NNAPI EP and/or the CPU EP at runtime in a mobile scenario. Add generic and QDQ specific helpers for selection and modification. Replace existing QDQ optimizers with optimizer based on new approach.
2021-07-09 06:11:43 +00:00
"${ONNXRUNTIME_ROOT}/core/optimizer/selectors_actions/*.h"
"${ONNXRUNTIME_ROOT}/core/optimizer/selectors_actions/*.cc"
Re-organize the transpose optimization and layout transformation files. (#16246) ### Description <!-- Describe your changes. --> Split out the more basic changes from #15552 for easier review. Re-organize to clarify the structure - Separate out generic base functionality from ORT specific components - pass in handlers for internal ORT ops to Optimize - Split out layout transformation from transpose optimization - Separate out level 1 transpose optimizer - Cleanup some naming to try and clarify things like an optimizer vs. general optimization code Most of the changes are from this movement of code. Two implementation changes: - the extended handlers are queried first in GetHandler - allows the extended handlers to override the default behaviour for an ONNX operator - simplify the Optimize function to remove OptimizerMode. - `can_modify_node` is used instead of `mode` and `ignore_assigned_nodes` and a long description of the current usage is added. I don't _think_ that changes the current behavior and hopefully clarifies what happens and when, and makes the base transpose optimizer implementation more generic. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Create a cleaner separation to support adding EP specific logic next to cleanly handle where an EP has additional layout sensitive behaviour required (e.g. it's Resize implementation only handles one layout).
2023-07-06 22:24:47 +00:00
"${ONNXRUNTIME_ROOT}/core/optimizer/transpose_optimization/*.h"
"${ONNXRUNTIME_ROOT}/core/optimizer/transpose_optimization/*.cc"
Initial changes to disable code in a minimal build (#4872) * Initial set of changes to start disabling code in the minimal build. Breaking changes into multiple PRs so they're more easily reviewed. Focus on InferenceSession, Model and Graph here. SessionState will be next. Needs to be integrated with de/serialization code before being testable so changes are all off by default. Changes are limited to - #ifdef'ing out code - moving some things around so there are fewer #ifdef statements - moving definition of some one-line methods into the header so we don't need to #ifdef out in a .cc as well - exclude some things in the cmake setup * Update session state and a few other places. The core code builds if ORT_MINIMAL_BUILD is specified.
2020-08-21 21:14:53 +00:00
)
endif()
Create a separate component for graph optimization. (#421) * Create a project for graph optimizer. Move optimizer related code to the folder optimizer. * Fix build failures. * rebase and fix build failures. * fix build failure. * fix build failure with cuda path. * fix python build failure. * Move two transformers(memcpy and insert_cast) from framework to optimizer. * rebase. * SessionState should not depend on optimizer.
2019-02-04 23:45:12 +00:00
Minimal Build for On-Device Training (#16326) 🛠️ __Changes in this pull request:__ This pull request introduces two significant changes to the project: - Changing on device training checkpoint format: The current implementation stores the on device training checkpoint as a sequence of tensors in multiple files inside a checkpoint folder, which can be inefficient in terms of storage and performance. In this PR, I have modified the checkpoint format to utilize the flatbuffer table to save the checkpoint to a single file, providing a more compact and efficient representation. The changes around this are twofold: - Add the checkpoint flatbuffer schema that will generate the necessary checkpoint source files. - Update the checkpoint saving and loading functionality to use the new format. - Adding support for onnxruntime minimal build: To support scenarios where binary size is a constraint, I made changes to ensure that the training build can work well with the minimal build. 🔍 __Open Issues:__ - In order to extract the optimizer type, the existing implementation re-loaded the onnx optimizer model and parsed it. This is no longer possible, since the model format can either be onnx or ort. One idea is to do the same for ort format optimizer model. This needs some investigation. - Changes to the offline tooling to generate ort format training artifacts. - End-to-end training example showcasing the use of the minimal training build. - Add support for export model for inferencing in a minimal build.
2023-06-22 19:27:23 +00:00
if (onnxruntime_ENABLE_TRAINING)
Enable selector action transformer infrastructure in minimal build. (#8804)
2021-08-27 07:16:05 +00:00
list(APPEND onnxruntime_optimizer_src_patterns
"${ORTTRAINING_SOURCE_DIR}/core/optimizer/*.h"
"${ORTTRAINING_SOURCE_DIR}/core/optimizer/*.cc"
Optimize SCE loss compute (#15401) ### Optimize SCE loss compute Compute optimization based on label data sparsity: - Insert ShrunkenGather before SCELoss node, to filter out invalid labels for compute. - Support ShrunkenGather upstream. - Added test for the above. - Added flag to enable label sparsity optimization with env var, by default disabled now. Will enable after comprehensive benchmarking later. - Extract common logic into test_optimizer_utils.h/cc from core/optimizer/compute_optimzier_test.cc, then the common functions can be shared by both core/optimizer/compute_optimzier_test.cc and orttraining/core/optimizer/compute_optimzier_test.cc - Extract common logic into shared_utils.h/cc: `GetONNXOpSetVersion` and `Create1DInitializerFromVector` ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. -->
2023-04-13 05:02:12 +00:00
"${ORTTRAINING_SOURCE_DIR}/core/optimizer/compute_optimizer/*.h"
"${ORTTRAINING_SOURCE_DIR}/core/optimizer/compute_optimizer/*.cc"
Memory optimization refactor and refinement (#17481) ### Memory optimization refactor and refinement Currently memory optimizer runs graph transformations and print recompute opportunities in INFO level, while ORT backend has many many INFO level logs making users hard to find those information. So we are looking for a Python binding API to retrieve the memory optimization opportunities instead of depending on the MemoryOptimizer's default logging. Then we can print ORTModule feature statistics using this information. Also, with such an API, we can create an ORT session created, where allocation plan is done, the analysis will consider buffer reuse as well. This can void giving some recomputation subgraphs that are reusing other subgraphs' output buffers. Check https://github.com/microsoft/onnxruntime/blob/pengwa/add_devinfo_level/docs/Memory_Optimizer.md for the new flow using `MemoryOptimizer`. This pull requests made following refactoring: 1. Print the log in ORTModule Python script, along with ORTModule feature enabling stats. This is implemented by exposing an API `get_serialized_ortmodule_memory_stat` to retrieve the memory optimization opportunities. 2. We are analyzing memory optimization opportunities considering ORT memory planning. This is done by firstly creating the execution graph without enabling MemoryOptimizer, then we call `execution_agent.get_serialized_ortmodule_memory_stat` which internally will consider the session memory allocation planner when analyzing memory optimization opportunity. As a direct result, the memory optimization opportunities can show those stashed activations that are reusing other buffers. 3. Move recompute analysis logic from memory_optimizer.h/cc to recompute_analysis.h/cc. 4. Abstract optimization strategies for their own implementation. This will make introducing new strategies (for example compression and decompression ) easier. New logging matrix (INFO Level), in WARNING level, the details will NOT show. ``` 2023-09-13 13:25:09,249 orttraining.rank-0 [WARNING] - ***** ONNX Runtime Training (ORTModule) is accelerating your model ***** ORTModule is enabled with following features ON/OFF for [training] mode: ATen Executor : ON : Dispatch ATen operators to ORT's ATen executor Cast Propagation : ON : Level 1 enabled Custom Function : ON : Support custom torch.autograd.Function export and execution Memory Optimizer : ON : RecomputeConfig: Reshape+Where+BiasSoftmax+:1:-1,Cast+:1:-1, ProbeLevel: 1, available configs: Config Freq Saving(B) Saving Symbolic(Bytes) - Plan 1 : ON : Reshape+Where+BiasSoftmax+:1:-1 5 671,088,640 640.0*inputs_input_ids_dim0*inputs_input_ids_dim1**2 - Plan 2 : ON : Cast+:1:-1 6 402,587,648 inputs_input_ids_dim0*inputs_input_ids_dim1*(384.0*inputs_input_ids_dim1 - 64.0) - Plan 3 : OFF : Reshape+Where+:1:-1 1 134,217,728 128.0*inputs_input_ids_dim0*inputs_input_ids_dim1**2 - Plan 4 : OFF : BiasSoftmax+:1:-1 1 134,086,656 128.0*inputs_input_ids_dim0*inputs_input_ids_dim1*(inputs_input_ids_dim1 - 1) - Plan 5 : OFF : BiasGelu+:1:-1 6 125,808,640 inputs_input_ids_dim0*(122880.0*inputs_input_ids_dim1 - 20480.0) - Plan 6 : OFF : FusedMatMul+:1:-1 6 125,808,640 inputs_input_ids_dim0*(122880.0*inputs_input_ids_dim1 - 20480.0) - Plan 7 : OFF : FusedMatMul+Add+FusedMatMul+Add+Add+Add+:1:-1 5 26,214,400 25600.0*inputs_input_ids_dim0*inputs_input_ids_dim1 - Plan 8 : OFF : Add+:1:-1 1 5,237,760 5120.0*inputs_input_ids_dim0*(inputs_input_ids_dim1 - 1) - Plan 9 : OFF : Reshape+Unsqueeze+Unsqueeze+Cast+Sub+Mul+Cast+:1:-1 1 4,096 4.0*inputs_input_ids_dim0*inputs_input_ids_dim1 - Plan 10 : OFF : Cast+:2:-1 1 2,048 2.0*inputs_input_ids_dim0*inputs_input_ids_dim1 Compute Optimizer : ON : Enable/Disable with env ORTMODULE_ENABLE_COMPUTE_OPTIMIZER=1/0 - FLOPReduction : ON : Reduce FLOPs by upstreaming shrinking-sized ops Auto Fallback : ON : Fallback to PyTorch when encountering unsupported ops TritonOp Enabled : OFF : ORT will switch to Triton for executing some ops to further accelerate training. ZeRO Stage3 Support : OFF : Enable/Disable with env ORTMODULE_ENABLE_ZERO_STAGE3=1/0 Total ORT initialization overhead is 10.73s where export takes 8.39s. Other overhead details: graph builder init takes 0.06s, runtime detection takes 0.01s, graph building takes 0.31s, session creation takes 1.96s Versions: ONNX Runtime - 1.16.0+cu118, ONNX - 1.11.0 Note 1: use comma to enable multiple plans at the same time. export ORTMODULE_MEMORY_OPT_CONFIG=<plan1 config>,<plan2 config>,... Note 2: saving is calculated based on the 1st batch symbolic dim values: inputs_input_ids_dim0=1, inputs_input_ids_dim1=1024, inputs_attention_mask_dim0=1, inputs_attention_mask_dim1=1024, inputs_labels_dim0=1, inputs_labels_dim1=1024, ************************************************************************ ``` If DEVINFO level is enabled, then more details about the memory optimizations are printed. ``` MemoryInsight Summary - User config: BiasGelu+:1:-1,Cast+:2:-1 ========================================================================================================================================== |Freq | Memory Optimization Opportunities (Clustered by node-level activation patterns) | |_ _ _ _|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ | |3 |For each row options are mutually exclusive, only one of them can be enabled. | | | | | |>>Option 1 : Recompute subgraph FusedMatMul+Add+Reshape+ | | | Status : Disabled. Enable with export ORTMODULE_MEMORY_OPT_CONFIG=FusedMatMul+Add+Reshape+:1:-1 | | | Stashed Activations: | | | - ReuseFreq : Output 0(3), | | | - Output 0 : [inputs_input_ids_dim0 x inputs_input_ids_dim1 x 32 x 240 x ], byte/elem: 2, 100% saved | |_ _ _ _|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ | |2 |For each row options are mutually exclusive, only one of them can be enabled. | | | | | |>>Option 1 : Recompute subgraph Reshape+ | | | Status : Disabled. Enable with export ORTMODULE_MEMORY_OPT_CONFIG=Reshape+:1:-1 | | | Stashed Activations: | | | - ReuseFreq : Output 0(2), | | | - Output 0 : [ x 2560 x ], byte/elem: 2, 100% saved | |_ _ _ _|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ | |2 |For each row options are mutually exclusive, only one of them can be enabled. | | | | | |>>Option 1 : Recompute subgraph FusedMatMul+ | | | Status : Disabled. Enable with export ORTMODULE_MEMORY_OPT_CONFIG=FusedMatMul+:1:-1 | | | Stashed Activations: | | | - Output 0 : [inputs_input_ids_dim0 x inputs_input_ids_dim1 x 10240 x ], byte/elem: 2, 100% saved | |_ _ _ _|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ | |2 |For each row options are mutually exclusive, only one of them can be enabled. | | | | | |>>Option 1 : Recompute subgraph Cast+ | | | Status : Disabled. Enable with export ORTMODULE_MEMORY_OPT_CONFIG=Cast+:1:-1 | | | Stashed Activations: | | | - Output 0 : [inputs_input_ids_dim0 x 32 x inputs_input_ids_dim1 x inputs_input_ids_dim1 x ], byte/elem: 2, 100% saved | |_ _ _ _|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ | |2 |For each row options are mutually exclusive, only one of them can be enabled. | | | | | |>>Option 1 : Recompute subgraph Reshape+Where+BiasSoftmax+ | | | Status : Disabled. Enable with export ORTMODULE_MEMORY_OPT_CONFIG=Reshape+Where+BiasSoftmax+:1:-1 | | | Stashed Activations: | | | - Output 0 : [inputs_input_ids_dim0 x 32 x inputs_input_ids_dim1 x inputs_input_ids_dim1 x ], byte/elem: 4, 100% saved | |_ _ _ _|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ | |2 |For each row options are mutually exclusive, only one of them can be enabled. | | | | | |>>Option 1 : Recompute subgraph BiasGelu+ | | | Status : Enabled, requested count=-1, actual applied count=2 | | | Stashed Activations: | | | - Output 0 : [inputs_input_ids_dim0 x inputs_input_ids_dim1 x 10240 x ], byte/elem: 2, 100% saved | |_ _ _ _|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ | |2 |For each row options are mutually exclusive, only one of them can be enabled. | | | | | |>>Option 1 : Recompute subgraph FusedMatMul+Add+FusedMatMul+Add+Add+Add+ | | | Status : Disabled. Enable with export ORTMODULE_MEMORY_OPT_CONFIG=FusedMatMul+Add+FusedMatMul+Add+Add+Add+:1:-1 | | | Stashed Activations: | | | - Output 0 : [inputs_input_ids_dim0 x inputs_input_ids_dim1 x 2560 x ], byte/elem: 2, 100% saved | |_ _ _ _|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ | |1 |For each row options are mutually exclusive, only one of them can be enabled. | | | | | |>>Option 1 : Recompute subgraph Reshape+Where+ | | | Status : Disabled. Enable with export ORTMODULE_MEMORY_OPT_CONFIG=Reshape+Where+:1:-1 | | | Stashed Activations: | | | - Output 0 : [inputs_input_ids_dim0 x 32 x inputs_input_ids_dim1 x inputs_input_ids_dim1 x ], byte/elem: 4, 100% saved | |_ _ _ _|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ | |1 |For each row options are mutually exclusive, only one of them can be enabled. | | | | | |>>Option 1 : Recompute subgraph FusedMatMul+ | | | Status : Disabled. Enable with export ORTMODULE_MEMORY_OPT_CONFIG=FusedMatMul+:1:-1 | | | Stashed Activations: | | | - Output 0 : [inputs_input_ids_dim0*(inputs_input_ids_dim1 - 1) x 10240 x ], byte/elem: 2, 100% saved | |_ _ _ _|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ | |1 |For each row options are mutually exclusive, only one of them can be enabled. | | | | | |>>Option 1 : Recompute subgraph Cast+ | | | Status : Disabled. Enable with export ORTMODULE_MEMORY_OPT_CONFIG=Cast+:1:-1 | | | Stashed Activations: | | | - Output 0 : [inputs_input_ids_dim0 x 32 x inputs_input_ids_dim1 - 1 x inputs_input_ids_dim1 x ], byte/elem: 2, 100% saved | |_ _ _ _|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ | |1 |For each row options are mutually exclusive, only one of them can be enabled. | | | | | |>>Option 1 : Recompute subgraph Reshape+Unsqueeze+Unsqueeze+Cast+Sub+Mul+Cast+ | | | Status : Disabled. Enable with export ORTMODULE_MEMORY_OPT_CONFIG=Reshape+Unsqueeze+Unsqueeze+Cast+Sub+Mul+Cast+:1:-1 | | | Stashed Activations: | | | - Output 0 : [inputs_input_ids_dim0 x 1 x 1 x inputs_input_ids_dim1 x ], byte/elem: 4, 100% saved | | | | | |>>Option 2 : RecomputeWithCompromise subgraph Cast+ | | | Status : Enabled, requested count=-1, actual applied count=1 | | | Stashed Activations: | | | - Output 0 : [inputs_input_ids_dim0 x 1 x 1 x inputs_input_ids_dim1 x ], byte/elem: 4, 50% saved | |_ _ _ _|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ | |1 |For each row options are mutually exclusive, only one of them can be enabled. | | | | | |>>Option 1 : Recompute subgraph BiasSoftmax+ | | | Status : Disabled. Enable with export ORTMODULE_MEMORY_OPT_CONFIG=BiasSoftmax+:1:-1 | | | Stashed Activations: | | | - Output 0 : [inputs_input_ids_dim0 x 32 x inputs_input_ids_dim1 - 1 x inputs_input_ids_dim1 x ], byte/elem: 4, 100% saved | |_ _ _ _|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ | |1 |For each row options are mutually exclusive, only one of them can be enabled. | | | | | |>>Option 1 : Recompute subgraph BiasGelu+ | | | Status : Enabled, requested count=-1, actual applied count=1 | | | Stashed Activations: | | | - Output 0 : [inputs_input_ids_dim0*(inputs_input_ids_dim1 - 1) x 10240 x ], byte/elem: 2, 100% saved | |_ _ _ _|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ | |1 |For each row options are mutually exclusive, only one of them can be enabled. | | | | | |>>Option 1 : Recompute subgraph Add+ | | | Status : Disabled. Enable with export ORTMODULE_MEMORY_OPT_CONFIG=Add+:1:-1 | | | Stashed Activations: | | | - Output 0 : [inputs_input_ids_dim0*(inputs_input_ids_dim1 - 1) x 2560 x ], byte/elem: 2, 100% saved | |_ _ _ _|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ | ========================================================================================================================================== Note: use comma as a separator for enabling more than one subgraphs. ************************************************************************ ``` ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. -->
2023-11-23 03:39:00 +00:00
"${ORTTRAINING_SOURCE_DIR}/core/optimizer/memory_optimizer/*.h"
"${ORTTRAINING_SOURCE_DIR}/core/optimizer/memory_optimizer/*.cc"
Enable selector action transformer infrastructure in minimal build. (#8804)
2021-08-27 07:16:05 +00:00
)
Introduce training changes.
2020-03-11 21:25:37 +00:00
endif()
Enable selector action transformer infrastructure in minimal build. (#8804)
2021-08-27 07:16:05 +00:00
file(GLOB onnxruntime_optimizer_srcs CONFIGURE_DEPENDS ${onnxruntime_optimizer_src_patterns})
Create a separate component for graph optimization. (#421) * Create a project for graph optimizer. Move optimizer related code to the folder optimizer. * Fix build failures. * rebase and fix build failures. * fix build failure. * fix build failure with cuda path. * fix python build failure. * Move two transformers(memcpy and insert_cast) from framework to optimizer. * rebase. * SessionState should not depend on optimizer.
2019-02-04 23:45:12 +00:00
source_group(TREE ${REPO_ROOT} FILES ${onnxruntime_optimizer_srcs})
support build option to include external graph transformers (#9478) * temp code * support external graph transformer from build script * remove debug code * add test case * support register rewrite rule * fix source_group issue if external source is not share any common prefix * fix python code style checker * resolve merge conflict Co-authored-by: Cheng Tang <chenta@microsoft.com>
2021-11-15 16:16:20 +00:00
if (onnxruntime_EXTERNAL_TRANSFORMER_SRC_PATH)
set(onnxruntime_external_transformer_src_patterns)
list(APPEND onnxruntime_external_transformer_src_patterns
"${onnxruntime_EXTERNAL_TRANSFORMER_SRC_PATH}/*.cc"
"${onnxruntime_EXTERNAL_TRANSFORMER_SRC_PATH}/*.cpp"
)
file(GLOB onnxruntime_external_transformer_src ${onnxruntime_external_transformer_src_patterns})
list(APPEND onnxruntime_optimizer_srcs ${onnxruntime_external_transformer_src})
endif()
Add static code analyzer to Windows CPU/GPU CI builds and fix the warnings (#7489)
2021-04-29 18:54:57 +00:00
onnxruntime_add_static_library(onnxruntime_optimizer ${onnxruntime_optimizer_srcs})
Initial changes to disable code in a minimal build (#4872) * Initial set of changes to start disabling code in the minimal build. Breaking changes into multiple PRs so they're more easily reviewed. Focus on InferenceSession, Model and Graph here. SessionState will be next. Needs to be integrated with de/serialization code before being testable so changes are all off by default. Changes are limited to - #ifdef'ing out code - moving some things around so there are fewer #ifdef statements - moving definition of some one-line methods into the header so we don't need to #ifdef out in a .cc as well - exclude some things in the cmake setup * Update session state and a few other places. The core code builds if ORT_MINIMAL_BUILD is specified.
2020-08-21 21:14:53 +00:00
Use target name for flatbuffers (#13991) ### Description Use target name for flatbuffers. Add version range for flatbuffers. It is similar to #13870 ### Motivation and Context To fix a build error: ``` CMake Error at onnxruntime_graph.cmake:88 (add_dependencies): The dependency target "flatbuffers" of target "onnxruntime_graph" does not exist. Call Stack (most recent call first): CMakeLists.txt:1490 (include) ``` It happens when flatbuffers library is already installed. For example, on Ubuntu people may get it from apt-get. But, the one provided by Ubuntu 20.04 is not compatible with our code. The one in Ubuntu 22.04 works fine.
2022-12-20 19:44:02 +00:00
onnxruntime_add_include_to_target(onnxruntime_optimizer onnxruntime_common onnxruntime_framework onnx onnx_proto ${PROTOBUF_LIB} flatbuffers::flatbuffers Boost::mp11 safeint_interface)
Create a separate component for graph optimization. (#421) * Create a project for graph optimizer. Move optimizer related code to the folder optimizer. * Fix build failures. * rebase and fix build failures. * fix build failure. * fix build failure with cuda path. * fix python build failure. * Move two transformers(memcpy and insert_cast) from framework to optimizer. * rebase. * SessionState should not depend on optimizer.
2019-02-04 23:45:12 +00:00
target_include_directories(onnxruntime_optimizer PRIVATE ${ONNXRUNTIME_ROOT})
Minimal Build for On-Device Training (#16326) 🛠️ __Changes in this pull request:__ This pull request introduces two significant changes to the project: - Changing on device training checkpoint format: The current implementation stores the on device training checkpoint as a sequence of tensors in multiple files inside a checkpoint folder, which can be inefficient in terms of storage and performance. In this PR, I have modified the checkpoint format to utilize the flatbuffer table to save the checkpoint to a single file, providing a more compact and efficient representation. The changes around this are twofold: - Add the checkpoint flatbuffer schema that will generate the necessary checkpoint source files. - Update the checkpoint saving and loading functionality to use the new format. - Adding support for onnxruntime minimal build: To support scenarios where binary size is a constraint, I made changes to ensure that the training build can work well with the minimal build. 🔍 __Open Issues:__ - In order to extract the optimizer type, the existing implementation re-loaded the onnx optimizer model and parsed it. This is no longer possible, since the model format can either be onnx or ort. One idea is to do the same for ort format optimizer model. This needs some investigation. - Changes to the offline tooling to generate ort format training artifacts. - End-to-end training example showcasing the use of the minimal training build. - Add support for export model for inferencing in a minimal build.
2023-06-22 19:27:23 +00:00
if (onnxruntime_ENABLE_TRAINING)
Introduce training changes.
2020-03-11 21:25:37 +00:00
target_include_directories(onnxruntime_optimizer PRIVATE ${ORTTRAINING_ROOT})
Flash attention recompute (#20603) ### Flash attn recompute 1. Allow PythonOp(FlashAttn) can be recomputed correctly. https://github.com/microsoft/onnxruntime/pull/20603/commits/45879ff5c20bf4cc11840b38b1808572126c5368 2. Use JSON to pass the selected-to-recompute subgraphs. https://github.com/microsoft/onnxruntime/pull/20603/commits/3c374da6788474cd09ba931eb0b00a45fa3f43e0 #### Better Memory Efficiency Customer model can run both PyTorch SPDA and Flash Attn, this PR make it possible to let the Flash Attn path work with ORTModule layerwise recompute. The peak drop from 45.xGB to 32.xGB if we only compare the layers (not including other pieces, BTW there are few more optimization targeting other pieces as well later). #### Better Perf Using Flash ATTN bring additionally 16% end to end time reduction, with highly aligned loss curve. ![image](https://github.com/microsoft/onnxruntime/assets/10530022/bb63894a-f281-49bc-a8e6-ff818439be38) #### Use JSON File to pass Recompute Plans To overcome the limitation of max length of the strings defined in session options. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. -->
2024-05-21 05:38:19 +00:00
onnxruntime_add_include_to_target(onnxruntime_optimizer nlohmann_json::nlohmann_json)
Support inplace update for PythonOp/Grad (#17687) ### Support inplace update for PythonOp/Grad This PR is based on another PR https://github.com/microsoft/onnxruntime/pull/17685's branch, to make it easier to review. With PR: PR https://github.com/microsoft/onnxruntime/pull/17685, By default all PythonOp inputs/outputs are assumed to not be inplaced, if during run, we found some inplace update happens (by checking output data address with all inputs data address), we add clone before set it as PythonOp/Grad's outputs. In this case, results are correct, but implicit copies overheads are introduced. This PR allow users to define output input reuse map, to let ORT know how to do the reuse map, avoid such unnecessary copies.
2023-10-11 04:36:45 +00:00
if (onnxruntime_ENABLE_TRAINING_TORCH_INTEROP)
onnxruntime_add_include_to_target(onnxruntime_optimizer Python::Module)
endif()
Introduce training changes.
2020-03-11 21:25:37 +00:00
endif()
Triton Codegen for ORTModule (#15831) Fuse connected elementwise and reduce Ops to TritonOp and codegen triton code to run the kernel. This PR is co-edited by @wejoncy and @er3x3
2023-07-13 10:17:58 +00:00
if (onnxruntime_ENABLE_TRITON)
target_link_libraries(onnxruntime_optimizer PRIVATE nlohmann_json::nlohmann_json)
onnxruntime_add_include_to_target(onnxruntime_optimizer Python::Module)
endif()
Create a separate component for graph optimization. (#421) * Create a project for graph optimizer. Move optimizer related code to the folder optimizer. * Fix build failures. * rebase and fix build failures. * fix build failure. * fix build failure with cuda path. * fix python build failure. * Move two transformers(memcpy and insert_cast) from framework to optimizer. * rebase. * SessionState should not depend on optimizer.
2019-02-04 23:45:12 +00:00
add_dependencies(onnxruntime_optimizer ${onnxruntime_EXTERNAL_DEPENDENCIES})
set_target_properties(onnxruntime_optimizer PROPERTIES FOLDER "ONNXRuntime")
[cmake] Export multi targets in static build (#11063) * [cmake] Export multi targets in static build * Install more components in static build, format some code * Fix code pos
2022-04-04 05:37:18 +00:00
if (NOT onnxruntime_BUILD_SHARED_LIB)
added support for cmake "find_package" (#8919) **Description**: Adds support for cmake find_package. **Motivation and Context** As mentioned in issue #7150 onnxruntime doesn't have support for CMake find_package, this PR adds that and also adds the CMake package version file. Now anyone can link onnxruntime like this: ```cmake find_package(onnxruntime) add_executable(test Source.cpp) target_link_libraries(test PRIVATE onnxruntime::onnxruntime) ``` this also simplifies #3124
2023-06-20 05:20:31 +00:00
install(DIRECTORY ${PROJECT_SOURCE_DIR}/../include/onnxruntime/core/optimizer DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/onnxruntime/core)
install(TARGETS onnxruntime_optimizer
[cmake] Export multi targets in static build (#11063) * [cmake] Export multi targets in static build * Install more components in static build, format some code * Fix code pos
2022-04-04 05:37:18 +00:00
ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR}
LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}
RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR}
FRAMEWORK DESTINATION ${CMAKE_INSTALL_BINDIR})
endif()
[ROCm] Fix hipify error: fast_divmod.h: No such file or directory (#19060) Fix error: ``` [ 48%] Built target onnxruntime_optimizer In file included from /onnxruntime_src/onnxruntime/core/providers/rocm/rocm_stream_handle.cc:5: /onnxruntime_src/onnxruntime/core/providers/rocm/rocm_common.h:11:10: fatal error: core/providers/rocm/shared_inc/fast_divmod.h: No such file or directory 11 | #include "core/providers/rocm/shared_inc/fast_divmod.h" | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ compilation terminated. ``` This error is due to onnxruntime_optimizer missing dependencies on hipify generated files.
2024-01-10 06:49:19 +00:00
if (onnxruntime_USE_ROCM)
add_dependencies(onnxruntime_optimizer generate_hipified_files)
endif()
Reference in a new issue Copy permalink