https://github.com/microsoft/onnxruntime/pull/4639 changed the default
behavior by removing optimizer state from state_dict/checkpoint APIs.
The reason for the previous change was to allow models trained on ORT to
be used for inference on PyTorch, which is an important feature.
Due to the change aforementioned, when resuming training from a checkpoint,
the optimizer would start with random weights, leading to a bad performance.
This behavior would also cause reproducibility issues, as the optimizer
wouldnt be able to resume from its previous state.
This PR adds a boolean flag to state_dict/save_xheckpoint API that
when True (default) it saves both model and optimizer state.
When False, only the model state is kept.
* enable rejecting models based on onnx opset
* enable unreleased opsets in linux and mac CI
* test fixes and more updates
* enable unreleased opsets in CI builds
* enable released opsets in linux cis
* try fix windows ci yml
* yml fixes
* update yml
* yml updates post master merge
* review comments
* bug fix
Disabling this test until it's intermittent failure is root caused, this is a function and does not have a dedicated op by itself. However, this op is not used in known model to the best of my knowledge to disabling this test for the sanity of CI until the investigation is over is probably reasonable.
* make tensorizer events measures
* throttle the events and add a new one SoftwareBitmapToGPUTensorTelemetryEvent
* factor out timing code into a class
* typo
* typo
* move eventimer class into its own header file
* add throttling to detensorization and remove variable timing
* make detensorization events measures as well
* add ConvertGPUTensorToSoftwareBitmapTelemetryEvent event
* de-duplicate event names
* fix comment
* PR feedback
* regsiter part of opset13 cpu kernels; fix a bug in func impl; adjust reshapefusion order
* remove useless function
Co-authored-by: Cheng Tang <chenta@microsoft.com>
Adds EinSum operator (purely an EP kernel, not a dedicated DML operator), which takes an equation string and depending on the specifics is capable of representing: identity, diag, trace, transpose, reduce sum, dot product, matmul, elementwise multiplication, inner product, outer product.
The DML EP recognizes many of them (identity, transpose, reduce sum, 1D dot product, matmul, elementwise multiplication), but defers to CPU when not supported (extended inner product, outer product, diag, trace, arbitrary batch ellipsis).
https://github.com/onnx/onnx/blob/master/docs/Operators.md#Einsum
WindowsAI PR: https://microsoft.visualstudio.com/DefaultCollection/WindowsAI/_git/WindowsAI/pullrequest/5100608
Related work items: #27469790
* Optimized MatMulGrad for dB when B's shape is 2D
* Refactor for ConstantScalarNode
Co-authored-by: Sherlock Huang <bahuang@OrtTrainingDev3.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>
* matching multiple choice between new and old apis
* update according to reviewer's comments
Co-authored-by: liqun <liqun@OrtTrainingDev4.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>
* support Normalized_0_1 and Normalized_1_1
* add tests for Normalized_1_1
* fix build error
* fix imagetests failure
* support denterization and add more tests
* fix build
* remove added models
* disable gpu tests for CPU pipeline
* refactor based on comments and moved two added models
* merge normalizer and Denomalizer into NominalRangeConverter
* add comments
* little change
* fix build failure for amd64
* cuda kernel support
* on comments
* test UT
* test UT
* revert settings
* attempt to fix broken UT
* corrected UT fix
Co-authored-by: Ethan Tao <ettao@microsoft.com>
* Extend C++ API for Map/Sequence Type Info (#3517)
Expose functionality to view type information about sequences/maps
to C++ API.
- Add functions
- `TypeInfo::GetSequenceTypeInfo`
- `SequenceTypeInfo::GetSequenceElementType`
- `TypeInfo::GetMapTypeInfo`
- `MapTypeInfo::GetMapValueType`
- `MapTypeInfo::GetMapKeyType`
- Add structs
- `SequenceTypeInfo`
- `MapTypeInfo`
Co-authored-by: Dudeldu <mustermann.informatik@gmail.com>
Co-authored-by: Jonas-Heinrich <Jonas@JonasHeinrich.com>
* Extend tests to cover new type info functionality for sequences and maps
- two new test case in test_nontensor_types for maps and sequences
Co-authored-by: Jonas-Heinrich <Jonas@JonasHeinrich.com>
