saymrwulf/onnxruntime
1
0
Fork 0
mirror of https://github.com/saymrwulf/onnxruntime.git synced 2026-06-04 23:59:56 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
onnxruntime/onnxruntime/test/python
History
kunal-vaishnavi cb69c59863
Add fusions for SigLIP and Conformer-Encoder (#23528) 
### Description
This PR adds fusions for [Google's SigLIP
model](https://huggingface.co/google/siglip-base-patch16-224/) and
Microsoft's internal conformer-encoder model.

Here is an example of how to run the ORT transformer optimizer for the
SigLIP model.
```
$ git clone https://github.com/microsoft/onnxruntime
$ cd onnxruntime/onnxruntime/python/tools/transformers
$ python3 optimizer.py --input /path/to/model.onnx --output /path/to/model_opt.onnx --model_type clip --num_heads 16 --hidden_size 1152 --use_external_data_format --opt_level 0 --disable_shape_inference
```

Here is an example of how to run the ORT transformer optimizer for the
conformer-encoder model.
```
$ git clone https://github.com/microsoft/onnxruntime
$ cd onnxruntime/onnxruntime/python/tools/transformers
$ python3 optimizer.py --input /path/to/model.onnx --output /path/to/model_opt.onnx --model_type conformer --num_heads 16 --hidden_size 1024 --use_external_data_format --opt_level 0 --disable_shape_inference --convert_attribute
```

### Motivation and Context
This PR helps optimize multi-modal models that use SigLIP for the vision
encoder and conformer-encoder for the speech encoder.

This PR uses changes from the following PRs:
- https://github.com/pytorch/pytorch/pull/144801
- https://github.com/microsoft/onnxscript/pull/2018
- https://github.com/microsoft/onnxscript/pull/2019
- https://github.com/microsoft/onnxscript/pull/2020
- https://github.com/microsoft/onnxscript/pull/2021
- https://github.com/microsoft/onnxscript/pull/2022
- https://github.com/microsoft/onnxscript/pull/2024
- https://github.com/microsoft/onnxscript/pull/2025
- https://github.com/microsoft/onnxscript/pull/2029
- https://github.com/microsoft/onnxscript/pull/2033

### Introduction of ONNX Script

This PR introduces [ONNX
Script](https://github.com/microsoft/onnxscript) into the ORT
transformer optimizer as an optional step via the
`fold_transpose_initializers()` method of the `DynamoOnnxHelper` class.
2025-01-31 09:17:49 -08:00
..
contrib_ops Target py310 and modernize codebase with ruff (#23401) 2025-01-16 19:10:14 -08:00
quantization Enable comprehension simplification in ruff rules (#23414) 2025-01-17 08:43:06 -08:00
testdata
transformers Add fusions for SigLIP and Conformer-Encoder (#23528) 2025-01-31 09:17:49 -08:00
helper.py
onnx_backend_test_series.py Target py310 and modernize codebase with ruff (#23401) 2025-01-16 19:10:14 -08:00
onnxruntime_test_collective.py
onnxruntime_test_distributed.py Target py310 and modernize codebase with ruff (#23401) 2025-01-16 19:10:14 -08:00
onnxruntime_test_engine_wrapper.py
onnxruntime_test_float8.py Enable comprehension simplification in ruff rules (#23414) 2025-01-17 08:43:06 -08:00
onnxruntime_test_float8_gemm8.py Enable comprehension simplification in ruff rules (#23414) 2025-01-17 08:43:06 -08:00
onnxruntime_test_python.py Enable dlpack by default (#23110) 2025-01-30 23:23:56 +01:00
onnxruntime_test_python_azure.py
onnxruntime_test_python_backend.py
onnxruntime_test_python_backend_mlops.py Ignore ruff rule N813 (#21477) 2024-07-24 17:48:22 -07:00
onnxruntime_test_python_cudagraph.py Target py310 and modernize codebase with ruff (#23401) 2025-01-16 19:10:14 -08:00
onnxruntime_test_python_dmlgraph.py Target py310 and modernize codebase with ruff (#23401) 2025-01-16 19:10:14 -08:00
onnxruntime_test_python_iobinding.py Use ruff as the formatter to replace black-isort (#23397) 2025-01-16 11:14:15 -08:00
onnxruntime_test_python_keras.py
onnxruntime_test_python_mlops.py Fix typos according to reviewdog report. (#21335) 2024-07-22 13:37:32 -07:00
onnxruntime_test_python_nested_control_flow_op.py Target py310 and modernize codebase with ruff (#23401) 2025-01-16 19:10:14 -08:00
onnxruntime_test_python_sparse_matmul.py Fix typos according to reviewdog report. (#21335) 2024-07-22 13:37:32 -07:00
onnxruntime_test_python_symbolic_shape_infer.py Enable comprehension simplification in ruff rules (#23414) 2025-01-17 08:43:06 -08:00
onnxruntime_test_scatternd.py Enable comprehension simplification in ruff rules (#23414) 2025-01-17 08:43:06 -08:00
requirements.txt Integrate onnx 1.17.0 (#21897) 2024-12-24 09:02:02 -08:00
test_pytorch_export_contrib_ops.py Target py310 and modernize codebase with ruff (#23401) 2025-01-16 19:10:14 -08:00