API¶
-API Overview¶
+API Overview¶
ONNX Runtime loads and runs inference on a model in ONNX graph format, or ORT format (for memory and disk constrained environments).
The data consumed and produced by the model can be specified and accessed in the way that best matches your scenario.
Load and run a model¶
+Load and run a model¶
InferenceSession is the main class of ONNX Runtime. It is used to load and run an ONNX model, as well as specify environment and application configuration options.
session = onnxruntime.InferenceSession('model.onnx')
@@ -135,10 +132,10 @@ profiling on the session:
Data inputs and outputs¶
+Data inputs and outputs¶
The ONNX Runtime Inference Session consumes and produces data using its OrtValue class.
Data on CPU¶
+Data on CPU¶
On CPU (the default), OrtValues can be mapped to and from native Python data structures: numpy arrays, dictionaries and lists of numpy arrays.
# X is numpy array on cpu
@@ -159,7 +156,7 @@ may not optimal if the input or output is consumed and produced on a device
other than CPU because it introduces data copy between CPU and the device.
-Data on device¶
+Data on device¶
ONNX Runtime supports a custom data structure that supports all ONNX data formats that allows users
to place the data backing these on a device, for example, on a CUDA supported device. In ONNX Runtime,
this called IOBinding.
@@ -261,15 +258,15 @@ Users can thus consume the ONNX Runtime allocated memory for the output
-API Details¶
+API Details¶
-InferenceSession¶
+InferenceSession¶
-
class onnxruntime.InferenceSession(path_or_bytes, sess_options=None, providers=None, provider_options=None, **kwargs)[source]¶
This is the main class used to run a model.
-- Parameters
+- Parameters:
path_or_bytes – filename or serialized ONNX or ORT format model in a byte string
sess_options – session options
Compute the predictions.
-- Parameters
+- Parameters:
output_names – name of the outputs
input_feed – dictionary { input_name: input_value }
run_options – See onnxruntime.RunOptions.
- Returns
+- Returns:
list of results, every result is either a numpy array,
a sparse tensor, a list or a dictionary.
Compute the predictions.
-- Parameters
+- Parameters:
iobinding – the iobinding object that has graph inputs/outputs bind.
run_options – See onnxruntime.RunOptions.
Compute the predictions.
-- Parameters
+- Parameters:
output_names – name of the outputs
input_dict_ort_values – dictionary { input_name: input_ort_value }
@@ -427,7 +424,7 @@ from numpy array or SparseTensor
run_options – See onnxruntime.RunOptions.
- Returns
+- Returns:
an array of OrtValue
@@ -441,7 +438,7 @@ from numpy array or SparseTensor
run_with_ortvaluevector(run_options, feed_names, feeds, fetch_names, fetches, fetch_devices)¶
Compute the predictions similar to other run_*() methods but with minimal C++/Python conversion overhead.
-- Parameters
+- Parameters:
run_options – See onnxruntime.RunOptions.
feed_names – list of input names.
Register the input list of execution providers. The underlying session is re-created.
-- Parameters
+- Parameters:
providers – Optional sequence of providers in order of decreasing
precedence. Values can either be provider names or tuples of
@@ -482,9 +479,9 @@ otherwise execute using CPUExecutionProvider.
-Options¶
+Options¶
-RunOptions¶
+RunOptions¶
-
class onnxruntime.RunOptions(self: onnxruntime.capi.onnxruntime_pybind11_state.RunOptions)¶
@@ -543,7 +540,7 @@ RunOptions instance. The individual calls will exit gracefully and return an err
-SessionOptions¶
+SessionOptions¶
-
class onnxruntime.SessionOptions(self: onnxruntime.capi.onnxruntime_pybind11_state.SessionOptions)¶
@@ -690,9 +687,9 @@ Serialized model format will default to ONNX unless:
-Data¶
+Data¶
-OrtValue¶
+OrtValue¶
-
class onnxruntime.OrtValue(ortvalue, numpy_obj=None)[source]¶
@@ -776,7 +773,7 @@ The new instance of OrtValue will assume the ownership of sparse_tensor
Factory method to construct an OrtValue (which holds a Tensor) from a given Numpy object
A copy of the data in the Numpy object is held by the OrtValue only if the device is NOT cpu
-- Parameters
+- Parameters:
numpy_obj – The Numpy object to construct the OrtValue from
device_type – e.g. cpu, cuda, cpu by default
Factory method to construct an OrtValue (which holds a Tensor) from given shape and element_type
-- Parameters
+- Parameters:
shape – List of integers indicating the shape of the OrtValue
element_type – The data type of the elements in the OrtValue (numpy type)
-SparseTensor¶
+SparseTensor¶
-
class onnxruntime.SparseTensor(sparse_tensor)[source]¶
@@ -897,7 +894,7 @@ You can query indices as:
static sparse_coo_from_numpy(dense_shape, values, coo_indices, ort_device)[source]¶
Factory method to construct a SparseTensor in COO format from given arguments
-- Parameters
+- Parameters:
dense_shape – 1-D numpy array(int64) or a python list that contains a dense_shape of the sparse tensor
must be on cpu memory
Factory method to construct a SparseTensor in CSR format from given arguments
-- Parameters
+- Parameters:
dense_shape – 1-D numpy array(int64) or a python list that contains a dense_shape of the
sparse tensor (rows, cols) must be on cpu memory
Returns a copy of this instance on the specified cuda device
-- Parameters
+- Parameters:
ort_device – with name ‘cuda’ and valid gpu device id
@@ -983,9 +980,9 @@ copies of the strings.
-Devices¶
+Devices¶
-IOBinding¶
+IOBinding¶
-
class onnxruntime.IOBinding(session)[source]¶
@@ -1002,7 +999,7 @@ copies of the strings.
-
bind_input(name, device_type, device_id, element_type, shape, buffer_ptr)[source]¶
-- Parameters
+- Parameters:
name – input name
device_type – e.g. cpu, cuda
-
bind_ortvalue_input(name, ortvalue)[source]¶
-- Parameters
+- Parameters:
name – input name
ortvalue – OrtValue instance to bind
-
bind_ortvalue_output(name, ortvalue)[source]¶
-- Parameters
+- Parameters:
name – output name
ortvalue – OrtValue instance to bind
-
bind_output(name, device_type='cpu', device_id=0, element_type=None, shape=None, buffer_ptr=None)[source]¶
-- Parameters
+- Parameters:
name – output name
device_type – e.g. cpu, cuda, cpu by default
-OrtDevice¶
+OrtDevice¶
-Internal classes¶
+Internal classes¶
These classes cannot be instantiated by users but they are returned
by methods or functions of this library.
-ModelMetadata¶
+ModelMetadata¶
-
class onnxruntime.ModelMetadata¶
@@ -1143,7 +1140,7 @@ facilitate the comparison.
-NodeArg¶
+NodeArg¶
-
class onnxruntime.NodeArg¶
@@ -1173,7 +1170,7 @@ including arg name, arg type (contains both type and shape).
-Backend¶
+Backend¶
In addition to the regular API which is optimized for performance and usability,
ONNX Runtime also implements the
ONNX backend API
@@ -1184,13 +1181,13 @@ The following functions are supported:
onnxruntime.backend.is_compatible(model, device=None, **kwargs)¶
Return whether the model is compatible with the backend.
-- Parameters
+- Parameters:
model – unused
device – None to use the default device or a string (ex: ‘CPU’)
- Returns
+- Returns:
boolean
@@ -1202,7 +1199,7 @@ The following functions are supported:
Load the model and creates a onnxruntime.InferenceSession
ready to be used as a backend.
-- Parameters
+- Parameters:
model – ModelProto (returned by onnx.load),
string for a filename or bytes for a serialized model
kwargs – see onnxruntime.SessionOptions
- Returns
+- Returns:
-
@@ -1223,7 +1220,7 @@ the compilation settings
onnxruntime.backend.run(model, inputs, device=None, **kwargs)¶
Compute the prediction.
-- Parameters
+- Parameters:
model – onnxruntime.InferenceSession returned
by function prepare
kwargs – see onnxruntime.RunOptions
- Returns
+- Returns:
predictions
@@ -1311,8 +1308,8 @@ In particular it’s used in the testing suite.
©2018-2021, Microsoft.
|
- Powered by Sphinx 5.3.0
- & Alabaster 0.7.12
+ Powered by Sphinx 6.1.3
+ & Alabaster 0.7.13
|
-
+
@@ -24,9 +24,6 @@
-
-
-
@@ -151,8 +148,8 @@
©2018-2021, Microsoft.
|
- Powered by Sphinx 5.3.0
- & Alabaster 0.7.12
+ Powered by Sphinx 6.1.3
+ & Alabaster 0.7.13
|
-
+
@@ -24,9 +24,6 @@
-
-
-
@@ -112,7 +109,7 @@ without using onnx.
The backend API is implemented by other frameworks
and makes it easier to switch between multiple runtimes
with the same API.
-Total running time of the script: ( 0 minutes 0.021 seconds)
+Total running time of the script: ( 0 minutes 0.019 seconds)
-Total running time of the script: ( 0 minutes 0.014 seconds)
+Total running time of the script: ( 0 minutes 0.011 seconds)
Download Python source code: plot_common_errors.py
@@ -284,8 +281,8 @@ ERROR with Shape=(2, 1, 2) - [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Invalid
©2018-2021, Microsoft.
|
- Powered by Sphinx 5.3.0
- & Alabaster 0.7.12
+ Powered by Sphinx 6.1.3
+ & Alabaster 0.7.13
|
-
+
@@ -24,9 +24,6 @@
-
-
-
@@ -61,14 +58,14 @@ starting with the training of a scikit-learn pipeline
which takes as inputs not a regular vector but a
dictionary { int: float } as its first step is a
DictVectorizer.
-
+
+
-Train a pipeline¶
+Train a pipeline¶
The first step consists in retrieving the boston datset.
import pandas
from sklearn.datasets import load_boston
@@ -143,12 +140,12 @@ and we show the confusion matrix.
print(r2_score(y_test, pred))
-0.8474132062469484
+0.92310182911155
-Conversion to ONNX format¶
+Conversion to ONNX format¶
We use module
sklearn-onnx
to convert the model into ONNX format.
@@ -200,12 +197,12 @@ ONNX Runtime expects one observation at a time.
print(r2_score(pred, pred_onx))
-0.9999999999999123
+0.9999999999999366
Very similar. ONNX Runtime uses floats instead of doubles,
that explains the small discrepencies.
-Total running time of the script: ( 0 minutes 1.374 seconds)
+Total running time of the script: ( 0 minutes 1.392 seconds)
-[array([[[0.72821426, 0.6339202 , 0.7272735 , 0.6409311 , 0.61518466],
- [0.7185246 , 0.640914 , 0.60131776, 0.6857518 , 0.70039463],
- [0.7135308 , 0.65056884, 0.58760184, 0.7135416 , 0.63284004],
- [0.7188247 , 0.5470599 , 0.58532 , 0.67812634, 0.6893187 ]],
+[array([[[0.6741172 , 0.7219819 , 0.61369073, 0.7058611 , 0.6851099 ],
+ [0.65471834, 0.59070265, 0.6932085 , 0.58628637, 0.68116176],
+ [0.6053132 , 0.6046966 , 0.5977106 , 0.5027518 , 0.6936471 ],
+ [0.584405 , 0.5433203 , 0.71175635, 0.5528623 , 0.66580206]],
- [[0.6211655 , 0.554035 , 0.55418974, 0.56652635, 0.62399405],
- [0.55780345, 0.6938668 , 0.5910147 , 0.59314 , 0.54391265],
- [0.7126377 , 0.65041703, 0.62936115, 0.69839984, 0.5651956 ],
- [0.6023196 , 0.6012137 , 0.7164181 , 0.59447944, 0.7121656 ]],
+ [[0.54814535, 0.5123932 , 0.68307173, 0.72042155, 0.63475394],
+ [0.64413285, 0.70860803, 0.6923877 , 0.5770614 , 0.53430355],
+ [0.57739794, 0.6555867 , 0.6817992 , 0.6585449 , 0.593924 ],
+ [0.64254063, 0.6924439 , 0.7242955 , 0.7274165 , 0.55333275]],
- [[0.6137416 , 0.65805656, 0.6757898 , 0.7231871 , 0.6186665 ],
- [0.662174 , 0.61860013, 0.6509645 , 0.57368857, 0.689906 ],
- [0.7140595 , 0.7226651 , 0.6117408 , 0.5281206 , 0.69103116],
- [0.7082236 , 0.7197187 , 0.6111821 , 0.6187154 , 0.53502613]]],
+ [[0.6204574 , 0.551248 , 0.54466 , 0.70989406, 0.5772743 ],
+ [0.67696744, 0.5659738 , 0.6278838 , 0.51225674, 0.6547737 ],
+ [0.6698583 , 0.527287 , 0.7128138 , 0.500769 , 0.6947695 ],
+ [0.5108964 , 0.694841 , 0.50497264, 0.6312783 , 0.7083145 ]]],
dtype=float32)]
-Total running time of the script: ( 0 minutes 0.013 seconds)
+Total running time of the script: ( 0 minutes 0.007 seconds)
-Total running time of the script: ( 0 minutes 0.010 seconds)
+Total running time of the script: ( 0 minutes 0.004 seconds)
Download Python source code: plot_metadata.py
@@ -186,8 +183,8 @@ version=0
©2018-2021, Microsoft.
|
- Powered by Sphinx 5.3.0
- & Alabaster 0.7.12
+ Powered by Sphinx 6.1.3
+ & Alabaster 0.7.13
|
-
+
@@ -24,9 +24,6 @@
-
-
-
@@ -61,14 +58,14 @@ in ONNX format than looking into its node with
onnx. This example demonstrates
how to draw a model and to retrieve it in json
format.
-
+
+
-Retrieve a model in JSON format¶
+Retrieve a model in JSON format¶
That’s the most simple way.
from onnxruntime.datasets import get_example
@@ -145,7 +142,7 @@ opset_import {
-Draw a model with ONNX¶
+Draw a model with ONNX¶
We use net_drawer.py
included in onnx package.
We use onnx to load the model
@@ -183,10 +180,10 @@ in a different way than before.
plt.imshow(image)
-
<matplotlib.image.AxesImage object at 0x7f3f51301840>
+<matplotlib.image.AxesImage object at 0x7f2b5427f280>
-Total running time of the script: ( 0 minutes 0.322 seconds)
+Total running time of the script: ( 0 minutes 0.290 seconds)
-onnxruntime_profile__2022-12-09_23-45-49.json
+onnxruntime_profile__2023-01-20_00-04-24.json
The results are stored un a file in JSON format.
@@ -122,20 +119,20 @@ Let’s see what it contains.
[{'args': {},
'cat': 'Session',
- 'dur': 65,
+ 'dur': 83,
'name': 'model_loading_array',
'ph': 'X',
- 'pid': 2734,
- 'tid': 2734,
- 'ts': 1},
+ 'pid': 2666,
+ 'tid': 2666,
+ 'ts': 2},
{'args': {},
'cat': 'Session',
- 'dur': 329,
+ 'dur': 335,
'name': 'session_initialization',
'ph': 'X',
- 'pid': 2734,
- 'tid': 2734,
- 'ts': 83}]
+ 'pid': 2666,
+ 'tid': 2666,
+ 'ts': 103}]
Total running time of the script: ( 0 minutes 0.007 seconds)
@@ -213,8 +210,8 @@ Let’s see what it contains.
©2018-2021, Microsoft.
|
- Powered by Sphinx 5.3.0
- & Alabaster 0.7.12
+ Powered by Sphinx 6.1.3
+ & Alabaster 0.7.13
|
-
+
@@ -24,9 +24,6 @@
-
-
-
@@ -58,16 +55,16 @@ to download the full example code
This example demonstrates an end to end scenario
starting with the training of a machine learned model
to its use in its converted from.
-
+
+
-Train a logistic regression¶
+Train a logistic regression¶
The first step consists in retrieving the iris datset.
from sklearn.datasets import load_iris
@@ -86,6 +83,16 @@ to its use in its converted from.
clr.fit(X_train, y_train)
+/home/runner/.local/lib/python3.10/site-packages/sklearn/linear_model/_logistic.py:444: ConvergenceWarning: lbfgs failed to converge (status=1):
+STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.
+
+Increase the number of iterations (max_iter) or scale the data as shown in:
+ https://scikit-learn.org/stable/modules/preprocessing.html
+Please also refer to the documentation for alternative solver options:
+ https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
+ n_iter_i = _check_optimize_result(
+
+
@@ -98,14 +105,14 @@ and we show the confusion matrix.
print(confusion_matrix(y_test, pred))
-[[ 9 0 0]
- [ 0 12 0]
- [ 0 1 16]]
+[[11 0 0]
+ [ 0 15 0]
+ [ 0 2 10]]
-Conversion to ONNX format¶
+Conversion to ONNX format¶
We use module
sklearn-onnx
to convert the model into ONNX format.
@@ -142,15 +149,15 @@ output name='output_label' and shape=[None]
print(confusion_matrix(pred, pred_onx))
-[[ 9 0 0]
- [ 0 13 0]
- [ 0 0 16]]
+[[11 0 0]
+ [ 0 17 0]
+ [ 0 0 10]]
The prediction are perfectly identical.
-Probabilities¶
+Probabilities¶
Probabilities are needed to compute other
relevant metrics such as the ROC Curve.
Let’s see how to get them first with
@@ -159,9 +166,9 @@ scikit-learn.
print(prob_sklearn[:3])
-[[9.74792469e-01 2.52074267e-02 1.04082095e-07]
- [2.04372454e-08 3.27727275e-03 9.96722707e-01]
- [3.58622821e-01 6.39558909e-01 1.81826980e-03]]
+[[2.51652941e-02 9.51347879e-01 2.34868266e-02]
+ [4.19825782e-02 8.96096750e-01 6.19206722e-02]
+ [1.35280393e-04 6.31904868e-02 9.36674233e-01]]
And then with ONNX Runtime.
@@ -174,9 +181,9 @@ The probabilies appear to be
pprint.pprint(prob_rt[0:3])
-[{0: 0.97479248046875, 1: 0.02520740032196045, 2: 1.0408190576072229e-07},
- {0: 2.043722346911636e-08, 1: 0.0032772724516689777, 2: 0.9967227578163147},
- {0: 0.35862284898757935, 1: 0.6395589113235474, 2: 0.0018182684434577823}]
+[{0: 0.02516530267894268, 1: 0.9513478875160217, 2: 0.02348681539297104},
+ {0: 0.041982587426900864, 1: 0.8960967659950256, 2: 0.06192063167691231},
+ {0: 0.00013528029376175255, 1: 0.06319047510623932, 2: 0.9366742372512817}]
Let’s benchmark.
@@ -200,11 +207,11 @@ The probabilies appear to be
Execution time for clr.predict
-Average 8.19e-05 min=7.17e-05 max=0.000104
+Average 5.2e-05 min=5.01e-05 max=6.67e-05
Execution time for ONNX Runtime
-Average 3.77e-05 min=3.46e-05 max=4.94e-05
+Average 3.26e-05 min=2.88e-05 max=4.42e-05
-3.7683185000219054e-05
+3.260208500009298e-05
Let’s benchmark a scenario similar to what a webservice
@@ -232,11 +239,11 @@ as opposed to a batch of prediction.
Execution time for clr.predict
-Average 0.00675 min=0.00645 max=0.00818
+Average 0.00474 min=0.00472 max=0.0048
Execution time for sess_predict
-Average 0.00192 min=0.00178 max=0.0025
+Average 0.00124 min=0.00122 max=0.00128
-0.0019168183650000968
+0.0012437381899999878
Let’s do the same for the probabilities.
@@ -253,11 +260,11 @@ Average 0.00192 min=0.00178 max=0.0025
Execution time for predict_proba
-Average 0.00959 min=0.00917 max=0.0106
+Average 0.00697 min=0.00695 max=0.00706
Execution time for sess_predict_proba
-Average 0.00197 min=0.00189 max=0.00221
+Average 0.0013 min=0.00129 max=0.00132
-0.001971747110000308
+0.0013020304249994296
This second comparison is better as
@@ -266,7 +273,7 @@ computes the label and the probabilities
in every case.
-Benchmark with RandomForest¶
+Benchmark with RandomForest¶
We first train and save a model in ONNX format.
from sklearn.ensemble import RandomForestClassifier
@@ -295,11 +302,11 @@ in every case.
Execution time for predict_proba
-Average 1.14 min=1.12 max=1.18
+Average 0.806 min=0.802 max=0.813
Execution time for sess_predict_proba
-Average 0.00248 min=0.00229 max=0.00335
+Average 0.00166 min=0.00163 max=0.00172
-0.002478402585000481
+0.0016625346749998473
Let’s see with different number of trees.
@@ -334,40 +341,40 @@ Average 0.00248 min=0.00229 max=0.00335
5
-Average 0.0958 min=0.0905 max=0.105
-Average 0.0017 min=0.0016 max=0.00179
+Average 0.0661 min=0.0649 max=0.0702
+Average 0.00122 min=0.00121 max=0.00125
10
-Average 0.149 min=0.147 max=0.15
-Average 0.00171 min=0.00159 max=0.0018
+Average 0.105 min=0.104 max=0.107
+Average 0.00125 min=0.00123 max=0.0013
15
-Average 0.196 min=0.193 max=0.198
-Average 0.00172 min=0.00162 max=0.00182
+Average 0.142 min=0.142 max=0.142
+Average 0.00124 min=0.00124 max=0.00127
20
-Average 0.246 min=0.243 max=0.248
-Average 0.00178 min=0.00167 max=0.00193
+Average 0.181 min=0.181 max=0.181
+Average 0.00126 min=0.00125 max=0.00129
25
-Average 0.318 min=0.313 max=0.323
-Average 0.00184 min=0.00173 max=0.00193
+Average 0.221 min=0.22 max=0.223
+Average 0.00129 min=0.00128 max=0.00132
30
-Average 0.358 min=0.352 max=0.364
-Average 0.00185 min=0.00177 max=0.0019
+Average 0.259 min=0.259 max=0.259
+Average 0.00129 min=0.00128 max=0.00133
35
-Average 0.408 min=0.405 max=0.411
-Average 0.00196 min=0.00187 max=0.00221
+Average 0.301 min=0.298 max=0.305
+Average 0.00133 min=0.00132 max=0.00136
40
-Average 0.481 min=0.477 max=0.484
-Average 0.00196 min=0.00191 max=0.002
+Average 0.336 min=0.335 max=0.336
+Average 0.00134 min=0.00132 max=0.00136
45
-Average 0.536 min=0.524 max=0.547
-Average 0.00197 min=0.00191 max=0.00207
+Average 0.378 min=0.376 max=0.381
+Average 0.00135 min=0.00133 max=0.00139
50
-Average 0.565 min=0.562 max=0.571
-Average 0.00203 min=0.00195 max=0.00217
+Average 0.414 min=0.414 max=0.416
+Average 0.00136 min=0.00134 max=0.0014
-<matplotlib.legend.Legend object at 0x7f3f3bce87c0>
+<matplotlib.legend.Legend object at 0x7f2b3cb88ee0>
-Total running time of the script: ( 5 minutes 17.899 seconds)
+Total running time of the script: ( 3 minutes 46.170 seconds)