2019-01-11 19:08:33 +00:00
2020-04-06 22:52:48 +00:00
<!DOCTYPE html>
2019-12-21 01:11:46 +00:00
2020-08-13 02:12:50 +00:00
< html lang = "en" >
2020-04-06 22:52:48 +00:00
< head >
< meta charset = "utf-8" / >
2021-02-11 00:34:31 +00:00
< meta name = "viewport" content = "width=device-width, initial-scale=1.0" / >
< title > API Summary — ONNX Runtime 1.6.0 documentation< / title >
< link rel = "stylesheet" href = "static/pygments.css" type = "text/css" / >
< link rel = "stylesheet" href = "static/alabaster.css" type = "text/css" / >
< link rel = "stylesheet" type = "text/css" href = "static/graphviz.css" / >
< link rel = "stylesheet" type = "text/css" href = "static/gallery.css" / >
< link rel = "stylesheet" type = "text/css" href = "static/gallery-binder.css" / >
< link rel = "stylesheet" type = "text/css" href = "static/gallery-dataframe.css" / >
< link rel = "stylesheet" type = "text/css" href = "static/gallery-rendered-html.css" / >
< script id = "documentation_options" data-url_root = "./" src = "static/documentation_options.js" > < / script >
< script src = "static/jquery.js" > < / script >
< script src = "static/underscore.js" > < / script >
< script src = "static/doctools.js" > < / script >
2020-04-06 22:52:48 +00:00
< link rel = "index" title = "Index" href = "genindex.html" / >
< link rel = "search" title = "Search" href = "search.html" / >
< link rel = "next" title = "Gallery of examples" href = "auto_examples/index.html" / >
< link rel = "prev" title = "Tutorial" href = "tutorial.html" / >
2021-02-11 00:34:31 +00:00
< link rel = "stylesheet" href = "static/custom.css" type = "text/css" / >
2020-04-06 22:52:48 +00:00
< meta name = "viewport" content = "width=device-width, initial-scale=0.9, maximum-scale=0.9" / >
< / head > < body >
< div class = "document" >
< div class = "documentwrapper" >
< div class = "bodywrapper" >
< div class = "body" role = "main" >
< div class = "section" id = "api-summary" >
2019-01-11 19:08:33 +00:00
< h1 > API Summary< a class = "headerlink" href = "#api-summary" title = "Permalink to this headline" > ¶< / a > < / h1 >
< p > Summary of public functions and classes exposed
in < em > ONNX Runtime< / em > .< / p >
< div class = "contents local topic" id = "contents" >
< ul class = "simple" >
2021-02-11 00:34:31 +00:00
< li > < p > < a class = "reference internal" href = "#ortvalue" id = "id1" > OrtValue< / a > < / p > < / li >
< li > < p > < a class = "reference internal" href = "#iobinding" id = "id2" > IOBinding< / a > < / p > < / li >
< li > < p > < a class = "reference internal" href = "#device" id = "id3" > Device< / a > < / p > < / li >
< li > < p > < a class = "reference internal" href = "#examples-and-datasets" id = "id4" > Examples and datasets< / a > < / p > < / li >
< li > < p > < a class = "reference internal" href = "#load-and-run-a-model" id = "id5" > Load and run a model< / a > < / p > < / li >
< li > < p > < a class = "reference internal" href = "#backend" id = "id6" > Backend< / a > < / p > < / li >
2019-01-11 19:08:33 +00:00
< / ul >
< / div >
2021-02-11 00:34:31 +00:00
< div class = "section" id = "ortvalue" >
< h2 > < a class = "toc-backref" href = "#id1" > OrtValue< / a > < a class = "headerlink" href = "#ortvalue" title = "Permalink to this headline" > ¶< / a > < / h2 >
< p > < em > ONNX Runtime< / em > works with native Python data structures which are mapped into ONNX data formats :
Numpy arrays (tensors), dictionaries (maps), and a list of Numpy arrays (sequences).
The data backing these are on CPU.< / p >
< p > < em > ONNX Runtime< / em > supports a custom data structure that supports all ONNX data formats that allows users
to place the data backing these on a device, for example, on a CUDA supported device. This allows for
interesting < em > IOBinding< / em > scenarios (discussed below). In addition, < em > ONNX Runtime< / em > supports directly
working with < em > OrtValue< / em > (s) while inferencing a model if provided as part of the input feed.< / p >
< p > Below is an example showing creation of an < em > OrtValue< / em > from a Numpy array while placing its backing memory
on a CUDA device:< / p >
< div class = "highlight-python notranslate" > < div class = "highlight" > < pre > < span > < / span > < span class = "c1" > #X is numpy array on cpu, create an OrtValue and place it on cuda device id = 0< / span >
< span class = "n" > ortvalue< / span > < span class = "o" > =< / span > < span class = "n" > onnxruntime< / span > < span class = "o" > .< / span > < span class = "n" > OrtValue< / span > < span class = "o" > .< / span > < span class = "n" > ortvalue_from_numpy< / span > < span class = "p" > (< / span > < span class = "n" > X< / span > < span class = "p" > ,< / span > < span class = "s1" > ' cuda' < / span > < span class = "p" > ,< / span > < span class = "mi" > 0< / span > < span class = "p" > )< / span >
< span class = "n" > ortvalue< / span > < span class = "o" > .< / span > < span class = "n" > device_name< / span > < span class = "p" > ()< / span > < span class = "c1" > # ' cuda' < / span >
< span class = "n" > ortvalue< / span > < span class = "o" > .< / span > < span class = "n" > shape< / span > < span class = "p" > ()< / span > < span class = "c1" > # shape of the numpy array X< / span >
< span class = "n" > ortvalue< / span > < span class = "o" > .< / span > < span class = "n" > data_type< / span > < span class = "p" > ()< / span > < span class = "c1" > # ' tensor(float)' < / span >
< span class = "n" > ortvalue< / span > < span class = "o" > .< / span > < span class = "n" > is_tensor< / span > < span class = "p" > ()< / span > < span class = "c1" > # ' True' < / span >
< span class = "n" > np< / span > < span class = "o" > .< / span > < span class = "n" > array_equal< / span > < span class = "p" > (< / span > < span class = "n" > ortvalue< / span > < span class = "o" > .< / span > < span class = "n" > numpy< / span > < span class = "p" > (),< / span > < span class = "n" > X< / span > < span class = "p" > )< / span > < span class = "c1" > # ' True' < / span >
< span class = "c1" > #ortvalue can be provided as part of the input feed to a model< / span >
< span class = "n" > ses< / span > < span class = "o" > =< / span > < span class = "n" > onnxruntime< / span > < span class = "o" > .< / span > < span class = "n" > InferenceSession< / span > < span class = "p" > (< / span > < span class = "s1" > ' model.onnx' < / span > < span class = "p" > )< / span >
< span class = "n" > res< / span > < span class = "o" > =< / span > < span class = "n" > sess< / span > < span class = "o" > .< / span > < span class = "n" > run< / span > < span class = "p" > ([< / span > < span class = "s2" > " Y" < / span > < span class = "p" > ],< / span > < span class = "p" > {< / span > < span class = "s2" > " X" < / span > < span class = "p" > :< / span > < span class = "n" > ortvalue< / span > < span class = "p" > })< / span >
< / pre > < / div >
< / div >
< / div >
2020-08-13 02:12:50 +00:00
< div class = "section" id = "iobinding" >
2021-02-11 00:34:31 +00:00
< h2 > < a class = "toc-backref" href = "#id2" > IOBinding< / a > < a class = "headerlink" href = "#iobinding" title = "Permalink to this headline" > ¶< / a > < / h2 >
2020-08-13 02:12:50 +00:00
< p > By default, < em > ONNX Runtime< / em > always places input(s) and output(s) on CPU, which
is not optimal if the input or output is consumed and produced on a device
other than CPU because it introduces data copy between CPU and the device.
< em > ONNX Runtime< / em > provides a feature, < em > IO Binding< / em > , which addresses this issue by
enabling users to specify which device to place input(s) and output(s) on.
Here are scenarios to use this feature.< / p >
< p > (In the following code snippets, < em > model.onnx< / em > is the model to execute,
< em > X< / em > is the input data to feed, and < em > Y< / em > is the output data.)< / p >
< p > Scenario 1:< / p >
2021-02-11 00:34:31 +00:00
< p > A graph is executed on a device other than CPU, for instance CUDA. Users can
2020-08-13 02:12:50 +00:00
use IOBinding to put input on CUDA as the follows.< / p >
< div class = "highlight-python notranslate" > < div class = "highlight" > < pre > < span > < / span > < span class = "c1" > #X is numpy array on cpu< / span >
< span class = "n" > session< / span > < span class = "o" > =< / span > < span class = "n" > onnxruntime< / span > < span class = "o" > .< / span > < span class = "n" > InferenceSession< / span > < span class = "p" > (< / span > < span class = "s1" > ' model.onnx' < / span > < span class = "p" > )< / span >
< span class = "n" > io_binding< / span > < span class = "o" > =< / span > < span class = "n" > session< / span > < span class = "o" > .< / span > < span class = "n" > io_binding< / span > < span class = "p" > ()< / span >
2021-02-11 00:34:31 +00:00
< span class = "c1" > # OnnxRuntime will copy the data over to the CUDA device if ' input' is consumed by nodes on the CUDA device< / span >
2020-08-13 02:12:50 +00:00
< span class = "n" > io_binding< / span > < span class = "o" > .< / span > < span class = "n" > bind_cpu_input< / span > < span class = "p" > (< / span > < span class = "s1" > ' input' < / span > < span class = "p" > ,< / span > < span class = "n" > X< / span > < span class = "p" > )< / span >
< span class = "n" > io_binding< / span > < span class = "o" > .< / span > < span class = "n" > bind_output< / span > < span class = "p" > (< / span > < span class = "s1" > ' output' < / span > < span class = "p" > )< / span >
< span class = "n" > session< / span > < span class = "o" > .< / span > < span class = "n" > run_with_iobinding< / span > < span class = "p" > (< / span > < span class = "n" > io_binding< / span > < span class = "p" > )< / span >
< span class = "n" > Y< / span > < span class = "o" > =< / span > < span class = "n" > io_binding< / span > < span class = "o" > .< / span > < span class = "n" > copy_outputs_to_cpu< / span > < span class = "p" > ()[< / span > < span class = "mi" > 0< / span > < span class = "p" > ]< / span >
< / pre > < / div >
< / div >
< p > Scenario 2:< / p >
2021-02-11 00:34:31 +00:00
< p > The input data is on a device, users directly use the input. The output data is on CPU.< / p >
< div class = "highlight-python notranslate" > < div class = "highlight" > < pre > < span > < / span > < span class = "c1" > #X is numpy array on cpu< / span >
< span class = "n" > X_ortvalue< / span > < span class = "o" > =< / span > < span class = "n" > onnxruntime< / span > < span class = "o" > .< / span > < span class = "n" > OrtValue< / span > < span class = "o" > .< / span > < span class = "n" > ortvalue_from_numpy< / span > < span class = "p" > (< / span > < span class = "n" > X< / span > < span class = "p" > ,< / span > < span class = "s1" > ' cuda' < / span > < span class = "p" > ,< / span > < span class = "mi" > 0< / span > < span class = "p" > )< / span >
< span class = "n" > session< / span > < span class = "o" > =< / span > < span class = "n" > onnxruntime< / span > < span class = "o" > .< / span > < span class = "n" > InferenceSession< / span > < span class = "p" > (< / span > < span class = "s1" > ' model.onnx' < / span > < span class = "p" > )< / span >
2020-08-13 02:12:50 +00:00
< span class = "n" > io_binding< / span > < span class = "o" > =< / span > < span class = "n" > session< / span > < span class = "o" > .< / span > < span class = "n" > io_binding< / span > < span class = "p" > ()< / span >
2021-02-11 00:34:31 +00:00
< span class = "n" > io_binding< / span > < span class = "o" > .< / span > < span class = "n" > bind_input< / span > < span class = "p" > (< / span > < span class = "n" > name< / span > < span class = "o" > =< / span > < span class = "s1" > ' input' < / span > < span class = "p" > ,< / span > < span class = "n" > device_type< / span > < span class = "o" > =< / span > < span class = "n" > X_ortvalue< / span > < span class = "o" > .< / span > < span class = "n" > device_name< / span > < span class = "p" > (),< / span > < span class = "n" > device_id< / span > < span class = "o" > =< / span > < span class = "mi" > 0< / span > < span class = "p" > ,< / span > < span class = "n" > element_type< / span > < span class = "o" > =< / span > < span class = "n" > np< / span > < span class = "o" > .< / span > < span class = "n" > float32< / span > < span class = "p" > ,< / span > < span class = "n" > shape< / span > < span class = "o" > =< / span > < span class = "n" > X_ortvalue< / span > < span class = "o" > .< / span > < span class = "n" > shape< / span > < span class = "p" > (),< / span > < span class = "n" > buffer_ptr< / span > < span class = "o" > =< / span > < span class = "n" > X_ortvalue< / span > < span class = "o" > .< / span > < span class = "n" > data_ptr< / span > < span class = "p" > ())< / span >
2020-08-13 02:12:50 +00:00
< span class = "n" > io_binding< / span > < span class = "o" > .< / span > < span class = "n" > bind_output< / span > < span class = "p" > (< / span > < span class = "s1" > ' output' < / span > < span class = "p" > )< / span >
< span class = "n" > session< / span > < span class = "o" > .< / span > < span class = "n" > run_with_iobinding< / span > < span class = "p" > (< / span > < span class = "n" > io_binding< / span > < span class = "p" > )< / span >
< span class = "n" > Y< / span > < span class = "o" > =< / span > < span class = "n" > io_binding< / span > < span class = "o" > .< / span > < span class = "n" > copy_outputs_to_cpu< / span > < span class = "p" > ()[< / span > < span class = "mi" > 0< / span > < span class = "p" > ]< / span >
< / pre > < / div >
< / div >
< p > Scenario 3:< / p >
2021-02-11 00:34:31 +00:00
< p > The input data and output data are both on a device, users directly use the input and also place output on the device.< / p >
< div class = "highlight-python notranslate" > < div class = "highlight" > < pre > < span > < / span > < span class = "c1" > #X is numpy array on cpu< / span >
< span class = "n" > X_ortvalue< / span > < span class = "o" > =< / span > < span class = "n" > onnxruntime< / span > < span class = "o" > .< / span > < span class = "n" > OrtValue< / span > < span class = "o" > .< / span > < span class = "n" > ortvalue_from_numpy< / span > < span class = "p" > (< / span > < span class = "n" > X< / span > < span class = "p" > ,< / span > < span class = "s1" > ' cuda' < / span > < span class = "p" > ,< / span > < span class = "mi" > 0< / span > < span class = "p" > )< / span >
< span class = "n" > Y_ortvalue< / span > < span class = "o" > =< / span > < span class = "n" > onnxruntime< / span > < span class = "o" > .< / span > < span class = "n" > OrtValue< / span > < span class = "o" > .< / span > < span class = "n" > ortvalue_from_shape_and_type< / span > < span class = "p" > ([< / span > < span class = "mi" > 3< / span > < span class = "p" > ,< / span > < span class = "mi" > 2< / span > < span class = "p" > ],< / span > < span class = "n" > np< / span > < span class = "o" > .< / span > < span class = "n" > float32< / span > < span class = "p" > ,< / span > < span class = "s1" > ' cuda' < / span > < span class = "p" > ,< / span > < span class = "mi" > 0< / span > < span class = "p" > )< / span > < span class = "c1" > # Change the shape to the actual shape of the output being bound< / span >
< span class = "n" > session< / span > < span class = "o" > =< / span > < span class = "n" > onnxruntime< / span > < span class = "o" > .< / span > < span class = "n" > InferenceSession< / span > < span class = "p" > (< / span > < span class = "s1" > ' model.onnx' < / span > < span class = "p" > )< / span >
2020-08-13 02:12:50 +00:00
< span class = "n" > io_binding< / span > < span class = "o" > =< / span > < span class = "n" > session< / span > < span class = "o" > .< / span > < span class = "n" > io_binding< / span > < span class = "p" > ()< / span >
2021-02-11 00:34:31 +00:00
< span class = "n" > io_binding< / span > < span class = "o" > .< / span > < span class = "n" > bind_input< / span > < span class = "p" > (< / span > < span class = "n" > name< / span > < span class = "o" > =< / span > < span class = "s1" > ' input' < / span > < span class = "p" > ,< / span > < span class = "n" > device_type< / span > < span class = "o" > =< / span > < span class = "n" > X_ortvalue< / span > < span class = "o" > .< / span > < span class = "n" > device_name< / span > < span class = "p" > (),< / span > < span class = "n" > device_id< / span > < span class = "o" > =< / span > < span class = "mi" > 0< / span > < span class = "p" > ,< / span > < span class = "n" > element_type< / span > < span class = "o" > =< / span > < span class = "n" > np< / span > < span class = "o" > .< / span > < span class = "n" > float32< / span > < span class = "p" > ,< / span > < span class = "n" > shape< / span > < span class = "o" > =< / span > < span class = "n" > X_ortvalue< / span > < span class = "o" > .< / span > < span class = "n" > shape< / span > < span class = "p" > (),< / span > < span class = "n" > buffer_ptr< / span > < span class = "o" > =< / span > < span class = "n" > X_ortvalue< / span > < span class = "o" > .< / span > < span class = "n" > data_ptr< / span > < span class = "p" > ())< / span >
< span class = "n" > io_binding< / span > < span class = "o" > .< / span > < span class = "n" > bind_output< / span > < span class = "p" > (< / span > < span class = "n" > name< / span > < span class = "o" > =< / span > < span class = "s1" > ' output' < / span > < span class = "p" > ,< / span > < span class = "n" > device_type< / span > < span class = "o" > =< / span > < span class = "n" > Y_ortvalue< / span > < span class = "o" > .< / span > < span class = "n" > device_name< / span > < span class = "p" > (),< / span > < span class = "n" > device_id< / span > < span class = "o" > =< / span > < span class = "mi" > 0< / span > < span class = "p" > ,< / span > < span class = "n" > element_type< / span > < span class = "o" > =< / span > < span class = "n" > np< / span > < span class = "o" > .< / span > < span class = "n" > float32< / span > < span class = "p" > ,< / span > < span class = "n" > shape< / span > < span class = "o" > =< / span > < span class = "n" > Y_ortvalue< / span > < span class = "o" > .< / span > < span class = "n" > shape< / span > < span class = "p" > (),< / span > < span class = "n" > buffer_ptr< / span > < span class = "o" > =< / span > < span class = "n" > Y_ortvalue< / span > < span class = "o" > .< / span > < span class = "n" > data_ptr< / span > < span class = "p" > ())< / span >
< span class = "n" > session< / span > < span class = "o" > .< / span > < span class = "n" > run_with_iobinding< / span > < span class = "p" > (< / span > < span class = "n" > io_binding< / span > < span class = "p" > )< / span >
< / pre > < / div >
< / div >
< p > Scenario 4:< / p >
< p > Users can request < em > ONNX Runtime< / em > to allocate an output on a device. This is particularly useful for dynamic shaped outputs.
Users can use the < em > get_outputs()< / em > API to get access to the < em > OrtValue< / em > (s) corresponding to the allocated output(s).
Users can thus consume the < em > ONNX Runtime< / em > allocated memory for the output as an < em > OrtValue< / em > .< / p >
< div class = "highlight-python notranslate" > < div class = "highlight" > < pre > < span > < / span > < span class = "c1" > #X is numpy array on cpu< / span >
< span class = "n" > X_ortvalue< / span > < span class = "o" > =< / span > < span class = "n" > onnxruntime< / span > < span class = "o" > .< / span > < span class = "n" > OrtValue< / span > < span class = "o" > .< / span > < span class = "n" > ortvalue_from_numpy< / span > < span class = "p" > (< / span > < span class = "n" > X< / span > < span class = "p" > ,< / span > < span class = "s1" > ' cuda' < / span > < span class = "p" > ,< / span > < span class = "mi" > 0< / span > < span class = "p" > )< / span >
< span class = "n" > session< / span > < span class = "o" > =< / span > < span class = "n" > onnxruntime< / span > < span class = "o" > .< / span > < span class = "n" > InferenceSession< / span > < span class = "p" > (< / span > < span class = "s1" > ' model.onnx' < / span > < span class = "p" > )< / span >
< span class = "n" > io_binding< / span > < span class = "o" > =< / span > < span class = "n" > session< / span > < span class = "o" > .< / span > < span class = "n" > io_binding< / span > < span class = "p" > ()< / span >
< span class = "n" > io_binding< / span > < span class = "o" > .< / span > < span class = "n" > bind_input< / span > < span class = "p" > (< / span > < span class = "n" > name< / span > < span class = "o" > =< / span > < span class = "s1" > ' input' < / span > < span class = "p" > ,< / span > < span class = "n" > device_type< / span > < span class = "o" > =< / span > < span class = "n" > X_ortvalue< / span > < span class = "o" > .< / span > < span class = "n" > device_name< / span > < span class = "p" > (),< / span > < span class = "n" > device_id< / span > < span class = "o" > =< / span > < span class = "mi" > 0< / span > < span class = "p" > ,< / span > < span class = "n" > element_type< / span > < span class = "o" > =< / span > < span class = "n" > np< / span > < span class = "o" > .< / span > < span class = "n" > float32< / span > < span class = "p" > ,< / span > < span class = "n" > shape< / span > < span class = "o" > =< / span > < span class = "n" > X_ortvalue< / span > < span class = "o" > .< / span > < span class = "n" > shape< / span > < span class = "p" > (),< / span > < span class = "n" > buffer_ptr< / span > < span class = "o" > =< / span > < span class = "n" > X_ortvalue< / span > < span class = "o" > .< / span > < span class = "n" > data_ptr< / span > < span class = "p" > ())< / span >
< span class = "c1" > #Request ONNX Runtime to bind and allocate memory on CUDA for ' output' < / span >
< span class = "n" > io_binding< / span > < span class = "o" > .< / span > < span class = "n" > bind_output< / span > < span class = "p" > (< / span > < span class = "s1" > ' output' < / span > < span class = "p" > ,< / span > < span class = "s1" > ' cuda' < / span > < span class = "p" > )< / span >
< span class = "n" > session< / span > < span class = "o" > .< / span > < span class = "n" > run_with_iobinding< / span > < span class = "p" > (< / span > < span class = "n" > io_binding< / span > < span class = "p" > )< / span >
< span class = "c1" > # The following call returns an OrtValue which has data allocated by ONNX Runtime on CUDA< / span >
< span class = "n" > ort_output< / span > < span class = "o" > =< / span > < span class = "n" > io_binding< / span > < span class = "o" > .< / span > < span class = "n" > get_outputs< / span > < span class = "p" > ()[< / span > < span class = "mi" > 0< / span > < span class = "p" > ]< / span >
< / pre > < / div >
< / div >
< p > Scenario 5:< / p >
< p > Users can bind < em > OrtValue< / em > (s) directly.< / p >
< div class = "highlight-python notranslate" > < div class = "highlight" > < pre > < span > < / span > < span class = "c1" > #X is numpy array on cpu< / span >
< span class = "c1" > #X is numpy array on cpu< / span >
< span class = "n" > X_ortvalue< / span > < span class = "o" > =< / span > < span class = "n" > onnxruntime< / span > < span class = "o" > .< / span > < span class = "n" > OrtValue< / span > < span class = "o" > .< / span > < span class = "n" > ortvalue_from_numpy< / span > < span class = "p" > (< / span > < span class = "n" > X< / span > < span class = "p" > ,< / span > < span class = "s1" > ' cuda' < / span > < span class = "p" > ,< / span > < span class = "mi" > 0< / span > < span class = "p" > )< / span >
< span class = "n" > Y_ortvalue< / span > < span class = "o" > =< / span > < span class = "n" > onnxruntime< / span > < span class = "o" > .< / span > < span class = "n" > OrtValue< / span > < span class = "o" > .< / span > < span class = "n" > ortvalue_from_shape_and_type< / span > < span class = "p" > ([< / span > < span class = "mi" > 3< / span > < span class = "p" > ,< / span > < span class = "mi" > 2< / span > < span class = "p" > ],< / span > < span class = "n" > np< / span > < span class = "o" > .< / span > < span class = "n" > float32< / span > < span class = "p" > ,< / span > < span class = "s1" > ' cuda' < / span > < span class = "p" > ,< / span > < span class = "mi" > 0< / span > < span class = "p" > )< / span > < span class = "c1" > # Change the shape to the actual shape of the output being bound< / span >
< span class = "n" > session< / span > < span class = "o" > =< / span > < span class = "n" > onnxruntime< / span > < span class = "o" > .< / span > < span class = "n" > InferenceSession< / span > < span class = "p" > (< / span > < span class = "s1" > ' model.onnx' < / span > < span class = "p" > )< / span >
< span class = "n" > io_binding< / span > < span class = "o" > =< / span > < span class = "n" > session< / span > < span class = "o" > .< / span > < span class = "n" > io_binding< / span > < span class = "p" > ()< / span >
< span class = "n" > io_binding< / span > < span class = "o" > .< / span > < span class = "n" > bind_ortvalue_input< / span > < span class = "p" > (< / span > < span class = "s1" > ' input' < / span > < span class = "p" > ,< / span > < span class = "n" > X_ortvalue< / span > < span class = "p" > )< / span >
< span class = "n" > io_binding< / span > < span class = "o" > .< / span > < span class = "n" > bind_ortvalue_output< / span > < span class = "p" > (< / span > < span class = "s1" > ' output' < / span > < span class = "p" > ,< / span > < span class = "n" > Y_ortvalue< / span > < span class = "p" > )< / span >
2020-08-13 02:12:50 +00:00
< span class = "n" > session< / span > < span class = "o" > .< / span > < span class = "n" > run_with_iobinding< / span > < span class = "p" > (< / span > < span class = "n" > io_binding< / span > < span class = "p" > )< / span >
< / pre > < / div >
< / div >
< / div >
2019-01-11 19:08:33 +00:00
< div class = "section" id = "device" >
2021-02-11 00:34:31 +00:00
< h2 > < a class = "toc-backref" href = "#id3" > Device< / a > < a class = "headerlink" href = "#device" title = "Permalink to this headline" > ¶< / a > < / h2 >
2019-01-11 19:08:33 +00:00
< p > The package is compiled for a specific device, GPU or CPU.
The CPU implementation includes optimizations
such as MKL (Math Kernel Libary). The following function
indicates the chosen option:< / p >
2020-08-13 02:12:50 +00:00
< dl class = "py function" >
2019-01-11 19:08:33 +00:00
< dt id = "onnxruntime.get_device" >
2021-02-11 00:34:31 +00:00
< code class = "sig-prename descclassname" > onnxruntime.< / code > < code class = "sig-name descname" > get_device< / code > < span class = "sig-paren" > (< / span > < span class = "sig-paren" > )< / span > → < a class = "reference external" href = "https://docs.python.org/3/library/stdtypes.html#str" title = "(in Python v3.9)" > str< / a > < a class = "headerlink" href = "#onnxruntime.get_device" title = "Permalink to this definition" > ¶< / a > < / dt >
2019-01-11 19:08:33 +00:00
< dd > < p > Return the device used to compute the prediction (CPU, MKL, …)< / p >
< / dd > < / dl >
< / div >
< div class = "section" id = "examples-and-datasets" >
2021-02-11 00:34:31 +00:00
< h2 > < a class = "toc-backref" href = "#id4" > Examples and datasets< / a > < a class = "headerlink" href = "#examples-and-datasets" title = "Permalink to this headline" > ¶< / a > < / h2 >
2019-01-11 19:08:33 +00:00
< p > The package contains a few models stored in ONNX format
used in the documentation. These don’
as they are installed with the package.< / p >
2020-08-13 02:12:50 +00:00
< dl class = "py function" >
2019-01-11 19:08:33 +00:00
< dt id = "onnxruntime.datasets.get_example" >
2021-02-11 00:34:31 +00:00
< code class = "sig-prename descclassname" > onnxruntime.datasets.< / code > < code class = "sig-name descname" > get_example< / code > < span class = "sig-paren" > (< / span > < em class = "sig-param" > < span class = "n" > name< / span > < / em > < span class = "sig-paren" > )< / span > < a class = "reference internal" href = "modules/onnxruntime/datasets.html#get_example" > < span class = "viewcode-link" > [source]< / span > < / a > < a class = "headerlink" href = "#onnxruntime.datasets.get_example" title = "Permalink to this definition" > ¶< / a > < / dt >
2019-01-11 19:08:33 +00:00
< dd > < p > Retrieves the absolute file name of an example.< / p >
< / dd > < / dl >
< / div >
< div class = "section" id = "load-and-run-a-model" >
2021-02-11 00:34:31 +00:00
< h2 > < a class = "toc-backref" href = "#id5" > Load and run a model< / a > < a class = "headerlink" href = "#load-and-run-a-model" title = "Permalink to this headline" > ¶< / a > < / h2 >
2019-01-11 19:08:33 +00:00
< p > < em > ONNX Runtime< / em > reads a model saved in ONNX format.
The main class < em > InferenceSession< / em > wraps these functionalities
in a single place.< / p >
2020-08-13 02:12:50 +00:00
< dl class = "py class" >
2019-01-11 19:08:33 +00:00
< dt id = "onnxruntime.ModelMetadata" >
2019-12-20 21:38:07 +00:00
< em class = "property" > class < / em > < code class = "sig-prename descclassname" > onnxruntime.< / code > < code class = "sig-name descname" > ModelMetadata< / code > < a class = "headerlink" href = "#onnxruntime.ModelMetadata" title = "Permalink to this definition" > ¶< / a > < / dt >
2019-01-11 19:08:33 +00:00
< dd > < p > Pre-defined and custom metadata about the model.
It is usually used to identify the model used to run the prediction and
facilitate the comparison.< / p >
2020-08-13 02:12:50 +00:00
< dl class = "py method" >
2019-01-11 19:08:33 +00:00
< dt id = "onnxruntime.ModelMetadata.custom_metadata_map" >
2019-12-20 21:38:07 +00:00
< em class = "property" > property < / em > < code class = "sig-name descname" > custom_metadata_map< / code > < a class = "headerlink" href = "#onnxruntime.ModelMetadata.custom_metadata_map" title = "Permalink to this definition" > ¶< / a > < / dt >
2019-01-11 19:08:33 +00:00
< dd > < p > additional metadata< / p >
< / dd > < / dl >
2020-08-13 02:12:50 +00:00
< dl class = "py method" >
2019-01-11 19:08:33 +00:00
< dt id = "onnxruntime.ModelMetadata.description" >
2019-12-20 21:38:07 +00:00
< em class = "property" > property < / em > < code class = "sig-name descname" > description< / code > < a class = "headerlink" href = "#onnxruntime.ModelMetadata.description" title = "Permalink to this definition" > ¶< / a > < / dt >
2019-01-11 19:08:33 +00:00
< dd > < p > description of the model< / p >
< / dd > < / dl >
2020-08-13 02:12:50 +00:00
< dl class = "py method" >
2019-01-11 19:08:33 +00:00
< dt id = "onnxruntime.ModelMetadata.domain" >
2019-12-20 21:38:07 +00:00
< em class = "property" > property < / em > < code class = "sig-name descname" > domain< / code > < a class = "headerlink" href = "#onnxruntime.ModelMetadata.domain" title = "Permalink to this definition" > ¶< / a > < / dt >
2019-01-11 19:08:33 +00:00
< dd > < p > ONNX domain< / p >
< / dd > < / dl >
2021-02-11 00:34:31 +00:00
< dl class = "py method" >
< dt id = "onnxruntime.ModelMetadata.graph_description" >
< em class = "property" > property < / em > < code class = "sig-name descname" > graph_description< / code > < a class = "headerlink" href = "#onnxruntime.ModelMetadata.graph_description" title = "Permalink to this definition" > ¶< / a > < / dt >
< dd > < p > description of the graph hosted in the model< / p >
< / dd > < / dl >
2020-08-13 02:12:50 +00:00
< dl class = "py method" >
2019-01-11 19:08:33 +00:00
< dt id = "onnxruntime.ModelMetadata.graph_name" >
2019-12-20 21:38:07 +00:00
< em class = "property" > property < / em > < code class = "sig-name descname" > graph_name< / code > < a class = "headerlink" href = "#onnxruntime.ModelMetadata.graph_name" title = "Permalink to this definition" > ¶< / a > < / dt >
2019-01-11 19:08:33 +00:00
< dd > < p > graph name< / p >
< / dd > < / dl >
2020-08-13 02:12:50 +00:00
< dl class = "py method" >
2019-01-11 19:08:33 +00:00
< dt id = "onnxruntime.ModelMetadata.producer_name" >
2019-12-20 21:38:07 +00:00
< em class = "property" > property < / em > < code class = "sig-name descname" > producer_name< / code > < a class = "headerlink" href = "#onnxruntime.ModelMetadata.producer_name" title = "Permalink to this definition" > ¶< / a > < / dt >
2019-01-11 19:08:33 +00:00
< dd > < p > producer name< / p >
< / dd > < / dl >
2020-08-13 02:12:50 +00:00
< dl class = "py method" >
2019-01-11 19:08:33 +00:00
< dt id = "onnxruntime.ModelMetadata.version" >
2019-12-20 21:38:07 +00:00
< em class = "property" > property < / em > < code class = "sig-name descname" > version< / code > < a class = "headerlink" href = "#onnxruntime.ModelMetadata.version" title = "Permalink to this definition" > ¶< / a > < / dt >
2019-01-11 19:08:33 +00:00
< dd > < p > version of the model< / p >
< / dd > < / dl >
< / dd > < / dl >
2020-08-13 02:12:50 +00:00
< dl class = "py class" >
2019-01-11 19:08:33 +00:00
< dt id = "onnxruntime.InferenceSession" >
2021-02-11 00:34:31 +00:00
< em class = "property" > class < / em > < code class = "sig-prename descclassname" > onnxruntime.< / code > < code class = "sig-name descname" > InferenceSession< / code > < span class = "sig-paren" > (< / span > < em class = "sig-param" > < span class = "n" > path_or_bytes< / span > < / em > , < em class = "sig-param" > < span class = "n" > sess_options< / span > < span class = "o" > =< / span > < span class = "default_value" > None< / span > < / em > , < em class = "sig-param" > < span class = "n" > providers< / span > < span class = "o" > =< / span > < span class = "default_value" > None< / span > < / em > , < em class = "sig-param" > < span class = "n" > provider_options< / span > < span class = "o" > =< / span > < span class = "default_value" > None< / span > < / em > < span class = "sig-paren" > )< / span > < a class = "reference internal" href = "modules/onnxruntime/capi/onnxruntime_inference_collection.html#InferenceSession" > < span class = "viewcode-link" > [source]< / span > < / a > < a class = "headerlink" href = "#onnxruntime.InferenceSession" title = "Permalink to this definition" > ¶< / a > < / dt >
2019-01-11 19:08:33 +00:00
< dd > < p > This is the main class used to run a model.< / p >
2019-10-30 16:35:16 +00:00
< / dd > < / dl >
2020-08-13 02:12:50 +00:00
< dl class = "py class" >
2019-01-11 19:08:33 +00:00
< dt id = "onnxruntime.NodeArg" >
2019-12-20 21:38:07 +00:00
< em class = "property" > class < / em > < code class = "sig-prename descclassname" > onnxruntime.< / code > < code class = "sig-name descname" > NodeArg< / code > < a class = "headerlink" href = "#onnxruntime.NodeArg" title = "Permalink to this definition" > ¶< / a > < / dt >
2019-01-11 19:08:33 +00:00
< dd > < p > Node argument definition, for both input and output,
including arg name, arg type (contains both type and shape).< / p >
2020-08-13 02:12:50 +00:00
< dl class = "py method" >
2019-01-11 19:08:33 +00:00
< dt id = "onnxruntime.NodeArg.name" >
2019-12-20 21:38:07 +00:00
< em class = "property" > property < / em > < code class = "sig-name descname" > name< / code > < a class = "headerlink" href = "#onnxruntime.NodeArg.name" title = "Permalink to this definition" > ¶< / a > < / dt >
2019-01-11 19:08:33 +00:00
< dd > < p > node name< / p >
< / dd > < / dl >
2020-08-13 02:12:50 +00:00
< dl class = "py method" >
2019-01-11 19:08:33 +00:00
< dt id = "onnxruntime.NodeArg.shape" >
2019-12-20 21:38:07 +00:00
< em class = "property" > property < / em > < code class = "sig-name descname" > shape< / code > < a class = "headerlink" href = "#onnxruntime.NodeArg.shape" title = "Permalink to this definition" > ¶< / a > < / dt >
2019-01-11 19:08:33 +00:00
< dd > < p > node shape (assuming the node holds a tensor)< / p >
< / dd > < / dl >
2020-08-13 02:12:50 +00:00
< dl class = "py method" >
2019-01-11 19:08:33 +00:00
< dt id = "onnxruntime.NodeArg.type" >
2019-12-20 21:38:07 +00:00
< em class = "property" > property < / em > < code class = "sig-name descname" > type< / code > < a class = "headerlink" href = "#onnxruntime.NodeArg.type" title = "Permalink to this definition" > ¶< / a > < / dt >
2019-01-11 19:08:33 +00:00
< dd > < p > node type< / p >
< / dd > < / dl >
< / dd > < / dl >
2020-08-13 02:12:50 +00:00
< dl class = "py class" >
2019-01-11 19:08:33 +00:00
< dt id = "onnxruntime.RunOptions" >
2019-12-20 21:38:07 +00:00
< em class = "property" > class < / em > < code class = "sig-prename descclassname" > onnxruntime.< / code > < code class = "sig-name descname" > RunOptions< / code > < a class = "headerlink" href = "#onnxruntime.RunOptions" title = "Permalink to this definition" > ¶< / a > < / dt >
2019-01-11 19:08:33 +00:00
< dd > < p > Configuration information for a single Run.< / p >
2020-08-13 02:12:50 +00:00
< dl class = "py method" >
2019-10-30 16:35:16 +00:00
< dt id = "onnxruntime.RunOptions.log_severity_level" >
2019-12-20 21:38:07 +00:00
< em class = "property" > property < / em > < code class = "sig-name descname" > log_severity_level< / code > < a class = "headerlink" href = "#onnxruntime.RunOptions.log_severity_level" title = "Permalink to this definition" > ¶< / a > < / dt >
2019-08-02 01:12:59 +00:00
< dd > < p > Log severity level for a particular Run() invocation. 0:Verbose, 1:Info, 2:Warning. 3:Error, 4:Fatal. Default is 2.< / p >
< / dd > < / dl >
2020-08-13 02:12:50 +00:00
< dl class = "py method" >
2019-10-30 16:35:16 +00:00
< dt id = "onnxruntime.RunOptions.log_verbosity_level" >
2019-12-20 21:38:07 +00:00
< em class = "property" > property < / em > < code class = "sig-name descname" > log_verbosity_level< / code > < a class = "headerlink" href = "#onnxruntime.RunOptions.log_verbosity_level" title = "Permalink to this definition" > ¶< / a > < / dt >
2019-10-30 16:35:16 +00:00
< dd > < p > VLOG level if DEBUG build and run_log_severity_level is 0.
2019-08-02 01:12:59 +00:00
Applies to a particular Run() invocation. Default is 0.< / p >
2019-01-11 19:08:33 +00:00
< / dd > < / dl >
2020-08-13 02:12:50 +00:00
< dl class = "py method" >
2019-10-30 16:35:16 +00:00
< dt id = "onnxruntime.RunOptions.logid" >
2019-12-20 21:38:07 +00:00
< em class = "property" > property < / em > < code class = "sig-name descname" > logid< / code > < a class = "headerlink" href = "#onnxruntime.RunOptions.logid" title = "Permalink to this definition" > ¶< / a > < / dt >
2019-01-11 19:08:33 +00:00
< dd > < p > To identify logs generated by a particular Run() invocation.< / p >
< / dd > < / dl >
2020-08-13 02:12:50 +00:00
< dl class = "py method" >
< dt id = "onnxruntime.RunOptions.only_execute_path_to_fetches" >
< em class = "property" > property < / em > < code class = "sig-name descname" > only_execute_path_to_fetches< / code > < a class = "headerlink" href = "#onnxruntime.RunOptions.only_execute_path_to_fetches" title = "Permalink to this definition" > ¶< / a > < / dt >
< dd > < p > Only execute the nodes needed by fetch list< / p >
< / dd > < / dl >
< dl class = "py method" >
2019-01-11 19:08:33 +00:00
< dt id = "onnxruntime.RunOptions.terminate" >
2019-12-20 21:38:07 +00:00
< em class = "property" > property < / em > < code class = "sig-name descname" > terminate< / code > < a class = "headerlink" href = "#onnxruntime.RunOptions.terminate" title = "Permalink to this definition" > ¶< / a > < / dt >
2019-01-11 19:08:33 +00:00
< dd > < p > Set to True to terminate any currently executing calls that are using this
RunOptions instance. The individual calls will exit gracefully and return an error status.< / p >
< / dd > < / dl >
2021-02-11 00:34:31 +00:00
< dl class = "py method" >
< dt id = "onnxruntime.RunOptions.training_mode" >
< em class = "property" > property < / em > < code class = "sig-name descname" > training_mode< / code > < a class = "headerlink" href = "#onnxruntime.RunOptions.training_mode" title = "Permalink to this definition" > ¶< / a > < / dt >
< dd > < p > Choose to run in training or inferencing mode< / p >
< / dd > < / dl >
2019-01-11 19:08:33 +00:00
< / dd > < / dl >
2020-08-13 02:12:50 +00:00
< dl class = "py class" >
2019-01-11 19:08:33 +00:00
< dt id = "onnxruntime.SessionOptions" >
2019-12-20 21:38:07 +00:00
< em class = "property" > class < / em > < code class = "sig-prename descclassname" > onnxruntime.< / code > < code class = "sig-name descname" > SessionOptions< / code > < a class = "headerlink" href = "#onnxruntime.SessionOptions" title = "Permalink to this definition" > ¶< / a > < / dt >
2019-01-11 19:08:33 +00:00
< dd > < p > Configuration information for a session.< / p >
2021-02-11 00:34:31 +00:00
< dl class = "py method" >
< dt id = "onnxruntime.SessionOptions.add_free_dimension_override_by_denotation" >
< code class = "sig-name descname" > add_free_dimension_override_by_denotation< / code > < span class = "sig-paren" > (< / span > < em class = "sig-param" > < span class = "n" > self< / span > < span class = "p" > :< / span > < span class = "n" > onnxruntime.capi.onnxruntime_pybind11_state.SessionOptions< / span > < / em > , < em class = "sig-param" > < span class = "n" > arg0< / span > < span class = "p" > :< / span > < span class = "n" > < a class = "reference external" href = "https://docs.python.org/3/library/stdtypes.html#str" title = "(in Python v3.9)" > str< / a > < / span > < / em > , < em class = "sig-param" > < span class = "n" > arg1< / span > < span class = "p" > :< / span > < span class = "n" > < a class = "reference external" href = "https://docs.python.org/3/library/functions.html#int" title = "(in Python v3.9)" > int< / a > < / span > < / em > < span class = "sig-paren" > )< / span > → < a class = "reference external" href = "https://docs.python.org/3/library/constants.html#None" title = "(in Python v3.9)" > None< / a > < a class = "headerlink" href = "#onnxruntime.SessionOptions.add_free_dimension_override_by_denotation" title = "Permalink to this definition" > ¶< / a > < / dt >
< dd > < p > Specify the dimension size for each denotation associated with an input’ < / p >
< / dd > < / dl >
< dl class = "py method" >
< dt id = "onnxruntime.SessionOptions.add_free_dimension_override_by_name" >
< code class = "sig-name descname" > add_free_dimension_override_by_name< / code > < span class = "sig-paren" > (< / span > < em class = "sig-param" > < span class = "n" > self< / span > < span class = "p" > :< / span > < span class = "n" > onnxruntime.capi.onnxruntime_pybind11_state.SessionOptions< / span > < / em > , < em class = "sig-param" > < span class = "n" > arg0< / span > < span class = "p" > :< / span > < span class = "n" > < a class = "reference external" href = "https://docs.python.org/3/library/stdtypes.html#str" title = "(in Python v3.9)" > str< / a > < / span > < / em > , < em class = "sig-param" > < span class = "n" > arg1< / span > < span class = "p" > :< / span > < span class = "n" > < a class = "reference external" href = "https://docs.python.org/3/library/functions.html#int" title = "(in Python v3.9)" > int< / a > < / span > < / em > < span class = "sig-paren" > )< / span > → < a class = "reference external" href = "https://docs.python.org/3/library/constants.html#None" title = "(in Python v3.9)" > None< / a > < a class = "headerlink" href = "#onnxruntime.SessionOptions.add_free_dimension_override_by_name" title = "Permalink to this definition" > ¶< / a > < / dt >
< dd > < p > Specify values of named dimensions within model inputs.< / p >
< / dd > < / dl >
< dl class = "py method" >
< dt id = "onnxruntime.SessionOptions.add_initializer" >
< code class = "sig-name descname" > add_initializer< / code > < span class = "sig-paren" > (< / span > < em class = "sig-param" > < span class = "n" > self< / span > < span class = "p" > :< / span > < span class = "n" > onnxruntime.capi.onnxruntime_pybind11_state.SessionOptions< / span > < / em > , < em class = "sig-param" > < span class = "n" > arg0< / span > < span class = "p" > :< / span > < span class = "n" > < a class = "reference external" href = "https://docs.python.org/3/library/stdtypes.html#str" title = "(in Python v3.9)" > str< / a > < / span > < / em > , < em class = "sig-param" > < span class = "n" > arg1< / span > < span class = "p" > :< / span > < span class = "n" > < a class = "reference external" href = "https://docs.python.org/3/library/functions.html#object" title = "(in Python v3.9)" > object< / a > < / span > < / em > < span class = "sig-paren" > )< / span > → < a class = "reference external" href = "https://docs.python.org/3/library/constants.html#None" title = "(in Python v3.9)" > None< / a > < a class = "headerlink" href = "#onnxruntime.SessionOptions.add_initializer" title = "Permalink to this definition" > ¶< / a > < / dt >
< dd > < / dd > < / dl >
< dl class = "py method" >
< dt id = "onnxruntime.SessionOptions.add_session_config_entry" >
< code class = "sig-name descname" > add_session_config_entry< / code > < span class = "sig-paren" > (< / span > < em class = "sig-param" > < span class = "n" > self< / span > < span class = "p" > :< / span > < span class = "n" > onnxruntime.capi.onnxruntime_pybind11_state.SessionOptions< / span > < / em > , < em class = "sig-param" > < span class = "n" > arg0< / span > < span class = "p" > :< / span > < span class = "n" > < a class = "reference external" href = "https://docs.python.org/3/library/stdtypes.html#str" title = "(in Python v3.9)" > str< / a > < / span > < / em > , < em class = "sig-param" > < span class = "n" > arg1< / span > < span class = "p" > :< / span > < span class = "n" > < a class = "reference external" href = "https://docs.python.org/3/library/stdtypes.html#str" title = "(in Python v3.9)" > str< / a > < / span > < / em > < span class = "sig-paren" > )< / span > → < a class = "reference external" href = "https://docs.python.org/3/library/constants.html#None" title = "(in Python v3.9)" > None< / a > < a class = "headerlink" href = "#onnxruntime.SessionOptions.add_session_config_entry" title = "Permalink to this definition" > ¶< / a > < / dt >
< dd > < p > Set a single session configuration entry as a pair of strings.< / p >
< / dd > < / dl >
2020-08-13 02:12:50 +00:00
< dl class = "py method" >
2019-01-11 19:08:33 +00:00
< dt id = "onnxruntime.SessionOptions.enable_cpu_mem_arena" >
2019-12-20 21:38:07 +00:00
< em class = "property" > property < / em > < code class = "sig-name descname" > enable_cpu_mem_arena< / code > < a class = "headerlink" href = "#onnxruntime.SessionOptions.enable_cpu_mem_arena" title = "Permalink to this definition" > ¶< / a > < / dt >
2019-01-11 19:08:33 +00:00
< dd > < p > Enables the memory arena on CPU. Arena may pre-allocate memory for future usage.
Set this option to false if you don’ < / p >
< / dd > < / dl >
2020-08-13 02:12:50 +00:00
< dl class = "py method" >
2019-08-02 01:12:59 +00:00
< dt id = "onnxruntime.SessionOptions.enable_mem_pattern" >
2019-12-20 21:38:07 +00:00
< em class = "property" > property < / em > < code class = "sig-name descname" > enable_mem_pattern< / code > < a class = "headerlink" href = "#onnxruntime.SessionOptions.enable_mem_pattern" title = "Permalink to this definition" > ¶< / a > < / dt >
2019-08-02 01:12:59 +00:00
< dd > < p > Enable the memory pattern optimization. Default is true.< / p >
< / dd > < / dl >
2020-08-13 02:12:50 +00:00
< dl class = "py method" >
2019-01-11 19:08:33 +00:00
< dt id = "onnxruntime.SessionOptions.enable_profiling" >
2019-12-20 21:38:07 +00:00
< em class = "property" > property < / em > < code class = "sig-name descname" > enable_profiling< / code > < a class = "headerlink" href = "#onnxruntime.SessionOptions.enable_profiling" title = "Permalink to this definition" > ¶< / a > < / dt >
2019-01-11 19:08:33 +00:00
< dd > < p > Enable profiling for this session. Default is false.< / p >
< / dd > < / dl >
2020-08-13 02:12:50 +00:00
< dl class = "py method" >
2019-10-30 16:35:16 +00:00
< dt id = "onnxruntime.SessionOptions.execution_mode" >
2019-12-20 21:38:07 +00:00
< em class = "property" > property < / em > < code class = "sig-name descname" > execution_mode< / code > < a class = "headerlink" href = "#onnxruntime.SessionOptions.execution_mode" title = "Permalink to this definition" > ¶< / a > < / dt >
2019-10-30 16:35:16 +00:00
< dd > < p > Sets the execution mode. Default is sequential.< / p >
2019-01-11 19:08:33 +00:00
< / dd > < / dl >
2021-02-11 00:34:31 +00:00
< dl class = "py method" >
< dt id = "onnxruntime.SessionOptions.execution_order" >
< em class = "property" > property < / em > < code class = "sig-name descname" > execution_order< / code > < a class = "headerlink" href = "#onnxruntime.SessionOptions.execution_order" title = "Permalink to this definition" > ¶< / a > < / dt >
< dd > < p > Sets the execution order. Default is basic topological order.< / p >
< / dd > < / dl >
< dl class = "py method" >
< dt id = "onnxruntime.SessionOptions.get_session_config_entry" >
< code class = "sig-name descname" > get_session_config_entry< / code > < span class = "sig-paren" > (< / span > < em class = "sig-param" > < span class = "n" > self< / span > < span class = "p" > :< / span > < span class = "n" > onnxruntime.capi.onnxruntime_pybind11_state.SessionOptions< / span > < / em > , < em class = "sig-param" > < span class = "n" > arg0< / span > < span class = "p" > :< / span > < span class = "n" > < a class = "reference external" href = "https://docs.python.org/3/library/stdtypes.html#str" title = "(in Python v3.9)" > str< / a > < / span > < / em > < span class = "sig-paren" > )< / span > → < a class = "reference external" href = "https://docs.python.org/3/library/stdtypes.html#str" title = "(in Python v3.9)" > str< / a > < a class = "headerlink" href = "#onnxruntime.SessionOptions.get_session_config_entry" title = "Permalink to this definition" > ¶< / a > < / dt >
< dd > < p > Get a single session configuration value using the given configuration key.< / p >
< / dd > < / dl >
2020-08-13 02:12:50 +00:00
< dl class = "py method" >
2019-05-07 19:11:16 +00:00
< dt id = "onnxruntime.SessionOptions.graph_optimization_level" >
2019-12-20 21:38:07 +00:00
< em class = "property" > property < / em > < code class = "sig-name descname" > graph_optimization_level< / code > < a class = "headerlink" href = "#onnxruntime.SessionOptions.graph_optimization_level" title = "Permalink to this definition" > ¶< / a > < / dt >
2019-05-07 19:11:16 +00:00
< dd > < p > Graph optimization level for this session.< / p >
< / dd > < / dl >
2020-08-13 02:12:50 +00:00
< dl class = "py method" >
2019-10-30 16:35:16 +00:00
< dt id = "onnxruntime.SessionOptions.inter_op_num_threads" >
2019-12-20 21:38:07 +00:00
< em class = "property" > property < / em > < code class = "sig-name descname" > inter_op_num_threads< / code > < a class = "headerlink" href = "#onnxruntime.SessionOptions.inter_op_num_threads" title = "Permalink to this definition" > ¶< / a > < / dt >
2019-10-30 16:35:16 +00:00
< dd > < p > Sets the number of threads used to parallelize the execution of the graph (across nodes). Default is 0 to let onnxruntime choose.< / p >
2019-01-11 19:08:33 +00:00
< / dd > < / dl >
2020-08-13 02:12:50 +00:00
< dl class = "py method" >
2019-10-30 16:35:16 +00:00
< dt id = "onnxruntime.SessionOptions.intra_op_num_threads" >
2019-12-20 21:38:07 +00:00
< em class = "property" > property < / em > < code class = "sig-name descname" > intra_op_num_threads< / code > < a class = "headerlink" href = "#onnxruntime.SessionOptions.intra_op_num_threads" title = "Permalink to this definition" > ¶< / a > < / dt >
2019-10-30 16:35:16 +00:00
< dd > < p > Sets the number of threads used to parallelize the execution within nodes. Default is 0 to let onnxruntime choose.< / p >
2019-08-02 01:12:59 +00:00
< / dd > < / dl >
2020-08-13 02:12:50 +00:00
< dl class = "py method" >
2019-10-30 16:35:16 +00:00
< dt id = "onnxruntime.SessionOptions.log_severity_level" >
2019-12-20 21:38:07 +00:00
< em class = "property" > property < / em > < code class = "sig-name descname" > log_severity_level< / code > < a class = "headerlink" href = "#onnxruntime.SessionOptions.log_severity_level" title = "Permalink to this definition" > ¶< / a > < / dt >
2019-10-30 16:35:16 +00:00
< dd > < p > Log severity level. Applies to session load, initialization, etc.
0:Verbose, 1:Info, 2:Warning. 3:Error, 4:Fatal. Default is 2.< / p >
2019-01-11 19:08:33 +00:00
< / dd > < / dl >
2020-08-13 02:12:50 +00:00
< dl class = "py method" >
2019-10-30 16:35:16 +00:00
< dt id = "onnxruntime.SessionOptions.log_verbosity_level" >
2019-12-20 21:38:07 +00:00
< em class = "property" > property < / em > < code class = "sig-name descname" > log_verbosity_level< / code > < a class = "headerlink" href = "#onnxruntime.SessionOptions.log_verbosity_level" title = "Permalink to this definition" > ¶< / a > < / dt >
2021-02-11 00:34:31 +00:00
< dd > < p > VLOG level if DEBUG build and session_log_severity_level is 0.
2019-10-30 16:35:16 +00:00
Applies to session load, initialization, etc. Default is 0.< / p >
2019-01-11 19:08:33 +00:00
< / dd > < / dl >
2020-08-13 02:12:50 +00:00
< dl class = "py method" >
2019-10-30 16:35:16 +00:00
< dt id = "onnxruntime.SessionOptions.logid" >
2019-12-20 21:38:07 +00:00
< em class = "property" > property < / em > < code class = "sig-name descname" > logid< / code > < a class = "headerlink" href = "#onnxruntime.SessionOptions.logid" title = "Permalink to this definition" > ¶< / a > < / dt >
2019-10-30 16:35:16 +00:00
< dd > < p > Logger id to use for session output.< / p >
2019-01-11 19:08:33 +00:00
< / dd > < / dl >
2020-08-13 02:12:50 +00:00
< dl class = "py method" >
2019-10-30 16:35:16 +00:00
< dt id = "onnxruntime.SessionOptions.optimized_model_filepath" >
2019-12-20 21:38:07 +00:00
< em class = "property" > property < / em > < code class = "sig-name descname" > optimized_model_filepath< / code > < a class = "headerlink" href = "#onnxruntime.SessionOptions.optimized_model_filepath" title = "Permalink to this definition" > ¶< / a > < / dt >
2021-02-11 00:34:31 +00:00
< dd > < p > File path to serialize optimized model to.
Optimized model is not serialized unless optimized_model_filepath is set.
Serialized model format will default to ONNX unless:< / p >
< blockquote >
< div > < ul class = "simple" >
< li > < p > add_session_config_entry is used to set ‘ ’ ‘ ’ < / p > < / li >
< li > < p > there is no ‘ ’ ‘ ’ < / p > < / li >
< / ul >
< / div > < / blockquote >
< / dd > < / dl >
< dl class = "py method" >
< dt id = "onnxruntime.SessionOptions.profile_file_prefix" >
< em class = "property" > property < / em > < code class = "sig-name descname" > profile_file_prefix< / code > < a class = "headerlink" href = "#onnxruntime.SessionOptions.profile_file_prefix" title = "Permalink to this definition" > ¶< / a > < / dt >
< dd > < p > The prefix of the profile file. The current time will be appended to the file name.< / p >
< / dd > < / dl >
< dl class = "py method" >
< dt id = "onnxruntime.SessionOptions.register_custom_ops_library" >
< code class = "sig-name descname" > register_custom_ops_library< / code > < span class = "sig-paren" > (< / span > < em class = "sig-param" > < span class = "n" > self< / span > < span class = "p" > :< / span > < span class = "n" > onnxruntime.capi.onnxruntime_pybind11_state.SessionOptions< / span > < / em > , < em class = "sig-param" > < span class = "n" > arg0< / span > < span class = "p" > :< / span > < span class = "n" > < a class = "reference external" href = "https://docs.python.org/3/library/stdtypes.html#str" title = "(in Python v3.9)" > str< / a > < / span > < / em > < span class = "sig-paren" > )< / span > → < a class = "reference external" href = "https://docs.python.org/3/library/constants.html#None" title = "(in Python v3.9)" > None< / a > < a class = "headerlink" href = "#onnxruntime.SessionOptions.register_custom_ops_library" title = "Permalink to this definition" > ¶< / a > < / dt >
< dd > < p > Specify the path to the shared library containing the custom op kernels required to run a model.< / p >
2019-05-07 19:11:16 +00:00
< / dd > < / dl >
2020-08-13 02:12:50 +00:00
< dl class = "py method" >
< dt id = "onnxruntime.SessionOptions.use_deterministic_compute" >
< em class = "property" > property < / em > < code class = "sig-name descname" > use_deterministic_compute< / code > < a class = "headerlink" href = "#onnxruntime.SessionOptions.use_deterministic_compute" title = "Permalink to this definition" > ¶< / a > < / dt >
< dd > < p > Whether to use deterministic compute. Default is false.< / p >
< / dd > < / dl >
2019-01-11 19:08:33 +00:00
< / dd > < / dl >
< / div >
< div class = "section" id = "backend" >
2021-02-11 00:34:31 +00:00
< h2 > < a class = "toc-backref" href = "#id6" > Backend< / a > < a class = "headerlink" href = "#backend" title = "Permalink to this headline" > ¶< / a > < / h2 >
2019-01-11 19:08:33 +00:00
< p > In addition to the regular API which is optimized for performance and usability,
< em > ONNX Runtime< / em > also implements the
< a class = "reference external" href = "https://github.com/onnx/onnx/blob/master/docs/ImplementingAnOnnxBackend.md" > ONNX backend API< / a >
for verification of < em > ONNX< / em > specification conformance.
The following functions are supported:< / p >
2020-08-13 02:12:50 +00:00
< dl class = "py function" >
2019-01-11 19:08:33 +00:00
< dt id = "onnxruntime.backend.is_compatible" >
2020-08-13 02:12:50 +00:00
< code class = "sig-prename descclassname" > onnxruntime.backend.< / code > < code class = "sig-name descname" > is_compatible< / code > < span class = "sig-paren" > (< / span > < em class = "sig-param" > < span class = "n" > model< / span > < / em > , < em class = "sig-param" > < span class = "n" > device< / span > < span class = "o" > =< / span > < span class = "default_value" > None< / span > < / em > , < em class = "sig-param" > < span class = "o" > **< / span > < span class = "n" > kwargs< / span > < / em > < span class = "sig-paren" > )< / span > < a class = "headerlink" href = "#onnxruntime.backend.is_compatible" title = "Permalink to this definition" > ¶< / a > < / dt >
2019-01-11 19:08:33 +00:00
< dd > < p > Return whether the model is compatible with the backend.< / p >
2019-12-20 21:38:07 +00:00
< dl class = "field-list simple" >
< dt class = "field-odd" > Parameters< / dt >
< dd class = "field-odd" > < ul class = "simple" >
< li > < p > < strong > model< / strong > – < / p > < / li >
< li > < p > < strong > device< / strong > – < cite > ‘ ’ < / cite > )< / p > < / li >
2019-01-11 19:08:33 +00:00
< / ul >
2019-12-20 21:38:07 +00:00
< / dd >
< dt class = "field-even" > Returns< / dt >
< dd class = "field-even" > < p > boolean< / p >
< / dd >
< / dl >
2019-01-11 19:08:33 +00:00
< / dd > < / dl >
2020-08-13 02:12:50 +00:00
< dl class = "py function" >
2019-01-11 19:08:33 +00:00
< dt id = "onnxruntime.backend.prepare" >
2020-08-13 02:12:50 +00:00
< code class = "sig-prename descclassname" > onnxruntime.backend.< / code > < code class = "sig-name descname" > prepare< / code > < span class = "sig-paren" > (< / span > < em class = "sig-param" > < span class = "n" > model< / span > < / em > , < em class = "sig-param" > < span class = "n" > device< / span > < span class = "o" > =< / span > < span class = "default_value" > None< / span > < / em > , < em class = "sig-param" > < span class = "o" > **< / span > < span class = "n" > kwargs< / span > < / em > < span class = "sig-paren" > )< / span > < a class = "headerlink" href = "#onnxruntime.backend.prepare" title = "Permalink to this definition" > ¶< / a > < / dt >
2019-01-11 19:08:33 +00:00
< dd > < p > Load the model and creates a < a class = "reference internal" href = "#onnxruntime.InferenceSession" title = "onnxruntime.InferenceSession" > < code class = "xref py py-class docutils literal notranslate" > < span class = "pre" > onnxruntime.InferenceSession< / span > < / code > < / a >
ready to be used as a backend.< / p >
2019-12-20 21:38:07 +00:00
< dl class = "field-list simple" >
< dt class = "field-odd" > Parameters< / dt >
< dd class = "field-odd" > < ul class = "simple" >
< li > < p > < strong > model< / strong > – < cite > onnx.load< / cite > ),
string for a filename or bytes for a serialized model< / p > < / li >
< li > < p > < strong > device< / strong > –
2019-01-11 19:08:33 +00:00
None means the default one which depends on
2019-12-20 21:38:07 +00:00
the compilation settings< / p > < / li >
< li > < p > < strong > kwargs< / strong > – < a class = "reference internal" href = "#onnxruntime.SessionOptions" title = "onnxruntime.SessionOptions" > < code class = "xref py py-class docutils literal notranslate" > < span class = "pre" > onnxruntime.SessionOptions< / span > < / code > < / a > < / p > < / li >
2019-01-11 19:08:33 +00:00
< / ul >
2019-12-20 21:38:07 +00:00
< / dd >
< dt class = "field-even" > Returns< / dt >
< dd class = "field-even" > < p > < a class = "reference internal" href = "#onnxruntime.InferenceSession" title = "onnxruntime.InferenceSession" > < code class = "xref py py-class docutils literal notranslate" > < span class = "pre" > onnxruntime.InferenceSession< / span > < / code > < / a > < / p >
< / dd >
< / dl >
2019-01-11 19:08:33 +00:00
< / dd > < / dl >
2020-08-13 02:12:50 +00:00
< dl class = "py function" >
2019-01-11 19:08:33 +00:00
< dt id = "onnxruntime.backend.run" >
2020-08-13 02:12:50 +00:00
< code class = "sig-prename descclassname" > onnxruntime.backend.< / code > < code class = "sig-name descname" > run< / code > < span class = "sig-paren" > (< / span > < em class = "sig-param" > < span class = "n" > model< / span > < / em > , < em class = "sig-param" > < span class = "n" > inputs< / span > < / em > , < em class = "sig-param" > < span class = "n" > device< / span > < span class = "o" > =< / span > < span class = "default_value" > None< / span > < / em > , < em class = "sig-param" > < span class = "o" > **< / span > < span class = "n" > kwargs< / span > < / em > < span class = "sig-paren" > )< / span > < a class = "headerlink" href = "#onnxruntime.backend.run" title = "Permalink to this definition" > ¶< / a > < / dt >
2019-01-11 19:08:33 +00:00
< dd > < p > Compute the prediction.< / p >
2019-12-20 21:38:07 +00:00
< dl class = "field-list simple" >
< dt class = "field-odd" > Parameters< / dt >
< dd class = "field-odd" > < ul class = "simple" >
< li > < p > < strong > model< / strong > – < a class = "reference internal" href = "#onnxruntime.InferenceSession" title = "onnxruntime.InferenceSession" > < code class = "xref py py-class docutils literal notranslate" > < span class = "pre" > onnxruntime.InferenceSession< / span > < / code > < / a > returned
by function < em > prepare< / em > < / p > < / li >
< li > < p > < strong > inputs< / strong > – < / p > < / li >
< li > < p > < strong > device< / strong > –
2019-01-11 19:08:33 +00:00
None means the default one which depends on
2019-12-20 21:38:07 +00:00
the compilation settings< / p > < / li >
< li > < p > < strong > kwargs< / strong > – < a class = "reference internal" href = "#onnxruntime.RunOptions" title = "onnxruntime.RunOptions" > < code class = "xref py py-class docutils literal notranslate" > < span class = "pre" > onnxruntime.RunOptions< / span > < / code > < / a > < / p > < / li >
2019-01-11 19:08:33 +00:00
< / ul >
2019-12-20 21:38:07 +00:00
< / dd >
< dt class = "field-even" > Returns< / dt >
< dd class = "field-even" > < p > predictions< / p >
< / dd >
< / dl >
2019-01-11 19:08:33 +00:00
< / dd > < / dl >
2020-08-13 02:12:50 +00:00
< dl class = "py function" >
2019-01-11 19:08:33 +00:00
< dt id = "onnxruntime.backend.supports_device" >
2020-08-13 02:12:50 +00:00
< code class = "sig-prename descclassname" > onnxruntime.backend.< / code > < code class = "sig-name descname" > supports_device< / code > < span class = "sig-paren" > (< / span > < em class = "sig-param" > < span class = "n" > device< / span > < / em > < span class = "sig-paren" > )< / span > < a class = "headerlink" href = "#onnxruntime.backend.supports_device" title = "Permalink to this definition" > ¶< / a > < / dt >
2019-01-11 19:08:33 +00:00
< dd > < p > Check whether the backend is compiled with particular device support.
In particular it’ < / p >
< / dd > < / dl >
< / div >
< / div >
2020-04-06 22:52:48 +00:00
< / div >
< / div >
< / div >
< div class = "sphinxsidebar" role = "navigation" aria-label = "main navigation" >
< div class = "sphinxsidebarwrapper" >
< p class = "logo" > < a href = "index.html" >
2021-02-11 00:34:31 +00:00
< img class = "logo" src = "static/ONNX_Runtime_icon.png" alt = "Logo" / >
2020-04-06 22:52:48 +00:00
< / a > < / p >
< h1 class = "logo" > < a href = "index.html" > ONNX Runtime< / a > < / h1 >
< h3 > Navigation< / h3 >
< ul class = "current" >
< li class = "toctree-l1" > < a class = "reference internal" href = "tutorial.html" > Tutorial< / a > < / li >
< li class = "toctree-l1 current" > < a class = "current reference internal" href = "#" > API Summary< / a > < / li >
< li class = "toctree-l1" > < a class = "reference internal" href = "auto_examples/index.html" > Gallery of examples< / a > < / li >
< / ul >
< div class = "relations" >
< h3 > Related Topics< / h3 >
< ul >
< li > < a href = "index.html" > Documentation overview< / a > < ul >
< li > Previous: < a href = "tutorial.html" title = "previous chapter" > Tutorial< / a > < / li >
< li > Next: < a href = "auto_examples/index.html" title = "next chapter" > Gallery of examples< / a > < / li >
< / ul > < / li >
< / ul >
< / div >
< div id = "searchbox" style = "display: none" role = "search" >
< h3 id = "searchlabel" > Quick search< / h3 >
< div class = "searchformwrapper" >
< form class = "search" action = "search.html" method = "get" >
< input type = "text" name = "q" aria-labelledby = "searchlabel" / >
< input type = "submit" value = "Go" / >
< / form >
2019-01-11 19:08:33 +00:00
< / div >
< / div >
2020-08-13 02:12:50 +00:00
< script > $ ( '#searchbox' ) . show ( 0 ) ; < / script >
2020-04-06 22:52:48 +00:00
< / div >
< / div >
< div class = "clearer" > < / div >
< / div >
< div class = "footer" >
2021-02-11 00:34:31 +00:00
© 2018-2021, Microsoft.
2020-04-06 22:52:48 +00:00
|
2021-02-11 00:34:31 +00:00
Powered by < a href = "http://sphinx-doc.org/" > Sphinx 3.4.2< / a >
2020-04-06 22:52:48 +00:00
& < a href = "https://github.com/bitprophet/alabaster" > Alabaster 0.7.12< / a >
|
2021-02-11 00:34:31 +00:00
< a href = "sources/api_summary.rst.txt"
2020-04-06 22:52:48 +00:00
rel="nofollow">Page source< / a >
< / div >
2019-01-11 19:08:33 +00:00
2020-04-06 22:52:48 +00:00
< / body >
2019-01-11 19:08:33 +00:00
< / html >
