API Summary

Summary of public functions and classes exposed in ONNX Runtime.

Device

The package is compiled for a specific device, GPU or CPU. The CPU implementation includes optimizations such as MKL (Math Kernel Libary). The following function indicates the chosen option:

onnxruntime.get_device() → str

Return the device used to compute the prediction (CPU, MKL, …)

Examples and datasets

The package contains a few models stored in ONNX format used in the documentation. These don’t need to be downloaded as they are installed with the package.

onnxruntime.datasets.get_example(name)[source]

Retrieves the absolute file name of an example.

Load and run a model

ONNX Runtime reads a model saved in ONNX format. The main class InferenceSession wraps these functionalities in a single place.

class onnxruntime.ModelMetadata

Pre-defined and custom metadata about the model. It is usually used to identify the model used to run the prediction and facilitate the comparison.

property custom_metadata_map

additional metadata

property description

description of the model

property domain

ONNX domain

property graph_name

graph name

property producer_name

producer name

property version

version of the model

class onnxruntime.InferenceSession(path_or_bytes, sess_options=None)[source]

This is the main class used to run a model.

end_profiling()[source]

End profiling and return results in a file.

The results are stored in a filename if the option onnxruntime.SessionOptions.enable_profiling().

get_inputs()[source]

Return the inputs metadata as a list of onnxruntime.NodeArg.

get_modelmeta()[source]

Return the metadata. See onnxruntime.ModelMetadata.

get_outputs()[source]

Return the outputs metadata as a list of onnxruntime.NodeArg.

run(output_names, input_feed, run_options=None)[source]

Compute the predictions.

Parameters

  • output_names – name of the outputs

  • input_feed – dictionary { input_name: input_value }

  • run_options – See onnxruntime.RunOptions.

sess.run([output_name], {input_name: x})
class onnxruntime.NodeArg

Node argument definition, for both input and output, including arg name, arg type (contains both type and shape).

property name

node name

property shape

node shape (assuming the node holds a tensor)

property type

node type

class onnxruntime.RunOptions

Configuration information for a single Run.

property run_log_severity_level

Log severity level for a particular Run() invocation. 0:Verbose, 1:Info, 2:Warning. 3:Error, 4:Fatal. Default is 2.

property run_log_verbosity_level

VLOG level if DEBUG build and run_log_severity_level is 0. Applies to a particular Run() invocation. Default is 0.

property run_tag

To identify logs generated by a particular Run() invocation.

property terminate

Set to True to terminate any currently executing calls that are using this RunOptions instance. The individual calls will exit gracefully and return an error status.

class onnxruntime.SessionOptions

Configuration information for a session.

property enable_cpu_mem_arena

Enables the memory arena on CPU. Arena may pre-allocate memory for future usage. Set this option to false if you don’t want it. Default is True.

property enable_mem_pattern

Enable the memory pattern optimization. Default is true.

property enable_profiling

Enable profiling for this session. Default is false.

property enable_sequential_execution

Enables sequential execution, disables parallel execution. Default is true.

property graph_optimization_level

Graph optimization level for this session.

property max_num_graph_transformation_steps

Runs optimization steps on the execution graph. Default is 5.

property session_log_severity_level

Log severity level. Applies to session load, initialization, etc. 0:Verbose, 1:Info, 2:Warning. 3:Error, 4:Fatal. Default is 2.

property session_log_verbosity_level

VLOG level if DEBUG build and session_log_verbosity_level is 0. Applies to session load, initialization, etc. Default is 0.

property session_logid

Logger id to use for session output.

property session_thread_pool_size

How many threads in the session thread pool. Default is 0 to let onnxruntime choose. This parameter is unused unless enable_sequential_execution is false.

set_graph_optimization_level(self: onnxruntime.capi.onnxruntime_pybind11_state.SessionOptions, arg0: int) → None

Graph optimization level for this session. 0 disables all optimizations. Whereas 1 enables basic optimizations and 2 enables all optimizations.

Backend

In addition to the regular API which is optimized for performance and usability, ONNX Runtime also implements the ONNX backend API for verification of ONNX specification conformance. The following functions are supported:

onnxruntime.backend.is_compatible(model, device=None, **kwargs)

Return whether the model is compatible with the backend.

Parameters

  • model – unused

  • device – None to use the default device or a string (ex: ‘CPU’)

Returns

boolean

onnxruntime.backend.prepare(model, device=None, **kwargs)

Load the model and creates a onnxruntime.InferenceSession ready to be used as a backend.

Parameters

  • model – ModelProto (returned by onnx.load), string for a filename or bytes for a serialized model

  • device – requested device for the computation, None means the default one which depends on the compilation settings

  • kwargs – see onnxruntime.SessionOptions

Returns

onnxruntime.InferenceSession

onnxruntime.backend.run(model, inputs, device=None, **kwargs)

Compute the prediction.

Parameters
Returns

predictions

onnxruntime.backend.supports_device(device)

Check whether the backend is compiled with particular device support. In particular it’s used in the testing suite.