Class InferenceSession

Represents an Inference Session on an ONNX Model. This is a IDisposable class and it must be disposed of using either a explicit call to Dispose() method or a pattern of using() block. If this is a member of another class that class must also become IDisposable and it must dispose of InferfenceSession in its Dispose() method.

Inheritance

System.Object

InferenceSession

Namespace: Microsoft.ML.OnnxRuntime

Assembly: cs.temp.dll.dll

Syntax

public class InferenceSession : IDisposable

Constructors

InferenceSession(Byte[])

Constructs an InferenceSession from a model data in byte array

Declaration

public InferenceSession(byte[] model)

Parameters

Type	Name	Description
System.Byte[]	model

InferenceSession(Byte[], PrePackedWeightsContainer)

Constructs an InferenceSession from a model data (in byte array) and it will use the provided pre-packed weights container to store and share pre-packed buffers of shared initializers across sessions if any.

Declaration

public InferenceSession(byte[] model, PrePackedWeightsContainer prepackedWeightsContainer)

Parameters

Type	Name	Description
System.Byte[]	model	Model as byte array
PrePackedWeightsContainer	prepackedWeightsContainer	Instance of PrepackedWeightsContainer. Lifetime of 'prepackedWeightsContainer' must be managed by the user and it must outlive any sessions reliant on it

InferenceSession(Byte[], SessionOptions)

Constructs an InferenceSession from a model data in byte array, with some additional session options

Declaration

public InferenceSession(byte[] model, SessionOptions options)

Parameters

Type	Name	Description
System.Byte[]	model
SessionOptions	options

InferenceSession(Byte[], SessionOptions, PrePackedWeightsContainer)

Constructs an InferenceSession from a model data (in byte array) with some additional session options and it will use the provided pre-packed weights container to store and share pre-packed buffers of shared initializers across sessions if any.

Declaration

public InferenceSession(byte[] model, SessionOptions options, PrePackedWeightsContainer prepackedWeightsContainer)

Parameters

Type	Name	Description
System.Byte[]	model	Model as byte array
SessionOptions	options	Session Options
PrePackedWeightsContainer	prepackedWeightsContainer	Instance of PrepackedWeightsContainer. Lifetime of 'prepackedWeightsContainer' must be managed by the user and it must outlive any sessions reliant on it

InferenceSession(String)

Constructs an InferenceSession from a model file

Declaration

public InferenceSession(string modelPath)

Parameters

Type	Name	Description
System.String	modelPath

InferenceSession(String, PrePackedWeightsContainer)

Constructs an InferenceSession from a model file and it will use the provided pre-packed weights container to store and share pre-packed buffers of shared initializers across sessions if any.

Declaration

public InferenceSession(string modelPath, PrePackedWeightsContainer prepackedWeightsContainer)

Parameters

Type	Name	Description
System.String	modelPath	Model path
PrePackedWeightsContainer	prepackedWeightsContainer	Instance of PrepackedWeightsContainer. Lifetime of 'prepackedWeightsContainer' must be managed by the user and it must outlive any sessions reliant on it

InferenceSession(String, SessionOptions)

Constructs an InferenceSession from a model file, with some additional session options

Declaration

public InferenceSession(string modelPath, SessionOptions options)

Parameters

Type	Name	Description
System.String	modelPath
SessionOptions	options

InferenceSession(String, SessionOptions, PrePackedWeightsContainer)

Constructs an InferenceSession from a model file, with some additional session options and it will use the provided pre-packed weights container to store and share pre-packed buffers of shared initializers across sessions if any.

Declaration

public InferenceSession(string modelPath, SessionOptions options, PrePackedWeightsContainer prepackedWeightsContainer)

Parameters

Type	Name	Description
System.String	modelPath	Model path
SessionOptions	options	Session options
PrePackedWeightsContainer	prepackedWeightsContainer	Instance of PrepackedWeightsContainer. Lifetime of 'prepackedWeightsContainer' must be managed by the user and it must outlive any sessions reliant on it

Properties

InputMetadata

Meta data regarding the input nodes, keyed by input names

Declaration

public IReadOnlyDictionary<string, NodeMetadata> InputMetadata { get; }

Property Value

Type	Description
IReadOnlyDictionary<System.String, NodeMetadata>

ModelMetadata

This property queries model metadata, constructs an instance of ModelMetadata and caches it

Declaration

public ModelMetadata ModelMetadata { get; }

Property Value

Type	Description
ModelMetadata	Instance of ModelMetdata

OutputMetadata

Metadata regarding the output nodes, keyed by output names

Declaration

public IReadOnlyDictionary<string, NodeMetadata> OutputMetadata { get; }

Property Value

Type	Description
IReadOnlyDictionary<System.String, NodeMetadata>

OverridableInitializerMetadata

Metadata regarding the overridable initializers, keyed by node names

Declaration

public IReadOnlyDictionary<string, NodeMetadata> OverridableInitializerMetadata { get; }

Property Value

Type	Description
IReadOnlyDictionary<System.String, NodeMetadata>

ProfilingStartTimeNs

Return the nanoseconds of profiling's start time On some platforms, this timer may not be as precise as nanoseconds For instance, on Windows and MacOS, the precision will be ~100ns

Declaration

public ulong ProfilingStartTimeNs { get; }

Property Value

Type	Description
System.UInt64

Methods

CreateIoBinding()

Create OrtIoBinding instance to bind pre-allocated buffers to input/output

Declaration

public OrtIoBinding CreateIoBinding()

Returns

Type	Description
OrtIoBinding	A new instance of OrtIoBinding

Dispose()

IDisposable implementation

Declaration

public void Dispose()

Dispose(Boolean)

IDisposable implementation

Declaration

protected virtual void Dispose(bool disposing)

Parameters

Type	Name	Description
System.Boolean	disposing	true if invoked from Dispose() method

EndProfiling()

Ends profiling for the session.

Declaration

public string EndProfiling()

Returns

Type	Description
System.String	Returns the profile file name.

Finalize()

Finalizer. to cleanup session in case it runs and the user forgets to Dispose() of the session

Declaration

protected void Finalize()

Run(IReadOnlyCollection<NamedOnnxValue>)

Runs the loaded model for the given inputs, and fetches all the outputs.

Declaration

public IDisposableReadOnlyCollection<DisposableNamedOnnxValue> Run(IReadOnlyCollection<NamedOnnxValue> inputs)

Parameters

Type	Name	Description
IReadOnlyCollection<NamedOnnxValue>	inputs	specify a collection of NamedOnnxValue that indicates the input values.

Returns

Type	Description
IDisposableReadOnlyCollection<DisposableNamedOnnxValue>	Output Tensors in a Collection of NamedOnnxValue. User must dispose the output.

Run(IReadOnlyCollection<NamedOnnxValue>, IReadOnlyCollection<NamedOnnxValue>)

Runs the loaded model for the given inputs and outputs.

Outputs need to be created with correct type and dimension to receive the fetched data.

Declaration

public void Run(IReadOnlyCollection<NamedOnnxValue> inputs, IReadOnlyCollection<NamedOnnxValue> outputs)

Parameters

Type	Name	Description
IReadOnlyCollection<NamedOnnxValue>	inputs	Specify a collection of NamedOnnxValue that indicates the input values.
IReadOnlyCollection<NamedOnnxValue>	outputs	Specify a collection of NamedOnnxValue that indicates the output values.

Run(IReadOnlyCollection<NamedOnnxValue>, IReadOnlyCollection<NamedOnnxValue>, RunOptions)

Runs the loaded model for the given inputs and outputs. Uses the given RunOptions for this run.

Outputs need to be created with correct type and dimension to receive the fetched data.

Declaration

public void Run(IReadOnlyCollection<NamedOnnxValue> inputs, IReadOnlyCollection<NamedOnnxValue> outputs, RunOptions options)

Parameters

Type	Name	Description
IReadOnlyCollection<NamedOnnxValue>	inputs	Specify a collection of NamedOnnxValue that indicates the input values.
IReadOnlyCollection<NamedOnnxValue>	outputs	Specify a collection of NamedOnnxValue that indicates the output values.
RunOptions	options

Run(IReadOnlyCollection<NamedOnnxValue>, IReadOnlyCollection<String>)

Runs the loaded model for the given inputs, and fetches the outputs specified in outputNames.

Declaration

public IDisposableReadOnlyCollection<DisposableNamedOnnxValue> Run(IReadOnlyCollection<NamedOnnxValue> inputs, IReadOnlyCollection<string> outputNames)

Parameters

Type	Name	Description
IReadOnlyCollection<NamedOnnxValue>	inputs	Specify a collection of NamedOnnxValue that indicates the input values.
IReadOnlyCollection<System.String>	outputNames	Specify a collection of string that indicates the output names to fetch.

Returns

Type	Description
IDisposableReadOnlyCollection<DisposableNamedOnnxValue>	Output Tensors in a Collection of NamedOnnxValue. User must dispose the output.

Run(IReadOnlyCollection<NamedOnnxValue>, IReadOnlyCollection<String>, IReadOnlyCollection<FixedBufferOnnxValue>)

Runs the loaded model for the given inputs and outputs.

Outputs need to be created with correct type and dimension to receive the fetched data.

Declaration

public void Run(IReadOnlyCollection<NamedOnnxValue> inputs, IReadOnlyCollection<string> outputNames, IReadOnlyCollection<FixedBufferOnnxValue> outputValues)

Parameters

Type	Name	Description
IReadOnlyCollection<NamedOnnxValue>	inputs	Specify a collection of NamedOnnxValue that indicates the input values.
IReadOnlyCollection<System.String>	outputNames	Specify a collection of string that indicates the output names. Should match outputValues.
IReadOnlyCollection<FixedBufferOnnxValue>	outputValues	Specify a collection of FixedBufferOnnxValue that indicates the output values.

Run(IReadOnlyCollection<NamedOnnxValue>, IReadOnlyCollection<String>, IReadOnlyCollection<FixedBufferOnnxValue>, RunOptions)

Runs the loaded model for the given inputs and outputs. Uses the given RunOptions for this run.

Outputs need to be created with correct type and dimension to receive the fetched data.

Declaration

public void Run(IReadOnlyCollection<NamedOnnxValue> inputs, IReadOnlyCollection<string> outputNames, IReadOnlyCollection<FixedBufferOnnxValue> outputValues, RunOptions options)

Parameters

Type	Name	Description
IReadOnlyCollection<NamedOnnxValue>	inputs	Specify a collection of NamedOnnxValue that indicates the input values.
IReadOnlyCollection<System.String>	outputNames	Specify a collection of string that indicates the output names. Should match outputValues.
IReadOnlyCollection<FixedBufferOnnxValue>	outputValues	Specify a collection of FixedBufferOnnxValue that indicates the output values.
RunOptions	options

Run(IReadOnlyCollection<NamedOnnxValue>, IReadOnlyCollection<String>, RunOptions)

Runs the loaded model for the given inputs, and fetches the specified outputs in outputNames. Uses the given RunOptions for this run.

Declaration

public IDisposableReadOnlyCollection<DisposableNamedOnnxValue> Run(IReadOnlyCollection<NamedOnnxValue> inputs, IReadOnlyCollection<string> outputNames, RunOptions options)

Parameters

Type	Name	Description
IReadOnlyCollection<NamedOnnxValue>	inputs	Specify a collection of NamedOnnxValue that indicates the input values.
IReadOnlyCollection<System.String>	outputNames	Specify a collection of string that indicates the output names to fetch.
RunOptions	options

Returns

Type	Description
IDisposableReadOnlyCollection<DisposableNamedOnnxValue>	Output Tensors in a Collection of NamedOnnxValue. User must dispose the output.

Run(IReadOnlyCollection<String>, IReadOnlyCollection<FixedBufferOnnxValue>)

Runs the loaded model for the given inputs, and fetches all the outputs.

Declaration

public IDisposableReadOnlyCollection<DisposableNamedOnnxValue> Run(IReadOnlyCollection<string> inputNames, IReadOnlyCollection<FixedBufferOnnxValue> inputValues)

Parameters

Type	Name	Description
IReadOnlyCollection<System.String>	inputNames	Specify a collection of string that indicates the input names. Should match inputValues.
IReadOnlyCollection<FixedBufferOnnxValue>	inputValues	Specify a collection of FixedBufferOnnxValue that indicates the input values.

Returns

Type	Description
IDisposableReadOnlyCollection<DisposableNamedOnnxValue>	Output Tensors in a Collection of NamedOnnxValue. User must dispose the output.

Run(IReadOnlyCollection<String>, IReadOnlyCollection<FixedBufferOnnxValue>, IReadOnlyCollection<NamedOnnxValue>)

Runs the loaded model for the given inputs and outputs.

Outputs need to be created with correct type and dimension to receive the fetched data.

Declaration

public void Run(IReadOnlyCollection<string> inputNames, IReadOnlyCollection<FixedBufferOnnxValue> inputValues, IReadOnlyCollection<NamedOnnxValue> outputs)

Parameters

Type	Name	Description
IReadOnlyCollection<System.String>	inputNames	Specify a collection of string that indicates the input names. Should match inputValues.
IReadOnlyCollection<FixedBufferOnnxValue>	inputValues	Specify a collection of FixedBufferOnnxValue that indicates the input values.
IReadOnlyCollection<NamedOnnxValue>	outputs	Specify a collection of NamedOnnxValue that indicates the output values.

Run(IReadOnlyCollection<String>, IReadOnlyCollection<FixedBufferOnnxValue>, IReadOnlyCollection<NamedOnnxValue>, RunOptions)

Runs the loaded model for the given inputs and outputs. Uses the given RunOptions for this run.

Outputs need to be created with correct type and dimension to receive the fetched data.

Declaration

public void Run(IReadOnlyCollection<string> inputNames, IReadOnlyCollection<FixedBufferOnnxValue> inputValues, IReadOnlyCollection<NamedOnnxValue> outputs, RunOptions options)

Parameters

Type	Name	Description
IReadOnlyCollection<System.String>	inputNames	Specify a collection of string that indicates the input names. Should match inputValues.
IReadOnlyCollection<FixedBufferOnnxValue>	inputValues	Specify a collection of FixedBufferOnnxValue that indicates the input values.
IReadOnlyCollection<NamedOnnxValue>	outputs	Specify a collection of NamedOnnxValue that indicates the output values.
RunOptions	options

Run(IReadOnlyCollection<String>, IReadOnlyCollection<FixedBufferOnnxValue>, IReadOnlyCollection<String>)

Runs the loaded model for the given inputs, and fetches the outputs specified in outputNames.

Declaration

public IDisposableReadOnlyCollection<DisposableNamedOnnxValue> Run(IReadOnlyCollection<string> inputNames, IReadOnlyCollection<FixedBufferOnnxValue> inputValues, IReadOnlyCollection<string> outputNames)

Parameters

Type	Name	Description
IReadOnlyCollection<System.String>	inputNames	Specify a collection of string that indicates the input names. Should match inputValues.
IReadOnlyCollection<FixedBufferOnnxValue>	inputValues	Specify a collection of FixedBufferOnnxValue that indicates the input values.
IReadOnlyCollection<System.String>	outputNames	Specify a collection of string that indicates the output names to fetch.

Returns

Type	Description
IDisposableReadOnlyCollection<DisposableNamedOnnxValue>	Output Tensors in a Collection of NamedOnnxValue. User must dispose the output.

Run(IReadOnlyCollection<String>, IReadOnlyCollection<FixedBufferOnnxValue>, IReadOnlyCollection<String>, IReadOnlyCollection<FixedBufferOnnxValue>)

Runs the loaded model for the given inputs and outputs.

Outputs need to be created with correct type and dimension to accept the fetched data.

Declaration

public void Run(IReadOnlyCollection<string> inputNames, IReadOnlyCollection<FixedBufferOnnxValue> inputValues, IReadOnlyCollection<string> outputNames, IReadOnlyCollection<FixedBufferOnnxValue> outputValues)

Parameters

Type	Name	Description
IReadOnlyCollection<System.String>	inputNames	Specify a collection of string that indicates the input names. Should match inputValues.
IReadOnlyCollection<FixedBufferOnnxValue>	inputValues	Specify a collection of FixedBufferOnnxValue that indicates the input values.
IReadOnlyCollection<System.String>	outputNames	Specify a collection of string that indicates the output names. Should match outputValues.
IReadOnlyCollection<FixedBufferOnnxValue>	outputValues	Specify a collection of FixedBufferOnnxValue that indicates the output values.

Run(IReadOnlyCollection<String>, IReadOnlyCollection<FixedBufferOnnxValue>, IReadOnlyCollection<String>, IReadOnlyCollection<FixedBufferOnnxValue>, RunOptions)

Runs the loaded model for the given inputs and outputs. Uses the given RunOptions for this run.

Outputs need to be created with correct type and dimension to accept the fetched data.

Declaration

public void Run(IReadOnlyCollection<string> inputNames, IReadOnlyCollection<FixedBufferOnnxValue> inputValues, IReadOnlyCollection<string> outputNames, IReadOnlyCollection<FixedBufferOnnxValue> outputValues, RunOptions options)

Parameters

Type	Name	Description
IReadOnlyCollection<System.String>	inputNames	Specify a collection of string that indicates the input names. Should match inputValues.
IReadOnlyCollection<FixedBufferOnnxValue>	inputValues	Specify a collection of FixedBufferOnnxValue that indicates the input values.
IReadOnlyCollection<System.String>	outputNames	Specify a collection of string that indicates the output names. Should match outputValues.
IReadOnlyCollection<FixedBufferOnnxValue>	outputValues	Specify a collection of FixedBufferOnnxValue that indicates the output values.
RunOptions	options

Run(IReadOnlyCollection<String>, IReadOnlyCollection<FixedBufferOnnxValue>, IReadOnlyCollection<String>, RunOptions)

Runs the loaded model for the given inputs, and fetches the specified outputs in outputNames. Uses the given RunOptions for this run.

Declaration

public IDisposableReadOnlyCollection<DisposableNamedOnnxValue> Run(IReadOnlyCollection<string> inputNames, IReadOnlyCollection<FixedBufferOnnxValue> inputValues, IReadOnlyCollection<string> outputNames, RunOptions options)

Parameters

Type	Name	Description
IReadOnlyCollection<System.String>	inputNames	Specify a collection of string that indicates the input names. Should match inputValues.
IReadOnlyCollection<FixedBufferOnnxValue>	inputValues	Specify a collection of FixedBufferOnnxValue that indicates the input values.
IReadOnlyCollection<System.String>	outputNames	Specify a collection of string that indicates the output names to fetch.
RunOptions	options

Returns

Type	Description
IDisposableReadOnlyCollection<DisposableNamedOnnxValue>	Output Tensors in a Collection of NamedOnnxValue. User must dispose the output.

RunWithBinding(RunOptions, OrtIoBinding)

This method runs inference on the OrtIoBinding instance The method does not return anything. This is a lightweight version of RunWithBindingAndNames(). When you bind pre-allocated buffers to the output values you may not want to fetch the outputs since you already have access to them so you can spare the expense of fetching them and pairing with names. You can still fetch the outputs by calling OrtIOBinding.GetOutputValues()

Declaration

public void RunWithBinding(RunOptions runOptions, OrtIoBinding ioBinding)

Parameters

Type	Name	Description
RunOptions	runOptions	runOptions
OrtIoBinding	ioBinding	ioBinding instance to use

RunWithBindingAndNames(RunOptions, OrtIoBinding, String[])

This method return a collection of DisposableNamedOnnxValue as in other interfaces Query names from OrtIoBinding object and pair then with the array of OrtValues returned from OrtIoBinding.GetOutputValues()

Declaration

public IDisposableReadOnlyCollection<DisposableNamedOnnxValue> RunWithBindingAndNames(RunOptions runOptions, OrtIoBinding ioBinding, string[] names = null)

Parameters

Type	Name	Description
RunOptions	runOptions	RunOptions
OrtIoBinding	ioBinding	OrtIoBinding instance with bindings
System.String[]	names	optional parameter. If you already know the names of the outputs you can save a native call to retrieve output names. They will be paired with the returned OrtValues and combined into DisposbleNamedOnnxValues. Otherwise, the method will retrieve output names from the OrtIoBinding instance. It is an error if you supply a different number of names than the returned outputs

Returns

Type	Description
IDisposableReadOnlyCollection<DisposableNamedOnnxValue>	A disposable collection of DisposableNamedOnnxValue that encapsulate output OrtValues