Class OrtIoBinding

This class enables binding of inputs and/or outputs to pre-allocated memory. This enables interesting scenarios. For example, if your input already resides in some pre-allocated memory like GPU, you can bind that piece of memory to an input name and shape and onnxruntime will use that as input. Other traditional inputs can also be bound that already exists as Tensors.

Note, that this arrangement is designed to minimize data copies and to that effect your memory allocations must match what is expected by the model, whether you run on CPU or GPU. Data copy will still be made, if your pre-allocated memory location does not match the one expected by the model. However, copies with OrtIoBindings are only done once, at the time of the binding, not at run time. This means, that if your input data required a copy, your further input modifications would not be seen by onnxruntime unless you rebind it, even if it is the same buffer. If you require the scenario where data is copied, OrtIOBinding may not be the best match for your use case.

The fact that data copy is not made during runtime also has performance implications.

Inheritance

System.Object

OrtIoBinding

Namespace: Microsoft.ML.OnnxRuntime

Assembly: cs.temp.dll.dll

Syntax

public class OrtIoBinding : SafeHandle

Properties

IsInvalid

Overrides SafeHandle.IsInvalid

Declaration

public override bool IsInvalid { get; }

Property Value

Type	Description
System.Boolean	returns true if handle is equal to Zero

Methods

BindInput(String, FixedBufferOnnxValue)

Bind the input with the given name as an OrtValue Tensor allocated in pinned managed memory. Instance of FixedBufferOnnxValue owns the memory and should be alive until the end of execution.

Declaration

public void BindInput(string name, FixedBufferOnnxValue fixedValue)

Parameters

Type	Name	Description
System.String	name	name of input
FixedBufferOnnxValue	fixedValue

BindInput(String, OrtExternalAllocation)

Bind externally (not from OrtAllocator) allocated memory as input. The model will read the specified input from that memory possibly avoiding the need to copy between devices. The user code continues to own the chunk of externally allocated memory, and the allocation should be alive until the end of execution.

Declaration

public void BindInput(string name, OrtExternalAllocation allocation)

Parameters

Type	Name	Description
System.String	name	name
OrtExternalAllocation	allocation	non ort allocated memory

BindInput(String, TensorElementType, Int64[], OrtMemoryAllocation)

Bind a piece of pre-allocated native memory as a OrtValue Tensor with a given shape to an input with a given name. The model will read the specified input from that memory possibly avoiding the need to copy between devices. OrtMemoryAllocation continues to own the chunk of native memory, and the allocation should be alive until the end of execution.

Declaration

public void BindInput(string name, TensorElementType elementType, long[] shape, OrtMemoryAllocation allocation)

Parameters

Type	Name	Description
System.String	name	of the input
TensorElementType	elementType	Tensor element type
System.Int64[]	shape
OrtMemoryAllocation	allocation	native memory allocation

BindOutput(String, FixedBufferOnnxValue)

Bind model output to a given instance of FixedBufferOnnxValue which owns the underlying pinned managed memory and should be alive for the time of execution.

Declaration

public void BindOutput(string name, FixedBufferOnnxValue fixedValue)

Parameters

Type	Name	Description
System.String	name	of the output
FixedBufferOnnxValue	fixedValue

BindOutput(String, OrtExternalAllocation)

Bind externally (not from OrtAllocator) allocated memory as output. The model will read the specified input from that memory possibly avoiding the need to copy between devices. The user code continues to own the chunk of externally allocated memory, and the allocation should be alive until the end of execution.

Declaration

public void BindOutput(string name, OrtExternalAllocation allocation)

Parameters

Type	Name	Description
System.String	name	name
OrtExternalAllocation	allocation	non ort allocated memory

BindOutput(String, TensorElementType, Int64[], OrtMemoryAllocation)

Bind model output to an OrtValue as Tensor with a given type and shape. An instance of OrtMemoryAllocaiton owns the memory and should be alive for the time of execution.

Declaration

public void BindOutput(string name, TensorElementType elementType, long[] shape, OrtMemoryAllocation allocation)

Parameters

Type	Name	Description
System.String	name	of the output
TensorElementType	elementType	tensor element type
System.Int64[]	shape	tensor shape
OrtMemoryAllocation	allocation	allocated memory

BindOutputToDevice(String, OrtMemoryInfo)

This function will bind model output with the given name to a device specified by the memInfo.

Declaration

public void BindOutputToDevice(string name, OrtMemoryInfo memInfo)

Parameters

Type	Name	Description
System.String	name	output name
OrtMemoryInfo	memInfo	instance of memory info

ClearBoundInputs()

Clear all bound inputs and start anew

Declaration

public void ClearBoundInputs()

ClearBoundOutputs()

Clear all bound outputs

Declaration

public void ClearBoundOutputs()

GetOutputNames()

Returns an array of output names in the same order they were bound

Declaration

public string[] GetOutputNames()

Returns

Type	Description
System.String[]	array of output names

GetOutputValues()

Declaration

public IDisposableReadOnlyCollection<OrtValue> GetOutputValues()

Returns

Type	Description
IDisposableReadOnlyCollection<OrtValue>

ReleaseHandle()

Overrides SafeHandle.ReleaseHandle() to properly dispose of the native instance of OrtIoBidning

Declaration

protected override bool ReleaseHandle()

Returns

Type	Description
System.Boolean	always returns true

SynchronizeBoundInputs()

Blocks until device completes all preceding requested tasks. Useful for memory synchronization.

Declaration

public void SynchronizeBoundInputs()

SynchronizeBoundOutputs()

Blocks until device completes all preceding requested tasks. Useful for memory synchronization.

Declaration

public void SynchronizeBoundOutputs()