mirror of
https://github.com/saymrwulf/onnxruntime.git
synced 2026-06-09 00:30:53 +00:00
a5dec8eedf
### Description Reduce a number of auxillary objects created to reduce GC pressure. Eliminate GCHandle type of memory pinning in most of the places. Improve string marshalling by allocating unmanaged memory that does not require pinning. Change native methods from `IntPtr` to `byte[]` (marshalling pinning is more efficient). Allocate input/output UTF-8 names in unmanaged heap for the lifetime of InferenceSession. So we do not keep converting them and pinning on every Run. Introduce a new native API that allows to allocate and convert/copy strings directly into a native tensor. The PR delivers around 50% latency improvements and less GC pauses. Inspired by: https://github.com/microsoft/onnxruntime/pull/15520 ### Motivation and Context Client experience GC pressure and performance degradation when dealing with string tensors. Co-Authored-By: @tannergooding |
||
|---|---|---|
| .. | ||
| targets | ||
| Tensors | ||
| Training | ||
| AssemblyInfo.shared.cs | ||
| DisposableNamedOnnxValue.shared.cs | ||
| Exceptions.shared.cs | ||
| FixedBufferOnnxValue.shared.cs | ||
| InferenceSession.shared.cs | ||
| ManagedProjections.shared.cs | ||
| Microsoft.ML.OnnxRuntime.csproj | ||
| Microsoft.ML.OnnxRuntime.sln | ||
| NamedOnnxValue.shared.cs | ||
| NativeApiStatus.shared.cs | ||
| NativeMethods.shared.cs | ||
| NativeOnnxValueHelper.shared.cs | ||
| OrtAllocator.shared.cs | ||
| OrtEnv.shared.cs | ||
| OrtIoBinding.shared.cs | ||
| OrtLoggingLevel.shared.cs | ||
| OrtThreadingOptions.shared.cs | ||
| OrtValue.shared.cs | ||
| OrtValueTensor.shared.cs | ||
| PrepackedWeightsContainer.shared.cs | ||
| ProviderOptions.shared.cs | ||
| RunOptions.shared.cs | ||
| SessionOptions.shared.cs | ||
| SessionOptionsContainer.shared.cs | ||