Train, convert and predict with ONNX Runtime

This example demonstrates an end to end scenario starting with the training of a machine learned model to its use in its converted from.

Train a logistic regression

The first step consists in retrieving the iris datset.

from sklearn.datasets import load_iris
iris = load_iris()
X, y = iris.data, iris.target

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y)

Then we fit a model.

from sklearn.linear_model import LogisticRegression
clr = LogisticRegression()
clr.fit(X_train, y_train)

Out:

LogisticRegression()

We compute the prediction on the test set and we show the confusion matrix.

from sklearn.metrics import confusion_matrix

pred = clr.predict(X_test)
print(confusion_matrix(y_test, pred))

Out:

[[11  0  0]
 [ 0 13  0]
 [ 0  0 14]]

Conversion to ONNX format

We use module sklearn-onnx to convert the model into ONNX format.

from skl2onnx import convert_sklearn
from skl2onnx.common.data_types import FloatTensorType

initial_type = [('float_input', FloatTensorType([None, 4]))]
onx = convert_sklearn(clr, initial_types=initial_type)
with open("logreg_iris.onnx", "wb") as f:
    f.write(onx.SerializeToString())

We load the model with ONNX Runtime and look at its input and output.

import onnxruntime as rt
sess = rt.InferenceSession("logreg_iris.onnx")

print("input name='{}' and shape={}".format(
    sess.get_inputs()[0].name, sess.get_inputs()[0].shape))
print("output name='{}' and shape={}".format(
    sess.get_outputs()[0].name, sess.get_outputs()[0].shape))

Out:

input name='float_input' and shape=[None, 4]
output name='output_label' and shape=[None]

We compute the predictions.

input_name = sess.get_inputs()[0].name
label_name = sess.get_outputs()[0].name

import numpy
pred_onx = sess.run([label_name], {input_name: X_test.astype(numpy.float32)})[0]
print(confusion_matrix(pred, pred_onx))

Out:

[[11  0  0]
 [ 0 13  0]
 [ 0  0 14]]

The prediction are perfectly identical.

Probabilities

Probabilities are needed to compute other relevant metrics such as the ROC Curve. Let’s see how to get them first with scikit-learn.

prob_sklearn = clr.predict_proba(X_test)
print(prob_sklearn[:3])

Out:

[[1.91824904e-05 3.92373474e-02 9.60743470e-01]
 [2.01308118e-02 8.63567570e-01 1.16301618e-01]
 [9.78657084e-01 2.13427283e-02 1.88123801e-07]]

And then with ONNX Runtime. The probabilies appear to be

prob_name = sess.get_outputs()[1].name
prob_rt = sess.run([prob_name], {input_name: X_test.astype(numpy.float32)})[0]

import pprint
pprint.pprint(prob_rt[0:3])

Out:

[{0: 1.9182492906111293e-05, 1: 0.0392373725771904, 2: 0.9607434272766113},
 {0: 0.02013082429766655, 1: 0.8635676503181458, 2: 0.1163015067577362},
 {0: 0.978657066822052, 1: 0.021342728286981583, 2: 1.881237352563403e-07}]

Let’s benchmark.

from timeit import Timer

def speed(inst, number=10, repeat=20):
    timer = Timer(inst, globals=globals())
    raw = numpy.array(timer.repeat(repeat, number=number))
    ave = raw.sum() / len(raw) / number
    mi, ma = raw.min() / number, raw.max() / number
    print("Average %1.3g min=%1.3g max=%1.3g" % (ave, mi, ma))
    return ave

print("Execution time for clr.predict")
speed("clr.predict(X_test)")

print("Execution time for ONNX Runtime")
speed("sess.run([label_name], {input_name: X_test.astype(numpy.float32)})[0]")

Out:

Execution time for clr.predict
Average 7.01e-05 min=6.33e-05 max=8.76e-05
Execution time for ONNX Runtime
Average 0.00123 min=0.000807 max=0.00149

0.0012285225000000111

Let’s benchmark a scenario similar to what a webservice experiences: the model has to do one prediction at a time as opposed to a batch of prediction.

def loop(X_test, fct, n=None):
    nrow = X_test.shape[0]
    if n is None:
        n = nrow
    for i in range(0, n):
        im = i % nrow
        fct(X_test[im: im+1])

print("Execution time for clr.predict")
speed("loop(X_test, clr.predict, 100)")

def sess_predict(x):
    return sess.run([label_name], {input_name: x.astype(numpy.float32)})[0]

print("Execution time for sess_predict")
speed("loop(X_test, sess_predict, 100)")

Out:

Execution time for clr.predict
Average 0.00421 min=0.00408 max=0.00438
Execution time for sess_predict
Average 0.0422 min=0.0417 max=0.043

0.042171193

Let’s do the same for the probabilities.

print("Execution time for predict_proba")
speed("loop(X_test, clr.predict_proba, 100)")

def sess_predict_proba(x):
    return sess.run([prob_name], {input_name: x.astype(numpy.float32)})[0]

print("Execution time for sess_predict_proba")
speed("loop(X_test, sess_predict_proba, 100)")

Out:

Execution time for predict_proba
Average 0.00612 min=0.00598 max=0.00655
Execution time for sess_predict_proba
Average 0.0432 min=0.0426 max=0.0444

0.04324414049999999

This second comparison is better as ONNX Runtime, in this experience, computes the label and the probabilities in every case.

Benchmark with RandomForest

We first train and save a model in ONNX format.

from sklearn.ensemble import RandomForestClassifier
rf = RandomForestClassifier()
rf.fit(X_train, y_train)

initial_type = [('float_input', FloatTensorType([1, 4]))]
onx = convert_sklearn(rf, initial_types=initial_type)
with open("rf_iris.onnx", "wb") as f:
    f.write(onx.SerializeToString())

We compare.

sess = rt.InferenceSession("rf_iris.onnx")

def sess_predict_proba_rf(x):
    return sess.run([prob_name], {input_name: x.astype(numpy.float32)})[0]

print("Execution time for predict_proba")
speed("loop(X_test, rf.predict_proba, 100)")

print("Execution time for sess_predict_proba")
speed("loop(X_test, sess_predict_proba_rf, 100)")

Out:

Execution time for predict_proba
Average 0.594 min=0.588 max=0.607
Execution time for sess_predict_proba
Average 0.0518 min=0.0511 max=0.0531

0.05180173199999999

Let’s see with different number of trees.

measures = []

for n_trees in range(5, 51, 5):
    print(n_trees)
    rf = RandomForestClassifier(n_estimators=n_trees)
    rf.fit(X_train, y_train)
    initial_type = [('float_input', FloatTensorType([1, 4]))]
    onx = convert_sklearn(rf, initial_types=initial_type)
    with open("rf_iris_%d.onnx" % n_trees, "wb") as f:
        f.write(onx.SerializeToString())
    sess = rt.InferenceSession("rf_iris_%d.onnx" % n_trees)
    def sess_predict_proba_loop(x):
        return sess.run([prob_name], {input_name: x.astype(numpy.float32)})[0]
    tsk = speed("loop(X_test, rf.predict_proba, 100)", number=5, repeat=5)
    trt = speed("loop(X_test, sess_predict_proba_loop, 100)", number=5, repeat=5)
    measures.append({'n_trees': n_trees, 'sklearn': tsk, 'rt': trt})

from pandas import DataFrame
df = DataFrame(measures)
ax = df.plot(x="n_trees", y="sklearn", label="scikit-learn", c="blue", logy=True)
df.plot(x="n_trees", y="rt", label="onnxruntime",
                ax=ax, c="green", logy=True)
ax.set_xlabel("Number of trees")
ax.set_ylabel("Prediction time (s)")
ax.set_title("Speed comparison between scikit-learn and ONNX Runtime\nFor a random forest on Iris dataset")
ax.legend()

Out:

5
Average 0.054 min=0.0512 max=0.0648
Average 0.0422 min=0.0414 max=0.0436
10
Average 0.0828 min=0.0797 max=0.0937
Average 0.042 min=0.0417 max=0.0423
15
Average 0.109 min=0.107 max=0.112
Average 0.0438 min=0.0425 max=0.0486
20
Average 0.14 min=0.137 max=0.143
Average 0.0432 min=0.0427 max=0.0437
25
Average 0.167 min=0.164 max=0.177
Average 0.044 min=0.0435 max=0.0448
30
Average 0.196 min=0.191 max=0.206
Average 0.0442 min=0.0434 max=0.0453
35
Average 0.225 min=0.219 max=0.231
Average 0.0447 min=0.0441 max=0.0458
40
Average 0.253 min=0.249 max=0.263
Average 0.0455 min=0.045 max=0.0467
45
Average 0.284 min=0.279 max=0.294
Average 0.0466 min=0.0459 max=0.0483
50
Average 0.31 min=0.306 max=0.321
Average 0.0467 min=0.0458 max=0.048

<matplotlib.legend.Legend object at 0x0000012207A74C50>