saymrwulf/pytorch
1
0
Fork 0
mirror of https://github.com/saymrwulf/pytorch.git synced 2026-05-14 20:57:59 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Threading and CPU Inference note

Browse source 
Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/23417

Test Plan:
cd docs; make html

Imported from OSS

Differential Revision: D16523781

Pulled By: ilia-cher

fbshipit-source-id: d6c09e8a85d39e6185bbdc4b312fea44fcdfff06
This commit is contained in:
Ilia Cherniavskii 2019-07-29 15:32:23 -07:00 committed by Facebook Github Bot
parent f4eb93f7bc
commit 41dfe7204b
2 changed files with 805 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

124
docs/source/notes/cpu_threading_torchscript_inference.rst Normal file
View file

@ -0,0 +1,124 @@
.. _cpu-threading-torchscript-inference:
CPU threading and TorchScript inference
=================================================
PyTorch allows using multiple CPU threads during TorchScript model inference.
The following figure shows different levels of parallelism one would find in a
typical application:
.. image:: cpu_threading_torchscript_inference.svg
:width: 75%
One or more inference threads execute a model's forward pass on the given inputs.
Each inference thread invokes a JIT interpreter that executes the ops
of a model inline, one by one. A model can utilize a ``fork`` TorchScript
primitive to launch an asynchronous task. Forking several operations at once
results in a task that is executed in parallel. The ``fork`` operator returns a
``future`` object which can be used to synchronize on later, for example:
.. code-block:: python
@torch.jit.script
def compute_z(x):
return torch.mm(x, self.w_z)
@torch.jit.script
def forward(x):
# launch compute_z asynchronously:
fut = torch.jit._fork(compute_z, x)
# execute the next operation in parallel to compute_z:
y = torch.mm(x, self.w_y)
# wait for the result of compute_z:
z = torch.jit._wait(fut)
return y + z
PyTorch uses a single thread pool for the inter-op parallelism, this thread pool
is shared by all inference tasks that are forked within the application process.
In addition to the inter-op parallelism, PyTorch can also utilize multiple threads
within the ops (`intra-op parallelism`). This can be useful in many cases,
including element-wise ops on large tensors, convolutions, GEMMs, embedding
lookups and others.
Build options
-------------
PyTorch uses an internal ATen library to implement ops. In addition to that,
PyTorch can also be built with support of external libraries, such as MKL_ and MKL-DNN_,
to speed up computations on CPU.
ATen, MKL and MKL-DNN support intra-op parallelism and depend on the
following parallelization libraries to implement it:
* OpenMP_ - a standard (and a library, usually shipped with a compiler), widely used in external libraries;
* TBB_ - a newer parallelization library optimized for task-based parallelism and concurrent environments.
OpenMP historically has been used by a large number of libraries. It is known
for a relative ease of use and support for loop-based parallelism and other primitives.
At the same time OpenMP is not known for a good interoperability with other threading
libraries used by the application. In particular, OpenMP does not guarantee that a single per-process intra-op thread
pool is going to be used in the application. On the contrary, two different inter-op
threads will likely use different OpenMP thread pools for intra-op work.
This might result in a large number of threads used by the application.
TBB is used to a lesser extent in external libraries, but, at the same time,
is optimized for the concurrent environments. PyTorch's TBB backend guarantees that
there's a separate, single, per-process intra-op thread pool used by all of the
ops running in the application.
Depending of the use case, one might find one or another parallelization
library a better choice in their application.
PyTorch allows selecting of the parallelization backend used by ATen and other
libraries at the build time with the following build options:
+------------+-----------------------+-----------------------------+----------------------------------------+
| Library | Build Option | Values | Notes |
+============+=======================+=============================+========================================+
| ATen | ``ATEN_THREADING`` | ``OMP`` (default), ``TBB`` | |
+------------+-----------------------+-----------------------------+----------------------------------------+
| MKL | ``MKL_THREADING`` | (same) | To enable MKL use ``BLAS=MKL`` |
+------------+-----------------------+-----------------------------+----------------------------------------+
| MKL-DNN | ``MKLDNN_THREADING`` | (same) | To enable MKL-DNN use ``USE_MKLDNN=1`` |
+------------+-----------------------+-----------------------------+----------------------------------------+
It is strongly recommended not to mix OpenMP and TBB within one build.
Any of the ``TBB`` values above require ``USE_TBB=1`` build setting (default: OFF).
A separate setting ``USE_OPENMP=1`` (default: ON) is required for OpenMP parallelism.
Runtime API
-----------
The following API is used to control thread settings:
+------------------------+-----------------------------------------------------------+---------------------------------------------------------+
| Type of parallelism | Settings | Notes |
+========================+===========================================================+=========================================================+
| Inter-op parallelism | ``at::set_num_interop_threads``, | ``set*`` functions can only be called once and only |
| | ``at::get_num_interop_threads`` (C++) | during the startup, before the actual operators running;|
| | | |
| | ``set_num_interop_threads``, | Default number of threads: number of CPU cores. |
| | ``get_num_interop_threads`` (Python, :mod:`torch` module) | |
+------------------------+-----------------------------------------------------------+ |
| Intra-op parallelism | ``at::set_num_threads``, | |
| | ``at::get_num_threads`` (C++) | |
| | ``set_num_threads``, | |
| | ``get_num_threads`` (Python, :mod:`torch` module) | |
| | | |
| | Environment variables: | |
| | ``OMP_NUM_THREADS`` and ``MKL_NUM_THREADS`` | |
+------------------------+-----------------------------------------------------------+---------------------------------------------------------+
For the intra-op parallelism settings, ``at::set_num_threads``, ``torch.set_num_threads`` always take precedence
over environment variables, ``MKL_NUM_THREADS`` variable takes precedence over ``OMP_NUM_THREADS``.
.. note::
``parallel_info`` utility prints information about thread settings and can be used for debugging.
Similar output can be also obtained in Python with ``torch.__config__.parallel_info()`` call.
.. _OpenMP: https://www.openmp.org/
.. _TBB: https://github.com/intel/tbb
.. _MKL: https://software.intel.com/en-us/mkl
.. _MKL-DNN: https://github.com/intel/mkl-dnn

681
docs/source/notes/cpu_threading_torchscript_inference.svg Normal file
View file

@ -0,0 +1,681 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<svg
xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:cc="http://creativecommons.org/ns#"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:svg="http://www.w3.org/2000/svg"
xmlns="http://www.w3.org/2000/svg"
viewBox="0 0 576 336"
height="336"
width="576"
xml:space="preserve"
id="svg2"
version="1.1"><metadata
id="metadata8"><rdf:RDF><cc:Work
rdf:about=""><dc:format>image/svg+xml</dc:format><dc:type
rdf:resource="http://purl.org/dc/dcmitype/StillImage" /></cc:Work></rdf:RDF></metadata><defs
id="defs6"><clipPath
id="clipPath18"
clipPathUnits="userSpaceOnUse"><path
style="clip-rule:evenodd"
id="path16"
d="M 0,6.1035e-5 H 432 V 252.00006 H 0 Z" /></clipPath><clipPath
id="clipPath34"
clipPathUnits="userSpaceOnUse"><path
style="clip-rule:evenodd"
id="path32"
d="M 6.437e-6,0 H 432.00001 V 252 H 6.437e-6 Z" /></clipPath><clipPath
id="clipPath52"
clipPathUnits="userSpaceOnUse"><path
style="clip-rule:evenodd"
id="path50"
d="M 6.437e-6,0 H 432.00001 V 252 H 6.437e-6 Z" /></clipPath><clipPath
id="clipPath64"
clipPathUnits="userSpaceOnUse"><path
style="clip-rule:evenodd"
id="path62"
d="M 6.437e-6,0 H 432.00001 V 252 H 6.437e-6 Z" /></clipPath><clipPath
id="clipPath118"
clipPathUnits="userSpaceOnUse"><path
style="clip-rule:evenodd"
id="path116"
d="M 6.437e-6,0 H 432.00001 V 252 H 6.437e-6 Z" /></clipPath><clipPath
id="clipPath130"
clipPathUnits="userSpaceOnUse"><path
style="clip-rule:evenodd"
id="path128"
d="M 6.437e-6,0 H 432.00001 V 252 H 6.437e-6 Z" /></clipPath><clipPath
id="clipPath142"
clipPathUnits="userSpaceOnUse"><path
style="clip-rule:evenodd"
id="path140"
d="M 6.437e-6,0 H 432.00001 V 252 H 6.437e-6 Z" /></clipPath><clipPath
id="clipPath154"
clipPathUnits="userSpaceOnUse"><path
style="clip-rule:evenodd"
id="path152"
d="M 6.437e-6,0 H 432.00001 V 252 H 6.437e-6 Z" /></clipPath><clipPath
id="clipPath166"
clipPathUnits="userSpaceOnUse"><path
style="clip-rule:evenodd"
id="path164"
d="M 6.437e-6,0 H 432.00001 V 252 H 6.437e-6 Z" /></clipPath><clipPath
id="clipPath184"
clipPathUnits="userSpaceOnUse"><path
style="clip-rule:evenodd"
id="path182"
d="M 6.437e-6,0 H 432.00001 V 252 H 6.437e-6 Z" /></clipPath><clipPath
id="clipPath202"
clipPathUnits="userSpaceOnUse"><path
style="clip-rule:evenodd"
id="path200"
d="M 6.437e-6,0 H 432.00001 V 252 H 6.437e-6 Z" /></clipPath><clipPath
id="clipPath214"
clipPathUnits="userSpaceOnUse"><path
style="clip-rule:evenodd"
id="path212"
d="M 6.437e-6,0 H 432.00001 V 252 H 6.437e-6 Z" /></clipPath><clipPath
id="clipPath238"
clipPathUnits="userSpaceOnUse"><path
style="clip-rule:evenodd"
id="path236"
d="M 6.437e-6,0 H 432.00001 V 252 H 6.437e-6 Z" /></clipPath><clipPath
id="clipPath252"
clipPathUnits="userSpaceOnUse"><path
style="clip-rule:evenodd"
id="path250"
d="M 6.437e-6,0 H 432.00001 V 252 H 6.437e-6 Z" /></clipPath><clipPath
id="clipPath278"
clipPathUnits="userSpaceOnUse"><path
style="clip-rule:evenodd"
id="path276"
d="M 6.437e-6,0 H 432.00001 V 252 H 6.437e-6 Z" /></clipPath><clipPath
id="clipPath290"
clipPathUnits="userSpaceOnUse"><path
style="clip-rule:evenodd"
id="path288"
d="M 6.437e-6,0 H 432.00001 V 252 H 6.437e-6 Z" /></clipPath><clipPath
id="clipPath302"
clipPathUnits="userSpaceOnUse"><path
style="clip-rule:evenodd"
id="path300"
d="M 6.437e-6,0 H 432.00001 V 252 H 6.437e-6 Z" /></clipPath><clipPath
id="clipPath330"
clipPathUnits="userSpaceOnUse"><path
style="clip-rule:evenodd"
id="path328"
d="M 6.437e-6,0 H 432.00001 V 252 H 6.437e-6 Z" /></clipPath><clipPath
id="clipPath342"
clipPathUnits="userSpaceOnUse"><path
style="clip-rule:evenodd"
id="path340"
d="M 6.437e-6,0 H 432.00001 V 252 H 6.437e-6 Z" /></clipPath><clipPath
id="clipPath354"
clipPathUnits="userSpaceOnUse"><path
style="clip-rule:evenodd"
id="path352"
d="M 6.437e-6,0 H 432.00001 V 252 H 6.437e-6 Z" /></clipPath><clipPath
id="clipPath366"
clipPathUnits="userSpaceOnUse"><path
style="clip-rule:evenodd"
id="path364"
d="M 6.437e-6,0 H 432.00001 V 252 H 6.437e-6 Z" /></clipPath><clipPath
id="clipPath378"
clipPathUnits="userSpaceOnUse"><path
style="clip-rule:evenodd"
id="path376"
d="M 6.437e-6,0 H 432.00001 V 252 H 6.437e-6 Z" /></clipPath><clipPath
id="clipPath390"
clipPathUnits="userSpaceOnUse"><path
style="clip-rule:evenodd"
id="path388"
d="M 6.437e-6,0 H 432.00001 V 252 H 6.437e-6 Z" /></clipPath><clipPath
id="clipPath402"
clipPathUnits="userSpaceOnUse"><path
style="clip-rule:evenodd"
id="path400"
d="M 6.437e-6,0 H 432.00001 V 252 H 6.437e-6 Z" /></clipPath><clipPath
id="clipPath414"
clipPathUnits="userSpaceOnUse"><path
style="clip-rule:evenodd"
id="path412"
d="M 6.437e-6,0 H 432.00001 V 252 H 6.437e-6 Z" /></clipPath><clipPath
id="clipPath426"
clipPathUnits="userSpaceOnUse"><path
style="clip-rule:evenodd"
id="path424"
d="M 6.437e-6,0 H 432.00001 V 252 H 6.437e-6 Z" /></clipPath><clipPath
id="clipPath438"
clipPathUnits="userSpaceOnUse"><path
style="clip-rule:evenodd"
id="path436"
d="M 6.437e-6,0 H 432.00001 V 252 H 6.437e-6 Z" /></clipPath><clipPath
id="clipPath450"
clipPathUnits="userSpaceOnUse"><path
style="clip-rule:evenodd"
id="path448"
d="M 6.437e-6,0 H 432.00001 V 252 H 6.437e-6 Z" /></clipPath><clipPath
id="clipPath462"
clipPathUnits="userSpaceOnUse"><path
style="clip-rule:evenodd"
id="path460"
d="M 6.437e-6,0 H 432.00001 V 252 H 6.437e-6 Z" /></clipPath><clipPath
id="clipPath474"
clipPathUnits="userSpaceOnUse"><path
style="clip-rule:evenodd"
id="path472"
d="M 6.437e-6,0 H 432.00001 V 252 H 6.437e-6 Z" /></clipPath><clipPath
id="clipPath486"
clipPathUnits="userSpaceOnUse"><path
style="clip-rule:evenodd"
id="path484"
d="M 6.437e-6,0 H 432.00001 V 252 H 6.437e-6 Z" /></clipPath><clipPath
id="clipPath498"
clipPathUnits="userSpaceOnUse"><path
style="clip-rule:evenodd"
id="path496"
d="M 6.437e-6,0 H 432.00001 V 252 H 6.437e-6 Z" /></clipPath><clipPath
id="clipPath510"
clipPathUnits="userSpaceOnUse"><path
style="clip-rule:evenodd"
id="path508"
d="M 6.437e-6,0 H 432.00001 V 252 H 6.437e-6 Z" /></clipPath><clipPath
id="clipPath524"
clipPathUnits="userSpaceOnUse"><path
style="clip-rule:evenodd"
id="path522"
d="M 6.437e-6,0 H 432.00001 V 252 H 6.437e-6 Z" /></clipPath></defs><g
transform="matrix(1.3333333,0,0,-1.3333333,0,336)"
id="g10"><g
id="g12"><g
clip-path="url(#clipPath18)"
id="g14"><path
id="path20"
style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none"
d="M 0,1.5259e-5 H 432 V 252.00002 H 0 Z" /></g></g><path
id="path22"
style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
d="m 95.58,212.04 h 96.5 v -1.56 h -96.5 z m 95.5,2.22 6,-3 -6,-3 z" /><path
id="path24"
style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
d="m 95.58,198.96 h 96.5 v -1.56 h -96.5 z m 95.5,2.22 6,-3 -6,-3 z" /><path
id="path26"
style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
d="m 96.3,171.48 h 96.5 v -1.56 H 96.3 Z m 95.5,2.22 6,-3 -6,-3 z" /><g
id="g28"><g
clip-path="url(#clipPath34)"
id="g30"><text
id="text38"
style="font-variant:normal;font-weight:normal;font-size:18px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
transform="matrix(1,0,0,-1,139.54,183.77)"><tspan
id="tspan36"
y="0"
x="0">…</tspan></text>
</g></g><path
id="path40"
style="fill:#4472c4;fill-opacity:0.12941003;fill-rule:evenodd;stroke:none"
d="m 9.72,196.11 h 54.54 v 6.45 l 12.9,-12.9 -12.9,-12.9 v 6.45 H 9.72 Z" /><path
id="path42"
style="fill:none;stroke:#000000;stroke-width:0.23999999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:10;stroke-dasharray:none;stroke-opacity:1"
d="m 9.72,196.11 h 54.54 v 6.45 l 12.9,-12.9 -12.9,-12.9 v 6.45 H 9.72 Z" /><path
id="path44"
style="fill:none;stroke:#000000;stroke-width:0.95999998;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:10;stroke-dasharray:none;stroke-opacity:1"
d="m 93.24,158.88 c -3.479,0 -6.3,0.47 -6.3,1.05 v 28.68 c 0,0.58 -2.821,1.05 -6.3,1.05 3.479,0 6.3,0.47 6.3,1.05 v 28.68 c 0,0.58 2.821,1.05 6.3,1.05" /><g
id="g46"><g
clip-path="url(#clipPath52)"
id="g48"><text
id="text56"
style="font-variant:normal;font-weight:normal;font-size:9.98400021px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
transform="matrix(1,0,0,-1,26.496,186.98)"><tspan
id="tspan54"
y="0"
x="0 2.5159681 7.7875199 13.059072 18.330624 21.675264">Inputs</tspan></text>
</g></g><g
id="g58"><g
clip-path="url(#clipPath64)"
id="g60"><text
id="text68"
style="font-variant:normal;font-weight:normal;font-size:9.96000004px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
transform="matrix(1,0,0,-1,87.888,229.97)"><tspan
id="tspan66"
y="0"
x="0 5.76684 11.02572 16.294559 18.58536 20.87616 25.059361 29.8302 33.206638 35.49744 40.746361 46.035118 48.07692 52.87764 58.146481 61.62252 66.542763 71.313599 76.61232 78.863281 84.04248 89.311317 94.580162">Application Thread Pool</tspan></text>
</g></g><path
id="path70"
style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none"
d="m 117.24,167.64 h 6.48 V 174 h -6.48 z" /><path
id="path72"
style="fill:none;stroke:#000000;stroke-width:0.95999998;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:10;stroke-dasharray:none;stroke-opacity:1"
d="m 117.24,167.64 h 6.48 V 174 h -6.48 z" /><path
id="path74"
style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none"
d="m 129.12,167.64 h 6.36 V 174 h -6.36 z" /><path
id="path76"
style="fill:none;stroke:#000000;stroke-width:0.95999998;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:10;stroke-dasharray:none;stroke-opacity:1"
d="m 129.12,167.64 h 6.36 V 174 h -6.36 z" /><path
id="path78"
style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none"
d="m 140.88,167.64 h 6.36 V 174 h -6.36 z" /><path
id="path80"
style="fill:none;stroke:#000000;stroke-width:0.95999998;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:10;stroke-dasharray:none;stroke-opacity:1"
d="m 140.88,167.64 h 6.36 V 174 h -6.36 z" /><g
id="g82"><path
id="path84"
style="fill:none;stroke:#000000;stroke-width:0.47999999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:10;stroke-dasharray:1.92, 1.44;stroke-dashoffset:0;stroke-opacity:1"
d="m 112.92,162.12 h 38.76 v 17.28 h -38.76 z" /></g><g
id="g86"><path
id="path88"
style="fill:none;stroke:#000000;stroke-width:0.47999999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:10;stroke-dasharray:1.92, 1.44;stroke-dashoffset:0;stroke-opacity:1"
d="m 112.92,162.24 22.56,-42.97" /></g><g
id="g90"><path
id="path92"
style="fill:none;stroke:#000000;stroke-width:0.47999999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:10;stroke-dasharray:1.92, 1.44;stroke-dashoffset:0;stroke-opacity:1"
d="M 151.68,160.92 295.3,120.04" /></g><g
id="g94"><path
id="path96"
style="fill:none;stroke:#000000;stroke-width:0.47999999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:10;stroke-dasharray:1.92, 1.44;stroke-dashoffset:0;stroke-opacity:1"
d="m 135.48,11.28 h 159.84 v 108 H 135.48 Z" /></g><path
id="path98"
style="fill:none;stroke:#000000;stroke-width:1.55999994;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:10;stroke-dasharray:none;stroke-opacity:1"
d="M 138.06,100.86 H 292.64" /><path
id="path100"
style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none"
d="m 161.88,98.52 h 6.36 v 6.36 h -6.36 z" /><path
id="path102"
style="fill:none;stroke:#000000;stroke-width:0.95999998;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:10;stroke-dasharray:none;stroke-opacity:1"
d="m 161.88,98.52 h 6.36 v 6.36 h -6.36 z" /><path
id="path104"
style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none"
d="m 173.76,98.52 h 6.36 v 6.36 h -6.36 z" /><path
id="path106"
style="fill:none;stroke:#000000;stroke-width:0.95999998;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:10;stroke-dasharray:none;stroke-opacity:1"
d="m 173.76,98.52 h 6.36 v 6.36 h -6.36 z" /><path
id="path108"
style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none"
d="m 250.92,98.52 h 6.36 v 6.36 h -6.36 z" /><path
id="path110"
style="fill:none;stroke:#000000;stroke-width:0.95999998;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:10;stroke-dasharray:none;stroke-opacity:1"
d="m 250.92,98.52 h 6.36 v 6.36 h -6.36 z" /><g
id="g112"><g
clip-path="url(#clipPath118)"
id="g114"><text
id="text122"
style="font-variant:normal;font-weight:normal;font-size:9.96000004px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
transform="matrix(1,0,0,-1,206.16,97.704)"><tspan
id="tspan120"
y="0"
x="0">…</tspan></text>
</g></g><g
id="g124"><g
clip-path="url(#clipPath130)"
id="g126"><text
id="text134"
style="font-variant:normal;font-weight:normal;font-size:8.03999996px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
transform="matrix(1,0,0,-1,160.66,108.98)"><tspan
id="tspan132"
y="0"
x="0 5.2799802">Op</tspan></text>
</g></g><g
id="g136"><g
clip-path="url(#clipPath142)"
id="g138"><text
id="text146"
style="font-variant:normal;font-weight:normal;font-size:8.03999996px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
transform="matrix(1,0,0,-1,172.34,108.82)"><tspan
id="tspan144"
y="0"
x="0 5.2799802">Op</tspan></text>
</g></g><g
id="g148"><g
clip-path="url(#clipPath154)"
id="g150"><text
id="text158"
style="font-variant:normal;font-weight:normal;font-size:8.03999996px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
transform="matrix(1,0,0,-1,249.19,109.1)"><tspan
id="tspan156"
y="0"
x="0 5.2799802">Op</tspan></text>
</g></g><g
id="g160"><g
clip-path="url(#clipPath166)"
id="g162"><text
id="text170"
style="font-variant:normal;font-weight:normal;font-size:9.96000004px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
transform="matrix(1,0,0,-1,61.512,97.632)"><tspan
id="tspan168"
y="0"
x="0 2.5099199 7.7887201 10.78668 15.70692 19.182961 24.103201 29.37204 33.585121 38.495399 41.005322 44.341919 49.63068 53.10672 58.026958 62.797798">Inference thread</tspan></text>
</g></g><path
id="path172"
style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
d="M 152.58,78.6 H 268.49 V 77.04 H 152.58 Z m 114.91,2.22 6,-3 -6,-3 z" /><path
id="path174"
style="fill:none;stroke:#000000;stroke-width:0.95999998;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:10;stroke-dasharray:none;stroke-opacity:1"
d="m 151.2,21.48 c -3.48,0 -6.3,0.47 -6.3,1.05 v 28.62 c 0,0.58 -2.82,1.05 -6.3,1.05 3.48,0 6.3,0.47 6.3,1.05 v 28.62 c 0,0.58 2.82,1.05 6.3,1.05" /><path
id="path176"
style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
d="M 176.38,98.52 V 80.958 h 1 V 98.52 Z m -1.5,-16.562 2,-4 2,4 z" /><g
id="g178"><g
clip-path="url(#clipPath184)"
id="g180"><text
id="text188"
style="font-variant:normal;font-weight:normal;font-size:6.96000004px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
transform="matrix(1,0,0,-1,162.79,86.064)"><tspan
id="tspan186"
y="0"
x="0 3.2363999 6.9460802 9.3472795">Fork</tspan></text>
</g></g><path
id="path190"
style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
d="M 242.42,77.356 V 95.52 h -1 V 77.356 Z m 1.5,17.164 -2,4 -2,-4 z" /><path
id="path192"
style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none"
d="m 238.68,98.52 h 6.48 v 6.36 h -6.48 z" /><path
id="path194"
style="fill:none;stroke:#000000;stroke-width:0.95999998;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:10;stroke-dasharray:none;stroke-opacity:1"
d="m 238.68,98.52 h 6.48 v 6.36 h -6.48 z" /><g
id="g196"><g
clip-path="url(#clipPath202)"
id="g198"><text
id="text206"
style="font-variant:normal;font-weight:normal;font-size:8.06400013px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
transform="matrix(1,0,0,-1,236.54,109.2)"><tspan
id="tspan204"
y="0"
x="0 5.2799678">Op</tspan></text>
</g></g><g
id="g208"><g
clip-path="url(#clipPath214)"
id="g210"><text
id="text218"
style="font-variant:normal;font-weight:normal;font-size:6.96000004px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
transform="matrix(1,0,0,-1,229.08,86.016)"><tspan
id="tspan216"
y="0"
x="0 2.2759199 5.9856 7.5446401">Join</tspan></text>
</g></g><path
id="path220"
style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none"
d="m 185.88,74.76 h 6.36 v 6.36 h -6.36 z" /><path
id="path222"
style="fill:none;stroke:#000000;stroke-width:0.95999998;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:10;stroke-dasharray:none;stroke-opacity:1"
d="m 185.88,74.76 h 6.36 v 6.36 h -6.36 z" /><path
id="path224"
style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none"
d="m 223.56,74.76 h 6.36 v 6.36 h -6.36 z" /><path
id="path226"
style="fill:none;stroke:#000000;stroke-width:0.95999998;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:10;stroke-dasharray:none;stroke-opacity:1"
d="m 223.56,74.76 h 6.36 v 6.36 h -6.36 z" /><path
id="path228"
style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none"
d="m 198.36,74.76 h 6.36 v 6.36 h -6.36 z" /><path
id="path230"
style="fill:none;stroke:#000000;stroke-width:0.95999998;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:10;stroke-dasharray:none;stroke-opacity:1"
d="m 198.36,74.76 h 6.36 v 6.36 h -6.36 z" /><g
id="g232"><g
clip-path="url(#clipPath238)"
id="g234"><text
id="text242"
style="font-variant:normal;font-weight:normal;font-size:9.96000004px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
transform="matrix(1,0,0,-1,210.7,74.544)"><tspan
id="tspan240"
y="0"
x="0">…</tspan></text>
</g></g><path
id="path244"
style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
d="M 152.58,59.64 H 268.49 V 58.08 H 152.58 Z m 114.91,2.22 6,-3 -6,-3 z" /><g
id="g246"><g
clip-path="url(#clipPath252)"
id="g248"><text
id="text256"
style="font-variant:normal;font-weight:normal;font-size:9.96000004px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
transform="matrix(1,0,0,-1,210.7,42.384)"><tspan
id="tspan254"
y="0"
x="0">…</tspan></text>
</g></g><path
id="path258"
style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
d="M 152.58,28.32 H 268.49 V 26.76 H 152.58 Z m 114.91,2.22 6,-3 -6,-3 z" /><path
id="path260"
style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
d="M 188.62,74.28 V 61.891 h 1 V 74.28 Z m -1.5,-11.389 2,-4 2,4 z" /><path
id="path262"
style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
d="M 227.3,58.948 V 71.76 h -1 V 58.948 Z m 1.5,11.812 -2,4 -2,-4 z" /><path
id="path264"
style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none"
d="m 193.08,56.04 h 6.36 v 6.36 h -6.36 z" /><path
id="path266"
style="fill:none;stroke:#000000;stroke-width:0.95999998;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:10;stroke-dasharray:none;stroke-opacity:1"
d="m 193.08,56.04 h 6.36 v 6.36 h -6.36 z" /><path
id="path268"
style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:none"
d="m 216.36,55.68 h 6.36 v 6.36 h -6.36 z" /><path
id="path270"
style="fill:none;stroke:#000000;stroke-width:0.95999998;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:10;stroke-dasharray:none;stroke-opacity:1"
d="m 216.36,55.68 h 6.36 v 6.36 h -6.36 z" /><g
id="g272"><g
clip-path="url(#clipPath278)"
id="g274"><text
id="text282"
style="font-variant:normal;font-weight:normal;font-size:9.98400021px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
transform="matrix(1,0,0,-1,50.928,50.4)"><tspan
id="tspan280"
y="0"
x="0 2.5159681 7.7875199 11.13216 16.064257">Inter</tspan></text>
</g></g><g
id="g284"><g
clip-path="url(#clipPath290)"
id="g286"><text
id="text294"
style="font-variant:normal;font-weight:normal;font-size:9.98400021px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
transform="matrix(1,0,0,-1,70.488,50.4)"><tspan
id="tspan292"
y="0"
x="0">-</tspan></text>
</g></g><g
id="g296"><g
clip-path="url(#clipPath302)"
id="g298"><text
id="text306"
style="font-variant:normal;font-weight:normal;font-size:9.98400021px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
transform="matrix(1,0,0,-1,73.512,50.4)"><tspan
id="tspan304"
y="0"
x="0 5.2615681 10.553088 12.809472 18.100992 22.883327 26.367744 31.170048 33.46637 35.732735 40.654846 42.951168 45.217537 49.061375">op parallelism</tspan></text>
</g></g><g
id="g308"><path
id="path310"
style="fill:none;stroke:#000000;stroke-width:0.47999999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:10;stroke-dasharray:1.92, 1.44;stroke-dashoffset:0;stroke-opacity:1"
d="m 248.4,96.36 h 11.28 v 10.8 H 248.4 Z" /></g><g
id="g312"><path
id="path314"
style="fill:none;stroke:#000000;stroke-width:0.47999999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:10;stroke-dasharray:1.92, 1.44;stroke-dashoffset:0;stroke-opacity:1"
d="M 255.93,156.72 248.4,107.47" /></g><g
id="g316"><path
id="path318"
style="fill:none;stroke:#000000;stroke-width:0.47999999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:10;stroke-dasharray:1.92, 1.44;stroke-dashoffset:0;stroke-opacity:1"
d="M 349.46,158.88 259.8,107.22" /></g><g
id="g320"><path
id="path322"
style="fill:none;stroke:#000000;stroke-width:0.47999999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:10;stroke-dasharray:1.92, 1.44;stroke-dashoffset:0;stroke-opacity:1"
d="m 255.96,158.88 h 93.84 v 61.56 h -93.84 z" /></g><g
id="g324"><g
clip-path="url(#clipPath330)"
id="g326"><text
id="text334"
style="font-variant:normal;font-weight:normal;font-size:9.96000004px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
transform="matrix(1,0,0,-1,255.65,232.3)"><tspan
id="tspan332"
y="0"
x="0 2.5099199 7.7887201 11.12532 14.60136">Intra</tspan></text>
</g></g><g
id="g336"><g
clip-path="url(#clipPath342)"
id="g338"><text
id="text346"
style="font-variant:normal;font-weight:normal;font-size:9.96000004px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
transform="matrix(1,0,0,-1,275.09,232.3)"><tspan
id="tspan344"
y="0"
x="0">-</tspan></text>
</g></g><g
id="g348"><g
clip-path="url(#clipPath354)"
id="g350"><text
id="text358"
style="font-variant:normal;font-weight:normal;font-size:9.96000004px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
transform="matrix(1,0,0,-1,278.09,232.3)"><tspan
id="tspan356"
y="0"
x="0 5.2987199 10.59744 12.75876 18.05748 22.86816 26.3442 31.184759 33.475559 35.766361 40.6866 42.977402 45.2682 49.112759">op parallelism</tspan></text>
</g></g><g
id="g360"><g
clip-path="url(#clipPath366)"
id="g362"><text
id="text370"
style="font-variant:normal;font-weight:normal;font-size:9px;font-family:ArialMT;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
transform="matrix(1,0,0,-1,261.82,207.34)"><tspan
id="tspan368"
y="0"
x="0">•</tspan></text>
</g></g><g
id="g372"><g
clip-path="url(#clipPath378)"
id="g374"><text
id="text382"
style="font-variant:normal;font-weight:normal;font-size:9px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
transform="matrix(1,0,0,-1,275.38,207.34)"><tspan
id="tspan380"
y="0"
x="0 5.1570001 9.585 14.022 18.702 22.176001 26.856001 31.167 34.307999 38.618999 40.653 42.695999 47.132999">ATen/Parallel</tspan></text>
</g></g><g
id="g384"><g
clip-path="url(#clipPath390)"
id="g386"><text
id="text394"
style="font-variant:normal;font-weight:normal;font-size:9.02400017px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
transform="matrix(1,0,0,-1,265.78,196.54)"><tspan
id="tspan392"
y="0"
x="0 2.734272 7.2011518 9.4751997 13.680384 15.954432 17.993856 22.316353 25.339392 27.721727 30.14016 34.805569 39.128063 42.241344 46.563839 48.603264 50.642689 55.082497 57.121922 61.561729 64.314049 69.114815 72.237122">(e.g. at::parallel_for)</tspan></text>
</g></g><g
id="g396"><g
clip-path="url(#clipPath402)"
id="g398"><text
id="text406"
style="font-variant:normal;font-weight:normal;font-size:9px;font-family:ArialMT;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
transform="matrix(1,0,0,-1,261.82,185.71)"><tspan
id="tspan404"
y="0"
x="0">•</tspan></text>
</g></g><g
id="g408"><g
clip-path="url(#clipPath414)"
id="g410"><text
id="text418"
style="font-variant:normal;font-weight:normal;font-size:9px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
transform="matrix(1,0,0,-1,275.38,185.71)"><tspan
id="tspan416"
y="0"
x="0 7.6950002 12.375">MKL</tspan></text>
</g></g><g
id="g420"><g
clip-path="url(#clipPath426)"
id="g422"><text
id="text430"
style="font-variant:normal;font-weight:normal;font-size:9px;font-family:ArialMT;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
transform="matrix(1,0,0,-1,261.82,174.91)"><tspan
id="tspan428"
y="0"
x="0">•</tspan></text>
</g></g><g
id="g432"><g
clip-path="url(#clipPath438)"
id="g434"><text
id="text442"
style="font-variant:normal;font-weight:normal;font-size:9px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
transform="matrix(1,0,0,-1,275.38,174.91)"><tspan
id="tspan440"
y="0"
x="0 7.6950002 12.375">MKL</tspan></text>
</g></g><g
id="g444"><g
clip-path="url(#clipPath450)"
id="g446"><text
id="text454"
style="font-variant:normal;font-weight:normal;font-size:9px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
transform="matrix(1,0,0,-1,291.58,174.91)"><tspan
id="tspan452"
y="0"
x="0">-</tspan></text>
</g></g><g
id="g456"><g
clip-path="url(#clipPath462)"
id="g458"><text
id="text466"
style="font-variant:normal;font-weight:normal;font-size:9px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
transform="matrix(1,0,0,-1,294.34,174.91)"><tspan
id="tspan464"
y="0"
x="0 5.5349998 11.277">DNN</tspan></text>
</g></g><g
id="g468"><g
clip-path="url(#clipPath474)"
id="g470"><text
id="text478"
style="font-variant:normal;font-weight:normal;font-size:9px;font-family:ArialMT;-inkscape-font-specification:ArialMT;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
transform="matrix(1,0,0,-1,261.82,164.11)"><tspan
id="tspan476"
y="0"
x="0">•</tspan></text>
</g></g><g
id="g480"><g
clip-path="url(#clipPath486)"
id="g482"><text
id="text490"
style="font-variant:normal;font-weight:bold;font-size:9px;font-family:Calibri;-inkscape-font-specification:Calibri-Bold;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
transform="matrix(1,0,0,-1,275.38,164.11)"><tspan
id="tspan488"
y="0"
x="0 2.4000001 4.8000002">...</tspan></text>
</g></g><g
id="g492"><g
clip-path="url(#clipPath498)"
id="g494"><text
id="text502"
style="font-variant:normal;font-weight:normal;font-size:9px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
transform="matrix(1,0,0,-1,382.58,201.26)"><tspan
id="tspan500"
y="0"
x="0 5.994 10.674 15.111 19.791 27.486">OpenMP</tspan></text>
</g></g><g
id="g504"><g
clip-path="url(#clipPath510)"
id="g506"><text
id="text514"
style="font-variant:normal;font-weight:normal;font-size:9px;font-family:Calibri;-inkscape-font-specification:Calibri;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
transform="matrix(1,0,0,-1,382.58,187.32)"><tspan
id="tspan512"
y="0"
x="0 4.428 9.3240004">TBB</tspan></text>
</g></g><path
id="path516"
style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
d="m 349.45,190.1 20.73,0.28 -0.01,1 -20.74,-0.28 z m 19.75,-1.23 3.97,2.05 -4.02,1.95 z" /><g
id="g518"><g
clip-path="url(#clipPath524)"
id="g520"><text
id="text528"
style="font-variant:normal;font-weight:bold;font-size:9px;font-family:Calibri;-inkscape-font-specification:Calibri-Bold;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none"
transform="matrix(1,0,0,-1,383.81,173.83)"><tspan
id="tspan526"
y="0"
x="0">…</tspan></text>
</g></g><g
id="g530"><path
id="path532"
style="fill:none;stroke:#000000;stroke-width:0.47999999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:10;stroke-dasharray:1.92, 1.44;stroke-dashoffset:0;stroke-opacity:1"
d="m 375,167.64 h 44.52 V 214.2 H 375 Z" /></g></g></svg>
Side by side

After

Width:  |  Height:  |  Size: 35 KiB