mirror of
https://github.com/saymrwulf/onnxruntime.git
synced 2026-07-19 19:00:47 +00:00
Migrate content from GitHub source to GitHub pages (#5053)
* Test re-using page layout from current ONNX Runtime website for docs * Add content for documentation on website * Fixed most broken links * Copy just-the-docs theme sources into repo * Remove local theme files as this did not work with GitHub * Remove nojekyll file * Move image assets into single location * Add Contents to markdown files and ensure only one h1 * Update after review * Fix img links * Add trailing slash to main nav links * Fix broken links on main docs page * Re-fix broken links on main docs page * Fix broken links #3 * Fix broken links #4 * Fix broken links #5 * Fix broken links #6 * Fix paths to global assets * Add updates since fork * Update custom op docs * Fix link
This commit is contained in:
Nat Kershaw (MSFT)
GitHub
parent
7841775e59
commit
15291ab4c9
63 changed files with 8496 additions and 13 deletions
4
.gitignore
vendored
Normal file
4
.gitignore
vendored
Normal file
|
|
@ -0,0 +1,4 @@
|
|||
_site
|
||||
.sass-cache
|
||||
.jekyll-metadata
|
||||
*.out
|
||||
16
404.html
16
404.html
|
|
@ -13,13 +13,13 @@
|
|||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
|
||||
<title>ONNX Runtime | Oops</title>
|
||||
<link rel="icon" href="./images/ONNXRuntime-Favicon.png" type="image/gif" sizes="16x16">
|
||||
<link rel="icon" href="/images/ONNXRuntime-Favicon.png" type="image/gif" sizes="16x16">
|
||||
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Open+Sans:300,400,700">
|
||||
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Lato:100,100i,300,300i,400,400i,700,700i,900,900i&display=swap">
|
||||
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css">
|
||||
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css">
|
||||
<link rel="stylesheet" href="css/custom.css">
|
||||
<link rel="stylesheet" href="css/responsive.css">
|
||||
<link rel="stylesheet" href="/css/custom.css">
|
||||
<link rel="stylesheet" href="/css/responsive.css">
|
||||
</head>
|
||||
|
||||
<body>
|
||||
|
|
@ -28,8 +28,8 @@
|
|||
<div class="top-banner-bg">
|
||||
<header class="fixed-top header-content">
|
||||
<nav class="navbar navbar-expand-md navbar-custom" aria-label="Main menu">
|
||||
<a id="ONNXLogo" class="navbar-brand" href="./index.html">
|
||||
<img src="images/svg/ONNX-Runtime-logo.svg" class="d-inline-block align-top onnx-logo" alt="ONNX Runtime Home" />
|
||||
<a id="ONNXLogo" class="navbar-brand" href="/index.html">
|
||||
<img src="/images/svg/ONNX-Runtime-logo.svg" class="d-inline-block align-top onnx-logo" alt="ONNX Runtime Home" />
|
||||
</a>
|
||||
<button class="navbar-toggler p-0" type="button" data-toggle="collapse" data-target="#navbarNav" aria-controls="navbarNav" aria-expanded="false" aria-label="Toggle navigation">
|
||||
<span class="navbar-toggler-icon"></span>
|
||||
|
|
@ -39,13 +39,13 @@
|
|||
<div class="my-md-2 mb-0 mt-2 my-lg-0 pl-3 pl-md-0">
|
||||
<ul class="navbar-nav navbar-nav mr-auto text-uppercase" id="navigation">
|
||||
<li class="nav-item">
|
||||
<a class="nav-link pr-3" href="./index.html#getStartedTable">Get Started</a>
|
||||
<a class="nav-link pr-3" href="/index.html#getStartedTable">Get Started</a>
|
||||
</li>
|
||||
<li class="nav-item">
|
||||
<a class="nav-link pr-3" target="_blank" href="https://cloudblogs.microsoft.com/opensource/tag/onnx">News</a>
|
||||
</li>
|
||||
<li class="nav-item active">
|
||||
<a class="nav-link pr-3" href="./about.html">About</a>
|
||||
<a class="nav-link pr-3" href="/about.html">About</a>
|
||||
</li>
|
||||
<li class="nav-item">
|
||||
<a class="nav-link" target="_blank" href="http://github.com/microsoft/onnxruntime">GitHub</a>
|
||||
|
|
@ -121,7 +121,7 @@
|
|||
</div>
|
||||
<div class="row pt-4 pb-4 text-center-sm">
|
||||
<div class="col-12">
|
||||
<a href="./index.html"><img class="mb-md-2 mb-0 onnx-footer-logo" src="images/svg/ONNX-Runtime-logo-white.svg" alt="ONNX Runtime Home"></a>
|
||||
<a href="/index.html"><img class="mb-md-2 mb-0 onnx-footer-logo" src="/images/svg/ONNX-Runtime-logo-white.svg" alt="ONNX Runtime Home"></a>
|
||||
<p class="text-copyright py-4 ml-md-5">Copyright © 2020 Microsoft. All rights reserved.
|
||||
</p>
|
||||
</div>
|
||||
|
|
|
|||
45
Gemfile
Normal file
45
Gemfile
Normal file
|
|
@ -0,0 +1,45 @@
|
|||
source "https://rubygems.org"
|
||||
|
||||
# Hello! This is where you manage which Jekyll version is used to run.
|
||||
# When you want to use a different version, change it below, save the
|
||||
# file and run `bundle install`. Run Jekyll with `bundle exec`, like so:
|
||||
#
|
||||
# bundle exec jekyll serve
|
||||
#
|
||||
# This will help ensure the proper Jekyll version is running.
|
||||
# Happy Jekylling!
|
||||
# gem "jekyll", "~> 3.9.0"
|
||||
|
||||
# This is the default theme for new Jekyll sites. You may change this to anything you like.
|
||||
gem "just-the-docs"
|
||||
gem "jekyll-toc"
|
||||
gem 'rake'
|
||||
gem 'html-proofer'
|
||||
|
||||
# If you want to use GitHub Pages, remove the "gem "jekyll"" above and
|
||||
# uncomment the line below. To upgrade, run `bundle update github-pages`.
|
||||
gem "github-pages", "~> 207", group: :jekyll_plugins
|
||||
|
||||
# If you have any plugins, put them here!
|
||||
group :jekyll_plugins do
|
||||
gem "jekyll-feed", "~> 0.6"
|
||||
end
|
||||
|
||||
# Windows does not include zoneinfo files, so bundle the tzinfo-data gem
|
||||
# and associated library.
|
||||
install_if -> { RUBY_PLATFORM =~ %r!mingw|mswin|java! } do
|
||||
gem "tzinfo", "~> 1.2"
|
||||
gem "tzinfo-data"
|
||||
end
|
||||
|
||||
# Performance-booster for watching directories on Windows
|
||||
gem "wdm", "~> 0.1.0", :install_if => Gem.win_platform?
|
||||
|
||||
# kramdown v2 ships without the gfm parser by default. If you're using
|
||||
# kramdown v1, comment out this line.
|
||||
gem "kramdown-parser-gfm"
|
||||
|
||||
# Link checker
|
||||
gem 'rake'
|
||||
gem 'html-proofer'
|
||||
|
||||
291
Gemfile.lock
Normal file
291
Gemfile.lock
Normal file
|
|
@ -0,0 +1,291 @@
|
|||
GEM
|
||||
remote: https://rubygems.org/
|
||||
specs:
|
||||
activesupport (6.0.3.2)
|
||||
concurrent-ruby (~> 1.0, >= 1.0.2)
|
||||
i18n (>= 0.7, < 2)
|
||||
minitest (~> 5.1)
|
||||
tzinfo (~> 1.1)
|
||||
zeitwerk (~> 2.2, >= 2.2.2)
|
||||
addressable (2.7.0)
|
||||
public_suffix (>= 2.0.2, < 5.0)
|
||||
coffee-script (2.4.1)
|
||||
coffee-script-source
|
||||
execjs
|
||||
coffee-script-source (1.11.1)
|
||||
colorator (1.1.0)
|
||||
commonmarker (0.17.13)
|
||||
ruby-enum (~> 0.5)
|
||||
concurrent-ruby (1.1.7)
|
||||
dnsruby (1.61.4)
|
||||
simpleidn (~> 0.1)
|
||||
em-websocket (0.5.1)
|
||||
eventmachine (>= 0.12.9)
|
||||
http_parser.rb (~> 0.6.0)
|
||||
ethon (0.12.0)
|
||||
ffi (>= 1.3.0)
|
||||
eventmachine (1.2.7)
|
||||
execjs (2.7.0)
|
||||
faraday (1.0.1)
|
||||
multipart-post (>= 1.2, < 3)
|
||||
ffi (1.13.1)
|
||||
forwardable-extended (2.6.0)
|
||||
gemoji (3.0.1)
|
||||
github-pages (207)
|
||||
github-pages-health-check (= 1.16.1)
|
||||
jekyll (= 3.9.0)
|
||||
jekyll-avatar (= 0.7.0)
|
||||
jekyll-coffeescript (= 1.1.1)
|
||||
jekyll-commonmark-ghpages (= 0.1.6)
|
||||
jekyll-default-layout (= 0.1.4)
|
||||
jekyll-feed (= 0.13.0)
|
||||
jekyll-gist (= 1.5.0)
|
||||
jekyll-github-metadata (= 2.13.0)
|
||||
jekyll-mentions (= 1.5.1)
|
||||
jekyll-optional-front-matter (= 0.3.2)
|
||||
jekyll-paginate (= 1.1.0)
|
||||
jekyll-readme-index (= 0.3.0)
|
||||
jekyll-redirect-from (= 0.15.0)
|
||||
jekyll-relative-links (= 0.6.1)
|
||||
jekyll-remote-theme (= 0.4.1)
|
||||
jekyll-sass-converter (= 1.5.2)
|
||||
jekyll-seo-tag (= 2.6.1)
|
||||
jekyll-sitemap (= 1.4.0)
|
||||
jekyll-swiss (= 1.0.0)
|
||||
jekyll-theme-architect (= 0.1.1)
|
||||
jekyll-theme-cayman (= 0.1.1)
|
||||
jekyll-theme-dinky (= 0.1.1)
|
||||
jekyll-theme-hacker (= 0.1.1)
|
||||
jekyll-theme-leap-day (= 0.1.1)
|
||||
jekyll-theme-merlot (= 0.1.1)
|
||||
jekyll-theme-midnight (= 0.1.1)
|
||||
jekyll-theme-minimal (= 0.1.1)
|
||||
jekyll-theme-modernist (= 0.1.1)
|
||||
jekyll-theme-primer (= 0.5.4)
|
||||
jekyll-theme-slate (= 0.1.1)
|
||||
jekyll-theme-tactile (= 0.1.1)
|
||||
jekyll-theme-time-machine (= 0.1.1)
|
||||
jekyll-titles-from-headings (= 0.5.3)
|
||||
jemoji (= 0.11.1)
|
||||
kramdown (= 2.3.0)
|
||||
kramdown-parser-gfm (= 1.1.0)
|
||||
liquid (= 4.0.3)
|
||||
mercenary (~> 0.3)
|
||||
minima (= 2.5.1)
|
||||
nokogiri (>= 1.10.4, < 2.0)
|
||||
rouge (= 3.19.0)
|
||||
terminal-table (~> 1.4)
|
||||
github-pages-health-check (1.16.1)
|
||||
addressable (~> 2.3)
|
||||
dnsruby (~> 1.60)
|
||||
octokit (~> 4.0)
|
||||
public_suffix (~> 3.0)
|
||||
typhoeus (~> 1.3)
|
||||
html-pipeline (2.14.0)
|
||||
activesupport (>= 2)
|
||||
nokogiri (>= 1.4)
|
||||
html-proofer (3.15.3)
|
||||
addressable (~> 2.3)
|
||||
mercenary (~> 0.3)
|
||||
nokogumbo (~> 2.0)
|
||||
parallel (~> 1.3)
|
||||
rainbow (~> 3.0)
|
||||
typhoeus (~> 1.3)
|
||||
yell (~> 2.0)
|
||||
http_parser.rb (0.6.0)
|
||||
i18n (0.9.5)
|
||||
concurrent-ruby (~> 1.0)
|
||||
jekyll (3.9.0)
|
||||
addressable (~> 2.4)
|
||||
colorator (~> 1.0)
|
||||
em-websocket (~> 0.5)
|
||||
i18n (~> 0.7)
|
||||
jekyll-sass-converter (~> 1.0)
|
||||
jekyll-watch (~> 2.0)
|
||||
kramdown (>= 1.17, < 3)
|
||||
liquid (~> 4.0)
|
||||
mercenary (~> 0.3.3)
|
||||
pathutil (~> 0.9)
|
||||
rouge (>= 1.7, < 4)
|
||||
safe_yaml (~> 1.0)
|
||||
jekyll-avatar (0.7.0)
|
||||
jekyll (>= 3.0, < 5.0)
|
||||
jekyll-coffeescript (1.1.1)
|
||||
coffee-script (~> 2.2)
|
||||
coffee-script-source (~> 1.11.1)
|
||||
jekyll-commonmark (1.3.1)
|
||||
commonmarker (~> 0.14)
|
||||
jekyll (>= 3.7, < 5.0)
|
||||
jekyll-commonmark-ghpages (0.1.6)
|
||||
commonmarker (~> 0.17.6)
|
||||
jekyll-commonmark (~> 1.2)
|
||||
rouge (>= 2.0, < 4.0)
|
||||
jekyll-default-layout (0.1.4)
|
||||
jekyll (~> 3.0)
|
||||
jekyll-feed (0.13.0)
|
||||
jekyll (>= 3.7, < 5.0)
|
||||
jekyll-gist (1.5.0)
|
||||
octokit (~> 4.2)
|
||||
jekyll-github-metadata (2.13.0)
|
||||
jekyll (>= 3.4, < 5.0)
|
||||
octokit (~> 4.0, != 4.4.0)
|
||||
jekyll-mentions (1.5.1)
|
||||
html-pipeline (~> 2.3)
|
||||
jekyll (>= 3.7, < 5.0)
|
||||
jekyll-optional-front-matter (0.3.2)
|
||||
jekyll (>= 3.0, < 5.0)
|
||||
jekyll-paginate (1.1.0)
|
||||
jekyll-readme-index (0.3.0)
|
||||
jekyll (>= 3.0, < 5.0)
|
||||
jekyll-redirect-from (0.15.0)
|
||||
jekyll (>= 3.3, < 5.0)
|
||||
jekyll-relative-links (0.6.1)
|
||||
jekyll (>= 3.3, < 5.0)
|
||||
jekyll-remote-theme (0.4.1)
|
||||
addressable (~> 2.0)
|
||||
jekyll (>= 3.5, < 5.0)
|
||||
rubyzip (>= 1.3.0)
|
||||
jekyll-sass-converter (1.5.2)
|
||||
sass (~> 3.4)
|
||||
jekyll-seo-tag (2.6.1)
|
||||
jekyll (>= 3.3, < 5.0)
|
||||
jekyll-sitemap (1.4.0)
|
||||
jekyll (>= 3.7, < 5.0)
|
||||
jekyll-swiss (1.0.0)
|
||||
jekyll-theme-architect (0.1.1)
|
||||
jekyll (~> 3.5)
|
||||
jekyll-seo-tag (~> 2.0)
|
||||
jekyll-theme-cayman (0.1.1)
|
||||
jekyll (~> 3.5)
|
||||
jekyll-seo-tag (~> 2.0)
|
||||
jekyll-theme-dinky (0.1.1)
|
||||
jekyll (~> 3.5)
|
||||
jekyll-seo-tag (~> 2.0)
|
||||
jekyll-theme-hacker (0.1.1)
|
||||
jekyll (~> 3.5)
|
||||
jekyll-seo-tag (~> 2.0)
|
||||
jekyll-theme-leap-day (0.1.1)
|
||||
jekyll (~> 3.5)
|
||||
jekyll-seo-tag (~> 2.0)
|
||||
jekyll-theme-merlot (0.1.1)
|
||||
jekyll (~> 3.5)
|
||||
jekyll-seo-tag (~> 2.0)
|
||||
jekyll-theme-midnight (0.1.1)
|
||||
jekyll (~> 3.5)
|
||||
jekyll-seo-tag (~> 2.0)
|
||||
jekyll-theme-minimal (0.1.1)
|
||||
jekyll (~> 3.5)
|
||||
jekyll-seo-tag (~> 2.0)
|
||||
jekyll-theme-modernist (0.1.1)
|
||||
jekyll (~> 3.5)
|
||||
jekyll-seo-tag (~> 2.0)
|
||||
jekyll-theme-primer (0.5.4)
|
||||
jekyll (> 3.5, < 5.0)
|
||||
jekyll-github-metadata (~> 2.9)
|
||||
jekyll-seo-tag (~> 2.0)
|
||||
jekyll-theme-slate (0.1.1)
|
||||
jekyll (~> 3.5)
|
||||
jekyll-seo-tag (~> 2.0)
|
||||
jekyll-theme-tactile (0.1.1)
|
||||
jekyll (~> 3.5)
|
||||
jekyll-seo-tag (~> 2.0)
|
||||
jekyll-theme-time-machine (0.1.1)
|
||||
jekyll (~> 3.5)
|
||||
jekyll-seo-tag (~> 2.0)
|
||||
jekyll-titles-from-headings (0.5.3)
|
||||
jekyll (>= 3.3, < 5.0)
|
||||
jekyll-toc (0.14.0)
|
||||
jekyll (>= 3.8)
|
||||
nokogiri (~> 1.10)
|
||||
jekyll-watch (2.2.1)
|
||||
listen (~> 3.0)
|
||||
jemoji (0.11.1)
|
||||
gemoji (~> 3.0)
|
||||
html-pipeline (~> 2.2)
|
||||
jekyll (>= 3.0, < 5.0)
|
||||
just-the-docs (0.3.1)
|
||||
jekyll (>= 3.8.5)
|
||||
jekyll-seo-tag (~> 2.0)
|
||||
rake (>= 12.3.1, < 13.1.0)
|
||||
kramdown (2.3.0)
|
||||
rexml
|
||||
kramdown-parser-gfm (1.1.0)
|
||||
kramdown (~> 2.0)
|
||||
liquid (4.0.3)
|
||||
listen (3.2.1)
|
||||
rb-fsevent (~> 0.10, >= 0.10.3)
|
||||
rb-inotify (~> 0.9, >= 0.9.10)
|
||||
mercenary (0.3.6)
|
||||
mini_portile2 (2.4.0)
|
||||
minima (2.5.1)
|
||||
jekyll (>= 3.5, < 5.0)
|
||||
jekyll-feed (~> 0.9)
|
||||
jekyll-seo-tag (~> 2.1)
|
||||
minitest (5.14.1)
|
||||
multipart-post (2.1.1)
|
||||
nokogiri (1.10.10)
|
||||
mini_portile2 (~> 2.4.0)
|
||||
nokogumbo (2.0.2)
|
||||
nokogiri (~> 1.8, >= 1.8.4)
|
||||
octokit (4.18.0)
|
||||
faraday (>= 0.9)
|
||||
sawyer (~> 0.8.0, >= 0.5.3)
|
||||
parallel (1.19.2)
|
||||
pathutil (0.16.2)
|
||||
forwardable-extended (~> 2.6)
|
||||
public_suffix (3.1.1)
|
||||
rainbow (3.0.0)
|
||||
rake (13.0.1)
|
||||
rb-fsevent (0.10.4)
|
||||
rb-inotify (0.10.1)
|
||||
ffi (~> 1.0)
|
||||
rexml (3.2.4)
|
||||
rouge (3.19.0)
|
||||
ruby-enum (0.8.0)
|
||||
i18n
|
||||
rubyzip (2.3.0)
|
||||
safe_yaml (1.0.5)
|
||||
sass (3.7.4)
|
||||
sass-listen (~> 4.0.0)
|
||||
sass-listen (4.0.0)
|
||||
rb-fsevent (~> 0.9, >= 0.9.4)
|
||||
rb-inotify (~> 0.9, >= 0.9.7)
|
||||
sawyer (0.8.2)
|
||||
addressable (>= 2.3.5)
|
||||
faraday (> 0.8, < 2.0)
|
||||
simpleidn (0.1.1)
|
||||
unf (~> 0.1.4)
|
||||
terminal-table (1.8.0)
|
||||
unicode-display_width (~> 1.1, >= 1.1.1)
|
||||
thread_safe (0.3.6)
|
||||
typhoeus (1.4.0)
|
||||
ethon (>= 0.9.0)
|
||||
tzinfo (1.2.7)
|
||||
thread_safe (~> 0.1)
|
||||
tzinfo-data (1.2020.1)
|
||||
tzinfo (>= 1.0.0)
|
||||
unf (0.1.4)
|
||||
unf_ext
|
||||
unf_ext (0.0.7.7)
|
||||
unicode-display_width (1.7.0)
|
||||
wdm (0.1.1)
|
||||
yell (2.2.2)
|
||||
zeitwerk (2.4.0)
|
||||
|
||||
PLATFORMS
|
||||
ruby
|
||||
|
||||
DEPENDENCIES
|
||||
github-pages (~> 207)
|
||||
html-proofer
|
||||
jekyll-feed (~> 0.6)
|
||||
jekyll-toc
|
||||
just-the-docs
|
||||
kramdown-parser-gfm
|
||||
rake
|
||||
tzinfo (~> 1.2)
|
||||
tzinfo-data
|
||||
wdm (~> 0.1.0)
|
||||
|
||||
BUNDLED WITH
|
||||
2.1.4
|
||||
23
_config.yml
23
_config.yml
|
|
@ -1 +1,22 @@
|
|||
|
||||
remote_theme: pmarsceill/just-the-docs
|
||||
color_scheme: onnxruntime
|
||||
search_enabled: false
|
||||
plugins:
|
||||
- jekyll-toc
|
||||
kramdown:
|
||||
parse_block_html: true
|
||||
logo: "/images/ONNX-Runtime-logo.svg"
|
||||
aux_links:
|
||||
"Get started":
|
||||
- "/docs/get-started/"
|
||||
"Tutorials":
|
||||
- "/docs/tutorials/"
|
||||
"Recipes":
|
||||
- "/docs/how-to/"
|
||||
"Reference":
|
||||
- "/docs/reference/"
|
||||
"Resources":
|
||||
- "/docs/resources/"
|
||||
"GitHub":
|
||||
- "https://github.com/microsoft/onnxruntime"
|
||||
ga_tracking: UA-156955408-1
|
||||
|
|
|
|||
2
_sass/color_schemes/onnxruntime.scss
Normal file
2
_sass/color_schemes/onnxruntime.scss
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
$link-color: $blue-000;
|
||||
$btn-primary-color: $blue-000;
|
||||
|
|
@ -69,14 +69,14 @@ header.scrolled .navbar-custom .navbar-nav .nav-item:not(:last-child) .nav-link{
|
|||
border-right: 2px solid #1a1a1a;
|
||||
}
|
||||
.navbar-toggler .navbar-toggler-icon {
|
||||
background-image: url('../images/svg/menu-icon.svg');
|
||||
background-image: url('/images/svg/menu-icon.svg');
|
||||
width: 3em;
|
||||
height: 3em;
|
||||
-webkit-transition:0.2s;
|
||||
transition:0.2s;
|
||||
}
|
||||
header.header-collapse .navbar-toggler .navbar-toggler-icon {
|
||||
background-image: url('../images/svg/menu-cross-icon.svg');
|
||||
background-image: url('/images/svg/menu-cross-icon.svg');
|
||||
width: 2.5em;
|
||||
height: 2.5em;
|
||||
-webkit-transition:0.2s;
|
||||
|
|
@ -193,7 +193,7 @@ a.link:active .link-arrow{
|
|||
border: 2px solid transparent;;
|
||||
}
|
||||
.main-wrapper .top-banner-bg{
|
||||
background-image:url(../images/top-banner-background.jpg);
|
||||
background-image:url(/images/top-banner-background.jpg);
|
||||
background-size: auto;
|
||||
background-position: top center;
|
||||
background-repeat: repeat no-repeat;
|
||||
|
|
@ -563,7 +563,7 @@ a .abbr[data-original-title], a abbr[title]{
|
|||
}
|
||||
.resources-img::before{
|
||||
content: '';
|
||||
background-image: url(../images/youtube-logo.png);
|
||||
background-image: url(/images/youtube-logo.png);
|
||||
background-position: center;
|
||||
background-size: 50%;
|
||||
background-repeat: no-repeat;
|
||||
|
|
|
|||
17
docs/get-started/index.md
Normal file
17
docs/get-started/index.md
Normal file
|
|
@ -0,0 +1,17 @@
|
|||
---
|
||||
title: Get Started
|
||||
has_children: true
|
||||
nav_order: 1
|
||||
---
|
||||
|
||||
# Get Started
|
||||
|
||||
ONNX Runtime is a cross-platform accelerator for training and inference, compatible with PyTorch, TensorFlow/Keras, scikit-learn, and more.
|
||||
|
||||
You can benefit from ONNX Runtime if you want to:
|
||||
|
||||
* Improve inference performance of machine learning models
|
||||
* Reduce time and cost of training large models
|
||||
* Train in Python but deploy into a C#/C++/Java app
|
||||
* Run on different hardware and operating systems
|
||||
* Support models created in several different frameworks
|
||||
69
docs/get-started/inference.md
Normal file
69
docs/get-started/inference.md
Normal file
|
|
@ -0,0 +1,69 @@
|
|||
---
|
||||
title: Inference
|
||||
parent: Get Started
|
||||
nav_order: 2
|
||||
---
|
||||
|
||||
# Use ONNX Runtime for Inference
|
||||
|
||||
## Docker Images
|
||||
|
||||
* [ONNX-Ecosystem](https://github.com/onnx/onnx-docker/tree/master/onnx-ecosystem): includes ONNX Runtime (CPU, Python), dependencies, tools to convert from various frameworks, and Jupyter notebooks to help get started
|
||||
* [Additional dockerfiles](https://github.com/microsoft/onnxruntime/tree/master/dockerfiles)
|
||||
|
||||
## API Documentation
|
||||
|
||||
|API|Supported Versions|Samples|
|
||||
|---|---|---|
|
||||
[Python](https://aka.ms/onnxruntime-python)| 3.5, 3.6, 3.7<br>[Python Dev Notes](https://github.com/microsoft/onnxruntime/tree/master/docs/Python_Dev_Notes.md)| [Samples](https://github.com/microsoft/onnxruntime/tree/master/samples/#python)|
|
||||
|[C#](../reference/api/csharp-api.md)| | [Samples](https://github.com/microsoft/onnxruntime/tree/master/samples/#C)|
|
||||
|[C++](https://github.com/microsoft/onnxruntime/blob/master/include/onnxruntime/core/session/onnxruntime_cxx_api.h)| |[Samples](https://github.com/microsoft/onnxruntime/tree/master/samples/#CC)|
|
||||
|[C](../reference/api/c-api.md)| | [Samples](https://github.com/microsoft/onnxruntime/tree/master/samples/#CC)|
|
||||
|[WinRT](../reference/api/winrt-api.md) | [Windows.AI.MachineLearning](https://docs.microsoft.com/en-us/windows/ai/windows-ml/api-reference)| [Samples](https://github.com/microsoft/windows-Machine-Learning)|
|
||||
|[Java](../reference/api/java-api.md)|8-13|[Samples](https://github.com/microsoft/onnxruntime/tree/master/samples/#Java)|
|
||||
[Ruby](https://github.com/ankane/onnxruntime) (external project)| 2.4-2.7| [Samples](https://ankane.org/tensorflow-ruby)|
|
||||
|[Javascript (node.js)](../reference/api/nodejs-api.md) |12.x | [Samples](https://github.com/microsoft/onnxruntime/blob/master/samples/nodejs) |
|
||||
|
||||
## Supported Accelerators
|
||||
|
||||
[Execution Providers](../reference/execution-providers)
|
||||
|
||||
|CPU|GPU|IoT/Edge/Mobile|Other|
|
||||
|---|---|---|---|
|
||||
|Default CPU - *MLAS (Microsoft Linear Algebra Subprograms) + Eigen*|NVIDIA CUDA|[Intel OpenVINO](../reference/execution-providers/OpenVINO-ExecutionProvider.md)|[Nuphar Model Compiler](../reference/execution-providers/Nuphar-ExecutionProvider.md) (*preview*)|
|
||||
|[Intel DNNL](../reference/execution-providers/DNNL-ExecutionProvider.md)|[NVIDIA TensorRT](../reference/execution-providers/TensorRT-ExecutionProvider.md)|[ARM Compute Library](../reference/execution-providers/ACL-ExecutionProvider.md) (*preview*)|[Rockchip NPU](../reference/execution-providers/RKNPU-ExecutionProvider.md) (*preview*)|
|
||||
|[Intel nGraph](../reference/execution-providers/nGraph-ExecutionProvider.md)|[DirectML](../reference/execution-providers/DirectML-ExecutionProvider.md)|[Android Neural Networks API](../reference/execution-providers/NNAPI-ExecutionProvider.md) (*preview*)|[Xilinx Vitis-AI](../reference/execution-providers/Vitis-AI-ExecutionProvider.md) (*preview*)|
|
||||
|Intel MKL-ML *(build option)*|[AMD MIGraphX](../reference/execution-providers/MIGraphX-ExecutionProvider.md)|[ARM-NN](../reference/execution-providers/ArmNN-ExecutionProvider.md) (*preview*)|
|
||||
|
||||
* [Roadmap: Upcoming accelerators](https://github.com/microsoft/onnxruntime/tree/master/docs/Roadmap.md#accelerators-and-execution-providers)
|
||||
|
||||
## Deploying ONNX Runtime
|
||||
|
||||
### Cloud
|
||||
|
||||
* ONNX Runtime can be deployed to any cloud for model inference, including [Azure Machine Learning Services](https://azure.microsoft.com/en-us/services/machine-learning-service).
|
||||
* [Detailed instructions](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-build-deploy-onnx)
|
||||
* [AzureML sample notebooks](https://github.com/Azure/MachineLearningNotebooks/tree/master/how-to-use-azureml/deployment/onnx)
|
||||
|
||||
* **ONNX Runtime Server (beta)** is a hosting application for serving ONNX models using ONNX Runtime, providing a REST API for prediction.
|
||||
* [Usage details](https://github.com/microsoft/onnxruntime/tree/master/docs/ONNX_Runtime_Server_Usage.md)
|
||||
* [Image installation instructions](https://github.com/microsoft/onnxruntime/tree/master/dockerfiles#onnx-runtime-server-preview)
|
||||
|
||||
### IoT and edge devices
|
||||
|
||||
* [Reference implementations](https://github.com/Azure-Samples/onnxruntime-iot-edge)
|
||||
|
||||
The expanding focus and selection of IoT devices with sensors and consistent signal streams introduces new opportunities to move AI workloads to the edge.
|
||||
This is particularly important when there are massive volumes of incoming data/signals that may not be efficient or useful to push to the cloud due to storage or latency considerations. Consider: surveillance tapes where 99% of footage is uneventful, or real-time person detection scenarios where immediate action is required. In these scenarios, directly executing model inference on the target device is crucial for optimal assistance.
|
||||
|
||||
### Client applications
|
||||
|
||||
* Install or build the package you need to use in your application. ([sample implementations](https://github.com/microsoft/onnxruntime/tree/master/samples/c_cxx) using the C++ API)
|
||||
|
||||
* On newer Windows 10 devices (1809+), ONNX Runtime is available by default as part of the OS and is accessible via the [Windows Machine Learning APIs](https://docs.microsoft.com/en-us/windows/ai/windows-ml/). ([Tutorials for Windows Desktop or UWP app](https://docs.microsoft.com/en-us/windows/ai/windows-ml/get-started-desktop))
|
||||
|
||||
## Build from Source
|
||||
|
||||
For production scenarios, it's strongly recommended to build only from an [official release branch](https://github.com/microsoft/onnxruntime/releases).
|
||||
|
||||
* [Instructions for additional build flavors](../how-to/build.md)
|
||||
86
docs/get-started/install.md
Normal file
86
docs/get-started/install.md
Normal file
|
|
@ -0,0 +1,86 @@
|
|||
---
|
||||
title: Install
|
||||
parent: Get Started
|
||||
nav_order: 1
|
||||
---
|
||||
|
||||
# Install ONNX Runtime
|
||||
{: .no_toc }
|
||||
|
||||
## Contents
|
||||
{: .no_toc }
|
||||
|
||||
* TOC placeholder
|
||||
{:toc}
|
||||
|
||||
Use this guide to install ONNX Runtime and its dependencies, for your target operating system, hardware, accelerator, and language.
|
||||
|
||||
For an overview, see this [installation matrix](https://onnxruntime.ai).
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### Linux / CPU
|
||||
|
||||
1. English language package with the `en_US.UTF-8` locale
|
||||
|
||||
* Install [language-pack-en package](https://packages.ubuntu.com/search?keywords=language-pack-en)
|
||||
* Run `locale-gen en_US.UTF-8`
|
||||
* Run `update-locale LANG=en_US.UTF-8`
|
||||
|
||||
2. OpenMP
|
||||
|
||||
* `apt-get install libgomp1`, which installs **libgomp.so.1**
|
||||
|
||||
### Linux / GPU
|
||||
|
||||
1. English language package with the `en_US.UTF-8` locale
|
||||
|
||||
* Install [language-pack-en package](https://packages.ubuntu.com/search?keywords=language-pack-en)
|
||||
* Run `locale-gen en_US.UTF-8`
|
||||
* Run `update-locale LANG=en_US.UTF-8`
|
||||
|
||||
2. CUDA 10.1 and cuDNN 7.6.5
|
||||
|
||||
* Version dependencies from older ONNX Runtime releases can be found in [prior release notes](https://github.com/microsoft/onnxruntime/releases).
|
||||
|
||||
### Windows / CPU
|
||||
|
||||
1. English language package with the `en_US.UTF-8` locale
|
||||
|
||||
2. [Visual C++ 2019 runtime](https://support.microsoft.com/en-us/help/2977003/the-latest-supported-visual-c-downloads)
|
||||
|
||||
3. OpenMP
|
||||
* Comes installed with Visual C++ runtime
|
||||
* Or is also available as redist packages [vc_redist.x64.exe](https://aka.ms/vs/16/release/vc_redist.x64.exe) and [vc_redist.x86.exe](https://aka.ms/vs/16/release/vc_redist.x86.exe)
|
||||
|
||||
### Windows / GPU
|
||||
|
||||
1. English language package with the `en_US.UTF-8` locale
|
||||
|
||||
2. [Visual C++ 2019 runtime](https://support.microsoft.com/en-us/help/2977003/the-latest-supported-visual-c-downloads)
|
||||
|
||||
3. CUDA 10.1 and cuDNN 7.6.5
|
||||
* Version dependencies from older ONNX Runtime releases can be found in [prior release notes](https://github.com/microsoft/onnxruntime/releases).
|
||||
|
||||
## Install
|
||||
|
||||
* Default CPU Provider (Eigen + MLAS)
|
||||
* GPU Provider - NVIDIA CUDA
|
||||
* GPU Provider - DirectML (Windows)
|
||||
* *On Windows, the [DirectML execution provider](https://github.com/microsoft/onnxruntime/tree/master/docs/execution_providers/DirectML-ExecutionProvider.md) is recommended for optimal performance and compatibility with a broad set of GPUs.*
|
||||
|
||||
If using pip, run `pip install --upgrade pip` prior to downloading.
|
||||
|
||||
|Repository|Official build|Nightly build|
|
||||
|---|---|---|
|
||||
|Python|CPU: [**onnxruntime**](https://pypi.org/project/onnxruntime)| [ort-nightly (dev)](https://test.pypi.org/project/ort-nightly)|
|
||||
||GPU: [**onnxruntime-gpu**](https://pypi.org/project/onnxruntime-gpu) | [ort-gpu-nightly (dev)](https://test.pypi.org/project/ort-gpu-nightly)|
|
||||
|C#/C/C++|CPU: [**Microsoft.ML.OnnxRuntime**](https://www.nuget.org/packages/Microsoft.ML.OnnxRuntime) | [ort-nightly (dev)](https://aiinfra.visualstudio.com/PublicPackages/_packaging?_a=feed&feed=ORT-Nightly)|
|
||||
||GPU: [**Microsoft.ML.OnnxRuntime.Gpu**](https://www.nuget.org/packages/Microsoft.ML.OnnxRuntime.gpu)|[ort-nightly (dev)](https://aiinfra.visualstudio.com/PublicPackages/_packaging?_a=feed&feed=ORT-Nightly)|
|
||||
|Java|CPU: [**com.microsoft.onnxruntime/onnxruntime**](https://search.maven.org/artifact/com.microsoft.onnxruntime/onnxruntime)|
|
||||
||GPU: [**com.microsoft.onnxruntime/onnxruntime_gpu**](https://search.maven.org/artifact/com.microsoft.onnxruntime/onnxruntime_gpu)|
|
||||
|nodejs|CPU: [**onnxruntime**](https://www.npmjs.com/package/onnxruntime)|
|
||||
|Other|[Contributed non-official packages](https://docs.microsoft.com/en-us/windows/ai/windows-ml/get-started-uwp) (including Homebrew, Linuxbrew, and nixpkgs)|
|
||||
||These are not maintained by the core ONNX Runtime team and may have limited support; use at your discretion.|
|
||||
|
||||
Note: Dev builds created from the master branch are available for testing newer changes between official releases. Please use these at your own risk. We strongly advise against deploying these to production workloads as support is limited for dev builds.
|
||||
47
docs/get-started/training.md
Normal file
47
docs/get-started/training.md
Normal file
|
|
@ -0,0 +1,47 @@
|
|||
---
|
||||
title: Training
|
||||
parent: Get Started
|
||||
nav_order: 3
|
||||
---
|
||||
|
||||
# Use ONNX Runtime for Training
|
||||
|
||||
The ONNX Runtime training feature enables easy integration with existing Pytorch trainer code to accelerate the execution. With a few lines of code, you can add ONNX Runtime into your existing training scripts and start seeing acceleration. The current preview version supports training acceleration for transformer models on NVIDIA GPUs.
|
||||
|
||||
**[ONNX Runtime pre-training sample](https://github.com/microsoft/onnxruntime-training-examples)**: This sample is setup to pre-train the BERT-Large model to show how ONNX Runtime training can be used to accelerate training execution.
|
||||
|
||||
## Train PyTorch model with ONNX Runtime
|
||||
|
||||
ONNX Runtime (ORT) has the capability to train existing PyTorch models through its optimized backend. For this, we have introduced an python API for PyTorch, called ORTTrainer, which can be used to switch the training backend for PyTorch models (instance of `torch.nn.Module`) to `orttrainer`. This requires some changes in the trainer code, such as replacing the PyTorch optimizer, and optionally, setting flags to enable additional features such as mixed-precision training. Here is a sample code fragment to integrate ONNX Runtime Training in your PyTorch pre-training script:
|
||||
|
||||
_NOTE: The current API is experimental and expected to see significant changes in the near future. Our goal is to improve the interface to provide a seamless integration with PyTorch training that requires minimal changes in users’ training code._
|
||||
|
||||
```python
|
||||
import torch
|
||||
...
|
||||
import onnxruntime
|
||||
from onnxruntime.capi.ort_trainer import IODescription, ModelDescription, ORTTrainer
|
||||
|
||||
# Model definition
|
||||
class Net(torch.nn.Module):
|
||||
def __init__(self, D_in, H, D_out):
|
||||
...
|
||||
def forward(self, x):
|
||||
...
|
||||
|
||||
model = Net(D_in, H, H_out)
|
||||
criterion = torch.nn.Functional.cross_entropy
|
||||
description = ModelDescription(...)
|
||||
optimizer = 'SGDOptimizer'
|
||||
trainer = ORTTrainer(model, criterion, description, optimizer, ...)
|
||||
|
||||
# Training Loop
|
||||
for t in range(1000):
|
||||
# forward + backward + weight update
|
||||
loss, y_pred = trainer.train_step(x, y, learning_rate)
|
||||
...
|
||||
```
|
||||
|
||||
## Build ONNX Runtime Training from source
|
||||
|
||||
To use ONNX Runtime training in a custom environment, like on-prem NVIDIA DGX-2 clusters, you can use these [build instructions](../how-to/build.md#training) to generate the Python package to integrate into existing trainer code.
|
||||
44
docs/how-to/add-custom-op.md
Normal file
44
docs/how-to/add-custom-op.md
Normal file
|
|
@ -0,0 +1,44 @@
|
|||
---
|
||||
title: Add a new operator
|
||||
parent: Recipes
|
||||
nav_order: 2
|
||||
---
|
||||
|
||||
# Add a new operator to ONNX Runtime
|
||||
{: .no_toc }
|
||||
|
||||
## Contents
|
||||
{: .no_toc }
|
||||
|
||||
* TOC placeholder
|
||||
{:toc}
|
||||
|
||||
A new op can be written and registered with ONNXRuntime in the following 3 ways
|
||||
|
||||
1. Using the experimental custom op API in the C API (onnxruntime_c_api.h)
|
||||
Note: These APIs are experimental and will change in the next release. They're released now for feedback and experimentation.
|
||||
|
||||
1. Using the experimental custom op API in the C API (onnxruntime_c_api.h)
|
||||
|
||||
Note: These APIs are experimental and will change in the next release. They're released now for feedback and experimentation.
|
||||
|
||||
* Create an OrtCustomOpDomain with the domain name used by the custom ops
|
||||
* Create an OrtCustomOp structure for each op and add them to the OrtCustomOpDomain with OrtCustomOpDomain_Add
|
||||
* Call OrtAddCustomOpDomain to add the custom domain of ops to the session options
|
||||
See [this](https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/test/shared_lib/test_inference.cc) for an example called MyCustomOp that uses the C++ helper API (onnxruntime_cxx_api.h).
|
||||
Currently, the only supported Execution Providers (EPs) for custom ops registered via this approach are the `CUDA` and the `CPU` EPs.
|
||||
|
||||
2. Using RegisterCustomRegistry API
|
||||
|
||||
* Implement your kernel and schema (if required) using the OpKernel and OpSchema APIs (headers are in the include folder).
|
||||
* Create a CustomRegistry object and register your kernel and schema with this registry.
|
||||
* Register the custom registry with ONNXRuntime using RegisterCustomRegistry API.
|
||||
|
||||
See
|
||||
[this](https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/test/framework/local_kernel_registry_test.cc) for an example.
|
||||
|
||||
3. Contributing the op to ONNXRuntime
|
||||
|
||||
This is mostly meant for ops that are in the process of being proposed to ONNX. This way you don't have to wait for an approval from the ONNX team
|
||||
if the op is required in production today.
|
||||
See [this](https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/contrib_ops) for an example.
|
||||
49
docs/how-to/add-execution-provider.md
Normal file
49
docs/how-to/add-execution-provider.md
Normal file
|
|
@ -0,0 +1,49 @@
|
|||
---
|
||||
title: Add a new execution provider
|
||||
parent: Recipes
|
||||
nav_order: 3
|
||||
---
|
||||
|
||||
# Add a new Execution Provider to ONNX Runtime
|
||||
{: .no_toc }
|
||||
|
||||
## Contents
|
||||
{: .no_toc }
|
||||
|
||||
* TOC placeholder
|
||||
{:toc}
|
||||
|
||||
## Create the Execution Provider
|
||||
|
||||
* Create a folder under onnxruntime/core/providers
|
||||
* Create a folder under include/onnxruntime/core/providers, it should has the same name as the first step.
|
||||
* Create a new class, which must inherit from [IExecutionProvider](https://github.com/microsoft/onnxruntime/tree/master/include//onnxruntime/core/framework/execution_provider.h). The source code should be put in 'onnxruntime/core/providers/[your_provider_name]'
|
||||
* Create a new header file under include/onnxruntime/core/providers/[your_provider_name]. The file should provide one function for creating an OrtProviderFactoryInterface. You may use 'include/onnxruntime/core/providers/cpu/cpu_provider_factory.h' as a template. You don't need to provide a function for creating MemoryInfo.
|
||||
* Put a symbols.txt under 'onnxruntime/core/providers/[your_provider_name]'. The file should contain all the function names that would be exported from you provider. Usually, just a single function for creating provider factory is enough.
|
||||
* Add your provider in onnxruntime_providers.cmake. Build it as a static lib.
|
||||
* Add one line in cmake/onnxruntime.cmake, to the 'target_link_libraries' function call. Put your provider there.
|
||||
|
||||
Examples:
|
||||
|
||||
* [CPU Execution Provider](https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/core/providers/cpu/cpu_execution_provider.h)
|
||||
* [CUDA Execution Provider](https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/core/providers/cuda/cuda_execution_provider.h)
|
||||
* [DNNL Execution Provider](https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/core/providers/dnnl/dnnl_execution_provider.h)
|
||||
|
||||
## Use the execution provider
|
||||
|
||||
* Create a factory for that provider, by using the c function you exported in 'symbols.txt'
|
||||
* Put the provider factory into session options
|
||||
* Create session from that session option
|
||||
|
||||
Example:
|
||||
|
||||
```c
|
||||
OrtEnv* env;
|
||||
OrtInitialize(ORT_LOGGING_LEVEL_WARNING, "test", &env)
|
||||
OrtSessionOptions* session_option = OrtCreateSessionOptions();
|
||||
OrtProviderFactoryInterface** factory;
|
||||
OrtCreateCUDAExecutionProviderFactory(0, &factory);
|
||||
OrtSessionOptionsAppendExecutionProvider(session_option, factory);
|
||||
OrtReleaseObject(factory);
|
||||
OrtCreateSession(env, model_path, session_option, &session);
|
||||
```
|
||||
1051
docs/how-to/build.md
Normal file
1051
docs/how-to/build.md
Normal file
File diff suppressed because it is too large
Load diff
143
docs/how-to/custom-python-operator.md
Normal file
143
docs/how-to/custom-python-operator.md
Normal file
|
|
@ -0,0 +1,143 @@
|
|||
---
|
||||
title: Use the PyOp operator
|
||||
parent: Recipes
|
||||
nav_order: 4
|
||||
---
|
||||
|
||||
# How to use the Python Operator (PyOp)
|
||||
{: .no_toc }
|
||||
|
||||
The Python Operator provides the capability to easily invoke any custom Python code within a single node of an ONNX graph using ONNX Runtime. This can be useful for quicker experimentation when a model requires operators that are not officially supported in ONNX and ONNX Runtime, particularly if there is already a Python implementation for the required functionality. This should be used with discretion in production scenarios, and all security or other risks should be considered beforehand.
|
||||
|
||||
## Design Overview
|
||||
|
||||
The feature can be in [language_interop_ops](https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/core/language_interop_ops).
|
||||
|
||||
Here is a chart of calling sequence:
|
||||
<pre>
|
||||
onnxruntime python capi script
|
||||
| | |
|
||||
| ------------------------------> | |
|
||||
| call with tensor(s) | ------------------------------> |
|
||||
| | call with numpy(s) |
|
||||
| | | compute
|
||||
| | <------------------------------ |
|
||||
| <------------------------------ | return numpys(s) |
|
||||
| return tensor(s) | |
|
||||
</pre>
|
||||
|
||||
## How to Use
|
||||
### Step 1
|
||||
Build onnxruntime with `--config Release --enable_language_interop_ops --build_wheel` and pip install the latest wheel file.
|
||||
|
||||
### Step 2
|
||||
Create an onnx model containing Python operator nodes:
|
||||
```python
|
||||
ad1_node = helper.make_node('Add', ['A','B'], ['S'])
|
||||
mul_node = helper.make_node('Mul', ['C','D'], ['P'])
|
||||
py1_node = helper.make_node(op_type = 'PyOp', #required, must be 'PyOp'
|
||||
inputs = ['S','P'], #required
|
||||
outputs = ['L','M','N'], #required
|
||||
domain = 'pyopmulti_1', #required, must be unique
|
||||
input_types = [TensorProto.FLOAT, TensorProto.FLOAT], #required
|
||||
output_types = [TensorProto.FLOAT, TensorProto.FLOAT, TensorProto.FLOAT], #required
|
||||
module = 'mymodule', #required
|
||||
class_name = 'Multi_1', #required
|
||||
compute = 'compute', #optional, 'compute' by default
|
||||
W1 = '5', W2 = '7', W3 = '9') #optional, must all be strings
|
||||
ad2_node = helper.make_node('Add', ['L','M'], ['H'])
|
||||
py2_node = helper.make_node('PyOp',['H','N','E'],['O','W'], domain = 'pyopmulti_2',
|
||||
input_types = [TensorProto.FLOAT, TensorProto.FLOAT, TensorProto.FLOAT],
|
||||
output_types = [TensorProto.FLOAT, TensorProto.FLOAT],
|
||||
module = 'mymodule', class_name = 'Multi_2')
|
||||
sub_node = helper.make_node('Sub', ['O','W'], ['F'])
|
||||
graph = helper.make_graph([ad1_node,mul_node,py1_node,ad2_node,py2_node,sub_node], 'multi_pyop_graph', [A,B,C,D,E], [F])
|
||||
model = helper.make_model(graph, producer_name = 'pyop_model')
|
||||
onnx.save(model, './model.onnx')
|
||||
```
|
||||
### Step 3
|
||||
Implement mymodule.py:
|
||||
```python
|
||||
class Multi_1:
|
||||
def __init__(self, W1, W2, W3):
|
||||
self.W1 = int(W1)
|
||||
self.W2 = int(W2)
|
||||
self.W3 = int(W3)
|
||||
def compute(self, S, P):
|
||||
ret = S + P
|
||||
return ret + self.W1, ret + self.W2, ret + self.W3
|
||||
class Multi_2:
|
||||
def compute(self, *kwargs):
|
||||
return sum(kwargs[0:-1]), sum(kwargs[1:])
|
||||
```
|
||||
### Step 4
|
||||
Copy mymodule.py into Python sys.path, then run the model with onnxruntime python API. On Windows, please set PYTHONHOME beforehand. It should point to directory where the python is installed, such as C:\Python37 or C:\ProgramData\Anaconda3\envs\myconda1 if it is in conda.
|
||||
|
||||
## Supported Data Types
|
||||
* TensorProto.BOOL
|
||||
* TensorProto.UINT8
|
||||
* TensorProto.UINT16
|
||||
* TensorProto.UINT32
|
||||
* TensorProto.INT16
|
||||
* TensorProto.INT32
|
||||
* TensorProto.FLOAT
|
||||
* TensorProto.DOUBLE
|
||||
|
||||
## Limitations
|
||||
* Inferencing and compiling environments must be installed with same version of python.
|
||||
* On Windows, `--config Debug` has known issues. Please build with `--config RelWithDebInfo` if debugging symbols are needed.
|
||||
* Due to Python C API restrictions, multi-threading is disabled so Python operators will run sequentially.
|
||||
|
||||
## Test Coverage
|
||||
The operator has been tested on multiple platforms, with or without conda:
|
||||
|
||||
Platform | Python 3.5 | Python 3.6 | Python 3.7
|
||||
----------- | ------------| ----------- | -----------
|
||||
Windows | (conda) passed | (conda) passed | passed
|
||||
Linux | (conda) passed | (conda) passed | passed
|
||||
Mac | (conda) passed | (conda) passed | (conda) passed
|
||||
|
||||
## Example
|
||||
Developers could resort to PyOp during model conversion for missing operators:
|
||||
```python
|
||||
import os
|
||||
import numpy as np
|
||||
from onnx import *
|
||||
from skl2onnx import convert_sklearn
|
||||
from skl2onnx.common.data_types import FloatTensorType
|
||||
from skl2onnx.common.utils import check_input_and_output_numbers
|
||||
|
||||
X = np.array([[1, 1], [2, 1], [3, 1.2], [4, 1], [5, 0.8], [6, 1]],dtype=np.single)
|
||||
nmf = NMF(n_components=2, init='random', random_state=0)
|
||||
W = np.array(nmf.fit_transform(X), dtype=np.single)
|
||||
|
||||
def calculate_sklearn_nmf_output_shapes(operator):
|
||||
check_input_and_output_numbers(operator, output_count_range=1, input_count_range=1)
|
||||
operator.outputs[0].type.shape = operator.inputs[0].type.shape
|
||||
|
||||
def convert_nmf(scope, operator, container):
|
||||
ws = [str(w) for w in W.flatten()]
|
||||
attrs = {'W':'|'.join(ws)}
|
||||
container.add_node(op_type='PyOp', name='nmf', inputs=['X'], outputs=['variable'],
|
||||
op_version=10, op_domain='MyDomain', module='mymodule', class_name='MyNmf',
|
||||
input_types=[TensorProto.FLOAT], output_types=[TensorProto.FLOAT], **attrs)
|
||||
|
||||
custom_shape_calculators = {type(nmf): calculate_sklearn_nmf_output_shapes}
|
||||
custom_conversion_functions = {type(nmf): convert_nmf}
|
||||
initial_types = [('X', FloatTensorType([6,2]))]
|
||||
onx = convert_sklearn(nmf, '', initial_types, '', None, custom_conversion_functions, custom_shape_calculators)
|
||||
with th open("model.onnx", "wb") as f:
|
||||
f.write(onx.SerializeToString())
|
||||
```
|
||||
mymodule.py:
|
||||
```python
|
||||
import numpy as np
|
||||
class MyNmf:
|
||||
def __init__(self,W):
|
||||
A = []
|
||||
for w in W.split('|'):
|
||||
A.append(float(w))
|
||||
self.__W = np.array(A,dtype=np.single).reshape(6,2)
|
||||
def compute(self,X):
|
||||
return self.__W
|
||||
```
|
||||
186
docs/how-to/export-pytorch-model.md
Normal file
186
docs/how-to/export-pytorch-model.md
Normal file
|
|
@ -0,0 +1,186 @@
|
|||
---
|
||||
title: Export PyTorch model
|
||||
parent: Recipes
|
||||
nav_order: 5
|
||||
---
|
||||
|
||||
## Export PyTorch model with custom ONNX operators
|
||||
{: .no_toc }
|
||||
|
||||
This document explains the process of exporting PyTorch models with custom ONNX Runtime ops. The aim is to export a PyTorch model with operators that are not supported in ONNX, and extend ONNX Runtime to support these custom ops.
|
||||
|
||||
Currently, a torch op can be exported as a custom operator using our custom op (symbolic) registration API. We can use this API to register custom ONNX Runtime ops under "com.microsoft" domain.
|
||||
|
||||
## Contents
|
||||
{: .no_toc }
|
||||
|
||||
* TOC placeholder
|
||||
{:toc}
|
||||
|
||||
### Export a Custom Op
|
||||
|
||||
In this example, we take Inverse operator as an example. To enable export of ```torch.inverse```, a symbolic function can be created and registered as part of custom ops:
|
||||
|
||||
```python
|
||||
from torch.onnx import register_custom_op_symbolic
|
||||
|
||||
def my_inverse(g, self):
|
||||
return g.op("com.microsoft::Inverse", self)
|
||||
|
||||
# register_custom_op_symbolic('<namespace>::inverse', my_inverse, <opset_version>)
|
||||
register_custom_op_symbolic('::inverse', my_inverse, 1)
|
||||
```
|
||||
|
||||
`<namespace>` is a part of the torch operator name. For standard torch operators, namespace can be omitted.
|
||||
|
||||
`com.microsoft` should be used as the custom opset domain for ONNX Runtime ops. You can choose the custom opset version during op registration.
|
||||
|
||||
All symbolics for ONNX Runtime custom ops are defined in `tools/python/register_custom_ops_pytorch_exporter.py`.
|
||||
|
||||
If you are adding a symbolic function for a new custom op, add the function to this file.
|
||||
|
||||
### Extend ONNX Runtime with Custom Ops
|
||||
|
||||
The next step is to add op schema and kernel implementation in ONNX Runtime.
|
||||
Consider the Inverse custom op as an example added in:
|
||||
https://github.com/microsoft/onnxruntime/pull/3485
|
||||
|
||||
Custom op schema and shape inference function should be added in https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/core/graph/contrib_ops/contrib_defs.cc using `ONNX_CONTRIB_OPERATOR_SCHEMA`.
|
||||
|
||||
```c++
|
||||
ONNX_CONTRIB_OPERATOR_SCHEMA(Inverse)
|
||||
.SetDomain(kMSDomain) // kMSDomain = "com.microsoft"
|
||||
.SinceVersion(1) // Same version used at op (symbolic) registration
|
||||
...
|
||||
```
|
||||
|
||||
To comply with ONNX guideline for new operators, a new operator should have complete reference implementation tests and shape inference tests.
|
||||
|
||||
Reference implementation python tests should be added in:
|
||||
https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/test/python/contrib_ops
|
||||
E.g.: https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/test/python/contrib_ops/onnx_test_trilu.py
|
||||
|
||||
Shape inference C++ tests should be added in:
|
||||
https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/test/contrib_ops
|
||||
E.g.: https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/test/contrib_ops/trilu_shape_inference_test.cc
|
||||
|
||||
The operator kernel should be implemented using ```Compute``` function
|
||||
under contrib namespace in `https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/contrib_ops/cpu/<operator>.cc`
|
||||
for CPU and `https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/contrib_ops/cuda/<operator>.cc` for CUDA.
|
||||
|
||||
```c++
|
||||
namespace onnxruntime {
|
||||
namespace contrib {
|
||||
|
||||
class Inverse final : public OpKernel {
|
||||
public:
|
||||
explicit Inverse(const OpKernelInfo& info) : OpKernel(info) {}
|
||||
Status Compute(OpKernelContext* ctx) const override;
|
||||
|
||||
private:
|
||||
...
|
||||
};
|
||||
|
||||
ONNX_OPERATOR_KERNEL_EX(
|
||||
Inverse,
|
||||
kMSDomain,
|
||||
1,
|
||||
kCpuExecutionProvider,
|
||||
KernelDefBuilder()
|
||||
.TypeConstraint("T", BuildKernelDefConstraints<float, double, MLFloat16>()),
|
||||
Inverse);
|
||||
|
||||
Status Inverse::Compute(OpKernelContext* ctx) const {
|
||||
... // kernel implementation
|
||||
}
|
||||
|
||||
} // namespace contrib
|
||||
} // namespace onnxruntime
|
||||
```
|
||||
|
||||
Operator kernel should be registered in https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/contrib_ops/cpu_contrib_kernels.cc for CPU and https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/contrib_ops/cuda_contrib_kernels.cc for CUDA.
|
||||
|
||||
Now you should be able to build and install ONNX Runtime to start using your custom op.
|
||||
|
||||
### ONNX Runtime Tests
|
||||
|
||||
ONNX Runtime custom op kernel tests should be added in: https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/test/contrib_ops/<operator>_test.cc
|
||||
|
||||
```c++
|
||||
namespace onnxruntime {
|
||||
namespace test {
|
||||
|
||||
// Add a comprehensive set of unit tests for custom op kernel implementation
|
||||
|
||||
TEST(InverseContribOpTest, two_by_two_float) {
|
||||
OpTester test("Inverse", 1, kMSDomain); // custom opset version and domain
|
||||
test.AddInput<float>("X", {2, 2}, {4, 7, 2, 6});
|
||||
test.AddOutput<float>("Y", {2, 2}, {0.6f, -0.7f, -0.2f, 0.4f});
|
||||
test.Run();
|
||||
}
|
||||
|
||||
...
|
||||
|
||||
} // namespace test
|
||||
} // namespace onnxruntime
|
||||
```
|
||||
|
||||
### Test model Export End to End
|
||||
|
||||
Once the custom op is registered in the exporter and implemented in ONNX Runtime, you should be able to export it as part of you ONNX model and run it with ONNX Runtime.
|
||||
|
||||
Below you can find a sample script for exporting and running the inverse operator as part of a model.
|
||||
|
||||
The exported model includes a combination of ONNX standard ops and the custom ops.
|
||||
|
||||
This test also compares the output of PyTorch model with ONNX Runtime outputs to test both the operator export and implementation.
|
||||
|
||||
```python
|
||||
import torch
|
||||
import onnxruntime
|
||||
import io
|
||||
import numpy
|
||||
|
||||
|
||||
class CustomInverse(torch.nn.Module):
|
||||
def forward(self, x):
|
||||
return torch.inverse(x) + x
|
||||
|
||||
x = torch.randn(3, 3)
|
||||
|
||||
# Export model to ONNX
|
||||
f = io.BytesIO()
|
||||
torch.onnx.export(CustomInverse(), (x,), f)
|
||||
|
||||
model = CustomInverse()
|
||||
pt_outputs = model(x)
|
||||
|
||||
# Run the exported model with ONNX Runtime
|
||||
ort_sess = onnxruntime.InferenceSession(f.getvalue())
|
||||
ort_inputs = dict((ort_sess.get_inputs()[i].name, input.cpu().numpy()) for i, input in enumerate((x,)))
|
||||
ort_outputs = ort_sess.run(None, ort_inputs)
|
||||
|
||||
# Validate PyTorch and ONNX Runtime results
|
||||
numpy.testing.assert_allclose(pt_outputs.cpu().numpy(), ort_outputs[0], rtol=1e-03, atol=1e-05)
|
||||
```
|
||||
|
||||
By default, the opset version will be set to ``1`` for custom opsets. If you'd like to export your
|
||||
custom op to a higher opset version, you can specify the custom opset domain and version using
|
||||
the ``custom_opsets argument`` when calling the export API. Note that this is different than the opset
|
||||
version associated with default ```ONNX``` domain.
|
||||
|
||||
```python
|
||||
torch.onnx.export(CustomInverse(), (x,), f, custom_opsets={"com.microsoft": 5})
|
||||
```
|
||||
|
||||
Note that you can export a custom op to any version >= the opset version used at registration.
|
||||
|
||||
We have a set of tests for export and output validation of ONNX models with ONNX Runtime custom ops in
|
||||
``tools/test/test_test_custom_ops_pytorch_exporter.py``. If you're adding a new custom operator, please
|
||||
make sure to include tests in this file.
|
||||
|
||||
You can run these tests using the command:
|
||||
|
||||
```bash
|
||||
PYTHONPATH=<path_to_onnxruntime/tools> pytest -v test_custom_ops_pytorch_exporter.py
|
||||
```
|
||||
8
docs/how-to/index.md
Normal file
8
docs/how-to/index.md
Normal file
|
|
@ -0,0 +1,8 @@
|
|||
---
|
||||
title: Recipes
|
||||
has_children: true
|
||||
nav_order: 3
|
||||
---
|
||||
|
||||
# ONNX Runtime Recipes
|
||||
{: .no_toc }
|
||||
203
docs/how-to/tune-performance.md
Normal file
203
docs/how-to/tune-performance.md
Normal file
|
|
@ -0,0 +1,203 @@
|
|||
---
|
||||
title: Tune performance
|
||||
parent: Recipes
|
||||
nav_order: 1
|
||||
---
|
||||
|
||||
# ONNX Runtime Performance Tuning
|
||||
{: .no_toc }
|
||||
|
||||
ONNX Runtime gives high performance across a range of hardware options by providing "Execution Providers" to interface to different execution environments. See: [design overview](../resources/high-level-design.md), [supported execution providers](../reference/execution-providers).
|
||||
|
||||
Along with this flexibility comes decisions for tuning and usage. For each model running with each execution provider, there are settings that can be tuned (e.g. thread number, wait policy, etc) to improve performance.
|
||||
|
||||
This document covers basic tools and knobs that can be leveraged to find the best performance for your model and hardware.
|
||||
|
||||
## Contents
|
||||
{: .no_toc }
|
||||
|
||||
* TOC placeholder
|
||||
{:toc}
|
||||
|
||||
## Performance Tuning Tools
|
||||
|
||||
The [ONNX Go Live "OLive" tool](https://github.com/microsoft/OLive) is an easy-to-use pipeline for converting models to ONNX and optimizing performance with ONNX Runtime. The tool can help identify the optimal runtime configuration to get the best performance on the target hardware for the model.
|
||||
|
||||
As a quickstart, please see the notebooks: [Python](https://github.com/microsoft/OLive/blob/master/notebook/Convert_Models_and_Tune_Performance_with_OLive_Python_SDK.ipynb), [Docker images](https://github.com/microsoft/OLive/blob/master/notebook/Convert_Models_and_Tune_Performance_with_OLive_Docker_Images.ipynb)
|
||||
|
||||
### Profiling and Performance Report
|
||||
|
||||
The onnxruntime_perf_test.exe tool (available from the build drop) can be used to test various knobs. Please find the usage instructions using `onnxruntime_perf_test.exe -h`.
|
||||
|
||||
You can enable ONNX Runtime latency profiling in code:
|
||||
|
||||
```python
|
||||
import onnxruntime as rt
|
||||
|
||||
sess_options = rt.SessionOptions()
|
||||
sess_options.enable_profiling = True
|
||||
```
|
||||
|
||||
If you are using the onnxruntime_perf_test.exe tool, you can add `-p [profile_file]` to enable performance profiling.
|
||||
|
||||
In both cases, you will get a JSON file which contains the detailed performance data (threading, latency of each operator, etc). This file is a standard performance tracing file, and to view it in a user friendly way, you can open it by using chrome://tracing:
|
||||
|
||||
* Open chrome browser
|
||||
* Type chrome://tracing in the address bar
|
||||
* Load the generated JSON file
|
||||
|
||||
## Using different Execution Providers
|
||||
|
||||
To learn more about different Execution Providers, see [docs/exeuction_providers](../reference/execution-providers).
|
||||
|
||||
### Python API
|
||||
|
||||
Official Python packages on Pypi only support the default CPU (MLAS) and default GPU (CUDA) execution providers. For other execution providers, you need to build from source. Please refer to the [build instructions](../how-to/build.md). The recommended instructions build the wheel with debug info in parallel.
|
||||
|
||||
For example:
|
||||
|
||||
`DNNL: ./build.sh --config RelWithDebInfo --use_dnnl --build_wheel --parallel`
|
||||
|
||||
` CUDA: ./build.sh --config RelWithDebInfo --use_cuda --build_wheel --parallel`
|
||||
|
||||
### C and C# API
|
||||
|
||||
Official release (nuget package) supports default (MLAS) and MKL-ML for CPU, and CUDA for GPU. For other execution providers, you need to build from source. Append `--build_csharp` to the instructions to build both C# and C packages.
|
||||
|
||||
For example:
|
||||
|
||||
`DNNL: ./build.sh --config RelWithDebInfo --use_dnnl --build_csharp --parallel`
|
||||
|
||||
`CUDA: ./build.sh --config RelWithDebInfo --use_cuda --build_csharp --parallel`
|
||||
|
||||
In order to use DNNL, nGraph, CUDA, or TensorRT execution provider, you need to call the C API OrtSessionOptionsAppendExecutionProvider. Here is an example for the CUDA execution provider:
|
||||
|
||||
C API Example:
|
||||
|
||||
```c
|
||||
const OrtApi* g_ort = OrtGetApi(ORT_API_VERSION);
|
||||
OrtEnv* env;
|
||||
g_ort->CreateEnv(ORT_LOGGING_LEVEL_WARNING, "test", &env)
|
||||
OrtSessionOptions* session_option;
|
||||
g_ort->OrtCreateSessionOptions(&session_options);
|
||||
g_ort->OrtSessionOptionsAppendExecutionProvider_CUDA(sessionOptions, 0);
|
||||
OrtSession* session;
|
||||
g_ort->CreateSession(env, model_path, session_option, &session);
|
||||
```
|
||||
|
||||
C# API Example:
|
||||
|
||||
```c#
|
||||
SessionOptions so = new SessionOptions();
|
||||
so.GraphOptimizationLevel = GraphOptimizationLevel.ORT_ENABLE_EXTENDED;
|
||||
so.AppendExecutionProvider_CUDA(0);
|
||||
var session = new InferenceSession(modelPath, so);
|
||||
```
|
||||
|
||||
Python API Example:
|
||||
|
||||
```python
|
||||
import onnxruntime as rt
|
||||
|
||||
so = rt.SessionOptions()
|
||||
so.graph_optimization_level = rt.GraphOptimizationLevel.ORT_ENABLE_ALL
|
||||
session = rt.InferenceSession(model, sess_options=so)
|
||||
session.set_providers(['CUDAExecutionProvider'])
|
||||
```
|
||||
|
||||
## Which Execution Provider will provide the best performance?
|
||||
|
||||
Performance is dependent on the specific model you're trying to run, the session and run options you've selected, and of course, your specific hardware target. Below you'll find some more information that may be helpful to select the right Execution Provider.
|
||||
|
||||
### CUDA (Default GPU) or CPU?
|
||||
|
||||
The CPU version of ONNX Runtime provides a complete implementation of all operators in the ONNX spec. This ensures that your ONNX-compliant model can execute successfully. In order to keep the binary size small, common data types are supported for the ops. If you are using an uncommon data type that is not supported, you can file an issue and/or contribute a PR (see examples - [PR #2112](https://github.com/microsoft/onnxruntime/pull/2112), [PR #2034](https://github.com/microsoft/onnxruntime/pull/2034), [PR #1565](https://github.com/microsoft/onnxruntime/pull/1565)). Please make sure you provide details on usage justification.
|
||||
|
||||
Additionally, not all CUDA kernels are implemented, as these have been prioritized on an as-needed basis. This means that if your model contains operators that do not have a CUDA implementation, it will fall back to CPU. Switching between CPU and GPU can cause significant performance impact. If you require a specific operator that is not currently supported, please consider [contributing](https://github.com/microsoft/onnxruntime/tree/master/CONTRIBUTING.md) and/or [file an issue](https://github.com/microsoft/onnxruntime/issues) clearly describing your use case and share your model if possible.
|
||||
|
||||
### TensorRT or CUDA?
|
||||
|
||||
TensorRT and CUDA are separate execution providers for ONNX Runtime. On the same hardware, TensorRT will generally provide better performance; however, this depends on the specific model and whether the operators in the model can be supported by TensorRT. In cases where TensorRT cannot handle the subgraph(s), it will fall back to CUDA. Note that the TensorRT EP may depend on a different version of CUDA than the CUDA EP.
|
||||
|
||||
### TensorRT/CUDA or DirectML?
|
||||
|
||||
DirectML is the hardware-accelerated DirectX 12 library for machine learning on Windows and supports all DirectX 12 capable devices (Nvidia, Intel, AMD). This means that if you are targeting Windows GPUs, using the DirectML Execution Provider is likely your best bet. This can be used with both the ONNX Runtime as well as [WinML APIs](../reference/api/winrt-api.md).
|
||||
|
||||
## Tuning performance for specific Execution Providers
|
||||
|
||||
### Thread management
|
||||
|
||||
* If ORT is built with OpenMP, use the OpenMP env variable to control the number of intra op num threads.
|
||||
* If ORT is not built with OpenMP, use the appropriate ORT API to control intra op num threads.
|
||||
* Inter op num threads (used only when parallel execution is enabled) is not affected by OpenMP settings and should
|
||||
always be set using the ORT APIs.
|
||||
|
||||
### Default CPU Execution Provider (MLAS)
|
||||
|
||||
The default execution provider uses different knobs to control the thread number.
|
||||
|
||||
For the default CPU execution provider, you can try following knobs in the Python API:
|
||||
|
||||
```python
|
||||
import onnxruntime as rt
|
||||
|
||||
sess_options = rt.SessionOptions()
|
||||
|
||||
sess_options.intra_op_num_threads = 2
|
||||
sess_options.execution_mode = rt.ExecutionMode.ORT_SEQUENTIAL
|
||||
sess_options.graph_optimization_level = rt.GraphOptimizationLevel.ORT_ENABLE_ALL
|
||||
```
|
||||
|
||||
* Thread Count
|
||||
* `sess_options.intra_op_num_threads = 2` controls the number of threads to use to run the model
|
||||
* Sequential vs Parallel Execution
|
||||
* `sess_options.execution_mode = rt.ExecutionMode.ORT_SEQUENTIAL` controls whether the operators in the graph run sequentially or in parallel. Usually when a model has many branches, setting this option to false will provide better performance.
|
||||
* When `sess_options.execution_mode = rt.ExecutionMode.ORT_PARALLEL`, you can set `sess_options.inter_op_num_threads` to control the
|
||||
number of threads used to parallelize the execution of the graph (across nodes).
|
||||
|
||||
* sess_options.graph_optimization_level = rt.GraphOptimizationLevel.ORT_ENABLE_ALL. Default is already ORT_ENABLE_ALL(99). Please see [onnxruntime_c_api.h](https://github.com/microsoft/onnxruntime/tree/master/include/onnxruntime/core/session/onnxruntime_c_api.h#L241) (enum GraphOptimizationLevel) for the full list of all optimization levels. For details regarding available optimizations and usage please refer to the [Graph Optimizations Doc](../resources/graph-optimizations.md).
|
||||
|
||||
### MKL_DNN/nGraph/MKL_ML Execution Provider
|
||||
|
||||
MKL_DNN, MKL_ML and nGraph all depends on openmp for parallelization. For those execution providers, we need to use the openmp environment variable to tune the performance.
|
||||
|
||||
The most widely used environment variables are:
|
||||
|
||||
* OMP_NUM_THREADS=n
|
||||
* Controls the thread pool size
|
||||
|
||||
* OMP_WAIT_POLICY=PASSIVE/ACTIVE
|
||||
* Controls whether thread spinning is enabled
|
||||
* PASSIVE is also called throughput mode and will yield CPU after finishing current task
|
||||
* ACTIVE will not yield CPU, instead it will have a while loop to check whether the next task is ready
|
||||
* Use PASSIVE if your CPU usage already high, and use ACTIVE when you want to trade CPU with latency
|
||||
|
||||
## Troubleshooting model performance issues
|
||||
|
||||
The answers below are troubleshooting suggestions based on common previous user-filed issues and questions. This list is by no means exhaustive and there is a lot of case-by-case fluctuation depending on the model and specific usage scenario. Please use this information to guide your troubleshooting, search through previously filed issues for related topics, and/or file a new issue if your problem is still not resolved.
|
||||
|
||||
### Performance Troubleshooting Checklist
|
||||
|
||||
Here is a list of things to check through when assessing performance issues.
|
||||
* Are you using OpenMP? OpenMP will parallelize some of the code for potential performance improvements. This is not recommended for running on single threads.
|
||||
* Have you enabled all [graph optimizations](../resources/graph-optimizations.md)? The official published packages do enable all by default, but when building from source, check that these are enabled in your build.
|
||||
* Have you searched through prior filed [Github issues](https://github.com/microsoft/onnxruntime/issues) to see if your problem has been discussed previously? Please do this before filing new issues.
|
||||
* If using CUDA or TensorRT, do you have the right versions of the dependent libraries installed?
|
||||
|
||||
### I need help performance tuning for BERT models
|
||||
|
||||
For BERT models, sometimes ONNX Runtime cannot apply the best optimization due to reasons such as framework version updates. We recommend trying out the [BERT optimization tool](https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/python/tools/transformers), which reflects the latest changes in graph pattern matching and model conversions, and a set of [notebooks](https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/python/tools/transformers/notebooks) to help get started.
|
||||
|
||||
### Why is the model graph not optimized even with graph_optimization_level set to ORT_ENABLE_ALL?
|
||||
|
||||
The ONNX model from IR_VERSION 4 only treats initializers that appear in graph input as non-constant. This may fail some of the graph optimizations, like const folding, operator fusion and etc. Move initializers out of graph inputs if there is no need to override them, by either re-generating the model with latest exporter/converter or with the tool [remove_initializer_from_input.py](https://github.com/microsoft/onnxruntime/tree/master/tools/python/remove_initializer_from_input.py).
|
||||
|
||||
### Why is my model running slower on GPU than CPU?
|
||||
|
||||
Depending on which execution provider you're using, it may not have full support for all the operators in your model. Fallback to CPU ops can cause hits in performance speed. Moreover even if an op is implemented by the CUDA execution provider, it may not necessarily assign/place the op to the CUDA EP due to performance reasons. To see the placement decided by ORT, turn on verbose logging and look at the console output.
|
||||
|
||||
### My converted Tensorflow model is slow - why?
|
||||
|
||||
NCHW and NHWC are two different memory layout for 4-D tensors.
|
||||
|
||||
Most TensorFlow operations used by a CNN support both NHWC and NCHW data format. The Tensorflow team suggests that on GPU NCHW is faster but on CPU NHWC is sometimes faster in Tensorflow. However, ONNX only supports NCHW. As a result, if the original model is in NHWC format, when the model is converted extra transposes may be added. The [tensorflow-onnx](https://github.com/onnx/tensorflow-onnx) and [keras-onnx](https://github.com/onnx/keras-onnx) converters do remove many of these transposes, but if this doesn't help sufficiently, consider retraining the model using NCHW.
|
||||
21
docs/index.md
Normal file
21
docs/index.md
Normal file
|
|
@ -0,0 +1,21 @@
|
|||
---
|
||||
title: Home
|
||||
nav_order: 0
|
||||
nav_exclude: true
|
||||
---
|
||||
|
||||
# Welcome to ONNX Runtime!
|
||||
|
||||
**ONNX Runtime** is a cross-platform **inferencing and training accelerator** compatible with popular ML/DNN frameworks, including PyTorch, TensorFlow/Keras, scikit-learn, and more.
|
||||
|
||||
You can benefit from ONNX Runtime if you want to:
|
||||
|
||||
* Improve inference performance for a wide variety of ML models
|
||||
* Reduce time and cost of training large models
|
||||
* Train in Python but deploy into a C#/C++/Java app
|
||||
* Run on different hardware and operating systems
|
||||
* Train and perform inference with models created in different frameworks
|
||||
|
||||
[ONNX Runtime inference](get-started/inference.md) APIs are stable and production-ready since the [1.0 release](https://github.com/microsoft/onnxruntime/releases/tag/v1.0.0) in October 2019 and can enable faster customer experiences and lower costs.
|
||||
|
||||
[ONNX Runtime training](get-started/training.md) feature was introduced in May 2020 in preview. This feature supports acceleration of PyTorch training on multi-node NVIDIA GPUs for transformer models. Additional updates for this feature are coming soon.
|
||||
82
docs/reference/api/c-api.md
Normal file
82
docs/reference/api/c-api.md
Normal file
|
|
@ -0,0 +1,82 @@
|
|||
---
|
||||
title: C API
|
||||
parent: API docs
|
||||
grand_parent: Reference
|
||||
nav_order: 1
|
||||
---
|
||||
|
||||
# ONNX Runtime C API
|
||||
{: .no_toc }
|
||||
|
||||
## Contents
|
||||
{: .no_toc }
|
||||
|
||||
* TOC placeholder
|
||||
{:toc}
|
||||
|
||||
## Features
|
||||
|
||||
* Creating an InferenceSession from an on-disk model file and a set of SessionOptions.
|
||||
* Registering customized loggers.
|
||||
* Registering customized allocators.
|
||||
* Registering predefined providers and set the priority order. ONNXRuntime has a set of predefined execution providers, like CUDA, DNNL. User can register providers to their InferenceSession. The order of registration indicates the preference order as well.
|
||||
* Running a model with inputs. These inputs must be in CPU memory, not GPU. If the model has multiple outputs, user can specify which outputs they want.
|
||||
* Converting an in-memory ONNX Tensor encoded in protobuf format to a pointer that can be used as model input.
|
||||
* Setting the thread pool size for each session.
|
||||
* Setting graph optimization level for each session.
|
||||
* Dynamically loading custom ops. [Instructions](../../how-to/add-custom-op.md)
|
||||
* Ability to load a model from a byte array. See ```OrtCreateSessionFromArray``` in [onnxruntime_c_api.h](https://github.com/microsoft/onnxruntime/blob/master/include/onnxruntime/core/session/onnxruntime_c_api.h).
|
||||
* **Global/shared threadpools:** By default each session creates its own set of threadpools. In situations where multiple
|
||||
sessions need to be created (to infer different models) in the same process, you end up with several threadpools created
|
||||
by each session. In order to address this inefficiency we introduce a new feature called global/shared threadpools.
|
||||
The basic idea here is to share a set of global threadpools across multiple sessions. Typical usage of this feature
|
||||
is as follows
|
||||
* Populate ```ThreadingOptions```. Use the value of 0 for ORT to pick the defaults.
|
||||
* Create env using ```CreateEnvWithGlobalThreadPools()```
|
||||
* Create session and call ```DisablePerSessionThreads()``` on the session options object
|
||||
* Call ```Run()``` as usual
|
||||
* **Share allocator(s) between sessions:** Allow multiple sessions in the same process to use the same allocator(s). This
|
||||
+allocator is first registered in the env and then reused by all sessions that use the same env instance unless a session
|
||||
+chooses to override this by setting ```session_state.use_env_allocators``` to "0". Usage of this feature is as follows
|
||||
* Register an allocator created by ORT using the ```CreateAndRegisterAllocator``` API.
|
||||
* Set ```session.use_env_allocators``` to "1" for each session that wants to use the env registered allocators.
|
||||
* See test ```TestSharedAllocatorUsingCreateAndRegisterAllocator``` in
|
||||
onnxruntime/test/shared_lib/test_inference.cc for an example.
|
||||
* **Share initializer(s) between sessions:**
|
||||
* *Description*: This feature allows a user to share the same instance of an initializer across
|
||||
+multiple sessions.
|
||||
* *Scenario*: You've several models that use the same set of initializers except the last few layers of the model and you load these models in the same process. When every model (session) creates a separate instance of the same initializer, it leads to excessive and wasteful memory usage since in this case it's the same initializer. You want to optimize memory usage while having the flexibility to allocate the initializers (possibly even store them in shared memory).
|
||||
* *Example Usage*: Use the ```AddInitializer``` API to add a pre-allocated initializer to session options before calling ```CreateSession```. Use the same instance of session options to create several sessions allowing the initializer(s) to be shared between the sessions. See [C API sample usage (TestSharingOfInitializer)](../onnxruntime/test/shared_lib/test_inference.cc) and [C# API sample usage (TestWeightSharingBetweenSessions)](../csharp/test/Microsoft.ML.OnnxRuntime.Tests/InferenceTest.cs).
|
||||
|
||||
## Usage Overview
|
||||
|
||||
1. Include [onnxruntime_c_api.h](/include/onnxruntime/core/session/onnxruntime_c_api.h).
|
||||
2. Call OrtCreateEnv
|
||||
3. Create Session: OrtCreateSession(env, model_uri, nullptr,...)
|
||||
- Optionally add more execution providers (e.g. for CUDA use OrtSessionOptionsAppendExecutionProvider_CUDA)
|
||||
4. Create Tensor
|
||||
1) OrtCreateMemoryInfo
|
||||
2) OrtCreateTensorWithDataAsOrtValue
|
||||
5. OrtRun
|
||||
|
||||
## Sample code
|
||||
|
||||
The example below shows a sample run using the SqueezeNet model from ONNX model zoo, including dynamically reading model inputs, outputs, shape and type information, as well as running a sample vector and fetching the resulting class probabilities for inspection.
|
||||
|
||||
* [C_Api_Sample.cpp](https://github.com/microsoft/onnxruntime/blob/master/csharp/test/Microsoft.ML.OnnxRuntime.EndToEndTests.Capi/C_Api_Sample.cpp)
|
||||
|
||||
## Deployment
|
||||
|
||||
### Windows 10
|
||||
|
||||
Your installer should put the onnxruntime.dll into the same folder as your application. Your application can either use [load-time dynamic linking](https://docs.microsoft.com/en-us/windows/win32/dlls/using-load-time-dynamic-linking) or [run-time dynamic linking](https://docs.microsoft.com/en-us/windows/win32/dlls/using-run-time-dynamic-linking) to bind to the dll.
|
||||
|
||||
#### Dynamic Link Library Search Order
|
||||
|
||||
This is an important article on how Windows finds supporting dlls: [Dynamic Link Library Search Order](https://docs.microsoft.com/en-us/windows/win32/dlls/dynamic-link-library-search-order).
|
||||
|
||||
There are some cases where the app is not directly consuming the onnxruntime but instead calling into a DLL that is consuming the onnxruntime. People building these DLLs that consume the onnxruntime need to take care about folder structures. Do not modify the system %path% variable to add your folders. This can conflict with other software on the machine that is also using the onnxruntme. Instead place your DLL and the onnxruntime DLL in the same folder and use [run-time dynamic linking](https://docs.microsoft.com/en-us/windows/win32/dlls/using-run-time-dynamic-linking) to bind explicitly to that copy. You can use code like this sample does in [GetModulePath()](https://github.com/microsoft/Windows-Machine-Learning/blob/master/Samples/SampleSharedLib/SampleSharedLib/FileHelper.cpp) to find out what folder your dll is loaded from.
|
||||
|
||||
## Telemetry
|
||||
|
||||
To turn on/off telemetry collection on official Windows builds, please use Enable/DisableTelemetryEvents() in the C API. See the [Privacy](https://github.com/microsoft/onnxruntime/blob/master/docs/Privacy.md) page for more information on telemetry collection and Microsoft's privacy policy.
|
||||
257
docs/reference/api/csharp-api.md
Normal file
257
docs/reference/api/csharp-api.md
Normal file
|
|
@ -0,0 +1,257 @@
|
|||
---
|
||||
title: C# API
|
||||
parent: API docs
|
||||
grand_parent: Reference
|
||||
nav_order: 2
|
||||
---
|
||||
|
||||
# ONNX Runtime C# API
|
||||
{: .no_toc }
|
||||
|
||||
The ONNX runtime provides a C# .Net binding for running inference on ONNX models in any of the .Net standard platforms. The API is .Net standard 1.1 compliant for maximum portability. This document describes the API.
|
||||
|
||||
## Contents
|
||||
{: .no_toc }
|
||||
|
||||
* TOC placeholder
|
||||
{:toc}
|
||||
|
||||
## NuGet Package
|
||||
|
||||
The Microsoft.ML.OnnxRuntime Nuget package includes the precompiled binaries for ONNX runtime, and includes libraries for Windows and Linux platforms with X64 CPUs. The APIs conform to .Net Standard 1.1.
|
||||
|
||||
## Sample Code
|
||||
|
||||
The unit tests contain several examples of loading models, inspecting input/output node shapes and types, as well as constructing tensors for scoring.
|
||||
|
||||
* [Microsoft.ML.OnnxRuntime.Tests/InferenceTest.cs](https://github.com/microsoft/onnxruntime/tree/master/csharp/test/Microsoft.ML.OnnxRuntime.Tests/InferenceTest.cs#L166)
|
||||
|
||||
## Getting Started
|
||||
Here is simple tutorial for getting started with running inference on an existing ONNX model for a given input data. The model is typically trained using any of the well-known training frameworks and exported into the ONNX format. To start scoring using the model, open a session using the `InferenceSession` class, passing in the file path to the model as a parameter.
|
||||
|
||||
```cs
|
||||
var session = new InferenceSession("model.onnx");
|
||||
```
|
||||
|
||||
Once a session is created, you can execute queries using the `Run` method of the `InferenceSession` object. Currently, only `Tensor` type of input and outputs are supported. The results of the `Run` method are represented as a collection of .Net `Tensor` objects (as defined in [System.Numerics.Tensor](https://www.nuget.org/packages/System.Numerics.Tensors)).
|
||||
|
||||
```cs
|
||||
Tensor<float> t1, t2; // let's say data is fed into the Tensor objects
|
||||
var inputs = new List<NamedOnnxValue>()
|
||||
{
|
||||
NamedOnnxValue.CreateFromTensor<float>("name1", t1),
|
||||
NamedOnnxValue.CreateFromTensor<float>("name2", t2)
|
||||
};
|
||||
using (var results = session.Run(inputs))
|
||||
{
|
||||
// manipulate the results
|
||||
}
|
||||
```
|
||||
|
||||
You can load your input data into Tensor<T> objects in several ways. A simple example is to create the Tensor from arrays.
|
||||
|
||||
```cs
|
||||
float[] sourceData; // assume your data is loaded into a flat float array
|
||||
int[] dimensions; // and the dimensions of the input is stored here
|
||||
Tensor<float> t1 = new DenseTensor<float>(sourceData, dimensions);
|
||||
```
|
||||
|
||||
Here is a [complete sample code](../csharp/sample/Microsoft.ML.OnnxRuntime.InferenceSample) that runs inference on a pretrained model.
|
||||
|
||||
## Reuse input/output tensor buffers
|
||||
|
||||
In some scenarios, you may want to reuse input/output tensors. This often happens when you want to chain 2 models (ie. feed one's output as input to another), or want to accelerate inference speed during multiple inference runs.
|
||||
|
||||
### Chaining: Feed model A's output(s) as input(s) to model B
|
||||
|
||||
```cs
|
||||
InferenceSession session1, session2; // let's say 2 sessions are initialized
|
||||
|
||||
Tensor<float> t1; // let's say data is fed into the Tensor objects
|
||||
var inputs1 = new List<NamedOnnxValue>()
|
||||
{
|
||||
NamedOnnxValue.CreateFromTensor<float>("name1", t1)
|
||||
};
|
||||
// session1 inference
|
||||
using (var outputs1 = session1.Run(inputs1))
|
||||
{
|
||||
// get intermediate value
|
||||
var input2 = outputs1.First();
|
||||
|
||||
// modify the name of the ONNX value
|
||||
input2.Name = "name2";
|
||||
|
||||
// create input list for session2
|
||||
var inputs2 = new List<NamedOnnxValue>() { input2 };
|
||||
|
||||
// session2 inference
|
||||
using (var results = session2.Run(inputs2))
|
||||
{
|
||||
// manipulate the results
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Multiple inference runs with fixed sized input(s) and output(s)
|
||||
|
||||
If the model have fixed sized inputs and outputs of numeric tensors, you can use "FixedBufferOnnxValue" to accelerate the inference speed. By using "FixedBufferOnnxValue", the container objects only need to be allocated/disposed one time during multiple InferenceSession.Run() calls. This avoids some overhead which may be beneficial for smaller models where the time is noticeable in the overall running time.
|
||||
|
||||
An example can be found at `TestReusingFixedBufferOnnxValueNonStringTypeMultiInferences()`:
|
||||
* [Microsoft.ML.OnnxRuntime.Tests/InferenceTest.cs#L1047](https://github.com/microsoft/onnxruntime/tree/master/csharp/test/Microsoft.ML.OnnxRuntime.Tests/InferenceTest.cs#L1047)
|
||||
|
||||
## Running on GPU (Optional)
|
||||
|
||||
If using the GPU package, simply use the appropriate SessionOptions when creating an InferenceSession.
|
||||
|
||||
```cs
|
||||
int gpuDeviceId = 0; // The GPU device ID to execute on
|
||||
var session = new InferenceSession("model.onnx", SessionOptions.MakeSessionOptionWithCudaProvider(gpuDeviceId));
|
||||
```
|
||||
|
||||
## API Reference
|
||||
### InferenceSession
|
||||
```cs
|
||||
class InferenceSession: IDisposable
|
||||
```
|
||||
The runtime representation of an ONNX model
|
||||
|
||||
#### Constructor
|
||||
```cs
|
||||
InferenceSession(string modelPath);
|
||||
InferenceSession(string modelPath, SessionOptions options);
|
||||
```
|
||||
|
||||
#### Properties
|
||||
```cs
|
||||
IReadOnlyDictionary<NodeMetadata> InputMetadata;
|
||||
```
|
||||
Data types and shapes of the input nodes of the model.
|
||||
|
||||
```cs
|
||||
IReadOnlyDictionary<NodeMetadata> OutputMetadata;
|
||||
```
|
||||
Data types and shapes of the output nodes of the model.
|
||||
|
||||
#### Methods
|
||||
```cs
|
||||
IDisposableReadOnlyCollection<DisposableNamedOnnxValue> Run(IReadOnlyCollection<NamedOnnxValue> inputs);
|
||||
```
|
||||
Runs the model with the given input data to compute all the output nodes and returns the output node values. Both input and output are collection of NamedOnnxValue, which in turn is a name-value pair of string names and Tensor values. The outputs are IDisposable variant of NamedOnnxValue, since they wrap some unmanaged objects.
|
||||
|
||||
```cs
|
||||
IDisposableReadOnlyCollection<DisposableNamedOnnxValue> Run(IReadOnlyCollection<NamedOnnxValue> inputs, IReadOnlyCollection<string> desiredOutputNodes);
|
||||
```
|
||||
Runs the model on given inputs for the given output nodes only.
|
||||
|
||||
### System.Numerics.Tensor
|
||||
The primary .Net object that is used for holding input-output of the model inference. Details on this newly introduced data type can be found in its [open-source implementation](https://github.com/dotnet/corefx/tree/master/src/System.Numerics.Tensors). The binaries are available as a [.Net NuGet package](https://www.nuget.org/packages/System.Numerics.Tensors).
|
||||
|
||||
### NamedOnnxValue
|
||||
```cs
|
||||
class NamedOnnxValue;
|
||||
```
|
||||
Represents a name-value pair of string names and any type of value that ONNX runtime supports as input-output data. Currently, only Tensor objects are supported as input-output values.
|
||||
|
||||
#### Constructor
|
||||
No public constructor available.
|
||||
|
||||
#### Properties
|
||||
```cs
|
||||
string Name; // get or set the name
|
||||
```
|
||||
|
||||
#### Methods
|
||||
```cs
|
||||
static NamedOnnxValue CreateFromTensor<T>(string name, Tensor<T>);
|
||||
```
|
||||
Creates a NamedOnnxValue from a name and a Tensor<T> object.
|
||||
|
||||
```cs
|
||||
Tensor<T> AsTensor<T>();
|
||||
```
|
||||
Accesses the value as a Tensor<T>. Returns null if the value is not a Tensor<T>.
|
||||
|
||||
### DisposableNamedOnnxValue
|
||||
```cs
|
||||
class DisposableNamedOnnxValue: NamedOnnxValue, IDisposable;
|
||||
```
|
||||
This is a disposable variant of NamedOnnxValue, used for holding output values which contains objects allocated in unmanaged memory.
|
||||
|
||||
### FixedBufferOnnxValue
|
||||
```cs
|
||||
class FixedBufferOnnxValue: IDisposable;
|
||||
```
|
||||
Class `FixedBufferOnnxValue` enables the availability to pin the tensor buffer. This helps to minimize overhead within each inference run.
|
||||
|
||||
`FixedBufferOnnxValue` can be used as either input or output. However, if used as output, it has to be a numeric tensor.
|
||||
|
||||
`FixedBufferOnnxValue` implements `IDisposable`, so make sure it get disposed after use.
|
||||
#### Methods
|
||||
```cs
|
||||
static FixedBufferOnnxValue CreateFromTensor<T>(Tensor<T>);
|
||||
```
|
||||
Creates a FixedBufferOnnxValue from a name and a Tensor<T> object.
|
||||
|
||||
|
||||
### IDisposableReadOnlyCollection
|
||||
```cs
|
||||
interface IDisposableReadOnlyCollection: IReadOnlyCollection, IDisposable
|
||||
```
|
||||
Collection interface to hold disposable values. Used for output of Run method.
|
||||
|
||||
### SessionOptions
|
||||
```cs
|
||||
class SessionOptions: IDisposable;
|
||||
```
|
||||
A collection of properties to be set for configuring the OnnxRuntime session
|
||||
|
||||
#### Constructor
|
||||
```cs
|
||||
SessionOptions();
|
||||
```
|
||||
Constructs a SessionOptions will all options at default/unset values.
|
||||
|
||||
#### Properties
|
||||
```cs
|
||||
static SessionOptions Default; //read-only
|
||||
```
|
||||
Accessor to the default static option object
|
||||
|
||||
#### Methods
|
||||
```cs
|
||||
SetSessionGraphOptimizationLevel(GraphOptimizationLevel graph_transformer_level);
|
||||
```
|
||||
See [ONNX_Runtime_Graph_Optimizations.md] for more details.
|
||||
|
||||
```cs
|
||||
SetSessionExecutionMode(ExecutionMode execution_mode);
|
||||
```
|
||||
* ORT_SEQUENTIAL - execute operators in the graph sequentially.
|
||||
* ORT_PARALLEL - execute operators in the graph in parallel.
|
||||
See [../how-to/tune-performance.md] for more details.
|
||||
|
||||
### NodeMetadata
|
||||
Container of metadata for a model graph node, used for communicating the shape and type of the input and output nodes.
|
||||
|
||||
#### Properties
|
||||
```cs
|
||||
int[] Dimensions;
|
||||
```
|
||||
Read-only shape of the node, when the node is a Tensor. Undefined if the node is not a Tensor.
|
||||
|
||||
```cs
|
||||
System.Type ElementType;
|
||||
```
|
||||
Type of the elements of the node, when node is a Tensor. Undefined for non-Tensor nodes.
|
||||
|
||||
```cs
|
||||
bool IsTensor;
|
||||
```
|
||||
Whether the node is a Tensor
|
||||
|
||||
### Exceptions
|
||||
```cs
|
||||
class OnnxRuntimeException: Exception;
|
||||
```
|
||||
|
||||
The type of Exception that is thrown in most of the error conditions related to Onnx Runtime.
|
||||
6
docs/reference/api/index.md
Normal file
6
docs/reference/api/index.md
Normal file
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
title: API docs
|
||||
parent: Reference
|
||||
has_children: true
|
||||
nav_order: 1
|
||||
---
|
||||
105
docs/reference/api/java-api.md
Normal file
105
docs/reference/api/java-api.md
Normal file
|
|
@ -0,0 +1,105 @@
|
|||
---
|
||||
title: Java API
|
||||
parent: API docs
|
||||
grand_parent: Reference
|
||||
nav_order: 3
|
||||
---
|
||||
|
||||
# ONNX Runtime Java API
|
||||
{: .no_toc }
|
||||
|
||||
The ONNX runtime provides a Java binding for running inference on ONNX models on a JVM, using Java 8 or newer.
|
||||
|
||||
## Contents
|
||||
{: .no_toc }
|
||||
|
||||
* TOC placeholder
|
||||
{:toc}
|
||||
|
||||
Release artifacts are published to Maven Central for use as a dependency in most Java build tools. The artifacts are built with support for some popular plaforms.
|
||||
|
||||
|
||||
|
||||
| Artifact | Description | Supported Platforms |
|
||||
|-----------|-------------|---------------------|
|
||||
| [com.microsoft.onnxruntime:onnxruntime](https://search.maven.org/artifact/com.microsoft.onnxruntime/onnxruntime) | CPU | Windows x64, Linux x64, macOS x64 |
|
||||
| [com.microsoft.onnxruntime:onnxruntime_gpu](https://search.maven.org/artifact/com.microsoft.onnxruntime/onnxruntime_gpu) | GPU (CUDA) | Windows x64, Linux x64 |
|
||||
|
||||
For building locally, please see the [Java API development documentation](https://github.com/microsoft/onnxruntime/tree/master/java/README.md) for more details.
|
||||
|
||||
For customization of the loading mechanism of the shared library, please see [advanced loading instructions](https://github.com/microsoft/onnxruntime/tree/master/java/README.md#advanced-loading).
|
||||
|
||||
## API Reference
|
||||
|
||||
The Javadoc is available [here](https://javadoc.io/doc/com.microsoft.onnxruntime/onnxruntime).
|
||||
|
||||
## Sample Code
|
||||
|
||||
An example implementation is located in
|
||||
[src/test/java/sample/ScoreMNIST.java](https://github.com/microsoft/onnxruntime/tree/master/java/src/test/java/sample/ScoreMNIST.java).
|
||||
Once compiled the sample code expects the following arguments `ScoreMNIST
|
||||
<path-to-mnist-model> <path-to-mnist> <scikit-learn-flag>`. MNIST is expected
|
||||
to be in libsvm format. If the optional scikit-learn flag is supplied the model
|
||||
is expected to be produced by skl2onnx (so expects a flat feature vector, and
|
||||
produces a structured output), otherwise the model is expected to be a CNN from
|
||||
pytorch (expecting a `[1][1][28][28]` input, producing a vector of
|
||||
probabilities). Two example models are provided in [testdata](https://github.com/microsoft/onnxruntime/tree/master/java/testdata),
|
||||
`cnn_mnist_pytorch.onnx` and `lr_mnist_scikit.onnx`. The first is a LeNet5 style
|
||||
CNN trained using PyTorch, the second is a logistic regression trained using scikit-learn.
|
||||
|
||||
The unit tests contain several examples of loading models, inspecting input/output node shapes and types, as well as constructing tensors for scoring.
|
||||
|
||||
* [https://github.com/microsoft/onnxruntime/tree/master/java/src/test/java/ai/onnxruntime/InferenceTest.java#L66](https://github.com/microsoft/onnxruntime/tree/master/java/src/test/java/ai/onnxruntime/InferenceTest.java#L66)
|
||||
|
||||
## Get Started
|
||||
|
||||
Here is simple tutorial for getting started with running inference on an existing ONNX model for a given input data. The model is typically trained using any of the well-known training frameworks and exported into the ONNX format.
|
||||
|
||||
Note the code presented below uses syntax available from Java 10 onwards. The Java 8 syntax is similar but more verbose.
|
||||
|
||||
To start a scoring session, first create the `OrtEnvironment`, then open a session using the `OrtSession` class, passing in the file path to the model as a parameter.
|
||||
|
||||
```java
|
||||
var env = OrtEnvironment.getEnvironment();
|
||||
var session = env.createSession("model.onnx",new OrtSession.SessionOptions());
|
||||
```
|
||||
|
||||
Once a session is created, you can execute queries using the `run` method of the `OrtSession` object. At the moment we support `OnnxTensor` inputs, and models can produce `OnnxTensor`, `OnnxSequence` or `OnnxMap` outputs. The latter two are more likely when scoring models produced by frameworks like scikit-learn.
|
||||
|
||||
The run call expects a `Map<String,OnnxTensor>` where the keys match input node names stored in the model. These can be viewed by calling `session.getInputNames()` or `session.getInputInfo()` on an instantiated session.
|
||||
The run call produces a `Result` object, which contains a `Map<String,OnnxValue>` representing the output. The `Result` object is `AutoCloseable` and can be used in a try-with-resources statement to
|
||||
prevent references from leaking out. Once the `Result` object is closed, all it's child `OnnxValue`s are closed too.
|
||||
|
||||
```java
|
||||
OnnxTensor t1,t2;
|
||||
var inputs = Map.of("name1",t1,"name2",t2);
|
||||
try (var results = session.run(inputs)) {
|
||||
// manipulate the results
|
||||
}
|
||||
```
|
||||
|
||||
You can load your input data into OnnxTensor objects in several ways. The most efficient way is to use a `java.nio.Buffer`, but it's possible to use multidimensional arrays too. If constructed using arrays the arrays must not be ragged.
|
||||
|
||||
```java
|
||||
FloatBuffer sourceData; // assume your data is loaded into a FloatBuffer
|
||||
long[] dimensions; // and the dimensions of the input are stored here
|
||||
var tensorFromBuffer = OnnxTensor.createTensor(env,sourceData,dimensions);
|
||||
|
||||
float[][] sourceArray = new float[28][28]; // assume your data is loaded into a float array
|
||||
var tensorFromArray = OnnxTensor.createTensor(env,sourceArray);
|
||||
```
|
||||
|
||||
Here is a [complete sample program](https://github.com/microsoft/onnxruntime/tree/master/java/src/test/java/sample/ScoreMNIST.java) that runs inference on a pretrained MNIST model.
|
||||
|
||||
## Run on a GPU or with another provider (optional)
|
||||
|
||||
To enable other execution providers like GPUs simply turn on the appropriate flag on SessionOptions when creating an OrtSession.
|
||||
|
||||
```java
|
||||
int gpuDeviceId = 0; // The GPU device ID to execute on
|
||||
var sessionOptions = new OrtSession.SessionOptions();
|
||||
sessionOptions.addCUDA(gpuDeviceId);
|
||||
var session = environment.createSession("model.onnx", sessionOptions);
|
||||
```
|
||||
|
||||
The execution providers are prioritized in the order they are enabled.
|
||||
47
docs/reference/api/nodejs-api.md
Normal file
47
docs/reference/api/nodejs-api.md
Normal file
|
|
@ -0,0 +1,47 @@
|
|||
---
|
||||
title: Node.js API
|
||||
parent: API docs
|
||||
grand_parent: Reference
|
||||
nav_order: 4
|
||||
---
|
||||
|
||||
# ONNX Runtime Node.js API
|
||||
{: .no_toc }
|
||||
|
||||
ONNX Runtime Node.js binding enables Node.js applications to run ONNX model inference.
|
||||
|
||||
## Contents
|
||||
{: .no_toc }
|
||||
|
||||
* TOC placeholder
|
||||
{:toc}
|
||||
|
||||
## Usage
|
||||
|
||||
Install the latest stable version:
|
||||
|
||||
```
|
||||
npm install onnxruntime
|
||||
```
|
||||
|
||||
Install the latest dev version:
|
||||
|
||||
```
|
||||
npm install onnxruntime@dev
|
||||
```
|
||||
|
||||
Refer to [Node.js samples](https://github.com/microsoft/onnxruntime/tree/master/samples/nodejs) for samples and tutorials.
|
||||
|
||||
## Requirements
|
||||
|
||||
ONNXRuntime works on Node.js v12.x+ or Electron v5.x+.
|
||||
|
||||
Following platforms are supported with pre-built binaries:
|
||||
|
||||
- Windows x64 CPU NAPI_v3
|
||||
- Linux x64 CPU NAPI_v3
|
||||
- MacOS x64 CPU NAPI_v3
|
||||
|
||||
To use on platforms without pre-built binaries, you can build Node.js binding from source and consume it by `npm install <onnxruntime_repo_root>/nodejs/`.
|
||||
|
||||
See also [build instructions](../../how-to/build.md#apis-and-language-bindings) for building ONNX Runtime Node.js binding locally.
|
||||
47
docs/reference/api/winrt-api.md
Normal file
47
docs/reference/api/winrt-api.md
Normal file
|
|
@ -0,0 +1,47 @@
|
|||
---
|
||||
title: WinRT API
|
||||
parent: API docs
|
||||
grand_parent: Reference
|
||||
nav_order: 5
|
||||
---
|
||||
|
||||
# Windows Machine Learning WinRT API
|
||||
|
||||
New in the ONNX Runtime Nuget package is the ability to use the full [WinML API](https://docs.microsoft.com/en-us/windows/ai/windows-ml/api-reference).
|
||||
|
||||
This allows scenarios such as passing a [Windows.Media.VideoFrame](https://docs.microsoft.com/en-us/uwp/api/Windows.Media.VideoFrame) from your connected camera directly into the runtime for realtime inference.
|
||||
|
||||
The WinML API is a WinRT API that shipped inside the Windows OS starting with build 1809 (RS5) in the Windows.AI.MachineLearning namespace. It embedded a version of the ONNX Runtime.
|
||||
|
||||
Many customers have asked for a way to use this offering as an application redistributable package.
|
||||
|
||||
With our [layered architecture](../../resources/high-level-design.md#the-onnx-runtime-and-windows-os-integration) you can now do this, with some limitations. The WinML APIs have been lifted and mirrored into the Microsoft.AI.MachineLearning namespace in the redistributable.
|
||||
|
||||
## Contents
|
||||
{: .no_toc }
|
||||
|
||||
## NuGet Package
|
||||
|
||||
The Microsoft.AI.MachineLearning [Nuget package](https://www.nuget.org/packages/Microsoft.AI.MachineLearning/) includes the precompiled binaries for using the ONNX runtime with the WinRT API. Support is compiled directly into *onnxruntime.dll*
|
||||
|
||||
Note: As of the 1.3 release, you can use all of the CPU and GPU functionality from these binaries.
|
||||
|
||||
## Sample Code
|
||||
|
||||
Any code already written for the Windows.AI.MachineLearning API can be easily modified to run against the Microsoft.ML.OnnxRuntime package. All types originally referenced by inbox customers via the Windows namespace will need to be updated to now use the Microsoft namespace. Check out these [existing samples](https://github.com/microsoft/Windows-Machine-Learning/tree/master/Samples/SqueezeNetObjectDetection/Desktop/cpp) in github.
|
||||
|
||||
## Deciding on whether to use WinML in the Windows SDK or the Redist
|
||||
To detect if a particular OS version of Windows has the WinML APIs, use the [IsApiContractPresent](https://docs.microsoft.com/en-us/uwp/api/windows.foundation.metadata.apiinformation.isapicontractpresent) method. This can be called from either UWP or native apps.
|
||||
|
||||
If the OS does not have the runtime you need you can switch to use the redist binaries instead.
|
||||
|
||||
|Release|API contract version|
|
||||
|--|--|
|
||||
|Windows OS 1809| 1|
|
||||
|Windows OS 1903| 2|
|
||||
|Windows OS 1909| 2|
|
||||
|ORT release 1.2| 3|
|
||||
|ORT release 1.3| 3|
|
||||
|ORT release 1.4| 3|
|
||||
|
||||
See [here](https://docs.microsoft.com/en-us/windows/ai/windows-ml/onnx-versions) for more about opsets and ONNX version details in Windows OS distributions.
|
||||
42
docs/reference/execution-providers/ACL-ExecutionProvider.md
Normal file
42
docs/reference/execution-providers/ACL-ExecutionProvider.md
Normal file
|
|
@ -0,0 +1,42 @@
|
|||
---
|
||||
title: ARM Compute Library (ACL)
|
||||
parent: Execution Providers
|
||||
grand_parent: Reference
|
||||
nav_order: 1
|
||||
---
|
||||
|
||||
# ACL Execution Provider
|
||||
|
||||
[Arm Compute Library](https://github.com/ARM-software/ComputeLibrary) is an open source inference engine maintained by Arm and Linaro companies. The integration of ACL as an execution provider (EP) into ONNX Runtime accelerates performance of ONNX model workloads across Armv8 cores.
|
||||
|
||||
## Build ACL execution provider
|
||||
|
||||
For build instructions, please see the [BUILD page](../../how-to/build.md#ARM-Compute-Library).
|
||||
|
||||
## Using the ACL execution provider
|
||||
|
||||
### C/C++
|
||||
|
||||
To use ACL as execution provider for inference, please register it as below.
|
||||
|
||||
```c++
|
||||
string log_id = "Foo";
|
||||
auto logging_manager = std::make_unique<LoggingManager>
|
||||
(std::unique_ptr<ISink>{new CLogSink{}},
|
||||
static_cast<Severity>(lm_info.default_warning_level),
|
||||
false,
|
||||
LoggingManager::InstanceType::Default,
|
||||
&log_id)
|
||||
Environment::Create(std::move(logging_manager), env)
|
||||
InferenceSession session_object{so, env};
|
||||
session_object.RegisterExecutionProvider(std::make_unique<::onnxruntime::ACLExecutionProvider>());
|
||||
status = session_object.Load(model_file_name);
|
||||
```
|
||||
|
||||
The C API details are [here](../api/c-api.md).
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
For performance tuning, please see guidance on this page: [ONNX Runtime Perf Tuning](../../how-to/tune-performance.md)
|
||||
|
||||
When/if using [onnxruntime_perf_test](https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/test/perftest), use the flag -e acl
|
||||
|
|
@ -0,0 +1,36 @@
|
|||
---
|
||||
title: ARM NN
|
||||
parent: Execution Providers
|
||||
grand_parent: Reference
|
||||
nav_order: 2
|
||||
---
|
||||
|
||||
## ArmNN Execution Provider
|
||||
|
||||
[ArmNN](https://github.com/ARM-software/armnn) is an open source inference engine maintained by Arm and Linaro companies. The integration of ArmNN as an execution provider (EP) into ONNX Runtime accelerates performance of ONNX model workloads across Armv8 cores.
|
||||
|
||||
### Build ArmNN execution provider
|
||||
For build instructions, please see the [BUILD page](../../how-to/build.md#ArmNN).
|
||||
|
||||
### Using the ArmNN execution provider
|
||||
#### C/C++
|
||||
To use ArmNN as execution provider for inferencing, please register it as below.
|
||||
```
|
||||
string log_id = "Foo";
|
||||
auto logging_manager = std::make_unique<LoggingManager>
|
||||
(std::unique_ptr<ISink>{new CLogSink{}},
|
||||
static_cast<Severity>(lm_info.default_warning_level),
|
||||
false,
|
||||
LoggingManager::InstanceType::Default,
|
||||
&log_id)
|
||||
Environment::Create(std::move(logging_manager), env)
|
||||
InferenceSession session_object{so, env};
|
||||
session_object.RegisterExecutionProvider(std::make_unique<::onnxruntime::ArmNNExecutionProvider>());
|
||||
status = session_object.Load(model_file_name);
|
||||
```
|
||||
The C API details are [here](../api/c-api.md).
|
||||
|
||||
### Performance Tuning
|
||||
For performance tuning, please see guidance on this page: [ONNX Runtime Perf Tuning](../../how-to/tune-performance.md)
|
||||
|
||||
When/if using [onnxruntime_perf_test](https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/test/perftest), use the flag -e armnn
|
||||
196
docs/reference/execution-providers/DNNL-ExecutionProvider.md
Normal file
196
docs/reference/execution-providers/DNNL-ExecutionProvider.md
Normal file
|
|
@ -0,0 +1,196 @@
|
|||
---
|
||||
title: Intel DNNL
|
||||
parent: Execution Providers
|
||||
grand_parent: Reference
|
||||
nav_order: 4
|
||||
---
|
||||
|
||||
# DNNL Execution Provider
|
||||
{: .no_toc }
|
||||
|
||||
Intel® Math Kernel Library for Deep Neural Networks (Intel® DNNL) is an open-source performance library for deep-learning applications. The library accelerates deep-learning applications and frameworks on Intel® architecture and Intel® Processor Graphics Architecture. Intel DNNL contains vectorized and threaded building blocks that you can use to implement deep neural networks (DNN) with C and C++ interfaces. For more, please see the DNNL documentation on (https://intel.github.io/mkl-dnn/).
|
||||
|
||||
Intel and Microsoft have developed DNNL Execution Provider (EP) for ONNX Runtime to accelerate performance of ONNX Runtime using Intel® Math Kernel Library for Deep Neural Networks (Intel® DNNL) optimized primitives.
|
||||
|
||||
|
||||
## Contents
|
||||
{: .no_toc }
|
||||
|
||||
* TOC placeholder
|
||||
{:toc}
|
||||
|
||||
## Build
|
||||
|
||||
For build instructions, please see the [BUILD page](../../how-to/build.md#dnnl-and-mklml).
|
||||
|
||||
## Supported OS
|
||||
|
||||
* Ubuntu 16.04
|
||||
* Windows 10
|
||||
* Mac OS X
|
||||
|
||||
## Supported backend
|
||||
|
||||
* CPU
|
||||
|
||||
## Using the DNNL Execution Provider
|
||||
|
||||
### C/C++
|
||||
|
||||
The DNNLExecutionProvider execution provider needs to be registered with ONNX Runtime to enable in the inference session.
|
||||
|
||||
```c++
|
||||
string log_id = "Foo";
|
||||
auto logging_manager = std::make_unique<LoggingManager>
|
||||
(std::unique_ptr<ISink>{new CLogSink{}},
|
||||
static_cast<Severity>(lm_info.default_warning_level),
|
||||
false,
|
||||
LoggingManager::InstanceType::Default,
|
||||
&log_id)
|
||||
Environment::Create(std::move(logging_manager), env)
|
||||
InferenceSession session_object{so,env};
|
||||
session_object.RegisterExecutionProvider(std::make_unique<::onnxruntime:: DNNLExecutionProvider >());
|
||||
status = session_object.Load(model_file_name);
|
||||
```
|
||||
|
||||
The C API details are [here](../api/c-api.md).
|
||||
|
||||
### Python
|
||||
|
||||
When using the python wheel from the ONNX Runtime built with DNNL execution provider, it will be automatically prioritized over the CPU execution provider. Python APIs details are [here](https://aka.ms/onnxruntime-python).
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
For performance tuning, please see guidance on this page: [ONNX Runtime Perf Tuning](../../how-to/tune-performance.md)
|
||||
|
||||
## Subgraph Optimization
|
||||
|
||||
DNNL uses blocked layout (example: nhwc with channels blocked by 16 – nChw16c) to take advantage of vector operations using AVX512. To get best performance, we avoid reorders (example. Nchw16c to nchw) and propagate blocked layout to next primitive.
|
||||
|
||||
Subgraph optimization achieves this in the following steps.
|
||||
|
||||
1. Parses ONNX Runtime graph and creates an Internal Representation of subgraph..
|
||||
2. Subgraph Operator (DnnlFunKernel) iterates through DNNL nodes and creates a vector DNNL Kernels
|
||||
3. Compute Function of DnnlFunKernel iterates and binds data to DNNL primitives in the vector and submits vector for execution.
|
||||
|
||||
### Subgraph (IR) Internal Representation
|
||||
|
||||
DnnlExecutionProvicer::GetCapability() parses ONNX model graph and creates IR (Internal Representation) of subgraphs of DNNL operators.
|
||||
Each subgraph contains a vector DnnlNodes, inputs, outputs and attributes for all its DnnlNodes. There can be attributes of same name. So, we prefix attribute names with Node name and its index. Unique id for subgraph is set as an attribute.
|
||||
|
||||
DnnlNode has an index to its inputs and outputs and pointer to its parent nodes. DnnlNode directly reads blocked memory from its parent to avoid data reordering.
|
||||
|
||||
<p align="left"><img src="/images/mkl-dnn_node.png" /></p>
|
||||
|
||||
### Subgraph Classes
|
||||
|
||||
Primitive like DnnlConv, DnnlPool, etc are derived from DnnlKernel base class.
|
||||
|
||||
The following UML diagram captures Subgraph classes.
|
||||
|
||||
<p align="left"><img src="/images/mkl-dnn_subgraph.png" /></p>
|
||||
|
||||
### Subgraph Execution
|
||||
|
||||
DnnlExecutionProvicer::Compute() function creates DnnlFuncKernel and call it’s Compute Function.
|
||||
|
||||
DnnlFuncKernel::Compute function creates SubgraphPrimitve pool and add the object to a map.
|
||||
|
||||
SubgraphPrimitve constructor calls the following member functions
|
||||
|
||||
```c++
|
||||
SubgraphPrimitve::CreatePrimitives()
|
||||
for (auto& mklnode : mklnodes) {
|
||||
if (mklnode.name == "Conv") {
|
||||
kernel.reset(new DnnlConv());
|
||||
kernels.push_back(kernel);
|
||||
} else if (mklnode.name == "BatchNormalization-Relu") {
|
||||
kernel.reset(new DnnlBatchNorm());
|
||||
context_.kernels.push_back(kernel);
|
||||
} else if (mklnode.name == "MaxPool") {
|
||||
kernel.reset(new DnnlPool());
|
||||
context_.kernels.push_back(kernel);
|
||||
}
|
||||
.
|
||||
.
|
||||
.
|
||||
```
|
||||
|
||||
In CreatePrimitives method, we iterate DnnlNodes and creates DnnlKernel objects and add DNNL primitive to a vector. It also reads attributes. This is done only once, at first iteration.
|
||||
|
||||
```c++
|
||||
SubgraphPrimitve::Compute()
|
||||
for (auto& kernel : kernels) {
|
||||
kernel->Bind(input_tensors, output_tensors);
|
||||
}
|
||||
stream->submit(net);
|
||||
```
|
||||
|
||||
In SubgraphPrimitve::Compute() method, we iterate thru Dnnl Kernels and bind input data. Then we submit the vector of Primitives to DNNL stream.
|
||||
|
||||
|
||||
### Subgraph Optimization
|
||||
|
||||
DNNL uses blocked layout (example: nhwc with channels blocked by 16 – nChw16c) to take advantage of vector operations using AVX512. To get best performance, we avoid reorders (example. Nchw16c to nchw) and propagate blocked layout to next primitive.
|
||||
|
||||
Subgraph optimization achieves this in the following steps.
|
||||
|
||||
1. Parses ONNX Runtime graph and creates an Internal Representation of subgraph..
|
||||
2. Subgraph Operator (DnnlFunKernel) iterates through DNNL nodes and creates a vector DNNL Kernels
|
||||
3. Compute Function of DnnlFunKernel iterates and binds data to DNNL primitives in the vector and submits vector for execution.
|
||||
|
||||
#### Subgraph (IR) Internal Representation
|
||||
|
||||
DnnlExecutionProvicer::GetCapability() parses ONNX model graph and creates IR (Internal Representation) of subgraphs of DNNL operators.
|
||||
Each subgraph contains a vector DnnlNodes, inputs, outputs and attributes for all its DnnlNodes. There can be attributes of same name. So, we prefix attribute names with Node name and its index.
|
||||
Unique id for subgraph is set as an attribute.
|
||||
|
||||
DnnlNode has an index to its inputs and outputs and pointer to its parent nodes. DnnlNode directly reads blocked memory from its parent to avoid data reordering.
|
||||
|
||||
<p align="left"><img src="/images/mkl-dnn_node.png" /></p>
|
||||
|
||||
#### Subgraph Classes
|
||||
|
||||
Primitive like DnnlConv, DnnlPool, etc are derived from DnnlKernel base class.
|
||||
|
||||
The following UML diagram captures Subgraph classes.
|
||||
|
||||
<p align="left"><img src="/images/mkl-dnn_subgraph.png" /></p>
|
||||
|
||||
#### Subgraph Execution
|
||||
|
||||
DnnlExecutionProvicer::Compute() function creates DnnlFuncKernel and call it’s Compute Function.
|
||||
|
||||
DnnlFuncKernel::Compute function creates SubgraphPrimitve pool and add the object to a map.
|
||||
|
||||
SubgraphPrimitve constructor calls the following member functions
|
||||
|
||||
```c++
|
||||
SubgraphPrimitve::CreatePrimitives()
|
||||
for (auto& mklnode : mklnodes) {
|
||||
if (mklnode.name == "Conv") {
|
||||
kernel.reset(new DnnlConv());
|
||||
kernels.push_back(kernel);
|
||||
} else if (mklnode.name == "BatchNormalization-Relu") {
|
||||
kernel.reset(new DnnlBatchNorm());
|
||||
context_.kernels.push_back(kernel);
|
||||
} else if (mklnode.name == "MaxPool") {
|
||||
kernel.reset(new DnnlPool());
|
||||
context_.kernels.push_back(kernel);
|
||||
}
|
||||
.
|
||||
.
|
||||
.
|
||||
```
|
||||
|
||||
In CreatePrimitives method, we iterate DnnlNodes and creates DnnlKernel objects and add DNNL primitive to a vector. It also reads attributes. This is done only once, at first iteration.
|
||||
|
||||
```c++
|
||||
SubgraphPrimitve::Compute()
|
||||
for (auto& kernel : kernels) {
|
||||
kernel->Bind(input_tensors, output_tensors);
|
||||
}
|
||||
stream->submit(net);
|
||||
```
|
||||
|
||||
In SubgraphPrimitve::Compute() method, we iterate thru Dnnl Kernels and bind input data. Then we submit the vector of Primitives to DNNL stream.
|
||||
126
docs/reference/execution-providers/DirectML-ExecutionProvider.md
Normal file
126
docs/reference/execution-providers/DirectML-ExecutionProvider.md
Normal file
|
|
@ -0,0 +1,126 @@
|
|||
---
|
||||
title: Direct ML
|
||||
parent: Execution Providers
|
||||
grand_parent: Reference
|
||||
nav_order: 3
|
||||
---
|
||||
|
||||
# DirectML Execution Provider
|
||||
{: .no_toc }
|
||||
|
||||
DirectML is a high-performance, hardware-accelerated DirectX 12 library for machine learning on Windows. DirectML provides GPU acceleration for common machine learning tasks across a broad range of supported hardware and drivers.
|
||||
|
||||
When used standalone, the DirectML API is a low-level DirectX 12 library and is suitable for high-performance, low-latency applications such as frameworks, games, and other real-time applications. The seamless interoperability of DirectML with Direct3D 12 as well as its low overhead and conformance across hardware makes DirectML ideal for accelerating machine learning when both high performance is desired, and the reliability and predictabiltiy of results across hardware is critical.
|
||||
|
||||
The *DirectML Execution Provider* is an optional component of ONNX Runtime that uses DirectML to accelerate inference of ONNX models. The DirectML execution provider is capable of greatly improving evaluation time of models using commodity GPU hardware, without sacrificing broad hardware support or requiring vendor-specific extensions to be installed.
|
||||
|
||||
The DirectML Execution Provider currently uses DirectML version 2.1.0.
|
||||
|
||||
## Contents
|
||||
{: .no_toc }
|
||||
|
||||
* TOC placeholder
|
||||
{:toc}
|
||||
|
||||
## Minimum requirements
|
||||
|
||||
The DirectML execution provider requires any DirectX 12 capable device. Almost all commercially-available graphics cards released in the last several years support DirectX 12. Examples of compatible hardware include:
|
||||
|
||||
* NVIDIA Kepler (GTX 600 series) and above
|
||||
* AMD GCN 1st Gen (Radeon HD 7000 series) and above
|
||||
* Intel Haswell (4th-gen core) HD Integrated Graphics and above
|
||||
|
||||
DirectML is compatible with Windows 10, version 1709 (10.0.16299; RS3, "Fall Creators Update") and newer.
|
||||
|
||||
## Building from source
|
||||
|
||||
For general information about building onnxruntime, see [BUILD.md](../../how-to/build.md).
|
||||
|
||||
Requirements for building the DirectML execution provider:
|
||||
1. Visual Studio 2017 toolchain (see [cmake configuration instructions](../../how-to/build.md))
|
||||
2. [The Windows 10 SDK (10.0.18362.0) for Windows 10, version 1903](https://developer.microsoft.com/en-us/windows/downloads/windows-10-sdk) (or newer)
|
||||
|
||||
To build onnxruntime with the DML EP included, supply the `--use_dml` parameter to `build.bat`. e.g.
|
||||
|
||||
```powershell
|
||||
build.bat --config RelWithDebInfo --build_shared_lib --parallel --use_dml
|
||||
```
|
||||
|
||||
The DirectML execution provider supports building for both x64 (default) and x86 architectures.
|
||||
|
||||
Note that building onnxruntime with the DirectML execution provider enabled causes the the DirectML redistributable package to be automatically downloaded as part of the build. Its use is governed by a license whose text may be found as part of the NuGet package.
|
||||
|
||||
## Using the DirectML execution provider
|
||||
|
||||
When using the [C API](../api/c-api.md) with a DML-enabled build of onnxruntime (see [Building from source](#building-from-source)), the DirectML execution provider can be enabled using one of the two factory functions included in `include/onnxruntime/core/providers/dml/dml_provider_factory.h`.
|
||||
|
||||
### `OrtSessionOptionsAppendExecutionProvider_DML` function
|
||||
|
||||
Creates a DirectML Execution Provider which executes on the hardware adapter with the given `device_id`, also known as the adapter index. The device ID corresponds to the enumeration order of hardware adapters as given by [IDXGIFactory::EnumAdapters](https://docs.microsoft.com/windows/win32/api/dxgi/nf-dxgi-idxgifactory-enumadapters). A `device_id` of 0 always corresponds to the default adapter, which is typically the primary display GPU installed on the system. A negative `device_id` is invalid.
|
||||
|
||||
```c
|
||||
OrtStatus* OrtSessionOptionsAppendExecutionProvider_DML(
|
||||
_In_ OrtSessionOptions* options,
|
||||
int device_id
|
||||
);
|
||||
```
|
||||
|
||||
### `OrtSessionOptionsAppendExecutionProviderEx_DML` function
|
||||
|
||||
Creates a DirectML Execution Provider using the given DirectML device, and which executes work on the supplied D3D12 command queue. The DirectML device and D3D12 command queue must have the same parent [ID3D12Device](https://docs.microsoft.com/windows/win32/api/d3d12/nn-d3d12-id3d12device), or an error will be returned. The D3D12 command queue must be of type `DIRECT` or `COMPUTE` (see [D3D12_COMMAND_LIST_TYPE](https://docs.microsoft.com/windows/win32/api/d3d12/ne-d3d12-d3d12_command_list_type)). If this function succeeds, the inference session once created will maintain a strong reference on both the `dml_device` and `command_queue` objects.
|
||||
|
||||
```c
|
||||
OrtStatus* OrtSessionOptionsAppendExecutionProviderEx_DML(
|
||||
_In_ OrtSessionOptions* options,
|
||||
_In_ IDMLDevice* dml_device,
|
||||
_In_ ID3D12CommandQueue* cmd_queue
|
||||
);
|
||||
```
|
||||
|
||||
### See Also
|
||||
|
||||
[DMLCreateDevice function](https://docs.microsoft.com/windows/win32/api/directml/nf-directml-dmlcreatedevice)
|
||||
[ID3D12Device::CreateCommandQueue method](https://docs.microsoft.com/windows/win32/api/d3d12/nf-d3d12-id3d12device-createcommandqueue)
|
||||
[Direct3D 12 programming guide](https://docs.microsoft.com/windows/win32/direct3d12/directx-12-programming-guide)
|
||||
|
||||
### ONNX opset support
|
||||
|
||||
The DirectML execution provider currently supports ONNX opset 11 ([ONNX v1.6](https://github.com/onnx/onnx/releases/tag/v1.6.0)). Evaluating models which require a higher opset version is not supported, and may produce unexpected results.
|
||||
|
||||
### Multi-threading and supported session options
|
||||
|
||||
The DirectML execution provider does not support the use of memory pattern optimizations or parallel execution in onnxruntime. When supplying session options during InferenceSession creation, these options must be disabled or an error will be returned.
|
||||
|
||||
If using the onnxruntime C API, you must call `DisableMemPattern` and `SetSessionExecutionMode` functions to set the options required by the DirectML execution provider.
|
||||
|
||||
See [onnxruntime\include\onnxruntime\core\session\onnxruntime_c_api.h](https://github.com/microsoft/onnxruntime/tree/master/include//onnxruntime/core/session/onnxruntime_c_api.h).
|
||||
|
||||
OrtStatus*(ORT_API_CALL* DisableMemPattern)(_Inout_ OrtSessionOptions* options)NO_EXCEPTION;
|
||||
|
||||
OrtStatus*(ORT_API_CALL* SetSessionExecutionMode)(_Inout_ OrtSessionOptions* options, ExecutionMode execution_mode)NO_EXCEPTION;
|
||||
|
||||
If creating the onnxruntime InferenceSession object directly, you must set the appropriate fields on the `onnxruntime::SessionOptions` struct. Specifically, `execution_mode` must be set to `ExecutionMode::ORT_SEQUENTIAL`, and `enable_mem_pattern` must be `false`.
|
||||
|
||||
Additionally, as the DirectML execution provider does not support parallel execution, it does not support multi-threaded calls to `Run` on the same inference session. That is, if an inference session using the DirectML execution provider, only one thread may call `Run` at a time. Multiple threads are permitted to call `Run` simultaneously if they operate on different inference session objects.
|
||||
|
||||
## Samples
|
||||
|
||||
A complete sample of onnxruntime using the DirectML execution provider can be found under [samples/c_cxx/fns_candy_style_transfer](https://github.com/microsoft/onnxruntime/tree/master/samples//c_cxx/fns_candy_style_transfer).
|
||||
|
||||
## Performance best practices
|
||||
The DirectML execution provider works most efficiently when tensor shapes are known at the time a session is created. This provides a few performance benefits:
|
||||
1) Because constant folding can occur more often, there may be fewer CPU / GPU copies and stalls during evaluations.
|
||||
2) More initialization work occurs when sessions are created rather than during the first evaluation.
|
||||
3) Weights may be pre-processed within DirectML, enabling more efficient algorithms to be used.
|
||||
4) Graph optimization occurs within DirectML. For example, Concat operators may be removed, and more optimal tensor layouts may be used for the input and output of operators.
|
||||
|
||||
Normally when the shapes of model inputs are known during session creation, the shapes for the rest of the model are inferred by OnnxRuntime when a session is created. However if a model input contains a free dimension (such as for batch size), steps must be taken to retain the above performance benefits.
|
||||
|
||||
In this case, there are three options:
|
||||
- Edit the model to replace an input's free dimension (specified through ONNX using "dim_param") with a fixed size (specified through ONNX using "dim_value").
|
||||
- Specify values of named dimensions within model inputs when creating the session using the OnnxRuntime *AddFreeDimensionOverrideByName* ABI.
|
||||
- Edit the model to ensure that an input's free dimension has a [denotation](https://github.com/onnx/onnx/blob/master/docs/DimensionDenotation.md) (such as "DATA_BATCH," or a custom denotation). Then when creating the session, specify the dimension size for each denotation. This can be done using the OnnxRuntime *AddFreeDimensionOverride* ABI.
|
||||
|
||||
## See also
|
||||
|
||||
[DirectML documentation \(docs.microsoft.com\)](https://docs.microsoft.com/en-us/windows/win32/direct3d12/dml)
|
||||
|
|
@ -0,0 +1,57 @@
|
|||
---
|
||||
title: AMD MI GraphX
|
||||
parent: Execution Providers
|
||||
grand_parent: Reference
|
||||
nav_order: 5
|
||||
---
|
||||
|
||||
# MIGraphX Execution Provider
|
||||
{: .no_toc }
|
||||
|
||||
ONNX Runtime's [MIGraphX](https://github.com/ROCmSoftwarePlatform/AMDMIGraphX/) execution provider uses AMD's Deep Learning graph optimization engine to accelerate ONNX model on AMD GPUs.
|
||||
|
||||
## Contents
|
||||
{: .no_toc }
|
||||
|
||||
* TOC placeholder
|
||||
{:toc}
|
||||
|
||||
## Build
|
||||
For build instructions, please see the [BUILD page](../../how-to/build.md#AMD-MIGraphX).
|
||||
|
||||
## Using the MIGraphX execution provider
|
||||
### C/C++
|
||||
The MIGraphX execution provider needs to be registered with ONNX Runtime to enable in the inference session.
|
||||
```
|
||||
string log_id = "Foo";
|
||||
auto logging_manager = std::make_unique<LoggingManager>
|
||||
(std::unique_ptr<ISink>{new CLogSink{}},
|
||||
static_cast<Severity>(lm_info.default_warning_level),
|
||||
false,
|
||||
LoggingManager::InstanceType::Default,
|
||||
&log_id)
|
||||
Environment::Create(std::move(logging_manager), env)
|
||||
InferenceSession session_object{so,env};
|
||||
session_object.RegisterExecutionProvider(std::make_unique<::onnxruntime::MIGraphXExecutionProvider>());
|
||||
status = session_object.Load(model_file_name);
|
||||
```
|
||||
You can check [here](https://github.com/scxiao/ort_test/tree/master/char_rnn) for a specific c/c++ program.
|
||||
|
||||
The C API details are [here](../api/c-api.md).
|
||||
|
||||
### Python
|
||||
When using the Python wheel from the ONNX Runtime build with MIGraphX execution provider, it will be automatically
|
||||
prioritized over the default GPU or CPU execution providers. There is no need to separately register the execution
|
||||
provider. Python APIs details are [here](/python/api_summary).
|
||||
|
||||
You can check [here](https://github.com/scxiao/ort_test/tree/master/python/run_onnx) for a python script to run an
|
||||
model on either the CPU or MIGraphX Execution Provider.
|
||||
|
||||
## Performance Tuning
|
||||
For performance tuning, please see guidance on this page: [ONNX Runtime Perf Tuning](../../how-to/tune-performance.md)
|
||||
|
||||
When/if using [onnxruntime_perf_test](https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/test/perftest#onnxruntime-performance-test), use the flag `-e migraphx`
|
||||
|
||||
## Configuring environment variables
|
||||
MIGraphX providers an environment variable ORT_MIGRAPHX_FP16_ENABLE to enable the FP16 mode.
|
||||
|
||||
|
|
@ -0,0 +1,44 @@
|
|||
---
|
||||
title: NNAPI
|
||||
parent: Execution Providers
|
||||
grand_parent: Reference
|
||||
nav_order: 7
|
||||
---
|
||||
|
||||
|
||||
# NNAPI Execution Provider
|
||||
{: .no_toc }
|
||||
|
||||
[Android Neural Networks API (NNAPI)](https://developer.android.com/ndk/guides/neuralnetworks) is a unified interface to CPU, GPU, and NN accelerators on Android.
|
||||
|
||||
## Contents
|
||||
{: .no_toc }
|
||||
|
||||
* TOC placeholder
|
||||
{:toc}
|
||||
|
||||
## Minimum requirements
|
||||
|
||||
The NNAPI EP requires Android devices with Android 8.1 or higher, it is recommended to use Android devices with Android 9 or higher to achieve optimal performance.
|
||||
|
||||
## Build NNAPI EP
|
||||
|
||||
For build instructions, please see the [BUILD page](../../how-to/build.md#Android-NNAPI-Execution-Provider).
|
||||
|
||||
## Using NNAPI EP in C/C++
|
||||
|
||||
To use NNAPI EP for inferencing, please register it as below.
|
||||
```
|
||||
string log_id = "Foo";
|
||||
auto logging_manager = std::make_unique<LoggingManager>
|
||||
(std::unique_ptr<ISink>{new CLogSink{}},
|
||||
static_cast<Severity>(lm_info.default_warning_level),
|
||||
false,
|
||||
LoggingManager::InstanceType::Default,
|
||||
&log_id)
|
||||
Environment::Create(std::move(logging_manager), env)
|
||||
InferenceSession session_object{so,env};
|
||||
session_object.RegisterExecutionProvider(std::make_unique<::onnxruntime::NnapiExecutionProvider>());
|
||||
status = session_object.Load(model_file_name);
|
||||
```
|
||||
The C API details are [here](../api/c-api.md).
|
||||
193
docs/reference/execution-providers/Nuphar-ExecutionProvider.md
Normal file
193
docs/reference/execution-providers/Nuphar-ExecutionProvider.md
Normal file
|
|
@ -0,0 +1,193 @@
|
|||
---
|
||||
title: Nuphar
|
||||
parent: Execution Providers
|
||||
grand_parent: Reference
|
||||
nav_order: 8
|
||||
---
|
||||
|
||||
# Nuphar Execution Provider (preview)
|
||||
{: .no_toc }
|
||||
|
||||
NUPHAR stands for Neural-network Unified Preprocessing Heterogeneous ARchitecture. As an execution provider in the ONNX Runtime, it is built on top of [TVM](https://github.com/dmlc/tvm) and [LLVM](https://llvm.org) to accelerate ONNX models by compiling nodes in subgraphs into optimized functions via JIT. It also provides JIT caching to save compilation time at runtime.
|
||||
|
||||
Developers can tap into the power of Nuphar through ONNX Runtime to accelerate inferencing of ONNX models. The Nuphar execution provider comes with a common ONNX to TVM lowering [library](https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/core/codegen) that can potentially be reused by other execution providers to leverage TVM. With the Nuphar execution provider, the ONNX Runtime delivers better inference performance on the same hardware compared to generic X64 CPU acceleration, especially for quantized recurrent neural networks. Various products at Microsoft have seen up to a 5x improvement in performance with no loss of accuracy, by running quantized LSTMs via the Nuphar execution provider in the ONNX Runtime.
|
||||
|
||||
## Contents
|
||||
{: .no_toc }
|
||||
|
||||
* TOC placeholder
|
||||
{:toc}
|
||||
|
||||
## Build
|
||||
For build instructions, please see the [BUILD page](../../how-to/build.md#nuphar).
|
||||
|
||||
## Using the Nuphar execution provider
|
||||
### C/C++
|
||||
The Nuphar execution provider needs to be registered with ONNX Runtime to enable in the inference session. The C API details are [here](../api/c-api.md).
|
||||
|
||||
### Python
|
||||
|
||||
You can use the Nuphar execution provider via the python wheel from the ONNX Runtime build. The Nuphar execution provider will be automatically prioritized over the default CPU execution providers, thus no need to separately register the execution provider. Python APIs details are [here](/python/api_summary).
|
||||
|
||||
## Performance and Accuracy Testing
|
||||
|
||||
You can test your ONNX model's performance with [onnxruntime_perf_test](https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/test/perftest/README.md), or test accuracy with [onnx_test_runner](https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/test/onnx). To run these tools with the Nuphar execution provider, please pass `-e nuphar` in command line options.
|
||||
|
||||
Please note that Nuphar uses TVM thread pool and parallel schedule for multi-thread inference performance. When building with OpenMP or MKLML, TVM thread pool would use gomp or iomp as its implementation; otherwise, TVM creates its own thread pool. Because of this, the current default parallel schedule policy is:
|
||||
- Default to on for USE_OPENMP or USE_MKLML. User can use OMP_NUM_THREADS/MKL_NUM_THREADS to control TVM thread pool, as well as TVM_NUM_THREADS
|
||||
- Default to off for none of above. User can use TVM_NUM_THREADS to control TVM thread pool.
|
||||
|
||||
This choice is to ensure to get ideal performance with the different build options. When build with USE_OPENMP or USE_MKLML, users would have to avoid thread confliction from OpenMP or MKL with their inference invocations anyway, so parallel schedule is enable to leverage existing thread pool. When not building with gomp or iomp, TVM thread pool is turned off to avoid confliction with user threads. If needed, user can set env or settings with [NUPHAR_PARALLEL_MIN_WORKLOADS](https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/core/providers/nuphar/common/nuphar_settings.cc#L61) to 0 to disable parallel schedule, or to some non-zero value to enable parallel schedule. The non-zero value indicates the minimal number of elements being computed per thread when parallel schedule would be turned on.
|
||||
|
||||
## Model Conversion and Quantization
|
||||
You may use Python script [model_editor.py](https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/core/providers/nuphar/scripts/model_editor.py) to turn LSTM/GRU/RNN ops to Scan ops for a given model, and then use [model_quantizer.py](https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/core/providers/nuphar/scripts/model_quantizer.py) to quantize MatMul ops into MatMulInteger ops.
|
||||
|
||||
We use dynamic per-row quantization for inputs of LSTM MatMul, so MatMul becomes three parts: quantization, MatMulInteger and dequantization. Weights for MatMulInteger are statically quantized per-column to int8. We have observed good speed-up and no loss of accuracy with this quantization scheme inside Scan for various LSTM models.
|
||||
|
||||
To convert models with LSTM/GRU/RNN ops to Scan ops:
|
||||
```
|
||||
python model_editor.py --input /path/to/input/model --output /path/to/output/model --mode to_scan
|
||||
```
|
||||
|
||||
To quantize MatMul ops to MatMulInteger ops (use option --only_for_scan to only quantize MatMuls inside Scan):
|
||||
```
|
||||
python model_quantizer.py --input /path/to/input/model --output /path/to/output/model --only_for_scan
|
||||
```
|
||||
|
||||
As an experiment, you may test conversion and quantization on [the BiDAF model](https://github.com/onnx/models/tree/master/text/machine_comprehension/bidirectional_attention_flow) from the ONNX model zoo. This model has 5 bidirectional LSTM ops, and long sequence lengths. Our test shows that the quantized model has comparable accuracy of F1 76.24, EM 68.08, vs. floating point model accuracy of F1 76.20, EM 68.11.
|
||||
|
||||
Speed-up in this model is ~20% on Intel Xeon E5-1620v4 (Note that AVX2 is required for Nuphar int8 GEMV performance), when comparing CPU execution provider with the floating point model with LSTM ops, vs. the Nuphar execution provider with quantized MatMulInteger inside Scan ops. Profile shows that most of the cost is in input projection outside of Scan ops, which uses MKL SGEMM. It's worth noting that MKL int8 GEMM is about the same speed as SGEMM in this model, so quantization of SGEMMs outside of Scan won't help performance. We are looking at ways to speedup int8 GEMM for better performance on quantized models.
|
||||
|
||||
## JIT caching
|
||||
|
||||
You may cache JIT binaries to reduce model loading time spent in JIT, using [create_shared.cmd](https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/core/providers/nuphar/scripts/create_shared.cmd) on Windows with Visual Studio 2017, or [create_shared.sh](https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/core/providers/nuphar/scripts/create_shared.sh) on Linux with gcc.
|
||||
|
||||
Windows
|
||||
|
||||
```
|
||||
REM You need to have Visual Studio 2017 for compile and link. Optionally, you can save model checksum to the output dll with FCIV tool from https://support.microsoft.com/en-us/help/841290
|
||||
set NUPHAR_CACHE_PATH=\path\to\jit\cache
|
||||
REM Then run Nuphar inference from either onnx_test_runner or onnxruntime_perf_test, or whatever inference using C++ or Python
|
||||
REM JIT object files would be saved to \path\to\jit\cache\<NUPHAR_CACHE_VERSION>
|
||||
create_shared.cmd \path\to\jit\cache\NUPHAR_CACHE_VERSION [optional_model_file_for_checksum] [optional_output_dll_name]
|
||||
REM If checksum is embedded in dll, set NUPHAR_CACHE_MODEL_CHECKSUM to FCIV output for the model to inference to pass checksum verification at runtime
|
||||
REM Checksum verification failure will cause Nuphar to fallback to JIT instead of loading binary from cache
|
||||
REM Run Nuphar inference again with cached JIT dll
|
||||
```
|
||||
|
||||
Linux
|
||||
|
||||
```bash
|
||||
# You need to have GCC of the same version Nuphar is built with, for compile and link. Optionally, you can save model checksum to jit.so with md5sum
|
||||
export NUPHAR_CACHE_PATH=/path/to/jit/cache
|
||||
# Then run Nuphar inference from either onnx_test_runner or onnxruntime_perf_test, or whatever inference using C++ or Python
|
||||
# JIT object files would be saved to /path/to/jit/cache/<NUPHAR_CACHE_VERSION>
|
||||
create_shared.sh -c /path/to/jit/cache/NUPHAR_CACHE_VERSION [-m optional_model_file_for_checksum] [-o optional_output_so_name]
|
||||
# If checksum is embedded in dll, set NUPHAR_CACHE_MODEL_CHECKSUM to md5sum output for the model to inference to pass checksum verification at runtime
|
||||
# Checksum verification failure will cause Nuphar to fallback to JIT instead of loading binary from cache
|
||||
# run Nuphar inference again with cached JIT dll
|
||||
```
|
||||
|
||||
## Debugging
|
||||
|
||||
### NGEMM
|
||||
|
||||
NGEMM (Nuphar GEMM) is an optimized low-precision GEMM implementation based on compiler techniques.
|
||||
Please refer to our paper for more details of NGEMM: ["NGEMM: Optimizing GEMM for Deep Learning via Compiler-based Techniques"](https://arxiv.org/abs/1910.00178).
|
||||
|
||||
#### NGEMM Tiling / Permutation Configuration
|
||||
|
||||
NGEMM has default tiling parameters, but users can overwrite them through environment variables:
|
||||
|
||||
* NUPHAR_IGEMM_TILE_M / NUPHAR_IGEMM_TILE_N / NUPHAR_IGEMM_TILE_K
|
||||
|
||||
These 3 parameters are the tiling sizes for the corresponding dimensions of GEMM ([M x K] x [K x N]).
|
||||
|
||||
Setting them to different values will generate GEMM with different tiling sizes.
|
||||
|
||||
* NUPHAR_IGEMM_PERMUTE
|
||||
|
||||
This environment variable is to control the loop permutation in GEMM.
|
||||
|
||||
The default is to not apply any loop permutation. Other options are "inner/outer/all",referring to apply permutations to only inner tile loops / only outer loops / both inner and outer loops, respectively.
|
||||
|
||||
There are several [environment variables](https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/core/codegen/common/settings.h) to dump debug information during code generation, plus [some more environment variables](https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/core/providers/nuphar/common/nuphar_settings.h) to dump/control the Nuphar execution provider. You can set environment variables prior to inference to dump debug info to the console. To list some most useful ones:
|
||||
|
||||
* CODEGEN_DUMP_LOWER
|
||||
|
||||
Dumps the lowered function from TVM.
|
||||
|
||||
Set it to "verbose" to dump all nodes, or node op_type to dump specific nodes. You may use "concise" to dump just the op_type of nodes.
|
||||
|
||||
* CODEGEN_DUMP_MODULE
|
||||
|
||||
Dumps compiled binary.
|
||||
|
||||
Set it to "ll" to dumps LLVM bit code, "asm" to dumps assembly.
|
||||
|
||||
* CODEGEN_DUMP_SCHEDULE
|
||||
|
||||
Dumps the schedule used in TVM nodes, like compute_root/compute_inline/compute_at.
|
||||
|
||||
Set it to "verbose" to dump all nodes, or node op_type to dump specific nodes. You may use "concise" to dump just the op_type of nodes.
|
||||
|
||||
* NUPHAR_DUMP_PARTITION
|
||||
|
||||
Dumps nodes in each partition.
|
||||
|
||||
Set it to "1" to dump partitions.
|
||||
|
||||
## Settings
|
||||
|
||||
When there are conflicts of environment variables running Nuphar in multiple processes, user can specify settings string when creating the Nuphar execution provider. The string comprises of comma separated key:value pairs. Keys should be lower cased environment variable names as shown above, and separated from corresponding values with colon. For example, the equivalent string of setting environment variables of NUPHAR_CACHE_PATH/NUPHAR_CACHE_MODEL_CHECKSUM would be "nuphar_cache_path:<path_to_cache>, nuphar_cache_model_checksum:<model_file_checksum>".
|
||||
|
||||
* Using in C/C++
|
||||
|
||||
Settings string could be specified when creating execution provider to specify JIT cache path, as well as model checksum:
|
||||
|
||||
```c++
|
||||
OrtStatus* status = OrtSessionOptionsAppendExecutionProvider_Nuphar(session_options, 1, "nuphar_cache_path:/path/to/cache, nuphar_cache_model_checksum:<model_checksum>"));
|
||||
```
|
||||
|
||||
* Using in C#
|
||||
|
||||
Settings string could be specified when creating session options:
|
||||
|
||||
```csharp
|
||||
SessionOptions.MakeSessionOptionWithNupharProvider("nuphar_cache_path:/path/to/cache, nuphar_cache_model_checksum:<model_checksum>")
|
||||
```
|
||||
|
||||
* Using in Python
|
||||
|
||||
Settings string should be passed in before InferenceSession is created, as providers are not currently exposed yet. Here's an example in Python to set cache path and model checksum:
|
||||
|
||||
```python
|
||||
nuphar_settings = 'nuphar_cache_path:{}, nuphar_cache_model_checksum:{}'.format(cache_dir, model_checksum)
|
||||
onnxruntime.capi._pybind_state.set_nuphar_settings(nuphar_settings)
|
||||
sess = onnxruntime.InferenceSession(model_path)
|
||||
```
|
||||
|
||||
## Known issues
|
||||
|
||||
* ONNX shape inference dependency
|
||||
|
||||
To save runtime JIT cost, Nuphar requires models to have shape inference information from ONNX after model is loaded. Some nodes in ONNX can generate dynamic output tensor shapes from input data value, i.e. ConstantOfShape, Tile, Slice in opset 10, Compress, etc. Those ops may block ONNX shape inference and make the part of graph after such nodes not runnable in Nuphar.
|
||||
|
||||
User may use Python script [symbolic_shape_infer.py](https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/python/tools/symbolic_shape_infer.py) to run symbolic shape inference in ONNX model. This script adds output tensor shapes in the model in graph.value_info field, by doing symbolic dimension computation using sympy when there are Shape ops in model. Besides, running symbolic shape inference on ONNX model would make the graph more readable. Note that when using [model_editor.py](https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/core/providers/nuphar/scripts/model_editor.py) to convert models with LSTM/GRU/RNN to Scan, the resulting model may have incomplete shape inference. Running symbolic_shape_infer.py is needed to get the Scan ops in the model to run in Nuphar. Besides, please note that quantization should be the last step, after verified accuracy and performance of the edited floating point model.
|
||||
|
||||
In addition, user may also manually add shapes to graph.value_info using [onnx.helper.make_tensor_value_info](https://github.com/onnx/onnx/blob/v1.5.0/onnx/helper.py#L290) with model specific knowledge. For example, if you have Hardmax output casted to bool as Compress input condition, then the unknown dimension of the output of Compress is actually 1.
|
||||
|
||||
* Performance benchmark
|
||||
|
||||
Current Nuphar's speed-up in quantized RNNs is optimized for AVX2, when running in single thread and batch size is 1. To help understand RNN performance in different configurations, please use Python script [rnn_benchmark.py](https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/core/providers/nuphar/scripts/rnn_benchmark.py). For older X64 CPUs that do not support AVX2, quantized model may have worse performance than non-quantized ones.
|
||||
|
||||
* Patches to TVM
|
||||
|
||||
There are some changes/bug fixes in TVM for Nuphar to work properly. We are in the process of contributing them back to TVM, but for now patches are used in [our forked TVM](https://github.com/microsoft/onnxruntime-tvm). To build cleanly from scratch, please run following commands before running build.bat or build.sh:
|
||||
|
||||
```bash
|
||||
git submodule sync
|
||||
git submodule foreach --recursive git stash
|
||||
git submodule foreach --recursive git clean -fd
|
||||
git submodule update --init --recursive
|
||||
```
|
||||
194
docs/reference/execution-providers/OpenVINO-ExecutionProvider.md
Normal file
194
docs/reference/execution-providers/OpenVINO-ExecutionProvider.md
Normal file
|
|
@ -0,0 +1,194 @@
|
|||
---
|
||||
title: OpenVINO
|
||||
parent: Execution Providers
|
||||
grand_parent: Reference
|
||||
nav_order: 9
|
||||
---
|
||||
|
||||
# OpenVINO Execution Provider
|
||||
{: .no_toc }
|
||||
|
||||
OpenVINO Execution Provider enables deep learning inference on Intel CPUs, Intel integrated GPUs and Intel<sup>®</sup> Movidius<sup>TM</sup> Vision Processing Units (VPUs). Please refer to [this](https://software.intel.com/en-us/openvino-toolkit/hardware) page for details on the Intel hardware
|
||||
supported.
|
||||
|
||||
## Contents
|
||||
{: .no_toc }
|
||||
|
||||
* TOC placeholder
|
||||
{:toc}
|
||||
|
||||
|
||||
## Build
|
||||
For build instructions, please see the [BUILD page](../../how-to/build.md#openvino).
|
||||
|
||||
## Onnxruntime Graph Optimization level
|
||||
OpenVINO backend performs both hardware dependent as well as independent optimizations to the graph to infer it with on the target hardware with best possible performance. In most of the cases it has been observed that passing in the graph from the input model as is would lead to best possible optimizations by OpenVINO. For this reason, it is advised to turn off high level optimizations performed by ONNX Runtime before handing the graph over to OpenVINO backend. This can be done using Session options as shown below:-
|
||||
|
||||
1. Python API
|
||||
```
|
||||
options = onnxruntime.SessionOptions()
|
||||
options.graph_optimization_level = onnxruntime.GraphOptimizationLevel.ORT_DISABLE_ALL
|
||||
sess = onnxruntime.InferenceSession(<path_to_model_file>, options)
|
||||
```
|
||||
|
||||
2. C++ API
|
||||
```
|
||||
SessionOptions::SetGraphOptimizationLevel(ORT_DISABLE_ALL);
|
||||
```
|
||||
|
||||
## Dynamic device selection
|
||||
When ONNX Runtime is built with OpenVINO Execution Provider, a target hardware option needs to be provided. This build time option becomes the default target harware the EP schedules inference on. However, this target may be overriden at runtime to schedule inference on a different hardware as shown below.
|
||||
|
||||
Note. This dynamic hardware selection is optional. The EP falls back to the build-time default selection if no dynamic hardware option value is specified.
|
||||
|
||||
1. Python API
|
||||
|
||||
```python
|
||||
import onnxruntime
|
||||
onnxruntime.capi._pybind_state.set_openvino_device("<harware_option>")
|
||||
# Create session after this
|
||||
```
|
||||
|
||||
2. C/C++ API
|
||||
|
||||
```c
|
||||
Ort::ThrowOnError(OrtSessionOptionsAppendExecutionProvider_OpenVINO(sf, "<hardware_option>"));
|
||||
```
|
||||
|
||||
## ONNX Layers supported using OpenVINO
|
||||
|
||||
The table below shows the ONNX layers supported and validated using OpenVINO Execution Provider.The below table also lists the Intel hardware support for each of the layers. CPU refers to Intel<sup>®</sup>
|
||||
Atom, Core, and Xeon processors. GPU refers to the Intel Integrated Graphics. VPU refers to USB based Intel<sup>®</sup> Movidius<sup>TM</sup>
|
||||
VPUs as well as Intel<sup>®</sup> Vision accelerator Design with Intel Movidius <sup>TM</sup> MyriadX VPU.
|
||||
|
||||
| **ONNX Layers** | **CPU** | **GPU** | **VPU** |
|
||||
| --- | --- | --- | --- |
|
||||
| Add | Yes | Yes | Yes |
|
||||
| ArgMax | Yes | Yes | Yes |
|
||||
| AveragePool | Yes | Yes | Yes |
|
||||
| BatchNormalization | Yes | Yes | Yes |
|
||||
| Cast | Yes | Yes | Yes |
|
||||
| Clip | Yes | Yes | Yes |
|
||||
| Concat | Yes | Yes | Yes |
|
||||
| Constant | Yes | Yes | Yes |
|
||||
| Conv | Yes | Yes | Yes |
|
||||
| ConvTranspose | Yes | Yes | Yes |
|
||||
| Div | Yes | Yes | Yes |
|
||||
| Dropout | Yes | Yes | Yes |
|
||||
| Flatten | Yes | Yes | Yes |
|
||||
| Floor | Yes | Yes | Yes |
|
||||
| Gather | Yes | Yes | Yes |
|
||||
| GatherND | Yes | Yes | Yes |
|
||||
| Gemm | Yes | Yes | Yes |
|
||||
| GlobalAveragePool | Yes | Yes | Yes |
|
||||
| Identity | Yes | Yes | Yes |
|
||||
| LeakyRelu | Yes | Yes | Yes |
|
||||
| Log | Yes | Yes | Yes |
|
||||
| LRN | Yes | Yes | Yes |
|
||||
| LSTM | Yes | Yes | Yes |
|
||||
| MatMul | Yes | Yes | Yes |
|
||||
| Max | Yes | Yes | Yes |
|
||||
| MaxPool | Yes | Yes | Yes |
|
||||
| Min | Yes | Yes | Yes |
|
||||
| Mul | Yes | Yes | Yes |
|
||||
| Pad | Yes | Yes | Yes |
|
||||
| Pow | Yes | Yes | Yes |
|
||||
| PRelu | Yes | Yes | Yes |
|
||||
| ReduceMax | Yes | Yes | Yes |
|
||||
| ReduceMean | Yes | Yes | Yes |
|
||||
| ReduceMin | Yes | Yes | Yes |
|
||||
| ReduceSum | Yes | Yes | Yes |
|
||||
| Relu | Yes | Yes | Yes |
|
||||
| Reshape | Yes | Yes | Yes |
|
||||
| Sigmoid | Yes | Yes | Yes |
|
||||
| Slice | Yes | Yes | Yes |
|
||||
| Softmax | Yes | Yes | Yes |
|
||||
| Squeeze | Yes | Yes | Yes |
|
||||
| Sub | Yes | Yes | Yes |
|
||||
| Sum | Yes | Yes | Yes |
|
||||
| Tanh | Yes | Yes | Yes |
|
||||
| TopK | Yes | Yes | Yes |
|
||||
| Transpose | Yes | Yes | Yes |
|
||||
| Unsqueeze | Yes | Yes | Yes |
|
||||
|
||||
|
||||
## Topology Support
|
||||
|
||||
Below topologies from ONNX open model zoo are fully supported on OpenVINO Execution Provider and many more are supported through sub-graph partitioning
|
||||
|
||||
## Image Classification Networks
|
||||
|
||||
| **MODEL NAME** | **CPU** | **GPU** | **VPU** | **FPGA** |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| bvlc_alexnet | Yes | Yes | Yes | Yes* |
|
||||
| bvlc_googlenet | Yes | Yes | Yes | Yes* |
|
||||
| bvlc_reference_caffenet | Yes | Yes | Yes | Yes* |
|
||||
| bvlc_reference_rcnn_ilsvrc13 | Yes | Yes | Yes | Yes* |
|
||||
| emotion ferplus | Yes | Yes | Yes | Yes* |
|
||||
| densenet121 | Yes | Yes | Yes | Yes* |
|
||||
| inception_v1 | Yes | Yes | Yes | Yes* |
|
||||
| inception_v2 | Yes | Yes | Yes | Yes* |
|
||||
| mobilenetv2 | Yes | Yes | Yes | Yes* |
|
||||
| resnet18v1 | Yes | Yes | Yes | Yes* |
|
||||
| resnet34v1 | Yes | Yes | Yes | Yes* |
|
||||
| resnet101v1 | Yes | Yes | Yes | Yes* |
|
||||
| resnet152v1 | Yes | Yes | Yes | Yes* |
|
||||
| resnet18v2 | Yes | Yes | Yes | Yes* |
|
||||
| resnet34v2 | Yes | Yes | Yes | Yes* |
|
||||
| resnet101v2 | Yes | Yes | Yes | Yes* |
|
||||
| resnet152v2 | Yes | Yes | Yes | Yes* |
|
||||
| resnet50 | Yes | Yes | Yes | Yes* |
|
||||
| resnet50v2 | Yes | Yes | Yes | Yes* |
|
||||
| shufflenet | Yes | Yes | Yes | Yes* |
|
||||
| squeezenet1.1 | Yes | Yes | Yes | Yes* |
|
||||
| vgg19 | Yes | Yes | Yes | Yes* |
|
||||
| vgg16 | Yes | Yes | Yes | Yes* |
|
||||
| zfnet512 | Yes | Yes | Yes | Yes* |
|
||||
| arcface | Yes | Yes | Yes | Yes* |
|
||||
|
||||
## Image Recognition Networks
|
||||
| **MODEL NAME** | **CPU** | **GPU** | **VPU** | **FPGA** |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| mnist | Yes | Yes | Yes | Yes* |
|
||||
|
||||
## Object Detection Networks
|
||||
| **MODEL NAME** | **CPU** | **GPU** | **VPU** | **FPGA** |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| tiny_yolov2 | Yes | Yes | Yes | Yes* |
|
||||
|
||||
*FPGA only runs in HETERO mode wherein the layers that are not supported on FPGA fall back to OpenVINO CPU.
|
||||
|
||||
## CSharp API
|
||||
|
||||
To use csharp api for openvino execution provider create a custom nuget package. Two nuget packages will be created
|
||||
Microsoft.ML.OnnxRuntime.Managed and Microsoft.ML.OnnxRuntime.Openvino.
|
||||
|
||||
1. Windows
|
||||
|
||||
Build a custom nuget package for windows.
|
||||
```
|
||||
.\build.bat --config Debug --build --use_openvino $Device --build_csharp
|
||||
msbuild csharp\OnnxRuntime.CSharp.proj /p:OrtPackageId=Microsoft.ML.OnnxRuntime.Openvino /p:Configuration=Debug /t:CreatePackage
|
||||
```
|
||||
The msbuild log will show the paths of the nuget packages created.
|
||||
|
||||
2. Linux
|
||||
|
||||
We currently do not have a process to build directly in Linux. But we can
|
||||
copy shared library <ORT linux repo>/build/Linux/<config>/libonnxruntime.so
|
||||
to onnxruntime source repository in windows and execute the same commands
|
||||
above to get custom nuget package for linux. Two nuget packages will be
|
||||
created Microsoft.ML.OnnxRuntime.Managed and Microsoft.ML.OnnxRuntime.Openvino.
|
||||
|
||||
On Linux Machine
|
||||
```
|
||||
./build.sh --config Debug --build_shared_lib --use_openvino $Device
|
||||
```
|
||||
|
||||
On Windows Machine
|
||||
```
|
||||
cp libonnxruntime.so onnxruntime/
|
||||
.\build.bat --config Debug --build --use_openvino $Device --build_csharp
|
||||
msbuild csharp\OnnxRuntime.CSharp.proj /p:OrtPackageId=Microsoft.ML.OnnxRuntime.Openvino /p:Configuration=Debug /t:CreatePackage
|
||||
```
|
||||
The msbuild log will show the path of the nuget packages created.
|
||||
|
|
@ -0,0 +1,91 @@
|
|||
---
|
||||
title: RKNPU
|
||||
parent: Execution Providers
|
||||
grand_parent: Reference
|
||||
nav_order: 10
|
||||
---
|
||||
|
||||
# RKNPU Execution Provider (preview)
|
||||
RKNPU DDK is an advanced interface to access Rockchip NPU. RKNPU Execution Provider enables deep learning inference on Rockchip NPU via RKNPU DDK.
|
||||
|
||||
## Contents
|
||||
{: .no_toc }
|
||||
|
||||
* TOC placeholder
|
||||
{:toc}
|
||||
|
||||
|
||||
## Supported platforms
|
||||
|
||||
* RK1808 Linux
|
||||
|
||||
*Note: RK3399Pro platform is not supported.*
|
||||
|
||||
|
||||
## Build
|
||||
For build instructions, please see the [BUILD page](../../how-to/build.md#RKNPU).
|
||||
|
||||
## Usage
|
||||
### C/C++
|
||||
To use RKNPU as execution provider for inferencing, please register it as below.
|
||||
```
|
||||
string log_id = "Foo";
|
||||
auto logging_manager = std::make_unique<LoggingManager>
|
||||
(std::unique_ptr<ISink>{new CLogSink{}},
|
||||
static_cast<Severity>(lm_info.default_warning_level),
|
||||
false,
|
||||
LoggingManager::InstanceType::Default,
|
||||
&log_id)
|
||||
Environment::Create(std::move(logging_manager), env)
|
||||
InferenceSession session_object{so,env};
|
||||
session_object.RegisterExecutionProvider(std::make_unique<::onnxruntime::RknpuExecutionProvider>());
|
||||
status = session_object.Load(model_file_name);
|
||||
```
|
||||
The C API details are [here](../api/c-api.md).
|
||||
|
||||
|
||||
## Supported Operators
|
||||
|
||||
The table below shows the ONNX Ops supported using RKNPU Execution Provider and the mapping between ONNX Ops and RKNPU Ops.
|
||||
|
||||
| **ONNX Ops** | **RKNPU Ops** |
|
||||
| --- | --- |
|
||||
| Add | ADD |
|
||||
| Mul | MULTIPLY |
|
||||
| Conv | CONV2D |
|
||||
| QLinearConv | CONV2D |
|
||||
| Gemm | FULLCONNECT |
|
||||
| Softmax | SOFTMAX |
|
||||
| AveragePool | POOL |
|
||||
| GlobalAveragePool | POOL |
|
||||
| MaxPool | POOL |
|
||||
| GlobalMaxPool | POOL |
|
||||
| LeakyRelu | LEAKY_RELU |
|
||||
| Concat | CONCAT |
|
||||
| BatchNormalization | BATCH_NORM |
|
||||
| Reshape | RESHAPE |
|
||||
| Flatten | RESHAPE |
|
||||
| Squeeze | RESHAPE |
|
||||
| Unsqueeze | RESHAPE |
|
||||
| Transpose | PERMUTE |
|
||||
| Relu | RELU |
|
||||
| Sub | SUBTRACT |
|
||||
| Clip(0~6)| RELU6 |
|
||||
| DequantizeLinear | DATACONVERT |
|
||||
| Clip | CLIP |
|
||||
|
||||
|
||||
## Supported Models
|
||||
|
||||
Below Models are supported from ONNX open model zoo using RKNPU Execution Provider
|
||||
|
||||
### Image Classification
|
||||
- squeezenet
|
||||
- mobilenetv2-1.0
|
||||
- resnet50v1
|
||||
- resnet50v2
|
||||
- inception_v2
|
||||
|
||||
### Object Detection
|
||||
- ssd
|
||||
- yolov3
|
||||
122
docs/reference/execution-providers/TensorRT-ExecutionProvider.md
Normal file
122
docs/reference/execution-providers/TensorRT-ExecutionProvider.md
Normal file
|
|
@ -0,0 +1,122 @@
|
|||
---
|
||||
title: TensorRT
|
||||
parent: Execution Providers
|
||||
grand_parent: Reference
|
||||
nav_order: 12
|
||||
---
|
||||
|
||||
# TensortRT Execution Provider
|
||||
{: .no_toc }
|
||||
|
||||
The TensorRT execution provider in the ONNX Runtime makes use of NVIDIA's [TensortRT](https://developer.nvidia.com/tensorrt) Deep Learning inferencing engine to accelerate ONNX model in their family of GPUs. Microsoft and NVIDIA worked closely to integrate the TensorRT execution provider with ONNX Runtime.
|
||||
|
||||
With the TensorRT execution provider, the ONNX Runtime delivers better inferencing performance on the same hardware compared to generic GPU acceleration.
|
||||
|
||||
## Contents
|
||||
{: .no_toc }
|
||||
|
||||
* TOC placeholder
|
||||
{:toc}
|
||||
|
||||
## Build
|
||||
|
||||
For build instructions, please see the [BUILD page](../../how-to/build.md#tensorrt).
|
||||
|
||||
The TensorRT execution provider for ONNX Runtime is built and tested with TensorRT 7.1.3.4.
|
||||
|
||||
## Using the TensorRT execution provider
|
||||
|
||||
### C/C++
|
||||
|
||||
The TensorRT execution provider needs to be registered with ONNX Runtime to enable in the inference session.
|
||||
|
||||
```c++
|
||||
string log_id = "Foo";
|
||||
auto logging_manager = std::make_unique<LoggingManager>
|
||||
(std::unique_ptr<ISink>{new CLogSink{}},
|
||||
static_cast<Severity>(lm_info.default_warning_level),
|
||||
false,
|
||||
LoggingManager::InstanceType::Default,
|
||||
&log_id)
|
||||
Environment::Create(std::move(logging_manager), env)
|
||||
InferenceSession session_object{so,env};
|
||||
session_object.RegisterExecutionProvider(std::make_unique<::onnxruntime::TensorrtExecutionProvider>());
|
||||
status = session_object.Load(model_file_name);
|
||||
```
|
||||
|
||||
The C API details are [here](../api/c-api.md).
|
||||
|
||||
#### Shape Inference for TensorRT Subgraphs
|
||||
|
||||
If some operators in the model are not supported by TensorRT, ONNX Runtime will partition the graph and only send supported subgraphs to TensorRT execution provider. Because TensorRT requires that all inputs of the subgraphs have shape specified, ONNX Runtime will throw error if there is no input shape info. In this case please run shape inference for the entire model first by running script [here](https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/python/tools/symbolic_shape_infer.py).
|
||||
|
||||
#### Sample
|
||||
|
||||
This example shows how to run Faster R-CNN model on TensorRT execution provider,
|
||||
|
||||
First, download Faster R-CNN onnx model from onnx model zoo [here](https://github.com/onnx/models/tree/master/vision/object_detection_segmentation/faster-rcnn).
|
||||
|
||||
Second, infer shapes in the model by running shape inference script [here](https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/python/tools/symbolic_shape_infer.py),
|
||||
|
||||
```bash
|
||||
python symbolic_shape_infer.py --input /path/to/onnx/model/model.onnx --output /path/to/onnx/model/new_model.onnx --auto_merge
|
||||
```
|
||||
|
||||
Third, replace original model with the new model and run onnx_test_runner tool under ONNX Runtime build directory,
|
||||
|
||||
```bash
|
||||
./onnx_test_runner -e tensorrt /path/to/onnx/model/
|
||||
```
|
||||
|
||||
### Python
|
||||
|
||||
When using the Python wheel from the ONNX Runtime build with TensorRT execution provider, it will be automatically prioritized over the default GPU or CPU execution providers. There is no need to separately register the execution provider. Python APIs details are .
|
||||
|
||||
#### Python Sample
|
||||
|
||||
Please see [this Notebook](https://github.com/microsoft/onnxruntime/blob/master/docs/python/notebooks/onnx-inference-byoc-gpu-cpu-aks.ipynb) for an example of running a model on GPU using ONNX Runtime through Azure Machine Learning Services.
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
For performance tuning, please see guidance on this page: [ONNX Runtime Perf Tuning](../../how-to/tune-performance.md)
|
||||
|
||||
When/if using [onnxruntime_perf_test](https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/test/perftest#onnxruntime-performance-test), use the flag `-e tensorrt`
|
||||
|
||||
## Configuring environment variables
|
||||
|
||||
There are four environment variables for TensorRT execution provider.
|
||||
|
||||
ORT_TENSORRT_MAX_WORKSPACE_SIZE: maximum workspace size for TensorRT engine.
|
||||
|
||||
ORT_TENSORRT_MAX_PARTITION_ITERATIONS: maximum number of iterations allowed in model partitioning for TensorRT. If target model can't be successfully partitioned when the maximum number of iterations is reached, the whole model will fall back to other execution providers such as CUDA or CPU.
|
||||
|
||||
ORT_TENSORRT_MIN_SUBGRAPH_SIZE: minimum node size in a subgraph after partitioning. Subgraphs with smaller size will fall back to other execution providers.
|
||||
|
||||
ORT_TENSORRT_FP16_ENABLE: Enable FP16 mode in TensorRT
|
||||
|
||||
ORT_TENSORRT_ENGINE_CACHE_ENABLE: Enable TensorRT engine caching. The purpose of using engine caching is to save engine build time in the cases that TensorRT may take long time to optimize and build engine. Engine will be cached after it's built at the first time so that next time when inference session is created the engine can be loaded directly from cache. Note each engine is created for specific settings such as precision (FP32/FP16/INT8 etc), workspace, profiles etc, and specific GPUs and it's not portable, so it's essential to make sure those settings are not changing, otherwise the engines need to be rebuilt and cached again. Also please clean up any old engine cache files (.engine) before enabling the feature for new models. Right now engine caching is only available for static shape models (subgraphs). For dynamic shape cases, since the engine is dynamically created at run-time it's hard to reuse it from previous run without knowing the profile the engine was created from. Dyanmic shape engine caching will be addressed in the future.
|
||||
|
||||
ORT_TENSORRT_ENGINE_CACHE_PATH: Specify path for TensorRT engine files if ORT_TENSORRT_ENGINE_CACHE_ENABLE is 1
|
||||
|
||||
By default TensorRT execution provider builds an ICudaEngine with max workspace size = 1 GB, max partition iterations = 1000, min subgraph size = 1, FP16 mode is disabled and TensorRT engine caching is disabled.
|
||||
|
||||
One can override these defaults by setting environment variables ORT_TENSORRT_MAX_WORKSPACE_SIZE, ORT_TENSORRT_MAX_PARTITION_ITERATIONS, ORT_TENSORRT_MIN_SUBGRAPH_SIZE, ORT_TENSORRT_FP16_ENABLE, ORT_TENSORRT_ENGINE_CACHE_ENABLE and ORT_TENSORRT_ENGINE_CACHE_PATH.
|
||||
e.g. on Linux
|
||||
|
||||
### override default max workspace size to 2GB
|
||||
export ORT_TENSORRT_MAX_WORKSPACE_SIZE=2147483648
|
||||
|
||||
### override default maximum number of iterations to 10
|
||||
export ORT_TENSORRT_MAX_PARTITION_ITERATIONS=10
|
||||
|
||||
### override default minimum subgraph node size to 5
|
||||
export ORT_TENSORRT_MIN_SUBGRAPH_SIZE=5
|
||||
|
||||
### Enable FP16 mode in TensorRT
|
||||
export ORT_TENSORRT_FP16_ENABLE=1
|
||||
|
||||
### Enable TensorRT engine caching
|
||||
export ORT_TENSORRT_ENGINE_CACHE_ENABLE=1
|
||||
|
||||
### Specify TensorRT engine cache path
|
||||
export ORT_TENSORRT_ENGINE_CACHE_PATH="cache"
|
||||
132
docs/reference/execution-providers/Vitis-AI-ExecutionProvider.md
Normal file
132
docs/reference/execution-providers/Vitis-AI-ExecutionProvider.md
Normal file
|
|
@ -0,0 +1,132 @@
|
|||
---
|
||||
title: Vitis AI
|
||||
parent: Execution Providers
|
||||
grand_parent: Reference
|
||||
nav_order: 13
|
||||
---
|
||||
|
||||
# Vitis-AI Execution Provider
|
||||
{: .no_toc }
|
||||
|
||||
<p align="center">
|
||||
<img src="/images/Vitis-AI.png">
|
||||
</p>
|
||||
|
||||
[Vitis-AI](https://github.com/Xilinx/Vitis-AI) is Xilinx's development stack for hardware-accelerated AI inference on Xilinx platforms, including both edge devices and Alveo cards. It consists of optimized IP, tools, libraries, models, and example designs. It is designed with high efficiency and ease of use in mind, unleashing the full potential of AI acceleration on Xilinx FPGA and ACAP.
|
||||
|
||||
The current Vitis-AI execution provider inside ONNXRuntime enables acceleration of Neural Network model inference using DPUv1. DPUv1 is a hardware accelerator for Convolutional Neural Networks (CNN) on top of the Xilinx [Alveo](https://www.xilinx.com/products/boards-and-kits/alveo.html) platform and targets U200 and U250 accelerator cards.
|
||||
|
||||
On this page you will find information on how to [build](#Build) ONNXRuntime with Vitis-AI and on how to [get started](#Getting-started) with an example.
|
||||
|
||||
## Contents
|
||||
{: .no_toc }
|
||||
|
||||
* TOC placeholder
|
||||
{:toc}
|
||||
|
||||
## Build
|
||||
|
||||
For building ONNXRuntime with the Vitis-AI execution provider, you will have to setup the hardware environment and build the docker, see [build steps](#Hardware-setup-and-docker-build).
|
||||
|
||||
### System requirements
|
||||
|
||||
The following table lists system requirements for running docker containers as well as Alveo cards.
|
||||
|
||||
|
||||
| **Component** | **Requirement** |
|
||||
|-----------------------------------------------------|------------------------------------------------------------|
|
||||
| Motherboard | PCI Express 3\.0\-compliant with one dual\-width x16 slot |
|
||||
| System Power Supply | 225W |
|
||||
| Operating System | Ubuntu 16\.04, 18\.04 |
|
||||
| | CentOS 7\.4, 7\.5 |
|
||||
| | RHEL 7\.4, 7\.5 |
|
||||
| CPU | Intel i3/i5/i7/i9/Xeon 64-bit CPU |
|
||||
| GPU \(Optional to accelerate quantization\) | NVIDIA GPU with a compute capability > 3.0 |
|
||||
| CUDA Driver \(Optional to accelerate quantization\) | nvidia\-410 |
|
||||
| FPGA | Xilinx Alveo U200 or U250 |
|
||||
| Docker Version | 19\.03\.1 |
|
||||
|
||||
### Hardware setup and docker build
|
||||
|
||||
1. Clone the Vitis AI repository:
|
||||
```
|
||||
git clone https://github.com/xilinx/vitis-ai
|
||||
```
|
||||
2. Install the Docker, and add the user to the docker group. Link the user to docker installation instructions from the following docker's website:
|
||||
* https://docs.docker.com/install/linux/docker-ce/ubuntu/
|
||||
* https://docs.docker.com/install/linux/docker-ce/centos/
|
||||
* https://docs.docker.com/install/linux/linux-postinstall/
|
||||
3. Any GPU instructions will have to be separated from Vitis AI.
|
||||
4. Set up Vitis AI to target Alveo cards. To target Alveo cards with Vitis AI for machine learning workloads, you must install the following software components:
|
||||
* Xilinx Runtime (XRT)
|
||||
* Alveo Deployment Shells (DSAs)
|
||||
* Xilinx Resource Manager (XRM) (xbutler)
|
||||
* Xilinx Overlaybins (Accelerators to Dynamically Load - binary programming files)
|
||||
|
||||
While it is possible to install all of these software components individually, a script has been provided to automatically install them at once. To do so:
|
||||
* Run the following commands:
|
||||
```
|
||||
cd Vitis-AI/alveo/packages
|
||||
sudo su
|
||||
./install.sh
|
||||
```
|
||||
* Power cycle the system.
|
||||
5. Build and start the ONNXRuntime Vitis-AI Docker Container.
|
||||
```
|
||||
cd {onnxruntime-root}/dockerfiles
|
||||
docker build -t onnxruntime-vitisai -f Dockerfile.vitisai .
|
||||
./scripts/docker_run_vitisai.sh
|
||||
```
|
||||
|
||||
Setup inside container
|
||||
```
|
||||
source /opt/xilinx/xrt/setup.sh
|
||||
conda activate vitis-ai-tensorflow
|
||||
```
|
||||
|
||||
## Getting started
|
||||
|
||||
### On-the-fly quantization
|
||||
|
||||
Usually, to be able to accelerate inference of Neural Network models with Vitis-AI DPU accelerators, those models need to quantized upfront. In the ONNXRuntime Vitis-AI execution provider we make use of on-the-fly quantization to remove this additional preprocessing step. In this flow, one doesn't need to quantize his/her model upfront but can make use of the typical inference execution calls (InferenceSession.run) to quantize the model on-the-fly using the first N inputs that are provided (see more information below). This will set up and calibrate the Vitis-AI DPU and from that point onwards inference will be accelerated for all next inputs.
|
||||
|
||||
### Config/Settings
|
||||
|
||||
A couple of environment variables can be used to customize the Vitis-AI execution provider.
|
||||
|
||||
| **Environment Variable** | **Default if unset** | **Explanation** |
|
||||
|----------------------------|---------------------------|---------------------------------------------------------|
|
||||
| PX_QUANT_SIZE | 128 | The number of inputs that will be used for quantization (necessary for Vitis-AI acceleration) |
|
||||
| PX_BUILD_DIR | Use the on-the-fly quantization flow | Loads the quantization and compilation information from the provided build directory and immediately starts Vitis-AI hardware acceleration. This configuration can be used if the model has been executed before using on-the-fly quantization during which the quantization and comilation information was cached in a build directory. |
|
||||
|
||||
### Samples
|
||||
|
||||
When using python, you can base yourself on the following example:
|
||||
|
||||
```python
|
||||
# Import pyxir before onnxruntime
|
||||
import pyxir
|
||||
import pyxir.frontend.onnx
|
||||
import pyxir.contrib.dpuv1.dpuv1
|
||||
|
||||
import onnxruntime
|
||||
|
||||
# Add other imports
|
||||
# ...
|
||||
|
||||
# Load inputs and do preprocessing
|
||||
# ...
|
||||
|
||||
# Create an inference session using the Vitis-AI execution provider
|
||||
session = onnxruntime.InferenceSession('[model_file].onnx', None,["VitisAIExecutionProvider"])
|
||||
|
||||
# First N (default = 128) inputs are used for quantization calibration and will
|
||||
# be executed on the CPU
|
||||
# This config can be changed by setting the 'PX_QUANT_SIZE' (e.g. export PX_QUANT_SIZE=64)
|
||||
imput_name = [...]
|
||||
outputs = [session.run([], {input_name: calib_inputs[i]})[0] for i in range(128)]
|
||||
|
||||
# Afterwards, computations will be accelerated on the FPGA
|
||||
input_data = [...]
|
||||
result = session.run([], {input_name: input_data})
|
||||
```
|
||||
6
docs/reference/execution-providers/index.md
Normal file
6
docs/reference/execution-providers/index.md
Normal file
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
title: Execution Providers
|
||||
parent: Reference
|
||||
has_children: true
|
||||
nav_order: 2
|
||||
---
|
||||
|
|
@ -0,0 +1,74 @@
|
|||
---
|
||||
title: nGraph
|
||||
parent: Execution Providers
|
||||
grand_parent: Reference
|
||||
nav_order: 6
|
||||
---
|
||||
|
||||
# nGraph Execution Provider
|
||||
{: .no_toc }
|
||||
|
||||
<p align="center">
|
||||
<img src="/images/ngraph-logo.png">
|
||||
</p>
|
||||
|
||||
## **Deprecation Notice**
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| Deprecation Begins | June 1, 2020 |
|
||||
| Removal Date | December 1, 2020 |
|
||||
|
||||
Starting with the OpenVINO™ toolkit 2020.2 release, all of the features previously available through nGraph have been merged into the OpenVINO™ toolkit. As a result, all the features previously available through ONNX RT Execution Provider for nGraph have been merged with ONNX RT Execution Provider for OpenVINO™ toolkit.
|
||||
|
||||
Therefore, ONNX RT Execution Provider for **nGraph** will be deprecated starting June 1, 2020 and will be completely removed on December 1, 2020. Users are recommended to migrate to the ONNX RT Execution Provider for OpenVINO™ toolkit as the unified solution for all AI inferencing on Intel® hardware.
|
||||
|
||||
[nGraph](https://github.com/NervanaSystems/ngraph) is a deep learning compiler from Intel®. The integration of nGraph as an execution provider (EP) into ONNX Runtime accelerates performance of ONNX model workloads across wide range of hardware offerings. Microsoft and Intel worked closely to integrate the nGraph EP with ONNX Runtime to showcase the benefits of quantization (int8). The nGraph EP leverages Intel® DL Boost and delivers performance increase with minimal loss of accuracy relative to FP32 ONNX models. With the nGraph EP, the ONNX Runtime delivers better inference performance across range of Intel hardware including Intel® Xeon® Processors compared to a generic CPU execution provider.
|
||||
|
||||
## Build
|
||||
|
||||
For build instructions, please see the [BUILD page](../../how-to/build.md#ngraph).
|
||||
|
||||
## Supported OS
|
||||
|
||||
While the nGraph Compiler stack supports various operating systems and backends ([full list available here](https://www.ngraph.ai/ecosystem)), the nGraph execution provider for ONNX Runtime is validated for the following:
|
||||
|
||||
* Ubuntu 16.04
|
||||
* Windows 10 (`DEX_ONLY` mode is only one supported for the moment, codegen mode is work-in-progress.)
|
||||
* More to be added soon!
|
||||
|
||||
## Supported backend
|
||||
|
||||
* CPU
|
||||
|
||||
## Using the nGraph execution provider
|
||||
|
||||
### C/C++
|
||||
|
||||
To use nGraph as execution provider for inferencing, please register it as below.
|
||||
|
||||
```c++
|
||||
string log_id = "Foo";
|
||||
auto logging_manager = std::make_unique<LoggingManager>
|
||||
(std::unique_ptr<ISink>{new CLogSink{}},
|
||||
static_cast<Severity>(lm_info.default_warning_level),
|
||||
false,
|
||||
LoggingManager::InstanceType::Default,
|
||||
&log_id)
|
||||
Environment::Create(std::move(logging_manager), env)
|
||||
InferenceSession session_object{so,env};
|
||||
session_object.RegisterExecutionProvider(std::make_unique<::onnxruntime::NGRAPHExecutionProvider>());
|
||||
status = session_object.Load(model_file_name);
|
||||
```
|
||||
|
||||
The C API details are [here](../api/c-api.md).
|
||||
|
||||
### Python
|
||||
|
||||
When using the python wheel from the ONNX Runtime built with nGraph execution provider, it will be automatically prioritized over the CPU execution provider. Python APIs details are [here](/python/api_summary).
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
For performance tuning, please see guidance on this page: [ONNX Runtime Perf Tuning](../../how-to/tune-performance.md)
|
||||
|
||||
When/if using [onnxruntime_perf_test](https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/test/perftest), use the flag -e ngraph
|
||||
8
docs/reference/index.md
Normal file
8
docs/reference/index.md
Normal file
|
|
@ -0,0 +1,8 @@
|
|||
---
|
||||
title: Reference
|
||||
has_children: true
|
||||
nav_order: 4
|
||||
---
|
||||
|
||||
# ONNX Runtime Reference
|
||||
{: .no_toc }
|
||||
2566
docs/reference/operators/ContribOperators.md
Normal file
2566
docs/reference/operators/ContribOperators.md
Normal file
File diff suppressed because it is too large
Load diff
662
docs/reference/operators/OperatorKernels.md
Normal file
662
docs/reference/operators/OperatorKernels.md
Normal file
|
|
@ -0,0 +1,662 @@
|
|||
---
|
||||
nav_exclude: true
|
||||
---
|
||||
|
||||
## Supported Operators Data Types
|
||||
*This file is automatically generated from the
|
||||
[def files](/onnxruntime/core/providers/cpu/cpu_execution_provider.cc) via [this script](/tools/python/gen_opkernel_doc.py).
|
||||
Do not modify directly and instead edit operator definitions.*
|
||||
|
||||
|
||||
|
||||
## Operators implemented by CPUExecutionProvider
|
||||
|
||||
| Op Name | Parameters | OpSet Version | Types Supported |
|
||||
|---------|------------|---------------|-----------------|
|
||||
**Operator Domain:** *ai.onnx.ml*
|
||||
|Abs|(*in* X:**T**, *out* Y:**T**)|6+|**T** = tensor(double), tensor(float), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
|Acos|(*in* input:**T**, *out* output:**T**)|7+|**T** = tensor(float)|
|
||||
|Acosh|(*in* input:**T**, *out* output:**T**)|9+|**T** = tensor(float)|
|
||||
|Add|(*in* A:**T**, *in* B:**T**, *out* C:**T**)|7+|**T** = tensor(double), tensor(float), tensor(int32), tensor(int64)|
|
||||
|Affine|(*in* X:**T**, *out* Y:**T**)|1+|**T** = tensor(float)|
|
||||
|And|(*in* A:**T**, *in* B:**T**, *out* C:**T1**)|7+|**T** = tensor(bool)|
|
||||
| | ||**T1** = tensor(bool)|
|
||||
|ArgMax|(*in* data:**T**, *out* reduced:**tensor(int64)**)|12+|**T** = tensor(float), tensor(int32)|
|
||||
| | |[1, 10]|**T** = tensor(float), tensor(int32)|
|
||||
| | |[11, 11]|**T** = tensor(float), tensor(int32)|
|
||||
|ArgMin|(*in* data:**T**, *out* reduced:**tensor(int64)**)|12+|**T** = tensor(float), tensor(int32)|
|
||||
| | |[1, 10]|**T** = tensor(float), tensor(int32)|
|
||||
| | |[11, 11]|**T** = tensor(float), tensor(int32)|
|
||||
|ArrayFeatureExtractor|(*in* X:**T**, *in* Y:**tensor(int64)**, *out* Z:**T**)|1+|**T** = tensor(double), tensor(float), tensor(int32), tensor(int64), tensor(string)|
|
||||
|Asin|(*in* input:**T**, *out* output:**T**)|7+|**T** = tensor(float)|
|
||||
|Asinh|(*in* input:**T**, *out* output:**T**)|9+|**T** = tensor(float)|
|
||||
|Atan|(*in* input:**T**, *out* output:**T**)|7+|**T** = tensor(float)|
|
||||
|Atanh|(*in* input:**T**, *out* output:**T**)|9+|**T** = tensor(float)|
|
||||
|AveragePool|(*in* X:**T**, *out* Y:**T**)|11+|**T** = tensor(float)|
|
||||
| | |[10, 10]|**T** = tensor(float)|
|
||||
| | |[7, 9]|**T** = tensor(float)|
|
||||
|BatchNormalization|(*in* X:**T**, *in* scale:**T**, *in* B:**T**, *in* mean:**T**, *in* var:**T**, *in* training_mode:**T1**, *out* Y:**T**, *out* output_mean:**T**, *out* output_var:**T**, *out* saved_mean:**T**, *out* saved_var:**T**) or (*in* X:**T**, *in* scale:**T**, *in* B:**T**, *in* mean:**T**, *in* var:**T**, *out* Y:**T**, *out* mean:**T**, *out* var:**T**, *out* saved_mean:**T**, *out* saved_var:**T**)|[7, 9]|**T** = tensor(double), tensor(float)|
|
||||
|Binarizer|(*in* X:**T**, *out* Y:**T**)|1+|**T** = tensor(float)|
|
||||
|BitShift|(*in* X:**T**, *in* Y:**T**, *out* Z:**T**)|11+|**T** = tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
|Cast|(*in* input:**T1**, *out* output:**T2**)|9+|**T1** = tensor(string)|
|
||||
| | ||**T2** = tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | |[6, 9]|**T1** = tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | ||**T2** = tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
|CastMap|(*in* X:**T1**, *out* Y:**T2**)|1+|**T1** = map(int64,tensor(float)), map(int64,tensor(string))|
|
||||
| | ||**T2** = tensor(float), tensor(int64), tensor(string)|
|
||||
|CategoryMapper|(*in* X:**T1**, *out* Y:**T2**)|1+|**T1** = tensor(int64), tensor(string)|
|
||||
| | ||**T2** = tensor(int64), tensor(string)|
|
||||
|Ceil|(*in* X:**T**, *out* Y:**T**)|6+|**T** = tensor(float)|
|
||||
|Clip|(*in* input:**T**, *in* min:**T**, *in* max:**T**, *out* output:**T**) or (*in* input:**T**, *out* output:**T**)|12+|**T** = tensor(double), tensor(float), tensor(int64), tensor(int8), tensor(uint64), tensor(uint8)|
|
||||
| | |[11, 11]|**T** = tensor(float)|
|
||||
| | |[6, 10]|**T** = tensor(float)|
|
||||
|Compress|(*in* input:**T**, *in* condition:**T1**, *out* output:**T**)|11+|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | ||**T1** = tensor(bool)|
|
||||
| | |[9, 10]|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | ||**T1** = tensor(bool)|
|
||||
|Concat|(*in* inputs:**T**, *out* concat_result:**T**)|11+|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | |[4, 10]|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
|ConcatFromSequence|(*in* input_sequence:**S**, *out* concat_result:**T**)|11+|**S** = seq(tensor(bfloat16)), seq(tensor(bool)), seq(tensor(double)), seq(tensor(float)), seq(tensor(float16)), seq(tensor(int16)), seq(tensor(int32)), seq(tensor(int64)), seq(tensor(int8)), seq(tensor(string)), seq(tensor(uint16)), seq(tensor(uint32)), seq(tensor(uint64)), seq(tensor(uint8))|
|
||||
|ConstantOfShape|(*in* input:**T1**, *out* output:**T2**)|9+|**T1** = tensor(int64)|
|
||||
| | ||**T2** = tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
|Conv|(*in* X:**T**, *in* W:**T**, *in* B:**T**, *out* Y:**T**)|11+|**T** = tensor(float)|
|
||||
| | |[1, 10]|**T** = tensor(float)|
|
||||
|ConvInteger|(*in* x:**T1**, *in* w:**T2**, *in* x_zero_point:**T1**, *in* w_zero_point:**T2**, *out* y:**T3**)|10+|**T1** = tensor(uint8)|
|
||||
| | ||**T2** = tensor(uint8)|
|
||||
| | ||**T3** = tensor(int32)|
|
||||
|ConvTranspose|(*in* X:**T**, *in* W:**T**, *in* B:**T**, *out* Y:**T**)|11+|**T** = tensor(float)|
|
||||
| | |[1, 10]|**T** = tensor(float)|
|
||||
|Cos|(*in* input:**T**, *out* output:**T**)|7+|**T** = tensor(float)|
|
||||
|Cosh|(*in* input:**T**, *out* output:**T**)|9+|**T** = tensor(float)|
|
||||
|Crop|(*in* input:**T**, *out* output:**T**)|1+|**T** = tensor(float)|
|
||||
|CumSum|(*in* x:**T**, *in* axis:**T2**, *out* y:**T**)|11+|**T** = tensor(double), tensor(float), tensor(int32), tensor(int64)|
|
||||
| | ||**T2** = tensor(int32), tensor(int64)|
|
||||
|DepthToSpace|(*in* input:**T**, *out* output:**T**)|11+|**T** = tensor(float)|
|
||||
| | |[1, 10]|**T** = tensor(float)|
|
||||
|DequantizeLinear|(*in* x:**T**, *in* x_scale:**tensor(float)**, *in* x_zero_point:**T**, *out* y:**tensor(float)**)|10+|**T** = tensor(int8), tensor(uint8)|
|
||||
|Det|(*in* X:**T**, *out* Y:**T**)|11+|**T** = tensor(float)|
|
||||
|DictVectorizer|(*in* X:**T1**, *out* Y:**T2**)|1+|**T1** = map(int64,tensor(double)), map(int64,tensor(float)), map(int64,tensor(string)), map(string,tensor(double)), map(string,tensor(float)), map(string,tensor(int64))|
|
||||
| | ||**T2** = tensor(double), tensor(float), tensor(int64), tensor(string)|
|
||||
|Div|(*in* A:**T**, *in* B:**T**, *out* C:**T**)|7+|**T** = tensor(double), tensor(float), tensor(int32), tensor(int64)|
|
||||
|Dropout|(*in* data:**T**, *in* ratio:**T1**, *out* output:**T**, *out* mask:**T2**) or (*in* data:**T**, *out* output:**T**, *out* mask:**T**) or (*in* data:**T**, *out* output:**T**, *out* mask:**T1**)|10+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
| | ||**T1** = tensor(bool)|
|
||||
| | |[7, 9]|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
| | ||**T1** = tensor(bool)|
|
||||
|DynamicQuantizeLinear|(*in* x:**T1**, *out* y:**T2**, *out* y_scale:**tensor(float)**, *out* y_zero_point:**T2**)|11+|**T2** = tensor(uint8)|
|
||||
|DynamicSlice|(*in* data:**T**, *in* starts:**Tind**, *in* ends:**Tind**, *in* axes:**Tind**, *out* output:**T**)|1+|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | ||**Tind** = tensor(int32), tensor(int64)|
|
||||
|Elu|(*in* X:**T**, *out* Y:**T**)|6+|**T** = tensor(float)|
|
||||
|Equal|(*in* A:**T**, *in* B:**T**, *out* C:**T1**)|11+|**T** = tensor(bool), tensor(float), tensor(int32), tensor(int64)|
|
||||
| | ||**T1** = tensor(bool)|
|
||||
| | |[7, 10]|**T** = tensor(bool), tensor(int32), tensor(int64)|
|
||||
| | ||**T1** = tensor(bool)|
|
||||
|Erf|(*in* input:**T**, *out* output:**T**)|9+|**T** = tensor(float)|
|
||||
|Exp|(*in* input:**T**, *out* output:**T**)|6+|**T** = tensor(double), tensor(float)|
|
||||
|Expand|(*in* input:**T**, *in* shape:**tensor(int64)**, *out* output:**T**)|8+|**T** = tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
|EyeLike|(*in* input:**T1**, *out* output:**T2**)|9+|**T1** = tensor(double), tensor(float), tensor(int32), tensor(int64), tensor(uint64)|
|
||||
| | ||**T2** = tensor(double), tensor(float), tensor(int32), tensor(int64), tensor(uint64)|
|
||||
|FeatureVectorizer|(*in* X:**T1**, *out* Y:**tensor(float)**)|1+|**T1** = tensor(double), tensor(float), tensor(int32), tensor(int64)|
|
||||
|Flatten|(*in* input:**T**, *out* output:**T**)|11+|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | |[1, 8]|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | |[9, 10]|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
|Floor|(*in* X:**T**, *out* Y:**T**)|6+|**T** = tensor(float)|
|
||||
|GRU|(*in* X:**T**, *in* W:**T**, *in* R:**T**, *in* B:**T**, *in* sequence_lens:**T1**, *in* initial_h:**T**, *out* Y:**T**, *out* Y_h:**T**)|7+|**T** = tensor(double), tensor(float)|
|
||||
| | ||**T1** = tensor(int32)|
|
||||
|Gather|(*in* data:**T**, *in* indices:**Tind**, *out* output:**T**)|11+|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | ||**Tind** = tensor(int32), tensor(int64)|
|
||||
| | |[1, 10]|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | ||**Tind** = tensor(int32), tensor(int64)|
|
||||
|GatherElements|(*in* data:**T**, *in* indices:**Tind**, *out* output:**T**)|11+|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | ||**Tind** = tensor(int32), tensor(int64)|
|
||||
|GatherND|(*in* data:**T**, *in* indices:**tensor(int64)**, *out* output:**T**)|11+|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | ||**Tind** = tensor(int64)|
|
||||
|Gemm|(*in* A:**T**, *in* B:**T**, *in* C:**T**, *out* Y:**T**)|11+|**T** = tensor(float)|
|
||||
| | |[7, 8]|**T** = tensor(float)|
|
||||
| | |[9, 10]|**T** = tensor(float)|
|
||||
|GlobalAveragePool|(*in* X:**T**, *out* Y:**T**)|1+|**T** = tensor(float)|
|
||||
|GlobalLpPool|(*in* X:**T**, *out* Y:**T**)|2+|**T** = tensor(float)|
|
||||
|GlobalMaxPool|(*in* X:**T**, *out* Y:**T**)|1+|**T** = tensor(float)|
|
||||
|Greater|(*in* A:**T**, *in* B:**T**, *out* C:**T1**)|9+|**T** = tensor(int32), tensor(int64)|
|
||||
| | ||**T1** = tensor(bool)|
|
||||
| | |[7, 9]|**T** = tensor(float)|
|
||||
| | ||**T1** = tensor(bool)|
|
||||
|HardSigmoid|(*in* X:**T**, *out* Y:**T**)|6+|**T** = tensor(float)|
|
||||
|Hardmax|(*in* input:**T**, *out* output:**T**)|11+|**T** = tensor(float)|
|
||||
| | |[1, 10]|**T** = tensor(float)|
|
||||
|Identity|(*in* input:**T**, *out* output:**T**)|1+|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
|If|(*in* cond:**B**, *out* outputs:**V**)|11+|**B** = tensor(bool)|
|
||||
| | ||**V** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | |[1, 10]|**B** = tensor(bool)|
|
||||
| | ||**V** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
|ImageScaler|(*in* input:**T**, *out* output:**T**)|1+|**T** = tensor(float)|
|
||||
|Imputer|(*in* X:**T**, *out* Y:**T**)|1+|**T** = tensor(float), tensor(int64)|
|
||||
|InstanceNormalization|(*in* input:**T**, *in* scale:**T**, *in* B:**T**, *out* output:**T**)|6+|**T** = tensor(float)|
|
||||
|IsInf|(*in* X:**T1**, *out* Y:**T2**)|10+|**T1** = tensor(double), tensor(float)|
|
||||
| | ||**T2** = tensor(bool)|
|
||||
|IsNaN|(*in* X:**T1**, *out* Y:**T2**)|9+|**T1** = tensor(float), tensor(float16)|
|
||||
| | ||**T2** = tensor(bool)|
|
||||
|LRN|(*in* X:**T**, *out* Y:**T**)|1+|**T** = tensor(float)|
|
||||
|LSTM|(*in* X:**T**, *in* W:**T**, *in* R:**T**, *in* B:**T**, *in* sequence_lens:**T1**, *in* initial_h:**T**, *in* initial_c:**T**, *in* P:**T**, *out* Y:**T**, *out* Y_h:**T**, *out* Y_c:**T**)|7+|**T** = tensor(double), tensor(float)|
|
||||
| | ||**T1** = tensor(int32)|
|
||||
|LabelEncoder|(*in* X:**T1**, *out* Y:**T2**)|2+|**T1** = tensor(float), tensor(int64), tensor(string)|
|
||||
| | ||**T2** = tensor(float), tensor(int64), tensor(string)|
|
||||
| | |[1, 1]|**T1** = tensor(int64), tensor(string)|
|
||||
| | ||**T2** = tensor(int64), tensor(string)|
|
||||
|LayerNormalization|(*in* X:**T**, *in* scale:**T**, *in* B:**T**, *out* Y:**T**, *out* mean:**U**, *out* inv_std_var:**U**)|1+|**T** = tensor(double), tensor(float)|
|
||||
|LeakyRelu|(*in* X:**T**, *out* Y:**T**)|6+|**T** = tensor(float)|
|
||||
|Less|(*in* A:**T**, *in* B:**T**, *out* C:**T1**)|9+|**T** = tensor(int32), tensor(int64)|
|
||||
| | ||**T1** = tensor(bool)|
|
||||
| | |[7, 9]|**T** = tensor(double), tensor(float)|
|
||||
| | ||**T1** = tensor(bool)|
|
||||
|LinearClassifier|(*in* X:**T1**, *out* Y:**T2**, *out* Z:**tensor(float)**)|1+|**T1** = tensor(double), tensor(float), tensor(int32), tensor(int64)|
|
||||
| | ||**T2** = tensor(int64), tensor(string)|
|
||||
|LinearRegressor|(*in* X:**T**, *out* Y:**tensor(float)**)|1+|**T** = tensor(float)|
|
||||
|Log|(*in* input:**T**, *out* output:**T**)|6+|**T** = tensor(float)|
|
||||
|LogSoftmax|(*in* input:**T**, *out* output:**T**)|11+|**T** = tensor(double), tensor(float)|
|
||||
| | |[1, 10]|**T** = tensor(double), tensor(float)|
|
||||
|Loop|(*in* M:**I**, *in* cond:**B**, *in* v_initial:**V**, *out* v_final_and_scan_outputs:**V**)|11+|**B** = tensor(bool)|
|
||||
| | ||**I** = tensor(int64)|
|
||||
| | ||**V** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | |[1, 10]|**B** = tensor(bool)|
|
||||
| | ||**I** = tensor(int64)|
|
||||
| | ||**V** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
|LpNormalization|(*in* input:**T**, *out* output:**T**)|1+|**T** = tensor(float)|
|
||||
|LpPool|(*in* X:**T**, *out* Y:**T**)|11+|**T** = tensor(float)|
|
||||
| | |[2, 10]|**T** = tensor(float)|
|
||||
|MatMul|(*in* A:**T**, *in* B:**T**, *out* Y:**T**)|9+|**T** = tensor(double), tensor(float), tensor(int32), tensor(int64), tensor(uint32), tensor(uint64)|
|
||||
| | |[1, 8]|**T** = tensor(double), tensor(float)|
|
||||
|MatMulInteger|(*in* A:**T1**, *in* B:**T2**, *in* a_zero_point:**T1**, *in* b_zero_point:**T2**, *out* Y:**T3**)|10+|**T1** = tensor(uint8)|
|
||||
| | ||**T2** = tensor(int8), tensor(uint8)|
|
||||
| | ||**T3** = tensor(int32)|
|
||||
|Max|(*in* data_0:**T**, *out* max:**T**)|12+|**T** = tensor(double), tensor(float), tensor(float16), tensor(int32), tensor(int64), tensor(uint32), tensor(uint64)|
|
||||
| | ||**T1** = tensor(double), tensor(float), tensor(float16), tensor(int32), tensor(int64), tensor(uint32), tensor(uint64)|
|
||||
| | |[6, 7]|**T** = tensor(float)|
|
||||
| | |[8, 11]|**T** = tensor(double), tensor(float)|
|
||||
| | ||**T1** = tensor(double), tensor(float)|
|
||||
|MaxPool|(*in* X:**T**, *out* Y:**T**) or (*in* X:**T**, *out* Y:**T**, *out* Indices:**I**)|12+|**I** = tensor(int64)|
|
||||
| | ||**T** = tensor(double), tensor(float), tensor(int8), tensor(uint8)|
|
||||
| | |[1, 7]|**T** = tensor(float)|
|
||||
| | |[8, 11]|**I** = tensor(int64)|
|
||||
| | ||**T** = tensor(double), tensor(float)|
|
||||
|MaxRoiPool|(*in* X:**T**, *in* rois:**T**, *out* Y:**T**)|1+|**T** = tensor(float)|
|
||||
|MaxUnpool|(*in* X:**T1**, *in* I:**T2**, *in* output_shape:**T2**, *out* output:**T1**)|11+|**T1** = tensor(float)|
|
||||
| | ||**T2** = tensor(int64)|
|
||||
| | |[9, 10]|**T1** = tensor(float)|
|
||||
| | ||**T2** = tensor(int64)|
|
||||
|Mean|(*in* data_0:**T**, *out* mean:**T**)|8+|**T** = tensor(float)|
|
||||
| | |[6, 7]|**T** = tensor(float)|
|
||||
|MeanVarianceNormalization|(*in* X:**T**, *out* Y:**T**) or (*in* input:**T**, *out* output:**T**)|9+|**T** = tensor(float)|
|
||||
| | |[1, 8]|**T** = tensor(float)|
|
||||
|Min|(*in* data_0:**T**, *out* min:**T**)|12+|**T** = tensor(double), tensor(float), tensor(float16), tensor(int32), tensor(int64), tensor(uint32), tensor(uint64)|
|
||||
| | ||**T1** = tensor(double), tensor(float), tensor(float16), tensor(int32), tensor(int64), tensor(uint32), tensor(uint64)|
|
||||
| | |[6, 7]|**T** = tensor(float)|
|
||||
| | |[8, 11]|**T** = tensor(float)|
|
||||
| | ||**T1** = tensor(float)|
|
||||
|Mod|(*in* A:**T**, *in* B:**T**, *out* C:**T**)|10+|**T** = tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
|Mul|(*in* A:**T**, *in* B:**T**, *out* C:**T**)|7+|**T** = tensor(double), tensor(float), tensor(int32), tensor(int64)|
|
||||
|Multinomial|(*in* input:**T1**, *out* output:**T2**)|7+|**T1** = tensor(float)|
|
||||
| | ||**T2** = tensor(int32), tensor(int64)|
|
||||
|Neg|(*in* X:**T**, *out* Y:**T**)|6+|**T** = tensor(double), tensor(float), tensor(int32), tensor(int64), tensor(int8)|
|
||||
|NonZero|(*in* X:**T**, *out* Y:**tensor(int64)**)|9+|**T** = tensor(bool), tensor(float), tensor(int32), tensor(int64), tensor(uint8)|
|
||||
|Normalizer|(*in* X:**T**, *out* Y:**tensor(float)**)|1+|**T** = tensor(double), tensor(float), tensor(int32), tensor(int64)|
|
||||
|Not|(*in* X:**T**, *out* Y:**T**)|1+|**T** = tensor(bool)|
|
||||
| | ||**T1** = tensor(bool)|
|
||||
|OneHot|(*in* indices:**T1**, *in* depth:**T2**, *in* values:**T3**, *out* output:**T3**)|11+|**T1** = tensor(float), tensor(int32), tensor(int64)|
|
||||
| | ||**T2** = tensor(float), tensor(int32), tensor(int64)|
|
||||
| | ||**T3** = tensor(float), tensor(int32), tensor(int64), tensor(string)|
|
||||
| | |[9, 10]|**T1** = tensor(float), tensor(int32), tensor(int64)|
|
||||
| | ||**T2** = tensor(float), tensor(int32), tensor(int64)|
|
||||
| | ||**T3** = tensor(float), tensor(int32), tensor(int64), tensor(string)|
|
||||
|OneHotEncoder|(*in* X:**T**, *out* Y:**tensor(float)**)|1+|**T** = tensor(double), tensor(float), tensor(int64), tensor(string)|
|
||||
|Or|(*in* A:**T**, *in* B:**T**, *out* C:**T1**)|7+|**T** = tensor(bool)|
|
||||
| | ||**T1** = tensor(bool)|
|
||||
|PRelu|(*in* X:**T**, *in* slope:**T**, *out* Y:**T**)|[7, 9]|**T** = tensor(float)|
|
||||
|Pad|(*in* data:**T**, *in* pads:**tensor(int64)**, *in* constant_value:**T**, *out* output:**T**) or (*in* data:**T**, *out* output:**T**)|11+|**T** = tensor(double), tensor(float), tensor(int32), tensor(int64)|
|
||||
| | |[2, 10]|**T** = tensor(float)|
|
||||
|ParametricSoftplus|(*in* X:**T**, *out* Y:**T**)|1+|**T** = tensor(float)|
|
||||
|Pow|(*in* X:**T**, *in* Y:**T**, *out* Z:**T**) or (*in* X:**T**, *in* Y:**T1**, *out* Z:**T**)|7+|**T** = tensor(double), tensor(float)|
|
||||
|QLinearConv|(*in* x:**T1**, *in* x_scale:**tensor(float)**, *in* x_zero_point:**T1**, *in* w:**T2**, *in* w_scale:**tensor(float)**, *in* w_zero_point:**T2**, *in* y_scale:**tensor(float)**, *in* y_zero_point:**T3**, *in* B:**T4**, *out* y:**T3**)|10+|**T1** = tensor(uint8)|
|
||||
| | ||**T2** = tensor(uint8)|
|
||||
| | ||**T3** = tensor(uint8)|
|
||||
| | ||**T4** = tensor(int32)|
|
||||
|QLinearMatMul|(*in* a:**T1**, *in* a_scale:**tensor(float)**, *in* a_zero_point:**T1**, *in* b:**T2**, *in* b_scale:**tensor(float)**, *in* b_zero_point:**T2**, *in* y_scale:**tensor(float)**, *in* y_zero_point:**T3**, *out* y:**T3**)|10+|**T1** = tensor(uint8)|
|
||||
| | ||**T2** = tensor(uint8)|
|
||||
| | ||**T3** = tensor(uint8)|
|
||||
|QuantizeLinear|(*in* x:**T1**, *in* y_scale:**tensor(float)**, *in* y_zero_point:**T2**, *out* y:**T2**)|10+|**T1** = tensor(float)|
|
||||
| | ||**T2** = tensor(int8), tensor(uint8)|
|
||||
|RNN|(*in* X:**T**, *in* W:**T**, *in* R:**T**, *in* B:**T**, *in* sequence_lens:**T1**, *in* initial_h:**T**, *out* Y:**T**, *out* Y_h:**T**)|7+|**T** = tensor(float)|
|
||||
| | ||**T1** = tensor(int32)|
|
||||
|RandomNormal|(*out* output:**T**)|1+|**T** = tensor(double), tensor(float)|
|
||||
|RandomNormalLike|(*in* input:**T1**, *out* output:**T2**)|1+|**T1** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | ||**T2** = tensor(double), tensor(float)|
|
||||
|RandomUniform|(*out* output:**T**)|1+|**T** = tensor(double), tensor(float)|
|
||||
|RandomUniformLike|(*in* input:**T1**, *out* output:**T2**)|1+|**T1** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | ||**T2** = tensor(double), tensor(float)|
|
||||
|Range|(*in* start:**T**, *in* limit:**T**, *in* delta:**T**, *out* output:**T**)|11+|**T** = tensor(double), tensor(float), tensor(int16), tensor(int32), tensor(int64)|
|
||||
|Reciprocal|(*in* X:**T**, *out* Y:**T**)|6+|**T** = tensor(float)|
|
||||
|ReduceL1|(*in* data:**T**, *out* reduced:**T**)|11+|**T** = tensor(float), tensor(int32)|
|
||||
| | |[1, 10]|**T** = tensor(float), tensor(int32)|
|
||||
|ReduceL2|(*in* data:**T**, *out* reduced:**T**)|11+|**T** = tensor(float), tensor(int32)|
|
||||
| | |[1, 10]|**T** = tensor(float), tensor(int32)|
|
||||
|ReduceLogSum|(*in* data:**T**, *out* reduced:**T**)|11+|**T** = tensor(float), tensor(int32)|
|
||||
| | |[1, 10]|**T** = tensor(float), tensor(int32)|
|
||||
|ReduceLogSumExp|(*in* data:**T**, *out* reduced:**T**)|11+|**T** = tensor(float), tensor(int32)|
|
||||
| | |[1, 10]|**T** = tensor(float), tensor(int32)|
|
||||
|ReduceMax|(*in* data:**T**, *out* reduced:**T**)|11+|**T** = tensor(float), tensor(int32), tensor(int64)|
|
||||
| | |12+|**T** = tensor(float), tensor(int32), tensor(int64), tensor(int8), tensor(uint8)|
|
||||
| | |[1, 10]|**T** = tensor(float), tensor(int32), tensor(int64)|
|
||||
|ReduceMean|(*in* data:**T**, *out* reduced:**T**)|11+|**T** = tensor(float), tensor(int32)|
|
||||
| | |[1, 10]|**T** = tensor(float), tensor(int32)|
|
||||
|ReduceMin|(*in* data:**T**, *out* reduced:**T**)|11+|**T** = tensor(float), tensor(int32), tensor(int64)|
|
||||
| | |12+|**T** = tensor(float), tensor(int32), tensor(int64), tensor(int8), tensor(uint8)|
|
||||
| | |[1, 10]|**T** = tensor(float), tensor(int32), tensor(int64)|
|
||||
|ReduceProd|(*in* data:**T**, *out* reduced:**T**)|11+|**T** = tensor(float), tensor(int32), tensor(int64)|
|
||||
| | |[1, 10]|**T** = tensor(float), tensor(int32), tensor(int64)|
|
||||
|ReduceSum|(*in* data:**T**, *out* reduced:**T**)|11+|**T** = tensor(float), tensor(int32), tensor(int64)|
|
||||
| | |[1, 10]|**T** = tensor(float), tensor(int32), tensor(int64)|
|
||||
|ReduceSumSquare|(*in* data:**T**, *out* reduced:**T**)|11+|**T** = tensor(float), tensor(int32)|
|
||||
| | |[1, 10]|**T** = tensor(float), tensor(int32)|
|
||||
|Relu|(*in* X:**T**, *out* Y:**T**)|6+|**T** = tensor(float)|
|
||||
|Reshape|(*in* data:**T**, *in* shape:**tensor(int64)**, *out* reshaped:**T**) or (*in* data:**T**, *out* reshaped:**T**)|5+|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | ||**shape** = tensor(int64)|
|
||||
|Reshape_1||[1, 4]|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
|Resize|(*in* X:**T**, *in* scales:**tensor(float)**, *out* Y:**T**) or (*in* X:**T1**, *in* roi:**T2**, *in* scales:**tensor(float)**, *in* sizes:**tensor(int64)**, *out* Y:**T1**)|11+|**T1** = tensor(float), tensor(int32), tensor(uint8)|
|
||||
| | |[10, 10]|**T** = tensor(float), tensor(int32), tensor(uint8)|
|
||||
|ReverseSequence|(*in* input:**T**, *in* sequence_lens:**tensor(int64)**, *out* Y:**T**)|10+|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
|RoiAlign|(*in* X:**T1**, *in* rois:**T1**, *in* batch_indices:**T2**, *out* Y:**T1**)|10+|**T** = tensor(double), tensor(float)|
|
||||
| | ||**T2** = tensor(int64)|
|
||||
|Round|(*in* X:**T**, *out* Y:**T**)|11+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|SVMClassifier|(*in* X:**T1**, *out* Y:**T2**, *out* Z:**tensor(float)**)|1+|**T1** = tensor(double), tensor(float), tensor(int32), tensor(int64)|
|
||||
| | ||**T2** = tensor(int64), tensor(string)|
|
||||
|SVMRegressor|(*in* X:**T**, *out* Y:**tensor(float)**)|1+|**T** = tensor(float)|
|
||||
|Scale|(*in* input:**T**, *out* output:**T**)|1+|**T** = tensor(float)|
|
||||
|ScaledTanh|(*in* input:**T**, *out* output:**T**)|1+|**T** = tensor(float)|
|
||||
|Scaler|(*in* X:**T**, *out* Y:**tensor(float)**)|1+|**T** = tensor(double), tensor(float), tensor(int32), tensor(int64)|
|
||||
|Scan|(*in* initial_state_and_scan_inputs:**V**, *out* final_state_and_scan_outputs:**V**) or (*in* sequence_lens:**I**, *in* initial_state_and_scan_inputs:**V**, *out* final_state_and_scan_outputs:**V**)|11+|**I** = tensor(int64)|
|
||||
| | ||**V** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | |[8, 8]|**I** = tensor(int64)|
|
||||
| | ||**V** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | |[9, 10]|**I** = tensor(int64)|
|
||||
| | ||**V** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
|Scatter|(*in* data:**T**, *in* indices:**Tind**, *in* updates:**T**, *out* output:**T**)|[9, 10]|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | ||**Tind** = tensor(int32), tensor(int64)|
|
||||
|ScatterElements|(*in* data:**T**, *in* indices:**Tind**, *in* updates:**T**, *out* output:**T**)|11+|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | ||**Tind** = tensor(int32), tensor(int64)|
|
||||
|ScatterND|(*in* data:**T**, *in* indices:**tensor(int64)**, *in* updates:**T**, *out* output:**T**)|11+|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | ||**Tind** = tensor(int64)|
|
||||
|Selu|(*in* X:**T**, *out* Y:**T**)|6+|**T** = tensor(float)|
|
||||
|SequenceAt|(*in* input_sequence:**S**, *in* position:**I**, *out* tensor:**T**)|11+|**I** = tensor(int32), tensor(int64)|
|
||||
| | ||**S** = seq(tensor(bfloat16)), seq(tensor(bool)), seq(tensor(double)), seq(tensor(float)), seq(tensor(float16)), seq(tensor(int16)), seq(tensor(int32)), seq(tensor(int64)), seq(tensor(int8)), seq(tensor(string)), seq(tensor(uint16)), seq(tensor(uint32)), seq(tensor(uint64)), seq(tensor(uint8))|
|
||||
| | ||**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
|SequenceConstruct|(*in* inputs:**T**, *out* output_sequence:**S**)|11+|**S** = seq(tensor(bfloat16)), seq(tensor(bool)), seq(tensor(double)), seq(tensor(float)), seq(tensor(float16)), seq(tensor(int16)), seq(tensor(int32)), seq(tensor(int64)), seq(tensor(int8)), seq(tensor(string)), seq(tensor(uint16)), seq(tensor(uint32)), seq(tensor(uint64)), seq(tensor(uint8))|
|
||||
| | ||**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
|SequenceEmpty|(*out* output:**S**)|11+|**S** = seq(tensor(bfloat16)), seq(tensor(bool)), seq(tensor(double)), seq(tensor(float)), seq(tensor(float16)), seq(tensor(int16)), seq(tensor(int32)), seq(tensor(int64)), seq(tensor(int8)), seq(tensor(string)), seq(tensor(uint16)), seq(tensor(uint32)), seq(tensor(uint64)), seq(tensor(uint8))|
|
||||
|SequenceErase|(*in* input_sequence:**S**, *in* position:**I**, *out* output_sequence:**S**)|11+|**I** = tensor(int32), tensor(int64)|
|
||||
| | ||**S** = seq(tensor(bfloat16)), seq(tensor(bool)), seq(tensor(double)), seq(tensor(float)), seq(tensor(float16)), seq(tensor(int16)), seq(tensor(int32)), seq(tensor(int64)), seq(tensor(int8)), seq(tensor(string)), seq(tensor(uint16)), seq(tensor(uint32)), seq(tensor(uint64)), seq(tensor(uint8))|
|
||||
|SequenceInsert|(*in* input_sequence:**S**, *in* tensor:**T**, *in* position:**I**, *out* output_sequence:**S**)|11+|**I** = tensor(int32), tensor(int64)|
|
||||
| | ||**S** = seq(tensor(bfloat16)), seq(tensor(bool)), seq(tensor(double)), seq(tensor(float)), seq(tensor(float16)), seq(tensor(int16)), seq(tensor(int32)), seq(tensor(int64)), seq(tensor(int8)), seq(tensor(string)), seq(tensor(uint16)), seq(tensor(uint32)), seq(tensor(uint64)), seq(tensor(uint8))|
|
||||
|SequenceLength|(*in* input_sequence:**S**, *out* length:**I**)|11+|**I** = tensor(int64)|
|
||||
| | ||**S** = seq(tensor(bfloat16)), seq(tensor(bool)), seq(tensor(double)), seq(tensor(float)), seq(tensor(float16)), seq(tensor(int16)), seq(tensor(int32)), seq(tensor(int64)), seq(tensor(int8)), seq(tensor(string)), seq(tensor(uint16)), seq(tensor(uint32)), seq(tensor(uint64)), seq(tensor(uint8))|
|
||||
|Shape|(*in* data:**T**, *out* shape:**T1**)|1+|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | ||**T1** = tensor(int64)|
|
||||
|Shrink|(*in* input:**T**, *out* output:**T**)|9+|**T** = tensor(bfloat16), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
|Sigmoid|(*in* X:**T**, *out* Y:**T**)|6+|**T** = tensor(float)|
|
||||
|Sign|(*in* input:**T**, *out* output:**T**)|9+|**T** = tensor(bfloat16), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
|Sin|(*in* input:**T**, *out* output:**T**)|7+|**T** = tensor(double), tensor(float)|
|
||||
|Sinh|(*in* input:**T**, *out* output:**T**)|9+|**T** = tensor(float)|
|
||||
|Size|(*in* data:**T**, *out* size:**T1**)|1+|**T** = tensor(bool), tensor(double), tensor(float), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | ||**T1** = tensor(int64)|
|
||||
|Slice|(*in* data:**T**, *in* starts:**Tind**, *in* ends:**Tind**, *in* axes:**Tind**, *in* steps:**Tind**, *out* output:**T**) or (*in* data:**T**, *out* output:**T**)|11+|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | ||**Tind** = tensor(int32), tensor(int64)|
|
||||
| | |[1, 9]|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | |[10, 10]|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | ||**Tind** = tensor(int32), tensor(int64)|
|
||||
|Softmax|(*in* input:**T**, *out* output:**T**)|11+|**T** = tensor(double), tensor(float)|
|
||||
| | |[1, 10]|**T** = tensor(double), tensor(float)|
|
||||
|Softplus|(*in* X:**T**, *out* Y:**T**)|1+|**T** = tensor(float)|
|
||||
|Softsign|(*in* input:**T**, *out* output:**T**)|1+|**T** = tensor(float)|
|
||||
|SpaceToDepth|(*in* input:**T**, *out* output:**T**)|1+|**T** = tensor(float)|
|
||||
|Split|(*in* input:**T**, *in* split:**T**, *out* outputs...:**T**) or (*in* input:**T**, *out* outputs:**T**)|11+|**T** = tensor(float), tensor(int32), tensor(int64), tensor(string)|
|
||||
| | |[2, 10]|**T** = tensor(float), tensor(int32), tensor(int64), tensor(string)|
|
||||
|SplitToSequence|(*in* input:**T**, *in* split:**I**, *out* output_sequence:**S**)|11+|**I** = tensor(int32), tensor(int64)|
|
||||
| | ||**S** = seq(tensor(bfloat16)), seq(tensor(bool)), seq(tensor(double)), seq(tensor(float)), seq(tensor(float16)), seq(tensor(int16)), seq(tensor(int32)), seq(tensor(int64)), seq(tensor(int8)), seq(tensor(string)), seq(tensor(uint16)), seq(tensor(uint32)), seq(tensor(uint64)), seq(tensor(uint8))|
|
||||
| | ||**T** = tensor(double), tensor(float), tensor(int32), tensor(string)|
|
||||
|Sqrt|(*in* X:**T**, *out* Y:**T**)|6+|**T** = tensor(double), tensor(float)|
|
||||
|Squeeze|(*in* data:**T**, *out* squeezed:**T**)|11+|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | |[1, 10]|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
|StringNormalizer|(*in* X:**tensor(string)**, *out* Y:**tensor(string)**)|10+|**T** = tensor(string)|
|
||||
|Sub|(*in* A:**T**, *in* B:**T**, *out* C:**T**)|7+|**T** = tensor(double), tensor(float), tensor(int32), tensor(int64)|
|
||||
|Sum|(*in* data_0:**T**, *out* sum:**T**)|8+|**T** = tensor(float)|
|
||||
| | |[6, 7]|**T** = tensor(float)|
|
||||
|Tan|(*in* input:**T**, *out* output:**T**)|7+|**T** = tensor(float)|
|
||||
|Tanh|(*in* input:**T**, *out* output:**T**)|6+|**T** = tensor(float)|
|
||||
|TfIdfVectorizer|(*in* X:**T**, *out* Y:**T1**)|9+|**T** = tensor(int32), tensor(int64), tensor(string)|
|
||||
| | ||**T1** = tensor(float)|
|
||||
|ThresholdedRelu|(*in* X:**T**, *out* Y:**T**)|10+|**T** = tensor(float)|
|
||||
| | |[1, 9]|**T** = tensor(float)|
|
||||
|Tile|(*in* input:**T**, *in* repeats:**T1**, *out* output:**T**) or (*in* input:**T**, *in* tiles:**T**, *in* axis:**T**, *out* output:**T**)|6+|**T** = tensor(bool), tensor(double), tensor(float), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | ||**T1** = tensor(int64)|
|
||||
|TopK|(*in* X:**T**, *in* K:**tensor(int64)**, *out* Values:**T**, *out* Indices:**I**) or (*in* X:**T**, *out* Values:**T**, *out* Indices:**I**)|11+|**I** = tensor(int64)|
|
||||
| | ||**T** = tensor(float), tensor(int64)|
|
||||
| | |[1, 9]|**I** = tensor(int64)|
|
||||
| | ||**T** = tensor(float)|
|
||||
| | |[10, 10]|**I** = tensor(int64)|
|
||||
| | ||**T** = tensor(float)|
|
||||
|Transpose|(*in* data:**T**, *out* transposed:**T**)|1+|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
|TreeEnsembleClassifier|(*in* X:**T1**, *out* Y:**T2**, *out* Z:**tensor(float)**)|1+|**T1** = tensor(double), tensor(float), tensor(int32), tensor(int64)|
|
||||
| | ||**T2** = tensor(int64), tensor(string)|
|
||||
|TreeEnsembleRegressor|(*in* X:**T**, *out* Y:**tensor(float)**)|1+|**T** = tensor(double), tensor(float)|
|
||||
|Unique|(*in* X:**T**, *out* Y:**T**, *out* indices:**tensor(int64)**, *out* inverse_indices:**tensor(int64)**, *out* counts:**tensor(int64)**)|11+|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
|Unsqueeze|(*in* data:**T**, *out* expanded:**T**)|11+|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | |[1, 10]|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
|Upsample|(*in* X:**T**, *in* scales:**tensor(float)**, *out* Y:**T**) or (*in* X:**T**, *out* Y:**T**)|[7, 9]|**T** = tensor(float), tensor(int32), tensor(uint8)|
|
||||
|Where|(*in* condition:**B**, *in* X:**T**, *in* Y:**T**, *out* output:**T**)|9+|**T** = tensor(float), tensor(int32), tensor(int64), tensor(string), tensor(uint8)|
|
||||
|Xor|(*in* A:**T**, *in* B:**T**, *out* C:**T1**)|7+|**T** = tensor(bool)|
|
||||
| | ||**T1** = tensor(bool)|
|
||||
|ZipMap|(*in* X:**tensor(float)**, *out* Z:**T**)|1+|**T** = seq(map(int64,tensor(float))), seq(map(string,tensor(float)))|
|
||||
| |
|
||||
| |
|
||||
**Operator Domain:** *com.microsoft*
|
||||
|Attention|(*in* input:**T**, *in* weight:**T**, *in* bias:**T**, *in* mask_index:**M**, *out* output:**T**)|1+|**T** = tensor(float)|
|
||||
|AttnLSTM|(*in* X:**T**, *in* W:**T**, *in* R:**T**, *in* B:**T**, *in* sequence_lens:**T1**, *in* initial_h:**T**, *in* initial_c:**T**, *in* P:**T**, *in* QW:**T**, *in* MW:**T**, *in* V:**T**, *in* M:**T**, *in* memory_seq_lens:**T1**, *in* AW:**T**, *out* Y:**T**, *out* Y_h:**T**, *out* Y_c:**T**)|1+|**T** = tensor(double), tensor(float)|
|
||||
| | ||**T1** = tensor(int32)|
|
||||
|BiasGelu|(*in* A:**T**, *in* B:**T**, *out* C:**T**)|1+|**T** = tensor(float)|
|
||||
|CDist|(*in* A:**T**, *in* B:**T**, *out* C:**T**)|1+|**T** = tensor(double), tensor(float)|
|
||||
|ConvTransposeWithDynamicPads|(*in* X:**T**, *in* W:**T**, *in* Pads:**tensor(int64)**, *in* B:**T**, *out* Y:**T**)|1+|**T** = tensor(float)|
|
||||
|CropAndResize|(*in* X:**T1**, *in* rois:**T1**, *in* batch_indices:**T2**, *in* crop_size:**T2**, *out* Y:**T1**)|1+|**T** = tensor(float)|
|
||||
| | ||**T2** = tensor(int32)|
|
||||
|DequantizeLinear|(*in* x:**T1**, *in* x_scale:**T2**, *in* x_zero_point:**T1**, *out* y:**T2**)|1+|**T1** = tensor(int8), tensor(uint8)|
|
||||
| | ||**T2** = tensor(float)|
|
||||
|EmbedLayerNormalization|(*in* input_ids:**T1**, *in* segment_ids:**T1**, *in* word_embedding:**T**, *in* position_embedding:**T**, *in* segment_embedding:**T**, *in* gamma:**T**, *in* beta:**T**, *in* mask:**T1**, *out* output:**T**, *out* mask_index:**T1**)|1+|**T** = tensor(float)|
|
||||
|ExpandDims|(*in* X:**T**, *in* axis:**tensor(int32)**, *out* Y:**T**)|1+|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | ||**axis** = tensor(int32)|
|
||||
|FastGelu|(*in* X:**T**, *in* bias:**T**, *out* Y:**T**)|1+|**T** = tensor(float)|
|
||||
|FusedConv|(*in* X:**T**, *in* W:**T**, *in* B:**T**, *out* Y:**T**)|1+|**T** = tensor(float)|
|
||||
|FusedGemm|(*in* A:**T**, *in* B:**T**, *in* C:**T**, *out* Y:**T**)|1+|**T** = tensor(float)|
|
||||
|GatherND|(*in* data:**T**, *in* indices:**Tind**, *out* output:**T**)|1+|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | ||**Tind** = tensor(int32), tensor(int64)|
|
||||
|Gelu|(*in* X:**T**, *out* Y:**T**)|1+|**T** = tensor(float)|
|
||||
|MatMulInteger16|(*in* A:**T1**, *in* B:**T2**, *out* Y:**T3**)|1+|**T1** = tensor(int16)|
|
||||
| | ||**T2** = tensor(int16)|
|
||||
| | ||**T3** = tensor(int32)|
|
||||
|MaxpoolWithMask|(*in* X:**T**, *in* M:**tensor(int32)**, *out* Y:**T**)|1+|**X** = tensor(float)|
|
||||
|MurmurHash3|(*in* X:**T1**, *out* Y:**T2**)|1+|**T1** = tensor(int32), tensor(string), tensor(uint32)|
|
||||
| | ||**T2** = tensor(int32), tensor(uint32)|
|
||||
|Pad|(*in* data:**T**, *in* pads:**tensor(int64)**, *in* value:**T**, *out* output:**T**)|1+|**T** = tensor(float)|
|
||||
|QuantizeLinear|(*in* x:**T1**, *in* y_scale:**T1**, *in* y_zero_point:**T2**, *out* y:**T2**)|1+|**T1** = tensor(float)|
|
||||
| | ||**T2** = tensor(int8), tensor(uint8)|
|
||||
|Range|(*in* start:**T**, *in* limit:**T**, *in* delta:**T**, *out* Y:**T**)|1+|**T** = tensor(double), tensor(float), tensor(int16), tensor(int32), tensor(int64)|
|
||||
|SampleOp|(*in* X:**T**, *out* Y:**T**)|1+|**T** = tensor(float)|
|
||||
|SkipLayerNormalization|(*in* input:**T**, *in* skip:**T**, *in* gamma:**T**, *in* beta:**T**, *in* bias:**T**, *out* output:**T**, *out* mean:**U**, *out* inv_std_var:**U**)|1+|**T** = tensor(double), tensor(float)|
|
||||
|Tokenizer|(*in* X:**T**, *out* Y:**T**)|1+|**T** = tensor(string)|
|
||||
|Unique|(*in* x:**T**, *out* y:**T**, *out* idx:**tensor(int64)**, *out* counts:**tensor(int64)**)|1+|**T** = tensor(float)|
|
||||
|WordConvEmbedding|(*in* Sequence:**T**, *in* W:**T1**, *in* B:**T1**, *in* C:**T1**, *out* Y:**T1**)|1+|**T** = tensor(int32)|
|
||||
| | ||**T1** = tensor(float)|
|
||||
| |
|
||||
| |
|
||||
**Operator Domain:** *com.microsoft.nchwc*
|
||||
|AveragePool|(*in* X:**T**, *out* Y:**T**)|1+|**T** = tensor(float)|
|
||||
|Conv|(*in* X:**T**, *in* W:**T**, *in* B:**T**, *in* Sum:**T**, *out* Y:**T**)|1+|**T** = tensor(float)|
|
||||
|GlobalAveragePool|(*in* X:**T**, *out* Y:**T**)|1+|**T** = tensor(float)|
|
||||
|GlobalMaxPool|(*in* X:**T**, *out* Y:**T**)|1+|**T** = tensor(float)|
|
||||
|MaxPool|(*in* X:**T**, *out* Y:**T**)|1+|**T** = tensor(float)|
|
||||
|ReorderInput|(*in* X:**T**, *out* Y:**T**)|1+|**T** = tensor(float)|
|
||||
|ReorderOutput|(*in* X:**T**, *out* Y:**T**)|1+|**T** = tensor(float)|
|
||||
|Upsample|(*in* X:**T**, *out* Y:**T**)|1+|**T** = tensor(float)|
|
||||
| |
|
||||
| |
|
||||
|
||||
|
||||
## Operators implemented by CUDAExecutionProvider
|
||||
|
||||
| Op Name | Parameters | OpSet Version | Types Supported |
|
||||
|---------|------------|---------------|-----------------|
|
||||
**Operator Domain:** *ai.onnx.ml*
|
||||
|Abs|(*in* X:**T**, *out* Y:**T**)|6+|**T** = tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
|Add|(*in* A:**T**, *in* B:**T**, *out* C:**T**)|7+|**T** = tensor(double), tensor(float), tensor(float16), tensor(int32), tensor(int64), tensor(uint32), tensor(uint64)|
|
||||
|Affine|(*in* X:**T**, *out* Y:**T**)|1+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|And|(*in* A:**T**, *in* B:**T**, *out* C:**T1**)|7+|**T** = tensor(bool)|
|
||||
| | ||**T1** = tensor(bool)|
|
||||
|ArgMax|(*in* data:**T**, *out* reduced:**tensor(int64)**)|11+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
| | |[1, 10]|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|ArgMin|(*in* data:**T**, *out* reduced:**tensor(int64)**)|11+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
| | |[1, 10]|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|AveragePool|(*in* X:**T**, *out* Y:**T**)|11+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
| | |[10, 10]|**I** = tensor(int64)|
|
||||
| | ||**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
| | |[7, 9]|**I** = tensor(int64)|
|
||||
| | ||**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|BatchNormalization|(*in* X:**T**, *in* scale:**T**, *in* B:**T**, *in* mean:**T**, *in* var:**T**, *in* training_mode:**T1**, *out* Y:**T**, *out* output_mean:**T**, *out* output_var:**T**, *out* saved_mean:**T**, *out* saved_var:**T**) or (*in* X:**T**, *in* scale:**T**, *in* B:**T**, *in* mean:**T**, *in* var:**T**, *out* Y:**T**, *out* mean:**T**, *out* var:**T**, *out* saved_mean:**T**, *out* saved_var:**T**)|9+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
| | |[7, 8]|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|Cast|(*in* input:**T1**, *out* output:**T2**)|9+|**T1** = tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | ||**T2** = tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | |[6, 8]|**T1** = tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | ||**T2** = tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
|Ceil|(*in* X:**T**, *out* Y:**T**)|6+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|Clip|(*in* input:**T**, *in* min:**T**, *in* max:**T**, *out* output:**T**) or (*in* input:**T**, *out* output:**T**)|12+|**T** = tensor(double), tensor(float), tensor(int64), tensor(int8), tensor(uint64), tensor(uint8)|
|
||||
| | |[11, 11]|**T** = tensor(float)|
|
||||
| | |[6, 10]|**T** = tensor(float)|
|
||||
|Compress|(*in* input:**T**, *in* condition:**T1**, *out* output:**T**)|11+|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | ||**T1** = tensor(bool)|
|
||||
| | |[9, 10]|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | ||**T1** = tensor(bool)|
|
||||
|Concat|(*in* inputs:**T**, *out* concat_result:**T**)|11+|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | |[4, 10]|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
|ConstantOfShape|(*in* input:**T1**, *out* output:**T2**)|9+|**T1** = tensor(int64)|
|
||||
| | ||**T2** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
|Conv|(*in* X:**T**, *in* W:**T**, *in* B:**T**, *out* Y:**T**)|11+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
| | |[1, 10]|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|ConvTranspose|(*in* X:**T**, *in* W:**T**, *in* B:**T**, *out* Y:**T**)|11+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
| | |[1, 10]|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|Crop|(*in* input:**T**, *out* output:**T**)|1+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|CumSum|(*in* x:**T**, *in* axis:**T2**, *out* y:**T**)|11+|**T** = tensor(double), tensor(float), tensor(float16), tensor(int32), tensor(int64), tensor(uint32), tensor(uint64)|
|
||||
| | ||**T2** = tensor(int32), tensor(int64)|
|
||||
|DequantizeLinear|(*in* x:**T**, *in* x_scale:**tensor(float)**, *in* x_zero_point:**T**, *out* y:**tensor(float)**)|10+|**T** = tensor(int8), tensor(uint8)|
|
||||
|Div|(*in* A:**T**, *in* B:**T**, *out* C:**T**)|7+|**T** = tensor(double), tensor(float), tensor(float16), tensor(int32), tensor(int64), tensor(uint32), tensor(uint64)|
|
||||
|Dropout|(*in* data:**T**, *in* ratio:**T1**, *out* output:**T**, *out* mask:**T2**) or (*in* data:**T**, *out* output:**T**, *out* mask:**T**) or (*in* data:**T**, *out* output:**T**, *out* mask:**T1**)|10+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
| | ||**T1** = tensor(bool)|
|
||||
| | |[7, 9]|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|DynamicSlice|(*in* data:**T**, *in* starts:**Tind**, *in* ends:**Tind**, *in* axes:**Tind**, *out* output:**T**)|1+|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | ||**Tind** = tensor(int32), tensor(int64)|
|
||||
|Elu|(*in* X:**T**, *out* Y:**T**)|6+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|Equal|(*in* A:**T**, *in* B:**T**, *out* C:**T1**)|11+|**T** = tensor(bool), tensor(int32), tensor(int64)|
|
||||
| | |[7, 10]|**T** = tensor(bool), tensor(int32), tensor(int64)|
|
||||
|Erf|(*in* input:**T**, *out* output:**T**)|9+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|Exp|(*in* input:**T**, *out* output:**T**)|6+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|Expand|(*in* input:**T**, *in* shape:**tensor(int64)**, *out* output:**T**)|8+|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
|EyeLike|(*in* input:**T1**, *out* output:**T2**)|9+|**T1** = tensor(double), tensor(float), tensor(int32), tensor(int64), tensor(uint64)|
|
||||
| | ||**T2** = tensor(double), tensor(float), tensor(int32), tensor(int64), tensor(uint64)|
|
||||
|Flatten|(*in* input:**T**, *out* output:**T**)|11+|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | |[1, 8]|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | |[9, 10]|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
|Floor|(*in* X:**T**, *out* Y:**T**)|6+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|GRU|(*in* X:**T**, *in* W:**T**, *in* R:**T**, *in* B:**T**, *in* sequence_lens:**T1**, *in* initial_h:**T**, *out* Y:**T**, *out* Y_h:**T**)|7+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
| | ||**T1** = tensor(int32)|
|
||||
|Gather|(*in* data:**T**, *in* indices:**Tind**, *out* output:**T**)|11+|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | ||**Tind** = tensor(int32), tensor(int64)|
|
||||
| | |[1, 10]|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | ||**Tind** = tensor(int32), tensor(int64)|
|
||||
|GatherElements|(*in* data:**T**, *in* indices:**Tind**, *out* output:**T**)|11+|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | ||**Tind** = tensor(int32), tensor(int64)|
|
||||
|Gemm|(*in* A:**T**, *in* B:**T**, *in* C:**T**, *out* Y:**T**)|11+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
| | |[7, 8]|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
| | |[9, 10]|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|GlobalAveragePool|(*in* X:**T**, *out* Y:**T**)|1+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|GlobalMaxPool|(*in* X:**T**, *out* Y:**T**)|1+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|Greater|(*in* A:**T**, *in* B:**T**, *out* C:**T1**)|9+|**T** = tensor(double), tensor(float), tensor(float16), tensor(int32), tensor(int64), tensor(uint32), tensor(uint64)|
|
||||
| | ||**T1** = tensor(bool)|
|
||||
| | |[7, 8]|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|HardSigmoid|(*in* X:**T**, *out* Y:**T**)|6+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|Identity|(*in* input:**T**, *out* output:**T**)|1+|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
|If|(*in* cond:**B**, *out* outputs:**V**)|11+|**B** = tensor(bool)|
|
||||
| | ||**V** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | |[1, 10]|**B** = tensor(bool)|
|
||||
| | ||**V** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
|ImageScaler|(*in* input:**T**, *out* output:**T**)|1+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|InstanceNormalization|(*in* input:**T**, *in* scale:**T**, *in* B:**T**, *out* output:**T**)|6+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|LRN|(*in* X:**T**, *out* Y:**T**)|1+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|LSTM|(*in* X:**T**, *in* W:**T**, *in* R:**T**, *in* B:**T**, *in* sequence_lens:**T1**, *in* initial_h:**T**, *in* initial_c:**T**, *in* P:**T**, *out* Y:**T**, *out* Y_h:**T**, *out* Y_c:**T**)|7+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
| | ||**T1** = tensor(int32)|
|
||||
|LayerNormalization|(*in* X:**T**, *in* scale:**T**, *in* B:**T**, *out* Y:**T**, *out* mean:**U**, *out* inv_std_var:**U**)|1+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
| | ||**U** = tensor(float)|
|
||||
|LeakyRelu|(*in* X:**T**, *out* Y:**T**)|6+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|Less|(*in* A:**T**, *in* B:**T**, *out* C:**T1**)|9+|**T** = tensor(double), tensor(float), tensor(float16), tensor(int32), tensor(int64), tensor(uint32), tensor(uint64)|
|
||||
| | ||**T1** = tensor(bool)|
|
||||
| | |[7, 8]|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|Log|(*in* input:**T**, *out* output:**T**)|6+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|Loop|(*in* M:**I**, *in* cond:**B**, *in* v_initial:**V**, *out* v_final_and_scan_outputs:**V**)|11+|**B** = tensor(bool)|
|
||||
| | ||**I** = tensor(int64)|
|
||||
| | ||**V** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | |[1, 10]|**B** = tensor(bool)|
|
||||
| | ||**I** = tensor(int64)|
|
||||
| | ||**V** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
|MatMul|(*in* A:**T**, *in* B:**T**, *out* Y:**T**)|9+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
| | |[1, 8]|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|MatMulInteger|(*in* A:**T1**, *in* B:**T2**, *in* a_zero_point:**T1**, *in* b_zero_point:**T2**, *out* Y:**T3**)|10+|**T1** = tensor(int8)|
|
||||
| | ||**T2** = tensor(int8)|
|
||||
| | ||**T3** = tensor(int32)|
|
||||
|Max|(*in* data_0:**T**, *out* max:**T**)|12+|**T** = tensor(double), tensor(float), tensor(float16), tensor(int32), tensor(int64), tensor(uint32), tensor(uint64)|
|
||||
| | |[6, 7]|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
| | |[8, 11]|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|MaxPool|(*in* X:**T**, *out* Y:**T**) or (*in* X:**T**, *out* Y:**T**, *out* Indices:**I**)|12+|**T** = tensor(double), tensor(float), tensor(float16), tensor(int8), tensor(uint8)|
|
||||
| | |[1, 7]|**I** = tensor(int64)|
|
||||
| | ||**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
| | |[10, 10]|**I** = tensor(int64)|
|
||||
| | ||**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
| | |[11, 11]|**I** = tensor(int64)|
|
||||
| | ||**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
| | |[8, 9]|**I** = tensor(int64)|
|
||||
| | ||**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|MemcpyFromHost|(*in* X:**T**, *out* Y:**T**)|1+|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
|MemcpyToHost|(*in* X:**T**, *out* Y:**T**)|1+|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
|Min|(*in* data_0:**T**, *out* min:**T**)|12+|**T** = tensor(double), tensor(float), tensor(float16), tensor(int32), tensor(int64), tensor(uint32), tensor(uint64)|
|
||||
| | |[6, 7]|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
| | |[8, 11]|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|Mul|(*in* A:**T**, *in* B:**T**, *out* C:**T**)|7+|**T** = tensor(double), tensor(float), tensor(float16), tensor(int32), tensor(int64), tensor(uint32), tensor(uint64)|
|
||||
|Neg|(*in* X:**T**, *out* Y:**T**)|6+|**T** = tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8)|
|
||||
|NonZero|(*in* X:**T**, *out* Y:**tensor(int64)**)|9+|**T** = tensor(bool), tensor(float), tensor(int32), tensor(int64), tensor(uint8)|
|
||||
|Not|(*in* X:**T**, *out* Y:**T**)|1+|**T** = tensor(bool)|
|
||||
| | ||**T1** = tensor(bool)|
|
||||
|Or|(*in* A:**T**, *in* B:**T**, *out* C:**T1**)|7+|**T** = tensor(bool)|
|
||||
| | ||**T1** = tensor(bool)|
|
||||
|PRelu|(*in* X:**T**, *in* slope:**T**, *out* Y:**T**)|7+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|Pad|(*in* data:**T**, *in* pads:**tensor(int64)**, *in* constant_value:**T**, *out* output:**T**) or (*in* data:**T**, *out* output:**T**)|11+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
| | |[2, 10]|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|ParametricSoftplus|(*in* X:**T**, *out* Y:**T**)|1+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|Pow|(*in* X:**T**, *in* Y:**T**, *out* Z:**T**) or (*in* X:**T**, *in* Y:**T1**, *out* Z:**T**)|7+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|QuantizeLinear|(*in* x:**T1**, *in* y_scale:**tensor(float)**, *in* y_zero_point:**T2**, *out* y:**T2**)|10+|**T1** = tensor(float)|
|
||||
| | ||**T2** = tensor(int8), tensor(uint8)|
|
||||
|RNN|(*in* X:**T**, *in* W:**T**, *in* R:**T**, *in* B:**T**, *in* sequence_lens:**T1**, *in* initial_h:**T**, *out* Y:**T**, *out* Y_h:**T**)|7+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
| | ||**T1** = tensor(int32)|
|
||||
|Range|(*in* start:**T**, *in* limit:**T**, *in* delta:**T**, *out* output:**T**)|11+|**T** = tensor(double), tensor(float), tensor(int16), tensor(int32), tensor(int64)|
|
||||
|Reciprocal|(*in* X:**T**, *out* Y:**T**)|6+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|ReduceL1|(*in* data:**T**, *out* reduced:**T**)|11+|**T** = tensor(double), tensor(float), tensor(float16), tensor(int32)|
|
||||
| | |[1, 10]|**T** = tensor(double), tensor(float), tensor(float16), tensor(int32)|
|
||||
|ReduceL2|(*in* data:**T**, *out* reduced:**T**)|11+|**T** = tensor(double), tensor(float), tensor(float16), tensor(int32)|
|
||||
| | |[1, 10]|**T** = tensor(double), tensor(float), tensor(float16), tensor(int32)|
|
||||
|ReduceLogSum|(*in* data:**T**, *out* reduced:**T**)|11+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
| | |[1, 10]|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|ReduceLogSumExp|(*in* data:**T**, *out* reduced:**T**)|11+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
| | |[1, 10]|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|ReduceMax|(*in* data:**T**, *out* reduced:**T**)|12+|**T** = tensor(double), tensor(float), tensor(float16), tensor(int32), tensor(int8), tensor(uint8)|
|
||||
| | |[1, 10]|**T** = tensor(double), tensor(float), tensor(float16), tensor(int32)|
|
||||
| | |[11, 11]|**T** = tensor(double), tensor(float), tensor(float16), tensor(int32)|
|
||||
|ReduceMean|(*in* data:**T**, *out* reduced:**T**)|11+|**T** = tensor(double), tensor(float), tensor(float16), tensor(int32)|
|
||||
| | |[1, 10]|**T** = tensor(double), tensor(float), tensor(float16), tensor(int32)|
|
||||
|ReduceMin|(*in* data:**T**, *out* reduced:**T**)|12+|**T** = tensor(double), tensor(float), tensor(float16), tensor(int32), tensor(int8), tensor(uint8)|
|
||||
| | |[1, 10]|**T** = tensor(double), tensor(float), tensor(float16), tensor(int32)|
|
||||
| | |[11, 11]|**T** = tensor(double), tensor(float), tensor(float16), tensor(int32)|
|
||||
|ReduceProd|(*in* data:**T**, *out* reduced:**T**)|11+|**T** = tensor(double), tensor(float), tensor(float16), tensor(int32)|
|
||||
| | |[1, 10]|**T** = tensor(double), tensor(float), tensor(float16), tensor(int32)|
|
||||
|ReduceSum|(*in* data:**T**, *out* reduced:**T**)|11+|**T** = tensor(double), tensor(float), tensor(float16), tensor(int32)|
|
||||
| | |[1, 10]|**T** = tensor(double), tensor(float), tensor(float16), tensor(int32)|
|
||||
|ReduceSumSquare|(*in* data:**T**, *out* reduced:**T**)|11+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
| | |[1, 10]|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|Relu|(*in* X:**T**, *out* Y:**T**)|6+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|Reshape|(*in* data:**T**, *in* shape:**tensor(int64)**, *out* reshaped:**T**) or (*in* data:**T**, *out* reshaped:**T**)|5+|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | ||**shape** = tensor(int64)|
|
||||
|Reshape_1||[1, 4]|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
|Resize|(*in* X:**T**, *in* scales:**tensor(float)**, *out* Y:**T**) or (*in* X:**T1**, *in* roi:**T2**, *in* scales:**tensor(float)**, *in* sizes:**tensor(int64)**, *out* Y:**T1**)|11+|**T1** = tensor(double), tensor(float), tensor(float16), tensor(int32), tensor(uint8)|
|
||||
| | |[10, 10]|**T** = tensor(double), tensor(float), tensor(float16), tensor(int32), tensor(uint8)|
|
||||
|ReverseSequence|(*in* input:**T**, *in* sequence_lens:**tensor(int64)**, *out* Y:**T**)|10+|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
|RoiAlign|(*in* X:**T1**, *in* rois:**T1**, *in* batch_indices:**T2**, *out* Y:**T1**)|10+|**T** = tensor(double), tensor(float)|
|
||||
| | ||**T2** = tensor(int64)|
|
||||
|Round|(*in* X:**T**, *out* Y:**T**)|11+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|ScaledTanh|(*in* input:**T**, *out* output:**T**)|1+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|Scan|(*in* initial_state_and_scan_inputs:**V**, *out* final_state_and_scan_outputs:**V**) or (*in* sequence_lens:**I**, *in* initial_state_and_scan_inputs:**V**, *out* final_state_and_scan_outputs:**V**)|11+|**I** = tensor(int64)|
|
||||
| | ||**V** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | |[8, 8]|**I** = tensor(int64)|
|
||||
| | ||**V** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(string), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | |[9, 10]|**I** = tensor(int64)|
|
||||
| | ||**V** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
|Scatter|(*in* data:**T**, *in* indices:**Tind**, *in* updates:**T**, *out* output:**T**)|[9, 10]|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | ||**Tind** = tensor(int32), tensor(int64)|
|
||||
|ScatterElements|(*in* data:**T**, *in* indices:**Tind**, *in* updates:**T**, *out* output:**T**)|11+|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | ||**Tind** = tensor(int32), tensor(int64)|
|
||||
|Selu|(*in* X:**T**, *out* Y:**T**)|6+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|Shape|(*in* data:**T**, *out* shape:**T1**)|1+|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | ||**T1** = tensor(int64)|
|
||||
|Shrink|(*in* input:**T**, *out* output:**T**)|9+|**T** = tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
|Sigmoid|(*in* X:**T**, *out* Y:**T**)|6+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|Slice|(*in* data:**T**, *in* starts:**Tind**, *in* ends:**Tind**, *in* axes:**Tind**, *in* steps:**Tind**, *out* output:**T**) or (*in* data:**T**, *out* output:**T**)|11+|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | ||**Tind** = tensor(float), tensor(int32), tensor(int64)|
|
||||
| | |[1, 9]|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | ||**Tind** = tensor(float), tensor(int32), tensor(int64)|
|
||||
| | |[10, 10]|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | ||**Tind** = tensor(float), tensor(int32), tensor(int64)|
|
||||
|Softmax|(*in* input:**T**, *out* output:**T**)|11+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
| | |[1, 10]|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|Softplus|(*in* X:**T**, *out* Y:**T**)|1+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|Softsign|(*in* input:**T**, *out* output:**T**)|1+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|Split|(*in* input:**T**, *in* split:**T**, *out* outputs...:**T**) or (*in* input:**T**, *out* outputs:**T**)|11+|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | |[2, 10]|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
|Sqrt|(*in* X:**T**, *out* Y:**T**)|6+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|Squeeze|(*in* data:**T**, *out* squeezed:**T**)|11+|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | |[1, 10]|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
|Sub|(*in* A:**T**, *in* B:**T**, *out* C:**T**)|7+|**T** = tensor(double), tensor(float), tensor(float16), tensor(int32), tensor(int64), tensor(uint32), tensor(uint64)|
|
||||
|Sum|(*in* data_0:**T**, *out* sum:**T**)|8+|**T** = tensor(double), tensor(float), tensor(float16), tensor(int32), tensor(int64), tensor(uint32), tensor(uint64)|
|
||||
| | |[6, 7]|**T** = tensor(double), tensor(float), tensor(float16), tensor(int32), tensor(int64), tensor(uint32), tensor(uint64)|
|
||||
|Tanh|(*in* input:**T**, *out* output:**T**)|6+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|ThresholdedRelu|(*in* X:**T**, *out* Y:**T**)|1+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
| | |10+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|Tile|(*in* input:**T**, *in* repeats:**T1**, *out* output:**T**) or (*in* input:**T**, *in* tiles:**T**, *in* axis:**T**, *out* output:**T**)|6+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
| | ||**T1** = tensor(int64)|
|
||||
|TopK|(*in* X:**T**, *in* K:**tensor(int64)**, *out* Values:**T**, *out* Indices:**I**) or (*in* X:**T**, *out* Values:**T**, *out* Indices:**I**)|11+|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | |[1, 9]|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | |[10, 10]|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
|Transpose|(*in* data:**T**, *out* transposed:**T**)|1+|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
|Unsqueeze|(*in* data:**T**, *out* expanded:**T**)|11+|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
| | |[1, 10]|**T** = tensor(bfloat16), tensor(bool), tensor(double), tensor(float), tensor(float16), tensor(int16), tensor(int32), tensor(int64), tensor(int8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(uint8)|
|
||||
|Upsample|(*in* X:**T**, *in* scales:**tensor(float)**, *out* Y:**T**) or (*in* X:**T**, *out* Y:**T**)|[7, 9]|**T** = tensor(double), tensor(float), tensor(float16), tensor(int32), tensor(uint8)|
|
||||
|Where|(*in* condition:**B**, *in* X:**T**, *in* Y:**T**, *out* output:**T**)|9+|**B** = tensor(bool)|
|
||||
| | ||**T** = tensor(float), tensor(float16), tensor(int32), tensor(int64), tensor(uint8)|
|
||||
|Xor|(*in* A:**T**, *in* B:**T**, *out* C:**T1**)|7+|**T** = tensor(bool)|
|
||||
| | ||**T1** = tensor(bool)|
|
||||
| |
|
||||
| |
|
||||
**Operator Domain:** *com.microsoft*
|
||||
|Attention|(*in* input:**T**, *in* weight:**T**, *in* bias:**T**, *in* mask_index:**M**, *out* output:**T**)|1+|**T** = tensor(float), tensor(float16)|
|
||||
|BiasGelu|(*in* A:**T**, *in* B:**T**, *out* C:**T**)|1+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|ConvTransposeWithDynamicPads|(*in* X:**T**, *in* W:**T**, *in* Pads:**tensor(int64)**, *in* B:**T**, *out* Y:**T**)|1+|**T** = tensor(float)|
|
||||
|DequantizeLinear|(*in* x:**T1**, *in* x_scale:**T2**, *in* x_zero_point:**T1**, *out* y:**T2**)|1+|**T1** = tensor(int8), tensor(uint8)|
|
||||
| | ||**T2** = tensor(float16)|
|
||||
|EmbedLayerNormalization|(*in* input_ids:**T1**, *in* segment_ids:**T1**, *in* word_embedding:**T**, *in* position_embedding:**T**, *in* segment_embedding:**T**, *in* gamma:**T**, *in* beta:**T**, *in* mask:**T1**, *out* output:**T**, *out* mask_index:**T1**)|1+|**T** = tensor(float), tensor(float16)|
|
||||
|FastGelu|(*in* X:**T**, *in* bias:**T**, *out* Y:**T**)|1+|**T** = tensor(float), tensor(float16)|
|
||||
|Gelu|(*in* X:**T**, *out* Y:**T**)|1+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|Irfft|(*in* X:**T**, *out* Y:**T**)|1+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|QuantizeLinear|(*in* x:**T1**, *in* y_scale:**T1**, *in* y_zero_point:**T2**, *out* y:**T2**)|1+|**T1** = tensor(float16)|
|
||||
| | ||**T2** = tensor(int8), tensor(uint8)|
|
||||
|Rfft|(*in* X:**T**, *out* Y:**T**)|1+|**T** = tensor(double), tensor(float), tensor(float16)|
|
||||
|SkipLayerNormalization|(*in* input:**T**, *in* skip:**T**, *in* gamma:**T**, *in* beta:**T**, *in* bias:**T**, *out* output:**T**, *out* mean:**U**, *out* inv_std_var:**U**)|1+|**T** = tensor(float), tensor(float16)|
|
||||
| |
|
||||
| |
|
||||
|
||||
|
||||
## Operators implemented by DnnlExecutionProvider
|
||||
|
||||
| Op Name | Parameters | OpSet Version | Types Supported |
|
||||
|---------|------------|---------------|-----------------|
|
||||
**Operator Domain:** *ai.onnx.ml*
|
||||
|Gemm|(*in* A:**T**, *in* B:**T**, *in* C:**T**, *out* Y:**T**)|7+|**T** = tensor(float)|
|
||||
| |
|
||||
| |
|
||||
18
docs/resources/compatibility.md
Normal file
18
docs/resources/compatibility.md
Normal file
|
|
@ -0,0 +1,18 @@
|
|||
---
|
||||
title: Operator compatibility
|
||||
parent: Resources
|
||||
nav_order: 1
|
||||
---
|
||||
|
||||
# ONNX and operator compatibility
|
||||
{: .no_toc }
|
||||
|
||||
Supporting models based on the standard [ONNX](https://onnx.ai) format, the runtime is compatible with PyTorch, scikit-learn, TensorFlow, Keras, and all other frameworks and tools that support the interoperable format.
|
||||
|
||||
* [Getting ONNX models - tutorials](https://github.com/onnx/tutorials#getting-onnx-models)
|
||||
|
||||
ONNX Runtime is up to date and backwards compatible with all operators (both DNN and traditional ML) since ONNX v1.2.1+. [(ONNX compatibility details)](https://github.com/microsoft/onnxruntime/blob/master/docs/Versioning.md). Newer versions of ONNX Runtime support all models that worked with prior versions, so updates should not break integrations.
|
||||
|
||||
* [Supported operators/types](https://github.com/microsoft/onnxruntime/blob/master/docs/OperatorKernels.md)
|
||||
* *Operators not supported in the current ONNX spec may be available as a [Contrib Operator](https://github.com/microsoft/onnxruntime/blob/master/docs/ContribOperators.md)*
|
||||
* [Extensibility: Add a custom operator/kernel](../how-to/add-custom-op.md)
|
||||
161
docs/resources/graph-optimizations.md
Normal file
161
docs/resources/graph-optimizations.md
Normal file
|
|
@ -0,0 +1,161 @@
|
|||
---
|
||||
title: Graph optimizations
|
||||
parent: Resources
|
||||
nav_order: 3
|
||||
---
|
||||
|
||||
# Graph Optimizations in ONNX Runtime
|
||||
{: .no_toc }
|
||||
|
||||
ONNX Runtime provides various graph optimizations to improve model performance. Graph optimizations are essentially graph-level transformations, ranging from small graph simplifications and node eliminations to more complex node fusions and layout optimizations.
|
||||
|
||||
Graph optimizations are divided in several categories (or *levels*) based on their complexity and functionality. They can be performed either *online* or *offline*. In online mode, the optimizations are done before performing the inference, while in offline mode, the runtime saves the optimized graph to disk. ONNX Runtime provides Python, C#, C++, and C APIs to enable different optimization levels and to choose between offline vs. online mode.
|
||||
|
||||
Below we provide details on the optimization levels, the online/offline mode, and the various APIs to control them.
|
||||
|
||||
## Contents
|
||||
{: .no_toc }
|
||||
|
||||
* TOC placeholder
|
||||
{:toc}
|
||||
|
||||
## Graph Optimization Levels
|
||||
|
||||
Graph optimizations are divided in three levels:
|
||||
* Basic
|
||||
* Extended
|
||||
* Layout Optimizations
|
||||
|
||||
The optimizations belonging to one level are performed after the optimizations of the previous level have been applied (e.g., extended optimizations are applied after basic optimizations have been applied).
|
||||
|
||||
**All optimizations are enabled by default.**
|
||||
|
||||
### Basic Graph Optimizations
|
||||
|
||||
These are semantics-preserving graph rewrites which remove redundant nodes and redundant computation. They run before graph partitioning and thus apply to all the execution providers. Available basic graph optimizations are as follows:
|
||||
|
||||
* Constant Folding: Statically computes parts of the graph that rely only on constant initializers. This eliminates the need to compute them during runtime.
|
||||
|
||||
* Redundant node eliminations: Remove all redundant nodes without changing the graph structure. The following such optimizations are currently supported:
|
||||
* Identity Elimination
|
||||
* Slice Elimination
|
||||
* Unsqueeze Elimination
|
||||
* Dropout Elimination
|
||||
|
||||
* Semantics-preserving node fusions : Fuse/fold multiple nodes into a single node. For example, Conv Add fusion folds the Add operator as the bias of the Conv operator. The following such optimizations are currently supported:
|
||||
* Conv Add Fusion
|
||||
* Conv Mul Fusion
|
||||
* Conv BatchNorm Fusion
|
||||
* Relu Clip Fusion
|
||||
* Reshape Fusion
|
||||
|
||||
### Extended Graph Optimizations
|
||||
|
||||
These optimizations include complex node fusions. They are run after graph partitioning and are only applied to the nodes assigned to the CPU or CUDA execution provider. Available extended graph optimizations are as follows:
|
||||
|
||||
| Optimization | Execution Provider | Comment |
|
||||
|---------------------------------|--------------------|-----------------------------------------------------------------------------|
|
||||
| GEMM Activation Fusion | cpu | |
|
||||
| Matmul Add Fusion | cpu | |
|
||||
| Conv Activation Fusion | cpu | |
|
||||
| GELU Fusion | cpu or cuda | |
|
||||
| Layer Normalization Fusion | cpu or cuda | |
|
||||
| BERT Embedding Layer Fusion | cpu or cuda | Fuse BERT embedding layer, layer normalization and attention mask length |
|
||||
| Attention Fusion | cpu or cuda | Attention mask has approximation in cuda execution provider |
|
||||
| Skip Layer Normalization Fusion | cpu or cuda | Fuse bias of fully connected layer, skip connection and layer normalization |
|
||||
| Bias GELU Fusion | cpu or cuda | Fuse bias of fully connected layer and GELU activation |
|
||||
| GELU Approximation | cuda | Erf is approximated by a formula using tanh function |
|
||||
|
||||
To optimize inference performance of BERT model, approximation is used in GELU approximation and Attention fusion for cuda execution provider. There might be slight difference in result. The impact on accuracy could be neglected based on our evaluation: F1 score for a BERT model on SQuAD v1.1 is almost same (87.05 vs 87.03).
|
||||
|
||||
GELU approximation is disabled by default.
|
||||
|
||||
### Layout Optimizations
|
||||
|
||||
These optimizations change the data layout for applicable nodes to achieve higher performance improvements. They are run after graph partitioning and are only applied to nodes assigned to CPU execution provider. Available layout optimizations are as follows:
|
||||
|
||||
* NCHWc Optimizer: Optimizes the graph by using NCHWc layout instead of NCHW layout.
|
||||
|
||||
## Online/Offline Mode
|
||||
|
||||
All optimizations can be performed either online or offline. In online mode, when initializing an inference session, we also apply all enabled graph optimizations before performing model inference. Applying all optimizations each time we initiate a session can add overhead to the model startup time (especially for complex models), which can be critical in production scenarios. This is where the offline mode can bring a lot of benefit. In offline mode, after performing graph optimizations, ONNX Runtime serializes the resulting model to disk. Subsequently, when new inference sessions are created for this model, we can instead use the already optimized model to reduce startup time.
|
||||
|
||||
**Notes**:
|
||||
|
||||
* When running in offline mode, make sure to use the exact same options (e.g., execution providers, optimization level) and hardware as the target machine that the model inference will run on (e.g., you cannot run a model pre-optimized for a GPU execution provider on a machine that is equipped only with CPU).
|
||||
* When layout optimizations are enabled, the offline mode can only be used on compatible hardware to the environment when the offline model is saved. For example, if model has layout optimized for AVX2, the offline model would require CPUs that support AVX2.
|
||||
|
||||
## Usage
|
||||
|
||||
### General Note
|
||||
**Levels**:
|
||||
ONNX Runtime defines the `GraphOptimizationLevel` enum to determine which of the aforementioned optimization levels will be enabled. Choosing a level enables the optimizations of that level, as well as the optimizations of all preceding levels. For example, enabling Extended optimizations, also enables Basic optimizations. The mapping of these levels to the enum is as follows:
|
||||
|
||||
* GraphOptimizationLevel::ORT_DISABLE_ALL -> Disables all optimizations
|
||||
* GraphOptimizationLevel::ORT_ENABLE_BASIC -> Enables basic optimizations
|
||||
* GraphOptimizationLevel::ORT_ENABLE_EXTENDED -> Enables basic and extended optimizations
|
||||
* GraphOptimizationLevel::ORT_ENABLE_ALL -> Enables all available optimizations including layout optimizations
|
||||
|
||||
**Online/Offline Mode**:
|
||||
To enable serialization of the optimized model to disk, set the SessionOptions option `optimized_model_path` to the desired path where the optimized model will be stored.
|
||||
|
||||
### Python API Usage
|
||||
```python
|
||||
import onnxruntime as rt
|
||||
|
||||
sess_options = rt.SessionOptions()
|
||||
|
||||
# Set graph optimization level
|
||||
sess_options.graph_optimization_level = rt.GraphOptimizationLevel.ORT_ENABLE_EXTENDED
|
||||
|
||||
# To enable model serialization after graph optimization set this
|
||||
sess_options.optimized_model_filepath = "<model_output_path\optimized_model.onnx>"
|
||||
|
||||
session = rt.InferenceSession("<model_path>", sess_options)
|
||||
```
|
||||
|
||||
### C API Example:
|
||||
```c
|
||||
const OrtApi* Ort::g_api = OrtGetApi(ORT_API_VERSION);
|
||||
OrtEnv* env;
|
||||
g_ort->CreateEnv(ORT_LOGGING_LEVEL_WARNING, "test", &env);
|
||||
OrtSessionOptions* session_options;
|
||||
g_ort->CreateSessionOptions(&session_options)
|
||||
|
||||
// Set graph optimization level
|
||||
g_ort->SetSessionGraphOptimizationLevel(session_options, ORT_ENABLE_EXTENDED);
|
||||
|
||||
// To enable model serialization after graph optimization set this
|
||||
const wchar_t* optimized_model_path = L"optimized_model_path";
|
||||
g_ort->SetOptimizedModelFilePath(session_options, optimized_model_path);
|
||||
|
||||
OrtSession* session;
|
||||
const wchar_t* model_path = L"model_path";
|
||||
g_ort->CreateSession(env, model_path, session_option, &session);
|
||||
```
|
||||
|
||||
### C# API Example:
|
||||
```c#
|
||||
SessionOptions so = new SessionOptions();
|
||||
|
||||
// Set graph optimization level
|
||||
so.GraphOptimizationLevel = GraphOptimizationLevel.ORT_ENABLE_EXTENDED;
|
||||
|
||||
// To enable model serialization after graph optimization set this
|
||||
so.OptimizedModelFilePath = "model_output_path\optimized_model.onnx"
|
||||
|
||||
var session = new InferenceSession(modelPath, so);
|
||||
```
|
||||
|
||||
### C++ API Example:
|
||||
```c++
|
||||
Ort::SessionOptions session_options;
|
||||
|
||||
// Set graph optimization level
|
||||
session_options.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_EXTENDED);
|
||||
|
||||
// To enable model serialization after graph optimization set this
|
||||
session_options.SetOptimizedModelFilePath("optimized_file_path");
|
||||
|
||||
auto session_ = Ort::Session(env, "model_file_path", session_options);
|
||||
```
|
||||
153
docs/resources/high-level-design.md
Normal file
153
docs/resources/high-level-design.md
Normal file
|
|
@ -0,0 +1,153 @@
|
|||
---
|
||||
title: High level design
|
||||
parent: Resources
|
||||
nav_order: 2
|
||||
---
|
||||
|
||||
# ONNX Runtime High Level Design
|
||||
{: .no_toc }
|
||||
|
||||
This document outlines the high level design of ONNX Runtime - a high performance, cross platform engine.
|
||||
|
||||
## Contents
|
||||
{: .no_toc }
|
||||
|
||||
* TOC placeholder
|
||||
{:toc}
|
||||
|
||||
## Key objectives
|
||||
|
||||
* Maximally and automatically leverage the custom accelerators and runtimes
|
||||
available on disparate platforms.
|
||||
* Provide the right abstraction and runtime support for custom accelerators and
|
||||
runtimes. We call this abstraction an [execution
|
||||
provider](https://github.com/microsoft/onnxruntime/tree/master/include//onnxruntime/core/framework/execution_provider.h). It defines and exposes a set of
|
||||
its capabilities to ONNXRuntime: a set of single or fused nodes it can
|
||||
execute, its memory allocator, and more. Custom accelerators and runtimes are
|
||||
instances of execution providers.
|
||||
* We don't expect that an execution provider can always run an ONNX model fully
|
||||
on its device. This means that ONNXRuntime must be able to execute a single
|
||||
model in a heterogeneous environment involving multiple execution providers.
|
||||
* Provide support for high-level optimizations that can be expressed as
|
||||
model-to-model transformations via a [graph-transformation
|
||||
API](https://github.com/microsoft/onnxruntime/tree/master/include//onnxruntime/core/optimizer/graph_transformer.h). Such
|
||||
transformations fall into two categories: global transformations, those that
|
||||
require analysis and transformation of the entire graph, and local
|
||||
transformations, which can be captured as simple (algebraic) [rewriting
|
||||
rules](https://github.com/microsoft/onnxruntime/tree/master/include//onnxruntime/core/optimizer/rewrite_rule.h).
|
||||
|
||||
## High-level system architecture
|
||||
|
||||
The flow is quite simple. Starting from an ONNX model, ONNXRuntime first
|
||||
converts the model graph into its in-memory graph representation. It then
|
||||
applies a number of graph transformations that a) perform a set of provider
|
||||
independent optimizations such cast transformations between float16 and float32, and b) partition the
|
||||
graph into a set of subgraphs based on the available execution providers. Each
|
||||
subgraph is assigned to an execution provider. We ensure that a subgraph can be
|
||||
executed by an execution provider by querying the capability of the execution
|
||||
provider using the GetCapability() API.
|
||||
|
||||
|
||||
|
||||
### More about partitioning
|
||||
|
||||
ONNXRuntime partitions a model graph into subgraphs based on the available execution providers, one for each distinct provider. ONNXRuntime provides
|
||||
a default execution provider that is used as the fallback execution for the
|
||||
operators that cannot be pushed onto the more specialized but more efficient
|
||||
execution providers. Intuitively we want to push computation to more
|
||||
specialized execution providers whenever possible.
|
||||
|
||||
We use a simple graph partitioning technique. The available execution providers
|
||||
will be considered in a specific order, and each will be assigned the maximal
|
||||
subgraphs (possibly more than one) that it is able to handle. The
|
||||
ONNXRuntime-provided default execution provider will be the last one
|
||||
considered, and it ensures completeness. More sophisticated optimizations can be
|
||||
considered in the future (or can even be implemented as a composite execution
|
||||
provider).
|
||||
|
||||
Conceptually, each partition is reduced to a single fused operator. It is
|
||||
created by invoking the execution provider's Compile() method and wraps it as a
|
||||
custom operator. Currently we support only synchronous mode of execution. An execution
|
||||
provider exposes its memory allocator, which is used to allocate the input
|
||||
tensors for the execution provider. The rewriting and partitioning transform the
|
||||
initial model graph into a new graph composed of operators assigned to either
|
||||
the default execution provider or other registered execution
|
||||
providers. The ONNXRuntime execution engine is responsible for running this graph.
|
||||
|
||||
## Key design decisions
|
||||
|
||||
* Multiple threads can invoke the Run() method on the same
|
||||
inference session object. See [API doc](../reference/api/c-api.md) for more details.
|
||||
* To facilitate this, the Compute() function of all kernels is const
|
||||
implying the kernels are stateless.
|
||||
* Implementations of the operators by execution providers are called
|
||||
kernels. Each execution provider supports a subset of the (ONNX)
|
||||
operators/kernels.
|
||||
* The ONNX Runtime guarantees that all operators are supported by the default
|
||||
execution provider.
|
||||
* Tensor representation: ONNXRuntime will utilize a standard representation for
|
||||
the tensor runtime values. The execution providers can internally use a
|
||||
different representation if they choose to, but it is their responsibility to
|
||||
convert the values from/to the standard representation at the boundaries of
|
||||
their subgraph.
|
||||
|
||||
## Extensibility Options
|
||||
|
||||
* [Add a custom operator/kernel](../how-to/add-custom-op.md)
|
||||
* [Add an execution provider](../how-to/add-execution-provider.md)
|
||||
* [Add a new graph
|
||||
transform](https://github.com/microsoft/onnxruntime/tree/master/include//onnxruntime/core/optimizer/graph_transformer.h)
|
||||
* [Add a new rewrite rule](https://github.com/microsoft/onnxruntime/tree/master/include//onnxruntime/core/optimizer/rewrite_rule.h)
|
||||
|
||||
## The ONNX Runtime and Windows OS integration
|
||||
|
||||
The ONNX runtime shipped with the Windows operating system in build 1809 (RS5). The runtime was embedded inside the Windows.AI.MachineLearning.dll and was exposed via that WinRT API (WinML for short). It includes CPU support and a DirectML execution provider for GPU support. Since then it has continued to ship in every version of Windows.
|
||||
|
||||
Starting with the ONNX Runtime 1.2 release we are bringing a new layered architecture to the ONNX Runtime and Windows ML.
|
||||
*Note: This feature is preview as of the 1.2 release*
|
||||
|
||||
The high level design looks like this
|
||||
|
||||
|
||||
|
||||
You can see we replaced the embedded ONNX runtime with the new ONNXRuntime.dll. With this new approach customers have flexibility on which API they choose to use and on how they want to distribute the binaries.
|
||||
|
||||
### API choice
|
||||
|
||||
Developers can now choose which API works best for their scenario.
|
||||
|
||||
||WinRT|C API|
|
||||
|--|--|--|
|
||||
|Type system| Integration with Windows RT types| Platform neutral types|
|
||||
|Language support| Language support via WinRT Projections| Language support via per language projections|
|
||||
|Tensorization| Accepts VideoFrames and converts to tensors (support for CPU and GPU)| Accepts tensors|
|
||||
|
||||
### Distribution choice
|
||||
|
||||
You can also choose to use runtimes included in the Windows OS, or use the redist nuget to ship the runtime with the app.
|
||||
|
||||
|Distribution|Inbox|App NuGet|
|
||||
|--|--|--|
|
||||
|Disk footprint| Included in the OS| Included in the App|
|
||||
|Servicing fixes| Serviced by OS updates| Serviced by the App|
|
||||
|Execution Providers| CPU & DirectML EP | App chosen EP|
|
||||
|Compatability testing| Tested with OS flights against supported GPUs and CPU's | App performs compatibility testing|
|
||||
|Opset| Refreshed in OS updates| App chooses|
|
||||
|
||||
### Using the NuGet WinRT API with other C-API distributions
|
||||
|
||||
The WinRT API NuGet is distributed with a curated build of the OnnxRuntime engine. App developers may wish to use the WinRT API, but find themselves limited to the functionality provided by the curated OnnxRuntime engine distributed as part of the WinRT API NuGet package. This can happen because the OnnxRuntime engine shipped with the WinRT API NuGet package only contains the CPU and DML execution providers.
|
||||
|
||||
App developers may additionally wish to use a custom build-from-source version of the OnnxRuntime engine as well, or use a prebuilt version of the OnnxRuntime engine from another distribution source like the Micorosoft.ML.OnnxRuntime.MKLML distribution.
|
||||
|
||||
To enable this, the WinRT API NuGet has been made to be compatible with a set of OnnxRuntime engines that ship in different NuGet packages.
|
||||
|
||||
Please refer to the following table listing the distributions with compatible OnnxRuntime engines.
|
||||
|
||||
* [Microsoft.ML.OnnxRuntime](https://www.nuget.org/packages/Microsoft.ML.OnnxRuntime)
|
||||
* [Microsoft.ML.OnnxRuntime.DirectML](https://www.nuget.org/packages/Microsoft.ML.OnnxRuntime.DirectML/)
|
||||
* [Microsoft.ML.OnnxRuntime.MKLML](https://www.nuget.org/packages/Microsoft.ML.OnnxRuntime.MKLML)
|
||||
|
||||
Note that compatible distributions must match in release version.
|
||||
|
||||
In order to use compatible engines, replace the onnxruntime.dll with the desired engine binary and its associated binaries.
|
||||
7
docs/resources/index.md
Normal file
7
docs/resources/index.md
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
---
|
||||
title: Resources
|
||||
has_children: true
|
||||
nav_order: 5
|
||||
---
|
||||
|
||||
# ONNX Runtime Resources
|
||||
184
docs/tutorials/fasterrcnn_csharp.md
Normal file
184
docs/tutorials/fasterrcnn_csharp.md
Normal file
|
|
@ -0,0 +1,184 @@
|
|||
---
|
||||
parent: Tutorials
|
||||
nav_order: 3
|
||||
---
|
||||
|
||||
# Object detection with Faster RCNN in C#
|
||||
{: .no_toc }
|
||||
|
||||
The sample walks through how to run a pretrained Faster R-CNN object detection ONNX model using the ONNX Runtime C# API.
|
||||
|
||||
The source code for this sample is available [here](https://github.com/microsoft/onnxruntime/blob/master/csharp/sample/Microsoft.ML.OnnxRuntime.FasterRcnnSample/Program.cs).
|
||||
|
||||
## Contents
|
||||
{: .no_toc }
|
||||
|
||||
* TOC placeholder
|
||||
{:toc}
|
||||
|
||||
## Prerequisites
|
||||
|
||||
To run this sample, you'll need the following things:
|
||||
|
||||
1. Install [.NET Core 3.1](https://dotnet.microsoft.com/download/dotnet-core/3.1) or higher for you OS (Mac, Windows or Linux).
|
||||
2. Download the [Faster R-CNN](https://github.com/onnx/models/blob/master/vision/object_detection_segmentation/faster-rcnn/model/FasterRCNN-10.onnx) ONNX model to your local system.
|
||||
3. Download [this demo image](/images/demo.jpg) to test the model. You can also use any image you like.
|
||||
|
||||
## Get started
|
||||
|
||||
Now we have everything set up, we can start adding code to run the model on the image. We'll do this in the main method of the program for simplicity.
|
||||
|
||||
### Read paths
|
||||
|
||||
Firstly, let's read the path to the model, path to the image we want to test, and path to the output image:
|
||||
|
||||
```cs
|
||||
string modelFilePath = args[0];
|
||||
string imageFilePath = args[1];
|
||||
string outImageFilePath = args[2];
|
||||
```
|
||||
|
||||
### Read image
|
||||
|
||||
Next, we will read the image in using the cross-platform image library [ImageSharp](https://www.nuget.org/packages/SixLabors.ImageSharp):
|
||||
|
||||
```cs
|
||||
using Image<Rgb24> image = Image.Load<Rgb24>(imageFilePath, out IImageFormat format);
|
||||
```
|
||||
|
||||
Note, we're specifically reading the `Rgb24` type so we can efficiently preprocess the image in a later step.
|
||||
|
||||
### Resize image
|
||||
|
||||
Next, we will resize the image to the appropriate size that the model is expecting; it is recommended to resize the image such that both height and width are within the range of [800, 1333].
|
||||
|
||||
```cs
|
||||
float ratio = 800f / Math.Min(image.Width, image.Height);
|
||||
using Stream imageStream = new MemoryStream();
|
||||
image.Mutate(x => x.Resize((int)(ratio * image.Width), (int)(ratio * image.Height)));
|
||||
image.Save(imageStream, format);
|
||||
```
|
||||
|
||||
### Preprocess image
|
||||
|
||||
Next, we will preprocess the image according to the [requirements of the model](https://github.com/onnx/models/tree/master/vision/object_detection_segmentation/faster-rcnn#preprocessing-steps):
|
||||
|
||||
```cs
|
||||
var paddedHeight = (int)(Math.Ceiling(image.Height / 32f) * 32f);
|
||||
var paddedWidth = (int)(Math.Ceiling(image.Width / 32f) * 32f);
|
||||
Tensor<float> input = new DenseTensor<float>(new[] { 3, paddedHeight, paddedWidth });
|
||||
var mean = new[] { 102.9801f, 115.9465f, 122.7717f };
|
||||
for (int y = paddedHeight - image.Height; y < image.Height; y++)
|
||||
{
|
||||
Span<Rgb24> pixelSpan = image.GetPixelRowSpan(y);
|
||||
for (int x = paddedWidth - image.Width; x < image.Width; x++)
|
||||
{
|
||||
input[0, y, x] = pixelSpan[x].B - mean[0];
|
||||
input[1, y, x] = pixelSpan[x].G - mean[1];
|
||||
input[2, y, x] = pixelSpan[x].R - mean[2];
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Here, we're creating a Tensor of the required size `(channels, paddedHeight, paddedWidth)`, accessing the pixel values, preprocessing them and finally assigning them to the tensor at the appropriate indicies.
|
||||
|
||||
### Setup inputs
|
||||
|
||||
Next, we will create the inputs to the model:
|
||||
|
||||
```cs
|
||||
var inputs = new List<NamedOnnxValue>
|
||||
{
|
||||
NamedOnnxValue.CreateFromTensor("image", input)
|
||||
};
|
||||
```
|
||||
|
||||
To check the input node names for an ONNX model, you can use [Netron](https://github.com/lutzroeder/netron) to visualise the model and see input/output names. In this case, this model has `image` as the input node name.
|
||||
|
||||
### Run inference
|
||||
|
||||
Next, we will create an inference session and run the input through it:
|
||||
|
||||
```cs
|
||||
using var session = new InferenceSession(modelFilePath);
|
||||
using IDisposableReadOnlyCollection<DisposableNamedOnnxValue> results = session.Run(inputs);
|
||||
```
|
||||
|
||||
### Postprocess output
|
||||
|
||||
Next, we will need to postprocess the output to get boxes and associated label and confidence scores for each box:
|
||||
|
||||
```cs
|
||||
var resultsArray = results.ToArray();
|
||||
float[] boxes = resultsArray[0].AsEnumerable<float>().ToArray();
|
||||
long[] labels = resultsArray[1].AsEnumerable<long>().ToArray();
|
||||
float[] confidences = resultsArray[2].AsEnumerable<float>().ToArray();
|
||||
var predictions = new List<Prediction>();
|
||||
var minConfidence = 0.7f;
|
||||
for (int i = 0; i < boxes.Length - 4; i += 4)
|
||||
{
|
||||
var index = i / 4;
|
||||
if (confidences[index] >= minConfidence)
|
||||
{
|
||||
predictions.Add(new Prediction
|
||||
{
|
||||
Box = new Box(boxes[i], boxes[i + 1], boxes[i + 2], boxes[i + 3]),
|
||||
Label = LabelMap.Labels[labels[index]],
|
||||
Confidence = confidences[index]
|
||||
});
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Note, we're only taking boxes that have a confidence above 0.7 to remove false positives.
|
||||
|
||||
### View prediction
|
||||
|
||||
Next, we'll draw the boxes and associated labels and confidence scores on the image to see how the model went:
|
||||
|
||||
```cs
|
||||
using var outputImage = File.OpenWrite(outImageFilePath);
|
||||
Font font = SystemFonts.CreateFont("Arial", 16);
|
||||
foreach (var p in predictions)
|
||||
{
|
||||
image.Mutate(x =>
|
||||
{
|
||||
x.DrawLines(Color.Red, 2f, new PointF[] {
|
||||
|
||||
new PointF(p.Box.Xmin, p.Box.Ymin),
|
||||
new PointF(p.Box.Xmax, p.Box.Ymin),
|
||||
|
||||
new PointF(p.Box.Xmax, p.Box.Ymin),
|
||||
new PointF(p.Box.Xmax, p.Box.Ymax),
|
||||
|
||||
new PointF(p.Box.Xmax, p.Box.Ymax),
|
||||
new PointF(p.Box.Xmin, p.Box.Ymax),
|
||||
|
||||
new PointF(p.Box.Xmin, p.Box.Ymax),
|
||||
new PointF(p.Box.Xmin, p.Box.Ymin)
|
||||
});
|
||||
x.DrawText($"{p.Label}, {p.Confidence:0.00}", font, Color.White, new PointF(p.Box.Xmin, p.Box.Ymin));
|
||||
});
|
||||
}
|
||||
image.Save(outputImage, format);
|
||||
```
|
||||
|
||||
For each box prediction, we're using ImageSharp to draw red lines to create the boxes, and drawing the label and confidence text.
|
||||
|
||||
## Running the program
|
||||
|
||||
Now the program is created, we can run it will the following command:
|
||||
|
||||
```bash
|
||||
dotnet run [path-to-model] [path-to-image] [path-to-output-image]
|
||||
```
|
||||
|
||||
e.g. running:
|
||||
|
||||
```bash
|
||||
dotnet run ~/Downloads/FasterRCNN-10.onnx ~/Downloads/demo.jpg ~/Downloads/out.jpg
|
||||
```
|
||||
|
||||
detects the following objects in the image:
|
||||
|
||||
|
||||
8
docs/tutorials/index.md
Normal file
8
docs/tutorials/index.md
Normal file
|
|
@ -0,0 +1,8 @@
|
|||
---
|
||||
title: Tutorials
|
||||
has_children: true
|
||||
nav_order: 2
|
||||
---
|
||||
|
||||
# ONNX Runtime Tutorials
|
||||
{: .no_toc }
|
||||
78
docs/tutorials/mnist_cpp.md
Normal file
78
docs/tutorials/mnist_cpp.md
Normal file
|
|
@ -0,0 +1,78 @@
|
|||
---
|
||||
parent: Tutorials
|
||||
nav_order: 4
|
||||
---
|
||||
|
||||
# Number recognition with MNIST in C++
|
||||
{: .no_toc }
|
||||
|
||||
This sample uses the MNIST model from the Model Zoo: https://github.com/onnx/models/tree/master/vision/classification/mnist
|
||||
|
||||
|
||||
|
||||
## Contents
|
||||
{: .no_toc }
|
||||
|
||||
* TOC placeholder
|
||||
{:toc}
|
||||
|
||||
## Requirements
|
||||
|
||||
Compiled Onnxruntime.dll / lib (link to instructions on how to build dll)
|
||||
Windows Visual Studio Compiler (cl.exe)
|
||||
|
||||
## Build
|
||||
|
||||
Run 'build.bat' in this directory to call cl.exe to generate MNIST.exe
|
||||
Then just run MNIST.exe
|
||||
|
||||
## How to use it
|
||||
|
||||
Just draw a number with the left mouse button (or use touch) in the box on the left side. After releasing the mouse button the model will be run and the outputs of the model will be displayed. Note that when drawing numbers requiring multiple drawing strokes, the model will be run at the end of each stroke with probably wrong predictions (but it's amusing to see and avoids needing to press a 'run model' button).
|
||||
|
||||
To clear the image, click the right mouse button anywhere.
|
||||
|
||||
## How it works
|
||||
|
||||
A single Ort::Env is created globally to initialize the runtime.
|
||||
https://github.com/microsoft/onnxruntime/blob/521dc757984fbf9770d0051997178fbb9565cd52/samples/c_cxx/MNIST/MNIST.cpp#L12
|
||||
|
||||
The MNIST structure abstracts away all of the interaction with the Onnx Runtime, creating the tensors, and running the model.
|
||||
|
||||
WWinMain is the Windows entry point, it creates the main window.
|
||||
|
||||
WndProc is the window procedure for the window, handling the mouse input and drawing the graphics
|
||||
|
||||
### Preprocessing the data
|
||||
|
||||
MNIST's input is a {1,1,28,28} shaped float tensor, which is basically a 28x28 floating point grayscale image (0.0 = background, 1.0 = foreground).
|
||||
|
||||
The sample stores the image in a 32-bit per pixel windows DIB section, since that's easy to draw into and draw to the screen for windows. The DIB is created here:
|
||||
https://github.com/microsoft/onnxruntime/blob/521dc757984fbf9770d0051997178fbb9565cd52/samples/c_cxx/MNIST/MNIST.cpp#L109-L121
|
||||
|
||||
The function to convert the DIB data and writ it into the model's input tensor:
|
||||
https://github.com/microsoft/onnxruntime/blob/521dc757984fbf9770d0051997178fbb9565cd52/samples/c_cxx/MNIST/MNIST.cpp#L77-L92
|
||||
|
||||
### Postprocessing the output
|
||||
|
||||
MNIST's output is a simple {1,10} float tensor that holds the likelihood weights per number. The number with the highest value is the model's best guess.
|
||||
|
||||
The MNIST structure uses std::max_element to do this and stores it in result_:
|
||||
https://github.com/microsoft/onnxruntime/blob/521dc757984fbf9770d0051997178fbb9565cd52/samples/c_cxx/MNIST/MNIST.cpp#L31
|
||||
|
||||
To make things more interesting, the window painting handler graphs the probabilities and shows the weights here:
|
||||
https://github.com/microsoft/onnxruntime/blob/521dc757984fbf9770d0051997178fbb9565cd52/samples/c_cxx/MNIST/MNIST.cpp#L164-L183
|
||||
|
||||
### The Ort::Session
|
||||
|
||||
1. Creation: The Ort::Session is created inside the MNIST structure here:
|
||||
https://github.com/microsoft/onnxruntime/blob/521dc757984fbf9770d0051997178fbb9565cd52/samples/c_cxx/MNIST/MNIST.cpp#L43
|
||||
|
||||
2. Setup inputs & outputs: The input & output tensors are created here:
|
||||
https://github.com/microsoft/onnxruntime/blob/521dc757984fbf9770d0051997178fbb9565cd52/samples/c_cxx/MNIST/MNIST.cpp#L19-L23
|
||||
In this usage, we're providing the memory location for the data instead of having Ort allocate the buffers. This is simpler in this case since the buffers are small and can just be fixed members of the MNIST struct.
|
||||
|
||||
3. Run: Running the session is done in the Run() method:
|
||||
https://github.com/microsoft/onnxruntime/blob/521dc757984fbf9770d0051997178fbb9565cd52/samples/c_cxx/MNIST/MNIST.cpp#L25-L33
|
||||
|
||||
|
||||
57
docs/tutorials/mnist_java.md
Normal file
57
docs/tutorials/mnist_java.md
Normal file
|
|
@ -0,0 +1,57 @@
|
|||
---
|
||||
parent: Tutorials
|
||||
nav_order: 5
|
||||
---
|
||||
|
||||
# Character recognition with MNIST in Java
|
||||
{: .no_toc }
|
||||
|
||||
Here is simple tutorial for getting started with running inference on an existing ONNX model for a given input data. The model is typically trained using any of the well-known training frameworks and exported into the ONNX format.
|
||||
|
||||
Note the code presented below uses syntax available from Java 10 onwards. The Java 8 syntax is similar but more verbose.
|
||||
To start a scoring session, first create the `OrtEnvironment`, then open a session using the `OrtSession` class, passing in the file path to the model as a parameter.
|
||||
|
||||
```java
|
||||
var env = OrtEnvironment.getEnvironment();
|
||||
var session = env.createSession("model.onnx",new OrtSession.SessionOptions());
|
||||
```
|
||||
|
||||
Once a session is created, you can execute queries using the `run` method of the `OrtSession` object.
|
||||
At the moment we support `OnnxTensor` inputs, and models can produce `OnnxTensor`, `OnnxSequence` or `OnnxMap` outputs. The latter two are more likely when scoring models produced by frameworks like scikit-learn.
|
||||
The run call expects a `Map<String,OnnxTensor>` where the keys match input node names stored in the model. These can be viewed by calling `session.getInputNames()` or `session.getInputInfo()` on an instantiated session.
|
||||
The run call produces a `Result` object, which contains a `Map<String,OnnxValue>` representing the output. The `Result` object is `AutoCloseable` and can be used in a try-with-resources statement to
|
||||
prevent references from leaking out. Once the `Result` object is closed, all it's child `OnnxValue`s are closed too.
|
||||
|
||||
```java
|
||||
OnnxTensor t1,t2;
|
||||
var inputs = Map.of("name1",t1,"name2",t2);
|
||||
try (var results = session.run(inputs)) {
|
||||
// manipulate the results
|
||||
}
|
||||
```
|
||||
|
||||
You can load your input data into OnnxTensor objects in several ways. The most efficient way is to use a `java.nio.Buffer`, but it's possible to use multidimensional arrays too. If constructed using arrays the arrays must not be ragged.
|
||||
|
||||
```java
|
||||
FloatBuffer sourceData; // assume your data is loaded into a FloatBuffer
|
||||
long[] dimensions; // and the dimensions of the input are stored here
|
||||
var tensorFromBuffer = OnnxTensor.createTensor(env,sourceData,dimensions);
|
||||
|
||||
float[][] sourceArray = new float[28][28]; // assume your data is loaded into a float array
|
||||
var tensorFromArray = OnnxTensor.createTensor(env,sourceArray);
|
||||
```
|
||||
|
||||
Here is a [complete sample program](https://github.com/microsoft/onnxruntime/blob/master/java/src/test/java/sample/ScoreMNIST.java) that runs inference on a pretrained MNIST model.
|
||||
|
||||
## Running on a GPU or with another provider (Optional)
|
||||
|
||||
To enable other execution providers like GPUs simply turn on the appropriate flag on SessionOptions when creating an OrtSession.
|
||||
|
||||
```java
|
||||
int gpuDeviceId = 0; // The GPU device ID to execute on
|
||||
var sessionOptions = new OrtSession.SessionOptions();
|
||||
sessionOptions.addCUDA(gpuDeviceId);
|
||||
var session = environment.createSession("model.onnx", sessionOptions);
|
||||
```
|
||||
|
||||
The execution providers are preferred in the order they were enabled.
|
||||
182
docs/tutorials/resnet50_csharp.md
Normal file
182
docs/tutorials/resnet50_csharp.md
Normal file
|
|
@ -0,0 +1,182 @@
|
|||
---
|
||||
parent: Tutorials
|
||||
nav_order: 2
|
||||
---
|
||||
|
||||
|
||||
# Image recognition with ResNet50v2 in C#
|
||||
{: .no_toc }
|
||||
|
||||
The sample walks through how to run a pretrained ResNet50 v2 ONNX model using the Onnx Runtime C# API.
|
||||
|
||||
The source code for this sample is available [here](https://github.com/microsoft/onnxruntime/tree/master/csharp/sample/Microsoft.ML.OnnxRuntime.ResNet50v2Sample).
|
||||
|
||||
## Contents
|
||||
{: .no_toc }
|
||||
|
||||
* TOC placeholder
|
||||
{:toc}
|
||||
|
||||
## Prerequisites
|
||||
|
||||
To run this sample, you'll need the following things:
|
||||
|
||||
1. Install [.NET Core 3.1](https://dotnet.microsoft.com/download/dotnet-core/3.1) or higher for you OS (Mac, Windows or Linux).
|
||||
2. Download the [ResNet50 v2](https://github.com/onnx/models/blob/master/vision/classification/resnet/model/resnet50-v2-7.onnx) ONNX model to your local system.
|
||||
3. Download [this picture of a dog](/images/dog.jpeg) to test the model. You can also use any image you like.
|
||||
|
||||
## Getting Started
|
||||
|
||||
Now we have everything set up, we can start adding code to run the model on the image. We'll do this in the main method of the program for simplicity.
|
||||
|
||||
### Read paths
|
||||
|
||||
Firstly, let's read the path to the model and path to the image we want to test in through program arguments:
|
||||
|
||||
```cs
|
||||
string modelFilePath = args[0];
|
||||
string imageFilePath = args[1];
|
||||
```
|
||||
|
||||
### Read image
|
||||
|
||||
Next, we will read the image in using the cross-platform image library [ImageSharp](https://www.nuget.org/packages/SixLabors.ImageSharp):
|
||||
|
||||
```cs
|
||||
using Image<Rgb24> image = Image.Load<Rgb24>(imageFilePath, out IImageFormat format);
|
||||
```
|
||||
|
||||
Note, we're specifically reading the `Rgb24` type so we can efficiently preprocess the image in a later step.
|
||||
|
||||
### Resize image
|
||||
|
||||
Next, we will resize the image to the appropriate size that the model is expecting; 224 pixels by 224 pixels:
|
||||
|
||||
```cs
|
||||
using Stream imageStream = new MemoryStream();
|
||||
image.Mutate(x =>
|
||||
{
|
||||
x.Resize(new ResizeOptions
|
||||
{
|
||||
Size = new Size(224, 224),
|
||||
Mode = ResizeMode.Crop
|
||||
});
|
||||
});
|
||||
image.Save(imageStream, format);
|
||||
```
|
||||
|
||||
Note, we're doing a centered crop resize to preserve aspect ratio.
|
||||
|
||||
### Preprocess image
|
||||
|
||||
Next, we will preprocess the image according to the [requirements of the model](https://github.com/onnx/models/tree/master/vision/classification/resnet#preprocessing):
|
||||
|
||||
```cs
|
||||
Tensor<float> input = new DenseTensor<float>(new[] { 1, 3, 224, 224 });
|
||||
var mean = new[] { 0.485f, 0.456f, 0.406f };
|
||||
var stddev = new[] { 0.229f, 0.224f, 0.225f };
|
||||
for (int y = 0; y < image.Height; y++)
|
||||
{
|
||||
Span<Rgb24> pixelSpan = image.GetPixelRowSpan(y);
|
||||
for (int x = 0; x < image.Width; x++)
|
||||
{
|
||||
input[0, 0, y, x] = ((pixelSpan[x].R / 255f) - mean[0]) / stddev[0];
|
||||
input[0, 1, y, x] = ((pixelSpan[x].G / 255f) - mean[1]) / stddev[1];
|
||||
input[0, 2, y, x] = ((pixelSpan[x].B / 255f) - mean[2]) / stddev[2];
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Here, we're creating a Tensor of the required size `(batch-size, channels, height, width)`, accessing the pixel values, preprocessing them and finally assigning them to the tensor at the appropriate indicies.
|
||||
|
||||
### Setup inputs
|
||||
|
||||
Next, we will create the inputs to the model:
|
||||
|
||||
```cs
|
||||
var inputs = new List<NamedOnnxValue>
|
||||
{
|
||||
NamedOnnxValue.CreateFromTensor("data", input)
|
||||
};
|
||||
```
|
||||
|
||||
To check the input node names for an ONNX model, you can use [Netron](https://github.com/lutzroeder/netron) to visualise the model and see input/output names. In this case, this model has `data` as the input node name.
|
||||
|
||||
### Run inference
|
||||
|
||||
Next, we will create an inference session and run the input through it:
|
||||
|
||||
```cs
|
||||
using var session = new InferenceSession(modelFilePath);
|
||||
using IDisposableReadOnlyCollection<DisposableNamedOnnxValue> results = session.Run(inputs);
|
||||
```
|
||||
|
||||
### Postprocess output
|
||||
|
||||
Next, we will need to postprocess the output to get the softmax vector, as this is not handled by the model itself:
|
||||
|
||||
```cs
|
||||
IEnumerable<float> output = results.First().AsEnumerable<float>();
|
||||
float sum = output.Sum(x => (float)Math.Exp(x));
|
||||
IEnumerable<float> softmax = output.Select(x => (float)Math.Exp(x) / sum);
|
||||
```
|
||||
|
||||
Other models may apply a Softmax node before the output, in which case you won't need this step. Again, you can use Netron to see the model outputs.
|
||||
|
||||
### Extract top 10
|
||||
|
||||
Next, we will extract the top 10 class predictions:
|
||||
|
||||
```cs
|
||||
IEnumerable<Prediction> top10 = softmax.Select((x, i) => new Prediction { Label = LabelMap.Labels[i], Confidence = x })
|
||||
.OrderByDescending(x => x.Confidence)
|
||||
.Take(10);
|
||||
```
|
||||
|
||||
### Print results
|
||||
|
||||
Next, we will print the top 10 results to the console:
|
||||
|
||||
```cs
|
||||
Console.WriteLine("Top 10 predictions for ResNet50 v2...");
|
||||
Console.WriteLine("--------------------------------------------------------------");
|
||||
foreach (var t in top10)
|
||||
{
|
||||
Console.WriteLine($"Label: {t.Label}, Confidence: {t.Confidence}");
|
||||
}
|
||||
```
|
||||
|
||||
## Running the program
|
||||
|
||||
Now the program is created, we can run it will the following command:
|
||||
|
||||
```
|
||||
dotnet run [path-to-model] [path-to-image]
|
||||
```
|
||||
|
||||
e.g.
|
||||
|
||||
```
|
||||
dotnet run ~/Downloads/resnet50-v2-7.onnx ~/Downloads/dog.jpeg
|
||||
```
|
||||
|
||||
Running this on the following image:
|
||||
|
||||
|
||||
|
||||
We get the following output:
|
||||
|
||||
```
|
||||
Top 10 predictions for ResNet50 v2...
|
||||
--------------------------------------------------------------
|
||||
Label: Golden Retriever, Confidence: 0.9212826
|
||||
Label: Kuvasz, Confidence: 0.026514154
|
||||
Label: Clumber Spaniel, Confidence: 0.012455719
|
||||
Label: Labrador Retriever, Confidence: 0.004103844
|
||||
Label: Saluki, Confidence: 0.0033182495
|
||||
Label: Flat-Coated Retriever, Confidence: 0.0032045357
|
||||
Label: English Setter, Confidence: 0.002513516
|
||||
Label: Brittany, Confidence: 0.0023459378
|
||||
Label: Cocker Spaniels, Confidence: 0.0019343802
|
||||
Label: Sussex Spaniel, Confidence: 0.0019247672
|
||||
```
|
||||
107
docs/tutorials/samples_catalog.md
Normal file
107
docs/tutorials/samples_catalog.md
Normal file
|
|
@ -0,0 +1,107 @@
|
|||
---
|
||||
parent: Tutorials
|
||||
toc: true
|
||||
nav_order: 1
|
||||
---
|
||||
|
||||
# Samples catalog
|
||||
{: .no_toc }
|
||||
|
||||
This page catalogs code samples for ONNX Runtime, running locally, and on Azure, both cloud and edge.
|
||||
|
||||
## Contents
|
||||
{: .no_toc }
|
||||
|
||||
* TOC placeholder
|
||||
{:toc}
|
||||
|
||||
## Python
|
||||
|
||||
* [Basic inference](https://microsoft.github.io/onnxruntime/python/tutorial.html)
|
||||
* [Resnet50 inference](https://github.com/onnx/onnx-docker/blob/master/onnx-ecosystem/inference_demos/resnet50_modelzoo_onnxruntime_inference.ipynb)
|
||||
* [Inference samples with ONNX-Ecosystem Docker image](https://github.com/onnx/onnx-docker/tree/master/onnx-ecosystem/inference_demos)
|
||||
* [ONNX Runtime Server: SSD Single Shot MultiBox Detector](https://github.com/onnx/tutorials/blob/master/tutorials/OnnxRuntimeServerSSDModel.ipynb)
|
||||
* [NUPHAR Execution Provider samples](https://github.com/microsoft/onnxruntime/tree/master/docs/python/notebooks/onnxruntime-nuphar-tutorial.ipynb)
|
||||
* [SKL tutorials](http://onnx.ai/sklearn-onnx/index_tutorial.html)
|
||||
* [Keras - Basic](https://microsoft.github.io/onnxruntime/python/auto_examples/plot_dl_keras.html#sphx-glr-auto-examples-plot-dl-keras-py)
|
||||
* [SSD Mobilenet (Tensorflow)](https://github.com/onnx/tensorflow-onnx/blob/master/tutorials/ConvertingSSDMobilenetToONNX.ipynb)
|
||||
* [BERT-SQuAD (PyTorch) on CPU](https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/python/tools/transformers/notebooks/PyTorch_Bert-Squad_OnnxRuntime_CPU.ipynb)
|
||||
* [BERT-SQuAD (PyTorch) on GPU](https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/python/tools/transformers/notebooks/PyTorch_Bert-Squad_OnnxRuntime_GPU.ipynb)
|
||||
* [BERT-SQuAD (Keras)](https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/python/tools/transformers/notebooks/Tensorflow_Keras_Bert-Squad_OnnxRuntime_CPU.ipynb)
|
||||
* [BERT-SQuAD (Tensorflow)](https://github.com/onnx/tensorflow-onnx/blob/master/tutorials/BertTutorial.ipynb)
|
||||
* [GPT2 (PyTorch)](https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/python/tools/transformers/notebooks/Inference_GPT2_with_OnnxRuntime_on_CPU.ipynb)
|
||||
* [EfficientDet (Tensorflow)](https://github.com/onnx/tensorflow-onnx/blob/master/tutorials/efficientdet.ipynb)
|
||||
* [EfficientNet-Edge (Tensorflow)](https://github.com/onnx/tensorflow-onnx/blob/master/tutorials/efficientnet-edge.ipynb)
|
||||
* [EfficientNet-Lite (Tensorflow)](https://github.com/onnx/tensorflow-onnx/blob/master/tutorials/efficientnet-lite.ipynb)
|
||||
* [EfficientNet(Keras)](https://github.com/onnx/keras-onnx/blob/master/tutorial/TensorFlow_Keras_EfficientNet.ipynb)
|
||||
* [MNIST (Keras)](https://github.com/onnx/keras-onnx/blob/master/tutorial/TensorFlow_Keras_MNIST.ipynb)
|
||||
* [BERT Quantization on CPU](https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/python/tools/quantization/notebooks/Bert-GLUE_OnnxRuntime_quantization.ipynb)
|
||||
* [Get started with training](https://github.com/microsoft/onnxruntime-training-examples/tree/master/getting-started)
|
||||
* [Train NVIDIA BERT transformer model](https://github.com/microsoft/onnxruntime-training-examples/tree/master/nvidia-bert)
|
||||
* [Train HuggingFace GPT-2 model](https://github.com/microsoft/onnxruntime-training-examples/tree/master/huggingface-gpt2)
|
||||
|
||||
## C/C++
|
||||
|
||||
* [C: SqueezeNet](https://github.com/microsoft/onnxruntime/tree/master/csharp/test/Microsoft.ML.OnnxRuntime.EndToEndTests.Capi/C_Api_Sample.cpp)
|
||||
* [C++: model-explorer](https://github.com/microsoft/onnxruntime/tree/master/samples/c_cxx/model-explorer) - single and batch processing
|
||||
* [C++: SqueezeNet](https://github.com/microsoft/onnxruntime/tree/master/csharp/test/Microsoft.ML.OnnxRuntime.EndToEndTests.Capi/CXX_Api_Sample.cpp)
|
||||
|
||||
## C#
|
||||
|
||||
* [Inference Tutorial](resnet50_csharp.md)
|
||||
* [ResNet50 v2 Tutorial](https://github.com/microsoft/onnxruntime/tree/master/csharp/sample/Microsoft.ML.OnnxRuntime.ResNet50v2Sample)
|
||||
* [Faster R-CNN Tutorial](https://github.com/microsoft/onnxruntime/tree/master/csharp/sample/Microsoft.ML.OnnxRuntime.FasterRcnnSample)
|
||||
|
||||
## Java
|
||||
|
||||
* [MNIST inference](https://github.com/microsoft/onnxruntime/tree/master/java/src/test/java/sample/ScoreMNIST.java)
|
||||
|
||||
## Node.js
|
||||
|
||||
* [Inference with Nodejs](https://github.com/microsoft/onnxruntime/tree/master/samples/nodejs)
|
||||
|
||||
---
|
||||
|
||||
## Azure Machine Learning
|
||||
|
||||
### Inference and deploy through AzureML
|
||||
|
||||
*For aditional information on training in AzureML, please see [AzureML Training Notebooks](https://github.com/Azure/MachineLearningNotebooks/tree/master/how-to-use-azureml/training)*
|
||||
|
||||
* Inferencing on **CPU** using [ONNX Model Zoo](https://github.com/onnx/models) models:
|
||||
* [Facial Expression Recognition](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/deployment/onnx/onnx-inference-facial-expression-recognition-deploy.ipynb)
|
||||
* [MNIST Handwritten Digits](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/deployment/onnx/onnx-inference-mnist-deploy.ipynb)
|
||||
* [Resnet50 Image Classification](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/deployment/onnx/onnx-modelzoo-aml-deploy-resnet50.ipynb)
|
||||
* Inferencing on **CPU** with **PyTorch** model training:
|
||||
* [MNIST](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/deployment/onnx/onnx-train-pytorch-aml-deploy-mnist.ipynb)
|
||||
* [BERT](https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/python/tools/transformers/notebooks/Inference_Bert_with_OnnxRuntime_on_AzureML.ipynb)
|
||||
* Inferencing on **CPU** with model conversion for existing (CoreML) model:
|
||||
* [TinyYolo](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/deployment/onnx/onnx-convert-aml-deploy-tinyyolo.ipynb)
|
||||
* Inferencing on **GPU** with **TensorRT** Execution Provider (AKS):
|
||||
* [FER+](https://github.com/microsoft/onnxruntime/tree/master/docs/python/notebooks/onnx-inference-byoc-gpu-cpu-aks.ipynb)
|
||||
|
||||
## Huggingface
|
||||
|
||||
[Export Tranformer models](https://github.com/huggingface/transformers/blob/master/notebooks/04-onnx-export.ipynb)
|
||||
|
||||
## Azure IoT Edge
|
||||
|
||||
* [Intel OpenVINO](http://aka.ms/onnxruntime-openvino)
|
||||
* [NVIDIA TensorRT on Jetson Nano (ARM64)](http://aka.ms/onnxruntime-arm64)
|
||||
* [ONNX Runtime with Azure ML](https://github.com/Azure-Samples/onnxruntime-iot-edge/blob/master/AzureML-OpenVINO/README.md)
|
||||
|
||||
## Azure Media Services
|
||||
|
||||
[Video Analysis through Azure Media Services using using Yolov3 to build an IoT Edge module for object detection](https://github.com/Azure/live-video-analytics/tree/master/utilities/video-analysis/yolov3-onnx)
|
||||
|
||||
## Azure SQL
|
||||
|
||||
[Deploy ONNX model in Azure SQL Edge](https://docs.microsoft.com/en-us/azure/azure-sql-edge/deploy-onnx)
|
||||
|
||||
## Windows Machine Learning
|
||||
|
||||
[Examples of inferencing with ONNX Runtime through Windows Machine Learning](https://docs.microsoft.com/en-us/windows/ai/windows-ml/tools-and-samples#samples)
|
||||
|
||||
## ML.NET
|
||||
|
||||
[Object Detection with ONNX Runtime in ML.NET](https://docs.microsoft.com/en-us/dotnet/machine-learning/tutorials/object-detection-onnx)
|
||||
BIN
favicon.ico
Normal file
BIN
favicon.ico
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 660 B |
148
images/ONNX-Runtime-logo.svg
Normal file
148
images/ONNX-Runtime-logo.svg
Normal file
|
|
@ -0,0 +1,148 @@
|
|||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<!-- Generator: Adobe Illustrator 23.0.3, SVG Export Plug-In . SVG Version: 6.00 Build 0) -->
|
||||
<svg version="1.1" id="Layer_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px"
|
||||
viewBox="0 0 217 74" style="enable-background:new 0 0 217 74;" xml:space="preserve">
|
||||
<style type="text/css">
|
||||
.st0{fill:#333333;}
|
||||
.st1{fill:#FFFFFF;}
|
||||
.st2{fill:#D1D1D1;}
|
||||
.st3{fill:#B2B2B2;}
|
||||
</style>
|
||||
<g>
|
||||
<g>
|
||||
<g>
|
||||
<path class="st0" d="M95.92,26.59V12.43c0-6.83,5.14-9.38,11.88-9.38c6.68,0,11.94,2.55,11.94,9.38v14.16
|
||||
c0,6.83-5.26,9.38-11.94,9.38C101.07,35.97,95.92,33.43,95.92,26.59z M116.79,12.43c0-4.96-3.37-7.24-8.99-7.24
|
||||
c-5.62,0-8.93,2.28-8.93,7.24v14.16c0,4.96,3.31,7.24,8.93,7.24c5.62,0,8.99-2.28,8.99-7.24V12.43z"/>
|
||||
<path class="st0" d="M149.02,34.05L131.63,8.32v26.4c0,0.67-0.71,0.98-1.48,0.98c-0.71,0-1.48-0.31-1.48-0.98V3.99
|
||||
c0-0.71,0.71-0.94,1.48-0.94c1.06,0,1.89,1.07,2.42,1.83l17.09,25.51V3.99c0-0.67,0.77-0.94,1.48-0.94
|
||||
c0.77,0,1.48,0.27,1.48,0.94v30.73c0,0.67-0.71,0.98-1.48,0.98C150.26,35.71,149.55,34.81,149.02,34.05z"/>
|
||||
<path class="st0" d="M182.3,34.05L164.92,8.32v26.4c0,0.67-0.71,0.98-1.48,0.98c-0.71,0-1.48-0.31-1.48-0.98V3.99
|
||||
c0-0.71,0.71-0.94,1.48-0.94c1.06,0,1.89,1.07,2.42,1.83l17.09,25.51V3.99c0-0.67,0.77-0.94,1.48-0.94
|
||||
c0.77,0,1.48,0.27,1.48,0.94v30.73c0,0.67-0.71,0.98-1.48,0.98C183.54,35.71,182.83,34.81,182.3,34.05z"/>
|
||||
<path class="st0" d="M204.3,21.46l-8.75,12.78c-0.41,0.63-0.89,1.52-1.77,1.52c-0.77,0-1.66-0.49-1.66-1.12
|
||||
c0-0.18,0.12-0.36,0.24-0.58l10.17-14.7l-9.99-14.56c-0.24-0.31-0.36-0.58-0.36-0.85c0-0.58,0.71-0.94,1.66-0.94
|
||||
c0.83,0,1.42,0.85,1.89,1.56l8.57,12.64l8.63-12.64c0.47-0.62,1.01-1.56,1.89-1.56c0.95,0,1.66,0.36,1.66,0.94
|
||||
c0,0.27-0.12,0.54-0.36,0.85l-10.05,14.56l10.23,14.7c0.12,0.22,0.24,0.4,0.24,0.58c0,0.63-0.83,1.12-1.66,1.12
|
||||
c-0.89,0-1.36-0.89-1.77-1.52L204.3,21.46z"/>
|
||||
</g>
|
||||
<g>
|
||||
<path class="st0" d="M109.47,53.62c0-2.28-0.65-3.95-1.94-4.97c-1.29-1.02-2.99-1.53-5.06-1.53l-5.65,0
|
||||
c-0.43,0.05-0.64,0.26-0.64,0.62v23.54c0,0.43,0.22,0.65,0.65,0.65s0.65-0.22,0.65-0.65V60.15h4.94l5.4,10.53l0.01,0.02h0.01
|
||||
c0.01,0.01,0.05,0.06,0.14,0.24l0.14,0.38c0.05,0.07,0.11,0.16,0.19,0.27c0.08,0.12,0.17,0.2,0.25,0.25
|
||||
c0.08,0.05,0.18,0.07,0.27,0.07c0.19,0,0.35-0.06,0.47-0.18c0.12-0.12,0.18-0.28,0.18-0.47c0-0.12-0.04-0.25-0.11-0.39
|
||||
l-5.55-10.8C107.57,59.61,109.47,57.44,109.47,53.62z M108.17,53.62c0,1.78-0.51,3.11-1.52,3.96c-1.01,0.85-2.42,1.28-4.18,1.28
|
||||
h-5V48.42h5c1.76,0,3.17,0.42,4.18,1.26C107.66,50.51,108.17,51.83,108.17,53.62z"/>
|
||||
<path class="st0" d="M127.25,47.12c-0.43,0-0.65,0.21-0.65,0.62v17.38c0,1.9-0.47,3.34-1.39,4.28c-0.92,0.95-2.26,1.42-3.98,1.42
|
||||
c-1.72,0-3.05-0.48-3.96-1.42c-0.91-0.95-1.38-2.39-1.38-4.28V47.73c0-0.41-0.22-0.62-0.65-0.62s-0.65,0.21-0.65,0.62v17.38
|
||||
c0,2.41,0.61,4.2,1.8,5.32c1.2,1.12,2.82,1.68,4.83,1.68s3.64-0.57,4.85-1.68c1.21-1.12,1.82-2.91,1.82-5.32V47.73
|
||||
C127.9,47.32,127.68,47.12,127.25,47.12z"/>
|
||||
<path class="st0" d="M147.3,47.12c-0.43,0-0.65,0.21-0.65,0.62V68.7l-10.27-20.34c-0.46-0.83-0.82-1.24-1.08-1.24
|
||||
c-0.43,0-0.65,0.21-0.65,0.62v23.54c0,0.43,0.22,0.65,0.65,0.65c0.43,0,0.65-0.22,0.65-0.65V50.3l10.34,20.38
|
||||
c0.42,0.83,0.75,1.24,1.02,1.24c0.43,0,0.65-0.22,0.65-0.65V47.73C147.95,47.32,147.74,47.12,147.3,47.12z"/>
|
||||
<path class="st0" d="M166.38,47.12h-13.22c-0.19,0-0.35,0.07-0.47,0.2c-0.12,0.13-0.18,0.28-0.18,0.45
|
||||
c0,0.17,0.06,0.32,0.18,0.45c0.12,0.13,0.28,0.2,0.47,0.2h5.98v22.85c0,0.43,0.22,0.65,0.65,0.65c0.43,0,0.65-0.22,0.65-0.65
|
||||
V48.42h5.95c0.43,0,0.65-0.23,0.65-0.68C167.03,47.32,166.81,47.12,166.38,47.12z"/>
|
||||
<path class="st0" d="M172.23,47.12c-0.43,0-0.65,0.21-0.65,0.62v23.54c0,0.43,0.22,0.65,0.65,0.65c0.43,0,0.65-0.22,0.65-0.65
|
||||
V47.73C172.88,47.32,172.66,47.12,172.23,47.12z"/>
|
||||
<path class="st0" d="M195.66,47.12c-0.27,0-0.62,0.41-1.08,1.24l-6.56,12.85l-6.53-12.85c-0.46-0.83-0.82-1.24-1.08-1.24
|
||||
c-0.43,0-0.65,0.21-0.65,0.62v23.54c0,0.43,0.22,0.65,0.65,0.65c0.43,0,0.65-0.22,0.65-0.65V50.3l6.28,12.16
|
||||
c0.21,0.45,0.43,0.67,0.68,0.67c0.22,0,0.44-0.22,0.68-0.66l6.31-12.16v20.97c0,0.43,0.22,0.65,0.65,0.65
|
||||
c0.43,0,0.65-0.22,0.65-0.65V47.73C196.31,47.32,196.09,47.12,195.66,47.12z"/>
|
||||
<path class="st0" d="M215.89,70.62H204.5V60.15h5.37c0.41,0,0.62-0.21,0.62-0.62c0-0.45-0.21-0.68-0.62-0.68h-5.37V48.42h11.39
|
||||
c0.41,0,0.62-0.23,0.62-0.68c0-0.4-0.21-0.62-0.62-0.62h-12.04c-0.43,0-0.65,0.22-0.65,0.65v23.5c0,0.43,0.22,0.65,0.65,0.65
|
||||
h12.04c0.41,0,0.62-0.21,0.62-0.62C216.5,70.85,216.29,70.62,215.89,70.62z"/>
|
||||
</g>
|
||||
</g>
|
||||
<g>
|
||||
<g>
|
||||
<polygon class="st1" points="58.11,20.09 39.33,41.08 39.33,1.31 "/>
|
||||
<polyline class="st2" points="58.11,20.09 46,33.49 39.33,20.09 "/>
|
||||
<path class="st3" d="M39.33,1.31l18.77,18.78H39.33C39.33,20.09,39.33,1.43,39.33,1.31z"/>
|
||||
</g>
|
||||
<g>
|
||||
<polygon class="st1" points="24.76,16.72 38.93,41.06 1.1,28.77 "/>
|
||||
<polyline class="st2" points="24.76,16.72 33.77,32.37 18.96,34.57 "/>
|
||||
<path class="st3" d="M1.1,28.77l23.66-12.05l-5.8,17.85C18.96,34.57,1.22,28.81,1.1,28.77z"/>
|
||||
</g>
|
||||
<g>
|
||||
<polygon class="st1" points="11.25,47.38 38.78,41.44 15.4,73.61 "/>
|
||||
<polyline class="st2" points="11.25,47.38 28.92,43.65 26.44,58.42 "/>
|
||||
<path class="st3" d="M15.4,73.61l-4.15-26.23l15.19,11.03C26.44,58.42,15.48,73.51,15.4,73.61z"/>
|
||||
</g>
|
||||
<g>
|
||||
<polygon class="st1" points="36.25,69.71 39.1,41.69 62.47,73.87 "/>
|
||||
<polyline class="st2" points="36.25,69.71 38.16,51.75 51.43,58.67 "/>
|
||||
<path class="st3" d="M62.47,73.87l-26.23-4.16l15.19-11.03C51.43,58.67,62.4,73.77,62.47,73.87z"/>
|
||||
</g>
|
||||
<g>
|
||||
<polygon class="st1" points="65.2,52.84 39.44,41.47 77.26,29.18 "/>
|
||||
<polyline class="st2" points="65.2,52.84 48.71,45.47 59.4,34.98 "/>
|
||||
<path class="st3" d="M77.26,29.18L65.2,52.84l-5.8-17.85C59.4,34.98,77.14,29.22,77.26,29.18z"/>
|
||||
</g>
|
||||
<path class="st0" d="M77.49,28.79c-0.13-0.15-0.33-0.21-0.52-0.15l-0.02,0.01L40.49,40.5l17.87-19.98c0,0,0,0,0-0.01
|
||||
c0.02-0.02,0.04-0.04,0.05-0.07c0,0,0-0.01,0.01-0.01c0.01-0.02,0.02-0.03,0.03-0.05c0,0,0-0.01,0-0.01
|
||||
c0-0.01,0.01-0.02,0.01-0.03c0.01-0.02,0.01-0.04,0.01-0.05c0-0.01,0-0.02,0-0.03c0-0.02,0-0.04,0-0.06c0-0.01,0-0.01,0-0.02
|
||||
c0-0.03,0-0.05-0.01-0.08c0-0.01,0-0.01-0.01-0.02c-0.01-0.02-0.01-0.04-0.02-0.06c0-0.01-0.01-0.02-0.01-0.03
|
||||
c-0.01-0.02-0.02-0.03-0.03-0.05c0-0.01-0.01-0.02-0.01-0.02c-0.02-0.02-0.03-0.05-0.05-0.07L39.56,1.08
|
||||
c-0.14-0.14-0.35-0.18-0.53-0.11c-0.18,0.08-0.3,0.25-0.3,0.45v0.02v38.34L25.26,16.61c0,0,0,0,0-0.01
|
||||
c-0.01-0.02-0.03-0.05-0.05-0.07c0,0-0.01-0.01-0.01-0.01c-0.01-0.01-0.03-0.03-0.04-0.04c0,0-0.01,0-0.01-0.01
|
||||
c-0.01-0.01-0.02-0.01-0.02-0.02c-0.02-0.01-0.03-0.02-0.05-0.03c-0.01,0-0.02-0.01-0.03-0.01c-0.02-0.01-0.04-0.02-0.06-0.02
|
||||
c-0.01,0-0.01,0-0.02-0.01c-0.03-0.01-0.05-0.01-0.08-0.02c-0.01,0-0.01,0-0.02,0c-0.02,0-0.04,0-0.06,0c-0.01,0-0.02,0-0.03,0
|
||||
c-0.02,0-0.04,0.01-0.06,0.01c-0.01,0-0.02,0-0.03,0.01c-0.03,0.01-0.05,0.02-0.08,0.03L0.96,28.48
|
||||
c-0.17,0.09-0.28,0.27-0.26,0.47c0.02,0.2,0.15,0.36,0.33,0.42l0.02,0.01l36.46,11.85l-26.2,5.66c0,0,0,0-0.01,0
|
||||
c-0.03,0.01-0.05,0.01-0.08,0.03c0,0-0.01,0-0.01,0.01c-0.02,0.01-0.03,0.02-0.05,0.03c0,0-0.01,0.01-0.01,0.01
|
||||
c-0.01,0.01-0.02,0.01-0.02,0.02c-0.01,0.01-0.03,0.02-0.04,0.04c-0.01,0.01-0.01,0.01-0.02,0.02c-0.01,0.01-0.03,0.03-0.04,0.05
|
||||
c0,0.01-0.01,0.01-0.01,0.02c-0.02,0.02-0.03,0.05-0.04,0.07c0,0.01,0,0.01-0.01,0.02c-0.01,0.02-0.01,0.04-0.02,0.06
|
||||
c0,0.01,0,0.02-0.01,0.03c0,0.02-0.01,0.04-0.01,0.06c0,0.01,0,0.02,0,0.03c0,0.03,0,0.06,0.01,0.09l4.15,26.23
|
||||
c0.03,0.19,0.17,0.35,0.37,0.4c0.19,0.05,0.39-0.03,0.51-0.19l0.01-0.02L38.5,42.84L35.79,69.5c0,0,0,0,0,0.01
|
||||
c0,0.03,0,0.06,0,0.09c0,0,0,0.01,0,0.01c0,0.02,0,0.04,0.01,0.06c0,0,0,0.01,0,0.01c0,0.01,0.01,0.02,0.01,0.03
|
||||
c0.01,0.02,0.01,0.03,0.02,0.05c0,0.01,0.01,0.02,0.01,0.03c0.01,0.02,0.02,0.04,0.03,0.05c0,0.01,0.01,0.01,0.01,0.02
|
||||
c0.02,0.02,0.04,0.04,0.06,0.06c0,0,0.01,0.01,0.01,0.01c0.02,0.01,0.03,0.03,0.05,0.04c0.01,0.01,0.02,0.01,0.02,0.01
|
||||
c0.02,0.01,0.03,0.02,0.05,0.02c0.01,0,0.02,0.01,0.03,0.01c0.03,0.01,0.05,0.02,0.08,0.02l26.23,4.16
|
||||
c0.19,0.03,0.39-0.06,0.49-0.23c0.1-0.17,0.09-0.38-0.02-0.54l-0.01-0.02L40.34,42.39l24.52,10.82c0,0,0,0,0.01,0
|
||||
c0.03,0.01,0.05,0.02,0.08,0.03c0,0,0.01,0,0.01,0c0.02,0,0.04,0.01,0.06,0.01c0,0,0.01,0,0.01,0c0.01,0,0.02,0,0.03,0
|
||||
c0.02,0,0.04,0,0.06,0c0.01,0,0.02,0,0.03,0c0.02,0,0.04-0.01,0.06-0.02c0.01,0,0.01,0,0.02-0.01c0.03-0.01,0.05-0.02,0.07-0.03
|
||||
c0.01,0,0.01-0.01,0.02-0.01c0.02-0.01,0.04-0.02,0.05-0.04c0.01-0.01,0.01-0.01,0.02-0.02c0.01-0.01,0.03-0.03,0.04-0.04
|
||||
c0.01-0.01,0.01-0.01,0.02-0.02c0.02-0.02,0.03-0.05,0.05-0.07l12.06-23.66C77.64,29.15,77.61,28.94,77.49,28.79z M46.01,32.75
|
||||
L40,20.69h16.9L46.01,32.75z M56.36,19.26l0.46,0.46H39.7c0-2.45,0-13.48,0-17.12L56.36,19.26z M39.7,22.26l5.65,11.35l-5.65,6.31
|
||||
V22.26z M2.44,28.81l21-10.7l0.58-0.29L18.72,34.1C16.4,33.34,5.9,29.94,2.44,28.81z M24.96,18.05l8.11,14.09l-13.33,1.98
|
||||
L24.96,18.05z M33.68,33.02l4.26,7.32l-16.8-5.46L33.68,33.02z M15.86,72.35l-3.68-23.28l-0.1-0.64L25.92,58.5
|
||||
C24.48,60.48,18,69.41,15.86,72.35z M12.58,47.61l15.91-3.36l-2.23,13.29L12.58,47.61z M27.42,56.44l2.1-12.5l8.28-1.79
|
||||
L27.42,56.44z M61.42,73.05l-23.28-3.69l-0.64-0.1l13.85-10.06C52.79,61.17,59.27,70.1,61.42,73.05z M36.87,68.52l1.72-16.17
|
||||
l11.95,6.23L36.87,68.52z M38.62,51.27l0.86-8.43l10.38,14.29L38.62,51.27z M40.65,41.47l16.8-5.46l-9.05,8.88L40.65,41.47z
|
||||
M49.42,45.25l9.62-9.44l5.22,16.07L49.42,45.25z M65.45,50.93l-0.29,0.58l-5.29-16.28c2.33-0.76,12.82-4.17,16.29-5.29
|
||||
L65.45,50.93z"/>
|
||||
</g>
|
||||
</g>
|
||||
<g>
|
||||
</g>
|
||||
<g>
|
||||
</g>
|
||||
<g>
|
||||
</g>
|
||||
<g>
|
||||
</g>
|
||||
<g>
|
||||
</g>
|
||||
<g>
|
||||
</g>
|
||||
<g>
|
||||
</g>
|
||||
<g>
|
||||
</g>
|
||||
<g>
|
||||
</g>
|
||||
<g>
|
||||
</g>
|
||||
<g>
|
||||
</g>
|
||||
<g>
|
||||
</g>
|
||||
<g>
|
||||
</g>
|
||||
<g>
|
||||
</g>
|
||||
<g>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 9.6 KiB |
BIN
images/Vitis-AI.png
Normal file
BIN
images/Vitis-AI.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 328 KiB |
BIN
images/demo.jpg
Normal file
BIN
images/demo.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 167 KiB |
BIN
images/dog.jpeg
Normal file
BIN
images/dog.jpeg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 6.7 KiB |
BIN
images/layered-architecture.png
Normal file
BIN
images/layered-architecture.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 12 KiB |
BIN
images/mkl-dnn_node.png
Normal file
BIN
images/mkl-dnn_node.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 50 KiB |
BIN
images/mkl-dnn_subgraph.png
Normal file
BIN
images/mkl-dnn_subgraph.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 42 KiB |
BIN
images/mnist-screenshot.png
Normal file
BIN
images/mnist-screenshot.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 6.1 KiB |
BIN
images/ngraph-logo.png
Normal file
BIN
images/ngraph-logo.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 22 KiB |
BIN
images/out.jpg
Normal file
BIN
images/out.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 308 KiB |
Loading…
Reference in a new issue