saymrwulf/onnxruntime
1
0
Fork 0
mirror of https://github.com/saymrwulf/onnxruntime.git synced 2026-07-25 19:48:11 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Update the operator documentation generation (#7787)

Browse source 
* Update the operator documentation generation
  - Make layout a little nicer
  - Update to latest supported operators including training
  - Fix some links that are broken when the docs content is copied to github-pages
  - Fix incorrect usage of 'onnx.ai.ml' as the default domain
    - ML ops are now separated from the real default domain of 'onnx.ai'
  - Include CPU, CUDA and training kernels
    - exclude DNNL as it's not an EP we own

* There are separate paths for CUDA and CUDNN as they are not guaranteed to be in the same location on a Windows machine. Use the CUDNN path when looking for the CUDNN library.

* Enable validation of both contrib ops and operator kernels in build
Filter generation so it's deterministic
Add ability for CI to publish the md files as build artifacts if they differ so a developer can download and add to their PR to resolve any diffs.
Remove workarounds for github-pages as that will now link to the github docs which display correctly
This commit is contained in:
Scott McKay 2021-06-02 17:47:40 +10:00 committed by GitHub
parent 3d734a1cdc
commit 0fbec1b9c1
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
8 changed files with 563 additions and 854 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

347
docs/ContribOperators.md
View file

@ -1,7 +1,6 @@
## Contrib Operator Schemas
*This file is automatically generated from the
[def files](/onnxruntime/core/graph/contrib_ops/contrib_defs.cc) via [this script](/tools/python/gen_contrib_doc.py).
Do not modify directly and instead edit operator definitions.*
*This file is automatically generated from the registered contrib operator schemas by [this script](https://github.com/microsoft/onnxruntime/blob/master/tools/python/gen_contrib_doc.py).
Do not modify directly.*
* com.microsoft
* <a href="#com.microsoft.Attention">com.microsoft.Attention</a>
@ -58,15 +57,6 @@
* <a href="#com.microsoft.Unique">com.microsoft.Unique</a>
* <a href="#com.microsoft.WordConvEmbedding">com.microsoft.WordConvEmbedding</a>
* <sub>experimental</sub> <a href="#com.microsoft.IsAllFinite">com.microsoft.IsAllFinite</a>
* com.microsoft.nchwc
* <a href="#com.microsoft.nchwc.AveragePool">com.microsoft.nchwc.AveragePool</a>
* <a href="#com.microsoft.nchwc.Conv">com.microsoft.nchwc.Conv</a>
* <a href="#com.microsoft.nchwc.GlobalAveragePool">com.microsoft.nchwc.GlobalAveragePool</a>
* <a href="#com.microsoft.nchwc.GlobalMaxPool">com.microsoft.nchwc.GlobalMaxPool</a>
* <a href="#com.microsoft.nchwc.MaxPool">com.microsoft.nchwc.MaxPool</a>
* <a href="#com.microsoft.nchwc.ReorderInput">com.microsoft.nchwc.ReorderInput</a>
* <a href="#com.microsoft.nchwc.ReorderOutput">com.microsoft.nchwc.ReorderOutput</a>
* <a href="#com.microsoft.nchwc.Upsample">com.microsoft.nchwc.Upsample</a>
## com.microsoft
### <a name="com.microsoft.Attention"></a><a name="com.microsoft.attention">**com.microsoft.Attention**</a>
@ -2762,336 +2752,3 @@ No versioning maintained for experimental ops.
</dl>
## com.microsoft.nchwc
### <a name="com.microsoft.nchwc.AveragePool"></a><a name="com.microsoft.nchwc.averagepool">**com.microsoft.nchwc.AveragePool**</a>
For internal use.
#### Version
This version of the operator has been available since version 1 of the 'com.microsoft.nchwc' operator set.
#### Attributes
<dl>
<dt><tt>auto_pad</tt> : string</dt>
<dd></dd>
<dt><tt>ceil_mode</tt> : int</dt>
<dd></dd>
<dt><tt>count_include_pad</tt> : int</dt>
<dd></dd>
<dt><tt>dilations</tt> : list of ints</dt>
<dd></dd>
<dt><tt>kernel_shape</tt> : list of ints (required)</dt>
<dd></dd>
<dt><tt>pads</tt> : list of ints</dt>
<dd></dd>
<dt><tt>strides</tt> : list of ints</dt>
<dd></dd>
</dl>
#### Inputs
<dl>
<dt><tt>X</tt> : T</dt>
<dd></dd>
</dl>
#### Outputs
<dl>
<dt><tt>Y</tt> : T</dt>
<dd></dd>
</dl>
#### Type Constraints
<dl>
<dt><tt>T</tt> : tensor(float)</dt>
<dd>Constrain input and output types to float tensors</dd>
</dl>
### <a name="com.microsoft.nchwc.Conv"></a><a name="com.microsoft.nchwc.conv">**com.microsoft.nchwc.Conv**</a>
For internal use.
#### Version
This version of the operator has been available since version 1 of the 'com.microsoft.nchwc' operator set.
#### Attributes
<dl>
<dt><tt>activation</tt> : string</dt>
<dd></dd>
<dt><tt>activation_params</tt> : list of floats</dt>
<dd></dd>
<dt><tt>auto_pad</tt> : string</dt>
<dd></dd>
<dt><tt>dilations</tt> : list of ints</dt>
<dd></dd>
<dt><tt>group</tt> : int</dt>
<dd></dd>
<dt><tt>kernel_shape</tt> : list of ints</dt>
<dd></dd>
<dt><tt>pads</tt> : list of ints</dt>
<dd></dd>
<dt><tt>strides</tt> : list of ints</dt>
<dd></dd>
</dl>
#### Inputs (2 - 4)
<dl>
<dt><tt>X</tt> : T</dt>
<dd></dd>
<dt><tt>W</tt> : T</dt>
<dd></dd>
<dt><tt>B</tt> (optional) : T</dt>
<dd></dd>
<dt><tt>Sum</tt> (optional) : T</dt>
<dd></dd>
</dl>
#### Outputs
<dl>
<dt><tt>Y</tt> : T</dt>
<dd></dd>
</dl>
#### Type Constraints
<dl>
<dt><tt>T</tt> : tensor(float)</dt>
<dd>Constrain input and output types to float tensors</dd>
</dl>
### <a name="com.microsoft.nchwc.GlobalAveragePool"></a><a name="com.microsoft.nchwc.globalaveragepool">**com.microsoft.nchwc.GlobalAveragePool**</a>
For internal use.
#### Version
This version of the operator has been available since version 1 of the 'com.microsoft.nchwc' operator set.
#### Inputs
<dl>
<dt><tt>X</tt> : T</dt>
<dd></dd>
</dl>
#### Outputs
<dl>
<dt><tt>Y</tt> : T</dt>
<dd></dd>
</dl>
#### Type Constraints
<dl>
<dt><tt>T</tt> : tensor(float)</dt>
<dd>Constrain input and output types to float tensors</dd>
</dl>
### <a name="com.microsoft.nchwc.GlobalMaxPool"></a><a name="com.microsoft.nchwc.globalmaxpool">**com.microsoft.nchwc.GlobalMaxPool**</a>
For internal use.
#### Version
This version of the operator has been available since version 1 of the 'com.microsoft.nchwc' operator set.
#### Inputs
<dl>
<dt><tt>X</tt> : T</dt>
<dd></dd>
</dl>
#### Outputs
<dl>
<dt><tt>Y</tt> : T</dt>
<dd></dd>
</dl>
#### Type Constraints
<dl>
<dt><tt>T</tt> : tensor(float)</dt>
<dd>Constrain input and output types to float tensors</dd>
</dl>
### <a name="com.microsoft.nchwc.MaxPool"></a><a name="com.microsoft.nchwc.maxpool">**com.microsoft.nchwc.MaxPool**</a>
For internal use.
#### Version
This version of the operator has been available since version 1 of the 'com.microsoft.nchwc' operator set.
#### Attributes
<dl>
<dt><tt>auto_pad</tt> : string</dt>
<dd></dd>
<dt><tt>ceil_mode</tt> : int</dt>
<dd></dd>
<dt><tt>dilations</tt> : list of ints</dt>
<dd></dd>
<dt><tt>kernel_shape</tt> : list of ints (required)</dt>
<dd></dd>
<dt><tt>pads</tt> : list of ints</dt>
<dd></dd>
<dt><tt>storage_order</tt> : int</dt>
<dd></dd>
<dt><tt>strides</tt> : list of ints</dt>
<dd></dd>
</dl>
#### Inputs
<dl>
<dt><tt>X</tt> : T</dt>
<dd></dd>
</dl>
#### Outputs
<dl>
<dt><tt>Y</tt> : T</dt>
<dd></dd>
</dl>
#### Type Constraints
<dl>
<dt><tt>T</tt> : tensor(float)</dt>
<dd>Constrain input and output types to float tensors</dd>
</dl>
### <a name="com.microsoft.nchwc.ReorderInput"></a><a name="com.microsoft.nchwc.reorderinput">**com.microsoft.nchwc.ReorderInput**</a>
For internal use.
#### Version
This version of the operator has been available since version 1 of the 'com.microsoft.nchwc' operator set.
#### Attributes
<dl>
<dt><tt>channels_last</tt> : int</dt>
<dd></dd>
</dl>
#### Inputs
<dl>
<dt><tt>X</tt> : T</dt>
<dd></dd>
</dl>
#### Outputs
<dl>
<dt><tt>Y</tt> : T</dt>
<dd></dd>
</dl>
#### Type Constraints
<dl>
<dt><tt>T</tt> : tensor(float)</dt>
<dd>Constrain input and output types to float tensors</dd>
</dl>
### <a name="com.microsoft.nchwc.ReorderOutput"></a><a name="com.microsoft.nchwc.reorderoutput">**com.microsoft.nchwc.ReorderOutput**</a>
For internal use.
#### Version
This version of the operator has been available since version 1 of the 'com.microsoft.nchwc' operator set.
#### Attributes
<dl>
<dt><tt>channels</tt> : int</dt>
<dd></dd>
<dt><tt>channels_last</tt> : int</dt>
<dd></dd>
</dl>
#### Inputs
<dl>
<dt><tt>X</tt> : T</dt>
<dd></dd>
</dl>
#### Outputs
<dl>
<dt><tt>Y</tt> : T</dt>
<dd></dd>
</dl>
#### Type Constraints
<dl>
<dt><tt>T</tt> : tensor(float)</dt>
<dd>Constrain input and output types to float tensors</dd>
</dl>
### <a name="com.microsoft.nchwc.Upsample"></a><a name="com.microsoft.nchwc.upsample">**com.microsoft.nchwc.Upsample**</a>
For internal use.
#### Version
This version of the operator has been available since version 1 of the 'com.microsoft.nchwc' operator set.
#### Attributes
<dl>
<dt><tt>coordinate_transformation_mode</tt> : string</dt>
<dd></dd>
<dt><tt>mode</tt> : string</dt>
<dd></dd>
<dt><tt>scales</tt> : list of ints</dt>
<dd></dd>
</dl>
#### Inputs
<dl>
<dt><tt>X</tt> : T</dt>
<dd></dd>
</dl>
#### Outputs
<dl>
<dt><tt>Y</tt> : T</dt>
<dd></dd>
</dl>
#### Type Constraints
<dl>
<dt><tt>T</tt> : tensor(float)</dt>
<dd>Constrain input and output types to float tensors</dd>
</dl>

802
docs/OperatorKernels.md
View file

File diff suppressed because it is too large Load diff

15
onnxruntime/python/onnxruntime_pybind_state.cc
View file

@ -828,17 +828,10 @@ void addGlobalMethods(py::module& m, Environment& env) {
std::vector<std::shared_ptr<onnxruntime::IExecutionProviderFactory>> factories = {
onnxruntime::CreateExecutionProviderFactory_CPU(0),
#ifdef USE_CUDA
onnxruntime::CreateExecutionProviderFactory_CUDA(
[&]() {
CUDAExecutionProviderInfo info{};
info.device_id = cuda_device_id;
info.gpu_mem_limit = gpu_mem_limit;
info.arena_extend_strategy = arena_extend_strategy;
info.cudnn_conv_algo_search = cudnn_conv_algo_search;
info.do_copy_in_default_stream = do_copy_in_default_stream;
info.external_allocator_info = external_allocator_info;
return info;
}()),
[]() {
OrtCUDAProviderOptions provider_options{};
return CreateExecutionProviderFactory_Cuda(&provider_options);
}(),
#endif
#ifdef USE_ROCM
onnxruntime::CreateExecutionProviderFactory_ROCM(

107
tools/ci_build/build.py
View file

@ -149,8 +149,7 @@ def parse_arguments():
help="Use parallel build. The optional value specifies the maximum number of parallel jobs. "
"If the optional value is 0 or unspecified, it is interpreted as the number of CPUs.")
parser.add_argument("--test", action='store_true', help="Run unit tests.")
parser.add_argument(
"--skip_tests", action='store_true', help="Skip all tests.")
parser.add_argument("--skip_tests", action='store_true', help="Skip all tests.")
# Training options
parser.add_argument(
@ -184,9 +183,10 @@ def parse_arguments():
available test data directories.""")
# generate documentation
parser.add_argument(
"--gen_doc", action='store_true',
help="Generate documentation on contrib ops")
parser.add_argument("--gen_doc", nargs='?', const='yes', type=str,
help="Generate documentation listing standard ONNX operators and types implemented by "
"various execution providers and contrib operator schemas. "
"Use `--gen_doc validate` to validate these match the current contents in /docs.")
parser.add_argument(
"--gen-api-doc", action='store_true',
@ -412,15 +412,11 @@ def parse_arguments():
"--use_full_protobuf", action='store_true',
help="Use the full protobuf library")
parser.add_argument(
"--skip_onnx_tests", action='store_true', help="Explicitly disable "
"all onnx related tests. Note: Use --skip_tests to skip all tests.")
parser.add_argument(
"--skip_winml_tests", action='store_true',
help="Explicitly disable all WinML related tests")
parser.add_argument(
"--skip_nodejs_tests", action='store_true',
help="Explicitly disable all Node.js binding tests")
parser.add_argument("--skip_onnx_tests", action='store_true',
help="Explicitly disable all onnx related tests. Note: Use --skip_tests to skip all tests.")
parser.add_argument("--skip_winml_tests", action='store_true', help="Explicitly disable all WinML related tests")
parser.add_argument("--skip_nodejs_tests", action='store_true', help="Explicitly disable all Node.js binding tests")
parser.add_argument(
"--enable_msvc_static_runtime", action='store_true',
help="Enable static linking of MSVC runtimes.")
@ -1770,52 +1766,49 @@ def build_protoc_for_host(cmake_path, source_dir, build_dir, args):
return expected_protoc_path
def generate_documentation(source_dir, build_dir, configs):
def generate_documentation(source_dir, build_dir, configs, validate):
# Randomly choose one build config
config = next(iter(configs))
cwd = get_config_build_dir(build_dir, config)
if is_windows():
cwd = os.path.join(cwd, config)
operator_doc_path = os.path.join(source_dir, 'docs', 'ContribOperators.md')
opkernel_doc_path = os.path.join(source_dir, 'docs', 'OperatorKernels.md')
shutil.copy(
os.path.join(source_dir, 'tools', 'python', 'gen_contrib_doc.py'), cwd)
shutil.copy(
os.path.join(source_dir, 'tools', 'python', 'gen_opkernel_doc.py'),
cwd)
run_subprocess(
[sys.executable,
'gen_contrib_doc.py',
'--output_path', operator_doc_path], cwd=cwd)
run_subprocess(
[sys.executable,
'gen_opkernel_doc.py',
'--output_path', opkernel_doc_path], cwd=cwd)
docdiff = ''
try:
docdiff = subprocess.check_output(['git', 'diff', opkernel_doc_path], cwd=source_dir)
except subprocess.CalledProcessError:
print('git diff returned non-zero error code')
if len(docdiff) > 0:
# Show warning instead of throwing exception, because it is
# dependent on build configuration for including
# execution propviders
log.warning(
'The updated opkernel document file ' + str(opkernel_doc_path) +
' is different from the checked in version. Consider '
'regenerating the file with CPU, DNNL and CUDA providers enabled.')
log.debug('diff:\n' + str(docdiff))
docdiff = ''
try:
docdiff = subprocess.check_output(['git', 'diff', operator_doc_path], cwd=source_dir)
except subprocess.CalledProcessError:
print('git diff returned non-zero error code')
if len(docdiff) > 0:
raise BuildError(
'The updated operator document file ' +
str(operator_doc_path) + ' must be checked in.\n diff:\n' +
str(docdiff))
contrib_op_doc_path = os.path.join(source_dir, 'docs', 'ContribOperators.md')
opkernel_doc_path = os.path.join(source_dir, 'docs', 'OperatorKernels.md')
shutil.copy(os.path.join(source_dir, 'tools', 'python', 'gen_contrib_doc.py'), cwd)
shutil.copy(os.path.join(source_dir, 'tools', 'python', 'gen_opkernel_doc.py'), cwd)
# limit to just com.microsoft (excludes purely internal stuff like com.microsoft.nchwc).
run_subprocess([sys.executable, 'gen_contrib_doc.py', '--output_path', contrib_op_doc_path,
'--domains', 'com.microsoft'], cwd=cwd)
# we currently limit the documentation created by a build to the CPU and CUDA EPs.
# Run get_opkernel_doc.py directly if you need/want documentation from other EPs that are enabled in the build.
run_subprocess([sys.executable, 'gen_opkernel_doc.py', '--output_path', opkernel_doc_path,
'--providers', 'CPU', 'CUDA'], cwd=cwd)
if validate:
try:
have_diff = False
def diff_file(path, regenerate_qualifiers=''):
diff = subprocess.check_output(['git', 'diff', path], cwd=source_dir)
if diff:
nonlocal have_diff
have_diff = True
log.warning('The updated document {} is different from the checked in version. '
'Please regenerate the file{}, or copy the updated version from the '
'CI build\'s published artifacts if applicable.'.format(path, regenerate_qualifiers))
log.debug('diff:\n' + str(diff))
diff_file(opkernel_doc_path, ' with CPU and CUDA execution providers enabled')
diff_file(contrib_op_doc_path)
if have_diff:
# Output for the CI to publish the updated md files as an artifact
print('##vso[task.setvariable variable=DocUpdateNeeded]true')
raise BuildError('Generated documents have diffs. Check build output for details.')
except subprocess.CalledProcessError:
raise BuildError('git diff returned non-zero error code')
def main():
@ -2052,10 +2045,13 @@ def main():
"currently through this script")
if not is_docker() and not args.use_acl and not args.use_armnn:
install_python_deps()
if args.enable_pybind and is_windows():
install_python_deps(args.numpy_version)
if args.enable_onnx_tests:
setup_test_data(build_dir, configs)
if args.use_cuda and args.cuda_version is None:
if is_windows():
# cuda_version is used while generating version_info.py on Windows.
@ -2064,6 +2060,7 @@ def main():
args.cuda_version = ""
if args.use_rocm and args.rocm_version is None:
args.rocm_version = ""
generate_build_tree(
cmake_path, source_dir, build_dir, cuda_home, cudnn_home, rocm_home, mpi_home, nccl_home,
tensorrt_home, migraphx_home, acl_home, acl_libs, armnn_home, armnn_libs,
@ -2142,7 +2139,7 @@ def main():
args.use_dnnl)
if args.gen_doc and (args.build or args.test):
generate_documentation(source_dir, build_dir, configs)
generate_documentation(source_dir, build_dir, configs, args.gen_doc == 'validate')
if args.gen_api_doc and (args.build or args.test):
print('Generating Python doc for ORTModule...')

4
tools/ci_build/github/azure-pipelines/win-ci-pipeline.yml
View file

@ -14,7 +14,7 @@ jobs:
UseOmp: '--use_openmp'
EnvSetupScript: setup_env.bat
buildArch: x64
additionalBuildFlags: --use_dnnl --build_java --build_nodejs --gen_doc
additionalBuildFlags: --use_dnnl --build_java --build_nodejs
msbuildPlatform: x64
isX86: false
x64_release:
@ -22,7 +22,7 @@ jobs:
UseOmp: ''
EnvSetupScript: setup_env.bat
buildArch: x64
additionalBuildFlags: --use_dnnl --build_java --build_nodejs --gen_doc
additionalBuildFlags: --use_dnnl --build_java --build_nodejs
msbuildPlatform: x64
isX86: false
x86_release:

23
tools/ci_build/github/azure-pipelines/win-gpu-ci-pipeline.yml
View file

@ -21,6 +21,7 @@ jobs:
msbuildPlatform: x64
isX86: false
ALLOW_RELEASED_ONNX_OPSET_ONLY: '0'
DocUpdateNeeded: false
timeoutInMinutes: 180
workspace:
clean: all
@ -94,7 +95,7 @@ jobs:
displayName: 'Generate cmake config'
inputs:
scriptPath: '$(Build.SourcesDirectory)\tools\ci_build\build.py'
arguments: '--config $(BuildConfig) --build_dir $(Build.BinariesDirectory) $(UseOmp) --skip_submodule_sync --build_shared_lib --update --cmake_generator "Visual Studio 16 2019" --build_wheel --disable_rtti --use_winml --build_shared_lib --enable_onnx_tests --enable_wcos $(additionalBuildFlags)'
arguments: '--config $(BuildConfig) --build_dir $(Build.BinariesDirectory) $(UseOmp) --skip_submodule_sync --build_shared_lib --update --cmake_generator "Visual Studio 16 2019" --build_wheel --disable_rtti --use_winml --build_shared_lib --enable_onnx_tests --enable_wcos --gen_doc $(additionalBuildFlags)'
workingDirectory: '$(Build.BinariesDirectory)'
- task: VSBuild@1
@ -147,12 +148,28 @@ jobs:
- powershell: |
Get-ChildItem -Path dist/*.whl | foreach {pip --disable-pip-version-check install --upgrade $_.fullname}
python $(Build.SourcesDirectory)\tools\ci_build\build.py --config $(BuildConfig) --build_dir $(Build.BinariesDirectory) $(UseOmp) --skip_submodule_sync --build_shared_lib --test --cmake_generator "Visual Studio 16 2019" --build_wheel --disable_rtti --use_winml --build_shared_lib --enable_onnx_tests --enable_wcos $(additionalBuildFlags)
python $(Build.SourcesDirectory)\tools\ci_build\build.py --config $(BuildConfig) --build_dir $(Build.BinariesDirectory) $(UseOmp) --test --enable_onnx_tests --gen_doc validate $(additionalBuildFlags)
workingDirectory: '$(Build.BinariesDirectory)\$(BuildConfig)\$(BuildConfig)'
condition: and(succeeded(), eq('${{ parameters.RunOnnxRuntimeTests}}', true))
displayName: 'Run tests'
# if the validation from --gen_doc failed it sets a variable so we can publish the latest version of the docs
# as an artifact, allowing a developer to download this and replace the current version instead of having to build
# and generate the docs locally themselves. handle each of the two md files separately - simpler than copying
# them to another location and publishing from there in a single task.
- task: PublishBuildArtifacts@1
condition: and(failed(), eq(variables['DocUpdateNeeded'], 'true'))
inputs:
pathtoPublish: '$(Build.SourcesDirectory)/docs/OperatorKernels.md'
artifactName: 'OperatorKernels.md'
- task: PublishBuildArtifacts@1
condition: and(failed(), eq(variables['DocUpdateNeeded'], 'true'))
inputs:
pathtoPublish: '$(Build.SourcesDirectory)/docs/ContribOperators.md'
artifactName: 'ContribOperators.md'
- task: PythonScript@0
displayName: 'Regenerate cmake config with STATIC_ANALYSIS=ON '
condition: and(succeeded(), eq(variables['BuildConfig'], 'RelWithDebInfo'))

42
tools/python/gen_contrib_doc.py
View file

@ -10,6 +10,7 @@ from __future__ import unicode_literals
from collections import defaultdict
import io
import os
import pathlib
import sys
import argparse
@ -18,7 +19,7 @@ import numpy as np # type: ignore
import onnxruntime.capi.onnxruntime_pybind11_state as rtpy
from onnxruntime.capi.onnxruntime_pybind11_state import schemadef # noqa: F401
from onnxruntime.capi.onnxruntime_pybind11_state.schemadef import OpSchema # noqa: F401
from typing import Any, Text, Sequence, Dict, List, Type, Set, Tuple
from typing import Any, Text, Sequence, Dict, List, Set, Tuple
from onnx import AttributeProto, FunctionProto
ONNX_ML = not bool(os.getenv('ONNX_ML') == '0')
@ -37,9 +38,13 @@ def display_number(v): # type: (int) -> Text
return Text(v)
def should_render_domain(domain): # type: (Text) -> bool
def should_render_domain(domain, domain_filter): # type: (Text) -> bool
if domain == ONNX_DOMAIN or domain == '' or domain == ONNX_ML_DOMAIN or domain == 'ai.onnx.ml':
return False
if domain_filter and domain not in domain_filter:
return False
return True
@ -308,19 +313,18 @@ def support_level_str(level): # type: (OpSchema.SupportType) -> Text
# "<sub>experimental</sub> " if status == OperatorStatus.Value('EXPERIMENTAL') else "" # type: ignore
def main(args): # type: (Type[Args]) -> None
def main(output_path: str, domain_filter: [str]):
with io.open(args.output, 'w', newline='', encoding="utf-8") as fout:
with io.open(output_path, 'w', newline='', encoding="utf-8") as fout:
fout.write('## Contrib Operator Schemas\n')
fout.write(
"*This file is automatically generated from the\n"
" [def files](/onnxruntime/core/graph/contrib_ops/contrib_defs.cc) via "
"[this script](/tools/python/gen_contrib_doc.py).\n"
" Do not modify directly and instead edit operator definitions.*\n")
"*This file is automatically generated from the registered contrib operator schemas by "
"[this script](https://github.com/microsoft/onnxruntime/blob/master/tools/python/gen_contrib_doc.py).\n"
"Do not modify directly.*\n")
# domain -> support level -> name -> [schema]
index = defaultdict(lambda: defaultdict(lambda: defaultdict(list)) # type: Dict[Text, Dict[int, Dict[Text, List[OpSchema]]]] # noqa: E501
)
index = defaultdict(lambda: defaultdict(lambda: defaultdict(list))) # type: Dict[Text, Dict[int, Dict[Text, List[OpSchema]]]] # noqa: E501
for schema in rtpy.get_all_operator_schema():
index[schema.domain][int(schema.support_level)][schema.name].append(schema)
@ -331,7 +335,7 @@ def main(args): # type: (Type[Args]) -> None
operator_schemas = list() # type: List[Tuple[Text, List[Tuple[int, List[Tuple[Text, OpSchema, List[OpSchema]]]]]]] # noqa: E501
exsting_ops = set() # type: Set[Text]
for domain, _supportmap in sorted(index.items()):
if not should_render_domain(domain):
if not should_render_domain(domain, domain_filter):
continue
processed_supportmap = list()
@ -384,13 +388,13 @@ def main(args): # type: (Type[Args]) -> None
if __name__ == '__main__':
parser = argparse.ArgumentParser(description='ONNX Runtime Operator Documentation Generator')
parser.add_argument('--output_path', help='output markdown file path',
default=os.path.join(os.path.dirname(os.path.realpath(__file__)), 'ContribOperators.md')
)
parser = argparse.ArgumentParser(description='ONNX Runtime Contrib Operator Documentation Generator')
parser.add_argument('--domains', nargs='+',
help="Filter to specified domains. "
"e.g. `--domains com.microsoft com.microsoft.nchwc`")
parser.add_argument('--output_path', help='output markdown file path', type=pathlib.Path, required=True,
default=os.path.join(os.path.dirname(os.path.realpath(__file__)), 'ContribOperators.md'))
args = parser.parse_args()
output_path = args.output_path.resolve()
class Args(object):
output = args.output_path
main(Args)
main(output_path, args.domains)

77
tools/python/gen_opkernel_doc.py
View file

@ -2,13 +2,12 @@
# Copyright (c) Microsoft Corporation. All rights reserved.
# Licensed under the MIT License.
from collections import defaultdict
import argparse
import io
import os
import argparse
import pathlib
from collections import defaultdict
from typing import Type
import onnxruntime.capi.onnxruntime_pybind11_state as rtpy
@ -44,36 +43,49 @@ def format_param_strings(params):
if firstparam:
firstparam = False
else:
s += ' or '
s += '<br><br>or<br><br>'
s += param
return s
def main(args): # type: (Type[Args]) -> None
def expand_providers(provider_filter: [str]):
providers = set()
if provider_filter:
for provider in provider_filter:
p = provider.lower()
if not p.endswith('executionprovider'):
p += 'executionprovider'
providers.add(p)
with io.open(args.output, 'w', newline='', encoding="utf-8") as fout:
fout.write('## Supported Operators Data Types\n')
return providers
def main(output_path: pathlib.Path, provider_filter: [str]):
providers = expand_providers(provider_filter)
with io.open(output_path, 'w', newline='', encoding="utf-8") as fout:
fout.write('## Supported Operators and Data Types\n')
fout.write(
"*This file is automatically generated from the\n"
" [def files](/onnxruntime/core/providers/cpu/cpu_execution_provider.cc) via "
"[this script](/tools/python/gen_opkernel_doc.py).\n"
" Do not modify directly and instead edit operator definitions.*\n")
"*This file is automatically generated from the registered kernels by "
"[this script](https://github.com/microsoft/onnxruntime/blob/master/tools/python/gen_opkernel_doc.py).\n"
"Do not modify directly.*\n\n")
opdef = rtpy.get_all_operator_schema()
paramdict = {}
for schema in opdef:
inputs = schema.inputs
domain = schema.domain
if (domain == ''):
domain = 'ai.onnx.ml'
domain = 'ai.onnx'
fullname = domain+'.'+schema.name
paramstr = '('
paramstr = ''
firstinput = True
if inputs:
for inp in inputs:
if firstinput:
firstinput = False
else:
paramstr += ', '
paramstr += '<br> '
paramstr += '*in* {}:**{}**'.format(inp.name, inp.typeStr)
outputs = schema.outputs
@ -82,10 +94,10 @@ def main(args): # type: (Type[Args]) -> None
if firstinput:
firstinput = False
else:
paramstr += ', '
paramstr += '<br> '
paramstr += '*out* {}:**{}**'.format(outp.name, outp.typeStr)
paramstr += ')'
paramstr += ''
paramset = paramdict.get(fullname, None)
if paramset is None:
paramdict[fullname] = set()
@ -96,12 +108,23 @@ def main(args): # type: (Type[Args]) -> None
for op in rtpy.get_all_opkernel_def():
domain = op.domain
if (domain == ''):
domain = 'ai.onnx.ml'
domain = 'ai.onnx'
index[op.provider][domain][op.op_name].append(op)
fout.write('\n')
# TOC
fout.write('## Execution Providers\n\n')
for provider in sorted(index.keys()):
if providers and provider.lower() not in providers:
continue
fout.write('- [{}](#{})\n'.format(provider, provider.lower()))
fout.write('\n---------------')
for provider, domainmap in sorted(index.items()):
fout.write('\n\n## Operators implemented by '+provider+'\n\n')
if providers and provider.lower() not in providers:
continue
fout.write('\n\n<a name="{}"/>\n\n'.format(provider.lower()))
fout.write('## Operators implemented by {}\n\n'.format(provider))
fout.write('| Op Name | Parameters | OpSet Version | Types Supported |\n')
fout.write('|---------|------------|---------------|-----------------|\n')
for domain, namemap in sorted(domainmap.items()):
@ -138,11 +161,13 @@ def main(args): # type: (Type[Args]) -> None
if __name__ == '__main__':
parser = argparse.ArgumentParser(description='ONNX Runtime Operator Kernel Documentation Generator')
parser.add_argument('--output_path', help='output markdown file path',
default=os.path.join(os.path.dirname(os.path.realpath(__file__)), 'OperatorKernels.md')
)
parser.add_argument('--providers', nargs='+',
help="Filter to specified execution providers. Case-insensitive. "
"Matches provider names from <ORT>/include/onnxruntime/core/graph/constants.h'. "
"'ExecutionProvider' is automatically appended as needed. "
"e.g. `--providers cpu cuda` will match CPUExecutionProvider and CUDAExecutionProvider.")
parser.add_argument('--output_path', help='output markdown file path', type=pathlib.Path, required=True,
default=os.path.join(os.path.dirname(os.path.realpath(__file__)), 'OperatorKernels.md'))
args = parser.parse_args()
class Args(object):
output = args.output_path
main(Args)
main(args.output_path, args.providers)