onnxruntime/tools/ci_build/gen_def.py

92 lines
3 KiB
Python
Raw Normal View History

2018-11-20 00:48:22 +00:00
#!/usr/bin/python3
import argparse
import os
2018-11-20 00:48:22 +00:00
def parse_arguments():
parser = argparse.ArgumentParser()
parser.add_argument("--src_root", required=True, help="input symbol file")
parser.add_argument("--output", required=True, help="output file")
2019-08-20 19:04:10 +00:00
parser.add_argument("--output_source", required=True, help="output file")
2018-11-20 00:48:22 +00:00
parser.add_argument("--version_file", required=True, help="VERSION_NUMBER file")
parser.add_argument("--style", required=True, choices=["gcc", "vc", "xcode"])
2018-11-20 00:48:22 +00:00
parser.add_argument("--config", required=True, nargs="+")
return parser.parse_args()
2018-11-20 00:48:22 +00:00
args = parse_arguments()
print(f"Generating symbol file for {args.config!s}")
Adopt linrtunner as the linting tool - take 2 (#15085) ### Description `lintrunner` is a linter runner successfully used by pytorch, onnx and onnx-script. It provides a uniform experience running linters locally and in CI. It supports all major dev systems: Windows, Linux and MacOs. The checks are enforced by the `Python format` workflow. This PR adopts `lintrunner` to onnxruntime and fixed ~2000 flake8 errors in Python code. `lintrunner` now runs all required python lints including `ruff`(replacing `flake8`), `black` and `isort`. Future lints like `clang-format` can be added. Most errors are auto-fixed by `ruff` and the fixes should be considered robust. Lints that are more complicated to fix are applied `# noqa` for now and should be fixed in follow up PRs. ### Notable changes 1. This PR **removed some suboptimal patterns**: - `not xxx in` -> `xxx not in` membership checks - bare excepts (`except:` -> `except Exception`) - unused imports The follow up PR will remove: - `import *` - mutable values as default in function definitions (`def func(a=[])`) - more unused imports - unused local variables 2. Use `ruff` to replace `flake8`. `ruff` is much (40x) faster than flake8 and is more robust. We are using it successfully in onnx and onnx-script. It also supports auto-fixing many flake8 errors. 3. Removed the legacy flake8 ci flow and updated docs. 4. The added workflow supports SARIF code scanning reports on github, example snapshot: ![image](https://user-images.githubusercontent.com/11205048/212598953-d60ce8a9-f242-4fa8-8674-8696b704604a.png) 5. Removed `onnxruntime-python-checks-ci-pipeline` as redundant ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Unified linting experience in CI and local. Replacing https://github.com/microsoft/onnxruntime/pull/14306 --------- Signed-off-by: Justin Chu <justinchu@microsoft.com>
2023-03-24 22:29:03 +00:00
with open(args.version_file) as f:
VERSION_STRING = f.read().strip()
2018-11-20 00:48:22 +00:00
print(f"VERSION:{VERSION_STRING}")
2018-11-20 00:48:22 +00:00
symbols = set()
for c in args.config:
file_name = os.path.join(args.src_root, "core", "providers", c, "symbols.txt")
Adopt linrtunner as the linting tool - take 2 (#15085) ### Description `lintrunner` is a linter runner successfully used by pytorch, onnx and onnx-script. It provides a uniform experience running linters locally and in CI. It supports all major dev systems: Windows, Linux and MacOs. The checks are enforced by the `Python format` workflow. This PR adopts `lintrunner` to onnxruntime and fixed ~2000 flake8 errors in Python code. `lintrunner` now runs all required python lints including `ruff`(replacing `flake8`), `black` and `isort`. Future lints like `clang-format` can be added. Most errors are auto-fixed by `ruff` and the fixes should be considered robust. Lints that are more complicated to fix are applied `# noqa` for now and should be fixed in follow up PRs. ### Notable changes 1. This PR **removed some suboptimal patterns**: - `not xxx in` -> `xxx not in` membership checks - bare excepts (`except:` -> `except Exception`) - unused imports The follow up PR will remove: - `import *` - mutable values as default in function definitions (`def func(a=[])`) - more unused imports - unused local variables 2. Use `ruff` to replace `flake8`. `ruff` is much (40x) faster than flake8 and is more robust. We are using it successfully in onnx and onnx-script. It also supports auto-fixing many flake8 errors. 3. Removed the legacy flake8 ci flow and updated docs. 4. The added workflow supports SARIF code scanning reports on github, example snapshot: ![image](https://user-images.githubusercontent.com/11205048/212598953-d60ce8a9-f242-4fa8-8674-8696b704604a.png) 5. Removed `onnxruntime-python-checks-ci-pipeline` as redundant ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Unified linting experience in CI and local. Replacing https://github.com/microsoft/onnxruntime/pull/14306 --------- Signed-off-by: Justin Chu <justinchu@microsoft.com>
2023-03-24 22:29:03 +00:00
with open(file_name) as file:
for line in file:
line = line.strip() # noqa: PLW2901
if line in symbols:
print("dup symbol: %s", line)
exit(-1)
symbols.add(line)
2018-11-20 00:48:22 +00:00
symbols = sorted(symbols)
symbol_index = 1
with open(args.output, "w") as file:
if args.style == "vc":
file.write("LIBRARY\n")
file.write("EXPORTS\n")
elif args.style == "xcode":
pass # xcode compile don't has any header.
2018-11-20 00:48:22 +00:00
else:
file.write(f"VERS_{VERSION_STRING} {{\n")
file.write(" global:\n")
for symbol in symbols:
if args.style == "vc":
file.write(" %s @%d\n" % (symbol, symbol_index))
elif args.style == "xcode":
file.write(f"_{symbol}\n")
else:
file.write(f" {symbol};\n")
symbol_index += 1
2018-11-20 00:48:22 +00:00
if args.style == "gcc":
file.write(" local:\n")
file.write(" *;\n")
file.write("}; \n")
2019-08-20 19:04:10 +00:00
with open(args.output_source, "w") as file:
file.write("#include <onnxruntime_c_api.h>\n")
for c in args.config:
# WinML adapter should not be exported in platforms other than Windows.
# Exporting OrtGetWinMLAdapter is exported without issues using .def file when compiling for Windows
# so it isn't necessary to include it in generated_source.c
# external symbols are removed, xnnpack ep will be created via the standard ORT API.
# https://github.com/microsoft/onnxruntime/pull/11798
if c not in (
"vitisai",
"winml",
"cuda",
"rocm",
"migraphx",
"qnn",
"snpe",
"xnnpack",
"cann",
"dnnl",
"tensorrt",
"azure",
"webgpu",
):
Adopt linrtunner as the linting tool - take 2 (#15085) ### Description `lintrunner` is a linter runner successfully used by pytorch, onnx and onnx-script. It provides a uniform experience running linters locally and in CI. It supports all major dev systems: Windows, Linux and MacOs. The checks are enforced by the `Python format` workflow. This PR adopts `lintrunner` to onnxruntime and fixed ~2000 flake8 errors in Python code. `lintrunner` now runs all required python lints including `ruff`(replacing `flake8`), `black` and `isort`. Future lints like `clang-format` can be added. Most errors are auto-fixed by `ruff` and the fixes should be considered robust. Lints that are more complicated to fix are applied `# noqa` for now and should be fixed in follow up PRs. ### Notable changes 1. This PR **removed some suboptimal patterns**: - `not xxx in` -> `xxx not in` membership checks - bare excepts (`except:` -> `except Exception`) - unused imports The follow up PR will remove: - `import *` - mutable values as default in function definitions (`def func(a=[])`) - more unused imports - unused local variables 2. Use `ruff` to replace `flake8`. `ruff` is much (40x) faster than flake8 and is more robust. We are using it successfully in onnx and onnx-script. It also supports auto-fixing many flake8 errors. 3. Removed the legacy flake8 ci flow and updated docs. 4. The added workflow supports SARIF code scanning reports on github, example snapshot: ![image](https://user-images.githubusercontent.com/11205048/212598953-d60ce8a9-f242-4fa8-8674-8696b704604a.png) 5. Removed `onnxruntime-python-checks-ci-pipeline` as redundant ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Unified linting experience in CI and local. Replacing https://github.com/microsoft/onnxruntime/pull/14306 --------- Signed-off-by: Justin Chu <justinchu@microsoft.com>
2023-03-24 22:29:03 +00:00
file.write(f"#include <core/providers/{c}/{c}_provider_factory.h>\n")
file.write("void* GetFunctionEntryByName(const char* name){\n")
for symbol in symbols:
if symbol != "OrtGetWinMLAdapter":
Adopt linrtunner as the linting tool - take 2 (#15085) ### Description `lintrunner` is a linter runner successfully used by pytorch, onnx and onnx-script. It provides a uniform experience running linters locally and in CI. It supports all major dev systems: Windows, Linux and MacOs. The checks are enforced by the `Python format` workflow. This PR adopts `lintrunner` to onnxruntime and fixed ~2000 flake8 errors in Python code. `lintrunner` now runs all required python lints including `ruff`(replacing `flake8`), `black` and `isort`. Future lints like `clang-format` can be added. Most errors are auto-fixed by `ruff` and the fixes should be considered robust. Lints that are more complicated to fix are applied `# noqa` for now and should be fixed in follow up PRs. ### Notable changes 1. This PR **removed some suboptimal patterns**: - `not xxx in` -> `xxx not in` membership checks - bare excepts (`except:` -> `except Exception`) - unused imports The follow up PR will remove: - `import *` - mutable values as default in function definitions (`def func(a=[])`) - more unused imports - unused local variables 2. Use `ruff` to replace `flake8`. `ruff` is much (40x) faster than flake8 and is more robust. We are using it successfully in onnx and onnx-script. It also supports auto-fixing many flake8 errors. 3. Removed the legacy flake8 ci flow and updated docs. 4. The added workflow supports SARIF code scanning reports on github, example snapshot: ![image](https://user-images.githubusercontent.com/11205048/212598953-d60ce8a9-f242-4fa8-8674-8696b704604a.png) 5. Removed `onnxruntime-python-checks-ci-pipeline` as redundant ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Unified linting experience in CI and local. Replacing https://github.com/microsoft/onnxruntime/pull/14306 --------- Signed-off-by: Justin Chu <justinchu@microsoft.com>
2023-03-24 22:29:03 +00:00
file.write(f'if(strcmp(name,"{symbol}") ==0) return (void*)&{symbol};\n')
file.write("return NULL;\n")
file.write("}\n")