onnxruntime/tools/ci_build/github/azure-pipelines
Justin Chu d834ec895a
Adopt linrtunner as the linting tool - take 2 (#15085)
### Description

`lintrunner` is a linter runner successfully used by pytorch, onnx and
onnx-script. It provides a uniform experience running linters locally
and in CI. It supports all major dev systems: Windows, Linux and MacOs.
The checks are enforced by the `Python format` workflow.

This PR adopts `lintrunner` to onnxruntime and fixed ~2000 flake8 errors
in Python code. `lintrunner` now runs all required python lints
including `ruff`(replacing `flake8`), `black` and `isort`. Future lints
like `clang-format` can be added.

Most errors are auto-fixed by `ruff` and the fixes should be considered
robust.

Lints that are more complicated to fix are applied `# noqa` for now and
should be fixed in follow up PRs.

### Notable changes

1. This PR **removed some suboptimal patterns**:

	- `not xxx in` -> `xxx not in` membership checks
	- bare excepts (`except:` -> `except Exception`)
	- unused imports
	
	The follow up PR will remove:
	
	- `import *`
	- mutable values as default in function definitions (`def func(a=[])`)
	- more unused imports
	- unused local variables

2. Use `ruff` to replace `flake8`. `ruff` is much (40x) faster than
flake8 and is more robust. We are using it successfully in onnx and
onnx-script. It also supports auto-fixing many flake8 errors.

3. Removed the legacy flake8 ci flow and updated docs.

4. The added workflow supports SARIF code scanning reports on github,
example snapshot:
	

![image](https://user-images.githubusercontent.com/11205048/212598953-d60ce8a9-f242-4fa8-8674-8696b704604a.png)

5. Removed `onnxruntime-python-checks-ci-pipeline` as redundant

### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->

Unified linting experience in CI and local.

Replacing https://github.com/microsoft/onnxruntime/pull/14306

---------

Signed-off-by: Justin Chu <justinchu@microsoft.com>
2023-03-24 15:29:03 -07:00
..
nodejs/templates Update to MacOS-12 (#14924) 2023-03-09 10:18:14 -08:00
nuget/templates fix_macos (#15018) 2023-03-14 21:54:44 -07:00
templates Add smoking tests for all CPU Packages. (#15153) 2023-03-24 12:30:05 +08:00
android-arm64-v8a-QNN-crosscompile-ci-pipeline.yml [QNN EP] Update QNN SDK to 2.8 (#14978) 2023-03-10 13:21:19 -08:00
android-x86_64-crosscompile-ci-pipeline.yml Update to MacOS-12 (#14924) 2023-03-09 10:18:14 -08:00
anybuild.yml
binary-size-checks-pipeline.yml Improve dependency management (#13523) 2022-12-01 09:51:59 -08:00
build-perf-test-binaries-pipeline.yml
c-api-noopenmp-packaging-pipelines.yml [Fix] Error in Linux_Packaging_combined_GPU of nuget packaing pipeline (#15060) 2023-03-16 08:49:37 +08:00
clean-build-docker-image-cache-pipeline.yml
linux-ci-pipeline.yml Update Gradle version (#14862) 2023-03-08 12:22:06 -08:00
linux-cpu-aten-pipeline.yml use build.sourceversion in cache image key (#15019) 2023-03-24 10:01:22 +08:00
linux-cpu-eager-pipeline.yml Disable LazyTensor-ORT Test (#14703) 2023-02-17 17:46:51 +08:00
linux-cpu-minimal-build-ci-pipeline.yml Improve dependency management (#13523) 2022-12-01 09:51:59 -08:00
linux-dnnl-ci-pipeline.yml Improve dependency management (#13523) 2022-12-01 09:51:59 -08:00
linux-gpu-ci-pipeline.yml Update Gradle version (#14862) 2023-03-08 12:22:06 -08:00
linux-gpu-tensorrt-ci-pipeline.yml [TensorRT EP] support TensorRT 8.5 (#13867) 2022-12-14 13:06:03 -08:00
linux-gpu-tensorrt-daily-perf-pipeline.yml Update package pipelines to support TRT 8.5 (#13998) 2022-12-16 15:01:50 -08:00
linux-migraphx-ci-pipeline.yml [ROCm] Update ROCm and MigraphX CI to ROCm5.4 (#14011) 2022-12-22 10:01:05 +08:00
linux-multi-gpu-ci-pipeline.yml Improve dependency management (#13523) 2022-12-01 09:51:59 -08:00
linux-multi-gpu-tensorrt-ci-pipeline.yml
linux-openvino-ci-pipeline.yml Openvino ep 2022.3 v4.3 (#14210) 2023-01-11 16:31:26 -08:00
linux-openvino-nightly-pipeline.yml
linux-qnn-ci-pipeline.yml [QNN EP] Update QNN SDK to 2.8 (#14978) 2023-03-10 13:21:19 -08:00
mac-ci-pipeline.yml make WITHCACHE as an option in MacOS workflow (#14188) 2023-01-10 10:54:19 +08:00
mac-coreml-ci-pipeline.yml Update to MacOS-12 (#14924) 2023-03-09 10:18:14 -08:00
mac-ios-ci-pipeline.yml Add compilation cache in iOS pipeline (#15070) 2023-03-16 21:43:18 +08:00
mac-ios-packaging-pipeline.yml Update to MacOS-12 (#14924) 2023-03-09 10:18:14 -08:00
mac-objc-static-analysis-ci-pipeline.yml Update clang-tidy path for updated Mac image. (#14760) 2023-02-22 09:00:42 -08:00
mac-react-native-ci-pipeline.yml
npm-packaging-pipeline.yml
orttraining-linux-ci-pipeline.yml Update Gradle version (#14862) 2023-03-08 12:22:06 -08:00
orttraining-linux-external-custom-ops.yml
orttraining-linux-gpu-amd-e2e-test-ci-pipeline.yml [ROCm] Fix azcopy issue on ROCm ci pipeline (#13365) 2022-10-20 12:08:57 +08:00
orttraining-linux-gpu-ci-pipeline.yml Update torch to 1.13.1 in CI and packaging pipelines for ort training (#14055) 2023-01-03 20:03:33 -08:00
orttraining-linux-gpu-distributed-e2e-test-pipeline.yml
orttraining-linux-gpu-docker-release-pipeline.yml
orttraining-linux-gpu-ortmodule-distributed-test-ci-pipeline.yml Introduce collective ops to ort inference build (#14399) 2023-02-07 13:47:48 -08:00
orttraining-linux-gpu-ortmodule-test-clear-cache-pipeline.yml
orttraining-linux-gpu-training-apis.yml [Java] Initial on device training support (#14027) 2023-03-08 10:01:08 -08:00
orttraining-linux-nightly-ortmodule-test-pipeline.yml Update acpt image in the training pipeline (#14855) 2023-03-07 14:10:32 -08:00
orttraining-mac-ci-pipeline.yml enable cache in orttraining-mac-ci (#14979) 2023-03-10 07:34:25 +08:00
orttraining-pai-ci-pipeline.yml Enable ccache for HIP objects (#14465) 2023-01-28 22:34:24 +08:00
orttraining-py-packaging-pipeline-cpu.yml Remove python37 and cuda37 packages in orttraing (#15041) 2023-03-15 08:54:15 +08:00
orttraining-py-packaging-pipeline-cuda116.yml Remove python37 and cuda37 packages in orttraing (#15041) 2023-03-15 08:54:15 +08:00
orttraining-py-packaging-pipeline-rocm.yml [ROCm] add rocm5.4.2 to python package pipeline (#15081) 2023-03-20 10:30:14 +08:00
post-merge-jobs.yml Migrating ORT Extensions from Git submodule to cmake FetchContent (#14298) 2023-02-22 19:42:36 -08:00
py-package-build-pipeline.yml
py-package-test-pipeline.yml Add smoking tests for all CPU Packages. (#15153) 2023-03-24 12:30:05 +08:00
py-packaging-pipeline.yml CloudEP (#13855) 2023-01-03 10:03:15 -08:00
sign_ov_ep_binaries.yml Move build machines with Nvidia M60 GPUs to Nvidia T4 (#13170) 2022-10-25 11:21:13 -07:00
snpe-ep-nuget-packaging-pipeline.yml Add yml file for Snpe EP build (#13494) 2022-10-28 19:47:50 -07:00
web-ci-pipeline.yml
web-packaging-pipeline.yml
win-ci-fuzz-testing.yml Improve dependency management (#13523) 2022-12-01 09:51:59 -08:00
win-ci-pipeline.yml Remove intermedia obj files once build finished (#14361) 2023-01-20 13:37:15 +08:00
win-eager-ci-pipeline.yml
win-gpu-ci-pipeline.yml Enable cache for msbuild (#14085) 2023-01-06 11:19:57 +08:00
win-gpu-reduce-op-ci-pipeline.yml Improve dependency management (#13523) 2022-12-01 09:51:59 -08:00
win-gpu-tensorrt-ci-pipeline.yml [TensorRT EP] support TensorRT 8.5 (#13867) 2022-12-14 13:06:03 -08:00
win-qnn-arm64-ci-pipeline.yml [QNN EP] Update QNN SDK to 2.8 (#14978) 2023-03-10 13:21:19 -08:00
win-qnn-ci-pipeline.yml [QNN EP] Update QNN SDK to 2.8 (#14978) 2023-03-10 13:21:19 -08:00