mirror of
https://github.com/saymrwulf/onnxruntime.git
synced 2026-06-03 23:49:44 +00:00
4dbaa73738
## Summary
* following inference's [set-up for end-to-end
tests](https://github.com/microsoft/onnxruntime/tree/main/js/web/test/e2e),
created an end-to-end test runner for training
* this test runner copies testdata from the [trainingapi
folder](https://github.com/microsoft/onnxruntime/tree/main/onnxruntime/test/testdata/training_api)
* then runs two tests (training session with evalModel & optimizer
model, and training session with the minimum options), and tests if the
ORT-web training package encompasses inference
* these tests check
* createTrainingSession
* runTrainStep
* runOptimizerStep if applicable
* the parameters methods (getParametersSize, loadParametersBuffer, and
getContiguousParameters)
## TL;DR
*
[`js/web/test/training/e2e/run.js`](https://github.com/microsoft/onnxruntime/compare/main...carzh:onnxruntime:carzh/training-e2e-runner?expand=1#diff-c1359c4d401f9ba69e937814219cefe5fd11b151a6ffd084c641af3c82e8216c)
is responsible for setting up and running the end to end tests
*
[`js/web/test/training/e2e/common.js`](https://github.com/microsoft/onnxruntime/compare/main...carzh:onnxruntime:carzh/training-e2e-runner?expand=1#diff-ee5452491b7b2563d175d13d81d10f2323b12b18589aa4c5798962a8b904a4a8)
contains the test function definitions (`testInferenceFunction`,
`testTrainingFunctionMin`, `testTrainingFunctionAll`)
## Flow
* entrypoint: user runs the following command in the terminal: `npm run
test:training:e2e`
*
[`js/web/package.json`](https://github.com/microsoft/onnxruntime/compare/main...carzh:onnxruntime:carzh/training-e2e-runner?expand=1#diff-79275844e75c3c410bb3a71c7f59b2b633e5a3e975c804ffc47220025084da28)
was modified to include an npm script that will run `run.js` which will
run the end to end tests
*
[`js/web/test/training/e2e/run.js`](https://github.com/microsoft/onnxruntime/compare/main...carzh:onnxruntime:carzh/training-e2e-runner?expand=1#diff-c1359c4d401f9ba69e937814219cefe5fd11b151a6ffd084c641af3c82e8216c)
is responsible for
* detecting and installing local tarball packages of ORT-web
* copying training data to the `js/web/training/e2e/data` folder
* starting two Karma processes. Karma is a test runner framework that
simulates testing in the browser.
* In this case, the tests happen in Chrome. We can configure the tests
to run in Edge and other browsers in the future.
* one of these karma processes is self-hosted, meaning it pulls the
ORT-web package from local
* the other karma process is not self-hosted, meaning it pulls the
ORT-web package from another source. In this case, we start an http
server that serves the ORT-web binaries.
*
[`js/web/test/training/e2e/simple-http-server.js`](https://github.com/microsoft/onnxruntime/compare/main...carzh:onnxruntime:carzh/training-e2e-runner?expand=1#diff-f798ab485f3ec26c299fe5b2923574c9e4b090200ba20d490bbf6c183286993c)
is responsible for starting the HTTP server and serving the ORT binary
files. This code almost identical to the same code in the inference E2E
tests.
*
[`js/web/test/training/e2e/karma.conf.js`](https://github.com/microsoft/onnxruntime/compare/main...carzh:onnxruntime:carzh/training-e2e-runner?expand=1#diff-436cfe8f670c768a04895bd4a1874a5e033f85e0e2d84941c62ff1f7c30a9f28)
Karma configuration file that specifies what happens when a karma
process is started. The config specifies Mocha as the testing framework,
which will go through all the loaded files and run any tests that exist
*
[`js/web/test/training/e2e/browser-test-wasm.js`](https://github.com/microsoft/onnxruntime/compare/main...carzh:onnxruntime:carzh/training-e2e-runner?expand=1#diff-13b6155e106dddc7b531ef671186e69b2aadb8a0f4b2f3001db0991567d78221)
File that contains the tests that Mocha will pick up on and run.
* The test functions (such as testInference and testTrainingFunctionAll)
are defined in
[`js/web/test/training/e2e/common.js`](https://github.com/microsoft/onnxruntime/compare/main...carzh:onnxruntime:carzh/training-e2e-runner?expand=1#diff-ee5452491b7b2563d175d13d81d10f2323b12b18589aa4c5798962a8b904a4a8).
## Notes
* I followed the [tests for training
core](b023de0bfc/orttraining/orttraining/test/training_api/core/training_api_tests.cc)
where they randomly generated input for the training session
* E2E tests are triggered by running `npm run test:training:e2e` --
suggestions for alternative script names are appreciated!!!
## Motivation and Context
- adding training bindings for web
54 lines
1.8 KiB
JavaScript
54 lines
1.8 KiB
JavaScript
// Copyright (c) Microsoft Corporation. All rights reserved.
|
|
// Licensed under the MIT License.
|
|
|
|
'use strict';
|
|
|
|
const args = require('minimist')(process.argv.slice(2));
|
|
const SELF_HOST = !!args['self-host'];
|
|
const ORT_MAIN = args['ort-main'];
|
|
const TEST_MAIN = args['test-main'];
|
|
if (typeof TEST_MAIN !== 'string') {
|
|
throw new Error('flag --test-main=<TEST_MAIN_JS_FILE> is required');
|
|
}
|
|
const USER_DATA = args['user-data'];
|
|
if (typeof USER_DATA !== 'string') {
|
|
throw new Error('flag --user-data=<CHROME_USER_DATA_FOLDER> is required');
|
|
}
|
|
|
|
module.exports = function(config) {
|
|
const distPrefix = SELF_HOST ? './node_modules/onnxruntime-web/dist/' : 'http://localhost:8081/dist/';
|
|
config.set({
|
|
frameworks: ['mocha'],
|
|
files: [
|
|
{pattern: distPrefix + ORT_MAIN},
|
|
{pattern: './common.js'},
|
|
{pattern: TEST_MAIN},
|
|
{pattern: './node_modules/onnxruntime-web/dist/*.wasm', included: false, nocache: true},
|
|
{pattern: './data/*', included: false},
|
|
],
|
|
plugins: [require('@chiragrupani/karma-chromium-edge-launcher'), ...config.plugins],
|
|
proxies: {
|
|
'/model.onnx': '/base/model.onnx',
|
|
'/data/': '/base/data/',
|
|
},
|
|
client: {captureConsole: true, mocha: {expose: ['body'], timeout: 60000}},
|
|
reporters: ['mocha'],
|
|
captureTimeout: 120000,
|
|
reportSlowerThan: 100,
|
|
browserDisconnectTimeout: 600000,
|
|
browserNoActivityTimeout: 300000,
|
|
browserDisconnectTolerance: 0,
|
|
browserSocketTimeout: 60000,
|
|
hostname: 'localhost',
|
|
browsers: [],
|
|
customLaunchers: {
|
|
Chrome_default: {base: 'ChromeHeadless', chromeDataDir: USER_DATA},
|
|
Chrome_no_threads: {
|
|
base: 'ChromeHeadless',
|
|
chromeDataDir: USER_DATA,
|
|
// TODO: no-thread flags
|
|
},
|
|
Edge_default: {base: 'Edge', edgeDataDir: USER_DATA}
|
|
}
|
|
});
|
|
};
|