mirror of
https://github.com/saymrwulf/pytorch.git
synced 2026-05-15 21:00:47 +00:00
9fae8fcdff
Summary: Generate serialized test inputs/outputs/backward graphs of tests inside `caffe2/python/operator_test` that call assertSerializedOperatorCheck(). Tests should be decorated with serialized_test.collect_tests.given_and_seeded to run hypothesis tests that are actually random and a single fixed seeded hypothesis tests. To use: 1. Refactor your test to be a SerializedTestCase 1a. Decorate it with given_and_seeded 1b. Call testWithArgs in main 2. Run your test with -g to generate the output. Check it in. 3. Subsequent runs of the test without generating the output will check against the checked in test case. Details: Run your test with `python caffe2/python/operator_test/[your_test].py -g` Outputs are in `caffe2/python/serialized_test/data`. The operator tests outputs are in a further subdirectory `operator_test`, to allow for other tests in the future (model zoo tests?) Currently, we've only refactored weighted_sum_test to use this, but in the next diff, we'll refactor as many as possible. The directory structure may also change as usually there are multiple tests in a single file, so we may create more structure to account for that. Pull Request resolved: https://github.com/pytorch/pytorch/pull/10594 Reviewed By: ezyang Differential Revision: D9370359 Pulled By: ajyu fbshipit-source-id: 2ce77389cd8bcc0255d3bccd61569833e545ede8 |
||
|---|---|---|
| .. | ||
| data/operator_test/weighted_sum_test.test_weighted_sum | ||
| __init__.py | ||
| README.md | ||
| serialized_test_util.py | ||
Serialized operator test framework
Major functionality lives in serialized_test_util.py
How to use
- Extend the test case class from
SerializedTestCase - Change the
@givendecorator to@given_and_seeded. This runs a seeded hypothesis test instance which will generate outputs if desired in addition to the unseeded hypothesis tests normally run. - Change a call to
unittest.main()in__main__totestWithArgs. - Run your test
python caffe2/python/operator_test/my_test.py -gto generate serialized outputs. They will live incaffe2/python/serialized_test/data/operator_test, one folder per test function - Thereafter, runs of the test without the flag will load serialized outputs and gradient operators for comparison against the seeded run. If for any reason the seeded run's inputs are different (this can happen with different hypothesis versions or different setups), then we'll run the serialized inputs through the serialized operator to get a runtime output for comparison.
If we'd like to extend the test framework beyond that for operator tests, we can create a new subfolder for them inside caffe2/python/serialized_test/data.