onnxruntime/orttraining/tools/ci_test/results/bert_base.convergence.baseline.mi100.csv

step,total_loss,mlm_loss,nsp_loss
0,11.2031,10.501,0.702177
5,9.54193,7.52403,2.01789
10,8.26154,7.56404,0.6975
15,8.28391,7.55602,0.727896
20,8.17255,7.4595,0.713049
25,8.2279,7.53246,0.69544
30,8.07992,7.38461,0.695307
35,7.96175,7.25041,0.711333
40,7.94631,7.25669,0.689622
45,7.92991,7.26444,0.665472
Add nightly pipeline for MI100 to run convergence and batch size test similar to V100. (#6611) * Partial updating of ROCM reduction code. * Update reduction_all.cu * Add reduce template parameters. * miopen common * Reuse CUDA's reduction_functions.cc * Reduction ops. * Update remaining reduction ops to use MIOpen. double datatype is not supported, so disable those typed kernels. * Disable a couple more unsupported tests. * Code formatting. * Delete ROCM-specific reduction code that is identical to CUDA reduction code. * Fix scratch buffer early free. * Fix merge conflict. * first attempt nightly amd ci pipeline * try fix bad yaml file * try again with corrected model directory * add convergence test as well * update reference loss for amd mi100 * include mi100 test results csv * update the mi100 convergence test reference values * update batch sizes for mi100 32g * fix gpu sku for run_convergence_test.py * undo unrelated changes to master * pr comments * pr comment Co-authored-by: Jesse Benson <jesseb@microsoft.com> 2021-02-12 21:22:06 +00:00			`step,total_loss,mlm_loss,nsp_loss`
[ROCm] update bert-L convergence reference file to fix CI (#15200) The change of layernorm lead to the change of bert-L convergence result. 2023-03-24 13:43:44 +00:00			`0,11.2031,10.501,0.702177`
			`5,9.54193,7.52403,2.01789`
			`10,8.26154,7.56404,0.6975`
			`15,8.28391,7.55602,0.727896`
			`20,8.17255,7.4595,0.713049`
			`25,8.2279,7.53246,0.69544`
			`30,8.07992,7.38461,0.695307`
			`35,7.96175,7.25041,0.711333`
			`40,7.94631,7.25669,0.689622`
			`45,7.92991,7.26444,0.665472`