Jamie King c23db9327a Smart Decay for Adam - Caffe2 (#61548) ... Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/61548 We want to decay learning parameters properly. Previously this was not done when a parameter is absent from a minibatch. We fix this by keeping track of missed minibatches and making decay catch up accordingly. The exponential moving averages (EMA) for the first and second moments used in Adam are updated only for parameters seen in a minibatch. Actually, for these parameters, 0 should be added to the EMAs and the EMAs should then be decayed by multiplying by beta1 and beta2 respectively. To avoid the computational overhead of touching every parameter for every minibatch, we: * keep track of the last time a parameter is seen * instead of decaying the EMAs by multiplying by beta1 and beta2, we multiply by beta1^k and beta2^k, where k is the number of minibatches since the parameter was last seen * we calculate the amount of momentum that would have been discharged over the missed minibatches and update the weight accordingly. Differential Revision: D29654246 fbshipit-source-id: 7a6cd7966eb1f31116d99dfce79a78b2d3ee9e3e		2021-07-14 10:22:38 -07:00
..
__init__.py
activation_ops_test.py	replace all remaining occurrences of deadline=1000, to prevent test flakiness	2021-05-25 15:55:30 -07:00
adadelta_test.py	replace all remaining occurrences of deadline=1000, to prevent test flakiness	2021-05-25 15:55:30 -07:00
adagrad_test.py	Weighted decay with frequency (count-based) (#60382)	2021-06-21 18:46:35 -07:00
adagrad_test_helper.py	Weighted decay with frequency (count-based) (#60382)	2021-06-21 18:46:35 -07:00
adam_test.py	Smart Decay for Adam - Caffe2 (#61548)	2021-07-14 10:22:38 -07:00
affine_channel_op_test.py
alias_with_name_test.py
apmeter_test.py
arg_ops_test.py
assert_test.py	replace all remaining occurrences of deadline=1000, to prevent test flakiness	2021-05-25 15:55:30 -07:00
async_net_barrier_test.py	[DPER] Introduce barrier operation to force synchronization of threads in async execution (#49322)	2020-12-15 16:13:42 -08:00
atomic_ops_test.py
basic_rnn_test.py
batch_box_cox_test.py
batch_bucketize_op_test.py
batch_moments_op_test.py
batch_sparse_to_dense_op_test.py	[caffe2] Add support for int32 lengths in BatchSparseToDense (#58062)	2021-05-26 10:33:32 -07:00
bbox_transform_test.py	replace all remaining occurrences of deadline=1000, to prevent test flakiness	2021-05-25 15:55:30 -07:00
bisect_percentile_op_test.py
blobs_queue_db_test.py	Drop unused imports from caffe2/python (#49980)	2021-01-05 13:17:46 -08:00
boolean_mask_test.py	replace all remaining occurrences of deadline=1000, to prevent test flakiness	2021-05-25 15:55:30 -07:00
boolean_unmask_test.py
box_with_nms_limit_op_test.py	replace all remaining occurrences of deadline=1000, to prevent test flakiness	2021-05-25 15:55:30 -07:00
bucketize_op_test.py	Drop unused imports from caffe2/python (#49980)	2021-01-05 13:17:46 -08:00
cast_op_test.py
ceil_op_test.py
channel_backprop_stats_op_test.py
channel_shuffle_test.py
channel_stats_op_test.py
checkpoint_test.py
clip_op_test.py	replace all remaining occurrences of deadline=1000, to prevent test flakiness	2021-05-25 15:55:30 -07:00
clip_tensor_op_test.py	replace all remaining occurrences of deadline=1000, to prevent test flakiness	2021-05-25 15:55:30 -07:00
collect_and_distribute_fpn_rpn_proposals_op_test.py
concat_split_op_test.py	Drop unused imports from caffe2/python (#49980)	2021-01-05 13:17:46 -08:00
conditional_test.py
conftest.py
conv_test.py	relax test deadlines	2021-05-25 15:02:52 -07:00
conv_transpose_test.py
copy_ops_test.py
copy_rows_to_tensor_op_test.py
cosine_embedding_criterion_op_test.py	Drop unused imports from caffe2/python (#49980)	2021-01-05 13:17:46 -08:00
counter_ops_test.py
crf_test.py	replace all remaining occurrences of deadline=1000, to prevent test flakiness	2021-05-25 15:55:30 -07:00
cross_entropy_ops_test.py	Drop unused imports from caffe2/python (#49980)	2021-01-05 13:17:46 -08:00
ctc_beam_search_decoder_op_test.py	Drop unused imports from caffe2/python (#49980)	2021-01-05 13:17:46 -08:00
ctc_greedy_decoder_op_test.py
cudnn_recurrent_test.py	Drop unused imports from caffe2/python (#49980)	2021-01-05 13:17:46 -08:00
data_couple_op_test.py
dataset_ops_test.py	[mlf][efficiency] add tensor inference function to last-n collector op (#46693)	2020-10-22 01:15:00 -07:00
decay_adagrad_test.py	optimizer exploration - v1 and v2 + fix position_weighted optimizer + decoupled weight decay (#54042)	2021-03-27 23:03:29 -07:00
deform_conv_test.py	Drop unused imports from caffe2/python (#49980)	2021-01-05 13:17:46 -08:00
dense_vector_to_id_list_op_test.py
depthwise_3x3_conv_test.py	Drop unused imports from caffe2/python (#49980)	2021-01-05 13:17:46 -08:00
detectron_keypoints.py
distance_op_test.py	Drop unused imports from caffe2/python (#49980)	2021-01-05 13:17:46 -08:00
dropout_op_test.py	replace all remaining occurrences of deadline=1000, to prevent test flakiness	2021-05-25 15:55:30 -07:00
duplicate_operands_test.py
elementwise_linear_op_test.py	Drop unused imports from caffe2/python (#49980)	2021-01-05 13:17:46 -08:00
elementwise_logical_ops_test.py
elementwise_op_broadcast_test.py	relax test deadlines	2021-05-25 15:02:52 -07:00
elementwise_ops_test.py	[caffe2] add tests for inplace elementwise ops (#60106)	2021-06-21 12:04:18 -07:00
emptysample_ops_test.py
enforce_finite_op_test.py	Drop unused imports from caffe2/python (#49980)	2021-01-05 13:17:46 -08:00
ensure_clipped_test.py
ensure_cpu_output_op_test.py
erf_op_test.py	replace all remaining occurrences of deadline=1000, to prevent test flakiness	2021-05-25 15:55:30 -07:00
expand_op_test.py	replace all remaining occurrences of deadline=1000, to prevent test flakiness	2021-05-25 15:55:30 -07:00
fc_operator_test.py	relax test deadlines	2021-05-25 15:02:52 -07:00
feature_maps_ops_test.py	Drop unused imports from caffe2/python (#49980)	2021-01-05 13:17:46 -08:00
filler_ops_test.py	replace all remaining occurrences of deadline=1000, to prevent test flakiness	2021-05-25 15:55:30 -07:00
find_op_test.py
flatten_op_test.py
flexible_top_k_test.py	replace all remaining occurrences of deadline=1000, to prevent test flakiness	2021-05-25 15:55:30 -07:00
floor_op_test.py
fused_nbit_rowwise_conversion_ops_test.py	replace all remaining occurrences of deadline=1000, to prevent test flakiness	2021-05-25 15:55:30 -07:00
fused_nbit_rowwise_test.cc
fused_nbit_rowwise_test_helper.py
gather_ops_test.py	replace all remaining occurrences of deadline=1000, to prevent test flakiness	2021-05-25 15:55:30 -07:00
gather_ranges_op_test.py	replace all remaining occurrences of deadline=1000, to prevent test flakiness	2021-05-25 15:55:30 -07:00
given_tensor_byte_string_to_uint8_fill_op_test.py
given_tensor_fill_op_test.py
glu_op_test.py	Drop unused imports from caffe2/python (#49980)	2021-01-05 13:17:46 -08:00
group_conv_test.py	Drop unused imports from caffe2/python (#49980)	2021-01-05 13:17:46 -08:00
group_norm_op_test.py
gru_test.py	Drop unused imports from caffe2/python (#49980)	2021-01-05 13:17:46 -08:00
heatmap_max_keypoint_op_test.py	Remove legacy constructor calls from pytorch codebase. (#54142)	2021-04-11 15:45:17 -07:00
histogram_test.py
hsm_test.py
hyperbolic_ops_test.py	Drop unused imports from caffe2/python (#49980)	2021-01-05 13:17:46 -08:00
im2col_col2im_test.py	Drop unused imports from caffe2/python (#49980)	2021-01-05 13:17:46 -08:00
image_input_op_test.py	Remove unused six code for Python 2/3 compatibility (#48077)	2020-12-22 18:07:08 -08:00
index_hash_ops_test.py
index_ops_test.py
instance_norm_test.py	replace all remaining occurrences of deadline=1000, to prevent test flakiness	2021-05-25 15:55:30 -07:00
integral_image_ops_test.py
jsd_ops_test.py	Drop unused imports from caffe2/python (#49980)	2021-01-05 13:17:46 -08:00
key_split_ops_test.py
lars_test.py
layer_norm_op_test.py	replace all remaining occurrences of deadline=1000, to prevent test flakiness	2021-05-25 15:55:30 -07:00
leaky_relu_test.py
learning_rate_adaption_op_test.py
learning_rate_op_test.py
length_split_op_test.py	replace all remaining occurrences of deadline=1000, to prevent test flakiness	2021-05-25 15:55:30 -07:00
lengths_pad_op_test.py	Drop unused imports from caffe2/python (#49980)	2021-01-05 13:17:46 -08:00
lengths_reducer_fused_nbit_rowwise_ops_test.py	Drop unused imports from caffe2/python (#49980)	2021-01-05 13:17:46 -08:00
lengths_tile_op_test.py	Drop unused imports from caffe2/python (#49980)	2021-01-05 13:17:46 -08:00
lengths_top_k_ops_test.py	Replace map(lambda constructs (#46462)	2020-10-22 09:50:22 -07:00
listwise_l2r_operator_test.py
load_save_test.py	Enable C2 load rate limiter [2/n] (#61551)	2021-07-14 08:27:05 -07:00
locally_connected_op_test.py	relax test deadlines	2021-05-25 15:02:52 -07:00
loss_ops_test.py	Drop unused imports from caffe2/python (#49980)	2021-01-05 13:17:46 -08:00
lpnorm_op_test.py	replace all remaining occurrences of deadline=1000, to prevent test flakiness	2021-05-25 15:55:30 -07:00
map_ops_test.py
margin_ranking_criterion_op_test.py	replace all remaining occurrences of deadline=1000, to prevent test flakiness	2021-05-25 15:55:30 -07:00
math_ops_test.py
matmul_op_test.py	replace all remaining occurrences of deadline=1000, to prevent test flakiness	2021-05-25 15:55:30 -07:00
mean_op_test.py	Lint trailing newlines (#54737)	2021-03-30 13:09:52 -07:00
merge_id_lists_op_test.py
mkl_conv_op_test.py
mkl_packed_fc_op_test.py
mod_op_test.py	[uhm][0/n] add cuda Mod Op (#46732)	2020-10-26 11:07:51 -07:00
moments_op_test.py	Drop unused imports from caffe2/python (#49980)	2021-01-05 13:17:46 -08:00
momentum_sgd_test.py
mpi_test.py	[pytorch] Update caffe2/python to eliminate Pyre errors (#52083)	2021-02-11 11:04:59 -08:00
mul_gradient_benchmark.py
negate_gradient_op_test.py
ngram_ops_test.py
normalize_op_test.py
numpy_tile_op_test.py	Drop unused imports from caffe2/python (#49980)	2021-01-05 13:17:46 -08:00
one_hot_ops_test.py	replace all remaining occurrences of deadline=1000, to prevent test flakiness	2021-05-25 15:55:30 -07:00
onnx_while_test.py	Drop unused imports from caffe2/python (#49980)	2021-01-05 13:17:46 -08:00
order_switch_test.py
pack_ops_test.py
pack_rnn_sequence_op_test.py	Drop unused imports from caffe2/python (#49980)	2021-01-05 13:17:46 -08:00
pad_test.py	Drop unused imports from caffe2/python (#49980)	2021-01-05 13:17:46 -08:00
partition_ops_test.py
percentile_op_test.py	Drop unused imports from caffe2/python (#49980)	2021-01-05 13:17:46 -08:00
piecewise_linear_transform_test.py
pooling_test.py	replace all remaining occurrences of deadline=1000, to prevent test flakiness	2021-05-25 15:55:30 -07:00
prepend_dim_test.py
python_op_test.py	replace all remaining occurrences of deadline=1000, to prevent test flakiness	2021-05-25 15:55:30 -07:00
quantile_test.py
rand_quantization_op_speed_test.py
rank_loss_operator_test.py
rebatching_queue_test.py
record_queue_test.py
recurrent_net_executor_test.py
recurrent_network_test.py	Drop unused imports from caffe2/python (#49980)	2021-01-05 13:17:46 -08:00
reduce_ops_test.py	relax test deadlines	2021-05-25 15:02:52 -07:00
reduction_ops_test.py	[C2] Add shape inference logic for ColwiseMax operator. (#51914)	2021-02-09 14:12:07 -08:00
reshape_ops_test.py	Remove unused six code for Python 2/3 compatibility (#48077)	2020-12-22 18:07:08 -08:00
resize_op_test.py
rmac_regions_op_test.py
rms_norm_op_test.py
rnn_cell_test.py
roi_align_rotated_op_test.py	Drop unused imports from caffe2/python (#49980)	2021-01-05 13:17:46 -08:00
rowwise_counter_test.py
scale_op_test.py	[pytorch] Update caffe2/python to eliminate Pyre errors (#52083)	2021-02-11 11:04:59 -08:00
segment_ops_test.py
self_binning_histogram_test.py	[MLF] Allow for computing prune quantile thresholds on absolute value of indicators in distributed-inference-compatible embedding LUT pruning (#46789)	2020-11-02 11:31:31 -08:00
selu_op_test.py	replace all remaining occurrences of deadline=1000, to prevent test flakiness	2021-05-25 15:55:30 -07:00
sequence_ops_test.py	fix AddPadding tensor shape inference (#59572)	2021-06-08 11:02:33 -07:00
shape_inference_test.py
sinusoid_position_encoding_op_test.py	replace all remaining occurrences of deadline=1000, to prevent test flakiness	2021-05-25 15:55:30 -07:00
softmax_ops_test.py	relax test deadlines	2021-05-25 15:02:52 -07:00
softplus_op_test.py	replace all remaining occurrences of deadline=1000, to prevent test flakiness	2021-05-25 15:55:30 -07:00
sparse_dropout_with_replacement_op_test.py
sparse_gradient_checker_test.py
sparse_itemwise_dropout_with_replacement_op_test.py	[itemwise-dropout][1/x][low-level module] Implement Itemwise Sparse Feature Dropout in Dper3 (#59322)	2021-06-04 19:59:17 -07:00
sparse_lengths_sum_benchmark.py
sparse_lp_regularizer_test.py
sparse_normalize_test.py	Clean up future imports for Python 2 (#53349)	2021-03-14 15:56:13 -07:00
sparse_ops_test.py	[1/n]support double for Caffe2 ScatterWeightedSum (#60402)	2021-06-29 14:17:04 -07:00
sparse_to_dense_mask_op_test.py	replace all remaining occurrences of deadline=1000, to prevent test flakiness	2021-05-25 15:55:30 -07:00
spatial_bn_op_test.py	[c2] Fix flaky test_spatial_bn_multi_batch_grad	2021-05-06 12:50:53 -07:00
specialized_segment_ops_test.py
square_root_divide_op_test.py	Drop unused imports from caffe2/python (#49980)	2021-01-05 13:17:46 -08:00
stats_ops_test.py
stats_put_ops_test.py
storm_test.py
string_ops_test.py	replace all remaining occurrences of deadline=1000, to prevent test flakiness	2021-05-25 15:55:30 -07:00
text_file_reader_test.py
thresholded_relu_op_test.py
tile_op_test.py
top_k_test.py	replace all remaining occurrences of deadline=1000, to prevent test flakiness	2021-05-25 15:55:30 -07:00
torch_integration_test.py	replace all remaining occurrences of deadline=1000, to prevent test flakiness	2021-05-25 15:55:30 -07:00
transpose_op_test.py
trigonometric_op_test.py
unique_ops_test.py
unique_uniform_fill_op_test.py
unsafe_coalesce_test.py	[C2] Revive unsafe CoalesceOp (#49402)	2020-12-17 04:31:29 -08:00
upsample_op_test.py
utility_ops_test.py	replace all remaining occurrences of deadline=1000, to prevent test flakiness	2021-05-25 15:55:30 -07:00
video_input_op_test.py
weight_scale_test.py
weighted_multi_sample_test.py
weighted_sample_test.py
weighted_sum_test.py	replace all remaining occurrences of deadline=1000, to prevent test flakiness	2021-05-25 15:55:30 -07:00
wngrad_test.py	replace all remaining occurrences of deadline=1000, to prevent test flakiness	2021-05-25 15:55:30 -07:00