saymrwulf/pytorch
1
0
Fork 0
mirror of https://github.com/saymrwulf/pytorch.git synced 2026-05-15 21:00:47 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
pytorch/caffe2/python
History
Huayu Li 793e2914e4 Support full id interations (#28769) 
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/28769

Support full id interaction.

Test Plan:
* unit-tests
  * buck test caffe2/caffe2/python/operator_test:pack_ops_test --
  * buck test caffe2/caffe2/fb/dper/layer_models/tests:sparse_nn_attention_test -- test_sparse_nn_full_id

* canary
  * apply SUM + full id with max_length as 20 on SPARSE_AD_MEDIA_XRAY_V11_TOPIC_ID: f147253340 (v1: f146340704)

# of embeddings for this features is 20:
{F219139816}

The corresponding ops: two lookups, which is as expected.
```
op {
  input: "nested/dot/SPARSE_AD_MEDIA_XRAY_V11_TOPIC_ID_AUTO_FIRST_X_AUTO_UNIGRAM/Pool_Option_0/Repeat_0/sparse_lookup/w"
  input: "feature_preproc/output_features:SPARSE_AD_MEDIA_XRAY_V11_TOPIC_ID_AUTO_FIRST_X_AUTO_UNIGRAM:values"
  input: "feature_preproc/output_features:SPARSE_AD_MEDIA_XRAY_V11_TOPIC_ID_AUTO_FIRST_X_AUTO_UNIGRAM:lengths"
  output: "nested/dot/SPARSE_AD_MEDIA_XRAY_V11_TOPIC_ID_AUTO_FIRST_X_AUTO_UNIGRAM/Pool_Option_0/Repeat_0/sparse_lookup/output"
  name: ""
  type: "SparseLengthsSum"
}
op {
  input: "nested/dot/SPARSE_AD_MEDIA_XRAY_V11_TOPIC_ID_AUTO_FIRST_X_AUTO_UNIGRAM/Pool_Option_1/Repeat_0/sparse_lookup/w"
  input: "feature_preproc/output_features:SPARSE_AD_MEDIA_XRAY_V11_TOPIC_ID_AUTO_FIRST_X_AUTO_UNIGRAM:values"
  output: "nested/dot/SPARSE_AD_MEDIA_XRAY_V11_TOPIC_ID_AUTO_FIRST_X_AUTO_UNIGRAM/Pool_Option_1/Repeat_0/sparse_lookup/output"
  name: ""
  type: "Gather"
}
op {
  input: "feature_preproc/output_features:SPARSE_AD_MEDIA_XRAY_V11_TOPIC_ID_AUTO_FIRST_X_AUTO_UNIGRAM:lengths"
  input: "nested/dot/SPARSE_AD_MEDIA_XRAY_V11_TOPIC_ID_AUTO_FIRST_X_AUTO_UNIGRAM/Pool_Option_1/Repeat_0/sparse_lookup/output"
  output: "nested/dot/SPARSE_AD_MEDIA_XRAY_V11_TOPIC_ID_AUTO_FIRST_X_AUTO_UNIGRAM/Pool_Option_1/Repeat_0/full_id/PackSegments/embedding_packed"
  name: ""
  type: "PackSegments"
  arg {
    name: "max_length"
    i: 20
  }
  arg {
    name: "pad_minf"
    i: 0
  }
}
op {
  input: "nested/dot/SPARSE_AD_MEDIA_XRAY_V11_TOPIC_ID_AUTO_FIRST_X_AUTO_UNIGRAM/Pool_Option_1/Repeat_0/full_id/PackSegments/embedding_packed"
  output: "nested/dot/SPARSE_AD_MEDIA_XRAY_V11_TOPIC_ID_AUTO_FIRST_X_AUTO_UNIGRAM/Pool_Option_1/Repeat_0/full_id/Reshape/reshaped_record"
  output: "nested/dot/SPARSE_AD_MEDIA_XRAY_V11_TOPIC_ID_AUTO_FIRST_X_AUTO_UNIGRAM/Pool_Option_1/Repeat_0/full_id/Reshape/old_shape"
  name: ""
  type: "Reshape"
  arg {
    name: "shape"
    ints: -1
    ints: 1280
  }
}
op {
  input: "nested/dot/SPARSE_AD_MEDIA_XRAY_V11_TOPIC_ID_AUTO_FIRST_X_AUTO_UNIGRAM/Pool_Option_1/Repeat_0/full_id/Reshape/reshaped_record"
  output: "nested/dot/SPARSE_AD_MEDIA_XRAY_V11_TOPIC_ID_AUTO_FIRST_X_AUTO_UNIGRAM/Pool_Option_1/Repeat_0/full_id/split/output_0"
  output: "nested/dot/SPARSE_AD_MEDIA_XRAY_V11_TOPIC_ID_AUTO_FIRST_X_AUTO_UNIGRAM/Pool_Option_1/Repeat_0/full_id/split/output_1"
  output: "nested/dot/SPARSE_AD_MEDIA_XRAY_V11_TOPIC_ID_AUTO_FIRST_X_AUTO_UNIGRAM/Pool_Option_1/Repeat_0/full_id/split/output_2"
  output: "nested/dot/SPARSE_AD_MEDIA_XRAY_V11_TOPIC_ID_AUTO_FIRST_X_AUTO_UNIGRAM/Pool_Option_1/Repeat_0/full_id/split/output_3"
  output: "nested/dot/SPARSE_AD_MEDIA_XRAY_V11_TOPIC_ID_AUTO_FIRST_X_AUTO_UNIGRAM/Pool_Option_1/Repeat_0/full_id/split/output_4"
  output: "nested/dot/SPARSE_AD_MEDIA_XRAY_V11_TOPIC_ID_AUTO_FIRST_X_AUTO_UNIGRAM/Pool_Option_1/Repeat_0/full_id/split/output_5"
  output: "nested/dot/SPARSE_AD_MEDIA_XRAY_V11_TOPIC_ID_AUTO_FIRST_X_AUTO_UNIGRAM/Pool_Option_1/Repeat_0/full_id/split/output_6"
  output: "nested/dot/SPARSE_AD_MEDIA_XRAY_V11_TOPIC_ID_AUTO_FIRST_X_AUTO_UNIGRAM/Pool_Option_1/Repeat_0/full_id/split/output_7"
  output: "nested/dot/SPARSE_AD_MEDIA_XRAY_V11_TOPIC_ID_AUTO_FIRST_X_AUTO_UNIGRAM/Pool_Option_1/Repeat_0/full_id/split/output_8"
  output: "nested/dot/SPARSE_AD_MEDIA_XRAY_V11_TOPIC_ID_AUTO_FIRST_X_AUTO_UNIGRAM/Pool_Option_1/Repeat_0/full_id/split/output_9"
  output: "nested/dot/SPARSE_AD_MEDIA_XRAY_V11_TOPIC_ID_AUTO_FIRST_X_AUTO_UNIGRAM/Pool_Option_1/Repeat_0/full_id/split/output_10"
  output: "nested/dot/SPARSE_AD_MEDIA_XRAY_V11_TOPIC_ID_AUTO_FIRST_X_AUTO_UNIGRAM/Pool_Option_1/Repeat_0/full_id/split/output_11"
  output: "nested/dot/SPARSE_AD_MEDIA_XRAY_V11_TOPIC_ID_AUTO_FIRST_X_AUTO_UNIGRAM/Pool_Option_1/Repeat_0/full_id/split/output_12"
  output: "nested/dot/SPARSE_AD_MEDIA_XRAY_V11_TOPIC_ID_AUTO_FIRST_X_AUTO_UNIGRAM/Pool_Option_1/Repeat_0/full_id/split/output_13"
  output: "nested/dot/SPARSE_AD_MEDIA_XRAY_V11_TOPIC_ID_AUTO_FIRST_X_AUTO_UNIGRAM/Pool_Option_1/Repeat_0/full_id/split/output_14"
  output: "nested/dot/SPARSE_AD_MEDIA_XRAY_V11_TOPIC_ID_AUTO_FIRST_X_AUTO_UNIGRAM/Pool_Option_1/Repeat_0/full_id/split/output_15"
  output: "nested/dot/SPARSE_AD_MEDIA_XRAY_V11_TOPIC_ID_AUTO_FIRST_X_AUTO_UNIGRAM/Pool_Option_1/Repeat_0/full_id/split/output_16"
  output: "nested/dot/SPARSE_AD_MEDIA_XRAY_V11_TOPIC_ID_AUTO_FIRST_X_AUTO_UNIGRAM/Pool_Option_1/Repeat_0/full_id/split/output_17"
  output: "nested/dot/SPARSE_AD_MEDIA_XRAY_V11_TOPIC_ID_AUTO_FIRST_X_AUTO_UNIGRAM/Pool_Option_1/Repeat_0/full_id/split/output_18"
  output: "nested/dot/SPARSE_AD_MEDIA_XRAY_V11_TOPIC_ID_AUTO_FIRST_X_AUTO_UNIGRAM/Pool_Option_1/Repeat_0/full_id/split/output_19"
  name: ""
  type: "Split"
  arg {
    name: "axis"
    i: 1
  }
}
```

Reviewed By: chonglinsun

Differential Revision: D18083520

fbshipit-source-id: f592fb7734dd4e3e712ba42dc0afcd0b32a4afa0
2019-10-29 14:56:18 -07:00
..
docs Fix several DeprecationWarning: invalid escape sequence (#15733) 2019-01-05 08:53:35 -08:00
examples Adding models to jenkins benchmark script (#21010) 2019-05-30 15:17:40 -07:00
helpers Add elementwise_affine for LayerNormGradientOp (#19982) 2019-05-03 15:33:46 -07:00
ideep caffe2 python ideep conv_op test_int8_convolution skip for python 3 2019-10-08 21:31:11 -07:00
layers Change dper3 loss module to match dper2 (#28265) 2019-10-18 10:08:38 -07:00
mint re-enable copy of python files, but be careful that the copy is only … (#14982) 2018-12-11 16:54:08 -08:00
mkl implement operators for DNNLOWP (#18656) 2019-04-10 12:04:39 -07:00
modeling Add more logging in net modifier 2019-10-21 17:53:00 -07:00
models Adding ShufflenetV2 to caffe2's benchmark suite. (#20180) 2019-05-23 20:40:17 -07:00
onnx Automatic update of fbcode/onnx to 23bb6ea1a71f08e200114a153f48bd7adb66d486 (#26441) 2019-09-24 05:38:52 -07:00
operator_test Support full id interations (#28769) 2019-10-29 14:56:18 -07:00
predictor Supporting Manifold DB in Predictor Exporter (#22334) 2019-07-01 16:17:02 -07:00
rnn Unify cuda and hip device types in Caffe2 python front end (#14221) 2018-11-29 14:00:16 -08:00
serialized_test Output sequence probability with CTC beam search, optional multiple output sequences (#21927) 2019-07-02 17:29:13 -07:00
test Enforce import order to make protobuf cpp implementation in python work (#18560) 2019-04-03 13:17:08 -07:00
trt Revert D17495965: TensorRT 6.0 support and PyTorch->ONNX->TRT6 unit test 2019-10-25 13:58:16 -07:00
__init__.py Revert #17191 and #17215 that no longer apply on Windows (#17567) 2019-03-01 10:37:27 -08:00
_import_c_extension.py Enforce import order to make protobuf cpp implementation in python work (#18560) 2019-04-03 13:17:08 -07:00
allcompare_test.py
attention.py
benchmark_generator.py
binarysize.py
brew.py Testing for folded conv_bn_relu (#19298) 2019-04-16 19:04:06 -07:00
brew_test.py
build.py
cached_reader.py Pass loop_over optional parameter for cached reader properly. (#21929) 2019-06-19 18:15:32 -07:00
caffe_translator.py Fix several ResourceWarning: unclosed file (#15746) 2019-01-09 15:36:53 -08:00
caffe_translator_test.py Fix several ResourceWarning: unclosed file (#15746) 2019-01-09 15:36:53 -08:00
checkpoint.py Remove setting logger level in caffe2.python.checkpoint (#19803) 2019-05-10 07:00:58 -07:00
checkpoint_test.py
CMakeLists.txt Fix CMakeLists.txt for Int8 python bindings (#15047) 2018-12-11 10:48:47 -08:00
cnn.py Unify cuda and hip device types in Caffe2 python front end (#14221) 2018-11-29 14:00:16 -08:00
compatibility.py
context.py
context_test.py
control.py
control_ops_grad.py DeviceScope support for CUDA and testing (#15357) 2019-01-30 18:42:12 -08:00
control_ops_grad_test.py Fix the weird bug in control_flow_op_test.py (#26931) 2019-09-26 20:44:03 -07:00
control_ops_util.py
control_test.py
convert.py New serialization format (#12384) 2018-10-16 16:36:58 -07:00
convert_test.py New serialization format (#12384) 2018-10-16 16:36:58 -07:00
convnet_benchmarks.py
convnet_benchmarks_test.py Skip convnets benchmark in rocm CI (#17331) 2019-02-20 21:12:24 -08:00
core.py BlobReference __getattr__ can only throw AttributeError (#26654) 2019-09-23 13:01:00 -07:00
core_gradients_test.py Back out "Back out "[Caffe2] Fix device_option propagation"" (#25908) 2019-09-17 04:01:36 -07:00
core_test.py Extend Net.RunAllOnGPU() to support RecurrentNetwork op (#15713) 2019-02-08 15:48:42 -08:00
crf.py
crf_predict.py Move crf in caffe2 from fb to oss (#12200) 2018-10-01 18:31:41 -07:00
crf_viterbi_test.py Move crf in caffe2 from fb to oss (#12200) 2018-10-01 18:31:41 -07:00
data_parallel_model.py skip import nccl and gloo_gpu in cpu machine (#22522) 2019-07-10 11:56:56 -07:00
data_parallel_model_test.py Unify cuda and hip device types in Caffe2 python front end (#14221) 2018-11-29 14:00:16 -08:00
data_workers.py
data_workers_test.py
dataio.py Rearrange stopping condition in CompositeReader (#20062) 2019-05-06 15:06:32 -07:00
dataio_test.py Fix for flaky caffe2 dataio test (test_time_limit_reader_with_short_limit) (#27592) 2019-10-10 13:53:58 -07:00
dataset.py
db_file_reader.py
db_test.py
device_checker.py
dlpack.h Upgrade DLPack 2018-11-12 15:59:46 -08:00
dyndep.py guard dyndep with a lock (#26153) 2019-09-13 11:38:14 -07:00
embedding_generation_benchmark.py
experiment_util.py
extension_loader.py always restore dlopen flag in dyndep (#22958) 2019-07-17 10:26:25 -07:00
filler_test.py caffe2 - Expose tensor filler util to Python (#18886) 2019-04-08 11:54:10 -07:00
functional.py
functional_test.py
fused_8bit_rowwise_conversion_ops_test.py
gradient_check_test.py Unify gpu_support variable in python tests (#16748) 2019-02-07 00:29:51 -08:00
gradient_checker.py Adding gradient to Boolean Mask operator (#21423) 2019-06-06 20:48:47 -07:00
gru_cell.py
hip_test_util.py Make CUDNN an alias of MIOPEN for HIP ops (#12278) 2018-10-24 17:07:31 -07:00
hsm_util.py
hypothesis_test.py Extend HasElements to support multiple inputs (#28717) 2019-10-27 23:00:07 -07:00
hypothesis_test_util.py Hypothesis tests: add ability to enforce shape inference (#23935) 2019-08-13 05:32:41 -07:00
ideep_test_util.py
layer_model_helper.py Integrate FC fp16 exporter into Dper2 (#26582) 2019-09-29 10:19:28 -07:00
layer_model_instantiator.py
layer_parameter_sharing_test.py Add validator for optimizers when parameters are shared 2019-04-17 21:10:38 -07:00
layer_test_util.py
layers_test.py Add support for attention weight in SparseLookup (#26748) 2019-10-08 20:22:25 -07:00
lengths_reducer_fused_8bit_rowwise_ops_test.py make the threshold for acurracy more precise (#17194) 2019-02-20 13:14:11 -08:00
lengths_reducer_rowwise_8bit_ops_test.py
lstm_benchmark.py Unify cuda and hip device types in Caffe2 python front end (#14221) 2018-11-29 14:00:16 -08:00
memonger.py Fix typo "properlyh" 2019-08-09 11:43:04 -07:00
memonger_test.py Unify gpu_support variable in python tests (#16748) 2019-02-07 00:29:51 -08:00
mkl_test_util.py
model_device_test.py Unify gpu_support variable in python tests (#16748) 2019-02-07 00:29:51 -08:00
model_helper.py Remove the identical if branch (#18019) 2019-03-15 13:14:26 -07:00
model_helper_test.py
modifier_context.py
mpi_python.cc
muji.py Unify cuda and hip device types in Caffe2 python front end (#14221) 2018-11-29 14:00:16 -08:00
muji_test.py Unify cuda and hip device types in Caffe2 python front end (#14221) 2018-11-29 14:00:16 -08:00
net_builder.py
net_builder_test.py
net_drawer.py Allow customization of blob node in net_drawer (#16915) 2019-02-12 15:02:50 -08:00
net_printer.py Fix spelling errors (#21665) 2019-06-13 15:21:55 -07:00
net_printer_test.py
nomnigraph.py createUniqueDataNode 2018-10-31 11:16:38 -07:00
nomnigraph_test.py nomnigraph - support subgraph visualization (#13795) 2018-11-16 08:19:20 -08:00
nomnigraph_transformations.py Add transpose network pass (#13437) 2018-11-01 14:27:07 -07:00
nomnigraph_transformations_test.py Add transpose network pass (#13437) 2018-11-01 14:27:07 -07:00
normalizer.py Enable alternative LayerNorm impl in FisherGan (#12178) 2018-10-11 17:36:11 -07:00
normalizer_context.py
normalizer_test.py
numa_benchmark.py Revert D13205604: Move numa.{h, cc} to c10/util 2018-12-07 10:01:25 -08:00
numa_test.py Move numa.{h, cc} to c10/util (#15024) 2018-12-12 12:21:10 -08:00
observer_test.py
operator_fp_exceptions_test.py Caffe2 - Add flag to fails if float point exceptions is detected in operator runs (#18040) 2019-03-16 12:28:05 -07:00
optimizer.py Add validator for optimizers when parameters are shared 2019-04-17 21:10:38 -07:00
optimizer_context.py
optimizer_test.py Unify gpu_support variable in python tests (#16748) 2019-02-07 00:29:51 -08:00
optimizer_test_util.py Unify cuda and hip device types in Caffe2 python front end (#14221) 2018-11-29 14:00:16 -08:00
parallel_workers.py get rid of deprecated thread.isAlive() to use py2.6 modern form is_alive() 2019-10-22 15:37:31 -07:00
parallel_workers_test.py Add event and event_counter columns to caffe2_usage_tracer table (#21739) 2019-06-13 12:06:02 -07:00
parallelize_bmuf_distributed_test.py Unify gpu_support variable in python tests (#16748) 2019-02-07 00:29:51 -08:00
pipeline.py
pipeline_test.py
predictor_constants.py
pybind_state.cc weight_names bug fix (#23848) 2019-08-06 10:58:43 -07:00
pybind_state.h Support unpickle py2 NetDef object in py3 (#26147) 2019-09-18 02:02:34 -07:00
pybind_state_dlpack.cc Upgrade DLPack 2018-11-12 15:59:46 -08:00
pybind_state_dlpack.h Remove PythonOp non-CPU path and PytorchOp (#15417) 2019-01-02 16:36:37 -08:00
pybind_state_gpu.cc add simple memory analyzer and log warning if GPU underutilized (#21024) 2019-05-28 19:58:54 -07:00
pybind_state_hip.cc Fix get_gpu_memory_info in non-cuda builds (#21054) 2019-05-28 23:05:15 -07:00
pybind_state_ideep.cc Upgrade mkldnn-bridge for dnnlowp support (#16308) 2019-04-03 12:47:17 -07:00
pybind_state_int8.cc Renaming meta() to dtype() - 2/2 (#13334) 2018-10-30 18:24:30 -07:00
pybind_state_nomni.cc nomnigraph - support subgraph visualization (#13795) 2018-11-16 08:19:20 -08:00
pybind_state_registry.cc
pybind_state_registry.h
python_op_test.py Clean up a couple of items in the C2 test scaffolding (WIP) (#7847) 2018-11-07 09:16:13 -08:00
queue_util.py
record_queue.py
recurrent.py
regularizer.py Initial commit for L0 norm approx (#27756) 2019-10-11 11:24:34 -07:00
regularizer_context.py
regularizer_test.py Implement "trimmed lasso" regularization and support all available regularization in a single interface (#22966) 2019-07-17 16:12:31 -07:00
rnn_cell.py Unify cuda and hip device types in Caffe2 python front end (#14221) 2018-11-29 14:00:16 -08:00
schema.py Pass LRU hash output evicted_values to SparseLookup (#21389) 2019-07-02 11:27:37 -07:00
schema_test.py Pass LRU hash output evicted_values to SparseLookup (#21389) 2019-07-02 11:27:37 -07:00
scope.py Add EmptyNameScope to allow you jump out from current scope. (#14631) 2018-12-12 01:39:50 -08:00
scope_test.py Add EmptyNameScope to allow you jump out from current scope. (#14631) 2018-12-12 01:39:50 -08:00
session.py
session_test.py
sparse_to_dense_mask_test.py Increase static tolerance for negative feature ids 2019-05-20 19:09:22 -07:00
sparse_to_dense_test.py
task.py A trivial typo fix in caffe2.python (#15907) 2019-01-17 04:57:34 -08:00
task_test.py caffe2/python/task: added __repr__ methods to all task definitions (#15250) 2018-12-17 16:02:16 -08:00
test_util.py caffe2 - support flaky operator tests for caffe2 build (#18155) 2019-03-25 16:58:34 -07:00
text_file_reader.py Create Node2Vec ModuleKeeper 2019-04-01 10:36:23 -07:00
timeout_guard.py
toy_regression_test.py Enable junk fill for the default CPU allocator (#13377) 2018-11-08 00:02:37 -08:00
transformations.py support pre-convert filter format for mkldnn training mode and change 'OptimizeForIdeep' to 'OptimizeForMkldnn' (#15171) 2019-03-29 19:00:48 -07:00
transformations_test.py Remove sinkMaxPool transformation (#17694) 2019-03-12 20:10:46 -07:00
tt_core.py
tt_core_test.py
utils.py Query caffe2 operator stats for detailed execution info (#20924) 2019-06-13 23:41:04 -07:00
utils_test.py Convert Arguments to dictionary (#13436) 2018-11-01 14:27:05 -07:00
visualize.py
workspace.py Fix get_gpu_memory_info in non-cuda builds (#21054) 2019-05-28 23:05:15 -07:00
workspace_test.py Pass Variable into Caffe2 ops, by requiring that the Variable doesn't require grad (#22473) 2019-07-08 11:31:10 -07:00