Summary:
We need a warm-up stage because otherwise first iteration
speds too much timedoing all the allocations
Reviewed By: akyrola
Differential Revision: D4986201
fbshipit-source-id: f60a75520988ff3f1540bb157cdc69634f307db4
Summary:
Layer to allow model to follow different paths for each instantiation context and join later. Together with tagging system cleanup (this is a separate issue), this should reduce the need to write a layer to differentiate between context.
Re: tagging system clean up, we should make exclusion more explicit: EXCLUDE_FROM_<CONTEXT>. This would simplify instation code. TRAIN_ONLY should become a set of all EXCLUDE_FROM_*, except EXCLUDE_FROM_TRAIN.
Reviewed By: kennyhorror
Differential Revision: D4964949
fbshipit-source-id: ba6453b0deb92d1989404efb9d86e1ed25297202
Summary: Make NCCL optional in data_parallel_model due to continuing reliablity (deadlock) issues.
Reviewed By: pietern
Differential Revision: D4988950
fbshipit-source-id: 8a2192f01b5f3c0e847137cd37aefc69e553a56f
Summary:
RFC. This is a naive implementation of Rebatchin Queue for MultiTask
effort. Full disclaimer, I'm very new to Caffe/Machine Learning and I'm doing
dodge science here (under Dmytros supervision), so please be extra tough on
this review so I
can learn best practices :)
Differential Revision: D4871970
fbshipit-source-id: 924820ef0fce45b5e2bdabeec9885cbafa23a880
Summary: I ran into this earlier and the debug messages were not helpful enuogh
Reviewed By: kennyhorror
Differential Revision: D4985754
fbshipit-source-id: b3d12b5e2cfa1b54fca9126768c84c902664ef28
Summary:
When appending net A to net B, an external input of net A should not be added as
an external input of net B if net B is outputting that blob.
Reviewed By: dzhulgakov
Differential Revision: D4975921
fbshipit-source-id: a5c0ada7b96d851e57d345244d322dd93c7be8e4
Summary: In certain situation, like in D4907916 where we insert additional step in the middle of a model, it's neccessary to keep the blob names constant across model helper so that it doesn't break communication schema.
Reviewed By: kennyhorror
Differential Revision: D4981527
fbshipit-source-id: 6b8d6d240279dd48f801cfacbaa1d320ba54d694
Summary: Inegration of the CRF Layer in DeepText wordmodels + Implementing the viterbi decode operator in C++ instead of python so that the CRF models can be deployed in production.
Differential Revision: D4912196
fbshipit-source-id: 64f499a1bd47e811e7a96dde839904dcd05cacb3
Summary: Calling `set()` or `set_value()` on Scalar is dangerous as something might be holding a reference to it. This is especially true with `LayerModel`, where instantiation is delayed. The code may still run but it will produce unexpected results, i.e., values maybe written to the wrong blob.
Reviewed By: kennyhorror
Differential Revision: D4955366
fbshipit-source-id: f5e8694a9a411ee319ca9f39a0fed632d180b8a5
Summary:
This is preamble for the "diagonal executor". Instead of creating a Net for each timestep, we have a single executor for the RecurrentNetworkOp that manages ops per timestep.
This will be used if net_type='rnn', so one can still use the old way by using a net type of 'simple' or 'dag' (so there is effective kill-switch if there are some issues with this).
Did this only for the forward-model. Gradient op will follow later on, but it is basically similar, just reverse order.
Reviewed By: salexspb
Differential Revision: D4979933
fbshipit-source-id: bda77918ec518cb6b29d7021ee036d59eb2dd303
Summary:
It turned out that we can not run PackedFC on a machine that does not have avx2 right now, as there is an known issue with MKL 2017.0.098 that produces wrong results on non-avx2 machines.
I just moved this test from here because this is not the purpose of this test
Reviewed By: salexspb
Differential Revision: D4974021
fbshipit-source-id: c5b82a41021defc9946a8219f59b28abb13d3beb
Summary: Previously, the code below would go out of bound.
Reviewed By: xianjiec
Differential Revision: D4968037
fbshipit-source-id: 3760e2cddc919c45d85ac644ac3fabf72dbaf666
Summary:
Implement NormalizeOP for GPU using CUDA, and re-write the graident to be a function of the output
so its more efficent specially for CUDA implemntation.
Reviewed By: akyrola
Differential Revision: D4971300
fbshipit-source-id: e0ab66462000988aaf1f26010ea550533d107167
Summary: As in the title + added scuba logging of the results.
Reviewed By: andrewwdye
Differential Revision: D4974261
fbshipit-source-id: 3e05b97133be95ffe37c8bcafd8a5a6bf3e7da93
Summary: Only CPU impl is available at the moment. Wrote simple cuda kernels.
Reviewed By: akyrola
Differential Revision: D4577736
fbshipit-source-id: c2540aa9d332fcdeac46cc7f89aab164d107d7a8
Summary: Both SquaredL2Distance and SquaredL2DistanceGradient had bad CUDA implementations. Use proper reductions and batched kernels.
Reviewed By: asaadaldien
Differential Revision: D4968527
fbshipit-source-id: f7cf82072d38bc127c757c5751863a9439aca8b5
Summary: Implement CPU and GPU gradient for Leaky ReLU op.
Differential Revision: D4943905
fbshipit-source-id: 541f13cd5f274a18b69ecf1362722b1bc0105ad9
Summary:
Instance norm failed grad check in some cases that needed a smaller step size. Decreased step size, but also increased threshold slightly.
Related diff: D4627379
Reviewed By: kennyhorror
Differential Revision: D4941827
fbshipit-source-id: d6f565340da92af40bfee90627960a3356c69412
Summary:
This is a naive layering approroach till we have a better
one. It could be c++ based and support diagonal execution. Not integrating into main LSTM API yet as this might be revised a bit. Would like to land so we can compare current implementation in the benchmark and also use this as an example of how LSTMs could be combined (as some folks are doing similar things with some variations).
Later we can LSTM() support API of layered_LSTM() and also change it under the hood so it stacks cells into a bigger cell instead. This way if we make RNN op use a kind of a DAG net, then RNN op can provide more parallelizm in stacked cells.
Reviewed By: urikz
Differential Revision: D4936015
fbshipit-source-id: b1e25f12d985dda582f0c67d9a02508027e5497f
Summary:
This is useful when data has standalone sequences which are
not connected to each other by any meaningful context
Reviewed By: yqwangustc
Differential Revision: D4835164
fbshipit-source-id: f95626acc26acc3eba3bca7efb08ed1dbdb36c83
Summary:
A new argument `blob_name_overrides` is added, which is to specify the
destination of loaded blob (in order to allow they have different names than
what are in the saved file/db).
This will be used for parameter initailization by pretrained model
in Dper 2. When loading a blob, we need to avoid name collision by assigning the
loaded blob with a new (temp) name.
Reviewed By: xianjiec
Differential Revision: D4952485
fbshipit-source-id: 4ce79bf40223314bb94981c22cbe537ae3f3d27c
Summary:
Free scratch blobs at data workers exit. Also add utility function that you can use to reset gradient blobs easily:
from caffe2.python import utils
grad_blobs = [b for b in workspace.Blobs() if b.endswith("_grad") or b.endswith("_shared")]
utils.ResetBlobs(grad_blobs)
Reviewed By: rpenggithub
Differential Revision: D4955531
fbshipit-source-id: d33b2bb2b5247dd2c4cff51c82b1257c871a4179
Summary: Current eval nets contain loss operators; see example: https://fburl.com/6otbe0n7, which is unnecessary. This diff is to remove them from the eval net.
Differential Revision: D4934589
fbshipit-source-id: 1ba96c20a3a7ef720414acb4124002fb54cabfc7
Summary: Now you can call coordinator.stop_coordinator("train") to stop the train model's data input and release its memory.
Reviewed By: rpenggithub
Differential Revision: D4955014
fbshipit-source-id: c1bc3ec67337b94aff8ea9b306c3b4158eeef42c
Summary:
The _param_init_net does not exist. All the other places reference
param_init_net instead. So far no one has encountered any problem
because all the passed params are BlobReferences. This diff makes
this assumption explicit.
Reviewed By: azzolini
Differential Revision: D4922930
fbshipit-source-id: e6dbd7a29ea640b7e62fcfec7ced3cc7d149f872
Summary:
ScaleGradient is a helper operator that does no actual numerical computation,
and in the gradient computation phase scales the gradient from being computed
through it.
Differential Revision: D4920719
fbshipit-source-id: 0e1e0888f79594be874fdbdda5ccef7389064c50
Summary:
Issue is that AliasOp doesn't work well with swaps that we do for
param.grad and param.accGrad. Tensors become the same if there is no
reallocation of the gradient tensor inside the backward cell net's
local workspace.
bug explanation from akyrola:
```
gpu_0/decoder/decoder_hidden_encoder_outputs_sum_grad: tensor A
on each timestap back to 0, we Alias
gpu_0/decoder/weighted_encoder_outputs_grad,
so then also
gpu_0/decoder/weighted_encoder_outputs_grad: tensor A
It's acc is:
gpu_0/decoder/weighted_encoder_outputs_grad_acc: tensor B
Now after timesteps, we swap (line 626) with _acc to get
gpu_0/decoder/weighted_encoder_outputs_grad: tensor B
gpu_0/decoder/weighted_encoder_outputs_grad_acc: tensor A
OPTION A -- batch size is same as before or smaller:
Then on next iteration, we do again the Alias to
gpu_0/decoder/decoder_hidden_encoder_outputs_sum_grad, so now
gpu_0/decoder/weighted_encoder_outputs_grad: tensor A
and also
gpu_0/decoder/weighted_encoder_outputs_grad_acc: tensor A
swapping them does nothing and they are the same
OPTION B -- batch size increases
gpu_0/decoder/decoder_hidden_encoder_outputs_sum_grad is reallocated,
becomes tensor C
gpu_0/decoder/weighted_encoder_outputs_grad becomes tensor C with
Alias
gpu_0/decoder/weighted_encoder_outputs_grad_acc: is tensor A
```
Reviewed By: urikz
Differential Revision:
D4946730
Tags: rnn, caffe2
fbshipit-source-id: b52d63cb238b81d2ad40e05e70deb32a81336f47
Summary: A layer that takes raw ids as inputs and outputs the indices which can be used as labels. The mapping will be stored with the model.
Reviewed By: kittipatv
Differential Revision: D4902556
fbshipit-source-id: 647db47b0362142cdba997effa2ef7a5294c84ee
Summary:
Adding add_weight_decay and image_input to brew module & remove `getWeights` and `getBias` from CNNModelHelper
With fbgs `useWeights`, the results show that noone but add_weight_decay is using this function. I checked with oculus people, their getWeights is a different function.
kennyhorror Please notice whether this is going to affect you :)
Reviewed By: salexspb
Differential Revision: D4945392
fbshipit-source-id: 4ef350fd81dd40a91847e9f3ebc5421eb564df32
Summary: printing resnet training loss and accuracy for each batch so that people will have better idea of what is going on
Reviewed By: pietern
Differential Revision: D4945390
fbshipit-source-id: 0fcd60f4735e81641355aba6e6cbf0e57e886e38
Summary:
lengthTile goes from 1 to multiple, the gradient op is simply the reverse,
by adding up the fanned-out rows of gradients together into 1
Reviewed By: kittipatv
Differential Revision: D4943375
fbshipit-source-id: deae9984e849974a0d484a10b94efdb1d30941cc
Summary:
Added optional support for using activation blobs for sharing as well. Doing this change revealed an non-optimal implementation in the blob sharing: we need to prefer to reuse freeblobs by prefering those blobs that are already shared by many other blobs. Otherwise the memory usage can increase when the pool of 'free blobs' grows.
Also, my first version only passed "free blobs" (i.e blobs in recycling pool) down the first branch when operators forked. But now we pass those blobs that were not used by the first branch down the second branch and so on.
Also added support for blob size information in the heuristic. This uses the shape inference mechanism.
I had to also do some small tweaks:
- use Sum() operator as a way to match shapes of blobs that had otherwise unknown shapes. This is related to the Sum() operator that is added to combine multiple incoming gradient inputs (with _autosplit gradients).
- a couple of random shape inference fixes
This reduces the Resnet-50 memory usage on 64 batch from 9.45 Gig to 8.5 Gig.
For a 32 batch, the memory usage is 4330 MiB, down from 4800 MB, compared to Torch's 6856MiB (thanks prigoyal for checking this for me).
This is unfortunately quite a bunch to review...
Reviewed By: asaadaldien
Differential Revision: D4393909
fbshipit-source-id: 9c7c94125f96512bea80463ebcb63c215ef95ff9
Summary:
This diff contains the following changes:
- implementing __repr__ on Field types; this makes it a little easier to see what broken in the unit tests
- preserve the shape of ndarray input to schema; previously, empty and scalar arrays lose their shape, while other keeps the shape.
- type-checking ndarray input; this ensures basic integrety of schema
Reviewed By: xianjiec
Differential Revision: D4913030
fbshipit-source-id: bd0f6b8722d95bfe800edf98ba05029c5b99d2af
Summary:
This PR is based on commit "977c6b3" as this version allows MKL to use all the cores available.
All MKL related files are added here after incorporating review comments, major changes include
1. usage of Clang-format(Linter) with --style = Google
2. usage of macros for checking input and filter dimension in the mkl operators
3. merged Max and Average pooling functions
4. created a new folder for mkl related python scripts in Python folder and moved them there
5. there is no mkl_alexnet_test.py as that was redundant while convnet_benchmark.py does the same thing
Closes https://github.com/caffe2/caffe2/pull/270
Differential Revision: D4905219
Pulled By: Yangqing
fbshipit-source-id: e5f5b189714a835b93b9ebda24c52e09572dfca7
Summary:
If exception is getting thrown inside of the namescope it won't be reset to
it's previous value. This diff is changing this behavior to expected one.
Reviewed By: kittipatv
Differential Revision: D4928621
fbshipit-source-id: 1d3579f2093ca60901b0d37ae3f2108deb2333ea
Summary: Instead of requiring gradient updates on GPU, this change will allow the usage when loss computation happens on GPU while all grad updates happen on CPU.
Reviewed By: jhcross
Differential Revision: D4943996
fbshipit-source-id: 1f2144c4277dfdb865877e0d0216ca1ac7dd7309
Summary:
Add a pointwise `IsMemberOf` operator to Caffe2.
The original idea was `In` but I think this is not so clear.
I used `UnaryElementwiseWithArgsOp` at some point, but it was making the code a bit more difficult to read without bringing any feature.
Reviewed By: ender-wieczorek
Differential Revision: D4912655
fbshipit-source-id: 716b66bb51468dd59db5f76f23d78cda85961b58
