-
Notifications
You must be signed in to change notification settings - Fork 421
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Ignore non-existent grad tensors for semi-sync (#2490)
Summary: During semi-sync, we need to ignore embedding tensor grads that are `None`, otherwise `torch.autograd.backward` will fail with error `grad can be implicitly created only for scalar outputs`. This is a valid scenario if, for example, the embeddings are looked up but never used for the final loss computation Reviewed By: che-sh Differential Revision: D63379382
- Loading branch information
1 parent
d1a2990
commit 953473e
Showing
2 changed files
with
37 additions
and
3 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters