Pytorch Autograd - Backward
The backward method of autograd provides an alternate way to compute gradients to the grad method. When called on a function that has a computational graph defined, (that is, a function which is defined in terms of torch tenros) the backward method will compute gradients for all tensors that have been marked with "requires_grad" to be True. The results will be stored in the "grad" attrlibute of the tensor for which the gradient was computed.Example 1
x = torch.tensor([1.0])
y = torch.tensor([1.0], requires_grad=True)
f = x + 2*y + 5
f.backward()
test1 = x.grad
test2 = y.grad
In this example, test1 is None, whereas test2 has a computed value because it was marked with "requires_grad".
Example 2
import torch
def func(x, y):
return x+2*y + 5
args = torch.tensor([1.0,2.0], requires_grad=True)
loss1 = func(*args)
loss1.backward()
test = args.grad
Accumulated Results and Zeroing
When bakward is called, the results are saved in each tensor that the function for which backward calls on depends on. However, successive calls to backward will throw an error, unless you specify that in the calls prior to the last call should retain their graph.
x = torch.tensor([1.0])
y = torch.tensor([1.0], requires_grad=True)
f = x + 2*y + 5
f.backward(retain_graph=True)
f.backward()
test1 = x.grad
test2 = y.grad
When this is done, the gradient gets accumulated in the tensors. Notice that test2 in this code will be 4, instead of 2. This ability to accumulate the gradient is useful for some models, but must in kept in mind for models for which you dont want to accumulate the result. In these cases, you may need to explicitly tell Pytorch to zero out the gradient. (see neural network traiing for an example which zeroes out the gradient )