Pytorch Autograd

Autograd is the heart of Pytorchs machine learning models. In particular, it is used to compute gradients and run optimizations. Users of Pytorchs models can mostly ignore Autograd, as it largely functions in the background, however, it does provide powerful capabilities for users who wish to dig into its details.

Computational Graph

When using the Pytorch's tensor to define a computation, Pytorch will create a computational graph that maps the relationships in the computation.

The following code shows a simple calculation using tensors. The end result will be tensor([8.0]).
When running this code, pytorch creates a computational graph which can be used by the methods below to calculate gradients.

The following code creates a function using def, and wraps it in a function that creates the computational graph.

Methods

AFter

• grad - simple method for computing gradients
• backward

Simple Example

Steps to Use

• Requries Grad - in order to use autograd, you need to create a tensor or tensors that are used in a computation and for which you indicate that it should participate in the autograd computation graph. This is indicated by setting the requires_grad parameter to be True
  
  
  x = torch.tensor(1.0, requires_grad=True)
• Define a Function that uses the Tensor
  f = x**4
  def model(a, b): return a**2 + 0.5*b**2
• Backward The backward function runs a gradient compuation.
  f.backward() args = torch.tensor([1,2], requires_grad=True) loss1 = model(*args) loss1.backward()

Zeroing out the Gradient

Once you have a tensor, you can instruct to compute a gradient using backpropagation by calling the backward function.

Full Scripts