5 Useful Functions in PyTorch

Introduction

Info

This is a test article, written to test how much I can accommodate in each page, this is adapted from my notebook for the zerotogans certification course.

PyTorch is a popular open-source machine learning library that is renowned for its flexibility and ease of use. It was developed primarily by Facebook’s AI Research lab (FAIR). It provides a rich set of functions for building, training, and deploying deep learning models. Its rich ecosystem includes modules for optimization, data loading, visualization, and integration with other popular libraries such as NumPy and SciPy. Below we will be taking a brief look at the following 5 functions:

• torch.tensor()
• torch.rand()
• torch.numel()
• torch.cat()
• torch.le()

# Uncomment and run the appropriate command for your operating system, if required
 
# Linux / Binder
# !pip install numpy torch==1.7.0+cpu torchvision==0.8.1+cpu torchaudio==0.7.0 -f [https://download.pytorch.org/whl/torch_stable.html](https://download.pytorch.org/whl/torch_stable.html)
 
# Windows
# !pip install numpy torch==1.7.0+cpu torchvision==0.8.1+cpu torchaudio==0.7.0 -f [https://download.pytorch.org/whl/torch_stable.html](https://download.pytorch.org/whl/torch_stable.html)
 
# MacOS
# !pip install numpy torch torchvision torchaudio

# Import torch and other required modules
import torch

Function 1 - torch.tensor()

# Example 1
torch.tensor([[1, 2], [3, 4.]])

tensor([[1., 2.],
        [3., 4.]])

Above code snippet creates a 2x2 PyTorch tensor with integer and float values. However notice that when the tensor is created, all elements are converted to floats, as all elements of PyTorch tensors need to be of the same datatype.

# Example 2
torch.tensor(4.)

tensor(4.)

Creates a PyTorch tensor containing a single floating-point number 4.

# Example 3 - breaking
torch.tensor([[1, 2], [3, 4, 5]])

---------------------------------------------------------------------------

ValueError                                Traceback (most recent call last)

<ipython-input-27-862912d5191b> in <cell line: 2>()
      1 # Example 3 - breaking
----> 2 torch.tensor([[1, 2], [3, 4, 5]])


ValueError: expected sequence of length 2 at dim 1 (got 3)

In PyTorch, tensors are a collection of matrices stacked on top of each other. Matrices require consistent dimensions for proper operations. Inconsistent dimensions lead to ambiguity and can cause errors during operations like arithmetic or reshaping. Above code snippet creates a PyTorch tensor with uneven inner lists, therefore causing an error due to inconsistent dimensions.

the torch.tensor() functions are very useful when one wants to manually create tensors for operations or demonstrations. It is also an excellent tool to demonstrate the functioning of tensors.

Let’s save our work using Jovian before continuing.

Function 2 - torch.rand()

The torch.rand() function takes the following parameters as inputs within the parentheses:

torch.rand(*size, *, generator=None, out=None, dtype=None, layout=torch.strided, device=None, requires_grad=False, pin_memory=False) → Tensor

It a function that takes the dimensions of a tensor under the size argument and returns a tensor of the same dimensions populated with floating point numbers randomly sampled from a uniform distribution on the interval $[0,1)$.

It also has dtype and required_grad parameters that need to be mentioned if the tensor needs to be populated with a specific data type or if autograd needs to keep track of the operations on the returned tensor elements, repectively.

For more information check out the docs.

# Example 1
torch.rand(1)

tensor([0.9667])

The above code snippet provides a 1 by 1 PyTorch tensor that has only one element samble from the uniform distribution. This is akin to randomly sampling just one element from the underlying uniform distribution.

# Example 2
torch.rand(1, 2, 3, 4)

tensor([[[[0.5317, 0.7106, 0.9012, 0.9234],
          [0.3289, 0.2093, 0.5495, 0.7578],
          [0.0761, 0.6425, 0.9418, 0.6874]],

         [[0.9743, 0.8962, 0.8319, 0.9644],
          [0.6160, 0.5674, 0.8149, 0.4495],
          [0.1996, 0.8394, 0.7290, 0.1429]]]])

Above code snippet creates a 4-dimensional PyTorch tensor with random values, of the shape: (1, 2, 3, 4).

# Example 3 - breaking (to illustrate when it breaks)
torch.rand(1, 2, dtype='str')

---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

<ipython-input-38-39d4ef9b50e7> in <cell line: 2>()
      1 # Example 3 - breaking (to illustrate when it breaks)
----> 2 torch.rand(1, 2, dtype='str')


TypeError: rand() received an invalid combination of arguments - got (int, int, dtype=str), but expected one of:
 * (tuple of ints size, *, torch.Generator generator, tuple of names names, torch.dtype dtype, torch.layout layout, torch.device device, bool pin_memory, bool requires_grad)
 * (tuple of ints size, *, torch.Generator generator, Tensor out, torch.dtype dtype, torch.layout layout, torch.device device, bool pin_memory, bool requires_grad)
 * (tuple of ints size, *, Tensor out, torch.dtype dtype, torch.layout layout, torch.device device, bool pin_memory, bool requires_grad)
 * (tuple of ints size, *, tuple of names names, torch.dtype dtype, torch.layout layout, torch.device device, bool pin_memory, bool requires_grad)

The code torch.rand(1, 2, dtype='str') raises an error because dtype='str' is not a valid data type for the torch.rand() function. The dtype argument in torch.rand() specifies the data type of the elements in the tensor, but PyTorch does not support creating tensors with string data type using this function. Instead, you can only specify numeric data types like torch.float32, torch.float64, torch.int32, etc.

The function torch.rand() is used to create tensors initialized with random values sampled from a uniform distribution ranging from 0 to 1. This function is commonly used for initializing weights in neural networks, creating random data for simulations, or any other application where random numbers are required.

Function 3 - torch.numel()

The torch.numel function takes a tensor as a parameter and returns the total number of elements inside that tensor.

This is a rather easy thing to calculate, the only thing we need it the dimensions of the tensors. The total number of elements in a tensor can be calculate by multiplying the size of each dimension together.

$$\left(\begin{array}{cccc}{a}_{11}& a_{12}& ...& a_{1n}\\ a_{21}& a_{22}& ...& a_{1n}\\ ...& ...& ...& ...\\ a_{m1}& a_{m2}& ...& a_{mn}\end{array}\right)$$

Since we know for a $m\times n$ matrix, the number of elements will be m*n, similary for tensors of higher dimensions we can get the total number of elements by multiplying the number of elements in each dimensions. An example in three dimensions is given below:

The dimensions of the tnesor given in the image is $4\times 4\times 2$, therefore we can similary see that the total number of elements in the tensor is 28.

# Example 1 - working
torch.numel(torch.rand(1, 2, 3, 4, 5))

120

Above code snipper calculates the total number of elements in a PyTorch tensor created by torch.rand() with dimensions $(1,2,3,4,5)$. Since the tensor has dimensions (1, 2, 3, 4, 5), it contains 1 × 2 × 3 × 4 × 5 = 120 elements.

Therefore, torch.numel(torch.rand(1, 2, 3, 4, 5)) returns the value 120.

# Example 2 - working
torch.numel(torch.tensor(1.))

1

Above code calculates the number of elements in a PyTorch tensor created with a single scalar value 1.0. Since the tensor has only one element, the function returns a 1.

# Example 3 - breaking (to illustrate when it breaks)
torch.numel(1, 2, 3, 4, 5)

---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

<ipython-input-42-ea60fee2f36f> in <cell line: 2>()
      1 # Example 3 - breaking (to illustrate when it breaks)
----> 2 torch.numel(1, 2, 3, 4, 5)


TypeError: numel() takes 1 positional argument but 5 were given

The function torch.numel() expects a tensor as its argument, but individual integers (1, 2, 3, 4, 5) were provided instead of a tensor. Therefore, calling torch.numel(1, 2, 3, 4, 5) will raise an error because it cannot compute the number of elements in non-tensor inputs.

torch.numel() is particularly useful in scenarios where we need to know the size of the tensor for reshaping operations, memory allocation, or understanding the size of the data. It can be used for validating that a tensor has a specific size before performing certain operations on it. It can also be used to provide a count of elements to aid iteration over a tensor. These are the main uses of the torch.numel() function.

Function 4 - torch.cat()

torch.cat(tensors, dim=0, *, out=None) → Tensor

torch.cat() is a PyTorch function that is used to concatenate tensors along a specified dimension. It takes a sequence (list, tuple, etc.) of tensors and concatenates them along the specified dimension.

# Example 1 - working
tensor1 = torch.tensor([[1, 2], [3, 4]])
tensor2 = torch.tensor([[5, 6]])
torch.cat((tensor1, tensor2), dim=0)

tensor([[1, 2],
        [3, 4],
        [5, 6]])

Explanation about example

# Example 2 - working
tensor1 = torch.tensor([[1, 2], [3, 4]])
tensor2 = torch.tensor([[5], [6]])
 
torch.cat((tensor1, tensor2), dim=1)

tensor([[1, 2, 5],
        [3, 4, 6]])

Code snippet given above concatenates along the second dimension (columns).

# Example 3 - breaking (to illustrate when it breaks)
tensor1 = torch.tensor([[1, 2], [3, 4]])
tensor2 = torch.tensor([[5, 6, 7]])
 
torch.cat((tensor1, tensor2), dim=0)

---------------------------------------------------------------------------

RuntimeError                              Traceback (most recent call last)

<ipython-input-46-07e4ddbe75d6> in <cell line: 5>()
      3 tensor2 = torch.tensor([[5, 6, 7]])
      4 
----> 5 torch.cat((tensor1, tensor2), dim=0)


RuntimeError: Sizes of tensors must match except in dimension 0. Expected size 2 but got size 3 for tensor number 1 in the list.

In this example, tensor1 has the shape (2, 2) whereas tensor2 has shape (1, 3). When attempting to concatenate along 0th dimension (rows), the error occurs because the sizes of the tensors don’t match along dimension 1 (columns), which violates the concatenation rule.

torch.cat() can be used in a lot of different places, starting from Data Preprocessing to concatenating two segmented datasets together. It can be used to add more layers with pretrained parameters to a model to boost its accuracy and make it a deeper network.

Function 5 - torch.le()

 torch.le(input, other, *, out=None) → Tensor

For any given tensor(s) tensor.le() performs element-wise comparison between the two tensors or a tensor and a scalar, and returns a new boolean tensor indicating whether each element in the input tensor(s) is less than or equal to the corresponding element in the other tensor or scalar.

# Example 1 - working
tensor = torch.tensor([1, 2, 3, 4, 5])
torch.le(tensor, 3)

tensor([ True,  True,  True, False, False])

Code shows a simple elementwise comparison of a tensor against a scalar number and returning a Boolean tensor.

# Example 2 - working
tensor1 = torch.tensor([[1, 2, 3], [4, 5, 6]])
tensor2 = torch.tensor([2, 3, 4])
 
result = torch.le(tensor1, tensor2)

Above code shows the elementwise comparison between two tensors. When we are comparing two tensors we must keep in mind that their dimensions must match for the elementwise comparison while broadcasting to take place successfully, otherwise the function will throw a RuntimeError saying that the dimensions do not match.

Read more about boradcasting at numpy docs and pytorch docs.

# Example 3 - breaking (to illustrate when it breaks)
tensor1 = torch.tensor([1, 2, 3])
tensor2 = torch.tensor([[2, 3]])
result = torch.le(tensor1, tensor2)

---------------------------------------------------------------------------

RuntimeError                              Traceback (most recent call last)

<ipython-input-50-7c2855fec316> in <cell line: 4>()
      2 tensor1 = torch.tensor([1, 2, 3])
      3 tensor2 = torch.tensor([[2, 3]])
----> 4 result = torch.le(tensor1, tensor2)


RuntimeError: The size of tensor a (3) must match the size of tensor b (2) at non-singleton dimension 1

This error occurs because the shapes of the tensors are not compatible for broadcasting. In broadcasting, PyTorch expands the smaller tensor’s dimensions to match the larger tensor’s dimensions, but the dimensions must be either equal or one of them must be 1 for broadcasting to work. In this case, neither condition is met, leading to the error.

The torch.le() function is a good way to generate binary mask, learn about broadcasting and do trivial comparison operations over tensors for analysis purposes.

Conclusion

These were just a few functions from PyTorch’s vast repertoire of functions. Although using each and every single one of them is not necessary, neither will they be of any daily usage. However, it is a good idea to keep these functions in mind as it reduces the effort involved in doing a lot of rudimentary analysis that can instead be implemented by functions already present in Torch’s library. There are a lot of useful functions and a lot of functions that will be useful only once or twice, but iterated over a lot of times, using these functions will save a lot of time and headache while debugging out code.

Reference Links

• Official documentation for tensor operations: https://pytorch.org/docs/stable/torch.html
• Read more about Broadcasting in NumPy and PyTorch: https://www.linkedin.com/pulse/broadcasting-pytorch-numpy-tanish-kumar-9eznf/