Pytorch [Basics] — Intro to Dataloaders and Loss Functions

Datasets and Dataloaders

A custom dataset class is created using 3 main components.

__init__
__len__
__getitem__

class CustomDataset(Dataset):
    def __init__(self):
        pass    def __getitem__(self, index):
        pass    def __len__(self):
        pass

__init__ : used to perform initializing operations such as reading data and preprocessing.

__len__ : returns the size of the input data.

__getitem__ : returns data (input and output) in batches.

A dataloader is then used on this dataset class to read the data in batches.

train_loader = DataLoader(custom_dataset_object, batch_size=32, shuffle=True)

Let’s implement a basic PyTorch dataset and dataloader. Assume you had input and output data as -

X : 1, 2, 3, 4, 5, 6, 7, 8, 9, 10

y : 0, 0, 0, 1, 0, 1, 1, 0, 0, 1

Let’s define the dataset class. We will return a tuple of (input, output).

class CustomDataset(Dataset):
    
    def __init__(self, X_data, y_data):
        self.X_data = X_data
        self.y_data = y_data
        
    def __getitem__(self, index):
        return self.X_data[index], self.y_data[index]
        
    def __len__ (self):
        return len(self.X_data)

Initialise the dataset object. The inputs have to be of the type Tensor.

data = CustomDataset(torch.FloatTensor(X), torch.FloatTensor(y))

Let’s use the methods __len__() and __getitem__() . __getitem__() takes the index as input.

data.__len__()################### OUTPUT #####################
10

Printing out the 4th element (3rd index) from out data.

data.__getitem__(3)################### OUTPUT #####################
(tensor(4.), tensor(1.))

Let’s initialise our dataloader now. Here we specify the batch size and shuffle.

data_loader = DataLoader(dataset=data, batch_size=2, shuffle=True)data_loader_iter = iter(data_loader)
print(next(data_loader_iter))################### OUTPUT #####################
[tensor([3., 6.]), tensor([0., 1.])]

Let’s use the dataloader with a for loop.

for i,j in data_loader:
    print(i,j)################### OUTPUT #####################tensor([ 1., 10.]) tensor([0., 1.])
tensor([4., 6.]) tensor([1., 1.])
tensor([7., 5.]) tensor([1., 0.])
tensor([9., 3.]) tensor([0., 0.])
tensor([2., 8.]) tensor([0., 0.])

Loss Functions

Following are the commonly used loss functions for different deep learning tasks.

Regression:

torch.nn.L1Loss()
torch.nn.MSELoss()

Classification:

torch.nn.BCELoss()
torch.nn.BCEWithLogitsLoss()
torch.nn.NLLLoss()
torch.nn.CrossEntropyLoss()

Learn more about the loss functions from the official PyTorch docs .

Import Libraries

import torch
import torch.nn as nn

Regression

Let’s begin by defining the actual and predicted output tensors in order to calculate the loss.

y_pred = torch.tensor([[1.2, 2.3, 3.4], [4.5, 5.6, 6.7]], requires_grad=True)print("Y Pred: \n", y_pred)
print("\nY Pred shape: ", y_pred.shape, "\n")print("=" * 50)y_train = torch.tensor([[1.2, 2.3, 3.4], [7.8, 8.9, 9.1]])
print("\nY Train: \n", y_train)
print("\nY Train shape: ", y_train.shape)
###################### OUTPUT ######################Y Pred: 
 tensor([[1.2000, 2.3000, 3.4000],
        [4.5000, 5.6000, 6.7000]], requires_grad=True)Y Pred shape:  torch.Size([2, 3]) ==================================================Y Train: 
 tensor([[1.2000, 2.3000, 3.4000],
        [7.8000, 8.9000, 9.1000]])Y Train shape:  torch.Size([2, 3])

Mean Absolute Error — torch.nn.L1Loss()

The input and output have to be the same size and have the dtype float .

y_pred = (batch_size, *) and y_train = (batch_size, *) .

mae_loss = nn.L1Loss()print("Y Pred: \n", y_pred)print("Y Train: \n", y_train)output = mae_loss(y_pred, y_train)
print("MAE Loss\n", output)output.backward()
###################### OUTPUT ######################Y Pred: 
 tensor([[1.2000, 2.3000, 3.4000],
        [4.5000, 5.6000, 6.7000]], requires_grad=True)
Y Train: 
 tensor([[1.2000, 2.3000, 3.4000],
        [7.8000, 8.9000, 9.1000]])
MAE Loss
 tensor(1.5000, grad_fn=<L1LossBackward>)

Mean Squared Error — torch.nn.MSELoss()

The input and output have to be the same size and have the dtype float .

y_pred = (batch_size, *) and y_train = (batch_size, *) .

mse_loss = nn.MSELoss()print("Y Pred: \n", y_pred)print("Y Train: \n", y_train)output = mse_loss(y_pred, y_train)
print("MSE Loss\n", output)output.backward()
###################### OUTPUT ######################Y Pred: 
 tensor([[1.2000, 2.3000, 3.4000],
        [4.5000, 5.6000, 6.7000]], requires_grad=True)
Y Train: 
 tensor([[1.2000, 2.3000, 3.4000],
        [7.8000, 8.9000, 9.1000]])
MSE Loss
 tensor(4.5900, grad_fn=<MseLossBackward>)

Binary Classification

y_train has two classes - 0 and 1. We use this BCE loss function in the situation when the final output from the network is a single value (final dense layer is of size 1) that lies between 0 and 1.

Binary classification can be re-framed to use NLLLoss or Crossentropy loss if the output from the network is a tensor of length 2 (final dense layer is of size 2) where both values lie between 0 and 1.

Let’s define the actual and predicted output tensors in order to calculate the loss.

y_pred = torch.tensor([[1.2, 2.3, 3.4], [7.8, 8.9, 9.1]], requires_grad = True)
print("Y Pred: \n", y_pred)
print("\nY Pred shape: ", y_pred.shape, "\n")print("=" * 50)y_train = torch.tensor([[1, 0, 1], [0, 0, 1]])
print("\nY Train: \n", y_train)
print("\nY Train shape: ", y_train.shape)
###################### OUTPUT ######################Y Pred: 
 tensor([[1.2000, 2.3000, 3.4000],
        [7.8000, 8.9000, 9.1000]], requires_grad=True)Y Pred shape:  torch.Size([2, 3]) ==================================================Y Train: 
 tensor([[1, 0, 1],
        [0, 0, 1]])Y Train shape:  torch.Size([2, 3])

Binary Cross Entropy Loss — torch.nn.BCELoss()

The input and output have to be the same size and have the dtype float .

y_pred = (batch_size, *) , Float (Value should be passed through a Sigmoid function to have a value between 0 and 1)

y_train = (batch_size, *) , Float

bce_loss = nn.BCELoss()y_pred_sigmoid = torch.sigmoid(y_pred)print("Y Pred: \n", y_pred)print("\nY Pred Sigmoid: \n", y_pred_sigmoid)print("\nY Train: \n", y_train.float())output = bce_loss(y_pred_sigmoid, y_train.float())
print("\nBCE Loss\n", output)output.backward()
###################### OUTPUT ######################Y Pred: 
 tensor([[1.2000, 2.3000, 3.4000],
        [7.8000, 8.9000, 9.1000]], requires_grad=True)Y Pred Sigmoid: 
 tensor([[0.7685, 0.9089, 0.9677],
        [0.9996, 0.9999, 0.9999]], grad_fn=<SigmoidBackward>)Y Train: 
 tensor([[1., 0., 1.],
        [0., 0., 1.]])BCE Loss
 tensor(3.2321, grad_fn=<BinaryCrossEntropyBackward>)

Binary Cross Entropy with Logits Loss — torch.nn.BCEWithLogitsLoss()

The input and output have to be the same size and have the dtype float . This class combines Sigmoid and BCELoss into a single class. This version is numerically more stable than using Sigmoid and BCELoss individually.

y_pred = (batch_size, *) , Float

y_train = (batch_size, *) , Float

bce_logits_loss = nn.BCEWithLogitsLoss()print("Y Pred: \n", y_pred)print("\nY Train: \n", y_train.float())output = bce_logits_loss(y_pred, y_train.float())
print("\nBCE Loss\n", output)output.backward()
###################### OUTPUT ######################Y Pred: 
 tensor([[1.2000, 2.3000, 3.4000],
        [7.8000, 8.9000, 9.1000]], requires_grad=True)Y Train: 
 tensor([[1., 0., 1.],
        [0., 0., 1.]])BCE Loss
 tensor(3.2321, grad_fn=<BinaryCrossEntropyWithLogitsBackward>)

Multiclass Classification

Let’s define the actual and predicted output tensors in order to calculate the loss.

y_train has 4 classes - 0, 1, 2, and 3.

y_pred = torch.tensor([[1.2, 2.3, 3.4], [4.5, 5.6, 6.7], [7.8, 8.9, 9.1]], requires_grad = True)
print("Y Pred: \n", y_pred)
print("\nY Pred shape: ", y_pred.shape, "\n")print("=" * 50)y_train = torch.tensor([0, 1, 2])
print("\nY Train: \n", y_train)
print("\nY Train shape: ", y_train.shape)
###################### OUTPUT ######################Y Pred: 
 tensor([[1.2000, 2.3000, 3.4000],
        [4.5000, 5.6000, 6.7000],
        [7.8000, 8.9000, 9.1000]], requires_grad=True)Y Pred shape:  torch.Size([3, 3]) ==================================================Y Train: 
 tensor([0, 1, 2])Y Train shape:  torch.Size([3])

Negative Log Likelihood — torch.nn.NLLLoss()

y_pred = (batch_size, num_classes) , Float (Value should be passed log probabilities which are obtained using the log_softmax function.

y_train = (batch_size) , Long (range of values = 0, num_classes-1). The classes must start from 0, 1, 2, ...

nll_loss = nn.NLLLoss()y_pred_logsoftmax = torch.log_softmax(y_pred, dim = 1)print("Y Pred: \n", y_pred)print("\nY Pred LogSoftmax: \n", y_pred_logsoftmax)print("\nY Train: \n", y_train)output = nll_loss(y_pred_logsoftmax, y_train)
print("\nNLL Loss\n", output)output.backward()
###################### OUTPUT ######################Y Pred: 
 tensor([[1.2000, 2.3000, 3.4000],
        [4.5000, 5.6000, 6.7000],
        [7.8000, 8.9000, 9.1000]], requires_grad=True)Y Pred LogSoftmax: 
 tensor([[-2.5672, -1.4672, -0.3672],
        [-2.5672, -1.4672, -0.3672],
        [-2.0378, -0.9378, -0.7378]], grad_fn=<LogSoftmaxBackward>)Y Train: 
 tensor([0, 1, 2])NLL Loss
 tensor(1.5907, grad_fn=<NllLossBackward>)

CrossEntropyLoss — torch.nn.CrossEntropyLoss()

This class combines LogSoftmax and NLLLoss into a single class.

y_pred = (batch_size, num_classes) , Float

y_train = (batch_size) , Long (range of values = 0, num_classes-1). The classes must start from 0, 1, 2, ...

ce_loss = nn.CrossEntropyLoss()print("Y Pred: \n", y_pred)print("\nY Train: \n", y_train)output = ce_loss(y_pred, y_train)
print("\nNLL Loss\n", output)output.backward()
###################### OUTPUT ######################Y Pred: 
 tensor([[1.2000, 2.3000, 3.4000],
        [4.5000, 5.6000, 6.7000],
        [7.8000, 8.9000, 9.1000]], requires_grad=True)Y Train: 
 tensor([0, 1, 2])NLL Loss
 tensor(1.5907, grad_fn=<NllLossBackward>)