PyTorch Activation Functions - ReLU, Leaky ReLU, Sigmoid, Tanh and Softmax - MLK - Machine Learning Knowledge (2024)

Introduction

In this tutorial, we will go through different types of PyTorch activation functions to understand their characteristics and use cases. We will understand the advantages and disadvantages of each of them, and finally, see the syntaxes and examples of these PyTorch activation functions. We will cover ReLU, Leaky ReLU, Sigmoid, Tanh, and Softmax activation functions for PyTorch in the article.

But before all that, we will touch upon the general concepts of activation function in neural networks and what are characteristics of a good activation function.

What is an Activation Function?

The concept of activation functions in the neural network is inspired by the biological neurons of the human brain. In the biological brain, neurons are fired or activated based on certain inputs from their previous connected neurons. The entire brain is a complex network of these biological neurons that are activated in a complex manner and help the functioning of the entire body.

In the artificial neural network, we have mathematical units known as artificial neurons that are connected with each other. And these neuron units are fired using the activation functions which is nothing but a mathematical function itself.

The below diagram explains this concept and comparison between the biological neuron and artificial neuron.

i) ReLU Activation Function

The ReLU or Rectified Linear Activation Function is a type of piecewise linear function.

Advantages of ReLU Activation Function

• ReLu activation function is computationally fast hence it enables faster convergence of the training phase of the neural networks.
• It is both non-linear and differentiable which are good characteristics for activation function.
• ReLU does not suffer from the issue of Vanishing Gradient issue like other activation functions. Hence it is a good choice in hidden layers of large neural networks.

Disadvantagesof ReLU Activation Function

• The main disadvantage of the ReLU function is that it can cause the problem of Dying Neurons. Whenever the inputs are negative, its derivative becomes zero, therefore backpropagation cannot be performed and learning may not take place for that neuron and it dies out.

ReLU() activation function of PyTorch helps to apply ReLU activations in the neural network.

Syntax of ReLU Activation Function in PyTorch

torch.nn.ReLU(inplace: bool = False)

Parameters

• inplace – For performing operations in-place. The default value is False.

Example of ReLU Activation Function

Now let’s look at an example of how the ReLU Activation Function is implemented in PyTorch. Here PyTorch’s nn package is used to call the ReLU function.

For input purposes, we are using the random function to generate data for our tensor. At last, we obtain the output.

iii) Sigmoid Activation Function

In PyTorch, the activation function for sigmoid is implemented using LeakyReLU() function.

Syntax of Sigmoid Activation Function in PyTorch

torch.nn.Sigmoid

Example of Sigmoid Activation Function

A similar process is followed for implementing the sigmoid activation function using the PyTorch library. The random data generated is passed to the Sigmoid() function of PyTorch and output is obtained.

iv) Tanh Activation Function

Tanh activation function is similar to the Sigmoid function but its output ranges from +1 to -1.

Advantages of Tanh Activation Function

• The Tanh activation function is both non-linear and differentiable which are good characteristics for activation function.
• Since its output ranges from +1 to -1, it can be used to transform the output of a neuron to a negative sign.

Disadvantages

• Since its functioning is similar to a sigmoid function, it also suffers from the issue of Vanishing gradient if there are too low or too high input values.

In PyTorch, the activation function for Tanh is implemented using Tanh() function.

Syntax of Tanh Activation Function in PyTorch

torch.nn.Tanh

Example of Tanh Activation Function

Once again, the Tanh() activation function is imported with the help of nn package.

Then, random data is generated and passed to obtain the output.

v) SoftMax Activation Function

Softmax function produces a probability distribution as a vector whose value range between (0,1) and the sum equals 1.

Advantages of Softmax Activation Function

• Since Softmax produces a probability distribution, it can be used as an output layer for multiclass classification.

In PyTorch, the activation function for Softmax is implemented using Softmax() function.

Syntax of Softmax Activation Function in PyTorch

torch.nn.Softmax(dim: Optional[int] = None)

Shape

• Input: (∗) where * means, Number of Dimensions
• Output: (∗), The output has an identical shape as an input.

Parameters

dim (int) – This is the dimension on which softmax function is applied.

Example – Softmax Activation Function

In the below example, we are using softmax activation function along with dim parameter set as ‘1’. Then, input data is produced to get the output.

Which Activation Function to use in Neural Network?

• SigmoidandTanh activation functions should not be used in hidden layers as they can lead to the Vanishing Gradient problem.
• Sigmoidactivation function should be used in the output layer in case of Binary Classification
• ReLU activation functions are ideal for hidden layers of neural networks as they don’t suffer from the Vanishing Gradient problem and help for faster convergence during training.
• Softmax activation function should be used in the output layer for multiclass classification.

• Also Read–PyTorch Tensor – Explained for Beginners
• Also Read – PyTorch Stack vs Cat Explained for Beginners

Conclusion

This tutorial talked about different kinds of activation functions in the PyTorch library. covered syntax of activation functions with examples along with pros and cons of each of them.

Reference –PyTorch Documentation