A function that enables neural networks to learn nonlinear (complex) relationships between features and the label.
Popular activation functions include:
The plots of activation functions are never single straight lines. For example, the plot of the ReLU activation function consists of two straight lines:
A plot of the sigmoid function looks as follows:
In a neural network, activation functions manipulate the weighted sum of all the inputs to a neuron. To calculate a weighted sum, the neuron adds up the products of the relevant values and weights. For example, suppose the relevant inputs to a neuron consist of the following:

So the weighted sum is:
weighted sum = (2)(-1.3) + (-1)(0.6) + (3)(0.4) = -2.0
Suppose the designer of this neural network chose the sigmoid function as the activation function. In that case, the neuron calculates the sigmoid of -2.0, which is approximately 0.12. So the neuron passes 0.12 (rather than -2.0) to the next layer in the neural network. The following figure illustrates the relevant part of the process:

Browse the book
Cite this book
Taima, Alaa. (2022). Activation function. Deep Learning in Arabic. https://dlarabic.com/en/articles/activation-function/
@book{taimaa2022_1480,
title = {Activation function},
author = {Taima, Alaa},
year = {2022},
publisher = {Deep Learning in Arabic},
url = {https://dlarabic.com/en/articles/activation-function/},
language = {arabic}
}


