More from the Deep Learning Glossary…

Activation function

A function that enables neural networks to learn nonlinear (complex) relationships between features and the label.

Popular activation functions include:

The plots of activation functions are never single straight lines. For example, the plot of the ReLU activation function consists of two straight lines:

A plot of the sigmoid function looks as follows:

In a neural network, activation functions manipulate the weighted sum of all the inputs to a neuron. To calculate a weighted sum, the neuron adds up the products of the relevant values and weights. For example, suppose the relevant inputs to a neuron consist of the following:

So the weighted sum is:

weighted sum = (2)(-1.3) + (-1)(0.6) + (3)(0.4) = -2.0

Suppose the designer of this neural network chose the sigmoid function as the activation function. In that case, the neuron calculates the sigmoid of -2.0, which is approximately 0.12. So the neuron passes 0.12 (rather than -2.0) to the next layer in the neural network. The following figure illustrates the relevant part of the process:

Book link

Browse the book

Cite this book

Taima, Alaa. (2022). Activation function. Deep Learning in Arabic. https://dlarabic.com/en/articles/activation-function/

Similar books you may like

Leave a comment

Your email address will not be published. Required fields are marked *