What is Leaky ReLU?

Leaky ReLU is a modified version of ReLU used in neural networks to overcome the problem of “dead neurons”, which occurs when outputs are permanently zero. In Leaky ReLU, small negative values are allowed through (instead of being set exactly to zero) by using a very small slope (such as 0.01).