What is ReLU?

ReLU (Rectified Linear Unit) is a very common activation function in neural networks: it outputs the value unchanged if it is positive and zero if it is negative. It is simple and fast to compute, speeds up training compared with older functions such as Sigmoid and Tanh, and reduces the vanishing gradient problem, but it can cause the “dying neurons” problem, when some units become permanently inactive.