Swish is a non-linear activation function developed by researchers at Google, and it is smoother than ReLU. It passes positive values through almost linearly while letting part of the negative values through instead of discarding them entirely, which helps the network learn more deeply and perform better on some tasks. It is often used in deep neural networks because it can improve training speed and accuracy compared with ReLU.
