In the family vacation example in [1], logistic regression feels natural. You gather signals like budget, weather, and school schedule, combine them into one score, and then turn that score into a clean yes or no decision. That final step is done by the sigmoid function, which squeezes any number into a probability between 0 and 1. It is perfect when your goal is a single, interpretable binary decision.
But once we move beyond one decision and start building neural networks, we face a new problem. A neural network is made of many layers, and each layer begins by computing a weighted sum of its inputs, just like logistic regression. If we keep stacking only these weighted sums, the entire network still behaves like one big linear model. No matter how many layers you add, it cannot represent truly complex patterns. To break that limitation, every layer needs a non-linear step after the weighted sum. That step is called an activation function, and it is what gives neural networks their real expressive power.
ReLU: The Game Changer
One of the most important activation functions is Rectified Linear Unit (ReLU). ReLU is simple: if the score is negative, it outputs zero; if the score is positive, it passes it through as is. You can think of it like a practical family rule during planning: if an idea clearly makes things worse, stop discussing it, but if it helps, keep it and build on it. This small rule turns out to be incredibly powerful.
Why ReLU Became Popular
ReLU became popular for two main reasons. First, it is hardware-friendly. Sigmoid needs expensive operations like exponentials and division, while ReLU is basically just a comparison and a selection. That makes it faster, cheaper, and easier to implement efficiently on real devices. Second, it helps training. Neural networks learn through gradients, and sigmoid often produces very small gradients when the input score is too large or too small, which can slow learning in deep models. ReLU avoids that problem on the positive side, allowing learning signals to flow more strongly through many layers.
ReLU's simplicity is its strength. By avoiding expensive computations and maintaining strong gradients, it makes training deep networks both practical and efficient.
The Limitations and Evolution
ReLU is not perfect though. If a neuron keeps seeing negative scores, it may output zero repeatedly and stop learning altogether, a problem sometimes called a "dead neuron". That is why variants exist, such as the Gaussian Error Linear Unit (GELU), which replaces ReLU's hard cutoff with a more gradual gating behavior. Instead of killing all negative values, GELU softly reduces them based on their magnitude, often written as GELU(z) = z*Φ(z) where Φ(z) is the standard normal CDF. In practice, this smoothness can improve training stability in some deep models, which is why GELU is commonly used in Transformer-based architectures.
The Right Tool for the Right Job
ReLU/GELU is ideal inside the hidden layers where the model is still forming its internal understanding. It helps the network decide what information is worth carrying forward, layer by layer, until that final decision becomes easy. Logistic regression makes the final call.
Understanding activation functions is not just about knowing the math. It is about appreciating how simple design choices can unlock powerful capabilities. ReLU's elegance lies in its simplicity, proving that sometimes the most practical solutions are also the most beautiful.