Learn AI /Neural Networks /Activation function
Activation function
- Deeper
- Neural Networks
- Technical readers
Also called: activation, ReLU, sigmoid, non-linearity.
Back to the milestone you came from
In five seconds
What it is
A neural network usually uses an activation function to transform a unit’s weighted input into an output, often adding the non-linear behaviour needed to learn complex patterns.
In plain words
A neural network unit first adds up evidence into one number, then passes that number through a rule. A non-linear rule might cut off negative values, like ReLU, or squeeze the number into 0 to 1, like sigmoid. It lets many layers model more than a single weighted sum and bias.
An everyday example
Imagine passing music through several devices. If each device only makes the sound twice as loud, chaining them still does nothing more complicated than turn the volume up. A single knob could produce the same result. If some devices instead cut off quiet sounds, compress loud ones or reshape certain frequencies, the chain can produce much richer effects. Linear activations are like stacking volume knobs. Non-linear activations give each stage a rule that can change the shape of the signal, not merely its overall scale.
People at a desk may use judgement, context and explicit criteria. An activation function does none of that. It is only a fixed mathematical transformation applied to a number. Also, a real road screening process may include thresholds, priorities and policy choices, while a network’s activation step is local to one unit and does not by itself rank whole images or explain damage.
Remember thisAn activation function transforms a unit’s weighted input. Non-linear activations let stacked layers represent patterns that a sequence of purely linear transformations cannot.
Understand it
Understand it
Why it matters
If you hear that a model uses ReLU, sigmoid or another activation, that is not a minor implementation detail. It affects what kinds of patterns the network can represent, how stable training may be, how outputs are scaled, and how easily signals pass through many layers. For a non-specialist making decisions about procurement, evaluation or risk, this matters because model behaviour can change for mathematical reasons that are not visible from the user interface. It also helps you avoid confusion: a sigmoid inside a hidden layer is not the same thing as a calibrated probability, and a softmax output layer serves a different purpose from an activation used throughout the network.
From weighted sum to non-linear output
After viewing this, a reader should be able to say what goes into an activation function, what comes out, and why non-linearity matters in a neural network.
01
Combine inputs
Several incoming values are multiplied by weights and added with a bias to form one number, often called z or the pre-activation.
Draw three input arrows labelled x1, x2, x3 entering a summing circle. Mark each arrow with w1, w2, w3. Add a small bias arrow labelled b. Output one box labelled z = w·x + b.
02
Apply a rule
The activation function takes z and transforms it. This is the step that changes the unit from a simple weighted sum into a non-linear component.
From the z box, draw arrows to two side-by-side function boxes labelled ReLU and sigmoid. Add the formulas max(0,z) and 1/(1+e^-z) in small text.
03
See different outputs
For the same input z, different activation functions can produce very different outputs.
Use one example value, such as z = -1.5. Show ReLU output = 0 and sigmoid output ≈ 0.18 in output boxes to the right.
04
Why this matters
Without non-linear activation functions, many stacked layers would behave like one linear transformation and could not represent many useful patterns.
Below the main flow, add a compact comparison: 'linear only' shown as several boxes collapsing into one straight arrow, and 'with non-linearity' shown as layers keeping distinct bends in the path.
Show one unit in a network as a left-to-right flow: inputs and weights combine into a single pre-activation value, then different activation functions transform it into different outputs. Read from left to right, then compare the two example functions at the end.
Hover or tap each function box to swap in a new example z value and update the displayed outputs, while keeping the structure of the flow unchanged.
Describe this without the diagram
The diagram is a left-to-right flow of one neural network unit. First, three inputs labelled x1, x2 and x3 are multiplied by weights w1, w2 and w3 and added with a bias b to produce one number called z, the pre-activation. Second, z is sent into an activation function. Two example functions are shown: ReLU, written as max of 0 and z, and sigmoid, written as 1 divided by 1 plus e to the power minus z. Third, for the same example value z equals minus 1.5, ReLU outputs 0 while sigmoid outputs about 0.18. A final comparison states that stacked linear operations collapse into one linear transformation, while adding non-linear activation functions lets layers represent more complex relationships.
With reduced motion: A static version shows the same four stages without animation: inputs and weights combine into z, z passes through ReLU or sigmoid, example outputs are displayed for one fixed z value, and a small comparison underneath explains linear-only versus non-linear stacks.
Inside a neural network, each unit first combines numbers from earlier units. Those numbers are multiplied by weights, added together, and usually shifted by a bias. That gives one intermediate value, often called the pre-activation.
An activation function then changes that intermediate value into the unit’s output. Common examples are ReLU, which keeps positive values and turns negative ones into zero, and sigmoid, which squeezes any value into a number between 0 and 1. The word activation can mean either the function itself or the output after that function is applied, so the context matters.
Non-linearity explains why commonly used activation functions matter. If every layer only used weighted sums and biases, stacking many layers would collapse into one equivalent weighted sum and bias. The network would remain limited to the relationships that one such operation can represent. A non-linear activation function breaks that limitation.
These words are related but not identical. Activation function is the rule. ReLU and sigmoid are specific kinds of that rule. Activation often names the resulting output value. Non-linearity describes the property that lets networks model more complex relationships than weighted sums can represent.
How “Volume knob versus sound effects” maps onto the real thing
| In the story | In the AI system |
|---|---|
| combined clue score for one image tile | pre-activation: weighted sum plus bias |
| rule that blocks negative scores or rescales scores | activation function such as ReLU or sigmoid |
| passed-on signal to the next desk | activation output sent to the next layer |
| using a rule that is not just a straight rescaling | non-linearity that lets deeper layers model complex patterns |
A worked example
At Headquarters, an information management officer is testing an image-quality model before a batch of publicly issued monitoring reports is uploaded to the portal. The model flags scanned pages that may need rescanning.
Starting from: One hidden unit receives three signals from the previous layer: blur pattern = 1.2, skewed edge = 0.8 and clean text line = 0.4.
- Apply the weights and bias: z = (1.2 × 1.5) + (0.8 × 0.7) + (0.4 × -1.0) + (-0.5) = 1.46.
- With ReLU, the output is max(0, 1.46) = 1.46. The positive signal therefore passes to the next layer unchanged.
- With sigmoid, the same pre-activation produces about 0.81 because sigmoid compresses the value into the range from 0 to 1.
- Later units combine this activation with other signals before the model produces its final rescan flag.
Result: The choice of activation changes how the same weighted input is represented. ReLU preserves its positive magnitude, while sigmoid produces a bounded activation. Neither value is, by itself, the model's final classification.
In a United Nations workplace
At a field presence where the connection is intermittent, the training officer is closing an audit observation on how course feedback is analysed. For a progress report due to the front office by close of business, the officer documents an approved model that runs locally and sorts de-identified, non-sensitive comments into themes. Its hidden units use non-linear activation functions to recognise combinations such as access problems plus requests for translated material, rather than merely adding keyword counts.
The activation function is why stacking the model's layers can add expressive power. If every layer used only a linear activation, the stack would still amount to one affine transformation. Using ReLU, sigmoid or another non-linear function changes the signals passed onwards and the patterns the model can learn.
Before you try thisCourse feedback may contain personal, personnel or disability-related information. Do not put names, personnel records, accommodation requests or identifiable comments into a model that is not approved for such data. A local installation is not approval. Use the authorised workflow, restrict access and review every theme before it enters the progress report.
Where you may hear it
- A vendor demo to an ICT governance board reviewing a proposed model architecture
- A technical review after a model's training results become unstable
- A procurement clarification meeting on how a proposed classifier works
- A handover between a data focal point and developers after outputs collapse into one class
Avoid confusion
Avoid confusion
Often confused with
Softmax
A numerical function that turns a list of model scores into positive values that sum to 1, highlighting which outputs are larger relative to the others.
Probability (in model outputs)
A number attached to a model’s possible outputs that shows how much of its total estimated likelihood the model assigns to each option.
Brain analogy (and its limits)
A comparison between neural networks and brains that can help with intuition, but breaks down because AI systems use simplified maths rather than living cells and biological processes.
Common misconceptions
Common misconception: ReLU, sigmoid and activation all mean the same thing.
Correction: They are related, but they refer to different levels. An activation function is the general idea: a rule applied to a unit’s pre-activation value. ReLU and sigmoid are specific examples of such rules. Activation often refers to the output value after the rule has been applied, not always the rule itself.
Common misconception: The activation function is only for keeping numbers small or tidy.
Correction: Some functions do change the range of values, but the central role is to introduce non-linearity. Without that, stacking many layers would still behave like one linear mapping, which sharply limits what the network can represent.
Common misconception: A sigmoid output is automatically a probability.
Correction: Only in certain settings does a sigmoid output get interpreted as a probability-like score, and even then calibration may be poor. Inside hidden layers, sigmoid is usually just a mathematical transformation. It should not be treated as a guaranteed probability statement.
Common misconception: More layers alone create complexity, even if everything is linear.
Correction: More linear layers do not create the kind of extra expressive power people usually mean. A composition of linear transformations is still linear. The non-linear activation between layers is what prevents the stack from collapsing into a single linear operation.
Quick check
Why do neural networks usually place an activation function after the weighted sum in each unit?
The main reason is non-linearity. Without it, many layers of weighted sums and biases would collapse into one affine transformation. Activations do not guarantee calibrated probabilities, replace weights or biases, or always keep outputs between 0 and 1.
Go deeper
Go deeper
The simple version
A neural network unit first adds up evidence into one number, then passes that number through a rule. A non-linear rule might cut off negative values, like ReLU, or squeeze the number into 0 to 1, like sigmoid. It lets many layers model more than a single weighted sum and bias.
The technical version
For one unit, a pre-activation value is computed as z = w·x + b, where x is the input vector, w the weights and b the bias. The activation function φ then produces a = φ(z). If φ is linear, such as φ(z)=cz, then compositions across layers remain equivalent to a single linear or affine transformation, so depth alone adds little representational power. Choosing a non-linear φ, such as ReLU or sigmoid, allows piecewise-linear or smooth non-linear mappings and therefore more expressive decision boundaries. ReLU is computationally simple and helps preserve positive signals, though it can produce units that stay at zero for many inputs. Sigmoid maps real numbers to the open interval from 0 to 1, which can be useful in some settings but may compress large-magnitude inputs strongly. The term activation may refer either to φ itself or to the resulting value a, so precise writing should distinguish the function from its output. This concept is also distinct from softmax, which is typically applied across multiple output logits to create a normalised distribution over classes.
Reviewed external resources
EncyclopaediaActivation function (opens in a new tab)Wikipedia
This Wikipedia page gives a broad reference-style overview of activation functions and suits readers who want definitions, examples and related terminology.
Intermediatelink checked 2026-07-31
InteractiveTensorFlow Playground (opens in a new tab)TensorFlow
This TensorFlow Playground interactive lets readers change activation functions and observe how decision boundaries change, which suits hands-on learners.
Starterlink checked 2026-07-31
VideoNeural networks, explained visually (opens in a new tab)3Blue1Brown
This 3Blue1Brown video gives a visual intuition for how neural networks transform numbers across layers and suits readers who learn best through animated explanation.
Intermediatelink checked 2026-07-31
VideoVideo explanations: Activation function (opens in a new tab)YouTube search
This YouTube search results page helps readers find multiple video explanations of activation functions from different teachers, so it suits people comparing teaching styles rather than seeking one authoritative source.
search results, not one pagelink checked 2026-07-31
DiscussionCommunity discussion: Activation function (opens in a new tab)Reddit search
This Reddit search results page helps readers browse community questions and practical discussions about activation functions, which suits readers curious about common confusions and informal troubleshooting.
search results, not one pagelink checked 2026-07-31
Where this came from
Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.
The one thing to remember
An activation function transforms a unit’s weighted input. Non-linear activations let stacked layers represent patterns that a sequence of purely linear transformations cannot.