Backpropagation can sound like one of the most intimidating parts of artificial intelligence. In practice, the idea is much simpler: a neural network makes a prediction, measures how wrong it was, calculates which weights contributed to that error, and adjusts those weights slightly. This worked example follows that entire process using three house-price predictions and two hidden neurons.
Backpropagation in One Sentence
A neural network learns by repeatedly answering three questions: What did I predict? How wrong was I? Which numbers inside the network should change?
The maths below is deliberately small enough to calculate by hand. Real neural networks can contain millions or billions of parameters, but the principle is the same.
The Worked Example: Three Houses and Two Hidden Neurons
Imagine a tiny neural network trying to predict house prices. Each house has three input features: the number of bedrooms, the floor area in square feet and a simplified location score.
Two bedrooms, 800 square feet and a location score of 5.
Three bedrooms, 1,200 square feet and a location score of 8.
One bedroom, 500 square feet and a location score of 3.
These are the real prices the network is trying to learn.
Each row is one house. Each column is one feature. This is the data the network receives.
The first layer weights and biases
The first hidden layer contains two neurons. Each neuron has a weight for every input feature. The weights decide how strongly each input contributes to that neuron’s calculation.
The bias is an extra adjustable starting value. It allows a neuron to shift its output independently of the input values.
The output layer
The two hidden-neuron outputs are then combined into one final house-price prediction.
A Quick Cheat Sheet Before We Calculate Anything
Step 1: The Forward Pass
The forward pass is simply the network using its current weights to make predictions. No learning happens yet. We are only moving information from the inputs towards the output.
Calculate the hidden-layer totals
Multiply every feature by its corresponding weight, add those products together, then add the neuron’s bias.
House 1, neuron 1:
(2 × 5,000) + (800 × 200) + (5 × 10,000) + 1,000 = 10,000 + 160,000 + 50,000 + 1,000 = 221,000.
House 1, neuron 2:
(2 × 10,000) + (800 × 150) + (5 × 20,000) + 2,000 = 20,000 + 120,000 + 100,000 + 2,000 = 242,000.
Repeating the same process for the other two houses gives:
Apply the ReLU activation function
We now apply ReLU. ReLU is one of the simplest activation functions: negative values become zero; positive values remain unchanged.
Every value in our hidden layer is already positive, so nothing changes:
The hidden neurons have converted the original house features into two new learned numerical representations. In this toy example those values are very large, but they are simply the intermediate information passed to the output layer.
Calculate the predicted house prices
House 1: (221,000 × 0.5) + (242,000 × 0.5) + 5,000 = 236,500.
House 2: (336,000 × 0.5) + (372,000 × 0.5) + 5,000 = 359,000.
House 3: (136,000 × 0.5) + (147,000 × 0.5) + 5,000 = 146,500.
The network over-predicts by £16,500.
The network under-predicts by £16,000.
The network over-predicts by £11,500.
Backpropagation tells us which weights should move and in which direction.
Step 2: Measure the Error with Mean Squared Error
A training algorithm needs one number that represents how well or badly the network performed. For this example we use Mean Squared Error, usually shortened to MSE.
Subtract the real value from each prediction, square each difference so negative and positive errors cannot cancel each other out, add them together, then divide by the number of examples.
The squared errors are:
- House 1: (236,500 − 220,000)² = 272,250,000
- House 2: (359,000 − 375,000)² = 256,000,000
- House 3: (146,500 − 135,000)² = 132,250,000
That number looks enormous because house prices are large values and MSE squares the errors. What matters during training is whether the loss moves down over repeated updates.
Step 3: Backpropagation — Work Backwards from the Error
We now know the network is wrong. Backpropagation asks a more useful question: how much did each adjustable parameter contribute to that error?
Calculus answers that using derivatives. A derivative measures how sensitive one quantity is to a small change in another. In neural networks, we call these derivatives gradients.
A gradient is a direction sign. Positive means increasing that parameter would increase the loss locally. Negative means increasing it would decrease the loss locally. Gradient descent then moves in the opposite direction.
First calculate the raw prediction errors
The factor-of-two trap in MSE
This is one of the easiest places to make a mistake when learning backpropagation. If our loss is exactly:
then differentiating the square produces a factor of 2:
Because n = 3, the error vector is multiplied by 2/3.
Some textbooks deliberately define the loss as (1 / 2n) × Σ(ŷ − y)². In that version the 2 cancels during differentiation, leaving (1/n) × (ŷ − y). Both conventions are valid, but the loss definition and the gradient formula must match.
If you write MSE as 1/n times the squared error, its derivative contains 2/n. If you want a 1/n gradient, define the loss as half-MSE: 1/(2n) times the squared error.
Calculate the gradient flowing out of the loss
Step 4: Calculate the Gradient for the Output Weights
Each output weight multiplied one of the hidden activations. Therefore the gradient for W₂ depends on both the prediction-error gradient and the values in a₁.
Transposing a₁ turns its two hidden-neuron columns into rows so the matrix dimensions line up with the three-house error vector.
Multiplying that matrix by the prediction gradient gives:
The gradient for the output bias is simpler because the bias contributes directly to every prediction:
What Do Negative Gradients Actually Mean?
Both output-weight gradients are negative. That does not mean the weights themselves should become negative. It means that, at the current point in the loss landscape, increasing those weights slightly would reduce the loss.
Gradient descent uses the following rule:
Because the gradients are negative, subtracting them increases the two output weights.
Step 5: Update the Weights with a Learning Rate
Our gradients are extremely large because the example uses raw square-foot values, large first-layer weights and hidden activations in the hundreds of thousands. We therefore need a very small learning rate for this hand-worked example.
Suppose we use:
The first output weight becomes:
The second becomes:
The bias changes by only a tiny amount at this learning rate:
Did the network actually improve?
Yes — slightly. Using only this one tiny output-layer update, the predictions become approximately:
- House 1: £236,567.70
- House 2: £359,103.66
- House 3: £146,541.32
The new MSE is approximately 220,128,130.93, down from 220,166,666.67.
The network made a prediction, converted the errors into gradients, nudged its weights in the direction that should reduce the loss, and produced a slightly lower error on the next forward pass. That cycle is the core of neural-network training.
Why Are the Gradients So Huge?
This worked example is intentionally easy to calculate by hand, but it also demonstrates an important practical issue: scale matters.
Square footage is measured in hundreds or thousands. The first-layer weights are also large. That produces hidden activations above 100,000. When those large activations are multiplied by prediction errors during backpropagation, the resulting gradients become enormous.
Floor-area values are much larger than bedroom counts or small location scores.
The first layer magnifies already unbalanced input scales.
These values feed directly into the output-weight gradient.
Modern ML pipelines commonly standardise or normalise features before training.
Feature scaling does not change the fundamental idea of backpropagation. It simply makes the optimisation problem easier to manage numerically.
Where the First Layer Fits In
So far we have calculated gradients only for the output layer. A full training step continues backwards.
The output error is propagated through W₂, then through the derivative of ReLU, and finally into the gradients for W₁ and b₁. Because all of the hidden values in this example are positive, the ReLU derivative is 1 for every hidden unit.
That is why the method is called backpropagation: the learning signal moves backwards through the same computational path used during the forward pass.
The Bigger Lesson: Neural Networks Learn by Credit Assignment
The most useful way to think about backpropagation is not as a collection of formulas. It is a system for credit assignment.
When the network makes a bad prediction, many different parameters may have contributed. Backpropagation calculates how sensitive the final loss is to each one. Gradient descent then uses those sensitivities to decide how to move the parameters.
The forward pass produces an output from the current parameters.
The loss function turns prediction quality into a numerical training signal.
Gradients distribute responsibility through the network so its parameters can be updated.
Key Takeaways
- A forward pass uses the current weights to produce predictions.
- MSE measures the average squared difference between predictions and real values.
- If MSE is defined as (1/n) × Σ(ŷ − y)², its prediction gradient contains the factor 2/n.
- The 1/n gradient commonly seen in examples corresponds to half-MSE: (1/2n) × Σ(ŷ − y)².
- Backpropagation applies the chain rule to determine how each parameter affected the loss.
- Gradient descent updates parameters in the opposite direction to the gradient.
- Large raw feature scales can create large gradients, which is one reason feature scaling matters in practical machine learning.
Neural-network learning is not magic. It is repeated measurement and adjustment: predict, measure the error, calculate responsibility, update the parameters, and try again.
What Comes Next?
The next natural step is to continue the same calculation through the first layer: propagate the output error through W₂, apply the ReLU derivative, calculate the gradients for W₁ and b₁, and then update the entire network.
Once that process is clear, the same logic scales from this two-neuron example to much larger architectures. Deep learning simply repeats the same chain-rule logic through many more layers.
