Building Neural Networks From Scratch

I wanted to understand the calculations behind the TensorFlow functions I’d been using, so I built a small neural network with NumPy. Working through the forward and backward passes gave me a way to connect the equations to running code. Michael Nielsen’s book and 3Blue1Brown’s neural network videos were useful references along the way.

The setup

In this network, each layer takes the previous layer’s outputs, forms weighted sums, and applies an activation function. Before the activation, a single neuron computes:

\[z = \sum_{i=1}^{n} w_i x_i + b\]

I used three layers with sigmoid activations and binary cross entropy loss, keeping the model small enough to work through each calculation.

1
2
3
4
5
6

import numpy as np

class NeuralNetwork:
    def __init__(self, layer_dims: list[int]) -> None:
        self.layer_dims = layer_dims
        self.parameters = self._initialize_parameters()

Initialization

The initial weights affect how signals and gradients travel through the network. If sigmoid inputs become too large in magnitude, the activations saturate and their gradients become small. Xavier initialization helps keep the scale of the activations more consistent across layers:

1
2
3
4
5
6
7
8
9

def _initialize_parameters(self) -> dict[str, np.ndarray]:
    parameters: dict[str, np.ndarray] = {}
    for l in range(1, len(self.layer_dims)):
        parameters[f"W{l}"] = np.random.randn(
            self.layer_dims[l],
            self.layer_dims[l-1]
        ) * np.sqrt(1./self.layer_dims[l])
        parameters[f"b{l}"] = np.zeros((self.layer_dims[l], 1))
    return parameters

Forward pass

The forward pass applies each layer in order. After computing the weighted sum, the sigmoid maps it to a value between 0 and 1:

\[\sigma(z) = \frac{1}{1 + e^{-z}}\]

1
2
3
4
5
6
7
8
9

def forward_propagation(self, X: np.ndarray):
    cache: dict[str, np.ndarray] = {}
    A = X
    for l in range(1, len(self.layer_dims)):
        Z = np.dot(self.parameters[f"W{l}"], A) + self.parameters[f"b{l}"]
        A = 1 / (1 + np.exp(-Z))
        cache[f"A{l}"] = A
        cache[f"Z{l}"] = Z
    return A, cache

I kept the intermediate Z and A values from each layer so the backward pass could reuse them.

Loss

Binary cross entropy measures how well the predicted probabilities match the labels. A confident wrong prediction contributes more to the loss than an uncertain one:

\[E = -\frac{1}{m}\sum_{i=1}^m \left[ y_i \log(a_i) + (1 - y_i) \log(1 - a_i) \right]\]

Tracking this loss during training gives us a way to check whether the updates are improving the predictions.

Backprop

Backpropagation computes the derivative of the loss with respect to each weight. The chain rule lets us break that calculation into the derivatives of the operations along the way:

\[\frac{\partial E}{\partial w_{i}} = \frac{\partial E}{\partial y} \cdot \frac{\partial y}{\partial z} \cdot \frac{\partial z}{\partial w_{i}}\]

We then subtract a fraction of the gradient from each weight, with the learning rate \(\alpha\) setting the step size:

\[w_{i} := w_{i} - \alpha \frac{\partial E}{\partial w_{i}}\]

Each training iteration runs the forward pass, computes the loss, and updates the parameters using the backward pass:

1
2
3
4
5
6
7
8
9

def train(self, X, Y, learning_rate: float = 0.1, epochs: int = 1000):
    losses: list[float] = []
    m = X.shape[1]
    for _ in range(epochs):
        A, cache = self.forward_propagation(X)
        loss = -1/m * np.sum(Y * np.log(A) + (1 - Y) * np.log(1 - A))
        losses.append(float(loss))
        self._backward_propagation(X, Y, cache, learning_rate)
    return losses

With sigmoid outputs and binary cross entropy, the output layer’s derivative simplifies to:

\[\frac{\partial L}{\partial z^{[L]}} = A^{[L]} - Y\]

For a hidden layer, we pass the gradient back through the next layer’s weights and multiply by the activation derivative:

\[\frac{\partial L}{\partial z^{[l]}} = W^{[l+1]T} \frac{\partial L}{\partial z^{[l+1]}} \odot \sigma'(z^{[l]})\]

The implementation walks the layers in reverse:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

def _backward_propagation(self, X, Y, cache, learning_rate):
    m = X.shape[1]
    L = len(self.layer_dims) - 1
    
    dA = -(np.divide(Y, cache[f"A{L}"]) -
           np.divide(1 - Y, 1 - cache[f"A{L}"]))
    
    for l in reversed(range(1, L + 1)):
        Z = cache[f"Z{l}"]
        A_prev = cache[f"A{l-1}"] if l > 1 else X
        
        dZ = dA * (cache[f"A{l}"] * (1 - cache[f"A{l}"]))
        dW = 1/m * np.dot(dZ, A_prev.T)
        db = 1/m * np.sum(dZ, axis=1, keepdims=True)
        
        self.parameters[f"W{l}"] -= learning_rate * dW
        self.parameters[f"b{l}"] -= learning_rate * db
        
        if l > 1:
            dA = np.dot(self.parameters[f"W{l}"].T, dZ)

The sigmoid derivative is \(\sigma'(z) = \sigma(z)(1 - \sigma(z))\), so we can calculate it from the activations saved during the forward pass.

Back to TensorFlow

Writing out both passes made it easier to reason about the TensorFlow models I was already using. When a network failed to learn, I had a clearer idea of how initialization, gradient scale, and the learning rate could affect it.

The implementation took more work than calling a library, but that was useful for this exercise. I could follow a prediction all the way from the input to the loss and then back through the weight updates.