Linear Algebra in ML

I started taking linear algebra more seriously when I began using it in machine learning. Matrix multiplication, gradient updates, and embeddings kept bringing me back to ideas from the course. These are a few of the connections that helped me understand what the models were doing.

Vectors and matrices, everywhere

We can represent a data point with \(n\) numerical features as a vector in \(\mathbb{R}^n\). A matrix \(A\) then gives us a linear transformation of that vector:

\[f(x) = Ax\]

Dense neural network layers use this operation, usually with a bias term and a nonlinear activation. Stacking those layers lets the network represent functions that a single linear transformation cannot.

1
2
3
4

import numpy as np

def linear_transform(X: np.ndarray, W: np.ndarray, b: np.ndarray) -> np.ndarray:
    return np.dot(X, W) + b

Writing the operation as \(Wx + b\) also makes the shapes explicit: the number of columns in \(W\) has to match the number of features in \(x\).

Where it keeps showing up

PCA

Principal component analysis finds directions along which the data varies most. For a centered data matrix \(X\), with one sample per row, we can form the covariance matrix using the normalization below:

\[C = \frac{1}{n} X^T X\]

then solve the eigenvalue problem:

\[Cv = \lambda v\]

The eigenvectors with the largest eigenvalues give the directions of greatest variance. Projecting onto a few of them reduces the number of dimensions while retaining as much variance as possible for that number of components.

1
2
3
4
5
6
7
8

def compute_pca(X: np.ndarray, n_components: int) -> tuple[np.ndarray, np.ndarray]:
    X_centered = X - np.mean(X, axis=0)
    cov_matrix = np.dot(X_centered.T, X_centered) / X.shape[0]
    eigenvals, eigenvecs = np.linalg.eigh(cov_matrix)
    idx = np.argsort(eigenvals)[::-1]
    eigenvals = eigenvals[idx][:n_components]
    eigenvecs = eigenvecs[:, idx][:, :n_components]
    return eigenvals, eigenvecs

Neural network layers

A fully connected layer combines a matrix multiplication, a bias, and an activation function:

\[h = \sigma(Wx + b)\]

The weights determine how the input features are combined, and training adjusts them to reduce the loss.

1
2

def neural_network_layer(X, W, b, activation):
    return activation(linear_transform(X, W, b))

Gradient descent

In gradient descent, we update the weight matrix using the gradient of the loss:

\[W_{t+1} = W_{t} - \alpha \frac{\partial L}{\partial W_{t}}\]

The gradient \(\partial L / \partial W\) has the same shape as \(W\). Each entry tells us how the loss changes with the corresponding weight, and \(\alpha\) controls the size of the update.

SVD

Singular value decomposition factorizes a matrix as:

\[A = U\Sigma V^T\]

SVD is useful for low-rank approximations, recommendation systems, and model compression. Truncated SVD keeps only the largest \(k\) singular values and their associated vectors:

1
2
3

def truncated_svd(X: np.ndarray, k: int):
    U, s, Vt = np.linalg.svd(X, full_matrices=False)
    return U[:, :k], s[:k], Vt[:k, :]

Why the math view is worth keeping

The matrix view is useful when I’m trying to understand a training problem. I can check whether the dimensions line up, whether values are growing too large through repeated transformations, or whether some features are redundant. Those checks give me specific things to investigate when a loss curve looks wrong.

I don’t need to work through a derivation every time I train a model, but being able to connect the code to these operations makes it easier to see what to check next.