Linear Algebra in ML

, ,

I used to think linear algebra was just another math class I had to get through. Then I started learning machine learning and realized it’s basically everywhere. All those matrix operations and vector spaces? They’re what actually run the algorithms behind modern AI.

Everything Is Vectors and Matrices

Machine learning is really just doing math on data. That data gets represented as vectors and matrices. Training a model? You’re doing a bunch of linear transformations on these mathematical objects.

Vector Spaces and Transformations

Say you have a dataset with $n$ features. Each data point lives in an $n$ dimensional vector space $\mathbb{R}^n$. When you apply a linear transformation, you multiply by a matrix $A$:

$$f(x) = Ax$$

This simple operation is what happens in every layer of a neural network. Just add a nonlinear activation function on top and you’ve got deep learning.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
import numpy as np

def linear_transform(X: np.ndarray, W: np.ndarray, b: np.ndarray) -> np.ndarray:
    """
    Applies a linear transformation to input data.
    
    Args:
        X: Input data matrix of shape (n_samples, n_features)
        W: Weight matrix of shape (n_features, n_output)
        b: Bias vector of shape (n_output,)
    
    Returns:
        Transformed data of shape (n_samples, n_output)
    """
    return np.dot(X, W) + b

Matrix Operations in Action

Here’s how matrix operations show up in some fundamental ML techniques.

Principal Component Analysis

PCA uses eigendecomposition to find the directions where your data varies the most. You compute the covariance matrix:

$$C = \frac{1}{n}X^TX$$

Then solve the eigenvalue equation:

$$Cv = \lambda v$$

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
def compute_pca(X: np.ndarray, n_components: int) -> tuple[np.ndarray, np.ndarray]:
    """
    Performs PCA dimensionality reduction.
    
    Args:
        X: Data matrix of shape (n_samples, n_features)
        n_components: Number of components to keep
        
    Returns:
        eigenvalues, eigenvectors of the covariance matrix
    """
    # Center the data
    X_centered = X - np.mean(X, axis=0)
    
    # Compute covariance matrix
    cov_matrix = np.dot(X_centered.T, X_centered) / X.shape[0]
    
    # Compute eigendecomposition
    eigenvals, eigenvecs = np.linalg.eigh(cov_matrix)
    
    # Sort in descending order
    idx = np.argsort(eigenvals)[::-1]
    eigenvals = eigenvals[idx][:n_components]
    eigenvecs = eigenvecs[:, idx][:, :n_components]
    
    return eigenvals, eigenvecs

Neural Networks

Deep learning networks are chains of linear transformations with nonlinear functions in between. Each layer does:

$$h = \sigma(Wx + b)$$

where $\sigma$ is the activation function, $W$ is the weight matrix, and $b$ is the bias vector.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
def neural_network_layer(
    X: np.ndarray,
    W: np.ndarray,
    b: np.ndarray,
    activation: callable
) -> np.ndarray:
    """
    Implements a single neural network layer.
    
    Args:
        X: Input data
        W: Weight matrix
        b: Bias vector
        activation: Activation function
        
    Returns:
        Layer output after activation
    """
    Z = linear_transform(X, W, b)
    return activation(Z)

Training

The training process also relies on linear algebra. Gradient descent computes partial derivatives with respect to matrices:

$$W_{t+1} = W_t - \alpha \frac{\partial L}{\partial W_t}$$

where $L$ is the loss function and $\alpha$ is the learning rate.

SVD and Other Stuff

Singular Value Decomposition is important for things like recommendation systems. SVD factorizes a matrix $A$ as:

$$A = U\Sigma V^T$$

where $U$ and $V$ are orthogonal matrices, and $\Sigma$ contains the singular values.

1
2
3
4
5
6
7
8
9
10
11
12
13
def truncated_svd(X: np.ndarray, k: int) -> tuple[np.ndarray, np.ndarray, np.ndarray]:
    """
    Performs truncated SVD for dimensionality reduction.
    
    Args:
        X: Input matrix
        k: Number of singular values to keep
        
    Returns:
        U, Sigma, V matrices of the truncated SVD
    """
    U, s, Vt = np.linalg.svd(X, full_matrices=False)
    return U[:, :k], s[:k], Vt[:k, :]

Wrapping Up

Linear algebra lets you express complicated transformations with simple matrix operations. I’ve been working with ML for a while now and I still find it pretty cool how these mathematical foundations make everything work. When you call model.fit() or torch.matmul(), there’s a lot of beautiful math happening under the hood.