Face Recognition with ResNet

I built a small face recognition system to understand how ResNet could be used to compare faces. The project involved finding and aligning faces as well as training the network, and masks became another problem to work on during COVID. The code is on GitHub.

Why face recognition

The same person can look quite different under different lighting or from a different angle. A recognition system needs to handle those changes while still distinguishing between people. I wanted to understand how the features learned by a convolutional network could help with that.

Convolutions are simpler than they look

We can represent a color image as an array with height, width, and RGB channels. A convolution moves a filter across that array and computes a weighted sum at each position:

\[(\mathbf{X} * \mathbf{K})(i,j) = \sum_{p=0}^{k_{h}-1} \sum_{q=0}^{k_{w}-1} \sum_{c=0}^{C-1} \mathbf{X}(i+p,j+q,c) \cdot \mathbf{K}(p,q,c)\]

The filter uses the same weights at each position. Early layers often learn features such as edges and textures, which later layers combine into more detailed representations.

ReLU

After each convolution we apply ReLU:

\[\text{ReLU}(z) = \max(0, z)\]

ReLU sets negative values to zero and leaves positive values unchanged. This gives the network a nonlinearity; composing only linear operations would still produce a linear operation, however many layers we added.

Multiple filters per layer

A single layer typically has many filters, each one looking for a different feature:

\[\mathbf{Y} = f(\mathbf{X} * \mathbf{K}_1, \mathbf{X} * \mathbf{K}_2, \ldots, \mathbf{X} * \mathbf{K}_N)\]

Each filter produces a feature map, and the next layer receives those maps as its input channels.

The ResNet trick

Adding layers to a plain network can make optimization harder, even when the deeper network should be able to represent the same function as the shallow one. ResNet introduces skip connections so a block can learn a change to its input:

\[\mathbf{X} = \mathbf{Z} + F(\mathbf{Z})\]

The block \(F\) applies its convolutions and batch normalization, and its output is added to the original input \(\mathbf{Z}\). This gives information and gradients a direct path through the block, which helps when training deeper networks.

A typical ResNet block looks like:

\[F(\mathbf{X}) = W_2 \sigma(\text{BN}(W_1 \mathbf{X}))\]

\[\mathbf{X}_{\text{out}} = \mathbf{X} + F(\mathbf{X})\]

The dimensions need to match for this addition. When they change between blocks, the skip path also needs a projection to the new shape.

Building a face recognition system

Finding and aligning faces

The system first detects faces in an image, then crops and aligns them. Detectors can use methods such as Haar cascades or convolutional networks. Alignment puts landmarks such as the eyes and nose in roughly consistent positions, reducing the variation that the recognition model has to handle.

Faces to embeddings

A network can represent a face as an embedding: a vector of features from one of its deeper layers. The aim is for different images of the same person to produce nearby vectors, while images of different people are farther apart.

Recognition

For identification among a fixed set of people, a softmax output assigns a probability to each class:

\[\hat{y} = \text{softmax}(\mathbf{z}) = \frac{e^{\mathbf{z}}}{\sum_{j=1}^K e^{z_{j}}}\]

We can train that classifier with cross entropy:

\[\mathcal{L} = -\sum_{i=1}^K y_{i} \log(\hat{y}_{i})\]

Verification asks whether two images show the same person, including people outside the training set. One approach compares their embeddings with cosine similarity or L2 distance and uses a threshold to decide whether they match.

Implementation

TensorFlow includes ResNet50, but I wrote a smaller version so I could follow the residual connections through the code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

import tensorflow as tf
from tensorflow.keras import layers, models, regularizers

def residual_block(x: tf.Tensor, filters: int, stride: int = 1) -> tf.Tensor:
    shortcut = x
    
    x = layers.Conv2D(filters, kernel_size=3, strides=stride, padding='same',
                      kernel_regularizer=regularizers.l2(1e-4))(x)
    x = layers.BatchNormalization()(x)
    x = layers.Activation('relu')(x)
    
    x = layers.Conv2D(filters, kernel_size=3, strides=1, padding='same',
                      kernel_regularizer=regularizers.l2(1e-4))(x)
    x = layers.BatchNormalization()(x)
    
    if stride != 1 or shortcut.shape[-1] != filters:
        shortcut = layers.Conv2D(filters, kernel_size=1, strides=stride,
                                 padding='same')(shortcut)
        shortcut = layers.BatchNormalization()(shortcut)
    
    x = layers.Add()([x, shortcut])
    x = layers.Activation('relu')(x)
    return x

def build_resnet(input_shape: tuple = (64, 64, 3),
                 num_classes: int = 16) -> tf.keras.Model:
    inputs = layers.Input(shape=input_shape)
    
    x = layers.Conv2D(16, kernel_size=3, strides=1, padding='same')(inputs)
    x = layers.BatchNormalization()(x)
    x = layers.Activation('relu')(x)
    
    x = residual_block(x, filters=16, stride=1)
    x = residual_block(x, filters=16, stride=1)
    x = residual_block(x, filters=32, stride=2)
    x = residual_block(x, filters=32, stride=1)
    
    x = layers.GlobalAveragePooling2D()(x)
    outputs = layers.Dense(num_classes, activation='softmax')(x)
    
    model = models.Model(inputs=inputs, outputs=outputs)
    model.compile(
        optimizer=tf.keras.optimizers.Adam(1e-3),
        loss='categorical_crossentropy',
        metrics=['accuracy'],
    )
    return model

The skip connection is in residual_block. The build_resnet function stacks these blocks after an initial convolution and uses global average pooling before the classifier. This is a small model for the project, with much less capacity than ResNet50.

Masks broke everything

Masks exposed a weakness in the model: it had learned from full faces and struggled when the mouth and chin were covered. I tried including masked faces in the training data and encouraging the model to use the visible regions around the eyes and forehead. Data augmentation worked better in my experiments.

Evaluating the whole system

Working on the whole system helped me understand how the parts fit together. The detector and alignment step prepare the input, the network extracts features, and the comparison method turns those features into a decision.

The masked-face experiments also made evaluation more concrete. A model that performs well on its usual inputs can struggle with a change that seems small to us, so the data and test conditions need as much attention as the architecture.