Two Phase Commit Protocol in Rust and Go

A friend and I got talking about what happens when an online payment fails partway through an order. We wanted to understand how separate services could agree on whether a transaction had completed, so we built a small two-phase commit system. We wrote the coordinator in Rust and the services in Go.

The basic idea

Two-phase commit (2PC) coordinates a transaction across several participants. The coordinator first asks whether each participant is prepared to commit. It can choose to commit only after every participant has agreed; otherwise it aborts the transaction.

What we built

Our system had three parts: a Rust coordinator, a Go wallet service that held balances, and a Go order service that managed inventory. Charging a user and reserving an item needed to succeed together or both be rolled back.

The coordinator

The coordinator sends prepare requests, collects the responses, and then sends its decision to the participants. The main control flow looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

struct Coordinator {
    wallet_conn: TcpStream,
    order_conn: TcpStream,
}

impl Coordinator {
    fn prepare_phase(&mut self, transaction: Transaction) -> Result<bool, Error> {
        self.wallet_conn.write_all(&transaction.serialize())?;
        self.order_conn.write_all(&transaction.serialize())?;
        
        let wallet_vote = self.wallet_conn.read_response()?;
        let order_vote = self.order_conn.read_response()?;
        
        Ok(wallet_vote == READY && order_vote == READY)
    }
    
    fn commit_phase(&mut self) -> Result<(), Error> {
        self.wallet_conn.write_all(COMMIT_MSG)?;
        self.order_conn.write_all(COMMIT_MSG)?;
        Ok(())
    }
}

During prepare, a refusal or timeout leads the coordinator to abort. If all participants agree, it can record a commit decision and tell them to apply it. Once that decision has been made, recovery needs to preserve it even if a participant is temporarily unreachable.

The microservices

Each service prepares its own part of the transaction. This is the wallet service’s prepare handler:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

type WalletService struct {
    db *sql.DB
}

func (ws *WalletService) handlePrepare(tx *sql.Tx, userId int, amount float64) error {
    var balance float64
    err := tx.QueryRow("SELECT balance FROM wallets WHERE user_id = ?", userId).Scan(&balance)
    if err != nil {
        return err
    }
    
    if balance < amount {
        return errors.New("insufficient funds")
    }
    
    _, err = tx.Exec("UPDATE wallets SET balance = balance - ? WHERE user_id = ?", amount, userId)
    return err
}

The prepare handler holds the local transaction open and locks the relevant rows while it waits for the coordinator. It must be able to carry out the eventual commit or abort decision, including after a failure.

What goes wrong, and there’s a lot

We spent much of our time working through failures at different points in the protocol:

Our tests covered nodes crashing during a transaction, dropped packets, and services responding too slowly. The point at which a failure happened often mattered as much as the failure itself.

The downsides of 2PC

A prepared participant may need to keep locks while waiting for the coordinator’s decision, so a coordinator failure can block other work. Recovery therefore needs durable transaction state and a way to recover the decision. Replicating that state with a consensus protocol can help, while approaches such as sagas use a different model of local transactions and compensating actions.

Running it on real infrastructure

We deployed the services on separate Google Cloud VMs to see how the protocol behaved outside localhost.

The prepare phase went from microseconds locally to milliseconds across regions, with some responses taking much longer than the median. That made timeout choices harder: a missing response could mean a failed service, a lost packet, or a service that was still working.

Testing was the hard part

A successful request from one service’s point of view could look like a timeout from another’s. We needed tests that controlled the order of events and introduced failures between protocol steps:

1
2
3
4
5
6
7
8
9
10
11
12

#[test]
fn test_node_failure_during_prepare() {
    let mut coordinator = Coordinator::new();
    let transaction = Transaction::new(user_id: 1, amount: 100.0);
    
    coordinator.order_conn.shutdown()?;
    
    assert!(matches!(
        coordinator.prepare_phase(transaction),
        Err(Error::Timeout)
    ));
}

Fault injection let us check whether participants reached the same decision after crashes and delayed responses, rather than only checking the successful path.

What we got out of it

Rust’s ownership model helped us keep track of the coordinator’s state, and the compiler caught mistakes before we had to diagnose them in a running system. Go’s goroutines were convenient for handling concurrent transactions in the services.

Running the system across VMs made the recovery requirements easier to understand. A timeout didn’t tell us what had happened to a transaction, so we had to rely on recorded state and explicit decisions instead of assumptions about response times.

The code is on GitHub. The README is in Norwegian because this was originally a class project.