Published on

A Look Back at WeChat's PhxSQL and the 'Fastest Majority'

Authors

It's 2016, and I'm working at WeChat (yes, that WeChat with over a billion users). Every day, servers are dying left and right – not because our hardware is garbage, but because when you have THAT many servers, even a 0.001% failure rate means something breaks every few minutes. It's like having a million light bulbs in your house; statistically, one of them is going to burn out while you're eating breakfast.

Fast forward to 2025, and I'm thinking about building a highly available secrets manager system. Suddenly, memories of PhxSQL come flooding back – that brilliant piece of engineering that WeChat open-sourced nine years ago, which somehow never got the attention it deserved. Maybe it's because not many companies have WeChat's "billion users with payment systems" problems. Most folks are still figuring out how to handle their first million users without their database catching fire.

Disclaimer: This article reflects my personal views and experiences. The content is not related to my previous or current employers.

The "Cross-Region Latency Curse" – Every DBA's Nightmare

Let me paint you a picture of distributed database hell. You've got your fancy "two regions, three centers" setup – maybe 2 nodes in Guangzhou, 1 node in Shanghai. You feel smart, you feel prepared for disasters. Then reality hits.

All consensus algorithms (Paxos, Raft, your uncle's homemade distributed protocol) work like a committee meeting where everyone needs to agree before anything gets done. In a cross-region setup, this means waiting for that one slow committee member in Shanghai to raise their hand and say "I agree" – over a network connection that takes 30+ milliseconds round trip.

It's like trying to have a conversation where every sentence takes 30 seconds to travel. By the time Shanghai says "yes," Guangzhou has already forgotten what they were talking about.

PhxSQL's Brilliant Solution: The "Fastest Majority" Hack

Here's where PhxSQL gets clever. Instead of waiting for the slowest kid in class, it just waits for the fastest majority. Let me break down how this works in our "Guangzhou 2 + Shanghai 1" setup:

The Magic Behind the Scenes

When you have 3 nodes, you need 2 votes to make any decision (that's your majority). Now, here's the genius part:

  1. The Leader is Usually Local: The master node will almost always be one of the Guangzhou nodes because they have better network connectivity to each other.
  2. The "Committee Meeting" Process: When Master Node A in Guangzhou wants to write something:
  • It sends the write request to both Guangzhou Node B and Shanghai Node C
  • Guangzhou Node B responds in ~2ms (same city, same coffee shop WiFi speed)
  • Shanghai Node C takes ~30ms to respond (cross-region, probably stuck in traffic)
  1. The "Don't Wait for the Slow Guy" Philosophy: As soon as Master Node A gets confirmation from Guangzhou Node B, it has 2 votes (itself + Node B). That's a majority! It can immediately:
  • Commit the transaction locally
  • Tell the client "success!"
  • Not care that Shanghai Node C is still processing the request

It's like starting the movie as soon as 2 out of 3 friends arrive, instead of waiting for the one who's "5 minutes away" for the past hour.

My Go + Raft Implementation: PhxSQL's Soul in Modern Clothes

While thinking about this secrets manager project, I decided to implement PhxSQL's core philosophy using Go, Raft, and SQLite. Here's the heart of the "Fastest Majority" implementation:

func (n *RaftNode) replicateAndWaitForMajority(entry LogEntry) bool {
    majorityNeeded := (len(peers) + 1) / 2
    
    var successCount struct {
        sync.Mutex
        count int
    }
    successCount.count = 1 // Count ourselves
    
    doneChan := make(chan bool, 1)
    
    // Send to all peers concurrently
    for _, peer := range peers {
        go func(peer string) {
            if n.sendAppendRequest(peer, req) {
                successCount.Lock()
                successCount.count++
                isMajority := successCount.count > majorityNeeded
                successCount.Unlock()
                
                if isMajority {
                    select {
                    case doneChan <- true: // We got majority!
                    default:
                    }
                }
            }
        }(peer)
    }
    
    // Wait for majority OR timeout
    select {
    case <-doneChan:
        return true // Success! We got our majority
    case <-time.After(2 * time.Second):
        return false // Timeout, no majority
    }
}

The beauty is in the select statement. We're racing all the network requests, and as soon as we get enough votes for a majority, we declare victory and move on. No waiting for stragglers!

Testing the Magic

Let me show you this in action. I started 3 nodes and ran some tests:

# Create a table
curl -X POST http://localhost:8001/sql -d '{"sql":"CREATE TABLE users (id INTEGER PRIMARY KEY, name TEXT)"}'
{"success":true,"result":{"rows":null,"rows_affected":0,"last_insert_id":0}}

# Insert some data
curl -X POST http://localhost:8001/sql -d '{"sql":"INSERT INTO users (name) VALUES (?)", "args":["Alice"]}'
{"success":true,"result":{"rows":null,"rows_affected":1,"last_insert_id":1}}

curl -X POST http://localhost:8001/sql -d '{"sql":"INSERT INTO users (name) VALUES (?)", "args":["Bob"]}'
{"success":true,"result":{"rows":null,"rows_affected":1,"last_insert_id":2}}

# Query from different nodes - data is replicated!
curl -X POST http://localhost:8002/sql -d '{"sql":"SELECT * FROM users"}'
{"success":true,"result":{"rows":[{"id":1,"name":"Alice"},{"id":2,"name":"Bob"}],"rows_affected":0,"last_insert_id":0}}

Node 1 logs show the election process:

The Trade-off: Speed vs. Absolute Durability

Now, let's address the elephant in the room. What if Guangzhou gets hit by a meteor right after the client receives "success" but before Shanghai gets the data? Well, that data is gone forever.

This isn't a bug – it's a feature! PhxSQL makes a deliberate choice: optimize for the 99.999% case where regions don't get destroyed, rather than the 0.001% disaster scenario.

For most applications, losing a few writes during a complete regional disaster is acceptable. The alternative – making every write operation 30ms slower to guard against meteor strikes – would make your users suffer every single day for a scenario that hopefully never happens.

Why PhxSQL Never Became the Next Big Thing

Nine years have passed since WeChat open-sourced PhxSQL, and it never really took off. Here's my theory:

  • The "WeChat Problem" is Rare: Not many companies have billion-user social networks with payment systems. Most companies are still figuring out how to handle their first million users.
  • Cloud-Native Happened: In 2025, most companies just use managed database services with built-in HA. Why build your own distributed database when AWS RDS or Google Cloud SQL does it for you?
  • Ecosystem Matters: PhxSQL never built the ecosystem. No fancy dashboard, no Docker images, no Kubernetes operators. If you can't deploy it with one YAML file, modern DevOps teams won't touch it.
  • On-Premise is Dead: Most companies moved to the cloud. Those who didn't probably went with TiDB or other modern distributed databases.

The Lasting Legacy: The Philosophy Still Matters

Even though PhxSQL itself might be a museum piece, its "Fastest Majority" philosophy is still relevant in 2025. Whether you're building microservices, distributed caches, or that secrets manager I mentioned, remember:

  • Optimize for the common case, not the disaster scenario
  • Don't let the slowest node dictate your performance
  • Sometimes "fast enough" is better than "perfectly consistent"

My WeChat Days: Learning from Giants

Back when I was part of the WeChat team, dealing with scale was a daily reality. When you have over a billion daily active users across social messaging, payments, and mini-programs, hardware failures aren't "if" – they're "when" and "how many today?" WeChat's approach to PhxSQL made perfect sense in that context. The team had:

  • Deep MySQL expertise (it's the most widely used database in China)
  • Extensive Paxos algorithm experience (they also open-sourced Paxos-based NoSQL and message queue systems)
  • A real need for automated failure handling and node replacement

Building a Paxos layer on top of MySQL wasn't just clever – it was necessary survival at that scale.

Sometimes the Old Ways Are Still Good Ways

Would I recommend PhxSQL for a new project in 2025? Probably not. Would I recommend understanding its philosophy? Absolutely.

In our rush toward cloud-native, serverless, and AI-everything, we sometimes forget that some of the best engineering solutions come from understanding fundamental trade-offs. PhxSQL's "Fastest Majority" approach is a masterclass in making the right compromises.

So the next time you're designing a distributed system, remember: you don't always need to wait for the slowest member of the committee. Sometimes, the fastest majority is all you need.

P.S. - The full Go implementation is available if you want to play with it. It's about 900 lines of code that capture the essence of a system that handled billions of users. Not bad for a weekend project!