Recently the Valkey community has been exploring a new feature: driving Cluster bus communication with the Raft protocol. I had just finished learning Raft and master–replica replication, and claimed an issue, so I wanted to dig deeper and try to solve more problems in this area.
Community discussion links:
It splits into multiple sub-issues. Summary:
1. Core foundation: Raft consensus and state-machine persistence #
This is the most orthodox part of Raft — ensuring the cluster does not lose state across power loss or crashes.
- #3857 State persistence: Persist the current Term, VotedFor, and the append-only Log. This is how a node recovers its identity.
- #3858 Log compaction and snapshots: Over time the Raft log grows without bound. You must periodically snapshot the state machine and trim old log entries so disks are not exhausted.
- #3859 Crash recovery: After restart, how a node smoothly rejoins by reading local log and fetching missing entries from the Leader.
2. Network resilience: partition tolerance and high availability #
This area stresses understanding of the network stack and split-brain: node liveness probes and isolation under abnormal conditions.
- #3860 Pre-Vote protocol: After a partition heals, prevent an isolated minority node from forcing a disruption of the healthy Leader with an inflated fake Term.
- #3861 Minority-partition detection and Leader step-down: When a Leader finds itself isolated in a minority partition, it must voluntarily give up leadership and stop serving.
- #3862 Replication-stream-based failure detection (decentralized lease): Drop the traditional high-frequency Ping-Pong heartbeat; reuse the data sync stream for liveness, greatly cutting network I/O.
3. Data control plane: sharding and state transitions #
This layer binds Valkey’s Slot concept tightly to the Raft engine for strongly consistent shard routing.
- #3863 Shard-level Epoch: Finer-grained version control for request fencing under network isolation, so clients cannot write dirty data to a stale Master.
- #3865 Persist MIGRATING/IMPORTING state in the log: Make slot migration an atomic consensus operation.
- #3874 Leader-side proposal pre-validation (new): Before packaging an operation into a log entry for broadcast, the Leader validates it first, rejecting invalid changes and cutting wasted network traffic.
4. Cluster topology and role evolution #
Break classic Raft limits for better scalability and more flexible operations.
- #3866 Non-voting members (Learner): Replicate data but do not vote — break the curse that more Raft nodes always means worse performance.
- #3868 Merging non-singleton clusters: Safely “marry” two clusters that already have independent state machines.
- #3869 Sync-replication promotion workflow: More precise state-machine transitions during primary/replica failover.
- #3864 Manual failover via REPLCONF: Keep an engineering backdoor for forced primary/replica switches.
5. Global ecosystem and client adaptation #
Provide a “god’s-eye” control plane — relatively more peripheral.
- #3867 CLI toolchain compatibility: Update the command-line client to strip
MULTI/EXECtransaction logic that conflicts with Raft’s async blocking. - #3875 Cluster-level ACL and Functions (new): A qualitative jump. Previously ACL (and similar) had to be applied node by node; with Raft the cluster finally has a globally reliable metadata plane — configure once, strong consistency everywhere.
Community discussion on motivation
Mainly to fix systemic design flaws of the first-generation Gossip-based cluster under very large deployments, cloud-native environments, and extreme load.
1. Control-plane vs data-plane resource contention (Resilience & Threading) #
- Single-thread bottleneck: In V1, Cluster Bus heartbeats and client data I/O all share the same Main Thread.
- False failovers: Under extreme client load or massive Pub/Sub traffic, the main thread is occupied so long that peers miss health-check Pings. This “control plane starved by data plane” pattern causes false death detection, pointless failovers, and cluster oscillation.
2. Topology state lacks strong consistency (Strong Consistency & Metadata) #
- Limits of eventual consistency: V1 propagates topology (who owns which Slot, who is primary) via Gossip. Under partitions, Epoch can be forced up without majority consensus. Topology is not strongly consistent, and in some edge cases not even eventually consistent — easy split-brain and cascading failure / data loss.
- Global config nightmare: Valkey has no centralized metadata store. Updating ACL, loading Functions, or changing global config requires manually pushing commands to every node — configuration drift is common.
3. Mathematical limits of network topology (Scalability & Gossip Overhead) #
- Communication complexity: Gossip requires a full-mesh of connections and frequent state exchange. As node count grows, redundant Cluster Bus traffic grows aggressively.
- ~500-node ceiling: This design caps V1 around ~500 nodes, far below modern large services that need thousands.
4. Poor fit for modern infrastructure (Cloud-Native & Operability) #
- Unfriendly to containers and dynamic IPs: Users report that in Kubernetes-style ephemeral, dynamic-IP environments, V1 bootstrap and node replacement are rigid. Config files accumulate stale IPs, and different queries (
cluster nodesvscluster shards) often disagree. - High HA bar: V1 election only lets Primaries vote, so a minimally resilient cluster needs at least 3 primaries for majority. V2 wants voting Replicas so tiny deployments (even single-shard) get built-in autofailover, absorbing classic Sentinel use cases.
- Passive route updates: Today clients often rediscover routes by hitting the wall (
MOVED). V2 plans RESP3 push of topology changes to cut SDK redirect churn.