Raft architecture review: see Raft Cluster Pre-Vote Protocol — our first Raft-related PR.
Also useful in parallel: feature: Redis Cluster.
Requirements from the issue:
Implement support for “Learner” or “Observer” non-voting roles in the Raft consensus group.
Implement learner or observer (effectively learner).
Non-voting nodes should still replicate the Raft log to receive topology updates so they can serve client requests like CLUSTER SLOTS accurately.
Learners should replicate the log / state machine and accept topology updates so they can serve client requests accurately (reads locally; writes forwarded).
They should not participate in write quorums or elections, preventing performance overhead and scale limitations on the voting core. In a cluster with hundreds of nodes, it’s beneficial if only ~5 nodes are voting members, to minimize the risk of many nodes simultaneously requesting votes for themselves in a leader election, resulting in high risk of split vote.
Do not participate in elections or write-quorum acks, so election performance does not degrade.
Why not participate in acks?
Because newly joined nodes have little log and cannot help commit; adding many at once stalls commits for a long time.
In a cluster of hundreds of nodes, only ~5 need to vote — reduce many nodes starting elections at once and split votes (randomized election timeouts already help a lot).
Support potential chaining-style replication of the Raft log where learners pull from other nodes rather than all overloading the leader. (This point seems like a separate feature, but we should think about the future possibility.)
If one voting node is hit by hundreds of learners,
network I/O becomes the bottleneck — consider chaining (classic idea; see Google File System dataflow / pipeline).
Later alignment with reviewers:
I think we should add all nodes as learners by default. When a node has caught up with the raft log, it can be promoted to a follower. I believe this is some recommendation some paper. Btw, does any of you know which paper describes Raft learners?
View: when a node joins, it should start as learner, and only become follower after catching up on the log (exact “identical vs close enough” still needs research).
Details need the thesis and etcd source.
1. Yes, a learner can be primary and replica.
In the data-plane sense, a learner can be primary (own slots) or replica.
Is it a problem if elections are not done by primaries?
2. I think it’s good to distinguish an entry that affects quorum from an entry that doesn’t, especially if we want to implement config-on-append later (Raft Cluster: Config-on-append for membership changes #3926). If we want to add all new nodes as learners first, then maybe NODE_JOIN should make the new node a learner. Or we add a flag to NODE_JOIN to say whether it’s a voter or not.
Regardless, we’ll need to be able to promote learners to followers and demote followers to learners, so we’ll need new entry types for that. When a node is removed, NODE_FORGET would apply to learners and voters or should we split that one as well, i.e. demote to follower before we append NODE_FORGET?
Distinguishing quorum-affecting vs non-quorum-affecting log entries is good, especially for future config-on-append.
If we add as learner first, maybe NODE_JOIN should default new nodes to learner.
Or add a flag on NODE_JOIN for voter vs learner.
Either way we need learner→follower and follower→learner, so new log entry types.
On remove: does NODE_FORGET apply to both, or demote to learner first then forget?
Not fully aligned yet — more research. → We should just remove the node, given Pre-Vote is already implemented.
3. Yeah, I think server.cluster->size should represent the voters only. In gossip cluster, the cluster size is only the primaries with slots, i.e. the voters, so we can keep the same convention in raft cluster.
Cluster size = voter count only. Contradiction? Learners can also be primaries holding slots.
Not really a contradiction — data plane and control plane are orthogonal.
4. Yeah PROMOTE <id> and something similar for DEMOTE <id> – one node at a time to match the current config-on-apply logic for quorum updates. But these words can also mean promote to leader, candidate, so maybe ADD_VOTER <id> and DEL_VOTER <id> is clearer?
Clearer naming?
5. I don’t think this problem applies to this issue. A singleton leader becomes a joiner (just waiting to be added to a cluster) and then reverts to singleton leader again after a timeout. There’s no learner role involved here. Maybe it’s a scenario that only affects Raft Cluster: Merging non-singleton clusters #3868? We can discuss it on that issue.
Leader demotion in cluster merge — separate issue.
Then, we’ll also need a flag in nodes.conf to say if a node is a voter or not. We could use one of these:
Add a cluster-config flag for voter vs not.
- node flags — use the first two
- aux fields- one of the unused columns ping-sent, pong-received
Just a though: (If we ever want to do joint consensus, we could list all the voters on every quorum change, e.g. an entry like CONFIG_VOTERS id1 id2 id3 .... Did we already say we don’t need joint consensus? If we just keep a list of uncommitted voters or something, maybe it’s not really that difficult? 🤔 We can change this later if we want to implement joint consensus though. If we do it, we can just use this entry type every time we add or remove voters. There shouldn’t be more than 5 or 7 voters in a cluster, normally.)
For safe topology changes, we chose single-server configuration change, dropped confusing PROMOTE/DEMOTE, and designed explicit Raft control commands:
NODE_JOIN: Change this so new nodes default to Learner, not immediately Voter.
ADD_VOTER <id> and DEL_VOTER <id>: New core cluster-internal / Raft state-machine log entries for learner ↔ follower transitions.
ADD_VOTER when Leader sees Learner nextIndex caught up — a config-change log.
Only after catching the log may it become follower (exact catch-up bar: see thesis).
After majority commits that entry, the Learner becomes Voter and server.cluster->size increases immediately.
About NODE_FORGET: Maintainer edge case: kick a Voter with direct NODE_FORGET, or force DEL_VOTER then NODE_FORGET?
In etcd, remove directly; if voter, recompute quorum; if learner, delete even more freely.
With Pre-Vote (Raft Cluster Pre-Vote Protocol) already in place, a deleted node will not disrupt the cluster.
So we delete directly — no two-phase demotion — and must immediately recompute Quorum.
Joint Consensus was covered in Raft-Extended Version 6–7.
Here we focus on single-server membership change.
Two RPCs:
AddServer: add a server to the cluster.
Leader side:
1. Check still leader
2. Catch up within fixed rounds, or last-round timeout → fail add
3. Wait previous config committed
4. Append new config log
Most membership change algorithms introduce additional mechanism to deal with such problems. This is what we did for Raft initially, but we later discovered a simpler approach, which is to disallow membership changes that could result in disjoint majorities.
Most membership algorithms add mechanism. Raft started that way.
Later found a simpler rule: forbid membership changes that can create disjoint majorities.
In the example above, old and new clusters have no intersection.
only one server can be added or removed from the cluster at a time.
That is single-server membership change.
Still Understandability.
Single-server scenarios look like:
Simple: (a) add one node to a 4-node cluster — old majority 3, new majority also 3; those two majorities of 3 in 5 must intersect — no split brain, safe to switch.
Cluster configuration is stored and communicated as special entries in the replicated log.
This reuses Raft’s existing replication and persistence.
It also lets the cluster keep serving clients during a config change by ordering config changes vs client requests (while still allowing pipelined/batched concurrent replication).
When the Leader receives a request to add/remove a server from the current config (Cold),
it appends the new config (Cnew) as a log entry
and replicates it with normal Raft.
As soon as Cnew is appended on a server, it takes effect there:
Cnew is replicated to servers in Cnew,
and the Leader uses Cnew’s majority to decide whether Cnew itself is committed.
Servers do not wait for the config entry to commit before using it; each always uses the latest config found in its log.
Once a config entry is replicated, it must take effect immediately — unlike data log entries.
Once Cnew commits, the configuration change is complete.
The Leader knows a majority of Cnew has adopted Cnew.
Any server that has not switched cannot form a majority, and no non-Cnew server can become Leader. Committing Cnew enables three things:
Leader can acknowledge to the client that the config change succeeded.
If the change removed a server, that server can now shut down.
The next config change can start. Before that, overlapping changes can degenerate into the unsafe case of Figure 4.2.
So config-change requests must be effectively one-at-a-time; otherwise the safety argument fails.
Do not start the next until the previous finishes.
As above, servers always use the latest config in the log, committed or not.
That lets the Leader easily avoid overlapping config changes by not starting a new one until the previous config entry commits.
If servers adopted Cnew only after learning it committed, the Leader would struggle to know when Cold’s majority had adopted it —
tracking which servers know about commit, and persisting commit index — mechanisms Raft otherwise does not need.
Instead, each server adopts Cnew as soon as it appears in the log.
Unfortunately this also means that after Leader changes, a config-change log entry may be overwritten/removed;
servers must be ready to fall back to the previous config in their log.
In Raft, reaching consensus (votes or log replication) uses the caller’s configuration:
Servers accept AppendEntries from a Leader not in their latest config. Otherwise a new server could never join (it would reject all log before the entry that adds it).
Servers also vote for a Candidate not in their latest config (if log is fresh enough and term is current). Sometimes that vote is required for availability — e.g. adding a 4th node to a 3-node cluster; if one node dies, the new node’s vote may be needed for majority. So when handling inbound RPCs, servers do not consult their current config.
Issue #3926 covers this; not in scope for us now — just be aware.
The phase we care about: new servers join as learners first, then promote to followers when caught up.
Join temporarily as a non-voting member.
How to catch up without infinite sync (Zeno’s paradox — Achilles vs turtle; learner never becomes follower if we require identical forever).
Mechanism:
New server joins; leader syncs.
Round 1: empty log, may take longer; meanwhile leader gets yellow entries → round 2.
Later incremental rounds shrink; never fully identical unless clients stop.
The leader needs to determine when a new server is sufficiently caught up to continue with the configuration change.
The leader must decide when a new server is ready to become a follower.
In Raft, tolerable unavailability is roughly one election timeout.
If the new server is unavailable or too slow, the leader should roll back the membership change.
Algorithm:
Log replication in rounds; each round fully copies all log present at round start.
Wait a fixed number of rounds (e.g. 10); if the last round finishes within one election timeout, promote — availability is acceptable. Otherwise roll back.
Short times need fast log rollback (already implemented; not our problem here).
