Raft Cluster valkey-cli Tooling Compatibility

Also a very interesting topic — Valkey recently introduced Raft.

This PR addresses the following: in a Raft cluster, the server-side execution model for topology-change commands has changed, but valkey-cli --cluster fix still sends commands using the old gossip transaction model, so the two are incompatible.

See https://github.com/valkey-io/valkey/issues/3856 for background.

start

#

First switch to the latest upstream branch:

# 拉取上游最新改动
git fetch upstream

# 基于上游的 cluster-v2 分支，在本地创建一个同名分支并切换过去
git checkout -b cluster-v2 upstream/cluster-v2

# 你还可以在自己的仓库内部保存一下进度
git push -u origin cluster-v2

Because the overall difficulty is high and the surface area is large, we need to learn and expand some background.

Start with feature: master–replica replication:

You also need to understand the Raft-Extended Version 1–5 algorithm in depth before going further into this feature.

This is the issue we need to solve: Raft Cluster: valkey-cli --cluster Tooling Compatibility

Workflow

#

This is our first collaborative development with the community.

Sync the feature branch first:

git fetch upstream
git checkout cluster-v2
git merge upstream/cluster-v2   # 或 git rebase upstream/cluster-v2

Or hard-reset to upstream’s latest:

# 1. 退出当前 rebase，恢复 rebase 前状态
git rebase --abort
# 2. 确认 upstream 最新
git fetch upstream
# 3. 将本地 cluster-v2 硬重置到 upstream（本地无独有 commit 时安全）
git checkout cluster-v2
git reset --hard upstream/cluster-v2

Then create a new feature branch:

git checkout -b cli-raft-fix-3867 # 之后就在这个上面进行开发

Keep origin and upstream in sync:

git push --force-with-lease origin cluster-v2

Background first

#

What does --cluster fix repair?

#

valkey-cli --cluster fix is used to repair abnormal cluster topology. Common cases include:

• A slot has no owner (uncovered)
• Multiple nodes claim the same slot
• A slot stuck in migrating / importing (half-migration)

One of the core operations is: explicitly assign ownership of a slot to a primary, via clusterManagerSetSlotOwner():

CLUSTER DELSLOTS <slot>   # 从当前节点摘掉（可能已是 unassigned，可忽略）
CLUSTER ADDSLOTS <slot>   # 正式赋给目标节点
CLUSTER SETSLOT <slot> STABLE  # 可选，清掉 migrating/importing 状态

Sending these commands to a given primary achieves that effect.

Several paths inside fix reach this function, e.g. clusterManagerFixOpenSlot(), clusterManagerFixSlotsCoverage(), and so on. This is how uncovered slots are handled.

Gossip era: why MULTI/EXEC

#

The so-called “transaction”.

In a gossip cluster (cluster-protocol gossip, classic Redis Cluster), topology changes roughly work as:

1. Each node updates its local slot table
2. Gossip propagates the change
3. Config epoch resolves conflicts

valkey-cli wants these steps to complete atomically so intermediate states are not visible to other operations, so clusterManagerSetSlotOwnerGossip() uses a transaction:

static int clusterManagerSetSlotOwnerGossip(clusterManagerNode *owner, int slot, int do_clear) {
	// 发送事务，然后执行。
    int success = clusterManagerStartTransaction(owner);
    if (!success) return 0;
    clusterManagerDelSlot(owner, slot, 1);
    clusterManagerAddSlot(owner, slot);
    if (do_clear) clusterManagerClearSlotStatus(owner, slot);
    clusterManagerBumpEpoch(owner);
    success = clusterManagerExecTransaction(owner, clusterManagerOnSetOwnerErr);
    return success;
}

On the wire the command sequence looks like:

For example, when you want to modify a node, commands are queued first, then executed all at once when EXEC arrives — no intermediate state.

MULTI
CLUSTER DELSLOTS 609    → QUEUED
CLUSTER ADDSLOTS 609    → QUEUED
CLUSTER SETSLOT 609 STABLE → QUEUED（可选）
CLUSTER BUMPEPOCH       → QUEUED
EXEC                    → 一次性执行，返回数组

Under gossip, CLUSTER ADDSLOTS / DELSLOTS are synchronous commands: they return +OK immediately and do not block the client. MULTI/EXEC is a reasonable optimization here.

1. V1 old approach: why was MULTI/EXEC used?

#

Under the current Gossip architecture, when you use valkey-cli --cluster create or rebalance to build/adjust a cluster, the CLI must send a series of topology-change commands to specific nodes (assign many slots, set epoch, etc.). To keep that batch atomic (all succeed or all fail), valkey-cli wraps them in a transaction block in C:

1. Send MULTI (start the transaction — called a “transaction” because multiple commands run as one batch)
2. Send CLUSTER ADDSLOTS 0 1 2 ...
3. Send CLUSTER SET-CONFIG-EPOCH 1
4. Send EXEC (execute once)

At this stage, the node handles these commands as fully synchronous, in-memory operations. The main thread runs them in one go with no network I/O wait in between — very smooth.

You send the batch to one machine; that machine’s main thread does not wait for consensus. It updates topology locally, and gossip later propagates it. If the desired end state needs many commands and you do not use a transaction, many intermediate states become visible.

2. V2 new conflict: why does Raft reject MULTI/EXEC?

#

After Raft (Cluster V2), topology changes are serious — a single node can no longer decide unilaterally.

Raft must wait for majority replies; we cannot freeze the server waiting for that. Topology changes became a big deal; the leader cannot just mutate freely.

When a node receives CLUSTER ADDSLOTS, it cannot simply mutate memory and return. It must:

1. Append the slot change as a Proposal into its Raft log.
2. Replicate the log over the network to Followers.
3. Block the current client request and wait for Quorum confirmation.
4. After quorum, apply to the in-memory state machine, then reply OK to the client.

In Valkey’s internals, operations that must suspend waiting for an async event (e.g. a network reply) are marked BLOCKED_ASYNC.

Here is the core conflict: Valkey’s core event loop is single-threaded. MULTI/EXEC semantics mean “own the main thread, run every queued command to completion, and never yield the CPU in between.”

Gossip can apply immediately and reach consensus later; Raft must wait until commit before replying.

If a BLOCKED_ASYNC command were allowed inside a MULTI block, while waiting for Raft network consensus it would freeze the entire Valkey main thread, and every other client request would stall.

To prevent that disaster, the Valkey server hard-codes a defense: any command marked BLOCKED_ASYNC is forbidden inside MULTI/EXEC. If it appears, reject with an error.

The idea is to stop using transactions and execute commands one by one; on the Raft side the leader replies to the client after the log commits, and the client can poll meanwhile.

Reproducing the problem

#

Build first:

make -j$(nproc) # 多线程进行编译

Then on the client side, on that connection:

~/Project/valkey cli-raft-fix-3867 ❯ ./src/valkey-cli -p 7000 
127.0.0.1:7000> MULTI
OK
127.0.0.1:7000(TX)> CLUSTER ADDSLOTS 0
QUEUED
127.0.0.1:7000(TX)> EXEC
1) (error) ERR This cluster command is not allowed inside MULTI # 目前MULTI操作是不支持的

Then verify with tests:

# Raft（验证 #3867）
./runtest --cluster-raft --single tests/unit/cluster/half-migrated-slot.tcl # 这个就一定会报错

# Gossip 回归
./runtest --single tests/unit/cluster/half-migrated-slot.tcl # 这个可以实现结束之后进行回归测试

At this point reproduction is done.

flowchart TB
    subgraph MULTI["MULTI/EXEC 模型"]
        Q[收到命令] --> Queue[只入队 queueMultiCommand]
        Queue --> R1[立刻回复 QUEUED]
        EXEC[EXEC] --> Loop[同步 for 循环 call]
        Loop --> Assert[assert blocked == 0]
        Assert --> Array[按顺序写进 EXEC 数组回复]
    end

    subgraph Raft["Raft 拓扑命令模型"]
        C[收到命令] --> Propose[立刻 clusterRaftPropose,Raft是不能被阻塞的，否则整个集群的状态机都会被直接积压住，这样会出现问题]
        Propose --> Block[blockClientAsync - 不回复]
        Block --> Wait[等 Raft commit]
        Wait --> CB[回调 addReply OK]
        CB --> Unblock[unblockClientAsync]
    end

    MULTI -.->|同一命令不能同时满足| Raft

Raft cannot wait: by design, as soon as a command arrives it must propose immediately. That is the core contradiction.