Valkey Over RDMA Internals

2. RDMA basics

How does communication actually work?

TCP is a stream protocol (byte stream); RDMA is message-oriented (elements).

We do not use two-sided ops (SEND/RECV) for business data. Instead, two-sided ops exchange signaling, and one-sided WRITE carries data.

[Valkey Client]                                               [Valkey Server (Listen RDMA Port)]
      |                                                                       |
      | ======================== 1. 建立 RDMA 连接 =========================>  |
      | <=======================    (RC QP 初始化)   ========================  |
      |                                                                       |
      | ================== 2. 交换 Feature 支持 [@IBV_WR_SEND] =============>  |
      | <================= 3. 交换 Feature 支持 [@IBV_WR_SEND] ==============  |
      |                                                                       |
      |                     [阶段 A：初始化远端内存写入权限] 这里都是双边操作        |
      | <---- 4. Server 发送其 RX Buffer 的地址/长度/R_Key [@IBV_WR_SEND] ---   | (Server 执行 ibv_post_recv 准备接收命令)
      | ---- 5. Client 发送其 RX Buffer 的地址/长度/R_Key [@IBV_WR_SEND] --->   | (Client 执行 ibv_post_recv 准备接收响应) 两边都通过send来发送R_key和内存地址,两边收到之后都可以绕过CPU直接进行读写的操作.
      |                                                                       |
      |                     [阶段 B：使用 WRITE 进行命令与数据交互]           |
      | --- 6. Valkey Command (如 PING) [@IBV_WR_RDMA_WRITE_WITH_IMM] ------> | (直接写入 Server 内存，触发 IMM 通知 Server CPU)
      | <--- 7. Valkey Response (如 PONG) [@IBV_WR_RDMA_WRITE_WITH_IMM] ----- | (直接写入 Client 内存，触发 IMM 通知 Client CPU)
      | --- 8. Valkey Command (如 SET) [@IBV_WR_RDMA_WRITE_WITH_IMM] -------> |
      | <--- 9. Valkey Response (如 OK) [@IBV_WR_RDMA_WRITE_WITH_IMM] ------- |
      |                                                                       |
      |                     [阶段 C：处理缓冲区写满 (RX is full)]             |
      | (假设 Client 发现 Server 之前给的 RX Buffer 空间不足了)               |
      |                                                                       |
      | <--- 10. Server 重新注册内存并发送新 RX Buffer 信息 [@IBV_WR_SEND] -- | (Server 发现自己消费完了，主动发新 Buffer，并 ibv_post_recv)
      | --- 11. Valkey Command (恢复发送) [@IBV_WR_RDMA_WRITE_WITH_IMM] ----> |
      |                                                                       |
      | (假设 Server 发现 Client 的 RX Buffer 不足了)                         |
      | ---- 12. Client 重新注册内存并发送新 RX Buffer 信息 [@IBV_WR_SEND]-> | (Client 主动发新 Buffer，并 ibv_post_recv)
      | <--- 13. Valkey Response (恢复发送) [@IBV_WR_RDMA_WRITE_WITH_IMM] --- |
      |                                                                       |
      | ======================== 14. 断开 RDMA 连接 ========================> |

1. What is an IMM notification?

An immediate value — covered in point 4 below.

2. Why not use two-sided ops for data?

In classic two-sided SEND/RECV, the receiver must guess how large the sender’s message will be. The receiver must ibv_post_recv a RECV WQE. If the Redis client sends a 10-byte PING but the server posted a 10MB receive buffer, that wastes memory; if the server posted 10 bytes and the client sends a 1MB SET KEY VALUE, the NIC reports Length Error and tears down the QP. For Redis-style streaming with unpredictable request sizes, pure SEND/RECV is a disaster. Handshake messages have predictable sizes, so two-sided is fine there.

3. Using one-sided WRITE to simulate a TCP stream?

RDMA op types make this vivid: 3. RDMA operation types Since message size is unpredictable, the design returns to TCP’s essence: maintain an application-level virtual receive buffer like a sliding window. Large memory pool: the server registers a relatively large region (e.g. 1MB) as a bucket (receive buffer). Signaling (role of SEND): the server uses a SEND to hand the client the bucket’s “key and address” (R_Key, VA, length). Client free writes: with the key, the client issues one-sided RDMA_WRITE into the server bucket at the current offset. 10 bytes or 100k bytes are fine as long as remaining capacity allows.

4. Why IBV_WR_RDMA_WRITE_WITH_IMM?

Pure RDMA_WRITE is a silent one-sided op — data lands in remote memory with no CQE on the remote side, so the remote CPU never knows. If Valkey Server does not know a command arrived, it cannot run GET/SET. IBV_WR_RDMA_WRITE_WITH_IMM silently writes data and attaches a 32-bit Immediate. When the NIC finishes the write, it wakes the remote and posts a CQE carrying that IMM on the Server CQ. (So I know when I can read, but not when I can write the peer’s registered memory.) In the Redis/Valkey engineering, clients typically put bytes written in that 32-bit IMM. Server epoll sees the NIC event, reads the CQE, sees IMM=25: “someone just wrote 25 bytes into my bucket — process them.” Data path is one-sided. As server, I usually know when to read, but not when I may write into the peer’s registered memory.

5. When full, refresh memory.

A fixed-size streaming receive buffer eventually fills. The Client tracks remaining Server bucket space and must stop when nearly full. After the Server drains commands and the region is exhausted, it reinitializes/resets the region and SENDs again: “bucket cleared (or new bucket), here is the new base and length — continue.” Does that RTT cause jitter? Yes — that is an open curiosity.