Table of Contents

Notes from trying to fix [CRASH] Does Valkey over RDMA not support I/O threads? #3112.

https://github.com/valkey-io/valkey/issues/3112

Summarizing the experimental workflow — longest campaign yet; over a month and still not merged.

Server side

# 解开当前进程的内存锁
sudo prlimit --memlock=unlimited --pid $$
# 创建在内存中工作的虚拟网卡
sudo modprobe dummy
sudo ip link add dummy0 type dummy
sudo ip link set dummy0 up
# 配置本地的IP
sudo ip addr add 10.0.0.1/24 dev dummy0
# 做RXE的绑定
sudo rdma link add rxe_dummy type rxe netdev dummy0
# 停止本来运行的 valkey
sudo systemctl stop valkey
# 编译一遍
make BUILD_RDMA=yes USE_FAST_FLOAT=yes
# 跑server
sudo ./src/valkey-server valkey.conf --rdma-bind 10.0.0.1 --rdma-port 6379

Client side

# 解开当前进程的内存锁
sudo prlimit --memlock=unlimited --pid $$
# 直接做暴力压测,还可以尝试比这个参数更高的
./src/valkey-benchmark -h 10.0.0.1 -p 6379 -d 256 --threads 16 -c 100 -P 32 -n 20000000 -t get --rdma

Because this is multithreaded, watch the Valkey config:

 io-threads 4
 save ""
 protected-mode no
 bind 0.0.0.0

How to capture logs for analysis

 # 拿一下server当前的PID:
 ./src/valkey-cli -h 10.0.0.1 -p 6379 INFO server | grep process_id
 # 拿到这个PID之后 抓线程栈
 sudo gdb -p $(PID) -ex 'thread apply all bt' -batch > /tmp/valkey_bt.txt
 # 看一下文件的前多少行
 head -n 60 /tmp/valkey_bt.txt
 
 
 # 抓取当前I/O队列的状态
 ./src/valkey-cli -h 10.0.0.1 -p 6379 INFO stats | grep -E 'clients_pending_io_read|clients_pending_io_write|io_threaded_reads_processed|io_threaded_writes_processed|instantaneous_ops_per_sec'
 # 看一下客户端的情况
 ./src/valkey-cli -h 10.0.0.1 -p 6379 CLIENT LIST | head -n 10
 # 查看系统积压的延迟
 ./src/valkey-cli -h 10.0.0.1 -p 6379 LATENCY LATEST
 
 # 看队列长度和事件计数,就是查事件的状态机
  ./src/valkey-cli -h 10.0.0.1 -p 6379 INFO stats | egrep 'clients_pending_io_read|clients_pending_io_write|io_threaded_reads_processed|io_threaded_writes_processed|instantaneous_ops_per_sec'

Also tidy the formatting

 clang-format -style=file -i src/networking.c

Reproducing the multithread RDMA crash
#

First build with RDMA:

  make BUILD_RDMA=yes USE_FAST_FLOAT=yes

Create a dummy NIC (pure in-memory sim; wlan0 can be flaky)

 sudo modprobe dummy
 sudo ip link add dummy0 type dummy
 sudo ip link set dummy0 up

Assign a local IP to the dummy NIC

 sudo ip addr add 10.0.0.1/24 dev dummy0 # 这个地址是否是可行的?

Bind SoftRoCE to dummy0

# 如果之前的 rxe0 还在，可以先删掉： sudo rdma link del rxe0
sudo rdma link add rxe_dummy type rxe netdev dummy0

Run the server:

./src/valkey-server valkey.conf --rdma-bind 10.0.0.1 --rdma-port 6379

If the port conflicts:

sudo systemctl stop valkey

Stress the client against the virtual IP (heavy load — watch memlock):

./src/valkey-benchmark -h 10.0.0.1 -p 6379 -d 256 --threads 5 -r 100000 -n 12000000 -c 30 -t get --rdma

Fails immediately — latest main still has the bug.

But the client error is:

~/Project/valkey unstable ⇡ ❯ ./src/valkey-benchmark -h 10.0.0.1 -p 6379 -d 256 --threads 5 -r 100000 -n 12000000 -c 30 -t get --rdma
Could not connect to server at 10.0.0.1:6379: RDMA: reg send mr failed

Aggressive stress. RDMA memory access is like DMA to userspace: lock pages so Linux cannot swap them out.

On Arch Linux, a normal user’s max locked memory (memlock) is usually limited. When the limit is hit, rdma_reg_mr() fails; the client sends incomplete frames or disconnects. That abnormal disconnect / bad frame can instantly trigger the server cleanup bug — another possible cause?

Check whether memlock is raised:
# On Linux memory is fluid and swaps with the swap partition
# So a process may lock at most 8MB — hostile to DMA, which needs locked pages for exchange
~ ❯ ulimit -l
 8192
# Try raising the limit — note: only for the current process
~ ❯ sudo prlimit --memlock=unlimited --pid $$
# Verify the process is unlimited — both soft/hard become unlimited
~ ❯ cat /proc/self/limits | grep "Max locked memory"
Max locked memory         unlimited            unlimited            bytes
# Apply this on both client and server
Client (sender): allocate a buffer for GET/SET commands and lock it. The NIC DMAs from that memory onto the wire.

Server (receiver): pre-allocate and lock a buffer. On receive the NIC writes past the CPU into that locked memory.

Bad client frames, or multithreading?
#

First run a single-thread test:

~/Project/valkey unstable ⇡ ❯ ./src/valkey-benchmark -h 10.0.0.1 -p 6379 -c 1 -n 1000 -t get --rdma
====== GET ======
  1000 requests completed in 0.07 seconds
  1 parallel clients
  3 bytes payload
  keep alive: 1
  host configuration "save": 3600 1 300 100 60 10000
  host configuration "appendonly": no
  multi-thread: no

Latency by percentile distribution:
0.000% <= 0.039 milliseconds (cumulative count 113)
50.000% <= 0.055 milliseconds (cumulative count 726)
75.000% <= 0.063 milliseconds (cumulative count 931)
93.750% <= 0.071 milliseconds (cumulative count 949)
96.875% <= 0.119 milliseconds (cumulative count 971)
98.438% <= 0.151 milliseconds (cumulative count 988)
99.219% <= 0.175 milliseconds (cumulative count 994)
99.609% <= 0.215 milliseconds (cumulative count 997)
99.805% <= 0.247 milliseconds (cumulative count 999)
99.902% <= 1.559 milliseconds (cumulative count 1000)
100.000% <= 1.559 milliseconds (cumulative count 1000)

Cumulative distribution of latencies:
96.500% <= 0.103 milliseconds (cumulative count 965)
99.600% <= 0.207 milliseconds (cumulative count 996)
99.900% <= 0.303 milliseconds (cumulative count 999)
100.000% <= 1.607 milliseconds (cumulative count 1000)

Summary:
  throughput summary: 14084.51 requests per second
  latency summary (msec):
          avg       min       p50       p95       p99       max
        0.056     0.032     0.055     0.079     0.159     1.559

Single-thread is fine. With memlock limited you can only push to about 3 before errors. Not the main point — even if the client OOMs / miscommunicates, the server should not exit like this!

After unlocking memlock, latest builds seem not to hit this.

But it is not 100% reproducible unless you use a config like:

io-threads 8 # 使用8个线程
save ""
protected-mode no
bind 0.0.0.0

Stress like this with -P. Default benchmark is ping-pong: send next only after the reply arrives.

With -P pipelining, 32 GETs go together; the server parses them all, then replies.

./src/valkey-benchmark -h 10.0.0.1 -p 6379 -d 256 --threads 16 -c 100 -P 32 -n 20000000 -t get --rdma

1. Production forces pipelining for throughput — so this test matters!

In high-performance networking the biggest killer is RTT. Ping-pong (one command, one reply) can spend ~100µs on the wire even if the server takes 1µs. To hit millions of QPS, real systems (cache warm, bulk write) force Pipelining — dozens/hundreds of commands in one socket write.

So Valkey must handle Pipelining perfectly — a core capability. Crashing on Pipelining is a P0.

2. How does cmd_queue work?

If Pipelining delivers many commands at once, how does the server handle them? Enter the crash protagonist: cmd_queue.

After Valkey 9.0’s new I/O multithread model:

NIC receive: RDMA DMAs a blob with 32 GETs into memory.

I/O thread parse: an I/O thread wakes, reads that memory, lexes into 32 command objects, pushes onto c->cmd_queue. Then c->cmd_queue.len is 32.

Main executes: main takes over, runs the 32 commands, writes replies, clears the queue (len → 0).

So I/O and main alternate? Round-robin?

Now the error is 100% reproducible — we can fix it.

git tips
#

Scenario:

We accidentally

git add .
git commit -m "......"

Then notice an experimental-only file like valkey.conf — restore it with

git checkout HEAD^ -- valkey.conf

restore that file, then amend the last commit

git commit --amend --no-edit

Done.

RDB pitfall
#

In valkey.conf:

save "" # 那么此时就是纯缓存系统,我们不会保存RDB文件.

Suppose:

save 3600 1 # 假如3600s之内,有一个key发生了变化,我们就fork来触发一个快照的任务.

RDMA perf testing hits this: if one branch’s test leaves dump.rdb (gitignored), switching branches still carries the file; server loads it on start and segfaults. (Would a random RDB still break a benchmark like ./src/valkey-benchmark -h 10.0.0.1 -p 6379 -d 256 --threads 16 -c 100 -P 32 -n 20000000 -t get --rdma under normal conditions?)

Another issue I found — file later / open an issue.

Fixing two CI crashes
#

Multithread crash fixed, but CI still shows two crashes. I suspect pizhenwei’s approach still has holes — yes, the state machine still has gaps.

Valkey tests use TCL (tool command language — common for E2E).

`test-ubuntu-io-threads`
#

Failure: log ends at Starting test WAITAOF when replica switches between masters, fsync: no, then the test client gets I/O error reading the reply — connection dropped / state corrupted while waiting.

Re-run several times:

for i in {1..100}; do
  echo "run $i"
  ./runtest --io-threads --accurate --verbose --dump-logs --single unit/wait || break
done

./runtest \
  --io-threads \
  --accurate \
  --verbose --verbose \
  --dump-logs \
  --dont-clean \
  --stop \
  --tags network \
  --single unit/wait \
  --only "WAITAOF when replica switches between masters, fsync: no" \
  --loops 500

Looks like replication path?

Cannot reproduce locally at all — Linux version dependent?

Understand CI: it merges your change onto latest, builds, and tests whether the merge breaks anything.

How to reproduce CI locally? See https://github.com/valkey-io/valkey/blob/unstable/.github/workflows/daily.yml for the workflow.

For example:

 test-ubuntu-io-threads:
    runs-on: ubuntu-latest
    if: |
      (github.event_name == 'workflow_call' || github.event_name == 'workflow_dispatch' ||
        (github.event_name == 'schedule' && github.repository == 'valkey-io/valkey') ||
        (github.event_name == 'pull_request' &&
          (github.event.pull_request.base.ref != 'unstable' ||
            contains(github.event.pull_request.labels.*.name, 'run-extra-tests')) &&
          (github.event.action != 'labeled' ||
            (github.event.pull_request.base.ref == 'unstable' && github.event.label.name == 'run-extra-tests'))
        )) &&
      !contains(github.event.inputs.skipjobs, 'iothreads')
    timeout-minutes: 1440
    steps:
      - name: prep
        if: github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call'
        run: |
          echo "GITHUB_REPOSITORY=${{inputs.use_repo || github.event.inputs.use_repo}}" >> $GITHUB_ENV
          echo "GITHUB_HEAD_REF=${{inputs.use_git_ref || github.event.inputs.use_git_ref}}" >> $GITHUB_ENV
          echo "skipjobs: ${{github.event.inputs.skipjobs}}"
          echo "skiptests: ${{github.event.inputs.skiptests}}"
          echo "test_args: ${{github.event.inputs.test_args}}"
          echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
        with:
          repository: ${{ inputs.use_repo || github.event.inputs.use_repo || github.repository }}
          ref: ${{ inputs.use_git_ref || github.event.inputs.use_git_ref || github.ref }}
      - name: Install libbacktrace
        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
        with:
          repository: ianlancetaylor/libbacktrace
          ref: b9e40069c0b47a722286b94eb5231f7f05c08713
          path: libbacktrace
      - run: cd libbacktrace && ./configure && make && sudo make install
      - name: make
        run: make SERVER_CFLAGS='-Werror' USE_LIBBACKTRACE=yes
      - name: testprep
        run: sudo apt-get install tcl8.6 tclx
      - name: test
        if: true && !contains(github.event.inputs.skiptests, 'valkey')
        run: ./runtest --io-threads --accurate --verbose --tags network --dump-logs ${{github.event.inputs.test_args}}
      - name: cluster tests
        if: true && !contains(github.event.inputs.skiptests, 'cluster')
        run: ./runtest-cluster --io-threads ${{github.event.inputs.cluster_test_args}}

That runs on Ubuntu — best: Ubuntu docker with matching env, then build/test.

Ubuntu Docker + a temporary worktree + compile/run tests the daily.yml way.

Ubuntu container for testing
#

# 在宿主机上准备worktree,防止产生影响
cd /home/ada/Project/valkey
git fetch upstream
git worktree add ../valkey-ci-repro fix-rdma-io-threads
cd ../valkey-ci-repro
git merge upstream/unstable

Start an Ubuntu container:

# 容器内部操作
docker run --rm -it \
  --name valkey-ci-repro \
  -v "$PWD":/work \
  -w /work \
  ubuntu:24.04 \
  bash
# 安装基础环境
apt-get update
DEBIAN_FRONTEND=noninteractive apt-get install -y \
  build-essential \
  pkg-config \
  git \
  wget \
  ca-certificates \
  tcl8.6 \
  tclx \
  libssl-dev
# 之后模拟workflow的操作
cd /tmp
git clone https://github.com/ianlancetaylor/libbacktrace.git
cd libbacktrace
git checkout b9e40069c0b47a722286b94eb5231f7f05c08713
./configure
make -j"$(nproc)"
make install
ldconfig
# 然后回仓库,按照CI的参数重新编译
cd /work
make distclean
make SERVER_CFLAGS='-Werror' USE_LIBBACKTRACE=yes -j"$(nproc)"

# 之后再使用容器的话可以这样
docker start -ai valkey-ci-repro

# 如果已经在后台运行的话
docker exec -it valkey-ci-repro bash

Still cannot reproduce locally even simulating this?

Later we can add a Dockerfile:

FROM ubuntu:24.04

RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y \
    build-essential \
    pkg-config \
    git \
    wget \
    ca-certificates \
    tcl8.6 \
    tclx \
    libssl-dev \
 && rm -rf /var/lib/apt/lists/*

RUN cd /tmp \
 && git clone https://github.com/ianlancetaylor/libbacktrace.git \
 && cd libbacktrace \
 && git checkout b9e40069c0b47a722286b94eb5231f7f05c08713 \
 && ./configure \
 && make -j"$(nproc)" \
 && make install \
 && ldconfig

WORKDIR /work

Cannot find it — park for now.

Still hunting state-machine timing holes.

crash2: TLS hang
#

~/Project/valkey fix-rdma-io-threads* 2m 17s ❯ ./runtest --io-threads --tls --accurate --verbose --dump-logs --single unit/wait

This test is problematic.

It hung — how to debug:

ps aux | grep valkey
# aux: a 表示显示所有用户的进程，u 表示以面向用户的详细格式输出（包含 CPU、内存占用等），x 表示显示没有控制终端的后台进程。

TLS build notes:

# 编译条件,注意写一些CI
make BUILD_TLS=yes SERVER_CFLAGS='-Werror' USE_LIBBACKTRACE=yes

# 生成测试证书
./utils/gen-test-certs.sh

# 检查TLS
tclsh <<< 'package require tls; puts OK'

Interesting contrast experiment:

Baseline:

./runtest --accurate --verbose --verbose --dump-logs --single unit/protocol

TLS only:

./runtest --tls --accurate --verbose --verbose --dump-logs --single unit/protocol

io-threads only:

./runtest --io-threads --accurate --verbose --verbose --dump-logs --single unit/protocol

4.TLS + io-threads：

./runtest --io-threads --tls --accurate --verbose --verbose --dump-logs --single unit/protocol

Only the fourth group fails — state-machine handoff bug.

Technique: understand timing differences between TLS and plain TCP state machines.

CI reported 24 errors?
#

Pick one CI run: https://github.com/valkey-io/valkey/actions/runs/23701289010/job/69045307291?pr=3335

All same class of failure — why the TCL test hangs:

test_slave_buffers {slave buffer are counted correctly} 1000000 10 0 1

Params: cmd_count=1000000, payload_len=10, limit_memory=0, pipeline=1

Full flow:

Start master + replica
Create 100 keys, 100KB each (~10MB)
Replica connects and finishes full sync
SIGSTOP the replica (pause_process $slave_pid) — fully frozen, no read/write
Create a valkey_deferring_client to master
Tight loop: send 1,000,000 SETRANGE key:0 0 AAAAAAAAAA (pipeline — send only)
Then read 1,000,000 replies
Check master’s mem_clients_slaves reflects the backed-up replica output buffer

TCP flow-control deadlock
#

Two independent TCP streams:

Stream A: Client → Server (commands)

Tcl → write() → Client kernel send buffer → [TCP] → Server kernel recv buffer → read() → Valkey Server

Stream B: Server → Client (replies)

Valkey Server → write() → Server kernel send buffer → [TCP] → Client kernel recv buffer → [nobody reading!]

Key: TCP receive window. When the receiver’s recv buffer fills, the sender cannot send more.

How the deadlock forms
#

Phase 1: normal

Client sends commands; Server reads and processes
Server generates replies (~5 bytes :10\r\n per SETRANGE) into the kernel send buffer
TCP delivers replies into the Client’s kernel recv buffer

Phase 2: Client recv buffer fills

Client never calls read() in its tight for-loop
Client TCP recv buffer keeps accumulating replies
Linux TCP recv buffer starts ~128KB (default of net.ipv4.tcp_rmem)
Auto-tuning can grow to ~6MB, but only if the app actively reads
Because Client does not read, auto-tuning barely grows the buffer

Phase 3: deadlock

When Client recv buffer is full (~128KB, ~26k replies):

Server → Client:

Client kernel recv full → TCP window 0 → Server write() returns EAGAIN

Server replies pile up in userspace client->reply

Client → Server:

Theoretically independent — but kernel behavior intervenes…

Critical Linux TCP behavior: full-duplex TCP still shares per-connection memory management. Heavy unconsumed data one way can constrain buffers the other way via tcp_mem / tcp_moderate_rcvbuf.

Note: Linux send/recv buffers share pressure accounting even though TCP is full duplex.

More direct: when Server cannot flush replies, its TCP send buffer fills; congestion/memory management can indirectly hurt receive-window advertisements on the same connection.

End result:

Client’s flush $fd (blocking write) stalls — Client kernel send buffer full
Server cannot write replies — Client kernel recv buffer full
Each waits for the other → deadlock

Why flaky (sometimes hangs, sometimes not)
#

Deadlock depends on precise timing:

Client send rate: ~54 bytes/cmd; flush is blocking write(). Fast CI runners may finish 1M cmds in 10–20s.
Server process rate: event loop read → execute → reply. If Server keeps recv-buffer headroom until Client finishes sending, no deadlock.
TCP buffer sizes: tcp_rmem / tcp_wmem / tcp_mem differ across CI runners.
32-bit extras: heavier allocs; slower Tcl; wider deadlock window.

This explains:

upstream #8453 test-ubuntu-32bit passed (47m44s) — faster runner finished sends before the window closed
your PR’s test-ubuntu-32bit failed (timeout ~22m) — slower runner hit the deadlock

In short: if processing is too slow, this deadlock can appear.

Still more CI errors to handle
#

Not sure if flaky or frequent.

docker run --rm -it \
  -v /home/ada/Project/valkey:/valkey \
  -w /valkey \
  alpine:latest sh -euxc '
  apk add --no-cache build-base git tcl procps tclx
  git clone --depth 1 https://github.com/ianlancetaylor/libbacktrace.git /tmp/libbacktrace
  cd /tmp/libbacktrace && git fetch --depth 1 origin b9e40069c0b47a722286b94eb5231f7f05c08713 && git checkout b9e40069c0b47a722286b94eb5231f7f05c08713
  cd /tmp/libbacktrace && ./configure && make -j"$(nproc)" && make install
  apk add --no-cache openssl-dev pkgconf # 加上openssl,装上开发依赖
  cd /valkey && make SERVER_CFLAGS="-Werror" USE_JEMALLOC=no CFLAGS=-DUSE_MALLOC_USABLE_SIZE USE_LIBBACKTRACE=yes -j"$(nproc)"
  cd /valkey && ./runtest --verbose --dump-logs
  '

Reproducing CI crashes across Linux versions — summary
#

Docker pulls missing images itself.

$ docker ps # 列出容器
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
$ docker ps -a # 列出所有容器,包括已经停止的
CONTAINER ID   IMAGE                  COMMAND       CREATED        STATUS                      PORTS     NAMES
c3146944eaa9   debian:bookworm-slim   "/bin/bash"   3 months ago   Exited (130) 3 months ago             modest_allen

To keep state after stopping the container:

docker run -it --name valkey-alpine \
  -v /home/ada/Project/valkey:/valkey \
  -w /valkey \
  alpine:latest sh

Later (if container is running):

docker start -ai valkey-alpine

If container is Exited:

docker start valkey-alpine
docker exec -it valkey-alpine sh

Tests passed — maybe only remote issues remain.

Watch for mount permission issues
#

Earlier builds in Docker (Alpine) with v /home/ada/Project/valkey:/valkey run as root and create root-owned artifacts under the mount, e.g. deps/libvalkey/obj/*, deps/libvalkey/lib/libvalkey.a.
On the host as a normal user, ./runtest --tls triggers make, which cleans deps/libvalkey. Logs already show:
```
rm: cannot remove 'obj/sockcompat.d': Permission denied

...

make[3]: *** [Makefile:312: clean] Error 1
```
Meaning root-owned artifacts cannot be deleted — clean incomplete.
Then building TLS libvalkey:
```
fatal error: opening dependency file obj/tls.d: Permission denied
```
Classic: deps/libvalkey/obj or files inside still root-writable only; current user cannot write .d files.
Later je_mallctl / je_malloc_usable_size implicit decls are usually fallout from a dirty deps tree / mismatched .make-settings vs built jemalloc. Fix ownership and clean deps, then full make — they usually vanish.

How to fix (on the host)
Pick one reliable approach:

Option A: fix ownership only (recommended; leave git-tracked files alone)

sudo chown -R "$(id -un):$(id -gn)" /home/ada/Project/valkey/deps/libvalkey/obj \
  /home/ada/Project/valkey/deps/libvalkey/lib

If more Permission denied, chown the whole tree once (note: changes ownership of all build artifacts):

sudo chown -R "$(id -un):$(id -gn)" /home/ada/Project/valkey

Then:

cd /home/ada/Project/valkey && make distclean && make BUILD_TLS=yes -j"$(nproc)"

Option B: map your user in Docker later (prevent recurrence)

docker run 时加 --user "$(id -u):$(id -g)"

(sometimes also $HOME) so new files under the mount are your UID; host builds won’t hit permissions.

Still a UAF?
#

I tried adding a test to test for use-after-free (took AI’s help) where some clients tried to send ping, and in the same batch there was a kill command issued.

Test: [a1b19c5](https://github.com/valkey-io/valkey/commit/a1b19c52a5946568343760f7df10d0179c36264d)

This test failed with ASAN - https://github.com/sarthakaggarwal97/valkey/actions/runs/24042292630/job/70116805349#step:7:530

Can we check if this is something we should investigate. The tests seems legitimate.

Try local reproduction first
#

We added a new TCL test and hit issues — add the test into our TCL suite first.

proc stress_same_batch_client_kill_on_handled_clients {} {
    set server_pid [s process_id]
    set victim_count 16
    set iterations 100

    for {set iter 0} {$iter < $iterations} {incr iter} {
        for {set i 0} {$i < $victim_count} {incr i} {
            set victim($i) [valkey_deferring_client]
            $victim($i) client id
        }
        for {set i 0} {$i < $victim_count} {incr i} {
            set victim_id($i) [$victim($i) read]
        }
        set killer [valkey_deferring_client]

        # Build one late CLIENT KILL command that can synchronously free many
        # already-handled clients in the same batch. If the handled_clients
        # post-pass still dereferences those raw client pointers, ASAN should
        # catch it.
        set kill_args [list kill id]
        for {set i 0} {$i < $victim_count} {incr i} {
            lappend kill_args $victim_id($i)
        }

        # Queue all victim reads first, then queue the killer command while the
        # server is stopped so they are eligible for the same IO-thread batch.
        pause_process $server_pid
        for {set i 0} {$i < $victim_count} {incr i} {
            $victim($i) ping
            $victim($i) flush
        }
        $killer client {*}$kill_args
        $killer flush
        resume_process $server_pid

        assert_equal $victim_count [$killer read]

        for {set i 0} {$i < $victim_count} {incr i} {
            catch {$victim($i) read}
            catch {$victim($i) close}
            unset victim($i)
            unset victim_id($i)
        }
        $killer close

        # Keep the loop making forward progress so sanitizer failures point at
        # the batching window instead of a later idle teardown.
        assert_equal {PONG} [r ping]
    }
}

start_server {config "minimal.conf" tags {"external:skip" "valgrind:skip"} overrides {enable-debug-command {yes} io-threads 5}} {
    # Skip if non io-threads mode - as it is relevant only for io-threads mode
    assert_equal {io-threads 5} [r config get io-threads]
@@ -102,3 +153,9 @@ start_server {config "minimal.conf" tags {"external:skip" "valgrind:skip"} overr
        }
    }
}

start_server {config "minimal.conf" tags {"external:skip" "valgrind:skip"} overrides {io-threads 2 events-per-io-thread 0 use-exit-on-panic yes crash-memcheck-enabled no}} {
    test {ASAN canary for same-batch CLIENT KILL vs handled_clients post-pass} {
        stress_same_batch_client_kill_on_handled_clients
    }
}

Root cause analysis:

  1. CLIENT KILL 在批处理中执行: 当在同一个批次中,有些客户端发送 PING,同时有一个 CLIENT KILL 命令杀死这些客户端时
  2. 客户端被添加到 handled_clients: 在 processClientsCommandsBatch 中 (memory_prefetch.c:250-252):
  if (beforeNextClient(c) == C_OK) {
      listAddNodeTail(handled_clients, c);
  }
  3. 客户端在批处理中被 CLIENT KILL 释放: CLIENT KILL 命令会调用 freeClient(),直接释放那些 victim 客户端
  4. UAF 发生: 在 processIOThreadsReadDone 最后的循环中 (networking.c:6456-6460):
  while ((handled_ln = listNext(&handled_li))) {
      client *c = listNodeValue(handled_ln);  // c 已经被释放!
      if (!c->conn) continue;  // <-- ASAN 在这里触发 heap-use-after-free
      connUpdateState(c->conn);
  }

Then build:

# 先清理之前的编译缓存，防止残留对象文件干扰
make distclean

# 使用 ASAN 选项编译，强制使用 libc 内存分配器
make SANITIZER=address -j$(nproc)

# 还要防止OOM杀死进程
sudo sysctl vm.overcommit_memory=1

MALLOC=libc: without this Valkey links jemalloc, and ASAN intercepts can fail or crash build/run.
fno-omit-frame-pointer: keep frame pointers so ASAN prints a deep, clear call stack — like the log you pasted — to pin the exact line.

For dlopen, ensure the dynamic library path is set.

export LD_LIBRARY_PATH=/home/ada/Project/valkey/src/modules/lua

Then run a single test case:

./runtest --single unit/io-threads

~/Project/valkey fix-rdma-io-threads* 4m 22s ❯ ./runtest --single unit/io-threads
Cleanup: may take some time... OK
Starting test server at port 21079
[ready]: 276318
Testing unit/io-threads
[ok]: Force the use of IO threads and assert active IO thread usage (1194 ms)
[err]: Sanitizer error: =================================================================
==276399==ERROR: AddressSanitizer: heap-use-after-free on address 0x7ce588be6388 at pc 0x55bc4c205b5f bp 0x7ffcdada6970 sp 0x7ffcdada6960
READ of size 8 at 0x7ce588be6388 thread T0
    #0 0x55bc4c205b5e in processIOThreadsReadDone /home/ada/Project/valkey/src/networking.c:6458
    #1 0x55bc4c31c9f1 in beforeSleep /home/ada/Project/valkey/src/server.c:1945
    #2 0x55bc4bf6a5ef in aeProcessEvents /home/ada/Project/valkey/src/ae.c:426
    #3 0x55bc4bf6a5ef in aeMain /home/ada/Project/valkey/src/ae.c:543
    #4 0x55bc4bf39efd in main /home/ada/Project/valkey/src/server.c:7687
    #5 0x7f8589dee6c0  (/usr/lib/libc.so.6+0x276c0) (BuildId: 7a8d41a2df4fde040b4c6ac2832311ab645a1e41)
    #6 0x7f8589dee7f8 in __libc_start_main (/usr/lib/libc.so.6+0x277f8) (BuildId: 7a8d41a2df4fde040b4c6ac2832311ab645a1e41)
    #7 0x55bc4bf3c494 in _start (/home/ada/Project/valkey/src/valkey-server+0x158494) (BuildId: 4805b42524cc62566a136ec6bcf2de4751ee59df)

0x7ce588be6388 is located 8 bytes inside of 576-byte region [0x7ce588be6380,0x7ce588be65c0)
freed by thread T0 here:
    #0 0x7f858a31f79d  (/usr/lib/libasan.so.8+0x11f79d) (BuildId: 0b96d08695bbce2da9d4770c29ad2e72fb536f47)
    #1 0x55bc4c207064 in freeClient /home/ada/Project/valkey/src/networking.c:2168
    #2 0x55bc4c210ea8 in freeClient /home/ada/Project/valkey/src/networking.c:2055
    #3 0x55bc4c210ea8 in clientKillCommand /home/ada/Project/valkey/src/networking.c:5262
    #4 0x55bc4c3335cf in call /home/ada/Project/valkey/src/server.c:3882
    #5 0x55bc4c33becb in processCommand /home/ada/Project/valkey/src/server.c:4568
    #6 0x55bc4c1eda61 in processCommandAndResetClient /home/ada/Project/valkey/src/networking.c:3808
    #7 0x55bc4c1eda61 in processPendingCommandAndInputBuffer /home/ada/Project/valkey/src/networking.c:3841
    #8 0x55bc4c130e12 in processClientsCommandsBatch /home/ada/Project/valkey/src/memory_prefetch.c:248
    #9 0x55bc4c130e12 in processClientsCommandsBatch /home/ada/Project/valkey/src/memory_prefetch.c:231
    #10 0x55bc4c2057c1 in processIOThreadsReadDone /home/ada/Project/valkey/src/networking.c:6451
    #11 0x55bc4c31c9f1 in beforeSleep /home/ada/Project/valkey/src/server.c:1945
    #12 0x55bc4bf6a5ef in aeProcessEvents /home/ada/Project/valkey/src/ae.c:426
    #13 0x55bc4bf6a5ef in aeMain /home/ada/Project/valkey/src/ae.c:543
    #14 0x55bc4bf39efd in main /home/ada/Project/valkey/src/server.c:7687
    #15 0x7f8589dee6c0  (/usr/lib/libc.so.6+0x276c0) (BuildId: 7a8d41a2df4fde040b4c6ac2832311ab645a1e41)
    #16 0x7f8589dee7f8 in __libc_start_main (/usr/lib/libc.so.6+0x277f8) (BuildId: 7a8d41a2df4fde040b4c6ac2832311ab645a1e41)
    #17 0x55bc4bf3c494 in _start (/home/ada/Project/valkey/src/valkey-server+0x158494) (BuildId: 4805b42524cc62566a136ec6bcf2de4751ee59df)

previously allocated by thread T0 here:
    #0 0x7f858a320cb5 in malloc (/usr/lib/libasan.so.8+0x120cb5) (BuildId: 0b96d08695bbce2da9d4770c29ad2e72fb536f47)
    #1 0x55bc4c479589 in ztrymalloc_usable_internal /home/ada/Project/valkey/src/zmalloc.c:156
    #2 0x55bc4c479589 in valkey_malloc /home/ada/Project/valkey/src/zmalloc.c:185
    #3 0x55bc4c1cfd99 in createClient /home/ada/Project/valkey/src/networking.c:282
    #4 0x55bc4c1d5347 in acceptCommonHandler /home/ada/Project/valkey/src/networking.c:1835
    #5 0x55bc4c36312c in connSocketAcceptHandler /home/ada/Project/valkey/src/socket.c:333
    #6 0x55bc4bf6a6d4 in aeProcessEvents /home/ada/Project/valkey/src/ae.c:486
    #7 0x55bc4bf6a6d4 in aeMain /home/ada/Project/valkey/src/ae.c:543
    #8 0x55bc4bf39efd in main /home/ada/Project/valkey/src/server.c:7687
    #9 0x7f8589dee6c0  (/usr/lib/libc.so.6+0x276c0) (BuildId: 7a8d41a2df4fde040b4c6ac2832311ab645a1e41)
    #10 0x7f8589dee7f8 in __libc_start_main (/usr/lib/libc.so.6+0x277f8) (BuildId: 7a8d41a2df4fde040b4c6ac2832311ab645a1e41)
    #11 0x55bc4bf3c494 in _start (/home/ada/Project/valkey/src/valkey-server+0x158494) (BuildId: 4805b42524cc62566a136ec6bcf2de4751ee59df)

SUMMARY: AddressSanitizer: heap-use-after-free /home/ada/Project/valkey/src/networking.c:6458 in processIOThreadsReadDone
Shadow bytes around the buggy address:
  0x7ce588be6100: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x7ce588be6180: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x7ce588be6200: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x7ce588be6280: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
  0x7ce588be6300: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x7ce588be6380: fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x7ce588be6400: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x7ce588be6480: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x7ce588be6500: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x7ce588be6580: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
  0x7ce588be6600: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==276399==ABORTING


Logged sanitizer errors (pid 276399):
=================================================================
==276399==ERROR: AddressSanitizer: heap-use-after-free on address 0x7ce588be6388 at pc 0x55bc4c205b5f bp 0x7ffcdada6970 sp 0x7ffcdada6960
READ of size 8 at 0x7ce588be6388 thread T0
    #0 0x55bc4c205b5e in processIOThreadsReadDone /home/ada/Project/valkey/src/networking.c:6458
    #1 0x55bc4c31c9f1 in beforeSleep /home/ada/Project/valkey/src/server.c:1945
    #2 0x55bc4bf6a5ef in aeProcessEvents /home/ada/Project/valkey/src/ae.c:426
    #3 0x55bc4bf6a5ef in aeMain /home/ada/Project/valkey/src/ae.c:543
    #4 0x55bc4bf39efd in main /home/ada/Project/valkey/src/server.c:7687
    #5 0x7f8589dee6c0  (/usr/lib/libc.so.6+0x276c0) (BuildId: 7a8d41a2df4fde040b4c6ac2832311ab645a1e41)
    #6 0x7f8589dee7f8 in __libc_start_main (/usr/lib/libc.so.6+0x277f8) (BuildId: 7a8d41a2df4fde040b4c6ac2832311ab645a1e41)
    #7 0x55bc4bf3c494 in _start (/home/ada/Project/valkey/src/valkey-server+0x158494) (BuildId: 4805b42524cc62566a136ec6bcf2de4751ee59df)

0x7ce588be6388 is located 8 bytes inside of 576-byte region [0x7ce588be6380,0x7ce588be65c0)
freed by thread T0 here:
    #0 0x7f858a31f79d  (/usr/lib/libasan.so.8+0x11f79d) (BuildId: 0b96d08695bbce2da9d4770c29ad2e72fb536f47)
    #1 0x55bc4c207064 in freeClient /home/ada/Project/valkey/src/networking.c:2168
    #2 0x55bc4c210ea8 in freeClient /home/ada/Project/valkey/src/networking.c:2055
    #3 0x55bc4c210ea8 in clientKillCommand /home/ada/Project/valkey/src/networking.c:5262
    #4 0x55bc4c3335cf in call /home/ada/Project/valkey/src/server.c:3882
    #5 0x55bc4c33becb in processCommand /home/ada/Project/valkey/src/server.c:4568
    #6 0x55bc4c1eda61 in processCommandAndResetClient /home/ada/Project/valkey/src/networking.c:3808
    #7 0x55bc4c1eda61 in processPendingCommandAndInputBuffer /home/ada/Project/valkey/src/networking.c:3841
    #8 0x55bc4c130e12 in processClientsCommandsBatch /home/ada/Project/valkey/src/memory_prefetch.c:248
    #9 0x55bc4c130e12 in processClientsCommandsBatch /home/ada/Project/valkey/src/memory_prefetch.c:231
    #10 0x55bc4c2057c1 in processIOThreadsReadDone /home/ada/Project/valkey/src/networking.c:6451
    #11 0x55bc4c31c9f1 in beforeSleep /home/ada/Project/valkey/src/server.c:1945
    #12 0x55bc4bf6a5ef in aeProcessEvents /home/ada/Project/valkey/src/ae.c:426
    #13 0x55bc4bf6a5ef in aeMain /home/ada/Project/valkey/src/ae.c:543
    #14 0x55bc4bf39efd in main /home/ada/Project/valkey/src/server.c:7687
    #15 0x7f8589dee6c0  (/usr/lib/libc.so.6+0x276c0) (BuildId: 7a8d41a2df4fde040b4c6ac2832311ab645a1e41)
    #16 0x7f8589dee7f8 in __libc_start_main (/usr/lib/libc.so.6+0x277f8) (BuildId: 7a8d41a2df4fde040b4c6ac2832311ab645a1e41)
    #17 0x55bc4bf3c494 in _start (/home/ada/Project/valkey/src/valkey-server+0x158494) (BuildId: 4805b42524cc62566a136ec6bcf2de4751ee59df)

previously allocated by thread T0 here:
    #0 0x7f858a320cb5 in malloc (/usr/lib/libasan.so.8+0x120cb5) (BuildId: 0b96d08695bbce2da9d4770c29ad2e72fb536f47)
    #1 0x55bc4c479589 in ztrymalloc_usable_internal /home/ada/Project/valkey/src/zmalloc.c:156
    #2 0x55bc4c479589 in valkey_malloc /home/ada/Project/valkey/src/zmalloc.c:185
    #3 0x55bc4c1cfd99 in createClient /home/ada/Project/valkey/src/networking.c:282
    #4 0x55bc4c1d5347 in acceptCommonHandler /home/ada/Project/valkey/src/networking.c:1835
    #5 0x55bc4c36312c in connSocketAcceptHandler /home/ada/Project/valkey/src/socket.c:333
    #6 0x55bc4bf6a6d4 in aeProcessEvents /home/ada/Project/valkey/src/ae.c:486
    #7 0x55bc4bf6a6d4 in aeMain /home/ada/Project/valkey/src/ae.c:543
    #8 0x55bc4bf39efd in main /home/ada/Project/valkey/src/server.c:7687
    #9 0x7f8589dee6c0  (/usr/lib/libc.so.6+0x276c0) (BuildId: 7a8d41a2df4fde040b4c6ac2832311ab645a1e41)
    #10 0x7f8589dee7f8 in __libc_start_main (/usr/lib/libc.so.6+0x277f8) (BuildId: 7a8d41a2df4fde040b4c6ac2832311ab645a1e41)
    #11 0x55bc4bf3c494 in _start (/home/ada/Project/valkey/src/valkey-server+0x158494) (BuildId: 4805b42524cc62566a136ec6bcf2de4751ee59df)

SUMMARY: AddressSanitizer: heap-use-after-free /home/ada/Project/valkey/src/networking.c:6458 in processIOThreadsReadDone
Shadow bytes around the buggy address:
  0x7ce588be6100: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x7ce588be6180: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x7ce588be6200: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x7ce588be6280: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
  0x7ce588be6300: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x7ce588be6380: fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x7ce588be6400: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x7ce588be6480: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x7ce588be6500: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x7ce588be6580: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
  0x7ce588be6600: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==276399==ABORTING

[exception]: Executing test client: I/O error reading reply.
I/O error reading reply
    while executing
"[srv $level "client"] {*}$args"
    (procedure "r" line 7)
    invoked from within
"r ping"
    (procedure "stress_same_batch_client_kill_on_handled_clients" line 48)
    invoked from within
"stress_same_batch_client_kill_on_handled_clients"
    ("uplevel" body line 2)
    invoked from within
"uplevel 1 $code"
    (procedure "test" line 63)
    invoked from within
"test {ASAN canary for same-batch CLIENT KILL vs handled_clients post-pass} {
        stress_same_batch_client_kill_on_handled_clients
    }"
    ("uplevel" body line 2)
    invoked from within
"uplevel 1 $code "
    (procedure "start_server" line 2)
    invoked from within
"start_server {config "minimal.conf" tags {"external:skip" "valgrind:skip"} overrides {io-threads 2 events-per-io-thread 0 use-exit-on-panic yes crash-..."
    (file "tests/unit/io-threads.tcl" line 157)
    invoked from within
"source $path"
    (procedure "execute_test_file" line 6)
    invoked from within
"execute_test_file $data"
    (procedure "test_client_main" line 10)
    invoked from within
"test_client_main $::test_server_port "

From the log above — reproduction succeeded.

Current fix approach:

git backport workflow
#

Current logic is incompatible with the new IO model, so we need a new operation: merge this into the old 9.0 tree (cherry-pick style).

E.g. see what your current branch history actually changed:

$ git log --author="Ada-Church-Closure" --oneline origin/fix-rdma-io-threads

Rebuild the branch and cherry-pick:

cd /home/ada/Project/valkey
git fetch upstream

# 在「纯净 9.0」上重建分支（名字仍用 PR 分支名，方便直接更新 PR）
git checkout -B fix-rdma-io-threads upstream/9.0

git cherry-pick 3fc639514 d8f8b3991 0dabb3dc9 df67b5d6d 0eb377393 ecf0dd743 \
  36dbf89a9 3a43bd4f5 7f139ef5c 454b41518 48611b619 a3e370894 685ed1ec4 95b637d29

Claude final retrospective
#

📋 Problem summary

Fixing Issue #3112: RDMA + I/O threads under high pipeline load (e.g. -P 32) hits two serious bugs:

Crash

Symptom: assertion c->cmd_queue.len == 0
Root cause: in processIOThreadsReadDone, connUpdateState(c->conn) ran before command execution. For RDMA that sync-pumps the CQ, re-entering parse while cmd_queue still holds many commands, violating the assert.

Hang (Lost Wakeup)

Symptom: server fully stuck
Root cause: I/O threads may leave unparsed data in querybuf due to batch limits. RDMA’s CQ is edge-triggered; returning to the event loop without draining the buffer means no further wakeups → deadlock.

🔧 Solution evolution

Initial fix (commit 3fc6395)

In src/networking.c:6378 processIOThreadsReadDone:

 // 1. 创建一个列表跟踪已处理的客户端
  list *handled_clients = listCreate();

  // 2. 延迟 connUpdateState 调用

  connUpdateState(c->conn);  // ❌ 移除过早的调用

  // 3. 收集处理过的客户端
  listAddNodeTail(handled_clients, c);

  // 4. 先批量执行命令队列
  processClientsCommandsBatch();

  // 5. 同步排空残留的 querybuf
  listRewind(handled_clients, &handled_li);
  while ((handled_ln = listNext(&handled_li))) {
      client *c = listNodeValue(handled_ln);
      if (c->querybuf && c->qb_pos < sdslen(c->querybuf)) {
          processPendingCommandAndInputBuffer(c);  // 🔑 关键：同步排空
      }
      connUpdateState(c->conn);  // ✅ 安全时机调用
  }

UAF fix (commit 0dabb3d → df67b5d)

Issue: @sarthakaggarwal97 noted storing client pointers can UAF because processPendingCommandAndInputBuffer may free the client
Attempt 1: client ID list + lookupClientByID (perf cost)
Final: change processClientsCommandsBatch / addCommandToBatchAndProcessIfFull so only live clients enter handled_clients

TLS compatibility (commits 0eb3773 + ecf0dd7)

Issue: TLS in CONN_STATE_ACCEPTING also needs state updates
Fix: in accept state, still call connUpdateState for TLS

Simplification (commit 7f139ef)

Let beforeNextClient return int directly instead of a wrapper

Cleanup (commits 48611b6 + a3e3708)

Drop unnecessary expire.h include
Minor formatting

🎯 Design takeaways

Edge-trigger awareness: RDMA CQ is edge-triggered — drain all buffered data before returning to the event loop
Reentrancy safety: connUpdateState sync-pumps CQ on RDMA — only safe when the command queue is empty
TCP/TLS compatible: harmless for TCP (slightly less epoll churn); TLS accept handled correctly
UAF defense: add to lists at the right time so freed clients are never touched

✅ Verification

Your stress script is excellent:

4 TCP + 4 RDMA benchmark processes at once
High concurrency (-c 200 -P 64)
Randomly kill some processes to simulate disconnects
Server should stay stable — no crash, no hang

📊 Current status

✅ Core tests pass (test-ubuntu-io-threads, test-ubuntu-tls-io-threads)
✅ RDMA-specific tests pass
⚠️ Two CI failures are flaky, unrelated to this change
👍 LGTM from @pizhenwei and @zuiderkwast
🎯 On the Valkey 9.0 / 9.1 backport list

💡 Assessment

High-quality fix:

Deep analysis: edge-trigger + reentrancy identified clearly
Iterative polish via code review
Backward compatible: zero impact on TCP/TLS
Solid testing: repro steps + stress script

Left: wait for maintainers on the two flaky tests, then merge.

Reproducing the multithread RDMA crash #

Bad client frames, or multithreading? #

git tips #

RDB pitfall #

Fixing two CI crashes #

test-ubuntu-io-threads #

Ubuntu container for testing #

crash2: TLS hang #

CI reported 24 errors? #

TCP flow-control deadlock #

How the deadlock forms #

Why flaky (sometimes hangs, sometimes not) #

Still more CI errors to handle #

Reproducing CI crashes across Linux versions — summary #

Watch for mount permission issues #

Still a UAF? #

Try local reproduction first #

git backport workflow #

Claude final retrospective #