RDMA Library Runtime Dynamic Loading (for client and benchmark)

Table of Contents

Two PRs:

Valkey main-repo change libvalkey upstream

We mainly cover dynamic loading of the two RDMA libraries (librdmacm, libibverbs) for client and benchmark.

Worth revisiting CSAPP’s dynamic-linking chapter later — program linking deserves a deep article of its own.

Background
#

ByteDance started Valkey Over RDMA development, but only decoupled the server side, not client and benchmark. That caused packaging pain on OpenSUSE and other Linux distros:

Description
- enable RDMA support

Comments & History
Marcus Rueckert (darix) created this request 2 months ago
Neal Gompa's avatar
Neal Gompa (Pharaoh_Atem) commented 2 months ago
target maintainer
I'm checking on this, please be patient. :)

Marcus Rueckert (darix) revoked request 2 months ago 
for some reasons it didnt just link the rdma libraries into the module but also into the main binary

Marcus Rueckert's avatar
Marcus Rueckert (darix) commented about 2 months ago
author
source maintainer
question:

in the fedora package you have split out the rdma module. in the newer versions they also link valkey-cli and valkey-benchmark with the rdma tools (probably to also test and connect via rdma) with that in mind I am not sure if we either need to split out the binaries or build valkey like this:

valkey:
owns everything but the binaries

Requires: valkey-implementation = %{version}
Suggests: valkey-no-rdma

%package no-rdma
Provides: valkey-implementation = %{version}
Conflicts: valkey-implementation
Removepathpostfixes: -no-rdma

%package rdma
Provides: valkey-implementation = %{version}
Conflicts: valkey-implementation
Removepathpostfixes: -rdma
and given that the cli and benchmark link the rdma code anyway we can also switch to rdma=yes

Neal Gompa's avatar
Neal Gompa (Pharaoh_Atem) commented about 2 months ago
target maintainer
I'll check on this with upstream, but I think we probably want to still do it the normal module way without doing duplicate builds. This might be something to fix upstream.

Marcus Rueckert's avatar
Marcus Rueckert (darix) commented about 2 months ago
author
source maintainer
i can understand that they also want the client and bencharmk to be able to talk rdma. and the module way i only would understand to avoid the dependency (7.7MB extra libraries for rdma enabled build) but if we cant get around that dependency anymore. we can also merge rdma support into the main package.

Neal Gompa's avatar
Neal Gompa (Pharaoh_Atem) commented about 2 months ago
target maintainer
I'm pretty sure this is a bug, and I'm checking with upstream about this.

Marcus Rueckert's avatar
Marcus Rueckert (darix) commented about 2 months ago
author
source maintainer
also did you see my asahi mail?

Neal Gompa's avatar
Neal Gompa (Pharaoh_Atem) commented about 2 months ago
target maintainer
No, I'll go look for it. Probably was buried over winter stuff.

OpenSUSE packagers had to pull in those two libraries — unreasonable for most users who never need RDMA (good RDMA NICs are expensive). Distro packagers filed issues asking for dynamic loading. The real change was mostly in Valkey’s submodule libvalkey: we switched to runtime loading and decoupled the hard dependency.

How static / hard linking works
#

1. Static / hard link (`USE_DLOPEN_RDMA=0`)
#

At Makefile build time:

# Makefile 里
RDMA_LDFLAGS=-lrdmacm -libverbs

Those libraries are pulled in automatically. The linker writes into valkey_rdma.so’s ELF .dynamic segment:

NEEDED  librdmacm.so.1
NEEDED  libibverbs.so.1 # 我需要这两个库,这是动态加载

At process start, ld-linux-x86-64.so.2 (the dynamic linker), before main:

mmap() maps both .so into the process VAS
Processes .rela.dyn / .rela.plt relocations
Fills GOT/PLT, resolving rdma_create_id etc. to real addresses

What are GOT (Global Offset Table) and PLT (Procedure Linkage Table)? To speed startup and save memory, Linux uses lazy binding: external function addresses are not resolved at startup, only on first call.

Flow:

First call: jump to the PLT stub for e.g. rdma_conn.

Trigger resolve: PLT hands off to the dynamic linker.

Find address: linker finds the real address in the shared library.

Update GOT: write that address into the matching GOT slot.

Run: function executes.

Later calls: PLT reads the filled GOT and jumps directly — no linker.

This is also central to security / reverse engineering (PLT/GOT Hook): rewrite a GOT slot to intercept or replace a call.

In other words, GOT/PLT get filled as needed:

高地址
┌─────────────────────────┐
│  valkey_rdma.so  .text  │  你的 RDMA 代码
├─────────────────────────┤
│  librdmacm.so.1  .text  │  rdma_create_id 等
├─────────────────────────┤
│  libibverbs.so.1  .text │  ibv_reg_mr 等
├─────────────────────────┤
│  libc.so ...
低地址

Downside: on machines without RDMA hardware/libs, ld.so fails at startup (error while loading shared libraries), even if you never call RDMA.

How we implement runtime dynamic loading
#

At compile time do not link -lrdmacm -libverbs:

  ifeq ($(USE_DLOPEN_RDMA),1)
      CFLAGS += -DDLOPEN_RDMA
      RDMA_LDFLAGS=
  else
      RDMA_LDFLAGS=-lrdmacm -libverbs
  endif

valkey_rdma.so’s .dynamic has no NEEDED entries for RDMA libs. The process starts normally; only valkeyInitiateRdma() attempts to load RDMA libraries.

What does `dlopen` do underneath?
#

Think of it as finding the .so object and returning a handle, similar to open.

How is our load logic written? This function is quite abstract:

static int rdma_dyn_load_libs(void) {
    if (rdmacm_handle && ibverbs_handle)
        return 0;

    const char *rdmacm_candidates[] = {"librdmacm.so.1", "librdmacm.so", NULL};
    const char *verbs_candidates[] = {"libibverbs.so.1", "libibverbs.so", NULL};

    for (int i = 0; !rdmacm_handle && rdmacm_candidates[i]; i++)
        rdmacm_handle = dlopen(rdmacm_candidates[i], RTLD_NOW);
    for (int i = 0; !ibverbs_handle && verbs_candidates[i]; i++)
        ibverbs_handle = dlopen(verbs_candidates[i], RTLD_NOW);

    if (!rdmacm_handle || !ibverbs_handle) {
        fprintf(stderr, "Error: Required RDMA libraries (librdmacm/libibverbs) not found on this system.\n");
        return -1;
    }

    LOAD_SYM(rdmacm_handle, rdma_create_event_channel);
    LOAD_SYM(rdmacm_handle, rdma_destroy_event_channel);
    LOAD_SYM(rdmacm_handle, rdma_create_id);
    LOAD_SYM(rdmacm_handle, rdma_create_qp);
    LOAD_SYM(rdmacm_handle, rdma_destroy_id);
    LOAD_SYM(rdmacm_handle, rdma_bind_addr);
    LOAD_SYM(rdmacm_handle, rdma_resolve_addr);
    LOAD_SYM(rdmacm_handle, rdma_resolve_route);
    LOAD_SYM(rdmacm_handle, rdma_connect);
    LOAD_SYM(rdmacm_handle, rdma_disconnect);
    LOAD_SYM(rdmacm_handle, rdma_get_cm_event);
    LOAD_SYM(rdmacm_handle, rdma_ack_cm_event);
    LOAD_SYM(rdmacm_handle, rdma_getaddrinfo);
    LOAD_SYM(rdmacm_handle, rdma_freeaddrinfo);
    LOAD_SYM(rdmacm_handle, rdma_event_str);

    LOAD_SYM(ibverbs_handle, ibv_alloc_pd);
    LOAD_SYM(ibverbs_handle, ibv_dealloc_pd);
    LOAD_SYM(ibverbs_handle, ibv_create_comp_channel);
    LOAD_SYM(ibverbs_handle, ibv_destroy_comp_channel);
    LOAD_SYM(ibverbs_handle, ibv_create_cq);
    LOAD_SYM(ibverbs_handle, ibv_destroy_cq);
    LOAD_SYM(ibverbs_handle, ibv_reg_mr);
    LOAD_SYM(ibverbs_handle, ibv_dereg_mr);
    LOAD_SYM(ibverbs_handle, ibv_get_cq_event);
    LOAD_SYM(ibverbs_handle, ibv_ack_cq_events);
    LOAD_SYM(ibverbs_handle, ibv_destroy_qp);

    return 0;
}

Syscall chain (Linux / glibc)
#

dlopen() is a glibc libdl API. Internally it roughly:

open() the .so (search LD_LIBRARY_PATH, /etc/ld.so.cache, etc.)
mmap() PT_LOAD segments into the current process address space
- Readable+executable → .text
- Readable+writable → .data / .bss / .got.plt
Parse ELF header, program headers, dynamic section
Recursively load any NEEDED deps
Relocate external symbol references
Run .init / .init_array constructors
Return a handle (often a struct link_map * in glibc’s loaded-module list)

RTLD_NOW: resolve all undefined symbols before dlopen returns; missing symbols fail immediately. RTLD_LAZY: resolve on first call (errors may surface mid-run). For a critical path like RDMA, RTLD_NOW is the right fail-fast choice.

What is the handle?
#

static void *rdmacm_handle = NULL;
static void *ibverbs_handle = NULL;

These void * are not file descriptors — they are handles into the dynamic linker’s data for already-mapped SOs. Later dlsym(rdmacm_handle, "xxx") searches only that SO (and its loaded deps).

We dlopen two libraries because rdma_* lives in librdmacm.so.1 and ibv_* in libibverbs.so.1 — different ELF objects.

What does `dlsym` do?
#

Our macro:

#define LOAD_SYM(handle, name)                                              \
    do {                                                                    \
        void *tmp_sym = dlsym(handle, #name);                               \
        if (!tmp_sym) {                                                     \
            fprintf(stderr, "RDMA Init Error: missing symbol %s\n", #name); \
            return -1;                                                      \
        }                                                                   \
        *(void **)(&rdma_syms.name) = tmp_sym;                              \
    } while (0)

Plainly: look up a symbol in the opened SO.

`dlsym(handle, "rdma_create_event_channel")`
#

String-search .dynsym of that SO
Read type (FUNC/OBJECT), binding, section
Combine with relocation results → final VA in this process
Return void * — for a function, the VA of the first instruction in .text

From the CPU’s view, later calls through the function pointer are ordinary indirect jumps:

call *(%rax); 跳转到 librdmacm.so 映射进来的 .text 地址

Different from the PLT/GOT path of ordinary dynamic linking — here we maintain our own function-pointer table; the call path is fixed at compile time.

What `*(void **)(&rdma_syms.name) = tmp_sym` does
#

C does not allow assigning void * directly to a function pointer (strict aliasing). The common pattern pivots through void **:

// 等价于：rdma_syms.rdma_create_id = (函数指针类型)dlsym(...)
*(void **)(&rdma_syms.rdma_create_id) = tmp_sym; // 这个中转也很细节

In practice: store the looked-up function address into our struct member.

Overall flow:

用户程序
   │
   ▼
valkeyInitiateRdma()                    // rdma.c:1275
   │
   ├─ [DLOPEN_RDMA] rdma_dyn_load_libs()
   │      │
   │      ├─ dlopen("librdmacm.so.1")  → mmap SO, 重定位, 返回 handle
   │      ├─ dlopen("libibverbs.so.1") → 同上
   │      │
   │      └─ 对每个 API: dlsym → 写入 rdma_syms.*
   │
   └─ valkeyContextRegisterFuncs(...)   // 注册 RDMA 连接类型

Short summary: We did not change RDMA business logic; we added a runtime PLT/GOT substitute: dlopen maps the SO into the process, dlsym manually resolves symbols, and a struct + macros redirect all API calls to a function-pointer table without changing call sites — so machines without RDMA can still load valkey_rdma.so, and only real RDMA init touches librdmacm / libibverbs.

If valkeyInitiateRdma is never called, the load path never runs.

Background #

How static / hard linking works #

1. Static / hard link (USE_DLOPEN_RDMA=0) #

How we implement runtime dynamic loading #

What does dlopen do underneath? #

Syscall chain (Linux / glibc) #

What is the handle? #

What does dlsym do? #

dlsym(handle, "rdma_create_event_channel") #

What *(void **)(&rdma_syms.name) = tmp_sym does #