The original production logs are no longer available, but the failure line was in this family:
Call to ibv_reg_mr_iova2 failed with error Cannot allocate memoryThe ibv prefix comes from InfiniBand Verbs, the userspace programming interface exposed by libibverbs. The same verbs model is also used by ConnectX adapters carrying RoCE; “verbs” names the programming interface, not necessarily the wire protocol.
The first searches led to familiar suspects: locked-memory limits, BAR1 exhaustion, driver mismatches, too many registered regions, and an unhealthy adapter. To distinguish them, we needed to understand what registration builds.
We’ll follow the address translations from a userspace buffer to the RNIC and GPU. The production failure comes in Part 2, followed by the Linux pinning conflict in Part 3.
Scope note. The hardware model here is an NVIDIA H100-class GPU and a ConnectX-7-class RDMA NIC on Linux. The interfaces are verified against the public NVIDIA Collective Communications Library (NCCL), rdma-core, Linux’s
mlx5ConnectX driver, and NVIDIA driver source. Hardware implementations change, so names such as MTT and PAS should be read as the concrete mlx5 form of a more general idea: a device-side translation from an address in a memory region to DMA-reachable pages.
1. Start with DMA, not RDMA
A CPU normally moves data with loads and stores:
CPU load from ACPU store to BCPU load from A+8CPU store to B+8...That is a terrible way to move a large packet. The CPU would spend its time acting as a copy engine.
Direct Memory Access, or DMA, gives the copy engine to the device. The CPU still sets the operation up. It allocates descriptors, tells the device where they are, rings a doorbell, and handles completion. But once the operation is running, the device issues the memory transactions itself.
setup data movement
CPU -----------------------------> device DMA engine descriptor: source, length, | destination, permissions | memory reads/writes v system memory“Direct” does not mean “the CPU has no involvement.” It means the CPU is not executing one instruction per transferred cache line.
A network adapter already needs DMA. For transmit, it reads packet bytes from memory. For receive, it writes packet bytes into memory. Conventional networking normally puts kernel-owned buffers in the middle:
application buffer | | copy / protocol processing vkernel socket buffer | | NIC DMA vnetworkRDMA changes who is allowed to name the final memory and how much of the CPU networking stack sits in the data path.
2. RDMA is network-triggered DMA into registered memory
Consider two machines, A and B. A process on B owns a buffer. It wants A to write directly into that buffer.
B cannot safely tell A, “write to virtual address 0x7f....” That address belongs to B’s process page tables. A’s NIC cannot walk them, and B may not even keep the same physical pages underneath that virtual range.
Instead, B registers the range with its local RDMA NIC—an RNIC, or RDMA-capable NIC. InfiniBand documentation often calls the same class of device a Host Channel Adapter (HCA). Registration creates a memory region, usually shortened to MR. B then gives A two important values:
remote address: where inside the MR to startrkey: the capability authorizing remote accessA posts an RDMA-write work request containing its local source buffer and B’s remote address and rkey.
RNIC registered memory region
The remote CPU does not copy the payload. It may have participated earlier—creating the queue pair, registering memory, exchanging metadata—but the data can arrive without a receive-side system call for every transfer. The registration mechanism described here is shared by InfiniBand and RDMA over Converged Ethernet (RoCE), even though their network transports differ.
RDMA lets a remote peer cause a local NIC to perform DMA against memory that was registered in advance.
3. PCIe is an addressed fabric
The GPU and RNIC are PCIe devices. It is tempting to picture PCIe as a collection of wires connecting devices to the CPU. A better model for this story is an addressed transaction fabric.
A PCIe requester can issue transactions such as:
Memory Read address=X length=NMemory Write address=Y payload=...Root ports and switches route those transactions through the PCIe topology. The destination may be system RAM, a device register window, or a peer device’s memory aperture.
A simplified machine-wide map might look like this:
system RAM RNIC / ConnectX GPU windows
The addresses differ by machine, but the routing depends on address windows assigned to devices. PCI configuration space and Base Address Registers describe those windows.
4. A PCI BAR is a request for an address window
Every PCIe function exposes configuration space. Among the standard fields are up to six Base Address Registers: BAR0 through BAR5.
A BAR describes an address window the device needs. The register stores the window’s configuration, rather than the data accessed through it.
At enumeration, firmware or the operating system performs a sizing exchange with the device, reserves an appropriately sized region in the host/PCI address map, and writes the chosen base address into the BAR. Linux then records that region as a PCI resource. Drivers can claim and map it.
PCI configuration space PCIe address space
BAR0 = 0xC000_0000 -------------------> [device window 0]BAR1 = 0x8000_0000_0000 -------------> [device window 1]BAR regions have a useful hardware constraint: their sizes are powers of two and their bases are naturally aligned to those sizes. That is why a GPU with 80 GiB of framebuffer can expose a 128 GiB BAR1 aperture. The BAR is an address-decoding window, not a byte-for-byte statement of installed VRAM. The next power-of-two aperture can contain unused address space or device-defined regions that do not correspond to usable framebuffer.
A 64-bit BAR consumes two adjacent 32-bit BAR slots because the base address itself needs 64 bits.
BAR0 and BAR1 on an NVIDIA GPU
The names are conventions tied to the device implementation, not universal PCI meanings. On NVIDIA data-center GPUs, the useful mental model is:
BAR0 control and register MMIOBAR1 aperture through which framebuffer memory can be reachedBAR0 lets software interact with the device’s control machinery. BAR1 makes selected GPU framebuffer pages visible in the PCIe address space so a CPU or peer device can access them. NVIDIA’s NVML documentation describes BAR1 as the mapping used for direct CPU or third-party-device access to framebuffer memory.
BAR1 is an address window through which the GPU exposes framebuffer mappings. It does not hold a second copy of VRAM.
GPU framebuffer pages PCIe-visible aperture slots
On a large-BAR system the aperture may be big enough to cover all framebuffer memory at once. On smaller-BAR systems the driver manages a limited window and consumes BAR1 space as peer mappings are created. nvidia-smi -q reports total, used, and free BAR1 space; NVIDIA’s GPUDirect documentation notes that mappings are managed in fixed-size chunks and that some space is reserved internally.
Resizable BAR
Classic BAR sizes are selected from capabilities exposed by the device. PCIe Resizable BAR lets software choose among multiple supported aperture sizes. Platform firmware still has to reserve enough address space—hence the familiar “Above 4G Decoding” and Resizable BAR firmware settings on systems with large GPU apertures.
None of this yet tells the RNIC which CUDA allocation it may access. BAR1 establishes that GPU memory can be represented in PCIe space. Registration establishes the exact pages, permissions, lifetime, and RNIC translation.
5. There is no single “physical address”
Most confusion in GPUDirect discussions comes from using the word “address” without naming the address space.
For this path, hold at least five different kinds of address:
1. CPU process virtual address2. CUDA / GPU virtual address3. CPU physical page number or host physical address4. DMA address visible to a particular device5. MR IOVA: the address the RNIC exposes through the memory keyThese addresses can have the same numeric value even though they belong to different address spaces.
CPU process virtual address
A normal pointer such as 0x7f23... is interpreted through a process’s CPU page tables:
CPU virtual address | | CPU page-table walk vhost physical pageLinux represents ordinary RAM pages with struct page objects and page frame numbers, or PFNs.
CUDA virtual address
A pointer returned by CUDA belongs to a CUDA-managed virtual-address space. For device memory, the CPU cannot simply walk its own page tables to discover ordinary RAM underneath it. The NVIDIA driver and GPU page tables own the mapping from the CUDA virtual range to framebuffer pages.
CUDA virtual address | | GPU/NVIDIA translation state vGPU framebuffer pagesUnified Virtual Addressing makes CPU and GPU pointers share a single-looking process address space, but it does not erase the different backing stores or page-table owners.
DMA address
Linux’s DMA API gives a device an address it can use for DMA. With the I/O Memory Management Unit (IOMMU) disabled or in passthrough, that value may closely resemble a host physical or peer PCIe address. With an IOMMU enabled, it may be an I/O virtual address translated again before the PCIe transaction reaches its target.
RNIC DMA address | | optional IOMMU translation vhost RAM or peer PCIe addressThis is why “the RDMA driver copies the GPU physical page numbers into the NIC” is too loose. The RNIC needs DMA addresses valid from that RNIC’s point of view. NVIDIA added nvidia_p2p_dma_map_pages() precisely because a peer resource’s CPU-visible physical address and a particular I/O device’s usable DMA address need not be identical.
MR IOVA
The memory region has its own externally visible address range. ibv_reg_mr_iova2() lets the caller specify the base IOVA that the RNIC should associate with the region.
mr = ibv_reg_mr_iova2(pd, addr, length, iova, access);addr identifies the userspace range whose backing memory must be registered. iova identifies the virtual base the RNIC will expose through the MR. NCCL commonly uses the same numeric value for both, but the API keeps them conceptually separate.
That distinction lets an application register one userspace range while presenting a chosen virtual base to the device.
6. What memory registration actually builds
Registration creates both access permissions and a translation object inside the RDMA stack and RNIC. Pinning the backing pages is one part of that work.
Conceptually:
Memory Region
owner: protection domain PD virtual base: IOVA length: N bytes permissions: local write, remote read, remote write, ... translations: IOVA page 0 -> DMA address A IOVA page 1 -> DMA address B IOVA page 2 -> DMA address C keys: lkey rkeyProtection domain
A protection domain, or PD, groups RDMA objects that are allowed to interact. Queue pairs and memory regions must belong to compatible protection domains. It is a software and hardware isolation boundary, not a Linux process namespace.
lkey
A local scatter/gather entry includes an address, length, and lkey. The RNIC uses the lkey to verify that the local work request is allowed to read or write that range.
rkey
A remote operation carries an rkey. The destination RNIC uses it to find the memory region, verify remote permissions and bounds, and reach the translation state.
An rkey is therefore a capability. Knowing an address without the matching key is insufficient.
MKey, MTT, and PAS
On mlx5 hardware, the kernel driver creates a memory key, or MKey. The MKey carries the access policy and points to address-translation information. Different generations and code paths use terms such as MTT—Memory Translation Table—and PAS arrays—physical-address lists—for the page translations loaded into that object.
A useful approximation is:
MKey = permissions + bounds + page geometry + pointer to translation entries
MTT/PAS entries: virtual page index 0 -> DMA page address 0 virtual page index 1 -> DMA page address 1 virtual page index 2 -> DMA page address 2The mlx5 Linux driver may create the MKey directly or use UMR—User-mode Memory Registration machinery—to load or update its translation. The implementation is more sophisticated than a flat array: it can select larger page sizes, cache MKeys, and update translations in hardware. But “the RNIC gets a protected page table for this MR” is the right first model.
The RNIC does not consult the CPU page table on every packet. Registration resolves and installs the alternate translation path ahead of time.
CPU / Linux MM translation RNIC translation state the one physical page
If Linux moved a backing page while the RNIC retained the old translation, the next packet would DMA into the wrong location. Registration therefore has to keep that mapping valid for as long as the device can use it.
7. Registering ordinary CPU memory
Now walk the ordinary host-memory path used by a classic mlx5 userspace MR.
NCCL eventually reaches its InfiniBand network plugin, which calls either ibv_reg_mr() or ibv_reg_mr_iova2(). The latter is used in the NCCL 2.17 generation when relaxed ordering is enabled.
NCCL -> libibverbs -> mlx5 userspace provider -> uverbs ioctl -> mlx5_ib kernel driverInside the kernel, mlx5_ib_reg_user_mr() asks RDMA core to obtain an ib_umem for the range:
mlx5_ib_reg_user_mr() | vib_umem_get()ib_umem_get() performs four important jobs.
7.1 Account the pin
Linux checks the process’s locked-memory allowance, represented by RLIMIT_MEMLOCK unless the process has the relevant capability. This is one reason ibv_reg_mr() can return ENOMEM even when the machine has plenty of free RAM: the word describes a resource-accounting failure, not necessarily exhausted DRAM.
7.2 Pin the pages
RDMA core calls pin_user_pages_fast() with FOLL_LONGTERM, and with FOLL_WRITE when the device may write the region.
userspace VA range | | pin_user_pages_fast(FOLL_LONGTERM) varray of struct page pointersPinning stabilizes the backing pages so they cannot be reclaimed, migrated, or replaced in ways that would invalidate the device mapping for the lifetime of the MR.
It does not make the pages physically contiguous. A 16 MiB virtual range can be backed by thousands of scattered 4 KiB pages.
7.3 Build a scatter/gather representation
RDMA core merges adjacent pages where possible into a scatter/gather table:
virtual range
page 0 -> PFN 91page 1 -> PFN 92 } one contiguous SG segmentpage 2 -> PFN 501page 3 -> PFN 9007.4 DMA-map it for the RNIC
The table is passed through the Linux DMA API for the RNIC. That step produces the DMA addresses valid for the ConnectX device, including any IOMMU mapping.
struct page / host PFN | | dma_map_sgtable(RNIC) vRNIC-visible DMA addressThe mlx5 driver then creates an MKey and loads those DMA addresses into its translation state.
The whole path is:
Linux MM state RNIC translation state capabilities returned to the caller
8. Why GPU memory needs a broker
For GPU memory, the ordinary path breaks at its first assumption.
ib_umem_get() knows how to resolve CPU virtual addresses backed by Linux-managed pages. A CUDA device pointer is backed by GPU framebuffer pages managed by the NVIDIA driver and GPU MMU. Linux RDMA core cannot call the normal GUP path and obtain an array of ordinary host struct page objects for VRAM.
The two drivers therefore need a memory-export protocol.
The legacy protocol on NVIDIA/ConnectX systems was nvidia-peermem.
9. Legacy GPUDirect registration with nvidia-peermem
nvidia-peermem registers as an RDMA peer-memory client. When the RDMA stack receives a userspace range, the peer-memory layer asks registered clients whether one of them owns it.
RDMA registration request | vpeer-memory clients | +-- nvidia-peermem: "is this an NVIDIA GPU range?"For an NVIDIA CUDA allocation, the public R525 module follows a path like this:
CUDA virtual range | | nvidia_p2p_get_pages() vNVIDIA P2P page table | | nvidia_p2p_dma_map_pages(RNIC PCI device) vRNIC-valid peer DMA addresses | vscatter/gather table | vmlx5 MKey translations9.1 Claim the range
nvidia-peermem aligns the CUDA virtual range to NVIDIA’s peer-page granularity and invokes nvidia_p2p_get_pages().
This call does more than “look up physical addresses.” It asks the NVIDIA driver to establish and hold a peer mapping for the GPU allocation and return a P2P page table describing it. The allocation must remain valid while a third-party device can access it.
The driver also has an invalidation story: if the CUDA allocation is freed or its mapping can no longer remain valid, the peer client must be notified or use a persistent-lifetime API with an explicit teardown contract.
9.2 Map for this RNIC
nvidia_p2p_dma_map_pages() takes the requesting PCI device—the ConnectX RNIC—and maps the GPU pages into addresses usable by that device.
The mapping must be valid for the requesting RNIC. A different peer device may need a different DMA address for the same GPU memory.
9.3 Hand the mappings to mlx5
nvidia-peermem converts the returned DMA addresses into a scatter/gather table. The RDMA driver can then build its MKey just as it would for any other DMA-mapped memory.
At the end, the RNIC holds translations that lead to PCIe addresses in the GPU’s peer-visible aperture.
CPU memory through the same probe
If the address is ordinary CPU memory, the NVIDIA peer client should decline ownership and the RDMA stack should continue through the normal ib_umem_get() path. In the public module, the ownership callback’s contract is effectively “one means mine, zero means not mine.”
That detail matters in debugging: seeing an NVIDIA peer-memory function reject an address does not automatically mean the MR failed. It may be the expected classification step before host registration.
10. DMA-BUF replaces the private ownership exchange
The newer GPU-registration path uses Linux DMA-BUF.
That path is gated by the whole software stack, not by the NVIDIA branch number alone. NVIDIA documents DMA-BUF GPUDirect RDMA as requiring the open kernel-module flavor, CUDA 11.7 or newer, Linux 5.12 or newer, and compatible network-driver support. NCCL 2.17 then probes support at runtime: the network plugin must expose regMrDmaBuf, the CUDA driver and GPU must report DMA-BUF capability, and the selected network device must advertise NCCL_PTR_DMABUF. If any gate fails, installing R535 does not by itself switch the buffer to DMA-BUF registration.
DMA-BUF is a kernel framework for sharing a memory object between drivers. One driver exports the object as a file descriptor. Another driver imports it, attaches its device, and asks for a DMA mapping.
For GPUDirect RDMA:
NVIDIA driver mlx5 RDMA driver
exports GPU allocation imports DMA-BUF fdas DMA-BUF fd ---------> attaches RNIC maps attachment receives SG tableAt userspace, NCCL can obtain a DMA-BUF file descriptor for a supported CUDA allocation and call:
ibv_reg_dmabuf_mr(pd, offset, length, iova, fd, access)The kernel path is conceptually:
ibv_reg_dmabuf_mr() | vmlx5_ib_reg_user_mr_dmabuf() | vib_umem_dmabuf_get() | +-- dma_buf_get(fd) +-- dma_buf_dynamic_attach(RNIC) +-- dma_buf_pin() +-- dma_buf_map_attachment() | vexporter-provided DMA SG table | vmlx5 MKeyThe NVIDIA driver remains responsible for the GPU allocation and its peer mapping. DMA-BUF does not remove the GPU driver; it replaces the special peer-memory-client handshake with a standard exporter/importer lifetime model.
The ownership and lifetime handling change as follows:
legacy:RDMA subsystem <-> out-of-tree/private peer-memory client <-> NVIDIA driver
DMA-BUF:RDMA importer <-> standard DMA-BUF framework <-> NVIDIA exporterIt also gives the kernel a standard place for attachment, reservation fences, invalidation, and unmapping.
What DMA-BUF does not change
DMA-BUF only applies to buffers that are exported through it. In stock NCCL 2.17, CUDA protocol buffers may take the DMA-BUF path, while host buffers continue through ordinary ibv_reg_mr_iova2().
Part 2 follows a failure in that ordinary host-memory path.
11. One RDMA write, all the way to VRAM
We can now follow a remote write into a registered GPU buffer.
Assume machine B has registered a GPU receive buffer and sent its address and rkey to machine A.
Step 1: A posts a work request
A’s software creates a work queue entry:
local source: address + length + lkey
remote destination: address + rkey
operation: RDMA WRITEThe lkey authorizes A’s RNIC to read the local source. The rkey will authorize B’s RNIC to write the remote destination.
Step 2: A’s RNIC reads the source
A’s RNIC resolves the local lkey, translates the local address, and DMA-reads the payload from A’s memory—possibly GPU memory on the sending side as well.
Step 3: packets cross the network
The RDMA transport carries the operation, destination virtual address, key, and payload according to the wire protocol. Large writes are segmented across packets.
Step 4: B’s RNIC validates the capability
B’s RNIC looks up the rkey and checks:
Does this key exist?Does it permit remote write?Is the requested range within the MR?Is the queue pair allowed to use it?A random address and guessed key should not be enough.
Step 5: the RNIC translates the MR address
The remote address must fall inside the MR’s IOVA range. The RNIC subtracts the MR base, applies the page geometry, and uses the MKey’s translation state to obtain one or more DMA addresses.
remote VA / MR IOVA | | MKey lookup | bounds + permissions vMTT/PAS entry | vRNIC DMA addressStep 6: the RNIC issues PCIe writes
For a GPU MR, the DMA address targets the peer-visible GPU aperture. The RNIC becomes a PCIe requester and emits memory-write transactions. At the PCIe protocol layer, those requests travel as Transaction Layer Packets (TLPs).
RNIC fast path GPU aperture and memory
Step 7: the GPU consumes the data
Transport completion and GPU visibility are related but not identical. PCIe posted writes, GPU cache/coherency rules, and CUDA synchronization determine when a GPU kernel may safely consume the new bytes. NCCL and CUDA contain explicit ordering mechanisms for this boundary.
The drivers establish the translations during registration. Each arriving packet uses the RNIC’s MKey and existing PCIe mappings, without asking the NVIDIA driver to translate it again.
12. Why BAR1 was such a plausible suspect
When a GPUDirect registration fails, BAR1 is a reasonable hypothesis.
Registration can consume BAR1 mapping space. Small-BAR GPUs can exhaust the aperture. Firmware can misassign a large BAR. A topology may not support peer routing. An IOMMU configuration may block the intended P2P mapping. NVIDIA’s own documentation recommends checking nvidia-smi -q and platform large-BAR support.
But BAR1 is only one resource in a long transaction:
userspace range -> allocation owner identified -> lifetime stabilized -> pages resolved -> peer mapping created -> DMA addresses produced -> SG table built -> MKey allocated -> translations loadedENOMEM can escape from many of those steps:
locked-memory accountingkernel allocation failurepage-pinning failurepage migration failureDMA mapping failureBAR aperture exhaustionMKey/cache exhaustionprovider bookkeeping failureThe top-level NCCL line does not preserve which one happened.
We still needed to find which step had returned the error in production.
13. The machine model to carry forward
The three registration paths differ in how they obtain the backing memory:
Ordinary host memory
process VA -> pin Linux pages long-term -> DMA-map for RNIC -> load MKey translationsGPU memory through nvidia-peermem
CUDA VA -> NVIDIA P2P page table -> map GPU pages for RNIC -> load MKey translationsGPU memory through DMA-BUF
CUDA allocation -> NVIDIA exports DMA-BUF -> mlx5 attaches and maps -> load MKey translationsAll three end in the same fast-path promise:
Given this key and an address in this region, the RNIC can reach stable DMA destinations without consulting a process page table.
Part 2 begins where that model misled us. The failing NCCL connection used GPUDirect RDMA, the logs mentioned memory registration, and the fleet had already spent time investigating GPU mappings.
The page that finally explained the failure was in host RAM.
Source map
The mechanism above is grounded in:
- NVIDIA, GPUDirect RDMA documentation: PCI BARs, BAR mappings,
nvidia_p2p_get_pages(), device-specific DMA mapping, synchronization, andnvidia-peermem. - NVIDIA NVML,
nvmlDeviceGetBAR1MemoryInfo: BAR1 as the framebuffer mapping used by CPUs and third-party PCIe devices. - rdma-core,
ibv_reg_mr(3): MR APIs, access flags, IOVA,lkey, andrkey. - Linux v6.6,
drivers/infiniband/core/umem.c: long-term page pinning, SG construction, and DMA mapping for ordinary userspace MRs. - Linux v6.6,
drivers/infiniband/hw/mlx5/mr.c: mlx5 user-MR and DMA-BUF-MR creation. - NCCL 2.17.1,
src/transport/net_ib.cc:ibv_reg_mr_iova2()andibv_reg_dmabuf_mr()selection in the IB plugin. - NCCL 2.17.1,
src/init.ccandsrc/transport/net.cc: DMA-BUF capability probing and per-network-device pointer-support checks. - NVIDIA GPU Operator, GPUDirect RDMA prerequisites: open kernel modules, CUDA, kernel, GPU, and network-driver requirements for the DMA-BUF path.
- NVIDIA R525,
nvidia-peermem.c: peer ownership, P2P page acquisition, and RNIC-specific DMA mapping. - Linux, DMA-BUF documentation: exporter/importer attachment and mapping model.