howtf.io

howtf can a device be both present and not found?

Mon, 13 Jul 2026 00:00:00 GMT

Linking & Loading, Part 2. Part 1, howtf does ./app reach main()?, traced the machinery: ELF, ld.so, symbol resolution, the PLT/GOT, relocations. This one is about what that machinery does to you when the same symbol exists twice.

1. Present, and not found

The failure that starts this story showed up the way these things do: in the logs and on the dashboards, as training jobs started crashing at startup. The line that mattered was the RDMA stack's classic no-device error, the moral equivalent of ibv_get_device_list() coming back empty:

No IB devices found

Paste that string into a search engine and every hit is hardware troubleshooting. Check the cable. Check the firmware. Check that the driver is loaded. Which is exactly the rabbit hole it aims you down, because everything about that error says the machine, and nothing about it says the binary.

But the machine was fine. The device was present by every check anyone could run: it showed up in enumeration, the driver was loaded. And the failing consumer was the last one anyone would suspect: NCCL, the most battle-tested RDMA consumer in the fleet, code nobody had touched, reporting No IB devices found at startup, every time, in some binaries and not others. All of them built from the same torch commit.

That's the howtf. Same source. Same fleet. Same device, verifiably there. Whether a binary could see it depended on the binary.

Some context, because the shape of the build matters later. This was at Meta. The binaries were application training binaries composed by Buck, with PyTorch built in-house and statically linked: hermetic builds and fast startup are worth a great deal at that scale, the exact case Part 1 made for why hyperscalers link statically.

"Composed by Buck" is doing real work in that sentence, and one fact about it decides this story, so here it is up front. Buck had two strategies for packaging a binary's native code, and the fleet was mid-migration between them. Binaries still on the older omnibus strategy hid the bundled verbs symbols. Binaries migrated to link groups published them.

The two shapes, briefly. Omnibus merges most of a binary's native code (torch and everything under it) "into a single giant shared library," then hides what it merged: the blob is linked behind a version script ending in local: *;, localizing every symbol except the exact set the Python-facing roots need. What omnibus swallows, it hides. Meta had written down the hazard of that kind of merge back in 2018, in the Android sibling of this exact machinery: native-library merging "works great, as long as there are no common symbols between the libraries being merged."

But a merged blob at training scale eventually outgrows x86-64's ±2 GiB relocation reach, the same wall Part 1 hit at the end of its static-linking detour. The way past it is link groups: carve the binary's native dependency graph into several shared libraries, each under the limit. A link group is still a merge, but it lands with the opposite symbol posture. Each group is a genuine shared library, wired to the binary through DT_NEEDED, and its boundary symbols are exported into the process's dynamic symbol tables so the pieces can find each other at runtime. The full machinery (the version script, the relocation arithmetic, the linker flags, and one public torch casualty) is in Appendix A.

fig. 1 · one bundled copy, two postures

the bundled verbs copy build artifact global scope exported names

The migration between the two ran application by application: some training binaries still composed as one libomnibus, others already carved into link groups. That matters here for two reasons. First, either composition sweeps the binary's native dependencies into the artifact itself. Among them, in these binaries, was the RDMA user-space stack: libibverbs and the mlx5 provider, the code that enumerates RDMA devices, bundled at build time rather than taken from the host the binary lands on.

Second, the posture of that bundled copy (localized inside an omnibus blob, or exported from a link group) depended on which side of the migration a given binary stood. Hold both thoughts; the second is where the opening riddle will find its answer.

Torch was one ingredient; the final artifact was each application's own training binary. And some of those binaries, depending on what they trained on, carried support for MTIA, Meta's own accelerator, served by an in-house collective-communication library newly enabled in the Torch backend. It was that library that pulled the verbs stack into the composition at all.

So: two collective libraries in one process. NCCL/NCCLX for the GPUs, the in-house library for MTIA. Both walk the same device list at startup. The new MTIA path was the one that worked.

2. The investigation

The first guess, always, is hardware. That is what the error says, that is where on-call muscle memory goes, and that is where this one went. We checked the hardware: no issues, everything looked fine. We ran the basic IP and connectivity tests: passed. The device was cabled, enumerated, and reachable, and other binaries were using it happily on the same machine, which cleared the host too.

Then came the fact that snapped the frame. The in-house library, running inside the very same processes, saw the full RDMA device list and worked. So the kernel was serving the device list correctly into the address space where NCCL reported it empty, and the hardware and driver stack were vouched for by a second, working consumer a few shared libraries away. That is the hinge of this whole story: the instant the bug stops being a hardware bug and starts being a linker bug, even though nobody is saying the word "linker" yet.

One detail about how each consumer reaches the verbs stack turns out to be the whole story. The in-house library was linked against the verbs copy bundled into the binary. NCCL in its stock build doesn't link the verbs library at all: it dlopens libibverbs at runtime and takes every entry point from that handle by versioned dlvsym, a handle-scoped lookup, the after-main() loading machinery from Part 1's Appendix H.

And every one of these binaries came from one torch commit, so the shared code was the same everywhere. Working versus broken didn't track hosts, and it didn't track that shared source. It tracked binaries. That doesn't eliminate every difference (two binaries can still diverge in dependencies, configuration, environment), but it aims the suspicion at the step where all of those become bits: how each binary was composed and linked.

fig. 2 · the elimination ladder: everything cleared except the link

suspect cleared the remaining suspect

I should be honest about the stakes, because they explain the depth of the eventual dig. This was a SEV, and a recurrence of an earlier SEV that had been mitigated without ever being fully root-caused. The failure had been here before, been made to go away, and come back.

Running it to ground this time meant descending through layers that don't usually share a whiteboard: how shared libraries are loaded for a binary, how Python links native extensions, and how the RDMA user-space drivers initialize. The root cause, once it surfaced, fit in a sentence: a symbol collision, from double inclusion of the same shared library.

Because once we looked inside those binaries, the constructors had run. A verbs stack (libibverbs, the mlx5 provider) initialized and registered its devices; the in-house library discovered them and ran happily on top. NCCL's discovery, in the same process, still came back empty. The state that initialization filled and the state that NCCL's discovery read had the same symbol names, and were not the same memory.

3. What was actually happening

Four ELF rules this section leans on (all from Part 1). One, the global scope wins a relocation lookup: for a newly loaded object, glibc searches the main program and its startup dependencies before the object's own group. Two, RTLD_LOCAL is not a private namespace: it keeps a dlopened library's names out of the global scope, but does not stop that library from seeing global definitions already there. Three, a handle lookup changes the search root: dlsym/dlvsym on a handle searches that object and its own dependencies, nobody else's. Four, a pointer returned by dlvsym is already an address: calling through it triggers no second lookup.

Two things were true at once, and they should not have been. Initialization had run: the verbs stack had walked the device list and written the results into its tables. And NCCL's discovery call, a moment later, found the tables it was reading empty. Both sides were using the same symbol names. They were not reaching the same state.

Evidence status. I no longer have the internal diffs, so the graph below is a best-fit reconstruction, and it is worth being explicit about what rests on what. Observed during the incident: the hardware was present, the in-house consumer saw the devices, NCCL did not, and working-versus-broken tracked the binary, not the host or the source. Verified from public source: NCCL dlopens verbs and takes its entry points by dlvsym, rdma-core keeps a per-instance driver registry, and glibc searches the global scope before a newly loaded object's own dependency group. Reconstructed: the bundled registration symbol was exported from a link-group DSO and captured the system provider's registration. Demonstrated separately: the two labs in section 4. Unavailable: the original failing binary's own binding trace and link map.

In Part 1 we traced how a dynamically linked program comes to life: ld.so maps the executable and its libraries, builds the lookup scope, and resolves every reference to exactly one definition, per lookup.

What Part 1 never had to confront is that "one definition per lookup" is not "one definition." If a strong, non-weak C symbol is defined in two modules, both definitions exist in the process image, and which one a reference binds to depends on where that reference lives and how its module was built. No error fires, because the two definitions never enter the same link: one is compiled into the executable, the other into a shared library, and a shared library's definitions do not collide at link time with the executable's. C does have a one-definition rule (§6.9 requires exactly one external definition per name in the whole program), but violating it is undefined behavior with no diagnostic required, and nothing in the separately-linked ELF pipeline is positioned to enforce it across that boundary.

Who wins when both copies exist? Whichever definition comes first in the global scope, and the scope is searched executable-first: this is Part 1's lookup-order rule doing exactly what it promised. So when the executable exports the name (and GNU ld puts it in the executable's dynamic symbol table as soon as a shared library on the link line references it, even without -rdynamic), every dynamically resolved reference to the duplicated name lands on the executable's copy. That's interposition, the same feature that lets LD_PRELOAD swap in a debugging allocator.

Crucially, in a default build it applies to the shared library's own references too: the library calls its own functions through the same lookup (the PLT indirection from Part 1), gets the executable's copy like everyone else, and the process stays consistent on one winner. Wasteful, but coherent.

The process can stop agreeing on one winner in more than one way. The cleanest trigger, the one the reproducer in section 4 uses to make the split happen on demand, is a library that opts out of being interposed. Build a shared library with -Bsymbolic-functions (or protected visibility) and its internal references are bound to its own definitions at link time, skipping the runtime lookup. Now the two copies stop agreeing: the library's constructor runs against the library's copy, while every other module's references still resolve through the global scope to the executable's copy.

The incident reached the same split through a different door: the migration from section 1. Its double inclusion, the one the root cause named, had two copies with names and a history. Copy A, the bundled copy, arrived with the in-house collective library: enabling MTIA support pulled libibverbs and the mlx5 provider into the binary's composition. In a binary still composed as libomnibus, that copy was merged into the blob and localized: section 1's local: *; version script kept the verbs symbols out of the process's dynamic symbol tables entirely, and every verbs call the in-house library made resolved to that internal copy, end to end.

(rdma-core is happy to live self-contained: a static build compiles the dlopen loader out entirely and pulls providers in through ibv_static_providers().)

Copy B was the system's: the same stack reached at runtime through dlopen. That is how NCCL gets its verbs, deliberately loading libibverbs at runtime instead of linking it, with a bare-name dlopen("libibverbs.so.1") that the loader's search resolved, on these hosts, to the system copy. And it is how that libibverbs in turn finds its hardware providers, dlopening the driver named in its config. Two instances, two consumers, zero contact: a hidden copy and a public copy have nothing to fight over. Every binary on that side of the migration worked.

Then a binary migrated to link groups, and the bundled verbs changed posture. Carved into a link group, libibverbs stopped being a localized region of a blob and became a genuine shared library whose symbols were exported, default visibility, into the process's global dynamic scope.

Precision matters here, because the naive next sentence ("and they collided with the system copy's identical names") is wrong in an instructive way. The system copy's names never entered the global scope: dlopen defaults to RTLD_LOCAL, which keeps a loaded library's names out of the global lookup.

A second precision, and it is load-bearing: why did that dlopen map a fresh system copy at all, with a verbs stack already resident in the image? Because the loader reuses libraries by name, not by contents. dlopen("libibverbs.so.1") first walks the objects already loaded, comparing the request against each one's names and SONAME (glibc's _dl_lookup_map), and maps a new file only when nothing answers. Had the bundled copy been shipped as a standalone libibverbs.so.1, it would have answered: NCCL's dlopen would have received the bundled copy, one instance, no split, no bug. But the bundled copy's symbols lived inside a link-group DSO carrying the group's own name. Nothing in the process answered to libibverbs.so.1, so the loader mapped the system file, and the image now held two.

That is the ledger to hold for everything that follows. Copy A: bundled, exported into the global scope, the in-house library linked to it. Copy B: the system copy, dlopened RTLD_LOCAL, NCCL pinned to its handle. The surprise, and the whole bug: both providers register into A.

fig. 3 · the two rails: every registration ran left, NCCL read right

copy A: bundled, exported copy B: system, private registration writes the captured import

Before the scope mechanics, the failure in one plain sentence: the system provider meant to register with the system libibverbs loaded right beside it, and the global-scope lookup sent that registration into the bundled copy instead. Here is how, step by step.

Neither consumer was ever confused about which instance it was calling. NCCL takes its entry points by dlvsym on its own handle, a lookup that sees only that handle's little world, so the pointers NCCL held were pinned to copy B, unreachable by interposition, by construction. The pinning covers the pointers NCCL holds, not the copy itself: copy B's own outward references still resolve like anyone else's, and that is about to matter. The in-house library was equally settled the other way: its verbs references resolved through the global scope to copy A, the only definition on offer there. Two consumers, each faithfully wired to one instance. So far, that is just the omnibus arrangement with the curtain open. The bug needs one more reference, one that has to cross between the worlds.

That reference lives in how rdma-core keeps its books. Every instance of libibverbs carries its own file-static state: the device list ibv_get_device_list() hands back lives in a static inside device.c, and the driver registry it is built from is a static driver_list inside init.c. And rdma-core's own version script marks the machinery around them local, so each instance's discovery is welded to its own tables at link time.

Providers are not handed over; they announce themselves: libmlx5 runs an ELF constructor, the PROVIDER_DRIVER macro, that calls verbs_register_driver_<N>() (rdma-core bakes its private-ABI number into the symbol name itself) to append the mlx5 driver to a registry. But which registry is not the loading instance's choice to make.

And ELF binds that call by lookup scope, not by which library loaded the caller. The constructor's call is a plain extern function call, not a function pointer, not a dlsym: an undefined import that the dynamic linker resolves against the global scope first and the dlopen's own dependency scope second. The provider's DT_NEEDED on libibverbs.so.1 decides what gets loaded beside it (Part 1's dependency-discovery step), not who wins the lookup. And RTLD_DEEPBIND, the one switch that flips the order, is nowhere in this stack. The global scope now offered copy A's exported definition.

So every registration in the process (the bundled provider's, the system provider's that copy B dlopened for NCCL, whichever fired first) resolved to copy A and filled copy A's registry. Interposition, doing exactly what Part 1 promised, to the one call that crossed the boundary. Copy B, the instance NCCL was pinned to, sitting right there as the provider's own DT_NEEDED dependency, kept a registry that no constructor could ever reach.

Discovery then walks sysfs and matches what the kernel reports against its own instance's registry, and a device that matches no registered driver is silently dropped from the result (the warning that would have named the problem hides behind an IBV_SHOW_WARNINGS environment variable).

Run the two consumers side by side. The in-house library reads copy A: the registrations landed there, every device matches, MTIA trains. NCCL reads copy B: registry empty, every device the kernel reported matches no driver, zero devices. No IB devices found, from the fleet's most trusted RDMA consumer, on a machine where the library that had smuggled the second copy in could see them all.

Note what the mechanism does not depend on: load order among the consumers. Copy A's exports entered the global scope at startup (DT_NEEDED wiring, in place before either consumer moved), so whichever consumer initializes first, the registration import binds by scope, not by caller. That is why every binary was consistent about failing, every time.

(The stability is a property of this startup topology, mind. A second copy arriving late, by an RTLD_GLOBAL dlopen mid-run, would not rebind references already resolved. Here there was nothing to rebind: the scope was set before the first constructor fired.)

And that is section 1's riddle solved: same commit, opposite behavior, because binaries still on omnibus carried a hidden copy that captured nothing, and binaries on link groups carried an exported copy that captured every registration. Whether NCCL could see the hardware depended on which side of a build-system migration its binary stood. (The dlopen road is also a preview: it is exactly the runtime-scope machinery that returns in section 7 as Route B.)

That is the whole disease, and it is worth naming in its own right: split-state linking, two live copies of one library's state in a single process, with references silently partitioned between them.

The topology did the capturing; on top of it, two conditions specific to rdma-core still had to hold for the collision to land at all: the copies must agree on rdma-core's private ABI number (it is baked into the registration symbol's name), and symbol versioning must not block the cross-copy match (it does not, for reasons glibc's own source comments on). Appendix B walks both.

4. Reproducing it

A claim like that is easy to state and easy to doubt, so here are two reproducers you can run in minutes, no special hardware needed. The first rebuilds section 3's binding topology directly and watches the registration get captured. The second strips the mechanism down to a model and builds the split six ways, to pin down exactly which ingredients produce it.

The production topology, reproduced

The first lab, scope-capture/ in the repo, restages the incident's own door: the split from scope alone, no self-binding flag, no copy in the executable. Three shared libraries, all default visibility:

libbundle.so is copy A, a DT_NEEDED dependency of the app, so its register_driver sits in the global scope from startup.
libregistryB.so.1 is copy B, the system libibverbs, dlopened RTLD_LOCAL so its names stay private, reached only through its own handle.
libproviderB.so is the provider. Its DT_NEEDED is copy B, and its constructor calls register_driver as a plain extern import.

That constructor call is the whole incident in one line. It is an undefined import, so the loader resolves it against the global scope first and the provider's own dependency group second. Copy A is global; copy B is only in the local group. The registration lands in copy A, captured, even though copy B is the provider's own dependency sitting right beside it. LD_DEBUG=bindings says so directly:

binding file libproviderB.so to libbundle.so: normal symbol `register_driver' [VERB_1.0]
binding file libregistryB.so.1 to libregistryB.so.1: normal symbol `get_device_list' [VERB_1.0]

The provider binds register_driver to libbundle, copy A. Copy B answers only its own get_device_list. Then each consumer reads where it was always going to: the in-house side, linked to copy A, sees the device; NCCL, by dlvsym on copy B's handle, reads copy B's empty registry.

[bundle / copy A] register_driver(mlx5_from_providerB) -> registry A @0x73da35e88040 now holds 1 device(s)
    in-house consumer sees 1 device(s)   [OK -- reads the copy the registration landed in]
[registryB / copy B] get_device_list  <- registry B @0x73da35e83060 holds 0 device(s)
    NCCL consumer sees 0 device(s)   *** No IB devices found -- the registration went to the OTHER copy ***

Two registry addresses, the write on one and the read on the other, no self-binding flag anywhere in the build. make fixed confirms the mechanism from the far side: localize copy A's register_driver so it leaves the global scope, and the provider's import falls through to copy B. Registration and discovery reunite there, and NCCL sees the device. That is also the lesson section 6's naive visibility fix misses: the copy you hide has to be the one wrongly winning the global lookup, the bundled copy A, not the system copy B.

The trigger, isolated

The scope-capture lab reproduces the exact door but fixes the topology. To map the boundary, when a split happens and when it doesn't, the second lab strips out the hardware and the scope machinery and models the mechanism directly: a small "verbs" library present in two copies (one static in the executable, one shared), a collective that performs discovery, and a couple of device names the library registers. Nothing links libibverbs. The point is the linker, not the hardware. make matrix builds the same scenario six ways and prints, for each, the address of the table the constructor wrote and the address of the table discovery read. Same address, no split. Different address, split. Whether a given build splits is a comparison of two hexadecimal numbers, not a matter of interpretation. (A second variant in the repo restages this split on real soft-RoCE (rdma_rxe) devices, through ibv_open_device, for anyone who wants the hardware path.)

This lab reaches the split through the cleanest switch that produces it on demand, a self-binding flag, rather than the scope-capture lab's dlopen scoping. Section 7 comes back to why that difference matters. Here is the splitting configuration's actual output:

[constructor in copy=SHARED] registering rxe_train, rxe_store
[register -> copy=SHARED table@0xffff90340028] now holds 2 device(s)
[get_list <- copy=STATIC table@0xaaaad8f00018] this copy holds 0 device(s)
collective: discovered 0 device(s)   *** DEVICE NOT FOUND -- but the
constructor DID register devices, into the OTHER copy ***

The constructor registered both devices. Discovery found zero. Different addresses, different copies. That is the production failure's shape, drawn small (the reproducer's simplified topology, not the production one: here the second copy is statically linked into the executable, where the incident's arrived by dlopen):

fig. 4 · the reproducer, simplified: the constructor filled one copy, discovery read the other

the exe's copy (read) the .so's copy constructor writes

The matrix pins down exactly when this happens and, just as important, when it doesn't. Config A, the default build with no special flags, concedes the obvious objection first: no split. The shared library's own constructor is interposed onto the executable's copy, so everyone agrees on one winner. Config B builds the shared library with -Bsymbolic-functions: split, the constructor writes the .so's copy while discovery reads the executable's, and the collective reports "device not found." Config C, protected visibility on the library's internals, is an equivalent self-binding trigger: split again. The remaining rows probe the edges (hidden visibility, and two data-symbol variants where copy relocation rescues one case); Appendix C walks them.

fig. 5 · the reproducer's gate: a split by this route needs all three conditions at once

condition present split: two live copies no split

So duplicate copies alone are not the bug; you need the self-binding trigger on top. And that trigger is not exotic: -Bsymbolic-functions is used by some build systems to cut binding overhead and symbol preemption, with one nasty property. Unlike full -Bsymbolic, which sets a DF_SYMBOLIC flag in the output, -Bsymbolic-functions leaves no explicit marker in the binary. No flag, no dynamic tag. The linker simply resolves the internal calls and moves on. What remains is an absence, the relocations those calls no longer need, and you cannot grep a .so for an absence. (Section 7's scanner learns to read exactly that residue.) The dangerous shape is a function (or state reached through one) in a self-binding library.

The reproducer also demonstrates the part that made the production incident so disorienting. Build the application twice from byte-identical source, once with the redundant static copy on the link line and once without:

### app_with_static    (redundant static copy linked):
  collective: discovered 0 device(s)   *** DEVICE NOT FOUND ***
### app_without_static (single copy):
  collective: discovered 2 device(s)

Same source, opposite behavior, decided entirely by link composition. That pair echoes the omnibus-to-link-groups migration: a different trigger, the same phenotype. In a fleet where composition varies per application, it is precisely "some binaries fine, some not, same commit."

Toolchain, for the record: gcc 13.3.0, binutils 2.42, Ubuntu 24.04, kernel 6.17-aws for the soft-RoCE variant; reproduced on both aarch64 and x86_64, and re-validated on a clean EC2 instance from the scripts alone, fresh RDMA GUIDs and all. The repo is at the reproducer repo.

5. Why nothing warned

The uncomfortable part is that every component behaved exactly to spec. The ELF gABI forbids multiple STB_GLOBAL definitions only among the objects that enter a link, and a shared library's definition never enters the executable's link. (Part 1's build-time flashback showed this from the other side: the linker consulted libmath.so only to verify the symbol existed and record DT_NEEDED. It never took the code.) The GNU ld manual describes archive members being pulled lazily, left to right, once. lld's documentation states the situation without alarm: two links can "both succeed but they have selected different objects from different archives that both define the same symbols." C's one-external-definition rule (§6.9) carries no required diagnostic and no COMDAT machinery to enforce it here; nothing even checks that the two definitions are the same code.

So no diagnostic fires by default, and the opt-in diagnostics that exist each miss this class. --warn-backrefs catches order-dependent archive resolution, not cross-boundary duplication. gold's --detect-odr-violations is scoped to C++ mangled names and weak definitions, and needs debug info. -z muldefs governs duplicates within a link, and this pair never shares one.

People have run into this before, of course. Sergei Trofimovich wrote up a shared-library collision breaking real programs and landed on the same verdict, that the toolchain does not help much here. What has been missing is the recognition that these one-off war stories are a single failure class with a describable trigger.

A failure that produces a crash gets a stack trace. A failure that produces a wrong answer gets silence.

6. Fixes: the folklore one that fails, and the ones that work

The instinct, once you know two copies of a symbol are colliding, is to reach for visibility: rebuild the shared library with -fvisibility=hidden, or slap a local: * version script on it, and the duplicate should stop being exported. It does not fix this. Verified against the reproducer, both leave the split fully in place:

NAIVE FIXES THAT DO NOT WORK:
  nofix-visibility     :   collective: discovered 0 device(s)   *** DEVICE NOT FOUND ***
  nofix-version-script :   collective: discovered 0 device(s)   *** DEVICE NOT FOUND ***

They hide the wrong copy. Visibility controls what the shared library exports; it does nothing about the executable's copy, which is the one discovery was binding to all along. (Hiding the library's symbols can also get it dropped by --as-needed entirely, trading a split for a constructor that never runs.)

What works, verified, is making the two copies stop being the same symbol, or stop being two:

FIXES THAT WORK:
  fix-drop-duplicate :   collective: discovered 2 device(s)
  fix-exclude-libs   :   collective: discovered 2 device(s)
  fix-prefix-rename  :   collective: discovered 2 device(s)

These are not equal in durability. The root-cause fix is one canonical copy, so there is only ever one instance to bind to. Where two copies genuinely must coexist, make them different symbols outright with objcopy --redefine-sym, so they can never collide. And a narrower, link-local measure is to stop the executable exporting its static copy: -Wl,--exclude-libs,libverbs_static.a, naming the offending archive rather than -Wl,--exclude-libs,ALL, which hides every archive's symbols and can suppress plugin or callback exports the process actually needs. The rename fix is not hypothetical. Meta's public torchcomms repository ships a rename_symbols.sh that prefixes every nccl* symbol, with a comment saying it exists to avoid conflicting with the OSS nccl* bundled with PyTorch. The ecosystem shipped the rename fix years before the disease had a name.

In the incident, both rungs got used, in the order SEV pressure dictates. The immediate mitigation was to make the in-house collective library opt-in: binaries that didn't need MTIA stopped pulling the verbs stack into the composition at all, so nothing exported a second copy into the global scope. The provider's registration import, with nothing left to capture it, fell through to the system libibverbs: registration and discovery reunited on the one copy NCCL was pinned to, and NCCL healed. The bug was defused by removing one of the two copies from most processes, not by fixing the collision.

The principled fix came after: statically link libibverbs and libmlx5, one canonical copy for the image, kept out of the dynamic namespace. That is the posture the omnibus merge had been providing by accident, restored on purpose. In the fix ladder's terms it removes the duplicate from the scope where the registration was being captured (the first rung's spirit), shipped in production before the reproducer existed to validate it.

7. How common is this, really?

One of the layers the SEV dig descended through was Python native linking: how the interpreter dlopens extension modules and the libraries bundled alongside them. That detour turns out not to be a detour at all, because the wheel ecosystem lives under the same pressure that built the incident and pushes back the opposite way. The monorepo merges (omnibus, link groups, one canonical copy per image) and breaks the day a canonical copy's symbols escape into a scope that already holds the same names, the hazard Meta's 2018 write-up warned about. The wheel ecosystem vendors (auditwheel grafts a private copy of every native dependency into each wheel) and breaks the day two of those private copies co-load and each runs its own state. Same pressure, opposite mitigations, one disease.

The two worlds also differ in route, and the distinction organizes everything measured below, because split-state linking arrives by two routes, not one. Route A (interposition capture) is the reproducer's shape: a duplicate strong symbol, a self-binding library, and an interposing module sharing one symbol scope. Route B (scope partition) needs neither self-binding nor interposition. If two modules are loaded into separate local scopes (RTLD_LOCAL, the default for every dlopen, which is how Python loads extension modules) and each carries its own vendored copy of a library, then each side binds its own copy and runs its own state. Same disease, reached without any special flag at all. The incident stood with one foot in each route: Route A's interposition did the capturing (the registration import, resolved through the global scope into the bundled copy), and Route B's scope machinery did the isolating (the victim pinned by handle to an RTLD_LOCAL copy the capture could never fill).

Measuring either route takes more than a duplicate-symbol lister, because the trigger leaves no marker in the binary (section 4) and the base rate of benign duplication is enormous: in a sweep of 788 stock system binaries, 468 had duplicate symbols somewhere in their closures, and not one was a split. So I built symsplit, a binding simulator: it models what ld.so actually does and flags a split only when two modules in one image would genuinely resolve the same name to different definitions. Against the reproducer matrix it flags exactly the splitting configuration and clears the rest. Against those 788 system binaries: zero flags (a quiet-corpus result rather than ground-truth validation; the corpus is presumed clean). Appendix D covers what it models, how it infers self-binding from a relocation absence, and where its limits lie.

Pointed at the manylinux ML-wheel ecosystem, the picture that comes back is specific.

Route B is live in stock wheels. Import faiss, scikit-learn, and torch into one Python process and /proc/self/maps shows two distinct builds each of libgomp, libgfortran, and libquadmath: two OpenMP runtimes, two Fortran runtimes, each with its own global state, resident in one process. And this is not merely structural: trace an actual compute workload under LD_DEBUG=bindings and 206 duplicated compute symbols bind to two different definitions at once in the same process, almost all of them OpenBLAS kernels, with faiss's statically embedded copy answering faiss's calls while numpy's libopenblas answers numpy's (partitioned binding caught live, not a demonstrated wrong answer). The ecosystem half-knows this. It's the "multiple OpenMP runtimes" problem, and Intel ships a runtime kill-switch for it, KMP_DUPLICATE_LIB_OK, silencing an error whose own text warns the duplication "can cause incorrect results." The full survey numbers live in Appendix E.

Route A's exact trigger is absent from public wheels, which is itself the finding. DF_SYMBOLIC is set on zero of the 366 libraries examined, and symsplit predicts zero Route A splits across all eight co-load configurations tested. The trigger lives where the incident lived: inside monorepo native-link builds (Buck, Bazel, symbolic-binding hardening, omnibus-to-link-group migrations) that you cannot download from PyPI. That inaccessibility is a good part of why the class went undiagnosed for so long. But the ingredient that promotes Route A is one line away in software everyone runs: import torch executes ctypes.CDLL("libtorch_global_deps.so", RTLD_GLOBAL), lifting torch's OpenMP into the global scope. An LD_DEBUG probe shows the consequence directly: import faiss alone and its extension module's OpenMP references bind faiss's bundled libgomp; import torch first and every one of those traced references rebinds to torch's copy instead. Which copy of a runtime your library gets is decided by Python import order.

The honest shape of the result: the preconditions are everywhere, the full Route A alignment is rare in public and lives behind corporate build systems, and Route B is quietly resident in the stock ML stacks tested here and, by the arithmetic of auditwheel's vendoring, in any process that co-loads two wheels carrying the same runtime. The training binary's disease, one import away, and nothing warns at any tier. The ecosystem survives by paying a scattered tax: KMP_DUPLICATE_LIB_OK, auditwheel's content-hashed sonames (which enable coexisting copies rather than prevent them), torchcomms' rename_symbols.sh, conda's one-copy-per-environment discipline. Four patches for one disease, none of them labeled with what they treat.

8. What should change

The diagnostic nobody built already has a name in the record. A --warn-interposition warning was floated on the GCC mailing list in May 2021 and never implemented in ld or lld. The reason it stalled is documented too: Fangrui Song (MaskRay), lld's maintainer, scoping the equivalent check, noted that the mechanics are easy but that "in the absence of an ignore list mechanism, this extension will not be useful". Interposition is a load-bearing ELF feature, and the base rate of benign duplication is enormous.

That missing ignore-list mechanism is exactly what symsplit is. The allowlist for intentional interposers (allocators, sanitizers), the weak/versioned/hidden/symtab-only filtering, the self-binding inference: all of it demonstrated against real binaries, silent across a 788-binary sweep of presumed-clean system binaries. The tool stands alone today; the question worth putting to the linker maintainers, and I intend to, is whether an opt-in, allowlist-first version of the check belongs in lld or ld proper.

Until then, the checklist for anyone shipping large statically-or-mixed-linked binaries. If a dependency is built -Bsymbolic or -Bsymbolic-functions, and a strong C symbol it defines also exists anywhere else in your image, you have a latent split-state hazard (it fires when the duplicated symbol guards state and both copies end up live in the same scope), and no default tool will flag it. Scan for it. Prefer one canonical copy, or make the copies different symbols outright. And file the lesson somewhere it will be found at 2 a.m.: No IB devices found can mean the devices are right there — enumerated, registered, waiting — in the copy of the world you didn't ask.

Reproducer, scanner, and survey artifacts: the reproducer repo (scanner at tools/symsplit). Everything quoted above (the scope-capture bindings, the address matrix, the fix ladder, the sweep, the wheel survey) is a captured artifact in the repo, rerunnable from scripts.

Appendices

Evidence lockers: the full dumps and gnarlier details the body text points at. Skip freely; return when a claim needs its receipts.

Appendix A: The Buck machinery (omnibus, link groups, and the 2 GiB wall)

Section 1 compressed the build story into one contrast: omnibus hides, link groups publish. Here is the machinery behind both halves, all of it readable in public tooling.

1) Omnibus: the merge

A Python training program pulls in an enormous amount of native code (torch and everything under it), and not all of it can be statically compiled into one executable. Buck's omnibus strategy does the next-best merge: statically link most of the native code "into a single giant shared library" (a libomnibus.so), leaving only the extensions Python imports directly as separate .so's. You pay the full static-link cost once, at build time; at runtime the binary dlopens roughly one big library instead of hundreds.

The merge comes with a symbol discipline you can read in the open-source prelude. The omnibus body is linked behind a generated version script whose last line is local: *;, localizing every symbol merged into the blob except the exact set the Python-facing roots need.

2) The 2 GiB wall

On x86-64 a PC-relative reference reaches ±2 GiB (R_X86_64_PC32 spans [-2³¹, 2³¹)), and a merged native library at training scale eventually outgrows it. The link fails with relocation truncated to fit. The medium and large PIC code models do exist on x86-64, but neither is a clean retrofit for an image this size: metadata like .eh_frame keeps 32-bit R_X86_64_PC32 relocations even in -mcmodel=large output, prebuilt small-model objects carry the same limit, and large-model code costs size and speed. So the documented way out is the move MaskRay's relocation-overflow survey prescribes: "partition the large monolithic executable into the main executable and a few shared objects."

3) Link groups: the partition

In buck2, that partitioning is link groups: a link_group_map carves the binary's native dependency graph into multiple shared libraries, each under the limit, with per-group control over what links statically and what dynamically. A link group is still a merge (each group statically links its members into one shared library), but its boundary symbols are exported: the same document describes public nodes linked --whole-archive so all their symbols survive, and --dynamic-list entries feeding names into the main binary's dynamic symbol table so the pieces can find each other at runtime.

None of this machinery is gentle around torch. The public tracker has buck2 #62, libomnibus turning a symbol undefined that libtorch_cpu, libc10, and libtorch_python all keep weak.

Appendix B: Two preconditions for the capture

Section 3 asserted that two preconditions had to hold for the registration to be captured. Here they are, with the receipts.

1) The copies must agree on rdma-core's private ABI number

The registration entry point is verbs_register_driver_<N>: rdma-core bakes its private-ABI number into the symbol name itself. Copies that disagree on the number define different symbols and cannot collide there at all. Same rdma-core lineage, same number: one name, two definitions, a collision waiting for a scope.

2) Symbol versioning does not block the match

The bundled definition had to be visible where the resolver looked, and symbol versioning (the version contract Part 1 read out of .gnu.version_r) does not prevent that. glibc's check_match takes an exact version match; failing that, an unversioned definition still gets in through two doors. If the defining object carries no version table at all, the symbol is accepted outright. The comment there reads, of all things, "This can happen during symbol interposition." And if it does carry one, a definition sitting at the global, unversioned index is kept as the fallback that wins when no exact match exists anywhere. What gets skipped is a definition under a different version tag. Same rdma-core lineage, same number, exported names: the capture follows.

Appendix C: The reproducer's edge rows (hidden visibility and copy relocation)

The body kept the matrix's headline rows: A (default, no split), B (-Bsymbolic-functions, split), C (protected visibility, split). The full matrix has three more rows, and each edge teaches something:

config	what changed	result
A	default build, no special flags	no split: same address; the shared library's own constructor is interposed onto the executable's copy, so everyone agrees
B	shared lib built `-Bsymbolic-functions`	SPLIT: constructor writes the .so copy, discovery reads the exe copy; "device not found"
C	protected visibility on the lib's internals	SPLIT: an equivalent self-binding trigger
C′	hidden visibility	the DSO is dropped by `--as-needed`, so the constructor never runs at all (a different failure); force it to load and the split reappears
D1	the colliding thing is a data table, static in the .so, global in the exe	SPLIT
D2	data table, global on both sides	no split: copy relocation quietly unifies everyone onto the executable's copy

C′ (hidden visibility) is a reminder that "just hide the symbols" changes the failure rather than removing it: with nothing exported, --as-needed drops the unreferenced DSO from the link, and the constructor never runs at all. Force the DSO to load and the split reappears.

D2 (copy relocation) is the trap turned inside out. The "obvious" version of this bug, a duplicated plain data global, is the one the toolchain saved us from here, because copy relocation unified the copies: the executable gets its own copy of the data, and the shared library's references are pointed at it. That rescue is itself configuration-dependent (it hinges on how the executable references the data, and flags like -z nocopyreloc switch it off), not a law. D1 shows the same data table splitting the moment the .so side's copy is a file-static the rescue cannot reach.

Appendix D: symsplit (a binding simulator, not a duplicate lister)

The distinction is the whole tool. nm | sort | uniq -d answers "does a duplicate exist," and on any real system it screams constantly about things that are fine: 468 of the 788 stock system binaries in the sweep had duplicate symbols somewhere in their closures, and not one was a split. bash defines its own getenv over libc's, which is benign because libc keeps an interposable reference to the name and unifies onto bash's copy. Thousands of weak libc aliases exist to be overridden. Versioned symbols with disjoint version sets can't collide.

symsplit models what ld.so actually does instead: .dynsym versus .symtab visibility, scope order, symbol versioning, and per-library self-binding inferred from relocations. The inference reads the absence section 4 described: a library that retains an interposable JUMP_SLOT or GLOB_DAT reference to one of its own exports demonstrably did not self-bind; one with none probably did. It flags a split only when two modules in one image would genuinely resolve the same name to different definitions. When it fires, it says why:

VERDICT  SEV     SYMBOL               WHY
SPLIT    MEDIUM  vx_get_device_list   libverbs_shared.so is probably
  self-binding (no JUMP_SLOT/GLOB_DAT to any own export =
  -Bsymbolic-functions signature); its own copy answers its constructor
  calls, while libcollective.so's reference resolves to app_B's copy
  -> two live copies diverge (split state)

It is honest about its own limits, too. -Bsymbolic-functions can't be proven from the ELF (a library with no self-references looks self-bound), so that inference carries a confidence label in the output. And dlopen scope is a runtime property the ELF doesn't record, so Route B modeling takes the scope layout as input rather than pretending to know it.

Appendix E: The wheel survey (full numbers)

The survey behind section 7's headlines, all of it rerunnable from the scripts in the reproducer repo.

1) Route B: resident duplicate runtimes

Import faiss, scikit-learn, and torch into one Python process and /proc/self/maps shows two distinct builds each of libgomp, libgfortran, and libquadmath. numpy plus scipy alone maps two libgfortran and two libquadmath.

Tracing an actual compute workload (numpy matmul, torch matmul, a faiss index search) under LD_DEBUG=bindings shows 206 duplicated compute symbols binding to two different definitions at once in the same process, almost all of them OpenBLAS kernels, with faiss's statically embedded copy answering faiss's calls while numpy's libopenblas answers numpy's. Those kernels are code, not divergent state (partitioned binding caught live, not a demonstrated wrong answer), but the runtimes underneath them carry genuinely mutable state, thread pools and locks, and the same partition carries each side's runtime state with it. That is the substance behind Intel's KMP_DUPLICATE_LIB_OK kill-switch: the error it silences warns that duplicate OpenMP runtimes "can cause incorrect results."

2) Route A: the trigger census

DF_SYMBOLIC is set on zero of the 366 libraries examined across the surveyed wheels, and symsplit predicts zero Route A splits across all eight co-load configurations tested (combinations of numpy, scipy, torch, faiss, and scikit-learn imported together).

3) The import-order probe

The ingredient that promotes Route A is observable directly. import torch executes ctypes.CDLL("libtorch_global_deps.so", RTLD_GLOBAL), lifting torch's OpenMP into the global scope. Under LD_DEBUG=bindings: import faiss alone and its extension module's OpenMP references bind faiss's bundled libgomp; import torch first and every one of those traced references rebinds to torch's copy instead.

howtf does ./app reach main()?

Mon, 16 Feb 2026 00:00:00 GMT

We have all been there. You deploy a binary that worked perfectly on your development machine, but the production environment crashes with:

/lib64/libc.so.6: version 'GLIBC_2.34' not found

error while loading shared libraries: libfoo.so: cannot open shared object file

You do a frantic search, blindly paste export LD_LIBRARY_PATH= commands, and install random packages until the error disappears. We often treat the execution process as a black box, something that "just works" until it doesn't. These errors are symptoms of a system most engineers never look at closely, and that lack of understanding compounds when you are debugging at scale.

In this post, we take a different approach. We trace the life of a command from the moment you hit Enter until it reaches main(), watching the kernel, linker, and loader coordinate to turn a file on disk into a running process. Then we flash back to build time (Parts V–VI) to see where the machinery was set up. At the end, we reproduce both errors above on purpose and read the diagnosis straight off the binary. Every dump in this post comes from one reproducible container; the demo and a regenerate.sh live in the site repo.

Scope & assumptions. This walkthrough uses Linux on x86‑64 as the concrete reference, with the glibc dynamic loader (ld-linux-x86-64.so.2) as "the loader" we talk about. The big ideas transfer to other architectures and libcs, but some details (relocation types, syscall entry, loader internals, memory-ordering constraints etc.) might differ.

Who is this for? If you have ever wondered what actually happens between hitting Enter and your code running, this is for you. Some comfort with C helps, and we will touch on assembly and kernel internals in places, but the main narrative is designed to be followed without deep expertise in either. The appendices are where the really gnarly details live.

fig. 0 · the relay race, keystroke to main()

our code kernel loader the destination

Follow Along

We will use a standard Linux environment. If you are on macOS or Windows, use Docker Desktop to get deterministic userspace behavior (specifically for x86‑64 relocation types).

A Note on Architecture (Apple Silicon & Windows ARM): If you are running on an ARM chip (M1/M2/M3, etc), you can still follow along.

macOS: Docker Desktop can run linux/amd64 containers using Rosetta‑based translation wired through binfmt_misc when configured to do so. This is documented by Apple and by Docker Desktop settings.

Windows (ARM): the common mechanism for running linux/amd64 binaries under an ARM64 Linux environment (including WSL2-based backends) is QEMU user-mode emulation wired through Linux's binfmt_misc. Whether it's already configured "out of the box" depends on the Docker/WSL2 setup, versions, and registration state, but most likely it is.

Curious how this cross-architecture magic works under the hood? See Appendix A.

A note on prompts: ❯ is my host machine; root@container:/code# is inside the container. Every dump in this post was captured in the container described below (gcc 11.4, glibc 2.35), by demo/elf-linking/regenerate.sh.

1. The Source Files

Three files. One program, one shared library, nothing hidden:

// main.c — the entire demo program. The interesting part is what links it.
#include <unistd.h>

extern int add(int a, int b);

int main(void) {
    int sum = add(5, 10);
    sleep(60); /* keeps the process alive so we can read /proc/<pid>/maps */
    return sum;
}

// math.c
int add(int a, int b) {
    return a + b;
}

CC = gcc

all: libmath.so dynamic_app dynamic_app_lazy

libmath.so: math.c
	$(CC) -shared -fPIC -o libmath.so math.c

dynamic_app: main.c libmath.so
	$(CC) -o dynamic_app main.c -L. -lmath -Wl,-rpath,'$$ORIGIN'

# explicit lazy-binding variant for the PLT/GOT walkthrough (Part III)
dynamic_app_lazy: main.c libmath.so
	$(CC) -o dynamic_app_lazy main.c -L. -lmath -Wl,-z,lazy -Wl,-rpath,'$$ORIGIN'

Two things here are load-bearing, and both will pay off later: we link with -L. -lmath (not by naming ./libmath.so directly, a difference that reproduces one of our two opening errors, as we'll see in Part VII), and we build a second binary with -Wl,-z,lazy (Part III explains why we need to ask for lazy binding explicitly in 2026).

2. Start the container

# Force x86-64 to align with our assembly examples
❯ docker run --rm -it \
  --platform=linux/amd64 \
  --cap-add=SYS_PTRACE \
  --security-opt seccomp=unconfined \
  -v "$PWD"/code:/code -w /code \
  ubuntu:22.04 bash

# Install tools
root@container:/code# apt-get update && apt-get install -y build-essential binutils gdb strace

3. Compile the project:

root@container:/code# make
gcc -shared -fPIC -o libmath.so math.c
gcc -o dynamic_app main.c -L. -lmath -Wl,-rpath,'$ORIGIN'
gcc -o dynamic_app_lazy main.c -L. -lmath -Wl,-z,lazy -Wl,-rpath,'$ORIGIN'

root@container:/code# ls
Makefile  dynamic_app  dynamic_app_lazy  libmath.so  main.c  math.c

You type ./dynamic_app and hit Enter.

Your shell calls fork() to create a child process. That child process calls execve("./dynamic_app"), and your app starts running. Simple, as long as nobody asks what execve actually did.

Part I: The Hardware Gate and Kernel Entry

1.1 The Wake Up

Your shell (bash/zsh) was actually asleep, blocked on a read() system call waiting for input. The kernel, tty, keyboard driver, etc. work together to let your shell know exactly what command the user executed.

How your keystroke actually reaches the shell (PTYs, the line discipline, and why Ctrl+C sometimes can't save you) is a whole story of its own. It's coming as a separate post: The keyboard dance.

1.2 The `fork()` syscall (cloning)

The shell parses your command and decides to run a new program. But first, it must duplicate itself. It calls fork().

This triggers a hardware transition.

The Trap: The CPU executes the syscall instruction (opcode 0F 05).
The Switch: The hardware instantly elevates privileges to Ring 0.
The Lookup: It consults the Model Specific Registers (MSRs) to jump straight into the kernel's entry point (entry_SYSCALL_64), stashing the user return address in RCX and the flags in R11 on the way. SYSCALL touches no stack; the kernel's entry stub switches to a kernel stack in software before it pushes anything.

The hardware gate deserves more than three bullets. IDT vs SYSCALL entry, TSS/IST stack rules, and KPTI are a separate post: Before the kernel answers.

It creates a near‑identical copy of the shell (the child process). In practice, the kernel does not duplicate physical memory. It marks the writable private pages as copy‑on‑write (COW), so the two processes share the same physical pages until one of them writes. This child is now running, but it is still running the shell's code.

1.3 The `execve` Syscall

The transition for the syscall remains the same as fork, but the handler will be different. execve kernel handler discards the child's old memory map (the shell code) and prepares to load the new binary.

The Operating System has taken the wheel. It is now sitting in Ring 0 with the file path ./dynamic_app and a mandate to start executing it.

1.4 Inside the Kernel: `fs/exec.c`

Once inside the kernel, execution eventually reaches do_execveat_common in fs/exec.c.

The kernel opens the file and iterates through a list of "binary handlers" to find one that understands the file format. Since this is an ELF file, it lands in load_elf_binary in fs/binfmt_elf.c.

1.5 The Magic Check

First, the kernel validates that this is actually an ELF file. It reads the first 4 bytes. If they aren't 0x7F 'E' 'L' 'F', it rejects the file immediately.

struct elfhdr *elf_ex = (struct elfhdr *)bprm->buf;

if (memcmp(elf_ex->e_ident, ELFMAG, SELFMAG) != 0)
    goto out;

(view at v6.8)

Part II: Mapping the Memory

The kernel does not care about "sections" (like .text or .data). Those are build/link time constructions, mainly for the linker. The kernel cares about segments (Program Headers), which tell the kernel what exactly to load and where.

fig. 1 · one file, two readings

file sections memory segments loader kernel

2.1 Iterating Segments (`load_elf_binary`)

The kernel loops over the program headers (PT_LOAD) to figure out what to map. (View Source in binfmt_elf.c)

for(i = 0, elf_ppnt = elf_phdata; i < elf_ex->e_phnum; i++, elf_ppnt++) {
    if (elf_ppnt->p_type == PT_LOAD) {
        // Create the memory mapping
        error = elf_map(bprm->file, load_bias + vaddr, elf_ppnt, ...);
    }
}

It is common to conceptually talk about "two main regions":

Code-ish mappings: read + execute (your code + PLT stubs + some read-only metadata).
Data-ish mappings: read + write (globals, .bss, GOT areas, dynamic data).

However, and this matters for correctness, modern toolchains frequently emit more than two PT_LOAD segments (e.g., separate read-only segments for constants, plus layouts that support RELRO cleanly). In our demo binary, readelf -l ./dynamic_app reveals four distinct PT_LOAD segments:

Read-Only Metadata (R): ELF headers and dynamic symbol tables.
The Text Segment (R E): Your actual code (.text) and the PLT stubs. This is the only memory executable by the CPU.
Read-Only Data (R): Constants (.rodata) and unwind info. Kept out of the executable mapping (-z separate-code is another modern default) so constant data can never be fetched as instructions. That shrinks the attack surface, though it does not by itself prevent ROP.
Writable Data (RW): Global variables (.data) and the Global Offset Table (GOT).

See Appendix D for the full readelf -l output and a detailed walkthrough.

2.2 Finding the Correct Address for the Segments

The code above already hints at the answer: each segment's virtual address is load_bias + vaddr, where vaddr comes straight from the program header's p_vaddr field. But look at the actual values in our binary (from Appendix D):

  LOAD           0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000638 0x0000000000000638  R      0x1000

A p_vaddr of 0x0? That would map over the NULL page. Something is off.

The explanation is that our binary is not a traditional fixed-address executable (ET_EXEC). On modern distros, GCC defaults to building Position-Independent Executables (PIE), which use type ET_DYN in the ELF header. This does not mean it is a shared library. It means the entire image can be loaded at an arbitrary base address, which is what enables ASLR.

root@container:/code# readelf -h ./dynamic_app | egrep 'Type:|Entry'
  Type:                              DYN (Position-Independent Executable file)
  Entry point address:               0x1080

This is where the variable load_bias in the kernel code is conceptually coming from:

With PIE: the kernel chooses a randomized base address (ASLR).
Runtime virtual address = load_bias + p_vaddr
Runtime entry point = load_bias + e_entry

See Appendix D to see this load_bias in action for our demo app.

At this point, assume that segments are loaded into the process's address space (or more precisely, mmapped).

A note on "mapped" vs "loaded": When we say "mapped," we do not mean "copied to RAM." The elf_map call essentially creates a VMA (Virtual Memory Area) that tells the kernel: "If the CPU asks for virtual address X, the bytes live in this file at offset Y." The physical RAM can be empty. When the CPU tries to execute the first instruction, a page fault fires. The kernel catches it, fetches the page from disk (via the page cache), and resumes execution as if nothing happened. This is demand paging.

2.3 The Fork in the Road: `PT_INTERP`

Then the kernel checks for a specific header: PT_INTERP. (View Source)

if (elf_ppnt->p_type == PT_INTERP) {
    interpreter = open_exec(interp_name); // e.g., /lib64/ld-linux-x86-64.so.2
    ...
    entry = load_elf_interp(&interp_elf_ex, interpreter, ...); // its own mapper, no recursion
}

Because dynamic_app has this header, the kernel maps the dynamic loader (ld-linux.so) into memory with a dedicated helper, load_elf_interp(). The interpreter's own PT_INTERP, if it had one, would be ignored (which is exactly why ld.so must bootstrap itself, as we'll see in Part III). The kernel then sets the instruction pointer to the loader's entry point, not your dynamic_app's. (View Source)

One more thing before the kernel leaves the stage: how will the loader know where our binary landed? The kernel writes the answer onto the new process's stack as the auxiliary vector: AT_PHDR (where the program headers were mapped), AT_ENTRY (the app's real entry point), AT_BASE (where the interpreter itself landed), and friends. That auxv is the kernel→loader handshake; you can watch it with LD_SHOW_AUXV=1 ./dynamic_app.

Part III: The Loader Takes Control (User Mode)

Control returns to User Mode. The program running is now the dynamic loader (ld-linux.so), appearing in glibc source as elf/rtld.c.

3.1 Self-Relocation (The Bootstrap)

The loader itself is also just a program, just a bit special one as it wakes up in a hostile environment. Because of ASLR, it has been loaded at a random address, meaning all its internal pointers to global variables are wrong. It cannot call functions or access static data yet. Before it can do anything else, the loader must fix these addresses. This happens in the _dl_start path. See Appendix E: The Loader's Bootstrap for more details.

3.2 Dependency Discovery

Once the loader has healed itself, it becomes a fully functional C program running inside your process. It can now inspect your dynamic_app. It reads the PT_DYNAMIC segment to find DT_NEEDED tags (libmath.so and libc.so.6 in our case), finds each library, and maps it into the process with mmap.

Where does it look? The precedence is specific, and worth stating exactly because our second opening error lives here: DT_RPATH (only honored if DT_RUNPATH is absent) → LD_LIBRARY_PATH → DT_RUNPATH (which applies only to the object's direct dependencies) → /etc/ld.so.cache → the default dirs (/lib, /usr/lib, …). And one rule that overrides all of it: if the stored name contains a /, it is treated as a path and no search happens at all. Hold that thought for Part VII.

You can watch the search happen. This is LD_DEBUG=libs running our binary, showing the RUNPATH-driven probe sequence for libmath.so:

root@container:/# LD_DEBUG=libs /code/dynamic_app
      3932:	find library=libmath.so [0]; searching
      3932:	 search path=/code/glibc-hwcaps/x86-64-v3:...:/code		(RUNPATH from file /code/dynamic_app)
      3932:	  trying file=/code/glibc-hwcaps/x86-64-v3/libmath.so
      3932:	  trying file=/code/tls/haswell/libmath.so
      ...

3.3 Filling the GOT: now, or later?

With every library mapped, the loader must make cross-object calls work. Your main() calls add(), but add lives in libmath.so at an address nobody knew until two milliseconds ago. The fix-up table for this is the GOT (Global Offset Table): a table of pointers, one per external thing, that the loader fills in with the real addresses. Calls and data accesses go through the GOT instead of embedding addresses directly.

There are two strategies for when the function-call slots get filled:

Eager (BIND_NOW): resolve every symbol at startup, before your code runs.
Lazy: leave function slots pointing at a resolver, and fix each one the first time it's called.

Textbooks (and the previous version of this post) describe lazy as "the default." On your distro, it probably isn't. Look at what Ubuntu's gcc actually passed to the linker (this is from gcc -v, Part V shows the full line): -pie -z now -z relro. That -z now means our default build is eager. The binary says so:

root@container:/code# readelf -d ./dynamic_app | grep -E 'FLAGS'
 0x000000000000001e (FLAGS)              BIND_NOW
 0x000000006ffffffb (FLAGS_1)            Flags: NOW PIE

root@container:/code# readelf -d ./dynamic_app_lazy | grep -E 'FLAGS'
 0x000000006ffffffb (FLAGS_1)            Flags: PIE

This is why our Makefile builds dynamic_app_lazy with -Wl,-z,lazy: in 2026 you have to ask for lazy binding to study it.

The difference is also a security posture, and it's visible in RELRO. RELRO (RELocation Read-Only) is the GNU_RELRO segment: after the loader finishes its patches, it mprotects that region read-only. With -z now you get full RELRO: every GOT slot is resolved up front, so all of them (including the function-call slots) sit inside the protected region. With lazy binding you get partial RELRO: the function-call slots (.got.plt) must stay writable so the resolver can patch them later. Our two builds show it directly: the R_X86_64_JUMP_SLOT entries for add and sleep land at 0x3fc8/0x3fd0 in the eager build, inside its RELRO region [0x3d90, 0x4000), but at 0x4018/0x4020 in the lazy build, past the end of its RELRO region [0x3dc8, 0x4000). Same program, same symbols; one layout locks the slots, the other leaves them writable forever. (That writable-GOT window is exactly the classic GOT-overwrite target, a point we'll return to below.)

3.4 Lazy binding, watched live

Eager binding is easy to imagine: a loop over relocation entries at startup (Appendix F walks it record by record). Lazy binding is the clever one, so let's watch it. Here is the machinery in the lazy binary, straight from objdump:

0000000000001169 <main>:
    ...
    1175:  be 0a 00 00 00        mov    $0xa,%esi
    117a:  bf 05 00 00 00        mov    $0x5,%edi
    117f:  e8 dc fe ff ff        call   1060 <add@plt>

0000000000001060 <add@plt>:                          ; .plt.sec
    1060:  f3 0f 1e fa           endbr64
    1064:  f2 ff 25 ad 2f 00 00  bnd jmp *0x2fad(%rip)   # 4018 <add's GOT slot>

0000000000001030 <.plt entry for add>:
    1030:  f3 0f 1e fa           endbr64
    1034:  68 00 00 00 00        push   $0x0             ; relocation index for 'add'
    1039:  f2 e9 e1 ff ff ff     bnd jmp 1020            ; the common resolver stub

main doesn't call add. It calls add@plt, a tiny trampoline that jumps through GOT slot 0x4018. And what does that slot contain before the first call? The file itself tells us. readelf -x .got.plt shows slot 0x4018 holding 0x1030: it points back into the PLT, at the very next instruction of the dance. So the first call goes main → add@plt → (through GOT) → push $0x0 → resolver, the resolver figures out which symbol relocation index 0 is, finds add in libmath.so, and patches the GOT slot so every later call jumps straight there.

Don't take my word for the patch: the demo binary can watch its own GOT slot change. got_watch.c (in the demo repo) reads the slot for add before and after the first call:

root@container:/code# ./got_watch
GOT slot for add lives at 0x555555558018
  before first call : 0x0000555555555030  <- points back into our own .plt
  add(5, 10) returns: 15      <- first call takes the resolver detour
  after first call  : 0x00007fffff7bd0f9  <- patched!
  libmath.so code   : 7fffff7bd000-7fffff7be000 r-xp ... /code/libmath.so

The before-value is our own image base plus 0x1030, exactly the push $0x0 stub in the .plt dump above. The after-value lands inside libmath.so's executable mapping: it's add itself.

One trap worth knowing (it bit this demo): got_watch.c is careful never to take &add. The moment a program takes a function's address, the linker must guarantee pointer equality across all objects, so it resolves that symbol eagerly through .plt.got and the lazy JUMP_SLOT you wanted to watch never exists.

fig. 3 · the same stub, two routes

our code GOT (data) loader

The full static evidence (the complete PLT disassembly, the initial .got.plt bytes, the relocation table) is in Appendix F. (If you'd rather drive this with gdb, use native x86-64 Linux: under Rosetta emulation ptrace is unavailable, which is exactly why the self-inspecting approach exists.)

3.5 Why not just call through the GOT directly?

A fair question: if calls go through a GOT slot anyway, why bother with the PLT stub at all? Why doesn't the compiler emit call *GOT_entry directly?

It can (-fno-plt does roughly that, and consequently forces eager binding). The traditional PLT exists to solve the "who called me?" problem that lazy binding creates. If an unresolved call *GOT_entry landed in the resolver, the resolver would have no idea which symbol you wanted: add? sleep? The PLT stub's push $0x0 is the missing ID: it pushes the relocation index so the resolver can look up exactly the right R_X86_64_JUMP_SLOT entry in DT_JMPREL and resolve precisely the intended symbol.

For the record-by-record version of everything above (how PT_DYNAMIC maps out the string/symbol/relocation tables, how R_X86_64_GLOB_DAT entries for things like __libc_start_main get resolved, and the full transcripts), see Appendix F.

Part IV: The Handoff (Loader → User)

The loader is now ready to hand control to your application. But it doesn't just call main(). In fact, it doesn't even know main exists.

The transition from the loader to your code happens in two steps.

1: The loader's exit (_dl_start_user) First, the loader runs the constructors (.init / .init_array) for all shared libraries (e.g., libmath.so) to ensure they are ready.

2: The application's entry (_start) The CPU lands at a function called _start. This is not your code. It is a small assembly stub provided by the C runtime (Scrt1.o, the position-independent sibling of the classic crt1.o, since our binary is a PIE) that was linked into your binary at build time. Its job is to set up the stack and pass arguments (argc, argv) to the C library helper __libc_start_main, which runs the constructors for your executable (the loader's _dl_init already ran the shared libraries' constructors) and finally calls your main.

(Curious what this assembly looks like? See Appendix G: The Assembly Handoff.)

That completes the relay from fig. 0: every leg of it has now crossed the page. Replay the whole thing, one step at a time; each caption should read as review, not news:

fig. 0b · the relay, replayed step by step

Part V: The Flashback (Build Time)

When we run gcc -c main.c, GCC acts as a driver. It runs cc1 (compiler) and as (assembler) to produce main.o.

At this stage, the compiler does not know where add is. It creates a relocation entry, basically a "to‑do" note for the linker.

Let's inspect main.o's relocation table, and the machine code it refers to:

root@container:/code# gcc -c main.c
root@container:/code# readelf -r main.o
Relocation section '.rela.text' at offset 0x1a0 contains 2 entries:
  Offset          Info           Type           Sym. Value    Sym. Name + Addend
000000000017  000400000004 R_X86_64_PLT32    0000000000000000 add - 4
000000000024  000500000004 R_X86_64_PLT32    0000000000000000 sleep - 4

root@container:/code# objdump -d main.o
0000000000000000 <main>:
   0:	f3 0f 1e fa          	endbr64
   ...
   c:	be 0a 00 00 00       	mov    $0xa,%esi
  11:	bf 05 00 00 00       	mov    $0x5,%edi
  16:	e8 00 00 00 00       	call   1b <main+0x1b>

Look at offset 0x16: a call instruction (e8) whose 4-byte operand is all zeroes. It "calls" the next instruction, because the compiler had nothing to put there. That's the hole.

Offset 0x17: the relocation points at the operand, one byte past the e8 opcode, the exact 4 bytes the linker must patch. (There are two entries because main also calls sleep.)
Type R_X86_64_PLT32: tells the linker: "I need a 32-bit PC-relative address to a PLT entry for symbol add."

5.1 The Hidden Startup Files

In Section 4 we saw that the real entry point is _start, not main(), and that it comes from the C runtime's startup object. But we never asked GCC to link that file. Where did it come from?

When you run gcc, it silently injects several startup objects provided by glibc: Scrt1.o (which contains _start; the plain crt1.o is used for non-PIE links), crti.o (init prologue), and crtn.o (init epilogue). The naming is historical: the original was called crt0.o (C RunTime, file zero), and the split into multiple files came later as initialization grew more complex. The S suffix marks the PIC/PIE variants.

You can see this hidden injection by running GCC with verbose flags:

root@container:/code# gcc -v -o dynamic_app main.o -L. -lmath 2>&1 | grep collect2
 .../collect2 ... -pie -z now -z relro -o dynamic_app
   .../x86_64-linux-gnu/Scrt1.o .../x86_64-linux-gnu/crti.o .../11/crtbeginS.o
   -L. ... main.o -lmath ... -lc ... .../11/crtendS.o .../x86_64-linux-gnu/crtn.o

Two things hide in that line. First, the startup files: since our binary is a PIE, gcc injects Scrt1.o (the position-independent variant of crt1.o; a non-PIE link would use crt1.o itself), plus crti.o/crtn.o and gcc's own crtbeginS.o/crtendS.o. Second, look again at the flags gcc chose without asking us: -pie -z now -z relro. That's the paper trail for both PIE-by-default (Part II) and eager-binding-by-default (Part III), sitting in one gcc -v invocation.

5.2 The Linker (`ld`)

Now ld runs. It has main.o, Scrt1.o, and libmath.so. It needs to create one file.

Step 1: The Blueprint (Linker Script)

The linker follows a script to decide memory layout.

root@container:/code# ld --verbose | grep -A 5 "SECTIONS"

Among many other directives, it tells the linker things like: "collect all input .text sections into one output .text section, all .rodata into one .rodata," and so on. It also defines segment boundaries, alignment, and the order things appear in the final binary.

Step 2: Weaving Sections Together

The linker maps the output file into memory (using mmap). It then performs a "scatter-gather" copy.

It copies Scrt1.o's .text to the beginning of the output buffer.
It copies main.o's .text right after it.
It updates its internal symbol map: main is no longer at offset 0; it is now at some final virtual address (and in PIE, that address is a relative virtual address that will receive a load bias at runtime).

Step 3: Synthesis (PLT & GOT)

The linker sees the R_X86_64_PLT32 relocation for add. It checks libmath.so and sees add is a shared symbol.

Allocate: it reserves space in .plt and .got.
Write: it writes the machine code instructions ("trampoline") into the PLT section, and reserves a GOT slot for the symbol.

But if add is resolved at runtime, why did the linker need libmath.so at all? It needs the file to verify that add actually exists, to record symbol and version requirements, and to write the DT_NEEDED tag so the loader knows to find and load libmath.so at runtime. (Linker flags like --allow-shlib-undefined or --unresolved-symbols can relax the existence check, but the default is to fail fast if a symbol can't be found.)

Step 4: Patching the Holes (Relocations)

Remember the relocation entry we saw earlier in main.o?

Offset 0x17    Type R_X86_64_PLT32    Symbol: add    Addend: -4

The linker now processes this. It does not scan the machine code looking for call instructions. It walks the .rela.text table, and for each entry it knows exactly which byte to patch and how.

For our add entry, the process is:

The linker looks at the Offset (0x17). That is where the placeholder bytes sit inside .text, the operand of the call at 0x16.
It knows from Step 3 that add@plt now lives at some address in the PLT section.
It computes: "how far is add@plt from this call site?" That distance is a 32-bit relative offset, which is what R_X86_64_PLT32 asks for. (The addend -4 accounts for the fact that x86 measures the offset from the end of the instruction, 0x17 + 4, not from the operand itself.)
It writes that offset into the 4 bytes at position 0x17, replacing the placeholder.

You can see the patched result in the final binary: the same instruction that read e8 00 00 00 00 in main.o reads e8 dc fe ff ff in dynamic_app_lazy: a PC-relative hop to add@plt (we saw it in Part III's disassembly).

Now when the CPU executes this call instruction at runtime, the offset points straight to add@plt.

Step 5: Sections to Segments

Finally, the linker maps output sections (.text, .data) to program headers (PT_LOAD).

It groups read-only sections (.text, .plt, .rodata) into segments so the kernel can protect them efficiently, and it may emit multiple PT_LOAD segments to match permissions and RELRO constraints.

Part VI: Static Linking

We have just spent multiple sections detailing the immense complexity of dynamic loading: the PLT, the GOT, the loader, runtime patching, and startup costs.

For many teams, this complexity is a feature: when a security vulnerability is found in a shared library like openssl, the OS can patch it once and every application that links against it picks up the fix without recompiling.

But for hyperscalers (like Meta, Google, and Netflix), this complexity is often a liability. They may opt for static linking, where every dependency is merged into a single executable file.

6.1 Why Hyperscalers Link Statically

Companies like Google and Meta prefer to statically link their production services. The reasons are practical:

Hermeticity (the "dependency hell" problem): Imagine a service that depends on PyTorch, which depends on libcuda.so, which depends on libgcc_s.so. If you deploy a dynamically linked binary to a production machine that has a slightly different version of libgcc, your service crashes at 3 AM. With static linking, the binary is self-contained: if it works on the build machine, it works in production.

Startup speed: Dynamic linking waits until runtime to resolve symbols, and for a large program this cost is not trivial. It can take seconds to calculate relocation mappings for large applications with hundreds of shared libraries. The loader must walk symbol hash tables, process relocations, and patch GOT entries, all the machinery we traced in Parts II through IV, before main() even starts. Static linking eliminates this entirely: there is no loader, no symbol resolution, and no PLT/GOT patching at runtime.

Real-world example: Meta's build system (Buck2) Meta's build systems (Buck/Buck2) were designed to manage these trade-offs using build modes. (This is based on my experience working on Meta's build infrastructure.)

@mode/dev (dynamic): used on developer laptops for fast iteration and quick incremental builds.
@mode/opt (static / more hermetic): used for production deployments to guarantee performance and hermeticity.

Buck2's open-source configuration system exposes dev and opt as standard constraint values, and the choice between them changes how every C++ dependency in the graph is linked.

6.2 The Consequence: The 2 GiB Relocation Barrier

Static linking sounds perfect until physics gets in the way. When you bundle an entire AI stack, or the transitive closure of a massive monorepo, into a single binary, it can grow to gigabytes. While working on Meta's build infrastructure, I regularly saw Buck2-built binaries exceed 25 GiB (including debug symbols) for large C++ services. At that point, a fundamental x86-64 limitation surfaces.

Recall from Step 4 in Section 5.2: the call instruction uses a 32-bit signed PC-relative offset. That gives it a reach of roughly ±2 GiB. If the linker cannot place a call target within that range, the link fails:

relocation truncated to fit: R_X86_64_PC32

This was not a theoretical problem. Large services with deep dependency graphs would hit this barrier, especially when built with instrumentation like -fprofile-generate or sanitizers that inflate code and data sections. Engineers sometimes refer to it colloquially as the "4 GB trap," but the underlying limit is the signed 32-bit reach of PC-relative relocations.

The brute-force fix is to compile with -mcmodel=large, which replaces the 5-byte relative call with a 12-byte movabs + call sequence that can reach any address. But this bloats every call site and increases register pressure, a steep price when you have millions of them.

The practical solution: link groups Rather than choosing between 10,000 tiny shared objects (too slow to load) or 1 giant static binary (too big to link), hyperscalers split the difference. They group related code into "islands": everything inside a group is statically linked together into one medium-sized .so, and the main binary dynamically links against just a handful of these groups.

Buck2 has this concept built directly into its C++ rules. A prebuilt_cxx_library_group bundles related libraries that must be linked together, and the auto_link_groups and link_group_map attributes on cxx_binary let the build system automatically partition the dependency graph into groups. The result: you might resolve 5 or 6 groups at startup instead of 50,000 individual DSOs, while keeping each group well within the 2 GiB barrier.

6.3 The Execution Flow (No Loader Involved)

If you build a fully static binary, the execution flow changes drastically. The loader (ld-linux.so) is removed from the picture entirely.

You can see this logic in the Linux kernel source fs/binfmt_elf.c. When you run a binary, the kernel checks for the PT_INTERP segment (which specifies the loader).

If PT_INTERP is missing (static binary):

No interpreter: the kernel does not map ld-linux.so into memory.
Direct entry: instead of setting RIP to the loader's _start, the kernel sets it directly to the binary's entry point (e_entry from the ELF header).
The new beginning: execution usually begins at _start (from the startup object, crt1.o for classic static builds), which sets up the stack and calls main.

There is no PLT indirection into shared libraries and no loader to wait for. (One nuance for the pedantic: even fully static glibc binaries perform a small amount of startup self-fixup, R_X86_64_IRELATIVE relocations for IFUNC symbols like the optimized memcpy variants, and a static-PIE binary relocates itself the way ld.so does, with no loader involved.) To a first approximation, though: the CPU just jumps straight into your code.

Everything we have covered so far happens before main() starts. But sometimes you need to load code after the program is already running: plugins, optional features, or hot-loaded extensions. This is what dlopen and dlsym provide. See Appendix H: Runtime Loading (dlopen/dlsym) for how the loader handles this and why it reuses much of the same machinery we have already seen.

Part VII: The Payoff — Both Errors, Solved

We opened with two production errors and a promise. Everything needed to keep it is now on the table.

7.1 `version 'GLIBC_2.34' not found`

Let's manufacture the error honestly: build the demo on a newer distro, run it on an older one.

# build on ubuntu:24.04 (glibc 2.39) ... then run on ubuntu:20.04 (glibc 2.31):
root@ubuntu20:/code# ./dynamic_app_glibc234
./dynamic_app_glibc234: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34'
    not found (required by ./dynamic_app_glibc234)

Where did the binary get the nerve to demand a specific glibc version? From the version tables we've been stepping around all post. Every dynamic symbol can carry a version requirement; readelf -V reads them straight out of our own default build:

root@container:/code# readelf -V ./dynamic_app
Version symbols section '.gnu.version' contains 8 entries:
  000:   0 (*local*)       2 (GLIBC_2.34)    1 (*global*)      1 (*global*)
  004:   1 (*global*)      1 (*global*)      3 (GLIBC_2.2.5)   3 (GLIBC_2.2.5)

Version needs section '.gnu.version_r' contains 1 entry:
  000000: Version: 1  File: libc.so.6  Cnt: 2
  0x0010:   Name: GLIBC_2.2.5  Flags: none  Version: 3
  0x0020:   Name: GLIBC_2.34  Flags: none  Version: 2

Read it as a contract: "I need libc.so.6, and from it I need symbols at version GLIBC_2.2.5 (that's sleep) and GLIBC_2.34 (that's __libc_start_main)." At link time, the linker recorded the version each symbol had in the libc it linked against. glibc 2.34 restructured its startup symbols, so anything linked against glibc ≥ 2.34 requires __libc_start_main@GLIBC_2.34. At load time, the loader checks .gnu.version_r against what the target's libc.so.6 actually exports (VERDEF tables), and refuses to start if a required version is missing. The error isn't mystical: it's the loader reading a table we can read ourselves.

The fix follows from the mechanism: build against the oldest glibc you must support (build in an old container, since glibc versions are backward-compatible, not forward), ship the runtime with the binary (containers), or take the loader out of the picture entirely (static linking, Part VI).

7.2 `cannot open shared object file`

Our Makefile's history contains this bug on purpose. Watch what happens if you link the "obvious" way, naming the file directly, instead of with -L. -lmath:

root@container:/code# gcc -o dynamic_app_broken main.c ./libmath.so -Wl,-rpath,'$ORIGIN'
root@container:/code# readelf -d ./dynamic_app_broken | grep -E 'NEEDED|RUNPATH'
 0x0000000000000001 (NEEDED)             Shared library: [./libmath.so]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000001d (RUNPATH)            Library runpath: [$ORIGIN]

root@container:/# cd / && /code/dynamic_app_broken
/code/dynamic_app_broken: error while loading shared libraries: ./libmath.so:
    cannot open shared object file: No such file or directory

There's our second opening error, self-inflicted. The DT_NEEDED entry became the literal string ./libmath.so. Remember the rule from Part III: a needed name containing / is used as a path, and every search mechanism is skipped. The RUNPATH [$ORIGIN] we carefully asked for is dead code; the binary only works when your current directory happens to contain the library. It ran fine in /code during development, then broke in production the first time someone ran it from anywhere else. Sound familiar?

The fixed link (-L. -lmath) stores a bare NEEDED [libmath.so], the search machinery engages, RUNPATH expands $ORIGIN to the binary's own directory, and it runs from anywhere:

root@container:/# /code/dynamic_app && echo "runs fine from /"

Triage order for this error in the wild: readelf -d the binary first. If DT_NEEDED contains a slash, you have this exact bug. If it's a bare name, run with LD_DEBUG=libs to watch the search and see which directories were probed (we did exactly this in Part III). Then fix it structurally (-Wl,-rpath,'$ORIGIN' for relocatable bundles, or ldconfig for system-wide installs) rather than exporting LD_LIBRARY_PATH in your shell profile and hoping.

Conclusion: The Full Cycle

Compiler: generates main.o with relocation entries ("holes").
Linker:
- injects Scrt1.o (the true entry point),
- weaves .text sections together based on the script,
- synthesizes PLT/GOT for dynamic symbols,
- patches the holes using the relocation table.
Running the application is a dance between user apps (terminal) and the kernel.
Kernel: maps segments, writes the auxv handshake, and invokes the interpreter (if PT_INTERP exists).
Loader: loads DSOs, applies relocations (eagerly under -z now, or lazily via the PLT), locks down RELRO. Calls _start → __libc_start_main → main().

The "simple" act of running ./app is a relay race passing the baton between the compiler, linker, kernel, and dynamic loader. And the two errors we started with are just the baton being dropped at two specific hand-offs: a version contract the loader can't satisfy, and a library search that never ran.

That follow-up now exists: Part 2, howtf can a device be both present and not found? traces a production incident where this machinery failed at scale: two collective communication libraries in one binary, a symbol collision that silently split one library's state into two live copies, and RDMA devices that were present, registered, and "not found." The resolution pipeline we just traced is the key to that diagnosis.

Appendices

Evidence lockers: the full dumps and gnarlier details the body text points at. Skip freely; return when a claim needs its receipts.

Appendix A: The Cross-Architecture Magic (Rosetta & QEMU)

If you ran this lab on an Apple Silicon Mac (M1/M2/M3) or a Windows ARM machine, you likely noticed that the x86-64 binary simply executed. It didn't crash, and it didn't require a manual emulator command.

Three pieces coordinate to make it happen:

a translation or emulation layer (Rosetta or QEMU),
the container/VM runtime (e.g., Docker Desktop / WSL2 / Apple's Virtualization Framework),
and the Linux kernel's binfmt_misc dispatch mechanism.

1) The Architecture Gap

Our host CPU speaks a different ISA than the guest binary. There are two broad approaches:

Emulation (QEMU-style): interpret/translate instructions and emulate architectural effects in software.
Translation (Rosetta-style): translate blocks of guest instructions into host instructions and cache/execute the translations.

Either way, the translator must preserve architectural semantics, not just instruction-by-instruction behavior. One example is memory ordering: x86's memory model is stronger (often described as TSO-like) than ARM's default. Translators must ensure the program observes x86-legal outcomes, which can require extra ordering constraints (i.e. inserting memory barriers) in the generated code or other clever mechanisms. That can affect performance.

Windows (QEMU Emulation): On Windows ARM, Docker commonly runs Linux containers inside a Linux VM (via WSL2). Cross‑arch support is frequently implemented by registering QEMU handlers with binfmt_misc, so that when the kernel encounters an x86‑64 ELF, it transparently invokes a QEMU interpreter (e.g., qemu-x86_64) to run it.
macOS (Rosetta + Hardware TSO): On macOS, Docker Desktop runs Linux containers inside a lightweight Linux VM and can integrate Rosetta into that VM so x86‑64 Linux binaries can run on Apple Silicon. Apple solved the memory ordering bottleneck at the silicon level. Their M-series chips include a hardware switch to enable Total Store Ordering (TSO). This allows the Rosetta translator to run without the heavy software barrier overhead, achieving near-native speeds.

2) How Rosetta Gets into the VM (VirtioFS Injection)

The Linux kernel inside our Docker VM does not ship with Rosetta. It is injected from macOS. Docker uses the Apple Virtualization Framework (AVF) to create the Linux VM. AVF exposes a specialized directory share called VZLinuxRosettaDirectoryShare. This is not a standard network share; it is a high-performance channel handled via VirtioFS (Virtual I/O File System).

When the VM boots, it detects this share and mounts it (usually to /run/rosetta). This makes the macOS rosetta binary visible and executable inside the Linux VM.

3) The Registration Command

Regardless of whether we use QEMU or Rosetta, the Linux Kernel mechanism is identical. It uses binfmt_misc (Binary Formats Miscellaneous).

When Docker Desktop starts (before our container is even created), its internal boot process sends a registration command to the kernel. Something equivalent of:

echo ':rosetta:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x3e\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/run/rosetta/rosetta:POCF' > /proc/sys/fs/binfmt_misc/register

(modeled on the registration example in Apple's documentation)

The kernel matches the file header against these bytes. The crucial part that identifies x86-64 is at Offset 18, which is 0x3e.

To verify this ourselves on an M1 with Docker Desktop (after making sure Rosetta is enabled in Settings):

# Start a privileged container
❯ docker run --rm -it --privileged ubuntu:22.04 bash

# Mount the binfmt filesystem
root@container:/# mount -t binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc

# Inspect the configuration
root@container:/# cat /proc/sys/fs/binfmt_misc/rosetta

interpreter /run/rosetta/rosetta
flags: POCF
magic 7f454c4602010100000000000000000002003e00

The POCF flags are documented in the kernel binfmt_misc docs: P (preserve argv[0]), O (open binary, pass an open fd to the interpreter), C (credentials, use the binary's credentials, not the interpreter's), and F (fix binary, keep the interpreter loaded so it works even inside mount namespaces/containers).

Appendix D: Segments Deep Dive

The full program-header dump behind Part II, plus the process's actual memory map.

readelf -lW ./dynamic_app (full output)

root@container:/code# readelf -lW ./dynamic_app

Elf file type is DYN (Position-Independent Executable file)
Entry point 0x1080
There are 13 program headers, starting at offset 64

Program Headers:
  Type           Offset   VirtAddr           PhysAddr           FileSiz  MemSiz   Flg Align
  PHDR           0x000040 0x0000000000000040 0x0000000000000040 0x0002d8 0x0002d8 R   0x8
  INTERP         0x000318 0x0000000000000318 0x0000000000000318 0x00001c 0x00001c R   0x1
      [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
  LOAD           0x000000 0x0000000000000000 0x0000000000000000 0x000670 0x000670 R   0x1000
  LOAD           0x001000 0x0000000000001000 0x0000000000001000 0x0001a5 0x0001a5 R E 0x1000
  LOAD           0x002000 0x0000000000002000 0x0000000000002000 0x0000e4 0x0000e4 R   0x1000
  LOAD           0x002d90 0x0000000000003d90 0x0000000000003d90 0x000280 0x000288 RW  0x1000
  DYNAMIC        0x002da0 0x0000000000003da0 0x0000000000003da0 0x000210 0x000210 RW  0x8
  NOTE           0x000338 0x0000000000000338 0x0000000000000338 0x000030 0x000030 R   0x8
  NOTE           0x000368 0x0000000000000368 0x0000000000000368 0x000044 0x000044 R   0x4
  GNU_PROPERTY   0x000338 0x0000000000000338 0x0000000000000338 0x000030 0x000030 R   0x8
  GNU_EH_FRAME   0x002004 0x0000000000002004 0x0000000000002004 0x000034 0x000034 R   0x4
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000 0x000000 RW  0x10
  GNU_RELRO      0x002d90 0x0000000000003d90 0x0000000000003d90 0x000270 0x000270 R   0x1

 Section to Segment mapping:
  Segment Sections...
   00     
   01     .interp 
   02     .interp .note.gnu.property .note.gnu.build-id .note.ABI-tag .gnu.hash .dynsym .dynstr .gnu.version .gnu.version_r .rela.dyn .rela.plt 
   03     .init .plt .plt.got .plt.sec .text .fini 
   04     .rodata .eh_frame_hdr .eh_frame 
   05     .init_array .fini_array .dynamic .got .data .bss 
   06     .dynamic 
   07     .note.gnu.property 
   08     .note.gnu.build-id .note.ABI-tag 
   09     .note.gnu.property 
   10     .eh_frame_hdr 
   11     
   12     .init_array .fini_array .dynamic .got

This output confirms that modern binaries are far more complex than the simple "Code vs. Data" model. The linker has split our binary into 4 distinct memory regions (LOAD segments) to maximize security and efficiency.

Explanation of the Output

1. The Header: DYN (Position-Independent Executable) This confirms our binary is a PIE. It has no fixed address. The Kernel will choose a random base address (ASLR) at runtime, and all VirtAddr values below (like 0x1000) are just offsets relative to that random base.

2. The INTERP Header

INTERP ... Requesting program interpreter: /lib64/ld-linux-x86-64.so.2

This is the first thing the kernel looks for. If found, the kernel maps this interpreter into memory and passes control to it.

3. The LOAD Segments (The Real Memory Map)

These 4 segments tell the Kernel exactly how to set up the Virtual Memory Areas (VMAs).

Segment	Flags	Offset	Content	Purpose
LOAD #1 (Metadata)	`R`	`0x000`	ELF Header, Program Headers, dynamic linking metadata (`.hash`, `.dynsym`)	Needed by the Loader, but should never be executed (security) or written to (integrity).
LOAD #2 (Code)	`R E`	`0x1000`	`.text` (your code), `.init`, `.plt`	The only region where the CPU can fetch instructions. Executing code anywhere else triggers an NX fault.
LOAD #3 (Constants)	`R`	`0x2000`	`.rodata` (string literals, constants), `.eh_frame` (unwind info)	Separated from executable code to prevent ROP gadgets from using data bytes as instructions.
LOAD #4 (Data)	`RW`	`0x3d90`	`.data` (globals), `.bss`, GOT (Global Offset Table)	The only writable memory. Backed by the file on disk until written, then Copy-on-Write kicks in.

4. The GNU_RELRO Segment (Security)

GNU_RELRO      0x...2d90 ... Flags R

This is a security overlay. Notice that its address (0x2d90) overlaps with the start of the LOAD #4 (RW) segment. See the RELRO section in Appendix F for more details.

5. GNU_STACK (NX Bit)

GNU_STACK ... Flags RW

The absence of the E flag here is critical. It tells the Kernel: "The stack is for data, not code." This prevents code-injection attacks on the stack.

Once the app starts running (that sleep(60) in main.c exists precisely so the process sticks around), we can read where everything actually landed. One honesty note: this capture comes from the emulated (Rosetta) container, where the kernel handed us 0x555555554000 (the canonical no-randomization PIE base) on every run. On native x86-64 Linux you'll see a different 0x55... bias per run; that per-run difference is ASLR, and load_bias from Section 2.2 is whatever the kernel picked. The structure below is identical either way: each LOAD segment became a VMA at load_bias + p_vaddr.

/proc/$pid/maps (rows for our binary, libmath, libc, ld-linux, stack)

root@container:/code# ./dynamic_app & pid=$!
root@container:/code# grep -E 'dynamic_app|libmath|libc|ld-linux|stack' /proc/$pid/maps
555555554000-555555555000 r--p 00000000 00:2d 86                         /code/dynamic_app
555555555000-555555556000 r-xp 00001000 00:2d 86                         /code/dynamic_app
555555556000-555555557000 r--p 00002000 00:2d 86                         /code/dynamic_app
555555557000-555555558000 r--p 00002000 00:2d 86                         /code/dynamic_app
555555558000-555555559000 rw-p 00003000 00:2d 86                         /code/dynamic_app
7fffff590000-7fffff5b8000 r--p 00000000 00:50 5451                       /usr/lib/x86_64-linux-gnu/libc.so.6
7fffff5b8000-7fffff74d000 r-xp 00028000 00:50 5451                       /usr/lib/x86_64-linux-gnu/libc.so.6
7fffff74d000-7fffff7a5000 r--p 001bd000 00:50 5451                       /usr/lib/x86_64-linux-gnu/libc.so.6
7fffff7a5000-7fffff7a6000 ---p 00215000 00:50 5451                       /usr/lib/x86_64-linux-gnu/libc.so.6
7fffff7a6000-7fffff7aa000 r--p 00215000 00:50 5451                       /usr/lib/x86_64-linux-gnu/libc.so.6
7fffff7aa000-7fffff7ac000 rw-p 00219000 00:50 5451                       /usr/lib/x86_64-linux-gnu/libc.so.6
7fffff7bc000-7fffff7bd000 r--p 00000000 00:2d 85                         /code/libmath.so
7fffff7bd000-7fffff7be000 r-xp 00001000 00:2d 85                         /code/libmath.so
7fffff7be000-7fffff7bf000 r--p 00002000 00:2d 85                         /code/libmath.so
7fffff7bf000-7fffff7c0000 r--p 00002000 00:2d 85                         /code/libmath.so
7fffff7c0000-7fffff7c1000 rw-p 00003000 00:2d 85                         /code/libmath.so
7ffffffc4000-7ffffffc6000 r--p 00000000 00:50 5433                       /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7ffffffc6000-7fffffff0000 r-xp 00002000 00:50 5433                       /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7fffffff0000-7fffffffb000 r--p 0002c000 00:50 5433                       /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7fffffffc000-7fffffffe000 r--p 00037000 00:50 5433                       /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7fffffffe000-800000000000 rw-p 00039000 00:50 5433                       /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
ffffedd69000-ffffedd8a000 rw-p 00000000 00:00 0                          [stack]

fig. 4 · the finished address space (from the maps dump above)

our app mapped libraries loader kernel

Appendix E: The Loader's Bootstrap (Self-Relocation)

In Section 3, we mentioned the loader must "fix itself." Here are the details.

The "Chicken and Egg" Problem

Normal programs rely on the loader to fix their addresses before they run. But ld-linux.so is the loader. Who loads the loader? No one.

When the kernel maps the loader, it just maps segments.

ASLR: loader is at a random address (e.g., 0x7f34...) instead of its link-time base.
Broken GOT: internal pointers may assume link-time addresses.
No libc: it can't call most libc routines yet.

The Solution: `_dl_start`

The entry point passes control to _dl_start in elf/rtld.c. This function is written with extreme care to avoid accesses that rely on unrelocated global state.

A simplified sketch:

/* elf/rtld.c */
// https://elixir.bootlin.com/glibc/glibc-2.39/source/elf/rtld.c#L517
static ElfW(Addr) __attribute_used__
_dl_start (void *arg)
{
    /* 1. Calculate the load bias */
    ElfW(Addr) l_addr = elf_machine_load_address ();

    /* 2. Apply bootstrap relocations (self-patch) */
    elf_machine_rela (l_addr, ...);   /* x86-64 is a RELA architecture */

    /* 3. Now the loader can safely run complex code */
    return _dl_start_final (arg, ...);
}

Step 1 finds the bias (often via RIP-relative tricks). Step 2 applies R_X86_64_RELATIVE-style relocations to itself. Once that's done, it becomes a "real program" and can load your app.

Appendix F: Loader's Relocation Mechanism

1) High-Level Sequence (What We're About to Zoom Into)

The loader starts running, but it itself is at a different location due to ASLR than what the linker had in mind.
Loader does the self-relocation as explained in Appendix E.
Now, it looks at the PT_DYNAMIC segment of the binary (note that these sections were already mapped as part of PT_LOAD mmap()ing by the kernel).
PT_DYNAMIC sort of creates a map of different entries (dynamic tags) that point the loader at the relevant tables/relocation lists:

map[DT_STRTAB] -> address of .dynstr (string table) map[DT_SYMTAB] -> address of .dynsym (dynamic symbol table, not .symtab) map[DT_NEEDED] -> list of libraries to load (actually, its an array of offsets into .dynstr, which contains these lib names) map[DT_RUNPATH] -> provided runpath (again, as an offset into .dynstr) map[RELA] -> address of .rela.dyn section, this has all the non-PLT relocations map[JMPREL] -> address of .rela.plt section, this has all the PLT relocations and a few more..

readelf -p .dynstr and readelf -d (string table + dynamic section)

# see how the lib names (libmath.so, libc.so.6) and $ORIGIN live here
root@container:/code# readelf -p .dynstr ./dynamic_app
String dump of section '.dynstr':
  [     1]  __cxa_finalize
  [    10]  _ITM_registerTMCloneTable
  [    2a]  _ITM_deregisterTMCloneTable
  [    46]  __gmon_start__
  [    55]  add
  [    59]  __libc_start_main
  [    6b]  sleep
  [    71]  libmath.so
  [    7c]  libc.so.6
  [    86]  GLIBC_2.2.5
  [    92]  GLIBC_2.34
  [    9d]  $ORIGIN

# inspect dynamic section
root@container:/code# readelf -d ./dynamic_app

Dynamic section at offset 0x2da0 contains 29 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libmath.so]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000001d (RUNPATH)            Library runpath: [$ORIGIN]
 0x000000000000000c (INIT)               0x1000
 0x000000000000000d (FINI)               0x1198
 0x0000000000000019 (INIT_ARRAY)         0x3d90
 0x000000000000001b (INIT_ARRAYSZ)       8 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x3d98
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0x3b0
 0x0000000000000005 (STRTAB)             0x498
 0x0000000000000006 (SYMTAB)             0x3d8
 0x000000000000000a (STRSZ)              165 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000015 (DEBUG)              0x0
 0x0000000000000003 (PLTGOT)             0x3fb0
 0x0000000000000002 (PLTRELSZ)           48 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x640
 0x0000000000000007 (RELA)               0x580
 0x0000000000000008 (RELASZ)             192 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000000000001e (FLAGS)              BIND_NOW
 0x000000006ffffffb (FLAGS_1)            Flags: NOW PIE
 0x000000006ffffffe (VERNEED)            0x550
 0x000000006fffffff (VERNEEDNUM)         1
 0x000000006ffffff0 (VERSYM)             0x53e
 0x000000006ffffff9 (RELACOUNT)          3
 0x0000000000000000 (NULL)               0x0

5. Iterates through `DT_NEEDED` entries. In our case: `libmath.so` and `libc.so.6`, as we can see in the output. (Note the `FLAGS: BIND_NOW` in this default build. The loader will resolve everything up front, per Part III.)

For libmath.so, the loader runs the search order from Part III: here, RUNPATH's $ORIGIN expands to the binary's directory and wins. It mmaps the library into the process, performing libmath's own relocations along the way, and does the same for libc.so.6. (Had the stored name contained a /, like Part VII's broken build, it would have been used as a literal path with no search at all.)
Then it will move to doing relocations for your executable. First it will look at map[RELA] (.rela.dyn) section.

2) Relocations for the Main Executable

First, let's see how the relocations information looks in our ELF binary.

readelf -rW ./dynamic_app (relocation tables)

root@container:/code# readelf -rW ./dynamic_app

Relocation section '.rela.dyn' at offset 0x580 contains 8 entries:

    Offset             Info             Type               Symbol's Value  Symbol's Name + Addend
0000000000003d90  0000000000000008 R_X86_64_RELATIVE                         1160
0000000000003d98  0000000000000008 R_X86_64_RELATIVE                         1120
0000000000004008  0000000000000008 R_X86_64_RELATIVE                         4008
0000000000003fd8  0000000100000006 R_X86_64_GLOB_DAT      0000000000000000 __libc_start_main@GLIBC_2.34 + 0
0000000000003fe0  0000000200000006 R_X86_64_GLOB_DAT      0000000000000000 _ITM_deregisterTMCloneTable + 0
0000000000003fe8  0000000400000006 R_X86_64_GLOB_DAT      0000000000000000 __gmon_start__ + 0
0000000000003ff0  0000000500000006 R_X86_64_GLOB_DAT      0000000000000000 _ITM_registerTMCloneTable + 0
0000000000003ff8  0000000700000006 R_X86_64_GLOB_DAT      0000000000000000 __cxa_finalize@GLIBC_2.2.5 + 0

Relocation section '.rela.plt' at offset 0x640 contains 2 entries:

    Offset             Info             Type               Symbol's Value  Symbol's Name + Addend
0000000000003fc8  0000000300000007 R_X86_64_JUMP_SLOT     0000000000000000 add + 0
0000000000003fd0  0000000600000007 R_X86_64_JUMP_SLOT     0000000000000000 sleep@GLIBC_2.2.5 + 0

It will go through each of the non-PLT relocations (i.e. in .rela.dyn) first.

Each entry is decoded into r_info first: r_info = index of this symbol into .dynsym (high 32 bits) || relocation type (low 32 bits).
The relocation type decides how to apply the relocation.
R_X86_64_RELATIVE (0x8) are relatively straightforward. It simply says that at this Offset + load_bias (remember load_bias from 2.2 Finding the Correct Address for the Segments?), put this value: addend + load_bias.
For R_X86_64_GLOB_DAT (0x6), it will look at the symbol in .dynsym. .dynsym usually contains symbol information like name (sym_name), type, visibility, value (sym_value) etc. For the name it points to an offset in .dynstr. Let's see it in action.
For __libc_start_main, it will look at the .dynsym[1] entry. 1 because, r_info = 0000000100000006 (first 32 bits is the index as mentioned before).

As we can see, index 1 has __libc_start_main.

readelf -sW --dyn-syms ./dynamic_app (dynamic symbol table)

root@container:/code# readelf -sW --dyn-syms ./dynamic_app

Symbol table '.dynsym' contains 8 entries:
   Num:    Value          Size Type    Bind   Vis      Ndx Name
     0: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT  UND
     1: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __libc_start_main@GLIBC_2.34 (2)
     2: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND _ITM_deregisterTMCloneTable
     3: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND add
     4: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND __gmon_start__
     5: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND _ITM_registerTMCloneTable
     6: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND sleep@GLIBC_2.2.5 (3)
     7: 0000000000000000     0 FUNC    WEAK   DEFAULT  UND __cxa_finalize@GLIBC_2.2.5 (3)
     ...

Loader will see that __libc_start_main is not defined (UND, usually indicated by sym_value being 0). It will try to find it. It will check where is this defined. Loader will get the symbol name (fetches it via .dynstr[dynsym[1].symbol_name]).
Each DSO's ELF would have maintained some sort of metadata (hash table) which loader will use to see which DSO defines this symbol. We have not dug deep into this part yet (we know .gnu.hash section does some fancy bloom filter things, but let's keep it for later).
Loader will find that libc.so defines __libc_start_main. It will find its value sym_value from .dynsym of libc.so, will add base_address_of_libc + sym_value and return that value. Note that, .dynsym contains a subset of symbols for that binary/DSO and not all symbols. It mainly contains the symbols used for import/export. The local symbols are not present/needed in .dynsym as they can be already resolved during build.
Then it will patch this value (i.e. __libc_start_main's absolute address) at (load_bias + 0000000000003fd8) address, which would be a GOT entry for this symbol.
Same thing for all the other R_X86_64_GLOB_DAT entries.
At runtime, the instruction will look like:

0000000000001080 <_start>:
    1080:   f3 0f 1e fa             endbr64
    1084:   31 ed                   xor    %ebp,%ebp
    1086:   49 89 d1                mov    %rdx,%r9
    ...
    1098:   48 8d 3d d9 00 00 00    lea    0xd9(%rip),%rdi        # 1178 <main>
    109f:   ff 15 33 2f 00 00       call   *0x2f33(%rip)        # 3fd8 <__libc_start_main@GLIBC_2.34>
    ...

this would automatically make a call to the address stored at relative 0x3fd8 address, which would now have absolute address of __libc_start_main in the process image.

Loader will go through PLT relocations next. map[JMPREL] points to .rela.plt table. Same way, we will go through these entries. For eager binding (when LD_BIND_NOW=1 is set) we will do the relocations at the startup time; for lazy binding (default) this sequence will take place at runtime when the first call is made. Regardless, the same sequence of events take place. R_X86_64_JUMP_SLOT type of entry indicates PLT entry.

3) The Lazy-Binding Path: the Full Evidence

Part III (§3.4) watches lazy binding happen live with got_watch. Here is the complete static evidence from the -Wl,-z,lazy build, so every number in that walkthrough can be checked against the file on disk.

The PLT relocations the resolver will service:

Relocation section '.rela.plt' at offset 0x640 contains 2 entries:
    Offset             Info             Type               Symbol's Value  Symbol's Name + Addend
0000000000004018  0000000300000007 R_X86_64_JUMP_SLOT     0000000000000000 add + 0
0000000000004020  0000000600000007 R_X86_64_JUMP_SLOT     0000000000000000 sleep@GLIBC_2.2.5 + 0

The PLT machinery itself (note the modern endbr64/bnd, which is CET/IBT hardening, and the .plt.sec split):

objdump -d dynamic_app_lazy, .plt and .plt.sec sections

./dynamic_app_lazy:     file format elf64-x86-64


Disassembly of section .plt:

0000000000001020 <.plt>:
    1020:	ff 35 e2 2f 00 00    	push   0x2fe2(%rip)        # 4008 <_GLOBAL_OFFSET_TABLE_+0x8>
    1026:	f2 ff 25 e3 2f 00 00 	bnd jmp *0x2fe3(%rip)        # 4010 <_GLOBAL_OFFSET_TABLE_+0x10>
    102d:	0f 1f 00             	nopl   (%rax)
    1030:	f3 0f 1e fa          	endbr64 
    1034:	68 00 00 00 00       	push   $0x0
    1039:	f2 e9 e1 ff ff ff    	bnd jmp 1020 <_init+0x20>
    103f:	90                   	nop
    1040:	f3 0f 1e fa          	endbr64 
    1044:	68 01 00 00 00       	push   $0x1
    1049:	f2 e9 d1 ff ff ff    	bnd jmp 1020 <_init+0x20>
    104f:	90                   	nop

Disassembly of section .plt.sec:

0000000000001060 <add@plt>:
    1060:	f3 0f 1e fa          	endbr64 
    1064:	f2 ff 25 ad 2f 00 00 	bnd jmp *0x2fad(%rip)        # 4018 <add@Base>
    106b:	0f 1f 44 00 00       	nopl   0x0(%rax,%rax,1)

0000000000001070 <sleep@plt>:
    1070:	f3 0f 1e fa          	endbr64 
    1074:	f2 ff 25 a5 2f 00 00 	bnd jmp *0x2fa5(%rip)        # 4020 <sleep@GLIBC_2.2.5>
    107b:	0f 1f 44 00 00       	nopl   0x0(%rax,%rax,1)

And the initial contents of .got.plt in the file, before the loader has touched anything. The slot for add (vaddr 0x4018) holds 0x1030, the address of its own PLT push-stub:

Hex dump of section '.got.plt':
 NOTE: This section has relocations against it, but these have NOT been applied to this dump.
  0x00004000 d83d0000 00000000 00000000 00000000 .=..............
  0x00004010 00000000 00000000 30100000 00000000 ........0.......
  0x00004020 40100000 00000000                   @.......

So the chain at first call is exactly: call add@plt → jmp *slot(0x4018) → lands back at 0x1030 → push $0x0 (the relocation index) → common stub at 0x1020 → _dl_runtime_resolve, which finds add in libmath.so, patches slot 0x4018, and jumps there. Every later call short-circuits: call add@plt → jmp *slot → add.

(If you want to poke this in gdb yourself, run it on native x86-64 Linux: under Rosetta emulation ptrace is unavailable, which is why the in-process got_watch approach exists.)

RELRO (RELocation Read-Only): partial vs full

At the end of relocation processing the loader mprotects the GNU_RELRO region read-only. How much protection that buys depends entirely on the binding mode. Compare our two builds:

# default build (-z now, eager):
GNU_RELRO      0x002d90 0x0000000000003d90 0x0000000000003d90 0x000270 0x000270 R   0x1
# lazy build (-Wl,-z,lazy):
GNU_RELRO      0x002dc8 0x0000000000003dc8 0x0000000000003dc8 0x000238 0x000238 R   0x1

Both regions end at 0x4000. In the eager build the R_X86_64_JUMP_SLOT entries live at 0x3fc8/0x3fd0, inside the region. Everything is resolved before main() runs, so the loader can seal the entire GOT: full RELRO. In the lazy build the jump slots live at 0x4018/0x4020, past the end. They must stay writable so the resolver can patch them at first call: partial RELRO.

The security consequence, stated precisely: under full RELRO, a memory-corruption bug cannot overwrite GOT entries to hijack library calls, because those pages are read-only by the time user code runs. Under partial RELRO, the data GOT is sealed but .got.plt remains writable for the process's lifetime, and a writable function-pointer table that the program jumps through is the canonical GOT-overwrite target. That trade, startup latency vs a locked GOT, is exactly why hardened distros ship -z now by default.

Once these relocations are done, we are ready to handoff to _start.

Appendix G: The Assembly Handoff (_start)

In Section 4, we glossed over the assembly handoff. Here are the exact mechanics of how the loader passes control to the user.

1) The Exit Stub (`_dl_start_user`)

The loader is written in C, but the final handoff requires assembly to manipulate registers precisely. This happens in architecture-specific glue (e.g., sysdeps/x86_64/dl-machine.h in glibc).

A schematic flow:

_dl_start_user:
    mov %rsp, %rdi         # Save stack pointer (argc/argv live here)
    call _dl_init          # Run init functions for DSOs
    jmp *%r12              # Jump to user entry point (_start)

2) The User Entry Point (`_start`)

The CPU lands at _start. This is provided by Scrt1.o (the PIE variant of crt1.o). Its primary job is to align the stack (16‑byte alignment required by the x86‑64 ABI) and set up arguments for __libc_start_main.

Conceptually:

_start:
    xor %ebp, %ebp         # End-of-stack marker for debuggers
    pop %rsi               # argc
    mov %rsp, %rdx         # argv
    and $-16, %rsp         # align stack
    call __libc_start_main

See the exact source code. Then __libc_start_main runs constructors for this binary (remember that the loader (_dl_init) already initialized shared libraries. __libc_start_main only runs constructors for the main executable) and calls our main.

Appendix H: Runtime Loading (dlopen/dlsym)

Everything in the main article happens before main() starts. But many real programs need to load code later: a web server that loads authentication modules on demand, a game engine that loads renderer backends based on the GPU it detects, or a language runtime loading compiled extensions. The mechanism for this is dlopen and dlsym.

1) Loading a Library After Startup

Suppose your program has an optional plugin system. At runtime, you decide to load a plugin:

void *handle = dlopen("./libplugin.so", RTLD_LAZY);

Under the hood, this calls back into the same dynamic loader (ld-linux.so) that set up your process at startup. The loader finds libplugin.so, maps it into the process's address space with mmap, resolves its dependencies (if libplugin.so itself depends on other libraries), and performs relocations, the same machinery we saw in Appendix F, just happening after main() instead of before it.

2) Initialization: Why `dlopen` Can Be Slow (or Crash)

Before dlopen returns, the loader must run the constructors (.init_array) of libplugin.so and all of its dependencies. This is the same initialization step the loader performs for startup libraries, but it happens synchronously inside your dlopen call.

This has a practical consequence: if libplugin.so contains a C++ global like MyClass instance;, that constructor runs inside dlopen. If it crashes, allocates a lot of memory, or takes a long time, your dlopen call inherits that behavior. The library must be fully initialized before you get the handle back.

3) Looking Up Symbols (`dlsym`)

Once dlopen returns successfully, you have an opaque handle. Internally, this is a pointer to the link_map structure the loader created when it mapped the library, the same structure it uses to track every shared library in the process.

To call a function from the loaded library:

void (*func)() = dlsym(handle, "run_plugin");
func();

The loader searches starting from that link_map (the object itself and its dependency subtree, not just the single object) and returns the memory address of run_plugin. From this point on, you call func() like any other function pointer.

This is conceptually how Python loads C extensions: import numpy eventually triggers a dlopen on the compiled NumPy shared object, and dlsym is used to find the entry points that bridge Python calls to the C implementation.

howtf.io

howtf can a device be both present and not found?

1. Present, and not found

2. The investigation

3. What was actually happening

4. Reproducing it

The production topology, reproduced

The trigger, isolated

5. Why nothing warned

6. Fixes: the folklore one that fails, and the ones that work

7. How common is this, really?

8. What should change

Appendices

Appendix A: The Buck machinery (omnibus, link groups, and the 2 GiB wall)

1) Omnibus: the merge

2) The 2 GiB wall

3) Link groups: the partition

Appendix B: Two preconditions for the capture

1) The copies must agree on rdma-core's private ABI number

2) Symbol versioning does not block the match

Appendix C: The reproducer's edge rows (hidden visibility and copy relocation)

Appendix D: symsplit (a binding simulator, not a duplicate lister)

Appendix E: The wheel survey (full numbers)

1) Route B: resident duplicate runtimes

2) Route A: the trigger census

3) The import-order probe

howtf does ./app reach main()?

Follow Along

Part I: The Hardware Gate and Kernel Entry

1.1 The Wake Up

1.2 The fork() syscall (cloning)

1.3 The execve Syscall

1.4 Inside the Kernel: fs/exec.c

1.5 The Magic Check

Part II: Mapping the Memory

2.1 Iterating Segments (load_elf_binary)

2.2 Finding the Correct Address for the Segments

2.3 The Fork in the Road: PT_INTERP

Part III: The Loader Takes Control (User Mode)

3.1 Self-Relocation (The Bootstrap)

3.2 Dependency Discovery

3.3 Filling the GOT: now, or later?

3.4 Lazy binding, watched live

3.5 Why not just call through the GOT directly?

Part IV: The Handoff (Loader → User)

Part V: The Flashback (Build Time)

5.1 The Hidden Startup Files

5.2 The Linker (ld)

Step 1: The Blueprint (Linker Script)

Step 2: Weaving Sections Together

Step 3: Synthesis (PLT & GOT)

Step 4: Patching the Holes (Relocations)

Step 5: Sections to Segments

Part VI: Static Linking

6.1 Why Hyperscalers Link Statically

6.2 The Consequence: The 2 GiB Relocation Barrier

6.3 The Execution Flow (No Loader Involved)

Part VII: The Payoff — Both Errors, Solved

7.1 version 'GLIBC_2.34' not found

7.2 cannot open shared object file

Conclusion: The Full Cycle

Appendices

Appendix A: The Cross-Architecture Magic (Rosetta & QEMU)

1) The Architecture Gap

2) How Rosetta Gets into the VM (VirtioFS Injection)

3) The Registration Command

Appendix D: Segments Deep Dive

Explanation of the Output

Appendix E: The Loader's Bootstrap (Self-Relocation)

The "Chicken and Egg" Problem

The Solution: _dl_start

Appendix F: Loader's Relocation Mechanism

1) High-Level Sequence (What We're About to Zoom Into)

2) Relocations for the Main Executable

3) The Lazy-Binding Path: the Full Evidence

RELRO (RELocation Read-Only): partial vs full

Appendix G: The Assembly Handoff (_start)

1) The Exit Stub (_dl_start_user)

2) The User Entry Point (_start)

Appendix H: Runtime Loading (dlopen/dlsym)

1.2 The `fork()` syscall (cloning)

1.3 The `execve` Syscall

1.4 Inside the Kernel: `fs/exec.c`

2.1 Iterating Segments (`load_elf_binary`)

2.3 The Fork in the Road: `PT_INTERP`

5.2 The Linker (`ld`)

7.1 `version 'GLIBC_2.34' not found`

7.2 `cannot open shared object file`

The Solution: `_dl_start`

1) The Exit Stub (`_dl_start_user`)

2) The User Entry Point (`_start`)

2) Initialization: Why `dlopen` Can Be Slow (or Crash)

3) Looking Up Symbols (`dlsym`)