Every debugging story I have starts the same way: staring at a bug, asking howtf did that happen?
This site is a collection of those moments. I try to demystify the bugs that live at the deepest levels of the systems stack. There is an immense, almost ridiculous pleasure in stripping away the magic and feeling that computers can actually be understood. Or at least I tell myself that, until the next crash.
The Linux memory-management mechanism underneath the incident: pages, pageblocks, CMA, FOLL_PIN, FOLL_LONGTERM, migration, and the second pin that returned ENOMEM. Part 3 of Memory Registration, All the Way Down.
A production debugging story across NCCL, ConnectX, nvidia-peermem, DMA-BUF, retsnoop, and an inherited six-gigabyte CMA reserve. Part 2 of Memory Registration, All the Way Down.
A ground-up walk through DMA, RDMA, PCIe BARs, memory registration, MKeys, nvidia-peermem, and DMA-BUF. Part 1 of Memory Registration, All the Way Down.
An NCCL 2.17.1 deep dive: an abort path waiting for a peer-visible close, a file descriptor duplicated by fork(), and the shutdown() that 2.18.1 added.
An RDMA deep dive: a GPUDirect visibility fence registered four bytes of host memory, and forked children started segfaulting on pointers the parent could read fine. The page-granular fork rule hiding under ibv_reg_mr().