NUMA Part 4: Closing the Xen dom0 I/O Gap

In the Part 3 of this series, we walked through the Xen paravirtual I/O architecture and what makes it expensive on a Non-Uniform Memory Access (NUMA) host: foreign-mapped pages with no NUMA metadata, kthreads landing wherever the scheduler happened to put them, an information gap between the dom0 kernel and the host's actual topology. We closed that part on a promise that the fix has two pieces - a structural one and a per-component one - and that part 4 would deliver both.

This part makes good on that - and along the way it covers two things we did not plan to fix. One we hit before the real work could start: dom0's own memory placement was so skewed that synthesising a topology for dom0 without fixing it first would have been an exercise in lying to a kernel that had no memory to back the lie with. That one is a prerequisite, not a detour - everything else in this part is built on top of it. The other we hit after the work was nominally done: a memory-placement bug in our own toolstack that had been silently broken for as long as multi-vnode vNUMA had existed in our tree, and only ever surfaced under a memory-bandwidth benchmark. That one is the genuine detour, and it waits until the end.

Let's start with the prerequisite.

The First Discovery: dom0's Memory Was in the Wrong Place

Stock Xen, when booting dom0, has to trim the host's BIOS memory map (the E820) down to the subset of memory dom0 is supposed to own. The trim walks the host E820 in physical address order, marking regions as dom0's RAM until it has set aside enough to satisfy dom0_mem. On a single-node host, this is harmless; dom0's memory comes from the only node available. On a multi-socket host, the same code is a real problem. The host's RAM regions are typically grouped by NUMA node in physical address order, so walking the E820 in order means dom0 gets all its memory from the lowest-address nodes first.

The worst case is severe. On a 128 GiB, 8-node host with dom0_mem=35% - the configuration our lab box runs - dom0 takes 100% of the lowest-address nodes' RAM and 0% of the others. Nodes 0 through 2 are almost entirely consumed by dom0; nodes 3 through 7 have no dom0-owned pages at all. Without dom0 being able to even attempt an allocation on the remote nodes, making dom0 NUMA-aware would have been synthesising a topology the kernel could never act on - the SRAT we generated would have described one node where 60% of dom0's memory lived alongside six nodes where it had nothing at all.

Our trim replaces this in-order grab with a proportional one, and it is one of the places our tree diverges from upstream Xen. It runs in two passes: pass one sums total host RAM; pass two gives each region a proportional share of the dom0_mem budget. The arithmetic is Bresenham-style, with rounding remainders accumulating so the total ends up exactly equal to dom0_mem. Dom0's memory ends up spread across every host node in proportion to that node's share of physical RAM.

With dom0 actually owning memory on every host node, the rest of the work could begin.

Making dom0 NUMA-aware

Three Pieces of NUMA Topology, for Any Xen Domain

Recall from part 2 that a NUMA-aware Xen domain sees three pieces of topology:

• The ACPI System Resource Affinity Table (SRAT), which maps CPUs and memory ranges to NUMA nodes.
• The ACPI System Locality Information Table (SLIT), which gives the inter-node distance matrix.
• The x2APIC IDs in CPUID, which encode the per-CPU package/core/thread identifiers.

Any Xen domain that wants accurate NUMA awareness needs all three to be present and consistent. Upstream Xen does most of this work for PVH and HVM domUs when the toolstack asks for a vNUMA layout via XEN_DOMCTL_setvnumainfo: SRAT and SLIT are synthesised to match the requested topology. Upstream Xen does not synthesise the x2APIC IDs in CPUID from the vNUMA layout, however. That third piece has been an Edera extension since before the work this post describes began, and we extended it further along the way to handle non-power-of-two vCPU counts in multi-node domUs.

Dom0 is a different story. Until the work described in this post, upstream Xen synthesised none of the three for dom0. Dom0 saw a single flat NUMA node regardless of how many physical nodes the host had. The work this section covers extends the same three-piece synthesis to dom0, under one specific set of conditions.

The conditions: dom0 must be PVH; it must boot with dom0_vcpus_pin=1 so each dom0 vCPU is hard-pinned 1:1 to a specific host pCPU; the dom0 vCPU count must equal the host pCPU count so that every pCPU has a corresponding dom0 vCPU; and the host must have more than one NUMA node. Those conditions also have a useful side effect for the implementation. Because every dom0 vCPU N is pinned to pCPU N, the vNUMA node assignment for vCPU N is simply the host's NUMA node for pCPU N. There is no layout algorithm to design, no placement decisions to make at boot, and no edge cases for partially-covered host topologies, because dom0 covers the whole host by definition. The implementation works out to a direct walk over the host's cpu_to_node table.

When the conditions are met, dom0 gets:

• A synthesised SRAT describing one proximity domain per host NUMA node that dom0 spans, with the appropriate CPU set and the appropriate memory regions in each.
• A synthesised SLIT sliced directly from the host's own SLIT - the distances dom0 sees are the real inter-node distances, not the default "10 local, 20 remote" stub that Linux substitutes when SLIT is absent.
• x2APIC IDs in CPUID encoded to match the synthesised topology, using the same machinery that handles the domU case.

After this, numactl -H inside dom0 finally shows the real topology - eight nodes where it used to report one, each with the CPUs and memory that actually belong to it. Storage daemons inside dom0 see the real topology. Container runtimes inside dom0 see the real topology.

And the kernel's own automatic NUMA balancing finally has something to act on: with an accurate map it samples where each task runs and, over time, migrates that task's pages onto the same node, healing remote accesses on its own. That self-healing does not exist against a flat one-node view - there is nothing to balance - so it is one more thing the topology work unlocks. It is a fallback rather than a plan, though: a sampling heuristic with its own cost, and the kernel's guidance is to switch it off for workloads that are already pinned - which is exactly what the later sections do to the I/O path.

The dom0 shell of every Edera deployment now operates on accurate information for the first time.

It's also worth noting that dom0 NUMA-awareness really matters on mixed nodes. Plenty of Edera deployments are not all-Edera: the same machine runs some pods as ordinary containers under the default runtime - directly in dom0 - and others under the Edera runtime, each sealed into its own zone. Those ordinary containers are dom0 workloads like any other, and before this work they ran NUMA-blind. A non-Edera pod could end up slower on an Edera host than it had been on the same box as plain Linux, purely because turning the machine into a Xen dom0 hid the topology its workloads used to see. Making dom0 NUMA-aware closes that gap: the containers that stay in dom0 get the topology they would have seen on bare metal, so adopting Edera does not quietly tax the workloads that were never sandboxed in the first place.

Why SRAT, SLIT, and CPUID All Matter

It is tempting to assume the ACPI tables alone are enough. The kernel reads them at boot, the NUMA infrastructure picks up the topology, you are done. For some consumers that is enough. The hwloc library, which most NUMA-aware userspace tooling is built on, derives its view of the topology from the kernel's sysfs view (which the kernel in turn populates from the SRAT and SLIT). Get those two right and the sysfs surface is populated correctly, and hwloc is happy.

Other consumers go further, and skipping CPUID quietly breaks them. The LLVM OpenMP runtime, used by parallel-loop-heavy workloads in compiled languages, derives its socket and core topology from the x2APIC CPUID leaf rather than from sysfs. On a Xen guest where CPUID has been left at the default single-socket layout but SRAT and SLIT are correct, OpenMP concludes the machine is single-socket and assigns its worker threads accordingly - which on a multi-socket host means stuffing every worker into one node's worth of cores. The kernel scheduler will rebalance over time, but OpenMP's initial assignment is set in stone, and the resulting placement is bad enough that benchmark numbers visibly suffer.

(LLVM OpenMP does have a compile-time option to use hwloc for topology discovery instead of reading CPUID directly, but it is not enabled by default and we have not found a binary distribution that turns it on. The CPUID path is what production builds run against.)

Conveniently, the runtime will show you its reasoning. Run an OpenMP workload with KMP_AFFINITY=verbose using the LLVM OpenMP runtime, and the runtime prints both the topology it discovered and the thread placement it chose; on an Edera guest, where the x2APIC IDs are synthesised to match the real layout, both come out correct - the same whether the workload runs in dom0 or inside a domU.

The lesson generalises beyond OpenMP. Making the topology visible to all of its consumers is the only way to make it actually useful. SRAT alone, SLIT alone, CPUID alone - none of those are enough on their own, because different consumers in different ecosystems read different sources. Doing two out of three would have meant choosing which class of workload to quietly disappoint.

Bugs Found While Building dom0 vNUMA

Getting the synthesis right turned up a handful of bugs we had not expected.

The first was a node-numbering bug. Xen renumbers proximity domain identifiers during boot into a dense internal node ID space, in SRAT scan order. The first version of the dom0 vNUMA build code used those internal Xen-side node IDs as the vNUMA node index, which then leaked into the SRAT we generated for dom0 and into the encoded APIC IDs. The result: a dom0 vCPU pinned to a pCPU on host proximity domain 1 might show up inside dom0 as belonging to "socket 3" if Xen's renumbering had landed proximity domain 1 into Xen's internal slot 3. Tools running inside dom0 would target the wrong physical socket if they were keying off proximity domain numbers. The fix was to index by host proximity domain directly, not by the Xen-internal node ID, so the SRAT, APIC IDs, and SLIT all use the same host-firmware numbering consistently.

The second was about how we built the memory ranges in the generated SRAT. The first version split each host RAM region equally across all the vNUMA nodes dom0 had, treating the split as proportional. This was wrong on two counts. Each host RAM region physically lives on exactly one NUMA node, so chopping it across vnodes claimed memory sat on nodes that could not actually host it. And Linux's NUMA registration code rejects an SRAT whose memory affinity entries do not fully cover its memblock view - the equal-split entries did not cover the full host E820, and dom0's dmesg complained about NUMA: no nodes coverage for 170175 MB of 261441 MB RAM before falling back to a faked single-node layout. The fix: emit one vmemrange per physical NUMA memblk, filtered to the nodes dom0 actually spans. Full coverage; honest ownership.

The third was the subtle one, and it was the original memory skew in a new hiding place: the proportional E820 trim itself. That trim assumes each host RAM region lives wholly on one NUMA node - true on a multi-socket host, where the firmware splits high RAM at node boundaries, but false on single-socket sub-NUMA layouts (Intel Sub-NUMA Clustering, or AMD Nodes Per Socket (NPS) of 2 or more), where one contiguous region can span two nodes. When the trimming pass cuts such a region down to its share, the memory it keeps all lands on the first node, leaving the second with almost nothing - about 1 GiB where the SRAT promised 32 - so dom0's per-node bindings have no memory to allocate against on that node. The skew was back, just harder to spot because it surfaced only on sub-NUMA hardware. The fix: split each region at NUMA node boundaries before taking its share.

After these fixes plus the topology synthesis, dom0 is NUMA-aware in the way the rest of the series has assumed throughout. That is half the fix. The other half is the per-request information that the dom0 backends need in order to do something useful with the topology.

Showing dom0 Where Foreign Frames Live

Even with dom0 fully NUMA-aware, the per-request placement problem from part 3 remains: when a domU grants dom0 access to a ring page, dom0's kernel has no way to learn what host node that page lives on. The struct page representing the ring in dom0's mem_map is a placeholder drawn from xen_alloc_unpopulated_pages. The stock-Linux version of that allocator registers its underlying memory section with NUMA_NO_NODE, so page_to_nid on any grant-mapped ring page returns NUMA_NO_NODE regardless of where the foreign frame actually lives.

That single missing number is the whole problem. Dom0 now has an accurate map of the host, but the one question that decides every backend's placement - which node is this ring on? - comes back "don't know". So the kthread that services the ring, the IRQ that wakes it, and the dom0 I/O stack behind them all get placed without reference to where the ring actually lives, and the path drops straight back into the every-hop-crosses case from part 3 - on a dom0 that is now fully NUMA-aware and has every fact it needs but this one.

The piece of information dom0 needs, the host's proximity domain for a given machine frame, is sitting inside the hypervisor. It just is not exposed to dom0 yet: the answer exists, one privilege boundary away. The first half of the structural fix is to expose it. We added a new Xen hypercall, XENMEM_get_mfn_pxms, that takes a batch of machine frame numbers and returns the proximity domain identifier for each. Permission is hardware-domain only: no untrusted guest has a legitimate reason to need this information, and exposing it broadly would leak details about the physical host topology.

There are three NUMA identifier namespaces floating around this code, which is worth a brief aside. Host proximity domain (the firmware-level identifier, the one ACPI SRAT uses), Xen-internal node IDs (assigned in SRAT scan order), and Linux node IDs (dom0's own NUMA node numbering). The hypercall returns proximity domain because that is the value space dom0's own SRAT already uses; pxm_to_node() is a single standard kernel call to translate the result to a Linux node ID. Returning Xen-internal node IDs would have forced every caller to maintain its own translation table.

With the hypercall in place, the second half of the structural fix is to teach dom0's unpopulated-page allocator to keep per-node pools. The allocator now keeps a separate free list per Linux node, and each section-sized backing region is registered with the correct node ID via memremap_pages(pgmap, node). Pages drawn from node N's pool report page_to_nid() == N by construction. Backend drivers asking for a ring page's host node call page_to_nid and get a real answer back.

One important piece of backwards compatibility ties the two fixes together. The hypercall is a downstream addition we ship in Edera's Xen build; a dom0 kernel that includes the new NUMA-aware code must still boot correctly against a stock upstream Xen that does not have the hypercall. We arrange for exactly that. The dom0 kernel probes for the hypercall at startup; if it is missing, the kernel falls back to the existing NUMA-oblivious behaviour across the entire backend driver stack. New Linux on old Xen is byte-equivalent to no patches at all - which is the only way we can ship Linux-side NUMA work that does not demand lockstep Xen updates across customers' deployments.

Teaching the Backends NUMA Affinity

With the structural fix in place, dom0's backend drivers can finally read a meaningful answer to "what host node owns this ring page?" and place themselves accordingly. The pattern across both xen-blkback and xen-netback is the same:

• At ring-map time, ask page_to_nid for the ring's host node.
• Create the per-ring (or per-queue) backend kthread with kthread_create_on_node so the task_struct and the initial kernel stack are allocated on the right node from the start. A bare set_cpus_allowed_ptr after the kthread is created leaves the stack on the caller's node and pays for it on every context switch.
• Pin the kthread to that node's cpumask via set_cpus_allowed_ptr.
• Bind the event-channel IRQ on the same node and set the IRQ's affinity hint to the node's cpumask.

That pattern was the original design. It did not quite work.

The first thing that did not work was the IRQ affinity. We were calling irq_set_affinity_and_hint correctly, but irqbalance, the userspace IRQ load-balancing daemon that ships and runs by default on most Linux distributions, was overwriting our affinity choice almost immediately. The root cause: the IRQ descriptor itself had been allocated with NUMA_NO_NODE baked into it by Xen's event-channel driver, so /proc/irq/N/node read -1, and irqbalance treats node-unaware IRQs as candidates for global rebalancing regardless of the affinity hint we had set.

The fix was to add _on_node variants of the four event-channel bind helpers Xen front/back ends use. The new variants thread the caller's node through to a new xen_allocate_irq_dynamic_node, so the IRQ descriptor is allocated with the correct node attribute from the start. After this change, /proc/irq/N/node reflects the node the caller asked for, and irqbalance respects the affinity hint as a NUMA-local subset of CPUs rather than treating the whole IRQ as floating.

The second thing that did not work was an interaction at teardown. free_irq warns when an IRQ being released still has an affinity hint set, on the theory that the hint is invalid after the IRQ has been freed. Our connect path was setting the hint; our disconnect path was calling unbind_from_irqhandler without clearing it first; every domU teardown generated a kernel WARN in xenwatch context. We added explicit irq_update_affinity_hint(irq, NULL) calls at every teardown site that pairs with a setup site. Not glamorous; necessary.

After these fixes, a dom0 backend kthread reliably runs on the same host node as the ring it services, the IRQ for that ring fires on a CPU in the same node, and the pages the backend touches - the ring, and the guest payload it maps - sit on that node too. The dom0 side of the I/O path is fully NUMA-local end to end.

Fixing the Frontends (and an XPS Steering Trap)

That fixes the dom0 side. There is a symmetric problem on the domU side, and a third problem that only matters once both ends are placed correctly.

The symmetric problem: stock-Linux frontend drivers (xen-blkfront, xen-netfront) allocate all of a multi-queue device's rings from whichever CPU the xenbus watch handler happens to be on at the moment of the device connect. On a single-node domU with well-placed memory, this is fine; there is only one node the rings can live on. On a multi-vnode domU, it means every queue's ring lands on the same vnode, which maps to the same host node, which funnels every dom0 backend kthread for that device onto one host node. Multi-queue parallelism evaporates.

We added a new xenbus helper, xenbus_setup_ring_node, that takes an explicit Linux node ID and draws the ring pages from that node's allocator. The frontends call a small companion helper that rotates queue indices over the set of nodes with online CPUs, so queue 0's ring lands on node 0, queue 1's ring lands on node 1, and so on. A natural-looking implementation using cpumask_local_spread(i, NUMA_NO_NODE) turns out not to actually rotate - it falls back to a linear walk of cpu_online_mask and collapses every queue onto the first node's CPUs. The working implementation uses for_each_node_state(node, N_CPU) explicitly.

Combined with the backend-side affinity work, this means each queue of a multi-queue device on a multi-vnode domU has its ring on a distinct guest node, its dom0 backend kthread on the matching host node, and its IRQ on a CPU within that host node. Multi-queue parallelism on a multi-vnode domU finally works as intended.

The third problem - the one that only matters with both ends placed correctly - is on the transmit side of networking. The Linux kernel's queue selector, __netdev_pick_tx, picks a TX queue by hashing the packet header when no per-CPU steering map is installed. A sender on vnode 0 can still hash to queue 7, whose ring lives on vnode 7, and the payload moves cross-node from the sender's CPU to the ring before it ever leaves the domU. The per-queue ring placement is wasted because the sender-to-queue pairing is effectively random.

Transmit Packet Steering (XPS) is the kernel mechanism that makes the queue choice CPU-aware. xen-netfront now installs an XPS map at connect time, mapping each queue's index to the cpumask of the node that owns its ring. A sender on vnode N selects queue N, whose ring is on vnode N, which maps to host node N, where the dom0 backend kthread for queue N is also pinned. The transmit path stays NUMA-local from the sender process inside the domU, through the domU kernel, through netfront, through netback in dom0, all the way to the point where dom0 hands the packet off to physical hardware.

Block storage does not need an XPS-equivalent. The block multi-queue layer (blk-mq) already maps hardware contexts to CPUs using the same cpumask_local_spread-style distribution the ring placement uses. A process on CPU C submits I/O via the hardware context mapped to C; that hardware context's ring lives on the node that owns C; the dom0 backend kthread for that ring is on the matching host node. Submitter, ring, and backend are aligned by construction without any extra steering layer.

The Whole Path, End to End

Let's put all of that together and follow a single domU-side packet transmission through the paravirtual network drivers (netfront/netback). Here's what happens, top-to-bottom, from dom0's perspective:

• An event-channel IRQ wakes the backend.
• netback's NAPI poll runs.
• It reads the shared TX ring to learn about the I/O.
• It maps the guest's outbound packet pages (transiently, to facilitate "zero-copy" transmission).
• It pushes the packet through dom0's egress queueing discipline (the qdisc).
• It hands the packet to the physical Network Interface Controller (NIC).

Strip that down and it is one execution context (netback's thread) reaching into the guest's memory through two separate mappings (the ring and the payload) and handing the result to the dom0-side networking stack, ultimately to be transmitted through the physical NIC on the host side.

Where the thread runs and where that memory lives are two separate placement decisions - the two affinities we covered in part one. Xen functionally only pins one piece of this whole process, the event channel IRQ, to a single dom0 CPU. It leaves everything else to the dom0 kernel scheduler and dom0 memory allocator. Of course, the guest memory is mapped into dom0's address space, but the pages stay resident wherever the guest is placed. Those pages don't move, they're just viewed by dom0. And that memory will be accessed from whatever CPU the netback thread happens to run on.

Networking is the stack that pays most for getting these pieces poorly affinitized. Disk I/O (the blkback/blkfront driver from part 3) maps its rings persistently and reuses them; the networking driver re-maps and unmaps the payload for every batch of packets.

Regardless, even with the event channel IRQ pinned, that pinning doesn't necessarily last long. The irqbalance daemon does its best to be helpful by balancing interrupt load across CPUs, moving the IRQ wherever it likes. On a NUMA-aware host it makes those moves from real topology; on a NUMA-oblivious dom0 that topology is hidden, so its choices are effectively blind.

What about the other components involved in handling the network I/O? Well, the NAPI poll runs on whatever CPU the dom0 scheduler picked. On the memory side, the TX ring is grant-mapped from the guest, so it lives on the guest's node. The egress qdisc is dom0's own memory, which would normally be node-local to whoever allocated it. But a NUMA-blind dom0 allocates it with no node preference at all, so it lands on whatever node the free list happened to be using.

There is also a difference between Edera and a stock Xen configuration worth noting. The dom0_vcpus_pin=1 flag we rely on (from the dom0 section) at least keeps the IRQ and the NAPI poll on stable cores - but it is not the default. A typical Xen install lets all dom0 vCPUs float across every logical CPU with no regard for locality, so on top of the scattered placement you pay for cache lines changing ownership and cross-socket translation lookaside buffer (TLB) shootdowns as guest memory is mapped and unmapped.

But even with that pinning, without our NUMA-awareness changes, the remaining parts in the dom0 paravirtual network handling could run on any CPU and start accessing guest memory from any host node. You could get lucky: all the components might run/allocate from the same NUMA domain by chance, but there's no kernel- or hypervisor-level intelligence behind that.

In the unlucky case, all five land somewhere different, and every hop reaches across the interconnect, like we show in the illustration below.

This diagram is a little bit simplified: not every traversal costs the same. Remember from part one that the cost of traversal between two nodes is given to us by the ACPI SLIT table, as a numeric "distance" value. A hop to the most "distant" node may be two to three times as expensive as a hop to a "close" neighbor node.

Reaching across the interconnect on every hop is brutal on a modern CPU, and the figure above, if anything, undersells it.

The most obvious cost is bytes moving from one node's memory to another. The less obvious part is the bookkeeping the hardware does to keep those scattered accesses honest. A ring touched from two nodes ping-pongs its cache lines back and forth, each read dragging a remote snoop behind it. A backend woken on the wrong node starts cold, every cache line it needs pulled from across the machine.

And networking adds a cost block storage does not. Recall from part 3 that transmit grant-maps the guest's outbound pages and then has to unmap them once the packet is sent - and tearing a mapping down forces a TLB shootdown: an inter-processor interrupt fired at every CPU that might hold a stale entry, each one spin-waited for. When those CPUs sit on other nodes the shootdown crosses the interconnect too, and it recurs with every batch of traffic. Block maps once and reuses; transmit pays this toll continuously.

Also, remember that the illustration traces a single request. A real host runs many virtual NICs and disks at once, each scattering its own path across the same fabric, and the costs do not just add up: they collide. The interconnect has finite bandwidth. Enough cross-node traffic will saturate the interface, and the coherence and shootdown overhead amplifies heavily under contention. One guest might never notice; but put a dozen busy guests on the fabric, or one guest with a dozen busy block and network queues, and the steady per-request tax becomes throughput and latency that swing with how loaded the neighbours are. None of it surfaces as a single dramatic stall. Most often it surfaces as highly variable latency and bandwidth.

So how do we fix all this cross-interconnect traffic? Once the dom0 kernel has NUMA-awareness, we can teach the paravirtual drivers to respect that information. And the fix is almost dull once you see it, but it is quite satisfying:

Every layer that touches the path now agrees on one node:

• The netback NAPI poll is affinitized to the host node backing the guest
• The event-channel IRQ is steered to a CPU on the same node
• The TX ring access is node-local because the domU is resident there
• The guest's payload pages map and unmap on that node too, so the shootdown stays local
• The egress qdisc is allocated from that node, and XPS for the host NIC steers the transmit onto node-local queues

The whole inter-domain conversation, guest to ring to backend to egress queue, happens without a single line leaving the node. The coherence traffic stays on-die, the caches stay warm, and the unmap shootdowns fire only at CPUs on the same node.

But you may notice there's one remaining step in the diagram above that we can't affinitize away: device locality. You can think of the "physical NIC" in the diagram as a PCIe-attached network card. Each PCIe slot has its own NUMA locality, and we have no choice but to deliver the traffic to that card through the processors that are local to it.

That means the floor is a single deliberate interconnect traversal, the honest cost of touching real hardware, instead of four accidental ones. A guest that happens to be resident on the same node as the device it is talking to does not even pay that one.

Remember from part two that Edera's domU placement policy prefers to put a guest local to any attached PCI passthrough device, and otherwise on the NUMA node that is least filled. It's possible to use PCI passthrough to attach a NIC directly to a guest, which would count towards that placement. But the shared PV path is the case where the hop remains: the physical NIC belongs to dom0 and serves multiple guests, so no single guest can be co-placed with it, and that one deliberate crossing is the price of sharing one card instead of dedicating it. (Storage is murkier still. A guest-attached block device could be backed by a file on a host volume that spans several controllers, so there may be no single node to be local to, which is why placement does not chase disk locality the way it chases passthrough-device locality.)

NIC and HBA Interrupt Steering on NUMA Hosts

Device locality runs deeper than that one data hop. The physical NIC is rarely a single queue: a modern server NIC exposes many hardware queues, each with its own message-signalled (MSI-X) interrupt, so receive and transmit work can fan out across CPUs. A serious SAS or SATA host bus adapter (HBA) is built the same way. Spreading those interrupts, and the kernel's receive- and transmit-side packet steering (RPS and XPS), onto CPUs near the card is ordinary Linux NUMA tuning, and the in-tree code does it on its own: the x86 interrupt allocator prefers the device's own node, and the steering maps follow.

All of that keys off a single number: the device's NUMA node, which the kernel reads from firmware (dev_to_node). On a NUMA-oblivious dom0 that sees one flat node, that number comes back "unknown" for every device, so the interrupt allocator has nothing to prefer and the steering maps have no node to aim at. An operator cannot paper over it by hand, because the node the device hangs off is genuinely not visible to the kernel. But if we give dom0 the real host topology, the number resolves: dom0 places the device's own interrupts and queue-steering on its local node, the same way it would on bare metal. It is the same locality we just chased through the PV path, now applied to the hardware's own machinery rather than the inter-domain hop.

The Latent vNUMA Memory Placement Bug We Found at the End

While running validation on the full stack, we noticed that the memory-bandwidth numbers on a scatter-strategy zone came in lower than they should. Every other indicator looked correct: lscpu inside the zone showed the multi-vnode topology we expected, the APIC IDs matched the layout we had asked for, the SRAT matched. The only number that disagreed with the rest was the bandwidth.

The bug had been silently broken since multi-vnode vNUMA was first added to our tree. The toolstack code that built the per-vmemrange allocation flags was passing vnode indices through to Xen's populate_physmap interface, where Xen expects physical node IDs. Xen has no notion of virtual nodes at that interface - the hypercall talks in host node IDs exclusively. The result: vnode N's memory always ended up on physical node N, regardless of which physical nodes the placement algorithm had actually selected. A 2-vnode zone placed on physical nodes 2 and 5 by scatter strategy was actually getting its memory on physical nodes 0 and 1.

The bug had been hiding for a fortunate reason. On a 2-socket, 1-NUMA-node-per-socket host - the most common production configuration in the field - the vnode indices and the physical node IDs happen to line up by default: vnode 0 to physical node 0, vnode 1 to physical node 1. The bug only manifests on systems where the placement algorithm picks a non-contiguous set of physical nodes, like our lab box (an 8-node EPYC host running scatter strategy). That is why the problem went unnoticed in production testing for so long, even though it had been latent the whole time.

It stayed buried for a second reason, too: every other piece of topology was correct. CPU pinning used the placement-selected physical node array directly; APIC IDs were derived from CPU pinning; the SRAT was derived from the APIC IDs. All of those reported what placement had asked for. The only thing that disagreed was the actual location of the physical memory, and the only diagnostic that would catch the disagreement was a memory-bandwidth benchmark sensitive enough to notice cross-socket DRAM traffic.

The lesson worth taking from this: topology bugs that do not affect CPU placement can hide for a long time, because every CPU-derived diagnostic stays consistent with the placement decision while the actual memory quietly lives on the wrong nodes.

Where the Code Lives

Most of what this series describes is public.

The hypervisor side - the proportional dom0 memory trim, the dom0 vNUMA synthesis, the SLIT slicing, the x2APIC encoding - lives in Edera's Xen branch:

• edera-dev/xen, branch edera/4.21 - Xen 4.21 with the Edera patches on top, rebased periodically.

One thing to know before reading it: Edera drives Xen from its own Rust toolstack, so none of this added new hypercalls to the upstream C toolstack (xl / libxl). The hypervisor pieces are all there for anyone who wants to wire them into their own toolstack; they are just not exposed through the stock one.

The kernel side - the per-queue frontend ring placement, the xenbus_setup_ring_node helper, the XPS map, and the backend affinity work - is a patch series against the Linux 6.18 long-term support (LTS) branch, covering both domU and dom0:

• edera-dev/linux-kernel-oci - the split-out series (a patch set today rather than a browsable tree).

The orchestration that ties the pieces together - how many nodes a zone gets, seeding the first node, and the compact and scatter strategies - lives in Edera's toolstack rather than in either upstream tree. Most of it would port to the stock Xen toolstack without much trouble; the parts that would not are the placement heuristics themselves - the least-full-node seeding and the compact/scatter selection - which lean on Edera-specific bookkeeping.

Wrapping Up

That is the technical end of the story. Across the series we went from why NUMA makes memory placement matter at all, to how Edera places a zone's CPUs and memory by default, to why Xen's paravirtual I/O path was blind to all of it, and finally to closing that gap so the placement decisions carry all the way to the bytes on the wire and the disk.

The takeaway is the one this part opened on: the work here is what makes Edera's NUMA placement actually pay off. Part 2 described the algorithm; this part is what makes its decisions survive contact with real I/O. The customer-visible effect is deliberately boring - less variance in I/O latency, more headroom under load, better density on the same hardware - and nobody has to think about NUMA to get it.

The placement is fixed at launch by design, so whether any form of rebalancing under load can be both safe and predictable is a question we keep circling back to. The heuristics that seed placement are deliberately simple, and there is room to teach them more about how a workload actually uses memory. None of that is a promise; it is just where the interesting questions are.

If you are running NUMA-sensitive workloads on Xen, or working on this kind of problem yourself, we would like to hear from you.

Read the Full NUMA Series

NUMA Part 1: Cores, memory, and the distance between them

NUMA Part 2: NUMA-Aware Zone Placement — How Edera Decides

NUMA Part 3: Xen's PV I/O Path, and Its NUMA Blind Spot