NUMA Part 3: Xen's PV I/O Path, and Its NUMA Blind Spot
In the previous part of this series, we walked through how Edera places zones on a host: the algorithm that picks which physical Non-Uniform Memory Access (NUMA) nodes a zone lives on, how its vCPUs are pinned, what topology the guest itself sees at boot. The story ended on a promise that placement is the first layer of a larger design and that each layer above it multiplies the value of getting the first layer right. This part walks through what sits immediately above placement: Xen's paravirtual I/O drivers, the ones that carry every byte of network and storage traffic between domUs and the world. Until recently, that layer was the single biggest reason good placement decisions did not carry through to the bytes hitting hardware.
There is a detective story coming later in this post. Two zones, identically configured, on the same host, performing measurably differently. A guess at where the actual work is landing. A diagnosis pointing at a piece of NUMA information that has been silently missing from every Xen dom0 in production for the entire history of the relevant kernel code. We will get to all of it. First, though, the architecture, because the diagnosis only makes sense once the moving parts are on the table.
How Does Xen Paravirtual I/O actually work?
Xen's split-driver design has been roughly stable since the original paravirtualization work in the early 2000s. The same set of moving parts appears across every PV driver - netfront and netback for networking, blkfront and blkback for storage, several others for less common roles - so once you have the skeleton, every PV driver is a variation on it. What differs between them, and what decides their NUMA cost, is how each one moves the payload.
There are four moving parts.
The Shared Ring Buffer
A small in-memory circular array of request and response slots, with separate producer and consumer indices on each side. The frontend (running inside the domU) pushes requests into the ring; the backend (running in dom0) drains them, processes them, and pushes responses back. The ring is the protocol surface between the two halves of the driver.
Grant Tables
A controlled mechanism by which a domU can expose a page of its own memory to dom0. The domU allocates a page in its own RAM, fills out a grant table entry saying "dom0 may access this page", and hands the reference number to dom0 via xenstore. Dom0 then asks the hypervisor to map the page into its own address space. On a PVH dom0, the foreign frame appears in dom0's view through the host's second-level address translation (SLAT); the result is that dom0's kernel sees a struct page for the ring as if it were ordinary memory, except that the underlying physical bytes live in the domU's RAM, not dom0's. Grant tables are the mechanism that exposes the ring.
Event Channels
Per-domain virtual interrupts. When the frontend has added work to the ring, it raises an event channel telling the backend "go check the ring". When the backend has produced a response, it raises an event channel telling the frontend. Without event channels, both sides would have to spin polling the ring to find new work; with them, either side can sleep until there is something to do.
Where the Actual Data Lives
The ring never carries the payload, only descriptors: which operation, where the result goes, and the grant references that point at the buffers. The payload itself is a separate set of domU-owned pages, granted to dom0 the same way the ring is. What differs from one driver to the next is how dom0 reaches those pages, and that one choice, made differently by block and by network, is what decides the NUMA cost. It gets its own section below.
The asymmetry to keep in mind: the frontend allocates the ring (real RAM inside the domU); the backend never owns the ring's underlying memory. The backend's struct page for the ring is a placeholder that lives in dom0's mem_map but is backed by the foreign frame through SLAT. To dom0's kernel the placeholder looks like a normal page - kmap, page_address, the usual accessors all work - but its physical home is in the domU's memory, not dom0's. Hold onto that distinction; it ends up doing the load-bearing work in the diagnosis section.
Map, unmap, or copy: how the payload moves
Dom0 has three different ways to get at a guest page it has been granted, and they cost wildly different amounts. The PV drivers choose among them, and block and network choose differently.
Grant-map rewrites dom0's own page tables so that one of its placeholder pages now resolves to the guest's physical frame. After that, dom0 reads and writes the guest's memory directly, as if it were its own. The cost is paid up front: a hypercall, a page-table write, a page pin. Once the mapping exists, using it is free, and it stays usable for as long as dom0 keeps it.
Grant-unmap tears that mapping back down, and the teardown is the expensive half. To guarantee no stale translation survives, Xen flushes the translation lookaside buffer (TLB) on every physical CPU the guest has ever run on - an inter-processor interrupt to each, with the caller spin-waiting for the acknowledgements. A single unmap is therefore a small interrupt storm across the host. This is the cost that persistent mappings and copying both exist to avoid.
Grant-copy sidesteps mapping entirely: dom0 asks the hypervisor to copy bytes between one of its own pages and a guest page, and the hypervisor performs the copy from inside its own address space. Dom0's page tables are never touched, so there is no map, no unmap, and no shootdown - only the cost of the bytes copied.
With those three on the table, the two stacks diverge cleanly.
Block Maps, and Keeps The Mapping
blkback maps the guest's request pages, builds the block I/O structure (the bio) directly on those foreign pages, and lets the disk perform its direct memory access (DMA) straight into the guest's memory. There is no copy and no dom0-side data buffer at all. And because a guest's storage traffic recurs over a bounded pool of buffers, blkback caches the mappings and reuses them - persistent grants, on by default - so the expensive map happens once per buffer and is amortised to almost nothing. In steady state the only thing that moves per request is the disk's DMA in and out of the guest's own pages.
Network Transmit Maps, but Cannot Keep It
When a guest sends a packet outward, netback maps the guest's outbound pages zero-copy (copying only the first 128 bytes of header into a small dom0 buffer) and hands the mapped pages to the Network Interface Controller (NIC). Then it has to unmap them: the guest needs its pages back, and the next packet will come from different ones, so there is nothing worth caching. Every batch of packets pays a map and, worse, an unmap, and the unmap is the shootdown described above. On a busy interface that interrupt storm recurs with the traffic, batch after batch.
Network Receive Copies
An inbound packet has already landed in a dom0 buffer from the NIC; to hand it to the guest, netback grant-copies it into receive pages the guest granted ahead of time. This is the one path that genuinely copies the payload, and its cost is the copy - no mapping, no shootdown.
So the same skeleton runs three very different errands. Block maps once and coasts. Receive copies, bounded and predictable. Transmit is the expensive one: a map and an unmap-with-shootdown on every batch, for the life of the workload. Of the two stacks, networking is where careful placement has the most to win or lose - a point part 4 builds its picture on.
And all of it is a NUMA question for the same underlying reason. In every case the payload's bytes physically live in the guest's memory (block and transmit) or are copied between dom0 and the guest (receive). Where those guest pages sit, relative to the device doing the DMA and the dom0 thread driving the transfer, is fixed entirely by placement. On the same node, each touch is a local memory access; a socket away, every byte crosses the interconnect. The grant primitive sets how often dom0 pays an overhead; placement sets how much each payment costs.
What upstream Xen knows about NUMA today
Upstream Xen does know about NUMA. The hypervisor reads the host's SRAT and SLIT at boot and tracks which physical pages belong to which node. Domains can be created with a vNUMA topology via the XEN_DOMCTL_setvnumainfo hypercall, and Xen synthesises matching ACPI SRAT and SLIT tables for the guest. PVH and HVM guests reading those tables see real node distances and per-node memory regions.
What upstream Xen does not synthesise from the vNUMA layout is the third piece of topology part 2 introduced - the x2APIC IDs in CPUID. A guest on upstream Xen gets correct memory-side NUMA information but a flat CPU topology, which matters for workloads that read CPUID directly rather than through the kernel's sysfs view. Closing that last piece is part of what part 4 covers.
So PVH and HVM guests are already most of the way NUMA-aware on upstream Xen.
What upstream Xen does not do is extend any of this to a few places it should.
PV guests cannot have a vNUMA topology, and the reason is structural: it comes down to PV having no firmware. A guest learns which node its memory and its CPUs belong to from the ACPI SRAT - the table maps both physical address ranges and CPU APIC IDs to nodes. A PV guest has no ACPI at all (it does not boot through firmware), so there is no SRAT, and the kernel has no other channel it will accept CPU-to-node mappings from: it reads them only from ACPI, from an AMD northbridge a guest cannot see, or from a device tree x86 does not use.
It is tempting to think CPUID could fill the gap, since PV CPUID is fully emulated and could report any topology you like - but the kernel never learns node membership from CPUID. CPUID says how to decode an APIC ID into cores and threads; it never says which node an APIC ID is on. That fact lives in the SRAT, and PV has none - and the APIC IDs a PV guest does present are a synthetic sequential count with no ACPI identity behind them for a SRAT to reference.
Making PV NUMA-aware would mean rebuilding the firmware, ACPI, and APIC model inside the guest, which is exactly what PVH is. So the answer is not "implement PV vNUMA", it is "use PVH". PV guests larger than a single NUMA node run on memory the host has spread across several physical nodes, with no way to see where it lives or which vCPUs are close to it; their NUMA-aware code paths are inert.
Dom0 has historically been a NUMA-oblivious guest. Dom0 is itself a guest in its own right - the privileged one, with device-driver access, but still a guest - and upstream Xen never synthesised a vNUMA topology for it. Part of why is historical: dom0 was for years exclusively a PV domain, and the PVH dom0 mode that makes ACPI-table synthesis tractable in the first place is a relatively recent addition that has not had its NUMA story extended upstream yet.
On a multi-socket host, dom0 saw a single flat node, no matter how many physical nodes the underlying hardware had - and crucially, no matter where dom0's own pages had actually landed. Xen places dom0's initial memory wherever it has free pages, which on a multi-socket host frequently means more than one node. Dom0 could not see that distribution, and so could not even make NUMA-aware decisions about its own working set - never mind the foreign-mapped pages from grant tables.
Everything dom0 does - storage daemons, container runtimes, observability agents, the I/O backends themselves - ran on a kernel that had no idea where any of its memory came from or which CPUs were close to it. numactl -H inside dom0 showed every page as equally close, so each of those daemons placed its threads and sized its caches as if that were true - while a steady fraction of their memory accesses were quietly crossing the interconnect to reach another socket.
The paravirtual I/O drivers themselves have no concept of NUMA, in either direction. Even on a hypothetical NUMA-aware dom0, the upstream code for xen-netback and xen-blkback (and their frontends inside guests) does not call kthread_create_on_node, does not consult page_to_nid on ring pages to drive placement, does not set IRQ affinity hints based on where the ring lives.
The hooks the rest of the Linux kernel uses for exactly this kind of thing - set_cpus_allowed_ptr, irq_set_affinity_and_hint, kthread_create_on_node - are simply not invoked.
The hypervisor knows; almost nothing above it does.
PCI passthrough: A NUMA Workaround, Not a Fix
If you are running a Xen-based stack today and you care enough about cross-node I/O cost to look for a way around the gap, there is a real workaround. Major enterprise Xen deployments have reached for it routinely, especially for high-packet-rate network workloads where the cross-node hit shows up as outright capacity loss. It is not subtle and it is not principled, but it works for the cases it works for.
The trick is to skip the dom0+PV path entirely. Attach the NIC directly to the domU via PCI passthrough. The domU now talks to the NIC's vendor driver in its own kernel, without going through netfront or netback. There is no ring buffer in dom0 to land on the wrong node. There is no dom0 backend kthread to schedule. The domU's vNUMA topology, if it has one, tells the kernel which queues to use for which CPUs, and the kernel runs the actual NIC driver on the right cores. Block storage admits the same approach with NVMe passthrough, with the predictable caveat that fewer storage devices are practical to pass through than network devices.
The workaround is operationally awkward: every device you pass through is normally one you cannot share, live migration becomes harder, the IOMMU has to be configured correctly, and the domU kernel needs the right vendor driver loaded.
The "you cannot share" part has an exception worth calling out. SR-IOV NICs - hardware that exposes a single physical NIC as multiple virtual PCI functions, each independently passthrough-able to a different domU - relax the one-NIC-per-domU constraint. The cost is allocator complexity that Xen VIFs do not have: SR-IOV virtual functions are pre-provisioned on the hardware at boot and bounded in number, where Xen VIFs are an unbounded software resource that the toolstack mints on demand. Block storage has an analogous SR-IOV story - the NVMe spec allows for virtual functions too - but such drives are rare outside specific datacenter parts.
Even with SR-IOV, passthrough is still a sidestep, not a solution. The PV path remains for every workload that cannot reach for passthrough, with all of its cross-node penalty intact. We have customers who reached for the passthrough workaround and paid for it in operational complexity and hardware count, and got the latency they wanted. The work the rest of this series describes is what makes the workaround stop being necessary.
Where the PV path falls apart
Let's set up a hypothetical situation. Two zones running on the same host, both built from the same image, both configured the same way, both running the same benchmark. One of them performs measurably worse than the other. Sometimes 20% worse. Sometimes identical to each other across a single run, but slower than they ran last week, on a host nobody touched in the meantime. The kind of difference that does not show up in synthetic microbenchmarks because the synthetic workload warms up fast enough that whatever cache or interconnect cost was hurting the real workload gets amortised away. The kind that absolutely shows up in latency-sensitive production workloads, and that takes weeks to attribute to its actual cause because every individual factor looks fine.
The natural question is the one from the top of this post: where is the actual work landing in physical space? Follow a single request through the PV path and ask the same thing at every component - does anything here know which host NUMA node this I/O is tied to?
Start in the domU. The frontend allocates the ring through alloc_pages_exact, from whichever CPU the xenbus watch handler happened to be on when the device connected - no node hint, no per-queue spread, so every queue's ring on a multi-queue device lands on the same guest node. It is not necessarily a good node, but the guest at least knows which one it is.
Cross into dom0 and the floor drops out. The foreign frame is mapped in through the grant table, and the placeholder struct page backing it comes from xen_alloc_unpopulated_pages, which in stock kernels registers its memory with NUMA_NO_NODE. So when dom0's kernel asks the obvious question - page_to_nid() on the ring page, which node is this? - the answer is NUMA_NO_NODE. Not "node 5." Not a wrong guess that could at least be corrected. Nothing.
That is the fact the top of this post promised. The host node of the foreign frame was never handed to dom0, so the single piece of information every downstream placement decision depends on is simply not there - and it has not been there in any stock Xen dom0, for the entire history of the relevant kernel code. Everything that follows is a consequence of that one hole.
Watch it compound. The backend kthread is created by xenwatch on whatever CPU processed the connect event, so kthread_run puts its task_struct on that node, with nothing tying it to the ring - because, to the kernel, the ring has no node. The event-channel IRQ is worse still: Xen's evtchn driver allocated its descriptor with NUMA_NO_NODE baked in, so /proc/irq/N/node reads -1, and irqbalance - which ships and runs by default on most distributions - treats the IRQ as floating and sends it wherever it likes. A backend that tried to hint at NUMA-local routing would simply be overruled.
So picture it on a busy host. The ring lands on host node 5, because xenbus was on a vCPU mapped there. The backend kthread lands on node 0, because that is where xenwatch happened to be. The IRQ fires on node 3, because irqbalance picked it out of the floating candidates. Three components that should share a node sit on three different ones, and every request crosses the interconnect at least once, often twice - not occasionally, not only under contention, but on every single request, deterministically, for the life of the device. An operator who knows the topology can script the pinning by hand after each connect, and some do; but the kernel has nothing of its own to act on, so it cannot help itself.
None of that scattering is unique to Xen. A plain Linux multi-queue NIC with no Transmit Packet Steering (XPS) or smp_affinity tuning splatters senders across queues and lands RX softirqs on the wrong cores in exactly the same way. What is unique to Xen is the ground it stands on: those other cases can be tuned, because the kernel knows what node everything is on. This one cannot, because the number it would need - the host node of the grant-mapped ring - was never exposed to it. The most careful NUMA-aware backend ever written could not have placed that kthread correctly, because the information it would have keyed off was not there to read.
What's Next in This Series
So how do we close the gap? The fix comes in two pieces. The first is structural: get the host node of foreign frames into dom0's view, so that page_to_nid on a grant-mapped ring page returns a real answer instead of NUMA_NO_NODE. That needs a small change in Xen itself (a new hypercall to expose what the hypervisor already knows) and a less small change in the dom0 unpopulated-page allocator (a per-node pool so the placeholder struct page is registered with the right node up front). The second piece is teaching every link in the chain to actually use that information: dom0's backends, the frontends inside domUs, the guest kernel's outgoing-traffic queue steering, and a few related corners we found along the way.
We will be talking about both pieces in part 4. Along the way we ran into a few things that were broken in less obvious ways than "the topology was just missing" - including a memory-placement bug in our own toolstack that had been silently sending vnode N's memory to physical node N instead of the actually-selected physical node, for as long as multi-vnode vNUMA had existed in our tree. The headline picture - what the data path looks like before and after, with the cross-node crossings counted explicitly - lives in part 4 too.
-3.avif)