We Rewrote the Last C Component in Our Stack. It Got Faster.

Edera runs containers as real virtual machines on a Type-1 hypervisor. We wrote every userspace component around that hypervisor from scratch in Rust. The daemon, the CRI shim, the network backend, and the init that boots inside each zone - these were all written from scratch. Except one.

Filesystem passthrough, which moves a ConfigMap, a Secret, or an emptyDir from the host into a running container, was still handled by QEMU's 9pfs. This C code sat on the critical path, mapping memory controlled by the guest. It was the last memory-unsafe part of an otherwise memory-safe stack, and it was by far the most stressful to work with. We decided to rewrite it anyway.

You’re Probably Already Running a 9pfs, and You ever Thought About It

If you use a sandboxed container runtime like Kata, gVisor's VM mode, Firecracker with a shared filesystem, or Edera, something needs to move files between the host and the guest. The container image, your mounted volumes, /etc/hosts, and the service account token your pod uses to talk to the API server all have to cross this boundary.

9P is the protocol that usually handles this. It originated from Plan 9 in the late 1980s, is small, well understood, and Linux has supported a client for it since 2005. In a paravirtualized Xen guest, it runs over a shared memory ring. The guest writes request frames into memory visible to both sides, the host reads them, does the filesystem work, and writes replies back.

It is essential, works quietly in the background, and almost nobody thinks about it. That is often where the most load bearing code hides in any system.

The Uncomfortable Part: Guest-Controlled Ring Buffers and Heap Overflows

The guest and the host share a ring buffer. The guest writes a producer index saying "I have put this many bytes in." The host reads a consumer index saying "I have taken this many out." The host subtracts one from the other to work out how much data is waiting, then reads that many bytes.

The guest controls those indices. A hostile guest can set them to anything.

In C, subtracting two attacker-controlled unsigned integers can wrap around. If it underflows, the backend interprets a huge amount of valid data in a small buffer. This leads to a heap overflow triggered from inside a container, on a process the host's filesystem has open.

QEMU's 9pfs has real CVE’s from this exact configuration:

• CVE-2021-20181 - a TOCTOU race in the 9pfs server leading to a use-after-free, allowing a malicious 9p client to escalate privileges on the host.
• CVE-2016-9102 - a memory leak in v9fs_xattrcreate, where a guest sending many Txattrcreate messages against the same fid could exhaust host memory and take down the process.
• CVE-2016-9103 - an information leak that lets a guest read host memory through extended attributes.

None of these issues are unusual. They are common failures that happen when parsing untrusted input in a language that does not prevent arithmetic mistakes. What is unusual is moving to a memory safe alternative without losing out on performance.

We Expected Rust to Cost Performance. It Didn't.

To be honest, 9P is proven and is load bearing for a significant amount of cloud infrastructure. QEMU's version has been tuned for twenty years, so rewriting it in a language with bounds checking seemed more likely to hurt performance rather than help it. The rewrite was for safety, and we expecting to trade some speed for security. So we decided to benchmark it.

How We Benchmarked 9pfs: Rust vs QEMU on EKS

We ran two EKS clusters on identical m5.xlarge hardware in the same AZ. One running Edera 1.9, with the QEMU filesystem passthrough, one running Edera 1.11 with the Rust implementation. Same protocol, same Xen transport, same stock Linux 9p client in the guest. Only the backend differs.

We benchmarked using a RAM-backed export. The m5.xlarge instance is EBS-only, with a baseline of about 143 MB/s. If you run fio on a disk-backed volume, both implementations hit the EBS limit before either becomes the bottleneck, so you’re really measuring the disk. Using host tmpfs for the export removes the disk from the equation and focuses on the datapath, which is the real comparison. We report the EBS numbers separately and note them where necessary.

We statically set the memory allocation. By default, Edera zones increase their memory usage over time. If left alone, the two guests would end up with different amounts of RAM during the run, which would be an uncontrolled variable in our comparison. So, both are set to 2 vCPU and 1930 MiB using a static resource policy.

Rust vs QEMU 9pfs: Sequential Read and Write Results

We set both clusters to msize=524288,cache=loose, so the backend implementation is the only thing that differs. Here are the results:

Of the roughly 2x upgrade, about 1.7x is mount tuning that the old implementation could have had, and about 1.18x is the rewrite. Getting a consistent 13 to 23% improvement by replacing a mature C implementation with a memory-safe one is a good result, especially since we expected to lose some performance for safety.

Directory and Metadata Operations: Where 9P's Round Trips Show

Directory operations is where 9P's round trip becomes most obvious. A readdir is a stream of requests, and walking a directory and stat'ing each entry means a request per file. This workload should show the biggest difference between implementations, and it is also where we nearly published something very misleading.

Here is the same directory benchmark over 10,000 files, run in both arms:

Why the Rust Rewrite Got Faster: Zero-Copy and io_uring

We use zero copy in both directions. For reads, the backend reserves a region directly in the guest's reply ring and reads from the file straight into grant-mapped memory, so the data never goes through a userspace buffer. For writes, we use a similar but more subtle approach: the request frame stays on the ring while pwritev reads from it into the file, and only after the write finishes does the consumer index move forward. Keeping the frame queued prevents the guest from reusing those bytes during the write. This flow control is the safety mechanism.

We did not get this right on the first try. Reads were zero-copy, but writes still copied data twice, which benchmarks revealed. The first implementation was faster on reads but slower on writes, so we fixed it.

We use one io_uring for everything the worker waits on. Each ring has its own worker thread and io_uring, and the ring's Xen event channel file descriptor is polled on that same io_uring. This way, a single io_uring_enter waits for both file completions and guest notifications. The worker is fully event-driven, and uses no resources when idle. A series of consecutive writes is grouped into one submission, so there is one io_uring_enter and one guest notification for the whole burst, instead of one for each request.

Is Rust Faster Than C for Filesystem Passthrough?

This does not prove Rust is faster than C. We’re not comparing two implementations with the same design. Instead, we replaced an old design with an improved one that happens to be written in Rust. Most of the improvement comes from architectural changes like zero-copy paths, io_uring batching, and NUMA-aware placement, which could have also been done in C.

What it does show is more specific and, in our view, more useful: moving to a memory-safe language did not cost us performance. The penalty we expected from the cost of safety never emerged. Copies, the datapath, and good engineering mattered much more than the language, and using Rust prevented us from introducing a whole class of bugs during development.

We are a small part of a much larger industry shift around Rust. Firecracker is Rust. Cloudflare retired nginx for Pingora. Android moved new native code to Rust and watched its memory-safety vulnerabilities tumble. Chromium is pulling Rust into the largest code sandbox on the planet, the web browser.

The last memory-unsafe component is usually the quiet one on the critical path, which is exactly why it takes so long to rewrite. Our stack is fully Rust now, and our benchmarks prove it was the right decision.