How we cut container image loading latency

Every compute surface Namespace runs (Devboxes, GitHub and Buildkite runners, RBE workers) boots from a base image. A slow load blocks whatever's waiting on it: a devbox that won't start, a CI job stuck pulling before it can run. So loading has to be extremely fast.

We wanted to make it instant. Instead of treating a load like a download that has to finish before anything starts, we built it to behave like a disk you mount, fetching only what's touched, as it's touched.

TL;DR: We bake each image into a bootable disk ahead of time and replicate it to every compute region, so a machine never has to reach across the planet for it. On boot, the VM starts immediately: pages are fetched as their blocks are read, and a miss triggers a background fetch instead of blocking the queue.

How are container images loaded?

Container images are usually stored in an OCI-compliant registry. Namespace built its own registry, nscr.io, for exactly this purpose. Owning the registry end-to-end lets us extend our offering without depending on a third party and, as we'll see, it opens the door to a few neat tricks.

To load an image you start with an image reference, either a tag or a digest. If it's a tag, the client first resolves it to a digest and then fetches the image's manifest. The manifest is a small JSON document that points at two things: a config and a set of layers.

The config is a JSON blob that describes how the image should run and how it was built. It carries the entrypoint and default command, environment variables, working directory, exposed ports, and the target OS/architecture, along with the ordered list of layer "diff IDs" that make up the root filesystem. In other words, the manifest tells you which pieces exist; the config tells you what they mean and how to execute them.

Each layer is a compressed tarball: a diff on top of the layer below it. To materialize the image, the client fetches every layer (often in parallel) and then unpacks them. It extracts each tarball in order, applying additions, modifications, and deletions (whiteouts) layer by layer until the complete root filesystem is reassembled on disk. Only then can a runtime such as Docker or containerd start the container.

This process has several flaws:

• It is heavily reliant on the container registry being fast and available.
• For virtual machines, the extracted filesystem still has to be turned into a disk our VM software can boot (an ext4 disk), which is extra work.
• The image must be fully assembled before the VM can start, adding significant latency to instance startup even though the individual layer pulls are parallelized.
• Most registries are backed by storage in a single region, so pulling an image from a region far from the source adds even more latency.

How does Namespace do better?

We've introduced a process we call optimization. It has two steps: baking and distribution.

Baking

Baking turns an ordinary OCI image into a ready-to-boot disk. First we pull the image and flatten it: all of its layers are collapsed into a single tarball representing the final filesystem, so none of the per-layer diffing has to happen at boot time. Then we build a filesystem image directly from that tarball.

For the ext4 format we run mkfs.ext4 straight against the flattened tar. The parameters to create this disk image are tuned for a compact, fast-to-mount image. We size the scratch file generously (roughly 10x the tar), then shrink it to its true minimum and truncate to the exact byte length. The finished disk is stored back in the registry as an OCI artifact so it lives right next to the original image and is addressed by the original image's digest plus a format suffix.

We've also experimented with an EROFS-based format that goes a step further by splitting the image into a small metadata layer and a data layer chunked into fixed-size (1 MiB) blocks. That split makes the data naturally block-addressable: the metadata is tiny and can be fetched whole up front, while the data blocks are fetched individually, on demand. As we'll see next, that is exactly the shape you want for distributing and loading an image a piece at a time.

Distribution

Once an optimized image exists, its blobs are replicated to every relevant region. Large blobs are split into fixed-size blocks (512 MiB by default), and each block is transferred, encrypted, and digest-verified independently before being written to site-local storage in each target region. Splitting the work this way means replication is parallel and self-healing: a single failed or corrupt block is re-fetched on its own rather than restarting the whole blob. Customers then pull from the registry replica closest to them, so a machine in one region never has to reach across the planet to fetch its image.

Baking and distribution together already eliminate two of our four problems: images are pre-converted into a bootable disk, and that disk is available locally in every region.

Incremental loading

That still leaves the biggest one: even with everything local, a VM traditionally has to pull the whole disk before it can boot. The pull is heavily parallelized, but it's still a barrier: nothing runs until it finishes.

To remove that barrier we introduced incremental loading. Instead of copying the disk up front, we expose it to the guest as a virtual block device. To the guest it looks like an ordinary disk. Under the hood, when the guest reads a region of that disk, we fetch exactly the part that's needed, and nothing else, from the registry. The VM starts immediately and pulls the data it actually touches, on demand, as it runs. A 100 GB image no longer means a 100 GB download before boot.

The block device

We expose the disk through one of two mechanisms, depending on the host:

• NBD, the Linux Network Block Device protocol. Our server implements the full handshake (negotiating export size, a preferred 4 KiB block size, a 32 MiB maximum request size, and multi-connection support so several kernel connections can drive the device in parallel) and then serves reads and writes. Crucially, reads are handled through an asynchronous callback: the handler hands the request to the backend and returns to the connection loop immediately, so a request that has to hit the network never blocks the requests behind it.
• Ublk, a newer userspace block driver that pairs with io_uring. Cached reads are satisfied inline; a cache miss is submitted as an async io_uring read that completes independently of the other in-flight I/O tags. This keeps queue depth high even while some reads are waiting on a page fetch.

Either way the guest just sees a disk. The interesting part is what happens behind it.

The block map: fetching bigger than you need

The disk is modeled as a grid of small, fixed-size blocks (4 KiB, matching the guest's page size) grouped into much larger pages (tens of megabytes). A block map records, for every block the guest might read, which page holds it and at what offset. Pages are the unit we fetch and cache; blocks are the unit the guest addresses. The mapping is stored run-length-encoded so that long stretches of consecutive blocks that live in the same page collapse to a single entry. This keeps the metadata for a multi-gigabyte disk small enough to fetch and hold entirely in memory.

The read path is built around that asynchronous design, and it resolves in three stages, fastest first:

1. In-memory hit. The requested block is in a page that's already resident in memory, copy it out and return.
2. Local-cache hit. The page is already in the node's on-disk cache, read it from the local file descriptor.
3. Miss. Spawn a background fetch of the page from the registry and let the block device keep serving other requests in the meantime. A slow fetch never stalls the queue.

Because a single fetched page contains many contiguous blocks, one network round trip typically satisfies a whole burst of nearby reads. Concurrency into the registry is bounded so a cold VM can't overwhelm the network.

Warming the filesystem metadata: There's a predictable pattern in the very first reads a guest makes: mounting an ext4 filesystem always touches the same structural blocks: the superblock and its backups, the group descriptor table, block and inode bitmaps, the start of the inode table, the root directory, and the journal. Rather than fault these in one cache-miss at a time, we analyze the filesystem's own layout at load time and prefetch precisely those blocks in a single sequential pass. This is static prefetching which is derived from the filesystem structure itself. This means the mount is already warm by the time the guest issues its first read.

Background hydration: While the guest is busy with its working set, a background worker walks the rest of the disk and pulls pages ahead of demand, so that reads which would otherwise miss increasingly find their page already local. Hydration is best-effort: it deprioritizes itself behind live guest I/O, and any page it fails to fetch is simply left for the on-demand path to retry.

Choosing the page size: Page size is a genuine trade-off, and we tuned it against real benchmarks. Larger pages amortize per-request overhead and lift throughput, but each one takes longer to fetch and wastes bandwidth if the guest only wanted a sliver of it; smaller pages are nimbler but multiply the request count. Measuring a ~30 GB image with 16 parallel readers:

Throughput plateaus in the 16 to 32 MiB range while per-page latency keeps climbing, so that's where we operate: near-peak bandwidth without paying the latency tax of very large pages. Under heavy concurrency the design scales too, as a 20 GB image driven by 100 parallel readers sustains roughly 350 MB/s aggregate with the fetcher running close to 90 pages in flight.

Where we're taking this next

Incremental loading gets a single VM booting from a local, on-demand disk. The next frontier is making the whole fleet cooperate so that pages are almost always a hop away.

• Prefetching from usage traces (PGO for I/O). The metadata prefetch above is static, but we can do better here by following a more dynamic approach. We could record each page read and build a heatmap of the pages per image over time. This would allow a prediction of which pages will be accessed before the guest asks for them. This idea is based on the profile-guided optimization in compilers, where real execution traces are used to perform branch prediction.
• A distributed, fleet-scaling page cache. Rather than every node fetching from a regional registry, nodes would serve cached pages to one another. Capacity then grows horizontally with the fleet, as each machine contributes the pages it already holds and popular images are absorbed by the fleet instead of hammering central storage.
• Topology-aware routing. Not all peers are equal. A cache that understands site → cluster → rack → node locality can prefer the nearest copy of a page, keeping traffic rack-local and minimizing expensive east-west bandwidth across the datacenter.
• NVMe-oF for page serving. Looking further out, serving pages between neighbors over an NVMe-over-Fabrics path would push peer-to-peer fetches close to local-disk latency.

Taken together, the direction is clear: an image should behave as if it were already on every machine that might ever need it, fetched lazily, cached everywhere it's useful, and always served from as close as physically possible. That's what loading container images at light speed means to us.

Stay tuned for the next write-up as we make image loading even faster.