Flintstack › Design notes

Design notes

Written up after the 0.7 rewrite and amended since. This page is the argument, not the manual: it says why the thing is shaped the way it is, including the parts that are a compromise and the parts that are simply a limitation.

Why a daemon

The first two releases had no resident process: the wrapper opened the store directly, took a lock, did its work and exited. That is simpler to reason about and it fell over at about twelve parallel jobs, because every invocation paid for opening the index, and the lock turned a parallel build into a slow queue.

A resident process holds the index in memory, keeps one writer per shard, and lets the wrapper be a 400-line program that writes a request to a socket and reads a reply. The cost is an operational component you have to supervise, and a failure mode where the socket exists but the process behind it is wedged. We pay that cost deliberately; the wrapper's five-second timeout and fall-through to the real compiler is what makes it survivable.

Composing the key

Three of the four inputs are easy. The toolchain is identified by version string or by a digest of the binary; the argument vector is normalized so that output paths and the working directory do not participate; input digests come from reading the files the preprocessor actually opened.

The environment is the hard one, and we solved it by being restrictive. Hashing the whole environment makes every entry unique in practice, because CI injects a job id. Hashing nothing is unsafe, because CFLAGS changes behaviour. So there is an allowlist, it is short, and it is in the config where a human can see it. The failure mode is a variable that matters to your build and is not on the list - which is a wrong-answer bug, so the list is documented on the configuration page rather than buried.

The non-hermetic build

A cache that keys on declared inputs is correct exactly as far as the declaration is correct. If a compile reads /usr/include/local-patch.h and the dependency scanner never saw it, the entry is keyed without it, and a machine with a different version of that header gets the first machine's object file. The result compiles, links, and is subtly wrong.

We do not solve this. Solving it means sandboxing every invocation and refusing reads outside the declared set, which is a different and much larger program. What we do instead is make the failure visible: flint key --explain lists every input that entered the hash, so when two machines disagree you can diff the two lists and see the missing header in about thirty seconds.

On-disk layout

Entries live in 256 shards by the first byte of the key. Each entry is a header, the compressed payload and a trailing digest of both. Writes go to a temporary file in the same shard and are moved into place with rename, so a reader either sees the previous entry or the new one, never half of either. The index is rebuilt from the shards at startup - on a 120 GiB store that takes about 2.8 seconds, which we consider cheap enough to avoid keeping a second source of truth.

There is no database. There was one, in 0.5, and it was the leading cause of bug reports: a crash at the wrong moment left the index and the shards disagreeing, and every recovery path we wrote was another thing to get wrong. Scanning is slower to start and impossible to corrupt.

Eviction

The default is least-recently-used, with a sweep that starts when the store passes 95 % of max_size and stops at 85 %. Hysteresis is the whole point: an eviction pass that runs at the ceiling spends the rest of the day running, and an eviction pass is the one operation that competes with builds for the same disk.

The alternative policy, size, removes the largest entries first. It is there for stores dominated by a few enormous link artifacts, where LRU keeps a thousand useful object files alive and then throws them all away to make room for one archive. If your average entry is under a megabyte, LRU is strictly better.

What we rejected

• Content-defined chunking. Deduplicating object files at the block level saved about 9 % of space on our corpus and cost 30 % of restore throughput. Object files are mostly unique; the win is in the headline, not in the workload.
• Cross-entry deduplication. Same conclusion by a different route: it makes eviction a graph problem, and eviction being a simple loop is why it is reliable.
• Splitting oversized entries. An entry larger than the limit is refused, loudly. Splitting means reassembly, reassembly means partial reads, and partial reads mean a corrupted object that still passes a digest check on each piece.
• A scheduler. Every few months somebody asks for work to be farmed out to idle machines. That is a build system, your project already has one, and the two would fight over the dependency graph.
• An admin socket. A live mutation interface is a live privilege-escalation surface on a component that already holds compiler output. The config file plus SIGHUP covers the real needs.

What we got wrong

Two decisions we would make differently if the store format were free to change again. The entry header has a 32-bit length field, which is where the 2 GiB entry ceiling actually comes from; it was described as a safety limit for two years before somebody read the code and found it was a structural one. And compression level is recorded per store rather than per entry, so changing compress does not re-encode what is already there - you end up with a store full of mixed levels and no way to tell which is which.

Neither is painful enough to justify a format break. Both are listed here because the next person to read the serializer deserves to know they are deliberate scars and not puzzles.

Last updated: 2026-07-04