loom▌
A distributed expert cache for large mixture-of-experts inference. Pool the RAM and disk of ordinary machines; serve a model none of them could hold alone.
The thesis: experts flow, not activations
A modern MoE model activates a small fraction of its weights per token — and only the routed experts change from token to token. Loom treats those experts as an immutable, content-addressed, sparsely accessed blob store. Every node runs its own full forward pass; the only thing that crosses the network is expert weight blocks — cacheable, prefetchable, digest-verified.
Because the blocks are immutable and Merkle-rooted, integrity is a free side effect: every shard is verified against the manifest digest before it touches a matmul or disk, a poisoned peer costs one rejected block, and a node that dies costs a re-fetch from a replica — never a broken pipeline.
Try it — join the live devnet
# one command: installs loom, syncs your share of the expert shards from # peers (no model download), serves an OpenAI-compatible API locally curl -fsSL https://raw.githubusercontent.com/ch4r10t33r/loom/main/scripts/join-devnet.sh | sh
The devnet serves Qwen3-30B-A3B (Q2_K, 6,262 content-addressed shards). Your node holds a random redundant fraction, fetches the rest from peers inside the token loop, and serves shards back. Chat UI on localhost:8555. Details: alpha guide.
Measured, not promised
Every design decision in loom carries a dated, append-only decision log entry with the measurement that forced it — including the negative results.
aggregate wall time on a 2.5 MB/s link (−9.0% at 119 MB/s), byte-identical output: a 12M-parameter head predicts every layer's experts from layer-0 state and prefetches them.
was 1–2 minutes: an origin adopts its previous boot's Merkle manifest and audits in the background; every read stays digest-verified.
a partial holder's output is byte-identical to a full node's; every optimization ships gated on that equality.
eager churn repair within one 2 s tick, plus systematic Reed–Solomon parity that rebuilds small holes — decode-then-verify, no trust in parity servers.
How it works
- Content-addressed store — the GGUF is partitioned into expert-aligned ranges, each digest-hashed under a Merkle root; the root is the model version.
- Committees — a bootnode assigns joiners least-covered shards, so complete coverage and redundancy hold by construction, not probability.
- Measured fetch path — RAM → disk → peers, with holder selection weighted by each peer's measured throughput; a layer's misses are fetched in parallel.
- Prediction — routing for every layer is predicted at layer 0 by a bolt-on head (no retraining), turning fetch stalls into overlapped prefetch.
- Version rollout — model upgrades are majority hardforks on a new manifest root; nodes on the old version are refused, never silently mixed.
Compared honestly
| System | Unit distributed | Crosses the network per token | A lost peer costs |
|---|---|---|---|
| Petals / exo / llama.cpp RPC | layer ranges | activations, every layer | the pipeline stalls |
| Browser swarms (WebRTC layer-split) | layer ranges | activations, every layer | the room stalls |
| Single-node offload (FreeToken, KTransformers) | nothing (one machine) | nothing | — |
| loom | expert blocks (immutable weights) | expert blocks, only on cache miss | a re-fetch from a replica |
Layer-split systems are deployed and effective; the distinction is structural. Single-node offload engines are complementary — they assume the whole model fits one machine's storage; loom removes exactly that assumption.
Research
- Pre-Gating Without Retraining — predicting every expert a token will need, from layer 0, with a bolt-on head. All traces, scripts, and trained heads released with the code.
- Whitepaper — architecture, trust model, bandwidth math, and the full decision log with every measurement (including the refuted ideas).