Blog
Technical deep-dives, case studies, and engineering insights from the Richland team.
One 180B MoE across five devices: two PCs, an M4 Pro MacBook, and two iPhones — 37–39 tok/s at 60k context via pipelined speculation
A practitioner chains five heterogeneous devices into a single Qwen3.8-Flash-Next instance, hiding cross-device latency with pipelined speculation — but the code isn't released yet.
Anthropic replaced git with Scion after agent merges hit 30,000 a day — the checkout, not the commit, was the bottleneck
A seven-person team built a server-side, tiered-cache VCS after agent merges grew 12x in four months — and Claude rewrote the migration tool itself.
NVIDIA opens the coding-agent midtraining recipe: 500B tokens at 256K context, and the ablation that shows why ~10% agentic traces matter
The Nemotron team's arXiv paper discloses the stage between pretraining and post-training — with data blend, eval protocol, and scaling curves.
capy-flash runs a 125B MoE in full NVFP4 on two 16 GB GPUs at 100 tok/s — and documents the TP=2 partial-sum rounding trap
Apache-2.0 engine offloads routed experts to host RAM; the author shows FP16 transport payloads restore single-GPU numerical parity at no speed cost
ORL: diffusion LLMs get a learnable "tokens per forward" — 2,788 tok/s on one RTX 5090, 7,708 on 8× B200, fully open source
Together AI's on-policy residual learning plus the dlmrt engine make diffusion decoding competitive with autoregressive speculative decoding — and the release is fully open, with a bit-for-bit reproducible QwenDLM recipe.
OpenGCUA: an Apache-2.0 harness lifts all 8 frontier models it tested on OSWorld-v2 (+19.6 points, −53% steps) — but real-device runs show the model gap it hides
Alibaba Tongyi-MAI's open computer-use stack ships harness, SDK, and trajectories — no weights — and its own 4-task real-Mac results contradict the benchmark's equalizing story.
COUNT: a 797 GB read-only n-gram table in host RAM buys a 1B LM the loss of 57% more compute — zero learned parameters
UW/AI2's COUNT builds n-gram embeddings from precomputed corpus counts in host RAM, adds no learned parameters, and swaps tables at inference for training-free domain adaptation.
When the KV cache is slower than the GPU: a 124× p99 fix for context-parallel prefill
A production GLM-5.3 teardown finds remote KV hits slower than recompute — break-even is 2.53 GB/s, and the fix is sharding by rank.
Latent Compaction: the KV cache as a trainable network — 96.7% on RULER at 32× (logical) compression, code released
Tsinghua's THUNLP optimizes cache entries per document with 2,000 gradient steps of test-time training and frozen weights; the per-document GPU cost is nowhere published.
jitllm relocates a running model's layers between GPUs mid-reply: 8 ms pause, KV cache moves with the layers
A Go engine treats layers like OS processes: JIT kernels, a paged KV cache that travels with them, and an 8 ms handoff on T4s.
Delta Tokenizer: exact delta tokenization cuts agent-session P99 TTFT 9.5× under load, 23× when long requests tie up the front end
ASU's CPU tokenization front end reuses the previous turn's token IDs instead of re-encoding the whole session — ~40× per-turn speedup, 59% median TTFT cut inside SGLang, reference-identical on every checked output.
openTPU: AI agents wrote the RTL, ISA, and compiler for the chip that runs their own inference — bit-exact across ten models on a Kintex-7
FeSens' Apache-2.0 monorepo pairs a deliberately simple, no-cache accelerator with a verification harness that measures a one-ulp floor instead of pretending exact equality.