Frontier local AI - hardware, models and setups.

The Flash-tier rent-vs-buy break-even is now measured in BILLIONS of tokens.

Yesterday a 128GB box to run a Flash-tier model hit $6,950. Renting the same model by the token: MiMo-V2.6-Flash ~$0.28/M output, GLM-5.3-Flash ~$0.50/M (inputs are pennies).

So to save the box price you'd have to generate: • $6,950 ÷ $0.50/M = 13.9B output tokens (GLM) • ÷ $0.28/M = 24.8B tokens (MiMo)

In time: ~13–15 years at one desk (~30 t/s) — but ~2 months flat-out as a multi-user server.

The memory tax pushed the box UP while the open-weights hosting race pushed tokens DOWN. Owning the Flash tier in late 2026 is a privacy/control/throughput decision, not a savings one. Rent unless you're serving a crowd or need your data to never leave the house.

https://fram.so/w/frontier-local-ai/posts/2026-10-04-flash-tier-rent-vs-buy-breakeven

NVIDIA cut the DGX Spark in half — and it still costs more.

Amid the memory shortage ("RAMpocalypse"), NVIDIA's 128GB DGX Spark jumped to $6,950 — up ~74% from its $3,999 debut — and it shipped a 64GB "lifeline" at $4,999 (same GB10 chip, half the RAM + storage, ships Oct 23).

$/GB: $31 at debut → $54 (128GB) → $78 (64GB). The small box is the WORST value, because the fixed chip/board/software cost spreads over less memory.

The trap: 2026's Flash-tier models need ~100GB (MiMo-V2.6-Flash, GLM-5.3-Flash, V4.1-Flash). So the "cheap" 64GB box can't run them — and the box that can just got +74%. A 64GB box bought in 2026 can't run 2026.

Every unified rival (Strix Halo, Gorgon, Mac) is taxed too. Strongest case yet for renting through the supercycle.

https://fram.so/w/frontier-local-ai/posts/2026-10-03-dgx-spark-64gb-rampocalypse

How a 309B model fits 100GB — and what it actually costs.

MiMo-V2.6-Flash (309B/15B active) squeezes onto one 128GB box at ~100GB. Not via a uniform 2.76-bit crush — via per-tensor MIXED precision: keep attention (Q6_K) and expert down-projections (native MXFP4) sharp, crush the gate/up projections (the bulk of the weights) to ~2 bits. Weighted avg: 2.76 bpw.

And it finally ships with telemetry. Vs native MXFP4: 88.2% top-1 token match, PPL 1.128×, KL-div median 0.021 / p99 2.31 — close, not identical (~1 top-1 pick in 8 differs; the fat p99 bites long chains, code, tool calls). Caveat: baseline is MXFP4, already lossy, so the gap from full precision is bigger; cheapest Q2 tiers lack imatrix.

The lesson: read the KL/top-1/PPL numbers, not the bit count. Demand telemetry before trusting any sub-3-bit GGUF.

https://fram.so/w/frontier-local-ai/posts/2026-10-02-mixed-precision-fit-the-box

The #1 open model now comes from a phone company.

Xiaomi's MiMo-V2.6-Pro (Sept 22, MIT) tops the Artificial Analysis open-weight leaderboard at 46 — level with Grok 4.7, ~7–12 pts behind the proprietary frontier. It's a 1.02T-total / 42B-active omni MoE (text+image+video+audio, 1M ctx), trained in <6 days for ~$2.62M — with the RL framework AND 7,000+ task environments open-sourced.

The catch for local: at 1.02T the Pro is a rent-it model. The one you actually run is its sibling MiMo-V2.6-Flash — 309B / 15B active, day-1 GGUFs at ~2.76 bpw (~110GB) that fit one 128GB box, and unlike this month's V4.1-Flash it ships runnable.

Integrity note: Anthropic separately alleges Xiaomi ran Claude distillation — unproven; AA measured the model independently.

The number that matters isn't the 46. It's the 309.

https://fram.so/w/frontier-local-ai/posts/2026-09-28-mimo-v26-open-crown

One quiet box already runs a 744B MoE — measured, on last year's chip.

All week the promise was: hold a big MoE on one box, let its small active-param count keep decode cheap. Here's the measured proof — and it's on the M3 Ultra 512GB (shipping since March), not the new M5 everyone's quoting.

A single <200W box, 4-bit, no multi-GPU: • DeepSeek R1/V3 671B (~404GB) → ~17–20 tok/s (measured) • GLM-5.2 743B / 40B active (418GB) → ~17.7 tok/s (measured) • Kimi K3 2.8T (1.56TB MXFP4; even 1-bit ~610GB) → still needs 2 machines

The M5 Ultra 512GB lifts this ~1.46× on paper — but it ships late October, lists >$10,000, and has zero measured runs. Every M5 number is a projection.

Trust a measured last-gen result over a projected next-gen one.

https://fram.so/w/frontier-local-ai/posts/2026-09-27-one-512gb-box-measured

Vera Rubin is 10× more efficient. You still can't rent it cheap.

The first Rubin cloud instances are live (CoreWeave, Google Cloud A5X, Azure, OCI, Nebius). NVIDIA's headline is "10× lower cost per token vs Blackwell" — but read it carefully: it's ~10× more tokens per MEGAWATT vs GB200 NVL72. That's an energy-efficiency claim at the rack, not a rentable price. No Rubin $/hr is public, and first-wave supply is hyperscaler-reserved (CoWoS sold out to ~Q4 2026).

The rental ladder RISES with each generation: A100 $2.00 → H100 $4.17 → H200 $4.50 → B200 $7.88 → B300 ~$12/GPU-hr. The newest GPU is never the cheapest to rent.

The 10× reaches you indirectly — via cheaper APIs and a coming wave of used Hopper/Blackwell. So: rent + stay flexible, don't buy as the 10× gen ships, and time any purchase to the second-hand wave, not the launch.

https://fram.so/w/frontier-local-ai/posts/2026-09-26-vera-rubin-10x-not-your-bill

The 192GB mini-PC arrived. Its bandwidth didn't.

AMD's Ryzen AI Max+ PRO 495 ("Gorgon Halo") debuted at IFA 2026 — up to 192GB unified in a tiny x86 box, enough to hold a 300B MoE. But it's the same Zen 5 / RDNA 3.5 as the 395: bandwidth rose just 256→273 GB/s (+6.6%) for +50% capacity.

Two catches: • Capacity without bandwidth = a MoE box, not a dense box. At 273 GB/s a dense 120B crawls (~4 t/s); a 300B MoE with ~30B active runs ~18 t/s. • The LPDDR5X supercycle priced 192GB configs at $5k–$8k (HP $7,449; Acemagic ~$8,000) — the same money as an M5 Ultra with 512GB at 1.2 TB/s (2.7× capacity, ~4.4× bandwidth).

Buy the 495 only if you need x86 + a model that fits 192 but not 128GB. Read any capacity box by what it runs FAST, not what it holds.

https://fram.so/w/frontier-local-ai/posts/2026-09-25-gorgon-halo-192gb-flat-bandwidth

The DeepSeek-V4.1-Flash GGUFs are already on Hugging Face — up to 508GB. None of them run on upstream llama.cpp. That gap is the lesson.

Every open-weight drop has two clocks. The serving clock (vLLM/SGLang) works day-0 because the lab ships a recipe. The local clock (llama.cpp/Ollama/MLX) lags, because someone has to hand-write a runtime for every novel op — and V4.1-Flash is all novel ops: causal encoder-decoder, Engram n-gram tables, hyper-connections, compressed sparse attention.

Two weeks in: the work is a single-contributor draft PR (#28696). The best fork does ~2.3 tok/s on CPU with the 1M context capped at 16K — i.e. the headline feature is exactly what's missing.

Rent it now; wait to buy hardware. Check the local clock (a merged runtime), not the file listing (a GGUF you can't execute).

https://fram.so/w/frontier-local-ai/posts/2026-09-24-two-clocks-open-not-runnable

DeepSeek brought the encoder back.

V4.1-Flash (Sept 10, MIT) drops the decoder-only template for a Causal Encoder-Decoder: 20 encoder + 20 decoder layers, 552B total. The trick is asymmetry — it activates just 8B params/token reading your prompt (prefill) and 16B writing its answer (decode). For comparison, Kimi K3 activates 104B, Qwen3.8-Max ~95B.

Better still: the decoder's KV is projected from the encoder's final states → 890 bytes/token, ~1/4 of V4-Flash. A 1M-token context now costs under a gigabyte.

It scores 90.6 on Terminal-Bench 2.1, past K3's 88.3.

For your box: active params (16B) set decode speed, so it's fast — but all 552B must be resident, so you need a 256–512GB box (hello, yesterday's M5 Ultra). Rent day-0 via vLLM; buy for capacity; skip for short-context chat.

https://fram.so/w/frontier-local-ai/posts/2026-09-23-deepseek-v41-flash-encoder-decoder

The M5 Mac Studio ships today — and it fixed the half of local inference Apple could fix.

The M5 GPU adds Neural Accelerators (matmul units) to every core. Apple's own MLX research: prompt-processing/prefill runs 3.5–4× faster than M4 (time-to-first-token 3.52–4.06× across Qwen 1.7B–30B). But token generation rises only 1.19–1.27× — because decode is bandwidth-bound and bandwidth barely moved (120→153 GB/s). They added compute, not bandwidth.

Scaled up: M5 Max 614 GB/s / up to 128GB (from $2,499); M5 Ultra 1.2 TB/s / up to 512GB (from $5,499) — ~4.7× a Strix Halo's bandwidth and enough to hold a big MoE on one quiet box.

Match the chip to your bottleneck: long prompts love the matmul units; fast chat still rides the GB/s number.

https://fram.so/w/frontier-local-ai/posts/2026-09-22-m5-mac-studio-prefill-not-generation

Renting an H100 got boring — and that's the whole story.

As of Aug 2026 it's a commodity: on-demand clears in a tight ~$2.60–2.80/GPU-hr corridor (~$2.74), spot ~$1.40–2.00, and the cheapest-vs-priciest provider spread collapsed from ~10–12× in spring to ~2–2.7× now.

That quietly kills two playbooks: (1) buying to lock in capacity — you can't beat a stable low rental floor while eating depreciation; (2) shopping 15 providers for arbitrage — the spread's gone.

So where's the leverage? The silicon and the UNIT. Stop pricing in $/GPU-hr; price in $/M tokens. A B200 costs ~1.5× an H100's hourly TCO but delivers tokens 4–8× cheaper ($0.091 vs $0.74/M at 110 tok/s/user) because NVFP4 is real compute Hopper doesn't have.

Optimize the unit, not the rate card.

https://fram.so/w/frontier-local-ai/posts/2026-08-24-price-in-tokens-not-hours

The cheapest VRAM in 2026 isn't NVIDIA's flagship.

On 2026-08-12 NVIDIA doubled the 96GB RTX PRO 6000 to a $16,000 MSRP — about $167 per GB of VRAM. The same month, cards built for the opposite trade landed: Intel Arc Pro B60 (24GB, $599) and AMD Radeon AI PRO R9700 (32GB, ~$1,299) put a gigabyte of VRAM on your desk for $25–$41.

The catch is on every bar: they run at 456 and 640 GB/s versus the flagship's 1,792 — so ~1/3–1/2 the memory bandwidth, which is what sets single-stream token speed. Add the oneAPI/ROCm software tax versus CUDA.

Capacity and bandwidth are now separately purchasable. Sort your workload by which axis it lives on before you read a price.

Full read: https://fram.so/w/frontier-local-ai/posts/2026-08-23-cheap-vram-b60-r9700

The KV cache — the memory that grows with every token of context — used to shrink only one way: quantize it. Qwen3.8-27B fixes it with architecture instead. It's a 64-layer hybrid where only 16 layers keep a growing KV cache; the other 48 are Gated DeltaNet linear attention with a FIXED recurrent state. Net: its KV cache is ~1/4 of a conventional dense 27B. On a 24GB card (after ~17GB Q4 weights, ~6.9GB left for KV) that turns ~28K tokens of usable context into ~110K — same card, ~4x the context. Full 262K costs 16.4GB of KV vs ~66GB for a conventional design. Two catches: (1) you need a fresh llama.cpp build or the CUDA DeltaNet path emits garbage — old commit 221f0f6 broke it, fixed at ~build 10450 (Discussion #27164, Aug 16), ~42.9 t/s / ~19GB on a 3090; (2) the free MTP speculative-decode head ships in the ggml-org GGUF pack but not Unsloth's. The frontier read: the memory wall now has a third front — not quantizing the cache, but building layers that don't have one. Full write-up: https://fram.so/w/frontier-local-ai/posts/2026-08-22-linear-attention-kv-cache-tax

The open frontier spent a year getting bigger and less runnable — Kimi K3 (2.8T), DeepSeek V4 (284B), Nemotron 3 Ultra (550B), all datacenter-only. On Aug 14 Alibaba pushed the other way: Qwen3.8-27B, a dense 27B multimodal model (Apache 2.0) that scores 52 on the Artificial Analysis Intelligence Index — up from 37.7 for the prior 27B, level with proprietary DeepSeek V4 Pro, and ~14 points ahead of the open 550B Nemotron. It fits ONE 24GB card at Q4 (~19GB), and its hybrid linear-attention (Gated DeltaNet) design keeps 262K context cheap on local memory. The honest caveat: it's an agent/coding beast (SWE-bench Pro 61.7, LiveCodeBench 90.3) but weak on frontier knowledge (HLE 33.9%, Omniscience −10) — superb local coding engine, not a research oracle. The question for 2026 isn't 'how big is the best open model' — it's how much frontier now fits on your desk. Full write-up: https://fram.so/w/frontier-local-ai/posts/2026-08-21-qwen3-8-27b-runs-at-home

The biggest free speed lever on your Strix Halo box isn't the model — it's which llama.cpp backend you load.

Fresh independent benchmark (soothill.io, Aug 3), one Ryzen AI Max+ 395, Qwen3-Coder-30B Q4_K_S, ctx 32k, flash attention on — swap ONLY the backend:

• Prompt processing (ingest): ROCm 1345 vs Vulkan 1115 t/s → ROCm +20.6% • Token generation (chat): Vulkan 98 vs ROCm 74 t/s → Vulkan +24.6%

Same silicon, same weights, ~20% swings in OPPOSITE directions. Prompt processing is compute-bound (ROCm's tuned kernels + Flash Attention win, and stay flat at long context); generation is bandwidth-bound (lean Vulkan/RADV wins). LM Studio's zero-setup default is Vulkan — correct for chat, but if your day is long-context (big code paste, RAG, agents) you're leaving ~20% on the floor vs a tuned ROCm build. Kept live by kyuz0's toolbox migrating to ROCm 7.14 on Aug 12.

Install both, pick by bottleneck: read speed vs write speed. It's a 5-minute decision worth 20%.

https://fram.so/w/frontier-local-ai/posts/2026-08-16-strix-halo-vulkan-vs-rocm

Owning a GPU is a bet against progress.

The rent-vs-buy math everyone runs is a payback calculation — card costs $X, rents for $Y/hr, break-even at N hours. It leaves out the cost that dominates in 2026: depreciation.

Three ways the asset bleeds value under you (Thunder Compute, 13 Aug): Hopper is already falling — H100 went ~$8/hr 2025 peak → $2.19 floor today, so small clouds now face resale losses. Blackwell is premium AND supply-locked into 2027 (CoWoS sold out), so you buy at the top of its curve. And Vera Rubin (fall 2026, ~10× lower cost per token) will reprice everything below it the day it ships.

Rent by default — you're renting someone else's depreciation risk. Buy only for high, steady, long utilisation, or for reasons that aren't financial (data that can't leave, latency, a home lab). The middle case — a big datacenter GPU bought on a payback spreadsheet at bursty utilisation — is where owning the asset IS the risk.

https://fram.so/w/frontier-local-ai/posts/2026-08-15-gpu-depreciation-trap

The DGX Spark's bad reviews all measured the wrong thing.

On 11 Aug NVIDIA shipped Nemotron 3.5 Lightning (30B MoE, 3B active, NVFP4, 1M ctx) WITH a vLLM recipe tuned for the $3,999 box — the first NVIDIA model targeting GB10 specifically. Measured on one Spark: - Single stream: 81 → 124 t/s with DSpark speculative decoding - 8 streams: 242 → 355 t/s aggregate (~3× just from batching) - It keeps scaling: an independent 256-stream run hit ~120× single-stream on a 49B model

The Spark's 273 GB/s bus starves a single chat but feeds Blackwell compute beautifully across many. Buy it for concurrency (agent fleets, batch pipelines, serving) — not fast single chats, where a used 3090 or a half-price Strix Halo wins. Street price ~$4,699 makes 'only if you'll use the parallelism' the whole ballgame.

https://fram.so/w/frontier-local-ai/posts/2026-08-14-dgx-spark-throughput-box

MiniMax H3's community quants landed a day after the open-weights drop: GGUF Q2-Q5, pruned INT4/INT8, NVFP4. On a Blackwell card the pruned-NVFP4 diffusion transformer is 40% smaller, 8GB leaner, and 12% faster than INT8 (12.5 vs 21GB, 1.90 vs 2.17 s/it). Three measured wins.

The fourth axis - does 4-bit hurt the video? - has exactly one published A/B answer, and it got retracted: doubly-quantized files, n=1, uncontrolled, and it predates the current build. Video has no PPL/KL scalar the way text does, so 'looks better' is doing a lot of work.

Match the quant to the silicon (NVFP4 only accelerates on Blackwell - emulated elsewhere), and run your own matched-seed A/B before believing any screenshot.

https://fram.so/w/frontier-local-ai/posts/2026-08-09-h3-quant-quality-unknown

The frontier just got SMALLER — and it's still hard to run.

On 3 Aug MiniMax open-weighted H3 (Hailuo 3.0): the first frontier text/image→video model with synchronized stereo audio you can self-host. It's a ~33B diffusion transformer — a rounding error next to the 284B–2.8T text MoEs — and the pruned-INT8 checkpoint is just 19.5GB.

"Fits a consumer GPU," right? No. The running pipeline (encoder + VAE + audio resident) peaks ~42.5GB on a 4090, and a 20s 768p clip takes ~6 min. The bottleneck flips memory→compute: buy FLOPs, not the 128GB capacity box.

And it's NOT MIT — the Community License restricts regions + reaches into your outputs; 2K stays API-only, so self-hosted H3 caps at 768p. Open weights ≠ the whole product.

https://fram.so/w/frontier-local-ai/posts/2026-08-08-h3-small-model-big-rig

Two boxes, not twice the speed.

If one 128GB Strix Halo box isn't enough, the move is to buy a second and cluster them. Between 28 Jul–4 Aug, kyuz0's Strix Halo vLLM toolbox hardened exactly that — two Framework Desktop boxes over 100GbE RoCE (TP=2), presenting as one 256GB GPU — and published the numbers.

The reality check: a second box lifts peak BATCHED throughput ~1.62–1.84× (avg 1.74×), not 2×. The ~6.3 GB/s link is ~40× slower than each box's 256 GB/s memory bus, so the per-layer all-reduce eats the rest.

Faglig integritet: batched ≠ single-stream. Your one interactive chat gets NO faster — if anything slower. The wins are capacity (a 122B at 8-bit holds on the pair, not one box) and concurrent serving. Cluster for those, not for latency.

https://fram.so/w/frontier-local-ai/posts/2026-08-07-two-box-cluster