Qwen released Qwen3.8-Flash-Next on August 26. By that evening it was running on a 128GB M5 Max MacBook, and the next day it completed a full benchmark sweep with the entire native 262,144-token context window allocated: a 4-bit quant of a model that is 176B parameters on disk, generating at 33 tokens per second on a fresh context and 11 tokens per second with a quarter-million tokens loaded, with no offload tricks.
What Qwen shipped
Qwen3.8-Flash-Next is an open-weight preview of the Qwen4 architecture (model card, README). The parameter accounting, stated once because the headline numbers confuse everyone: a 125B mixture-of-experts main model plus a 51B n-gram embedding table makes the quoted 176B, and a 4B multi-token-prediction head for speculative decoding rides along. The MoE runs 512 experts with 10 routed plus 1 shared active per token, about 6B active parameters. It is multimodal, with a vision tower included. Context is 262K native, extensible to 1M with YaRN.
Qwen's README claims it trains at about 1/9 the cost of Qwen3.7-Plus while outperforming it, and the model card shows the 6B-active model beating much larger models on agentic benchmarks like CoWorkBench and JobBench. Vendor-curated numbers, but the announced pricing tells the same story: the hosted version is priced at $0.16 per million input tokens and $0.47 per million output. This model exists to be cheap to run, which is what makes it interesting to run at home.
Why 262K tokens fit in a laptop
Long context normally dies on the KV cache. A dense 70B model stores about 320 KiB of cache per token; at 262K tokens that is roughly 86 GB before you load a single weight.
Flash-Next attacks this with a hybrid layer stack. Of its 48 layers, only 12 are real attention layers. The other 36 are Gated DeltaNet layers: linear-attention layers that keep one fixed-size "fast weight" state per head instead of an ever-growing token cache. Per token, a DeltaNet layer reads what its memory currently predicts for the incoming key, then writes back only the correction, with learned gates deciding how fast to forget. The state never grows, so cost per layer is the same at 4K tokens and at 262K.
That state is a lossy summary of history, which is why every fourth layer is still genuine attention with full recall. Those 12 layers use 2 KV heads at head dimension 256, so the entire growing cache costs 24 KiB per token:
- 262,144 tokens (native): 6.4 GB
- 1,000,000 tokens (YaRN): ~25 GB
The attention layers are themselves sparse. Qwen Sparse Attention keeps the full KV around but attends selectively: a small indexer compresses the key cache 4:1 into micro-block summaries, scores every block against the current query, and runs real softmax attention over only the top 512 blocks, about 2,048 tokens, no matter how deep the context is. Below that budget it is identical to dense attention.
With the cache solved, the limiting factor is the weights: 93.7 GB on disk at 4-bit, which fits in 128 GB of unified memory with room left for the full-context KV cache.
The 51B n-gram table
The n-gram embedding table is 51B parameters that work as lookup rows rather than compute weights. For each token, the model hashes the previous two and three tokens into 16 lookups against a table of 320 million rows, concatenates the results, and injects them into the second decoder block. Qwen's stated reasoning: embeddings scale capacity with almost no compute, and unlike MoE experts the lookup depends only on token identities, so it can be prefetched or offloaded cheaply.
At inference the access pattern is radically sparse, exactly 16 rows touched per token out of 320 million, so llama.cpp simply memory-maps the table: cold pages never load and hot n-grams live in page cache.
The table dominated the release-day discussion. The top comment on the Hacker News thread doubted a usable sub-100GB quant could exist at all. In the Hugging Face discussions someone did the napkin math on streaming the table from SSD, concluded it meant random reads on every token and maybe 7 tokens per second, and called it impractical. About 16 hours later another developer posted a working Metal I/O implementation streaming the table from disk with no measurable decode loss, and the original skeptic ported it to CUDA.
The recipe
Nothing was merged anywhere when I set this up on release night. The steps, which got easier within a day:
1. llama.cpp with qwen4exp support. Unsloth opened PR #27742 on release day; it merged into mainline about 24 hours later, complete with the delta-net layers, the sparse attention, the memory-mapped n-gram table, and vision. I built the PR branch; you can now just build master:
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j2. The unsloth 4-bit GGUF (three shards, 93.7 GB), from unsloth/Qwen3.8-Flash-Next-GGUF:
hf download unsloth/Qwen3.8-Flash-Next-GGUF --include "UD-IQ4_XS/*" --local-dir ./qwen383. Raise the GPU wired-memory limit. macOS caps GPU-wired memory around 75% of RAM by default, about 96 GB on a 128GB machine. Weights plus full-context KV need ~105 GB resident:
sudo sysctl iogpu.wired_limit_mb=118784That grants the GPU 116 GB and leaves 12 GB for the OS. It resets on reboot.
4. Serve at full native context. Point llama-server at the first shard; it finds the rest:
llama-server -m qwen38/UD-IQ4_XS/Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf \
-c 262144 --host 127.0.0.1 --port 8089One trap for scripted use: the current llama-cli drops into an interactive chat prompt after answering. Pass -st (single turn) or a background invocation will sit at that prompt forever. Mine did, overnight.
The benchmarks
llama-bench at build 213df585b (the PR branch as merged, Metal backend), pp512 and tg128 at increasing KV depths, single run per depth because the deep fills are expensive:
llama-bench -m <shard1> -p 512 -n 128 -d 0,4096,16384,32768,65536,131072,262144 -r 1
| KV depth | Prefill pp512 (t/s) | Generation tg128 (t/s) |
|---|---|---|
| 0 | 966 | 33.0 |
| 4,096 | 776 | 30.7 |
| 16,384 | 456 | 27.9 |
| 32,768 | 324 | 24.8 |
| 65,536 | 241 | 23.0 |
| 131,072 | 154 | 17.0 |
| 262,144 | 80 | 10.9 |
The 262K row is the point of the exercise: the full native window allocates, fills, and generates on one machine. Memory at depth peaked at 106 GB wired out of the 116 GB allowance, about 126 GB total in use. macOS compressed roughly 10 GB of inactive memory and stayed responsive for the whole sweep. Tight, but stable.
Against the other 128GB machines in the day-0 reports (numbers from unsloth's speed thread and the HN thread; quants and harnesses differ, as labeled):
On the same 4-bit quant, the M5 Max leads the Strix Halo box on both generation (30.7 vs 23.3) and prefill (966 vs 533 at pp512). The M1 Ultra report used the 1-bit quant, so treat that bar as its owner's setup rather than the machine's ceiling. Corroborating long-context reports from the same threads: another 128GB M5 Max holding 180K context at around 100 GB resident, and a DGX Spark paging the n-gram table from NVMe at about 80 GB resident.
The real price of full context is prefill. Integrating the measured curve, filling all 262K tokens one-shot takes about 28 minutes. With prompt caching that is a once-per-corpus cost rather than per-message, but it reframes what the window is for: load a codebase or a book once, then interrogate it.
Caveats
- The architecture support is a day old. The merge itself shook out a graph-reuse bug and three quantizer fixes during review, MTP speculative decoding is deferred to a follow-up (meaning decode speed has real headroom), a Vulkan flash-attention incompatibility surfaced post-merge, and a vision bug slipped through. Expect churn for a couple of weeks.
- Quantized KV cache has open issues with this architecture. That is what keeps the 1M YaRN mode out of reach here: fp16 KV at 1M is ~25 GB and the total lands at the very edge of 128 GB. Native 262K is comfortable today; 1M wants either the KV fix or more memory.
- I benchmarked speed, not quality. The 4-bit quant answered a brief generation check sanely; I have not tested retrieval quality at depth, and nobody has rigorous numbers yet on what IQ4_XS costs against the FP8 reference on this architecture.
- Nobody has reported completing a real million-token workload locally yet. The deepest day-0 reports are 180K on a Mac and a DGX Spark run at 256K with SSD offload.
Verdict
64K context is where the interactive experience holds, which is why I call that the daily-driver line rather than the full window: a cold 64K fill costs about three minutes and generation runs at 23 tokens per second, comfortably faster than reading. Past 64K both rates slide, and by 262K a session opens with 28 minutes of prefill and answers arrive at 11 tokens per second, slower than most people read. The full native window is still a working tool, but for jobs that justify that entry cost, not for conversation. The larger point is architectural: a hybrid stack where only a quarter of the layers pay for recall, plus a sparse attention budget, means the KV cache for a quarter-million tokens now costs less than the operating system running it. The next unlock is concrete: when quantized KV lands for this architecture, the 1M window comes into range on the same 128GB machine.
Comments
// Comments are reviewed before appearing. No spam. No noise.