A small lantern-lit guide boat leading a huge freighter through thick night fog
Fringe Tech 16 Oct 2026 8 min read

Qwen3.8-Flash-Next at 60 Tokens per Second Across Its Full 262K Context on a 128GB MacBook

oMLX with the MTP head and the n-gram table on SSD holds about 60 tok/s from empty to 245K; llama.cpp with MTP runs 59 fresh and 46 at 245K. Full window, vision on, identical output.

In August I ran Qwen3.8-Flash-Next at its full 262,144-token context on a 128GB M5 Max MacBook Pro and measured about 33 tokens per second on a fresh context and about 11 with the window nearly full. Same laptop, same model, same 262K window with vision loaded, same harness, it now decodes at about 60 tokens per second at every depth: 64.5 tok/s fresh and 59.7 at 245K on oMLX with the model's MTP head enabled. llama.cpp with the same head runs 59.3 fresh and 45.6 at 245K. The output is unchanged, because speculative decoding verifies every drafted token; I compared greedy output with and without MTP at all four fills and it matched.

Three things got it there, in order of how much each mattered:

  1. Updating llama.cpp. I was still on an early build of the support branch.
  2. Loading the model's own MTP (multi-token prediction) head as a speculative drafter. Both runtimes support it.
  3. Testing the "MLX is 2x faster" claim with the setup it describes: oMLX with MTP and the 51B n-gram table on SSD, at full context.

Update llama.cpp first

The support PR (ggml-org/llama.cpp#27742) merged to mainline on 2026-08-27. My August numbers came from an earlier build of the PR branch. Mainline alone moved decode from 34.0 to 46.3 tok/s on an empty context (llama-bench tg128), a 36% gain, and from 31.7 to 42.3 at 16K. Prefill went from 1051 to 1140 tok/s empty and from 563 to 774 at 16K.

If you built this model's support from a branch in late August, rebuild master before you benchmark anything else.

The MTP head

Qwen3.8-Flash-Next ships with a 4B multi-token-prediction head, and unsloth/Qwen3.8-Flash-Next-GGUF publishes GGUF versions of it in an MTP/ folder. A friend suggested I try it as a speculative drafter. The head proposes the next few tokens cheaply; the main model verifies them in one forward pass and keeps the ones it would have produced anyway. Accepted drafts are free decode steps, so the win depends on acceptance.

unsloth's README recommends the "shared" heads and says stock mainline cannot use them, pointing to #28243 or their fork. That PR was superseded by #29761, which merged, so current mainline already has MTP. On mainline b76b133:

I draft 2 tokens per step (--spec-draft-n-max 2, unsloth's default). The README claims 1.3x to 1.7x; across the full context range I measured 1.32x to 1.49x on top of mainline.

One hand writing ahead with a quill while a second hand follows with a red pen, checking each word
The head drafts, the full model checks every token. Illustration.

llama.cpp with MTP, full context

The server as I run it daily: llama-server -c 262144 -fa on --jinja -np 1, BF16 mmproj loaded, 1024 generated tokens per point, greedy, real C++ source as the context filler. The stock model decodes at 44.4 tok/s empty and 34.6 at 245K without the head, and 59.3 empty, 57.1 at 32K, 55.3 at 128K and 45.6 at 245K with it. Vision read a serial off a label image correctly at every point.

Draft acceptance climbs as the context fills, from 70% empty to 82% at 128K, which is why the gain holds at depth. Temperature 0.7 drops acceptance a few points but barely moves throughput. Prefill with the head loaded is 5 to 11% slower (619 vs 677 tok/s on the deepest chunk).

Under my everyday 116 GB wired limit (iogpu.wired_limit_mb=118784), the full window plus the 4.14 GB head fits: 63.6 / 55.1 / 51.6 / 46.1 tok/s at empty / 32K / 128K / 245K, vision correct. Run-to-run noise is about plus or minus 5 tok/s.

A laptop on a desk at night beside a paper scroll that spills off the table onto the floor
A quarter-million tokens of context, decoding at about 60 tokens per second. Illustration.

The "MLX is 2x faster" claim

Two posts put MLX far ahead of llama.cpp on this model. One is a video titled "Qwen3.8-Flash-Next 125B: 60 TPS MLX vs 30 TPS llama.cpp on M5 Max"; its summary says the MLX side ran oMLX with MTP and the 51B n-gram table offloaded to SSD, in under 90 GB. The other is a Medium post I could not load. The video's 30 tok/s for llama.cpp matches an early build (33 to 34), and current llama.cpp with MTP runs 59 to 63 on an empty context, so most of that gap was build age and the missing drafter.

Plain mlx-vlm 0.7.8 at 4-bit, no MTP, beat mainline llama.cpp without MTP by 1.1 to 1.2x (56.4 vs 46.3 empty), but its weights are 111.5 GB against 94 for the GGUF and the 262K window did not fit on 128GB. So I ran the video's configuration.

oMLX with MTP, full context

oMLX is a server built on MLX. I ran 0.7.1rc1 with Jundot/Qwen3.8-Flash-Next-oQ4e-mtp, a mixed-precision quant at 4.72 bits per weight, 106.3 GB on disk including the vision tower and the MTP head. Two per-model settings matter: mtp_enabled, and qwen4_ple_ssd_offload, which maps the 51B n-gram table from SSD instead of keeping it resident. Same wired limit, same harness.

With the table on SSD, decode holds flat: 64.5 empty, 64.6 at 32K, 59.9 at 128K, 59.7 at 245K, against llama.cpp + MTP falling from 59.3 to 45.6. That is about 1.1x fresh and 1.3x at 245K. Without MTP, oMLX runs 50 to 58 tok/s. The log reports acceptance between 80 and 93% with adaptive draft depth up to 3. Resident memory was 69.7 GB.

Prefill is the larger lead: 1386 to 1667 tok/s per chunk on oMLX against 733 to 619 on llama.cpp + MTP, about 2.5x on prompt reading.

Keeping the n-gram table in RAM is faster while the context is short, 100 to 108 tok/s empty and 67 at 128K at 99.5 GB resident, but on 128GB it is a short and medium context mode. A fresh 245K prompt was refused by oMLX's prefill guard (about 110 GB peak against a 108.3 GB limit); the 245K run succeeded only from a cached prefix.

Caveats: the quants differ (oQ4e at 4.72 bits per weight, UD-IQ4_XS about 4.25), so some of oMLX's edge may be the quant rather than the runtime. 0.7.1rc1 is a release candidate. On oMLX, temperature 0.7 ran faster than greedy (79.7 vs 64.5 empty) and I have not explained that.

A large cargo ship easing into a narrow canal lock at night with inches to spare
Weights, KV cache for 262K tokens, and the drafter head all have to fit under the wired-memory limit. Illustration.

The abliterated variant (llama.cpp only)

huihui-ai/Huihui-Qwen3.8-Flash-Next-abliterated-v2-GGUF (released 2026-10-11, same UD-IQ4_XS quant, its own BF16 mmproj) matched stock on my small harness: tool calls 18/24 against 17/24, reasoning, instruction following and vision equal, and 4 of 5 benign but commonly refused prompts answered against stock's 3. No measurable capability loss on this small set, and it decodes at the same speed with the same MTP acceptance, so the llama.cpp recipe below applies to either. There is no oMLX build of it.

Then vs now

I ran the August build through today's harness at the same fills. It reproduces the August numbers, 32.6 tok/s fresh and 11.3 at 245K.

Line chart of decode speed in tokens per second against context fill for four setups
Decode speed by context fill, same harness, vision loaded.

Best setup against August: 2.0x fresh, 2.4x at 32K, 3.7x at 128K, 5.3x at 245K. The gain grows with depth because mainline's long-context attention is much faster than the early branch's, and oMLX's decode barely slows as the window fills.

Prompt processing improved even more. Filling to 245K took about 23 minutes on the August build, about 6.3 on llama.cpp with the head loaded, and about 2.5 on oMLX.

Bar chart of minutes needed to fill a 245K-token prompt
Time to read a 245K-token prompt.

Which one to run

For long context on a 128GB Mac: oMLX with MTP and the n-gram table on SSD. It holds about 60 tok/s to the end of the window, fills a 245K prompt in about 2.5 minutes, and leaves more than 45 GB of headroom under the wired limit.

When the context stays short: oMLX with the n-gram table in RAM, for 80 to 108 tok/s, knowing the guard will refuse a cold prompt near the full window.

When you want the abliterated GGUF, or already run a llama.cpp stack: mainline with the MTP head. It is about 5 tok/s behind oMLX on a fresh context and holds 46 tok/s at 245K, four times the August number.

The recipes

Both need the GPU wired-memory limit raised. It resets on reboot; I set it at boot with a LaunchDaemon:

sudo sysctl iogpu.wired_limit_mb=118784

oMLX

Install the release wheel into a Python 3.12 venv (or the macOS app from the DMG); nothing to build:

python3.12 -m venv omlx && source omlx/bin/activate
pip install omlx-0.7.1rc1-cp312-cp312-macosx_15_0_universal2.whl
hf download Jundot/Qwen3.8-Flash-Next-oQ4e-mtp

oMLX finds models in the Hugging Face cache. Set three per-model values, in the admin UI or in model_settings.json:

"mtp_enabled": true,
"qwen4_ple_ssd_offload": true,
"max_context_window": 262144

Then serve with the memory guard:

omlx serve --memory-guard-gb 114

Set qwen4_ple_ssd_offload to false for the faster short-context mode.

llama.cpp

Build mainline with the normal cmake Metal build (steps in the August post). Download the self-contained head:

hf download unsloth/Qwen3.8-Flash-Next-GGUF --include "MTP/mtp-Qwen3.8-Flash-Next-Q8_0.gguf"

The daily command, with your own paths:

llama-server \
  -m Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf \
  --mmproj mmproj-Qwen3.8-Flash-Next-BF16.gguf \
  -md mtp-Qwen3.8-Flash-Next-Q8_0.gguf \
  --spec-type draft-mtp --spec-draft-n-max 2 \
  -c 262144 -ngl 999 -fa on --jinja

Point -m at the first shard and llama-server finds the rest. Swap in the huihui shards and mmproj for the abliterated variant; the -md line stays the same.

If you have MTP numbers on an M4 Max or M5 Max, on either runtime, post them in the comments.

Sources

Fringe Tech Local LLMs

Comments

// Comments are reviewed before appearing. No spam. No noise.