A Used Desktop as a Local LLM Server: a 125B Model With Vision and 262K Context on an 8 GB GPU

A used gaming desktop with an 8 GB RX 7600 and 128 GB of DDR5 runs Qwen3.8-Flash-Next, a 125B-parameter mixture-of-experts model with vision and a 262,144-token window, at about 12 tokens per second. The model is 94 GB and lives in system RAM; the card holds attention, the KV cache and the vision tower. It is served as an OpenAI-compatible endpoint on the LAN.

Generation speed on an empty context, mainline llama.cpp with Vulkan:

Qwen3.5-4B, entirely on the GPU: 67 tok/s
Qwen3.6-35B-A3B: 27.6 tok/s, 10.4 at 255K
Qwen3-Coder-Next (80B-A3B): 21.5 tok/s, 8.6 at 255K
Qwen3.8-Flash-Next (125B-A6B): 13.0 tok/s, 5.7 at 64K
Qwen3.8-27B dense: 4.45 tok/s

The hardware

CPU: Intel i9-14900K, 8 performance cores with hyperthreading plus 16 efficiency cores.
RAM: 128 GB DDR5, four 32 GB sticks at 4000 MT/s. DDR5-4800 and 5200 failed memory training with four sticks populated. Measured streaming read: 59 GB/s against a 64 GB/s ceiling.
GPU: AMD Radeon RX 7600, 8 GB, PCIe 4.0 x8.
Storage: two 4 TB NVMe drives in RAID0.
OS: Ubuntu with the Mesa RADV Vulkan driver.

Macro photograph of four memory modules seated side by side in a dark motherboard — Four memory sticks, seated and ready.

Memory bandwidth sets the speed. A mixture-of-experts model reads only its active parameters per token (6B of 125B for the flagship, 3B of 35B for the mid model), and with an 8 GB card most of those bytes come out of system RAM on every token. Decode speed is roughly RAM bandwidth divided by active weight bytes per token.

Build llama.cpp with Vulkan

ROCm is not needed.

sudo apt install build-essential cmake git libvulkan-dev glslc vulkan-tools \
  mesa-vulkan-drivers spirv-headers libcurl4-openssl-dev
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build -DGGML_VULKAN=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j

Built at commit 547208dca. Four things that bit:

Configure fails without spirv-headers; glslc does not pull it in.
The Intel iGPU shows up as a second Vulkan device. Pin the Radeon with --device Vulkan0; picking the iGPU does not error, the numbers are just wrong.
Headless, Vulkan sees only llvmpipe until the account is in the render group: sudo usermod -aG render,video $USER, and SupplementaryGroups=render video in the systemd unit.
Deep prefills (about 120K tokens on the 35B) hung the GPU with amdgpu: ring comp_1.x.0 timeout. The compute-ring timeout defaults to 2 seconds. Raise it and reboot:

# /etc/modprobe.d/amdgpu-lockup.conf
options amdgpu lockup_timeout=10000,300000,10000,10000

Settings that matter

Expert placement. --n-cpu-moe N keeps the expert weights of the first N layers in RAM and puts everything else on the card. Start with N at the layer count and step down. The lowest value that loads is not the best one: when the card overcommits, RADV spills silently into system memory and decode collapses. Watch mem_info_gtt_used in sysfs; it should stay near 300 MB. At 262K with vision loaded, the 35B needs 38 and the flagship needs all 48.

Wall of small wooden apothecary drawers with one pulled open in window light — Many small choices, one drawer open at a time.

Threads. P-cores only. -t 8 -C 0x5555 for the 3B-active models, -t 16 -C 0xFFFF for the flagship. Letting llama.cpp use all 32 threads cut decode 20 to 47 percent; the E-cores drag the token step.

KV cache. q8_0 fits the 35B and Coder-Next at 262K. The flagship needs q4_0: with q8_0 it spills and decodes at 7 tok/s, with q4_0 it fits and decodes at 12.7. Needle retrieval passed with q4_0 at every depth tested.

Batch size. -ub 2048 -b 2048 more than doubles prefill with experts on the CPU (500 vs 213 tok/s on the 35B), since each expert is read once per batch. The bigger buffer costs VRAM, so it only fits at short context.

Results

Method: llama-bench for pp512 and tg128 on an empty context, then llama-server filled in steps to 262K with a needle near the top, plus smoke tests for coding, tool calls, reasoning and vision.

Generation speed against tokens already in context for the four models. Dashed segments span depths that were not measured. — Generation speed by context depth, llama-server on the box. The flagship stops at 64K because the longer fills were cut; dashed segments cross depths that were not measured.

Qwen3.8-Flash-Next (125B-A6B, UD-IQ4_XS, 94 GB, vision).

Empty context: 91 tok/s prefill, 13.0 generation. 11.8 at 32K, 5.7 at 64K, needle found.
Prefill drops to 21 to 25 tok/s past 32K; 32K to 64K took 21 minutes. The 131K and 262K fills were cut for time. The full window works for a conversation that grows; filling 262K cold extrapolates to over 4 hours.
Smoke: all four pass. Q3_K_XL was no faster than IQ4_XS.

Qwen3.6-35B-A3B (UD-Q4_K_XL, 23 GB, vision).

Empty context: 212 prefill, 27.6 generation. 32K: 172 and 22.0. 255K: 10.4 generation, needle found; the fill took 64 minutes.
Fastest of the CPU-offloaded models at depth. Smoke: all four pass.

Qwen3-Coder-Next (80B-A3B, UD-Q3_K_XL, 36 GB, text only).

Empty context: 114 prefill, 21.5 generation. 32K: 17.8. 128K: 12.4. 255K: 8.6, needle found at every depth; the full fill took about 40 minutes.
Smoke: tools and reasoning pass; coding passed after removing one placeholder import. No thinking block.

Qwen3.8-27B dense (UD-Q4_K_XL, 17.6 GB, vision).

146 prefill, 4.45 generation, 2.5 at 32K. Every token reads about 12 GB of weights from RAM, so 4.4 tok/s is the bandwidth limit.

Qwen3.5-4B (UD-Q4_K_XL, 3 GB, vision, all on the GPU).

1591 prefill, 67 generation. 52 at 32K, 31 at 131K, 21.6 at 262K with a q4_0 cache.
CPU-only for comparison: 12.8 tok/s. The GPU is worth 5x on a model that fits in it.

Multi-token prediction

4B, GPU only: 105 vs 61 tok/s at 2K (1.7x), 67 vs 49 at 32K, no gain at 131K.
35B: 93 percent acceptance and no speed gain, while prefill fell from 172 to about 65. Verifying drafted tokens touches more experts, and expert reads are the bottleneck.
Flagship: 5.5 vs 12.6 tok/s at 40K. The 4 GB head pushes the card into spill.

MTP helps only when the whole model fits on the GPU.

The server

llama-server \
  -m Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf \
  --mmproj mmproj-BF16.gguf \
  --device Vulkan0 -ngl 99 --n-cpu-moe 48 \
  -fa on -c 262144 -ctk q4_0 -ctv q4_0 \
  -t 16 -tb 16 -C 0xFFFF -b 2048 -ub 512 \
  -np 1 --jinja --metrics --alias qwen38 \
  --host 0.0.0.0 --port 8080

Runs as a systemd service. Point -m at the first shard and llama-server finds the rest. Expect about 12 tok/s generation and 45 tok/s prefill on short prompts, about 6 and 20 past 64K.

The ceiling

Active bytes per token times measured tok/s puts effective decode bandwidth at roughly 50 GB/s for the 35B, 40 for the flagship and 28 for Coder-Next, against the 59 GB/s the RAM can deliver. These are estimates, since part of each model sits on the GPU. The GPU doubled decode on the 35B over CPU-only (12.6 to 27.7); beyond holding attention and some experts, VRAM has little left to give on this box.

Aerial photograph of a reservoir backed up behind a small sluice gate with a thin stream escaping — A lot of capacity behind one narrow gate.

Estimated effective decode bandwidth per model against the 59 GB/s measured RAM read ceiling — Estimated effective decode bandwidth, from active bytes per token times measured tok/s. Part of each model's weights is on the GPU, so these are rough.