GLM-4.7-Flash on One Consumer GPU: Why It Runs My Homelab Agent

I run GLM-4.7-Flash, a ~31B Mixture-of-Experts reasoning model from Z.ai, as the brain of a small read-only homelab and farm monitoring agent. It serves at 4-bit from a single consumer 32GB GPU at roughly 198 tokens per second, and on the real Berkeley Function-Calling Leaderboard dataset it scored 84%, with a dense 32B at 82% and the model it replaced at 80%. That spread is inside my sample's noise floor, so the case for it rests on speed and on two specific categories rather than on overall accuracy. This post covers what the model is, what it is good for, why the model I started with had to go, the three-way benchmark, and two lessons about evaluating models before deploying them.

What the model is

GLM-4.7-Flash is MIT licensed, so there are no usage constraints on self-hosting it for anything. Architecturally it is a Mixture-of-Experts model with about 31B total parameters but only about 3B active per token. That sparsity is the source of the speed: it carries the knowledge of a 30B-class model while paying the per-token compute of a small one.

It is also a reasoning model. It thinks before it answers. On tool-calling requests it barely reasons and stays fast. On open-ended chat it reasons heavily and needs a generous token budget to finish a thought, which matters for deployment.

I run it at 4-bit (Q4_K_M GGUF, about 17GB) served by llama.cpp on a single RTX 5090-class card. Generation runs around 198 tokens per second. The published tau2-Bench score is 79.5, a multi-turn tool-use-with-policy benchmark. I did not take that number on faith for the 4-bit build; I re-ran the benchmark on my own quant.

What it does all day

The agent's entire workload is agentic tool calling. It answers questions over a chat channel by querying a metrics database and a weather service: pick the right tool, build the right query, read the result, report the number, and know when not to call a tool at all.

I reach it over Signal, on a free Google Voice number, with signal-cli wired up from scratch so the whole message path stays mine. That build is its own post.

The multi-source case is where the model earns its keep. Asked "is everything ok," it pulls weather alerts, equipment alarms, tank level, temperatures across buildings, and a sensor battery, and folds them into one short status where every number matches the live data. That is five tool calls, correctly chosen and correctly synthesized, from a model running on one consumer card.

For this workload the requirements are specific: local, fast, accurate at function calling, and safe in the sense of not inventing tool calls when none are warranted. GLM-4.7-Flash is the best fit I have measured.

Why the model I started with had to go

The agent originally ran on a 30B coding model. It was the fastest of everything I tested, and it still had to go, because of how it failed.

It did not fail by erroring out. It failed by guessing. Asked for the heat pump water temperatures, it returned two confident, plausible numbers and had made zero tool calls to get them. The real values were roughly thirty degrees away from what it reported. Asked for a tank level, it once produced a number below my low-water warning threshold that appeared in no query anywhere, which is a false alarm on a system I actually act on. And when I pushed back and said a number looked wrong, it defended the number instead of re-running the query.

That is the worst possible failure mode for a monitoring agent, and it is worse than the model simply falling over. Misinformation is worse than no information. A failed tool call is loud: I see it, I go look at the dashboard myself, no harm done. A fabricated reading is silent and it looks exactly like a real one. It gets believed, and on a farm it gets acted on.

So the property I care most about is not raw capability. It is whether the model reaches for its tools instead of its priors, and whether it will say it does not know. That reframed the whole evaluation, and it is why one specific benchmark category ended up mattering more than the headline score.

The benchmark: three models, same quant, same server

Published leaderboard numbers are measured on full-precision weights, and the 4-bit quant on your hardware is a different artifact that can score differently. So I pulled the real Berkeley Function-Calling Leaderboard (BFCL) v3 dataset, graded it with BFCL's own AST methodology (function-name match, every parameter value inside its acceptable list, no hallucinated parameters, and the irrelevance category passes only when the model correctly makes no call), and ran it against three models at the same 4-bit quant on the same server. Sample of 100 cases per category, temperature 0.2.

Three things stand out.

Accuracy, and what it does not show. GLM-4.7-Flash scored highest at 84%, but I ran a two-proportion test on the gaps and none of them clears the bar. 84 against the dense 32B's 82 is p=0.40. 84 against the coding model's 80 is p=0.10. The 95% interval on that 84% runs from roughly 81 to 87 and overlaps all three models, so at 100 cases per category I cannot claim GLM is the most accurate of the three. What the numbers do support is that the dense 32B bought nothing measurable for its 3-to-4x slowdown. It activates all of its parameters per token while the two MoE models activate a fraction, and that extra compute did not buy extra correctness.

The live-simple gap. The coding model was fastest but least accurate, and its weakest category by far was live-simple, at 68% against GLM's 82%. Live-simple uses real-world user phrasing, the messy way people actually ask questions, which is exactly what a chat-facing agent gets all day. A coding model doing conversational operations work shows its seams there.

Irrelevance. This category measures whether a model correctly makes no call when none is warranted, and after the guessing problem above it is the number I read first. It is the closest thing BFCL has to a "will it invent something" score. GLM measured highest here at 88%.

Two lessons from the evaluation

A homegrown harness scored the right model as a reject. Before running BFCL, I evaluated GLM with an eight-case tool-calling smoke test written by hand around the agent's actual tools. It scored 62%, clearly below the coding model, and read literally that says reject it. The real BFCL data put the same model at 84%, more than twenty points higher. The harness was too small and two of its cases tested prompt wording more than model capability. A homegrown smoke test is fine for catching regressions. It should never carry a keep-or-kill decision. Use recognized benchmark data with published answer keys.

Benchmark the artifact you will actually deploy. The 79.5 tau2 number is a full-precision score; my 84% is a 4-bit number on my card and my serving stack, and it is the one I can act on. The same verify-everything rule applied to the vendor's recommended sampling settings for tool calling: measured against near-greedy low temperature on the full BFCL sample, they were a wash, 83% versus 84%, every category within a few points. Near-greedy is fine for structured tool output and more deterministic. Recommendations are hypotheses. Measure them.

How I run it

Two pieces: a GGUF of the model, and llama.cpp's server.

The model. I use Unsloth's GGUF build, unsloth/GLM-4.7-Flash-GGUF, specifically the GLM-4.7-Flash-Q4_K_M.gguf file, about 17GB on disk. The base weights are zai-org/GLM-4.7-Flash, MIT licensed, and the quant carries the same license. On a 32GB card, 4-bit leaves roughly 13GB for context and overhead, which is exactly why it is the right call here. Heavier quants exist and they do not leave enough room for a useful context window on a single consumer card.

The server. llama.cpp, using a prebuilt CUDA binary rather than building it from source. Building it yourself on a current distro is its own adventure, and that is the other post.

llama-server \
  --model GLM-4.7-Flash-Q4_K_M.gguf \
  --host 0.0.0.0 --port 8000 \
  -ngl 999 \
  --jinja \
  --ctx-size 32768 \
  --metrics \
  --api-key YOUR_KEY

The flags that actually matter:

--jinja, with a correction. I first wrote that this flag is mandatory. On the build I run it is not. llama-server --help reports --jinja, --no-jinja ... (default: enabled), so on current llama.cpp passing it is redundant. It was required on older builds, and the failure it prevents is still worth recognising because --no-jinja or an older binary reproduces it: without the model's chat template, tool calls come back as raw text that nothing parses and the agent silently does nothing. If you see that symptom, look at template handling first.

-ngl 999 offloads every layer to the GPU. With a 17GB model on a 32GB card there is no reason to leave layers on the CPU, and any that stay there cost a lot of speed.

--ctx-size 32768 sets the context window. The model supports far more, around 200k, but context costs VRAM, and on one card you are trading context length against headroom. 32k is generous for an agent whose prompt is a system prompt plus a few tool results. If you run out of memory, this is the first knob to turn down. If your prompts are larger, raise it and watch VRAM.

--metrics exposes a Prometheus endpoint on the same port, so serving throughput and queue depth land in existing monitoring with no separate exporter. It requires the API key, so the scrape config needs the bearer token.

--api-key because the port is open on the LAN.

Sampling: near-greedy. I run temperature 0.2 for tool calling. I measured the vendor's recommended tool-calling settings, a higher temperature plus top-p and min-p adjustments, against near-greedy across the full BFCL sample: 83% versus 84%, a wash. Low temperature is fine here and more deterministic, which is what structured output wants.

Token budget, the gotcha that will cost you an hour. Because this model reasons before answering, a tight max_tokens cuts it off mid-thought and you get an empty response with no error. On tool calls it barely reasons, so this never bites there. On open-ended chat it will spend several hundred tokens thinking, so give conversational requests 2000 or more. An empty reply from this model usually means the budget, not a failure.

Run it under a supervisor. Whatever your platform's service manager is, use it, with restart-on-failure and start-at-boot. A model server that dies with your terminal session is not infrastructure.

Caveats

The reasoning behavior cuts both ways. On tool calls the model thinks briefly and stays fast, which is the common path for this agent. On open-ended chat it reasons at length, and if the token budget is tight it stops mid-thought. Give it headroom for conversational responses or cap the workload to structured tasks.

Guessing is a spectrum, not a solved problem. GLM leads the irrelevance category, it does not ace it. I still write the agent's prompt to hand it pre-built queries rather than let it improvise them, because every model I tested will invent a plausible value when the prompt leaves room for one.

Where it lands

GLM-4.7-Flash sits at the top of a statistically tied group on function-calling accuracy, and it gets there at a fraction of the dense model's cost, with the best measured score on the do-nothing-when-appropriate axis that matters most for a read-only monitoring agent, under a license with no strings. For local agentic tool calling on a single consumer GPU, it is the model I would recommend testing first, with the emphasis on testing: run the real cases on your own quant, because your hardware and serving stack are part of the artifact you are scoring.

Getting it to serve at all required working around a serving stack that refused connections for this one model, which I will cover in a separate post.