Qwen3.6-27B Scores Higher on BFCL, Ties on My Workload, and Runs 4.9x Slower

I published benchmark numbers for local tool-calling models on a single 32GB card. Three claims came back at the post. That it read as generated filler. That the model I picked was outdated. That Qwen3.6-27B would beat it.

Two of those are testable, so I tested them. The first one is answerable only by writing less and measuring more, which is what this is.

What I left out of the original benchmark

Qwen3.6-27B was released on 2026-04-21. My benchmark ran three months after that and did not include it. I had ruled it out early, on the assumption that a dense 27B would be too slow to sit in front of an interactive agent, and I never went back to check whether that was true.

The measurements below say the assumption was correct. Qwen3.6-27B takes 4.9x longer per query on the real workload. That does not make skipping the test defensible, and the reason it does not is the whole point of this post. An untested assumption that happens to be right is still an untested assumption, and I did not know it was right until today. I reasoned from architecture instead of measuring, on the one model that most deserved the measurement.

A second correction predates all of this. The original post said GLM was "the most accurate of the three." At 84%, 82% and 80% over n=500 per model, that sentence was not supportable by the data under it. I corrected the published post on my own initiative before running any of the work below. I raise it because reading a spread as a ranking is the exact failure this post is about, and I committed it first.

What fits on one 32GB card

GLM-4.7-Flash was released 2026-01-19. It is roughly a 31B MoE with about 3B active, MIT licensed, run here as the unsloth Q4_K_M GGUF at 18.3 GB on disk, built from the zai-org base weights. Qwen3.6-27B was released 2026-04-21. Dense 27B, Apache 2.0, unsloth Q4_K_M at 16.8 GB.

Newer GLMs exist. GLM-5 landed 2026-02-11, GLM-5.1 on 2026-04-03, GLM-5.2 on 2026-06-16. None of them fit. GLM-5.2 is 78 layers with 256 experts and 8 active, which is a datacenter shape rather than a 32GB shape. zai-org has shipped no newer Flash-class model since January. Within the one-card constraint, GLM-4.7-Flash is still the current small option. The model that actually contests that slot is Qwen3.6-27B, which is why leaving it out was the real error.

Full BFCL v3, every case, both models

I rebuilt the AST harness from scratch and ran every case in five BFCL v3 categories rather than sampling 100 from each: simple 400, multiple 200, parallel 200, irrelevance 240, live_simple 258, for n=1298 per model. Identical harness, identical quant (Q4_K_M), identical server (llama.cpp b10075), identical flags, temp 0.2, max_tokens 2048, same box, same session. GLM was re-run as a control on the new harness so the comparison is like for like.

Qwen3.6-27B scored higher in four categories and tied the fifth. On the benchmark I chose, the model I did not include scored higher than the model I did, and I am not going to soften that. Scored higher is the claim the table supports. Whether that difference is real is a separate question and it gets its own section.

The statistics, which I also got wrong the first time

Originally I ran a two-proportion z-test and got p=0.318. That was the wrong test. Both models answered the same 1298 cases, so the data is paired and McNemar's test on the discordant pairs is the correct one.

GLM right with Qwen wrong: 49 cases. Qwen right with GLM wrong: 66 cases. McNemar exact two-sided p = 0.1354. The two models agreed on 91.1% of cases, with 1074 both correct, 109 both wrong and 115 discordant. Qwen took the larger share of discordant pairs in four of the five categories, though those five splits are not independent tests and none of them is individually significant, so read that as texture rather than as corroboration.

Using the correct paired test roughly doubles the strength of evidence in Qwen's favour compared to the wrong test I ran first. It still does not clear 0.05.

The power calculation explains why. At n=1298 per arm, 80% power and alpha 0.05, the minimum detectable effect is 3.70pp. The observed difference is 1.31pp. Running every case in five categories was not enough, and no amount of care in the harness fixes that. Doubling the case count would not have been enough either, because the detectable effect only falls with the square root of n.

I want to be precise about what that means, because the sloppy phrasings available here all happen to flatter me. I am not going to claim the two models perform the same, and I am not going to claim the question is unanswerable in principle. Failing to reject a null is not evidence of equivalence. The honest statement is narrower: the evidence leans toward Qwen3.6-27B being slightly more accurate, my study could not resolve whether that lean is real, and being unable to demonstrate a difference is not the same as demonstrating there is not one.

The in-domain test I should have built first

The agent this was all for has exactly one tool, an http_request GET, plus a recipe table in its system prompt mapping question types to exact URLs against a metrics database and a weather service. Its real job is to take a question about the farm and emit the correct URL, or correctly emit nothing.

I built 59 cases directly from the production recipe table: weather 18, cistern 8, outside temperature 6, heat pump 5, alarms 4, multi-source 8, and 10 cases where no tool call is warranted at all. Same system prompt as production.

GLM-4.7-Flash: 58/59, 98.31%. Qwen3.6-27B: 58/59, 98.31%.

Identical overall, identical per category, and they missed the same single case with the same wrong answer. Both read "is anything alarming" as a general status check and fired all five status URLs instead of the alarm query alone. On reflection that is probably my test case being too strict about an ambiguous phrasing rather than either model getting it wrong.

Both models land near 98% here against 86.52% and 87.83% on BFCL. The recipe table constrains the task enormously, and that gap is the whole argument about why a public leaderboard is not a workload. BFCL is not my workload. That is why I built the second test.

The sample is the obvious weakness. 59 cases can detect large differences and nothing else. I am not claiming these two models are equal to within a point. I am claiming that at the resolution this test has, nothing separates them, and that it can tell me neither model is unfit for the job, which is what it did.

Speed, decomposed

My original headline of 8.7x was wrong in two directions at once. On the real workload:

2.65 multiplied by 1.85 gives 4.91. Roughly a third of the wall-clock gap comes from Qwen3.6 emitting more tokens rather than computing more slowly, and those are two separate claims that my single number had fused together. A dense 27B on one card generates slower per token than a 31B MoE with about 3B active, and it also emits more tokens per answer. Both are real, and they multiply.

The ratio is also workload dependent: 8.7x on BFCL, 4.9x on the real job. Quoting one of those without the workload attached says very little. The 201.0 tok/s figure independently confirms the roughly 198 tok/s I reported the first time. In practice, per-query latency is 0.87s against 4.27s.

What I checked before believing any of it

Zero request errors across all 2596 BFCL cases and all 118 in-domain cases. On categories that require a call, the zero-call rate was 1.80% for GLM and 1.89% for Qwen, near enough identical that neither score is being depressed by one model running into the token budget. That check catches total silence only. It does not fully rule out truncation mid-reasoning, and I am not going to claim more from it than that.

A correction to the original post

The first post said "--jinja is not optional, this is the flag people miss." That is stale. On b10075, llama-server --help lists --jinja and --no-jinja with the default already enabled, so the flag is redundant on current builds. I have marked it as a correction in the published post. The underlying failure still reproduces with --no-jinja or an older binary: without the model's chat template, tool calls come back as raw text and nothing parses them.

What this work cannot support

My grader is mine. I wrote it from scratch and I have not validated it against the official BFCL scorer, so every percentage and every p-value above is conditional on my implementation being right. That is the largest uncontrolled variable in the whole exercise.

I ran plain Q4_K_M for both models, while unsloth's own headline recommendation for that repo is the UD dynamic quant, UD-Q4_K_XL, which specifically targets instruction following. Qwen3.6 might do better under it. I did not test that, and it is the first thing I would run next.

I excluded BFCL's multi-turn and agentic categories, which are arguably the closest thing in the suite to a real agent workload.

I am also retracting something. I had claimed that GLM moving from 84% to 86.52% was pure sampling noise. It is not clean, because I rebuilt the harness between those two runs, with a new grader and new prompt construction. Those two numbers are confounded and the claim is withdrawn. The only comparison I stand behind is the one where both models ran on the same harness in the same session.

The speed figures are a single run each with no repeats. They are also warm-path numbers. The harness fired its cases back to back, so the weights stayed resident the whole time. On an idle box this driver pages the model out to system memory between requests, so the first query after a quiet stretch pays a load cost that none of these figures include. That penalty applies to both models and does not move the comparison, but 0.87 seconds is the warm number, not the number someone gets on their first message of the morning.

On benchmaxxing

Someone will read this as choosing a benchmark that flatters my pick, so it is worth answering directly.

Benchmaxxing means selecting a metric you already win. My headline accuracy metric shows my model scoring lower, in four categories out of five. Nobody picks a benchmark hoping to lose it.

More to the point, the decisive number in this post is not a benchmark score. It is a stopwatch reading of 0.87s against 4.27s per query, which anyone can reproduce in one command without trusting a line of my grader code.

And the conclusion does not need the benchmark at all. A human waiting on a chat reply notices four seconds and does not notice one. That is a property of the workload, not of any leaderboard. The benchmark's job here was to eliminate accuracy as the deciding variable, and once it is eliminated, cost is what remains.

What actually decides it

Model discussion online runs heavily on people repeating the headline from the most recent announcement they read, without asking whether that model is the right tool for the job actually in front of them. The data above is the strongest rebuttal of that habit I can offer, because it uses the model a newness-first argument would pick.

Qwen3.6-27B is newer. It scored higher on the public-style benchmark. It is a genuinely excellent model and I would recommend it without hesitation for plenty of work. It is still the wrong choice for this particular job, because on the job itself it is exactly as accurate and it costs five times the latency per query. This is why actual use cases and real world tests matter.

Which model is best is a malformed question until a workload is attached to it. On BFCL the newer model scored higher and my study could not resolve whether that difference is real, though the lean is there. On the actual work the two scored the same, 58 out of 59 each, same miss, same wrong answer. The only difference that survives contact with the real job is cost. The agent keeps GLM-4.7-Flash, and it keeps it for a reason that has nothing to do with which model is newer or which one scored higher on somebody else's leaderboard.

Opinions about which model is better are cheap and there is an unlimited supply of them. 2596 graded BFCL cases and 118 in-domain cases against a rebuilt harness are not cheap. Weigh the method rather than the confidence of whoever is asserting, mine included.