A concrete dam spillway at blue hour with a row of closed gates and one gate raised, white water pouring through

AbliteratedOutTheBox: A llama.cpp Fork That Abliterates Per Prompt and Leaves the Weights Alone

A look-ahead gate, two directions read from the prompt, and a second prefill. One recipe for 131 llama.cpp architectures, with stock output on prompts the gate leaves alone.

AbliteratedOutTheBox is a two-day-old fork of llama.cpp that removes a model's refusals at inference time, per prompt, with the weights on disk left as they are. On prompts the model would answer anyway, the output is bit-identical to stock llama.cpp. I have not run it yet; this is a read of the repo and README.

What abliteration usually means

The technique comes from a 2024 paper, Refusal in Language Models Is Mediated by a Single Direction (Arditi et al.): a model's decision to refuse lives along one direction in its activations. FailSpy, and then Maxime Labonne's post Uncensor any LLM with abliteration, popularized the recipe: find that direction, edit the weights to project it out, and save the result. I keep one such model on my laptop, a weight-baked abliterated Qwen3.8, next to the stock copy.

That approach has three costs. It changes each answer a little, including answers to prompts the model was fine with. It needs per-architecture surgery. And it leaves you with a second copy of the weights.

What this fork does instead

llama-abliterate handles a prompt in two passes.

  1. A look-ahead check generates up to 24 of the model's own tokens and measures the peak strength of the refusal direction, averaged across a band of middle layers.
  2. If that exceeds --ff-threshold (default 0.70), the tool reads two directions from the prompt itself, by forcing a few comply versus refuse openings and clean versus disclaimered openings and diffing the activations. It then re-runs the prompt with both removed from the middle layers, and generates.
  3. If the check does not fire, there is no second pass and the output matches the stock model.

The edit sits at llama.cpp's build_cvec layer-output site, the hook control vectors use. The README says 131 of 165 architectures share that site, so one recipe covers them with no per-model code. The other knobs are --edit-strength (default 2.0) and --disc-strength (default 0.4); the second removes safety disclaimers, and pushing it higher can degrade long outputs.

Pruning shears snipping one thin glowing cyan wire that runs along a dew-covered grapevine
One direction removed, the rest of the model left as it is. Illustration.

Self-reported numbers

These are the author's own measurements on Qwen3-8B, read with a hidden-state probe rather than string matching or a judge model.

The detail I keep thinking about

The README says the refusal decision is committed into the prompt's KV cache during prefill. Editing the generated tokens alone leaves refusal near 95%. Editing the prompt positions too brings it to single digits. That is why the tool needs a second prefill, and it is a tidy observation about where the model makes up its mind: before it has written a word.

A railway switch at dusk with the points set to divert one track, green signal lamps lit
The route is set during prefill, before the first token. Illustration.

Limits

It runs one sequence at a time, since the gate is per context. It is a command-line tool; I found no OpenAI-style server in the repo. A flagged prompt costs one extra prefill, small at 8B with a short prompt and real time at long context. The repo is two days old (37 stars, 2 forks, MIT), and llama.cpp arrived as one squashed commit, so the upstream base version is not visible in the history.

Where I land

What the design would buy me is one copy of the weights with stock behavior on ordinary prompts, instead of a second model I switch to on purpose. What I would want first is a server mode, and a run on a large MoE model at long context to see what the extra prefill costs there. The part I find elegant is the idea itself: leave the model alone until it is about to refuse, and intervene then.

Sources

Fringe Tech Local LLMs

Comments

// Comments are reviewed before appearing. No spam. No noise.