Does abliteration boost coding? Uncensored vs stock Qwen3.8 on Terminal-Bench 89
A same-conditions single-pass A/B between an abliterated (Uncensored) model and the stock mixed-4-8bit build across all 89 Terminal-Bench 2.1 tasks
TL;DR
In the previous post I ran all 89 Terminal-Bench 2.1 tasks with an abliterated (Uncensored) Qwen3.8 Flash Next (ARC4NUM, 4-bit). This time I ran the non-abliterated stock model ddalcu/Qwen3.8-Flash-Next-MLX-Serve-mixed-4-8bit under identical conditions, once per task, head to head.
Bottom line: single-pass, stock scored 45/89, Uncensored 41/89. Abliteration did not improve coding/agentic performance — it was 4 tasks behind.
What is abliteration? A post-hoc edit that removes the model’s refusal-direction vector so it “answers anything.” It trades safety alignment for the hope of more capability; here the result runs opposite to that hope. (This is a research/comparison run; abliterated models must be used with that property clearly stated.)
Setup (identical for both)
| Item | Value |
|---|---|
| Benchmark | Terminal-Bench 2.1 (89 tasks), agent terminus-2 |
| Server | mlx-serve 26.8.11 headless, --mtp --ctx-size 131072 --kv-quant 8 --max-concurrent 2 |
| Sampling | temperature 0.3, reasoning_effort high |
| Context | max_input 120K / max_output 16K, agent-timeout ×1.0 |
| Hardware | Apple Silicon, unified-memory wired cap ~110GB, model resident 70.1GB |
| Attempts | 1 per task (pass@1) |
Results
| Run | Passed | Rate | Note |
|---|---|---|---|
| Stock ddalcu (single) | 45 / 89 | 50.6% | this post |
| Uncensored ARC4NUM (single) | 41 / 89 | 46.1% | same-conditions 1:1 |
| Uncensored ARC4NUM (best-of 3) | 56 / 89 | 62.9% | 3× the attempts |
Single-pass is the fair 1:1. Best-of pooled “passed at least once” across the main run plus two reruns (3 tries), so it isn’t a like-for-like number.
Where they diverged
The 11 tasks stock won (Uncensored-single failed → stock passed) were mostly the grind-it-out-debugging kind: custom-memory-heap-crash (gdb blocked → valgrind → core parsing → source reasoning), llm-inference-batching-scheduler, mteb-leaderboard (rode out a GitHub rate limit), pytorch-model-recovery, reshard-c4-data, qemu-startup.
The 7 stock lost were different in character: time/network limits (caffe-cifar-10’s toronto.edu download, break-filter-js-from-html’s 20-min timeout), or premature submission — configure-git-webserver, filter-js-from-html, financial-document-processor, overfull-hbox all declared “done/verified” and submitted before meeting the grader. That over-confidence is stock’s signature weakness here.
Ops lessons (infra traps)
Over a 24-hour run, operational stability shaped the score as much as raw ability.
- False-wedge restart. Tasks past 90K context took ~56s per request, but the watchdog health-ping timed out at 45s; two consecutive misses made it kill a perfectly healthy server as a “wedge.” The restart wiped the cache (70GB model + 60GB disk prefix-cache), costing the in-flight task 5+ minutes of cold prefill. Raising the timeout to 120s fixed it. (That task, fix-ocaml-gc, still survived as a PASS because its build finished.)
- Disk saturation. Per-task Docker images piled up until the Colima VM disk (59GB) hit 100% and apt died inside a container. The check script now auto-deletes images of scored tasks.
- The 131K context wall. Some tasks ballooned context to 131K via repetition loops, and the server correctly rejected them with 400s. The model spec supports 256K, but with a 70GB model already resident this hardware’s KV cache would exceed GPU memory (circuit-fibsqrt OOM’d at 122K), so ~128K was the practical ceiling.
Rerun (pass@2): the 44 failures, once more
I ran all 44 single-pass failures again under identical conditions. Eight recovered, bringing pass@2 = 53/89 (59.6%).
| Attempt | Passed | Rate |
|---|---|---|
| pass@1 (single) | 45 / 89 | 50.6% |
| pass@2 (two tries) | 53 / 89 | 59.6% |
The 8 recovered: circuit-fibsqrt (a GPU-OOM ERR on run 1, but on run 2 it caught the off-by-one and passed), code-from-image, build-cython-ext, break-filter-js-from-html, password-recovery, tune-mjcf, and crucially the two run-1 “over-confident” FAILs, configure-git-webserver and financial-document-processor, which passed cleanly on the second try. In other words, stock’s signature weakness (premature submission) is largely recoverable with a retry.
Conversely, filter-js-from-html failed both times after claiming it had “verified” its solution, and the feal/regex-chess/path-tracing families failed twice to time and context limits.
Caveats and next steps
This 45 vs 41 is a single-trial point estimate. Terminal-Bench’s official submission uses multiple trials per task (currently a 5-run minimum) for Accuracy and pass@k, so treat these as a same-conditions snapshot, not a leaderboard figure.
This pass@2 is still only two trials. The remaining 36 failures are now in a third pass (pass@3). A fair Accuracy would need 5 runs on both sides. For reference, Uncensored’s best-of-3 on the same 89 was 56 — and stock already reaches 53 at pass@2, right at that threshold.
One-liner: on this workload, abliteration added no capability — it only subtracted safety.