Closing the table — stock vs Uncensored Qwen3.8 on Terminal-Bench 89, pass@5 64 to 59
Round-5 (pass@5) results for the abliterated Uncensored Qwen3.8 Flash Next on Terminal-Bench 2.1, plus a series wrap-up: 59/89 vs stock 64/89, the 13 tasks that split them, speed, and harness failures
Summary
Part 5 ended with “5 vs 4, one round left.” I ran the Uncensored build’s (ARC4NUM abliterated 4bit) 31 pass@4 failures a fifth time under the same conditions as the stock model (timeout 1.0×, reasoning high, temperature 0.3, ctx 120K, terminus-2). Result: 1 recovered.
Bottom line: Uncensored pass@5 = 59/89 (66.3%), stock pass@5 = 64/89 (71.9%). Both sides now have the official 5 trials, so this table closes here.
| Attempt | Stock ddalcu | Uncensored ARC4NUM |
|---|---|---|
| 1 (pass@1) | 45 / 89 (50.6%) | 41 / 89 (46.1%) |
| 2 cumulative | 53 / 89 (59.6%) | — |
| 3 cumulative | 58 / 89 (65.2%) | 56 / 89 (62.9%, best-of-3) |
| 4 cumulative | 60 / 89 (67.4%) | 58 / 89 (65.2%) |
| 5 cumulative (pass@5) | 64 / 89 (71.9%) | 59 / 89 (66.3%) |
Uncensored’s increments: (best-of-3) → 2 → 1. Where stock flipped 4 tasks in its fifth round, Uncensored effectively saturated. The gap narrowed 8 → 6 → 5, but every point of that narrowing was Uncensored catching up on tasks stock had already passed. Of the 21 tasks stock never solved, Uncensored flipped zero in five tries.
What happened in round 5
31 tasks: 1 recovered, 7 FAIL, 22 TO, 1 ERR. 13h 24m, 24.7 min per task on average.
mteb-leaderboard (58.1 min, 97 turns, ctx 96K) — find the top model on the August-2025 MTEB leaderboard among those with results on all 28 tasks. Stock passed it in round 4 in 32.6 minutes; Uncensored had hit the 60-minute timeout four times. This trajectory snapshotted the embeddings-benchmark/results repo at the right commit, parsed every model’s result files, narrowed to the 8 models with all 28 tasks and ranked GritLM-7B (0.6408) first. The decisive part came next — it noted that “this matches the current leaderboard API’s meanTask to six decimals” as a cross-check of its own method, then still went looking for models with 28 files that were missing from its list, found two (multilingual-e5-small, all-MiniLM-L6-v2), traced them to a legacy file format, and only then submitted. The last 20 of 58 minutes were verification. The previous four attempts burned their time scraping the API.
That’s it. The other 30 failed, and 23 of those are tasks stock also failed five times, so the outcome was expected. What’s worth looking at is not why they lost but how.
Where they split (all 89, five rounds each)
| Group | Count | Tasks |
|---|---|---|
| Both pass | 55 | — |
| Both fail all rounds | 21 | chess-best-move, compile-compcert, dna-assembly, extract-moves-from-video, feal-linear-cryptanalysis, gcode-to-text, gpt2-codegolf, install-windows-3.11, make-doom-for-mips, make-mips-interpreter, path-tracing, path-tracing-reverse, protein-assembly, qemu-alpine-ssh, query-optimize, regex-chess, rstan-to-pystan, sanitize-git-repo, schemelike-metacircular-eval, video-processing, write-compressor |
| Stock only | 9 | circuit-fibsqrt, extract-elf, llm-inference-batching-scheduler, qemu-startup, raman-fitting, sparql-university, torch-pipeline-parallelism, train-fasttext, tune-mjcf |
| Uncensored only | 4 | feal-differential-cryptanalysis, filter-js-from-html, model-extraction-relu-logits, polyglot-rust-c |
Gap 5 = 9 − 4. The nine stock-only tasks come in two kinds.
- Four 15-minute tasks (qemu-startup, raman-fitting, torch-pipeline-parallelism, tune-mjcf). Stock cleared these narrowly, in 3–15 minutes (torch-pipeline via a file left on disk at timeout). Uncensored used the full 15 minutes all five times. raman-fitting round 5 is the type specimen — stock noticed on turn 3 that “the x axis is wavelength and the decimal separator is a comma” and solved it in 3 minutes; Uncensored again spent 20 turns re-fitting with different laser wavelengths and ended with “the x-axis interpretation is unclear.” In two of its five attempts it mentioned the correct interpretation in the trajectory and discarded it.
- Five precision tasks (extract-elf, llm-inference-batching-scheduler, sparql-university, train-fasttext, circuit-fibsqrt). Time left over, answer wrong. sparql-university was submitted in 3.6 minutes again; the grader listed rows with multi-country answers like ‘CH, ES, US’ and ‘GR, US’ as missing (the model returned ‘GR’ alone). train-fasttext, a 19-minute FAIL, was worse than before: the grader reported
/app/model.bin cannot be opened, i.e. the model was never saved to the required path. extract-elf, 1.6 min · 11 turns, submitted with the same posture that produced “0.00% match” in round 4.
Conversely the four Uncensored-only tasks stayed exactly four across all five rounds. Differential cryptanalysis, a JS filter, ReLU-logit extraction, a Rust–C polyglot — if there’s a common thread it’s “construction problems with a single right answer,” but four samples don’t support a claim about abliteration. Likewise, a gap of 5 sits inside the per-round variance of an 89-task benchmark (stock round 5: +4, Uncensored round 5: +1). The honest conclusion is not “stock is better” but “abliteration buys nothing on this benchmark.”
Anatomy of 30 failures — this time the harness and the server star
① The fallback loop, third appearance. video-processing, 60 min · 1,127 turns · ctx 130K. Longer than round 4 (822 turns). After crossing 100K and hitting the server’s 131K limit, the rest of the hour filled with “Technical difficulties” retries at several per second. This one task produced 2,800 400-overflow responses, 99.7% of the round’s 2,809. After stock’s path-tracing-reverse and Uncensored’s round-4 video-processing and circuit-fibsqrt this is the fourth instance, so it reproduces. The condition is the same: one large tool output that jumps the 11K margin between max_input_tokens 120K and the server’s 131K in a single turn, so the handoff-summary request itself gets a 400.
There is a control. regex-chess, 60 min · 182 turns, got its handoff on turn 136 at ctx 117.5K and reset to 3.1K, then ran 46 more normal turns (it didn’t solve the task — it conceded on the last turn that “the rules must generalize to any position”). Same harness, same model. Whether it stops inside the margin or not is one tool output’s difference.
② The server refused a request — a new ERR type. schemelike-metacircular-eval, 39.2 min · 297 turns · ctx 119K. As in round 4 it grew its context through a repetition loop, but this time it died before the 131K wall. Server log:
prompt 119609 tokens needs ~11908MB (KV+working+margin), ~10653MB available — rejectingmlx-serve’s memory admission couldn’t reserve room for the KV cache and rejected the request; harbor recorded a litellm BadRequestError as ERR. Why a 128 GB machine was short 10 GB is answered under memory below. The trajectory was headed for a timeout anyway, so the score is unaffected, but it’s the first time server state leaked into a grading result.
③ The three-turn repetition loop, to the end. tune-mjcf, 15 min · 35 turns. Its last three turns are the same analysis to the character (“mass-matrix minimum eigenvalue 7.5e-9, stiff mode ω≈3310, period 1.9 ms ≈ timestep”). The diagnosis is right — this is the task where round 4 got as far as “switch to the PGS solver and it passes.” While it re-emitted the diagnosis it never wrote the file. The same disease showed in llm-inference-batching-scheduler (69K), make-mips-interpreter (106K) and dna-assembly (118K).
④ The rest are known patterns. Premature confident submissions (sanitize-git-repo 1.9 min · 9 turns, install-windows-3.11 4.0 min, query-optimize 7.1 min), compile-compcert spending its 40 minutes on an environment build (a Coq source build, again), and ten 15-minute tasks all timing out.
Speed was a wash
The series looked only at accuracy, so here is speed to close. Both models ran on the same mlx-serve 26.8.11, MTP on, 8-bit KV, M5 Max 128 GB.
Server throughput (request log, median by prompt length, tok/s):
| Prompt | Stock prefill / decode | Uncensored prefill / decode |
|---|---|---|
| < 8K | 162 / 73.3 | 234 / 57.8 |
| 8–32K | 350 / 65.5 | 391 / 61.9 |
| 32–64K | 378 / 50.8 | 324 / 53.0 |
| 64K+ | 173 / 53.3 | 248 / 44.0 |
Stock decodes 10–25% faster on short prompts. That the larger mixed-4-8bit isn’t slower says the bottleneck at this size is KV/attention, not weight reads. Caveats: the Uncensored numbers include round 5’s swap stretch (decode at 20–27 tok/s), and the stock server log is a 1,675-request sample, so this isn’t a precise comparison.
Task completion speed (all rounds, stock 227 runs · Uncensored 240 runs):
| Stock | Uncensored | |
|---|---|---|
| Minutes per turn (median) | 0.38 | 0.39 |
| Output tokens per turn (median) | 345 | 334 |
| 55 tasks both passed, fastest PASS (median) | 4.6 min / 17 turns | 5.4 min / 16 turns |
| Sum of fastest PASS over those 55 | 562 min / 1,372 turns | 527 min / 1,309 turns |
| Tasks where it was faster | 30 | 25 |
0.38 vs 0.39 minutes per turn — in an agent loop most of the time is command execution inside the container and prefill, and the decode difference disappears into it. Per-task wins split 30:25, and Uncensored’s 6% lower total isn’t skill, it’s search-path luck on a few tasks (distribution-search 19→4 min, custom-memory-heap-crash 16→5 min). Going the other way, stock was much faster on mteb-leaderboard (33→58 min) and large-scale-text-editing (9→20 min). Speed can’t tell these two apart. The only difference is accuracy, 64 to 59.
Ops notes
Zero wedge restarts in 13h 24m; the server ran 28 hours straight including round 4. This time it’s all about memory.
- A resident 100K context puts this machine into swap. The chain: 100K+ prompt → mlx-serve’s hot cache blows through its 8 GB cap to 13.6 GB (it evicts in whole entries and there are only two, so the cap can’t hold) → macOS pages out mlx-serve itself (RSS 14 → 2 GB) → the swap file grows 41 → 47 GB → decode drops 55 → 20–27 tok/s. schemelike’s admission rejection (② above) came out of this window. It recovers within 10 minutes of the task ending. Next time with this setup, start with
--prefix-cache-mem 4GB. - 2,809 400-overflows, 2,800 of them video-processing. No server load (under a second per request); the cost was about 30 minutes of bench time.
- 11 admission rejections, all in the swap window. That the server refused requests instead of dying is mlx-serve doing the right thing — four days ago this would have been a wedge.
- Round 4’s “one memory warning” (free 0.3 GB · 16% headroom) didn’t recur. The rule (free under 1 GB and headroom under 20%) never tripped both at once; free near 1 GB happened several times.
Closing the series
A week ago this started as “how far can Qwen3.8 Flash Next go on an M5 Max,” and ended up as Terminal-Bench 89 tasks × two models × five rounds = 467 runs. Three things remain.
- Abliteration isn’t free on this benchmark. pass@1 −4, pass@5 −5. The gap held in single digits through all five rounds, and the Uncensored-only set was four tasks from start to finish. “Removing refusals makes a better agent” does not hold on these 89 tasks, at least.
- pass@k measures variance more than models. Stock 45 → 64, Uncensored 41 → 59: +19 and +18 from five retries. The same task solves in 3 minutes one run and times out at 30 the next. Comparing two models on single-run numbers is playing inside that variance.
- A lot of the failures belong to the harness. Four fallback loops (30 minutes of nothing each), an 11K max_input_tokens margin, no repetition-loop detection, and the “install the environment vs. write the answer first” time split on 15-minute tasks. Fixing those four will raise pass@1 more than swapping models.
One line: five rounds each, 64 to 59. Same speed; a gap of 5 means abliteration bought nothing; and on both sides, 21 tasks were blocked by the harness and the clock, not the model.