Still ahead after three tries: stock Qwen3.8 pass@3 58/89 vs Uncensored best-of-3 56
Round-3 (pass@3) Terminal-Bench 2.1 results for stock Qwen3.8 Flash Next mixed-4-8bit, a like-for-like three-trial comparison with the abliterated build, and why 31 tasks still fail
TL;DR
In the previous post the stock ddalcu/Qwen3.8-Flash-Next-MLX-Serve-mixed-4-8bit scored 45/89 single-pass and 53/89 cumulative after a second try. Round 3 — the remaining 36 failures, rerun once more under exactly the same conditions — is done.
Result: pass@3 = 58/89 (65.2%). Last time I set the Uncensored (ARC4NUM, abliterated) best-of-3 of 56/89 aside as “not comparable, different trial count”. Now the trial counts match — and the stock model is ahead.
| Trials | Stock ddalcu | Uncensored ARC4NUM |
|---|---|---|
| 1 (pass@1) | 45 / 89 (50.6%) | 41 / 89 (46.1%) |
| 2 cumulative | 53 / 89 (59.6%) | — |
| 3 cumulative (pass@3 / best-of-3) | 58 / 89 (65.2%) | 56 / 89 (62.9%) |
+4 single-pass, +2 at three trials. The gap narrowed; the direction did not change. Abliteration still added no capability.
What happened in round 3
36 tasks: 5 recovered, 10 FAIL, 21 TO (timeout). 14 h 49 min wall-clock, 23 min per task on average.
The five recoveries have distinct stories.
- sparql-university (3.5 min, 15 turns) — rounds 1 and 2 wrote the query and submitted immediately; both failed. Round 3 added a step: “let me actually run it and check the output.” Passed. The Uncensored build failed this task all three times, every time by submitting unverified.
- regex-log (2.7 min, 14 turns) — two FAILs, then exact on the third. Short tasks like this are pure sampling luck.
- count-dataset-tokens (13.5 min) — TO, then FAIL, then a clean run that avoided the tokenizer bottleneck.
- mcmc-sampling-stan (full 30 min) — the artifact submitted just before the deadline passed grading. Logged as a timeout, but reward > 0, so it counts.
- caffe-cifar-10 (38 min, 81 turns) — twice killed by the toronto.edu download stall noted last time. This time the download finished and training completed. Network luck.
Where they diverge (three trials each, all 89 tasks)
| Count | Tasks | |
|---|---|---|
| Both pass | 50 | — |
| Both fail ×3 | 25 | path-tracing family, regex-chess, feal-linear, make-*-mips, install-windows-3.11, compile-compcert, … |
| Stock only | 8 | circuit-fibsqrt, custom-memory-heap-crash, dna-insert, llm-inference-batching-scheduler, mteb-leaderboard, qemu-startup, sparql-university, tune-mjcf |
| Uncensored only | 6 | feal-differential-cryptanalysis, filter-js-from-html, model-extraction-relu-logits, overfull-hbox, polyglot-rust-c, winning-avg-corewars |
The eight stock-only wins match the earlier observation — tasks that yield to persistent debugging (route around a stuck gdb via valgrind, ride out a rate limit, add a verification step). Of the six Uncensored-only wins, three (filter-js-from-html, overfull-hbox, model-extraction) are tasks the stock model lost all three times by declaring itself done and submitting early. Retries partly compensate for that “premature confidence” weakness (sparql-university, configure-git-webserver), but it hasn’t gone away.
Anatomy of the 31 that still fail
Three buckets.
① Long-horizon tasks hitting the context wall (most of the 21 TOs). Six tasks peaked above 100K context: path-tracing-reverse 129K, regex-chess 122K (322 turns!), video-processing 122K, winning-avg-corewars 112K, make-mips-interpreter 108K, schemelike-metacircular-eval 101K. 131K is the physical ceiling on this hardware, and near it a single request takes 20–56 s, so the remaining budget evaporates. terminus-2’s context summarization did kick in around 122K (regex-chess 122K→44K, video-processing 122K→33K), so there was exactly one 400 overflow error in the whole round.
② Repetition loops. winning-avg-corewars burned the last 45 of its 60 minutes emitting a byte-identical message every turn (“I need to break out of this loop… the scanner t3 gives stone 69, vampire 88…”). The model knows it is looping and still cannot issue a new command. The Uncensored build passed this task on its 1.0× rerun.
③ Fast premature-submission FAILs. overfull-hbox 3.3 min, sanitize-git-repo 3.5 min, model-extraction-relu-logits 4.5 min, query-optimize 5.6 min — under a quarter of the time budget, then “verified, done.” Self-verification is looking at something the grader isn’t.
25 of the 31 were also 0-for-3 for the Uncensored build. So roughly 80% of the remaining failures are not a model difference; they are problems this size of model, at this context limit, with this harness, does not solve.
Ops notes
The infrastructure fixes from last time (watchdog timeout 120 s, automatic image cleanup) held. Zero server restarts over the 15-hour round, two days of continuous uptime, no wedge. Memory headroom repeatedly dipped from ~80% to 19% while a 100K+ context task ran and recovered when it finished — never crossing the alert threshold (free < 1 GB). The 60 GB disk prefix-cache sat at its limit and rotated via LRU.
Next
Round 4 (pass@4) on the remaining 31 is already running. Expectations are low — a task that timed out three times rarely flips on a retry; if anything recovers it will come from bucket ③. A fair Accuracy figure (the official leaderboard uses 5 trials) would also require running the Uncensored build for rounds 4 and 5, which is another day and a half on this machine.
One line: match the trial counts and the stock model still wins. 58 vs 56.