Six filled on try five — Qwen3.8-27B 4bit on Terminal-Bench 89, pass@5 54 past Uncensored
Round 5 of Qwen3.8-27B MLX 4bit (17GB): rerun the 41 round-4 failures. Six recoveries reach pass@5 54/89, fills rebound +0/+2/+6, the gap to Uncensored (59) narrows to 5, ERRs vanish 13/1/0.
Summary
In round 4, Qwen3.8-27B 4bit (17GB) recovered 2 to reach pass@4 48, and I wrote that “most of the remaining 41 are cells all three models miss, so further recoveries will be sparse.” That was wrong. Same method — restart only the server, everything else unchanged — and round 5 filled six. 20 hours 44 minutes.
Result: pass@5 = 54 / 89 (60.7%) — 6 recovered. And the ERRs vanished completely: 13 (round 3) → 1 (round 4) → 0 (round 5).
Of the 41: PASS 5, PASS(TO) 1 (correct at the timeout boundary), FAIL 7, TO 28, ERR 0.
| pass@k | 27B 4bit (17GB) | Flash Next stock (100GB) | Uncensored 4bit |
|---|---|---|---|
| pass@1 | 35 (39.3%) | 45 | 41 |
| pass@2 | 46 (51.7%) | 53 | — |
| pass@3 | 46 (51.7%) | — | — |
| pass@4 | 48 (53.9%) | — | — |
| pass@5 | 54 (60.7%) | 64 | 59 |
Per-round fills: 35 → +11 → +0 → +2 → +6. The curve that flat-lined in round 3 came back to life, and at pass@5 the gap to Uncensored (59) narrows to 5. Stock (64) is still 10 away.
The six recoveries — why the count went up
| Task | Round 5 | Time | Round 4 | Stock pass@5 | Uncensored pass@5 |
|---|---|---|---|---|---|
| raman-fitting | PASS | 4.1 min | FAIL | PASS | FAIL |
| mteb-retrieve | PASS | 5.5 min | FAIL | PASS | PASS |
| crack-7z-hash | PASS | 8.5 min | TO | PASS | PASS |
| bn-fit-modify | PASS | 22.6 min | FAIL | PASS | PASS |
| llm-inference-batching-scheduler | PASS | 24.9 min | FAIL | PASS | FAIL |
| rstan-to-pystan | PASS(TO) | 30.0 min | TO | FAIL | FAIL |
Median recovery time was 22.6 minutes. In round 4 these were 4 FAILs and 2 TOs; in round 5 all six passed. Again it’s variance, not attempt count — give the same problem once more and some fraction walks a different path to the answer. crack-7z-hash, a 30-minute TO in round 4, finished in 8.5 via a straight 7z2john+john crack; bn-fit-modify correctly concluded the partial correlation was essentially zero, so there was no direct edge.
The rebound “+0 → +2 → +6” runs against intuition — pass@k usually dries up over rounds. But round 3’s +0 was an artificial floor: half that round was lost to ERR, and rounds 4–5 cleared that factor by restarting the server clean. The remaining failure pool still held several “high-variance, mid-difficulty” tasks, and those hit this time.
Two cells past Uncensored, and a third only 27B solved
What stands out here are the cells where the ranking flips.
- llm-inference-batching-scheduler and raman-fitting — stock Flash Next passed both, but Uncensored never solved them through pass@5. 27B filling them sharpens the picture: a 4bit model 6× smaller leads a same-lineage 4bit (Uncensored) on specific cells.
- rstan-to-pystan — neither stock nor Uncensored solved it through pass@5; only 27B did. It ported the R Stan model to PyStan and fit 4 MCMC chains right at the deadline for a PASS(TO). After compile-compcert and query-optimize in round 2, that’s the third cell only 27B solved among the three.
By aggregate score (54 < 59 < 64) 27B still trails, but the question of “who solves this specific problem” keeps flipping. Three of them is hard to call a coincidence.
The last scene of the vanishing ERRs
Round 3’s post corrected the 13 ERRs from “the server aged” to harbor’s setup apt install hitting a 120s timeout (network). Round 4, server restarted, dropped it to 1. Round 5: zero.
ERR: 13 → 1 → 0. This three-round decline settles the diagnosis for good: the ERRs were neither the model nor the server, but that night’s container network. Restarting the server didn’t fix them — they were server-independent all along and simply don’t occur on a night the apt mirror is healthy. Across three nights the server state differed wildly (74h uptime → restart → restart), yet the ERRs tracked only the mirror’s condition.
The 35 still failing
Even with the second-biggest rebound (+6), the remaining 35 (FAIL 7 · TO 28) fail the same ways.
- heredoc trap — circuit-fibsqrt got the terminal stuck feeding a Python script via heredoc and looped Ctrl+D/Ctrl+C for the full 60 minutes into a TO; make-mips-interpreter never finished its interpreter either.
- 130K context wedge — video-processing kept reloading frames until it hit the context ceiling (131072), spun 337 turns, and TO’d. Over 1,300 of the server’s 400-overflows came from this one task.
- optimization / hardest cells — feal-differential/linear-cryptanalysis burned the clock designing differential/linear attacks; the path-tracing tasks failed to reverse the render’s pixel patterns.
And many of these 35 are cells all three models (27B, stock, Uncensored) miss through pass@5. The conclusion stands: these need a harness or prompt change, not more attempts — a single loop detector alone would save hours on the heredoc and 130K wedges.
Onto a sixth
pass@1 35 → 46 → 46 → 48 → pass@5 54. Drawing the curve out to stock (64) and Uncensored (59) keeps the table honest, so as I write this, round 6 is running on the remaining 35. This time I set expectations lower — the leftover pool skews harder toward “all three miss,” so recoveries should dry further. Still, 5 cells separate 27B from Uncensored (59), and how many close by variance only running tells.
One thing this small-model-many-runs experiment confirms each time: the gap in aggregate score and the question of “who solves this problem” are two different things — just as neither stock nor Uncensored solved rstan-to-pystan, and 27B did.