Two more on try six — Qwen3.8-27B 4bit on Terminal-Bench 89, pass@6 56 and the round we wasted
Round 6 of Qwen3.8-27B MLX 4bit (17GB): rerun 35 round-5 failures. Two recoveries reach pass@6 56/89, three off Uncensored. Round three was lost to network ERRs, so this is really an honest pass@5.
Summary
In round 5, Qwen3.8-27B 4bit (17GB) recovered 6 to reach pass@5 54. Round 6 reran the remaining 35 with everything unchanged but a clean server restart. About 18.5 hours.
Result: pass@6 = 56 / 89 (62.9%) — 2 recovered. custom-memory-heap-crash (26.5 min) and mailman (17.6 min) passed. ERRs were again zero.
Of the 35: PASS 2, FAIL 6, TO 27, ERR 0.
| pass@k | 27B 4bit (17GB) | Flash Next stock (100GB) | Uncensored 4bit |
|---|---|---|---|
| pass@1 | 35 (39.3%) | 45 | 41 |
| pass@2 | 46 (51.7%) | 53 | — |
| pass@3 | 46 (51.7%) | — | — |
| pass@4 | 48 (53.9%) | — | — |
| pass@5 | 54 (60.7%) | 64 | 59 |
| pass@6 | 56 (62.9%) | — | — |
Per-round fills: 35 → +11 → +0 → +2 → +6 → +2. At pass@6 the gap to Uncensored (59) narrows to 3. Stock (64) is still 8 away.
But there’s a dishonest corner in this table. We only ran six rounds because the third one was lost whole.
Why six rounds — the one that got thrown away
The rule of a pass@k run is simple: retry only what failed. But round 3 lost 13 of 43 tasks to ERR — not the model failing to emit a token, but harbor’s agent-setup apt-get install hitting a 120-second timeout: a transient problem with that night’s container network (the apt mirror). Those 13 weren’t “couldn’t solve,” they were “never attempted,” and round 3’s recovery came out to +0.
So round 3 was essentially a whole round spun for nothing. Round 4 restarted the server and reran the same 43; ERR dropped to 1, and by round 5 to 0 — three rounds settling the diagnosis that it was the network, not the server. But we paid for it with an extra round.
Count honestly, then. The rounds that actually filled anything are 2, 4, 5, 6; round 3 is a null round eaten by the network. Drop it and the effective attempts are:
pass@1 35 → 46 → 48 → 54 → 56Five. This pass@6 56 is nominally the sixth try, but it’s really closer to an honest pass@5. (To be fair, 30 of round 3’s 43 tasks were attempted normally and failed, so dropping the whole round is generous to 27B.) With that caveat, right there sits a comparison: Uncensored’s pass@5 is 59 — the same 4bit lineage.
At equal effective attempts, a 6×-smaller 17GB 4bit trails a same-lineage 4bit (Uncensored) by three cells. By aggregate score it still loses, but “how many does it solve given five tries” is a different question, and there the gap is down to 3. Had we not burned round 3, we’d have reached this conclusion a round sooner.
The two recoveries — variance, again
| Task | Round 6 | Time | Round 5 | Stock lineage (ddalcu) | Uncensored (ARC4NUM) |
|---|---|---|---|---|---|
| custom-memory-heap-crash | PASS | 26.5 min | TO 30m | PASS | pass@5 PASS |
| mailman | PASS | 17.6 min | TO 30m | PASS | PASS |
No “only 27B solved it” cell this time. Both were already passed by the other two models; 27B just caught up on the sixth try. So this round is no ranking-flip drama — it’s clearing the last two twigs variance left on the tree.
- custom-memory-heap-crash — reproduce and fix a heap crash. GDB doesn’t work in the container, so it hung a high-priority
__attribute__((destructor))to catch the problem by destructor order, and got both binaries compiling cleanly inside the 26.5-minute limit. Round 5 burned the full 30 minutes to a TO; this time a single change of approach fit inside the clock. - mailman — a round-5 TO stuck on LMTP mail delivery. This time it pinned the failure to postfix
mydestinationmissing the local domain, breaking direct delivery, and fixed it. Again a matter of path, not attempt count.
Both were TOs at round 5 and PASSes at round 6. Give the same problem once more and some fraction walks a different road to the answer — this experiment’s recurring lesson held to the last.
The 33 still failing
The remaining 33 (FAIL 6 · TO 27) are now mostly cells all three models (27B, stock, Uncensored) miss, and they fail the same ways they have all along.
- 130K context wedge — video-processing chased an over-eager background subtractor into the context ceiling (131072), spun 464 turns, and TO’d; over 1,300 of the server’s overflows came from this one task (nearly identical to round 5). regex-chess likewise burned 60 minutes across 179 turns on castling and check detection.
- optimization / hardest cells — feal-differential/linear-cryptanalysis ran the clock designing differential/linear attacks; winning-avg-corewars couldn’t clear the pmars win threshold (75+); the path-tracing tasks again failed to reverse the render’s pixel patterns.
- environment / network walls — qemu-alpine-ssh hit container apt 404s, protein-assembly hit blocked external DBs, install-windows-3.11 spent 60 minutes on a black QEMU screen.
The conclusion stands: these need a harness or prompt change, not more attempts. A single loop detector or one context-compaction pass alone would save hours on the 130K wedges and the heredoc family.
Closing the campaign
pass@1 35 → 46 → 46 → 48 → 54 → pass@6 56. I’m closing it here. The remaining 33 are cells all three models miss in common; the curve won’t climb further without touching the harness.
After six rounds, two sentences remain. One: what killed round 3 was the network that night, not the server — pile up samples and the cause separates. The other: a 17GB 4bit reaches, at the pass@5 it would have hit five tries in without the wasted round, a score of 56 — three cells behind the same-lineage Uncensored (59). In this small-model-many-runs experiment, the gap in aggregate score and the question of “who solves this under equal conditions” stayed two different questions to the very end.