How much does a retry recover — Qwen3.8-27B 4bit on Terminal-Bench 89, pass@2 46 vs 53
Round 2 of Qwen3.8-27B MLX 4bit (17GB): re-running the 54 round-1 failures reaches pass@2 46/89 (51.7%). 11 recoveries (median 6.5 min), two solved only by 27B, and why retrying failures is slower.
Summary
Yesterday’s post ran Qwen3.8-27B 4bit (17GB) once against Terminal-Bench 2.1’s 89 tasks under the same conditions as the two Flash Next models, landing at 35/89 (39.3%). pass@k is a game of retrying only what failed. So I re-ran the 54 tasks that didn’t pass in round 1 under exactly identical settings (terminus-2, ctx 120K, temperature 0.3, reasoning xhigh, timeout 1.0×). 23 hours 49 minutes.
Result: pass@2 = 46 / 89 (51.7%) — round 1’s 35 plus 11 recovered here.
Of the 54 retries: PASS 9, PASS(TO) 2 (correct file at the timeout boundary), FAIL 10, TO 33.
| pass@k | 27B 4bit (17GB) | Flash Next stock (100GB) | Uncensored 4bit |
|---|---|---|---|
| pass@1 | 35 (39.3%) | 45 | 41 |
| pass@2 | 46 (51.7%) | 53 | — |
| pass@5 | — | 64 | 59 |
A second pass took 27B from 35 to 46, filling 11. Under the same conditions the original went pass@1 45 → pass@2 53, so the gap narrowed from 10 to 7. A model 6× smaller clearing half the suite on a retry is meaningful in itself, but it did not catch the original.
The 11 recoveries — what got filled
| Task | Round 2 | Time | Round 1 | Other models |
|---|---|---|---|---|
| nginx-request-logging | PASS | 1.5 min | FAIL | both ddalcu & Uncensored had passed it |
| merge-diff-arc-agi-task | PASS | 2.9 min | FAIL | 〃 |
| sqlite-with-gcov | PASS | 3.9 min | FAIL | 〃 |
| polyglot-c-py | PASS | 4.9 min | TO | 〃 |
| query-optimize | PASS | 6.0 min | FAIL | only 27B, of all three |
| regex-log | PASS | 6.5 min | TO | had passed at pass@5 |
| distribution-search | PASS | 8.3 min | TO | ddalcu & Uncensored passed |
| torch-tensor-parallelism | PASS(TO) | 15.0 min | FAIL | 〃 |
| caffe-cifar-10 | PASS | 19.9 min | TO | 〃 |
| compile-compcert | PASS | 28.7 min | TO | only 27B, of all three |
| mcmc-sampling-stan | PASS(TO) | 30.0 min | TO | ddalcu & Uncensored passed |
The median time of the 11 recoveries is 6.5 minutes. Half finished within ten. These tasks weren’t “too hard to solve” — they were “stumbled once, walked straight to it the second time.” Things that ended as short FAILs in round 1 (nginx-request-logging at 4 min, merge-diff at 5 min) came back as PASS in 1–3 minutes. It means 27B has high variance on short-path tasks: give it the same problem twice and one of the two lands.
At the other end are slow recoveries — compile-compcert (29 min), caffe-cifar-10 (20 min), mcmc-sampling-stan (30 min). It configured CompCert with Coq 8.16.1 and built it with make; the MCMC run fit 4 chains × 100k iterations just under the deadline for a PASS(TO).
Two tasks only 27B solved
Nine of the 11 recoveries are tasks a Flash Next variant (original or Uncensored) had already passed in some round. Two are different.
- compile-compcert — the original timed out through pass@5, Uncensored FAILed even at pass@5. Across the entire series, 27B is the first and only model to pass it.
- query-optimize — both original and Uncensored FAILed through pass@5. Again, only 27B, in 6 minutes.
Evidence that a small model doesn’t trail a large one everywhere — on some tasks it leads. 27B loses on the aggregate pass@k score, but on “which model solves this specific problem” there are cells where the ranking flips.
Why retrying only failures took longer
Mid-run I got asked: “why does re-running the 54 failures (24 h in round 2) take longer than all 89 (31 h in round 1)?” It looks backwards but it’s expected. Split round 1’s actual runtime into passes and failures and the reason is clear.
| Group | Count | Mean time | Total |
|---|---|---|---|
| PASS | 35 | 6.9 min | 4.0 h |
| Failures | 54 | 28.2 min | 25.4 h |
A passing task ends the instant it finds the answer — mean 7 min, and those 35 accounted for only 4 hours of the whole 89-task run. A failing task, by contrast, burns its full limit (15/30/60/200 min) before timing out — those 54 ate 25 of round 1’s 29 hours. A pass@k retry by definition selects exactly these “time-eating 54,” with no fast tasks mixed in to pull the average down. Running only the failures, serially, taking as long as or longer than the full suite is exactly what you’d expect.
The 43 still failing
The 43 that didn’t pass even on a second try (FAIL 10 · TO 33) failed the same way as round 1.
- 130K context wedge — custom-memory-heap-crash never caught the static-destructor crash and spun 287 turns into a 130K context.
- heredoc trap — make-mips-interpreter and regex-chess got the terminal stuck in a heredoc, unable to send the terminator, looping.
- “verified” then wrong — mailman self-reported “all 3 tests passed” then FAILed grading; sparql-university claimed “query correct, saved” and FAILed. 27B’s habit of over-trusting its own verification, again.
- repetition loops — build-pov-ray hammered the same mirror for all 200 minutes after SourceForge blocked it.
And 19 of these 43 are tasks that all three models (27B, original, Uncensored) failed through pass@5. Those cells are hard for the benchmark itself; model size and retry count don’t fill them.
Onto a third
pass@1 35 → pass@2 46. The expected recovery rate of the next retry drops fast. Half of the remaining 43 (19) are untouched by the entire series, and the rest are failures — 130K wedges, heredoc traps, self-verification errors — that need a harness or prompt change, not more attempts. Still, drawing the pass@k curve out to where the original (64) and Uncensored (59) sit keeps the comparison honest. Round 3 is running on the remaining 43.
One thing was clear again: unless loop detection goes into the harness, the same task burns its full time limit the same way every round. That was most of the 24 hours.