Third try recovered nothing — Qwen3.8-27B 4bit on Terminal-Bench 89, pass@3 46 and the 13 ERRs
Round 3 of Qwen3.8-27B MLX 4bit (17GB): re-running the 43 round-2 failures recovers zero, pass@3 stays 46/89. 30 failed exactly as before; 13 never ran a turn — and the real cause was not the server.
Summary
In the round 2 post, one retry took Qwen3.8-27B 4bit (17GB) from 35 to 46. Same recipe again: the 43 tasks that failed round 2, re-run under exactly identical settings (terminus-2, ctx 120K, temperature 0.3, reasoning xhigh, timeout 1.0×). 16 hours 26 minutes.
Result: pass@3 = 46 / 89 (51.7%) — zero recoveries. Same number as round 2.
The 43 broke down as PASS 0, FAIL 6, TO 24, ERR 13. ERR is a new status; there was not a single one across the 143 trials of rounds 1 and 2.
| pass@k | 27B 4bit (17GB) | Flash Next stock (100GB) | Uncensored 4bit |
|---|---|---|---|
| pass@1 | 35 (39.3%) | 45 | 41 |
| pass@2 | 46 (51.7%) | 53 | — |
| pass@3 | 46 (51.7%) | — | — |
| pass@5 | — | 64 | 59 |
The round 2 post ended with “the expected recovery rate of the next retry drops fast.” It didn’t drop — it hit zero. But 13 of the 43 never got a chance, so the honest count of “failed three times” is 30. The 13 get their own section below.
Why 30 tasks failed the same way a third time
Line the 30 properly graded tasks (FAIL 6 · TO 24) up against their round 1 and 2 trajectories and the conclusion is simple: same trap, same way in. At temperature 0.3, identical observations produce near-identical responses, and the early observations of a task are identical every time. Tasks with room to branch already branched in round 2; what’s left is the set that doesn’t.
| Failure mode | Task | Round 3 trajectory |
|---|---|---|
| heredoc trap | regex-chess | 589 of 605 turns were the same command in a row — “terminal is stuck in a heredoc, send Ctrl+C” for the full 60 minutes |
| 〃 | feal-differential-cryptanalysis | 115 turns, 96 identical in a row |
| 〃 | feal-linear-cryptanalysis | 89 turns, tried Ctrl+Z to escape, 30 minutes |
| 130K context wedge | path-tracing-reverse | 319 turns, hit 130K, then 252 “Technical difficulties” retries |
| 〃 | video-processing | 287 turns, same |
| 〃 | make-mips-interpreter | 119 turns, started as a heredoc trap, ended as a wedge |
| repetition loop | build-pov-ray | 193 turns, checked “render still in progress” 74 times in a row, 200 minutes |
| optimization trap | winning-avg-corewars | 75 turns, ctx 103K. Found a winning warrior and kept tuning for 60 minutes instead of submitting |
| “verified” then wrong | mteb-retrieve / sanitize-git-repo / sparql-university / bn-fit-modify | declared “verified” in 3–7 minutes, FAILed grading. Same answer all three rounds |
The 6 FAILs are the clearest case. mteb-retrieve ended all three rounds in about 9 turns and 3–4 minutes with “task completed successfully,” and was wrong all three times. A wrong answer that passes the model’s own verification does not get fixed by retrying — the verification criterion itself is wrong, so the same answer passes the same check every time.
The 24 timeouts alone consumed 860 minutes in round 3 — 87% of the run. The 8 classified as heredoc / wedge / repetition loop account for 470 of those. The round 2 post said “unless loop detection goes into the harness, the same task burns its full time limit every round.” It did.
What the 13 ERRs actually were — not the server
I suspected the server first. mlx-serve had been running for 74 hours straight since round 1, and the ERR message was Command timed out after 120 seconds, so “long uptime made the first response slow enough to trip the harness’s command timeout” sounded right. I restarted the server before round 4 on that theory.
Only after reading the traceback to the end did I see it was wrong. The timeout wasn’t in a model call. It was in harbor terminus-2’s agent setup.
trial.py: _prepare() → _setup_agent()
terminus_2.py: setup() → self._session.start()
tmux_session.py: _attempt_tmux_installation() → _install_recording_tools()
docker.py: _compose_exec(...) → RuntimeError: Command timed out after 120 secondsBefore it calls the model, terminus-2 enters the task container and apt-get installs tmux and asciinema. That install command has a 120-second cap, and all 13 ERRs died there. agent_execution is None — the model never generated a single token. That’s why all 13 show 0.0 minutes in result.json.
The evidence is in the setup-time distribution. Of the 43 round 3 trials, the 30 normal ones took a median of 11 seconds (min 1.4 s) to set up the agent; all 13 ERRs took exactly 120.47 seconds. The same tasks set up in 18 seconds in rounds 1 and 2. Same container image, same apt cache state — the only thing different was the network at that moment.
The timing says the same. The 13 are clustered in batches 2–7, between 15:53 and 23:10 on September 11, with none in batches 1, 8, or 9. In batch 2, configure-git-webserver (15:53), db-wal-recovery (15:56), and crack-7z-hash (15:58) died back to back at 2–3 minute intervals — each one waiting 120 seconds on an apt mirror and giving up. For those seven hours, outbound apt requests from inside the containers were intermittently stalling, while the model server was happily working other tasks the whole time.
harbor 0.22.0’s code calls this step “best-effort.” There’s a guard: if installation exceeds a 240-second budget, log a warning and proceed without recording. But that guard only catches asyncio.TimeoutError. The per-command 120-second timeout inside it surfaces as a RuntimeError, which sails straight through the guard and kills the whole trial. A failed install of a recording tool takes down an entire model evaluation.
The problem is that 7 of those 13 were recovery candidates: configure-git-webserver, crack-7z-hash, db-wal-recovery, dna-insert, fix-ocaml-gc, mailman, polyglot-rust-c — tasks another model in the series had passed in some round. The 30-task saturation is the model’s limit; these 7 were simply never attempted. So pass@3 46 is a lower bound, not a ceiling.
Where the time went
| Group | Count | Total |
|---|---|---|
| TO | 24 | 860 min |
| FAIL | 6 | 50 min |
| ERR | 13 | 26 min (120 s wait each) |
| batch overhead | — | 50 min |
14 h 20 min of the 16 h 26 min is the 24 timeouts. Same structure as round 2 — failing tasks burn their entire limit. Had the 13 ERRs run normally, they’d have added roughly 4–5 hours.
Onto a fourth
pass@1 35 → pass@2 46 → pass@3 46. The curve has flattened, and the reasons split three ways.
- 30 tasks won’t be solved by retrying. Heredoc escape, context wedges, loop detection, an instruction to “submit and stop” — all of that belongs to the harness or the prompt.
- 13 tasks were never evaluated. That’s infrastructure loss, not a model score.
- 7 of those were solved by other models, so they could genuinely recover.
Round 4 re-runs all 43. The server is already restarted (a misdiagnosis, but harmless), so if ERRs reappear the network is confirmed as the cause, and if they don’t we get to see whether the lost 7 actually pass. That’s the last piece needed to honestly place where 27B’s pass@k curve stops in this series.
One lesson: don’t decide the cause of a benchmark ERR from the message alone. “Timed out after 120 seconds” was enough for me to restart a server, and the answer was 40 lines further down the traceback.