Two filled on try four — Qwen3.8-27B 4bit on Terminal-Bench 89, pass@4 48 and the vanished ERRs
Round 4 of Qwen3.8-27B MLX 4bit (17GB): restart only the server, rerun the 43 failures. Two recoveries reach pass@4 48/89 and the 13 round-3 ERRs drop to 1 — the server was never the cause.
Summary
In round 3, Qwen3.8-27B 4bit (17GB) recovered nothing and stayed at pass@3 46, with 13 of the 43 tasks lost as ERR — the model never generated a single token. I corrected the cause: not the server, but harbor’s agent-setup apt-get install hitting a 120-second timeout (container network). I wrote that “restarting the server is harmless, but whether ERR recurs is up to the network.”
So round 4 restarted only the server (kill watchdog + mlx-serve → wipe KV cache → relaunch with identical args) and reran the same 43 tasks with everything else untouched. 21 hours 49 minutes.
Result: pass@4 = 48 / 89 (53.9%) — 2 recovered. And the ERRs fell from 13 to 1.
Of the 43: PASS 1, PASS(TO) 1 (correct at the timeout boundary), FAIL 10, TO 30, ERR 1.
| pass@k | 27B 4bit (17GB) | Flash Next stock (100GB) | Uncensored 4bit |
|---|---|---|---|
| pass@1 | 35 (39.3%) | 45 | 41 |
| pass@2 | 46 (51.7%) | 53 | — |
| pass@3 | 46 (51.7%) | — | — |
| pass@4 | 48 (53.9%) | — | — |
| pass@5 | — | 64 | 59 |
Per-round fills: 35 → +11 → +0 → +2. The curve is flattening as expected, but the jump back from round 3’s dead stop (+0) has a specific reason: the candidates lost as ERR in round 3 finally got a real attempt this time.
The two recoveries
| Task | Round 4 | Time | Stock pass@5 | Uncensored pass@5 |
|---|---|---|---|---|
| fix-ocaml-gc | PASS | 30.2 min | PASS | PASS |
| model-extraction-relu-logits | PASS(TO) | 15.0 min | TO (never solved) | PASS |
fix-ocaml-gc fixes a heap-walking bug in the OCaml runtime GC. 27B corrected the heap-pointer increment (the line adding Whsize_hd(hd)), rebuilt the whole compiler, and passed the 40-test basic suite with zero failures. Build + full test ate 30 minutes but fit inside the 60-minute limit. A task both Flash Next models had already passed — caught by 27B on the fourth try.
model-extraction-relu-logits is different. It reverse-extracts a neural net’s ReLU kinks (activation boundaries) from logits, and stock Flash Next (the pass@5-64 model) timed out on all five attempts — it never solved it. 27B fixed the kink clustering (737 kinks piling into one cluster) right at the timeout boundary for a PASS(TO). Uncensored did solve this one, so it isn’t “only 27B,” but the fact stands: a task the big original never cracked was filled by a model 6× smaller. After compile-compcert and query-optimize in round 2, that’s one more cell where the ranking flips by task.
From 13 ERRs to 1 — the server was never the cause
The key correction in the round-3 post was: “the 13 ERRs weren’t from a long-running server, they were harbor’s setup apt install hitting a 120s timeout.” Back then that was a hypothesis backed by a traceback. Round 4 became its natural test.
- The server was restarted. If round-3 ERRs were really from server uptime, a clean server should drive ERR toward zero.
- Everything else is identical. If ERRs were network (apt mirror), they’d recur or not independent of the restart, depending on that night’s network.
Actual result: zero apt-setup-timeout ERRs in round 4. The single ERR (db-wal-recovery) died after running normally for 9.7 minutes and 41 turns — a different failure entirely, not the setup stage. So the 13 that clustered into round-3’s batches 2–7 were a transient container-network issue that night, not reproduced on another. Exactly consistent with “server-independent.”
One sample can’t nail causation. But the contrast across two rounds makes one thing clear: the first intuition — “the server aged and threw ERRs” — was wrong. The spot where ERRs clustered (setup apt) came back completely clean.
What the lost 7 did on a real attempt
Of the 13 lost to ERR in round 3, 7 were flagged as “recoverable” because another model had passed them at pass@5. This time all 7 got a normal attempt:
| Task | Round 4 | Stock · Uncensored pass@5 |
|---|---|---|
| fix-ocaml-gc | PASS | both PASS |
| configure-git-webserver | FAIL (1.4 min) | both PASS |
| dna-insert | FAIL (6.2 min) | both PASS |
| crack-7z-hash | TO (30 min) | both PASS |
| mailman | TO (30 min) | both PASS |
| polyglot-rust-c | TO (15 min) | Uncensored only |
| db-wal-recovery | ERR again | both PASS |
Only 1 of 7 passed. The rest were things 27B fails even with a proper attempt — configure-git-webserver and dna-insert diagnosed the problem correctly (dna-insert even pinpointed a 39bp insertion at position 215) but the final artifact didn’t pass grading; mailman fixed the import error but stalled on LMTP mail delivery. ERR meant “never attempted,” not “couldn’t solve” — but restoring the attempt still leaves most of them failing. Round 3’s +0 wasn’t only about the lost ERRs.
When monitoring lied — the trajectory isn’t the truth
Watching the last task, winning-avg-corewars (raise a Core War bot’s average win rate by dueling in pmars), I nearly misread it. The check script printed:
winning-avg-corewars: 41.4 min elapsed / 60 limit · 6 turns · ctx 8K · last turn 48.5 min ago
⚠ no new turn for 48 min — long command running, or server stalledBy trajectory steps alone: stuck at 6 turns, no new turn for 48 minutes — a textbook wedge. But the server metrics in the same output said the opposite — request count climbing 3167 → 3294 → 3403 → 3499, and the prompt being processed growing 55K → 73K → 88K → 103K. The agent was running hundreds of pmars duels within a single turn and piling the results into context. A trajectory “step” only advances when the model stops to issue the next command; this task ran a long battle loop inside one step, so the step counter looked frozen.
The lesson is clear. Whether an agent is alive is told by rising server requests and prompt size, not by trajectory turn count. A 130K wedge (path-tracing-reverse at 213 turns, video-processing at 229, hammering the context ceiling and spilling overflows) and a healthy long-running command both look like “turns not advancing” — but reading the server log alongside separates them instantly. Baking that check into the monitor is what killed the false alarm this time.
Onto a fifth
pass@1 35 → pass@2 46 → pass@3 46 → pass@4 48. Most of the remaining 41 (FAIL 10 · TO 30 · ERR 1) are still failures that need a harness or prompt change, not more attempts — heredoc traps, 130K wedges, optimization traps, self-verification errors. Still, drawing the curve out to where stock (64) and Uncensored (59) sit keeps the table honest, so as I write this, round 5 is running on the remaining 41.
One thing was confirmed again: a small model doesn’t trail a large one everywhere. Just as 27B filled model-extraction-relu-logits that stock missed all five times, the gap in the aggregate score and the question of “who solves this specific problem” are two different things.