Round five: the "saturated" curve bent up again — stock Qwen3.8 pass@5 64/89 (71.9%)
Round-5 (pass@5) Terminal-Bench 2.1 results for stock Qwen3.8 Flash Next mixed-4-8bit: 4 of 29 retries recovered (two timeout-with-the-right-file), a 120K context handoff and a fallback loop
Summary
In part 3 the increments had gone 8 → 5 → 2 and I called the curve “saturating”. I ran the remaining 29 tasks a fifth time under identical conditions and four flipped. I was wrong.
Bottom line: pass@5 = 64/89 (71.9%). Official Terminal-Bench reports 5 trials, so this completes the stock model’s attempts.
| Attempt | Stock ddalcu | Uncensored ARC4NUM |
|---|---|---|
| 1 (pass@1) | 45 / 89 (50.6%) | 41 / 89 (46.1%) |
| 2 cumulative | 53 / 89 (59.6%) | — |
| 3 cumulative | 58 / 89 (65.2%) | 56 / 89 (62.9%, best-of-3) |
| 4 cumulative | 60 / 89 (67.4%) | — |
| 5 cumulative (pass@5) | 64 / 89 (71.9%) | — |
Increments 8 → 5 → 2 → 4. To be honest about it: two of the four are PASS(TO) — the run ended with AgentTimeoutError, but the files left on disk passed the grader. Half of this round’s wins are ones the model never claimed to have finished.
What happened in round 5
29 tasks: 4 recovered, 5 FAIL, 20 TO. 12 h 30 min, 24.2 min per task on average.
raman-fitting (3.2 min, 14 turns) — fit the G and 2D peaks of a graphene Raman spectrum. Uncensored failed all three tries; stock had “verified” and submitted wrong answers in 3–4 minutes four times running. This trajectory diverged from turn three: the model noticed the data file uses European decimal commas and a wavelength (nm) x-axis, converted to wavenumber (1/λ), and the laser line (~525 cm⁻¹), G (1580) and 2D (2670) landed exactly where the textbook says. It then fit both Lorentzian+offset and Gaussian and picked by residual. Same three minutes as before; the difference was suspecting the data format first.
overfull-hbox (9.3 min, 25 turns) — remove seven overfull hbox warnings from a narrow-textwidth LaTeX document using only a synonym list. This was one of the six tasks only Uncensored had passed; stock caught up on try five. It started like the earlier rounds, swapping each flagged word for a shorter synonym — and fell into the whack-a-mole where shifted line breaks create new overfulls. Turn 15 is where it departs from the previous four runs: it concluded “line breaking is a global optimisation”, wrote a greedy search that compiles ~600 times per iteration over the 110 synonym families, and converged 7 → 5 → 3 → 2 → 1 → 0. First time I have seen this model switch from hand-editing to building a tool.
torch-pipeline-parallelism (15.0 min TO, 19 turns) — implement an AFAB pipeline-parallel training step. The container has neither torch nor pip, and all four previous runs burned the whole budget trying to install them. This time it gave up on installation after four minutes, wrote the implementation first, then went to verify — and ran out of time on its own test harness failing to push a requires_grad loss tensor through a queue. But the train_step_pipeline_afab it had already written passed all four grader tests. “Write the answer before fixing the environment” was the whole difference.
winning-avg-corewars (60.0 min TO, 93 turns, ctx 63K) — write a Core War warrior that beats each of five opponents above a per-opponent win threshold. This was part 3’s poster child for repetition (146 turns · ctx 127K · 107 looping turns). This time there was no loop at all. It patched pMARS to log where its processes die, ran five warrior archetypes against each opponent to measure wins, settled on a g2-clear variant with dec=2700, and had all five thresholds cleared at the 45-minute mark. Then it spent the last 15 minutes tightening the “stone margin is only +1” — having written, itself, that with a fixed seed +1 is enough — until the clock ran out. Grader: 5/5 opponents.
Where they diverge (stock ×5 vs Uncensored ×3, all 89) — final
| Bucket | Count | Tasks |
|---|---|---|
| Both pass | 52 | — |
| Both fail every time | 21 | path-tracing family, regex-chess, feal-linear, make-*-mips, install-windows-3.11, compile-compcert, schemelike, rstan-to-pystan, query-optimize, video-processing, … |
| Stock only | 12 | circuit-fibsqrt, custom-memory-heap-crash, dna-insert, extract-elf, llm-inference-batching-scheduler, mteb-leaderboard, qemu-startup, sparql-university, train-fasttext, tune-mjcf, raman-fitting, torch-pipeline-parallelism |
| Uncensored only | 4 | feal-differential-cryptanalysis, filter-js-from-html, model-extraction-relu-logits, polyglot-rust-c |
Uncensored-only went from 6 to 4. The remaining four are tasks stock never cleared in five tries, so I read them as a real gap between the two models. But the attempt counts are 5 vs 3, which makes this table unfair — see “Next”.
Anatomy of the remaining 25 failures
① Repetition loops and context blow-ups. Four of the 20 timeouts were loops, and this round I saw the harness both break a loop and fail to.
- path-tracing-reverse 30 min · 230 turns · ctx 129K. The worst shape. Once the prompt exceeded 131K the server returned 400, and terminus-2 cycled through its four-step fallback (full request 400 → parse-error retry 400 → handoff summary 400 → “Briefly continue” as a 724-token single message) for eleven minutes. The server’s 400-overflow counter went up by 335 from this one task. After the context overflows, even the summary can’t be sent, so there is no recovery path.
- regex-chess 60 min · 194 turns · ctx 120K. Crossed 100K at turn 95 and was repeating the same message four times in a row near turn 120. Then terminus-2’s context handoff fired at 120K, ctx dropped to 47K, and the loop broke — the following 74 turns were all distinct. Only nine minutes remained, so the result is still TO, but the “forced change of action” I asked for in part 3 got an accidental test and it worked.
- schemelike-metacircular-eval 40 min · 169 turns · ctx 120K and polyglot-rust-c 15 min · 60 turns — same pattern as part 3.
② Near misses. Four.
- query-optimize 5.4 min — fifth time passing 5 of 6 tests. Query runtime 0.980 s, threshold 0.765 s (golden 0.729 × 1.05). Five times it declared the query “fast enough” and submitted.
- video-processing 7.1 min — extract take-off and landing frames from a jump video. It got a clean parabola for the feet and submitted takeoff=55, landing=60 as “physically accurate”; the grader accepts takeoff 50–54 and landing 62–64. The grader is looking at the start of the push-off and full ground contact, not the moment the feet lift. A definition mismatch.
- rstan-to-pystan 30 min TO — this time the cause is crisp. Sampling reached 100 %, but httpstan’s four workers plus the parent each held the full sample set, hit the 8 GB cgroup limit (7.7 GB) and hung at teardown. The model Ctrl-C’d, rewrote the script to run chains sequentially, re-ran, and ran out of time. Sequential from the start would have passed.
- model-extraction-relu-logits 11.7 min — recovered 20 of 30 rows; it assumed 20 hidden neurons where there are 30.
③ Premature confident submits. sanitize-git-repo 4.3 min (“all four AWS keys verified removed” → 2 of 3 tests fail), filter-js-from-html 9.7 min. Same as part 3.
Operations notes
12.5 hours, zero server restarts, zero wedges, uptime past three days. Three new things this round.
- 400 overflow +335, all during path-tracing-reverse’s eleven-minute fallback loop. The server was fine and answered each with a 400 in seconds; this was not load.
- While regex-chess sat at 122K, free memory hit 0.2 GB with memory headroom at 15 %, and decode halved from 41 to 20.7 tok/s. After the handoff brought ctx down to 47K, it recovered to free 1 GB / 22 % / 42 tok/s within ten minutes. Even with 8-bit KV, 120K is the swap boundary on this machine.
- At 00:43 swap jumped +2.1 GB in one step and mlx-serve’s RSS dropped from 46 to 40.7 GB (the kernel paged model pages out). Decode speed did not change at all — the evicted pages were evidently not the active weights — and swap then only drifted down for the next twelve hours.
Next
The problem with the table above is attempt count. Stock has had five tries; Uncensored three (and two of those were retries with different timeout multipliers, so not strictly the same conditions). So right now I am running Uncensored’s 33 best-of-3 failures a fourth time under exactly the stock conditions — 1.0× limits, reasoning high, temperature 0.3. If Uncensored also flips four or more, the “saturation” story is dead for both models.
One line: four flipped on the fifth try, 64 vs 56. Saturation was my mistake, and now the other side gets the same number of attempts.