Giving the other side the same number of tries — Uncensored Qwen3.8 pass@4 58/89 (65.2%), gap of 6

Summary

Part 4 ended with “the other side has to get the same number of tries.” So I took the 33 tasks the Uncensored build (ARC4NUM abliterated 4-bit) had failed in its best-of-3 and ran them a fourth time under exactly the stock model’s conditions (timeout ×1.0, reasoning high, temperature 0.3, ctx 120K, terminus-2). Result: 2 recovered.

Bottom line: Uncensored pass@4 = 58/89 (65.2%), a gap of 6 behind the stock build’s pass@5 of 64/89 (71.9%). Comparing at the same four tries, the stock build was at 60, so it’s still 2 behind.

Attempt	Stock ddalcu	Uncensored ARC4NUM
1 (pass@1)	45 / 89 (50.6%)	41 / 89 (46.1%)
2 cumulative	53 / 89 (59.6%)	—
3 cumulative	58 / 89 (65.2%)	56 / 89 (62.9%, best-of-3)
4 cumulative	60 / 89 (67.4%)	58 / 89 (65.2%)
5 cumulative (pass@5)	64 / 89 (71.9%)	(running)

After the stock model flipped 4 in round five, I wrote that if Uncensored flipped 4 or more, the “saturation” story would be dead for both. It flipped 2. And both are tasks the stock build had already passed, so the gap merely narrowed from 8 to 6 — the Uncensored-only set stays at 4.

What happened in round 4

33 tasks: 2 recovered, 7 FAIL, 24 TO. 14h45m (including 7 minutes lost to a relaunch), 24.1 minutes per task on average.

dna-insert (8.5 min, 29 turns, ctx 40K) — compare input/output plasmid sequences, locate the insertion, and design a PCR primer pair that produces it. Three straight TO/FAILs before; the stock build passed it on try one. This time, when naive prefix/suffix matching failed (the insert shares homology with the insertion site, so the boundary is ambiguous), it switched approach at turn 5, installed edlib, and pinned the 39-nt insertion by alignment. Then it satisfied every constraint — annealing lengths 15/32 nt, Tm 59.54/59.74 °C, ΔTm 0.21 °C — and simulated the PCR to confirm the output sequence was reproduced before submitting. Earlier rounds had spent their time hand-tuning the boundary.

custom-memory-heap-crash (4.8 min, 18 turns, ctx 15K) — a C++ program that segfaults only in RELEASE on top of an in-house-patched libstdc++. Two 30-minute TOs and one FAIL before. This time it read locale_init.cc.patch at turn 3 and named the cause at turn 6: locale facets allocated via ::operator new while the custom heap was active get released with std::free in a static destructor after main returns. The clinching observation was that the crash (exit 139) happens after “completed successfully” prints. After the fix: exit 0 on both builds, 0 valgrind errors, submit. A five-minute task that had previously eaten 30 minutes a round — that’s the variance of this model in one line.

Where they diverge (stock ×5 vs Uncensored ×4, all 89)

Bucket	Count	Tasks
Both pass	54	—
Both fail every time	21	the path-tracing pair, regex-chess, feal-linear, make-*-mips, install-windows-3.11, compile-compcert, schemelike, rstan-to-pystan, query-optimize, video-processing, …
Stock only	10	circuit-fibsqrt, extract-elf, llm-inference-batching-scheduler, mteb-leaderboard, qemu-startup, raman-fitting, sparql-university, torch-pipeline-parallelism, train-fasttext, tune-mjcf
Uncensored only	4	feal-differential-cryptanalysis, filter-js-from-html, model-extraction-relu-logits, polyglot-rust-c

Gap of 6 = 10 − 4. “Stock only” went 12 → 10; “Uncensored only” stays at 4. Still 5 tries vs 4 — one more round to go.

Anatomy of the 31 failures

① Fallback loops — the headline of this round. Two tasks each burned their full 60-minute budget, and half of that time the model did nothing at all.

• video-processing 60 min · 822 turns · ctx 130K. Crossed 100K at turn 85 (23 min) and the server’s 131K hard limit at turn 114 (32 min). That’s where terminus-2’s handoff summary should have fired — but the prompt was already over the limit, so the summary request itself got a 400. For the next 28 minutes the trajectory accumulated 706 copies of “Technical difficulties. Please continue with the task.” In the server log: a 131K request → 400 → a ~760-token retry → repeat, several times per second. This one task added +2,120 to the 400-overflow counter.
• circuit-fibsqrt 60 min · 716 turns · ctx 130K. Same pattern: 100K at turn 84 (22 min), over the limit at turn 129 (31 min), then 585 repeats and +1,757 400s. Right up to the last real turn it was a perfectly sane trajectory — instrumenting the simulator to trace rem0=20032, root0=20064.

3,877 HTTP 400s between them. There’s a control case: regex-chess (60 min · 247 turns) crossed 100K at turn 68 (20 min) but the handoff fired before 129K, resetting ctx to 3K at turn 82, and the remaining 165 turns were used normally (it had its own 68-repeat loop, but that’s a different disease). The stock model’s path-tracing-reverse in round 5 died the same way, so this isn’t a model difference — it’s the harness. With max_input_tokens set to 120K, a single tool output over ~11K blows straight through the wall with no chance to summarize. Either the margin needs to be wider (say 100K), or the harness needs to truncate the last tool output and retry when it gets a 400.

② Timeouts spent building the environment.

• compile-compcert 40 min · 51 turns · ctx 13K. Stock failed this one all five times too. At turn 17, configure warned that Coq 8.18.0 is unsupported, so the agent started building Coq 8.16.1 from source via opam, and the rest of the budget was eight consecutive “Still compiling Coq. Continue waiting.” Putting a Coq build inside a 40-minute limit was the losing move.
• train-fasttext 38.9 min FAIL · 39 turns. The stock build cleared this as PASS(TO) in round 4. This time the agent ran a grid search and submitted in time (dim=32, wordNgrams=2, epoch=4, self-reported 0.6412), but the grader’s held-out accuracy was 0.569 against a 0.62 bar. It had validated on a 2% slice of the same Yelp data and trusted a number that was really training-distribution fit.

③ Early-confidence submits. sanitize-git-repo 1.2 min (8 turns!), sparql-university 3.0 min (filtered by a VALUES list of 27 EU country codes; the reference set has multi-country entries like ‘GR, US’), query-optimize 6.6 min, install-windows-3.11 4.6 min. Same disease as the stock build, and repetition doesn’t cure it. extract-elf (6.3 min · 43 turns) is a slightly different case — the task the stock build cracked in round 4 via the endbr64 bytes. Uncensored hypothesized “read the whole file as 4-byte words from base 0x400000”, produced 3,990 keys, and submitted. Grader: 0.00% of expected values matched. It matched the example’s address pattern and never checked a value.

④ Near misses. tune-mjcf 15 min · 19 turns — make a MuJoCo simulation faster within an accuracy tolerance. At turn 18 it found the answer (“switching to the PGS solver: 0.328 → 0.156 s, max error 3.9e-7, passes”) and ran out of time before writing it to the file. raman-fitting 15 min · ctx 73K — the task the stock build solved in 3 minutes in round 5 by spotting decimal commas and converting nm to wavenumbers. Uncensored spent 15 minutes sweeping laser wavelengths of 3103/3140/3164 nm trying to make G and 2D land simultaneously. It had raised the possibility that the x-axis was already in wavenumbers at turn 5 — and dropped it.

Operations notes

Zero wedge restarts in 14h45m (15 hours of continuous uptime). Three new things this round.

• 400-overflow total 3,938 — 12× the stock build’s entire round 5 (335). All of it from the two fallback loops. The server answered each 400 in under a second, so it wasn’t load; the real cost was 56 minutes of bench time.
• Lost 7 minutes on batch 8. While writing the script to rerun the 9 tasks that ended in ERR mid-round, I omitted the terminal-bench/ prefix on task names, and harbor died in one minute with “No tasks matched the filter”. The packaged dataset requires org/name form.
• One memory warning. At 17:23, while dna-assembly was at ctx 99K: free 0.3 GB, 16% headroom. Even with 8-bit KV, a 65K cache on top of a 70 GB resident model puts this 128 GB machine at the swap edge — the same thing regex-chess at 122K did in the stock build’s round 5. Recovered within 10 minutes of the task ending.
• Unrelated to the bench, a Spotlight index merge (mds_stores) overlapped with cloud-drive hydration for much of the day and pushed load average past 13. Decode speed was unaffected.

Next

5 vs 4. Uncensored round 5 is running on the remaining 31 now. When it finishes, both sides will be at the official 5-trial count and this series’ table can finally close. The stock build flipped 4 in its fifth round; for Uncensored to close a gap of 6, it has to flip tasks from the 21 that stock never solved.

One-liner: same conditions, fourth try — Uncensored recovers 2 to reach 58. Still 2 behind stock at equal tries (4 vs 4), and two fallback loops that fired requests into a wall for 28 minutes each say the harness needs fixing before the model does.