The price of a 6x smaller model — Qwen3.8-27B 4bit on Terminal-Bench 89, single pass 35 vs 45

Summary

In the September 1 post I called Qwen3.8-27B 4bit “the one to use if memory is tight, quality is on par.” That verdict came from a handful of short prompts. Having closed the Terminal-Bench series, there was no reason not to drop this model into the same harness.

ddalcu/Qwen3.8-27B-MLX-Serve-4bit (17GB, affine 4bit g64, built-in MTP head), all 89 tasks, one pass, identical conditions to the two Flash Next models (terminus-2, ctx 120K, temperature 0.3, reasoning xhigh, timeout 1.0×). 31 hours 11 minutes.

Result: 35 / 89 (39.3%) — 31 PASS, 4 PASS(TO) (file was already correct when the timeout hit), 16 FAIL, 38 TO, 0 ERR.

Single pass (pass@1)	Passed	Rate	Wall time
Flash Next stock mixed-4-8bit (100GB)	45 / 89	50.6%	24.7h
Flash Next Uncensored 4bit (ARC4NUM)	41 / 89	46.1%	—
27B 4bit (17GB)	35 / 89	39.3%	31.2h

For reference, the two Flash Next builds reached pass@5 of 64 and 59. The 27B has run once, so it doesn’t belong in that table; this post compares single pass to single pass. Whether to run it five times is discussed at the end.

One line: one-sixth the disk, ten fewer correct answers, and no speed gain. What those ten answers were is the body of this post.

One correction on conditions

Earlier posts said reasoning_effort=high; this run uses xhigh. That looks like a different setting, but measured directly, mlx-serve 26.8.11 maps high — which the Qwen3.8 template doesn’t know — to xhigh (identical prompt token count, 53). So the whole series actually ran at xhigh, and this run just says so explicitly. Conditions are equal.

Architecture, for the record. Both are hybrids at 3 linear-attention layers to 1 full-attention layer. The 27B is 64 dense layers (16 full attention); Flash Next is 48 MoE layers (10 of 512 experts active, 12 full attention plus an attention indexer). By active parameters the 27B is roughly 4× Flash Next (~6B), but by total parameters and disk it is one-sixth.

Where they split

Overlaying the three single passes:

Group	Count	Tasks
All three pass	24	the short ones — fix-git, git-leak-recovery, openssl-selfsigned-cert, kv-store-grpc, …
Both Flash Next pass, only 27B fails	10	bn-fit-modify, db-wal-recovery, distribution-search, fix-ocaml-gc, mailman, merge-diff-arc-agi-task, mteb-retrieve, polyglot-c-py, sqlite-with-gcov, torch-tensor-parallelism
Both Flash Next fail, only 27B passes	6	build-cython-ext, code-from-image, count-dataset-tokens, extract-elf, password-recovery, tune-mjcf
One Flash Next passes, 27B passes	5	filter-js-from-html, overfull-hbox, pytorch-model-recovery, reshard-c4-data, financial-document-processor
One Flash Next passes, 27B fails	13	caffe-cifar-10, crack-7z-hash, mcmc-sampling-stan, qemu-startup, …
All three fail	31	the 21 nobody solved in five rounds + regex-log, torch-pipeline-parallelism, raman-fitting, …

The gap of 10 (stock 45 − 27B 35) breaks down, task by task, as 9 the 27B solved that stock didn’t and 19 stock solved that the 27B didn’t. Single-pass comparisons wobble this much — the series showed a 19-task spread between pass@1 and pass@5.

So I overlaid the pass@5 results too. Of the 27B’s 35, 34 sit inside stock’s pass@5 set; the one exception is filter-js-from-html (stock failed it five times, Uncensored passed). Against Uncensored’s pass@5, extract-elf and tune-mjcf were Uncensored’s five-time failures. Of the 21 tasks outside the union of both Flash Next pass@5 sets (68 tasks), the 27B solved zero. The 27B opened no new doors. Still worth noting: a model one-sixth the size got a few “big model needs five tries” tasks on the first try.

How the six 27B-only passes went:

tune-mjcf (14.2 min, 30 turns) — a 15-minute task: make a MuJoCo cable simulation run in ≤60% of reference time with identical final state. Stock Flash Next timed out on pass 1 and got it on pass 2; Uncensored failed all five. The 27B profiled at turn 8, found the solver was the bottleneck, tried Newton then PGS, and landed solver="PGS" — one line, 2.18× (46%). Solved on the first try, with 48 seconds to spare on the limit.
extract-elf (PASS(TO), 15.0 min, 35 turns) — extract memory values from an ELF. The example shows 0x400000 but the binary’s .data is at 0x4000, so it spent 30 turns digging through headers, then in the last three turns switched to “emit every address scheme at once.” The timeout cut it off, but the file on disk at that moment passed. It didn’t understand the answer; it won on coverage.
code-from-image (11.5 min) — pseudocode inside a PNG, no PIL, no tesseract. It pip-installed easyocr, cropped line by line, recovered SALT = b'0000TBENCH-SALT', computed the sha256 chain, checked its own answer against the hint (“starts with bee26a”) and stopped.
count-dataset-tokens (10.4 min) — the README returned 401, so it read the cached copy, narrowed “science domain” to biology·chemistry·physics (26 rows), and answered 79,586.
password-recovery (5.9 min) — found a ZIP local header (PK\x03\x04) inside a .dat file and pulled app/launchcode.txt out of it.
build-cython-ext (PASS(TO)) — port pyknotid to NumPy 2.x. “18 tests pass” at turn 24, then git diff got stuck in a pager for four turns and the timeout hit. The files were already right.

There’s no common thread among the six. If forced: “the search path happened to be right the first time” — and these are all tasks the big model eventually got within five tries.

38 timeouts — this time the star is the repetition loop

By time limit: 15 min 13 · 30 min 13 · 60 min 8 · 40 min 2 · 20 min 1 · 200 min 1. The 26 at 15/30 minutes are mostly the “died installing the environment” pattern seen with Flash Next (mcmc-sampling-stan compiled RStan for the full 30 minutes; adaptive-rejection-sampler installed R and got 11 turns). That’s been covered. What’s new this time is the same turn repeating dozens of times.

I counted runs of consecutive turns whose message was byte-identical to the previous one. 15 of the 38 timeouts contain a run of 10 or more.

Task	Repeats	Total turns	What the turn was
regex-chess	156	172	“The terminal appears stuck. Let me send Ctrl+C” — started at turn 14, ctx 21K, and sent Ctrl+C for the remaining 60 minutes
build-pov-ray	143	168	“SourceForge returns ‘no’. Let me try other mirrors” — the same wget 143 times, all of a 200-minute limit
custom-memory-heap-crash	64	141	see ② below
extract-moves-from-video	57	136	the same `grep -A 30 'FRAME 16'` frame check, 57 times
protein-assembly	41	62	41 attempts to reach the OAS database
break-filter-js-from-html	34	40	the same BeautifulSoup test script from turn 6 to the end
polyglot-rust-c	30	36	“`// #if 0` doesn’t work, I need a different strategy” — then writes the same file, 30 times
winning-avg-corewars · install-windows-3.11 · caffe-cifar-10 · path-tracing	24–31
distribution-search · polyglot-c-py · path-tracing-reverse	10–17		writes “I’ve been going in circles” and issues the same command

regex-chess is the archetype. At turn 14 some command ran long, the model decided the terminal was stuck and sent Ctrl+C; it looked at the resulting observation (an empty prompt) and decided the terminal was stuck. Same observation → same response at temperature 0.3 → same observation. A 324-token turn, 156 times, 60 minutes. build-pov-ray is more expensive: it’s a 200-minute task, so 3 hours 20 minutes went into one wget line.

The Flash Next series had repetition loops too. But across 467 runs the visible ones were mostly fallback loops after hitting the 130K context wall — not pure repetition starting at ctx 20K. The 27B cannot get out unless the observation changes on its own. polyglot-rust-c’s repeated message is telling: it analyzes “this approach fails, I need a different strategy” and then writes the same file. Analysis and action have come apart.

The bigger model in the same harness got caught less, so this is a model problem. But a harness rule as simple as “intervene if the command matches the last N turns” would have pushed most of these 15 into at least trying something else. The series’ closing line, “no repetition-loop detection,” becomes fatal with a small model.

② The 130K wall, twice. custom-memory-heap-crash (30 min, 141 turns) and video-processing (60 min, 747 turns). Both hit 130K context, got a server 400 (prompt 131,790 tokens exceeds ctx_size 131,072), and the harness spent the rest of the time inserting “Technical difficulties. Please continue” and retrying. video-processing alone produced 2,082 400-overflows, 94% of the run’s total (2,223). Fifth time this pattern has shown up in the series; nothing new, and the 27B was no different.

16 FAILs — “all verified,” and wrong

14 of the 16 FAILs finished in under 15 minutes. Time to spare, answer wrong, and the last turn almost always reads “all requirements verified.” A few of what the grader said:

mteb-retrieve (3.1 min, 8 turns) — return the document with the 5th-highest cosine similarity. It answered HumanEval: Benchmarking Python code generation…; the expected value was MTEB: Massive Text Embedding Benchmark. Confident submission at turn 8, ranking off by one.
sqlite-with-gcov (5.6 min) — final turn: “sqlite3 is in PATH.” Grader: sqlite3 not found in PATH. The PATH existed only in its own shell.
nginx-request-logging (3.5 min) — Status code missing in logs. It reported at turn 26 that $status was in the log_format; the actual log lines didn’t have it.
db-wal-recovery (9.7 min) — claimed 11 records recovered, but Apple should have updated value 150 from WAL. It emitted the original value (100) without applying the WAL update.
torch-tensor-parallelism (14.5 min) — ran its own world_size 1/2/4 tests and passed them; the grader said bias slicing and the allclose on outputs were off.
mteb-leaderboard (30.8 min, 63 turns) — stock got this on pass 1, Uncensored only on pass 5. Answered codefuse-ai/F2LLM-v2-14B (0.7110); expected GritLM/GritLM-7B. Like Uncensored four times over, it missed the “models with all 28 tasks present” filter.
configure-git-webserver (1.8 min) — 10 turns, “end-to-end test passes.” Grader: 404. It deleted the site along with its test data.
merge-diff-arc-agi-task, mailman, bn-fit-modify, dna-insert, sparql-university, raman-fitting, train-fasttext — same pattern. bn-fit-modify learned the wrong DAG outright; mailman spent 76 turns and still had the announce-message count wrong.

Flash Next had this type too (sparql-university’s dual nationalities, extract-elf’s 0.00%). The difference is frequency: 10 of the 27B’s 16 single-pass FAILs were stock’s single-pass PASSes. The rate at which “I verified it” fails to match actual verification goes up with a smaller model.

One more: regex-log (15 min TO, 5 turns). Turns 4 and 5 were each ~5,300-token responses — a Python test script stuffed with cases, written again at length right after saying “my previous attempt was too long.” Five turns, fifteen minutes. Output-length control is another thing that wobbles first in a small model.

Speed — 17GB was not faster than 100GB

This is what many people would run it for. Server request log (mlx-serve 26.8.11, MTP on, 8-bit KV, M5 Max 128GB), medians by prompt length, tok/s:

Prompt	Stock prefill / decode	Uncensored	27B 4bit
< 8K	162 / 73.3	234 / 57.8	206 / 53.9
8–32K	350 / 65.5	391 / 61.9	508 / 56.4
32–64K	378 / 50.8	324 / 53.0	407 / 41.1
64K+	173 / 53.3	248 / 44.0	301 / 33.3
100K+			256 / 27.0

On short prompts the 27B decodes at 74% of stock; past 64K, 63%. Prefill is actually faster on the 27B in the 8–64K range. The decode gap is simple: active parameters per token, 27B vs ~6B. The MoE’s “only 6B actually runs” beats a 4bit dense 27B. The widening at long context is presumably the 16-vs-12 full-attention layers plus Flash Next’s attention indexer — a guess, I haven’t taken the architectures apart.

In the agent loop: 27B median time per turn 0.36 min (stock 0.38, Uncensored 0.39), median output 301 tokens per turn (345, 334). Same. On the 27 tasks both the 27B and Uncensored passed, median time was 3.0 vs 2.8 minutes, with the 27B faster on 12. Exactly the series’ conclusion — agent time goes to command execution inside the container and to prefill, and decode differences get buried. What the small model buys is memory, and this machine didn’t need it.

Ops notes

Originally started 09-08 at 21:07, restarted from scratch at 23:49. Within the first five tasks decode halved from 43 to 20 tok/s — not the model but a Finder runaway (148% CPU, 7.6GB, fileproviderd at 87%). killall Finder fixed it instantly. I restarted to keep conditions clean; Finder behaved for the next 31 hours.
Server: 4,870 requests, 0 wedge restarts, 0 memory admission rejections, RSS pinned at 17GB. Started with --prefix-cache-mem 4GB as recommended in the Uncensored round-5 post: with 100K contexts resident the hot cache held its cap (2,601 evictions) and swap didn’t grow. The earlier 47GB swap incident was indeed the 8GB setting.
2,223 400-overflows, all from the two tasks above. No server load.
31.2 hours vs stock’s 24.7 — 6.5 hours longer. Not because the model is slower, but because timeouts were 38 vs 29. build-pov-ray’s 200 minutes alone is half the difference.

Run it five times?

Putting the 27B in the pass@5 table means four more rounds. Retrying only the 54 single-pass failures (16 FAIL + 38 TO) is roughly 20 hours a round — four days. The series’ lesson was +18–19 from pass@1 to pass@5, so the 27B would probably land around 50. But the 15 repetition loops above are likely to fall into the same observation → same response at temperature 0.3 on a retry, which makes the increment smaller than that.

Decision for now: on hold. If I run it, harness repetition detection comes first — same command with the same observation three times, intervene. Running four more rounds without it means handing build-pov-ray another 13 hours.

One line: 27B 4bit, single pass 35 vs 45. It opened no new tasks; the gap came from sub-15-minute FAILs that said “verified” and were wrong, and from pure repetition loops starting at ctx 20K; and the speed gain bought with one-sixth the disk was zero on this machine.