Uncensored Qwen3.8 Flash Next: all 89 Terminal-Bench tasks

The previous post was an interim report at 70 tasks. This one is the final record: all 89 tasks finished, then the failures rerun twice under changed conditions. It started on the night of 08-31 and ended on the evening of 09-03.

One thing needs to be said first. The previous post left it out.

This is not an official Qwen model

The model under test is ARC4NUM/Qwen3.8-Flash-Next-Uncensored-MLX-Serve-4bit. The word Uncensored is the point. It is an abliterated derivative of the Flash Next that the Qwen team released, with the refusal direction removed, then requantized by a community user to 4bit for mlx-serve. It is not an official Qwen release.

Abliteration finds the direction vectors inside the model associated with refusals and removes them. Because it edits the original weights, even the publisher does not know exactly how much capability was lost. So the scores in this post are not Flash Next scores. They mix 4bit quantization loss with abliteration loss, and nothing in this experiment separates the two.

I picked it because, when the experiment started, it was the only Flash Next 4bit that ran on mlx-serve with MTP out of the box. It was used for research only, and I will come back to this in the conclusion.

Test setup

Timeouts differ per task. Some have a 900 s base (raman-fitting, qemu-startup, tune-mjcf and others), some 3600 s (regex-chess, train-fasttext, video-processing and others), and the rest 1800 s. --agent-timeout-multiplier scales that base. The 0.5× and 1.0× below refer to this multiplier.

Full timeline

The conditions changed four times over three days. Here is the whole thing in one table, then step by step.

The single-pass score is 41/89, 46.1%. Adding the 15 tasks recovered in the two reruns gives a best-of score of 56/89, 62.9%. Those two numbers mean different things, and the conclusion treats them separately.

Step 1: overnight, halving the context

The first night is covered in the previous post, so only the setting changes here.

The server started at --ctx-size 131072 with the harness max_input_tokens at 110000. The server stalled twice, at 22:03 and 01:35. It reports state: ready but cannot serve even a 14-token request, and RSS collapses from 70 GB to 3.5 GB. Both times it happened on the first prefill of a new task, right after the hot cache reached 15 GB. This post calls that symptom a wedge.

One cache entry scales with context length, so at the 01:35 recovery I cut --ctx-size to 65536. The next four hours had no wedge.

Cutting the server while leaving the harness limit at 110000 created a new problem. terminus-2 trusts max_input_tokens, computes “free tokens 53104”, sends the request, and the server rejects it for exceeding 65536. The agent summarizes the conversation, resends, gets rejected again. In dna-assembly this loop ran 55 times, and at 04:42 I cancelled batch 5 outright. Its five tasks went into the “remaining 19” later.

Lowering max_input_tokens to 55000 made batch 6 clean, with zero fallbacks. One lesson: if you change the server ctx-size, change the harness model_info with it.

Step 2: the main run, wedges kept coming

The restart at 20:05 on 09-01 failed within a minute: batches 7 through 13 all logged “finished” instantly. Colima was down, so there was no Docker daemon. I brought it up and restarted at 20:08.

This time a watchdog was attached: ping every two minutes, restart the server after two consecutive failures. It caught four wedges on 09-02, at 01:24, 03:12, 08:14 and 08:36. So shrinking ctx to 65536 did not remove wedges. It avoided the 15 GB cache condition, but the pattern of the next request stalling right after a request of around 50K tokens stayed.

Batches 7–16 passed 23 of 48. With the overnight results that is 36 of 70, the 51.4% from the previous post. Three of those 70 were not the model’s fault, though. nginx-request-logging and model-extraction-relu-logits ended in RuntimeError because docker compose could not bring up the environment, and sanitize-git-repo was cut off at the deadline with CancelledError.

Step 3: the remaining 19 at half timeout

Three tasks lost to wedges in batch 2, five from the cancelled batch 5, and eleven that never got their turn after batch 16 made 19. To save time I ran them at 0.5× from 10:36 to 15:21 on 09-02. Five passed, ten timed out, four failed.

That completed the single pass over 89 tasks: 41/89, 46.1%. It is lower than the 51.4% of the previous post because the last 19 were concentrated on hard tasks and had half the time. The 48 failures break down as 29 timeouts, 16 finished-but-wrong, and 3 infrastructure errors.

Step 4: rerunning the 48 failures with changed settings

With more than half the failures being timeouts, I wanted to know whether the harness settings were holding the model back or whether it was a capability limit. I took only the 48 failures and changed three things.

• Server ctx back to 131072, max_input_tokens up to 120000, to keep long conversations without summarizing.
• reasoning_effort from medium to high, temperature from 0.7 to 0.3.
• Timeout left at 0.5×. A compromise to finish 48 tasks in one night.

It ran from 15:25 on 09-02 to 01:59 on 09-03, and nine tasks flipped to PASS.

Timed out but scored PASS is possible. Harbor grades the container state at the moment the agent hits the limit. If the files are already correct, it is a PASS.

nginx-request-logging was an infrastructure error in the main run, so the real flips are eight. Whether that came from lower temperature and higher reasoning, or simply from rolling the dice again, this experiment cannot tell. One attempt per task means the variance was never measured.

Wedges continued during this rerun: three, at 20:55, 21:02 and 23:23. The first two are seven minutes apart. The same large request came back right after the restart and stalled it again.

Step 5: the remaining 39 at full timeout

The 39 tasks that still failed in rerun 1 were run again with the same settings but the timeout back at 1.0×. From 02:02 to 18:36 on 09-03, sixteen and a half hours. Six flipped.

The other 33 moved like this.

• Timeout → timeout, 23 tasks. Eight of them went past 100K context: path-tracing-reverse 130K, mteb-leaderboard 126K, make-mips-interpreter 123K, regex-chess 122K, schemelike-metacircular-eval 118K, circuit-fibsqrt 116K, dna-assembly 109K, custom-memory-heap-crash 95K.
• Timeout → fail, 3 tasks. compile-compcert 26.8 min, install-windows-3.11 41.2 min, sanitize-git-repo 3.4 min. Twice the time, and they finished wrong.
• Fail → fail, 5 tasks. dna-insert, extract-elf, query-optimize, sparql-university, train-fasttext (41.7 min).
• Fail → timeout, 2 tasks. protein-assembly, video-processing (130K).

Four wedges in this phase: 07:43, 10:45, 13:13 and 14:21.

The new failure mode 131K context created

Restoring ctx to 131072 surfaced a problem distinct from wedges. Since 01:24 on 09-02 the server log holds 5,266 responses of prompt N tokens exceeds ctx_size 131072, almost all of them from the two reruns.

Once the agent fills the conversation to around 120K, the next request exceeds 131072. terminus-2 tries to summarize, but the summarization request itself exceeds the context and fails. The log repeats Even fallback chat failed. That line appears 1,386 times in the rerun 2 log. From there the agent does nothing until the timeout.

video-processing looked stranger still. From the point its context passed 92K until 112K, it produced the identical 620-token output every turn. Same command, unchanged, over and over. Then it crossed 131K, got a 400, and that was it. This long-context degradation is why none of the eight tasks that went past 100K ever flipped.

13 wedges, 16 false alarms

Across three days the server stalled 13 times. Two manual recoveries on the first night, eleven automatic ones by the watchdog. All the timestamps:

Two patterns. One is the next request stalling right after a long output at 60K+ context. The other is memory pressure: it happened with RSS down near 50 GB and free memory at 0.4 GB or less.

The watchdog also logged 16 cases where one ping failed and the next succeeded. Those were the server busy with a large prefill, unable to return four tokens within 45 seconds. Without the two-consecutive-failures rule it would have killed a healthy server 16 more times.

One more note on memory. Swap went from 17.4 GB at the start to 29.7 GB three days later. In the middle, by my own mistake, another inference server on the same machine loaded a 20 GB model. Free memory dropped to 1.0 GB and the mlx-serve RSS was paged out to 1.4 GB, but I cleaned it up within seven minutes and the server never dropped a request. Lesson learned the hard way: there is no room for anything next to a 70 GB model.

How to read the numbers

If you ask whether 41/89 (46.1%) or 56/89 (62.9%) is this model’s score, both come with conditions.

The single pass of 41 was not run under uniform conditions: 70 tasks at 1.0×, 19 at 0.5×, and 3 infrastructure errors. The best-of 56 counts any task that passed once in up to three attempts, so it must not be compared with official leaderboards. The honest version is: this model lands around 46% on this harness, and can reach 63% given more time and tuned settings.

15 of the 48 failures (31%) flipped in the reruns, and every flipped task stayed at or below 70K context. Conversely, tasks that finished within the time and were wrong never flipped even with double the time. Tasks that went past 100K context never flipped either. So what time and settings can recover is timeouts on short tasks; the rest looks like a model-side limit.

Whether that limit comes from 4bit quantization or from abliteration is the open question. As said at the top, this is an Uncensored derivative and its score must not be read as the original Flash Next’s. I am preparing to run ddalcu/Qwen3.8-Flash-Next-MLX-Serve-mixed-4-8bit (experts 4bit, attention 8bit, MTP kept), which has no abliteration, on the same harness. The 107 GB download is in progress, and all 89 tasks under the rerun 2 conditions will take 35–40 hours. Only with that result can the two losses be separated.

If I did this again

• Server ctx and harness max_input are a pair. Change one alone and you get a summarization loop.
• Shrinking ctx does not remove wedges. Attach the watchdog from the start, and keep the two-consecutive-failures rule.
• Halving the timeout mostly costs you short tasks. If time is short, cut the task count instead.
• Opening 120K context gave this model nothing. Every task that filled past 100K failed.
• One attempt cannot measure variance. I do not know whether the eight flips were settings or luck.