Engineering

Running Terminal-Bench overnight against Qwen3.8 Flash Next

Measuring how much real terminal work a local LLM on an M5 Max can actually complete. The pass rate was less interesting than the server wedging four times and recovering itself every time.

삽질하는개발자

Summary of an overnight Terminal-Bench run on Qwen3.8 Flash Next — 70/89 completed, 51.4 percent pass rate, timeouts the leading failure cause at 13, and four mlx-serve wedges all recovered by the watchdog

In the previous post I wrote about getting Qwen3.8 Flash Next onto an M5 Max and making it fast. Being fast and being good at the work are different things, though, so I wanted to measure how much terminal work this local model can actually complete. I ran Terminal-Bench overnight.

What Terminal-Bench is

It hands an agent a real terminal environment (a Docker container), gives it a task — fix this, build that — and grades the result with a verification script. There are 89 tasks: recovering git repositories, configuring servers, building compilers, implementing algorithms. I ran it with the terminus-2 agent using this local model as its backend.

Results first

70/89 completed · 36 passed · 51.4 percent

Overnight it graded 70 of the 89 and passed about half. Hard tasks ate enough time that it never finished the full set.

Honestly, the pass rate itself is hard to read much into. It is not the complete sample, and it is a local 4bit model being given a benchmark designed for frontier cloud models. Two patterns were more interesting than the number.

Passes and failures split by kind

Passed: git repository recovery, web server configuration, issuing a self-signed certificate, data merging, log summarization, simple polyglot interop. Broadly, work where following a defined procedure exactly is enough.

Failed: building DOOM for MIPS, booting QEMU, implementing chess in regular expressions, Raman spectroscopy fitting, a metacircular evaluator in Scheme. Clustered in build/compile, numerical algorithms, emulation, and regex work.

Digging into the failure causes, more than half were timeouts. For reference, a side-by-side comparison of this model against 27B at 4bit and 8bit is in Qwen3.8 27B — 4bit vs 8bit vs Flash Next.

25 failures (overnight portion)
  timeout            13
  wrong answer        9
  execution error     3

Each task has a 15–60 minute limit, and 13 were cut off before finishing. If the model grasps the problem it solves it in minutes; if not, it wanders and burns the clock until it fails. Anything not done within about 10 minutes generally failed — a pattern that repeated across both runs. install-windows-3.11 sat at 100 percent CPU for 65 minutes before timing out.

The real story was on the server side

Watching through the night, what concerned me was not the pass rate but the server. mlx-serve stopped four times.

The symptom is distinctive. Server state reports ready but it cannot process any request. The log shows only <- 0+0 tokens, RSS drops sharply, and CPU sits near zero. I call this a wedge. There is no fix but a restart.

All four had something in common: they hit immediately after processing a large request of roughly 50,000 tokens. The schemelike-metacircular-eval task in particular produced an unusually large trajectory, and that one task triggered two consecutive wedges later in the run.

The watchdog recovered all four

Having hit this wedge in an earlier run, this time I attached a monitoring script. It pings the server every two minutes and, after two consecutive failures, kills and relaunches it.

# check every 2 minutes, restart after 2 consecutive failures
while true; do
  if curl -s --max-time 45 .../v1/chat/completions -d '{...,"max_tokens":4}' | grep -q choices
  then fail=0
  else fail=$((fail+1))
    if [ $fail -ge 2 ]; then
      pkill -9 -f "mlx-serve serve"; sleep 3
      rm -rf ~/.mlx-serve/kv-cache
      nohup mlx-serve serve ... &   # same production options
      sleep 60; fail=0
    fi
  fi
  sleep 120
done

Overnight, this watchdog recovered all four unattended. In the earlier run I had to restart manually after each wedge; this time I woke up to find it had died and revived four times while the benchmark kept running. The longest stretch without a restart was 4 hours 51 minutes.

There was one trap. If the watchdog’s restart command omits the production options (--kv-quant 8 and the rest), the server comes back untuned after every restart. Keeping the restart command identical to the real launch options mattered.

The pings created fake wedges

One thing confused me while reading the logs. <- 0+0 tokens appeared more than 30 times overnight, but there were only four actual wedges.

The rest came from the watchdog’s own pings. When the server is busy with a long request, the ping waits 45 seconds behind it and cancels itself, which is recorded as 0+0 tokens. To distinguish a real wedge (RSS collapse + consecutive 0+0 + server restart) from that, you have to look at server uptime alongside it. If uptime is intact, that 0+0 is just a cancelled ping.

Summary

A local 4bit model passing half of a terminal agent benchmark was better than I expected. What held the pass rate down was failing to finish hard tasks in time, and that is a model capability problem — not something infrastructure can fix.

The infrastructure side, on the other hand, taught me something concrete. I still have not fully identified why mlx-serve occasionally stops after a large request, but not being able to prevent the stop turned out not to matter, because reviving it automatically was enough to make unattended overnight runs possible. That was the real result of this run.

(Measured 2026-09-02. The pass rate is based on 70 graded tasks rather than all 89, so read it as a trend.)

More in Engineering