Local Model Auto-Eval

I built a daily watcher that checks HuggingFace for new releases from publishers I trust and matches them against the two local model lanes I actually run in production on the Mac Studios: Qwen for agent and briefing work, Gemma for edge extraction and summarisation. It's the free half of Model Price Watch's swap logic; a new model release triggers the check instead of a price event, and the question is whether hardware I already own can run something better on it than it's running today.

A match still has to clear real hurdles before I hear about it. It has to fit the Metal memory ceiling once the incumbent it would replace is stopped, and it needs an MLX build available; conversions lag origin releases by days, so a candidate that clears size but has no MLX build yet sits in a pending queue and gets rechecked for up to 14 days before I drop it. Only then does it move to the real test: stop the incumbent's launchd job, download and serve the candidate, run it against the same seven production-faithful tasks the incumbent handles for real (inbox classification, edge extraction, judge calls, music prose, news scoring, summarisation, tool-calling), score it, then restart the incumbent.

The watcher runs daily at 03:35, diffing HuggingFace's model API against the previous day's snapshot so nothing depends on a review cadence. Uptime Kuma gets a maintenance window for the swap, so a legitimate stop and restart doesn't fire a false DOWN alert, and the whole sequence, SSH over a dedicated health-recovery key, stop, download, serve, evaluate, teardown, restart, has to complete inside that window. Every remote call in the sequence counts against the same wall-clock budget, not just the two big waits; an earlier version of that budget missed five one-shot SSH calls worth 390 seconds between them and still passed.

The eval budget itself moved from 20 minutes to 40 after a live dry run against the Gemma lane ran past the original ceiling on a 2B candidate against the real task set, including one roughly 20,000-token scoring task; the run failed closed rather than fudge a result. A stale lockfile recovers itself, an unsafe HuggingFace model id gets rejected before it ever reaches a shell command, and a notification outage doesn't change what the eval does. The most recent candidate through the pipeline, a 30B model, ran both a budget-starved pass and a fixed-context pass; both sets of results are in the repo.

• Python
• MLX
• HuggingFace API
• launchd
• SSH (cross-host)
• Uptime Kuma API