Model Price Watch
I built a pipeline that watches what every model I actually run in production costs, and tells me the moment a cheaper one is trustworthy enough to swap in. It checks pricing daily across the providers I use, flags genuine drops against three live roles (a SonarQube auto-fix routine, wiki enrichment, and an app's chat worker), and never switches anything itself; it notifies me and I decide.
A price drop alone isn't a recommendation. Before I ever see one, a candidate has to clear a live reachability and tool-calling smoke test (the same privacy gate that keeps untrusted providers away from anything but code), a free intelligence-index pre-filter, and a fixed task-set evaluation judged by Claude against the incumbent, capped at $5 a week so a bad week of candidates can't run up a bill.
A daily job diffs the provider pricing API against the previous day's snapshot, so a price event is caught the moment it happens rather than on some arbitrary review cadence. A weekly run checks whether that diff plus a Claude judge call found anything worth escalating; a bi-weekly forced backstop re-runs the deeper tiers even on a quiet week, so a real opportunity can't go silent just because the price event that triggered it aged out of a seven-day lookback.
The part that actually matters is the last mile. A lightweight Claude-judged eval is a filter, not a verdict, so before I trust a switch for a routine that auto-merges unattended, I run it through a separate bake-off harness that measures real patch success against a frozen fixture and its actual build gate, not a text comparison. One candidate looked clean on the cheap eval and then silently dropped a whole file's worth of fixes in the real bake-off; it stayed rejected. Another held up under the same real measurement, about 30 times cheaper and 4 times faster than the model it replaced with no reliability regression, so that one's live in production now.
- Python
- OpenRouter
- Artificial Analysis API
- Claude Sonnet 5 (judge)
- InfluxDB
- Uptime Kuma
- launchd