SpecJudge

Most teams pick their coding model the same way every time: whatever the largest model they have access to, applied uniformly across a task list that mixes one genuinely hard architectural decision with twenty mechanical edits. That habit fails in both directions at once, because the oversized model quietly bills frontier prices for renaming variables while the undersized one burns an afternoon of retries on the single task that actually needed the capability, and in neither case do you find out until the work is done. SpecJudge is a local-first CLI that puts that decision before implementation instead of after the invoice.

What it does

Point it at a project that already has Spec-Driven Development artifacts (constitution, spec, plan and tasks, as produced by spec-kit) and it reads the whole task set in a single pass, works out what the project demands across several dimensions, and compares that demand against a catalog of models and their current prices. The result is a ranked table in your terminal and, if you want it, an HTML report with the full matrix in your browser. When a project has no artifacts at all, SpecJudge says so rather than inventing a recommendation from nothing, and when it has some but not enough to judge confidently, it tells you that too.

$ specjudge examples/task-manager

                   Model comparison (SpecJudge) - judge: devstral-small-2
┏━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃    ┃ Model                         ┃ Rating   ┃ Price                       ┃ Priced on  ┃
┡━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ 🥇 │ Mistral Small 3.2 24B (local) │ good     │ open-source/free            │ 2026-07-20 │
│ 🥈 │ DeepSeek V4 Flash             │ good     │ 0.28 out / 0.14 in USD/1M   │ 2026-07-28 │
│ 🥉 │ Gemma 4 26B MoE               │ good     │ 0.33 out / 0.06 in USD/1M   │ 2026-07-20 │
│    │ Claude Fable 5                │ overkill │ 50.00 out / 10.00 in USD/1M │ 2026-07-28 │
└────┴───────────────────────────────┴──────────┴─────────────────────────────┴────────────┘

The rating scale is deliberately coarse — poor, regular, good, overkill — because the decision you are actually making is "is this model enough" versus "am I overpaying", and a two-decimal score would be false precision dressed up as rigour. Prices live in a YAML catalog with the date they were checked attached to each entry, and SpecJudge warns you when the figures behind a recommendation have gone stale.

Why the judge runs on your machine

The judge is a model you already have installed in Ollama. On the first run SpecJudge lists your installed models and asks which one you want to use for judging, and from that point on nothing about your project leaves your hardware. This is not a privacy checkbox added for marketing reasons: specs are usually the most sensitive artifact in the repository, because they spell out the architecture, the business rules and often the client by name long before any of it reaches the code. SpecJudge requires Ollama 0.5.0 or newer.

How the judge is kept honest

This is where most of the engineering went, because "a local model said so" is not a recommendation anyone should act on. Every dimension the judge rates has to quote a fragment of your spec that genuinely exists in the text, and that citation is then verified deterministically against the source rather than taken on trust, so a fabricated quote causes the entire evaluation to be discarded instead of being handed back to you with a plausible-looking number attached to it. Dimensions the judge cannot ground in the text come back marked unsupported and drop out of the fit calculation altogether, rather than being silently treated as easy, which is precisely the failure mode that makes tools like this recommend something too small.

The judge also has its own regression suite, built on twelve reference projects and run at two levels: a deterministic layer that runs in CI, and a local eval_judge.py for the parts that need a real model behind them. It has already paid for itself by surfacing a bug where 8B judges were failing on every single project because of JSON formatting, which now passes nine out of nine after the schema became a hard requirement rather than a suggestion. Sampling is pinned, so running SpecJudge twice on the same project gives you the same answer twice.

Version 0.3.0: usable as a library, not just a CLI

The most requested change after the first release was being able to build on top of SpecJudge rather than parsing its terminal output, so 0.3.0 ships a public Python API (from specjudge import api, then api.analyze(...)) with sixteen documented names in docs/api.md and everything else under specjudge.* declared explicitly internal, which means the internals can keep moving without breaking anyone downstream. The JSON contract is versioned independently of the package version, carries a schema_version in every payload, and the real JSON Schema is obtainable with specjudge --print-schema: adding a field is a minor bump, removing one or changing what it means is a major bump. Errors now carry exit codes that distinguish "this project cannot be judged" from "your environment is broken", which matters as soon as you put SpecJudge inside a pipeline. None of this breaks anything from 0.2.0.

What is not there yet

Multi-provider support is the honest gap: the judge currently runs through Ollama only, and running it against any OpenAI-compatible endpoint (OpenRouter and friends) is scoped in issue #4, with local remaining the default and remote strictly opt-in with a runtime warning, because sending your specs to a third party should never happen by accident. Input is also still tied to the spec-kit layout, and reading CLAUDE.md, AGENTS.md and other SDD conventions is the next direction I want to take it.

Contributing without writing any code

The most useful contribution to SpecJudge needs no Python at all, because the model catalog and the prices are plain YAML files that anyone can edit: adding a model that came out last week, or correcting a price that changed, is a one-file pull request. The recommendations are only ever as good as that catalog, so keeping it current is genuinely the highest-leverage thing anyone can do here.

If you have run this on a real project, I would like to know where the recommendation felt wrong, and which SDD formats beyond spec-kit you would want it to read.

MIT licensed. Install with pip install specjudge, source at github.com/JoaquinRuiz/SpecJudge. Built by Joaquín Ruiz, who also explains the whole thing on video (in Spanish).