Measuring models
Benchmarks tell you how a model does on someone else's code. forge records every run you make, so over time you know how each model does on yours.
The run dataset
Run folders get deleted, so every run also records its inputs and outcomes in ~/.local/share/forge/dataset/, outside every repo: the models it resolved, prompts, each attempt's diff, verdicts and findings, tokens, and whether merged work held up later. FORGE_DATASET=off turns it off; deleting a run keeps what it taught the dataset.
forge dataset status # size, counts, anything that looks incomplete forge dataset forecast "$PLAN" # likely tokens, time and first-try pass rate, per task forge dataset routing # pass rates per model and effort, per difficulty forge dataset reviewers # how often each reviewer's PASS was later reverted or rewritten forge dataset outcomes # which integrated lines survived, from git history
What it's used for
- Forecasts
- Before a run dispatches, forge prints a forecast from past tasks at the same difficulty and model, and flags tasks unlikely to pass first time. Advisory only.
- Exploration
- Now and then a task you left to forge gets the neighbouring tier's model from your own pools, marked in the table. That's how routing data gets fair comparisons.
- Reviewer report card
- How often work each reviewer passed was later reverted or rewritten. It says "not enough data" rather than make up a rate from a handful of tasks.
- Paired reviews
- With
--shadow-qa, a sample of reviews gets a second reviewer.forge dataset pairsreports agreement by model, family or effort, and what each side found and cost. - Finding facts
forge dataset factsreplays candidate blocking rules on past reviews, such as "existing problems don't block", and lists what each would have let through.
forge-ui's Models screen shows the same numbers as a leaderboard. It warns, as you should remember, that routing is observational: a model sent the hard tasks looks worse than it is.
Decision gates
forge decide can put a small external model at a few decision points: scope drift, a vague prompt, a retry that didn't address its findings, which merge broke verification. Each gate is off, shadow (it answers, nothing acts) or on, and turning one on requires it to have passed a replay against past runs. Off by default; each gate sends a narrow slice of the run to the provider you choose, and a repo can be set to never send.
forge decide status forge decide gate drift shadow
Full reference Every flag and edge case: docs/dataset.md