A private benchmark for coding agents
The referee.
Point it at your repository. Guignet mines your own git history into replayable, validity-checked tasks, runs any model, agent or config against them in isolated worktrees — entirely on your machine — and hands you the only leaderboard that matters: on your code.
Guignet’s green · hydrated chromium oxide
Leaderboard
Measured where?
- 01Frontierhigh effort——unpriced72.1%
- 02Frontierstandard——unpriced69.4%
- 03Mid-tierhigh effort——unpriced63.0%
- 04Mid-tierstandard——unpriced58.7%
- 05Open-weights70B——unpriced44.2%
Standardise on Frontier · high effort.
Everyone is choosing on someone else’s evidence.
01 — Measured on other people’s repositories
Public benchmarks
SWE-bench and its descendants are saturated, partially gamed, and scored against codebases that are not yours. The deltas demonstrably do not transfer to your stack, your conventions, your weird build.
02 — Requires the upload you cannot make
Eval SaaS
Every hosted evaluator begins by asking for the source. That upload is a non-starter for exactly the organisations spending the most on AI tooling — which is to say, the ones with the most to measure.
03 — A large decision made on vibes
The net effect
Models, agents and spend tiers get chosen on marketing plus a feeling, annually, at six figures. The buyer holds the budget and has no instrument. That absence is the whole opportunity.
mine → gate → run → score → report
Five stages, each idempotent and resumable. Every stage writes to disk and can be re-entered, so interrupting a run costs nothing but the attempt in flight.
- 01
guignet mine
History becomes tasks
Walks your git history for commits that carry a fix and the tests that prove it, and rebuilds each into a prompt, a base commit, a held-out fix and a held-out verifier.
- 02
guignet gate
Reality gets replayed
A task is admitted only once the verifier fails at base and passes at the real fix, k times each, in a clean worktree. Anything that flakes is discarded.
- 03
guignet run
Agents take their turn
N attempts per task, each in a disposable worktree, through an adapter — Claude Code headless, or any command at all via the generic escape hatch.
- 04
guignet score
The verdict is binary
Each solution meets the held-out verifier. Pass or fail, no judge. Cost is parsed from the harness’s own transcript and never self-reported.
- 05
guignet report
One file, offline
A self-contained HTML report — no framework, no CDN — plus a --json twin of every number in it. Regenerates from stored runs at any time.
Most of your history is not a benchmark.
A reconstructed task is admitted only after Guignet proves it reproduces reality: the held-out verifier must fail at the parent commit and pass at the real fix, twice each, in a clean worktree running your own setup and test commands.
Anything that flakes is discarded, never patched. Discarding is the feature — it is the entire reason the surviving suite can be trusted.
87
admitted of 312 candidates
27.9%
soundness rate
Discarded — by recorded reason
- Flaky verifier
- 96
- No reproduction at base
- 71
- Verifier already passed
- 34
- Setup failure
- 16
- Timeout
- 8
The model may have trained on your history.
Unhandled, that inflates every score and destroys trust in the whole result. Guignet handles it four ways — and reports all four, whether or not they flatter the outcome.
- 01Cutoff splits Reported always
- Every task is dated, and every score is split by each candidate model’s training cutoff. The post-cutoff column is the clean measure, and it is the headline.
- 02Regurgitation detection Flag rate per config
- Solutions whose token-level similarity to the held-out fix runs suspiciously high on pre-cutoff tasks are flagged. A verbatim fix is memory, not skill.
- 03Private-era preference Corpus selection
- For repositories with a public-to-private transition, Guignet prefers commits from the era no crawler ever saw.
- 04Mutation mode Optional
- Symbol and path renames applied consistently across base and verifier, breaking memorisation while leaving the task itself intact.
Five commands, one config file.
Drop a .guignet/config.json into the target repository and run the pipeline. Every command accepts --json and prints exactly one object on stdout.
- exit 0
- ok
- exit 1
- failure
- exit 2
- usage
- exit 3
- soft-blocked
$ guignet doctor # validate the repo + suite discovery ✓ bun 1.3.14 ✓ git worktrees ✓ testCmd resolves $ guignet mine # reconstruct tasks from git history 312 candidates discovered $ guignet gate # replay validity → the admitted suite ✓ 87 admitted ✕ 225 discarded (flaky, no repro, setup failure) $ guignet run --config run.json$ guignet score$ guignet report → .guignet/reports/2026-08-02/guignet-report.htmlNotes — on host citizenship
A benchmark run is background work, and Guignet treats your machine that way. Every subprocess it spawns — agents, installs, verifiers — runs at reduced scheduling priority, so your foreground always wins under contention.
- 1.
Reduced priority by default. taskpolicy -c utility on macOS, nice on Linux. Opt out with host.priority: "normal" on a dedicated box.
- 2.
Host-aware run pool. Admission reads CPU load average and the kernel’s own memory-pressure verdict — never a self-invented heuristic.
- 3.
Degrades, never stalls. Critical pressure holds concurrency down to sequential progress. No freeze, no stall, no lost work.
- 4.
Fails open where it must. On a host with no pressure signal the memory gate quietly disarms and the CPU gate stands alone.
Which model should we standardise on — proven on our code?
The harness is MIT and yours to run. When you would rather have the answer than the instrument, the Model Selection Report is a fixed engagement: Guignet runs on your repositories, and you get the leaderboard, the recommendation and the tuned configs.