Evaluation
Measure review quality on AACR-Bench with ocra-eval.
ocra-eval replays AACR-Bench: 200 real pull requests from 50 open-source projects in 10 languages, with 1,505 expert-verified review comments as ground truth.
Running
export GEMINI_API_KEY=...
export OCRA_MODEL_STANDARD=google/gemini-flash-lite-latest
node packages/eval/dist/main.js list --limit 20 --max-change-lines 300 # free preview
node packages/eval/dist/main.js run --limit 20 --max-change-lines 300 --label baseline --max-cost-usd 5
node packages/eval/dist/main.js score .ocra/eval/baseline # re-score only| Option | Meaning |
|---|---|
--limit, --seed, --languages, --max-change-lines, --ids | Seeded, reproducible selection |
--label | Run name; rerunning the same label resumes |
--max-cost-usd | Stop starting new PRs once review spend reaches the cap |
--retry-failed | Review PRs again that failed in an earlier attempt of the same run |
--reviewers | Passed to ocra review --reviewers, for comparing reviewer sets |
--pr-max-cost-usd | Spend limit per PR, passed to ocra review --max-cost-usd; --max-cost-usd only stops starting new PRs, so use both to bound a run |
--mock-judge | Offline word-overlap judge for pipeline checks (not comparable) |
When a pull request fails because the model's quota is spent (for example a free-tier daily limit), the run stops starting new ones and marks them skipped_quota; run the same command later to resume them.
Reviews always run with --no-repo-config, so benchmark repositories cannot load plugins; models come from OCRA_MODEL_*. Repositories are cached as blobless clones under ~/.cache/ocra/aacr-bench/repos. PRs whose commits can no longer be fetched are reported as unavailable and excluded from scoring.
Recall ceiling (free)
node packages/eval/dist/main.js ceiling --limit 20 --max-change-lines 300Classifies every annotated issue by what ocra's deterministic stages decide, with no model call: the file was excluded by selection, is not in the change, no reviewer covers it, the issue is out of scope by design (maintainability and readability), it is a security or performance issue without that reviewer, it lies outside the changed lines, or it is reachable. The reachable share is an upper bound on recall; a lower number means a model cannot fix it, only selection, the matrix or the scope can.
Scoring
A port of the benchmark's official matching: same file, same diff side, line ranges at most one line apart, then an LLM judge decides whether two comments express the same concern. Each generated comment counts once.
| Metric | Definition |
|---|---|
| Precision | matched findings / generated findings |
| Recall | matched findings / annotated comments |
| F1 | harmonic mean of the two |
The judge uses JUDGE_BASE_URL, JUDGE_API_KEY and JUDGE_MODEL, or a Gemini key. Answers are cached per run, so re-scoring is free. Reports break results down by language, issue category and context level, with tokens, cost and latency.