Was the final/current PR head still the exact commit that was approved?
DETERMA analyzed 200 independently inspectable approval histories across 10 open-source repositories for agentic / AI-agent projects. The benchmark makes one narrow observation: in 30 histories, the captured final/current PR head SHA differed from the exact commit ID bound to the latest captured approval.
V4: what is actually inside the 30 mismatches?
V4 enriches the 30 mismatches instead of treating a different commit ID as equivalent to a meaningful code change.
Interpretation boundary: a different tree proves content-state difference, not that the difference was material, unsafe, or policy-violating. Human materiality labels are not yet part of this benchmark.
GitHub already has relevant controls
GitHub can optionally dismiss stale approvals when reviewable commits change the pull-request diff, and it can optionally require approval of the most recent reviewable push. Those controls are part of the baseline DETERMA must be compared against — not ignored.
GitHub protected-branch documentation · GitHub ruleset documentation
Control-snapshot boundary: these rule values are a point-in-time snapshot captured on 2026-09-28. They do not reconstruct which rules were active when the historical approvals occurred. A branch with no visible pull-request ruleset rule is not treated as unprotected; classic branch-protection details are not inferred from absence.
The captured approvals still report APPROVED
30 / 30 exact review objects captured by V3 currently return review state APPROVED through the GitHub API.
This is useful state evidence, but it is not evidence that a PR was mergeable, that all required checks passed, or that any repository bypassed its policy.
The candidate DETERMA response — not yet a product proof
Observed signal: APPROVAL_COMMIT_DIFFERS_FROM_CAPTURED_FINAL_OR_CURRENT_HEAD
Architecture response under evaluation: REVALIDATE_EXACT_ACTION_BEFORE_EXECUTION
The public corpus demonstrates the signal. It does not yet measure whether DETERMA makes the correct ALLOW / DENY / NEEDS_REVIEW decision, whether it blocks at runtime, or whether receipts and authoritative readback are correct.
What this benchmark still does not prove
- That any specific tree-changing case was materially risky.
- That GitHub controls were absent, bypassed, or misconfigured at the historical approval time.
- That any project violated its own policy.
- That DETERMA would outperform GitHub's built-in controls on these cases.
- Any false-positive or false-negative rate for DETERMA.
- Runtime enforcement, readback, receipt correctness, customer validation, or commercial outcomes.
Next proof required: two independent human materiality labels per tree-changing case, adjudication of disagreements, then a frozen counterfactual DETERMA run on the same cases and comparison against the observable GitHub baseline.
Explore the 200 cases
All source records point to public GitHub histories. The V4 tag is descriptive only; it is not a risk score.
Try it on your repo — self-serve is live
Give DETERMA one public or explicitly authorized repository. A bounded historical scan is processed automatically and the evidence report is returned to the same GitHub issue before any production integration.
Methodology
Ten preselected repositories for agentic / AI-agent projects; 20 most recently updated PR histories with an approved review per repository; search window updated:>=2026-01-01; latest retrievable APPROVED review bound to a commit ID; compare that exact commit ID with the captured final/current head SHA.
V4 then compares Git tree IDs, commit-graph relation, file-level compare metadata, the current state of the exact captured review object, and a current public Rules API snapshot where observable.
Statistical boundary: purposive benchmark corpus, not a random population sample. Percentages describe this corpus only.