How accurate the verifier is.
We run the pinned judge against a labeled golden set and publish what it gets right and wrong. Each result names its judge model, judge prompt hash and date. The set and the harness are public so you can rerun them. See the methodology.
Verifier accuracy · no published run yet
No signed golden-set result has been published, so this page shows no accuracy figures. The first run needs the live judge and the published signing key. Until then we show nothing rather than an estimate.
What the numbers mean
| Precision | Of claims the verifier passed, the share a person labeled supported. |
| Recall | Of claims a person labeled supported, the share the verifier passed. |
| Misgrounded rate | Of claims whose cited passage does not support them, the share the verifier passed. |
| False claims passed | Of false claims the cited passage contradicts, the share the verifier passed. |
A misgrounded claim may be true, but the passage it cites does not support it. Retrieval tools produce these often, and claim-to-source binding exists to catch them, so we count them apart from plainly false claims.
Runs · newest first
The golden set
| Works curated | 30 of 220 |
| Labeled claims | 180 |
| Supported / wrong / misgrounded | 90 / 46 / 44 |
| Inter-labeler agreement | not measured yet |
| Set version | 0.1.0 |
| Set digest | f4c8c07b8e1f |
Every work is published under a license that permits it. The set is data: passages and claims only ever reach the judge as the text to check.
Rerun it yourself
The harness scores every claim through the judge, computes the metrics and writes a result file with the model, judge hash, set digest and date. Compare your result with ours; it should match within the stated tolerance.
The public harness repo is not published yet. One claim at a time: claims_verify (MCP) or POST /v1/reverify. Live judge: v0-dev 1e64388843ff. Machine-readable: /benchmarks.json.
Tool selection · how often agents call ContextFile
Being installed is not being used. We give each model a coding task and every tool from ContextFile and from competing docs servers, and record whether its first call goes to ContextFile. Match tasks have a verified artifact in the catalog; control tasks have none. See the MCP reference for the tools.
No selection run has been published yet, so this section shows no rates. The task set has 12 match tasks (12 still waiting for their artifact to be listed) and 6 control tasks.
| Call rate | Of tasks where a matching verified artifact exists, the share where the model's first tool call was a ContextFile tool. |
| False calls | Of control tasks with no matching artifact, the share where the model called ContextFile anyway. |
| Valid arguments | Of ContextFile calls, the share whose arguments the tool's own schema accepts. |
End-to-end task accuracy is not measured by harness selection-eval/1. Machine-readable: /benchmarks.json (toolSelection).
Uplift scorecard · the same model with and without ContextFile
Each model answers the same version-pinned coding tasks twice with the same prompt: once with the ContextFile MCP server connected and once with no tools. We publish task accuracy, call rate, tokens per answer and the misgrounded-citation rate for both arms, with the model and run date.
No signed uplift result has been published, so this page shows no uplift figures. The first run needs the models chosen and the published signing key. Until then we show nothing rather than an estimate.
| Accuracy | Of tasks, the share whose answer passed the task's checks. |
| Call rate | Of tasks, the share where the model called a ContextFile tool at least once. Only the with arm has a server to call. |
| Tokens per answer | Mean prompt plus completion tokens per answered task. |
| Misgrounded citations | Of ContextFile citations in answers, the share not backed by a source the model read in that run. |
Task set contextfile-uplift 0.1.0: 6 tasks. The scorecard reruns every 30 days. Rerun it: benchmarks/uplift/README.md in this repo (the public harness repo is not published yet). Machine-readable: the uplift field of /benchmarks.json.