Verbatim Citation Gate

tom@tomrochette.com (Tom Rochette) — Sat, 10 Oct 2026 00:00:00 +0000

Verbatim Citation Gate is an MIT two-stage citation auditor that catches fabricated RAG quotes deterministically before any model call, then hands only quotes that actually exist to a skeptical LLM judge.

The mechanism is the order of the checks: a zero-dependency, zero-token verbatim gate rejects fabricated, stitched (“frankenquote”), and misattributed quotes for free, so the expensive judge only ever sees quotes the sources contain, and the judge itself starts at “unsupported” and fails closed when its output does not parse.

What it is
#

A pip-installable Python package (from the repository, not yet on PyPI) with stage one in gate.py: pure re plus substring matching over a normalized form (case, smart quotes, dashes, and whitespace folded, numbers and percent preserved), returning not_found or misattributed with no network and no model. Stage two in judge.py takes any (system, user) -> str callable, so it wires to Claude, GPT, Gemini, Mistral, Cohere, or a local Qwen without adapters, and returns supports, partial, unrelated, or contradicts. The judge’s prompt encodes three rules: default-refute (the verdict starts at unsupported and ties break against the claim), outside knowledge is inadmissible (a claim can be true and still unsupported by this source), and full-strength support or the verdict caps at partial. Each named failure mode has its own test, including test_audit_fabrication_never_calls_judge, which proves a fabricated quote costs zero model calls. The docs page describes the maintainer as the Palo Alto AI Research Lab, a self-styled name; the repository itself belongs to a personal GitHub account (tonydzi), and no GitHub organization of that lab name exists as of 2026-10-10.

Status
#

Active but barely adopted: 4 stars, 2 forks, and 4 open issues as of 2026-10-10, created 2026-07-24, pushed 2026-10-09, with two releases (v0.2.0 on 2026-08-25, v0.1.0 on 2026-08-04) and CI running a tests workflow. A Hacker News search returns zero threads as of 2026-10-10, which is the adoption signal here: this is a solo project with a clean mechanism and no independent users on record yet.

Strengths
#

The free first stage is an effective filter: three quote-failure classes (fabricated, frankenquote, misattributed) each get a named test, and the contiguous-match design is what makes stitched quotes fail.
Fail-closed judge: an unparseable judge response never counts as support, which is the right default for a verification component.
The smallest possible judge contract ((system, user) -> str) means the gate outlives any vendor SDK.
Misattribution is surfaced as its own verdict rather than collapsed into not-found, because the two failures need different fixes upstream.

Cautions
#

Near-zero adoption: 4 stars and zero discussion as of 2026-10-10 means nobody has independently validated the approach, so treat it as a promising pattern to copy, not a dependency to trust.
Exact-verbatim matching misses legitimate paraphrase and cross-sentence citations by design, so it fits quote-style citation systems only.
The distribution is a git URL install with no PyPI package, and the two releases stopped in August 2026 even though pushes continue.
The “Palo Alto AI Research Lab” branding on the docs is uncorroborated by any organization, funding, or publication record a reader can check.

Pricing
#

Does not apply: MIT-licensed, free, stage one costs zero tokens and stage two costs whatever model you attach.

Compared to
#

deepeval: its faithfulness metrics judge grounding with an LLM on every check; choose Verbatim Citation Gate to make the deterministic substring check absorb the obvious fabrications first, and keep deepeval for everything non-verbatim.
Jevals: typed decision-model judges that calibrate per question; Jevals is the general judge layer, this gate is the free pre-filter that shrinks what any judge sees.
Phoenix: RAG evals and tracing at platform scale; run the platform for visibility and bolt this gate onto the citation path where fabrication actually hurts.

Bottom line
#

Recommended as a pattern and a small library for quote-citing pipelines: adopt the two-stage order (deterministic verbatim check, then skeptical judge) even if you reimplement it. Not for paraphrase-tolerant citation systems, and not as a battle-tested dependency until someone other than the author has run it.

Changes
#

2026-10-10 - Created from the evaluation-review resolution pass as the differentiated judge mechanism: the deterministic zero-token citation-fabrication gate ahead of an LLM judge.

References
#

https://github.com/tonydzi/verbatim-citation-gate - repository: README, two-stage design, test names, license
https://api.github.com/repos/tonydzi/verbatim-citation-gate - stars, forks, issues, creation and push dates as of 2026-10-10
https://api.github.com/repos/tonydzi/verbatim-citation-gate/releases - v0.2.0 (2026-08-25) and v0.1.0 (2026-08-04)
https://tonydzi.github.io/verbatim-citation-gate/ - the docs page: two-stage diagram, known limits, and the self-styled lab attribution
https://hn.algolia.com/api/v1/search?query=verbatim-citation-gate&hitsPerPage=3 - the zero-thread community-footprint check as of 2026-10-10

hallucination on tomrochette.com

Verbatim Citation Gate

What it is
#

Status
#

Strengths
#

Cautions
#

Pricing
#

Compared to
#

Bottom line
#

Changes
#

See also
#

References
#

hallucination on tomrochette.com

Verbatim Citation Gate

What it is #

Status #

Strengths #

Cautions #

Pricing #

Compared to #

Bottom line #

Changes #

See also #

References #

What it is
#

Status
#

Strengths
#

Cautions
#

Pricing
#

Compared to
#

Bottom line
#

Changes
#

See also
#

References
#