All projects
AGENT EVALUATIONResearch protocol

EviSuff Finance

Does an agent change its answer when necessary evidence disappears? A counterfactual protocol for evidence-sufficiency evaluation in long financial documents.

Key metric
4

paired conditions: full, gold-only, necessary removal, irrelevant removal

One question, four contexts
FULL

All evidence

GOLD ONLY

Sufficient evidence only

NECESSARY REMOVAL

Critical evidence missing

IRRELEVANT REMOVAL

A distractor removed

Protocol schematic · no empirical model results claimed.

Architecture

EviSuff Finance

Components, data flows and trust boundaries of the system.

Benchmark constructionPaired conditionsEvaluationadjudicateJSONLfreezemanifestS-1 / F-1 filingsIPO Finance AgentAnnotationclaims · evidence setsDouble annotation≥ 25% adjudicatedSchema validationJSONL · hitting setsFrozen benchmarkbefore any model runGitHub Actionstests · fixture checksPaper V0protocol & auditfullall evidencegold_onlyminimal sufficient setnecessary_removalcritical evidence cutirrelevant_removaldistractor removedAgents under testprovider-agnosticScoringabstention · citationsPaired bootstrapconfidence intervalsModel metricsevidence sensitivity

Scroll the diagram →

Technologies

  • Python
  • JSONL schema
  • Bootstrap statistics
  • GitHub Actions
  • LaTeX

Design decisions

  1. 01

    Each question carries one or more minimal sufficient evidence sets; “necessary removal” deletes the smallest hitting set that breaks all of them.

  2. 02

    The benchmark is frozen before any experiment, and LLM judges stay auxiliary until calibrated against human labels.

  3. 03

    Predictions go through a provider-agnostic manifest, so every model is scored the same way.

  4. 04

    Synthetic tests validate the software, not the findings: no model figure is published before the human benchmark.

What I built

JSONL annotation schema, validation of minimal evidence sets, deterministic context construction, sensitivity, abstention and citation metrics, paired bootstrap intervals and consistency tests.

Scope and limitations

Infrastructure and protocol implemented. Synthetic tests validate the software; no empirical model results are claimed before a human-verified benchmark exists.
Next project
Smart Bidding