cat projects/dispute-engine/README.md
Dispute Engine
- Status
- In progress
- Stack
- Python, Pydantic, Claude API, FastAPI
Given a dispute, the system classifies it, judges whether it’s worth fighting, and lists the evidence required and what’s missing. I’m using it to work through the AI-engineering stack over ten weeks.
How it works
DisputeCase ──▶ classify (versioned prompt) ──▶ Classification
│
└─ response cache, per-model request shaping
dispute classifyturns one case into a validatedClassification.dispute generatebuilds a synthetic corpus offline. Cases are generated from a chosen dispute family, so every case has a gold label without hand-labeling. 200 cases across three difficulty tiers. 11% are flagged as having a contestable label and kept out of headline numbers.dispute evalreports per-class precision, recall and F1, a confusion matrix, calibration (ECE), a per-tier breakdown, and a failure list. CI gates read the saved scorecard, so they need no API key.dispute serveis a local review UI that compares models side by side and highlights fields that differ.
Current results
| Metric | Value | Reading |
|---|---|---|
| Classification macro F1 | 1.00 | Benchmark is too easy |
| Winnability accuracy | 0.43 | The real problem |
| Calibration error (ECE) | 0.27 | Underconfident in every bin |
grep -l dispute-engine notes/
Two eval runs for $0.55 turned up a data bug, a saturated benchmark, 43% accuracy on winnability, and underconfident calibration.
On a mislabeled "fraud" chargeback, both models found the real category. Only Opus asked for the evidence that would decide the case.
Haiku 4.5 rejects adaptive thinking and the effort parameter, so model-agnostic code needs per-model request shaping.