Classifier evaluation
Forty labelled borrower replies, run through the deterministic baseline and through the model. The headline accuracy is the least interesting number here. What the set is for is finding the misses that cost a lender money or a complaint, and there is one.
Recorded 2026-08-21T23:23:47Z. The numbers on this page are a recorded run, not a live one — forty uncached calls take about two minutes, which is not something to do on a page load. Use the box below to watch a single reply go through live.
Try one live
Where it is wrong
Not all misses are equal. Filing a borrower's question as unclear costs a servicer thirty seconds. Reading a hardship as a promise to pay schedules a draft against an account the borrower has just told you is empty, which returns, which charges them another NSF fee, which is the thing this product exists to reduce. The eval marks that second class separately.
costly adv-negotiation — expected hardship, read as promise to pay at 80% confidence
I can do 300 a month, that is genuinely all I have. Take it or take the house.
an offer, but the operative fact is inability to pay the contractual amount
Why this one matters. The borrower is not promising, they are telling you what they can survive. The model is confident — 80% — and confidence is exactly what makes it dangerous: a threshold rule would let it straight through. This is the case that decides the architecture. A classification here never becomes a scheduled promise on its own; a servicer promotes it, and the promotion is the event. That boundary is not caution for its own sake, it is the response to a failure that this set reproduces every run.
Per intent
| Intent | Baseline | gemini-3.6-flash | n |
|---|---|---|---|
| dispute | 10.0% | 100.0% | 10 |
| promise to pay | 28.6% | 100.0% | 7 |
| question | 83.3% | 100.0% | 6 |
| hardship | 60.0% | 80.0% | 5 |
| unclear | 80.0% | 100.0% | 5 |
| wrong person | 25.0% | 100.0% | 4 |
| unsubscribe | 100.0% | 100.0% | 3 |
Confusion matrix — rows are the label, columns are the reading
| expected \ read as | promise to pay | dispute | hardship | wrong person | unsubscribe | question | unclear |
|---|---|---|---|---|---|---|---|
| promise to pay | 7 | · | · | · | · | · | · |
| dispute | · | 10 | · | · | · | · | · |
| hardship | 1 | · | 4 | · | · | · | · |
| wrong person | · | · | · | 4 | · | · | · |
| unsubscribe | · | · | · | · | 3 | · | · |
| question | · | · | · | · | · | 6 | · |
| unclear | · | · | · | · | · | · | 5 |
On the labelled set
The first thirty cases were written to sit on the boundaries a servicing shop trips over. The model scored 30 of 30 on them. That is not a result, it is a signal that the set stopped discriminating, so ten adversarial cases were added: a promise that appears only inside quoted history, a sarcastic refusal shaped like agreement, a bookkeeper who says "I will forward this", a conditional that commits to nothing, and the negotiation above.
Accuracy fell from 100% to 97.5%, which is the number worth having. An eval that only ever goes up is measuring the wrong thing. The next ten cases should be drawn from real returned mail rather than written, because the failures that matter are the ones nobody thought to imagine.