TypeSafe's Jev told me it was 75% confident.
Humans agreed 10% of the time.

A decision model returns a probability, and your code writes a threshold against it. So I checked whether the number is honest — 8,000 judgments measured against human annotations.

0.91AUC
ranking quality

0.156ECE
calibration error

96%of that error
removable

$0.05total
cost

The result

Jev doesn't generate text. You hand it state, it hands back a typed probability. That makes the probability the entire product. TypeSafe's docs say “Start with conservative thresholds, test with your own data, and adjust as you observe results.” This is what that looks like when you actually do it — and the tool that does it for you.

Reliability diagram: every confidence band falls below the perfect-calibration diagonal. — Every confidence band sits below the diagonal. A systematic lean toward “yes”, not noise — and it holds against the raw annotator fractions too, so it isn't an artefact of where I drew the threshold.

Two ways to be good, and they fail separately

	Question	Metric	If it's bad
Ranking	Can it tell flagged from clean?	AUC 0.91	the model can't do the task
Calibration	Do the numbers mean what they say?	ECE 0.156	the numbers need rescaling

A model can sort almost perfectly while its probabilities are nonsense. Reporting accuracy alone hides both facts. On one question, Jev at the default threshold scored 61% where answering “no” every time scores 68% — worse than a constant, while ranking at AUC 0.83.

“Your prompts were just badly written”

The obvious objection, so I tested it: four wordings of the same question in a single call, over identical state, so only the phrasing varied.

Wording	Mean p	ECE	AUC
Original	0.445	0.397	0.896
Mirrors the annotators' own definition	0.566	0.519	0.881
Strict — high bar, explicit exclusions	0.259	0.211	0.913
Bare — no criteria at all	0.488	0.441	0.888

True base rate: 4.8%. Wording matters a lot — but the wording mirroring the human raters' own published definition scored worst of the four. AUC barely moved across all of them: wording shifts the scale, not the signal. And nothing fixed it — the strictest still predicted 26% where reality was 5%.

So I re-ran everything with the best wording I found. Error fell from 0.209 to 0.156 and required human review roughly halved. The bias survived.

The fix

Reliability curves before and after recalibration; the corrected curve tracks the diagonal. — Fitted on one half of the data, scored on the half it never saw.

A logistic fit — two parameters, about twenty lines — removed 96% of the calibration error on held-out data. AUC went 0.918 → 0.918, because a monotonic rescale reorders nothing. The ranking was always there; only the units were wrong.

What I'd want asked back at me

One dataset, one domain, one pinned model version (jev-1.13.0).
One question has a single positive example in 400. Every number attached to it is meaningless. It's in the repo labelled as such rather than quietly dropped.
Four wordings is four, not all of them. Someone may find a better one.
Majority-vote human labels aren't neutral ground truth — annotator pools carry their own biases, particularly around dialect and reclaimed slurs.
I nearly reported “95% accuracy” as a win, until I noticed that flagging nothing scores 97% at this base rate. Accuracy is a trap on imbalanced data; the honest read is precision 0.17 at recall 0.74.

None of this is an argument against the model. As a high-recall pre-filter with a calibration layer on top, it works well. As if p > 0.9, it doesn't do what it looks like it does.

Code, charts & raw data Slides (PDF)