A vendor calls its confidence scores calibrated. I audited them against human annotators for $0.05, then shipped the fix as a one-file tool.
What I didRan the audit, then wrote the tool that fixes what it found.
Every number on this page links to the file it came from, pinned to does-jev-confidence-mean-anything @ def9fac and jevcal @ 468a4df.
Headline results
The short version
The problem
A vendor’s docs call its confidence scores calibrated. Engineers read that as: threshold at 0.9 and you get 90%. I wanted to know whether that reading survives contact with human-labelled data, and what it costs to find out.
What I did
I used civil_comments as ground truth, because every comment carries the share of a human panel that flagged it. I sent 8,000 judgments to Jev, across four wordings and two base rates, for five cents, and measured ranking (AUC) and calibration (ECE and Brier score) separately.
Jev ranks well. Its probabilities lean toward yes, the error grows as positives get rarer, and a two-parameter fit removes 96% of it without changing a single ranking. Then I turned the check into a tool: jevcal, one file, numpy only, about 100 labelled rows.
What this is not
It is not an argument against the model. An AUC of 0.91 means it understands the task, and as a high-recall pre-filter with a calibration layer on top it works. As if p > 0.9 it does not do what it looks like it does.
What I would do next
Run jevcal on a second model that emits probabilities, such as an LLM judge, to find out whether the claim that nothing in it is Jev-specific holds.
How it works
Pick a request and watch it go through the system. The red ones are the requests I use to attack it. They end where the system is supposed to stop them.
AUC 0.91, so it ranks well, but ECE is 0.156 to 0.209 and every confidence band sits below the diagonal.
- civil_comments
- Jev API
- Predictions
- Measure
- Reliability chart
One pipeline, two outcomes. The audit trace measures the problem. The fix trace learns a correction. The red trace is the bet that the audit disproves.
Test results
11 claims, each with the bar I held it to and what I measured. 5 failed and 1 are unclear. They stay in, because a list where everything passes tells you nothing.
| Result | Claim | Expected | What I measured | Source |
|---|---|---|---|---|
| PASS | Jev ranks flagged comments above clean ones | AUC well above 0.5 | AUC 0.903 to 0.912 across three runs | README.md ↗ @ def9fac |
| FAIL | Stated confidence is calibratedThe vendor's sentence that higher confidence means higher accuracy is true here, since rank correlation is 0.96. What fails is reading 0.9 as a threshold you can bet on. | ECE near 0 | ECE 0.156 to 0.209, and every confidence band sits below the diagonal | README.md ↗ @ def9fac |
| FAIL | A stated 0.75 means about 75% | roughly 75% of those comments flagged | annotators flagged 10% of the judgments in the 0.7 to 0.8 band (n=40) | README.md ↗ @ def9fac |
| PASS | A two-parameter fit fixes the units without touching the ranking | n/a | ECE 0.157, 0.209 and 0.156 become 0.023, 0.007 and 0.006, and AUC stays 0.918 | README.md ↗ @ def9fac |
| PASS | The fix is scored on data it never saw | n/a | fitted on one half of the rows and scored on the other half | README.md ↗ @ def9fac |
| FAIL | A better prompt wording fixes the calibration | ECE near 0 with the right wording | no; across four wordings ECE spans 0.211 to 0.519 and the strictest still said 26% where reality was 5% | README.md ↗ @ def9fac |
| FAIL | Accuracy at the default threshold shows the model worksThe signal is there and the threshold is in the wrong place. That is the whole argument for measuring calibration. | n/a | 61.0% accuracy, worse than answering no every time (67.8%), while ranking at AUC 0.83 | README.md ↗ @ def9fac |
| FAIL | A low ECE means the model is goodSo jevcal prints that baseline next to your result and makes its own decisions on Brier score. | n/a | no; a model that ignores its input and returns the base rate scores ECE 0.0000 at AUC 0.5 | README.md ↗ @ 468a4df |
| PASS | About 100 labelled rows are enough to calibrate | n/a | 94% of the calibration error removed at 100 rows, against 69% at 25, 90% at 50 and 93% at 800 | README.md ↗ @ 468a4df |
| PASS | jevcal says when it cannot be trusted | n/a | warns under 50 rows or 10 positives, and writes no file if the fix does not improve held-out Brier score | README.md ↗ @ 468a4df |
| UNCLEAR | The correction works for models other than JevThe README says nothing in it is Jev-specific. That is a design claim, and I have not run it on a second model. | n/a | not tested; the repo's only example is the real Jev output from this audit | README.md ↗ @ 468a4df |
The numbers
Each headline is the conclusion. Open “View the data” under a chart for the exact values.
Every confidence band sits below the diagonal. Jev's top band, scored 0.94, was right 33% of the time.
Stated confidence against observed rate
1,600 judgments on natural comment traffic (2.9% positive), tightened wording. Dot size shows how many judgments fall in the band.
Mirroring the annotators' own definition was the worst wording. No wording fixed it.
Calibration error by question wording
Four wordings of one question in a single call, over identical state. True base rate 4.8%. Lower is better.
About 100 labelled rows are enough. More does not help.
Labelled rows needed to calibrate
Share of calibration error removed, measured on real data.
What went wrong
My first guess at a better prompt made it worse
- How it surfaced
- Mirroring the annotators' own published definition, which should have been the fairest wording, gave the highest ECE of four wordings (0.519).
- What I did
- I ran four wordings of the same question in one call over identical state, so only the phrasing varied, then re-ran the whole pipeline with the best wording instead of the first. The bias survived.
Accuracy would have said the model was fine
- How it surfaced
- At a 2.9% base rate, flagging nothing scores 97.1% accuracy, so accuracy on this data is worse than useless.
- What I did
- The measuring step prints the do-nothing baseline before anything else and reports precision and recall.
ECE can be gamed
- How it surfaced
- A model that ignores its input and returns the base rate for every row scores a perfect ECE of 0.0000 with AUC 0.5.
- What I did
- jevcal prints that baseline beside your result and makes its own decisions on Brier score, which cannot be gamed that way.
Invented data could be mistaken for real data
- How it surfaced
- A no-API-key mode needs simulated data to test the plumbing, and simulated data looks exactly like data.
- What I did
- Simulated output goes to separate SIMULATED files and every chart is stamped, so the two cannot be confused.
Choices I would defend
Use human ground truth, not a model's opinion
civil_comments records what fraction of a panel flagged each comment, so a toxicity of 0.7 means 7 in 10 real people agreed.
Test the obvious objection before making the claim
Someone will say the prompt was badly worded. Step 7 runs four wordings in one call and the headline uses the best, not the first.
Ship the fix, not only the finding
jevcal fits a correction on half the labelled rows, scores it on the other half, and prints the threshold you should actually write.
Implement the maths directly
Platt scaling and isotonic regression are about sixty lines each, and the API client is plain urllib, so every number traces to code in the repo.