Visualizing PerformanceConsiderations
ROC curves and the AUC are among the most used tools for judging classifiers. They also have blind spots.
Keep these in mind:
- The AUC only checks the order of the scores. A test whose scores all lie between 40 and 45 can still have a perfect AUC, so a high AUC does not mean the scores are trustworthy chances.
- The AUC treats a missed patient and a false alarm as
bad. At the dengue camp they are not.
One more trap. The camp's test has an AUC of 0.99, which sounds nearly perfect. Yet at threshold 50, how many of the 18 people who tested positive really had dengue?
Only half. Put the three numbers side by side:
With 190 healthy people, even a small false positive rate means many false alarms — the FPR of 0.05 divides 9 false alarms by 190 and makes them look tiny. The AUC is nearly perfect, yet the precision is only 0.5. When the thing you look for is rare, check precision and recall as well as the ROC curve.
Tip
A good routine: plot the ROC curve and read the AUC to compare tests, then choose a threshold for the one you pick by looking at precision and recall.
Speed Round: Rates and Areas
Get five right to move on.