Visualizing PerformanceIntroduction
Back to Dr. Anjali's dengue camp in Kochi. In the last chapter every threshold gave the quick test a different confusion matrix: threshold 50 caught 9 of the 10 patients with 9 false alarms, threshold 65 caught 8 with only 2 false alarms. So which one is the test's real score?
Neither. The threshold is a choice Dr. Anjali makes; how good the test is, is something else. In this chapter you will draw the test at every threshold at once as one curve, the ROC curve, and sum that curve up in one number, the AUC.
The name is older than computers. In the Second World War, radar operators had to tell enemy aircraft from flocks of birds and storm clouds on a flickering screen, and this curve was invented to measure how good each operator was. ROC stands for
Today ROC curves are used to judge medical tests, weather warnings and, above all, classifiers in machine learning.
The Two Core Rates
An ROC curve is built from two rates, each a fraction between 0 and 1. At threshold 50 the camp's matrix was TP = 9, FN = 1, FP = 9 and TN = 181.
The true positive rate (TPR) is the share of the real patients that the test catches. It is the recall from the last chapter:
The false positive rate (FPR) is the share of the healthy people who get a false alarm:
A good test has a high TPR and a low FPR: it catches the patients and leaves the healthy people alone. At threshold 50 the camp's test does both: a tall TPR bar and a tiny FPR bar.
Note
The FPR is 1 − specificity. At threshold 50 the specificity was 181 ÷ 190 ≈ 0.95, and 1 − 0.95 = 0.05.