Welcome back to our series on ROC, other related statistics and charts. Last time, we discussed several statistics associated with ROCs. In this installment, we will add some charts (graphs) that can be drawn from those ROC statistics and study more statistics.
The chart below (Fig. 1, below) shows two different plots (ROC curves) -- one for sensitivity, one for specificity. The cut off is on the x-axis and the per cent value on the y-axis. There are 6 cut off points Figure 1from low to high values. For example, looking at point 3, the sensitivity is about (78 percent) and the specificity is about 68 percent. What are the sensitivity and specificity at point 4? Which cut-off would you pick? What are the pros and cons of your choice?
Note that, as the cut off increases, the sensitivity goes down (thus more FN) and the specificity goes up (more TN). It is important to keep in mind when reading articles where these statistics are given that sensitivity and specificity are cut off dependent.
Imagine that you want 100 percent sensitivity -- all those with the disease would be identified by the laboratory test. To do that, we might start with cut off 1. This would yield the 100 percent sensitivity, but a specificity of only about 40 percent. After considering that, you might decide to use cut off 2. If there were still too many FP (more than 40 percent of the group without the disease being tested for), you might move to cut off 3 -- giving up some sensitivity to keep FP lower. In order to achieve a higher specificity, we move to cut off 3 and raise the specificity. This reduces our sensitivity from 100 to under 80 percent. This means that the test misses about 20 percent of the patients with the disease, leaving the clinician to rely on other tests as well as signs and symptoms and his/her own acumen. The color of medicine is grey.
Part of the decision about whether to use this test and which cut off to assign to it will be based on the known number of patients in the two groups (with and without the disease). If the ratio of diseased to non-diseased is quite low, it might be impractical to attempt to find 100 diseased patients.
Unless we are interested only in percent data, these lines may not answer the question of whether to use the test -- they do not tell us the number of persons that would be correctly identified -- only the percent. We could go back to the table we used to draw the sensitivity and specificity chart and look at the raw data. We have added a table here with the overall efficiency, the NPV and PPV, as well as number of patients correctly identified at each cut off. It is important to factor the numbers in with the plots as it is unlikely the number of diseased persons is the same as 'normal' or those without the disease being looked for.
Here is a table with TP and TN, as well as FP and FN; followed by the plot of FP and FN; followed by the plot of NPV and PPV.
You see that, when all of the diseased patients are detected, the (squares in Fig. 1) number of FP (triangles in Fig. 1) is quite high. To reduce FP, we need to use a higher cut off. But that lowers the number of TP. Again, there is nothing special about the point where the lines cross.
At this point, we may wonder how best to assign a cut off. The clinicians will get a major vote in this dialogue. We have mentioned the Positive and Negative Predictive Values earlier (PPV and Figure 2NPV). The statistics indicate the probability of a positive result, being from a person with the disease and not a person without the disease. For example, a PPV of 90 means that 9 out of 10 positive laboratory values that were above a given cut off came from persons with the disease. An NPV of 80 means that 8 of 10 negative laboratory values came from patients the clinician said did not have the disease. Two negatives came from patients the clinician had labeled as with the disease.
Here is data from a study of BNP and NT-proBNP in heart failure patients: Both markers were significantly higher in HF patients than in healthy subjects. For a cut-off of 32 pg/ml, BNP had a 91.7 percent sensitivity 93.7 percent specificity, 75.5 percent PPV and 98.3 percent NPV. A NT-proBNP cut-off of 126 pg/ml had 96.3 percent sensitivity 98.8 percent specificity, 94.3 percent PPV and 99.3 percent NPV. Both tests had an excellent ability to distinguish HF from non-HF subjects. "NT-proBNP was more sensitive and specific."1
Do you agree with their conclusion? NT-BNP seems to have a much better PPV than BNP. What does this mean to the clinician when they see a positive value? Which method had more FP? [The BNP with the lower specificity has more FP.]
Here is data on 226 patients who underwent diagnostic coronary angiography after sudden cardiac arrest and were analyzed retrospectively: Levels of TnT, CK and CK-MB on admission and 6h, 24h and 36h later and compared with the results of coronary angiography. Applying a cut-off of 1 µg/l the 6h TnT measurement had a sensitivity of 71 percent and specificity of 61 percent to diagnose AMI after cardiac arrest. Using CK 800U/l as cut-off level resulted in a sensitivity of 62 percent and specificity of 74 percent, CK-MB levels higher than 100U/l yielded a sensitivity of 59 percent and specificity of 73 percent. The authors concluded that "Cardiac injury markers cannot be used to reliably diagnose or rule out AMI after resuscitation."2 Weak diagnostic performance of troponin, creatine kinase and creatine kinase-MB to diagnose or exclude myocardial infarction after successful resuscitation.3
Do you agree with their conclusion? Do you think that one or more of the other statistics we discussed in this installment might be more useful than the sensitivity and specificity? If so, which would you use and why? If not, why not? Would the curves help to understand the conclusion? Why?
Next time, we will look at the question of the dollar cost of "errors" and look at another plot and the area under the curve (AUC).
- Comparative value of BNP and NT-proBNP in diagnosis of heart failure. Fonseca C, Sarmento PM, Minez A, Rev Port Cardiol. 2004 Jul-Aug;23: 979-91.
- From Kruse JM, Enghard P, Schröder T, et al.
- Int J Cardiol. 2014 Feb 28.