ABSTRACT
Human behavior is challenging to predict. People commonly make cognitive pseudo-prediction errors, such as the confusion of inverse probabilities. People also tend to ignore base rates when making predictions. When the base rate of a behavior is very low or very high, you can be highly accurate in predicting the behavior by predicting from the base rate. Thus, you cannot judge how accurate your prediction is until you know how accurate your predictions would be by random chance. Moreover, maximizing percent accuracy may not be the ultimate goal because different errors have different costs. Though there are many types of accuracy, there are two broad types: discrimination and calibration—and they are orthogonal. Discrimination accuracy is frequently evaluated with the area under the receiver operating characteristic curve, or with sensitivity and specificity, standardized regression coefficients. Calibration accuracy is frequently evaluated graphically and with various indices. Sensitivity and specificity depend on the cutoff. Therefore, the optimal cutoff depends on the purposes of the assessment and how much one weights the various costs of the different types of errors: false negatives and false positives. It is important to evaluate both discrimination and calibration when evaluating prediction accuracy.
