How Are Predictive Models Trained and Evaluated?

This chapter discusses the main principles behind the training and evaluation of a predictive model. The evaluation of a model's performance using a test set alone has the limitation that not all the available data is used in the training step. The chapter reviews the metrics most commonly used to evaluate the performance of both classification and regression models while discussing their limitations and optimal areas of application. In general a predictive model should be as simple as possible. The Fukushima nuclear disaster is the accident that occurred at the Fukushima Daiichi Nuclear Power Plant in Okuma, Japan, immediately preceded by the Tohoku earthquake and tsunami that took place on March 11, 2011. The performance of a predictive model on its own training set alone cannot possibly be representative of its actual performance because of the overfitting problem.