ABSTRACT
This chapter evaluates the application of Cochran-Weiss-Shanteau (CWS) to this real-world environment where expertise in sensemaking can take years to decades to develop competence. CWS integrates two necessary conditions for expert skill, namely consistency and discrimination. CWS integrates these two by taking the ratio of discrimination to inconsistency, such that higher CWS scores are indicative of better performance. CWS has been successfully applied as a performance measure to several preexisting datasets, three of these applications are described in Shanteau et al., drawing on data from: auditing, personnel hiring and livestock judging. The research question is how to assess performance that keeps improving 'beyond perfection'? To answer this question, various measures were compared for evaluating skilled performance that is 'better than perfect'. Five measures were used to assess performance: mean number of errors per session; mean time-through-sector for aircraft in the airspace; consistency of behaviors over repetitions of the same scenarios; discrimination between different scenarios; and discrimination/inconsistency.
