ABSTRACT
This purpose of this chapter is to highlight methodological issues that are not always considered when trying to understand how children come to regulate their cognition and behavior during the preschool period. To give some background, there has been a rapid increase in the number of papers addressing the development of executive control in very young, preschool children. This expansion has resulted from four primary issues. First, it is now widely recognized that executive dysfunction is a central feature, or a strong contributor to outcome, of many neurodevelopmental disorders that either emerge or are detectable during this age range (Anderson, Anderson, Grimwood, & Nolan, 2004; Espy, Kaufmann, & Glisky, 1999; Espy et al., 2002; Ewing-Cobbs, Prasad, Landry, Kramer, & DeLeon, 2004; Pennington & Ozonoff, 1996). Second and concomitantly, executive control plays a critical role in other cognitive processes, including memory, attention, consciousness=theory of mind, reading and mathematics, and social skills (Baddeley & Hitch, 1994; Bull & Scerif, 2001; Desimone & Duncan, 1995; Espy et al., 2004; Gathercole & Pickering, 2000; Hughes, 1998a; Hughes, White, Sharpen, & Dunn, 2000; Isquith, Gioia, & Espy, 2004; Perner, Lang, & Kloo, 2002; Posner & Petersen, 1990). Third, the application of cognitive neuroscience paradigms had enabled the examination of executive abilities in this age range in a novel and more accessible manner (Blair, Zelazo, & Greenberg, 2005; Diamond, Prevor, Callendar, & Druin, 1997; Espy, 2004). Finally, with the protracted course of prefrontal development (Benes, 2001; Giedd et al., 1999; Huttenlocher & Dabholkar, 1997; Sowell, Thompson, Holmes, Jernigan, & Toga, 1999), there is the potential to uncover the fundamental principles
of executive control in its ontogeny at a period where cognition is less complicated, which should further illuminate the fundamental nature of executive control in humans (Espy, 2004; Hughes, 2002). Although a substantial literature base has developed, the precise nature of execu-
tive control in children, or adults for that matter, is far from resolved. Differing accounts vary fundamentally according to (1) whether executive control is viewed as a unitary process (Allport & Wylie, 2000; Duncan & Owen, 2000; Miller & Cohen, 2001; Munakata, 1998) or composed of ‘‘fractionated,’’ interdependent subprocesses (Carlson & Moses, 2001; Miyake, Friedman, Rettinger, Shah, & Hegarty, 2001; Pennington, 1997); (2) the relative weights or uniqueness that these executive constructs are ascribed (Bishop, Aamodt-Leaper, Creswell, McGurk, & Skuse, 2001; Friedman &Miyake, 2004; Nigg, 2000); (3) the differential localization within the brain (Casey et al., 2000; Durston et al., 2002); or (4) the manner or pattern of development during childhood (Kirkham, Cruess, & Diamond, 2003; Zelazo, Mueller, Frye, & Marcovitch, 2003). There remains a paucity of commercially available, norm-referenced, psychometric tests to measure executive function in this age, with the only specific individually administered battery including specific ‘‘executive’’ measures being the NEPSY (Korkman, Kirk, & Kemp, 1998). Therefore, executive control typically has been studied in preschool children with experimental tasks, where different stimuli or response requirements are varied systematically to reveal critical performance differences that shed light on the nature of the executive cognitive process. Such tasks include rule governed, attribute-based sorting tasks (Espy, Kaufmann, McDiarmid, & Glisky, 1999; Hughes, 1998b) such as the Dimensional Change Card Sorting (DCCS) task (Zelazo, Frye, & Rapus, 1996); tasks with manual selection or verbal naming of stimuli that conflict or interfere on the basis of natural associations (Carlson & Moses, 2001; Davidson, Amso, Anderson, & Diamond, 2006; Diamond, Briand, Fossella, & Gehlbach, 2004; Diamond, Kirkham, & Amso, 2002; Gerstadt, Hong, & Diamond, 1994; Prevor & Diamond, 2005; Wright, Waterman, Prescott, & Murdoch-Eaton, 2003); manual search tasks with working memory maintenance demands (Diamond et al., 1997; Espy, Kaufmann, Glisky, & McDiarmid, 2001; Hughes, 1998b) and with inhibiting prepotent or prohibited somatic motor responses (Carlson & Moses, 2001; Diamond & Taylor, 1996; Espy, Kaufmann, McDiarmid et al., 1999; Kochanska, Murray, Jacques, Koenig, & Vandegeest, 1996; Korkman et al., 1998; Reed, Pien, & Rothbart, 1984). Although this study has revealed important insights, such as the critical role of
conflict (Brooks, Hanauer, Padowska, & Rosman, 2003; Diamond et al., 2002; Perner & Lang, 2002; Rennie, Bull, & Diamond, 2004), interference=distraction (Espy & Bull, 2005; Espy & Cwik, 2004; Zelazo et al., 2003), and information maintenance (Alloway & Gathercole, 2005; Hughes, 1998b), we argue that further progress in understanding the nature of executive control in very young children has been hampered by a lack of attention to psychometric theory, the methodological tenets upon which all instruments rest. From a methodological standpoint, notice that the majority of work done to date has focused on characterizing performance in preschoolers by manipulating task conditions to better isolate important cognitive mechanisms that are=are not operative in these contrasting conditions. This type of design has a critical place in developmental cognitive neuroscience and has lead to
Executive functions and
the development of several important theories of executive control development (Kirkham et al., 2003; Munakata, 1998; Zelazo et al., 2003). The advantages of this design include the ability to make comparisons within subjects, thereby taking advantage of the reduced error variance and maximizing the likelihood of detecting condition-related differences. Furthermore, from a theoretical perspective, it is quite compelling when it can be demonstrated that children behave in one way in one condition but completely differently in another, when relatively small alterations to the tasks are made. These discrepancies can be extended further by the inclusion of a contrast group that performs in an alternative manner in response to this condition manipulation. Such findings reveal a ‘‘double-dissociation’’ that is considered the ‘‘gold standard’’ of behavioral neurology (Van Orden, Pennington, & Stone, 2001). This approach, for example, has been used productively to reveal differences in executive control among children with attention-deficit=hyperactivity disorder (ADHD) (Berwid et al., 2005), head injury (Ewing-Cobbs et al., 2004), preterm birth (Espy et al., 2004; Luciana, 2003), phenylketonuria (PKU) (Diamond et al., 1997), and autism (Ozonoff, South, & Provencal, 2005) to name just a few. As compelling as these findings are, there are other considerations derived from
clinical science in the application of this method that are often ignored. From the perspective of psychometric test theory, it is important to keep in mind that the experimental tasks and their various conditions do not differ from commercially available, psychometric tests. Like psychometric tests, experimental tasks also have demands that necessarily elicit specific cognitive processes that are necessary to perform the task. In like fashion, in both psychometric tests and experimental tasks, the demands of any test=task typically do not elicit only one cognitive process or construct, regardless how ‘‘discrete’’ the test=task is designed to be. For example, although one might be interested in assessing visual-spatial processing with the commercially available, psychometric test, Block Design (The Psychological Corporation, 2003), which has considerably more task demands than just visual-spatial processing. At a minimum, independent of the visual-spatial demands, the child must visually perceive the differential color that comprise the pictured stimulus, and manually place the blocks in the desired configuration, in temporally integrated and proficient manner. To reduce the extraneous motor task demands, one might select a visual matching task to assess visual-spatial processing. Even here, minimally, the child must visually perceive the pictured stimuli and formulate and select the appropriate verbal response. In the cases of both experimental tasks and psychometric tests, the relation
between task performance and task demands can be represented as ‘‘true score’’ variance according to classic test theory, that is, the portion of performance that is related reliably to the underlying cognitive process or latent construct of interest. The portion that is non-reliable is considered error. ‘‘Reliably’’ is important-in that it is the proportion of the variance that is reproducible across test items, administrations and participants. True score variance can be further decomposed into smaller portions of variance that are related to specific constructs, and into error variance that includes subject-specific factors, administration-specific factors, time-specific factors, and random chance. In most of the studies described earlier that have used experimental tasks, however, the task and its experimental conditions typically are
treated as fully representing the latent cognitive construct of interest in a one-to-one fashion. When a performance difference is observed between a clinical and control group or between children of differing ages, this difference is attributed solely to the discrete cognitive processes that the authors are trying to study by experimental task administration. However, because the true relations between the latent cognitive construct and experimental task performance are unknown, it is equally plausible that performance differed on the second condition related to subject-specific factors that were not considered. Using the overly simple Block Design example, younger children might perform more poorly on Block Design because of less precise motor skills in manipulating the blocks, not related to fundamental differences in visualspatial processing. Critically in the case of executive control, where the executive control of cognition inherently involves the regulation of other cognitive processes, such as language, visual-spatial ability, memory, etc., it is easy to conceive of experimental task manipulation that might results in performance differences that in fact are related to other differences in cognitive processes, rather than to the intended executive control itself. Without knowing the relation of observed task performance to the latent construct, even with experimentally manipulated conditions, there can be unexpected differences in demands that can yield differential results that are unrelated to the question of interest. This issue is particularly prominent when studying groups of very young children who exhibit substantial variation in language, motor, memory, and social skills, all of which can affect the obtained pattern of task performance results. The differential attention to the reliable, ‘‘true score’’ portion of the variance in
performance comes from the difference in intended purpose between psychometric tests and experimental tasks. Most psychometric tests initially were developed to be used in the clinical context where one needs to make inferences about individual children-inferences about whether a child’s level of performance is so discrepant from age peers to warrant a clinical diagnosis or treatment. With such a ‘‘highstakes’’ outcome, it is not surprising that reliability in the relation between latent cognitive construct and observed performance needs to be highly specific and reproducible. On the other hand, experimental tasks do not require such a burden, and are more commonly used to make inferences about groups of children in a general sense. Therefore, commonly, experimental task performance is treated to fully represent the cognitive process of interest, both in terms of its reliability and validity. This treatment can be justified to a degree in that there are no real ‘‘consequences’’ for mis-specification, as scientific knowledge is built through replication. Presumably, if observed differences were spuriously due to some cognitive process other than what was attributed, results from other studies over time would be discrepant from the original to challenge the erroneous inference. In this ‘‘high-stakes’’ context, considerable efforts are made to reliably identify the
cognitive processes that comprise a child’s psychometric test performance at a given point in time. These efforts typically are composed to two basic approaches: (1) characterizing reliability by identifying the consistency in content and performance stability over time; and (2) examining the evidence for convergent and discriminative validity by characterizing the relations between the target test and other measures that are similar=discordant in demands. Because those using experimental tasks
Executive functions and
traditionally have not considered test theory or the resultant psychometric issues regarding reliability as is commonly done with commercially available psychometric tests, here, we apply these two approaches to reliability of experimental tasks that have been used to understand the development of executive control in both typically developing preschool children, as well as those with neurodevelopmental conditions, to elucidate the consequences of treating task performance to fully represent the latent executive cognitive process.
