ABSTRACT
Chapter 4 describes the research questions, dataset, and data analysis procedures of the study. The study aimed to examine differences (across task types and student proficiency levels) and changes (across test occasions) in the linguistic and discourse characteristics of 276 essays written by 85 Chinese EFL (English as a foreign language) learners who each completed the same TOEFL-iBT® independent and integrated writing tasks before and, again, after a period of English language instruction. Only 48 students completed both tasks on both test occasions, creating an unbalanced design. Students were divided into four proficiency groups based on their TOEFL-iBT® scores before instruction. The 276 essays were analyzed using a combination of human rating, manual coding, and computer analysis (e.g., Coh-Metrix, Criterion, TAALES) in terms of six competencies and that match the TOEFL-iBT® rating criteria: grammatical, discourse, sociolinguistic, strategic, content, and source use. The chapter describes the various indices used to operationalize and measure each of the six competencies. Results from essay analyses were then compared, using non-parametric analysis of variance (PERMANOVA), in order to examine differences and changes in L2 writing performance across tasks, proficiency levels, and test occasions. Issues with the design of the study and computer and statistical analyses are also discussed. The chapter also describes procedures used to ensure and evaluate data quality, such as rater and coder training and reliability.
