ABSTRACT
Recent advances in the fields of machine learning and natural language processing (NLP) allow for analyzing scientific reasoning and argumentation by means of automatic coding. As opposed to manual coding, automatically generated codes can be applied with low effort at large scale, and to a broad range of domains. This advantage, however, usually comes at the cost of a decrease in validity as compared with expert codings. In this chapter, we pursue the question of whether automated text classification systems can capture domain-specific patterns of scientific reasoning and argumentation. We report on corresponding automatic coding experiments in different domains and discuss the implications for research and practice. Particularly, we model argumentation and reasoning in a data-driven fashion and look at the results of the automatic coding methodology with respect to scientific reasoning and argumentation in certain domains. To this end, we performed our experiments on datasets reflecting different operationalizations of reasoning and argumentation as well as different skill levels. The data types encompass news text, think-aloud protocols, and collaborative discourse of teacher students, as well as think-aloud protocols of social work students and professionals. We report results from both within- and cross-domain experiments, effectively modeling the impact of the underlying domain on automatic coding. For the detection of relevant linguistic signals in the automatic coding pipeline, we apply a range of NLP techniques, covering semantic, structural, and lexical information. Our results show that domain-specific factors have significant impact on the performance of automatically coding reasoning and argumentation. While the same linguistic signals can be used to automatically code scientific reasoning and argumentation in very different domains and operationalizations of argumentation and reasoning, transferring domain-specific knowledge between domains by means of automatic coding remains very challenging.
