Argumentation is an important practice in science by which scientists generate, justify and evaluate scientific claims. Argumentation also plays a central role in science education because it leads students toward deep learning by engaging them in the practice of constructing and evaluating scientific arguments. Students therefore need to be explicitly taught and assessed in scientific argumentation (SA) to acquire the competency to engage in the practice. There has, however, been little study on developing assessment instruments for this practice despite the growing interest in the topic. More knowledge is needed about how items can be written and what factors should be considered to promote the demonstration of SA.

This article is based on a three round qualitative study in an iterative process of developing an instrument to assess scientific argumentation competency (SAC) in the context of Physics. In this study, SAC is decomposed into three components with hypothesized increasing cognitive demand: Identification of SA, Evaluation of SA and Production of SA. The focus of this article is to explore how students interact with the assessment instrument and the factors that should be considered to improve the assessment.  

In the first round of the study, the author conducted one semi-structured interview with one Chinese Physics teacher, and two think-aloud interviews and two follow-up interviews with students. In the second round of the study, 6 panels including 10 teachers were organized for the instrument review, 4 students were invited for the think-aloud interview, and 30 students attempted the test with 4 students taking part in a follow-up interview. In the third round of the study, 9 students participated in the think-aloud interview while 2 students were invited for the follow-up interview. All the interview data were collected using video/audio calls, and transcribed. The author analyzed all the interview transcripts using thematic analysis.  

Five factors emerged from the first round of the study are considered for the revision of the instrument. These are Language, Scenario arrangement, Argument opportunity, Scoring rubric grain size and Construct definition. The five factors were categorized into three aspects, namely, Item design, Scoring rubrics and Construct map according to Wilson (2013)’s “Four Building Blocks” approach. Findings from the first round of the study contribute to the revision of the instrument for the next round. In terms of the second round of the study, both teachers’ and students’ interview generated 4 factors respectively, with 3 factors overlapped. No new factors contributing to the instrument improvement appeared in the third round of the study. In total, there are 10 factors emerging from the study, specifically, Scenario arrangement, Item interrelationship, Explicating the problem to be argued, SA-related terms clarification and expression, Information provided in the task, Understandable and clear language, Scientific accuracy, Test length, Scoring rubric grain size and Construct definition. The author argues that these factors provide detailed information in supporting SAC assessment research although its universality has not been tested.

SAC Conceptualisation This study takes SAC as the competency of using scientific evidence and language to defend one’s scientific claims reasonably, meanwhile using scientific evidence to evaluate the advantages and weaknesses of others’ arguments. Drawing on Toulmin (1958)’s argumentation pattern, Erduran (2004)’s analytical framework, and Kuhn (2013)’s idea of developing argument competency, SAC is decomposed into three components with hypothesised increasing cognitive demand: Identification of SA, Evaluation of SA and Production of SA. Each component contains several elements: claim, use of evidence, explanation and rebuttal. For each element, there are indicators referring to the aspects from which the element is assessed. For example, the indicators for “use of evidence” are relevance, accuracy and sufficiency. The detailed description of elements and indicators would be progressively evaluated and modified according to the emerging evidence of its appropriateness during the research process. The Iterative Process of Assessment Development Drawing upon Wilson (2004)’s four building blocks approach of developing assessment, and Newton (2017)’s approach of developing validation arguments that emphasises on both the outcome and the design process of assessment, the instrument here includes the construct map that articulates the competency to be measured, a written test, and scoring rubric for each item, and its design follows the “trial in the field-feedback-reflection-revision” cycle until the instrument is valid enough for the research. All the students were from the north of China and were in grade 11 when participating in the study. Besides, all of them were 16-17 years old. Teachers were from different places of China, two of them were science education researchers and the rest of them were Physics teachers in high school. For teachers who would like to join in a group discussion, an item panel discussion was conducted; others participated in a semi-structured interview. Teachers were given a brief research summary and the instrument around 7 days before interview, so they had enough time to read the materials. Students who participated in the think aloud interview were given the test after the interview begins. The interviewer told participants how to think aloud and provided them with an example before asking them to speak their thinking aloud. All the interview transcripts were analysed using Nvivo. In each round of the study, teachers’ and students’ interviews were analysed separately to search for emerged influencing factors, then factors were taken together for instrument modification.

Results Factors identified from the study Five factors emerging in the first round of the study contribute to instrument revision. Combined with factors found in the second round of the study, a total of 10 factors emerged. Test design before and after modification Initially, 8 task examples, each of which contains one or more items, were designed to see how they work in the field. Each item adopting either open-ended form or matching form assesses one SAC element. The content knowledge involved in these tasks is Motion and Force that had been learned by participants. After considering the factors that emerged from the interview data in all the three rounds of study, the modified test includes 9 tasks with three items for each SAC element. Each task, comprising of several items, focuses on one problem to be argued. Conclusion In this study, we presented influencing factors found in an iterative instrument development process that contribute to the improvement of SAC assessment. Based on the findings of the study, we conclude that the iterative process is reasonable in designing the instrument, both teachers and students provided useful feedback. Besides, the instrument performed better in the field after considering these factors and brought positive consequences to participants. Students were interested in taking this “unusual formed” test and felt much less confusing and provided more positive feedback about interacting with the modified instrument that was revised according to the factors shown above. Although students seldom been engaged in SA, they found it helpful in promoting their thinking ability and learning. We argue that this study provides detailed information in supporting SAC assessment research, and further studies are encouraged to test its application in other contexts.

Diana, N. Y. P. (2019). Assessment of Argumentation in Chemistry: A Model for Designing Items. In Argumentation in Chemistry Education (pp. 106-141). Newton, P. E. (2017). An approach to understanding validation arguments. Ofqual, Coventry. Erduran, S., Simon, S., & Osborne, J. (2004). TAPping into argumentation: Developments in the application of Toulmin's argument pattern for studying science discourse. Science education, 88(6), 915-933. Berland, L. K., & McNeill, K. L. (2010). A learning progression for scientific argumentation: Understanding student work and designing supportive instructional contexts. Science Education, 94(5), 765-793. Wilson, M. (2004). Constructing measures: An item response modeling approach: Routledge. Toulmin, S. (1958). The uses of argument. Cambridge: Cambridge University Press. Kuhn, D., Zillmer, N., Crowell, A., & Zavala, J. (2013). Developing norms of argumentation: Metacognitive, epistemological, and social dimensions of developing argumentive competence. Cognition and Instruction, 31(4), 456-496. Osborne, J. F., Henderson, J. B., MacPherson, A., Szu, E., Wild, A., & Yao, S. Y. (2016). The development and validation of a learning progression for argumentation in science. Journal of Research in Science Teaching, 53(6), 821-846.