A Qualitative Analysis of Differential Effect of Multiple-Choice and Open-ended Questions on Metacognition and Affect

Author(s):

Bengi Birgili(presenting / submitting)Ercan Kiraz(presenting)

Conference:

ECER 2016

Network:

Emerging Researchers' Group (for presentation at Emerging Researchers' Conference)

Format:

Paper

Session Information

ERG SES H 12, Studies on Education

Paper Session

Time:

2016-08-23

11:00-12:30

Room:

OB-H2.20

Chair:

George Head

Contribution

In the context of a large-scale Turkish examination system, numerous conflicts and disagreements occur in the process of establishing the best evaluation strategies that measure achievement of pupils’ pedagogical development and academic orientations. The strongest and most productive polemics held between educators and decision makers is about the necessity of reform in developing an effective exam format that ameliorates internal problems of conceptual understanding about evaluating student success, and is about the crises of traditional formats employed. Hence, this study investigates the problem from the perspectives of two different exam styles, multiple-choice (MC) and open-ended (OE). Interestingly, there is a contradictive comprehension among different government officials about these two exam types. Most people consider MC as traditional while consequences of uncertainty forces most others to believe that OE as more contemporary. The emergence of new movements is essential since existing country-wide MC exam is no longer efficient in this era for our millennium kids. On the other hand, are efforts which could be categorized as reform movements really alternatives of the MC approach?

Differential effect of question formats on metacognitive and affective processes of middle school students could be very heterogeneous and, according to course objectives, often diametrically opposite, but they all had one characteristic in common: one is not better than the other. As if one form is better than the other, without any pedagogical-theoretical praxis, rapid changes is about being made on the question format of the examination system in Turkey. Although frequent and sharp public, academic, and politic conflicts and discussions were held throughout the history, according to recent news (TRTNews, 2013; Vatan, 2013), Minister of National Education in Turkey has embarked by exposing to answer open-ended questions instead of multiple choice question formats. However, it seem to be important to note and warn officials that making this process, which gets its support from political courage of ignorance, effective without theoretical and scientific background will be another short-term change, and tendency of unprofessionalism. It should also be noted that a change in statewide examination type requires a great effort and reciprocal contribution of scientists with regard to “what really works” on large-scale examination system.

In this context, the question that arises is whether exam styles are alternatives to each other. There is a point to remember that no examination can be a sole determinant for a better and more accurate result in measuring achievement than the other. Almost all types of questioning formats have their own constraints; Multiple-choice, for example, causes a tendency for guessing or studying to the test rather than studying for the test. On the other hand, Open-ended formats techniques measure somewhat separable constructs. Each measurement technique requires different applications of cognitive strategies and learners, however, develop both cognitive and exam taking strategies congruent with the type of exam. In addition, there was a high tendency to assess achievement through the use of MC question types. MC concentrates on remembering, distinguishing, selecting, and so forth whereas OE focuses on understanding, explaining, justifying, and creating. In both question types, cognitive, metacognitive and affective constructs are seem to be related. For instance, self-checking and cognitive strategy had been selected an important aspect of metacognition because the development of individual’s capacity to think about how they learn through the process by being self-awareness can be reflected with these strategies. Moreover, worry and effort are seen as significantly important due to the fact that worry is important for having cognitive relation to affective experiences, and effort invested in a task is highly related the affective predictions and success in this task.

Method

In this study, cognitive strategy and self-checking dimensions of metacognition, and worry and effort dimensions of affect were explored through common experiences of 8th-grade students, their teachers, and academicians. For this purpose, they were selected purposefully for the intention of qualitative inquiry. The researchers used phenomenological approach and applied cognitive interview with 32 participants. The primary data collection method was in-depth interviews. For instance, as so involving overall dimensions such questions were imposed to the participants; “How do you control computations while solving open-ended problems?”, “How do you test your own test-taking and cognitive performances while solving multiple-choice questions?”, “Do open-ended questions require more effort? If so, why? How?” The participants were included in to cognitive interview (Willis, 1999), and focus group interviews in order to reveal the participants’ perceptions on overall dimensions of metacognitive and affective aspects of examination process. More importantly, since one of the critical aspects of qualitative inquiry is to provide trustworthiness, interview questions and instruments that were developed throughout the study were reviewed by experts. To provide credibility, which is the degree the researcher’s analysis find participant agreement, member check technique was utilized (Lincoln & Guba, 1999). Each participant were given exclusive access to their interview transcripts and read them thoroughly for clarity, accuracy so that they provided additional insight and information when necessary. Furthermore, the role of the researcher in the study was not to generate replicability, rather it was to describe the environment through those who experience it. Thus, member checks also enhanced the level of dependability of this qualitative study (Merriam, 2009). While examining the differential effect of MC and OE question formats in terms of cognitive strategy, self-checking, worry and effort; “think aloud" (Ericsson & Simon, 1990) and "verbal probing" approach were employed to understand how participants differentiate between two types of exams. Detailed information and descriptions of exam phenomena from the participants’ perspectives were asked throughout the study. Data gathered during the process were analyzed through content analyses method (Krippendorff, 2004). Theoretical framework of this study was confined to discuss differentiated effects of MC and OE consistent with cognitive strategy, self-checking, worry, and effort sub-dimensions from the experiences of 8th-grade students, middle school teachers, and academicians. Codes, categories, and themes were developed based on information gathered from interviews. Hence, twenty-three categories and four themes emerged to explicate the purpose and results of the study.

Expected Outcomes

This current research aimed to reveal the differential effect of OE and MC question formats in terms of metacognition and affect qualitatively. The main research question guided this study was “what are the perceptions of middle school students, teachers, and academicians about MC and OE exams in relation to metacognition and affect?” The declaration from Minister of National Education of Turkey thematized that “OE would be preferred radically instead of MC since they intend to measure student achievement better.” (Tartanoglu, 2013; Vatan, 2013). However, results of this study revealed that both types have important strength and weaknesses and there is no need for exaggerated inclination toward one. According to twenty-three categories of four themes from this phenomenological study indicated that 1) the students prefer to use their own constructed responses or solution steps; 2) higher number of different cognitive strategies are used; 3) the children have a re-wording tendency through the problem root; 4) they spend more time to understand the question, and 5) they prefer re-reading the question or solution steps. In addition, common perceptions of the participants justified that the learners: 1) check their work comparatively more 2) go over their solution steps or answer choices equally in both MC and OE; 3) judge their correctness of the solution; 4) consciously ask themselves how well they are doing; 5) tend to find and correct their errors more; and 6) remind themselves to stay on track. It can be remarked that OE may promote metacognition of children more than MC about cognitive strategy and self-checking skill. Moreover, in terms of worry and effort, the study showed that both OE and MC may initiate worry over the students similarly.

References

Birenbaum, M., & Pinku, P. (1997). Effects of test anxiety, information organization, and testing situation on performance on two test formats. Contemporary Educational Psychology, 22, 23-38. Cheng, H. (2004). A comparison of multiple-choice and open-ended response formats for the assessment of listening proficiency in English. Foreign Language Annals, 37(4), 544-553. Efklides, A., Papadaki, M., Papantoniou, G., & Kiosseoglou, G. (1997). The effects of cognitive ability and affect on school mathematics performance and feelings of difficulty. American Journal of Psychology, 110, 225–258. Efklides, A. (2006). Metacognition and affect: What can metacognitive experiences tell us about learning process? Educational Research Review, 1, 3-14. Ericsson, K. A., & Simon, H. A. (1990). Verbal reports as data. Psychological Review, 87, 215-250. Gültekin, S., & Çıkrıkçı-Demirtaşlı, N. (2012). Comparing the test information obtained through multiple-choice, open-ended and mixed item tests based on item response theory. Elementary Education Online, 11(1), 251-263. Krippendorff, K. (2004). Content analysis: An introduction to its methodology. Thousand Oaks, CA: Sage. Lincoln, Y.S., & Guba, E.G. (1999). Judging the quality of case study reports. Qualitative Studies in Education, 3 (1), 53-59. Martinez, M. E. (2006). What is metacognition?. Phi Delta Kappan, 696-699. Merriam, S. B. (2009). Qualitative research: A guide to design and implementation. San Francisco: Jossey-Bass. O’Neil, H. F., & Brown, R. S. (1998). Differential effects of question formats in math assessment on metacognition and affect. Applied Measurement in Education, 11(4), 331-351. PISA (2012) PISA 2012 Results in Focus. Retrieved on March 10, 2014, from http://www.oecd.org/pisa/keyfindings/pisa-2012-results-overview.pdf Tartanoğlu, S. (2013, April). SBS’de Kazak sistemi. [Kazakh system in SBS] Cumhuriyet Eğitim, Retrieved on May 10, 2013, from: http://www.cumhuriyet.com.tr/?hn=413814 TRT News (2013, April). SBS’de açık uçlu sorular üzerinde çalışılıyor. Retrieved on May 16, 2013, from: http://www.trthaber.com/haber/egitim/sbsde-acik-uclu-sorular-uzerinde-calisiliyor-84084.html TUSIAD (2013). PISA 2012 değerlendirmesi: Türkiye için veriye dayalı eğitim reform önerileri. [Evaluation of PISA 2012: Education Reform Suggestions Depending on Data for Turkey.] Vatan (2013, April). SBS’nin yerine geliyor. [Coming instead of SBS] Vatan Gündem. Retrieved on May 15, 2013, from: http://haber.gazetevatan.com/sbsnin-yerine-geliyor/533852/1/gundem. Willis, G. B. (1999). Cognitive interviewing: A “How To” guide. Meeting of the American Statistical Association. Yıldırım, T. P. (2011) Understanding the modeling skill shift in engineering: The impace of self-efficacy, epistemology, and metacognition. (Unpublished Doctoral Dissertation). University of Pittsburgh. Zeidner, M. (1987). Essay versus multiple-choice type classroom exams: The student’s perspective. The Journal of Educational Research, 80(6), 352-358.

Author Information

Bengi Birgili (presenting / submitting)

MEF University, Turkey

Ercan Kiraz (presenting)

Abdullah Gul University, Faculty of Educational Sciences, Turkey

Search the ECER Programme

Search for keywords and phrases in "Text Search"
Restrict in which part of the abstracts to search in "Where to search"
Search for authors and in the respective field.
For planning your conference attendance you may want to use the conference app, which will be issued some weeks before the conference
If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.