Validating the Interpretations of University Students' Mental Processes in Business and Economics using a Mixed-Methods Approach

Author(s):

Sebastian Brückner(presenting / submitting)

Conference:

ECER 2015

Network:

09. Assessment, Evaluation, Testing and Measurement

Format:

Paper

Session Information

09 SES 14 B, Developing and Scrutinizing Tests in the Domains of Finance, Accountancy and Economics

Paper Session

Time:

2015-09-11

15:30-17:00

Room:

328. [Main]

Chair:

Luc Le

Contribution

For many years, validation practices have been criticized for not generating comprehensive explanatory results that would allow test users to draw causal inferences from test scores (Kane, 2013; Zumbo, 2009). Hence, recent validity concepts highlight the need to draw valid inferences from the broadest possible basis of logical and empirical evidence. This means test developers should not only rely on quantitative validation practices based on the final test scores (e.g., correlations of test scores with students’ study progress), but also analyze the underlying mental processes (Borsboom, Mellenbergh & van Heerden, 2004; Zumbo, 2009). In the domain of business and economics, standardized testing instruments are frequently used to draw inferences on students’ economic knowledge based on test scores. However, there is still a lack of research on the mental processes and particularly on “cognitive validity” in this domain (Shavelson, 2013). Cognitive validity is a substantive aspect of validity that reflects the extent to which empirical evidence of mental processes is related to construct-relevant features (e.g., students’ progress of studies) and item scores.

To address this research deficit in business and economics, this paper analyzes cognitive validity of business and economics test items focusing on the following three research questions:

(1) What evidence of mental processes can be found by observing students who think aloud while responding to economics test items?

(2) Do students who are more advanced in their studies use construct-relevant mental processes more frequently?

(3) Are the item scores influenced by these processes?

In relevant international research, the mental processes that occur when university students solve economic problems are addressed mainly in research based on phenomenography (Pang & Marton, 2005), conceptual change (Davies & Lundholm, 2012), and educational psychology (Leiser, 2009; Vernooij, 1995). These fields of research have explored how mental processes can be identified for specific economic topics or concepts (e.g., inflation, pricing, computing selling prices) and have compared the structure of economic reasoning within or between relevant sub-groups with different expertise (e.g., students at different stages of their studies). These approaches assume that understanding economics can be described as a mainly cognitive process. However, from problem-solving research, it is known that this process can also be accompanied and initiated by both affective and metacognitive processes.

When solving specific problems in economics, students generate individual mental models based on their memory of concepts taught in university classes and in various situations and contexts (e.g., the concept of SWOT analysis is taught in marketing and management classes, and students have to apply this concept to solve specific problems). However, several studies (e.g., Leiser, 2009; Vernooij, 1995) have emphasized the fact that “for many students [their individual] mental models are not accurate reflections of the conceptual models [taught in class] required to solve the problems presented” (Vernooij 1995, p. 74). While advanced students use more abstracting and contextualizing processes, the mental models of novice students are rather based on individual experiences from daily life and can be described as superficial economical heuristics (e.g., “Good begets good” (Leiser, 2009)). Hence, research on students’ understanding of economics needs to consider which mental models the students use to solve economic problems. Accordingly, it is important to investigate the extent to which the test items enable university students to build mental models that reflect suitable conceptual models that indicate the correct response (Pellegrino, Baxter, & Glaser, 1999). If students develop mental models containing the intended concepts and if they respond correctly to the items, this will support the validity argument.

Method

In the study, 19 standardized items in different closed-ended item formats (e.g., complex multiple choice) and from different content areas (e.g., accounting, microeconomics) were selected according to IRT-based item parameters from the first field study (N=1,113). They were selected from the item pool of the WiwiKom project, which translated and adapted business and economics test items from the Mexican test Examen General para el Egreso de la Licenciatura (EGEL) and the American Test of Understanding in College Economics (TUCE). The items were administered to 20 students at different stages of their studies. The students’ level of expertise was determined according to different validation criteria (e.g., their current year of studies and the classes attended up to that point). The students were selected through a purposeful random sampling (Patton, 2002), in which the random sample was drawn based on the validation criteria of the field study (e.g., distribution of students’ progress of studies). In total, 380 item response processes were assessed. First, the students had a training phase for getting familiar with the situation. Then, the think-aloud interviews were conducted in a concurrent and a retrospective phase. In the concurrent phase, the students were asked to think aloud while responding to items, and no interactions with the interviewer were allowed. In the retrospective phase, the students summarized their way of responding to the items and responded to verbal probes by the interviewer. Afterwards, the audio-taped data were transcribed, coded inductively, and re-coded in order to answer the qualitative research question (1). Moreover, the qualitative data were transformed into quantitative data (Chi, 1997) in which each occurrence of a process was dichotomously coded. Afterwards, chi-square independence tests and regression analysis were used to explore the significance of the relationships between the coded mental processes, the validation criteria, and the item scores. These analyses supported the credibility of the qualitative data and provided answers to research questions (2) and (3).

Expected Outcomes

The presentation will feature a qualitative and a quantitative part. To answer research question (1), the speaker will highlight different kinds of mental processes from the coded qualitative data. In total, more than 22 processes were identified in the analyses, including cognitive, metacognitive, and affective processes, as well as test-taking strategies. The most frequent ones were cognitive and metacognitive processes. For example, construct-relevant cognitive processes included recalling economic concepts learned primarily at university (e.g., algorithms, terms, research methods), making deductive, inductive, and abductive inferences, and general complexity-reducing heuristics (e.g., “good begets good”). Metacognitive processes consisted mainly in planning or self-checking activities. Furthermore, the analyses revealed construct-irrelevant affective processes, including aspects of motivation and resignation, and test-taking strategies, such as guessing and eliminating. With regard to research questions (2) and (3), exploratory analyses showed that advanced students outperformed novice students, measured in terms of both construct-relevant processes and item scores. Although all mental processes were found in all subgroups, cognitive and metacognitive processes showed the highest effects on the item scores (Brückner, in prep.). The presentation will summarize the current progress of the study and, in addition to the descriptive results, the speaker will discuss some practical aspects of item development (e.g., elimination of distractors with equivalent meaning). Furthermore, the speaker will discuss how the results from the study can contribute to the development of cognitive diagnostic models.

References

Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004). The Concept of Validity. Psychological Review, 111(4), 1061–1071. Brückner, S. (in preparation). Die mentalen Prozesse von Studierenden bei der Bearbeitung wirtschaftswissenschaftlicher Testaufgaben. [The mental processes of university students during responding to business and economics test items.] Dissertation. Johannes Gutenberg-Universität Mainz. CENEVAL. (2010). Examen General para el Egreso de la Licenciatura en Administración (EGEL-ADMON). Chi, M. T. H. (1997). Quantifying Qualitative Analyses of Verbal Data: A Practical Guide. Journal of the Learning Sciences, 6(3), 271–315. Davies, P., & Lundholm, C. (2012). Students’ understanding of socio-economic phenomena: Conceptions about the free provision of goods and services. Journal of Economic Psychology, 33(1), 79–89. Kane, M. T. (2013). Validating the Interpretations and Uses of Test Scores. Journal of Educational Measurement, 50(1), 1–73. Leiser, D. (2009). Lay Understanding of Macroeconomic Causation: The Good‐Begets‐Good Heuristic. Applied Psychology, 58(3), 370-384. Pang, M. F., & Marton, F. (2005). Learning Theory as Teaching Resource: Enhancing Students? Understanding of Economic Concepts. Instructional Science, 33(2), 159–191. Patton, M. Q. (2002). Qualitative Research & Evaluation Methods. Thousand Oaks: Sage Publishing. Pellegrino, J. W., Baxter, G. P., & Glaser, R. (1999). Addressing the “two disciplines” problem: Linking theories of cognition and learning with assessment and instructional practice. Review of Research in Education, 24(1), 307–353. Shavelson, R. J. (2013). An Approach to Testing & Modeling Competence. In S. Blömeke, O. Zlatkin-Troitschanskaia, C. Kuhn & J. Fege (Eds.), Modeling and Measuring Competencies in Higher Education (pp. 29-43). Rotterdam: Sense Publishers. Vernooij, A. (1995). Problem Solving Strategies. In W. H. Gijselaers, D. T. Tempelaar, P. K. Keizer, J. M. Blommaert, E. M. Bernard, & H. Kasper (Eds.), Educational Innovation in Economics and Business: Vol. 1. Educational Innovation in Economics and Business Administration. The Case of Problem-Based Learning (pp. 69–77). Dordrecht: Springer Netherlands. Walstad, W. B., Watts, M., & Rebeck, K. (2007). Test of understanding in college eco-nomics: Examiner's manual (4th ed). New York, NY: National Council on Economic Education. Zumbo, B. D. (2009). Validity as contextualized and pragmatic explanation, and its implications for validation practice. In R. W. Lissitz (Ed.), The concept of validity. Revisions, new directions, and applications (pp. 65–82). Charlotte, NC: Information Age Pub.

Author Information

Sebastian Brückner (presenting / submitting)

Johannes Gutenberg-Universität Mainz

Mainz

Search the ECER Programme

Search for keywords and phrases in "Text Search"
Restrict in which part of the abstracts to search in "Where to search"
Search for authors and in the respective field.
For planning your conference attendance you may want to use the conference app, which will be issued some weeks before the conference
If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.