While the measurement and modeling of competencies typically focuses on task outcomes, behavioral differences during task completion are often not considered. With digital technologies, competence assessments can provide process data as additional information about the skills and strategies of test takers. Funded by the German Research Foundation (DFG) we focus on the so-called purchasing literacy of children and explore how process and product data can be used in an extended competence modelling.

Children begin to actively make purchase decisions between the age of six and eight. As an important target group, children are exposed to a multitude of marketing messages from manufacturers and dealers and to an increasing number of advertisements directed at them and geared to their needs. However, their significance as market participants is in contradiction to their lack of market experience and knowledge as well as to their cognitive skills and abilities that are still under development (Schuhen, Mau, Schramm-Klein, & Hartig, 2017). Purchasing literacy comprises the cognitive and motivational-affective skills that enable consumers to manage the entire purchasing process. We assume that purchasing literacy requires the knowledge of different concepts (e.g. pricing), certain mathematical skills (e.g. price-quantity comparisons), learning opportunities, self-regulation, domain-specific problem solving strategies, as well as certain attitudes. However, the extent to which these individual aspects jointly affect purchasing decisions in terms of purchasing literacy has not yet been investigated. In our study we are therefore systematically analyzing the buying competence, buying behavior, and buying decision processes of primary school children for the first time and are testing hypotheses on the structure of purchasing literacy.

Based on our assumptions we developed a computer-based measuring instrument that includes not only achievement and personality tests but also a computer-based simulated supermarket. A comprehensive shift towards computer-based testing (CBA) can currently be observed in the field of educational psychology (Greiff, Scherer, & Kirschner, 2017). CBA opens up the possibility of new innovative item formats and scoring procedures, which can also be observed in international large-scale assessments such as the Programme for International Student Assessment (PISA). Unlike standard paper-pencil assessment, CBAs offer the possibility to consider not only performance (i.e., the correctness of responses), but also the steps in the task completion process. In so-called log files, all actions in a computer-based task can be recorded and all (observable) steps taken towards problem solutions can be retraced afterwards. While the measurement and modeling of competencies usually focuses on outcome indicators, in our study we want to investigate how behavioral differences in achieving a particular outcome (process data) can be included in competency modeling in addition to interindividual differences in task outcomes (product data).

Using a shopping list and a maximum budget, a total of 136 school children aged 7-12 were asked to shop as cheaply as possible for their parents. During the processing time, all interactions of the children with the simulation interface were recorded in a log file. In order to buy the products on the shopping list as cheaply as possible, the children must first find and open the right shelves, identify the products on the shelves that match the term on the shopping list, compare the final prices of relevant products, identify the cheapest product, add it to the shopping cart, and bring the whole purchase to the checkout. With two shelves, there is also the additional requirement to choose between two products that are cheapest but have different price-quantity ratios. Here, in addition to the final price, the base price has to be taken into account. To conduct a psychometric analysis of the data set, and thus a psychometric evaluation of test functioning, a Rasch partial credit model (Masters, 1982) was chosen. One partial credit item was created for each of the eight products on the shopping list. In addition, another item was created to cover whether products were purchased that were not on the shopping list. We used a three-step approach for the scoring of the assessment: (1) Meaningful events of students’ item solving process were defined and extracted from the log data, (2) each process pattern were assigned to a partial credit score and (3) a partial credit model was used for item response theory calibrating of our instrument. In addition to the scoring of the assessment and the psychometric evaluation of the test functionality, we will present our analysis of further process data that reflect processing behavior and allow conclusions about cognitive processing. As the simulation requires a high level of interactivity, the granularity of our log files is high enough for investigating the task completion process. From this, clues were gained as to why a child was able to successfully solve the task or why he or she failed. Using the finite state machine approach and different psychometric methods, we will present results of our theory-based and exploratory process data analyses.

Individual scores for purchasing literacy were estimated using Weighted Maximum Likelihood Estimates (WLE; Warm, 1989) for each student. The fit of data to our model was evaluated by assessing two types of MSNQ (mean squared residuals) fit statistics were infit and outfit. It was found that all items had good MNSQ values (0.66 < MNSQ < 1.00). Reliability was examined as an additional aspect of evaluating our instrument’s quality. WLE reliability was acceptable with .75. Cronbach’s Alpha was even higher with .88. Subsequently, it was investigated whether patterns of behavior can be identified that are not necessary to successfully complete the task, but that can nevertheless predict the children's task success (WLE). Results show that complex construct-relevant cognitive processes, such as domain-specific problem-solving strategies and self-regulation, can be captured via process data. We will present these identified behaviors, their frequency, and their correlations with task success (WLE) and other student variables. Our research demonstrates how process data of an interactive, computer-based task can be used in an extended modelling of competencies and how it enhances the theoretical model and its measurement. We present how process data generate diagnostic added value in computer-based testing. Methodologically, this is not only significant for the measurement and modeling of competencies in general, but also highly relevant for the research areas of educational process mining and digital assessment in schools.

Greiff, S., Scherer, R., & Kirschner, P. A. (2017). Some critical reflections on the special issue: Current innovations in computer-based assessments. Computers in Human Behavior, 76, 715-718. Masters, G. (1982). A rasch model for partial credit scoring. Psychometrika, 47(2), 149–174. Schuhen, M., Mau, G., Schramm-Klein, H., & Hartig, J. (2017). When children become purchasers: A qualitative study for describing the purchasing literacy of children. Zeitschrift für ökonomische Bildung, 6, 171-192. Warm, T. A. (1989). Weighted likelihood estimation of ability in item response theory. Psychometrika, 54(3), 427-450.