A number of studies have examined how situational factors may affect the way a person responds to test items. These are factors such as a test taker’s mood, hunger, fatigue, physical comfort, time and peer pressure. Unlike systematic conditions like malnutrition or test anxiety, these situational factors represent unstable constructs that can change, from a statistical point of view, randomly. Indeed that is why they are modeled as random error variance in psychometric models, with lower error meaning higher reliability.

As early as 1967, Sattler and Theye highlighted various situational variables, including mood, as having a significant impact on intelligence test performance. In subsequent studies mood has been related to performance in a number of areas, including test performance (Lun, Yeung & Ku, 2023). Hoque and Weil (2016) reported that higher perceived comfort was related to better academic performance.

A tempting conclusion to make would be that well rested test takers in a good mood, working without distractions would perform better. But can we really generalize these observations onto standardized assessment conditions? After all, standardized tests are designed specifically to minimize the effects of every variable except for the construct they are built to measure.

In this study we set out to examine how situational variables such as mood, hunger and physical comfort interact with various aspects of the process of responding to test items. Our target audience was school children in grades 7-9 undertaking a typical standardized assessment. The data was collected as part of a pilot study of a new computerized assessment of Digital Literacy (DL), a complex 21^st century skill that may reductively be described as problem solving in a digital environment.

We were interested in learning to what extent a test taker’s mood, restfulness and physical comfort may be related to different observable aspects of test taking behaviour. That includes overall skill estimates, test taking speed, proportion of skipped items, and individual item functioning.

Our dataset contains 1785 students from 25 schools. We are interested in examining situational variables in the context of a typical standardized assessment, in this case a pilot study of a new tool for measuring DL. This is a computerized scenario-based assessment consisting of 4 large tasks. Each task represents a typical learning-related activity students undertake in their day-to-day, like preparing a presentation for class or planning a school trip, while using simulated digital tools, internet resources and even messengers (with simulated classmates). The scenarios are fully “single-player” and constructed so that every student has to undertake activities in the same way, so that proper standardized measurement is possible. Performance is evaluated with 81 hidden observable indicators (items) derived from test takers’ work products. Each situational factor is measured with a single question. Students could report their mood as being good, normal or bad, and the questions about getting enough sleep and being comfortable had binary response options (yes/no). In terms of statistical analysis, first we are using regression models to quantify the relationship situational variables have with achievement on the DL assessment, amount of skipped items, speed and person fit. We are also using confirmatory factor analysis to model the amount of error variance (of the latent construct and each individual indicator separately) explained by each situational variable. Finally, we are using IRT linear logistic models to estimate whether the relationship between each situational variable and item parameters changes based on how early the item appears in the assessment.

Our large sample size and study design lend themselves well to an investigation of how mood, restfulness and physical comfort interact with different aspects of responding to test items on a standardized test. The assessment situation itself provided plenty of room for observing situational effects. The stakes for students were low. Data was collected in several different schools, so test administration was uneven both in terms of instruction and working conditions for students. Assessment reliability was high, but not too high at (OmegaH=0.7, Alpha=0.78). So although our measures of the situational factors were by necessity brief, we still have a good opportunity of observing significant interactions between our variables of interest. We expect our results to add to the body of work related to how situational factors, usually modelled in assessment as unpredictable random effects, can influence the process of responding to test items. Many studies have highlighted how a person’s mood, for example, can impact performance in specific activities, such as creative thinking or problem solving. However these studies were often designed specifically to find that relationship, and many relied on small sample sizes of below 100 participants. So at least for reproducibility purposes it is important to keep adding to the body of knowledge around the issue. Beyond that, the implications of any outcome for standardized testing are exciting. If there is a significant relationship between situational factors and test taking, it means educators and test administrators can use that to bring the best possible performance out of their students. Simultaneously, this information could help inform our expectations about assessment reliability. If, on the other hand, the relationship is not significant, that is also interesting to know, since then we don’t have to worry about controlling for these situational factors in assessment procedures, guidelines and user interfaces.

Sattler, J. M., & Theye, F. (1967). Procedural, situational, and interpersonal variables in individual intelligence testing. Psychological Bulletin, 68(5), 347–360. https://doi.org/10.1037/h0025153 Lun, V. M. C., Yeung, J. C., & Ku, K. Y. L. (2023). Effects of mood on critical thinking. Thinking Skills and Creativity, 47, 101247. Hoque, S., & Weil, B. (2016). The relationship between comfort perceptions and academic performance in university classroom buildings. Journal of Green Building, 11(1), 108-117.