Many works have been attempted to transform more adaptive physics education. For example, implementing adaptive tutoring system, administering computerized adaptive test, and providing constructive feedback for addressing students’ learning difficulties. Physics teachers often create important decisions to implement appropriate learning strategies to address students’ diversity. In practice, this decision could be thought of as teachers’ task to make a future prediction of students’ learning outcomes. The teachers’ decision in the classroom would be useful for students if it can enhance students’ learning. Unfortunately, human judgment made by physics teachers often produces inaccurate and overestimated feedback to predict students’ learning (Li et al., 2022).

To tackle this problem, we offer a solution through the implementation of machine learning (ML) technology that is a branch of artificial intelligence (AI) studies. The ML has been gaining popularity in the physics education research (PER) field for recent years (Aiken et al., 2019; Stewart et al., 2022; Yang et al., 2020; Zabriskie et al., 2019). It can be offered to provide data driven feedback through predictive information. Therefore, physics teachers could improve their prediction accuracy and it can be expected to overcome aforementioned challenges. 

Nevertheless, former studies on the ML implementation still face limitations on feature selection and hyperparameter optimization methods during the training process of the ML model. 

Feature selection is an optimization task to select the most important features for producing the most accurate prediction by the trained ML model. Former method which is based on the wrapper method such as recursive feature elimination (RFE), as attempted by previous scholars (Aiken et al., 2019; Stewart et al., 2022; Yang et al., 2020; Zabriskie et al., 2019), discovers accuracy bias since it might be dependent to the selected ML technique. Hence, we offer an alternative approach to the filter method using the item response theory (IRT) framework (Kline et al., 2021). We extract features through studying the psychometric properties of each feature/ variable in producing the output or the prediction target (students’ learning outcome).

Furthermore, there are some hyperparameters of an ML model that should be tuned thus we can achieve the most optimum ML performance. For example, this study employed one of the most employed ML techniques in former PER studies, i.e. random forest (RF) (Breiman, 2021). We should set some hyperparameters such as number of trees and number of selected features on each split prior to the training of the RF model. Hyperparameter tuning through the random and grid search methods is relatively simple but computationally inefficient. Therefore, a more efficient strategy should be offered and we choose a novel approach of genetic algorithm (GA).

To guide this study achieving the goals, we address research questions below.

1. What are the psychometric properties of the features analyzed using the IRT framework to predict students’ performance on physics through the RF model?

2. What are the optimized RF hyperparameters using the GA approach to predict students’ performance on physics?

3. How is the RF performance after it has been optimized by IRT and GA?

4. Is there a significant difference of prediction accuracy between the human judgment made by physics teachers and the trained ML model?

Our dataset to train the RF model was collected from some research based assessments (RBAs) established in the field. They include multidimensional aspects of students’ learning on high school physics. The conceptual aspect was probed using the Force Concept Inventory (FCI), the Force and Motion Conceptual Evaluation (FMCE), the Rotational and Rolling Motion Conceptual Survey (RRMCS), the Fluid Mechanics Concept Inventory (FMCI), the Mechanical Waves Conceptual Survey (MWCS), the Thermal Concept Evaluation (TCE), and the Survey of Thermodynamic Processes and First and Second Laws (STPFaSL). We also measured learning outcomes in terms of attitude toward physics and students’ skill in the laboratory. They were probed using the Scientific Abilities Assessment Rubrics (SAAR) and the Colorado Learning Attitudes about Science Survey (CLASS). A total of 497 students were involved from three large and a small public high schools located in a suburban district of a high-populated province in Indonesia. Some variables related to demographics, accessibility to literature resources, and students’ physics identity were also studied. We decided to incorporate variables of literature accessibility and physics identity because some scholars suggest their potential impact on students’ learning in physics.

In summary, IRT was successfully implemented to extract the psychometric properties of features in the dataset for predicting students’ performance at the end of the learning process which was independent of the selected ML technique. Thus, it can mitigate the issue of accuracy bias as reported by the former methods. The GA implemented in this study obtained the best hyperparameter of the RF model. Thus, it could optimize the predictive performance in terms of accuracy, sensitivity, specificity, recall, F1-measure, and area under receiver operating characteristic curve (AUROC). The ML performance achieved a plausible performance after the optimization using the IRT and GA and outperformed the human judgment made by physics teachers.

Aiken, J. M., Henderson, R., & Caballero, M. D. (2019). Modeling student pathways in a physics bachelor’s degree program. Physical Review Physics Education Research, 15(1). https://doi.org/10.1103/PhysRevPhysEducRes.15.010128 Breiman, L. (2001). Random Forests. Machine Learning, 45(1), 5–32. https://doi.org/10.1023/A:1010933404324 Kline, A. S., Kline, T. J. B., & Lee, J. (2021). Item response theory as a feature selection and interpretation tool in the context of machine learning. Medical and Biological Engineering and Computing, 59 (2). https://doi.org/10.1007/s11517-020-02301-x Li, Q., Xu, S., Chen, Y., Lu, C., & Zhou, S. (2022). Detecting preservice teachers’ visual attention under prediction and nonprediction conditions with eye-tracking technology. Physical Review Physics Education Research, 18(1), 10134. https://doi.org/10.1103/PhysRevPhysEducRes.18.010134 Raji, I. D., Bello-Salau, H., Umoh, I. J., Onumanyi, A. J., Adegboye, M. A., & Salawudeen, A. T. (2022). Simple Deterministic Selection-Based Genetic Algorithm for Hyperparameter Tuning of Machine Learning Models. Applied Sciences, 12(3). https://doi.org/10.3390/app12031186 Stewart, J., Hansen, J., & Burkholder, E. (2022). Visualizing and predicting the path to an undergraduate physics degree at two different institutions. Physical Review Physics Education Research, 18(2), 20117. https://doi.org/10.1103/PhysRevPhysEducRes.18.020117 Yang, J., Devore, S., Hewagallage, D., Miller, P., Ryan, Q. X., & Stewart, J. (2020). Using machine learning to identify the most at-risk students in physics classes. Physical Review Physics Education Research, 16(2). https://doi.org/10.1103/PhysRevPhysEducRes.16.020130 Zabriskie, C., Yang, J., Devore, S., & Stewart, J. (2019). Using machine learning to predict physics course outcomes. Physical Review Physics Education Research, 15(2). https://doi.org/10.1103/PhysRevPhysEducRes.15.020120