INTRODUCTION

Assessment is essential in education since it identifies and tracks students' knowledge and skills while also improving their academic performance and learning (Wiggins, 1998). Commonly used assessment methods, like classical test theory or item response theory, offer a single score, making it simple to rank, compare, and even estimate students' future performance. However, these models give limited diagnostic information regarding students' strengths and weaknesses (Choi, 2010). Effective assessments should contribute to students' learning when they provide immediate and effective feedback. The need to provide data-driven feedback has led to an increasing awareness of the importance of diagnostic assessments based on Cognitive Diagnostic Models (CDMs), which offer detailed information about students' strengths and weaknesses on learning attributes (Rupp et al., 2010).

CDMs offer diagnostic information by categorizing students according to their mastery of skills (DiBello et al., 2007). In CDMs, variables are observable and measurable by using Cognitive Diagnostic Assessments (CDAs) (Cheng, 2010). CDMs offer a multidimensional view of cognitive strengths and weaknesses. CDMs are considered latent class models since students are classified depending on how similar their answers to items are (Haagenars and McCutcheon, 2002). CDMs are called restricted latent class models because the number of latent classes is restricted by the number of attributes in test items (Ravand & Robitzsch, 2015).

Despite their benefits, CDMs are not widely implemented in classrooms, largely due to limited awareness and practical applications. Recent interest in CDMs stems from initiatives like the No Child Left Behind Act, emphasizing formative assessments to improve student learning outcomes (NRC, 2001).

According to Polya (1962), a problem is a situation a person or organization faces with no apparent or clear solution. Problem-solving requires mental processes that need a defined and directed aim to achieve, even if the solution to the problem is unknown (Van de Walle, 2014). Problem-solving is a process that includes reading and comprehending the problem, selecting relevant data from the available options, expressing the data using mathematical notation, and performing the requisite operations to obtain the answer. Mathematics problems are classified as routine and non-routine problems. Routine problems are computational challenges that are well-known and have standardized strategies. On the other hand, non-routine problems require advanced thinking skills because there is no quick, clear way to solve them (Arikan et al., 2020).

Significance and Aim of Study

The current study answers the question, “What are the cognitive strengths and weaknesses of 4^th grade students for the attributes of routine and non-routine problem-solving skills with four operations (addition, subtraction, multiplication, and division)?” This study uses CDMs to examine 4^th grade students' strengths and weaknesses in problem-solving skills. The study aims to develop a CDA-based instrument to provide alternative feedback and assessment for both learners and instructors.

This study may benefit the field for several reasons. Unlike other CDA research, which employs items from large-scale assessments (Li et al., 2020), this study creates items particularly for diagnostic purposes based on predefined attributes. Second, the current study gives students diagnostic feedback on four-operation mathematical problem-solving. Although several CDM studies concentrate on mathematics success (Sen & Arican, 2015), there is no study on designing a CDA for mathematical problem-solving. Additionally, the current study seeks to highlight ability differences among students with the same total score. Differentiating student profiles is important for problem-solving, and CDAs allow for customized feedback based on each person's unique strengths and weaknesses.

METHODS Participants Samples of this study consist of 511 4th grade students from six various private and public schools in Istanbul. Before administering the test, 10 students were voluntarily selected for interviews about the test items. The clarity of the items and timing were evaluated, and necessary revisions were made based on these interviews. The participants answered 20 questions in 60 minutes in their classrooms. Instrument and Test Design According to the aim of the study, a cognitive diagnostic mathematics test was developed and administered to 4th-grade students. The test includes 6 attributes and 20 items. These items were prepared by reviewing 2021-2022 4th-grade mathematics textbooks, Ministry of Education resources, and various textbooks with routine and non-routine questions. The development of non-routine problems was assisted by cooperation with the Bogazici University Adaptive Test Laboratory. Initially, 30 items were developed, but during the elimination process, timing, clarity, and appropriateness for the students’ level were considered, resulting in a 20-item test. To prevent cheating and minimize teacher-student interactions, four booklets with the same items in different orders were created. There are different suggested paths while designing a CDA test. Regarding different pathways to develop a CDA test, the steps for the current study are determined as follows: identifying the aims and attributes of the study, design selection, writing items, Q-matrix construction, test administration, model fit, and validity check and revision. Attributes The attributes are problem-solving skills involving four operations: addition, subtraction, multiplication, and division. Cognitive domains include routine and non-routine problems. Similar studies (Su et al., 2013) were reviewed to validate the attributes, which are categorized as follows: adding 2-, 3-, and 4-digit numbers; subtracting 2-, 3-, and 4-digit numbers; multiplying 2-, 3-, and 4-digit numbers; dividing 2-, 3-, and 4-digit numbers; solving routine problems; and solving non-routine problems. Q-Matrix and Data Analysis The initial Q-matrix, containing 20 multiple-choice items for 6 attributes, was developed by the researcher. 26 attribute profiles will be constructed by the package to show students’ weaknesses and strengths. Q-matrix was reviewed by one academic, one homeroom teacher, and two math teachers, leading to some revisions. Regarding their suggestions, some revisions are made to the Q-matrix. The data of the study is analyzed with the GDINA model using the GDINA package, version 2.8.8. (Ma and de la Torre, 2022) and the CDM package (Robitzsch et al., 2022) in R software (4.2.0).

RESULTS The results of the study indicated that routine problem-solving (75%) was the most mastered attribute, whereas non-routine problem-solving (17%) was the least mastered. GDINA parameters demonstrated that students could resolve routine problems without acquiring a comprehensive understanding of all related attributes, whereas non-routine problems required a complete understanding of foundational attributes. The research shows that multiplication and division are more difficult to master, as subtraction was more frequently mastered than other operations. Regarding the attribute prevalence, it can be said that the mastery probability for subtracting skills (.57) was highest followed by division (.33), multiplication (.33), and addition (.31) skills. Item analysis determined that most items had acceptable estimating and slippage parameters, except for items 14 and 19, which exhibited greater slipping rates, suggesting their difficulty. Non-routine problem-solving items had larger sliding parameters, indicating their difficulty. The study identified 64 latent attribute classes, ranging from the completely non-mastery profile (000000) to the entire mastery profile (111111). The most frequent attribute profiles observed were 000010, 010010, 111111, 111110, 000000, and 110010. Notably, non-routine problem-solving was mastered only in the 111111 profile (6.16%). Similarly, multiplying and dividing were mastered only in profiles 111111 (6.16%) and 111110 (5.89%). Additionally, the research used CDA to offer individual-level feedback, identifying different strengths and weaknesses even among students with the same score. Students with equal scores on a standard examination, for example, have different mastery profiles; they could be very good at subtraction but have trouble with addition or handling non-routine problems. This highlights the limitations of traditional assessments and the benefits of diagnostic tools for tailored feedback. In conclusion, the study demonstrated the use of GDINA and diagnostic assessments to identify specific weaknesses and strengths. These tools enable educators to adjust teaching strategies and support students more effectively, fostering individualized learning.

REFERENCES Arikan, S., Erktin, E., & Pesen, M. (2020). Development and validation of a STEM competencies assessment framework. International Journal of Science and Mathematics Education, 20(3), 1-24. Cheng, Y. (2010). Improving cognitive diagnostic computerized adaptive testing by balancing attribute coverage: The modified maximum global discrimination index method. Educational and Psychological Measurement, 70(6), 902-913. Choi, H. J. (2010). A model that combines diagnostic classification assessment with mixture item response theory models (Doctoral dissertation). University of Georgia. DiBello, L., Roussos, L., & Stout, W. (2007). Review of cognitively diagnostic assessment and a summary of psychometric models. In Handbook of Statistics, 26(1), 979-1030. Haagenars, J., & McCutcheon, A. (2002). Applied latent class analysis. Cambridge University Press. Li, L., Zhou, X., Huang, J., Tu, D., Gao, X., Yang, Z., & Li, M. (2020). Assessing kindergarteners’ mathematics problem solving: The development of cognitive diagnostic tests. Studies in Educational Evaluation, 66(1), 1-10. Ma, W., & de la Torre, J. (2022). GDINA: The generalized DINA model framework. R package version 2.8.8. Retrieved from https://CRAN.R-project.org/package=GDINA National Research Council. (2001). Knowing what students know: The science and design of educational assessment. Committee on the Foundations of Assessment. Reston, VA. Polya, G. (1962). Mathematical discovery: On understanding, teaching, and learning problem solving. John Wiley. Ravand, H., & Robitzsch, A. (2015). Cognitive diagnostic modeling using R. Practical Assessment, Research, and Evaluation, 20(1), 11-24. Robitzsch, A., Kiefer, T., George, A. C., & Ünlü, A. (2022). CDM: Cognitive diagnosis modeling. R package version 8.1-12. Retrieved from https://CRAN.R-project.org/package=CDM Rupp, A. A., Templin, J., & Henson, R. (2010). Diagnostic measurement: Theory, methods, and applications. Guilford Press. Sen, S., & Arican, M. (2015). A diagnostic comparison of Turkish and Korean students’ mathematics performances on the TIMSS 2011 assessment. Journal of Measurement and Evaluation in Education and Psychology, 6(2), 238-253. Su, Y. L., Choi, K. M., Lee, W. C., Choi, T., & McAninch, M. (2013). Hierarchical cognitive diagnostic analysis for TIMSS 2003 mathematics. Centre for Advanced Studies in Measurement and Assessment, 35(1), 1-71. Van De Walle, J. (2014). Assessment through problem-solving-elementary & middle school mathematics: Teaching. Pearson Education. Wiggins, G. (1998). Educative assessment: Designing assessments to inform and improve student performance. Jossey-Bass Publishers.