Introduction

Textbook evaluation studies examine the quality of textbooks based on various criteria related to content, pedagogy and visual design (Lemmer, Edwards, & Sello, 2008; Maun, 2006; Vinisha & Ramadas, 2013). However, there is limited research on examining multimedia learning textbook design from an instructional design perspective (Cheng, Chou, Wang, & Lin, 2015; Mayer, Bove, Bryman, Mars, & Tapangco, 1996). While existing research focuses on examining learning outcomes through experimental methods (Cheng et al., 2015; Mayer et al., 1996), eye-tracking methodology allows an examination of student learning processes with multimedia learning materials in a unique way (Mayer, 2010). The purposes of this study were to (1) investigate multimedia learning design and visual design of a 6th grade Science textbook as a multimedia learning material through eye-tracking, and to (2) create a model and several principles for evaluating multimedia learning of science textbook design. 

Theoretical Background

Textbook evaluation studies in the literature constitute three types: (1) survey method, (2) textbook analysis, and (3) experimental evaluation (Mikk, 2002). Some researchers also used qualitative methods such as interviews with teachers or students to investigate thoughts related to textbook use (Maun, 2006; Lemmer et al., 2008). There were several textbook analysis studies in the literature that examined visuals and the integration of text and visuals in science textbooks. The common evaluation criteria used include type of used visuals, proximity of text and visuals, and the integration of text and visuals (Slough, McTigue, Kim, & Jennings, 2010; Vinisha & Ramadas, 2013).

The cognitive theory of multimedia learning provides some principles to integrate text and pictures in addition to visual design principles, and it facilitates learning (Mayer, 2005; Renkl & Scheiter, 2015). Previous research suggests that students’ performance on science topics increases when science textbooks are designed to align with multimedia learning design principles (Cheng et al., 2015, Mayer et al., 1996). However, there are limitations to cognitive theory of multimedia learning (Mayer, 2005) and eye-tracking methodology allows one to investigate how multimedia design principles work during the learning process in order to examine its limitations (Lai et al., 2013; Lin, Holmqvist, Miyoshi, & Ashida, 2017; Mayer, 2010).

We investigated the student learning process using a 6th grade science textbook that is used in public schools as a multimedia learning material.  This was accomplished by using eye-tracking as well as bringing the perspectives of both experts (authors and graphical designer, etc.) and users (students and teachers) together utilizing multimedia learning design and visual design of science textbooks to extend the perspectives based on the literature. This study helps (a) to understand how multimedia learning design principles and visual design principles facilitate student learning while using a science textbook and by examining the learning process, (b) to expose principles that facilitate the design of science textbooks for student learning needs, and (c) to refine design principles by bringing different perspectives of multimedia learning science textbook design together, thus providing an exemplary model for designing other multimedia learning materials. 

This study employed design-based research (DBR) to design an evaluation model and associated principles, and to investigate multimedia learning and visual designs of a 6th grade science textbook by using eye-tracking. DBR was used in this study since the designing evaluation model and principles required iterative analysis, design, development, and redesign (Wang & Hannafin, 2005). The participant group of the study consisted of 5th grade middle school students, science teachers teaching 6th grade science at schools, and experts experienced in science textbook design or learning materials. Eye-tracking sessions were conducted with student participants while experts and science teachers were interviewed to evaluate 6th grade science textbook by using semi-structured interview protocols. In the first cycle of eye-tracking sessions 12 students participated in eye-tracking sessions. Prior to the session, each participant completed a demographic survey and took a ten-minute prior knowledge test that included six questions about cell biology basics. Subsequently, each participant underwent a brief orientation session during the eye-tracking session. The eye-tracking session began immediately after calibration of the eye-tracker. Participants were presented with six learning task questions identical to those in the prior knowledge test. Participants studied learning tasks such as looking through the textbook individually for an average of 26 minutes. Participants tried to complete learning tasks by using a printed copy of a cell biology chapter in the 6th grade science textbook. After that, a retrospective interview protocol was conducted to learn about a participant’s experience during the eye-tracking process. In the second cycle, there were two groups of participants consisting of up to 35 participants in each group. Participants were randomly assigned to one of the groups. Participants in one group used the "old" version of cell biology chapter in the textbook and the second group used the revised version of the same chapter after the initial revision of the chapter from feedback based on collected data. The other research procedures were the same as described above for the first phase of the study. Learning task performance and eye-tracking data were analyzed using descriptive statistics for the first cycle of eye-tracking sessions. For the second cycle of eye-tracking sessions, the data will be analyzed by using one-way ANOVA statistics to compare the eye-tracking measures and learning task performance of two groups. Qualitative data was analyzed by using content analysis and 7 categories emerged: learner level, contextual, content, motivation, multimedia learning design, pedagogy, and visual design.

Results revealed that all participants completed the tasks partially and stated that the images of the cell on the task sheets were not like those in the book so, some relevant images don’t represent some of the relevant texts in the book properly. They also couldn’t find the true answer or made mistakes because one of the answers was not represented in images or the cues didn’t indicate the relevant part of the text properly. Content analysis results revealed 42 issues related to the science textbook. The most coded issue was “irrelevant images”. Some of the images in the book were also not clear, representative for the topic, distinctive and helpful to transfer of knowledge to different cell images so those caused misconceptions. The second critical issue was that the science textbook was not motivating the students. Three teachers suggested using comic characters for motivation so, cell factory analogy and comic characters were used in the scope of emotional design principle of multimedia learning during revision of science textbook chapter. Seven categories of principles were established: learner level, contextual, content, motivation, multimedia learning design, pedagogy, and visual design. The principles can be used as criteria for evaluating science textbooks. The present research included the combined perspectives of students, teachers and experts differently than those found in the literature. These research findings can be used by experts in the design process to develop a science textbook based on learner needs as well as by teachers to select appropriate science textbooks for students. The research also provides an exemplary model for designing other multimedia learning materials. The final version of the paper will include a complete and detailed model for science textbook evaluation, and the validation of current principles based on empirical data analysis.

Cheng, M., Chou, P., Wang, Y., & Lin, C. (2015) learning effects of a science textbook designed with adapted cognitive process principles on grade 5 students. International Journal of Science and Mathematics Education, 13, 467-488. Lai, M-L., Tsai, M-J., Yang, F-Y., Hsu, C-Y., Liu, T-C., Lee, S. W., Lee, M-H., Chiou, G-L., Chiou, J-C., & Tsai, C-C. (2013). A review of using eye-tracking technology in exploring learning from 2000 to 2012. Educational Research Review, 10, 90-115. Lemmer, M., Edwards, J., & Sello, R. (2008). Educators’ selection and evaluation of natural sciences textbooks. South African Journal of Education, (28), 175–187. Lin, Y. Y., Holmqvist, K., Miyoshi, K., & Ashida, H. (2017). Effects of detailed illustrations on science learning: an eye-tracking study. Instructional Science, 45, 1-25. Maun, I. (2006). Penetrating the surface: The impact of visual format on readers’ affective responses to authentic foreign language texts. Language Awareness, 15(2), 110-127. Mayer, R. E., Bove, W., Bryman, A., Mars, R., & Tapangco, L. (1996). When less is more: meaningful learning from visual and verbal summaries of science textbook lessons. Journal of Educational Psychology, 88(1) 64-73. Mayer, R. E. (2005). Cognitive Theory of Multimedia Learning. In R. E. Mayer (Ed.), Cambridge Handbook Of Multimedia Learning (pp. 31-48). Cambridge University Press. Mayer, R. E. (2010). Unique contributions of eye-tracking research to the study of learning with graphics. Learning and Instruction, 20(2), 167-171. Mikk, J. (2002).Experimental evaluation of textbooks and multimedia. In S.Selander & M. Tholey (Eds.), New educational media and textbooks, The 2nd IARTEM Volume (pp. 121 -140). Stockholm Institute of Education Press. Renkl, A., & Scheiter, K. (2015). Studying visual displays: How to instructionally support learning. Educational Psychology Review, 1-23. Slough, S. W., McTigue, E. M., Kim S., & Jennings, S. K. (2010). Science textbooks' use of graphical representation: A descriptive analysis of four sixth grade science texts, reading. Reading Psychology, 31(3), 301-325. Vijapurkar, J., Kawalkar, A. & Nambiar, P. (2014). What do cells really look like? An inquiry into students’ difficulties in visualising a 3-d biological cell and lessons for pedagogy. Research in Science Education, 44(2), 307-333. Vinisha, K, & Ramadas, J. (2013). Visual representations of the water cycle in science textbooks. Contemporary Education Dialogue, 10(1), 7-36. Wang, F., & Hannafin, M. J. (2005). Design-based research and technology-enhanced learning environments. Educational Technology Research and Development, 53(4), 5-23.