16 SES 11 A, Foreign Language Learning and ICT
Emotion incitement as a strategy to increase levels of participation and identification of the audience has always been one of the main aims in the field of audiovisual creation and production. Sound design has the capacity, thanks to the veridiction pact (Zunzunegui, 1991) and to the semi-conscious perception of sound (Murch, 2003), of generating emotions that intensify the involvement and participation of the audience. The emergence of more interactive audiovisual content introduces new techniques for the creation and organization of sound elements, increasing the audience's levels of immersion through sound. The appearance of spherical sound production systems (three-dimensional listening through headphones) is a sample of this progression: "Both gaming and film are moving quickly into providing a sense of total audio immersion. […] Market realities suggest that the primary thrust of 3D audio innovation will occur over headphones" (LaGrou, 2014: 7). Significantly, several studies have shown that a greater emotional impact is achieved by immersive sound systems (Grimshaw, 2008, Jørgensen, 2006, Murphy and Pitt, 2001 or Stockburger, 2006).
Sound design can have a significant development beyond the audiovisual industry, with applicability in fields such as therapeutic rehabilitation, stress reduction or reinforcement in learning. Regarding the learning process, it has been found that an emotionally marked experience is best remembered, especially because of the connection between the hippocampus and the amygdala (Mora, 2005). However, little research has been done on the possibilities of using sound as a resource for learning improvement. We can highlight the study carried out at the MOVES Institute (Shilling et al, 2002), which verified the emotional impact of sound in video games and concluded that emotional activation has a positive impact on learning, performance and the feeling of immersion in virtual environments.
According to the previous findings, the hypothesis of this study was the following: the increase of the audience immersion in an audiovisual narration with a soundtrack in 3D audio format has a positive impact on listening comprehension in a foreign language (L2).
From this hypothesis, the specific objectives of this research study were twofold: first, to evaluate the influence that the spatial sound mix configuration (stereo mix vs 3D audio mix) has on the listening comprehension of the story. Second, to measure the student's emotional and attentional impact achieved by the version of the stimulus they had watched and listened.
For the experimental study, a three minute fiction audiovisual sequence was created, containing a complete soundtrack based on sound effects, ambiences and dialogues, the latter recorded taking into account the English level of the participants. Two different versions of the sequence were elaborated for each English level: one mixed in stereo format and the other mixed in surround 3D format (from a 5.1 channel configuration), both produced to be listened through headphones. Nuendo 7.1 and Spatial Audio Designer (from New Audio Technology) were used to produce both mixes. Participants were randomly assigned a specific version of the audiovisual stimulus, and watched it using an 8” tablet with stereo over-ear headphones. The sample consisted of 330 students from 4th Primary and 1st and 4th Secondary Education coming from two different Schools in Seville. The students’ English level had been previously tested: A2 for 4th Primary and 1st Secondary and B1 for 4th Secondary Education. For the evaluation of listening comprehension, a five multiple-choice question test was used (based on Cambridge examination standard). The emotional and attentional impact was measured using three different instruments: physiological (electrodermal response during exposition to the stimulus), Self-assessment mannequin test – SAM (pre and post stimulus), and Think aloud (after exposition). As for the quantitative data, MANOVA and Mann-Whitney (non-parametric of independent samples) analyses were carried out. The physiological data were analysed using ANOVA, Turkey and Mann-Whitney. Finally, the analysis of the qualitative data obtained from the Thinkaloud was performed using NVivo 11, identifying the categories of emotion, attention and immersion.
As a main result, statistically significant differences have been found between the students watching the sequence with the 3D surround mix and the students watching it with the stereo version. The former obtained better results in listening comprehension. This supports the conclusions from other studies about improvements when the learning experience is emotionally marked. In this sense, the immersive experience of 3D sound listening can be considered as a way to improve learning, which opens a great amount of possibilities in the design and use of media products (audiovisual, video games) and sound based products (podcasts, audio games) as learning resources, both on formal and on informal learning contexts. Furthermore, the impact of the different sound mixes on the emotional level is not self-perceived by the participants themselves, according to SAM test results. It reinforces the consideration of the semi-conscious perception of sound as a factor for the involvement of the audience in the narration. In connection to this, 3D sound also increases the impact of the veridiction pact, as surround sound mixes achieve a greater level of fidelity with listening conditions in real life contexts.
•Grimshaw, M. (2008). Sound and immersion in the first-person shooter. International Journal of Intelligent Games Technology, 119–124. •Jørgensen, K. (2006). On the functional aspects of computer game audio, Audio Mostly 2006, Piteå, Sweden. •LaGrou, J. (2014). The future of audio engineering. Tape op. http://tapeop.com/interviews/100/the-future/ Consultada el 20 de septiembre de 2015. •Mora, F. (2005). Cómo funciona el cerebro. Madrid. Alianza •Murch, W. (2003): En el momento del parpadeo. Madrid: Ocho y Medio, Libros de Cine. •Shilling, R., Zyda, M. Y Wardynsky, E. (2002). Introducing Emotion into Military Simulation and Videogame Design: America ’ s Army : Operations and VIRTE. Comunicación presentada en GameOn Conference. London. •Stockburger, A. (2006). The rendered arena: Modalities of space in video and computer games, unpublished PhD thesis, London, University of the Arts. •Zunzunegui, S. (1995). Pensar la imagen, Madrid, Ediciones Cátedra. Revista
- Search for keywords and phrases in "Text Search"
- Restrict in which part of the abstracts to search in "Where to search"
- Search for authors and in the respective field.
- For planning your conference attendance you may want to use the conference app, which will be issued some weeks before the conference
- If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.