In the second language (L2) field, research on pronunciation instruction has gradually captured the attention of L2 researchers. Influenced by the dominance of communicative language teaching theory, the focus of pronunciation instruction has shifted towards the development of overall communication skills. Three assessment principles proposed by numerous L2 scholars for measuring L2 learners’ pronunciation are intelligibility, comprehensibility, and accentedness (Derwing & Munro, 2005; Levis, 2005, 2020). Review studies on L2 pronunciation instruction primarily employ measurements involving human listeners and acoustic measures. Due to their close relationship with all three directions, human listeners’ judgments have constituted the majority of assessment methods for measuring outcomes in L2 pronunciation instruction. The use of acoustic measures is limited, as L2 pronunciation scholars believe acoustic analyses may not predict intelligibility accurately (Kermad & Kang, 2018). In recent years, combining advanced acoustic measures with native speakers’ judgments of specific pronunciation features has proven productive in elucidating the relationship between pronunciation features and the three principles mentioned above (Kang, Rubin & Pickering, 2010; Trofimovich & Isaacs, 2012). There has been very limited research conducted on exploring the overall effectiveness of pronunciation instructions in the L2 field and very few studies have followed an evaluation framework to guide the assessment of pronunciation instructions. Chapelle, Enright, and Jamieson (2010) proposed an argument-based validation model, suggesting a network of inferences needed to be verified to support test score interpretation and use. Six inferences were included in this model to develop a validity argument for a language test, that are domain definition, evaluation, generalization, explanation, extrapolation, and utilization (Fan & Yan, 2020). Saito and Plonsky (2019) suggested a framework for measuring the effectiveness of L2 pronunciation teaching, including the constructs, the scoring method, and the type of knowledge elicited method. Based on these two frameworks, we will propose a new evaluation framework as a theoretical base to provide a standard for designing assessments. Three domains are presented in the new proposed evaluation framework, that are the pronunciation constructs of assessments, scoring methods and reliability, and gender difference.

Three questions are discussed:

1) Do the effects of assessments vary when focusing on different pronunciation constructs?

2) Do the effects of assessments vary with scoring methods performed by human raters and acoustic scoring?

3) Do the effects of assessments vary in terms of reliability between human raters and acoustic scoring?

4) Do pronunciation instructions employ gender differences in assessments?

A carefully screened selection of studies pertaining to pronunciation instruction within the L2 research field is conducted. This ensures that the included studies align with the research questions. By following the proposed evaluation framework, we analyzed a total of 35 published studies between 1980 and 2023 on pronunciation assessments in three domains: a) the pronunciation constructs of assessments (pronunciation aspects); b) scoring methods and reliability (human raters versus acoustic scoring); and c) gender differences (female versus male). Pronunciation instructions that are conducted in the CFL context are included. Studies are selected that have examined the effectiveness of pronunciation instructions with a pretest-posttest design and experimental-control group design. Key words are used to screen the databases (e.g., Chinese as a foreign language, CFL, pronunciation, instruction, second language, foreign language, tone, initial, final, intonation, stress). Kappa and Cronbach’s alpha results will be used to check the reliability among the human examiners’ scoring methods.

L2 pronunciation instructions focusing on explicit pronunciation aspects (segmental/suprasegmental accuracy) are more dominant compared to global pronunciation instructions (intelligibility, perceived fluency, etc.), and expert ratings significantly dominate as compared to acoustic machines. Effect sizes between human examiners vs. acoustic machines show that both assessments are capable of comparing the within-group differences and between-groups differences. Gender differences are ignored in the selected pronunciation instructions, and no assessments has examined the differences.

Chapelle, C. A., Enright, M. K., & Jamieson, J. (2010). Does an argument‐based approach to validity make a difference?. Educational measurement: Issues and practice, 29(1), 3-13. Derwing, T. M., & Munro, M. J. (2005). Second language accent and pronunciation teaching: A research‐based approach. TESOL quarterly, 39(3), 379-397. Fan, J., & Yan, X. (2020). Assessing speaking proficiency: a narrative review of speaking assessment research within the argument-based validation framework. Frontiers in psychology, 11, 330. Isaacs, T., & Trofimovich, P. (2012). Deconstructing comprehensibility: Identifying the linguistic influences on listeners’ L2 comprehensibility ratings. Studies in Second Language Acquisition, 34(3), 475-505. Kang, O., Rubin, D. O. N., & Pickering, L. (2010). Suprasegmental measures of accentedness and judgments of language learner proficiency in oral English. The Modern Language Journal, 94(4), 554-566. Kermad, A., & Kang, O. (2019). Effect of classroom assessment stakes on English language learners’ oral performance. Tesol Journal, 10(2), e00392. Levis, J. M. (2005). Changing contexts and shifting paradigms in pronunciation teaching. TESOL quarterly, 39(3), 369-377. Levis, J. (2020). Revisiting the intelligibility and nativeness principles. Journal of Second Language Pronunciation, 6(3), 310-328. Saito, K., & Plonsky, L. (2019). Effects of second language pronunciation teaching revisited: A proposed measurement framework and meta‐analysis. Language Learning, 69(3), 652-708.