Recent years have seen a notable rise in the interest in international reading literacy assessments, the results of these assessments often being the basis on which nations make educational decisions. However, for the results of these assessments to be valid, all the different-language versions of the test have to be equivalent, or comparable, to each other; they must not be biased, unfairly favouring some language groups over some others. To ensure equivalence, several quality control procedures are implemented in these assessments. An important part of these procedures are statistical and judgemental item analyses. Statistical DIF (differential item functioning) analyses are conducted to detect the items that function differentially across countries. These are followed by qualitative, judgmental analyses, where the flagged items are reviewed for bias and those interpreted as biased are removed from the final analyses. In international assessments an important potential source of bias are translation-related differences, i.e. translation problems and errors (e.g. Allalouf, 2003; Elosua & López-Jaúregui, 2007; Ercikan, 2002; Gierl & Khaliq, 2001; Grisay, Gonzalez & Monseur, in press). Given the significant role that judgmental item analyses and the identification of the items that are biased because of translation-related differences have in ensuring validity in international assessments, it is important to make sure that the analyses are conducted in such a way that they are able to disclose the items that show translation-related DIF. However, very little research exists on these analyses and on their ability to identify translation-related DIF. In addition, the research that exists suggests that making these analyses and identifying sources of DIF is extremely challenging and that the analyses have often not been able to consistently and accurately identify translation-related DIF (Gierl & Khaliq, 2001; Roussos & Stout, 1996). The purpose of the study was to examine whether it would be possible to improve the ability of judgmental item analyses conducted in international reading literacy assessments to identify translation-related sources of DIF. The ultimate goal was to develop judgmental item analyses conducted in international reading literacy assessments – although the study can also be used when developing other types of international studies – and to add to the validity of these assessments.

A judgmental analysis was conducted of three Finnish items which in the PISA 2000 reading test had been identified as DIF (harder or easier than expected) but for which no clear translation-related explanations had been suggested. The items were compared linguistically to the corresponding items in the English and French source versions, at the same time taking into account the cognitive processes required to answer them. The objective was to explore whether there were linguistic elements in the Finnish versions which deviated so much from the corresponding elements in the source versions that they could have explained the differential functioning of the items. When assessing item difficulty the analysis drew mainly on the work by Kirsch and Mosenthal (e.g. Kirsch, 2001; Mosenthal & Kirsch, 1991), and on reading research and on the factors that in this research have been found to influence text difficulty (e.g. Alderson, 2000).

In the analysis several translation-related explanations were found which may have affected students’ performance on the three items. Most of the explanations had to do with language-specific differences in either grammar or meanings and connotations. However, some arose from differences between the English and French source versions or from a disregard of the translation instructions and unduly free translation. The study thus suggests that it is possible to make judgmental item analyses conducted in international assessments better able to identify translation-related sources of DIF, especially those deriving from differences between languages, but also those resulting from differences between the two source versions and unfaithfulness to the translation instructions. To be able to do this, however, a more rigorous and standardized judgmental item analysis procedure needs to be developed in these assessments which, moreover, should be followed in all participating countries. Development also seems to be needed in other quality control procedures implemented in these assessments. Suggestions are given as to how to develop not only judgmental item analyses but also other quality control procedures in international assessments.