The international large-scale assessments and surveys (ILSAs) have complex sampling and assessment designs. ILSA use multistage stratified cluster sampling design where the schools are sampled at the first stage and students are sampled at the second one. The probability of first-stage sampling is proportional to the size of the primary units (PPS) (LaRoche et al., 2017; Tieck, 2020b, 2020a). ILSAa also use complex assessment design with multiple matrix rotation of blocks of items (and complex tasks) across multiple booklets/combinations, linking consecutive booklets through common blocks (Foy & Yin, 2017; Fraillon, 2020). The design issues have to be taken into account when analyzing ILSAs’ data.

The goal of this workshop is to introduce participants to analysis of ILSAs’ data using the R programming language and statistical software (The Comprehensive R Archive Network, n.d.). More specifically, the workshop will introduce the participants to a newly developed R package, the R Analyzer for Large-Scale Assessments. The package can handle all ILSAs design issues which have an impact on the analysis routines. RALSA has multiple advantages compared to other available software products. RALSA converts the originally provided SPSS (or text) data into native R data sets. The converted data sets also contain the user-defined missing values for the variables, which is different from the typical way R handles the missing data. RALSA also has the capability to recognize the study, its cycle and the available respondent types to select the appropriate design variables and apply the pertinent computational routines for the study in scope. Further, the package has a graphical user interface which eases the analysis for users with limited technical skills. Last, but not least, RALSA has a comprehensive output system which exports the results in MS Excel workbook with multiple embedded sheets (estimates, analysis information, model statistics, and the used analysis syntax). The entire package, including the graphical user interface and the output system, was built entirely in R without relying on any other platform or programming language. The package was built for user experience with a flexible design and architecture which permit quick addition of new studies and functionality. RALSA is useful for all European educational researchers and analysts worldwide.

Currently, the package can process and analyze data from all cycles of the following studies:

• CivED;

• ICCS;

• ICILS;

• RLII;

• PIRLS (including PIRLS Literacy and ePIRLS);

• TIMSS (including TIMSS Numeracy, eTIMSS will be added with the upcoming release of TIMSS 2019);

• TiPi (TIMSS and PIRLS joint study);

• TIMSS Advanced;

• SITES;

• TEDS-M;

• PISA;

• TALIS;

• TALIS Starting Strong Survey (a.k.a. TALIS 3S); and

• Responses to Educational Disruption Survey (REDS).

The following data preparation and analysis functionality is supported:

• Prepare data for analysis

  • Convert data (SPSS, or text in case of PISA prior 2015)

  • Merge study data files from different countries and/or respondents

  • View variable properties (name, class, variable label, response categories/unique values, user-defined missing values)

  • Recode variables

• Perform analyses (more analysis types will be added in future)

  • Percentages of respondents in certain groups and averages on variables of interest, per group

  • Cross-tabulations with Rao-Scott chi-square adjustments

  • Percentiles of variables within groups of respondents

  • Percentages of respondents reaching or surpassing benchmarks of achievement

  • Correlations (Pearson or Spearman)

  • Linear regression

  • Binary logistic regression

RALSA also introduces a graphical user interface for the users with limited technical skills. It is written entirely in R without relying to any external platform or programming language. RALSA can work on any operating system where R can be installed (e.g. Linux, MacOS and Windows).

The workshop will be divided into two parts. The first part will introduce participants to ILSAs: what are they, what is their goal and intent, what purposes their data serve for. This part will also provide an overview of the studies’ design and field operations. A specific stress will be put on the complex sampling and assessment designs ILSAs employ to collect reliable data from all participating countries. The statistical complexities and issues for analysis stemming from the complex sampling and assessment designs will be introduced with multiple examples. The analysis methodologies will be introduced with all relevant details for analyzing the data, and the issues arising from the complex designs. The second part will introduce the analysis software (RALSA). All features will be explained and multiple examples will be provided during the demonstrations. The examples will be demonstrated using both the command line and the graphical user interface. The last part of the workshop will provide assignments for the participants which they will complete on their own with assistance from the instructor. The workshop will use data from multiple European countries participating in the IEA’s International Computer and Information Literacy Study (ICILS) 2018 and the digital component of the IEA’s Progress in International Reading Literacy Study (ePIRLS) 2016. These two studies have gained extreme importance due to the new demands COVID-19 posed in the last year. By the end of the workshop the participants are expected to have gained the following: 1. Knowledge on the modern ILSAs; 2. Knowledge, understanding and appreciation on the ILSAs design and methodology, as well as the statistical complexities and issues for analyzing their data; 3. Gather knowledge and understanding on the computational routines used in ILSAs; and 4. Obtain skills to analyze data using R software, tailored for ILSAs design. The workshop is intended both for analysts who do not have yet the knowledge and experience using ILSAs’ data, as well as for researchers with more experience. Knowledge on R programming language is not required, RALSA has intuitive and easy to use syntax, as well as interface. However, working knowledge on basic statistic is required. All materials for the workshop, including the software, will be provided free of charge. The participants will need a computer to install the software and perform the sample analyses.

ILSAs have became one of the main drivers of policy making in education. Many of the European educational systems use the findings provided by the ILSAs’ data for initiating new or guiding existing reforms in education. Due to their complex sampling and assessment designs, however, analysis needs using special techniques to obtain correct population estimates. This, in turn, requires availability of software to perform the computations. This workshop aims towards enriching the competencies of researchers already using ILSAs’ data, and give the knowledge, understanding and skills to newcomers to ILSAs. It will also enlarge the network of scientists who are willing to analyze data with specific research questions to support policies and reforms in education. The free software provided will also promote the use of ILSAs’ data in research.

Foy, P., & Yin, L. (2017). Scaling the PIRLS 2016 Achievement Data. In M. O. Martin, I. V. S. Mullis, & M. Hooper (Eds.), Methods and Procedures in PIRLS 2016 (p. 12.1-12.38). TIMSS & PIRLS International Study Center. Fraillon, J. (2020). ICILS 2018 test development. In J. Fraillon, J. Ainley, W. Schulz, T. Friedman, & D. Duckworth (Eds.), IEA International Computer and Information Literacy Study 2018: TECHNICAL REPORT (pp. 11–29). IEA. LaRoche, S., Joncas, M., & Foy, P. (2017). Sample Design in PIRLS 2016. In M. O. Martin, I. V. S. Mullis, & M. Hooper (Eds.), Methods and Procedures in PIRLS 2016 (p. 3.1-3.34). Lynch School of Education, Boston College. The Comprehensive R Archive Network. (n.d.). Retrieved October 11, 2019, from http://cran.r-project.org/ Tieck, S. (2020a). Sampling design and implementation. In J. Fraillon, J. Ainley, W. Schulz, T. Friedman, & D. Duckworth (Eds.), IEA International Computer and Information Literacy Study 2018: TECHNICAL REPORT (pp. 59–78). IEA. Tieck, S. (2020b). Sampling weights, non-response adjustments, and participation rates. In J. Fraillon, J. Ainley, W. Schulz, T. Friedman, & D. Duckworth (Eds.), IEA International Computer and Information Literacy Study 2018: TECHNICAL REPORT (pp. 79–93). IEA.