Educators and policymakers around the world use comparative international data, such as PISA, to identify relative strengths and weaknesses in their national educational progress. Although these data traditionally focus on verbal and quantitative skills, there is growing recognition that students will need a wider array of skills to prepare for higher education and careers in the 21^st century. Perhaps the strongest advocate for the importance of richer and more varied skills was the consortium formed under the “Assessment and Teaching of 21^st Century Skills” by Cisco, Intel, and Microsoft and administered through the University of Melbourne.  They identified ten kinds of skills in four broad categories (Binkley, M., Erstad, O., Hermna, J., Raizen, S., Ripley, M., Miller-Ricci, M., & Rumble, M., 2012).

            The continuing challenge is how to assess these complex skills in ways that are fair, valid, and reliable across a wide variety of people, languages, and countries because assessments are crucial for (a) providing benchmarks of performance, (b) indicators of growth and change, and (c) guidance for instruction and training.  Moreover, the assessments need to be delivered, completed, collected, scored, and reported using computer-based technologies in order for the assessments to be brought to global scale with efficiency, low cost, and quick feedback. 

         My presentation will focus on innovations being developed at the Educational Testing Service in the USA to measure some of these 21^st century skills. It is important to share this information with a global audience because (a) some of the methods are already planned for international assessments such as PISA, (b) the technologies are heuristic for other agencies and countries building new assessments of complex skills, and (c) the consequences of the new assessments are intended to inform training and instruction. In my presentation I will describe advances in technology-enabled assessments of collaboration, communication, and life and career (i.e., intrapersonal and interpersonal) skills.

Collaboration and collaborative problem solving skills

            An important 21^st century skill is to collaborate effectively in teams. Workplace research shows that routine cognitive and manual tasks have declined over the past 30 years, while expert problem-solving tasks and complex communication tasks have increased (Autor, Levy, & Murnane, 2003; Levy & Murnane, 2004). A survey of employers found that “teamwork and collaboration” was ranked second (behind “work ethic”) as the applied skill employers were most likely to rate as “very important” (Casner-Lotto & Barrington, 2006).  As the workforce becomes increasingly team oriented, it is important for educational systems to ensure that students learn collaborative skills to be prepared to face future challenges. At ETS we have initiated several projects focused on developing assessments that enable students to build and demonstrate collaborative skills.

In this presentation I will review some of the key issues in collaborative assessment and then demonstrate two systems designed to assess collaboration.  The demonstrations are online game-like task environments involving teachers and student colleagues, either simulated (with a script) or live (communicating through a chat window). One of the lessons is set in a science context (learning about volcanoes) and the other is an English lesson. The demonstrations are designed to show both how curricular content is presented in a collaborative context, and also how students’ collaboration skills can be assessed. The research also recognizes the need for new measurement methods to analyze CPS task data, and draws on approaches outside traditional psychometrics such as data visualization, social network analysis, and dynamic models from engineering.

Autor, D. H., Levy, F., & Murnane, R.J.. (2003). The skill content of recent technological change: An empirical exploration. Quarterly Journal of Economics 118, (4) (a): 1279-333. Binkley, M., Erstad, O., Hermna, J., Raizen, S., Ripley, M., Miller-Ricci, M., & Rumble, M. (2012). Defining Twenty-First Century Skills. In Griffin, P., Care, E., & McGaw, B. Assessment and Teaching of 21st Century Skills, Dordrecht, Springer. Cai, Z., Graesser, A.C., Millis, K.K., Halpern, D., Wallace, P., Moldovan, C., Forsyth, C. (2009) ARIES!: An Intelligent Tutoring System Assisted by Conversational Agents. In Artificial Intelligence in Education: Building Learning Systems that Care: From Knowledge Representation to Affective Modeling, p. 796. IOS Press, Amsterdam. CASEL — Collaborative for Academic, Social, and Emotional Learning (2013). CASEL Guide: Effective Social and Emotional Learning Programs. Preschool and Elementary School Edition. Chicago, IL. Casner-Lotto, J., & Barrington, L. (2006). Are they really ready to work? Employers' perspectives on the basic knowledge and applied skills of new entrants to the 21st century U.S. workforce. U.S.: Conference Board, Partnership for 21st Century Skills, Corporate Voices for Working Families, Society for Human Resource Management. Durlak, J.A., Weissberg, R.P., Dymnicki, A.B., Taylor, R.D., Schellinger, K.B. (2011). The impact of enhancing students' social and emotional learning: a meta-analysis of school-based universal interventions. Child Development, 82(1), 405-432. Levy, F., & Murnane, R.J. (2004). The new division of labor: How computers are creating the next labor market. Princeton, N.J.: Princeton University Press.