This paper offers a methodological reflection on different recipes (ways) of studying data practices with a focus on organisations in the K-12 educational domain. The starting point for our paper is the observation that by now it has been widely acknowledged that data are not “raw” (Gitelman 2013) but rather that they are “cooked” (Bowker 2005): Data do not provide a window to the social world and represent independently existing phenomena, but rather produce specific subject and object positions. Their agency is hence important when considering the transformation of education (Jarke and Breiter 2019). So far sociologies of education focus on practices of software and data use, the inquiry about how data are constructed and produced within software is only emerging (e.g. Decuypere 2021, Jones and McCoy 2019, Hartong and Förschler 2019, Macgilchrist 2019, Ratner and Gad 2019, Selwyn 2015, Williamson 2016, 2017). They study how “software and code connect people, things, systems, places and events in a pervasive and sinuous fabric” (Mackenzie 2013: 392). In many studies however, the software in which these data are produced, processed and circulate are conceived as given and to-some-extent stabilised objects that ‘configure’ their users and ‘inscribe’ certain data use practices. The contingencies in the design and implementation of such software are mostly neglected. This paper contributes to this line of research and extends the methodological work in the sociologies of education with an empirical examination and methodological reflection on the study of school information systems.

To this point, software studies, “have developed various ways of mapping and tracking these trajectories [of software design] by observing coding practices as well as tracking tropes and figures of control and power through software“ (Mackenzie 2013: 393). Of course, such approaches require ample time and access for ethnographic field work. What poses a methodological challenge is how and to what extent the process in which the negotiations about the software design take place as well as subsequent design activities can be reconstructed, if the research embarks only after a first prototype has been developed. One way to investigate the emergence of data (structures) within software and related data practices is to consider the movements of data within and beyond systems and organisations. 

In this paper, we propose to approach educational data movements and practices methodologically through the framework of data journeys (Leonelli, 2014; Bates et al., 2016). 

Data journeys render visible not only how data circulate and are made circulate within various information systems and organisations, but also the changes and frictions that occur as data move. The framework of data journeys can be applied as both “a way of theorising data movement and as a methodological tool to investigate it” (Leonelli, 2020, p. 5). Using both accounts of actual data practices and software design documentation, data journeys illustrate both how data are meant to move according to the software developers and how these movements are enacted in organisational practices of schools.

We demonstrate that to be able to reconstruct the ways in which school information systems (re)configure educational actors, relations and processes through its data movements, structures, and routines, it is important to consider their development and design as situated, continuous and contingent processes. In our paper, we hence attend to the following research questions (RQs):

RQ1. In what ways can data journeys render visible the movements of data that are put to work in decision-making practices of educational organisations?

RQ2. What can we learn about the transformation of education from studying software as continuous design, implementation, and (re)use processes?

This paper is based on a study of four school information systems, which are deployed in public schools of four federal states in Germany. In each of the federal states, the Ministries of Education were leading the development and design of these systems. There are differences across the states with respect to the members and size of the development teams, the duration of the design processes, its framing conditions and roll-out, amongst others. Based on our study, we demonstrate how a process-understanding of software enables researchers to uncover (so far hidden) data practices, and the socio-technical assemblages in which they are meant and made to work. They allow to ask which problem a software ought to solve; how central relations, roles and tasks of educational actors are defined and which values and priorities are being inscribed or omitted. In order to conceptualise and visualise software design in its becoming, we create data journeys based on analysing software design documents and on semi-structured interviews with developers and ministry officials. In the case of our study, this includes 1) legal frameworks (e.g. data protection), 2) policy documents (e.g. education governance regulations), 3) software documentation (e.g. requirements, software role models, data(base) models, process models and anticipated use cases), 4) technical documentation, 5) design documentation (e.g. scenarios and personas for software development including pedagogical and didactical concepts), 6) prototypes, 7) user tests guidelines, and of course 8) user manuals. Our approach places analytical focus on spatial and temporal movements between organisations, actors, systems, artefacts in-the-making.

This paper aims to extend our understanding of software and data practices, by viewing software as a contingent process of design, implementation, and use. We propose to analyse educational software such as school information systems according to two types of data journeys, each producing different kinds of knowledges. First, assemblage-based data journeys, conceptually aligned with Kitchin’s (2014) notion of data assemblage, explore educational actors’ boundary making practices, making certain school actors present 'in-here' and leaving others 'out-there' (Law 2004). We demonstrate that different data practices produce differently enacted realities of educational organisations (e.g. with respect to the performance of organisational membership or the enactment of the organisational boundaries of schools). Second, we depict temporality of software and data movements in process-oriented data journeys. In addition to temporal aspects, process-oriented research artefacts make the agential forces of specific human or non-human actors visible. Such journeys also uncover what problem ought to be solved by an educational software system and in what ways various actors come to their solutions. Comparing these artefacts across our field sites allows to reconstruct what became black-boxed in different software design processes. Software development and design is a multifaceted, contingent process in which multiple actors (human and non-human) negotiate and create imaginaries of education, schooling and learning. It requires a process-oriented research methodology, as it is important to not only study the use of software but also its design. By deconstructing socio-technical assemblages of software design in the data journey, we shed light on the contingencies in which software operates in the educational domain.

Bates, Jo, Yu-Wei Lin, and Paula Goodale. 2016. Data Journeys: Capturing the Socio-Material Constitution of Data Objects and Flows. Big Data & Society 3(2): 2053951716654502. Bowker, Geoffrey C. 2005. Memory Practices in the Sciences. Inside Technology. Cambridge, Mass: MIT Press. Decuypere, Mathias. 2021. The Topologies of Data Practices: A Methodological Introduction. Journal of New Approaches in Educational Research, 10(1), 67–84. Gitelman, Lisa. 2013. Raw Data Is an Oxymoron. MIT Press. Hartong, Sigrid, and Annina Förschler. 2019. Opening the Black Box of Data-Based School Monitoring: Data Infrastructures, Flows and Practices in State Education Agencies. Big Data & Society 6(1): 205395171985331. Jarke, Juliane, and Andreas Breiter. 2019. Editorial: The datafication of education. Learning, Media and Technology, 44(1), 1–6. Jones, Kyle M. L., and Chase McCoy. 2019. Reconsidering Data in Learning Analytics: Opportunities for Critical Research Using a Documentation Studies Framework. Learning, Media and Technology 44(1): 52-63. Kitchin, Rob. (2014). The Data Revolution: Big Data, Open Data, Data Infrastructures and Their Consequences. SAGE. Law, John. 2004. After method: Mess in social science research. Routledge. Leonelli, Sabina. (2014). What difference does quantity make? On the epistemology of Big Data in biology: Big Data & Society. https://doi.org/10.1177/2053951714534395 Leonelli, Sabina. (2020). Learning from Data Journeys. In S. Leonelli & N. Tempini (Eds.), Data Journeys in the Sciences (pp. 1–24). Springer International Publishing. Macgilchrist, Felicitas. 2018. Cruel Optimism in Edtech: When the Digital Data Practices of Educational Technology Providers Inadvertently Hinder Educational Equity. Learning, Media and Technology 44(1): 77–86. Mackenzie, Adrian. 2013. Programming Subjects in the Regime of Anticipation: Software Studies and Subjectivity. Subjectivity 6(4): 391–405. Ratner, Helene, and Christopher Gad. 2018. Data Warehousing Organization: Infrastructural Experimentation with Educational Governance: Organization. Sage UK: London, England. Selwyn, Neil. 2015. Data Entry: Towards the Critical Study of Digital Data and Education. Learning, Media and Technology 40(1): 64–82. Williamson, Ben. 2016. Digital Methodologies of Education Governance: Pearson Plc and the Remediation of Methods. European Educational Research Journal 15(1): 34–53. Williamson, Ben. 2017. Decoding ClassDojo: Psycho-Policy, Social-Emotional Learning and Persuasive Educational Technologies. Learning, Media and Technology 42(4): 440–453.