The extent to which education systems can deliver on their commitments to quality, equity, access (to name only some) depends a great deal on education policy and even more so on its implementation by education participants – teachers, students, parents, administrators. Some researchers even suggest using the term ‘policy appropriation’ over “policy implementation” since education participants are “active constructors” of educational policies and reforms as they refract them through the prism of their own experiences, contexts, and identities.

The exploration of policy appropriation experiences can deliver important insights about the ways in which education policy supports (or hinders) the proliferation of desirable professional practices in line with education sector commitments. An important, yet still under-explored source of evidence which can feed into such insights, is social media. Education practitioners and stakeholders use it to share resources, carry out collegial discussions, and connect to students and parents who, for their part, also use it to interact and exchange on their own experiences with education.

In the wake of the COVID19 pandemic, many teachers, learners, parents, and school administrators intensified their use of social media platforms, in particular Facebook, even further. Their posts and comments in dedicated social media groups have transformed these groups into rich repositories of data in the form of first-hand accounts, stories, and reflections from the frontline of teaching, learning and family life with schoolchildren.

To tap into the richness of such sources, we developed a methodology for the sampling and collection of content data from platforms such as Facebook for the purpose of computer-assisted qualitative analysis, which capture the experiences of education participants on the frontline of education provision, i.e.  how new policy measures are experienced and appropriated in the practice of teaching, learning, and school management in times of crisis.

In our contribution, we first outline the conceptual underpinnings of the methodology, which draws on digital ethnography, social media research, and sociocultural approach to education policy. We then explain in detail a string of methodological decisions which help with the framing, defining, and collecting samples of relevant content data. We conclude by illustrating the potential of the methodology on the example of recent social media data harvested in several countries in Eastern and South Eastern Europe, the Caucasus, and Central Asia.

We developed and applied an approach which allows for the collection of content data that meet four specifications: a) they are generated by education participants; b) they are concerned with experiences of policies and measures which apply to education sector in their country or region; c) they are not compromising the privacy of individual users; d) they are feasible to collect in terms of volume and permissible to collect in terms of means and methods. To meet these requirements, we designed a sequence of routines which include determining where to find the relevant target groups for the data collection, selecting a sample of data from these locations to ensure that the content captured through the data is of relevance; and collecting the data in that sample in ways that are replicable, reliable, and compliant with the ToRs of the targeted social media platform. The sampling stage comprises two routines. The first establishes a sampling frame for the data collection, understood as the range of virtual “locations” where to find the target population and from which to select a sample of data for the evidence collection. These locations are limited to pages and groups as places where people communicate about shared interests with other people, where content data is not filtered by the selective algorithm that manages the timeline content on Facebook, and where personal information about users is not being disclosed. Through a second routine, we designed a sample based on the timestamps of posts as a proxy of relevance. The dates in the sample were chosen to coincide with dates on which countries have introduced policy measures of research interest. Our assumption was that new policies are likely to be triggers of sector-specific communication between members of our target population and consequently, that content posted on this narrow selection of dates is more likely to reflect on the new policy than posts and comments posted on other, unrelated dates. Finally, we developed a routine for the manual collection of data in our sample. We opted for a manual solution as a way of ensuring that the approach is accessible to, and replicable by, peers irrespective of their resources, programming skills, and the changing policies of companies running the social media platforms. In addition, manual collection is the only approach which does not collide with the Terms of Service of social media platforms which forbid automated scrapping of data.

We piloted our methodology in Armenia, Kazakhstan, Russia, Serbia, and Ukraine. With its help we established a sampling frame covering a target population of 564 497 users across a total of 20 locations (Facebook groups and pages) in the five countries. The content data from these locations comprised 1 932 posts and 24 516 comments from a sample of 13 to 14 target dates per country, within a reference period covering the first school closures from March to July 2020. Our experience shows that the methodology is robust enough to deliver comparable and reliable data even when applied in different national and cultural contexts, by a diverse group of researchers in terms of specialisation, experience, background, and research interest. Between a quarter and a third of all data in each country were directly related to policy appropriation experiences in education. The data was also ready for computer-assisted qualitative analysis without the need for further processing, such as data cleaning, formatting, etc., which suggest that the methodology can be of help in circumstances which require evidence and insight about policy decisions as quickly as possible, such as in times of crisis. Although our research questions emerged in the rather specific context of demand for insights during the pandemic, one of our lessons learned is that our approach to sampling and collecting content data from social media is likely to be of relevance beyond the purposes of our own research. It provides a stand-alone solution for the identification and collection of qualitative evidence from social media on the policy experiences of stakeholders, which can be applied to any sector and/or policy and in any context in which there are users of social media who are members of the target population.

Bartlett, L., & Vavrus, F. (2014). Transversing the vertical case study: A methodological approach to studies of educational policy as practice. Anthropology & Education Quarterly, 45(2), 131-147. Bergdahl, N. & Nouri, J. (2020). Covid-19 and Crisis-Promoted Distance Education in Sweden. Technology, Knowledge and Learning, 1-17. Beręsewicz, M., Lehtonen, R., Reis, F., Di Consiglio, L., & Karlberg, M. (2018). An overview of methods for treating selectivity in big data sources. Luxembourg: Publications Office of the European Union. Braun, A., Maguire, M., & Ball, S. J. (2010). Policy enactments in the UK secondary school: Examining policy, practice and school positioning. Journal of education policy, 25(4), 547-560. Braun, V. & Clarke, V. (2006) Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77-101. de Jong, E. J. (2008). Contextualizing Policy Appropriation: Teachers’ Perspectives, Local Responses, and English-only Ballot Initiatives. Urban Review 40, 350-370. Freelon, D. (2018). Computational Research in the Post-API Age. Political Communication, 35(4), 665-668. Gudmundsdottir, G. B., & Hathaway, D. M. (2020). " We Always Make It Work": Teachers' Agency in the Time of Crisis. Journal of Technology and Teacher Education, 28(2), 239-250. Hodges, C., Moore, S., Lockee, B., Trust, T., & Bond, A. (2020). The difference between emergency remote teaching and online learning. Educause Review, 27. Levinson, B. A., Sutton, M., & Winstead, T. (2009). Education policy as a practice of power: Theoretical tools, ethnographic methods, democratic options. Educational Policy, 23(6), 767-795. Lundin, M., Lantz-Andersson, A., & Hillman, T. (2017). Teachers’ reshaping of professional identity in a thematic FB-group. Qwerty-Open and Interdisciplinary Journal of Technology, Culture and Education, 12(2), 12-29. Mancosu, M., & Vegetti, F. (2020). What You Can Scrape and What Is Right to Scrape: A Proposal for a Tool to Collect Public Facebook Data. Social Media + Society, 6(3), 1-11. OECD. (2020). OECD Policy Responses to Coronavirus (COVID-19): Combatting COVID-19’s effect on children. Paris: OECD Publishing. Turner, A. G. (2003). Sampling frames and master samples. New York: United Nations Secretariat - Statistics Division. Trust, T. & Whalen, J. (2020). Should Teachers be Trained in Emergency Remote Teaching? Lessons Learned from the COVID-19 Pandemic. Journal of Technology and Teacher Education, 28(2), 189-199. Trust, T., Carpenter, J. P., Krutka, D. G., & Kimmons, R. (2020). # RemoteTeaching &# RemoteLearning: Educator Tweeting During the COVID-19 Pandemic. Journal of Technology and Teacher Education, 28(2), 151-159. Varis, P. (2014). Digital ethnography. (Tilburg Papers in Culture Studies; No. 104)