The purpose of this paper is to examine the methodological dilemmas that researchers face in their attempts to improve learning and teaching through learning analytics. Learning analytics is "the process of collecting and studying usage data in order to make instructional decisions that will support student success" (Becker, 2013, p. 63). With the availability of an unprecedented amount of user data from Massive Open Online Courses (MOOCs), Learning Management Systems (LMSs), gaming environments, and other sources, the analysis of these so-called "big data" presents particular methodological challenges. In particular, a measurement framework for dealing with the impact of big data on education has yet to be fully developed. In this paper we refer to the seminal measurement research of Cattell (1952) to develop a systematic measurement framework for understanding the divergent inferences that arise from analyzing different sources of variation in big learning data: variation over time, variation across persons, variation across contexts, and variation across tasks. We will apply cluster analysis of simulated data to show how inference may diverge based on different mixtures of variation across time, tasks, contexts, and persons. This paper will demonstrate that each way of analyzing the data (e.g. variation over time, variation across persons, variation across context, and variation across tasks) supports divergent conclusions that can be made from the results. Without the theorizing process to guide the analysis of big data, learning analytics researchers are confronted with the difficulty in making sense of their research results. Although larger and larger datasets may have great promise, how theory is used and essentially, what constitutes proof becomes more, not less important in the research process.

Becker, B. (2013). Learning analytics: Insights into the natural learning behavior of our students. Behavioral and Social Sciences Librarian, 32(1), 63-67. doi:10.1080/01639269.2013.751804 Cattell, R. B. (1952). The three basic factor-analytic research designs—their interrelations and derivatives. Psychological Bulletin, 49(5), 499-520. doi:10.1037/h0054245