Using Temporal Learner Analytic Data to Develop a Casual Predictive Model

Author(s):

John Busch(presenting / submitting)Neil Anderson Aidan McGowan Philip Hanna Mathew Collins

Conference:

ECER 2016

Network:

03. Curriculum Innovation

Format:

Paper

Session Information

03 SES 02 A, Curriculum and Student Performance Measurement

Paper Session

Time:

2016-08-23

15:15-16:45

Room:

NM-A106

Chair:

Wilmad Kuiper

Contribution

The mass storage of raw student information has caught the attention of academics, researches and Higher Education (HE) management (Richards, Bilgin & Marrone 2013, p68). This novel research area is widely referred to as Learner Analytics (LA). The purpose of LA is to collect, explore, study and analyse data to reveal a coherent meaning (Philips, 2011, p997). In other words LA can measure, store, analysis and report data about learners and their direct behaviour. This can be used to enhance both the pedagogic experience of the learner and their learning environment.

This new field in research has provided opportunities to improve the student learning experience by informing changes in course material, structure and assessment (Siemens & Long, 2011, p444 and Diaz & Brown, 2012, p4). Further studies have also suggested that intervention mechanisms can be triggered using LA (Beer, Tickner & Jones 2014, p247). However, Pardo and Siemens (2014, p444) assert that LA can easily become immorally intrusive into a student’s personal behaviour, raising issues of privacy and ethics. Even more worrying, Beer, Jones, & Clark (2012, p82) suggest the analysis of such large sets of data at a macro-level can lead to unsound conclusions. The abstraction of general patterns can ignore the underlying complexity that is not apparent in a broad linear relationship analysis.

Learning management systems (LMS) automatically generate and store a large amount of log data about students’ clickstream activities (Jones, Beer & Clark 2013, p2). They primarily record macro-level data, such as access-time data for digital resources e.g. web pages, videos and assessments. Furthermore, they can also record student test scores and attempts made, if the LMS is capable of such functionality.

The study uses retroactive data from a 2^nd year undergraduate cohort studying web software development. The data was aggregated from the web server which was used for the development of web sites and thus became a development web server for students. Our work is similar to a study by Tian, Rudraraju & Li (2004, p756) that used this data to evaluate the reliance of web code. The study will instead evaluate the development cycle of students web develop files and code. This exploratory data analysis will attempt to create a predictive analytics model, an analytical approach to uncover patterns and relationships which could predict student performance. A limitation of only studying the records within the development environment is that they do not explain why a particular behaviour occurs.

Method

From October 2014 to December 2014 log server data was recorded from 135 students studying a web development module at Queen's University Belfast. Students would implicitly request or 'hit' their web server pages to render their web programming code. The logs are generated and stored in an access log file for each individual student using the Apache web server for their development. Identity information was collected along with the following data recorded in the access logs that the study will find useful to our workload analysis: • IP number of the machine making the request; • date and time of the request; • HTTP status code. The data at the macro-level, that all other educational research uses, can show details of when, where and what web files a student requested. Since the context of what the Apache web server data is used for is different, instead of serving learning content and files to students, it is a development tool rather than service tool. The Apache web server is used by the students to develop their own understanding of web programming, because of this they implicitly use it for development purposes. The log data now sits within the micro-level because it records data about a student's temporal engagement with the learning material. Students must use the Apache web server to parse and execute their web programming code. It is assumed that the abstraction of general patterns within the logs cannot ignore complexity, since that analytics would be within the micro-level context, as suggested by Beer, Jones, & Clark (2012, p84). An individual record of performance from the final individual-based assessment project was also collected so that a collation can sought between the student's behaviour and their performance mark. Building on Romero, Ventura & García (2008, p370) work on analytics, we propose a five-step data-analytics process: 1) Collect the each individual web server user logs; 2) Pre-process the data, it is filtered to reduced and remove; 3) General analysis; creating a broad linear relationship; 4) Regression analysis, developing a linear regression; 5) Interpret and evaluate results. From following this process a predictive casual model maybe hypothesised.

Expected Outcomes

We have already implemented the first three of our five-step data-analytics process (as described in the previous section). Our preliminary results suggest significant assessment score differences between those that logged early date and times 'hits' from the web server. Early engagements level can be calculated by averaging the amount of hits within a set period, e.g. number of hits/days. Of those that received a 70%+ final module mark tended to log an average of 0.5 hits on the web server in the early to mid-term of the semester. Of those who were within the 40%-49% final module mark, their average is 0.21 hits per day. Of those how were awarded a fail in the module, <39%, their average was only 0.08% hits per day. Grouping the logged data within a set time frame allowed general analysis to suggest some correlation between when students engage with the web server and their performance. It would indeed suggest that the more highly engaged a student's development work is at the start to mid-term of a semester does greatly affect their final module grade. A linear regression can be used on the initial analysis results with additional grouped data will build a more complete summary. Thereafter a casual model should emerge; of whether or not early adapter activity within the logs can predict student performance. In future the model could be used as a benchmark into developing software development teaching and learning strategies and technologies.

References

Beer, C., Jones, D., & Clark, D. (2012). Analytics and complexity: Learning and leading for the future. In Proceedings of the 29th Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education (ASCILITE 2012) (pp. 78-87). Australasian Society for Computers in Learning in Tertiary Education (ASCILITE). Diaz, V., & Brown, M. (2012). Learning analytics: A report on the ELI focus session. In Educause (Ed.), Educause Learning Initiative (Paper 2, 2012 ed., Vol. ELI Paper 2: 2012, pp. 18). Educause: Educause. Available from: http://www.educause.edu/library/resources/learning-analytics-report-eli-focus-session Last accessed: 14/01/2016. Jones, D., Beer, C. & Clark, D. (2013). The IRAC framework: Locating the performance zone for learning analytics. In H. Carter, M. Gosper and J. Hedberg (Eds.), Electric Dreams. Proceedings ascilite 2013 Sydney. (pp.446-450) Pardo, A., & Siemens, G. (2014). Ethical and privacy principles for learning analytics. British Journal of Educational Technology, 45(3), 438-450. Phillips, R., Maor, D, Cumming-Potvin., Roberts, P., Herrington, J., Preston, G. & Moore, E.(2011). Learning analytics and study behaviour: A pilot study. In G. Williams, P. Statham, N. Brown & B. Cleland (Eds.), Changing Demands, Changing Directions. Proceedings ascilite Hobart 2011. (pp.997-1007). Romero, C., Ventura, S., & García, E. (2008). Data mining in course management systems: Moodle case study and tutorial. Computers & Education, 51(1), 368-384. Siemens, G., & Long, P. (2011). Penetrating the Fog: Analytics in Learning and Education. EDUCAUSE review, 46(5), 30. Tian, J., Rudraraju, S., & Li, Z. (2004). Evaluating web software reliability based on workload and failure data extracted from server logs. Software Engineering, IEEE Transactions on, 30(11), 754-769. Wohlers, S. D., & Jamieson, J. (2014). “What in me is Dark, Illumine”: developing a semantic URL learning analytics solution for Moodle. In B. Hegarty, J. McDonald, & S.-K. Loke (Eds.), Rhetoric and Reality: Critical perspectives on educational technology. Proceedings ascilite Dunedin 2014 (pp. 110-119).