Learning and Educational Analytics
-
Upload
university-of-southern-queensland -
Category
Education
-
view
4 -
download
2
description
Transcript of Learning and Educational Analytics
Learning analytics and Moodle: So much we could measure, but what do we want to measure?
Associate Professor Michael Sankey, EdDDirector, Learning Environments and Media
SAF
Embarking on a project that involves: Establishing common codebase across all our 3
Moodle environments fully aligned with Mahara Extending the functionality of eAssessment within
Moodle (replacing EASE, CMA, EMS) Establishing a suite repositories in Equella Create new digital rights management workflow Enhance discoverability Establish learning analytics across L&T systems Align help resources to new regime and Provide PD
Learning analytics for our systems
Which systems? What tools? How big do we want the data? Is it just our USQ systems? What do we want to know? How do we want to use this data? Who gets involved? Who makes the decisions? I’ll come back to these questions at the end But first some background…
Siemens, G. 2013. Structure and logic of analytics. Available from http://www.learninganalytics.net/
Three levels within the institution
Learning Analytics
Educational Data mining
Educational
Analytics
All focus on the learner to some degree, either as an individual or in context to the institution
Educational Analytics
Academic and Learning Analytics
http://www.educause.edu/ero/article/penetrating-fog-analytics-learning-and-education
Siemens, G. 2013. Structure and logic of analytics. Available from http://www.learninganalytics.net/
Developing an analytics framework
Identify Tier 1 data
Data that exists (technical
analytical)
Form Tier 2 data Issues
intelligence
Create Tier 3 data
Context intelligence
Adapted from Terenzini, 2013; Padró & Frederiks, 2013
SBMI and Peoplesoft• AUSSE/UES• Grades• Graduation rate• Persistence• Retention• Student demographics• Student satisfaction data• Transfer rates
L&T Systems & RightNow• Co-curricular student
engagement activities data• Course interactions• Systems data• Learning Centre data• Other student learning
support activities data
Institutional emphasis for data collection & analysis:Customer service (transactional) or Student development
National policy preference
Learning analytics is the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs.
http://www.solaresearch.org/
Where did it originate
SoLAR exists to ensure that there is an expansive, transformative vision for what analytics might mean for the future of learning and to promote a very critical discourse that is non-partisan, and grounded as far as possible in practice-based research. SoLAR is a non-profit organization. Incorporation is currently underway.
Scope
At what level do we pitch? LMS data analytics
Easier to implement Limited data so not the whole picture Logs in Moodle are good, but not comprehensive
The Learning ecosystem analytics Complex – needs an open standards model and
potentially access to external repositories Much more holistic picture In our case, Mahara, Equella, BB Collaborate,
EASE, Library, lecture capture, etc
LMS is one of the primary providers of the data, since it preserves digital footprints of student interactions which can be mined for patterns of learning behaviour and teaching practice, and this allows for benchmarking and the monitoring of institutional quality initiatives.
Predictive analysis indicates that some students are higher risk than others; for those who are first in family or from a low socio-economic background, the risk of failure increases. There is a question as to what constitutes quality learning. Analytics is a key player in this field, given that it provides a vast amount of data and techniques for its analysis. Learners and their context are vitally important in this discussion.
How big is the data?
Typically in our Moodle we generate between 50-100 million log records per year
What is the Aim Finding out we have a problem before it fully
manifests If we accept this it gives us a framework to
consider our options
An perspective on big data OUA
As with other fields the key questions to ask are: what do you want to know; why do you want to know it and what are you going to do next?
Analytics makes sound educational and financial sense, it increases retention and encourages students to enrol again.
The biggest factor in student’s retention is intent of purpose; why are they doing what they want to do? There are things that can be done to aid them to achieve their intent of purpose.
Previous education is the single biggest predictor of success. So the question becomes, what supports can we put in place for those without this. E.g.: Invigilated exams in a student’s first unit decreases their chance of
success. This raises learning design issues for our introductory units. Other data shows that older students and female students are more likely
to succeed in their first Course. Coaching and contact are also predictors of retention and success, along
with preparatory units.
QUT study – Wendy Harper
Overall, the conclusions she drew were: The key predictor of success in a unit is GPA. The number of hits and days visiting a QUT Blackboard unit
site also predicts unit success. Students who are likely to fail a unit often do not engage early
enough with their online environment. Students who fail a unit often have alternating high peaks of
engagement and total disengagement. Factors such as gender, international or domestic enrolment,
and age make very little difference to student behaviour in online units.
‘Narrowly failing’ students often perform a much greater amount of online activities in the unit they are struggling in.
‘Narrowly failing’ students often show high engagement around early assessment pieces, but this drops off as the semester progresses.
“Learning and knowledge creation is often distributed across multiple media and sites in networked environments. Traces of such activity may be fragmented across multiple logs and may not match analytic needs. As a result, the coherence of distributed interaction and emergent phenomena are analytically cloaked”
Suthers, Rosen, 2011
Athabasca's approach
Siemens, G. 2013. Structure and logic of analytics. Available from http://www.learninganalytics.net/
Ethics
The ethical professional Respecting the rights of students Stepping in to
provide pastoral support advise about risks of failure advise about increasing chances of success
Research ethics Risk minimisation Needed for publication Issues with:
accessing ‘databanks’ anonymity
Privacy Issues
The Greater Good vs Big Brother Teacher:
“It’s unethical not to tell a student they are at risk of failing”
Student: “I don’t want you to be looking over my shoulder. I
can make my own choices about my study.” Reports to staff vs dashboards for students
Usefulness of analytics
What is the question to which analytics is the answer? Don’t just buy a product
Learning analytics are just indicators of behaviour They don’t explain behaviour
A single source of analytic data is probably insufficient Combine data into a data warehouse
Time to look at some different options
The Engagement Analytics block
http://docs.moodle.org/22/en/report/analytics/index It provides information about student progress against a range of
indicators. It provides feedback on the level of "engagement" of a student. “Engagement" refers to activities which have been identified by current research to have an impact on student success in an online course.
The plugin was developed as part of a NetSpot Innovation Fund project by Monash University (Dr Phillip Dawson), with code by NetSpot developers (Ashley Holman & Adam Olley).
It is a block that teachers can add to their Moodle course that will provide them with a quick graphical snapshot of which students are at risk.
It is important to note that the purpose of the plugin is to provide teachers with information only, it does not automatically take any action based on the indicators e.g. NO email or notification is sent to students automatically.
If desired the teacher would follow up on the information themselves, based on what they know about the student and their other communications.
GISMO
It is a visualization tool for Moodle that obtains tracking data, transforms the data into a form convenient for processing, and generates graphical representations that can be explored and manipulated by course instructors to examine social, cognitive, and behavioral aspects of distance students.
It can be included in any Moodle course as side block. Since it is aimed to help instructors, this block will be visible only to users who have the instructor role (students don't see it).
Each time the Moodle cron jobs runs, GISMO fetches students' data from Moodle logs, and performs some statistical calculations. The lifetime of GISMO data corresponds to the length of time of your Moodle logs.
It has Accesses overview
A graph reporting the student's accesses to the course. Accesses to the course
A graph reporting accesses for each student in a timeline. Accesses overview on resources
A graph reporting the number of accesses made by the students to the resources of the course
Assignments overview A graph reporting the submission of assignments. Color is mapped to the grade
assigned by the teacher. Quizzes overview
A graph reporting the submission of quizzes. Color is mapped to the grade. Resources accesses overview
A graph reporting an overview of the number of accesses to resources of the course.
Resources accessed by a particular student A graph reporting an overview of the student's accesses to resources on a timeline.
Students' accesses to resources A graph reporting, for each student, the number of accesses to resources of the
course.
A graph reporting the student's accesses to the course.
A graph reporting accesses for each student in a timeline.
A graph reporting the number of accesses made by the students to the resources of the course
A graph reporting the submission of assignments. Color is mapped to the grade assigned by the teacher.
A graph reporting an overview of the number of accesses to resources of the course.
Macquarie, UNSW and Netspot are working on a new student centric analytics tool for Moodle
SNAPP
http://www.snappvis.org/
SNAPP
The Social Networks Adapting Pedagogical Practice (SNAPP) tool performs real-time social network analysis and visualization of discussion forum activity within popular commercial and open source Learning Management Systems (LMS).
It essentially serves as a diagnostic instrument, allowing teaching staff to evaluate student behavioral patterns against learning activity design objectives and intervene as required a timely manner.
Dawson, S., Macfadyen, L., Lockyer, L., & Mazzochi-Jones, D. (2011). Using Social Network Metrics to Assess the Effectiveness of Broad-Based Admission Practices. Australasian Journal of Educational Technology, 27(1), 16-27. Also available from: http://www.snappvis.org/?page_id=4
Info from Shane Dawson (UniSA) Social interaction is one of the most important
of student behaviours and predictors of success. Student networks are the “single most potent source
of influence.” The tool provides a visualisation of social
networking. Different patterns are available to the individual, and mechanics which allow the data to be manipulated for different purposes.
It demonstrates that with students, like responds to like; they form self-regulating structures. It may be possible to manipulate group structures so that high-
performing students can assist low-performing ones. It may also be possible to direct teachers’ time to areas of need.
Dawson, S., Macfadyen, L., Lockyer, L., & Mazzochi-Jones, D. (2011). Using Social Network Metrics to Assess the Effectiveness of Broad-Based Admission Practices. Australasian Journal of Educational Technology, 27(1), 16-27.
Learning Analytics for Understanding
Low 10% student located in network
Students with a grade >75% < 90%
Social Network Analysis
Available from http://grsshopper.downes.ca/about.htm
Available from http://grsshopper.downes.ca/about.htm
Refining the signals from the Twitter feed
http://mashe.hawksey.info/2012/11/cfhe12-analysis-summary-of-twitter-activity/
BIM
David Jones FoE BIM (BAM into Moodle). BAM = Blog Aggregation Management. BIM is a Moodle module that supports an activity where:
Each student registers an individual external web feed. The feed might be generated by a blog, twitter or any other tool that produces a web feed. It's the student's choice what they use.
Each student uses that external feed to respond to a set of questions. Currently, those questions usually encourage the student in reflecting on their learning, often in the form of a reflective journal.
There is no need to have a set of questions. it maintains a copy of each students web feed, and attempts to
allocate student posts to the questions. it allows different teachers to track, manage and mark posts for
different groups of students. Allows a coordinating teacher to allocate teaching staff to different
groups, track their marking progress and all student activity. Student results can be sent to the Moodle gradebook.
Jones, D. 2013. BIM – Feed Aggregation. Available from http://davidtjones.wordpress.com/research/bam-blog-aggregation-management/
ACODE prepared a literature review containing 165 categorised references in an Endnote library
Learning analytics for our systems
The big Q Your big Answer
Which systems?
What tools?
How big do we want the data?
Is it just our USQ systems?
What do we want to know?
How do we want to use this data?
Who gets involved?
Who makes the decisions?