2016-05-30 Venia Legendi (CEITER): Luis Pablo Prieto
42
Orchestration and Learning Analytics for Educational Innovation Luis P. Prieto Venia Legendi for a Senior Research Fellow position Tallinn University, 30 May 2016
-
Upload
ifi8106tlu -
Category
Education
-
view
88 -
download
2
Transcript of 2016-05-30 Venia Legendi (CEITER): Luis Pablo Prieto
Orchestration and Learning Analytics
for Educational Innovation
Luis P. Prieto
Venia Legendi for a Senior Research Fellow position
Tallinn University, 30 May 2016
1. Orchestration and LearningAnalytics: A lecture
2. A vision to apply them in CEITER (and beyond)
2
1. Orchestration and Learning Analytics
3
Orchestration is…
“the process of productively coordinatingsupportive interventions across multiple learning activities occurring at multiple social levels”
(Dillenbourg, Järvelä & Fischer, 2009)
5
why does orchestrationmatter?
6
Lack of educational technologyadoption in authentic settings• … in learning design (Mor, Craft & Hernández-Leo, 2013)
• … in collaborative learning (Looi, So, Toh & Chen, 2011)
• … in teacher inquiry (Emin-Martínez et al., 2014)
• …
7
8
9
How can orchestrationbe used?
10
Application to edutech design
• GLUE!-PS: From an open architecture to solve thefragmentation of learning design authoring tools…
• … to a tool to support orchestration of existingVLEs and external tools (e.g., runtime changes, etc.)
(Prieto et al., 2014) 11
What does orchestration entail?The ‘5+3 Aspects’ framework
(Prieto, Holenko-Dlab, Gutiérrez, Abdulwahed & Balid, 2011)12
Application to edutech evaluation
• Does GLUEPS-AR support the orchestration of mobile AR-based learning activities?
(Muñoz-Cristóbal et al., 2015)13
… but technology aloneis not enough to guarantee adoption
14
Modelling teacher orchestration: observational studies
(Prieto et al., 2011)15
Applications to teacherprofessional development
Observationalstudies
Successfulroutines/practices/patterns
Professional development
workshops
(Prieto et al., 2013)
16
Modelling teacher orchestration: diving deeper with eye-tracking
17
Modelling teacher orchestration: diving deeper with eye-tracking• Study teacher cognitive
(orchestration) load• Class-level interactions are
higher load
• Reading faces is higher load
• Novice teachers have clearerload trends than experts
• …
(Prieto, Wen, Caballero, Sharma & Dillenbourg, 2014)(Prieto, Sharma, Wen & Dillenbourg, 2015)
(Prieto, Sharma & Dillenbourg, 2015)
18
Learning Analytics
• Main goal: Aiding educators in understanding and improving teaching and learning processes
• Main difference with Educational Data Mining(EDM): human in the loop
• Aimed at interventions/supportive actions
• Cycle of data gathering, analysis, feedback/visualization
• Hence, LA can be seen as a very useful tool fororchestration
• Awareness/Assessment aspect in ‘5+3’ framework
(Siemens & Long, 2011)
19
(Clow, 2012)
Orchestration + LA = Teaching Analytics• First attempt at automating the
observational modelling of orchestrationgraphs using multimodal analysis
• Audio, video, eyetracking, accelerometers, EEG
• Exploration of basic, general-purposefeatures and algorithms
(Prieto, Sharma, Rodríguez-Triana & Dillenbourg, 2016) 20
Orchestration + Learning Analytics= Teaching Analytics• First attempt at automating the observational
modelling of orchestration graphs usingmultimodal analysis
• Audio, video, eyetracking, accelerometers, EEG
• Exploration of basic, general-purpose features and algorithms
• Results:• Predicting teacher activity accuracy: 65%• Predicting social plane accuracy: 90%• Audio-video channels most useful• Random forest and GBM as best algorithms
(Prieto, Sharma, Rodríguez-Triana & Dillenbourg, 2016) 22
2. A vision to apply themin CEITER (and beyond)
23
Previously, on this
Venia Legendi…
• Orchestration as a (complex) metaphor forteaching practice
• Focus on an innovation’s potential for adoptionwithin ecosystem of authentic educational setting
• Applications to educational technologies as well as the analysis of (adoption) practice
• Learning Analytics (LA)• “Orchestration-flavored” take on EDM?
• Multimodal LA to model what happens outside thebox (face-to-face)
25Image from https://www.flickr.com/photos/65092514@N08/18679295525
Have orchestration and learning analytics researchsolved adoption?Well, not quite…
26
An example orchestration/LA research project• Prolearning: simple app to foster PD conversations
based on everyday data gathering by teachers
• Students are asked simple questions about theirexperience in terms of school-emphasized practices& teachers have to predict student response
• http://prolearning.realto.ch
• Successes:• Teachers used it in ~70% of all their lessons for 2 weeks
• Recorded evidence of changes in student experience
• Takes 2 mins: “I can’t see how it can be more efficient”
27
What didn’t work
Teacher interviews:
“… any time you’re teachingthere is a hundred variablesthat you have to accountfor”
“[I would use it] if you can demonstrate that these data are reliable…”
28
Unsolved challenges
1. Trust, privacy, agency and other ethical factors• Traditional top-down innovations quickly get subverted
2. The quest for added value• Hard: Evidence of benefits for student learning• Soft: Help with existing “chores”, social value…
3. Still does not scale well!• Measure blended learning, but with ecological validity? • Slow: cycles of analysis and reflection are loooooong
• Can we do better than final course assessment?
• Fail to create new practices that go with the technology• Routines are how we deal with our complex environments
29
The CEITER project
• From project description:• Teaching 21st century skills• Innovative methods & learning environments tailored to
learner’s development/needs/capabilities• Improving the research staff capacity & new generation of
researchers• Use Learning Analytics
• From Tobias’s talk (06.04.2016):• Tech/Tools cannot be separated from teaching and learning
practices• Educational innovation is taken up at different social entities• Multiple levels of analysis (institutional, PD, learner…)
30
Can we change Estonianeducation from the“ivory tower”?Probably not…
31
Fresh perspective
on Learning
Analytics
Ecosystem of
tools for evidence-
based orchestration
CEITER
Agency &
Ethics
Clear added
value
Adoption
& scale1 2 3
Blended LA in
distributed
LEs + ‘just
enough’
multimodal LA
Focus on
assessing
learning
Focus on
everyday
evidence
Capture
successful
practices
Teacher
training and
community
of practice
Open, privacy-
conscious
architecture +
specific tools
Integrate
existing
tools and
practices
Lightweight
assessment
techniques
Evidence-
based
training
Orchestr
ation-
aware
tools
32
33
CEITER’s
virtuous
cycle
Personal
practice
analysis
tools
Facilitate
certain
evaluation
“chores”
Evidence on
tool/practice
effectivenessEffective
practices
fed back
to CoP
Ecologically
-valid
educational
research
Evidence-
based policy
making to
reward/support
practices
Learning
Analytics’
vicious
cycle
Modus operandi
• Design-based research, mixed methods, participatory…
• Lean startup model: start qualitative, slowly buildquantitative indicators, minimal prototypes
• Cross-functional teams/projects
• Eat your own dog food!
• Promote international exchanges (students, faculty)
• Focus on EduTech entrepreneurship to attractstudents & researchers
34
A path for (part of) CEITER
35
Phase 1
1st LA prototypes
Applicationin PD
(teachers & researchers)CEITER as
a 1st evidence-based CoP
Basic practicecapture
Phase 2
1st orchestrati
on tools
Applicationin teachereducation
Pre-serviceteacherseb-CoP
Focusedcapture, multi-
classroom
To phase 3
Why me?
Contributions to wide variety of orchestration-related researchcommunities
• From CSCL to LD to HCI to LA…
Experience in internationalprojects & collaboration
Quanti & quali methods
Used to Enjoys inter-disciplinarywork
Motivated by innovationpractice, teacher PD, …
FP6
FP7LLP
36
… but not only me!
• Other CEITER team members:• Existing expertise in lifelong & workplace learning
• Infrastructure expert to communicate different data sources and databases
• Psychology profile: to determine the “building blocks” of learning and how to (micro-)assess them
• Pedagogical approach profile: to model existingpractices/tools and propose new ones
• Other researchers at U. Tallinn (or T.U. Tallinn)• Partnerships with local HCI, Signal processing, Machine
learning, Sensors, (Estonian) voice recognition… experts
37
… but not only me! (II)
• My existing network of relevant contacts, e.g.:• Orchestration: P. Dillenbourg (CH), P. Tchounikine (FR)
• Pedagogies, e.g., CSCL: Y. Dimitriadis (ES), D. Persico (IT)
• Multimodal Learning Analytics: X. Ochoa (EC), S. D’Mello (US)
• Large-scale school innovation: C.K. Looi (SNG), J. Roschelle (US)
• U. Tallinn’s own network of national & internationalcontacts
• Incl. Estonian teacher/school networks, policy-makers
38
Beyond CEITER
• This vision can be too ambitious for CEITER itself
• Once we have initial results, use them to “pitch” consortium and proposals
• EU calls:• Evidence-based policy aspect: SwafS-21-2017• Explore prototypes’ market potential: SMEInst-12-2016-2017• If things go well, ERC-STG-2018/2019…
• Other funding for non-EU countries, e.g.:• US: Partnerships for International Research and Education (NSF
PIRE) • Singapore (parallel for systemic change): advisors/expert for
national project (visit planned 2017)
39
This is just a modular vision…
CEITERBlended LA in
distributed
LEs + ‘just
enough’
multimodal LA
Teacher
training and
community
of practice
Open, privacy-
conscious
architecture +
specific tools
Lightweight
assessment
techniques
40
… within an ecosystem of interests
Blended LA in
distributed
LEs + ‘just
enough’
multimodal LA
Teacher
training and
community
of practice
Open, privacy-
conscious
architecture +
specific tools
Lightweight
assessment
techniques
Agile &
entrepreneurship
methods in
research
Scientific writing
& communication
support
Researcher
communities
of practice
MOOCs
Machine
learning
Learning
design
Creative
writing
Tangible
& paper
UIsData
literacy
Data
science
CSCL
Dissemination
to the public41
Thank you! Questions?
Email: [email protected]: https://people.epfl.ch/luis.prietoLinkedIn: https://www.linkedin.com/in/lprisanGoogle Scholar:
https://scholar.google.ch/citations?user=ySpnj0MAAAAJ
42
