e-AFFECT: What have we learnt?
-
Upload
allegra-mendez -
Category
Documents
-
view
31 -
download
3
description
Transcript of e-AFFECT: What have we learnt?
Phase 1 •3 programme areas•65 staff and 1722 students
Phase 2 •7 programme areas•95 staff and 1482 students
Phase 3 •4 programme areas•95 staff and 1287 students
PROJECT MANAGEMENT
DISSEMINATION
Phas
e 1
Phas
e 2
Phas
e 3
2011-2012 2012-2013 2013-2014
Baseline Plan Develop
Baseline Plan Develop
Baseline Plan Develop
Evaluation/refinement
Evaluation/refinement
Evaluation/refinement
Intervention 1 Intervention 2
Intervention 1 Intervention 2
Evaluation Evaluation
Evaluation Evaluation
On going activity
EVALUATION
EMBEDDING
Participating programmesAHSS EPS MHLS
Business Management Civil Engineering Biomedical SciencesDrama Computer Science Midwifery
English Environmental Planning Pharmacy (baseline only)
Film Studies Psychology
Law
Music
Social Policy
Managing assessment and feedback
• Not all Schools have assessment policies • Sometimes there is one for an area within a School• Some are very detailed • Others are very prescriptive
• Hard copy versus electronic submission of coursework• Variation in the release of feedback and marks on coursework• Variation in the provision of feedback on exams
Assessment and feedback timelines
Represents medium stakes assessment
Represents low stakes assessment
Represents high stakes assessment
F
F
F
Represents low stakes formative assessment
Represents medium stakes formative assessment
Represents high stakes formative assessment
Where students have opportunity to act on one or more formative/summative assessments in a
future a summative assessment
Adapted from the ESCAPE project
2 4 6 8 10 12 week
Assessment and feedback timelines
2 4 6 8 10 12 week
2 4 6 8 10 12 week
2 4 6 8 10 12 week
SEMESTER 1 LEVEL 1
SEMESTER 2 LEVEL 1
2 4 6 8 10 12 week
2 4 6 8 10 12 week
2 4 6 8 10 12 week
2 4 6 8 10 12 week
2 4 6 8 10 12 week
2 4 6 8 10 12 week
F
F
F
F
F
F
F
F
F
SEMESTER 2 LEVEL 2
Level 1
2 4 6 8 10 12 week
2 4 6 8 10 12 week
2 4 6 8 10 12 week
Level 3
2 4 6 8 10 12 week
2 4 6 8 10 12 week
2 4 6 8 10 12 week
F
F F
Level 2
2 4 6 8 10 12 week
2 4 6 8 10 12 week
2 4 6 8 10 12 week
2 4 6 8 10 12 week
Attendance and presentation of weekly
seminar – one chosen at random
Level 3
2 4 6 8 10 12 week
2 4 6 8 10 12 week
F
F
F
F
F
F
Student and staff questionnaires
Student questionnaire• Study and learning strategies• Extent to which practice is
occurring• About themselves
• Response rate = 20.4%
Staff questionnaire• Assessment and feedback
strategies• Extent to which practice is
occurring• About themselves
• Response rate = 47.3%
Student profileFaculty Year of study Gender Age
AHSS 28.7% Level 1 36.9% Female 74.5% 18-22 years
77.1%
EPS 47.2% Level 2 33.5% Male 25.7% 23+ years
22.9%
MHLS 24.1% Level 3+ 29.6%
Study and learning strategies
• Amount and distribution of study effort 6 statements• Assignments and learning 8 statements• Quantity and timing of feedback 6 statements• Quality of feedback 6 statements• What students do with feedback 6 statements• The examination and learning 6 statements
Based on Gibbs and Simpson (2004)
Analysis Faculty Year of study Gender Age
Amount and distribution of study effort 6 significant 5 significant 2 significant
Assignments and learning 5 significant 1 significant 2 significant 4 significant
Quantity and timing of feedback 6 significant 2 significant 1 significant 4 significant
Quality of feedback 6 significant 6 significant 4 significant
What you do with the feedback4 significant 5 significant 2 significant 5 significant
The examination and learning 6 significant 2 significant 1 significant 4 significant
Significance p ≤ 0.05
Key findings Faculty Year of
studyGender Age
On this course, it is possible to do quite well without studying much.
Moderately strong ns Moderately
strong ns
In weeks when the assignments are due I put in many more hours. Weak ns Moderate Weak
The assignments are not very challenging.Moderate Weak Strong Weak
The feedback does not help me with any subsequent assignments.
Moderately strong Moderate ns ns
I understand things better as a result of the exam. Weak ns ns Moderate
• 27 statements based on ATLAB questionnaire (Whitelock and Cross, 2011)• Fully occurring – not in a position to say• Assessment criteria are written and used• All marking criteria are shared with students in advance and
feedback refers to them• Feedback is intended to acknowledge, consolidate and promote
student learning• Students get feedback which corrects errors and supplies further
information• Students have opportunities for cooperative and collaborative
assessment
Analysis
Students Student - Staff
Faculty Year of study
Gender Age
Number of significant statements
17 16 4 6 23
Significance p ≤ 0.05
Students
Faculty Year of study
Gender Age
Students have opportunities for cooperative and collaborative assessment
Moderately strong Moderate ns Weak
Assessment activities are situated within real-world contexts (i.e. they reflect the problems, tasks or competencies a practitioner would face)
Moderate ns ns ns
Students and staff Staff - studentsStudents have opportunities for a feedback dialogue (peer or teacher-student) around assessment tasks on the course
Moderate
Assessment includes the use of a portfolio or similar practice Moderately strong
A variety of communication channels are used to provide feedback to students
Moderately strong
Students are told what type of feedback they can expect and when and how they will receive it
Moderate
Opportunity to act on feedback
Help clarify good performance
Development of self-assessment and reflection
Civil Engineering 1
0-34 35-39 40-49 50-59 60-69 70-79 80-1000.0
5.0
10.0
15.0
20.0
25.0
30.0
35.0
2011-12 2012-13 2013-14
%
Opportunity to act on feedback
Help clarify good performance
Development of self-assessment and reflection
Encourage time and effort on challenging tasks
0-34 35-39 40-49 50-59 60-69 70-79 80-1000
5
10
15
20
25
30
35
40
45
2011-12 2012-13
% a
chie
ving
Civil Engineering 2
also includevideo and audio stimuli, to target
particular phoneticfeatures.
‘Fast feedback’s really important in phonetics. You need to establish good habits right from the start.’
Phonetics
Opportunity to act on feedback
Help clarify good performance
Development of self-assessment and reflection
Encourage time and effort on challenging tasks
Encourage interaction and dialogue around learning0-34 35-39 40-49 50-59 60-69 70-79 80-100
0.00
10.00
20.00
30.00
40.00
50.00
60.00
2011-12 2012-13 2013-14
%
Phase 2 activities• Some of the activities are represented in the Marketplace• Other activities include:• Marking and feedback workshops with students• Students indicating how they have used feedback• The use of Vimeo for uploading films• E-submission/marking/feedback• QuestionMark Perception for formative and summative activities• Peer review using PeerMark
Phase 3 – planned activities
• Development of feedback comments bank• Student generated MCQs using PeerWise• QuestionMark Perception for online objective testing• Vodcasts• Screen capture feedback on draft work • Clarification of assessment criteria and standards – guides and
workshops
– Collaborate project
A Cross-Institutional Approach to Assessment Redesign:Embedding Work-Integrated Assessments within the Curriculum
Designing Work-Integrated Assessments
Dimensions of a Work-Integrated Assessment
Aligning Technologies to the Model: ‘Top Trumps’ Using a Model for Work-Integrated Assessment:Six Dimensions
School of History and Anthropology: Student generated MCQs
‘The lecturer got us to sign up to a site called 'Peerwise' which involves students creating multiple choice
questions and the other students are able to answer them. I thought this was a brilliant idea as it’s great for
revision.’ Module evaluation, Deviance History, QUB, 2013
Centre for Medical EducationA strategy to help students to actively engage with feedback provided: sharing experience from the undergraduate medical student selected component (SSC) programme
Student responses
I constructed a more balanced argument…
I incorporated more cases to illustrate..
I cut down… to allow more room for….
I put more emphasis on this point….
Examples of feedback
justify reasons for and against this ….
make greater use of relevant cases and sources
to support discussion here….
background should be more focused on…
e-AFFECT
...Shared and jointly owned standards leading
to consistency of feedback
C described his practice in giving feedback when students
previously had had to provide a reflection on the strengths and weaknesses of their work when
they requested feedback. He read their reflective statements
and incorporated these comments into the feedback he had already written. A dialogue
was thus developed.
Environmental Planning - JingImplementation
1. As a method of delivering formative feedback on visual work to enhance skill development
2. To enhance skill development with Jing Screencast Tutorials
Environmental Planning – Acrobat Pro
Implementation
Annotation over drawings is a standard way of giving feedback for design based modules. Students can access this feedback through their tablets, computers or smart phones.
The module consisted of three stages and Acrobat Pro was used to give feedback for the first of the three submissions in the module.
Environmental Planning - VoiceThreadImplementation
For this module we made four 15 minute videos on a range of planning subjects and asked specific questions that students need to answer. All students can see all the comments that other students and the tutors make.
Drama – VoiceThread for student engagementHow did I use it?
• PowerPoint lectures were put into VoiceThread with my commentary as a voiceover.
• Students were asked to watch the lectures before the class.
• Students were required to add substantive comments or questions to the VoiceThread.
• 15% of the marks were allocated for this activity
Centre for Medical Education – Virtual Patients
Turning flat word-processed patient cases into interactive learning objects
Storyboard
Computer Science - Offline objective marking
105 Level 3 students take a series of assessed practicals and a final assessment in QMP. As the students work through the practicals they complete a worksheet in QMP.
The teaching team used Excel to score the responses
References• Cooperrider, D & Whitney, D 2005 Appreciative Inquiry: a positive
revolution in change Berrett-Koehler Publishers, Inc. San Francisco• Gibbs, G & Simpson, C (2004) Conditions under which assessment
supports learning, Learning and Teaching in Higher Education 1, 3-31 • Nicol, D (2009)
Transforming assessment and feedback: enhancing integration and empowerment in the first year, The Quality Assurance Agency for Higher Education, Mansfield• Whitelock, D. & Cross, S. (2011). Assessment Benchmarking:
Accumulating and accelerating institutional know-how for best practice. International Journal of e-Assessment (IJEA), Vol 1. No. 1 http://journals.sfu.ca/ijea/index.php/journal/article/view/18
Resources
• e-AFFECT Jisc Design Studio http://jiscdesignstudio.pbworks.com/w/page/50671059/e-AFFECT%20Project • e-AFFEECT Resources Jisc Design Studio http://
jiscdesignstudio.pbworks.com/w/page/68656954/e-AFFECT%20Resources • e-AFFECT blog http://blogs.qub.ac.uk/e-affect/ • REAP project http://www.reap.ac.uk/ • Viewpoints Project Jisc Design Studio http://
jiscdesignstudio.pbworks.com/w/page/29227748/Viewpoints%20project and http://wiki.ulster.ac.uk/display/VPR/Home