STUDENTS’ ASSESSMENTS

IN MEDICAL EDUCATION

AMAL AL OTAIBICCP, MME

OBJECTIVES Define assessment. Identify purpose and importance of

assessment. Enumerate different methods used in

medical education. Differentiate between formative &

summative assessment.

WHY DISCUSS ASSESSMENT?

WHAT IS ASSESSMENT?

The word ‘assess’ comes from the Latin verb ‘assidere’ meaning ‘to sit with’.

In assessment one is supposed to sit with the learner. This implies it is something we do ‘with’ and ‘for’ students and not ‘to’ students (Green, 1999).

WHAT IS ASSESSMENT?Assessment in education is the process of gathering, interpreting, recording, and using information about students’ responses to an educational task. (Harlen, Gipps, Broadfoot, Nuttal,1992)

VALUES AND ATTITUDES ABOUT ASSESSMENT1. Teachers value and believe in students.2. Sharing learning goals with the

students.3. Involving students in self-assessment.4. Providing feedback that helps students

recognize their next steps and how to take them.

5. Being confident that every student can improve.

6. Providing students with examples of what we expect from them.

TYPES OF ASSESSMENT Formative Assessment

Summative Assessment

FORMATIVE ASSESSMENT Assessment for learning Taken at varying intervals throughout a course

to provide information and feedback that will help improve the quality of student learning the quality of the course itself

Provides information on what an individual student needsTo practiceTo have re-taughtTo learn next

KEY ELEMENTS OF FORMATIVE ASSESSMENT

1. The identification by teachers & learners of learning goals, intentions or outcomes and criteria for achieving these.

2. Rich conversations between teachers & students that continually build and go deeper.

3. The provision of effective, timely feedback to enable students to advance their learning.

4. The active involvement of students in their own learning.

5. Teachers responding to identified learning needs and strengths by modifying their teaching approach(es).

Black & Wiliam, 1998

SUMMATIVE ASSESSMENT Assessment of learning Generally taken by students at the end

of a unit or semester to demonstrate the "sum" of what they have or have not learned.

Summative assessment methods are the most traditional way of evaluating student work.

"Good summative assessments--tests and other graded evaluations--must be demonstrably reliable, valid, and free of bias" (Angelo and Cross, 1993).

Formative and summative assessment are interconnected. It is widely and empirically argued that formative assessment has the greatest impact on learning and achievement.

Formative ‘… often means no more than that the assessment is carried out frequently and is planned at the same time as teaching.’ (Black and Wiliam, 1999)

‘… provides feedback which leads to students recognizing the (learning) gap and closing it … it is forward looking …’ (Harlen, 1998)

‘ … includes both feedback and self-monitoring.’ (Sadler, 1989)

‘… is used essentially to feed back into the teaching and learning process.’ (Tunstall and Gipps, 1996)

Summative‘…assessment (that) has increasingly been used to sum up learning…’(Black and Wiliam, 1999)

‘… looks at past achievements … adds procedures or tests to existing work ... involves only marking and feedback grades to student … is separated from teaching … is carried out at intervals when achievement has to be summarized and reported.’ (Harlen, 1998)

PURPOSES OF ASSESSMENT

FORMATIVE To provide a profile of what a student has learnt. To provide feedback to students so they know where to

improve their learning. To motivate students. To help students to develop their skills of self-

assessment. To improve teaching.

SUMMATIVE To evaluate a student’s strengths and weaknesses. To pass or fail a student. To grade or rank a student. To license to proceed. To license to practice. To select for future employment.

IMPORTANCE OF ASSESSMENT FOR LEARNING

[Brown & Knight] “Assessment is at the heart of the student

experience”

[Chalmers]“Assessments define what students see as

important”

[C. Van der Vleuten] - “Tests drive learning”

IMPORTANCE OF ASSESSMENT FOR LEARNING (CONT’D)

Students take cues from what is assessed rather than from what their lecturers assert is important

“If you want to change student learning then change the methods of assessment” [Chalmers]

WEAKNESSES IN ASSESSMENTS - not be representative of the student’s

capabilities- not match learning objectives of the

course- draw on too narrow a domain (e.g., only

written exams used) - be over-weighted with particular skills &

underweight others

COMMON WEAKNESSES IN ASSESSMENT SYSTEMS

Overload of students and staff. Too many assignments with the same deadline set in the

department/school Insufficient time for students to complete assignments Insufficient time for staff to mark the assignments Inadequate or superficial feedback provided to students Wide variations in assessment demands by different

courses/modules Wide variations in marking across courses/modules.

[Brown, Bull and Pendlebury]

OTHER COMMON WEAKNESSES IN ASSESSMENT SYSTEMS Wide variations in marking within a module by tutors Fuzzy or non-existent criteria. Students do not know what is expected of them Students do not know what counts as a good or bad

assignment/project. Assessment viewed by some departments as an extra

rather than a recognized use of staff time

Brown, Bull and Pendlebury

Validity

CHARACTERISTICS OF INSTRUMENTS

Reliability

Educationalimpact

Acceptability

Cost

Validity

Reliability

Educationalimpact

RELIABILITY

(the degree to which the measurement is accurate and reproducible)

VALIDITY

(WHETHER THE ASSESSMENT MEASURES WHAT IT CLAIMS TO MEASURE)

A SIMPLE MODEL OF COMPETENCE

Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine (Supplement) 1990; 65: S63-S7.

Knows

Shows how

Knows how

Does

Pro

fess

iona

l aut

hent

icity

Cognition

Behaviour

A SIMPLE MODEL OF COMPETENCE

Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine (Supplement) 1990; 65: S63-S7.

Knows

Shows how

Knows how

Does

Pro

fess

iona

l aut

hent

icity

Written, Oral orComputer based assessment

Performance or hands on assessment

CLIMBING THE PYRAMID......

Knows

Shows how

Knows how

Does

Knows Factual tests: MCQ, essay type, oral…..

Knows how (Clinical) Context based tests:MCQ, essay type, oral…..

Shows howPerformance assessment in vitro:OSCE, SP-based test…..

Does Performance assessment in vivo: Masked SPs, Video, Audits…..

ASSESSMENT METHODS Written examinations

Multiple choiceTrue/False Matching Short answer Essay

Oral

NEW ASSESSMENT METHODS Modified essay question (MEQ) Objective Structured Clinical

Examination (OSCE) Mini-Clinical Evaluation Exercise

(Mini-CEX) 360 degree (multi source feedback) Portfolio

MODIFIED ESSAY QUESTION (MEQ)

An assessment method where you have a scenario followed by sequential questions.

Strengths Good reliability. It measure students’ skills in clinical reasoning and

problem solving abilities using what they know about the topic.

 Weaknesses Need resources (human) Time consuming A lot of paper work. It can’t be used alone.

WHAT IS AN OSCE

Objective Structured Clinical Examination

Harden RG and Gleeson FA Assessment of clinical competence using an objective structured clinical examination (OSCE) Medical Education,1979, Vol 13: 41-54

OSCE - OBJECTIVE

All the candidates are presented with the same test

Specific skill modalities are tested at each station History taking Clinical examination Procedures

OSCE - STRUCTURED

The marking sheet for each station is structured.

Structured interaction between examiner and student

OSCE – CLINICAL EXAMINATION

Test of performance of clinical skills candidates have to demonstrate their

skills, not just describe the theory

SKILLS – WHICH SKILLS

Clinical Diagnostic

Procedural skills

Communication skills

OSCE STATIONS TYPESI. Clinical stations involving interaction

between the student & a real or simulated patient.

II. Practical stations involving performance of technical procedures (e.g. performing CPR).

III. Static stations do not involve any physical interaction but the task may be very varied.

ESSENTIAL ELEMENTS OF OSCE STATIONS

I. A clear definition of the task to be performed by student.

II. Precise and clear instructions to students,examiners and simulated patients.

III. A rating form or marking sheet.IV. A list of requirements (e.g.

Equipments,scripts for the simulated patients.

TITLE SHEET

CANDIDATE INSTRUCTION

EXAMINER INSTRUCTION

MARKING SPECIFICATION

MARKING SHEET

REQUIREMENTS LIST

SIMULATED PATIENT INSTRUCTIONS

CLASSIFICATION

PABXRoom

Fire ExitFire Exit

Secretary

Secretary

CR

ConferenceRoom

MIT Room

Tutorial Rm.. 7

Tutorial Rm. 6

Auditorium

Students ASSEMBLYCR

CR

Kitchen

StockRoom

AliHadwer

FemaleWaiting

ElectricalRoom

Laboratory 1

Laboratory 2

CRStockRoom

Rear Entrance

Main Entrance

Library

Mr.Shakil

Con

trol

Roo

m

Computer Lab.

Tutorial Rm. 4

Station - 1

Tutorial Rm. 3

Station - 2

Tutorial Rm. 2

Station - 3

Tutorial Rm. 1

Station - 4

Station - 6

OSC

E

Exam

. Flow

PABXRoom

Rest &

Refreshm

entA

rea

Tutorial Rm. 5

Station - 7

Station - 5

Extra Rest Stationfor Round - 3

COLLEGE OF MEDICINE OSCE REGULATION

Each station is 7 minutes. 1 minute is allowed to students to move

between stations & examiners to arrange themselves.

Minimum 7 stations in each exam, maximum 9.

Training of simulated patients & examiners is obligatory.

Feedback session among the examiners & Block

Coordinator after the OSCE exam.

Every candidate does the same examinationGreater opportunity for objectivityOSCE Stations are re-useable

Advantages of OSCE

DISADVANTAGES OF OSCE

Costly in terms of resources & manpower

Time consuming to construct & administer

Appropriate space & facilities

Experience needed to administer

MINI-CLINICAL EVALUATION EXERCISE (MINI-CEX) is a rating scale developed by the

American Board of Internal Medicine (ABIM) in the 1990s to assess six core competencies of residents. These are; medical interviewing skills, physical examination skills, professionalism, clinical judgment, counseling skills, organization & efficiency.

MINI-CLINICAL EVALUATION EXERCISE (MINI-CEX) Strengths

Direct observation of candidate performance Allows global evaluation of performance Good inter-rater reliability Practical & easy to use Provide immediate feedback Authentic

Weaknesses Faculty training is needed to improve

reliability It is not possible to assess all aspects of

competencies through a single encounter Needs lot of resources

360 DEGREE (MULTI SOURCE FEEDBACK): consists of measurement tools

completed by multiple individuals in a person’s sphere of influence (ACGME, 2000). Usually, it assesses how frequently a behavior or an action is performed by a candidate using a rating scale (e.g. 1= frequently, 5= never). The observation is done by many different individuals, and generally includes the supervising physicians, peers, and nurses.

PORTFOLIO: is a collection of one’s professionals and

personal goals, achievements, and methods of achieving these goals (Amin and Khoo, 2003). It may contain items such as one’s best essay, written or research project, log books, letter of reflection and evidence of professional growth, to support individual accomplishment and progression (Friedman et al., 2001)

FACTORS INHIBITING ASSESSMENT A tendency for teachers to assess

quantity and presentation of work rather than quality of learning.

Greater attention given to marking and grading, much of it tending to lower self esteem of students, rather than providing advice for improvement.

A strong emphasis on comparing students with each other, which demoralizes the less successful learners.

THANK YOU