Post on 16-Feb-2017
Plenary 2
Tim McNamaraSchool of Languages and Linguistics, University of Melbourne
What language assessment
theorists and practitioners have
learned about Occupation-
specific Language Assessments
Tim McNamara
School of Languages and Linguistics &
Language Testing Research Centre
University of Melbourne
Working with language across boundaries
What is the role of language in the competence
of health professionals?
Do language professionals and health
professionals (‘domain experts’) share an
understanding of this role?
How can language professionals come to
understand the thinking of domain experts?
How can language specialists understand the
the specific purpose domain?
The problem of reconciling the different
perspectives of language and non-
language professionals is a perennial
concern for Languages for Specific
Purposes practitioners.
Basturkmen & Elder (2004, p. 677)
Occupational English Test (OET)
Specific-purpose English language and
communication (ESP) test for migrant health
professionals in 12 professions whose training has
not been in English
4 skills test designed to assess the readiness of candidates
to manage the communicative tasks of the health workplace
– Speaking, writing – communicative tasks specific to each
profession
– Listening, reading – communicative tasks common to all health
professions
– Recognized for professional registration of migrant health
professionals in Australia, New Zealand and Singapore
The policy question…
Need to ‘get it right’ about who to admit, who to
exclude
Involves balancing
patient safety standards vs
not creating unreasonable restrictions on IMGs’
access to clinical experience
Evidence that the decisions are not always right –
aggrieved failing candidates
complaints from clinical supervisors
By what criteria should candidates be judged?
2 studies: on speaking criteria, on writing criteria
Australian Research Council Linkage grant (LP0991153)
Ethics approval for study given by University of Melbourne
Languages & Linguistics Human Ethics Advisory Group
7
Towards improved healthcare communication:
Development and validation of language
proficiency standards for non-native English
speaking health professionals
Catherine Elder, Robyn Woodward-Kron, Tim McNamara,
Elizabeth Manias, Geoff McColl, Gillian Webb
John Pill, Sally O’Hagan, Diana van Die
OET Speaking sub-test
Test task: role-plays between health professional and patient
– profession-specific
Test content: common situations encountered in the workplace
– accuracy and currency confirmed by health professional advisors
Assessment: linguistic criteria applied by language professionals to
audio recorded performances of interaction
Overall Communicative Effectiveness
Intelligibility
Fluency
Appropriateness of Language
Resources of Grammar and
Expression
SAMPLE MATERIALS - SPEAKING SUB-TEST
(PHYSIOTHERAPISTS): Candidate’s card
SETTING: Hospital clinic
PATIENT: An elderly person who is recovering from a stroke (CVA). The patient is making slow progress in learning to walk again.
TASK: Talk to the patient about the following pieces of equipment:
• a wheelchair
• a walking frame
• a walking stick.
– Explain the advantages and disadvantages of each one.
– You would like the patient to be as independent in his/her movements as possible. You feel the frame is not appropriate.
– You want the patient to have a stick. You do not want the patient to have a wheelchair at this stage.
SAMPLE MATERIALS - SPEAKING SUB-TEST
(PHYSIOTHERAPISTS): Role-player’s card
SETTING: Hospital clinic
PATIENT: You are an elderly person who is recovering from a stroke. You feel you are making painfully slow progress, and don't really expect to be able to walk again.
You feel you should be allowed to have a wheelchair.
TASK: Ask the physiotherapist when you will be given a
wheelchair.
Insist on your need for this equipment. Explain that you feel that
the painful exercises you are doing at the moment are pointless,
and that you are pessimistic about your chances of making real
progress.
Be difficult!
3 yr study of ‘indigenous’ assessment criteria for
judging communication in medical workplace
Investigation of feedback by clinical educators on
clinical spoken interaction between trainee health
professionals (HPs) and patients:
to investigate what qualities are valued by health
professionals in spoken clinical communication
and
to see if these should be and can be included in
a test of English for spoken clinical
communication
Towards improved healthcare communication
(Australian Research Council grant, 2009-2013)
Jacoby (1998): Indigenous assessment
Indigenous assessment – naturally
occurring, routine instances where
communication is judged
Indigenous assessment criteria – criteria
which emerge in these contexts
Investigating views of content experts
Jacoby 1998:
Studied feedback from expert physicist to junior
physicists rehearsing conference papers
Presenters both native and non-native speakers
of English S and NNS (some with limited
language proficiency)
No distinction between these groups in
feedback – occasional remarks on language
Organization, timing etc more important
Towards improved healthcare communication
(Australian Research Council grant, 2009-2013)
Phase 1
Identifying criteria indigenous to workplace context
Translating them into assessment criteria
Phase 2
Implementing new criteria
Investigating how they function
Phase 3
Setting cut-scores (e.g., pass/fail)
3 Professions: Medicine, Nursing and Physiotherapy
WHAT
HOW
HOW
MUCH
Phase 1
Phase 1
Identifying criteria indigenous to the workplace context
for judging the adequacy of clinical communication skills
Eliciting and analysing health professionals HP
judgements of actual clinical communication
What do (HPs) themselves value in communication with
patients?
Result: model of what HPs value (Pill 2013) – shows
strong differences from linguistic criteria
Translating this into assessment criteria to be used in
judging performances on the Occupational English Test
Model of what health professionals value in
patient communication (Pill 2012)
OET Speaking assessment criteria
Current
language-related
criteria
Proposed
interaction-related
criteria
IntelligibilityClinician
EngagementFluency
Appropriateness of
LanguageManagement of
InteractionResources of
Grammar and
Expression
Proposed new criteria
Clinician Engagement
This criterion refers to the ability to use language to demonstrate a professional
manner towards and awareness of the patient.
You should consider whether the candidate demonstrates:
• a suitably positive manner towards the patient (i.e., attentive, tolerant, respectful,
non-judgemental)
• awareness of the patient’s situation, narrative, opinions and expectations,
acknowledging the patient’s concerns, needs and emotions.
(4)• Professional manner generally strong and positive.
• Generally high level of awareness of the patient.
(3)• Professional manner sufficiently positive.
• Generally adequate level of awareness of the patient.
(etc.) …
18
Clinician Engagement
Management of Interaction
Phase 2: Can this model inform OET criteria?
What is at stake in the statistical analysisCan assessment of these two new
aspects be handled successfully by
language professionals?
How do the new criteria relate to the
existing criteria?
Are the new criteria measuring a new
aspect of the same construct or
introducing a different construct?
New criteria represent related but somewhat different
dimension of competence
The addition of two new criteria in a revised test would
represent more of what health professionals value in
clinical spoken communication with patients/relatives
Existing OET (language-focused) assessors can be
trained to apply these new criteria consistently (though
further training necessary)
Two options to report the speaking sub-test results:
• As two separate grades for the new and existing criteria;
• As a single grade for existing criteria supplemented by a
diagnostic report relating to the scores given on the new
criteria.
Summary of findings – Phases 1 & 2
Findings of Phase 3
Fewer test takers would pass using new criteria
But more of those would achieve an A (“strong”) grade
Cut-scores for nursing were a bit lower than for medicine
Moderate to good correlation between
– Health professionals’ (HPs’) holistic judgement
– OET assessors’ grading using six criteria
(2 interactional and 4 language-related)
The new criteria appear to capture what matters to HPs
Phase 2:
The two additional criteria allow claims that a revised
test would represent more of what health professionals
value
OET (language-focused) assessors can be trained to
apply these criteria consistently
Phase 3:
Implementation of new criteria would change current
outcomes for some candidates
Promises better decisions about who to admit, who not
to admit, to clinical practice
Summary and conclusion
Australian Research Council Linkage grant (LP130100171)
Ethics approval for study given by University of Melbourne
Languages & Linguistics Human Ethics Advisory Group
23
Toward improved quality of written patient
records: Language proficiency standards for
non-native English speaking health
professionals
Tim McNamara, Ute Knoch, Robyn Woodward-Kron, Catherine
Elder, Elizabeth Manias, Eleanor Flynn, Barbara Zhang
Sharon Yahalom, Simon Davidson, Annemiek Huisman
OET Writing sub-test
Test task: Letter of referral/discharge
Test content: Based on case notes
Assessment: Linguistic criteria applied by language professionals
Overall Task Fulfilment
Appropriateness of Language
Comprehension of Stimulus
Linguistic Features (Grammar & Cohesion)
Presentation Features (Spelling, Punctuation & Layout)
Overview of the project
Phase 1
• Interviews with stakeholders who contribute to or read medical records
Phase 2
• Establishing healthcare professional relevant criteria
Phase 3
• Applying the professionally relevant criteria to the OET writing task
Phase 4
• Setting minimum standards on the OET writing task
Phase 2
Conducted workshops in which doctors were asked to
comment on the strengths and weaknesses of a range of
handover documents
Participants: 61 health professionals (doctors, nurses,
health information managers)
18 experienced doctors from a range of sub-disciplines and
contexts
Average years of experience: 25 years
Average years as educators: 17 years
Stimulus material: 10 handover documents (referral letters
and discharge summaries) selected from 200 patient
medical records
Referral letter
Discharge summary
Sample medical record
Extraction of all referral letters and discharge summaries
Preselection of documents
Researcher meeting for final selection
Methodology
Data collection procedures
Workshops conducted in small groups or individually
Audio recorded
Data analysis
Audio recordings transcribed
Coded for salient themes and sub-themes
Methodology
Results
1. Appropriateness
Writer Text Reader
1. Appropriateness
Appropriateness of the writing to the text
type/ style of the document
Any clinician would be able to understand it but it
is not the sort of language you want in a
discharge summary (DW1, M2, D1,2,3)
It is almost like somebody has had a private
investigator following him but it doesn’t scream
of medical assessment
(DW1, M22, D4,5)
1. Appropriateness
Appropriateness of writing to audience
I mean give it a medical term because it is going to the
GP (DW1, M12, D1,2,3)
P: I also find it quite difficult sometimes when you are
referring to a consultant, I don’t like to be saying ‘Well
this is what I would like’ because sometimes
consultants don’t like being told what to do
P: Yeah, don’t tell me what to do, I will decide what to
do, so therefore often I will say, ‘I would be pleased
for your expert management’ or something like that.
(DW2, R13, D11,12)
1. Appropriateness
Appropriateness of the writing to the
sub-discipline of the writer
… when I went to psych that is what I was doing, I
was doing bullet points and they said, “No, you have
to write novels”. So then I started writing novels and
then coming back here, I am like I don’t have to write
novels anymore. (DW2, M25, D8,9,10)
2. Conciseness
• Conciseness:
So it is succinct which is also highly valued and as you get
more experienced you get more succinct.
(DW1, R66, D4,5)
2. Conciseness
This discharge summary falls into the trap of trying to be
succinct with bullet points but at the same time being very
verbose (DW2, M12, D8,9,10)
3. Sufficiency
Sufficiency:
You have got the information mostly that you need
here and so he/she has done that well (DW2, M20,
D11, 12)
P: Um, is there too much information sometimes?
P: No, I think as an initial letter because this is the first
time they have seen him in clinic, I think that is perfect.
P: and this would give me all the information I need
about his background (DW4, M25, D14)
4. Accuracy of content
P: when you get particular
things like antibiotics that
they would have prescribed
five years ago and it is still
listed on their medication
list
P: so I would find that a
little bit embarrassing to
send it. It reflects poorly on
you if you send off a letter
which is incorrect.
5. Clarity
Clarity of content
– Being explicit
– Making purpose of letter clear (in case of referrals)
– Making request or follow up plans clear (both referrals
and discharge summaries)
– Prioritizing information
It is not clear who is doing what or even what has
really happened. It is just a jumble of thoughts
really (DW1, R66, D1,2,3)
Towards the end I think when you really want to lay
it out, clarity is really crucial (DW1, M25, D1,2,3)
5. Clarity
Clarity of organisation & presentation
– Logical organisation of content
– Prioritization of information
– Use of paragraphing, dot points, spacing when
appropriate
– Legibility & general ‘tidiness’
The information is there and the information is good
but actually that still doesn’t do the job because of
how it is presented (DW2, R13, D8,9,10)
So I think it is very well structured in terms of you
could pick this up and get a good picture of the
patient and what they look like (DW4, M22, D14)
5. Clarity
Clarity of language
– Grammatical accuracy, spelling, sentence
structure were all mentioned
So the grammar and spelling and English is not 100%
but that is not an issue in terms of understanding what is
there (DW4, R36, D14)
You could ignore the sentence structure, provided what
was written down was comprehensive and useful (DW2,
R36, D8, 9, 10)
5. Clarity
Clarity of language
– Use of acronyms
– Use of technical language
P: I don’t like acronyms, I don’t know what BRAF is?
P: no
P: there is a reference to BRAF, which is obviously an abbreviation
but within the oncology field (DW2_M25_D11, D12)
P: But it doesn’t read very easily because there is a lot of acronyms,
there is a lot of shortcuts (DW2_R13_D8, 9, 10)
Discussion – what doctors value in handover
documentation
AppropriatenessWriter
Text
Reader
Discussion – what doctors value in handover
documentation
Conciseness
&
Sufficiency
Appropriateness
All key information
present
No irrelevant
information
Discussion – what doctors value in handover
documentation
ClarityConciseness
&
Sufficiency
Appropriateness
content
organisation
presentation
language
Discussion – what doctors value in handover
documentation
ClarityConciseness
&
Sufficiency
Appropriateness
Accu
racy o
f conte
nt
Discussion
Values of domain experts cannot always be directly
translated do not all transfer into the (more limited)
language testing domain
Many of the aspects mentioned by the domain experts are
represented in the current criteria
Some aspects not present
Grouping might have to be re-examined
Weighting will have to be re-examined
Next steps:
Examining how well descriptors are reflected in our empirical data
Develop checklist indicators
Apply checklist indicators to OET writing data
Strong and weak performance tests
Two types of test:
‘in the strong sense, tasks will represent real-world tasks, and
performance will primarily be judged on real-world criteria, that
is, the fulfilment of the task set. Aspects of language ability as
such may or may not be assessed at all’
‘in the weak sense, the focus is on language performance.
Task may resemble or simulate real-world tasks … however,
the capacity to perform the task is not actually the focus of
assessment. Rather, the purpose … is to elicit a language
sample so that second language proficiency … may be
assessed.’
McNamara, T. F. (1996). Measuring second language performance. Harlow: Longman. (pp. 43-44)
[The body] responsible for the OET is required by law to
assess the English language proficiency of overseas-trained
health professionals, but is not permitted to make
assessments of their professional competence (McNamara,
1996, p. 40)
(Impossible) separation of communicative skill and
professional competence
Assessment of communicative skill in hands of language
professionals (language teachers, language testing experts)
But are language assessors and clinicians oriented to the
same qualities in communication in interactions between
clinicians and patients?
Construct issue: Divided responsibility for
assessment
© Copyright The University of Melbourne 2011
Plenary 2
Tim McNamaraSchool of Languages and Linguistics, University of Melbourne