Rss Oct 2011 Mixed Modes Pres5
-
Upload
gerrynicolaas -
Category
Documents
-
view
91 -
download
2
description
Transcript of Rss Oct 2011 Mixed Modes Pres5
![Page 1: Rss Oct 2011 Mixed Modes Pres5](https://reader033.fdocument.pub/reader033/viewer/2022042817/559dcdc21a28ab69368b47a9/html5/thumbnails/1.jpg)
Is it a good idea to optimise question format for mode of data collection?
Gerry Nicolaas National Centre for Social ResearchPamela Campanelli Independent Survey Methods ConsultantSteven Hope University College LondonAnnette Jäckle University of EssexPeter Lynn University of Essex
![Page 2: Rss Oct 2011 Mixed Modes Pres5](https://reader033.fdocument.pub/reader033/viewer/2022042817/559dcdc21a28ab69368b47a9/html5/thumbnails/2.jpg)
Overview
• Background
• Research question• ‘Mark all that apply’ vs ‘yes/no for each item’
• Literature, Hypotheses, Methods, Results
• Branched questions vs non-branching• Literature, Hypotheses, Methods, Results
• Cognitive interview results• Methods, Results
• Conclusion
![Page 3: Rss Oct 2011 Mixed Modes Pres5](https://reader033.fdocument.pub/reader033/viewer/2022042817/559dcdc21a28ab69368b47a9/html5/thumbnails/3.jpg)
Background
• Face-to-face interviewing is dominant mode in UK• Interviewer presence• Aural stimuli & Visual stimuli (e.g. show cards)
• Many questions designed to be ‘optimal’ for face-to-face interviews• abundant use of show cards, long response lists, ‘mark all
that apply’, fully labelled scales, etc
• Push towards cheaper data collection modes and mixed modes• Some questions used in face-to-face interviews need to be
adapted for other modes
![Page 4: Rss Oct 2011 Mixed Modes Pres5](https://reader033.fdocument.pub/reader033/viewer/2022042817/559dcdc21a28ab69368b47a9/html5/thumbnails/4.jpg)
Research question
• Is it a good idea to ‘optimise’ question format for mode of data collection? • Increasing risk of differences in measurement across modes?
• 2 question formats that tend to be changed for telephone interviews• Mark all that apply • Scales
![Page 5: Rss Oct 2011 Mixed Modes Pres5](https://reader033.fdocument.pub/reader033/viewer/2022042817/559dcdc21a28ab69368b47a9/html5/thumbnails/5.jpg)
‘Mark all that apply’versus
‘yes/no for each item’
![Page 6: Rss Oct 2011 Mixed Modes Pres5](https://reader033.fdocument.pub/reader033/viewer/2022042817/559dcdc21a28ab69368b47a9/html5/thumbnails/6.jpg)
Example of ‘mark all that apply’
F2F & Self-CompletionThis card shows a number of different ways for reducing poverty. In your opinion, which of the following would be effective in reducing poverty?
SHOW CARD / MARK ALL o Increasing pensionso Investing in education for childreno Improving access to childcare o Redistribution of wealtho Increasing trade union rightso Reducing discriminationo Increasing income supporto Investing in job creation o None of these
Telephone interview
o Yes o No
Would investing in education for children reduce poverty?
Would increasing pensions reduce poverty?
o Yes o No
etc……….
I am now going to ask you a number of questions about different methods for reducing poverty. In your opinion, which of the following would be effective?
![Page 7: Rss Oct 2011 Mixed Modes Pres5](https://reader033.fdocument.pub/reader033/viewer/2022042817/559dcdc21a28ab69368b47a9/html5/thumbnails/7.jpg)
‘mark all’ vs ‘yes/no for each’
• Difficult to interpret absence of being marked with ‘mark all’• Does not apply or item non-response?• Sudman & Bradburn (1982)
• Deeper processing of each item with ‘yes/no’• More items selected with ‘yes/no’• Respondents take more time to complete ‘yes/no’• Primacy effects with ‘mark all’• Rasinski et al (1994), Smyth et al (2006), Smyth et al
(2008)
• ‘yes/no’ performs similarly in tel & web modes• Smyth et al (2008)
![Page 8: Rss Oct 2011 Mixed Modes Pres5](https://reader033.fdocument.pub/reader033/viewer/2022042817/559dcdc21a28ab69368b47a9/html5/thumbnails/8.jpg)
Our research
• Replicate & extend Smyth et al (2006 & 2008)• Face-to-face mode as well as telephone and web• UK general population with internet access• Cognitive interviews as well as experimental data
![Page 9: Rss Oct 2011 Mixed Modes Pres5](https://reader033.fdocument.pub/reader033/viewer/2022042817/559dcdc21a28ab69368b47a9/html5/thumbnails/9.jpg)
Hypotheses
• Higher % of items chosen in tel mode with ‘y/n’than in F2F & web with ‘mark all’
• If using ‘y/n’ in all 3 modes, no differences • If using ‘mark all’ in F2F & web, no differences• Deeper processing with ‘y/n’
• ‘y/n’ takes longer to complete than ‘mark all’• more primacy with ‘mark all’ respondents with below
average completion times• ‘mark all’ respondents with at least average completion
times select as many items as ‘y/n’ respondents
• Differences greater for difficult questions
![Page 10: Rss Oct 2011 Mixed Modes Pres5](https://reader033.fdocument.pub/reader033/viewer/2022042817/559dcdc21a28ab69368b47a9/html5/thumbnails/10.jpg)
Methods
• 2 split ballot experiments• Easy attitude question about neighbourhood with 8 items• Difficult attitude question about poverty with 8 items
• Mean number of endorsements for each format in each mode
• Response latencies for each format in each mode
• OLS regression
• Control variables for differential non-response
![Page 11: Rss Oct 2011 Mixed Modes Pres5](https://reader033.fdocument.pub/reader033/viewer/2022042817/559dcdc21a28ab69368b47a9/html5/thumbnails/11.jpg)
2 split ballot experiments
(1) Neighbourhood
What are the things that you like about your neighbourhood?
SHOW CARDo Its community spirito It feels safeo The neighbourso The character of its buildingso Its cleanlinesso Its locationo Its quietnesso Its transport facilities
(2) Reducing povertyThis card shows a number of different ways for reducing poverty. In your opinion, which of the following would be effective in reducing poverty?
SHOWCARD o Increasing pensionso Investing in education for childreno Improving access to childcare o Redistribution of wealtho Increasing trade union rightso Reducing discriminationo Increasing income supporto Investing in job creation o None of these
![Page 12: Rss Oct 2011 Mixed Modes Pres5](https://reader033.fdocument.pub/reader033/viewer/2022042817/559dcdc21a28ab69368b47a9/html5/thumbnails/12.jpg)
Results
• Hypotheses supported:• higher % of items selected in tel with ‘y/n’ than F2F &
web with ‘mark all’• ‘y/n’ takes longer to complete than ‘mark all’• no differences when using ‘mark all’ in F2F & web
• But:• when using ‘y/n’ in all modes, differences observed
between tel (& F2F) and web• differences were greater for the difficult question• no primacy among ‘mark all’ respondents with less than
average completion times• ‘mark all’ respondents with at least average completion
times still selected fewer items
![Page 13: Rss Oct 2011 Mixed Modes Pres5](https://reader033.fdocument.pub/reader033/viewer/2022042817/559dcdc21a28ab69368b47a9/html5/thumbnails/13.jpg)
Differences with Smyth et al (2006 & 2008)
• Differences in study population• Smyth et al: students at USA university• This study: general population who used internet
• Differences in question design• Smyth et al: 9-15 items • This study: 8 items for both questions
![Page 14: Rss Oct 2011 Mixed Modes Pres5](https://reader033.fdocument.pub/reader033/viewer/2022042817/559dcdc21a28ab69368b47a9/html5/thumbnails/14.jpg)
Full response list in 1 step versus
Branching into 2 steps
![Page 15: Rss Oct 2011 Mixed Modes Pres5](https://reader033.fdocument.pub/reader033/viewer/2022042817/559dcdc21a28ab69368b47a9/html5/thumbnails/15.jpg)
Example of scalar question
Fully labelled scale
Please tell me whether you consider your local shopping facilities to be …….o extremely poor,o very poor,o poor,o good,o very good, oro extremely good?
Branched question
Please tell me whether you consider your local shopping facilities to be …….o poor oro good?
Would this be …….o (poor/good), o very (poor/good), oro extremely (poor/good)?
![Page 16: Rss Oct 2011 Mixed Modes Pres5](https://reader033.fdocument.pub/reader033/viewer/2022042817/559dcdc21a28ab69368b47a9/html5/thumbnails/16.jpg)
Full scale vs branching
• Branching makes the task easier for respondents• Decomposition principle• Armstrong et al (1975), Groves & Kahn (1979), Krosnick &
Berent (1993), Yu et al (2003), Malhotra et al (2008)
• Branching produces more extreme responses• Reduces reluctance to select extreme responses• Nicolaas et al (2000), Yu et al (2003)
![Page 17: Rss Oct 2011 Mixed Modes Pres5](https://reader033.fdocument.pub/reader033/viewer/2022042817/559dcdc21a28ab69368b47a9/html5/thumbnails/17.jpg)
Hypotheses
• More extreme responses in tel mode with branching than in F2F & web with no branching• This effect is more prevalent for attitudinal than factual
questions
• Within each mode, more extreme responses with branching• This effect is more prevalent for attitudinal than factual
questions
• If using branching in all 3 modes, no differences
![Page 18: Rss Oct 2011 Mixed Modes Pres5](https://reader033.fdocument.pub/reader033/viewer/2022042817/559dcdc21a28ab69368b47a9/html5/thumbnails/18.jpg)
Methods
• 4 split ballot experiments• Easy attitude question about shopping facilities• Difficult attitude question about standard of living• Easy factual question about rent/mortgage• Difficult factual question about grocery shopping
• Proportion of extreme & non-extreme responses• Highest versus the lowest category• Two highest versus the two lowest categories
• Logistic regression
• Control variables for differential non-response
![Page 19: Rss Oct 2011 Mixed Modes Pres5](https://reader033.fdocument.pub/reader033/viewer/2022042817/559dcdc21a28ab69368b47a9/html5/thumbnails/19.jpg)
4 split ballot experiments
(1) Please tell me whether you consider your local shopping facilities to be ...
READ OUT oextremely poor,o very poor,o poor,o good,o very good or o extremely good?
(2) Thinking back to the last general election, would you say that the standard of living has ...
SHOW CARD oincreased by a large amount,o increased by a medium amount,o increased by a small amount,o stayed the same,o decreased by a small amount,o decreased by a medium amount, oro decreased by a large amount?
![Page 20: Rss Oct 2011 Mixed Modes Pres5](https://reader033.fdocument.pub/reader033/viewer/2022042817/559dcdc21a28ab69368b47a9/html5/thumbnails/20.jpg)
4 split ballot experiments (continued)
(3) How much did your household spend last month in rent or mortgage for the accommodation you live in?
READ OUT oless than £100o £100 - £199o £200 - £299o £300 - £399o £400 - £499o or £500 or more?
(4) How much did your household spend last month on grocery shopping?
READ OUTo less than £100o £100 - £199o £200 - £299o £300 - £399o £400 - £499o or £500 or more?
![Page 21: Rss Oct 2011 Mixed Modes Pres5](https://reader033.fdocument.pub/reader033/viewer/2022042817/559dcdc21a28ab69368b47a9/html5/thumbnails/21.jpg)
Results
• Hypotheses supported or partly supported:• more extreme responses in tel with branching than F2F
& web with no branching• difference was greater for attitudinal than factual
questions• within each mode, branching produced more extreme
responses but only for attitudinal questions
• Hypotheses not supported:• within each mode no branching seemed to produce
more extreme responses than branching for factual questions
• many differences observed across modes when using branched format in all 3 modes
![Page 22: Rss Oct 2011 Mixed Modes Pres5](https://reader033.fdocument.pub/reader033/viewer/2022042817/559dcdc21a28ab69368b47a9/html5/thumbnails/22.jpg)
Cognitive interviews
![Page 23: Rss Oct 2011 Mixed Modes Pres5](https://reader033.fdocument.pub/reader033/viewer/2022042817/559dcdc21a28ab69368b47a9/html5/thumbnails/23.jpg)
Mode effects with Y/N (CAPI vs CAWI) (1)
Subtleties going on that could affect aggregatemode comparisons. 1. Instances of clear and possible satisficing
• Most of these were in the ‘yes’ category• Most of these were in CAWI
2. Instances of Rs in the middle ground between ‘yes’ and ‘no’ (e.g., qualified their answer or said it depends)• Almost all chose ‘yes’ rather than ‘no’• More in CAPI than CAWI
![Page 24: Rss Oct 2011 Mixed Modes Pres5](https://reader033.fdocument.pub/reader033/viewer/2022042817/559dcdc21a28ab69368b47a9/html5/thumbnails/24.jpg)
Mode effects with Y/N (CAPI vs CAWI) (2)
3. Of 6 questions, 2 were more sensitive (increasing income support benefit and redistribution of wealth), • slightly more ‘yes’ answers in CAPI than in CAWI
Finding (1) –• Opposite to quant findings, but not surprising - more
satisificing in CAWI on other hypotheses
Findings (2) and (3) –• In the direction of the quantitative findings
Resultant quantitative effects would depend on prevalence of respondents in these different categories
![Page 25: Rss Oct 2011 Mixed Modes Pres5](https://reader033.fdocument.pub/reader033/viewer/2022042817/559dcdc21a28ab69368b47a9/html5/thumbnails/25.jpg)
Branching versus No Branching (1)
• Needed direct evidence rather than aggregate evidence
• 12 Rs asked the 2 attitude questions in branched format as part of the survey questions
• Much later in cognitive interviews, R given a showcard with ‘non-branched’ format of same question
• Interviewer was to probe for any inconsistencies
![Page 26: Rss Oct 2011 Mixed Modes Pres5](https://reader033.fdocument.pub/reader033/viewer/2022042817/559dcdc21a28ab69368b47a9/html5/thumbnails/26.jpg)
Branching versus No Branching (2)
• Some Rs were inconsistent• 2 themes:
• Vagueness of the answer categories• EXAMPLE: “To me, extremely good and
very good are the same thing”• Confusion in scope of question
• EXAMPLE: Were ‘local shopping facilities’for food shopping or clothes shopping?
• Although not explaining the branching effect, per se, findings suggest how variable respondents’answers can be.
![Page 27: Rss Oct 2011 Mixed Modes Pres5](https://reader033.fdocument.pub/reader033/viewer/2022042817/559dcdc21a28ab69368b47a9/html5/thumbnails/27.jpg)
Conclusions
![Page 28: Rss Oct 2011 Mixed Modes Pres5](https://reader033.fdocument.pub/reader033/viewer/2022042817/559dcdc21a28ab69368b47a9/html5/thumbnails/28.jpg)
Conclusions
• ‘Y/N for each’ and branching formats are not necessarily functionally equivalent across all three modes
• Optimising question format for each mode should not be done without empirical evidence that this will produce comparable data across modes
![Page 29: Rss Oct 2011 Mixed Modes Pres5](https://reader033.fdocument.pub/reader033/viewer/2022042817/559dcdc21a28ab69368b47a9/html5/thumbnails/29.jpg)
Thank you
www.natcen.ac.uk