Research Methodology Workshop 22 May 2013Hampshire Teaching School Alliances

Wan Ching Yee

Questionnaire design & analysis

The research question frames the rest

• … the concepts you are researching

• … the relevant research literature

• … design considerations

• … samples

• … methodologies

• … the conclusions that you can draw!

Types of survey

• Descriptive – produce relevant data

• Analytic – examine relationships

• Exploratory – patterns explored without prior theory

• Confirmatory – model or causal relationships are

tested

What are the principles of good survey design?

• Realistic and feasible

• KISS

• Select only those variables that are most likely to

affect outcomes

• Minimise cost and effort to respondents

• Credibility is a factor

5

Presentation & Layout:

• Clear presentation e.g use of fonts

• Open questions used as little as possible

• Clear instructions about how to respond e.g. give

examples

• Vertical or horizontal closed answers?

• Keep question and answers together

6

Why use questionnaires?

• Cheap to administer

• Quick to administer

• Absence of interviewer effects

• No interviewer variability

• Convenience for respondents

7

Disadvantages of questionnaires

• Cannot prompt• Cannot probe• Cannot ask too many questions that are not salient

to respondents• Do not know who answers• Not appropriate for some kinds of respondents• Greater risk of missing data• Lower response rates

8

Improving response rates

• Good (personalising?) covering letter • Accompanied by SAE if postal questionnaire• Follow up non-respondents• Shorter questionnaires• Clear instructions and attractive layout• Avoid ‘bulky’ questionnaire • Begin with questions which are likely to be of

interest to the respondent• Incentives

9

Types of questions

• Personal factual questions

• Factual questions about others

• Questions about attitudes

• Questions about beliefs and values

• Questions about knowledge

• Branching question for more detailed questions

10

General rules for designing questions

• Always bear in mind your research questions

(these are at a different level to your questions on

your questionnaire!)

• What do you want to know?

• How would you answer it?

• Aim for clarity, simplicity & neutrality

11

Improve the quality of your questions

Try to avoid:• Long complex questions• Double negatives• Jargon• Abbreviations• Culture specific terms• Words with double meaning• Leading questions• Emotionally loaded words

12

Improve the quality of your questions

May want to consider:

Including a response category:

• Don't know• Not applicable• Do not want to answer

13

Data gathering

Question types: Closed and open ended text or open-ended numeric

• Closed: Yes/No, Male/Female+ Easy to convert to numerical format if using

statistical packages- Limited choices• Open ended: What is your major source of

anxiety at the moment?+ Respondents have more freedom to respond in

their own way - Responses need to be summarised (more time)

Young respondents

14

15

Formatting

• Group together questions on major subject areas

• Begin with emotionally neutral questions which

ideally, are easy and pleasant to answer

• More sensitive or difficult questions in the middle

• Personal characteristics eg income, at the end

16

Piloting and pre-testing questions

• Open questions used to generate fixed-choice

answers

• Identify questions that don’t work e.g skipped,

unclear

• Identify clarity of instructions

• Use questions employed by other researchers?

• http://www.natcen.ac.uk

• http://surveynet.ac.uk/sqb/surveys/introduction.asp

17

Steps in questionnaire design

• Make a list of variables• Assemble file of questions or instruments for

measuring each variable e.g available in books, on web

• Compose a draft• Shorten the set of instruments• Pretest• Validate• Consider your sample – how will you access them?

And how will you recruit them?

18

Different types of dataNominal or categorical

Examples Sex, preference type of coffee, colour

Characteristics No sense of orderCan be stored as a word, text or numerical code

Summarise Frequency or percentage

Cannot use Mean or average value

Display Pie chart Column chartBar chart

19

Different types of dataOrdinal

Examples Satisfaction, rank, fanciness

Characteristics Have a meaningful order but the intervals between the values in the scale may not be equal

Summarise Frequency or percentage

Cannot use Mean or average value?

Display NOT as a pie chart Column chartBar chart

20

Different types of data

Interval/Ratio Examples Number of customers, weight, age, size

Characteristics Includes things that can be measured rather than classified or orderedData can be discrete whole numbers (5 customers) or continuous with fractional numbers (1.2 miles)

Summarise MeanMedianStandard deviation

Display HistogramBar chartLine chart (data that occur over time)

21

Data gathering and processing

• Data gathering – Quantification/measurement

• Data modification –grouping/categorising, ordering, taxonomies

Displaying – looking for trendsComparing – with each other, with the

populationRelationships between variablesUse of statistics

22

Grouping

Define and label variables (sex, age)

Assign numbers to each of the possible responses

23

Grouping

Sex 1=Males2=Females

Age 1= 21-25 years2= 26-30 years3= 31-35 years4= above 36 years5= No answer given

Optimising scale (items 1-4) 1 (strongly agree)2 (agree)3 (disagree)4 (strongly disagree)

Closed questions

Grouping Open ended questions

What is your major source of anxiety at the moment?

Scan through questionnaires and try to find common themes.

Anxiety 1=Grade2=Pass the course3=Lack of time

25

Displaying data

• Basic diagrams can be created with word processing programmes such as Windows

• Most useful diagrams are typically tables, pie charts, bar charts, trend lines

• Diagrams are best if simple • Diagrams do not tell you about the importance of the

size of the quantities involved

9

27

Student numbers 2007: ft/pt, European+Home/International

Inft Inpt EHft EHptCounselling 14 0 6 0EDLP 17 0 7 9Psych 14 0 17 23Special Ed 1 0 7 9

28

MEd and MSc Pathways 2007

0

10

20

30

40

50

60

Coun EDLP ICP Math Psyc ResM SpEd ETS SURE TESOL

O/seas ft O/seas pt EH ft EH pt

29

30

Basic Techniques

• Identification of variablesDependent/IndependentDiscrete/Continuous

• Sampling• Piloting

ReliabilityValidity

31

Analysis

• Hypotheses may require comparisons, e.g. difference between groups, difference between treatments, relation between scores

• There is always some variation• Possibility of measurement error, sample error, etc• Hence we use statistics for an assessment of how

likely it is that the difference/relation found is not worth thinking about

32

Probability

• Statistical analyses assess probability that found result is unlikely to be result of random variation in population

• Convention is that result needs to be such that it would not occur by chance more than 1 time in 20 (5%) or 1 time in 100 (1%) - or even less likely – before we work on interpreting it

33

Enhancing chances of getting Significance

• Carefully define your prediction (effectively bet on outcome, don’t just look for difference say which direction the difference should be)

• Use good reliable measures• Select participants properly• Use a large enough sample to be able to detect the

result you predict

Enhancing chances of making a real significant discovery

Never ever ‘cherry-pick’ the data that suit you

Never ever ‘cherry-pick’ the results that suit you

Replicate, replicate, replicate

Test, test, test

35

The core point

Scientific research needs to be seen for what it truly is; a way of preventing me from deceiving myself in regard to my creatively formed subjective hunches which have developed out of the relationship between me and my material.

Carl Rogers 1961

Paired task

Considering the examples of “what not to do” from the

start of the session:

• Using the research question on your proforma,

start to construct a questionnaire to help you

answer this question. Think about the different

types of questions you could use, and how best to

phrase them.

• Pilot your questionnaire with another pair. What

improvements could you make?

Useful references

• Dillman, D.A. (1978) Mail and telephone surveys: The total

design method. New York: John Wiley and Sons.

• Oppenheim, A.N. (1992) Questionnaire design, interviewing

and attitude measurement. London: Pinter

• Sheatsley, P.B. (1983) Questionnaire construction and item

writing. In Ross, P.H., Wright, J.D. and Anderson, A.B.(Eds.),

Handbook of Survey Research. New York: Academic Press.

• Sudman, S. and Bradburn, N.M. (1989) Asking questions: A

practical guide to questionnaire design. San Francisco:

Jossey Bass.