Research Design

Gaining deeper insight

Colleen RollerAuthor & Practitioner

Areas of research

Behavioral decision theory The study of how people judge, decide and behave

Cognitive psychology The study of how people perceive, think, remember

and learn Social psychology

The study of how people are influenced and affected by other people

No neutral design

Preferences, judgments, and decisions are remarkably sensitive to the context in which they are made.

No neutral design

No neutral design

No neutral design

This presentation

Uses research findings to inform research design

Clever Hans

The power of the subconscious

Rosenthal/Jacobsen 1968

‘When we expect certain behaviors of others, we are likely to act in ways that make the expected behavior more likely to occur.’

- Rosenthal and Babad

Confirmation bias

The tendency to seek out and assign more weight to evidence that confirms one’s hypothesis, and ignore or underweight evidence that could disconfirm it.

So what?

We may be influencing participants – and research findings – in ways we’re not even aware of.

FRAMING

What is framing?

Framing

‘I’ve got great news. You managed to avoid a salary decrease.’

Frame the study

The purpose of this study is to test the design, not you.

Child custody

Parent AAverage incomeAverage healthAverage working hoursReasonable rapport with the childRelatively stable social life

Shafir, Simonson, Tversky, 1993

Which parent should get sole custody?

Award – 64%

Parent BAbove-average incomeMinor health problemsLots of work-related travelVery close relationship with the childExtremely active social life

Child custody

Parent AAverage incomeAverage healthAverage working hoursReasonable rapport with the childRelatively stable social life

Shafir, Simonson, Tversky, 1993

Which parent should not get sole custody?

Award – 64%Deny – 55%

Parent BAbove-average incomeMinor health problemsLots of work-related travelVery close relationship with the childExtremely active social life

Positive/negative

What did you like/dislike… How useful, or not,… How confident, or not… Why would you continue/stop using…

How we ask the question

Positive/negative Breadth/depth – broad/narrow

Did you notice anything different about this process…

How does this process compare to how you currently do this?

What information do you need in order to feel confident about doing x…

What would make you feel confident about doing x…

How often do you use the x website…

Tell me about how and when you use the x website…

Counterintuitive frame

What do you think of the brand?

What does the brand think of you?

‘A small question can never hold a large answer.’

- Gerald Zaltman (How Customers Think)

Assumptions

Be careful that your assumptions aren’t built into the question

Turn your assumptions into questions

Rating scales

How successful have you been in life, so far?

0 1 2 3 4 5 6 7 8 9 10Not satisfied Very satisfied

-5 -4 -3 -2 -1 0 1 2 3 4 5Not satisfied Very satisfied

34%

13%

Schwartz et al, 1991

Scholarships

Two scholarships are available. You prefer a scholarship with less waiting

time and more money. Which one would you pick?

Sun, Li, Bonini 2010

Graphical scale

Figure 1: wait time salient – preferred A Figure 2: $ amount salient – preferred B

Sun, Li, Bonini 2010

Graphical comparison

Toothpaste B

Toothpaste A

0 5 10 15 20 25 30 35

People (per 5000) with serious gum disease

E.R. Stone et al. 2003

Graphical comparison

Toothpaste A30 out of 5000

Toothpaste B15 out of 5000

E.R. Stone et al. 2003

So what?

How does the frame itself influence the meaning that people take away?

ORDERING OF QUESTIONS

Survey design

Please rate your overall satisfaction.

How many times have you had repairs?

German students

How happy are you these days?

How many dates did you have last month?

Strack et al, 1988

Reverse order

How happy are you these days? How many dates did you have last month?

How many dates did you have last month? How happy are you these days?

Strack et al, 1988

‘The sequential arrangement of information often creates the very meaning of that information.

The order of presentation determines the reaction.’

- Dr. Frank Luntz (Words That Work)

So what?

In what ways are you creating (unintended) meaning?

What are you measuring?

THE POWER OF WORDS

Different meanings

Gambling v. gaming Liberal v. progressive Liquor v. spirits Used v. pre-owned Etc.

An auto accident

At what speed did car A contact car B?

At what speed did car A smash into car B?

Loftus/Palmer 1974

31.8 mph

40.5 mph

Actual: 12 mph

Power of words

Make contact Smash into Crash into Bump into Run into Collide with

So what?

How are your words impacting user perception – and consequently, research

findings? how your business partners perceive the

findings?

Delivering findings

Users were… Aggravated… Annoyed… Confused… Disappointed… Frustrated… Irritated… Surprised…

Neutral

Users… Said… Expected… Thought… Anticipated…

IN SUMMARY

Research design

‘The function of a research design is to ensure that the evidence obtained enables us to answer the initial question as unambiguously as possible.’

The design matters

There is no neutral design

‘All research methods involve compromises with reality.’

- Gerald Zaltman (How Customers Think)

The solution

Be aware Control what we can

How we ask questions Ordering of questions Choice of words Assumptions and their effect Use of ‘scale’ – charts, graphs, rating scales Comparison – graphs, ratings

UXmatters.com UX magazine (uxmag.com)

Colleen Roller