Research Methods: Design and Analysis

Research Methods: Design and Analysis

by Dr. James Lani

Today’s Webinar

To schedule a consultation call 877-437-8622 or email [email protected]

• Why Do Research?• The Research Cycle• Research Questions and (Testable) Hypotheses

• Operationalization of Variables• Literature Review Resources• Research Designs• Sampling Method and Sample Assignment• Instrumentation• Method of Data Collection (Procedures)• Validity

• Internal Validity• External Validity

• Reliability• Data Analysis Plan• Sample Size• Q & A

Why Do Research?


OK, to get out of school….and also to examine questions we care about. We want to test a theory, evaluate a program or intervention, describe a phenomenon.

Does Violence on TV cause children to be more or less aggressive?Do opposites attract or birds of a feather flock together?Does treatment X predict academic achievement?Is there a difference on depression scores by group (exercise vs. not)?

What is True???Goal of research: describe, explain, and predict. To establish knowledge, you need methods and statistics.

The Research Cycle


LiteratureReview

MethodHypotheses and

Operational Variables

MethodDesign Study and

Collect Data

ResultsAnalyze data and

accept or reject Ho’s

Discussion

Research Questions and (Testable) Hypotheses


• Hypotheses: State relationships among variables (constructs). Typically in terms of Differences, Relationships, or Prediction.

• Keep research questions and hypotheses aligned (in number and language).

• Questions must be testable.• Keep it tentative: no proof—just support

or non-support

Operationalization of Variables

(or how do you measure what you’re studying?)


Constructs—objects that are not directly observable• Constructs are measured with variables• Should be quantifiable • (e.g., depression can be measured with a Beck

depression scale or by the observation of the number of times someone laughs)

Confounding Variables• Provide alternative explanation of response in the DV

Levels of Measurement: Nominal, Ordinal, Interval, and Ratio

Literature ReviewResources


See: UW—Madison—the writing centerUC Santa Cruz—UNW library—write a literature reviewWashington and Lee University—literature reviewStatistics Solutions’ Free membership—sample dissertation

Research Designs


EXPLORATORY

EXPERIMENTALDescriptive

Longitudinal

CAUSAL Cross-sectionalCorrelational Cohort

designs

Non-experimental

Sequential

Research Designs


Its about how you structure your inquiry

Research Designs


Experimental vs. Non-experimental Designs

Non-Experimental DesignsCorrelational studies, observational

studies, quasi-experimental studies, and surveys

Experimental (or True) DesignsManipulate the IV(e.g., Control vs. Experimental groups)Random assignmentConfounds are controlled or eliminated

Research Designs


Experimental Designs

One shot case studies X O

One group pretest posttest O1 X O2

Statistic group X O1 O2 R=random assignment, O=observation, X=treatment group

Research Designs


True Experimental DesignsPretest-posttest control group design

R O1 X O2

R O3 -- O4

R=random assignment, O=observation, X=treatment groupGroup Time 1 Time 2 Time3

Random Treatment Mean X Mean

Assignment Control Mean Mean

No diff Sig diff

Pretest Posttest0

5

10

ControlTreatment

Research Designs


For great research design references, see:

Cook and Campbell ‘79 Campbell and Stanley ‘63 Creswell ‘05 Leedy and Ormrod ‘10 SocialResearchMethods.net

A word about Sampling Method and Sample Assignment


Random sample: each individual in the population has an equal chance to become a participant of the sample.

Random assignment: each participant of the sample has an equal chance of being assigned to a group (treatment or control).

Instrumentation


These are the tools that measure the outcomes—typically variables in the tools are categorized into independent variables (IV’s) and dependent variables (DV’s).

IV’s (or treatment variables) are typically manipulated by researcher.

DV’s (also called outcome variables or criterion variables) are responses to IV, caused by IV, or predicted by IV.

The variables should be reliable and valid.

Don’t make your own.

See Google scholar, review articles, Statistics Solutions directory.

Instrumentation


Questionnaires Interviews Observations Rating scales Achievement tests Personality

inventories Historical reports or

data

Method of Data Collection (Procedures)


After describing the sample and the instruments, tell the reader how you are going to systematically and objectively administer these instruments to these participants.

ParticipantsMeasures

Validity


You measure what you say you are measuring.

(Validity is more important than reliability)

Internal Validity


Eliminate confounds or competing explanations of differences on DV (E.g., Differences on Self-efficacy by gender…but the woman were 35 y.o. and men were 75 y.o.)

Threats to internal validity• Selection bias—you think they represent the population (Safeguard tip: randomly sample and randomly assign)• Pre-post studies without a control group

o History/maturation—something else occurred between the assessments

• Drop-out/attrition • Reliability of measure• Low power• Order effect• Multiple tests of significance • John Henry Effect

External Validity


Def: The ability to generalize outcomes to the population

Threats• Population validity—ability to generalize to

population. (E.g., Kohlberg moral development—males value individual rights, where Gilligan found that females valued relationships).

• Ecological validity—ability to generalize from lab to everyday environments.

Reliability


Inter-rater reliability (kappa coefficient)

Test-retest reliability Internal consistency—

Cronbach’s alpha

Data Analysis Plan(what to include)


Reliability of scales: Cronbach’s alpha

Descriptive statistics: Means, standard deviations, frequency and percentages.

A data analysis plan for EACH research hypothesis.

• Inferential: ANOVA, MANOVA, regression, time-series, SEM, Chi-square, correlation

• Non-inferential: descriptives (e.g., M, SD, %)

Assumptions of analyses and justification for analyses

Sample Size


N = ?

1. Effect Size—importance of differences

2. Power—likelihood of finding differences (typically .80)

3. Alpha (typically .05)

1-1 Consulting


[email protected]

877-437-8622

mailto:[email protected]

We’re interested… Webinar Feedback and

What You’d LikePlease provide feedback on the webinar and tell us what

else you’d like to learn about in upcoming webinars.

Send comments to: [email protected]

mailto:[email protected]

Thank You for your Participation

and Attention!

Join us for our next webinar on Wednesday, November 20th at

8:30pm ETResults Chapter:

Conducting, Interpreting, and Writing

Research Methods: Design and Analysis

Education

Transcript of Research Methods: Design and Analysis