Lecture 1 Educational Research Design

Fang Chen East China Normal University

PART I: INTRODUCTION

2015/10/6 陈芳 华东师大英语系 2

1. Empirical 2. Systematic 3. Valid 4. Reliable 5. Variety of forms

General characteristics of educational research

陈芳

华东师大英语系

2015/10/6 3

Figure 1.2: Validity vs. reliability

2015/10/6 陈芳 华东师大英语系 4

• Classification of educational research – Basic vs. applied research

• Extension of knowledge vs. immediate practical problem

– Qualitative vs. quantitative • Words vs. numbers

– General methods of research

5. Variety of forms

2015/10/6 陈芳 华东师大英语系 5

• Theory/hypothesis – 理论/假设 – Substantive and statistical hypothesis

• Variables – Independent variable 自变量 – Dependent variable 因变量

• Research design – 研究设计

• Literature review guides you on all the above-mentioned points.

Some Terminologies

2015/10/6 陈芳 华东师大英语系 6

PART II:RESEARCH METHODS

2015/10/6 陈芳 华东师大英语系 7

1. Experimental research design 2. Quasi-experimental research design 3. Survey methodology 4. Historical research 5. Ethnographic research 6. Delphi method

2015/10/6 陈芳 华东师大英语系 8

① Posttest-Only Control Group Design: – 后测控制组

② Pretest-Posttest Control Group Design： – 前测后测控制组

③ Solomon Four-Group Design： – 所罗门四组

④ Factorial Designs： – 多因素设计

⑤ Repeated Measures Design： – 重复测量

⑥ Time-series Design： – 时间序列

1. Experimental research design

2015/10/6 陈芳 华东师大英语系 9

• We cannot control anything but we still want to answer a set of research questions. A. EX post facto Research B. Survey Research

2. Quasi-experimental research design

2015/10/6 陈芳 华东师大英语系 10

• Ex post facto research design ① Posttest-Only, Nonequivalent control group design

② Pretest-Posttest, Nonequivalent control group

design

③ Time Series design

④ Single-subject designs

⑤ Action research

2015/10/6 陈芳 华东师大英语系 11

• Possible issues： a. if pretest scores are different across

groups? b. This design makes the basic assumption

that the pre and post tests are equivalent. What if they are not? • Define subgroups based on the scores and

build the subgroup factor into research design. • Statistical control.

Lucky you…

2015/10/6 陈芳 华东师大英语系 12

a) Single group time series design G1 – O1 – O2 – O3 – X – O4 – O5

b) Multiple group time series design G1 – O1 – O2 – O3 – X – O4 – O5 G2 – O1 – O2 – O3 – – O4 – O5

③ Time series design

2015/10/6 陈芳 华东师大英语系 13

• Typically a repeated-measure design. • Single Subject Design

a) A-B Design b) A-B-A Design

④ Single-subject design

2015/10/6 陈芳 华东师大英语系 14

• Sounds qualitative but nothing prevents it from using quantitative measures.

• The key thing is that it is design to solve an immediate and local issue. Generalizability is not a concern.

⑤ Action research

2015/10/6 陈芳 华东师大英语系 15

3.SURVEY METHODOLOGY 调研法

QUANTITATIVE AND/OR QUALITATIVE

2015/10/6 陈芳 华东师大英语系 16

1. Longitudinal-(Given Multiple Times) a) Trend Study b) Cohort Study c) Panel Study

2. Cross-Sectional Design- (Given Once)

Survey as a research method

2015/10/6 陈芳 华东师大英语系 17

• Total survey design: – Quality of the sample : the frame, the size,

the design, and the response rate – Quality of the questions as measures – Quality of data collection: especially the use

of effective interviewer training and supervising procedures

Total survey design

2015/10/6 陈芳 华东师大英语系 18

Inference and errors

2015/10/6 陈芳 华东师大英语系 19

Characteristics of Population

Sample of population members who answer

questions

Answers respondents give

Validity Issue: How well are the

answers?

Generalizability Issue: Who answered the

questions?

• What? • Why?

– Selection framework – Question quality: 3rd lecture – Interviewer quality

• How to reduce? – Access: flexible, multi-mode – Cooperation

Nonresponse

2015/10/6 陈芳 华东师大英语系 20

QUALITATIVE RESEARCH IN GENERAL

Historical Ethnographic Delphi method

2015/10/6 陈芳 华东师大英语系 21

• 5 basic descriptors as the underlying epistemology of qualitative research.

① Holistic ② Natural Settings ③ Perceptions are Important ④ Favor Post Hoc conclusions ⑤ Phenomena are loosely modeled

2015/10/6 陈芳 华东师大英语系 22

Reliability

Quantitative research 定量 • External reliability

– Can replicate the study

• Internal reliability – Based on the internal

constancy of experiment

Qualitative research 定性

2015/10/6 陈芳 华东师大英语系 23

• External reliability: – specific to natural settings, but can

make a persuasive argument so that the reader can evaluate whether it can be replicated

• Internal reliability:

– Typically, there are observations from multiple observers so can compare the agreement rates or the agreement of conclusions.

• Interrater reliability

Validity

Quantitative research 定量 • External validity

– Generalizability • Internal validity

– Can make a reasonable conclusion

Qualitative research 定性

2015/10/6 陈芳 华东师大英语系 24

• External validity: – Comparability or translatability of

the research – Not a concern for the researcher

• Internal validity:

– Can the data provide a conclusion that still apply?

• Cross-validation

4. HISTORICAL RESEARCH 历史研究

Qualitative method

2015/10/6 陈芳 华东师大英语系 25

ETHNOGRAPHIC RESEARCH 人种志研究

2015/10/6 陈芳 华东师大英语系 26

• The process of ethnographic research is an integrated process in which procedures are conducted concurrently. ① Identify the Phenomena to be studied:

What? ② Identify Subjects: Who? ③ Hypotheses: What? ④ Data Collection: How? ⑤ Analysis: How and what? ⑥ Drawing Conclusions: How and why?

The process

2015/10/6 陈芳 华东师大英语系 27

• Field notes example

From data to hypothesis

2015/10/6 陈芳 华东师大英语系 28

• Triangulation.

2015/10/6 陈芳 华东师大英语系 29

DELPHI METHOD 德尔菲法 Good for policy-related research

2015/10/6 陈芳 华东师大英语系 30

• Groupthink tends to occur when a group strive to reach consensus despite organizational flaws, combined with a high degree of homogeneity of member's social background and ideology. Extreme pressure or stress exacerbates the situation!

• Historical lessons from:

2015/10/6 陈芳 华东师大英语系 31

1. Illusions of invulnerability : 2. Ignoring warnings 3. Unquestioned belief 4. Stereotyping 5. Direct pressure to conform. 6. Self censorship 7. Illusions of unanimity: 8. Mindguards –

8 Symptoms of group think

2015/10/6 陈芳 华东师大英语系 32

• Research category • Validity and reliability • Experimental and quasi-experimental

research design • Typical qualitative research methods • Next:

– Quantitative models and modeling: General approaches and examples in language-related research

– Data collection methods for language-related research

Summary for today

2015/10/6 陈芳 华东师大英语系 33