Chapter 11+12faculty.ecnu.edu.cn/picture/article/220/0a/d4/9eb... · ECNU Educational Research...
Transcript of Chapter 11+12faculty.ecnu.edu.cn/picture/article/220/0a/d4/9eb... · ECNU Educational Research...
ECNU Educational Research Methodology
Sampling design Measurement and data collection
Chapter 11+12
2016/12/7 1 陈芳 华东师大英语系
ECNU Educational Research Methodology
Plan
We have discussed many different research designs and their advantages and disadvantages.
Now, we will assume that you know which
design you will be using and we will learn how to actually go about getting our subjects, collecting data, analyzing it, and making conclusions.
2016/12/7 2 陈芳 华东师大英语系
ECNU Educational Research Methodology
Chapter 11
Sampling Design
2016/12/7 3 陈芳 华东师大英语系
ECNU Educational Research Methodology
Good Sampling
In general good sampling is
1. Goal Oriented – tailored to design 2. Measurable – provides needed data 3. Practical – it is feasible 4. Economical – have resources
Often, some of these are compromised. The
important overall criterion is that the design be feasible and adequately accommodate the research problem.
2016/12/7 4 陈芳 华东师大英语系
ECNU Educational Research Methodology
1. Random sample 随机抽样(suiji chouyang) This was typically in experimental or quantitative
research ①Simple random sampling ②Stratified random sampling ③Cluster sampling ④Systematic sampling
2. Purposeful Sampling 目的抽样 (mudi chouyang) We presented this idea for qualitative research
mainly ethnographic.
Introduction
2016/12/7 陈芳 华东师大英语系 5
ECNU Educational Research Methodology
1. Random Sampling
Random Sampling or Probability Sampling In this case all members if the population have
some probability of being selected • These probabilities are not necessarily all equal but they
are non-zero
Computer software makes the sampling
process rather easy
2016/12/7 6 陈芳 华东师大英语系
ECNU Educational Research Methodology
SPSS function for simple random sampling
2016/12/7 陈芳 华东师大英语系 7
ECNU Educational Research Methodology
2016/12/7 陈芳 华东师大英语系 8
ECNU Educational Research Methodology
2016/12/7 陈芳 华东师大英语系 9
ECNU Educational Research Methodology
Random selection vs. random assignment We could randomly select people but not randomly
assign them to groups We could also have a sample (not random) and
randomly assign them to tasks. (think of a classroom situation)
Some terminologies
2016/12/7 陈芳 华东师大英语系 10
ECNU Educational Research Methodology
More terminologies
After we have a sample we must differentiate the difference between Sampling Error: Random
Sampling Bias: systematic
2016/12/7 11 陈芳 华东师大英语系
ECNU Educational Research Methodology
② Stratified Random Sampling 分层随机抽样
Imagine that we have a sample that consists of: 50% Subpopulation A 40% Subpopulation B 5% Subpopulation C 5% Subpopulation D
2016/12/7 12 陈芳 华东师大英语系
ECNU Educational Research Methodology
Two types of stratified sampling
We want a sample of 100. 1. Non-proportional allocation
a) randomly select 25 people from each strata --- equal allocation
b) select more persons from the stratum where there is more variability: maybe 30, 30, 20, 20
2. Proportional allocation: 50 people from strata A 40 people from strata B 5 people from strata C 5 people from strata D
2016/12/7 13 陈芳 华东师大英语系
ECNU Educational Research Methodology
Example of proportional allocation
Unless a large survey is being conducted, stratification seldom involves more than two stratifying variables, and the total number of strata would seldom exceed 20 and usually is considerably less.
2016/12/7 陈芳 华东师大英语系 14
ECNU Educational Research Methodology
③ Cluster Sampling
Clusters / strata. In cluster sampling we first divide the population
into clusters and randomly sample from the clusters. However, once the cluster is selected, all the members in it are selected. This is the difference between cluster sampling and stratified sampling.
Another difference is that the sample size is usually unknown until after the sample is selected because clusters are usually not of the same size. 2016/12/7 15 陈芳 华东师大英语系
ECNU Educational Research Methodology
Book example
Clusters may be City
blocks Schools Counties
2016/12/7 陈芳 华东师大英语系 16
ECNU Educational Research Methodology
④ Systematic Sampling
This last method of sampling requires that the individual already has a list of people Imagine an alphabetical list or something like that; Compute the sampling fraction n/N=1/k Once we know k, we randomly select someone
from the first k people Starting with that person, we select every kth
person Once the first person has been selected the whole
sample is determined
2016/12/7 17 陈芳 华东师大英语系
ECNU Educational Research Methodology
Periodicity: Every kth member of the population has some
unique characteristics that are related to, or have an effect on, the dependent variable. In that case, the sample becomes biased.
Student name list ordered by ability within each
class.
Pro: this method requires less work.
Systematic Sampling and periodicity
2016/12/7 陈芳 华东师大英语系 18
ECNU Educational Research Methodology
Sample size
1. It is related to costs and time involved. Practicality
2. It is related to statistical precision and statistical power.
2016/12/7 19 陈芳 华东师大英语系
ECNU Educational Research Methodology
Statistical Precision and Statistical Power
Statistical Precision Larger sample size, better precision. Confidence interval.
Statistical Power Larger sample size enables us to be more likely to
find a difference when there is difference.
2016/12/7 20 陈芳 华东师大英语系
ECNU Educational Research Methodology
1. Sudman 1976: for regional or special surveys (thesis or dissertation) with few subgroups, 200-500 is typical.
2. The more variable the dependent variable the greater the number required in each group.
3. e.g. 8, or 10, 15-30 per group may be adequate. For two groups, maybe totally need 100.
4. Factorial, 10 or more in each cell. 5. …… 6. Kraemer & Thiemann 1987: How many
subjects? Statistical power analysis in research.
Guidelines?
2016/12/7 陈芳 华东师大英语系 21
ECNU Educational Research Methodology
2. Purposeful Sampling
The other kind of sampling we discussed was purposeful sampling when we want to find a sample that will be most informative for our purpose and a random sample may not make sense or may be infeasible. ① E.g. Impossible logistically to select a random
sample from all graduating high school. ② E.g. Unethical to assign randomly patients for
treatment or nontreatment. Ethnographic research, Generalizability and external validity?
2016/12/7 22 陈芳 华东师大英语系
ECNU Educational Research Methodology
Purposeful Sampling
There are 6 basic kinds of purposeful sampling discussed in the book Comprehensive: e.g. all handicapped Maximum Variation: Use a set of schools that are all quite
different (low SES, Medium SES, High SES) Extreme Case Typical Case Homogeneous Other
Examples?
2016/12/7 23 陈芳 华东师大英语系
ECNU Educational Research Methodology
Extreme Case Sampling
Make selections based on unique or special cases of a particular characteristic.
The logic of using extreme case sampling is
that what is learned from the extreme cases may be applied to typical cases. E.g. Studying a new method for school
• Could use a traditionally low achievement school
2016/12/7 24 陈芳 华东师大英语系
ECNU Educational Research Methodology
Typical Case Sampling
This is the opposite of Extreme case sampling. So an example of the average case is used Going back to the school example
• One with average performance is used
2016/12/7 25 陈芳 华东师大英语系
ECNU Educational Research Methodology
Homogeneous Sampling
A sample is selected to represent a specific subgroup of people In studying teachers, only beginning teachers are
used
2016/12/7 26 陈芳 华东师大英语系
ECNU Educational Research Methodology
Sample Size
Unlike random sampling, we are not as concerned with sample sizes
Typically, they are small and are selected for their specific purpose
2016/12/7 27 陈芳 华东师大英语系
ECNU Educational Research Methodology
Summary
So we have specified 4 types of random sampling (simple, stratified, cluster, systematic)
And we specified 6 types of purposeful sampling (Comprehensive, Maximum Variation, Extreme Case, Typical Case, Homogeneous, Other
2016/12/7 28 陈芳 华东师大英语系
ECNU Educational Research Methodology
Chapter 12 Measurement and Data Collection
2016/12/7 29 陈芳 华东师大英语系
ECNU Educational Research Methodology
Introduction
What is measurement? Concept of measurement and measurement scales Reliability of Measurement Validity of Measurement Tests and Inventories used for measurement
2016/12/7 30 陈芳 华东师大英语系
ECNU Educational Research Methodology
Measurement
Kerlinger (1986, p.391): Measurement is a process of assigning
numerals to events or objects according to rules.
Examples: Achievement Attitudes Intelligence
2016/12/7 31 陈芳 华东师大英语系
ECNU Educational Research Methodology
Measurement Scales
Depending on the rules and the properties of those numbers our measurement scale can be one of four types Nominal: just a name Ordinal: there is rank Interval: equal distance Ratio: true zero
2016/12/7 32 陈芳 华东师大英语系
Nominal Ordinal Interval Ratio
ECNU Educational Research Methodology Absolute zero means absence of electron movement, thus no heat
2016/12/7 33 陈芳 华东师大 FANG CHEN ECNU
ECNU Educational Research Methodology
Measurement Characteristics
There are two basic characteristics that we use to characterize the quality of our measurement Reliability – Consistency of Measurement
Validity – Quality of Measurement (i.e., are we
really measuring what we mean to measure)
2016/12/7 34 陈芳 华东师大英语系
ECNU Educational Research Methodology
Reliability
We have many different ways to measure reliability empirically
Parallel forms Test-Retest Split-Half Kuder-Richardson Cronbach alpha
2016/12/7 35 陈芳 华东师大英语系
ECNU Educational Research Methodology
Reliability
While reliability varies depending on what is being measured, most of the time we aim for about .85 although Achievement is a little higher Psychology is a little lower
Correlation analysis; Factor analysis…
2016/12/7 36 陈芳 华东师大英语系
ECNU Educational Research Methodology
2016/12/7 陈芳 华东师大英语系 37
ECNU Educational Research Methodology
Validity
Are we measuring what we want to measure?
Two parts: Internal – Can we make the conclusions
that our experiment is intended to show External – Generalizability
2016/12/7 38 陈芳 华东师大英语系
ECNU Educational Research Methodology
Validity
Unitary concept + a matter of degree Content-Related: content sampling Criterion-Related Evidence
• Concurrent • Predictive
Construct-Related Face validity Consequential
2016/12/7 39 陈芳 华东师大英语系
ECNU Educational Research Methodology
2016/12/7 陈芳 华东师大英语系 40
ECNU Educational Research Methodology
Variables Measured in Education
There are several types of variables, typically quantitative, that are measured frequently in education. ① Achievement Tests in Academic and Skills Areas ② Attitude Inventories ③ Aptitude Tests ④ Rating Scales ⑤ Observation Systems
2016/12/7 41 陈芳 华东师大英语系
ECNU Educational Research Methodology
① Achievement tests
Achievement is one of the principal outcomes of schooling.
They come in three forms Norm-referenced (compared to others in the
normative group) Criterion-Referenced (compared to the standards) Proficiency (only testing a certain level of
proficiency)
2016/12/7 42 陈芳 华东师大英语系
ECNU Educational Research Methodology
② Attitude
Achievement tests generally are designed to measure an individual’s best or maximum performance, whereas an attitude inventory is intended to measure typical performance.
Attitude is toward something.
Measurement of the attitude is intended to place individuals on this continuum.
2016/12/7 43 陈芳 华东师大英语系
ECNU Educational Research Methodology
Attitude
Because attitude is usually thought to exist on a continuum these tests usually are Likert Scales Semantic Differential
• http://psc.dss.ucdavis.edu/faculty_sites/sommerb/sommerdemo/
In either case, if writing your own you should
pilot the test and evaluate the items and its reliability
2016/12/7 44 陈芳 华东师大英语系
ECNU Educational Research Methodology
1. Using the right response option The ABC institute had a very positive impact on my
teaching. • Strongly disagree --- Strongly agree
Rate the extent to which the ABC institute has had
a positive effect on your teaching. • None --- Some --- Much
Issues in Likert scale item writing
2016/12/7 陈芳 华东师大英语系 45
ECNU Educational Research Methodology
2. Using the right scoring SA A U D SD It is fun to work with math puzzles. I do not like to work fraction problems.
Issues in Likert scale item writing continued
2016/12/7 陈芳 华东师大英语系 46
ECNU Educational Research Methodology
3. Items with little variance will not differentiate between individuals or groups; may also prevent programs from running.
4. Items that are frequently omitted may be unclear or offensive to the respondents.
5. Fake responses.
6. The middle-of-the-roader.
2016/12/7 陈芳 华东师大英语系 47
ECNU Educational Research Methodology
The semantic differential provides a series of bipolar adjective scales relative to a word or concept; the respondents indicate their feelings on the continuum of each scale.
Format Scoring: Remember your scale
Semantic differential
2016/12/7 陈芳 华东师大英语系 48
ECNU Educational Research Methodology
③ Aptitude
These tests are designed to measure the potential for achievement. Intelligence tests General scholastic ability General mental ability
2016/12/7 49 陈芳 华东师大英语系
ECNU Educational Research Methodology
Other Measures
④ Rating Scales
⑤ Observation Scales • High Inference: • Low Inference: • Low inference observations tend to be
somewhat more reliable than high-inference observations because less judgment is involved.
2016/12/7 50 陈芳 华东师大英语系
ECNU Educational Research Methodology
Where to Find These Tests
Mental Measures Yearbooks Tests In Print Buros Desk Reference: Psychological
Assessment in the School ETS Disertations Abstracts International And many others…
2016/12/7 51 陈芳 华东师大英语系
ECNU Educational Research Methodology
Variables Measured in Education
Published tests are usually well constructed and should come with a reliability measure and other details Mental Measurements Yearbooks (MMY) Tests in Print (TIP)
2016/12/7 52 陈芳 华东师大英语系
ECNU Educational Research Methodology
2016/12/7 陈芳 华东师大英语系 53
ECNU Educational Research Methodology
Scoring and Data Prep
Once we have collected the data we will need to code it or score it before any analysis.
Codebook is an important part of quantitative research and analyses. It is the notebook to record the type of numerals you have assigned to each variable in your data. They have to be interpreted back into words for your conclusions.
2016/12/7 54 陈芳 华东师大英语系
ECNU Educational Research Methodology
2016/12/7 陈芳 华东师大英语系 55
ECNU Educational Research Methodology
Weighting http://applied-survey-methods.com/weight.html
2016/12/7 陈芳 华东师大英语系 56