Pengukuran, Kesahan (Validity) Dan Kebolehpercayaan (Reliability) Instrumen
Validity and Reliability -...
Transcript of Validity and Reliability -...
Validity and Reliability
PD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com
Validity
� Is the translation from concept to operationalization accurately representing the underlying concept.
� Does your variables measure what you think in abstract concepts.
� This is more familiarly called Construct Validity.� empirical study with high construct validity would ensure the studied parameters are relevant to the research questions.
�Without a valid design, valid scientific conclusions cannot be drawn
PD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com
Types of construct validity
� Translation validity (Trochims term)�Face validity �Content validity
� Criterion-related validity�Predictive validity �Concurrent validity �Convergent validity �Discriminant validity
PD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com
Translation validity
� Is the operationalization a good reflection of the construct?
� This approach is definitional in nature � assumes you have a good detailed definition of the
construct � and you can check the operationalization against it.
� Example software success. � Does your definition representative of SW success
construct?� E.g. Application software is a software used to assist
end users.
PD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com
Face Validity
� “On its face" does it seems like a good translation of the construct.
� If the respondent knows what information we are looking for,
� they can use that “context” to help interpret the questions and provide more useful, accurate answers
PD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com
Content Validity
� Check the operationalization against the relevant content domain for the construct.
� For example, � a depression measure should cover the checklist
of depression symptoms� World history – its content must include major
histories from all continents or countries� Interface Usability – should include all valid
usability measures: � learnability, efficiency, memorability (low
cognitive overload), error recovery and the like
PD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com
Criteria-Related Validity
� Check the performance of operationalization against some criterion.
� it compares the test with other measures or outcomes (the criteria) already held to be valid.
� For example, employee selection tests are often validated against measures of job performance (the criterion), and
� IQ tests are often validated against measures of academic performance (the criterion).
PD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com
Predictive Validity
� Assess the operationalization's ability to predict something it should theoretically be able to predict. � A high correlation would provide evidence for predictive validity
� Examples � Measures of job applicant is supposed to measure the new applicant
performance at work. � If the applicant performs well at his job when measured after one
year, our applicant measurement is a good predictive measure.� Measures of Interface Usability can predict later SW utilization. � High correlation is an indication of measures of predictive validity
PD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com
Concurrent Validity
� Assess the operationalization'sability to distinguish between groups that it should theoretically be able to distinguish between.
� It is similar to predictive validity but the measures are taken at the same time.
� If measure of subordinate rating and supervisor rating positively correlate on job performance, it has high concurrent validity
� Compares the results of two measures
PD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com
Convergent Validity
� Examine the degree to which the operationalization is similar to (converges on) other operationalizations that it theoretically should be similar to.
� This compares two or more attributes of the same construct
� To show the convergent validity of a test of arithmetic skills, � one might correlate the scores on a Math test with scores on other tests (e.gproblem solving ability) that support to measure basic math ability,
� The measure learnability should have high correction with efficiency, memorability, errors and satisfaction
� All measures measure the same construct
� There is also instrument measure convergence – If measure of Interview and questionnaire produce the same result to say the instruments are convergent
PD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com
Discriminant Validity
� Examine the degree to which the operationalization is not similar to (diverges from) other operationalizationsthat it theoretically should be not similar to.
� A test of a concept is not highly correlated with other tests designed to measure theoretically different concepts.
PD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com
Discriminate …
� where rxy is correlation between x and y,� rxx is the reliability of x, and� ryy is the reliability of y:� a result less than .85 tells us existence of
discriminant validity� > .85, the two constructs overlap greatly and they are
likely measuring the same thing.
PD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com
Discriminate …
� Measur ing the concept of N arcissism and Self-esteem� N arcissism is a term with a wide range of meanings,
usually is used to describe some kind of problem in a person or group's relationships with self and others.
� Self-esteem is a term in psychology to reflect a person's overall evaluation or appraisal of her or his own worth.
� Self-esteem encompasses beliefs (for example, "I am competent", "I am worthy") and emotions such as triumph, despair, pr ide and shame
� The Researchers show that their new scale measures N arcissism and not simply Self-esteem.
PD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com
Internal and External Validity
PD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com
Internal Validity
� Inferences are said to possess internal validity if a causal relation between two variables is properly demonstrated.
� A causal inference may be based on a relation when three criteria are satisfied:
1. the "cause" precedes the "effect" in time (temporal precedence),
2. the "cause" and the "effect" are related (covariation), and
3. there are no plausible alternative explanations for the observed covariation
PD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com
Example - Internal
� The researcher hypothesized that computer training will increase software usability
� Training (IV) and usability (DV)
� Positive correlation between the two indicates high internal validity.
� This can be done with Spearman Rank Correlation or Pearson Correlation
� Can be easily done with SPSS software
PD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com
Internal …
� In many cases, however, themagnitude of effects found in the dependent variable may not just depend on � variations in the independent variable,� thepower of the instruments and statistical procedures used
to measure and detect the effects, and� the choice of statistical methods
� Other variables or circumstances uncontrolled for (or uncontrollable) may lead to additional or alternative explanations � (a) for the effects found and/or � (b) for the magnitude of the effects found.
PD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com
Internal …
� highly controlled true experimental designs, i.e� random selection, � random assignment to either the control or experimental
groups, � reliable instruments, � reliable manipulation processes, and � safeguards against confounding factors may be the "gold
standard" of scientific research.
� the very strategies employed to control these factors may also limit the generalizability or External Validity of the findings.
PD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com
External validity
� external validity refers to the applicability of study or experimental results to realms beyond those under immediate observation.
� Refers to generalizability of the research finding to other similar cases
� Does the software solution for one case is also applicable to other similar cases in other organization or country.
� Does the solution has wider application and audience or acceptance. We need that solution!
� Researchers prize studies with external validity, since the results can be widely applied to other scenarios.
PD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com
External …
� External validity for a given study has several aspects:
1. whether the study generalizes to other subjects in the domain
2. whether there exist enough evidence and arguments to support the claimed generalizability
3. whether the study outcomes validate predicted theories
PD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com
Reliability
� Means "repeatability" or "consistency". � A measure is considered reliable if it would give us the
same result over and over again (assuming that what we are measuring isn't changing!).
� Measuring the same distance at different times should give the same result if the instrument (e.g. meter) is reliable.
� There are four general classes of reliability estimates, each of which estimates reliability in a different way.
PD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com
Types of Reliability Estimation22
� There are four measures of reliability
1. Inter-rater or inter-observer reliability
2. Test-retest reliability
3. Parallel-forms reliability
4. Internal consistency (Split Half) reliability
PD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com
Inter-Rater or Inter-Observer Reliability
� Used to assess the degree to which different raters/observers give consistent estimates of the same phenomenon.
� Establish reliability on pilot data or a subsample of data and retest often throughout.
� For categorical data, � X2 (Chai sqaure) can be used and
� For continuous data, � a R (such as linear regression) can be used
PD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com
Test-Retest Reliability
� Used to assess the consistency of a measure from one time to another.
� This approach assumes that there is no substantial change in the construct being measured between the two occasions.
� The amount of time allowed between measures is critical.
� The shorter the time gap, the higher the correlation; the longer the time gap, the lower the correlation
PD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com
Parallel-Forms Reliability
� Used to assess the consistency of the results of two tests constructed in the same way from the same content domain.
� Create a large set of questions that address the same construct and then � randomly divide the questions into two sets and administer both instruments to the same sample of people.
� The correlation between the two parallel forms is the estimate of reliability.
� One major limitation of this approach is that you generate lots of items that reflect the same construct.
PD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com
Split Half Reliability
� Collect your data with the instrument to measure your construct.
� Split the data into halve and do correlation between the two data sets
� Positive correlation indicates high reliability
PD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com
Reliability and Validity27
PD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com
Research Ethics
PD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com
Ethics – a definition29
“Research should avoid causing harm, distress, anxiety, pain or any other negative feeling to participants. Participants should be fully informed about all relevant aspects of the research, before they agree to take part” [1]
PD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com
ARE YOU Corrupted Person?
24 Nov 2008Research Methodology
30
THIS IS A HYPOTHETICAL QUESTION -
DO NOT ANSWER THISPD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com
Research questions – ethical or not?
31
� Research may ask a taboo or personal question� What if you were asked if you are asked are corrupted person?� How would you feel if you were asked this?�Would you feel awkward?�Would you lie?�Would you answer truthfully?
� Why are we asking this question anyway?� Could we rephrase the question better?
PD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com
Pause for thought32
Is it morally correct to carry out research by any means whatsoever providing that the end result increases the sum of human knowledge or provides some tangible benefit to mankind?
Does the end justify the means?
DISCUSS
PD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com
Ethics before Research begins33
� Inform all participants fully�What about children
�Mentally deficient people
� Those with poor language skills
� Obtain consent
� Craft your research methods carefully� No distortion of the data
PD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com
Ethics during Research
Research Methodoogy
34
� Field notes – what are they?
� Do we need these?
� Can we use these in our research?
�Consent issues
�Content issues
�Moral issues
�You have heard about a crime – do you report it?
DISCUSS
PD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com
Confidentiality of respondent data35
� How do we keep track of respondents?
� Should we keep track of respondents?
� How do we de-personalise gathered data?
� If data are depersonalised, is it morally correct to reuse this data for a new research project?
DISCUSS
PD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com
Ethics after Research36
� Disposal of data – paper or digital?
� Freedom of Information Act
� Reuse of data – is this ethical?
� Are there occasions where reuse of gathered data for another purpose is ok?
� Requesting permission from respondents
� Difficulty of contacting original respondents
PD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com
Engineering and Ethics37
� Confidentiality of data
� Ownership of research results
� Consider research results
�Is a cure for a disease as the direct result of research ‘good’?
�Is the creation of a powerful bomb as the direct result of research ‘good’?
�e.g. the atom bombDISCUSS
PD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com
Research Ethics Committees38
� Monitor ethical issues in research programmes Before during and after research
� Makes decisions and enforces these
� Gives researchers organisational support
�Reassurance to researchers about moral issues related to a particular research project
PD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com
Plagiarism39
�What is plagiarism?
� How do we avoid plagiarism?
�What are the dangers that plagiarism causes?
� State some examples of plagiarism.
DISCUSS
PD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com
Responsibilities of Actors
Researchers Sponsors
Respondents
Ethics
PD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com
29 August 2005MBA III (Research Methodology) Course Instructor: Dr.
Aurangzeb Z. Khan
41
Ethical Considerations: Respondents
� Truthfulness in giving information to the researcher if a research subject or respondent gives his or her consent to participate in a research study
� Sustained cooperativeness with the researcher throughout the course of the research study
� Adhere to responsibility if informed consent is given to the researcher
� State any constraints or limitations in advance
PD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com
29 August 2005MBA III (Research Methodology) Course Instructor: Dr.
Aurangzeb Z. Khan
42
Ethical Considerations: Researchers
� No deception, be forthright and do not conceal the true purpose of the research
� Maintain objectivity, courtesy and high professional standards through scientific process
� No falsification, alteration or misrepresentation of data for political or other purposes
� Protect the confidentiality of the research subjects and research sponsors
� No faulty conclusions
� No inclusion or use of information or ideas contained in competing research proposals
PD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com
29 August 2005MBA III (Research Methodology) Course Instructor: Dr.
Aurangzeb Z. Khan
43
Ethical Considerations: Research Sponsors
� No request for submission of competitive bids by researchers if selection of the researcher has already been made
� Avoid manipulation and influencing of the researcher with a view to discrediting individuals or organizations
� The conclusions drawn from research work should be consistent with the data and not influenced by other undesirable conditions or motives
� Observe the confidentiality of the research subjects and researcher
� Avoid Advocacy Research
PD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com
End!
PD
F C
reated with deskP
DF
PD
F W
riter - Trial :: http://w
ww
.docudesk.com