We can assess reliability by four ways: Show
Some conditions need to be fulfilled in the repetition of measurement, such as same location; repetition over a short period of time; same administration procedures. However, it raises concerns for educational testing which is the practice effect. The correlation between two sets of scores is used as the reliability index:
2. Parallel forms reliability Parallel/Alternative/Equivalent forms should be built based on the same test specifications but contain different items. For instance, having same set of domains, or same types of questions (multiple-choice vs. essay types) or corresponding questions at the same difficulty level. The administration of the tests can be counterbalanced to minimize the variation due to environmental factors.
Correlation between two forms is used as the reliability index. 3. Split-half reliability The correlation between two separate half-length tests is used to estimate the reliability.
3. Internal consistency reliability The idea is that each item in a test can be considered as a one-item test. The total test of n items is seen as a set of n parallel tests. Then we estimate the reliability depending on the consistency of each person’s performance from item to item. This is called the Coefficient Alpha, also known as Cronbach Alpha. The coefficient alpha is interpreted asthe degree to which all of the items measure a common construct The Kuder-Richardson formula 20 (K 20) is used when each item is scored dichotomously (either 0 or 1), the item variance (for a Bernoulli distribution) can be expressed as pi= Proportion of correct responses qi = 1 – pi ; the proportion of incorrect responses Then the equation becomes:
Validity
Validity is composed of different forms→ the intended uses need to be justified from different aspects. The three major ones:
Content-Related Validity
➢ Description of content to be covered by the test. Criterion-Related Validity: The criterion-related validity focus on the degree to which it correlates with some chosen criterion measure of the same construct (relations to other variables). There are two broad classes of this validity form.
Example: Use spelling test scores to predict reading test scores, the validity of the SAT scores for predicting First-Year Grades given high-school GPA.
Example: A new test vs. an old test measuring the same construct. Usually the scores on both tests are obtained at essentially the same time. Construct-Related Validity: construct validation requires collecting multiple types of evidence. Four commonly used approaches to construct validation are:
The valid measures of a construct will indicate that it should be strongly related to certain measures (Convergent validity), and it should be weakly related to others (Discriminant validity).An explicit method for studying the patterns of high and low correlations among a set of measures is called the analysis of Multi- Trait Multi-Method (MTMM) matrix of correlations.
The factor analysis investigates the construct validity from the perspective of examining the Internal structure of the construct. It investigates if the items “hang together” to measure the construct. The two primary classes of factor analytic methods are exploratory factor analysis (EFA) and confirmatory factor analysis (CFA). Exploratory factor analysis (EFA): EFA explores factor structures without a consideration of the theoretical expectations of the researcher, even when such expectations are available.
Confirmatory factor analysis (CFA) CFA is used to validate a pre-specified structure and to quantify the fit of each model to the data. In EFA, a single model is tested, but CFA can readily be used to test several competitive models and compare the fit among the models. It is strongly encouraged to test all plausible models using CFA and report which model fits better than others based on fit indices. Useful resources: Cortina, J. M. (1993). What is coefficient alpha? An examination of theory and applications, Journal of Applied Psychology, 78(1), 98-104. Streiner, D. L. (2003). Being inconsistent about consistency: When coefficient alpha does and doesn’t matter. J Pers Assess, 80(3), 217-222. https://pdfs.semanticscholar.org/992c/b5766720f4465a9bfbf736f0484290dc0f23.pdf
References Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. New York: Harcourt Brace. What type of validity is measured by correlation coefficients?Correlation coefficients
It measures the relationship between two variables rather than the agreement between them, and is therefore commonly used to assess relative reliability or validity. A more positive correlation coefficient (closer to 1) is interpreted as greater validity or reliability.
What is the type of validity that is judged on the basis of the correlation between the measure and a future external criterion?Criterion Validity
Predictive validity refers to the extent to which a survey measure forecasts future performance.
What is validity in correlation?Criterion validity is the extent to which people's scores on a measure are correlated with other variables (known as criteria) that one would expect them to be correlated with.
What type of reliability is determined by correlating the scores from the same test given to the same individual at two different times?Test-retest reliability is a measure of reliability obtained by administering the same test twice over a period of time to a group of individuals.
|