304: Test Reliability

1. A test that is relatively free of measurement error is considere _______
1. Reliable
2. Conceptualization of Error
1. Measurement Error:
1. There will ALWAYS be error in measurement
2. Goal: design tests relatively free of error
3. The observed score consists of the true score plus measurement error (O = T + E)
4. "Rubber Yardstick" comparison
1. carpenter will never get the same measurement with a rubber yardstick
5. Systematic Error is biased
6. When can we increase measurement error?
1. how the test was created and situational factors
3. What is Reliability
1. Methods to test for Reliability
1. Test-retest reliability
1. Test someone now and then test them later 
2. Will the same person take the same test in the same way
2. Alternate-form/parallel-forms reliability
1. Example
1. Vocabulary test and then give another vocab test but with content slightly altered
2. Different versions of the same test
3. Split-half reliability
1. Take the scores on the first half of the test and compare them to the scores on the second half of the test
2. Often test get harder as you go on so often people will say to split the test into even and odd questions
4. Inter-item consistency: Cronbach's Alpha
1. What does it mean if your cronbach's alpha is .95?
1. It means everything is telling you the same thing so you probably don't need so many items because some of your items are telling you the same thing.  
2. The more items you have the more reliable your test will be...but at what cost?
3. So if your cronbach's alpha is .95 you could say that it is too reliable because you aren't getting much information.  But if it is below .7 then it is too low.
2. Kappa coefficient 
1. is similar to cronbach's alpha but it also takes chance into account
5. Inter-rater reliability
1. This is when you have multiple testers doing ratings to make sure you are getting accurate information
2. Conceptual Definitions of Reliability
1. The degree to which test-takers' scores reflect "true" abilities
4. Domain Sampling Model
1. Domain: extremely large collection of items
2. The larger the sample the more accurately it measures the domain
3. Might help to think of a test item as a person in a study
5. Classical Test Score Theory
1. Because we assume error is random we also make the assumption that the distribution of error is the same for everyone
1. If we have a wide variance in the test = lots of error
2. Less variance = less error
6. Test Construction
7. Test Administration
1. It's just as interesting in some tests to not just know their score but to know how they got that score and what influenced it.
2. Test Environment
1. different environments affect scores
3. Test-taker variables
1. What if the test-taker doesn't eat breakfast
4. Examiner-related variables
1. Perhaps the test-taker is being defiant so the examiner gives her an ultimatum to either take the test or he will call the police.  Will this affect the test scores?  How so?
2. What if you are having a bad day or if you are biased in some way?  We need to be aware of our biases because we all have them.
5. Neuropsych majors give simple tests for malingering.  For example: asking someone with brain damage to put their fingers together and then pull them apart.  Anyone can do this but someone who is malingering will pretend to not be able to.