You are viewing a free preview of this lesson.
Subscribe to unlock all 10 lessons in this course and every other course on LearningBro.
Two questions decide whether a study's findings can be trusted. Is the measurement consistent? — that is reliability. Is the study measuring what it claims to measure, and does it generalise? — that is validity. These two questions, and the precise vocabulary that answers them, form the single most important evaluation toolkit in the whole A-Level: almost every AO3 mark you earn in Components 02 and 03 will draw on the concepts introduced here, applied to a named study. Alongside these sit the specific threats that undermine them: demand characteristics, social desirability, researcher and observer bias. This lesson gives you the full vocabulary OCR expects, and — because these terms are the backbone of AO3 evaluation — the ability to deploy them precisely when you evaluate any study in Components 02 and 03.
We cover the types of reliability (internal, external; inter-rater, test-retest, split-half), the many types of validity (internal, face, construct, concurrent, criterion, external, population, ecological), the threats of demand characteristics, social desirability and researcher/observer bias and effects, and the linked ideas of representativeness and generalisability. Precision matters: examiners reward candidates who name the specific type of reliability or validity relevant to a study, not just "it wasn't reliable".
| This lesson covers | OCR H567 Component 01 sub-area | AO focus |
|---|---|---|
| Reliability: internal, external, inter-rater, test-retest, split-half | 1.3 — methodological issues: reliability | AO1; AO3 |
| Validity: internal, face, construct, concurrent, criterion, external, population, ecological | 1.3 — validity | AO1; AO3 |
| Demand characteristics, social desirability, researcher/observer bias & effects | 1.3 — methodological issues | AO1; AO3 |
| Representativeness and generalisability | 1.3 — representativeness; generalisability | AO1; AO3 |
Referenced descriptively; see the official OCR H567 specification document for exact wording. This lesson develops AO1 (defining each form of reliability, validity and bias) and AO3 (using these concepts to evaluate the trustworthiness and generalisability of research — the dominant evaluation vocabulary of the whole A-Level).
Reliability is the consistency of a measurement. A reliable measure gives the same result under the same conditions. OCR distinguishes two overarching kinds and three ways of assessing/improving reliability. The word "reliability" in everyday speech means "dependability", and the technical sense preserves this: a reliable thermometer, questionnaire or observation schedule can be depended upon to give the same reading for the same underlying reality each time it is used. It is this dependability that lets a researcher — and later a reader — trust that a difference in scores signals a difference in the world rather than a wobble in the instrument.
Internal reliability concerns consistency within a measure — do all the items of a test measure the same underlying thing consistently? External reliability concerns consistency over time or occasions — would the measure give the same result if used again?
The importance of reliability is easy to state: an unreliable measure is worthless, because you can never tell whether a change in scores reflects a real change in what you are measuring or just the instrument's own inconsistency. If a questionnaire gives one person wildly different anxiety scores on two occasions a week apart — with no real change in their anxiety — then any result obtained with it is untrustworthy, because the "signal" is drowned in measurement noise. Reliability is thus a precondition for meaningful research: only once a measure is shown to be consistent can differences it detects be attributed to the variables under study rather than to the measure's own unpredictability. This is why establishing reliability is a routine early step in developing any psychological test or observation schedule, and why "how could the researcher check/improve the reliability of this measure?" is such a common exam question.
The distinction between internal and external reliability maps neatly onto the checks used to assess each. Internal reliability — consistency within the measure — is assessed by split-half (and, at undergraduate level, by internal-consistency statistics), because these examine whether the parts of the measure agree with one another. External reliability — consistency across occasions — is assessed by test-retest, which examines whether the measure gives the same result when applied again later. Inter-rater reliability sits slightly apart, addressing consistency across observers rather than across items or time, and is the check most relevant to observational and rating-based research. Keeping straight which form of reliability a given check assesses is exactly the kind of precise knowledge examiners test.
The three specific forms:
Each of these methods works by looking for agreement — the recurring idea being that a reliable measure produces consistent results across observers, occasions or items, and this consistency is quantified by a correlation. Inter-rater reliability correlates the records of two observers: if they largely agree, the coding is not just one observer's idiosyncratic interpretation. Test-retest correlates the same test's scores on two occasions: a strong correlation shows the measure is stable over time (with the caveat that the gap must be long enough that participants do not simply remember their earlier answers, but not so long that the thing being measured has genuinely changed). Split-half correlates two halves of a single test: a strong correlation shows the items hang together and measure the same underlying construct. In every case the remedy for poor reliability follows from the cause — inter-rater disagreement is improved by clearer, more operationalised coding categories and observer training; low internal consistency is improved by removing or rewriting items that do not correlate with the rest; test-retest instability may point to an ambiguous measure that needs tightening. Being able to name the appropriate check and the matching improvement for a described problem is precisely what the exam rewards.
| Form | What it checks | How assessed | Internal or external |
|---|---|---|---|
| Inter-rater | Agreement between observers | Correlate raters' records | (Assesses observer consistency) |
| Test-retest | Consistency over time | Re-test and correlate | External |
| Split-half | Consistency within a test | Correlate two halves | Internal |
A recurring exam demand: identify which type of reliability is at stake. If two observers disagree, it is inter-rater; if a questionnaire gives different scores when repeated a fortnight later, it is test-retest; if a scale's items are inconsistent with each other, it is split-half/internal.
Inter-rater reliability deserves special emphasis because it is the linchpin of observational research, which is inherently vulnerable to subjectivity. Whenever data depend on human judgement — coding behaviours, rating open-ended responses, classifying qualitative material — there is a risk that the "data" reflect the observer's interpretation as much as the phenomenon. Establishing inter-rater reliability by having two or more observers work independently and then correlating their records is the safeguard: a high correlation shows that the coding is reproducible and not merely one person's idiosyncratic reading. When agreement is poor, the fix lies in the design of the coding frame, not in the observers' effort — vaguely-defined categories force judgement calls that different observers resolve differently, so operationalising the categories more tightly and training observers on a shared standard is what raises agreement. This is exactly the link back to Lesson 4: the quality of the coding frame designed there determines the inter-rater reliability achievable here, so reliability is built at the design stage and merely verified by the correlation.
Validity is whether a study measures what it claims to measure and whether its findings are legitimate and generalisable. OCR names an unusually full set of validity types; know a one-line definition of each. The reason validity matters is that a study can be beautifully reliable, statistically significant and immaculately controlled, yet still be worthless if it is not measuring what it claims or if its findings do not apply beyond the study itself. Validity is the ultimate test of whether research tells us anything true about the world, which is why questions asking how a researcher could improve the validity of a study — by controlling confounds (internal), by using measures that genuinely capture the construct (construct), or by broadening the sample and setting (external) — are so central to Component 01.
Reliability and validity, though often mentioned together, are genuinely different properties, and keeping them apart is essential. Reliability is about consistency; validity is about accuracy and truth. A measure can be perfectly reliable yet completely invalid: a bathroom scale that reads five kilograms heavy gives you the same wrong weight every time — consistent (reliable) but wrong (invalid). In psychology, a poorly-constructed questionnaire might reliably produce the same score for a person on every occasion while measuring something quite different from the construct it claims to assess. The reverse — valid but unreliable — is essentially impossible, because a measure that gives inconsistent results cannot be consistently accurate. This is why reliability is often described as necessary but not sufficient for validity: you need consistency before accuracy is even possible, but consistency alone does not guarantee you are measuring the right thing. Examiners reward candidates who deploy this distinction precisely rather than treating the two terms as interchangeable synonyms for "good".
Internal validity — whether the study genuinely measured the effect of the IV on the DV, free from confounding variables. High internal validity means we can trust the cause-and-effect claim. It is threatened by anything that offers a rival explanation for the results — confounding variables (Lesson 2), demand characteristics, or a measure that does not really capture the DV.
Face validity — whether a measure appears, on the face of it, to measure what it intends (a simple, superficial check — does an anxiety scale look like it measures anxiety?). Despite being superficial, face validity has practical value: a measure that looks irrelevant to participants may reduce their engagement or arouse suspicion, whereas one that plainly relates to the topic feels credible.
Construct validity — whether the measure genuinely captures the underlying theoretical construct it targets (does the anxiety scale actually tap the psychological construct of anxiety?).
Concurrent validity — whether a new measure produces results that agree with an established measure of the same thing tested at the same time (does the new anxiety scale correlate with a well-established one?). It is a practical way to validate a new, perhaps shorter or cheaper, instrument: if it correlates strongly with a trusted existing measure, that is evidence it taps the same construct.
Criterion validity — the broader family of validating a measure against an external criterion or outcome (concurrent and predictive validity are types of criterion validity). Predictive validity, for instance, asks whether a measure predicts a relevant future outcome — an aptitude test has predictive validity if high scorers really do go on to perform well — while concurrent validity checks agreement with a criterion measured now. Both anchor the measure to something external and observable rather than relying on appearance alone.
The distinction between face and construct validity is worth dwelling on because it is frequently confused. Face validity is the weakest, most superficial form: it asks only whether a measure looks like it measures the target, a judgement anyone could make at a glance. It is easy to establish but proves little, because a measure can look plausible yet fail to capture the construct. Construct validity is deeper and harder to establish: it asks whether the measure genuinely taps the underlying theoretical construct, judged through patterns of evidence — does it correlate with related measures, distinguish groups it should distinguish, and behave as theory predicts? A questionnaire might have good face validity (its items obviously concern anxiety) but poor construct validity (it actually measures general negative mood rather than anxiety specifically). Recognising that "looks right" and "genuinely measures the construct" are different claims is a hallmark of a sophisticated understanding.
External validity — whether findings generalise beyond the study, subdividing into:
Subscribe to continue reading
Get full access to this lesson and all 10 lessons in this course.