You are viewing a free preview of this lesson.
Subscribe to unlock all 10 lessons in this course and every other course on LearningBro.
Two questions decide whether a study's findings are worth anything. Is the measure consistent — would it give the same result if we did it again? And is the measure accurate — does it actually capture what it claims to? These are the twin concepts of reliability and validity, and they are among the most frequently examined ideas in Edexcel Paper 3, because every study — experimental or not — lives or dies by them. A finding that cannot be reproduced is a fluke; a finding produced by a measure that does not really measure the target is precise nonsense. This lesson works systematically through the two concepts: the types of reliability and validity you must know, how each is assessed, and — the part examiners reward most — how each is improved. A recurring theme, worth fixing at the outset, is that reliability is necessary but not sufficient for validity: a measure can be perfectly consistent and still entirely wrong.
Key Definition: Reliability is the consistency of a measure — whether it produces the same results when repeated under the same conditions. Validity is the accuracy of a measure — whether it measures what it claims to, and whether its findings generalise beyond the study.
By the end of this lesson you will be able to:
Edexcel 9PS0 — Paper 3: Psychological Skills (Research Methods). This lesson develops the reliability and validity strand assessed in Section A of Paper 3. Our sequence teaches each concept as a pair of questions (is it consistent? is it accurate?) and then works through the sub-types and their remedies, rather than following the specification's own order.
| Our lesson covers | Edexcel 9PS0 research-methods area |
|---|---|
| Internal and external reliability; test–retest, inter-observer, split-half | Reliability and how it is assessed and improved |
| Internal and external validity; face, concurrent, ecological, temporal, population validity | Validity and how it is assessed and improved |
| Ways of improving reliability and validity | Improving the quality of research |
Assessment Objectives. These items are AO1 (define a type of reliability/validity, describe an assessment method) and AO2/AO3 (apply the right check to a described study, and evaluate or improve a study's reliability or validity in context). "Explain how you could assess/improve the reliability of this study" is a stock Paper 3 question, and it is almost entirely applied, so a scenario-free answer scores poorly.
Connects to…
Reliability is about consistency. A reliable bathroom scale gives the same reading for the same object each time; a reliable psychological measure gives the same result for the same person (or the same behaviour) on repeated measurement. It is useful to split reliability into two forms.
Key Definition: Internal reliability concerns consistency within a measure — whether all parts of a test measure the same thing consistently. External reliability concerns consistency over time or across users — whether a measure gives the same result when repeated or applied by different people.
| Method | What it checks | How it works | Criterion |
|---|---|---|---|
| Test–retest | External reliability (over time) | Give the same test to the same people on two occasions, then correlate the two sets of scores | A high positive correlation (r≥+0.80) indicates good reliability |
| Inter-observer / inter-rater | External reliability (across users) | Two or more observers/raters record or score the same behaviour independently, then their records are correlated | r≥+0.80 (or 80%+ agreement) is acceptable |
| Split-half | Internal reliability | Split the items of a test into two halves (e.g. odd vs even items), score each half separately, and correlate the two half-scores | A high correlation indicates the items are internally consistent |
Test–retest must leave enough time between administrations that participants do not simply remember their earlier answers, but not so long that the thing being measured has genuinely changed (a personality trait should be stable; a mood should not). Inter-observer reliability matters wherever human judgement enters the data — coding behaviour, rating open responses, diagnosing — and is improved by operationalising categories and training observers together. Split-half reliability checks that a test is internally coherent: if the odd-numbered items and the even-numbered items of an anxiety scale produce very different scores, the scale is not measuring one consistent construct.
Worked example — assessing the reliability of an observation. Suppose two researchers observe classroom "on-task" behaviour. To check inter-observer reliability, they first agree operationalised categories (e.g. "eyes on the teacher or task", "writing", "off-task talking"), then watch the same lesson simultaneously but independently, each tallying behaviour every 30 seconds (time sampling). Afterwards, their two sets of tallies are correlated. Imagine the correlation comes out at r=+0.62 — below the +0.80 threshold, so reliability is unacceptable. The remedy is diagnostic: they inspect where they diverged, find that "off-task talking" was being interpreted differently (one counted whispering, the other did not), redefine that category more tightly, retrain together on a practice clip, and then re-check. This concrete cycle — measure, correlate, locate the disagreement, refine, re-check — is exactly what an exam answer on "how would you improve reliability?" should describe, rather than a vague "train the observers".
A useful further distinction is between reliability of the measure and reliability of the findings. A single questionnaire can be internally reliable (its items cohere) yet a study's overall findings be unreliable if the study cannot be replicated — which is why replication, powered by standardisation and operationalisation, is the ultimate test of a finding's consistency across time, samples and laboratories.
graph TD
A[Reliability = consistency] --> B[Internal: within the measure]
A --> C[External: over time / across users]
B --> B1[Split-half:<br/>correlate two halves of a test]
C --> C1[Test-retest:<br/>same test, same people, later]
C --> C2[Inter-observer:<br/>correlate two observers records]
style A fill:#2980b9,color:#fff
When a measure proves unreliable, the remedy depends on the source of the inconsistency:
Exam Tip: If a question asks how to assess reliability, name the correct check (test–retest for a test over time; inter-observer for an observation; split-half for internal consistency) and state the r≥+0.80 criterion. If it asks how to improve reliability, tie a specific fix (operationalise categories, standardise instructions, pilot the questionnaire) to the study in front of you.
Validity is about accuracy — whether a measure truly captures what it claims to, and whether the findings extend beyond the specific study. Like reliability, it divides into an internal and an external form.
Key Definition: Internal validity is whether a study genuinely measured what it set out to — whether the effect on the DV was really caused by the IV, and not by confounds, demand characteristics or investigator effects. External validity is whether the findings generalise beyond the study — to other people, settings and times.
Internal validity is undermined by anything that offers a rival explanation for the results: confounding variables that vary with the IV, demand characteristics (participants guessing and playing to the aim), investigator effects (the researcher's expectations shaping the outcome), and social desirability bias in self-report. A study high in internal validity has controlled these, so that the measured effect can be trusted to reflect the intended variable.
To make this concrete: imagine a study claiming that a new revision technique improves test scores, but the technique group happened to be tested in the morning and the control group in the afternoon. Time of day is now a confounding variable — it varies systematically with the IV, so any score difference could be due to alertness rather than the technique, and the study's internal validity collapses. The fix is to control the confound (test both groups at the same time, or counterbalance). Notice how the language of internal validity simply reuses the variable-control concepts from the experimental-methods lesson: protecting internal validity is controlling extraneous variables so they never become confounds.
External validity has three commonly examined sub-types:
| Type | Question it asks | Threatened when… |
|---|---|---|
| Population validity | Do the findings generalise to other people beyond the sample? | The sample is unrepresentative (e.g. all students, all one culture — see the sampling lesson) |
| Ecological validity | Do the findings generalise to other settings, especially everyday life? | The task or setting is artificial and unlike real behaviour (a criticism of many lab studies) |
| Temporal validity | Do the findings still hold across time? | Attitudes, norms or conditions have changed since the study (an old study may no longer apply) |
Each sub-type can be illustrated with a real study. Population validity is the concern raised by any all-student or single-culture sample: findings on Western undergraduates may not describe older adults or collectivist cultures (the WEIRD-samples problem from the sampling lesson). Ecological validity is the concern raised whenever a laboratory task is unlike everyday behaviour — recalling nonsense syllables, or judging line lengths in an Asch-style setup, tells us less about real-world memory or conformity than we might wish. Temporal validity is the concern raised by older attitude research: a 1950s study of conformity or gender roles may capture the norms of its era rather than a timeless truth, so its findings may not replicate today. A precise evaluation names which sub-type is threatened and why, rather than asserting a vague "the study lacked validity".
Whereas reliability is assessed by correlation, validity is assessed in several distinct ways:
Key Definition: A measure has construct validity when it genuinely captures the underlying psychological construct it is intended to measure, rather than something adjacent to it. Face and concurrent validity are two ways of building the case for construct validity.
Ecological validity is worth dwelling on, because it is both heavily examined and frequently misunderstood. It concerns two related things: the setting (is the environment like a real-world one?) and the task (does the measured behaviour resemble what people actually do in life?). A study can be high on one and low on the other. A memory experiment that asks participants to recall lists of unconnected words in a lab is low on task realism — we rarely memorise random word-lists — even before we consider the setting. This is a standard criticism of some early memory research, and the counter is to use mundane realism: tasks and materials that mirror everyday demands (recalling a shopping list, a face, or the details of a witnessed event). The reverse case matters too: a field study set in a real street still lacks ecological validity if the behaviour demanded is contrived. The examiner-friendly formulation is therefore not "lab = low, field = high", but "does the task and setting together resemble the real behaviour we want to generalise to?" — a more discriminating judgement that separates strong answers from formulaic ones.
Subscribe to continue reading
Get full access to this lesson and all 10 lessons in this course.