Non-Parametric Inference for Related Samples: Key Techniques

Every M.Ed or Ph.D scholar working with pre-test and post-test scores eventually hits the same wall: the paired t-test assumes the differences between scores are normally distributed, and with small classroom samples, that assumption often collapses. When it does, the paired t-test’s non-parametric cousins step in. Two of the most widely used options are the Sign Test and the Wilcoxon Matched-Pairs Signed-Ranks Test. Both are built for related, or “matched,” samples, but they read the data very differently, and knowing the difference can shape how convincingly you argue your findings.

Why related samples need their own tests 🔗

Related, or dependent, samples occur when the same participants are measured twice, or when participants are matched on relevant traits before being assigned to two conditions. A teacher testing a new pedagogy on the same batch of students before and after an intervention is generating related-sample data. So is a researcher pairing students by prior academic performance before comparing two teaching methods. Because the two sets of scores come from the same or matched individuals, the pairs are not independent of each other, and that changes which statistical test applies.

When those paired differences follow a roughly normal distribution, the paired-samples t-test is appropriate. When they do not, or when the data is only ordinal, non-parametric alternatives are designed specifically for this matched-pairs situation. The Sign Test and the Wilcoxon Matched-Pairs Signed-Ranks Test are the two most common choices, and both work directly on the differences between paired observations rather than the raw scores themselves.

The sign test: comparing direction of change 🔗

The Sign Test is, by design, the most stripped-down non-parametric option available for paired data. It ignores how large a change is and looks only at its direction, whether a score went up or down between the two measurements.

How the sign test works 🔗

The procedure is straightforward. For every pair of observations, calculate the difference between the second score and the first. If the difference is positive, record a plus sign; if negative, record a minus sign; if there is no change at all, that pair is dropped from the analysis. Under the null hypothesis, the number of plus signs and minus signs should be roughly equal, since neither direction is expected to dominate by chance. The test statistic is simply the smaller of the two sign counts, and this is compared against a critical value from the binomial distribution, or a normal approximation once the number of pairs is reasonably large.

A classroom example 🔗

Suppose a researcher records reading comprehension scores for 15 students before and after a six-week reading programme. Rather than analysing the actual score gains, the Sign Test only asks how many students improved and how many declined. If 12 students improved, 2 declined, and 1 showed no change, the test checks whether 12 out of 14 usable pairs is an outcome unlikely to happen by chance alone. This kind of before-and-after comparison on the same group is exactly the setting the test was built for, and it applies equally well to attitude ratings, behavioural checklists, or any paired outcome where only the direction of change is meaningful.

Strengths and limitations of the sign test 🔗

The appeal of the Sign Test lies in its simplicity. It makes almost no assumptions about the underlying distribution, works with ordinal data, and can even be used when exact scores are unavailable but direction of change is known. That simplicity, however, comes at a cost. Because it discards information about how much a score changed, it is considerably less sensitive to real effects than tests that use magnitude. A student who improved by one mark and one who improved by twenty marks count exactly the same in a Sign Test, which is a real loss of statistical power, especially with small samples typical of classroom-based research.

The Wilcoxon matched-pairs signed-ranks test 🔗

Frank Wilcoxon designed this test to capture what the Sign Test leaves out: the magnitude of change, not just its direction. It remains distribution-free with respect to the raw scores, but it does assume that the differences are drawn from a roughly symmetric distribution, which makes it a genuine step up in the amount of information it uses.

Step-by-step: how the wilcoxon test works 🔗

The process builds directly on the logic of the Sign Test but adds a ranking stage. First, calculate the difference for every pair, exactly as before, and discard any pairs with a difference of zero. Next, take the absolute value of each remaining difference and rank them from smallest to largest, ignoring their original sign. Once ranked, reattach the original sign to each rank. Finally, sum the ranks that carry a positive sign and separately sum the ranks that carry a negative sign. The test statistic is based on comparing dependent samples using these ranked differences rather than the raw values, and the smaller of the two rank sums, often labelled W, is checked against Wilcoxon’s tables or, for larger samples, approximated using a normal distribution.

Why the wilcoxon test is more powerful 🔗

Because it incorporates the size of each difference through ranking, the Wilcoxon test can detect effects that the Sign Test would miss entirely. If most students show small negative changes but a handful show very large positive ones, the ranking process reflects that imbalance in a way that a simple sign count cannot. This is precisely why it is treated as the non-parametric counterpart to the paired-samples t-test and is generally preferred over the Sign Test whenever the data allows it.

Applying it to an education research example 🔗

Consider a study assessing whether a new instructional method improves numeracy skills in a group of 25 students, measured on a 0 to 20 scale before and after the intervention. If a Shapiro-Wilk test shows the score differences are not normally distributed, the paired t-test’s normality assumption is violated. This is a common situation in education and social science research, where ordinal scales and non-normal distributions are the norm rather than the exception, and it is exactly where the Wilcoxon test earns its place in a thesis methodology chapter. Rather than discarding the informative pre-post gap in scores, the researcher can rank the gaps and still test whether the improvement is statistically meaningful.

Choosing between the sign test and the Wilcoxon test 🔗

Deciding which test to use is not simply a matter of preference. A few practical questions can guide the choice.

What kind of data do you have? 🔗

If the outcome is purely ordinal, or if only the direction of change can be meaningfully judged, such as a preference between two teaching aids, the Sign Test is the more honest choice, since it does not pretend to measure magnitudes that were never really available. If the data is interval or ratio, such as marks out of 100, the Wilcoxon test extracts more information from the same dataset.

Does the assumption of symmetry hold? 🔗

The Wilcoxon test assumes the distribution of differences is roughly symmetric around the median. This is why it is best suited to examining within-group differences where scores are expected to shift in a broadly consistent pattern, such as comparing pre-test and post-test ranks for the same group. If this symmetry assumption looks shaky, and especially with very small samples, the Sign Test remains the safer fallback.

How much statistical power do you need? 🔗

Given a genuine choice between the two, the Wilcoxon test is almost always preferred for its greater sensitivity to real differences. Many statistical software packages, including SPSS, default to the Wilcoxon Signed-Ranks Test under the two related-samples non-parametric menu for exactly this reason.

Why this distinction matters for your research 🔗

For an M.Ed or Ph.D scholar analysing intervention studies, action research, or programme evaluations, the choice between these two tests is not a technicality buried in an appendix. It directly affects whether a modest but real improvement in student outcomes gets detected or gets lost in an underpowered analysis. Reporting that a Sign Test was chosen because only directional data was available, or that a Wilcoxon test was used because interval-scaled scores permitted a more sensitive analysis, signals to examiners that the choice of statistical tool was deliberate rather than default. That kind of methodological clarity strengthens a dissertation’s data analysis chapter considerably.

What do you think? If you were evaluating a classroom intervention with a small, non-normal sample of paired scores, would you be comfortable trading some statistical power for the simplicity of the Sign Test, or would you push for the Wilcoxon test even with the extra assumptions it carries?

References

1. https://sphweb.bumc.bu.edu/otlt/MPH-Modules/BS/BS704_Nonparametric/BS704_Nonparametric5.html
2. https://statistics.laerd.com/spss-tutorials/sign-test-using-spss-statistics.php
3. https://www.technologynetworks.com/informatics/articles/the-wilcoxon-signed-rank-test-370384
4. https://www.sciencedirect.com/topics/medicine-and-dentistry/wilcoxon-signed-ranks-test
5. https://www.statisticssolutions.com/free-resources/directory-of-statistical-analyses/how-to-conduct-the-wilcox-sign-test/
6. https://usq.pressbooks.pub/statisticsforresearchstudents/chapter/paired-samples-wilcoxon-sign-rank-test/