Precision Does Not Remove Bias

The concept

Random sampling uncertainty and systematic bias are different sources of error. Under suitable assumptions, increasing sample size can improve precision. It cannot repair a sample that systematically excludes relevant people or a measurement process that consistently records the wrong quantity. Before asking how many responses you have, ask how those responses were produced.

Worked example

A fictional community survey asks about evening bus service only through a daytime email newsletter. Ten thousand replies may yield a stable percentage among respondents, but people who work evenings or rarely use email may be underrepresented. Collecting another ten thousand replies through the same channel can sharpen the wrong picture. The organizers add other contact methods and report how recruitment affects the population their result can describe.

Limits and judgment

Correcting representation is not always easy. Weighting depends on information and assumptions, and nonresponse can remain related to the outcome. Report recruitment, missingness, and measurement limits alongside numerical uncertainty. A small well-designed study can sometimes answer a bounded question more credibly than an enormous convenience sample whose connection to the target population is unclear.

RETRIEVE. REASON. REMEMBER.

Knowledge check.

Choose one answer for each question, then check your reasoning.