StatisticsInferential Statistics

Knowledge guide

How to Assess Opinion Poll Bias and Sampling Quality

Assess a poll through its target population, sampling frame, selection, response patterns, and uncertainty rather than respondent count alone.

Start with who the poll is meant to describe

An opinion poll estimates an attitude or preference in a target population. A large respondent count is useful only alongside a design that connects those respondents to that population. Distinguish the people you want to describe, the sampling frame you can reach, the selected sample, and the people who actually answer.

Coverage, selection, and nonresponse

Coverage error occurs when the frame misses some of the intended population. Selection bias can arise when inclusion favors people whose opinions differ from those of other eligible people. Nonresponse bias arises when selected people who answer differ in the measured opinion from those who do not.

A short response fraction is a warning to investigate, not a formula for the amount of bias. You need information about the differences between groups and the survey's design or adjustments. Modern standardized response rates also require eligibility and contact-outcome rules; simply dividing returns by invitations may describe participation without being a fully standardized rate.

A new example: a campus transport poll

Suppose a university wants the views of all enrolled students about an evening bus. It sends a survey only to students living in residence halls. Commuters are missing from the frame, so increasing the number of residence-hall replies cannot directly establish what commuters think.

If every enrolled student can be selected but only students enthusiastic about the bus tend to respond, coverage may be adequate while nonresponse remains a concern. Neither scenario supplies enough information to calculate the exact campus-wide support rate. These are hypothetical mechanisms, not collected survey results.

Random sampling error is different

Repeated probability samples can yield different estimates even when the design is sound. That random variation is sampling error. Coverage and selective nonresponse are separate sources of error. A small conventional sampling margin of error does not certify freedom from them, and it should not be mechanically attached to an opt-in or quota sample as though selection were random.

Quotas and random strata

Quota sampling targets subgroup counts but generally permits nonrandom selection within groups. Stratified random sampling uses a probability-based selection within strata. Matching demographic proportions can be useful, but it does not establish representativeness for every opinion. Weighting can help with measured differences under suitable assumptions; it cannot guarantee correction for every missing factor.

Questions to ask before trusting an estimate

Who is the target population? Who could be selected? How were people selected within groups? What happened to selected nonrespondents? How were estimates adjusted? Keep respondent counts, participation, selection method, and uncertainty together when reporting results.

Related question

Apply this knowledge

Use the concept guide to understand the reasoning, then return to the complete question and worked answer.

Opinion Poll Bias: The 1936 Literary Digest Sample

Sources

These references support the core concepts and interpretation boundaries explained above.