The Quiet Margin: Why Non-Response Deserves a Place in the Board Pack
Photo: IAEA Imagebank, CC BY 2.0, via Wikimedia Commons
Most board discussions about survey evidence begin with the sample size. A figure of 1,000 UK adults sounds reassuring; 2,000 sounds stronger still. The margin of error is quoted, the confidence interval noted, and the conversation moves to the headline finding. Yet the most consequential number in the pack may be the one that never appears: the proportion of the intended sample that did not take part.
Non-response is not a rounding issue. It is a bias issue, and it operates before the analyst opens the dataset. The people who decline, drop out or are never reached are not a random subset of the population. They differ in ways that often matter to the question being asked. A board that reads only the completed interviews is reading a partial transcript and treating it as the whole conversation.
Three kinds of silence
Fieldwork produces at least three distinct absences. The first is non-contact: people who were selected but never reached. The second is refusal: people who were reached and declined. The third is attrition: people who began but did not finish. Each carries different information.
Non-contact is shaped by channel. Online panels can under-represent older and less digitally engaged groups. Telephone surveys confront declining answer rates and caller identification. Face-to-face work is expensive and geographically concentrated. In business-to-business research, gatekeepers and shared inboxes mean the buying committee may never see the invitation. Refusal is shaped by salience: a survey about financial security will attract some people and repel others. Attrition is shaped by burden: long questionnaires lose the time-poor first.
None of this is noise to be averaged away. It is evidence about who is outside the room.
Why weighting is not a cure
Weighting is necessary, but it is not a magic remedy. It adjusts the margins to match known totals. If the missingness is unrelated to the outcome, weighting can reduce bias. If the people who are absent differ on the very thing being measured, weighting by age, sex and region may leave the bias intact.
Consider a UK survey on household financial resilience. If people under financial stress are less likely to complete a long questionnaire, the achieved sample will over-represent the comfortable. Weighting to population totals on age and region will not fix that if financial stress cuts across those categories. The same logic applies to health, housing, employment and technology adoption. The relevant benchmark is not only who the population is, but who the population is on variables connected to the research question.
A simple benchmark diagnostic
Before the findings are presented, compare the achieved sample with independent benchmarks. In the UK, the ONS Census 2021 provides detailed population totals; the Labour Force Survey offers employment and economic activity data; the ONS Opinions and Lifestyle Survey and Ofcom research provide regular attitudinal and digital-use benchmarks. The point is not to chase perfect representation. It is to identify material departures and ask what they might mean.
A practical diagnostic has three steps. First, profile the achieved sample on core demographics and on topic-relevant variables such as tenure, employment status, health status or internet use. Second, compare those profiles with the best available independent benchmarks, noting where deviations exceed what the design would lead you to expect. Third, examine paradata: the number of contact attempts, the time of day, the device used and the difference between early and late responders. Late responders are often closer to non-responders in their characteristics, so a comparison between the two can be informative.
The diagnostic does not produce a single correction factor. It produces a defensible account of who is missing and why it matters.
Three questions for the board to ask
When a supplier presents survey evidence, the board should ask three questions about the quiet margin.
First: can you show the disposition of every case in the issued sample, including non-contact, refusal, ineligible and unknown eligibility? A completed-interview count is not enough. The board needs to see the whole fieldwork funnel.
Second: which independent benchmarks did you use to profile the achieved sample, and why those? Did the profiling include variables related to the topic, not just standard demographics? A supplier that can answer this has thought about non-response; one that cannot has likely left it to the weighting stage.
Third: what does your non-response analysis show, and how has it changed the weighting, the reported margins or the language of the findings? The answer may be that the bias is modest. That is a legitimate answer, provided it is evidenced. What is not legitimate is silence about silence.
What to do with the answer
Boards should not demand zero non-response. That is neither achievable nor affordable in most UK fieldwork. They should demand transparency about it. A short non-response note, appended to the main findings, can record the achieved sample profile, the benchmarks used, the scale of any deviations and the implications for interpretation.
Where the risk is material, consider design changes: mixed-mode fieldwork, targeted follow-up, shorter instruments or incentives aimed at under-represented groups. These are procurement decisions as much as research decisions. They belong in the brief, not in the post-hoc apology.
The central point is counter-intuitive but simple. The people who did not answer are still telling you something. They are telling you who is missing, and that is evidence the board can use before it reads the results.
