Reading the Error Bars: A Boardroom Decoder for UK Survey Significance
Photo: Titanic Belfast, CC BY 2.0, via Wikimedia Commons
A single line in an insight pack can travel a long way. '62% of UK finance directors back the proposal, up from 58% a year ago' appears on a slide, is repeated in a steering meeting, and within a fortnight has become the evidential basis for a budget decision. Somewhere in the appendix, in eight-point type, sits a caveat: base 214, margin of error plus or minus 6.7 percentage points. The two statements are not compatible, and nobody in the room has been handed the means to notice.
This is not a story about dishonest research. It is a story about a gap between how survey evidence is produced and how it is consumed. Suppliers are careful with their caveats; buyers are busy; and the numbers that survive the journey from appendix to action are almost always the ones without error bars attached. Closing that gap is less about statistical training than about a handful of questions asked consistently, at the right moment.
What the margin of error actually measures
The margin of error describes sampling uncertainty: the range within which the true figure for the population would fall if the same survey were repeated many times, assuming a perfectly random sample and no other sources of error. It is a narrow claim dressed up as a broad one. It says nothing about whether the right people were interviewed, whether the question was understood as intended, whether non-respondents differ systematically from respondents, or whether the panel's recruitment method skewed the pool before the first question was asked.
Those omissions matter. Total survey error is the sum of sampling error and everything else, and in commercial research the everything else is frequently the larger part. A perfectly executed probability sample of 2,000 with a badly worded question about intention to switch suppliers will still mislead. A skewed online panel with a tight confidence interval is precisely wrong.
There is a second, quieter issue: the interval applies to the estimate, not to the comparison. A finding that support rose from 58% to 62% is not a finding at all unless the difference survives the uncertainty attached to both figures. Two overlapping confidence intervals are the statistical equivalent of two people shouting from opposite ends of a warehouse; proximity is not agreement.
The subgroup problem
Most board-level decisions are not made on the total sample. They are made on a segment: finance directors in mid-market firms, procurement leads in the public sector, decision-makers in Scotland, buyers of a particular category. Each cut reduces the base, and the margin of error widens quickly.
| Unweighted base | Approximate margin of error (percentage points, 95% confidence) |
|---|---|
| 2,000 | 2.2 |
| 1,000 | 3.1 |
| 500 | 4.4 |
| 250 | 6.2 |
| 100 | 9.8 |
| 50 | 13.9 |
These figures assume a simple random sample and a result near 50%, which is the most conservative case. Real studies do worse. Weighting to correct for region, sector, firm size or job role reduces the effective sample size, sometimes substantially; a design effect of 1.5 means a nominal base of 300 behaves like 200. Clustered or quota-based fieldwork compounds the problem further.
The practical consequence is that many of the most quotable lines in a UK insight pack rest on bases of 60 to 120 respondents, where the interval spans a fifth of the scale. A finding that 'rural customers are more satisfied than urban ones' may be real, but at those bases the survey alone cannot tell you so.
Significant to the statistician, invisible to the customer
Statistical significance answers a narrow question: how likely is a result of this size if there were truly no difference? It is a filter, not a verdict. Commercial significance asks something else entirely: is the effect large enough to justify the cost, risk and disruption of acting on it?
The two frequently diverge. A two-point rise in stated preference can be statistically robust at a very large base and commercially meaningless if the switching cost runs to six figures. Conversely, a consistent pattern of anecdotal feedback from twelve major accounts may carry no statistical significance whatsoever and still be the most decision-relevant evidence in the building.
A useful discipline is to state, before the research is read out, what size of effect would change the decision. If a four-point movement would not alter the investment case, then a survey showing a four-point movement has told the board nothing, however tidy its confidence interval.
A one-page decoder for decision-makers
When a poll-led recommendation lands on the table, six questions will do most of the work:
- What is the base for this specific number? Not the total sample. The base for the cut being quoted, and the unweighted rather than weighted figure.
- What is the interval around each figure being compared? If the intervals overlap, the difference is not established by this evidence alone.
- Has the effective sample size been reported? Weighting and clustering shrink it, and the shrinkage should be disclosed rather than absorbed.
- What would a commercially meaningful difference look like? Written down in advance, ideally.
- What other sources point the same way? Sales data, win-loss reviews, complaint logs, operational metrics. A survey is stronger corroborated than alone.
- What decision is this number being asked to support, and is it reversible? Proportionate evidence for a proportionate decision is the whole game.
None of these questions requires a statistics qualification. They require a chair willing to ask them before the slide is accepted rather than after the budget is committed.
What to write into the next brief
The supply side responds to demand. A brief that asks for unweighted bases, effective sample sizes, question wording, topline data and an explicit statement of what the study cannot support will receive more careful work than one that asks for 'robust, nationally representative insight' and a headline figure. Asking for the limitations section to be presented alongside the findings, rather than buried in the appendix, changes the tenor of the discussion in the room.
The margin of error is not the enemy. It is the most honest number in the pack. The trap is treating it as a footnote when it is, in fact, the qualification that determines how much weight the headline can bear.
