Two Truths in One Board Pack: Separating Supplier Method from Market Movement

A board pack lands with two consumer surveys inside it. One, a syndicated tracker, puts category confidence at 34 per cent. The other, a bespoke agency study fielded three weeks later, puts it at 43. The conversation moves immediately to what has changed in the market. It rarely pauses to ask whether anything changed at all. Two reputable suppliers, asking roughly the same question of roughly the same public, produced numbers that differ by nine points — and nobody in the room is holding a note explaining why.

That gap has a name worth using in the boardroom. A house effect is a persistent, directional offset attributable to the supplier's instrument rather than to the population being measured. It is not error in the sense of a mistake, and it is not sampling variance, which is random and shrinks with sample size. It is a stable characteristic of how a particular house collects and reports its evidence. Recognising it is the difference between reading a market and reading a method statement.

Why two defensible numbers can coexist

It is tempting to assume that one of the two figures must be wrong. In practice, both can be defensible and neither need be recent. A survey is an instrument, and instruments carry signatures. A telephone study with interviewer probing will read differently from a mobile-completion online study, even where the questionnaire is identical. A trade body survey of its own members will read differently from a nationally weighted panel, because the population differs before a single question is asked.

The board's task is not to adjudicate which supplier is better. It is to establish whether the difference between the two numbers is a property of the market or a property of the measurement. The four checks below are the practical version of that question.

Check one: who was asked, and how

Begin with the sample frame. An online access panel recruited through a rewards scheme, a random-probability address sample, a directory of named finance directors and a trade association membership list are four different populations wearing the same label. Coverage matters most where the behaviour being measured is unevenly distributed — among older consumers, among small businesses, among the self-employed.

Fieldwork mode then shapes the answer. Interviewer-administered surveys tend to produce more socially desirable responses on questions about health, finance and sustainability. Self-completion on a mobile handset invites satisficing: short attention, straightlining, midpoint drift. Where one supplier mixes modes and the other does not, expect an offset before you expect a trend.

Check two: the wording and the scale

The most common source of a spurious nine-point gap is a scale that was never identical. A five-point agreement scale with a labelled midpoint behaves differently from a seven-point scale without one. Offering a don't know option explicitly depresses the proportion expressing a view; omitting it forces a lean. Prompted awareness produces higher figures than spontaneous recall, by design rather than by insight.

The defensive move is straightforward: put the two questionnaires side by side and read the stems aloud. If one asks whether the respondent is confident about the next twelve months and the other asks about the year ahead, they are not the same question, however similar the tabulated output looks.

Check three: weighting and the base

Weighting is where comparability is quietly won or lost. Two suppliers may both rim-weight to ONS mid-year population estimates and still diverge, because one trims extreme weights more aggressively, weights on additional attitudinal variables, or applies a design effect that widens the reported interval. Where a figure is drawn from a sub-group of four hundred respondents, the interval around it is wider than the headline table implies.

Ask, too, whether the published number is a single wave or a rolling average of several. A three-wave average will lag real movement, which is a virtue in a volatile category and a nuisance when the board is trying to read the last fortnight.

Check four: reporting conventions

Net agreement, top-two-box and mean scores are three different summaries of the same underlying distribution and they do not rank respondents identically. Rounding cascades can shift a total by a point. Net promoter scores are comparable between suppliers only where the scale and the question structure match exactly, which is less often than the market assumes. A percentage expressed on a filtered base is not comparable with one expressed on all respondents.

A one-page house effect note

Before either number reaches a board paper, attach a short annex covering six lines: sample source and any known coverage limitation; fieldwork mode and dates; the exact question wording and scale; weighting variables and base sizes; the reporting convention used; and the interval around the figure. Where two suppliers are being compared, the same six lines should be completed for each. The annex is unglamorous and it will prevent at least one unnecessary strategy discussion a year.

Reconcile rather than choose

Having completed the checks, a board has three sensible options. It can report both figures with an explicit method note and a stated range, which is honest and often sufficient. It can ask both suppliers for a short bridging exercise — the same three questions fielded on both instruments — which isolates the offset empirically rather than by argument. Or it can treat the midpoint of the two as a working assumption, though averaging methods removes neither bias nor mode effect and should be labelled as a convenience, not a correction.

What a board should resist is the quiet choice of the more congenial number, or the more familiar supplier, on the grounds that the other one is anomalous. Anomaly is a claim requiring evidence.

When the gap really is movement

None of this argues that differences are never real. Two signals deserve attention: where both instruments move in the same direction across consecutive waves, and where a gap widens rather than persists at a stable level. A stable offset is a house effect; a widening offset is a candidate trend. Fieldwork dates matter here too — a three-week gap in a fast-moving category can produce genuine movement rather than methodological noise.

The discipline Jordan Research would press for is modest but consequential. Every number in a board pack should arrive with enough method attached to be interpreted, and where two numbers disagree, the disagreement itself should be explained before it is acted upon. A supplier's signature is not a market signal. Boards that can tell the two apart make better decisions on the evidence they already hold.