The Cost of a Confident Guess: Auditing DIY Survey Evidence in UK SMEs

UK small and medium-sized enterprises have never had easier access to the machinery of survey research. A template, a link, a modest media budget and a dashboard can be assembled in a morning. The result often looks like evidence: clean charts, percentage-point differences and a summary that can be pasted into a board pack. That polish is precisely the risk. A survey is not evidence because it carries a percentage sign. It is evidence when the sample, the questions and the analysis can bear the weight of the decision.

At Jordan Research, the practical concern is not that DIY platforms exist. They are useful tools. The concern is that they encourage a category error: treating a quick consultation with a self-selecting group as if it were a measured estimate of a market. For a UK SME weighing a pricing change, a new location, a product launch or a shift in service model, that error can be expensive. The savings on research are rarely large enough to offset a decision made on misleading data.

Sampling: who answered, and who did not

The most common weakness is not a badly worded question. It is an unrepresentative respondent base. A survey promoted through a company newsletter, a social media post or a customer list will over-represent the engaged, the loyal and the digitally comfortable. It will under-represent lapsed customers, prospects, quieter segments and people who have no reason to care about the brand. That matters when the decision depends on the size of a market or the preferences of a group the business does not already know well.

Probability sampling is not a luxury. It is the mechanism that allows a result to be projected beyond the people who replied. Without it, a survey can still generate useful anecdotes and hypotheses, but it cannot reliably tell a UK SME that 62 per cent of its target market will pay more for a service. The number may be accurate for the people who answered. It is not a market estimate.

Non-response is the quieter problem. Even a well-drawn sample becomes less trustworthy when particular groups decline. A short field period, an incentive that appeals only to some customers, or a topic that feels irrelevant to busy owners can all skew the profile. DIY dashboards rarely show a non-response analysis. They show completion counts. Those are not the same thing.

The leading question problem

Question design is where confidence is manufactured. A prompt such as 'How much would you value a faster service?' invites agreement. A prompt such as 'Would you pay £15 more for next-day delivery?' asks a sharper, more decision-relevant question. The difference is not cosmetic. It changes the cognitive task from expressing a vague preference to making a trade-off.

Common DIY errors include double-barrelled questions, unbalanced answer scales, missing 'don't know' options and brand-loaded language. Each one nudges the respondent. The resulting data can look statistically tidy while measuring little more than the wording of the instrument. For SMEs, the risk is acute because the survey is often designed by the person who most wants a particular answer. That is not a criticism of intent. It is a structural bias.

A useful discipline is to draft the decision first, then the question. If the decision is whether to introduce a paid support tier, the survey must test willingness to pay, current workarounds, switching behaviour and the size of the unserved need. If the decision is whether to open a second site, the survey must capture catchment, travel tolerance and competing alternatives. A general satisfaction score will not do.

The illusion of statistical significance

DIY platforms increasingly offer significance testing, confidence intervals and segment comparisons. These are valuable features in the right setting. They are also easy to misread. A difference between two small subgroups can be presented as significant when the underlying sample is too thin to support a stable conclusion. The language of statistics can create an aura of rigour that the design does not deserve.

Statistical significance is not the same as commercial significance. A two-point difference may be statistically detectable in a large sample and operationally irrelevant. A ten-point difference may be commercially important but too uncertain to act on because the sample is small. UK SMEs need both readings: the precision of the estimate and the size of the effect that would change the decision. A dashboard rarely frames the question that way.

There is also the multiple-comparisons trap. Slice a dataset by region, age, sector and tenure, and some differences will appear by chance. Without a pre-specified analysis plan, the most dramatic finding may be the least robust. That is not a reason to avoid segment analysis. It is a reason to treat exploratory cuts as hypotheses rather than verdicts.

A decision framework for UK SMEs

DIY research is fit for purpose when the decision is low-stakes, reversible and internally focused. Examples include gathering feedback from a known customer panel, testing whether a proposed message is intelligible, generating a list of objections for a sales team, or running a quick pulse check after a service change. In these cases, speed and proximity to the customer may matter more than representativeness. The output should be labelled as indicative, not projected.

Commissioned or professionally designed research becomes the better route when one or more of the following applies: the decision is difficult to reverse; the financial exposure is material relative to turnover; the result will be used in a regulatory, investor or board context; the target audience is unfamiliar or hard to reach; the business needs to size a market or segment; or the finding is likely to be contested. In those settings, the cost of a flawed survey is not the survey fee. It is the cost of the wrong price, the wrong site, the wrong product or the wrong message.

A simple test is to ask what would happen if the survey were wrong by ten percentage points. If the answer is 'we would adjust next month', DIY may be adequate. If the answer is 'we would have committed to a lease' or 'we would have mispriced a contract', the research needs a stronger design.

Making DIY honest

Where SMEs do use DIY tools, they should impose a few governance rules. State the decision the survey will inform. Document how respondents were recruited. Report the profile of the achieved sample against the target population. Avoid projecting results beyond the respondent base. Treat subgroup findings as exploratory unless the sample supports them. Keep the questionnaire and raw data so that a later review can see what was actually asked. These practices do not turn a convenience sample into a probability sample, but they reduce the risk of overclaiming.

The bottom line

The DIY research delusion is not that small firms should never run their own surveys. It is that a familiar interface can make an unfamiliar methodological problem feel solved. UK SMEs need inexpensive insight as much as anyone. They also need a clear line between exploration and evidence. Drawing that line before the survey goes live is cheaper than discovering it after the decision has been made.