'Don't you agree the council is doing a terrible job?' versus 'How would you rate the council?' Same topic, wildly different answers. A survey question can lead a witness — and most do, by accident.
A survey gathers data by asking people questions. Its accuracy depends on three things: who you ask (sampling), what you ask (question design), and who actually answers (response rate).
Politics, market research, public health, customer feedback, social science — surveys are how organisations 'hear' large populations. Badly designed ones produce confident nonsense.
Which question is least likely to bias the answer?
Question-design traps
- Leading questions — phrasing that suggests the answer ('Don't you agree…?').
- Loaded words — emotionally charged terms ('death tax', 'pro-life').
- Double-barrelled — two questions in one ('Is the food cheap and tasty?').
- Vague scales — 'often / sometimes / rarely' mean different things to different people.
- Order effects — earlier questions colour later answers.
What's wrong with: 'How satisfied are you with our fast, friendly service?'
Why do response rates matter as much as sample size?
If 90% of those contacted refuse, the 10% who answered are unusual — more motivated, more opinionated, often angrier. Non-response bias can wreck a survey with a huge nominal sample. A small survey with a high response rate often beats a big one with a low one.
Treating a self-selected online poll as a real survey. 'Vote in our poll!' attracts the motivated and the organised, not a random cross-section. It measures who clicked, not what the public thinks.
Before believing a survey result, ask: *who was sampled, how were the questions worded, and what was the response rate?* If the article won't tell you, be sceptical.
- Survey quality = sampling + question wording + response rate.
- Avoid leading, loaded, double-barrelled, and vague questions.
- Self-selected online polls aren't surveys — they measure who bothered to click.