Biased survey questions: leading, loaded and double-barreled examples (and fixes)

Biased survey questions, with examples: leading, loaded, double-barreled, agree-disagree and lopsided scales, plus how to rewrite each so the answers mean something.

Instant Expert EditorialPublished Updated 7 min read

Biased survey questions nudge people toward one answer, and the nudge is usually small: a hopeful adjective, a hidden assumption, two questions jammed into one, or answer options that lean positive. The fix is almost always to remove the hint and ask one thing at a time, with options that give every reasonable answer an equal chance.

Small nudges move big numbers. In a Pew Research Center experiment described in its guide to writing survey questions, 68% of respondents favored military action in Iraq when asked plainly, but only 43% favored it when the question added "even if it meant that U.S. forces might suffer thousands of casualties." Same topic, one added clause, a 25-point swing. Your pricing survey is open to the same effect.

Six kinds of biased question, with fixes

TypeBiased exampleNeutral rewrite
Leading: the question suggests the answer"How much time does our scheduling tool save you each week?""How, if at all, has the scheduling tool changed the time you spend on bookings?"
Loaded: the question assumes something that may be false"What do you find most frustrating about your current billing software?""Do you use software for billing? (Yes / No)" then, for Yes: "What, if anything, would you change about it?"
Double-barreled: two things, one answer"How satisfied are you with our pricing and support?"Two questions: one about pricing, one about support
Agree-disagree statement"Our product is easy to use. (Agree / Disagree)""How easy or difficult is the product to use? Very easy / Somewhat easy / Somewhat difficult / Very difficult"
Unbalanced scaleExcellent / Very good / Good / PoorExcellent / Good / Fair / Poor, or a scale with equal positive and negative options
Pleading context"We're aiming for five stars. How would you rate us?""How would you rate your experience?"

Double-barreled questions get one answer to two questions

"How satisfied are you with our pricing and support?" is the classic double-barreled question. Someone who loves your support and resents your prices has no honest answer, so they pick something in the middle and you learn nothing about either.

NN/g's survey best practices uses "How easy and intuitive was this website to use?" as its example: people answer one word or average the two, and you can't tell which. Pew and NN/g both say to split them.

More double-barreled question examples, all made up, with the split versions:

Double-barreledSplit into
"Was checkout fast and secure?""How fast was checkout?" and "How secure did checkout feel?"
"How likely are you to renew and recommend us?""How likely are you to renew?" and "How likely are you to recommend us?"
"Are the reports accurate and useful?""How accurate are the reports?" and "How useful are the reports?"
"Should we add dark mode or a mobile app?"Ask people to rank the two

To find them, search your draft for "and" and "or" joining two things to be rated.

Leading questions and lopsided scales

  • Leading questions include or imply the desired answer, as NN/g defines them. Name the topic without naming a feeling: "How might this affect your efficiency, if at all?"
  • Agree-disagree statements. Some people agree with whatever statement they are shown, which Pew calls acquiescence bias, so it recommends a choice between alternative statements. AAPOR adds that people may lean toward "agree" to please the person asking.
  • Unbalanced scales. Three positive options and two negative ones push people toward a positive answer (NN/g). Keep the sides equal.
  • Pleading context. NN/g's example opens with "We are committed to achieving a 5-star satisfaction rating." Anyone with a mediocre experience now feels rude saying so. Delete the sentence.

Loaded words carry an argument inside them

Pew reports that 51% of respondents favored "making it legal for doctors to give terminally ill patients the means to end their lives," but only 44% favored "making it legal for doctors to assist terminally ill patients in committing suicide."

In business surveys the loaded words are quieter: "outdated," "clunky," "manual," "AI-powered." "Paper forms" is a description. "Outdated paper forms" is an argument. AAPOR also warns against presenting only one side: "Would you pay more for faster delivery?" does. "Which matters more to you: lower delivery fees or faster delivery?" doesn't.

Earlier questions change how people read later ones

Pew found that 88% of people said they were dissatisfied with the way things were going in the country when that question came right after one about approval of the president, compared with 78% when it came without that context. In a business survey, five questions about a competitor's problems will color the satisfaction question that follows. AAPOR recommends general questions before specific ones on the same topic.

A worked example: four biased questions, rewritten

Here is a made-up example. A founder building appointment-reminder software for dental clinics drafts four questions for office managers:

  1. "How much revenue do no-shows cost your practice each month?" (assumes a cost most managers haven't calculated)
  2. "How satisfied are you with your current reminder and scheduling system?" (double-barreled)
  3. "Automated text reminders would reduce our no-shows. (Strongly agree to Strongly disagree)" (asks people to agree with the founder's solution)
  4. "Would you consider switching to a better reminder tool?" (calls the alternative "better")

A neutral version:

  1. "In the past four weeks, roughly how many scheduled appointments were missed without notice?" None / 1-5 / 6-15 / 16-30 / More than 30 / Don't know
  2. "How do you currently remind patients about appointments?" (options from a few earlier conversations, plus "Other")
  3. "How well does your current reminder method work for your practice?" Very well / Somewhat well / Not very well / Not at all well
  4. "In the past year, have you looked at other ways of reminding patients?" Yes / No / Don't know

The second draft can't tell the founder that dentists want the product. It tells them how common missed appointments are and whether anyone wants a change, which is more useful.

A ten-minute bias check before you send

For each question, ask:

  • Does it contain an adjective that carries an opinion?
  • Does it assume something about the respondent that might be false?
  • Does "and" or "or" join two things to be rated?
  • Could someone who dislikes the product answer it comfortably?
  • Are the scale's sides equal, with "Don't know" or "Not applicable" where needed?
  • Would an earlier question change how someone reads this one?

Then pretest, as AAPOR recommends: have a few people like your respondents explain out loud how they understood each question. And check who gets it. A neutral survey sent only to your happiest customers is still skewed (market research survey covers sampling, and types of survey questions covers which format to use for each question).

Your next step

Run your draft through the six-type table and rewrite anything that matches. If the neutral version of a question keeps turning out to be "why?", run interviews instead (interviews vs. surveys). For the dental founder, that means office managers at independent dental practices; operations professionals in dental services is one place to start. On Instant Expert you can ask each person for a 15-minute call or for a written or voice answer to one open question, such as "How do you currently remind patients about appointments?" You set what each person receives and our fee (20% of what you pay) is added on top, so a $40 offer costs you $50, charged only if they book or answer (how it works). Keep the invitation as neutral as the survey: a note that calls your product "innovative" leads people before they have said a word.