Quantitative market research: who answered matters more than how many
Quantitative market research counts how many and how much. Pick a method, write countable questions, size a survey with a margin of error table and report it honestly.
Quantitative market research measures how many people do, think or would buy something, and how much. The common methods are surveys, your own sales and product data, experiments such as A/B tests or preorders, and public statistics. The numbers are only as good as the questions you asked and the people who answered.
So two things matter more than sample size: knowing what you are counting (which usually comes from interviews first) and knowing who answered. Get either wrong and a bigger sample just gives you a more precise wrong answer.
A worked example: after-hours repairs
Here is a made-up example. You sell maintenance-request software to property management companies. Earlier interviews showed three ways companies handle after-hours repair requests: an answering service, an on-call staff member, or voicemail until morning. You want to know how common each is before deciding which to build for first.
You have a list of 1,500 property management companies from a trade event, and you plan to email them a short survey.
The main quantitative methods
In Nielsen Norman Group's overview of research methods, quantitative data is gathered through an instrument, such as a survey or analytics tool, and analyzed mathematically. It answers how many and how much better than qualitative work does.
| Method | What it measures | Main limit |
|---|---|---|
| Surveys | How common a view, behavior or setup is | People report what they think they do |
| Your sales and product data | What existing customers actually did | Silent on people who never became customers |
| Experiments and A/B tests | Whether a change makes people act differently | Needs enough traffic to read a result |
| Preorders, pilots and pricing tests | Whether people pay, as opposed to saying they would | Slower; needs a believable offer |
| Public statistics | Counts of businesses, workers or households | Rarely broken down to your exact customer |
For the property managers, use a survey. Most of them aren't customers yet, so your own data can't count their setups.
Know what you are counting before you write questions
The American Association for Public Opinion Research (AAPOR) best practices start by asking whether a survey is the right method at all and whether existing data already answers the question. For the questions themselves, AAPOR recommends asking about one concept at a time, using words your audience understands, and making closed answer options mutually exclusive and complete, with "don't know" where it fits.
This is where the interviews pay off. You list the three known setups as answers, plus "other" with a text box. Without the interviews you'd be guessing, and a missing option quietly pushes people into the nearest wrong one.
AAPOR also recommends pretesting, typically through cognitive interviews: people like your respondents answer the questions aloud and explain how they read them. Does "after hours" include weekends? You think so. Some property managers won't.
How many responses you need
The margin of error describes how far a survey result might be from the true value because you asked a sample instead of everyone. Pew Research Center explains that a margin of plus or minus 3 points at the 95% confidence level means that if the same survey were run 100 times, the result would be within 3 points of the true value about 95 of those times.
For a simple random sample, the 95% margin of error on a percentage is roughly 1.96 × √(p × (1 − p) ÷ n), where n is the number of responses and p is the result. It is largest when p is 50%. We calculated it for a few sample sizes using that worst case:
| Responses | Margin of error at 95% confidence |
|---|---|
| 50 | ±13.9 points |
| 100 | ±9.8 points |
| 200 | ±6.9 points |
| 400 | ±4.9 points |
| 1,000 | ±3.1 points |
Going from 100 to 400 responses roughly halves the margin. Going from 400 to 1,000 buys much less. Pew also notes that subgroups have larger margins, and the margin for a difference between two groups is generally about twice the margin for each. If 400 companies answer and you compare the 100 with fewer than 50 units against the rest, each small-company estimate carries a margin near ±10 points.
Who answered matters more than how many
That table assumes a random sample, where everyone had a known chance of being picked. AAPOR calls that a probability sample. Opt-in panels and people recruited through social media or personal networks are nonprobability samples, which often need special statistical techniques and a clear account of the method.
The trade-event list is a nonprobability sample. It over-represents companies that go to events, and the ones that reply may be the ones with an after-hours problem worth complaining about. Pew notes that sampling error is only one source of error: who can be reached, who responds and question wording all affect results, and the reported margin rarely includes them.
So if 400 of the 1,500 companies reply and 32% say they use an answering service, write "32% of the 400 companies that responded," not "32% of property managers." The math gives about ±4.6 points for a random sample, but it does nothing about who chose to answer. If the decision hangs on the exact share, check it against a second source, such as a trade association.
AAPOR's transparency guidance lists what to disclose with results, including the sample size, the population, how the sample was recruited, the survey mode and the full question wording. For an internal decision, a short note covering who you asked, how many answered and the exact questions does the job.
Go back to interviews for the reasons
Say the survey also shows 40% of respondents let after-hours calls go to voicemail. The survey can't tell you whether that is a deliberate cost decision or a problem nobody has had time to fix. Call a few of those companies and ask them to walk through the last after-hours request they handled. Interviews versus surveys and checking interviews against product data cover combining the two.
For pretesting or follow-up calls, the procurement professionals in property management directory page shows the kinds of people you might invite. You can also search for people by describing who you need, and review who it finds before anything is sent.
The two jobs suit different formats on Instant Expert. A cognitive pretest needs someone reading the questions aloud, so book a 15 or 30 minute call. A follow-up on one survey result, such as why after-hours calls go to voicemail, can be a single question answered in writing or by voice note on the person's own time. Either way you set what each person receives and our fee (20% of what you pay) is added on top: a $100 offer costs you $125, charged only when they book or answer.