Stop Flying Blind - A book by Michael Mace

9. Quantitative market research

The three perspectives a company needs in order to map the future are competitive analysis, market research, and advanced technology analysis. This week we continue our look at market research with a discussion of quantitative research – surveys and other studies that give you statistically reliable numbers. Or that would, if they were conducted properly…

Uses of quantitative research

Because quantitative research gives you accurate numbers, it can be used to keep score for your business. How many people are aware of your products? What do they like and dislike about them? Do they like your products better than the competition’s? Do they plan to buy from you in the next six months?

All of these things are relatively easy to measure, but make sure the survey is very well constructed. These statistics are basically report cards on the work being done by the company’s marketing and product teams. If you find bad news, you need to be sure it’s completely accurate. Besides screwing up the company’s decisions, producing flawed research can easily get you fired.

One of the most important statistics your company will want to track is purchase intent. If you can gaze into the future and estimate how many people will buy your products in the next quarter or year, those figures can be driven straight into business plans and sales goals. But purchase intent is also one of the toughest numbers to interpret. There’s a long path from someone thinking about buying a product to actually purchasing it, and any interruption along that process can throw off your findings. I’ve seen studies that showed rising purchase intent even though actual sales were dropping. It’s best to use this sort of research to check for potential warning signs of trouble, but don’t let good results lull you into a false sense of security, and be very careful about building these figures into the business plan.

It’s also commonplace to use surveys to test things like reactions to new products and new pricing. Like purchase intent, this research can be very tricky research to interpret, because conditions almost always change from the time you conduct a study until the time you take action on it. For example, in the computer hardware industry we usually set the pricing for fall’s products in the spring. The research has to start even sooner, so you’ll have time to collect the results and study them. Pretty soon you’re surveying people in February for a decision that won’t be implemented until October. People might tell you they love a price in Spring, but by the time the product ships in fall, there are three new competitors at lower prices, one of the competitors has launched an aggressive new promotional campaign, and economic conditions have changed.

You can of course try to anticipate all these things in your research study, but pretty quickly you have to make so many future assumptions that you’re conducting an academic exercise rather than testing something in the real world.

When I was at Apple, I spent some time as the head of marketing for the home and education business unit. I tried using a quantitative survey to forecast that fall’s sales and set pricing, and the exercise turned out to be a waste of time – the results were out of date by the time we could act on them. Today, Internet-based surveying might let a company move faster.

The other challenge to keep in mind in price research is that people almost always overstate how much they’re willing to pay. Think about it — in the very process of conducting the survey, you have to describe the product in some detail, focusing the subject’s attention on it much more than would normally happen. This is almost certain to make them more interested than they would be in the real world, where your messages will be lost in a flood of other things being communicated to them.

People also just plain tend to get cheaper as they go further into the buying process. They might say $299 was an ideal price when surveyed, but when they go to actually buy the product that money feels a lot more important to them. Maybe there’s some other item across the store that they might like to buy instead, or maybe they want to go out to dinner and a show this weekend.

This doesn’t mean it’s pointless to do any research on pricing, but I think it’s better to try to research price bands — what range of prices are people willing to pay for certain classes of product — rather than trying to set the exact price of a single product. And if the research does indicate that a certain price is optimal, treat that as the upper limit on your pricing rather than the midpoint.

Things to look for in quantitative research

It’s very easy to screw up a quantitative research study. Even small errors in methodology can make the results meaningless, so it’s best to work with someone who knows research. There are more potential pitfalls than I can list here, but a couple of prominent things to watch out for include:

–Make sure you’re surveying enough people so you can be reasonably sure that the results represent the population as a whole. In research terms, you want a large enough sample so that your findings will be statistically significant. Preferably, the margin of error in the study should be plus or minus five percentage points at the 95% confidence level. That means that if you see a five percentage point difference in a question (52% say yes, 47% say no), there’s a 95% chance that the majority of people actually would say yes if you surveyed everyone in the country.

For consumer tech products in the US, that usually means you need to survey a couple of thousand people minimum. For a corporate product, about 200-300 people may be sufficient, since the world of corporate buyers is a lot smaller than the world of consumers.

–You need to be sure the list of potential respondents (the people you’re surveying) doesn’t have any biases. If you’re surveying a pool of people who are inclined toward a particular answer, it’s sure to skew your results. You see this all the time when, for example, magazines survey their own readers and then report the results as if they represent the country as a whole.

–Work with the researcher closely on crafting the implications of the study. A great market researcher has to be meticulous about methodology, but that same focus on detail can make them reluctant to draw conclusions that reach beyond the basics of the data. This is especially likely to happen when you use outside research consultants, who won’t understand your industry as well as you do. They’ll tend to give you implications that are straight-line projections of their findings, without much context.

For example, if people have less than positive opinions of your product, the researchers might report “you need to improve impressions of this product” or even “the product is a failure.” But you might have other information — perhaps there was a product recall that temporarily hurt opinions of the product; or maybe your company launched the product as a stop-gap, knowing there would be problems. It can be tremendously demoralizing to have an outside researcher come into your company and beat up a product without the right context on what its goals were and what else is happening in the market.

–Beware of buried assumptions. Sometimes a researcher’s unstated assumptions about the market will slip into their selection of what to emphasize and how to phrase it. For example, suppose you did a survey showing that 19% of the population liked your product. A researcher could report that fact with either of the following sentences: “Unfortunately, only 19% of adults want the product” or “Fortunately, nearly one in five adults want the product.” One sentence makes the finding sound bad, the other makes it sound good. Sometimes an outside researcher will make assumptions about what your company’s goals are, and editorial comments like this will slip into their report without them even realizing it.

Before the launch of the original Palm Pilot, Palm commissioned a survey to determine how many people would want the product. The survey showed that two percent of US adults were extremely interested. Many researchers would interpret that as a terrible result; 98% of adults weren’t entranced by the product. But Palm looked at those numbers and decided they were good news — two percent of US adults is about five million handhelds, a very attractive figure for a small hardware company. So the launch went ahead and the rest was history.

–Be very cautious of “off the shelf” customer segmentations. Several market research companies have conducted very large studies on people around the industrialized world. Based on these studies, they have divided the population into segments — usually about a dozen of them — with various demographic and interest profiles. The segments are usually given offbeat, vaguely disturbing titles like “Sultry Seniors” and “TechnoTweens.”

These companies specialize in mapping your products to their demographic segmentations, and using that information to tell you what to do. There’s nothing wrong with this, and sometimes the segments can be useful.

But very often the segmentation of your customers will be specific to the products you make. For example, one of the hottest trends in advanced phones today is building in e-mail capability. The people who want this feature don’t fit into a particular lifestyle category, they’re just people who obsess about communication. The target market for an e-mail phone won’t show up in most standardized segmentations. If you can afford it, you’re much better off doing a segmentation study that’s specific to your industry.

–Price is the biggest drawback of quantitative research. I’ve rarely seen a consumer study in the US that cost less than $30,000, and they can easily go over $100,000 if you want to get specific detail on multiple market segments. Expanding a survey into Europe typically costs about $30,000 per country, and Asia is even more expensive than the US.

You can sometimes find companies that will charge you less, but usually they’re taking some hidden shortcut that will reduce the value of the research. Typically they’re identifying respondents on the cheap, in ways that could bias the findings.


Please click here to rate this section (the link will open a one-screen anonymous survey, and you’ll get to see the results after you take it).

Next week: How to get along with market researchers.

3 Responses to “9. Quantitative market research”

  1. Mark Schraad said on June 26th, 2006 at 9:18 am

    Segmentation is the aggregation of customers into groups with that desire a common set of product attributes. It is NOT based upon demographics, psychographics or sociographics. This later data, however, can be used to determine what marketing messages to deliver, to whom they should be delivered and how to deliver them.

  2. Copy Dvd To Dvd On Imac | The Blu-Ray Way said on January 24th, 2013 at 1:14 am

    [...] enclosure an access vis USB or Firewire. Or burn files to dvd. Good luckPowered by Yahoo! AnswersMark asks…how can i restore a imac with a external disc drive?I have a imac that is barely 2 years…p>how can i restore a imac with a external disc drive?I have a imac that is barely 2 years old. I [...]

  3. corporate marketing consultants | business internet sales & marketing consulting strategies said on May 10th, 2013 at 4:51 am

    [...] For you reading this post and familiar with my writings, you will know that this topic of the use of…to slash budgets and in many cases make many key individuals with very specific skills leave the company. This is where I see in the near future the use of corporate business strategy consultants [such as myself] becoming more and more in demand as companies look for that vital input of skills and strategies. The problem that businesses face of cours is that of needing a particular role fulfilled and, not having the time to train staff to be able to perform that role. In these instances will turn increasingly to external expertise. The beauty of using external expertise is the vast amount of perspective, knowledge and experience that they bring. Also, I have found one great advantage being that I often come having worked in multiple industries whereas we see that typically an executive may have been with one company and in that industry for over a decade. This perceived strength I have found, is something that in fact acts as a weakness as consultants are able to use strategies and tactics from other industries, modify them and bring benefit to completely different industries. If you are a manager, person of influence, agency executive maybe looking for someone to help kick start a particular business circumstance, consider this post and – I'd be more that interested in hearing your perspective. To your success, -Stephen C Campbell [...]

Leave a Reply