Stop Flying Blind - A book by Michael Mace

12. The online revolution in market research

Working in the high tech industry, I have become very jaded about claims that the Internet is going to revolutionize something. Grocery shopping, book-reading, even going to the bathroom were all supposed to be transformed by the Web.1 So I hesitate to say this, but I really believe it: The Internet is producing a revolution in quantitative market research.

Web-based companies are driving incredible reductions in the cost and time needed to collect quantitative information. It’s now becoming possible for even a small company to create the sort of studies that previously were available only to the largest companies and political organizations.

To explain how dramatic the change is, I first have to describe the steps you take to conduct a traditional quantitative research study. First, you work with a researcher to create and review the questionnaire. This alone can take weeks. The questionnaire has to be specially coded and formatted so that phone attendants can understand it. Then the researcher identifies a list of people to call, the call screeners have to be briefed, the phone calls have to be made (and often re-made if no one’s home), the results have to be tabulated and analyzed, and then the researchers create a presentation of the results. The process generally takes 2-3 months and costs at least $30,000 in the US — sometimes a lot more depending on what you want to learn.

Several companies on the Web have recently automated this process. Examples include SurveyMonkey and Zoomerang, and I know there are a lot of others. In all of them, the process generally works the same way: you create your own survey online, send e-mails to recruit the respondents, they fill out the survey on the Web, and then you download your results. I did my first survey this way about a year ago, and the process took less than a week:

On Monday afternoon I wrote the survey. At 8 pm Monday we sent the e-mails inviting respondents. At 11 pm that night, I checked the results from home just before I went to bed. We already had 800 responses, and I could check the tabulated results for each question. There were several interesting trends that I decided to watch closely. I let the survey run until Thursday, and we got a total of 5,400 responses, 60% of them from people outside the US. On Friday morning I showed the results to our CEO, and he picked out four questions that he wanted made into charts for a speech the next week. I downloaded the results into Excel, and sent him the charts before I went home that night.

Total time elapsed: Five business days. Total cost to the company: About $800.

Oh and by the way, that $800 included a year’s subscription to the survey service, which will let me do follow-up studies for free.

Total reduction in time: About 92%. Total reduction in cost: At least 95%.

Now, this is not an invitation to go cut your company’s market research budget by 95%. There are some significant limitations to online research. Here’s are the most important, plus some ideas on how to work around them.

Drawback #1: It’s limited to web-users. Conventional market research uses phone calls or the good old postal service to contact people. Although this can be slow and expensive, it reaches almost everyone. Online market research reaches only people who use the web. Although in 2006 that was about 70%-77% of adults in the US, and a rising percentage elsewhere, it’s not everyone.2
Web usage is generally lower outside the US. Below are some Web usage rates for some prominent countries:

Percent of population who have access to the Internet3

The penetration limit may not matter if you’re selling a high tech product in the developed countries — almost all of your target customers will probably be online anyway. But it’s a big deal to a company selling, for example, discount eyeglasses for elderly people (only about 40% of people over age 65 in the US are online).

Even if you are selling a technology product, you shouldn’t make the mistake of projecting to the whole adult population from online results. For example, if 10% of people in an online study say they like your product, you can’t conclude that 10% of adults in the country like your stuff, only that 10% of the web users like it.

There are a couple of workarounds for this.

If you’re trying to decide whether there’s a big enough market for your product, try the math assuming that the only people you’ll sell to are those who are online. For example, if in an online study you find that 10% of the people surveyed want a product, your estimated available market is 10% of the online population (in the US, that would be 10% of about 210 million online users, or 21 million people). You know the real market will be bigger than that, but at least you have a floor. If that’s enough people to make your product profitable, you have the info you need to launch it.

(Even in some low-penetration countries such as China and India, the actual number of Internet users is large enough that it adds up to a substantial potential market. If you believe the statistics, China is second only to the US in number of Internet users, and India’s Internet population is slightly larger than the UK’s. I’d read these numbers with a bit of caution, though — in some countries Internet access is through Internet cafes rather than a PC that’s in the home or office. That means usage patterns, and your ability to market to these people online may be very different.)

Another tactic you can try is supplementing your online research with other studies. For example, you could conduct one conventional market research study side by side with an online one. By comparing the results, you’d get an idea of how you need to adjust the online results to account for the full population. Then you could probably get away with just online studies for a year or 18 months before you’d need another conventional study to recalibrate.

You should also ask about online access whenever you collect information from your customers. For example, rather than just asking age and education on a product registration card, also ask your customers if they use the web. That will tell you what percent of your users you’re reaching with online studies.

Drawback #2: You need a good list. To conduct an effective online survey, you have to send e-mail invitations to a random list of web users. If there’s a bias in the list of people you contact, your results will be biased. In my experience, this is the biggest source of errors in online studies. For example, I’ve seen online surveys made of people who register to post messages on a particular website. People who post actively online are far different from the average web user, and the results you get from them will not reflect “normal” people.

I’ve also seen industry analyst companies trumpet the results of surveys of their own subscribers, as if those people represented average customers. Remarkably, those surveys seem to always reflect back whatever messages the analyst firm has been preaching. This happens both because people repeat the messages they’ve been told, and because people tend to subscribe to industry analysis services that they agree with in the first place.

But the worst way to get a list of customers is to post a message to an online forum inviting people to come respond to your survey. When respondents can select themselves, you inevitably get flooded with fanatics and cranks. That isn’t a survey, it’s a popularity contest.

Finding a good list is hard. There are companies that specialize in collecting lists of people willing to fill out online surveys. You can rent access to these lists (although it’s often expensive). I use this technique a lot in my own work, and the results seem quite solid, but I do sometimes worry that people who go out of their way to volunteer for surveys may not be a great proxy for the public as a whole.

If you have friends in research at other companies, you may be able to share access to customer lists with companies that don’t compete with you. For example, if you were looking to evaluate the market for a new video camera, a list of television owners might be a good starting point. Unfortunately, there are legal restrictions on how companies can share e-mail lists; make sure you don’t violate the law.

Over time, it’s best to compile your own customer list. That won’t help you size the market for a completely new product, but it will let you track your current ones, and evaluate the market for derivative products. Chances are your company’s marketing department is already compiling a customer list. They probably regard it as a marketing tool, something they can milk like a cow to get additional revenue. But the list is also a valuable source of information, and can save you a lot of money on research. Reward your customers for sharing their e-mail addresses with you, and make sure the marketing department doesn’t drive those people away by spamming them with too many “special” offers.

Drawback #3: You need to know how to design and analyze surveys. This is trickier than it looks. It’s easy to accidentally bias a survey by asking a question in the wrong way. For example, if you ask people to rate something on a numerical scale (i.e., “one to six, with six being best”), it’s a good idea to give an even number of choices. If you give an odd number, a lot of people will cop out by choosing the middle, neutral, option.

If you haven’t had some training, it’s best to get help from someone who knows how to construct a survey. (This is the point where I should probably do a shameless promotion for the consulting company I work for.)


Fun with online research

Now that I’ve listed the challenges with online research, let’s talk about the opportunities. The first opportunity is frequency. You’ve seen the “tracking polls” that professional politicians use in election campaigns. You can now do your own tracking studies. If you have a good list of e-mail addresses, you can easily survey a subset of them every week, watching for changes in attitudes and tracking the effect of things like ad and PR campaigns. Do this right, and you should never be caught by surprise by a market trend again.

It’s very helpful if you can collect demographic information about the people on your customer list. Once you have this, you can use it to aim targeted surveys at particular segments — for example, people with a certain income level, or in a particular age group. This will let you learn much more detailed and subtle information on the market than you could have collected in traditional studies. Be sure you understand the limits on information collection in the regions where you operate, though — some countries are adopting restrictive rules on the use of customer information, and the rules change frequently.

The other opportunity, I think, is driving the use of research more deeply into your company. In a situation where market research is virtually free and almost instant, your use of it can change dramatically. Trying to make a tough decision on whether or not to build in a feature? You can have data in a week. Thinking about a price change? You can get customer feedback almost instantly. I’m not saying you should let research make your business decisions for you. As I’ve said elsewhere in this book, I think the best decisions come from a mix of data and gut instinct. But now there’s no excuse for basing a decision only on guesswork. The data’s almost free. Why would you possibly want to operate without it?

Here’s your chance to check out SurveyMonkey. Please click here to rate this post. The link will open a one-screen survey hosted by SurveyMonkey, and you’ll get to see the results after you take it.

Next week: Using market research to plan product strategy.

  1. In 2004, there was allegedly a project at Microsoft to create an Internet-equipped public restroom, so people could browse while otherwise occupied. There’s some disagreement over whether it was a serious project, or just a joke. Everyone agrees that if there was such a project, it was killed soon after it was first disclosed to the public. Meanwhile, those of us in the mobile industry know that a significant number of people use smartphones to browse and send text messages while they’re using the facilities. So maybe this is an area where the Internet is breeding a revolution after all. [↩ back]
  2. Harris Interactive measured US penetration at about 77% in May 2006. The other US statistics in this section also come from Harris. You can check out a summary of their study here. Other sources put the US figure at about 69%. The difference may be due to variations in the way different studies define Internet penetration. [↩ back]
  3. These are generally 2005 figures, as collected by [↩ back]

2 Responses to “12. The online revolution in market research”

  1. Donald Richardson said on September 14th, 2006 at 11:44 am

    Your article touches on all the important survey taking points and is a good addition to the how-to’s out there.

    Specifically, I’ve seen little focus on international markets, and penetration issues in regards to online surveying. I’m glad someone’s delving in to that.

    Seeing that you know what you’re talking about, and furthermore, you know how to express it well, I was hoping to see if you’d noticed anything that could have been improved during the “create survey -> distribute invitation -> track responses -> analyze results” process.

    The company I work for, Tercent Inc., recently released, yet another online survey tool. As you know, the market is quite full of these online survey tools. We hope to distinguish ourselves from competitors by providing features that facilitate the process at a more than competitive price. We want to be easy and cheap. Although we are focusing on the educational market, I’m sure that any improvements on the general process that you may offer will be invaluable to improving our service.

    I thank you now for the time you’ve spent reading this, and in advance for any time spent offering your advice.


  2. Mike Mace said on September 14th, 2006 at 12:04 pm

    Hi, Donald.

    Thanks for the comment. I want to think about your question some more, but my first thought is that the online survey creation process stinks. Even with a fast web connection, it can be painfully slow to create adn edit a survey, especially whenit goes to more than a few questions. It’s hard to detect errors in skip logic in the tools I’ve used (I need a flowchart view), and I have had lots of problems getting complicated filters to work. There are a lot of services, though, and it may well be that I just haven’t tried the right ones yet.

    Once the survey’s completed, the process of getting it out to people is pretty easy and quick — you just e-mail the link to the survey, or have your list vendor send it out. Some of the survey tools don’t report results the way the list vendors want, so that can be an issue.

    Thanks again for asking, and I’ll try to think about it some more.


Leave a Reply