Working in the high tech industry, I have become very jaded about claims that the Internet is going to revolutionize something. Grocery shopping, book-reading, even going to the bathroom were all supposed to be transformed by the Web.1 So I hesitate to say this, but I really believe it: The Internet is producing a revolution in quantitative market research.
Web-based companies are driving incredible reductions in the cost and time needed to collect quantitative information. It’s now becoming possible for even a small company to create the sort of studies that previously were available only to the largest companies and political organizations.
To explain how dramatic the change is, I first have to describe the steps you take to conduct a traditional quantitative research study. First, you work with a researcher to create and review the questionnaire. This alone can take weeks. The questionnaire has to be specially coded and formatted so that phone attendants can understand it. Then the researcher identifies a list of people to call, the call screeners have to be briefed, the phone calls have to be made (and often re-made if no one’s home), the results have to be tabulated and analyzed, and then the researchers create a presentation of the results. The process generally takes 2-3 months and costs at least $30,000 in the US — sometimes a lot more depending on what you want to learn.
Several companies on the Web have recently automated this process. Examples include SurveyMonkey and Zoomerang, and I know there are a lot of others. In all of them, the process generally works the same way: you create your own survey online, send e-mails to recruit the respondents, they fill out the survey on the Web, and then you download your results. I did my first survey this way about a year ago, and the process took less than a week:
On Monday afternoon I wrote the survey. At 8 pm Monday we sent the e-mails inviting respondents. At 11 pm that night, I checked the results from home just before I went to bed. We already had 800 responses, and I could check the tabulated results for each question. There were several interesting trends that I decided to watch closely. I let the survey run until Thursday, and we got a total of 5,400 responses, 60% of them from people outside the US. On Friday morning I showed the results to our CEO, and he picked out four questions that he wanted made into charts for a speech the next week. I downloaded the results into Excel, and sent him the charts before I went home that night.
Total time elapsed: Five business days. Total cost to the company: About $800.
Oh and by the way, that $800 included a year’s subscription to the survey service, which will let me do follow-up studies for free.
Total reduction in time: About 92%. Total reduction in cost: At least 95%.
Now, this is not an invitation to go cut your company’s market research budget by 95%. There are some significant limitations to online research. Here’s are the most important, plus some ideas on how to work around them.
Drawback #1: It’s limited to web-users. Conventional market research uses phone calls or the good old postal service to contact people. Although this can be slow and expensive, it reaches almost everyone. Online market research reaches only people who use the web. Although in 2006 that was about 70%-77% of adults in the US, and a rising percentage elsewhere, it’s not everyone.2
Web usage is generally lower outside the US. Below are some Web usage rates for some prominent countries:
Percent of population who have access to the Internet3
The penetration limit may not matter if you’re selling a high tech product in the developed countries — almost all of your target customers will probably be online anyway. But it’s a big deal to a company selling, for example, discount eyeglasses for elderly people (only about 40% of people over age 65 in the US are online).
Even if you are selling a technology product, you shouldn’t make the mistake of projecting to the whole adult population from online results. For example, if 10% of people in an online study say they like your product, you can’t conclude that 10% of adults in the country like your stuff, only that 10% of the web users like it.
There are a couple of workarounds for this.
If you’re trying to decide whether there’s a big enough market for your product, try the math assuming that the only people you’ll sell to are those who are online. For example, if in an online study you find that 10% of the people surveyed want a product, your estimated available market is 10% of the online population (in the US, that would be 10% of about 210 million online users, or 21 million people). You know the real market will be bigger than that, but at least you have a floor. If that’s enough people to make your product profitable, you have the info you need to launch it.
(Even in some low-penetration countries such as China and India, the actual number of Internet users is large enough that it adds up to a substantial potential market. If you believe the statistics, China is second only to the US in number of Internet users, and India’s Internet population is slightly larger than the UK’s. I’d read these numbers with a bit of caution, though — in some countries Internet access is through Internet cafes rather than a PC that’s in the home or office. That means usage patterns, and your ability to market to these people online may be very different.)
Another tactic you can try is supplementing your online research with other studies. For example, you could conduct one conventional market research study side by side with an online one. By comparing the results, you’d get an idea of how you need to adjust the online results to account for the full population. Then you could probably get away with just online studies for a year or 18 months before you’d need another conventional study to recalibrate.
You should also ask about online access whenever you collect information from your customers. For example, rather than just asking age and education on a product registration card, also ask your customers if they use the web. That will tell you what percent of your users you’re reaching with online studies.
Drawback #2: You need a good list. To conduct an effective online survey, you have to send e-mail invitations to a random list of web users. If there’s a bias in the list of people you contact, your results will be biased. In my experience, this is the biggest source of errors in online studies. For example, I’ve seen online surveys made of people who register to post messages on a particular website. People who post actively online are far different from the average web user, and the results you get from them will not reflect “normal” people.
I’ve also seen industry analyst companies trumpet the results of surveys of their own subscribers, as if those people represented average customers. Remarkably, those surveys seem to always reflect back whatever messages the analyst firm has been preaching. This happens both because people repeat the messages they’ve been told, and because people tend to subscribe to industry analysis services that they agree with in the first place.
But the worst way to get a list of customers is to post a message to an online forum inviting people to come respond to your survey. When respondents can select themselves, you inevitably get flooded with fanatics and cranks. That isn’t a survey, it’s a popularity contest.
Finding a good list is hard. There are companies that specialize in collecting lists of people willing to fill out online surveys. You can rent access to these lists (although it’s often expensive). I use this technique a lot in my own work, and the results seem quite solid, but I do sometimes worry that people who go out of their way to volunteer for surveys may not be a great proxy for the public as a whole.
If you have friends in research at other companies, you may be able to share access to customer lists with companies that don’t compete with you. For example, if you were looking to evaluate the market for a new video camera, a list of television owners might be a good starting point. Unfortunately, there are legal restrictions on how companies can share e-mail lists; make sure you don’t violate the law.
Over time, it’s best to compile your own customer list. That won’t help you size the market for a completely new product, but it will let you track your current ones, and evaluate the market for derivative products. Chances are your company’s marketing department is already compiling a customer list. They probably regard it as a marketing tool, something they can milk like a cow to get additional revenue. But the list is also a valuable source of information, and can save you a lot of money on research. Reward your customers for sharing their e-mail addresses with you, and make sure the marketing department doesn’t drive those people away by spamming them with too many “special” offers.
Drawback #3: You need to know how to design and analyze surveys. This is trickier than it looks. It’s easy to accidentally bias a survey by asking a question in the wrong way. For example, if you ask people to rate something on a numerical scale (i.e., “one to six, with six being best”), it’s a good idea to give an even number of choices. If you give an odd number, a lot of people will cop out by choosing the middle, neutral, option.
If you haven’t had some training, it’s best to get help from someone who knows how to construct a survey. (This is the point where I should probably do a shameless promotion for the consulting company I work for.)
Fun with online research
Now that I’ve listed the challenges with online research, let’s talk about the opportunities. The first opportunity is frequency. You’ve seen the “tracking polls” that professional politicians use in election campaigns. You can now do your own tracking studies. If you have a good list of e-mail addresses, you can easily survey a subset of them every week, watching for changes in attitudes and tracking the effect of things like ad and PR campaigns. Do this right, and you should never be caught by surprise by a market trend again.
It’s very helpful if you can collect demographic information about the people on your customer list. Once you have this, you can use it to aim targeted surveys at particular segments — for example, people with a certain income level, or in a particular age group. This will let you learn much more detailed and subtle information on the market than you could have collected in traditional studies. Be sure you understand the limits on information collection in the regions where you operate, though — some countries are adopting restrictive rules on the use of customer information, and the rules change frequently.
The other opportunity, I think, is driving the use of research more deeply into your company. In a situation where market research is virtually free and almost instant, your use of it can change dramatically. Trying to make a tough decision on whether or not to build in a feature? You can have data in a week. Thinking about a price change? You can get customer feedback almost instantly. I’m not saying you should let research make your business decisions for you. As I’ve said elsewhere in this book, I think the best decisions come from a mix of data and gut instinct. But now there’s no excuse for basing a decision only on guesswork. The data’s almost free. Why would you possibly want to operate without it?
Here’s your chance to check out SurveyMonkey. Please click here to rate this post. The link will open a one-screen survey hosted by SurveyMonkey, and youâ€™ll get to see the results after you take it.
Next week: Using market research to plan product strategy.
- In 2004, there was allegedly a project at Microsoft to create an Internet-equipped public restroom, so people could browse while otherwise occupied. There’s some disagreement over whether it was a serious project, or just a joke. Everyone agrees that if there was such a project, it was killed soon after it was first disclosed to the public. Meanwhile, those of us in the mobile industry know that a significant number of people use smartphones to browse and send text messages while they’re using the facilities. So maybe this is an area where the Internet is breeding a revolution after all. [↩ back]
- Harris Interactive measured US penetration at about 77% in May 2006. The other US statistics in this section also come from Harris. You can check out a summary of their study here. Other sources put the US figure at about 69%. The difference may be due to variations in the way different studies define Internet penetration. [↩ back]
- These are generally 2005 figures, as collected by internetworldstats.com. [↩ back]