Stop Flying Blind - A book by Michael Mace

10. How to work with market researchers

This week we continue our look at market research, with a discussion of how to work with market researchers. The typical market researcher has a very specialized skill set that’s not fully understood, or necessarily valued, by the company as a whole. If there’s an MR team in your organization, you need to spend some time learning how they work and what makes a good research study.

In the tech industry there’s an informal rule that if you want to get along with hardware engineers, you have to learn how to appreciate their block diagrams. A block diagram is a drawing that shows how the various components of a circuit or computing device work together. If you can understand the basics of an engineer’s block diagram, his or her respect for you will go way up, and you might even be treated like a sentient being.

This is a block diagram of the Data Translation DT9840, a “low-cost real-time data acquisition USB module with an embedded DSP for high-accuracy noise and vibration testing.” Check out the dual 24-bit analog inputs. Sweet!

The equivalent of block diagrams for a market researcher is something called a crosstab. Crosstabs are documents the size of a regional phone book, listing every question asked in a quantitative survey and every response, cut by a myriad of different statistical groupings — age, income, and so on. Reading crosstabs can feel about like, well, reading a phone book. But there’s a hidden beauty to them. As you look through the questions and answers, you’ll start to pick up subtle patterns and get a feel for how the customers actually think. Here’s a simplified example of something you might see in a crosstab:

This is a little excerpt from a study that looked at Internet usage in the US. In this question, people were asked if they had browsed the web in the last three months. The vertical columns across the top divide the results by the age of the respondents and their sex. The horizontal row labeled “Total” shows the total number of people surveyed in each category. For example, the survey talked to 433 people aged 65 and older. The row labeled “Have browsed web” shows the number of people who answered yes to the question, “have you browsed the Web in the last three months?” So, 75 out of 433 people aged 65 or older said yes, or 17% of the sample.

To me, there are two important findings in this sample, one of them a surprise and one not. The thing that didn’t surprise me is that elderly people are less likely to use the Web. The surprising finding was that the rate of Web usage was very flat for people under age 54. For most technology products, young people are more enthusiastic adopters.

You can’t get this sort of intimate familiarity with the data in a study just by reading a summary; you have to look at the crosstabs. Unfortunately, it’s a lot of work to read them. A typical crosstab document for a major research study could easily have more than 400 pages, each of them looking about like this:

This is a typical crosstab page. Don’t worry that it’s unreadable; the original Word document is in about five point type.

Chances are you won’t be reading crosstabs every day. But you should do enough of it that you get comfortable with the formatting and can pick out important information. If nothing else, it’ll help you ask the researchers much more informed questions.

A caution about crosstabs.
Because they’re the mother lode of information about a study, crosstabs can be dangerous in the wrong hands. If someone misreads a crosstab, they might completely misinterpret a research study. Because of this, some researchers don’t like to show anyone their crosstabs, and I think you should be very reluctant to circulate them freely in your company. If a researcher is kind enough to share their crosstabs with you, keep in mind that it’s an act of trust. Be sure to check with them if you form any conclusions about the data, and don’t give the crosstabs to anyone else without telling them.

Become a methodology groupie. This is the other key to getting along with market researchers. Methodology means the way the study was conducted — how the people in the survey were chosen, how the questions were asked, and how the results were tabulated. I gave you a start on understanding methodology in the first part of this chapter, but if you’ll be working with researchers regularly you should do a little more study on your own. If you’re not a born researcher, methodology is about as interesting as double-entry bookkeeping, but it’s hideously important. If it’s done wrong, it can completely skew the results of a study, so researchers spend a huge amount of time agonizing about it. If you want to understand their world, you should know enough about methodology so you can at least tell the difference between a reasonably well structured study and one that belongs in a circus.

How to organize a market research team

Reporting structure. A market research team can vary in size tremendously, depending on the size of the company it serves. In a very small company, you can get away with having no full-time researchers at all. In this case you contract out your research to an external expert who manages the projects for you and delivers the findings. I don’t like this model because researchers pick up a lot of information and insight along the way that never makes it into a formal report at the end. If the researcher lives outside your company, that insight will be lost.

In a multi-division company with several business units, you’ll need several researchers. The first question is whether to have those people report to a central team, or to distribute them into the business units. If there’s any business synergy at all between the BUs, I think it’s best to have the team located centrally. This has several advantages:

–First, it’s more efficient. Depending on how much work there is, a single researcher can often handle the needs of two or more business units. If you parcel out the researchers to each BU, you’ll have to hire more people.

–Second, if the team is centralized, there’s a growth path for the researchers. Market research is a specialized skill, and it’s very hard for a researcher to “graduate” from that role to something else. Most of them don’t want to do anything else anyway. But they would like to have the opportunity for promotion, something they can get in a central team. It’s also very helpful to have researchers supervised by someone who’s a professional researcher themselves. There’s a huge amount of expertise needed in market research, and it’s very hard for a non-expert to evaluate the quality of a researcher’s work and give them meaningful feedback on their projects. Having a non-researcher lead a market research team is like having a non-doctor lead a medical research team.

–Third, if the researchers work together, comparing notes and talking about their work, they’ll be able to spot trends and information that crosses multiple studies. Often these serendipitous discoveries are the most useful.

The drawback of a central market research team is that the business units tend to view it as distant and not focused on their needs. This is a genuine risk. One way to make the central team more acceptable is to have the researchers report “dotted line” into the business units. The researchers sit in on the BU staff meetings, so they feel like a part of the team and are responsive to its needs. But their formal reporting structure still runs back through the central MR team.

Allow only one source of customer truth in the company.
As I mentioned above, strategic market research that focuses on understanding how customers think can be the most valuable output of a market research team. But you should not focus all of your group’s efforts on that sort of research. In fact, it’s very important to make sure that your group is also the exclusive source of tactical market research services for the company. If someone needs a study on sales of a particular product, or customer attitudes in a particular company, you should never turn away that request.

If you don’t have enough people in your team to do all the research the company wants, you should pre-qualify a couple of outside vendors who can take on the extra work. Make sure they understand the projects you’re doing, and the main themes that you’re trying to educate the company about. You should also supervise loosely the work they do for your company. In particular, you should take a look at their conclusions before they deliver a research report to the company. Be ruthless about enforcing this rule.

This is a contrast to what I said a competitive analysis team should do. For competitive analysis, one of the biggest challenges is not getting consumed by trivial support requests from the company. For a market research team, one of the biggest challenges is making sure there’s only one unified source of customer “truth” for the company. In my experience, if you let parts of the company start doing their own market research without supervision, you’ll quickly end up with competing versions of the “truth” floating around the firm. If you leave a business unit to its own devices, inevitably it will contract with a low-cost researcher who produces poor findings, or who tells them what they want to hear. This mangled research will conflict with some of the things you’ve found about the market, so you’ll end up arguing against the BU’s research. This can get very ugly. The average employee at your company doesn’t have the knowledge to tell the difference between a good study and a bad one, so your argument can quickly degenerate into a mud-slinging match about who has the biggest methodology. Even if you win the argument, you’ll make enemies.

Far better to prevent the problem from happening in the first place by making sure all research comes through you and is professionally conducted.

Work style. Many market researchers are extroverts. That’s not surprising, since their profession focuses on understanding people. But much of the actual work of market research — designing a study and analyzing the results — is pretty solitary, involving a researcher wrestling one-on-one with a computer and sometimes hundreds of pages of tabulated numbers.

Unlike competitive analysts, a market research team shouldn’t be pushed into collaborative work all the time. Researchers need a balance between opportunities to work alone and interaction with their team. The interaction is mostly at the start and end of a study. At the start, a study design and questionnaire should be reviewed by others in the group. If you’re doing focus groups, it’s good to have several people from the research team attend some of the groups, just to get a feel for what customers are saying. And at the end of a study, the researcher’s conclusions and presentation of findings should always be previewed, and defended, in front of the entire group.

It’s also pretty common in any large study to have a few strange results that seem out of place or are hard to explain. For example, I’ve seen cases where people gave very different answers when small changes were made in the wording of a question. If you have any cases like that, it’s very important to discuss them with the group, and figure out what they mean, before the findings go to anyone outside. Some people in your company will be basically skeptical about market research, and will jump on any error or ambiguity as an excuse to dismiss the entire study. Protect your team’s credibility by reviewing studies carefully before they’re delivered.

Don’t call them analysts. As an example of how specialized the market research world can be, it’s important to be careful with the job titles you give to market researchers. I once made the mistake of referring to them as “analysts.” I thought that would be a compliment, because analysis is a more active and valuable activity than just running a research study. It turned out to be an insult. In the market research world, an analyst is a junior trainee who massages data after a more senior researcher has conducted a study. Calling a researcher an analyst is like calling a corporate vice-president an administrator.

This is another example of why it’s good to have a professional researcher manage an MR team.


Please click here to rate this section (the link will open a one-screen anonymous survey, and you’ll get to see the results after you take it).

Next week: Picking researchers and presenting findings.

One Response to “10. How to work with market researchers”

  1. Barbara Ballard said on June 13th, 2006 at 3:09 pm

    This chapter is interesting and useful – but is missing a key component. The title is “how to work with …” and not “how to get along with …”. The latter is a subset of the former.

    Reading this book as I am, in segments, leaves me dissociated with earlier chapters. So:

    - what type of projects should each group do?
    - when, if ever, should the teams cooperated on a project?
    - what projects should one group refer to the other?
    - when should each group sub-contract to the other?
    - how should the results of each group’s work be published to the other?

Leave a Reply