This has significant
managerial implications: if the interactive build-up [of team
brainstorming] is not leading to better ideas, an organization might be
better off relying on asynchronous idea generation by individuals
using, for example, web-based idea management systems.
That quote is from a report by three researchers from the INSEAD and Wharton business schools. They published a study, Idea Generation and the Quality of the Best Idea, that analyzes a mainstay of corporate life: the brainstorming session.
Is it effective in generating quality ideas?
To find out, the researchers conducted a field experiment in which they compared two models of generating ideas:
Team structure: Group works together at the same time together in a room to generate ideas.
Hybrid structure: Individuals generate their ideas independently, then meet together in a group.
Their objective was to determine
which of those two structures generated more ideas, ideas of higher
quality and is better able to discern the quality of ideas. They found
in all cases that the hybrid structure outperformed the team structure.
Extreme Value Theory
The success of idea
generation in innovation usually depends on the quality of the best
opportunity identified. For most innovation challenges, an organization
would prefer 99 bad ideas and 1 outstanding idea to 100 merely good
ideas. In the world of innovation, the extremes are what matter, not
the average or the norm.
The researchers – Karan Girotra, Christian Terwiesch, Karl T. Ulrich – were interested in determining what methods generate the best
ideas. They distinguish their approach from previous research which
analyzed the quantity or average quality of ideas generated.
They use extreme value theory
to understand the factors impacting the quality of ideas. Extreme value
theory shows that the maximum value of an idea from a set of ideas is
based on:
- The sheer volume of ideas generated
- Average quality of all ideas generated
- The level of variance in the quality of generated ideas
These concepts are put together nicely in this graphic:
Once you understand this framework
for innovation, it becomes a matter of maximizing the values for each
component. Watching, of course, for correlative impacts between them.
Field Research Experiment
The three researchers conducted an
exhaustive experiment to determine which of the two methods – team
structure or hybrid structure – generated the highest quality ideas at
the top end of the scale. Here is the summary of their experiment.
Subjects: 44 juniors, seniors and grad students at the University of Pennsylvania
Challenges: They generated 443 ideas around two challenges.
- You have been retained by a
manufacturer of sports and fitness products to identify new product
concepts for the student market. The manufacturer is interested in any
product that might be sold to students in a sporting goods retailer. - You have been retained by a
manufacturer of dorm and apartment products to identify new product
concepts for the student market. The manufacturer is interested in any
product that might be sold to students in a home-products retailer.
Idea generation formats:
Subjects were split into four clusters. Half the clusters did the team
structure first, half did the hybrid structure first. The clusters then
switched structures for the different ideation challenges.
Idea quality: The quality of the ideas was assessed in two ways.
- Business value: Panel of 41 Wharton MBA students each assessed the business value of the ideas on a 1 – 10 scale
- Purchase intent:
Panel of 88 college students (the target market for the ideas) each
assessed their own likelihood of buying a given product proposal on a 1
– 10 scale
Experiment format: Subjects conducted idea generation exercises as follows.
- Team structure: 30 minutes together in a room to generate ideas together. Then 5 minutes of assessing and selecting the best 5 ideas.
- Hybrid structure:
10 minutes of generating ideas on their own. Then 20 minutes of
discussing these and new ideas. Finally, 5 minutes of assessing and
selecting the best 5 ideas.
Results: Hybrid Structure Tops Team Brainstorming
The results of the experiment are eye-opening. The researchers analyzed the two approaches on the three components of extreme value theory.
They find hybrid is better on the individual components of the theory,
and in the ultimate test: quality of the top ideas produced.
Number of ideas generated. Hybrid structure generates three times more ideas than does the team structure. Researchers attribute this result to three dynamics:
- Free riding: it’s easy enough to ride the idea coattails of the group
- Evaluation apprehension: the fear of negative reaction when proposing an idea in front of a group
- Production blocking: participants have to wait while one person is speaking, limiting idea generation throughput
Idea quality: The
average quality of the hybrid structure ideas was higher than that of
the team structure. Specifically, 0.25 points better in business value,
0.35 points better in purchase intent. To put this in perspective,
these differences translate into roughly a 30 point differential in
percentile rankings. In other words, the difference between the 1st and
30th idea in a pool of 100 ideas.
Researchers attribute the decrease in idea quality for team structures to the same free riding dynamic that reduces the quantity of ideas.
Idea quality variance: The researchers found no discernible difference in idea quality variance between the hybrid and team structures.
What this means is that from extreme value theory, the quantity and average quality of ideas are the key drivers of generating the highest-ranked ideas.
Best ideas: Here’s where the rubber meets the road. Which approach had the highest ranked ideas? Hybrid structure, by a landslide.
The researchers looked at the top 5
ideas, by quality scores, that emerged from the two approaches. The
hybrid structure ideas were of much higher quality than those generated
from the team structure. This finding held for looking at the top 3, 4
and 6 ideas as well.
To recap:
The hybrid structure produced:
- More ideas
- Ideas of better quality on average
- Highest rated ideas
Ability to Select Best Ideas
Perhaps the one down note from the study is the ability of the group
to select the best ideas. Remember that in both the team and hybrid
structures, the group did a consensus selection of the top ideas. Participants weren’t asked to select the top ideas individually.
The researchers found a small advantage in the hybrid structure
group’s ability to select the top 5 ideas resulting from their ideation
exercises. But it wasn’t material. Indeed, they note:
The hybrid process may generate better ideas, but that due
to the noisy selection process, its relative advantage is much
diminished, to the point of becoming statistically insignificant for
one of our quality metrics.
“Noisy selection process”, indeed. Ever been in a brainstorming
session where you’re supposed to rank the ideas at the end? Imagine the
dynamics of resolving differences of opinion, time constraints and the
extraordinary influence of certain individuals that drowns out other
opinions. This is not an optimal way to determine the ideas that define
innovation for your organization.
What This Means for Companies Seeking Innovation
As we described previously in Crowdsourcing Is the New Collaboration,
there are many benefits to taking a new approach to idea generation,
peer collaboration and integrating innovation more deeply into an
organization’s culture.
As this study confirms, distributing the idea generation process, as
well as the idea selection process, results in higher quality ideas for
organizations. This study dovetails well with another study by Professor Ron Burt, that found that employees with access to a wider range of viewpoints and feedback generate higher quality ideas.
Brainstorming does have its benefits in terms of face-to-face interactions. Perhaps the nature of what
is brainstormed needs to change. Brainstorming can be valuable for
project-oriented tasks and problem-solving. But don’t consider it your
go-to activity for the best ideas.
(Cross-posted @ Spigit )
The hybrid process may generate better ideas, but that due to the noisy selection process, its relative advantage is much diminished…
Then what is the optimal way to rank or determine the best ideas?
Hi Joanne –
I think it’s important to understand the “noisy” part of the in-person process. Group dynamics can undercut the hard work of getting the good ideas.
What would be better? Let people step away from the process. On their own they can decide which ideas have the most merit. From the study, there’s a tantalizing finding (pg 19 of 41):
“In further analysis, we compared the self evaluation provided in the individual phase of the hybrid treatment to the independent judges’ quality ratings, and find that these individual ratings are better predictors of ―true quality‖ than are either of the group evaluations, lending further support to the idea that some aspect of the group interaction leads to poor assessments of quality.”
In other words, people were better off assessing the idea outside the group dynamics that can affect their judgment. Or, more likely, their ability to affect the decision.
Allow people to judge the ideas independently of one another. That doesn’t mean feedback should be in a vacuum. There’s also a lot of value in the diversity of viewpoints brought to ideas, and in collaboration around them.
Obviously, the social software systems of today, such as Spigit, enable this crowdsourced approach to innovation.
Finally, I’ll add that the identification of top ideas also requires the considered judgment of authorities. Not exclusively, but their input should be part of the process as well. Just make sure these people are open to innovation, not protecting turf or agendas.
Helpful answer?
Hutch