How Companies Strangle Innovation

I just watched a very smart company try to manage innovation by hiring a global consulting firm to offload engineering from “distractions.” They accomplished their goal, but at a huge, unanticipated cost: the processes and committees they designed ended up strangling innovation.

There’s a much better way.

An existing company or government organization is primarily organized for day-to-day execution of its current business processes or mission. From the point of view of the executors, having too many innovation ideas gets in the way of execution.

The Tidal Wave of Unfiltered Ideas

Pete Newell and I were working with a company that was getting its butt kicked from near-peer competitors as well as from a wave of well-funded insurgent startups. This was a very large and established tech company; its engineering organization developed the core day-to-day capabilities of the organization. Engineering continually felt overwhelmed. They were trying to keep up with providing the core services necessary to run the current business and at the same time deal with a flood of well-meaning but uncoordinated ideas about new features, technologies and innovations coming at them from all directions. It didn’t help that “innovation” was the new hot-button buzzword from senior leadership, and incubators were sprouting in every division of their company, it just made their job more unmanageable.

One of the senior engineering directors I greatly admire (who at one time or another had managed their largest technology groups) described the problem in pretty graphic terms:

“The volume of ideas creates a denial of service attack against capability developers, furthers technical debt, and further encumbers the dollars that should be applied towards better innovation.”

Essentially, the engineering organization was saying that innovation without a filter was as bad as no innovation at all. So, in response the company had hired a global consulting firm to help solve the problem. After a year of analysis and millions of dollars in consulting fees, the result was a set of formal processes and committees to help create a rational innovation pipeline. They would narrow down the proposed ideas and choose which ones to fund and staff.

Build the Wall

I took one look at the process they came up with and could have sworn that it was invented by the company’s competitors to throttle innovation.

The new innovation process had lots of paperwork – committees, application forms and presentations, and pitches. People with ideas, technology or problems pitched in front of the evaluation committee. It seemed to make sense to have have all the parties represented at the committee, so lots of people attended – program managers who controlled the budget, the developers responsible for maintaining and enhancing the current product and building new ones, and representatives from the operating divisions who needed and would use these products. Someone with an idea would fill out the paperwork justifying the need for this innovation, it would go to the needs committee, and then to an overall needs assessment board to see if the idea was worth assigning people and budget to. And oh, since the innovation wasn’t in this year’s budget, it would only get started in the next year.

Seriously.

As you can guess in the nine months this process has been in place the company has approved no new innovation initiatives. But new unbudgeted and unplanned threats kept emerging at a speed their organization couldn’t respond to.

At least it succeeded in not distracting the developers.

This was done by smart, well-intentioned adults thinking they were doing the right thing for their company and consultants who thought this was great innovation advice.

What went wrong here? Three common mistakes.

First, this company (and most others) viewed innovation as unconstrained activities with no discipline. In reality for innovation to contribute to a company or government agency, it needs to be designed a process from start to deployment.

Second, the company had not factored in that their technology advantage attrited every year, and new threats would appear faster than their current systems could handle. Ironically, by standing still, they were falling behind.

Third, this company had no formal innovation pipeline process before proposals went to the committee. Approvals tended to be based on who had the best demo and/or slides or lobbied the hardest. There was no burden on those who proposed a new idea or technology to talk to customers, build minimal viable products, test hypotheses or understand the barriers to deployment. The company had a series of uncoordinated tools and methodologies as activities, but nothing to generate evidence to refine the ideas, technology or problems as an integrated innovation process (though they did have a great incubator with wonderful coffee cups). There were no requirements for the innovator. Instead the process dumped all of these “innovations” onto well-intentioned, smart people sitting in a committee who thought they could precompute whether these innovation ideas were worth pursuing.

An Innovation Process and Pipeline

What the company needed was a self-regulating, evidence-based innovation pipeline. Instead of having a committee vet ideas, they needed a process that operated with speed and urgency, and innovators and stakeholders who curated and prioritized their own problems/idea/technology.

All of this would occur before any new idea, tech or problem hit engineering. This way, the innovations that reached engineering would already have substantial evidence – about validated customer needs, processes, legal, security and integration issues identified — and most importantly, minimal viable products and working prototypes already tested. A canonical Lean Innovation process inside a company or government agency would look something like this:

Curation

As the head of the U.S. Army’s Rapid Equipping Force, Pete Newell built a battle-tested process to get technology solutions deployed rapidly. This process, called Curation, gets innovators to work through a formal process of getting out of their offices and understanding:

Internal and External Survey

• Other places the problem might exist in a slightly different form
  • Internal projects already in existence
  • Commercially available solutions
• Legal issues
• Security issues
• Support issues

Use Cases/Concept of Operations

• Who are the customers? Stakeholders? Other players?
• How did they interact? Pains/Gains/Jobs to be done?
• How does the proposed solution work from the viewpoint of the users?
• What would the initial minimal viable products (MVPs) – incremental and iterative solutions – look like?

In the meantime, the innovators would begin to build initial minimal viable products (MVPs) – incremental and iterative tests of key hypotheses. Some ideas will drop out when the team itself recognizes that they may be technically, financially or legally unfeasible or they may discover that other groups have already built a similar product.

Prioritization

One of the quickest ways to sort innovation ideas is to use the McKinsey Three Horizons Model. Horizon 1 ideas provide continuous innovation to a company’s existing business model and core capabilities. Horizon 2 ideas extend a company’s existing business model and core capabilities to new customers, markets or targets. Horizon 3 is the creation of new capabilities to take advantage of or respond to disruptive opportunities or disruption. And we added a new category, Horizon 0, which defers or graveyards ideas that are not viable or feasible.

At the end of this prioritization step, the teams meet another milestone: is this project worth pursing for another few months full time? A key concept of prioritization across all horizons is that this ranking is not done by a remote committee, but by the innovation teams themselves as an early step in their discovery process.

Solution Exploration/Hypotheses Testing

The ideas that pass through the prioritization filter enter an I-Corps incubation process. I-Corps was adopted by all U.S. government federal research agencies to turn ideas into products. Over a 1,000 teams of our country’s best scientists have gone through the program taught in over 50 universities. (Segments of the U.S. Department of Defense and Intelligence community have also adopted this model as the Hacking for Defense process.)

This six- to ten-week process delivers evidence for defensible, data-based decisions. It tests the initial idea against all the hypotheses in a business model (or for the government, the mission model) canvas. This not only includes the obvious — is there product/market (solution/mission) fit? — but the other “gotchas” that innovators always seem to forget. The framework has the team talking not just to potential customers but also with regulators, and people responsible for legal, policy, finance, support. It also requires that they think through compatibility, scalability and deployment long before this gets presented to engineering. There is now another major milestone for the team: to show compelling evidence that this project deserves to be a new mainstream capability and inserted into engineering. Or does it create a new capability that could be spun into its own organization? Or does the team think it should be killed?

Incubation

Once hypothesis testing is complete, many projects will still need a period of incubation as the teams championing the projects need to gather additional data about the application as well as may need to mature as a team before they are ready to integrate with a horizon 1 engineering organization or product division. Incubation requires dedicated leadership oversight from the horizon 1 organization to insure the fledgling project does not die of malnutrition (a lack of access to resources) or become an orphan (no parent to guide them).

Integration/Refactoring

Trying to integrate new, unbudgeted and unscheduled Horizon 1 and 2 innovation projects into an engineering organization that has line item budgets for people and resources results in chaos and frustration. In addition, innovation projects not only carry technical debt, but also organizational debt.

Technical debt describes what happens when software or hardware is built quickly to validate hypotheses and find early customers. This quick and dirty development results in software that can become unwieldy, difficult to maintain and incapable of scaling. Organizational debt is all the people/culture compromises made to “just get it done” in the early stages of an innovation project. You clean up technical debt through refactoring, by going into the existing code and restructuring it to make the code stable and understandable. You fix organizational debt by refactoring the team, realizing that most of the team who built and validated a prototype may not be the right team to take it to scale but is more valuable starting the next innovation initiative.

Often when an innovation pipeline runs head-on into a process-driven execution organization, chaos and finger-pointing ensues and adoption of new projects stall. To solve this problem we acknowledge that innovation projects will need to refactor both technical and organizational debt to become a mainstream product/service. To do so, the innovation pipeline has engineering set up a small refactoring organization to move these validated prototypes into production. In addition, to solve the problem that innovation is always unscheduled and unbudgeted, this group has a dedicated annual budget.

Disruptive Products

Some products and services going through the pipeline create new capabilities or open new markets. These Horizon 3 disruptive innovations need to separate from the existing development organizations and be allowed to grow and develop in physically separate spaces. They need the support and oversight of the CEO.

—

Fast forward a year, and slowly, like turning a supertanker, the innovation pipeline we proposed is taking shape. The company has adopted Lean language and process: curation, prioritization, three horizons, I-Corps – business/mission model canvas, customer development and agile engineering.

Lessons Learned

• Every large company and government agency is dealing with disruption
• Most have concluded that “business as usual” can’t go on
• Yet while the top of the organization gets it, and the innovators on the bottom get it, there has been no relief for the engineering groups trying to keep the lights on
• Innovation isn’t a single activity; it is a process from start to deployment
• In “execution engines,” committees and broad stakeholder involvement make sense because experience, knowledge, and data from the past allow better decision-making
• In “innovation engines” there isn’t the data to decide between competing ideas/projects (since nobody’s been in the future), so the teams need to gather facts outside their cubicle or building quickly
• A self-regulating, evidence-based Lean Innovation process will deliver continuous innovation and disruptive breakthroughs with speed and urgency

[This post by Steve Blank first appeared on the official website and has been reproduced with permission.]

Note: We at Inc42 take our ethics very seriously. More information about it can be found here.