Agile Zone is brought to you in partnership with:

My engineering career spans 25 years and includes research on relational databases, designing geographic map software, building email products, and creating scalable web applications. I am currently CTO at Rearden Commerce. Dan is a DZone MVB and is not an employee of DZone and has posted 16 posts at DZone. You can read more from them at their website. View Full User Profile

Consistency vs. Innovation

11.15.2010
| 1988 views |
  • submit to reddit

I've been pondering the problem of how does an organization achieve innovation and at the same time have some level of consistency in the patterns and components that are in use across teams. Considering the very nature of innovation is to solve existing problems in new ways, there is clearly a conflict between architectural consistency and innovation. This begs the question, how does one create an environment for innovation at all.

When faced with a new problem, creative people will try to find several ways to solve it. The nature of creative work is the need for autonomy to work through many possible solutions before arriving at one that is considered optimal. When organizations start with a selected list of solutions before the process even begins, the autonomy that creativity thrives is removed. The creative process itself is dampened which not only leads to substandard solutions, but lowers the morale of those forced to stay inside the box.

The counter argument to allowing complete autonomy is that your company will wind up with several ways to solve every problem. There will be no consistency and a lot of wasted effort as different groups solve the same problems over and over in different ways. This does in fact happen, but I also believe it may be the result of other root causes and not people's inherent desire to reinvent the wheel constantly. I believe that the real cause is one of four basic issues:

  1. The knowledge that a perfectly reasonable solution exists isn't widely known in the organization.
  2. The available solution isn't well understood or is difficult to understand.
  3. The available solution is imperfect but difficult to adapt to the problem.
  4. There really is a better solution to the problem.


The first 2 are the same basic root cause. The knowledge about what solutions have been created is either poorly communicated or not communicated at all. Components are built by one team, poorly documented, and not broadly socialized. Team A creates an amazing component, but the documentation is terrible and they are too busy to tell anybody about it. Team B, when faced with the same problem, proceeds to build another solution. Not because they want to, but because they don't understand why they don't have to. In fact, neither team views the component as core to their goals, but rather a necessity. Organizations cannot have enough opportunities to communicate what teams are working on.

The 3rd issue is a classic software design problem. The solution is narrowly focused and will require as much work to adapt as creating a new component. The biggest risk here is that another narrowly focused component will be built that has overlap but solves a different subset of a broader problem. This is an area where architectural oversight can help teams see the value of creating a new component that is broad enough to encompass the relevant subsets of the problem area.

The 4th is called innovation. This is where leaders have to be open minded, let people explore better solutions. Accepting that you may be creating the 2nd, 3rd, or even 4th way to solve the same problem at your organization is key. It's only through this acceptance that innovation can occur and your technical solutions can move forward.

The traditional role of architectural governance can be very useful though in sorting out which of the 4 scenarios exist when someone wants to introduce a new solution to a problem. Most teams have a tremendous backlog of work, so looking for ways to ensure they aren't inventing for invention's sake is useful. It is also an opportunity to mentor and mature your teams.

Ultimately, I believe the answer is consistency through meritocracy not governance. The key challenge to a meritocratic process though is communication. Your organization must have open, consistent, frequent, and quality communication between teams. Wikis and issue tracking tools are a minimum. Social networking tools like Yammer are hugely valuable and an improvement over email because they provide broader visibility and context. Email suffers from two major flaws as an engineering communication tool:

  1. If you aren't on the distribution list for a conversation, you will never know about it.
  2. The historical record is hidden away in individual mailboxes, inaccessible to new team members.


Examine how your technology organization communicates. Encourage open and frequent communication. Attempt to over communicate (you can't). And consider shifting from a process of governance to achieve consistency and reuse to one of meritocracy.

References
Published at DZone with permission of Dan Pritchett, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Tags: