It’s not a well-kept secret. It’s not a costly investment. It won’t harm the environment. It doesn’t require a particular toolset. But it is often divisive and triggers repercussions. Nonetheless, it could differentiate those who thrive from those who dive in their industries. What is it? “It” is the set of interrelated and interdependent components in organizations that have sustained success in their use of agile approaches for both business innovation and software development.
But first, some industry experts have affirmed that agile software development has surpassed traditional—usually implicating “waterfall-based”—methodologies.  One may question whether that ascent is reflected in the number of projects, dollars, headcounts, or mere intuition. Regardless of the basis of measure, most IT leaders will agree that agile development is already an option for them, if not a desired future state of their organization. So then why do so many organizations continue to struggle in their agile transition? Why have others “hit the wall” or “plateaued” in their agile transformation? Why have others adopted hybrid methods that are characterized as scrumfall or watergile?
The focus of this article is scrum-based agile development. Scrum is used in over 75 percent of software development projects that describe themselves to be “agile” today.  Another recent survey depicts that percent at 82.  These include scrum, scrum with extreme programing (XP), and scrumban, which intertwines scrum techniques with kanban. However, confusion still abounds regarding what constitutes agile.  A recent informal poll of agile workshop attendees revealed that nearly 71 percent thought that “agile = scrum.”  Whether scrum is your chosen agile approach or you’ve selected one of the numerous other varieties, the impingements to further maturation of agile practice are likely the same.
Scrum teams have three primary roles: the scrum master, the product owner, and the development team. A well-seasoned, business-savvy, technology-aware, and servant leader-driven scrum master would be welcomed by most scrum teams. The scrum master leads by example, and is a process and tool mentor when needed. The scrum master champions engagement with management and eliminates impediments. This skillset is an asset to the organization but is not enough to ensure the success of the project.
Continuing this example, the scrum master is accompanied by a talented co-located team. Team members are cross-functional; each of them can ably assist on numerous aspects of the development work. They resist being siloed into traditional roles that impede productivity and induce delays. They are constantly learning and continuously incorporating feedback to further increase their value to the team and the organization. They mentor each other. They are creative in finding solutions to challenges that would derail other teams. They understand that their integrity and their commitments are inseparable. They own the quality of their product. They recognize that the success of the business is the goal, not merely their own velocity. They are fully aligned with the scrum master, but again, they are not capable of guaranteeing the success of the project.
Additionally, our scrum master and development team have a dedicated product owner. The product owner is available, accessible, collaborative, and usually steadfast, though urgent business needs may shift priorities from one sprint to another. But ongoing reprioritization of needs is fully embraced by the team as a whole. The product owner is realistic, has a grasp of the significance of the needs of the business, and is a trusted “voice of the customer” for the product’s many stakeholders. The product owner acknowledges the acceptance criteria associated with stories as the determinant for accepting and rejecting stories at the end of the sprint. As a result, stories are richly developed; the backlog is groomed, meetings are attended and productive, metrics are accepted and trending in the right direction, and commitments are met. While all may seem well to the casual observer, the project is still missing necessary elements associated with its success.
Elements like process and tools are likely suspects yet to be considered for contributing to the project’s success. The talented team and the skillful scrum leader have thoughtfully attended to planning needs: tailoring their practices to best fit this project, defining their sprint cycles, road mapping, story pointing, retrospective reflection, and configuration management, as examples. They groom the product backlog, work the sprint backlog, use meeting times for intended purposes, and demonstrate with pride and a tad of trepidation their committed work. The product owner understands the process, its cycles, and boundaries like time-boxing. The product owner is “present” and engaged at all team functions. Enough process is evident, yet it is insufficient for the team’s success.
The presence or absence of tools does not impair this team. They have discovered that some tools necessitate more work and attention than is useful for their performance. The words “high touch, low tech” remind them that a hand-drawn burndown chart satisfies the needs of the team and its stakeholders. The team does not avoid capturing details when they are discovered, but they are not thwarted when details are scarce for work not yet atop the product backlog. The team understands that managing the work and entering details about ongoing work may not be of equal value. They have avoided allowing the tool to drive the process, rather than to sustain it. While tool usage appears to be an asset to the team, and all the roles are fulfilled, and the process is stable, the team may still be trending to failure.
Early in 2016, the McKinsey group compiled their conclusions from a study of agile companies.  McKinsey ranked the top practices that differentiated “most agile” companies from “least agile” companies. Their top practices included the following:
- Role clarity
- Top-down innovation
- Capturing external ideas
- Process-based capabilities
- Operationally disciplined
- Internally competitive
- Meaningful values
- Knowledge sharing
- Inspirational leaders
- People performance review
Note that the hypothetical team described earlier, combined with their processes and tool usage, explicitly address only a few of the items on the McKinsey list. Practices 1, 4, and 8 correlate most closely with the team description above. One could argue that the product owner owned and exercised practice 3, and that practice 7 included a shared set of values among the team. More likely, any conflicting values of the organization would take precedence of those of the team. Perhaps as many as half of the practices can be attributed to the efforts of the team, and in turn, contribute to team success.
At least equally important are the practices that transcend the boundary of the team. Consider the McKinsey report: top-down innovation, meaningful values, and inspirational leaders. Each of these are rooted in the leadership and the culture of the institution. Individual teams are often helpless for effecting changes that facilitate agile usage and agile thinking at the executive level. I assert that the unwillingness or inability of an organization to innovate, inspire, and interpret its value for current markets is impairing organizations—in some cases they are already extinct while others are enduring a slow but certain similar destiny. Note that an organization’s inability to attain timely mastery of those three i’s drives the same unfortunate outcome as those that are unwilling. Limited cultural adaption is often the most overlooked insufficiency. Alignment with and velocity of institutional change will enhance the likelihood of excelling in agile cultures.
Note again then the relevance of inhibitors to agile adoption cited by VERSIONONE  in 2016. Respondents were able to cite more than one inhibitor accounting for percentages that exceed the sum of 100. Only the first five are included here, though the report contains several others:
- The ability to change organizational culture (55%)
- General organizational resistance to change (42%)
- Pre-exiting rigid / waterfall framework (40%)
- Not enough personnel with the necessary agile experience (39%)
- Management support (38%)
Correlating the importance of the McKinsey and the VERSIONONE data, we can more clearly see how critical leadership and culture are for agile success—not merely the success of agile projects, but more significantly, the success of an agile enterprise.
Table 1 – Assessing the significance of the 3 i’s of agile adoption inhibitors
|The ability to change organizational culture|
|General organizational resistance to change|
|Pre-exiting rigid / waterfall framework|
|Not enough personnel with the necessary agile experience|
I accept the probability that specific projects and organizations may assign different values to the cells in Table 1. As some in the agile community might suggest—it prompts a conversation. I might suggest it provokes an overdue discussion. Reexamining an enterprise’s culture is seldom easy. It may produce undesired outcomes. I’m not a fan of the saying “fail fast” or “fail often.” I’d rather promote learning from pilots or frequent experimenting. Failure to learn is perhaps the greatest failure.
A certain “fail fast” (and often) approach is for leadership to declare, “We’re going agile.” Often leadership doesn’t know how that impacts the culture. Agile may have an immediate appeal to the business partner, often followed with frustration when poorly defined stories lead to unrealized expectations and failed releases. The absence of training and ongoing coaching for teams and leadership compounds other cultural barriers. Agile is not merely a (not so) new way to deliver software; it’s also a potent approach to business development.
Organizations might want to include in their agile road mapping a review of the Capability Maturity Model Integration® (CMMI) v1.3.  The CMMI has a number of potentially valuable practices specific to establishing an environment for improvement. Many of these are found at Maturity Level 3 among the Process Management Category. They include Organizational Process Focus, Organizational Process Definition, and Organizational Training. I’m not suggesting that organizations subscribe to the CMMI to become agile. I’m suggesting that organizational level practices in the CMMI are beneficial for defining the scope of activities that help to establish organizational capability, agile or otherwise.
I have attempted to make a case for the need for leadership to embrace cultural change—a cultural reexamination, a shake-up—to effectively champion agile adoption and success. Agile cannot persist as a grass-roots-only initiative because its rippling impact touches so many other functions in the organization, not the least of which is the business itself. Adjustments must be introduced for setting expectations, financing work, reporting, scheduling, and even the vernacular used within the organization. Leadership can’t promote agile while asking for traditional work products, for demanding that requirements be “signed off” before the project proceeds, or for tracking agile projects with waterfall-like critical path management. These are examples of what organizations experience when they describe “culture” as a “blocker” to agile success. Yours may be different. Do you know what they are? Have you asked your agile teams how the culture and leadership itself is putting a drag on success?
Leadership and cultural transformation are likely to also produce unintended consequences. The journey may not be comfortable. I like to encourage groups undergoing agile transformation to “get comfortable with being uncomfortable.” Transformation is not about recklessness—it’s about restlessness. Change can be your ally or you can be its prey.
References: Software Risk Master™ (SRM) Estimating Examples For Quality and Schedules; Capers Jones; September 29, 2015  10th Annual State of Agile Report; VERSIONONE; 2016  2015 State of Scrum Report; AgileAlliance; 2015; pg 2  Keep the Baby (Examining Agile); Schofield; MetricViews; Winter, 2014 / 2015  Rally e-mail for Discover Agile webinar; 9/24/2015  McKinsey quarterly, 2/2016; top ten values for successful agile organizations (161 companies)  2015 State of Scrum Report; AgileAlliance; 2015; pg 13 – 71% of teams acknowledge tension between how their project executes and the rest of the organization  http://www.sei.cmu.edu/reports/10tr033.pdf; retrieved 9/10/2016