Considering Agile? Then understand the risks

Agile, in the world of software development refers to a set of values and principles that result in a more collaborative and iterative development process together with the customer.  Agile is sometimes seen as THE alternative to traditionally “planned” software development projects that usually run over budget, are late and deliver a product that hardly ever matches up to what the customer actually wants.

agile-boat

I recently had a catch up discussion with a colleague and friend who works in software and it turned out that her company is now changing their development methodology to adopt agile.   Now I know the company concerned well,  and started thinking about agile and the risks faced when embarking on agile in an environment that has in the past always been very standard process oriented.  Their specific SDLC (software development life cycle) was always prescribed by the customer and regarded for decades as the “only way” to deliver good software products.  So what changed?

Coincidentally, later in the week I spent 3 days with another client who was also embarking on exactly the same initiative, changing the established SDLC and introducing agile processes into a big ERP software rewrite and upgrade project.

Clearly companies are increasingly adopting agile in preference to plan driven development – but at what risk?

A transition to agile can be exciting and yield many benefits, provided the transition itself is well managed in the specific circumstances.  If not, agile can unfortunately prove to be an unmitigated disaster.  There has been little evidence that software project failures will be eliminated through agile, in fact there is real discontent around agile, for example:

Agile promises solutions it cannot deliver. It promotes sloppy requirements, hides the true cost of development and prevents effective management. Contrary to what we’re told to expect, this leads to long-running projects, dissatisfied customers and an overall IT ineffectiveness. – Lajos Moczar

 

 

Waterfall and agile

Simply stated, there are two well known approaches to managing software projects – waterfall and agile.

The waterfall methodology is familiar to most project managers.  These projects follow a prescribed set of activities in a logical sequence; optimised around resource constraints, timing and cost. Due to the sequential nature of the project, the end product usually does not emerge until after integration and testing.

In long duration projects the customer does not get to see the finished product until much of the investment is complete; and then invariably there is a gap between what is required / expected by the customer and what was delivered.

Agile on the other hand is an approach that seeks to favour the following:

  • collaboration,
  • individual contribution,
  • flexible planning,
  • total focus on delivering finished working software

Planned projects (waterfall) are characterised by rigid process, extensive documentation, detailed planning and inflexible contracts.

Agile in contrast is at its heart a set of values and principles; it is not a prescribed framework.

Agile is seen by many to be the best approach for delivering software in an environment of fast changing technology and radically evolving customer requirements.  I don’t disagree, but I advise caution because it is also a significant culture change.

Agile is not new, many companies increasingly started applying agile to their whole software development lifecycle about 4 years ago but the principles go back 20 years or more.

Agile can be implemented in most areas in a software company, including development,  enterprise architecture design, project office, service management and operations.

In both examples I encountered last week, the adoption of agile was driven by the need to upgrade the underlying technology from Oracle and ensure that the application continues to function on modern cloud and mobile platforms.

In both cases the companies are known for their mature, working and established processes in place for software development. These processes had previously followed a waterfall approach and projects could last 12-24 months.

The problem was that the estimated costs for the technology upgrade using the traditional waterfall techniques were orders of magnitude higher than the business or customers could ever afford.  The project durations were also ridiculously long when measured against client expectations.

In short, both companies needed a different approach.  And they both independently chose agile in preference to other alternatives.

Agile is not without risk

My own experience with agile has been positive. But this was in a start up business with a small and dynamic team. It was relatively easy in this environment (especially as the leader) to establish the cultural norms for agile because we were small, the business was young and we shared a common vision.

Agile evolved in our company naturally, not by edict or prescription; but because it was the best approach.  We never even knew we were implementing agile until we discovered that what we were doing was described by others as “agile”.

But agile is not without risk.

As you scale up a business or project the complexity becomes significantly greater. The strategies and tactics that apply to a small development team of about 10 people do not necessarily scale to a team of 50 or 100 people.

Include outsourced partners and offshore development and you are exponentially increasing the complexity and risk.

The complex parts still need to be designed and delivered

Adopting agile does not mean that the real complexities in the software simply fall away.

If you implement agile because you are failing to deliver planned projects on target then perhaps you need to look more deeply at the capabilities within your organisation – agile projects are not necessarily a remedy for poor project management and could amplify shortcomings in skills and process maturity.

Keeping the customer close in a participative collaboration does have its benefits; but can also introduce new complexities.  Some customers like to be beta testers, others cannot afford to work with incomplete software products.

At the client I was with last week there are over 30 customers who want to contribute to the development roadmap. Each customer is at a different maturity level in their use of the system and therefore has their own priority.  Some customers are long standing (over 25 years),  others new (less than 1 year).  They are also based in several countries.

Involving this many customers with conflicting agendas in formulating your development roadmap can be extremely difficult and ultimately counter-productive.

The solution we agreed on last week was to establish SIG (special interest groups) with the most experienced users representing the customers to collaborate with the development teams. Managing SIG’s however is a special skill and requires strong product leadership.

Successful agile boils down to “cultural factors”

In a survey in 2012 – 53% of companies state that organisational culture is the biggest barrier to agile adoption.

The shift from a highly governed process driven culture towards an agile culture is full of risk.

My experience in this regard was amplified when we merged a smaller “semi-agile” company together with a larger “process driven”, project oriented development team. After 2 years the sub-cultures continued to clash: the “agile cowboys” never respected the “gantt chart plodders” (my terminology!) and visa versa.

By the way,  cowboy approaches are easy to identify – just look at the customer satisfaction and track record of failed projects in the relevant business unit or team.  If the team is adopting agile and the customers are happy references I can guarantee you that there are no “cowboys” involved.  If the customers are dissatisfied then you are in the old wild west without doubt.

In any significant cultural shift, good leadership is essential. Most of the issues I experienced in software teams over the years were not related to technical challenges at all; they boiled down to personalities and “culture”.

Leadership remains accountable

As mentioned, moving to agile is a huge cultural shift for many organisations.  And culture is strongly influenced by leadership.

Coherent governance in an agile environment can become a real problem and can also introduce a real risk of failure.  Agile is not for cowboys. Agile requires discipline and process.  A foundation of empowerment and trust in your people is necessary for agile teams to succeed.

Process driven cultures are not always based on empowerment and trust – project managers become the policemen of delivery and people are expected to deliver repeated results in work packages according to established protocol and recipes.

Leaders cannot simply hope that agile introduced in a department will succeed without their support.  Agile needs strategic commitment and involvement.

Abdication by leadership to operational managers “responsible for agile” is simply not an option if you want to succeed.

Leaders need to spend the time and make the effort to properly understand the agile manifesto and proactively address the necessary cultural shift.

Leaders need to start with an understanding of the maturity of the existing organisational culture before bolting on agile in the hope that all problems will now vanish.

Leaders must treat the move to agile as a strategic project requiring deep understanding, proper change management and consultation with affected stakeholders.