The benefit of incremental, adaptive and lightweight processes such as Agile is its focus on producing value added releases and addressing architectural risk early in the project. This helps the project manager to ensure that the development team is working on those aspects important to the client as well as those that provide the most value to the business and increase the likelihood of delivering the project within the restraints of schedule and budget.
Summary
Software project failure is often devastating to an organization. Missed deadlines and releases containing serious flaws and missing features can mean the end of the project or even financial disaster for a company. It is economics that determine the success of any software project and its value to a company with the amount of money spent on development determining the cost of the asset. The return generated by the product is its value, with the difference between the return and the cost being the “return on investment”.
The Standish Group’s Chaos Report is a landmark study of IT project failure. The Standish Group research shows a staggering 31.1% of projects will be cancelled before completion. Further results indicate that 52.7% of projects will cost over 189% of their original estimates. The cost of these failures and overruns are just the tip of the proverbial iceberg. The lost opportunity costs are not measurable, but could easily be in the trillions of dollars in the United States alone.
The traditional project methodologies, such as the SDLC (Systems Development Life Cycle) approach, that many top corporations use are considered to be bureaucratic or “predictive” in nature, and they have resulted in many unsuccessful projects. These “heavyweight” methodologies are becoming increasingly unpopular. They can be so laborious that design, development and deployment can actually be slowed down.
Agile software development is an increasingly prevalent alternative to traditional, process centric software development processes differentiated by a focus on people, results, minimal methods and maximum collaboration. It is geared towards the high pace and the rapidly changing environments of today’s business projects.
The purpose of this process analysis is to identify how Agile software development can benefit organizations and individuals within that organization from stakeholders and business persons to project managers and engineers, with Revolution being no exception. This benefit will be realized through a deeper understanding of how traditional software development methodologies are not delivering on the promise of creating a framework for the successful delivery of critical software systems and how alternative methodologies can help software initiatives realize their maximum return on investment.
Introduction
With much of the fundamental project infrastructure (scope, priorities, estimates, schedules, risks, etc.) in a constant state of flux, it has never been more important to steer and manage decisions within an overall business context and drive project goals towards business value.
An extreme may occur in traditional software development processes when the planning process is at such a detailed level that the project never really gets started. Once the scope, plan, and objectives are fully defined and agreed upon, the team realizes it used 50% of the time and budget for the project to define the plan. Subsequently, the dates and costs of the plan are obsolete. The team needs to go back and revise the plan again. One realizes this approach does not work because the planning phase never ends and work never begins.
Agile development is a methodology used to describe how to deliver projects on time and budget based on the scope. Given the rapid change in technology and business needs, it is impossible to accurately plan and schedule a multiple month software project up front. This approach offers the necessary planning and control but also the flexibility to accommodate change.
The scope of this analysis will be limited to an examination of the traditional process to software development and where this approach fails to yield the desired results. This process will be contrasted to an alternative approach known as Agile development . This approach that has been gaining momentum in the software development and business communities, demonstrating success in addressing the shortcomings of a more traditional process.
This process analysis will attempt to demonstrate how Agile development will allow organizations to deliver projects on time and within budget, while simultaneously ensuring greater customer satisfaction.
Research Method – Statistics on IT projects failure rate
Before embarking on a “strategic” project, every organization should be aware of its chances of success. Statistics of IT project failure rate provide a good measure of those chances. The purpose is to make executives undertaking large projects ponder on how to approach this endeavor to maximize their chances of success.
The following surveys provide statistical data about the rate of failure of IT projects.
- The KPMG Canada Survey (1997)
- The Chaos Report (1995)
- The OASIG Survey (1995)
The KPMG Canada Survey (1997)
In April 1997, KPMG Canada sent a survey questionnaire focusing on IT project management issues to Canada’s leading 1,450 public and private sector organizations. The main purpose was to outline the reasons behind the failure of Information Technology projects.
Survey Scope
Out of 1,450 questionnaires sent, 176 were analyzed. Of these, 61 % reported details on a failed IT project.
Key Findings
Over 61 % of the analyzed projects had failed according to the respondents. More than three quarters went over their schedules by 30% or more; more than half exceeded their budgets by a substantial margin. Considering that an estimated $25 billion is spent on IT application development in Canada annually, the survey data indicated that unbudgeted IT project expenditures must run into the billions of dollars.
The Chaos Report (1995)
The Chaos Report was a landmark survey made by the Standish Group. This report is the study of IT project failure and is widely cited when IT project failures are being discussed.
Scope of the Study
The respondents to the Standish Group survey were IT executive managers. The sample included large, medium, and small companies across major industry segments: banking, securities, manufacturing, retail, wholesale, heath care, insurance, services, local, state, and federal organizations. The total sample size was 365 respondents representing 8,380 applications. In addition, The Standish Group conducted focus groups and personal interviews to provide a qualitative context for the survey results.
Key Findings
The Standish Group research showed a staggering 31.1% of projects would be cancelled before they ever get completed. Further results indicated that 52.7% of projects will cost over 189% of their original estimates. Based on this research, The Standish Group estimated that in 1995 American companies and government agencies would spend $81 billion for cancelled software projects and paid an additional $59 billion for software projects that would be completed, but exceeded their original time estimates and projects completed by the largest American companies had only approximately 42% of the originally-proposed features and functions.
The OASIG Study (1995)
This study was undertaken under the auspices of OASIG, a Special Interest Group in the UK concerned with the Organizational Aspects of Information Technology.
Scope of the Study
Information was collected in 1995 in the United Kingdom from a sample of 45 experts employed primarily by Universities or Consultancies. On average they had each over 20 years personal experience representing a cumulative knowledge base of over 900 years. The OASIG drew their opinion from a sample of approximately 14,000 user organizations. 31 of these interviewees (69%) included consultancy work as a major component of their work and 27 (60%) include research; many did both. Their professional areas of expertise covered the domains of management, business, and social science. A small number of those interviewed had a background in engineering.
Key Findings
The IT project success rate quoted revolved around 20-30% based on the most optimistic interviews. Ultimately, 7 out of 10 IT projects “failed” in some respect.
Process Analysis
Traditional software lifecycle development processes grew out of a need to control ever larger development projects, and the difficulties of estimating and managing these efforts to reliably deliver results. These methodologies drew heavily on the principles from engineering such as construction management. As a result, they stressed predictability (one has to plan every last detail of a bridge or building before it is built), and linear development cycles; requirements led to analysis which led to design which in turn led to development . Along with predictability, traditional methodologies inherited a deterministic approach that relied on task breakdown, and was predicated on stability: stable requirements, analysis and stable design. This rigidity was also marked by a tendency towards slavish process “compliance” as a means of project control.
While these methodologies may have worked for some organizations in the past and may still work in some circumstances, for many companies these methodologies only added cost and complexity while providing a false sense of security that management was “doing something” by exhaustively planning, measuring, and controlling. Huge costs were sunk in premature planning, without the rapid iterative development and continuous feedback from customers that we have come to realize are prerequisites for success today.
Waterfall Process Model
The waterfall model emphasizes producing a “correct” design prior to implementation, proceeding through phases in a linear, sequential manner. Appropriate only when requirements can be well understood up front and requirements won’t change.
Agile development has little in common with the waterfall model. In some eyes the waterfall is discredited, but many large scale software development takes place today under this model. The waterfall model is the most predictive of the methodologies, stepping through requirements capture, analysis, design, coding, and testing in a strict, planned sequence. Progress is generally measured in terms of deliverable artifacts – requirement specifications, design documents, test plans, code reviews and the like. The waterfall model can result in a substantial integration effort toward the end of the cycle, a time period typically extending from several months to several years. The size and difficulty of this integration effort is one cause of waterfall project failure. Agile methods, in contrast, produce completely developed features (but a small subset of the total) every few weeks or months. The emphasis is on obtaining a simple but working system early and continually improving it.
Agile Process Model
The Plan, Do Check, Act (Deming) Cycle is the foundation of empirical, continuous improvement methods such as TQM, Six Sigma, Lean Manufacturing and Agile Development. Agile is a nonlinear process characterized by responding to change, adapting based on feedback, focusing on value and being lightweight and incremental, and is appropriate when requirements are not well understood up front and will change.
Agile processes are a system of methods designed to minimize the cost of change, especially in a context where important facts emerge late in a project, or where we are obliged to adapt to important uncontrolled factors. Non Agile processes, by comparison, are ones that seek to achieve efficiency by anticipating, controlling, or eliminating variables so as to eliminate the need for changes and associated costs of changing.
The basic problem is that in a traditional waterfall project, risk remains high throughout the majority of the development lifecycle. It may only be at the coding or integration stage that a problem is uncovered and this can have a major impact on all of the work undertaken thus far. All too often this results in major delays or even outright cancellation of the project. The graph below illustrates this problem :
The graph below illustrates a typical agile software development approach. The x-axis shows time while the y-axis plots the amount of working software and the level of project risk. With an iterative approach the software is developed in stages, with core functionality developed first and additional features added as the product evolves. Since the early emphasis is on putting the technical infrastructure in place, the early iterations produce only limited working functionality. However, the real benefit is that they serve to tackle the project risk much earlier in the lifecycle.
Conclusion
Heavyweight methodologies try to plan out a large part of a software project in great detail over a long span of time. Project managers want to see every technical detail because they want to predict every conceivable project milestone. This leads managers to demand a variety of specifications, plans, reports, checkpoints, and schedules. This strategy is only effective as long as there are not any unexpected changes.
Traditional methodologies are typically known as “heavy” or “monumental”. These methodologies typically follow what is called and SDLC (Systems Development Life Cycle). They impose a strong emphasis on process especially upfront planning. Even though they have been around for quite some time, they are not noted for being very successful or popular. The unpopularity of these “heavy” methodologies is a result of the massive effort required throughout the process which can actually slow down development and often lead to the failure of the project.
Today’s Information Technology manager is under ever-increasing pressure to deliver results – in the form of applications that drive improvements to the bottom line – even while IT budgets are being significantly slashed. Business environments continue to change at a rapid pace leaving many IT shops struggling to keep up with the pace of change. These changes have led to an increased interest in agile software development processes for rapid product delivery and flexibility while maintaining quality.
Agile methodologies strive to reduce the cost of change throughout the software development process. For example, rapid iterative planning and development cycles in order to force trade offs and deliver the highest value features as early as possible. In addition, the constant, systemic testing ensures high quality via early defect detection and resolution.
The benefit of lightweight processes such as Agile development is its focus on producing value added releases and addressing architectural risk early in the project. This helps ensure that the finished product meets expectations and that the stakeholders will perceive it to be “a good value.” The project manager can therefore give the client a release of the application much earlier in the project plan than would be possible with a heavyweight methodology. In this way, the project manager ensures that the development team is working on those aspects important to the client and those provide the most value to the business, and increases the likelihood of delivering the project within the constraints of schedule and budget.
