The Importance of Requirements Definition in IT Systems Development

by Ray Phariss

IS 6840, Systems Analysis

November 29, 2006

 

Numerous surveys and studies indicate that one of the major challenges in IT system development is the determination of system requirements.  It might be surprising that after several decades of industry experience in these endeavors, developers would still pursue projects for which the requirements are not abundantly clear.  However, these problems persist and the findings are validated by a long list of infamous IT project failures ranging from the very recent to those reaching back nearly as far as computers have existed. 

 

 While we may be inclined to associate such failures with certain types of companies, certain areas of the world, or certain types of business practices, Robert N. Charette notes, “failures are universally unprejudiced:  they happen in every country; to large companies and small; in commercial, nonprofit, and governmental organizations; and without regard to status or reputation.” [11] This being the case, the failure of IT projects represents a complex problem that must be thoroughly studied from many perspectives to be fully understood.  Further, the insights gained from such studies must be carefully applied in order for future projects to be successful.  As information technology continues to expand into nearly every form of human activity, this is a problem that must be addressed.  The scope of the problem is not trivial and neither are the costs.  One estimate is that “the yearly tab for failed and troubled software conservatively runs somewhere from $60 billion to $70 billion in the United States alone.” [11]  

 

In addition to direct financial costs, failed projects can also lead to expensive litigation, and some people make a career out of helping to sort out the mess.  One such individual is Brooks Hilliard, who “has been a consulting and testifying expert witness, on computer fraud, for both plaintiffs and defendants, in more than 30 cases involving computer systems that were alleged not to have performed as promised.” [2] One should also consider that some failures are never publicized because the parties would prefer to cover up the failure to avoid loss of face, as well as a possible loss in their company’s value, which can often be considerable.  One notorious example of staggering financial loss is Oxford Health Plans Inc.  In 1997, its automated billing system failed and the result was a one-day loss in its stock value from $68 to $26 per share, a 62% drop which equaled $3.4 billion in the corporation’s overall value [11].

 

This paper will explore the challenges associated with requirements definition as a major cause of failure in IT systems development by drawing on documented examples, existing analysis and studies, and theoretical concepts.  Additionally, it will offer suggested remedies to mitigate the risk associated with the challenges inherent in requirements definition and management.  The ultimate purpose of the paper is to learn from others’ mistakes as a means to reducing the rate of failure in IT projects. [19]

 

 

What are the Challenges?

While there are potentially endless reasons why IT projects fail in general, this paper focuses on the requirements definition stage of development.  This is a critical stage in the process – if not done right, all other activity is irrelevant and a waste of time and money.  Partly, this is due to the fact that requirements are necessarily defined early in the project development schedule.  Unfortunately, this presents the opportunity for the early introduction of errors that can result in terrible consequences later in the project.  As Dave Gluch notes, “a good percentage of errors occur at the requirements stage, and if they’re not discovered until later on, the cost of fixing them is substantially higher.” [10]  While this idea might seem obvious, the concept is easily forgotten in practice when other factors begin to influence the process.  Often, a failure to properly define the project’s requirements is only discovered in the postmortem analysis, i.e. the case study.  Accordingly, it is from these and other sources that the following subjects were selected for further analysis in this paper:

        IT Project Team Performance

        Size and Complexity of the Project

        Inadequate Understanding of Business Processes

        Selecting the Right Development Methodology

        Organizational Culture

 

Together, these factors account for a substantial portion of mistakes made in the requirements definition stage of IT project development.  As such, they merit attention to better understand and, hopefully, minimize their effects on future projects.

 

 

IT Project Team Performance

            IT projects nearly always require teams of people for their implementation.  The level of performance within these teams contributes greatly to the overall outcome of the project.  In her paper titled “Failed IT Projects: The Human Factor,” Sheila Wilson examines several case studies and concludes that the following factors affect team performance in the context of IT projects: skill level of the team members, varying levels of commitment of team members, insufficient or too many team members, turnover of team members (especially those in key leadership positions) and lack of use of a structured methodology [20].  If asked directly, most project managers would agree that these factors are important points to consider when building a team for a specific project.  However, managers may not always conduct a deliberate analysis of such considerations in actual practice.  Rather, teams are often assembled based upon the personnel available to them at the time, a process that might disregard some or all of the aforementioned factors. 

 

In his article “Maine’s Medicaid Mistakes”, Allan Holmes describes a failed $25 million project that was built to process Medicaid claims and comply with new requirements mandated by the Health Insurance Portability and Accountability Act of 1996 (HIPAA). [15] A major cause of the system’s failure was traced back to poor requirements, which in turn were traced back to numerous problems with the implementation team.  First, Dick Thompson, who was then the head of procurement for the state of Maine, said the team suffered from a staff shortage, but he was reluctant to request additional personnel.  This put additional strain on a team that was already struggling to meet an aggressive HIPAA compliance deadline.  Second, Holmes points out critical shortages in the skills of the understaffed team:  “The 65-person team composed of DHS IT staffers and CNSI representatives assigned to the project had difficulty securing time with the dozen Medicaid experts in the Bureau of Medical Services to get detailed information about how to code for Medicaid rules.” [15] Thompson goes on to admit that “it was something we should have done from the start: have someone who knew the business [of Medicaid] working full-time on the project.” [15] The failure to incorporate the needed expertise into the requirements phase of the project resulted in disastrous results later, after the flash cut-over implementation began to produce errors at an alarmingly unexpected rate.  Holmes explains that “without adequate guidance from Medicaid experts, the system had been designed to accept files with up to 1,000 lines of claim data.  But many claims were much larger, some containing up to 10,000 lines, and the server was rejecting them automatically.” [15] This example shows how the lack of expertise on a project team can result in poorly defined requirements that ultimately threaten the overall success of the entire project.

 

Another challenge to effective and successful team performance, especially in the requirements stage, is the distributed nature of many project teams.  As a result of companies outsourcing software development, especially offshore, project teams often consist of geographically dispersed members across multiple time zones.  This makes it difficult for team members to stay synchronized on continuously evolving requirements documentation, often complicated by multiple copies of the same information being exchanged between members until they lose track of which one is the most current.  Microsoft has recognized the challenges inherent in requirements gathering, especially in a distributed development environment.  It proposes a “requirements authoring solution” based on its nearly ubiquitous Microsoft Office application suite, claiming that it “can enable development teams to more easily author and edit requirements documents for complex projects.”  [7] This represents recognition by a major software company of the difficulty in effectively capturing and managing requirements documentation.  When applicable, a keen project manager should seek out tools and methods to help mitigate these types of challenges.

 

 

Size and Complexity

            A challenge primarily associated with large projects is that they are often quite complex when one considers all the intricate business rules and dependencies that exist, as well as interconnections with other systems, some of which might be external to the organization.  This complexity can become a serious problem as it  increases the possibility of errors, because no one really understands all the interacting parts of the whole or has the ability to test them.” [11] As a result, developers may build the system around the parts they were able to identify and accept risk in the other areas, hoping that it will all work out in the end.  However, such hope is often in vain as “studies indicate that large-scale projects fail three to five times more often than small ones.”  [11]  Larger projects also take more time to complete.  Unfortunately, business needs change so frequently that the original requirements, no matter how well they might have met the business needs at the time they were identified and documented, may no longer meet the business needs by the time the project is completed.  Therefore, large projects may require a development methodology that allows for some flexibility in requirements during the course of the project.

 

Another challenge that arises due to the large size of a project is confusion or lack of agreement about who is in charge.  As Warne and Hart point out, “the larger the scope of a proposed information system, the more likely it is to be attended by organizational conflict regarding who should own and control it.” [13] In the article “Anatomy of Failure”, Lauren Gibbons Paul presents an example of a failed data warehouse project for a large organization. [18] Paul explains, “the challenge was to get the distinct and autonomous business units to settle on…critical definitions.” [18] Further, internal disagreement was created “because the managers looked at things differently…defining the fact groups therefore became a hellish ordeal.”  In the end, the project was cancelled after spending three times the original budget.  This example illustrates how the size of a project can lead to internal disagreement, thus contributing to the complexity of both the design and implementation of the project.

 

 

Inadequate Understanding of Business Processes

Successful requirements definition relies heavily on a thorough and accurate understanding of the business processes a project is supposed to model.  When access to the necessary stakeholders and subject matter experts is limited or non-existent, as was the case with Maine’s Medicaid billing project, “developers, having no input from the users, build what they believe is needed, without having any real knowledge of the business.” [3] When this occurs, as Scott Ambler points out in his writing about Agile modeling, as a project manager you must “recognize that you are going at risk if you do not include some direct users in your software development efforts.”  [1] In order to minimize risk to the project, therefore, it is imperative that IT project managers ensure they have adequate stakeholder involvement in the requirements stage, and that the “users…know what it is they want, and be able to specify it precisely.”  [3]

 

Regrettably, customers themselves may not always have the experience and/or knowledge about IT systems necessary to adequately define their requirements.  Ambler notes that “project stakeholders will very often sense that there is a problem or opportunity that should be addressed but they don’t know how.” [1] As a result of their inability to fully grasp their own needs, customers might sign off on requirements they do not understand or that do not adequately meet their business needs - they just do it to keep the project moving along.  Later, when the project is completed and they are disappointed with the results, they immediately point blame at the developer, who in turn falls back on the requirements document that was approved by the customer.  Such was the case when Prudential Europe “terminated a $50 million contract with Unisys following the collapse of its Unite project, which aimed to deliver real-time processing of policies and pensions over the Internet.  Unisys argued that some specifications and requirements were not signed off or fully defined by Prudential.”  [6]  

 

Sometimes, even well understood and documented requirements change.  In the case of Maine’s Medicaid billing system previously mentioned, Medicaid rules kept changing at federal and state levels as the project was on-going, requiring re-programming of the system during development. (Holmes)  This type of re-work can add unexpected delays and costs to already strained projects.  Many projects involve some degree of requirements discovery during the implementation phase of development, and Watts Humphrey, a fellow at the Software Engineering Institute points out, “you can’t design a process that assumes [requirements] are stable.” [17]  

    

Another complicating factor of requirements changing during a project is the effect it can have on the ability to determine whether or not the requirements have been met.  Derek Sisson’s explains in an article about requirements and specifications for web site development that “requirements tend to change through the course of a project, with the result that the product as delivered may not adhere to the available requirements – this is a constant and annoying facet to the quality assurance process.” [9] Effective project managers must ensure that changes in requirements are captured and maintained so that this documentation can be referenced later to validate that the requirements have indeed been met.

 

 

Selecting the Right Development Methodology

In order to provide the flexibility necessary to accommodate large, complex projects and the changes in requirements that often occur throughout their development cycle, project managers should carefully select the development methodology that is most appropriate for these projects.  In his article Challenges in Government Software Development, Scott Cronenweth presents interviews he conducted to examine effects of the recently-created Office of Electronic Government within the U.S. government’s Office of Management and Budget (OMB), an organization that is pushing for greater integration of information systems among the various government departments. [4]

 

Notorious for its bureaucratic tendencies and stove-piped information systems, the federal government would seem like a hostile environment for anyone wishing to integrate its information systems across departmental boundaries.  Further, the sheer scale and complexity of such an endeavor might make the task prohibitively difficult for any single company to successfully accomplish.  As Michael Burkett, one of Cronenweth’s interviewees, notes, “no one manufacturer can deliver an entire system anymore.  Companies have consolidated their businesses around core competencies.  So a prime contractor, such as Lockheed-Martin, must inevitably work with many subcontractors and suppliers, each of which has critical input into the design and manufacturing of specific system components.” [4]  Since entire systems now require multiple companies in order to deliver an end-to-end solution, the level of complexity involved in their development has increased.

 

Notwithstanding the complexities involved, the Office of Electronic Government persists in trying to force the governmental agencies to work together in the interest of the citizens.  Cronenweth describes an example of a system that would allow a citizen to find out what money is owed to him by the government in a consolidated way, even though several distinct agencies might each owe him some amount (IRS, Social Security Administration, etc.). [4] Scott Frohman, another of Cronenweth’s interviewees, explains that “the first challenge…concerns requirements management: getting the right people collaborating to determine what the system needs to look like.  The goal is a unified, agreed-upon set of requirements that defines an agency’s business process and reflects a common vision.” [4] The agreement of the various participants is fundamental to all subsequent activity – without it, the project will certainly fail.  However, this does not mean that every last detail must be agreed upon from the start.  Frohman explains that “with an iterative approach, you don’t assume that requirements are sparkling perfect when you start; you know they’re subject to ongoing adjustment.  This greatly mitigates the risk of building the wrong thing.  In addition, you start testing early on to find and eliminate bugs that would be very costly to fix later on.  Iterative development mitigates all forms of project risk.” [4]

 

Another methodology that appears to be growing in popularity is the open-source development model.  In this methodology, no single person or entity owns the software.  Rather, it is publicly available to everyone with the stipulation that it must remain free and that the source code must remain available, or “open” so that others may modify it.  This model relies on the combined talents of programmers who form communities of interest based on a common goal of developing applications for various purposes.  One of the most widely known examples of this is the Linux operating system, originally conceived as an alternative to Microsoft Windows.  While the number of Linux users today is not nearly as large as the number of users running Windows, Linux has penetrated niche functions such as web servers.  This has led some people to suggest that the open-source model might be well-suited for other purposes.  One example of this is revealed in a post by an individual identified only by a username (Saint) and an email address, who suggests on a Linux-based forum for medical IT professionals that “a closed-source medical software development model is either doomed to failure, or doomed to be prohibitively expensive.” [8] Citing first-hand knowledge of numerous failed medical IT projects, Saint suggests that the medical industry does not represent a large enough market to justify the high cost of traditional, closed-source development for what he considers to be an extremely complex business application.  He claims that “there is no single undertaking other than writing an entire operating system that rivals the complexity and knowledge required for a successful clinical computing system.” [8] Although Saint does not provide any factual evidence to support his position, he does raise interesting points of consideration, especially the notion that we should consider alternative methodologies for some projects that might not be suitable for the more traditional models.

 

 

Organizational Culture

Another factor that can create challenges in determining requirements for IT projects is organizational culture.  The leap from manual processes to an automated system requires a cultural shift in the mindset of employees, many of whom are often incapable of grasping IT-related concepts due to their lack of experience with them.  Further, this lack of experience may manifest itself in a culture that is not conducive to IT system development.  In fact, it might even become a barrier to progress.  One such customer was the FBI, whose recent public failure with its Virtual Case File (VCF) project has come under intense scrutiny and is widely used as a case study of a large-scale IT project failure. 

 

 “The VCF was supposed to automate the FBI’s paper-based work environment, allow agents and intelligence analysts to share vital investigative information, and replace the obsolete Automated Case Support (ACS) system.  Instead, the FBI claims, the VCF’s contractor, Science Applications International Corp. (SAIC), in San Diego, delivered 700,000 lines of code so bug-ridden and functionally off target, that this past April, the bureau had to scrap the US $170 million project, including $105 million worth of unusable code.”  [12]

 

One of the major reasons the VCF project failed was because the culture at the FBI did not place a high value on IT and this affected its ability to work with SAIC to adequately define the project’s requirements.  Some have also concluded that the FBI had a somewhat arrogant attitude toward requirements definition.

 

“Poor benchmarking and requirements gathering…affected the project, with FBI overseers…relying upon a ‘I’ll know it when I see it’ attitude to determine when the project would be ready.”  [14]

 

The FBI’s cavalier approach during the development process was a major factor in the VCF’s ultimate failure.  It was the organizational culture that arguably doomed the project to failure from the beginning by preventing developers from effectively capturing workable requirements.  In numerous investigative findings, the lack of well-defined requirements is listed as one of the major causes:

 

“In a devastating 81-page audit, released in 2005, Glenn A. Fine, the U.S. Department of Justice’s inspector general, described eight factors that contributed to the VCF’s failure.  Among them:  poorly defined and slowly evolving design requirements…”  [12]

 

“It was a classic case of not getting the requirements sufficiently defined in terms of completeness and correctness from the beginning.  And so it required a continuous redefinition of requirements that had a cascading effect on what had already been designed and produced.”  [12]

 

“The requirements and design documentation were incomplete, imprecise, requirements and design tracings have gaps, and the software cannot be maintained without difficulty.  And it is therefore unfit for use.” [12]

 

This example reveals that the challenges of determining requirements for an IT project are not just technical in nature.  Rather, these efforts can be thwarted by the culture of the organization attempting the project, perhaps even unknowingly and unintentionally.  While the employees themselves cannot be reasonably expected to identify or mitigate such cultural biases, a good IT project manager should look for indicators of such conditions and take steps to deal with them. 

 

 

Conclusion

 Requirements determination is a critical stage in any software development process due to its fundamental importance to making sure the project ultimately solves the right problem in the right way.  As many studies have shown, errors in this stage of development often lead to overall project failure with disastrous consequences.  Therefore, prudence requires that IT project managers search for ways to mitigate the common mistakes that occur in requirements definition so that they can improve their likelihood of success in future projects.  This paper has discussed the following factors of successful requirements gathering:

        IT Project Team Performance

        Size and Complexity of the Project

        Inadequate Understanding of Business Processes

        Selecting the Right Development Methodology

        Organizational Culture

 

Project managers must create teams of highly skilled, committed persons who work well together, are able to communicate with each other (even while distributed throughout the world and across different time zones) and share a range of expertise applicable to the success of the particular project at hand.  The size of a project must also be taken into consideration, and the complexity of a particular project based on its large-scale size must be accounted for.  The time, resources and cooperation/communication needed to accommodate large scale projects is often orders of magnitude greater than those needed for small scale projects, an aspect of requirements determination that is often overlooked but is ultimately imperative to the success of a project. 

 

Adequate understanding of business processes is also key throughout the design and implementation of the project.  Project managers must be able to determine when customers may or may not have appropriate understanding of the complexity of project development and must make applicable adjustments to ensure the proper communication and determination of the project’s end goals.  Understanding the many different methodologies and applying the correct methodology for each project is also particularly important in meeting the project’s end goals.  Finally, an organization’s culture must be taken into consideration by project managers to ensure a project’s smooth implementation.  Understanding each organization’s culture will help managers plan and design systems that work best for each of those organizations and will help managers avoid cookie-cutter project designs that ultimately end in failure. 

 

While project failures often go unreported or underreported due to fear of loss of reputation or financial value, we still are able to find ample evidence that the factors discussed throughout the paper are key elements of a project’s ultimate failure or success.  In examining these examples, we are able to use past experiences to enhance our understanding not only of what not to do, but also what project managers can and should do to ensure a project’s success.  Examining in particular the requirements determination stage of project design allows managers to avoid unnecessary pitfalls and mistakes early on in the project’s development, thus ensuring a smoother, more successful project further later, as it moves through its many stages toward completion. 

 

 

 

 

 

 


 Works Cited

 Web:

 

1.      Ambler, Scott W. “Overcoming Requirements Modeling Challenges.” April 29, 2006. Online.  http://www.agilemodeling.com/essays/requirementsChallenges.htm

 

2.      BizAuto. “Web Advertisement for Computer Expert Witness, Litigation Support & Computer Forensics.” Online.  http://www.bizauto.com/expert.htm

 

3.      Coley Consulting. “Why Projects Fail.” 2001 – 2005. Online.  http://www.coleyconsulting.co.uk/failure.htm#nav

 

4.      Cronenweth, Scott. “Challenges in Government Software Development.” The Rational Edge. January 15, 2003. Online.http://www-128.ibm.com/developerworks/rational/library/2223.html

 

5.      Factor, Phil. “Failed Project?” Simple-Talk.com, December 22, 2005. Online.  http://www.simple-talk.com/sql/t-sql-programming/failed-project/

 

6.      IT Cortex. “Failure Examples of IT Projects.” IT Cortex. Online. http://www.it-cortex.com/Examples_f.htm

 

7.      Microsoft. “Managing Requirements Authoring for Complex Application Development Using the Microsoft Office System.” Online. http://www.microsoft.com/office/showcase/rapidauthoring/default.mspx

 

8.      Saint. “Medicine’s Dirty Software Secret.” LinuxMedNews. Online. (ivaldes@hal-pc.org): http://www.linuxmednews.com/955063770/index_html

 

9.      Sisson, Derek. “A Thoughtful Approach to Web Site Quality: Requirements and Specifications.” 1999 – 2002. Online. http://www.philosophe.com/design/requirements.html

 

10. Thomas, Bill. “Meeting the Challenges of Requirements Engineering.” SEI Interactive, March, 1999. Online.  www.sei.cmu.edu/news-at-sei/features/1999/mar/Spotlight.mar99.pdf

 

 

Non-Web:

 

11. Charette, Robert N. “Why Software Fails.” IEEE Spectrum, September 2005.

 

12. Goldstein, Harry. “Who Killed the Virtual Case File?” IEEE Spectrum, September 2005.

 

13. Hart, Dennis and L. Warne. “The Impact of Organizational Politics on Information Systems Project Failure – A Case Study.”  Conference Study: 29th Annual Hawaii International Conference on System Sciences. 1996. 

 

14. Hickson, Thomas B. “Failure at the Speed of Light: Project Escalation and De-Escalation in the Software Industry.” Thesis: The Fletcher School, Tufts University. Spring 2006.

 

15. Holmes, Allen. “Maine’s Medicaid Mistakes.” CIO Magazine, April 15, 2006.

 

16. Lyytinen, Kalle and D. Robey. “Learning Failure in Information Systems Development.”  Info Systems J (1999) 9, pgs. 85-101.

 

17. May, Lorin J. “Major Causes of Software Project Failures.” May, 07, 1998.

 

18. Paul, Lauren Gibbons. “Anatomy of Failure.” CIO Enterprise. November 15, 1997.

 

19. Web, Allen. “Project Failure and How to Avoid It.” March, 2001.

 

20. Wilson, Sheila. “Failed IT Projects: The Human Factor.” University of Maryland/Bowie State University. May 16, 1998.