There’s no question that a commercial Requirements Management tool is very useful; but, can it pay for itself at your company? In this article we’ll look at a model to help you calculate ROI on Requirements Management tools.
Overview of ROI
Return On Investment (ROI) is a popular method of measuring the success of process improvements and IT investments. It is a measure of the dollars returned on dollars invested. As Jeffery E. Payne points out in his article “Quality Meets the CEO: How to Get Management Buy-in,” ROI is an effective approach for arguing the need for, or demonstrating the success of, process improvements and IT investments.
Though there are a number of methods of calculating ROI, one straightforward, simple to understand method is the Benefit to Cost Ratio, which is simply dividing the benefits in dollars of process improvement or IT investment by the costs. Benefit to Cost Ratio = Benefits/Cost. So, a Benefits to Cost Ratio of say, three, would mean that for each $1 spent on the cost of process change and IT, $3 in benefits were realized.
In doing an ROI assessment, typical sources of cost include:
Typical benefits that are considered in an ROI assessment include:
Let’s look at how to do a Benefit to Cost model for process change and IT investment of putting a requirements management tool in place at your company. Though this model was developed in conjunction with the rollout of a commercial tool, it should be readily adaptable to development and rollout of “home grown” tools.
Requirements Management Tools
Requirements Management tools are to requirements what Defect Tracking Tools are to defects. They provide an environment for, and database approach to, managing large numbers of requirements related in potentially complex
ways (traceability). Requirements tools provide continuity through time, across projects, through staff changes, and through company re-orgs. They are a corporate memory for requirements.In a project, setting a Requirements Management tool is used in a number of contexts:
For a large company, dealing with thousands of requirements, moving to a database approach to managing requirements, allows a metrics-based management of requirements that is simply not practical with paper document-oriented approaches.
Calculating the ROI
No question that a requirements management tool is very useful, but can it pay for itself at your company? In his paper Calculating Your Return on Investment from More Effective Requirements Management, Dean Leffingwell advances this model for estimating ROI (numbers are from his example ):
One can then make an argument that if the requirements management process was improved by say 10%, that is a cost savings of $504,000 x 10% = $50,400. If the cost of process improvement (e.g. training), plus tool purchase was a total of $19,900 (again, numbers from his example) then the Cost to Benefit Ratio is $50,400 / $19,900 = 2.53.
The model I will present here is a bit different from Leffingwell’s; it will provide specific improvements one might expect from installing a requirements management tool. I will then try to quantify the benefits from that perspective. The model is based on one I developed for a company of about 500 R&D staff, 1000 employees overall. The tool, a commercially available product, was deployed in four cities. The ROI assessment was done about 1.5 years into the rollout.
Assumptions and Conventions
We begin by making a number of working assumptions and conventions that will be used throughout the ROI model. The model is developed in an Excel spreadsheet.
Cells with black text are ones for which you must input values that make sense for your company.
Cells with red text are ones with values computed via some formula.
Cells that are gray filled have values that are subjective.
These fields are the main source of uncertainty in the model. In interviewing staff as part of the rollout, the values for these fields varied widely.
We begin by calculating the cost of a fully burdened employee, per work day, per hour, and per minute. By “fully burdened” we simply mean the total cost to
the company: salary + total benefits. As a rule of thumb, the fully burdened cost is usually about 1.5 times the salary.
Next, we capture the actuals in terms of numbers of requirements that were entered into the requirements management tool as of the date of the ROI assessment; about 1.5 years after initial rollout. Of the approximately 5000
requirements entered, about 1000 had already been implemented and shipped as part of some project, and some 200 had been rejected, meaning a decision had been made that these requirements would never be addressed in any future release.
The Cost
An ROI assessment must be done for a fixed period of time; both the cost and the benefits must be calculated for the same fixed period. This study was done about 1.5 years into the rollout of the new requirements management
process and tool.
Consulting cost was for onsite support beyond classes, i.e. additional support that was purchased to aid in the rollout of the tool and process. The Rollout Team was a core set of staff that spent a portion of their working time in support of the process and tool rollout. This number is probably a bit low as it doesn’t account for travel expenses.
Another cost we account for is tool use overhead. Just as the proper entry of a defect into a defect tracking tool requires some time, the proper entry of requirements into a requirements management tool requires more time than the capture of requirements on the back of a napkin, or excel spreadsheet. As noted above, the values entered in the gray cells are highly subjective.
Finally, we account for the added rigor in requirements management that we are asking teams to make. For example, requirements that would otherwise not be recorded, or recorded on a white-board in an office, are now entered into a public record. This increase in easy-to-access requirements leads to additional review, discussion, test planning, and change control. The cost of doing the job right is, nevertheless, a cost, and is captured in the ROI
model here.
In Ed Weller’s Calculating the Economics of Inspections (StickyMinds, Jan 2002) he states the recommended rate for preparation and inspection of a requirements specification is about seven pages an hour. This seems a reasonable heuristic for estimating the average cost of additional review and rigor on requirements that were actually implemented. The requirements that were entered, but not implemented, receive much less scrutiny, here modeled as one-tenth the effort, or a factor of 10 increase in speed at which they are reviewed. For the model we will assume 1 requirement = 1 page.
Benefits
Let’s move to calculating the benefits of the requirements management tool. Of the two broad categories of benefits discussed above, increased sales and reduced operating costs, this one will focus solely on reduced operating expenses for the company. This is fairly common for most IT investment ROI models. In the case of a requirements management tool one could probably make an argument that increased sales, or at least retention of sales are realized due to increased customer satisfaction resulting from better requirements management.
Cost Savings from Staff Working More Efficiently
We begin by calculating the savings realized from staff on projects having a readily available, always-up-to-date, common source of requirements upon which they can base their work. It is a cost reduction from staff working
more efficiently to plan, develop, test, document, and develop training materials for a product.
Scenarios where inefficiencies occur when requirements are not readily available to staff include:
Note that this part of the model is not about doing a better job; it’s about doing the same job more efficiently.
Avoiding the Cost of Lost Requirements
Staff churn is a common problem for projects. Staff quit the company; staffs move to other projects or are promoted; whole projects get re-staffed through company reorganization. When requirements are not recorded and managed in a companywide system, they are subject to loss due to staff churn. The loss means that the requirements must be “rediscovered” and re-engineered again and again. Human Resource groups report that 12 percent staff churn per year is not uncommon. Requirements kept on laptops are subject to loss when the laptop is stolen at the airport!
Avoiding Cost of Unnecessary Development
One of the benefits of a companywide requirements management tool is the increased visibility that requirements receive. On Projects employees are able to see what each other is doing; redundancy is spotted; priorities are better managed. The result: requirements get rejected. This leads to cost savings in avoiding unnecessary work.
Reducing the Cost of Requirements Related Defects
Finally, let’s look at the cost savings in terms of fixing requirements-related defects. For this part of the model we’ll use a few concepts also used in doing inspection ROI assessments, e.g. Ed Weller’s “Calculating the Economics of Inspections” (StickyMinds, Jan 2002).
We’ll do this by:
the cost of which we paid for as Added Review & Rigor above.
It’s almost an industry clich that the later a defect is found in the development lifecycle, the higher the cost to fix. The relative cost in the increase varies from industry to industry. Here, I use a simple three phase
defect removal model:
The relative cost of fixing defects will be 1 staff day, 5 staff days, and 25 staff days; an increase of a factor of 5 from phase to phase, well within the industry averages that are cited in the literature.
Now let’s look at the model. We start by estimating the number of requirements-related defects. Unless we are willing to acknowledge the possibility of a perfect requirement, it’s safe to assume that all of the defects have at least one defect. If you are not comfortable with that assumption, adjust the percent as needed.
Next, we estimate the number of requirements related defects removed prior to commitment to code, and the associated cost. A removal effectiveness of 50% means that of the total population of defects, 50% were caught and fixed. We’ll assume that on average a defect at this stage can be found and fixed in 1 staff day. Changes that don’t involve code are simply cheaper to make.
Next, we estimate the number of requirements related defects removed from the code itself (e.g. unit, integration and system test) and the associated cost. The calculation starts with the number of defects that remain
undetected and unfixed from the previous stage. Because we are now dealing with code, the cost of finding and fixing a defect rises from 1 staff day per defect to 5.
At this point let’s review the total removal effectiveness of the first two stages.
Finally, the cost of defects shipped with the product to the customer. At this stage “finding” the bug is not so much a factor in the cost; the customer does that for you! Here, the cost of defects is determined by
factors such as customer support calls, loss of sales from unhappy customers, and, of course the increased cost to patch software in the field. The cost of defects at this point will vary greatly depending on the industry,
safety-critical products being an example where the cost can be very high.
Our baseline cost for finding, supporting, and fixing
requirements-based defects is $2,608,696. Then we rerun the baseline, but with a 10% increase in defect detection and removal at each of the first two stages. This is the added review and rigor we paid for previously (see above under
costs).
Our new cost is $2,024,348; a cost savings of $584,348.
Bottom Line
All that is left is to tally the Benefit to Cost Ratio, and as they say, “your mileage may vary.”
Estimated savings: $1,884,565
Estimated cost: $427,657
Benefit to Cost Ratio: 4.4 to 1
Recall, that cost at this point, includes one-time expenses such as initial software purchase, and initial on-site consulting and training. The ratio should get even better once those one-time expenses are out of the way.
Summary
As noted previously, cells that are gray filled are ones where the values entered are highly subjective, and vary depending on your company and industry. I found that a useful approach to using this model in talking with people is to have them provide values for these cells, kind of a Wideband Delphi approach. In that way they can see if the values they are willing to buy into result in a good benefit to cost ratio. After you’ve talked with a number of
people, use their responses to build a minimum and maximum version of the model. If you want to get a bit more sophisticated, you can even run the model with a
Monte Carlo simulation tool. This would provide a probability distribution function of the Benefit to Cost Ratio.
And while this model was developed to demonstrate ROI after the fact, the ideal use of such a model is to help in selling the process change in the first place, and to forecast that point at which benefits would begin to outweigh the costs of the effort.
Want to Learn More?
David Gelperin, Maybe We Shouldn’t Write Requirements, StickyMinds Original Article, October 2002.
A brief discussion-and lot’s of reader comments-on how best to record and manage requirements: paper documents vs. spreadsheets vs. database etc.
Jeffery E. Payne, Quality Meets the CEO: How to Get Management Buy-in, STQE magazine, May/June 1999.
David F. Rico, How to Estimate ROI for Inspections, PSPsm, TSPsm, SW-CMM, ISO 9000, and CMMIsm, StickyMinds, Sept 2002.
Ed Weller, Calculating the Economics of Inspections, StickyMinds, Jan 2002.
Dean Leffingwell, “Calculating Your Return on Investment from More Effective Requirements Management” http://www.rational.com/media/whitepapers/roi1.pdf
First appeared at StickyMinds.com
Richard Denney is a software process management consultant, emphasizing requirements and testing management. Richard is an affiliate of TeraQuest Metrics (www.TeraQuest.com), a leader in CMM consulting, holds a BA and MS in CS from the University of Texas, and is certified with the Product Development and Management Association. Contact info: visit http://www.rdenney.bizland.com or e-mail rdenney@sbcglobal.net.
