Most middle managers love to see source code. They love to see reams upon reams of source code, the earlier in the project lifecycle the better. This to them represents progress. If “90%” of the project code has been written, then the project must be “90% complete”.
The same managers balk at the notion of spending time up front gathering requirements, writing specifications, laying down the groundwork for a project where everyone is on the same page. This to them is non-work, as the project does not appear to be any nearer completion. Six months might have gone by without any code having been written.
This problem is exacerbated by a common tendency to take a “waterfall” approach to development. All the functional specs must be written, then all the design specs, then all the code. Then everything is integrated in one big swoop, and integration testing takes place for the entire system, then user acceptance testing takes place. The customer does not get to see any real evidence of progress until right at the end of the project.
It’s understandable that management balk and panic at the prospect of this sort of high-risk process. They don’t see any working code until late in the project. There is no sign that the project will be brought in on time, even with the project manager’s earnest-faced assurance that everything will “suddenly come together on day 347 or therabouts”. To the senior manager, this is a terrifying prospect, and understandably so. Managers at this level need to see tangible progress, and they don’t have time to learn the finer aspects of process engineering in order to appreciate that the project really is on track.
Of course, it could be argued that they should at least understand the basics, because after all, they are in charge of a software project. But does that mean that the customer should also be educated in the same way? And the “power behind the customer”, the moneybags to whom they must justify the project? The answer is probably yes to an extent, if this was an ideal world – but of course in most cases it just won’t happen.
Therefore, if the customer won’t “play ball”, or cannot reasonably be expected to learn our rules, we need to change our ball game to suit them. To translate that into English: the customer doesn’t have the time or inclination to learn about software methodologies. We are the programmers, after all. They just want working software. So we need to give them that, and to actively demonstrate they we are in the process of giving them that, throughout the project.
The customer will want assurance that you are following some sort of process (and a low-risk one at that), but beyond that they probably don’t care about the details. All they care about is what they are going to get, not how they are going to get it. That’s your problem. The customer will make appreciative noises and agree wholeheartedly that they need to know more about the process your team are following – but then the next time you speak to him, it will be as if the conversation never happened. Life is like that sometimes.
Fear of non-progress, then, is a real problem. Clueless managers (who really should know better) will encourage you to leap straight into the programming. To them, code = progress, therefore coding = appreciable activity. If the requirements gathering has taken too long, they will push for you to skip the design phase altogether, and to just “start making some progress.”
So what’s the answer? How do you get those all-important requirements and design phases out of the way without terrifying your own managers and the customer?
The answer is really in two parts. The first, and most important part, is communication, conveying a sense of rapid development. We’ll cover that in a moment. First, though, it’s worth reviewing the process itself, and examining an insidious problem that can manifest, even on projects that don’t set out to follow a “waterfall” process…
The Waterfall Effect
Here’s an interesting paradox: managers and customers of software projects, particularly in larger companies, appear to love the waterfall process because it offers a formal, seemingly low-risk approach that can easily be planned. Waterfall projects fit very neatly into Microsoft Project. However, management are at the same time wary of the waterfall process because it’s high risk – they don’t get to see evidence of real progress until late in the project.
We all know this, but project teams still sail over the waterfall, sometimes without even realising it. Any project which goes “Step 1… then step 2… then step 3…” has an element of waterfall. Step 1 provides the input into step 2, and so on. Different methodologies provide alternative approaches, but this basic pattern is mostly inescapable (you even see it in agile methodologies). Why? Because it always pays to know what you are doing before you do it – to look before you leap.
One of the biggest problems is this: if Step 1 is delayed, so is the rest of the project. That’s one uber-critical path. Worse still, there is a risk that Step 1 might not be sufficient for Step 2 to begin. Step 2 (and all subsequent steps) are entirely dependent on the previous step being completed, and having been done well, i.e. flawlessly.
If your software process needs to be re-engineered, the primary goal should be to reduce risk.
Part of the answer is, of course, to break the project into smaller iterations, or mini-waterfalls. RUP takes this approach, albeit with a big up-front inception and requirements gathering exercise. This initial phase is still important, as it helps to get a bigger picture of what you are trying to achieve, and to make sure you and the customer are both on the same page before you set out to do anything else. The high-level architecture should also be created in one go (but fleshed out in more detail in later iterations). Then the project can be confidently broken into its constituent sub-systems. With well defined interfaces, sub-systems can be developed in parallel, virtually quashing the evil critical path (in theory, at least).
Extreme Programming (XP) takes the approach that customers are afraid even of these initial phases, and that we should leap straight into programming (with a small amount of warm-up analysis, and some hand-written user stories). The design then “evolves” as the project progresses. The theory is then that the customer gets to see “progress” throughout the project, as small amounts of the project are released on a regular basis. A nice idea in theory, except that this progress, however tangible it might seem, is illusory – first, because it is impossible to say precisely how much more needs to be coded, and second because any code that has been written will probably need to be redesigned and rewritten (refactored) many times over (click here to see what I really think of XP).
This brings us to our first and most important answer to the problem of perceived slow development: communication.
Increase Communication
There is always a danger that your project, however well it is progressing, may give an impression of slow progress.
This can give rise to all sorts of problems, e.g. senior managers decide to “crack down” on the project; insist on unnecessary and time-consuming daily updates; try to embellish the process with their own “best practices” which just hinder the project; mix staff around; reassign the project; or worse still get cold feet and can the project entirely. Frustrating, when you knew that the project was doing well. All you had to do was convince them. This is where communication comes in. It’s an essential part of the mix.
Steve McConnell writes about the perception of slow development in Rapid Development: Taming Wild Software Schedules (Microsoft Press). The first thing to do is eliminate wishful thinking, and make planned schedules more realistic. If expectations are reasonable, everyone can relax and get on with it without resorting to that most time-consuming of development modes, panicky hacking.
It isn’t always possible to convince management to allow you a reasonable deadline. Market pressures mean that the deadline is regarded as immovable (e.g. go live in mid-September ready for the Christmas rush). In this case, the only possible way to achieve this is to cut down the amount that will be delivered. The customer can have some of what they want by their proposed deadline (and get the rest later), or they can “go for broke” and almost certainly end up with none of what they want.
Taking this reckless approach, the quality of the product inevitably suffers. They might get what they want eventually, but it probably won’t be until Christmas the following year. It’s even more likely that they will end up with a buggy mess that has to be scrapped.
In this instance, it is vital to make the customer understand that leaving something out of this delivery phase doesn’t mean they will never receive it. They may just have to wait a few months. Prioritisation is key. Use DSDM-style MoSCoW rules if you must (but don’t be surprised if everything ends up as a “Must-Have”). You need to find a polite way of saying “You can have some of what you want now, or you can try to have it all and end up with nothing.” If the customer can be made to understand this, then 90% of the battle is won.
However, what happens if your project really is suffering from slow development, and not just the illusion of slowness?
If your project is experiencing the real thing – if it really is suffering from slow development, and not just the illusion of slowness, then you need to tackle the problem head-on.
For example, ICONIX Process (described in the book Use Case Driven Object Modeling With UML, A Practical Approach by Doug Rosenberg and Kendall Scott) provides many ways to identify and deal with “analysis paralysis” – the problem of getting bogged down in a particular section of the project (such as, painstakingly drawing state diagrams for every class in your static model).
A major part of communicating rapid development also ties in with our re-engineered process (which luckily gives this article some much-needed cohesion):
To the manager, code = progress. So make sure they see lots of code. You can show them lots of reports that demonstrate that the project really is on track. But what they really want to see is new software. This is, after all, the objective of the project. Paper to them is not progress, new software is.
To the person for whom we are writing this system, working functionality = progress. So make sure they see frequent small releases. This does not necessarily mean giving them actual production code to “go live” with every two to three weeks (as would be the rather tiring case in an XP project). This would drive most customers mad. Consider a blue-chip client that likes to spend six months performing their own acceptance tests for any new software. Are they going to initiate a new set of tests every three weeks? Even scaled down to a smaller project for a less quality-obsessed organisation (the cads!), this sort of rapid-fire release schedule represents a major logistical headache.
Instead, make sure the customer sees progress in action. Make sure they can push buttons and see things happen, see message queues filling and emptying and invoices being processed (even if they aren’t actually doing anything, or hooked up to anything, at this early stage in the project). If it’s visual, show it to them. As soon as something has been connected up to something else, and it works, show them. At the very least, regularly email them the latest screenshots (whether they ask for them or not) so they can see the project progressing, and provide feedback.
Start Producing the Front End Early
A neat trick to help convey this sense of progress, and hence ease the customer’s jittery nerves, is to produce the front end early. This could be a Visual Basic or Swing GUI if it’s a desktop application (once considered old-school in the face of groovy new HTML front-ends, but now making a welcome comeback). If your project isn’t GUI-centric (e.g. a server product or a data processing system), you’ll have to make do with whatever material you can use that still conveys a feeling of progress.
That really is the key point: you need to convey a sense of steady progress throughout the project. This means regularly releasing new sections of the user interface, which will delight the customer and keep them enthralled as they discover new areas of their burgeoning new product, as if they are exploring an adventure game. Keep rewarding them with new areas to explore.
Rapid GUI prototyping is a popular method of quickly reaching a common understanding of the customer’s requirements. For example, each iteration of the ICONIX Process (mentioned above) actually starts with a GUI prototype (or in some cases even a prototype user manual), from which the use cases are identified. This initial effort may not end up being anything like the real GUI, as the use cases will be honed and modified as the analyst develops a greater understanding of the customer’s requirements.
However, the GUI prototypes are a great starting point, because they help the customer to visualise what it is that they really want the system to do. As an added bonus, they can also help to convince them that things really are progressing smoothly (especially when they see more and more of the agreed-upon user interface steadily becoming “real”).
It is possible to plan the entire project lifecycle around the perception of rapid development. Remember, keeping the customer happy is as important to the project’s success as delivering the project for real.
Matt Stephens is a systems architect and software team leader working in
Central London. He’s the co-author of
Extreme Programming Refactored: The Case Against XP (due out in
August 2003) – a hard-hitting book which challenges the hype surrounding
XP (the book also suggests some better ways of achieving XP’s agile
goals). Matt is also editor of (and regular contributor to) SoftwareReality.com, a
satirical website for software developers.
