Feb 062014
Ed Sumerfield and Chris Nelson of Gaslight Sof...

Ed Sumerfield and Chris Nelson of Gaslight Software presenting “Why Agile Fails” (Photo credit: RubyNuby)

How the fuck is this coming around again?

George Westwater on Waterfall

Every so often, someone will put forth the idea that agile practices are great and all, but they fail at certain things like providing an up-front budget, a well-researched scope before work begins, a full project timeline with milestones, etc.  These sorts of things are such staples to the way most orgs approach their projects that there’s a certain tension to make agile practices better at providing them, and the path of least resistance to trying to supply these is to use the practices that used to provide them – typically some type of phase-gate approach like Waterfall.  You hear things like this:

  • We complete all the requirements and design up front, but then we do our development in an agile way.
  • We do agile, but in a way that can be managed with Microsoft Project.
  • We’re agile, except we do our testing at the end, but at least we release every six months.

And so on.  The basic idea is that doing a project without these traditional Waterfall additions is somehow inherently more risky or will fail to provide a company what it needs to make intelligent project decisions.

I’ve already spent a lot of electronic ink on this blog talking about how the benefits Waterfall purports to offer are largely illusory (kind of like the illusion of security offered by being employed at a large company) and how various agile methods can actually do a better job at providing intelligence to the business, so I don’t want to rehash all that.

However, the point I want to make here is that Waterfall and Agile arose from very different systems and operate off very different assumptions about the work.  For instance:

Waterfall’s Assumptions

  • The work to be done is mostly static.  Changes are rare and always occasion for re-examination.
  • Business needs / the market is not volatile; our priorities and methodologies today will be very similar three years from now.
  • The work is extremely similar to work that we have done before.  Almost everything is a known quantity.

Agile’s Assumptions

  • The work to be done changes often and progressively as more clarity is gained or needs evolve.
  • Business needs and the market is volatile.  The thing on the front burner today might be on the back burner in three months.  Our methodologies are evolving.
  • The work is to develop something new.  Similar technologies and methods might be used, but the business problems we’re solving technically have not been solved before.  Otherwise, we’d be using that.

The problems with Waterfall are not necessarily inherent to Waterfall as a methodology.  On paper, it looks all right.  The problem is that the set of circumstances under which it operates best is pretty rare in software development.  Are there software development projects where the domain is pretty static, changes are rare, and the business changes very little?  Sure, those do exist, and Waterfall would work just fine.  In fact, it or something similar might be the best choice for those kinds of projects.

However, a lot of organizations believe Waterfall’s assumptions to be true of their situation, and they are not at all.  Most real organizations have work that looks more like that list of Agile Assumptions, and in that context, agile methods will probably fare much better.

The issue with combining the methodologies isn’t a religious one; it’s that you’re trying to combine two things that are appropriate in two very different contexts.  You’ll end up crippling Agile by trying to make it more Waterfallish because you’re trying to drag in a context that doesn’t apply.  It’s like adding a sail to your car.  Sails are pretty good in their context, but they’re not objectively good for all contexts.

The fact is, becoming agile will challenge a lot of illusions and assumptions we make about our work that are not accurate.  If you’re hearing something like, “Agile makes it difficult to budget for the whole project up front,” that’s your cue to examine if budgeting a whole project up front is really the smartest thing to do as opposed to paying for value as you go.  If you’re hearing, “Agile makes it hard to set project milestones,” that’s a good opportunity ask yourself why you feel you need project milestones, what value they offer, and could you get that value in better ways?  Agile is meant to change your understanding, your values, and ultimately your practices at an organizational level.  It is not designed to accommodate them.

So, before you decide to add a sail to your car, just give driving the car a shot.  You might find that it doesn’t need that sort of improvement.



Enhanced by Zemanta
Dec 032012
5GW Waterfall Model

5GW Waterfall Model (Photo credit: Dreaming 5GW)

If you get into any kind of open forum discussion regarding the benefits of more agile or more lean approaches to traditional software development, it’s not uncommon for someone to say that the problems organizations experience with Waterfall are largely because they aren’t doing Waterfall right (incidentally, this is also what most of the Scrum folks say when organizations experience problems doing Scrum, but I digress). The idea is that, if organizations were disciplined and knowledgeable enough to implement Waterfall properly, the way it was designed, that most or all of the high risk issues in Waterfall wouldn’t be issues.

It reminds me of martial arts forums. Someone will post twenty videos of Bad Won Do practitioners getting their tails kicked, and someone will comment that what they’re doing isn’t “real” Bad Won Do.  “Real” Bad Won Do is actually quite effective (and apparently undocumented) and, if they’d just do the real stuff right, they’d be powerhouses.

But here’s the thing. Even if you did Waterfall with 100% perfect execution, it remains an extremely high risk, high fail project delivery methodology.

1. Testing Comes Too Late in the Process

Winston Royce, the man who brought a lot of visibility and popularity to phase-gate approaches to software development (and accidentally contributed to the success of Waterfall) wrote this in “Managing the Development of Large Software Systems:”

The testing phase which occurs at the end of the development cycle is the first event for which timing, storage, input/output transfers, etc., are experienced as distinguished from analyzed. These phenomena are not precisely analyzable. They are not the solutions to the standard partial differential equations of mathematical physics for instance. Yet if these phenomena fail to satisfy the various external constraints, then invariably a major redesign is required. A simple octal patch or redo of some isolated code will not fix these kinds of difficulties. The required design changes are likely to be so disruptive that the software requirements upon which the design is based and which provides the rationale for everything are violated. Either the requirements must be modified, or a substantial change in the design is required. In effect the development process has returned to the origin and one can expect up to a l00-percent overrun in schedule.

In other words, since you’re testing the whole system at the end, you run a very high risk of major rewrites to fix issues.  This is Waterfall done right.

2. Value delivery comes at the end.

You release at the end of the project. Nobody gets any value out of your work until everything is done. This is a huge risk for a number of reasons.

First of all, the project might not get done. In fact, many projects meet this fate. Funding runs out. Priorities change. Dissatisfaction with the vendor. When this happens, the business is often left with nothing. They just eat the cost of a ton of work that got them nothing but stacks of documentation and some half-baked code. Since value wasn’t delivered incrementally, they have nothing.

Second, there is an enormous risk that what was valuable nine months ago is no longer valuable, either due to changes in the business or just the fact that the value of requests expires over time. It would have been useful when it was requested, but not nearly as useful now.

Third, the funder of the project has no real option but 100% success or 100% failure. You either get everything, or you get nothing, and this leads to some very unhealthy decisions like sticking with a vendor who isn’t performing or continuing to pump funding into a project that should have been abandoned or thoroughly revised months ago.

This is Waterfall done right.

3. Waterfall is predicated on a faulty premise.

Waterfall assumes that I can fairly accurately predict timelines and resource needs for a project that will involve several people extending over a large amount of time.  The entire notion of a phase-gate approach like Waterfall is the idea that I can get all the requirements, lock those down, do all my design, etc. And all the trappings of my project management revolve around this presupposition.

And this might have a glimmer of hope to it if software requirements were cookie-cutter kinds of things that all required the same resources, same effort, same complexity, same outcomes. But as we know, software development is way, way more variable than that.

Can you imagine a general making a battle plan for a year long campaign and sticking to that plan the entire year? It doesn’t happen. Yes, having a plan is important, but the plan has to be intrinsically adaptable because conditions on the ground are changing. Some are predictable, some aren’t – the variables in war are endless.

I’m not saying software development is as volatile as war (most of the time), but the principle is very similar. I mean, what if I asked you to plan down to the hour your work days for the next year.  Could you do it?  What about the next month?  Next week?  I think you’d find this challenging, because each day of work brings up new variables, emergencies, challenges, etc.  So, I ask you, if you couldn’t even plan the hours of your work week in advance, how realistic is it to assume that you can do that for several people, departments, and resources over the course of six months to a year and expect the outcome to still be relevant to all stakeholders?

This is Waterfall done right, and this factor is, I believe the most problematic, because it gives an illusion of certainty that does not actually exist.

Enhanced by Zemanta
Jan 172012
Waterfall Model

Image via Wikipedia

In recent discussions with groups of agile coaches, mentors, Scrumbots, etc., I’ve noticed a sort of ad hoc dividing line between people who really know what they’re doing and people who are still struggling with it.  It’s not the only line, but it’s one of them, and that line (how many times can I use the word “line” in the opening paragraph?) is this:

Does this person confuse processes with principles and/or value processes over principles?

Businesses love processes because they are comforting.  You can document them, replicate them, measure them, and enforce them.  You don’t have to rely on people thinking very much, because there’s a process.  As much as people might try to deny it, the overwhelming majority of people actually like being told what to do.  It takes a lot of pressure off.  So, I do get the attraction and, for some of the reasons above, I’m not anti-process.

However, a process should always be built around certain principles or goals and critiqued by their effectiveness in maintaining those principles / achieving those goals.  It is never ok to stop asking the question: Is our current process still the best way to accomplish what we’re trying to accomplish?  Once you stop asking that question, the process then becomes an end unto itself.  It is now inherently good to follow the process.

Remember the story about the goose who laid the golden eggs?  The reason you valued the goose wasn’t because geese rock; it was because it laid golden eggs.  What you really valued were the golden eggs, and if the goose ever stopped laying golden eggs, you wouldn’t think anything more about that goose than any other.

This is similar to processes and principles.  What you really value are the principles.  Processes only have value if they effectively deliver on those principles.

This applies to many, many aspects of life, but in software development, it applies a lot to companies putting implicit faith in a process rather than critically examining if the process is actually accomplishing what they wanted in the first place.

A commitment to Waterfall obviously comes to mind, but far more insidious is a commitment to old school, textbook Scrum.

Companies want to be more agile.  Scrum is a process that purports to bring you agility, so companies trade blind dedication to one process for blind dedication to another – something Scrum tends to encourage, incidentally.  You even have a person (the Scrum Master) whose entire mission in life is to make sure everyone is following the process.

Companies don’t want to critically think about why they aren’t agile and what things they might need to correct to get them there – they want someone to tell them what to do.  “Do this stuff, and you will be agile.  You will deliver working software faster.  You will value people over processes, with the possible exception of this one.  You will be able to call your team agile.  All you have to do is follow this process to the letter, and magic will happen.”

Unfortunately, that’s not how it goes.  Enforcing Scrum to become agile is like trying to cure all sickness with penicillin.  You can’t just follow the process and assume faster, higher quality software will be the inevitable result.

I’m not anti-Scrum per se, but I am very much anti-making Scrum the new Waterfall.  Don’t stress about your process.  Don’t stress about the integrity of your process.  Think about your actual goals for your team and constantly correct your course to get there.  Change often.  Improve constantly.  There is no textbook procedure that will guarantee a cornucopia of agility just by virtue of being implemented.

Dec 072011
Armlock shown is a straight arm bar (juji-gatame)

Image via Wikipedia

I practiced several different martial arts in my lifetime before finding a home in silat, and one thing that came up in virtually all of them was how joint locks work, which is kind of a fascinating study of the way people react.

A joint lock basically involves bending someone’s joints in a way that they’re not supposed to go or to a degree they were never meant to bend, such as the infamous “chicken wing” with the arm bent up behind the back, or the “gooseneck” with the elbow immobilized and the wrist pulled down. This creates pain and a degree of immobility in your opponent, hence the “lock.”

Here’s the funny part: almost any lock can be escaped relatively easily, and the vast majority of them can be reversed – locking the locker – without much more trouble.  A joint lock works because, when a person feels that pressure and pain, they think, “Crap, I’m locked,” and generally struggle against the locking pressure, which generally just makes the pain worse. They don’t think of a way they can move that would relieve that pressure; they become fixated on trying to push or pull their way out of it.

In essence, people lock themselves; the person executing the lock has just created the circumstances that allow you to lock yourself.

Across the wide world of software development, developers and their clients have locked themselves.

We have all assumed that giving an up front time and cost estimate for a project is the only way to measure progress, control cost, and compare companies.

We have all assumed that we have to get all our requirements before we begin work.

We have all assumed that testing and debugging is something you do after completion.

We have all assumed that deadlines have inherent value and meaning, even if there’s no business rules driving it.

As a result, we have come to accept that changes are painful and expensive, projects will virtually always miss deadlines (or require a lot of overtime crunch to get them done on time), and that every release will have bugs.

We don’t stop to think about ways we might move to relieve these pressures – we just push and pull against them, accepting the “reality” that we are locked into this way of thinking and doing things. This is just the way you do software development. Estimate – Requirements – Design – Build – Test – Deploy.

Believe it or not, software does not have to be built in this way. This is not your only option. This is not the only workflow that makes sense. Missed deadlines, expensive changes, 12 hour work days the week before a release, and bug filled releases do not have to be just “how things are.”

The way you get out of a joint lock is by being relaxed, fluid, and moving with and around an opposing force. Folks: welcome to Agile.

Sep 302011
BRADENTON, FL - FEBRUARY 28:  A napkin holder ...

Image by Getty Images via @daylife

Diner 1: Hey, do either of you have an extra napkin?

Waterfall Guy 1: I have made detailed plans and estimates on exactly how many napkins I will need for this meal.  Therefore, no.  If I finish the meal and have overestimated my napkin needs, that’s a very good thing, and I’ll be happy to give you my extra napkins.  Historically, I tend to underestimate these needs, which will lead to either a request for more napkins or drawn out sessions where I use the same napkins I have until they dissolve into component particles.  Either way, I won’t know until I’m done, but I wouldn’t hold my breath.

Agile Guy 1: Any napkin I’m not using right now is an extra napkin.  Take as many as you like.

May 242011
Example of an American grocery store aisle. 

Image via Wikipedia

Cast: Bill and Ted – roommates

Bill: Hey, Ted.  I just noticed that we’re getting kind of low on groceries.  Do you think you could run out to the store?

Ted: No problem.

Bill: Great.  How long will that take?

Ted: What?

Bill: How long will that take you?  I need to plan my day.

Ted: Well, I don’t know.  I don’t know how many groceries we need, and even if I did, I don’t know what traffic will be like or how busy the store will be, or even if I’ll have to go to two stores because they’ll be out of something.

Bill: Look, you’ve gone to to grocery store every week for two years since I’ve known you.  How can you not know how long this is going to take?  Please give me a number so I can plan my day.  Also tell me how much the bill will be.

Ted: I have no idea!

Bill: Come on, Ted.  You’ve done this a hundred times, and you don’t know about how much it’ll cost?  How am I supposed to budget without knowing how much you’re going to spend?

Ted: Ok, it’ll take an hour and I’ll spend $200.

Bill: That seems like an awfully long time.


Bill: Wow.  You have real communication issues.  Anyway, do you think you could get it done in half an hour?

Ted: Sure.

Bill: Great.  You’re committed to half an hour with a budget of $200.

Ted: Fine.  Ok, now, what groceries do we need?

Bill: Whoa!  Hold on there, sport.  You can’t just go charging in to that sort of thing.  We need to figure out what car you’re going to take, how much gas you’re going to use and how much that will cost, and what the optimal route is to the store.  Then we need to make a list of our ideal grocery stock and compare it to all the items we currently have on hand to create a list of deltas.  Then we need…

One week later…

Ted: All right.  We’ve mapped out the route I’m going to take, made all the logistical decisions, and attached a timeline and dollar figure to each item.  I now have a plan where every detail has been accounted for, and I am ready to begin the process.

Bill: Well, the problem is we’ve eaten more groceries since you’ve started that process, so your list is wrong.

Ted: That’s a real shame, because our entire plan hinges around it being correct.  If you want to make changes this late in the game, we’re going to have to do some more requirements gathering and figure out the impact to our times and budget, and just that process alone will have impact to the time and budget, so maybe we should vote on whether or not we even want to…

Bill: All right, all right.  Nevermind.  Just get the groceries we planned on, I guess, and we’ll just live with it.

Ted goes to the store…

Ted: (singing) Implementation phase, implementation phase, oh how I love the implementation phase.   (stops singing) Wow, traffic sure is terrible.

At the store…

Ted: I didn’t know today was National Broccoli Day.  This place is packed.  It’s taking forever just to get down one aisle.  Crap, they’re out of Baco-Bits.

Back at home…

Bill: Ted, your trip to the store took 50 minutes – almost twice what you estimated.  Also, you spent more on gas with your trip to another store to get Baco-Bits, which put you over budget.  Furthermore, what you did get fell far short of the groceries we actually needed by that time.  Our plan that you, yourself, helped to create spelled everything out, so I can only assume that you’re a bad grocery shopper who can’t even follow simple directions.  You’re fired.

Ted: Thank you.