In our latest adventure at work, we’re racing headlong and headfirst down the road of “emergent design” in Agile software development. We’re learning how to “fail fast and fail often” the hard way, and it’s been a challenge to determine whether we’re failing “correctly” — if there really is such a thing!

Traditionally, software was developed in some form of “waterfall process” in which a relatively firm set of requirements were defined by sales or marketing, which then flowed down to engineering who then designed, developed, and tested the product. The product was usually well-defined, designed, developed, and tested with long-term considerations in mind, and wasn’t released (usually) until sales/marketing had received the product or feature they wanted. Then the entire process would start over with the next set of marketing requirements flowing down from on “high” again.

Unfortunately, the problem with the waterfall process is that it’s often slow, cumbersome, filled with latency/bottlenecks, and the products can become quickly outdated and often difficult to maintain over time. Sometimes the market is simply changing faster than the company can keep up with, and the product has become irrelevant before it’s even released. In certain markets, the product has become outdated even before the requirements are finished being written! Most software products are like any other perishable product: they have a certain shelf-life of a few years (or months!) and then they’ve lost their usefulness. In the worst cases, it’s like trying to bake a batch of cookies in which the ingredients expire before they’re done being mixed!

One of the more popular alternatives to the waterfall process is emergent design, in which the software development life-cycle is greatly shortened but sped up, and the product rapidly “emerges” in a series of small steps instead of one big release. In the end, the same product may be made available, but the user-base sees the product or feature evolve in a series of frequent steps with more functionality being added every few weeks instead of several months — or longer.

From Wikipedia (the final authority of all that is known on the Internet), “With emergent design, a development organization starts delivering functionality and lets the design emerge. Development will take a piece of functionality A and implement it using best practices and proper test coverage and then move on to delivering functionality B. Once B is built, or while it is being built, the organization will look at what A and B have in common and refactor out the commonality, allowing the design to emerge. This process continues as the organization continually delivers functionality. At the end of an agile release cycle, development is left with the smallest set of the design needed, as opposed to the design that could have been anticipated in advance. The end result is a simpler design with a smaller code base, which is more easily understood and maintained and naturally has less room for defects.”

While I’m still getting used to working in the emergent design world, I can see its benefits — and it’s drawbacks. The product/features receive frequent feedback from the user-base and the software changes to meet their needs and/or demands accordingly. Emergent design gets a larger user-base involved in shaping the product earlier and more often rather than at the beginning and end of the process. However, while emergent design does solve some problems, it also causes some that could be avoided with the more traditional waterfall process, namely in the design phase. Going back to the cookies example, sometimes emergent design feels like you’re trying to bake a batch of cookies in which there’s no written recipe (just general guidelines) and the batter is being mixed by trial and error! Not only that, but the recipe is being continually refined based upon your kids’ taste-buds at any given moment!

I hate having to do a job twice – especially jobs that I’ve put a lot of time and effort and heart into. I was taught that if you have to do something twice, then you didn’t do it right the first time. I dislike writing “disposable” code, functionality that I know will likely be thrown away in a few months or even weeks! It’s one thing to write code in such a way as to prepare for new features you know will be coming soon, but quite another to write code that you know will be deleted or never even seen by the users!

Needless to say, that’s become a big part of my job over the last year or so – continually reworking and refactoring roughly the same code over and over again. A tweak here, a tweak there, with sometimes an ugly gutting of all the innards and practically starting over. While I understand and mostly agree with the new process, sometimes emergent design goes against every common-sense software-development bone in my body! Before we code up something, how about we slow down for a minute and make a good design that will last longer? Instead of swinging from one extreme (waterfall) to the other (emergent), can’t we come up with a happy medium somewhere in between?

In both worlds, the user-stories/requirements change and the software has to be modified accordingly. Customer A likes this feature but Customer B doesn’t, so we try to find a way to make both happy. In emergent design, the process is sped up and both customers end up getting a portion of what they really want. They may want a Cadillac, but they have to settle for a 10-speed, then a Kia, and then a Lexus (if they’re lucky). The software doesn’t necessarily become better and better as much as different and more inline with what the customers want (or can somewhat agree to), though that often changes over time too. I’ve been working on this project from Line 1 on Day One and while it’s continually changing, it’s not always evolving in a certain defined direction as much as being more “fleshed out” as time goes on.

Maybe that’s the way it is with us – maybe we’re not always improving in any tangible, particular ways as much as being stretched, deepened, thickened, and being more fleshed out. Circumstances outside our control change and we have to respond accordingly. People are constantly coming and going, feeling, responding, changing, and affecting us (for good or bad). Maybe we who are being molded and shaped by the Divine Potter don’t always become better and better in the way we think we should be as much as different and more inline with what He requires of us at that particular point in time, though that often changes too.

Maybe it’s more important to be walking with God along this long, winding, and bumpy road — wherever He may be leading us — than it is to get to what we think should be our position or destination as quickly and directly as we can. We like all those shortcuts to make things faster and easier on ourselves more than how the long, difficult journey will prove our character and faith — but He doesn’t. I think He is much more concerned with how we’re responding to Him and those around us at any given moment and seeing us draw closer to Him with every step along the way. I think He’s more concerned with how we emerge from our circumstances and how we conform to Him than anything else.

Maybe God is more concerned with our character and how it’s being shaped day-in and day-out than how quickly and easily we get Home, because that’s already been assured once we become part of His Family. He expects us to get battered, bruised, and calloused along the way, doesn’t He? After all, those blisters prove we’ve been walking a long way for a long time, don’t they? He expects us to live fully and take risks just like the men the shrewd master entrusted his talents to (Matthew 25:14-30). He’s already promised to bring us Home and be with us all along the way on this Journey, but He hasn’t said much about the ardors of the Journey itself!

Though our lives and our walk may indeed resemble software being written by “emergent design”, we can trust that the Potter knows exactly what He’s doing and that He’s shaping all of us precisely how He wants us to be not only at any given moment, but the final product of our lives.

Now to Him who is able to keep you from stumbling, and to present you faultless before the presence of His glory with exceeding joy, to God our Savior, Who alone is wise, be glory and majesty, dominion and power, both now and forever. Amen. — Jude 24-25