Agile murdered babies

The agile movement started as a counter movement that sought to free the developers from the heavy-weight mammoth processes of the late 90’s and the start of the millennium. While there were many heavy-weight software development processes, some of them even iterative and incremental, the antagonist and the bogeyman of the agile movement has always been the nefarious Mr. Waterfall, a straw man of many vices.

While the agile movement has brought about many good things, and I agree with most of the agile manifesto, it has also thrown away many babies with the bath water. One of the frameworks I especially blame is Scrum, which, while describing a useful way of working for a development team, has left its followers to believe that product owners are Moses-like beings, who can dream up business requirements while meditating on a mountain. Namely, Scrum is to blame for our community’s negligence on the methods of requirement elicitation and problem analysis.

The problem with strong movements like Scrum, or more recently the Lean Startup, is not that they would be completely wrong or that they wouldn’t work in some specific settings. The problem is that they clearly do not work in all cases, but their authors and proponents do not acknowledge this. All things start to look like nails when you are holding a hammer. All we can say is that this phenomenon is not restricted to software engineering, as you should realize when you hear prisons calling prisoners their customers.

It seems that we, as an engineering discipline, are doomed to repeat our mistakes and so the counter-counter movement to agile has already begun with the advent of heavy-weight frameworks such as the Disciplined Agile Development – which has its roots in the Unified Process – and the Scaled Agile Framework. Having taken a short look at them, I predict a counter-counter-counter movement in a few years’ time.

Getting back to the murdered babies, another area of software methodology that was overrun by the agile movement is software architecture. While the agile manifesto per se says nothing about not doing architectural design, many of the zealots have interpreted any kind of planning to be a sign of the lowly BDUF, Big Design Up Front. Agile architecture is supposed to emerge, not be planned up-front. You are simply not supposed to think if you do agile and you definitely should not look beyond your nose. Kent Beck, I blame you!

Architecture is a philosophical question

So what exactly is software architecture? A very philosophical question that escapes the mortal programmer. In fact, there are hundreds of scholars that have proposed their own definition and the Software Engineering Institute at CMU is keeping a record of these attempts. The list is long and growing. One of the most common themes in these attempts revolves around components and connectors, for which we can thank Mary Shaw. For me, that is not enough since it neglects the existence of architectural textures, things that you cannot componentize – like, for example, error handling or logging – which are present in all components.

For me, the things that the well-known object-oriented programmer Martin Fowler has said about software architecture hit home:

  • the highest-level breakdown of a system into its parts
  • the decisions that are hard to change;
  • there are multiple architectures in a system
  • what is architecturally significant can change over a system’s lifetime
  • and, in the end, architecture boils down to whatever the important stuff is

Software is not a house

I have always hated it when people compare developing software to building houses. The analogy has many fallacies. For one, building a house takes months, while the software build process takes usually some seconds. That is, unless you are a TDD-ist, in which case you are probably struggling with your build times.

At least if you are building anything big, like enterprise big, a better analogy would be that software is like a town or a city – and thus doing software architecture is like urban planning. I’m not implicating that all towns have to be planned. The world is full of towns whose architecture has emerged as opposed to having been designed. In fact, some of the most intriguing small towns have grown this way. But most of them have burned down because they didn’t have a proper architecture to prevent fire from spreading.

Likewise, at the other end there are many small and large cities that have been over engineered. Small towns where hundreds of millions have been wasted in building the grandest five-level high-way crossings where a simple one-level crossing would have sufficed.

Just like with urban planning, you will have many concerns that you need to listen to when doing software architecture. Should we have a city where you can use your own car to get about, should it have a working public transportation system, or both? The answer changes depending on who you ask.

The common thing about software and city architecture is that while some changes are easy, others are near impossible. For example, old houses can be demolished one or two at a time and new ones built in place without disturbing the whole city. However, try building a new railroad line or a decent bicycle network into a city that didn’t have a reservation for one and you will be stuck with the project for the next 20 years. And you will be hated by at least half of the city.

With software it is the same. Unless you have designed for your cloud-based-micro-service-oriented system to be used off-line in a mobile environment, the transition will most probably mean writing a new system from scratch. That is not to say that you should design all your software to be ready for off-line operation or that writing things from scratch would be a bad thing.

The analogy goes even further. Just like an urban planner can’t be bothered with what the individual homes look like, software architecture shouldn’t concern itself with the details of individual components. When an urban planner leaves a reservation for a high-way around the city, it doesn’t mean the high-way needs to be built on day one. The same goes for software. You don’t need to design everything up front, and even when you are smart and design something up front, you don’t need to implement it right away. Be warned though, even reservations can have a hidden cost.

Here, have some silver bullets

So how should one design software architecture in an agile project? Here are some of the things that have been said about doing architecture in our agile day and age.

Plan ahead, think big, know where you are heading. The cost of thinking ahead is very small and it absolutely cannot harm you. Beware though, you should not fall in love with your plans. Be ready to change them. Planning ahead isn’t the same thing as daydreaming or fantasizing.

Avoid premature commitment, act small, delay decision until the last responsible moment. Just because you plan something, doesn’t mean you need to commit to your plan. Commitment comes in many forms and you need to be aware of them. The most deceptive kind of commitment is mental. Other types include financial (you already own the Oracle license) and contractual commitment (your customer chose Liferay for you) and of course code itself.

Do the simplest thing that could possibly work. That is, follow the Occam’s razor. Software is complicated as it is, you don’t need to complicate it further with a fancy software architecture that supports features no-one asked for.

Fail fast, be risk driven. If you have committed yourself to an architecture, you want to know quickly if it is going to work or not. If you are not sure something will work, you better test it out. The agile way of proving that it works is to code it, do an architectural spike, a possibly throw-away prototype.

Prototypes and code is certainly a good way to test things out, but while spiking your way to success might work, it sounds like a very steep and thorny way to heaven. Personally I will first do a gedankenexperiment and only then resort to code.


I hold true all the above guidelines, but they are not enough. How do you decide what is simple enough and what is too simple? What shall you spikes or though experiments be about?

Making educated decisions about software architecture is not easy. It is even more difficult if you have nothing to go by.

Architecture should always be based on the requirements of the system. Not all of them mind you, just those that are architecturally significant. I call such requirements architectural concerns. The idea is by no means original. Philippe Kruchten had a similar idea in his famous 4+1 paper, where he described the use of scenarios (specific instances of use cases). Likewise Kent Beck says that the first iteration of a project should put the architecture in place by implementing a set of stories that force the team to create “the whole system”.

So what are they really? How do you know something is a concern? Architectural concerns are the system-level things that keep you awake during the night. How will you do X in your system, be that functional or non-functional? How will the system replicate data to the off-line clients? How will the system handle the estimated load? How do we deploy changes without affecting end-users? How will we distribute the CPU or memory intensive process X?

The one mistake I advice you against is to think of all the “-ilities” (scalability, reliability, availability, modifiability, etc.) as your concerns. Don’t make a fuss about something that is not an issue. If you are only going to have few users or very little data, then you probably don’t need to think about scalability. You can do a thought experiment to decide if something is an architectural concern or not. Ask yourself what happens if you don’t consider the thing now? Will the software be forever doomed or will things most probably just work out? If the latter, then don’t worry. It is not a concern.

After you have identified a concern, you can look at them and validate your current thinking about the system architecture. I like to use concerns for simulations. I take a concern and I reason to myself how the system is someday, through a series of refactoring, going to answer to the concern.

That someday is probably not now. The plan is just a goal that you can start working towards. Along the path you will learn new things that will make you change your goals. Remember, you should not commit prematurely to any plan. And as what goes into working towards a goal, you need to understand that outside the beaten track and on the rocky waters, the fastest path is seldom straight.

Heikki Vesalainen
Chief Architect, TrademarkNow

by Heikki Vesalainen.