Managing complexity and communication in software projects

Most software projects fail because they're too large and teams can't communicate effectively. I wish there were a more complex explanation, one full of crazy metaphors, but there isn't. Large projects are hard to manage because complexity doesn't grow linearly, but exponentially. Regardless of the process, whether it's more agile or more waterfall, creating a software product is hard.

I've been thinking about complexity management a lot lately. As humans, one of the hardest things we do every day is to construct and deconstruct thoughts and ideas. A successful software project, that is, a project that successfully solves user needs, is planned, executed, delivered, and enjoyed by users, requires multiple groups of people to deconstruct and construct ideas from different angles, in tandem, using the same vocabulary and semantics. If this sounds hard, it's because it is.

Large projects: the cycle of deconstructing and constructing

Large projects don't work for a straightforward reason: complexity build-up isn't linear, but exponential.

If a simple screen with a chart has 4 engineering tickets and it takes 1 sprint to build, a screen with data fetching won't be 4 tickets: it'll be 10. And so on. And so on. By the time the project gets to QA, half of the team doesn't even remember why the project started in the first place, QA testers can't feasibly test all interactions with other sub-systems plus all the components, and users... don't care.

For the project above to work, you'd need a team of 3 super-gifted machine-humans with the greatest capacity for insight gathering, recall, and synthesis. Those 3 humans don't exist. (If you know these 3 humans, please let me know)

“You don't need 20 friends, you only need 3 motherf*ers to take over a country” - Joey Diaz

Agile can't save you. Lean won't save you. Waterfall will kill you.

Silver lining: Bite-sized, user-tested iterations

Not to be Occam's razor of product building, but in this case, the simplest answer is the best. Make your iterations smaller. No individual iteration of a project should be longer than a month, at most, unless it's an architecture project impossible to break down.

Each iteration should add value independently – quite literally a minimum viable product –, should be scalable onto the next iteration, should be measurable, and should be testable with users. Doing this means that any given iteration will never go past the Limit of Despair, where the complexity of the project is such that testing it successfully becomes impossible.

This is also an issue with communication. Speccing and designing a product is nothing other than creating an abstraction for other people to do their work, which is to plan and develop, which in itself is an abstraction for the next person, ideally the user. Having smaller, bite-sized iterations means you can articulate a project better and everyone involved understands why you're building it.

Silver lining: Communicating the why

If by the product spec review meeting, the team doesn't understand why you're meeting, and why you want to build something, you've failed. One of the goals of having smaller iterations is that you can communicate more and more effectively with the whole team.

The problem with longer, larger projects is that by the time they're done – if they ever get done – the team loses awareness of why the project is being built in the first place.

In exceptions like the ones mentioned above, where you just can't feasibly break down a project into smaller chunks, part of your job is to make sure you reduce the risk of failure by communicating early and often.

Conclusions

There is a lot more to managing complexity in a software project because it's why a lot of software projects fail. A treatise on this would be a few volumes long. I assume this maxim could be applied to most business problems, but I'm a software person and it's what I know.

Building software, at its core, consists of automating and simplifying user problems. The amount of work that goes into it is massive, and most endeavors fail even in the earliest stages, by misunderstanding what the user needed in the first place. Bite-size iterations should be tested with users; which users and how is probably worth another article.