Common Mistakes when Planning a New Software Project

Once Upon a Time

Years ago I was asked by a information security firm to build a custom scanning software system. I submitting an initial sketch of the plan with an estimate in time and cost. This was not sufficient, however, as the firm needed more exhaustive diagrams and descriptions before committing.

After agreeing to an initial analysis phase for us to figure out all the minute details, we spent a couple weeks drawing database and class diagrams. We pored over the requirements document, picking out tiny details to clarify and generating several meetings, emails, and Skype chats (before everyone was using Slack or Discord or Riot). In the end we had a forty page PDF describing every step of the way of the development requires. We got the green light.

It took 72 hours to discover a few unexpected twists that caused significant deviation from our meticulous plan. For example, the data structures one of the other scanners with which we had to interface was not in line with documentation we used during our analysis phase. If we had read source code over the maintainer’s documentation instead, our analysis phase would have taken longer than the actual build itself! There are always hazards beneath the surface.

“No battle plan ever survives contact with the enemy.” - Helmuth von Moltke

I found this to be a repeated pattern for many more client interactions on planning new software projects. The premise, it seems, is that extensive planning—down to class diagrams and database schema—reduces risk for stakeholders. The contrary is the case. I believe this premise arises from some mistaken beliefs about IT management as well as mistaken attitudes about the work of programming.

Mistake #1: Programmers are strictly left-brained analysts

The myth perhaps starts on the observations that computer science is placed as a branch of mathematics. Mathematics is traditionally believed to be a left-brained, analytical, and largely non-creative activity. Programmers are thus analytical thinkers who behave and work with pure logic only. Therefore, they can be concatenated together on an assembly line to feed each other the one true and best solution to a given problem as part of a team.

This idea leads to folly.

Mathematics has hundreds of branches. New branches appear frequently. For every theorem, there may be dozens of proofs—each reflecting the creativity of the mathematician who authors them. Theorems and proofs appear inspired from observing snowflakes, grains of sugar floating in coffee, or watching birds. And the most important work is inductive in nature—rather than deducing facts from broad truths, mathematicians much search inductively to understand why a bundle of facts happen to be.

Programmers are similarly creative. If you provide a business problem to be solved to ten programmers (or the same programmer on different days), you will get ten different solutions. The more challenging a problem, the more divergent solutions appear. Of those ten solutions, some might be optimal, and of those optimal ones they might still appear very different.

Planning the activity of programming means taking into consideration the variances that arise from the creative aspect of this work. Programmers are not logical assembly line workers; they are poets and painters who need context and space to produce. They need a posteriori knowledge to produce.

Mistake #2: Requirements specification is where we start

Extensive business requirements documents are useful for gaining context. User stories are included here. But these do no capture the world and the actual value the business solution is seeking to provide.

By starting with a single source of truth in the form of written requirements, one will start building an artificial mental environment about what these requirements intend to do.

A business solves problems. A solution is a thing of value. Creating value is the business goal of software engineering. In order for all stakeholders, including engineers, to create value that is on target requires involvement in both defining the problem and designing its solution.

If you have worked with a contractor on a smaller project, for example, you’ve likely been questioned about your goals and objectives more than you expect. Engineers are aware of their role in value creation and will often have product-altering ideas that better suit the problem being solved by the business.

By boxing value creation within the confines of a written requirements document, and not involving engineers in the bigger philosophy behind the business and project, one creates an a priori environment. This environment is the opposite of that which was discussed in Mistake #1.

Requirements documentation is necessary. But it isn’t where the project starts. The project starts in discussing, with an open mind and willingness to introduce alterations by management, the fundamental problems the business is solving.

This position of keeping engineering eyes on the value it is creating in the world, rather than ticking boxes off a requirements sheet, will set up the agility you will need when presented with wild unknown unknowns. When hit with a difficulty, engineers will be able to formulate multiple options to proceed, rather than pass the difficulty to management where an opportunity can be lost in the absence of engineering strength.

Mistake #3: Take engineering estimates unfiltered

The unknown unknowns will always appear. Engineers will take this into consideration when providing estimates. There are even more unkonwn unknowns. An estimate of time or cost will always be a range; and usually on the high end or even higher end of the range.

When planning a project in terms of cost and time, it helps to add a coefficient for engineering estimates. As the project complexity or size increases, so must that coefficient. The engineering team will decide a project will take 1 month, but adds an extra week to be safe. The project manager may offer to add an extra week on top for more room. You can expect even then that engineering will need a full 2 months to complete the project.

Sometimes estimates are right on target. But the farther one gets from the engineering team, the greater that coefficient must be.

Mistake #4: Exclude end users from planning

A few years back we were part of a start up. This start up with more complex than most, but after finishing many of the minimal parts of the product and ensuring all the plumbing was in order, we were ready for the next part. The next part, however, was more building and more tweaking of features. This continued for several more months until a large product was complete and ready for end users.

We asked when the end users will be introduced to the product to make sure it aligns with what they want to see. This was delayed and delayed. Unfortunately by the time end users were introduced to the product, some big parts landed flat.

Without all team members, management, and end users working together, we found up with an “eh” reaction to the product instead of the “wow!” we wanted to see. For too long we worked disconnected from end users.

Similar to keeping engineers connected to the real world value they are creating, keeping engineers and end users in close contact is crucial. Often there is a tendency to silo product management to the end users, and silo the engineers to their project managers and supervisors. Product managers draw up what they’ve gathered, and pass this to the engineers. This is the same process as building extensive requirements documentation and boxing the entire creative part of the business into those requirements.

Engineers should have access to raw end user feedback, if not access to end users directly, to better understand the work to be done.

During planning, having an end user available or a some feedback from an end user available, all team members from multiple departments can make adjustments to plans that will align closer to the actual value the business will produce.

Start Planning

Planning starts by understanding the world and the place your business will have in it. The problems it will solve, the way it will solve them, and they way users will feel about them. All team members must have the same a posteriori grasp of the solution you are planning.

For some, it is tempting to be withholding of information about the big picture. The result is always an artificial attempt that will stray from the real world intention they have. Such attempts are expensive, take longer, and annoy the creative people needed to make it a reality.

Always start planning by having an open discussion of your objectives and goals and what you wish to see manifest itself. Be ready to take feedback on those objectives from colleagues and subordinates. This will lay a foundation for giving individuals on the team a sense of ownership and responsibility to make the big idea happen.

On reflection of the security firm case, one can see how a focus on the reason for the scanner should have been the starting point. An experiment working with software dependencies should have been next. The top-down approach of class diagrams, planning modules months in advance, and imposing a database schema before getting our hands dirty with the underlying data relations was destined to be disappointing within days of starting.

Do not consider software project planning to be a top-down affair of meeting a priori requirements. Software projects do no exist in a vacuum. Start by looking at the world.