If you haven’t read Part 1, you may want to start there.
Now that everyone’s clear what exactly needs to be done, it’s time to start on more thorough design and planning.
I’ve split this post into two parts:
Technical Design: addresses what the change should look like, and
Planning: preparing to deliver the changes smoothly.
But first…
Why should you do technical design and planning?
Again, some of you may be tempted to jump straight into building and figure things out as you go. That will work for a small deliverable that you’re doing by yourself. However, if you skip the design and planning for larger projects, you risk running into problems like this:
Halfway through, it turns out you need another team to do something, and their backlog is full until the end of the quarter.
Mid-sprint you discover that one of the developers disagrees with you on the design. Now both of you need to stop your work and debate it. The rest of the team has to stop too because their work depends on the outcome of the debate.
The code that worked for the first two parts of the project doesn’t work for the third, and now you need to rewrite most of it.
The job of technical design and planning is to discover issues like these early when they’re cheapest to address. It won’t prevent all surprises; you could still get last-minute feedback that the users are confused by the interface and end up having to rewrite it. But at least you can minimize the delays and rework from the foreseeable problems.
Technical design
With that in mind, when I go into technical design, my goals are:
Uncover as many technical surprises and as much unexpected work as possible—plus any missed requirements.
Uncover any hidden dependencies, especially dependencies on other teams or outside vendors.
Make sure that the team is clear on what we’re doing and agrees with the approach.
Get a better sense for the size of the work, how many people are needed, and how it can be broken up and parallelized.
We’re going to do this by diving deep into the design and, again, having some meetings.
Discovering surprises and dependencies
From everything I’ve tried so far, there isn’t really an easy way to do this except to go and design the system in detail.
To keep track of that, I usually start a technical design document. If your organization already has a technical design template, try to follow it; if not, feel free to start another document similar to the requirements one. I find the exact format doesn’t really matter, as long as the document is reasonably readable and follows basic principles of technical writing like putting the summary and most important information first, then gradually expanding into details.
In this document, start sketching out the system design in a bit more excruciating detail than you’d first think is necessary. Like, not the individual functions and classes, but maybe a level above that.
For example, in my recent project that delivered a new user-facing feature across several front-ends and back-end services, the technical design involved writing out:
The full list of the front-ends and the full list of the back-end services involved, how each one would change, which new REST API endpoints would need to be created, request/response sketches, and what authentication would be used in each case.
A detailed sketch of how each of a couple dozen or so user flows would be handled by the system, including how it would move through the front-ends, back-ends, DBs, auth used, vendor interactions, and so on. Since the system involved moving around and submitting complex data structures, the design also included listing where each data field would come from and how we made sure that it was correct.
Any data model changes and data migrations, the exact list of the background jobs that would need to be present, and any notable infrastructure changes beyond the standard setup.
Plus a few other details.
Other things worth including, depending on your situation:
Expected scale, latency, and throughput.
Security, privacy, authentication, and permissions.
Any dependencies on other teams or vendors/partners. You’ll want to start conversations with them early.
Whether any organizational reviews need to be passed, such as security, legal, compliance, or architecture reviews.
What’s the right level of detail to go into when writing everything out? Well, consider the earlier goals: we’re trying to uncover major surprises like unexpected work, dependencies, or difficult technical questions. So focus on the details where getting things wrong would cause substantial rework, delay, or a dependency surprise. Don’t overdesign details that are cheap to change later.
Tracking open questions and risks
Just as with the requirements, when writing a technical design, it’s good to keep an eye on spots where you’re not totally clear on the details. This could be either questions about requirements, or questions about how to implement something, or maybe questions you need to ask other teams, or architects, or vendors.
In these cases, write these down so you don’t forget them. And make sure to follow up on them.
Also, remember the Risks section from Part 1? Now is a good time to sit down and ask yourself: what are some of the ways the project can go wrong? These could result in more open questions or follow-up meetings.
Getting the team on board
As mentioned before, one of the goals of the design process is to make sure that all team members who’d be working on the project understand how it would be implemented and agree with the approach.
The best way to do that, I find, is to have a meeting with all project participants and walk them through what you have in mind so far. Then ask for questions or opinions and be genuinely open to feedback and criticism. Take those away, think them through, and update the document as needed.
In cases of major disagreement or concerns, I found it useful to have follow-up discussions with the person who brought them up, maybe even working with them over a few days or weeks to resolve the issues or update the designs.
Simplifying the design
As you’re working through the design, it’s worth periodically asking questions like:
Is there a simpler way to do this? Or, do I even need to do this?
Now that I know the cost of this requirement, should we still do it?
The latter question has often led to conversations with product managers or other decision makers who, after realizing the cost, ultimately said “you know what, maybe we don’t need this.”
Planning
I usually start transitioning to planning once the design starts stabilizing, though depending on the time constraints you may have to start planning out some pieces sooner.
Each organization has its own way to track large features/epics/projects together with their subtasks, and its own processes for refinement or planning. You’re more familiar with those than I am, so I won’t give you advice on how to use them. Instead, I’ll just give some general pointers for planning larger projects.
Unlike design, I found that it’s OK not to go too deep into planning upfront and instead take it sprint-by-sprint. That said, there are still a few questions that you’d need to answer before planning the first batch of work:
What is the dependency between the different deliverables? Which ones are blocking—i.e. must be delivered first, before any work that depends on them?
Now that you know the design, what’s the total effort? Do you think you’d meet your deadlines? Should you go to your manager or other decision makers and renegotiate the scope, the timeline, or the resources (i.e. the number of people working on the project)?
If you have more than one person working on this project, can you parallelize things in a way that keeps everyone busy without any gaps or downtime?
There may still be surprises lurking; what’s the best way to uncover them as early as possible?
For this purpose, I find it’s helpful to at least do a rough sizing of the various components of the project with the team, add some padding for the inevitable problems and unknowns, and sketch out what pieces should be delivered in the first few sprints.
Integrating and deploying the work
One of the questions that also needs to be addressed upfront is what your overall approach to integration and testing will be. By that I don’t just mean CI and automated tests (I assume that’s a given these days), but for the entire project as a whole.
I’ve seen projects delivered two ways:
Approach 1: Everyone writes various pieces separately, possibly coding/testing against pre-agreed interfaces. Then the entire system is put together, tested in the test environment, and then deployed to production.
Approach 2: First, the thinnest possible sliver of end-to-end functionality is deployed to the test and production environments. Then everyone builds out the various pieces around it, doing integration testing in the full system as they go.
From personal experience, Approach 2 is far superior. This is for a couple of reasons:
Deploying to prod as soon as you can allows you to do end-to-end testing earlier, show the system to the prospective users earlier, and adjust course sooner. Also, some bugs show up only in the end-to-end tests; would you rather find them early in the project, or a week before launch?
Production deployment is often more complex than you’d first expect, especially in larger organizations and for large projects. You may have to wait for approvals, satisfy unforeseen requirements, or (in at least one case I’ve experienced) switch to an entirely new deployment method due to a recent directive by a CTO that you weren’t aware of until you tried to deploy. Ideally, you’d deal with these surprises early.
For this reason, in the first couple of sprints on large projects, I prefer to focus not just on building out the initial critical pieces of code, but also on deploying shells of all components to production and making them demonstrate end-to-end connectivity.
Pieces that often get missed
When planning the work, even senior engineers sometimes forget that “Done” is not just when the code is written or the PR is merged, but when:
Someone confirms in the test environment that the change does what it’s supposed to do,
If it doesn’t, any follow-up adjustments and fixes are made, and
The change is deployed to production, where it’s also proven to work as expected.
Often these steps take nearly as much time and effort as the original code, so make sure to include them when planning both the individual tasks and larger components.
How long should design and planning take?
On shorter, simpler projects I’ve had design and planning take a day or two, plus one or two meetings with the team. For larger multi-quarter projects, I’ve had it take up to a couple of months with multiple follow-up meetings with various individuals.
Moving on
With the tech designs firmed up and the initial work planned out, now it’s time for the fun part: building.
Next up, coming soon: Intro to project management for developers: Part 3 - Staying on top of delivery