Picking a Tech Stack

Understanding our product’s domain is the first thing in our engineering checklist because it will help us identify the specific technical problems we have to solve. Scale, availability, features - they all stem from business problems.

Now it’s our job to pick the tools that will help us solve these problems.

We’ve all heard the saying that we should pick the right tool for each job but how do we know which one it is? Most advice ends here, expecting you to have some druid-like intuition that will tingle when you open the docs of the correct framework.

Unfortunately, unless you’ve spent a decade setting up projects, that’s not the case. And you will need a system to help you narrow down the numerous options you have. A system that will turn you into an independent thinker.

So we won’t be looking at a predefined tech stack. I won’t advocate for one technology over another.

Because picking a stack is an exercise in identifying the correct level of engineering complexity you need. And the technologies that will allow us to build at that level.

Let’s see what this means.

A Quick Aside

This is not just an article full of code examples, it’s a chapter of my upcoming book “The Full-Stack Tao” which I’m writing in public.

Here are all published chapters so far:

Over and Under Engineering

Even if you haven’t done it yourself, you’ve heard other engineers complaining about over-engineered codebases. Some products are built in a way more complex way than is needed.

And even if their creators had good intentions, they’ve made things worse rather than better.

Complexity makes things harder to understand. They force us to keep more context in our heads and achieve less with more code. We have to deal with confusing development processes where working with multiple layers of abstractions, types, and boilerplate code grinds development down to a halt.

All this, in turn, leads to a higher chance of errors.

Imagine having a simple presentational website that has a couple of forms, images, and maybe a simple booking system. Now imagine it built using a micro-frontend architecture where every page is a separate application, and there are a couple of microservices handling the form submissions.

Technically, that sounds amazing. But it’s entirely unneeded and all the additional complexity makes development harder, deployments more confusing, and opens the door to a whole lot of network errors that we wouldn’t have to deal with.

And while we’re used to hearing people complain about over-engineered products, under-engineered ones are just as problematic. In contrast, they are far too simple for the problems we’re trying to solve.

We may fail to foresee basic challenges that the technologies we’ve picked can’t answer. A simple example is deciding to build our UI with vanilla JavaScript because we haven’t thought of all the complex components we’d have to build.

They can have scalability or performance problems. An under-engineered codebase can become too messy as engineers have shied away from creating abstractions. In our attempts to keep things simple, we can end up with the same result - slower and harder development.

Imagine that instead of writing the simple booking system we mentioned above using an SQL database, we decided to write the data directly in a file or just embed a spreadsheet that clients can fill their data into.

The time we will have to spend on fixing and managing faulty data will take us a lot more than the time we saved with this implementation.

While over-engineering is a result of overthinking and planning too far into the future, under-engineering is a sign of rushed development, limited budgets, or a lack of understanding of the domain.

But both extremes are bad because we’re failing to adequately address the requirements of our product.

Choose the Lesser Evil

Before we move on, I need to point out that I’ve found under-engineering to be the lesser of the two evils. We can always make a system more complex. The opposite is not true.

I’ve seen more projects struggle because of over-engineering rather than the opposite. Our focus on the future, fighting potential problems that may never occur is incredibly successful at making codebases too complex to work on.

Over-engineered codebases are a lot worse in terms of development speed. And slower development doesn’t make a software better.

In fact, we should be aiming to create an environment in which we can make such good sense of everything that we can move fast. We work in iterations, drafting and editing our code and architectures over and over again.

The slower that cycle is, the less frequently we’d be able to iterate.

So when in doubt, always lean towards under-engineering rather than over-engineering.

Engineered Just Enough

We don’t want to be in any of these two extremes. They’re both detrimental. We want to be in this middle space that’s “engineered just enough” and allows us to both work productively and use powerful enough tools and abstractions.

Virtue is always in the balance.

We don’t get the drilling machine when a simple hammer would be enough to put a bolt in the wall. But programming can be a lot more complex than that, and picking the “best tool” involves more variables.

Because of that, I’ve prepared a few questions that help us narrow down our technical options.

How Long Will This Live in Prod?

The most important question you need to ask yourself is how long will this product be living in production. This may sound like a weird way to start thinking about it, so I will ask you to trust me and read on.

An application that lives for years will be subject to constant change. It will be refactored, and extended, its libraries will be updated, and even its environment may change. The business will make pivots and changes in strategy, and they will then be reflected in the codebase one way or another.

You need a technology that gives you good structure and readability. One that is has a broad community because many engineers will be passing this project down and they need to make sense of it.

In other words, you need to be more conservative.

On the other hand, you can be a lot more careless with a prototype that may go in the bin in a month or even a year. There’s little point in obsessing over the right application structure when you’re working for a startup struggling to find product-market fit.

If what you’re building has a shorter lifespan, you don’t need to think long-term. You can write code that doesn’t scale, try out a new technology, and experiment. Take on tech debt more aggressively if you won’t have to pay it off.

There’s a reason why certain technologies are far more prevalent in corporations than startups (and vice-versa). Their needs are different because the applications’ lifespans are different.

Failing to recognize this leads to over-engineered startup applications, and messy projects getting handed over from team to team in a corp.

Now I’m sure that as you read this, there are already technologies popping up in your head that you’d use in one case and not the other.

Sometimes they may overlap. For example, at the time of this writing, I’d use React both in a corporate setting and a startup environment. But only because it can give me both a stable community, and the necessary flexibility I’d need in a fast-paced environment.

It all comes down to how long you expect this software to live.

Can You Use Tools You Know?

At this point, I’m convinced that FOMO is one of the biggest factors behind the rapid pace of the industry.

You think you can’t possibly build an application unless you use the latest version of a front-end framework, a low-level language, and a scalable database.

We keep churning technologies to no objective benefit out of fear that we won’t have access to an important feature. We don’t stop to think that very complex software has already been written using tools far less powerful than the ones we have available.

Keep in mind that your experience with a technology will beat the potential technical benefits of another. Knowing what pitfalls to avoid and what patterns to use will help you build a good structure and be productive at the same time.

An engineer will output a far better product using a familiar language, than a new one. They already know the ecosystem and the supporting tooling. They can communicate with community members and know the common anti-patterns.

It takes time to get to know your tooling, no matter how experienced you are.

A different read on this same question would be - what tools is your team experienced with? In a team with 3 knowledgeable Angular devs, there’d be little point in picking React. Anything you can build with one front-end framework, you can build with another. In the same way you can write a good enough REST API in most languages.

So unless the domain has a problem solvable only by specific technologies, we should value previous experience highly in picking our stack.

How is Your Team Organized?

Conway’s law states that every organization will inevitably create a system that mirrors the way its teams are organized. We underestimate the influence of external factors over our technical decisions, but this is a great example.

The easier we communicate, the fewer problems we will have working on the same part of a product. The harder it is to talk and make decisions, the more we will need autonomy so we can work productively.

So we split products into smaller parts managed by specific teams.

In a small team, a person will have ownership over a module in a monolithic application. In a bigger team, this may be an entire service instead. In a large company, a team may own a few services in their entire system.

This happens naturally as a team evolves and we start drawing seams between our work. But to make your architecture better, you can short-circuit that process and design your product around the way your team is structured.

This is also known as the Inverse Conway Monoeuvre and it’s a relatively popular pattern.

In practice, this would mean that if you have three full-stack engineers working from the same office on the same product, you will do just fine with a modular monolith. If you have separate back-end and front-end teams, having two separate repos and communicating via REST API would be a lot easier.

The easier the communication, the more productive the development process.

Balance is a Moving Target

This is not a static questionnaire that you fill in once and use as a north star. The right level of engineering complexity is a moving target and it will change together with the company.

Suddenly, your prototype was a success and now the business wants you to continue building on top of it.

But this changes the fundamental idea of how long the codebase will live in production. You’ve cut some corners here and there knowing that your work will be rewritten. Now, when the product is validated it’s living longer than expected.

Your initial decisions are not good enough to support the product and you need to refactor.

Maybe the company hired a team in a different timezone and now pull requests drag on for days. An engineer from one team broke the build before leaving for the day and now the other team has to figure out what went wrong. Scheduling meetings becomes harder yet more necessary.

And slowly a good initial design can become obsolete at a moment’s notice.

An organizational problem can be solved with the technical solution of modularizing the project or even splitting it into separate ones.

This mustn’t be taken as a criticism of the above events. A startup looking for a quick way to go to market, and a company expanding to new locations is perfectly normal. It’s important how we as engineers reflect these changes in the code.

The Stack Doesn’t Matter

The exact technologies you pick don’t matter… as long as they fit the answers to the questions above. If the technologies match your needs of speed, stability, community, and prior experience, then it’s a matter of personal taste what you pick.

A library’s API won’t make or break your product so don’t obsess over them.

But after all this, I need to at least offer some technical guidance in terms of technologies. At the time of this writing, you need a good reason not to pick React for your front-end, and Postgres for your database.

They seem to check all the boxes and are used both in small companies and large-scale enterprises. If everything else fails, start from there.