When I started my career, REST was at the peak of its adoption. It had become the de facto standard for designing web APIs with vast amounts of tutorials, resources, and tools to support engineers. There was no question what paradigm you’d follow when building a service - it was REST.
I’ve been asked to explain its principles too many times to count during interviews, and I’ve asked it myself twice as much.
I knew the theory so well that I could easily recite it if you woke me up at 3 AM. And my fingers had developed muscle memory of their own from typing GET, POST, and the rest of the methods so many times.
But I was already in a senior position when I learned I wasn’t building REST APIs properly
We’ll follow the same approach we did in my previous write-up. We’ll write a first draft of our implementation, and make sure it works. Then we’ll edit and flesh it out, improving its design, and making sure we end up with a clean implementation
For this application, we will use Express, a minimalistic HTTP router. It doesn’t interfere with our application’s design, allowing us to structure and build it in a way that supports our domain the best. And, as a side benefit, I will be able to show you as many techniques as possible when we have only the bare necessities of routing solved.
A Quick Aside
This is not just an article full of code examples, it’s a chapter of my upcoming book “The Full-Stack Tao” which I’m writing in public.
Here are all published chapters so far:
- 1. Start with the Domain
- 2. Picking a Tech Stack
- 3. Setting Up the Project
- 4. Clean Architecture in React
- 5. Building a Proper REST API
- 6. How to Style a React Application
The Business Requirements
Let’s recap our requirements:
“We need to build a page that allows users to read the currently active prompt in a writing app, and then answer it. They should be able to see other users’ answers, but only after they submit their own. Before they do, they should only see the prompt and the input field to submit their response.”
While the front-end is making a simple request to retrieve the prompt without any parameters, we need to find a way to identify which one is currently active. This is a decision primarily driven by the business requirements, and here understanding our domain well will play a huge part.
How do we define the active prompt?
We have two approaches:
- Mark it manually - we can add a column in the database called
activeand write a job that updates it at a given schedule - Derive it based on other data - we can use a field that marks when that prompt is going to be available and query the store based on it every time.
The former makes fetching easier, but it requires additional complexity to set it up. The latter is a lot simpler since it won’t require a continuously running job, but we need to make sure we can derive it consistently.
To make a more informed technical decision, we need to consult with the business.
The requirements we have tell us that there will be only one active prompt each week. However, we know that the company is still validating this new product, and the details around the active prompt might change.
Communicate to Make Engineering Easier
So we go back and ask about future plans, and what the product team is cooking up.
To our surprise, the business tells us that they were considering changing it to every 3 days rather than every week, and perhaps experimenting with varying prompt lengths - one for the week, another for the weekend. There’s also the idea of having small daily prompts and a large weekly prompt active at the same time.
This new information immediately throws the manual option out of the window.
That’s a lot of context we were missing and without asking, the business doesn’t know what details are relevant to us. If we hadn’t drilled down and asked questions, we could’ve made an under-engineered implementation that wouldn’t have supported us in the future. We would’ve piled tech debt from the start.
You could say that we should make all our implementations a bit more complex, so they can support the business better. But over-engineering everything is not a solution, since the complexity and reduced speed of development will make our work a lot harder.
Communication gives us better insight into when we want to over-engineer or under-engineer something. Here we definitely need to keep our options open.
The Decision
With that information, we know that we can’t use a column to identify the week the prompt should be active because that will most likely change.
Instead, we could use a combination of two columns - start_date and type. The date will tell us when the prompt will become active, and it’s not tied to a particular schedule. We can change the prompts with any frequency.
We won’t use the second column yet, but knowing that the business would like to try out different kinds of prompts in the future, we will need this information. It would be better if we added it now and avoid running an extra database migration in the near future.
The Handler
Since we figured out what endpoint we would need when we were working on the UI, we can get straight to it. I need to note that there are more moving parts in the REST API than in the UI, so if we gloss over some details, be sure that we will come back to them later. We’ll do that so we can focus on design immediately.
Right now we need a single endpoint to retrieve the currently active prompt.
The two fundamental REST API principles are using resource-based URLs, and implementing HTTP endpoints to provide context for operations. Following that, we will create a GET handler for the currently active prompt and inline the implementation.
app.get('/prompts', (req, res) => {
const user = { id: 1 }
const currentDate = new Date()
// ...
})Since we’ll be working on authentication in a separate chapter, we will use a mocked user object for now. For the current date, we will create a Date instance which will default to the current date and time.
Next up, we have the data access.
Retrieving the prompt
We’ll be using a light ORM called Drizzle for the examples below. I’ve chosen it because it doesn’t affect our application’s design but still gives us a powerful API to interact with the database. The next article will be focused entirely on data access, ORMs, and query builders so we will go into more detail there.
Following the business logic, we can fetch the current prompt, then check if the user has answered it, and retrieve the answers if they have.
app.get('/prompts', (req, res) => {
// ...
const prompt: Prompt = await db
.select()
.from(prompts)
.where(lt(prompts.start_date, currentDate))
.orderBy(desc(prompts.start_date))
.limit(1)[0]
const hasAnswered = await db
.select()
.from(answers)
.where(
eq(answers.user_id, user.id),
eq(answers.prompt_id, prompt.id)
)
if (!hasAnswered) {
return res.json({ ...prompt, answers: [] })
}
const answers: Array<Answer> = await db
.select()
.from(answers)
.where(eq(answers.prompt_id, prompt.id))
return res.json({ ...prompt, answers })
})And that’s pretty much the whole endpoint.
Improving the design
We’ve written the first draft for our new endpoint, and we’ve made sure it returns the correct data. As I mentioned in the previous chapter, this is a good place to stop if you’re prototyping, you’re pressed for time, or you expect this functionality to be deleted or changed immediately.
However, if this is an application that’s meant to be maintained, we need to think about its structure and edit it a little bit.
This is how our handler looks now.
app.get('/prompts', (req, res) => {
const user = { id: 1 }
const currentDate = new Date()
const prompt: Prompt = await db
.select()
.from(prompts)
.where(lt(prompts.start_date, currentDate))
.orderBy(desc(prompts.start_date))
.limit(1)[0]
const hasAnswered = await db
.select()
.from(answers)
.where(
eq(answers.user_id, user.id),
eq(answers.prompt_id, prompt.id)
)
if (!hasAnswered) {
return res.json({ ...prompt, answers: [] })
}
const answers: Array<Answer> = await db
.select()
.from(answers)
.where(eq(answers.prompt_id, prompt.id))
return res.status(200).json({ ...prompt, answers })
})Our first handler doesn’t need to read anything from the request body or the parameters, but it does need a couple of external pieces of data in order to work correctly - the current user and the current date.
Following what we said about mocking in the previous chapter we don’t want to mock a user object here. We want to do it a level above where the hardcoded data is used so our work’s design is not affected by the mock.
We will need to create a function that will validate and retrieve a user by the contents of the request object, and do the mocking there.
We pass the entire request object to the function because we don’t want to couple the handler to the specific way the user is passed to the API. Authentication can be done via a token passed as a header or a cookie - it’s not the handler’s responsibility. We want to keep it as generic as possible, and we will let the function decide how to retrieve the user.
app.get('/prompts', (req, res) => {
const user = getCurrentUser(req)
// ...
})But we immediatelly notice a big potential for duplication because the vast majority of our routes will need access to the current user. Even though we’ve extracted the logic to a function, all these handlers will still have to start with a call to it.
Because of that, we will create a middleware - a function that will execute before our route, and we will use it to retrieve the user and pass it to the handlers
const authenticate = (req, res, next) => {
req.user = { id: 1 }
next()
}
app.get('/prompts', authenticate, (req, res) => {
// We can now access the user object from the request object - req.user
// ...
})But this is still a function call, what do we gain by doing it like this instead of inside the handler? Normally, I favor explicit before implicit code, but this is one of the exceptions to my rule.
Middleware is a common pattern used to split some of the HTTP-related logic into multiple functions. It helps us hide details from the handlers, letting them focus on the transport-specifics of their route. Otherwise, every handler function we write would have to become aware of how the user identifier is passed to the API, and if we, potentially, changed that, we’d have to update all of our handlers as well.
By using a middleware, we rely on convention - every route with this middleware will have access to the user object. And on top of that, we don’t need to worry about error handling and unauthenticated users in our handlers. They can focus on the happy path.
We need to remember that the middleware is still only a part of the HTTP layer, though. We’re only delegating the responsibility to extract the correct value from the request object, and then either enrich it with the user or return a bad response. We don’t want the middleware itself to execute business logic - and retrieving and authenticating users is a part of that logic.
Because of that, we will still use the nested function we made earlier, but call it from the middleware and react to the value it returns.
const getCurrentUser = (req) => {
// Get the token from the request object and validate it
}
const authenticate = (req, res, next) => {
const user = getCurrentUser(req)
if (!user) {
return res
.status(http.Unauthorized)
.json({ message: 'Unauthorized' })
}
req.user = user
next()
}This is true about any middleware, not only the one dealing with authentication.
We’re still working with mocked data here, but we can see that we’re already starting to structure our application in a way that will support our domain logic and reduce the amount of boilerplate we have to write.
Dates
Extracting the date retrieval functionality comes down to the philosophical question of whether you see it as a part of the domain logic or not.
I like to treat dates as an external piece of data because we once again need to communicate with something that is outside of the confines of our own application - the operating system.
We fetch the user from a database, and in the same way, we fetch the date from the OS. Right now we only create a new date object via new Date(), but we might also want to retrieve the date in a specific format, based on the format our database needs. This would be the equivalent of mapping the data coming from the database.
Since working with dates would most likely require a third-party library, and some configuration, the simplest way to abstract it would be to wrap everything in a function and keep it in a separate file.
// dates.ts
import dayjs from 'dayjs'
import 'dayjs/locale/en-gb'
dayjs.locale('en-gb')
export function getCurrentDate() {
return dayjs(new Date()).format('YYYY-MM-DD')
}
// handler.ts
app.get('/prompts, authenticate, (req, res) => {
const currentDate = getCurrentDate()
// ...
})Many developers suffer from analysis paralysis, figuring out what design pattern to use when they’re extracting such functionality, how to name the file, and how to structure it. In most cases, they end up with a dates-manager or a dates-service or put it in the utils bin together with everything else.
But not everything has to be something.
It’s just a module (file) with functionality related to dates, called dates.ts and it exports a function. We might add more there in the future, who knows? But right now that’s all it has to be. Don’t overthink it, and don’t try to force patterns. Learn to recognize them when they present themselves.
Improving Data Access
Something to keep in mind when you’re working on your database queries is that business operations don’t always map well to data access logic. If we implement them based on how we word our requirements, most likely we won’t take advantage of the database’s full capabilities.
Right now we fetch the prompt, check if it was answered, and then either retrieve the complete list of answers or we return the prompt. But we could take a different angle and do the check first. Then we will have the information we need to either retrieve the prompt alone or together with the articles.
That’s more efficient.
app.get('/prompts', (req, res) => {
// ...
const hasAnswered = await db
.select()
.from(answers)
.where(
eq(answers.user_id, user.id),
eq(answers.prompt_id, prompt.id)
)
if (!hasAnswered) {
const prompt: Prompt = await db
.select()
.from(prompts)
.where(lt(prompts.start_date, currentDate))
.orderBy(desc(prompts.start_date))
.limit(1)[0]
return res.json({ ...prompt, answers: [] })
}
const promptAndAnswers = await db
.select()
.from(prompts)
.leftJoin(answers, eq(answers.prompt_id, prompts.id))
.where(lt(prompts.start_date, currentDate))
.orderBy(desc(prompts.start_date))
// ...
})And while this reduces the maximum number of queries to two, we can go even further.
We should make our database do as much work as possible without running actual business logic in it. Why make two queries for something we could do with one? If we can have the store check for an answer’s existence as a part of the retrieval, we’d spare ourselves the latency cost of making two requests.
app.get('/prompts', (req, res) => {
// ...
const currentPrompt = db
.select()
.from(prompts)
.where(lt(prompts.start_date, currentDate))
.orderBy(asc(prompts.start_date))
.limit(1)
.as('cp')
const result = await db
.select()
.from(currentPrompt)
.leftJoin(answers, eq(currentPrompt.id, answers.promptId))
})Databases can do a lot of work for us that we’d otherwise have to implement in the application layer. In this case, we’re not only making a performance improvement by removing an unnecessary query, but we’re also solving an issue with potential data inconsistency. We have no guarantee that the data in the database won’t change between the check and the follow-up query, and that could make us return a response that doesn’t reflect the state of the store.
I want to stress again that there’s a fine line between making the store do more work and implementing actual business logic inside of it. I avoid database triggers and stored procedures because they hinder our ability to maintain and test the product. More on that in follow-up chapters.
The repository
We extracted our authentication logic, we saw how important it is to move out logic related to time and dates as well, and we even improved the effectiveness of our database operations. But we still have our data access logic sitting in the middle of our handler.
We have to move it out too in order to establish clear layers between our domain logic and the rest of our application’s responsibilities.
Business logic needs to be database agnostic. Our goal when we’ve refactored this would be for it to be as close as possible to the business language we use.
When you talk with the product people, they don’t say “Make a check for an answer, and join the prompt with the answers if it exists”. They say “Retrieve the current prompt and the answers if the user has contributed to it”.
That’s the language that we want to keep. So, following the advice of clean/hexagonal/layered architecture, we will extract our database logic away from the service, and give it domain-focused methods whose names reflect what we’re trying to achieve.
// repository.ts
export function getPromptByDate(date) {
const currentPrompt = db
.select()
.from(prompts)
.where(lt(prompts.start_date, currentDate))
.orderBy(asc(prompts.start_date))
.limit(1)
.as('cp')
const result = await db
.select()
.from(currentPrompt)
.leftJoin(answers, eq(currentPrompt.id, answers.promptId))
return result
}
// handler.ts
app.get('/prompts', (req, res) => {
const currentDate = getCurrentDate()
const prompt = getPromptByDate(currentDate)
return res.json(prompt)
})This file we just created is what you’d call a repository.
It’s nothing more than an abstraction over our database logic. It could be a module with exported functions, like the one we just created, or an object.
// repository.ts
export default {
getPromptByDate(date) {
// ...
},
}
// handler.ts
import promptRepository from './repository'
app.get('/prompts', (req, res) => {
const currentDate = getCurrentDate()
const prompt = promptRepository.getPromptByDate(currentDate)
return res.json(prompt)
})It can be a class.
class PromptRepository {
getPromptByDate(date) {
// ...
}
}Or whatever structure we have available in the language we’re using.
The purpose of repositories
Repositories aim is to decouple our domain logic from ORMs, query builders, table structures, and other database specifics. The exact implementation varies between languages, frameworks, and paradigms, and since we’re aiming to have a light architecture that doesn’t introduce unnecessary boilerplate - a simple function is more than enough.
Put simply, we want to wrap our database operations in functions to make them easier to read.
The word repository may sound too daunting for something as simple as that, but we need a common vocabulary to communicate our intentions with different engineers, and this is the term that the industry has settled down on.
A mistake some teams make is rushing to extract operations in a repository before they’ve decided on an exact implementation. Had we created the repository immediately as a part of our “first draft”, we would’ve had something like this.
// repository.ts
export function hasUserAnsweredPrompt() {}
export function getPromptByDateWithAnswers() {}
export function getPromptByDate() {}But when we explored the idea deeper, we ended up with a single method and a much simpler API than the above.
// repository
export function getPromptByDate() {}This once again highlights the importance of seeing software development as an iterative effort. We won’t come up with the best possible design on the first try. It’s important to get something working, and then improve it, think about it, then improve it again.
A good test for whether we’ve implemented the repository pattern successfully is our ability to swap the database. In practice, situations when we’d really need to change the store are extremely rare, and our abstractions will be the least of our worries then.
But as long as the repository function returns a response with the proper format, we won’t even notice the difference in our domain logic if we change the ORM or the store underneath.
Data formats
Right now, because of the way we join our data, our repository layer returns an array of objects where each of which has the following structure:
[
{
prompt: {
id: 2,
title: 'What is the meaning of life?',
start_date: '2023-04-13',
},
answers: {
id: 1,
text: '42',
},
},
{
...
}
]This is normal since we’re using a query builder and not an ORM that maps the data for us.
But if we go back to the previous chapter, we will see that this is not the format we decided to use in the browser. More experienced front-end engineers will be quick to add a light mapping layer between the HTTP call and the component, structuring the data in the proper format before displaying it.
But while this will solve the immediate problem, it’s only curing a symptom, not addressing the root cause.
It’s never right for one application in a system to address the flaws of another. This puts a maintenance burden on the client because we will have to implement mapping for every future endpoint we need to interact with. But if this service ever gets used by other front-end applications, then they’d have to duplicate the effort.
Not to mention that this couples the front-end to the database structure, and if it ever changes we’d have to do numerous updates across multiple apps. So in the long term, we will end up doing more work if we use the query result directly.
export function getPromptByDate(date) {
const currentPrompt = db
.select()
.from(prompts)
.where(lt(prompts.start_date, currentDate))
.orderBy(asc(prompts.start_date))
.limit(1)
.as('prompt')
const result = await db
.select()
.from(currentPrompt)
.leftJoin(answers, eq(currentPrompt.id, answers.promptId))
return {
prompt: result[0].prompt,
answers: result.map((r) => r.answers),
}
}We will map the data to the format we want our users to consume inside the repository. To make the method a bit simpler, we could also extract the actual mapping to a separate function.
function mapPromptToDomainObject(result) {
return {
prompt: result[0].prompt,
answers: result.map((r) => r.answers),
}
}
export function getPromptByDate(date) {
const currentPrompt = db
.select()
.from(prompts)
.where(lt(prompts.start_date, currentDate))
.orderBy(asc(prompts.start_date))
.limit(1)
.as('prompt')
const result = await db
.select()
.from(currentPrompt)
.leftJoin(answers, eq(currentPrompt.id, answers.promptId))
return mapPromptToDomainObject(result)
}Now we’re done with our repository layer.
Repository method naming
While the repository gives us a good abstraction over our database logic, it’s still important for our function names to explain clearly and honestly what’s happening underneath.
In this case getPromptByDate doesn’t convey information about the check that we’re doing prior to the data fetching. It also doesn’t say that we’re retrieving the answers together with the prompt.
Naming the function something like getPromptAndAnswersIfUserContributed would add that additional context. Describing what we’re getting the prompt by is not necessary since the date parameter in the function signature makes it obvious.
This is a long name for a function but when I’m on the fence about one, I always go for the more descriptive one. If saving characters on a name would lead me to write additional documentation or comments, I’d rather just be verbose.
On the other side, functions with the word ‘and’ in them are generally a code smell because it means they’re doing too many things at once. But in this case, the complexity is handled by the database, not the application layer.
This is why I like to inline my code first.
I never write an abstraction directly because it might influence the final design of the implementation. I want to see the code in its most raw form first, and then I can start to think about how to structure it. We went through three different approaches before ending up with this repository method, and if we had created a set of functions for the first one, we would’ve either had to change it twice or we would’ve become invested in a suboptimal design.
The service
We’ve structured our handler very well. Authentication logic is out of its responsibilities. We’ve created a separate function to retrieve the current date, and we’ve abstracted our database logic into a repository.
But our transport logic is still intertwined with our domain logic, and to fix this we will extract it into a “service”.
// prompt-service.ts
export function getCurrentPrompt(user) {
const currentDate = getCurrentDate()
const prompt = promptRepository.getPromptByDate(currentDate)
return prompt
}
// handler.ts
app.get('/prompts', authenticate, (req, res) => {
const prompt = promptService.getCurrentPrompt(req.user)
return res.status(200).json(prompt)
})Just like the repository, you can use any structure you want for your service. It can be a module, an object, or a class.
class PromptService {
getCurrentPrompt(user) {
const currentDate = getCurrentDate()
const prompt = promptRepository.getPromptByDate(currentDate)
return prompt
}
}I favor simple functions and rely on the file as a module to group them together, because in this usecase we don’t have a problem that OOP’s a good solution for. The important thing is that your service is separated.
The term “service” is used broadly and carries different meanings depending on the context.
In a front-end application, you may see a service object dealing with HTTP requests. In the scope of a system, a service is a whole REST API that communicates with other APIs. But in the scope of a back-end application, the term service is inspired by the concept of a “domain service” in Domain Driven Design and is most often used to describe an object that encapsulates the business logic.
Something else we’d need to extract out of our new service is the error handling. It is no longer responsible for returning an error response and setting status codes. It’s only responsible for returning the data.
So we can move out the try/catch block from the service and into the handler.
app.get('/prompts', authenticate, (req, res) => {
try {
const prompt = promptService.getCurrentPrompt(req.user)
return res.status(200).json(prompt)
} catch (error) {
return res
.status(500)
.json({ message: 'Failed to retrieve the current prompt' })
}
})Not we have the following layers:
- A handler responsible for the HTTP logic.
- A service responsible for the domain logic.
- A repository responsible for the database logic.
Where to map data?
Even though it’s nothing more than an object with a single method, right now the promptService is responsible for our domain logic.
The handler calls a method on it, passing the user object, and the database maps its data to a specific format before returning it. But why are we going through all these hoops in order to keep our business logic simple? Couldn’t we just pass the whole request, and return the raw data, letting the business layer figure it out?
In other words, whose responsibility is it to map data to a domain-specific format?
We essentially have a layered architecture, and there’s a fundamental rule we need to follow - outer layers can know details about inner layers, but not vice versa.
The domain logic is the heart, the core of an application and we “wrap” it with HTTP and database logic so it can communicate with the outside world. Imagine that your business logic must remain oblivious about anything outside of its boundaries.
The only way to achieve this is if the transport layer only passes down the data needed by the core, and the database layer maps its data in the format used by the core.
The end goal is to write our business logic the same way regardless of what other technologies we’re using. So while this article is focused on REST APIs, it’s actually more about teaching you that the environment shouldn’t matter for the way you build your application.
Same core, different shell
We could’ve stopped our refactoring 4K words ago and called it a day. But there are two reasons why we’re going through all of this:
- Make sure that changes in one layer, be it transport, data access, or domain, won’t impact the others.
- Decouple our business logic from technologies and frameworks as much as possible to make it easier to evolve.
Most applications will never have to drastically change their data access or transport layers, but their business logic will be modified daily.
There’s no reason why we should write our domain logic in a different way based on the libraries, the transport, or the database we use.
const server = new grpc.Server()
server.addService(promptsProto.PromptsService.service, {
getCurrentPrompt: (call, callback) => {
const user = getUserFromCall(call)
const prompt = promptService.getCurrentPrompt(user)
// ...
},
})If we had chosen gRPC for our application our domain logic would be implemented in the same way. The only difference would be in the transport layer.
const consumer = new KafkaConsumer({
topic: 'prompts',
onMessage: (message) => {
const user = getUserFromMessage(message)
const prompt = promptService.getCurrentPrompt(user)
// ...
},
})Had we had to implement our service as a Kafka consumer, the domain logic would still remain the same. The changes are contained in the transport layer.
And you can imagine that if we had to change our database, the changes would be contained to the repository.
Don’t get carried away thinking about preparing your application to move to another transport or database, though. Your abstractions will be the least of your worries if this happens.
Making our code easier to migrate is only a side effect of what we’re actually trying to achieve here - decoupling.
HATEOAS
That’s how most REST APIs look and work. We’ve done a pretty good job decoupling our domain logic from everything else and we’ve structured our application well. But our API has not tapped into REST’s most powerful decoupling tool - HATEOAS.
We’ve been focusing on the domain logic so far, but we will shift our focus to the transport layer by examining something called the Richardson Maturity Model.
This is a framework that classifies services according to their adherence to the principles of REST, and it consists of four levels.
- Level 0: a web service that follows no RESTful standards and is essentially executing remote procedure calls.
- Level 1: structures its API around resources - each entity has its own URLs that communicate the operations that can be done with it.
- Level 2: utilizes HTTP methods for the different actions that we can take with a resource. It follows the standard semantics like using GET to retrieve data, POST to create, and so on.
Our service, like most other REST APIs, is a level 2 service according to the Richardson Maturity Model. Having a service at this level of REST adherence is perfectly acceptable and that’s how the majority of APIs are written in our industry.
In fact, I’d dare argue that most applications needn’t go further than that.
- Level 3: serviceß at this level uses hypermedia (Hypertext As The Engine Of Application State aka HATEOAS) to include links related to the current operation to help clients discover other resources dynamically.
So far we’ve been sending a simple response, containing the prompt and answers. But if we want to take full advantage of REST, we should include additional information about the endpoints our service supports.
app.get('/prompts', (req, res) => {
// ...
res.json({
prompt,
links: [
{ rel: 'self', href: '/v1/prompts' },
{ rel: 'answers', href: `/v1/prompts/${prompt.id}/answers` },
],
})
})I’ve added the second answers endpoint purely for illustrative purposes - we don’t have this functionality right now.
If this is your first time encountering HATEOAS, you might be doubtful. We’ve spoken about complexity so much in previous chapters that your spidey sense should be tingling, telling you how this will increase it.
Why do we even need this?
HATEOAS is another decoupling mechanism between the client and the server.
Making clients dynamic
Clients don’t need to be aware of the exact structure and URLs of the endpoints, and they can use them dynamically from the responses. The biggest benefit behind HATEOAS is that our REST API will be able to change its endpoints without alerting the applications consuming it.
You can add any link the clients can use to work with the entity you’ve sent their way. It’s handy when it comes to CRUD operations:
{
"prompt": {
// ...
},
"links": [
// We can pass the CRUD links, so the UI doesn't have to stitch them
{ "rel": "create", "link": "..." },
{ "rel": "update", "link": "..." },
{ "rel": "delete", "link": "..." },
// We can also pass links about related resources
{ "red": "author", "link": "..." },
{ "red": "most-liked", "link": "..." }
]
}For many companies, HATEOAS can be a solution to organizational problems. If a REST API is evolving rapidly and needs to change its endpoints frequently, communicating this with the teams consuming that API would be a bottleneck. Every change will have to be announced, you’d have to maintain both endpoints until teams migrate, and then decommission the old one.
By implementing HATEOAS your architecture becomes more dynamic, allowing your clients to “discover” the structure of your API.
I’ve mostly seen HATEOAS used in 3rd party public APIs that are meant to be widely consumed. The links act as in-code documentation that makes browsing resources a lot easier. Hypermedia is also useful in large complex systems that consist of many RESTful services that communicate with each other.
We’ll talk about microservices later in the book, but they’re a good example of an architecture that would demand dynamic behavior. A system in which services are developed and deployed by independent teams needs to be decoupled.
HATEOAS and complexity
In a previous chapter we discussed the importance of picking the right level of engineering complexity for a product - neither over-engineering nor under-engineering it. Deciding whether you want to use HATEOAS is a decision that should be made carefully.
Using hypermedia comes at the cost of a lot of complexity.
Every endpoint will have to follow the standards, and we’d need additional guardrails to make sure we haven’t broken a URL or the structure of the object. The REST API will be solely responsible for its accessibility now.
It also means that clients (the front-end or other services) now have the implicit requirement to consume hypermedia which adds more complexity on their side. This puts additional stress on code reviews and documentation - you don’t want an engineer to hardcode an endpoint and the server to break it unintentionally with a change.
Architectural trade-offs
When we’re faced with an architectural decision like this, we need to boil it down to what we’re taking and what we’re giving. In this situation, we’re trading off complexity for decoupling. So we need to consider whether decoupling the server from its client(s) is important enough for us to warrant the additional complexity.
For the small application we’re building right now, this is largely unneeded. The time we’ll spend implementing HATEOAS and consuming the links in the front-end will be better spent working on our next feature. This is the same reason why most other teams never reach this state of REST API maturity.
While resource-based endpoints and methods provide much-needed structure to our back-end, hypermedia is situational.
But what about the future? What if we need to decouple in the future?
This is where knowledge about the domain can help us make a decision. Do we realistically expect to find ourselves in a situation where we have a rapidly evolving API? Do we expect the business to need a large system of independent services that need to communicate with one another?
Be painfully honest, talk with business people to learn more about what the business expects in terms of growth, and then make the call.
API Versioning
There comes a moment when you want to make a breaking change to the endpoint. Maybe you’d want to remove a field from the response or add a required parameter in the request. Unfortunately, that will mean that you’ll break the existing clients.
And while HATEOAS can help us if we’re just changing the endpoint, if we’re changing the structure of the response, we don’t want to do that dynamically.
Instead of planning and organizing a convoluted migration in all clients consuming the API, we can create a new version of the API and let them decide when they want to migrate to it.
This is a common practice in REST APIs and it’s usually done by adding a version prefix to the URL of the endpoint. Instead of having the clients call /prompts directly, we can have them call /v1/prompts.
router.get('/v1/prompts', authenticate, (req, res) => {
// ...
})Then if we have to make a breaking change, we can register a new endpoint prefixed with /v2.
Are we over-engineering this? We haven’t shipped our current API and we’re already insuring ourselves against breaking changes. Even though this is the exact thing we want to avoid, the cost of adding this safeguard is so small that it’s better to do so.
The /v1 prefix won’t affect our application’s design and it’s not adding any additional boilerplate or complexity.
However, it’s inconvenient to add the prefix manually to every endpoint. So we will leverage a feature of the Express router to do this for us.
// Create a router to handle the requests
const router = express.Router()
router.get('/prompts', (req, res) => {
// ...
})
// Add the router to the app with a common prefix
app.use('/v1', router)We essentially create another middleware that will handle all the requests that start with /v1 and pass them to the router which has all our handlers defined on it. This ensures we won’t mess up a prefix, and it helps us keep the versioning configuration in one place.
Also, the handler doesn’t need to be aware of what version it’s used in.
Most routers and frameworks will allow you to either add a prefix or group your routes so you can apply the versioning.
If a service fails and no one is around to see it, has it failed?
The philosophical equivalent of this question has been asked for centuries, questioning our knowledge of reality. In the context of our REST API, this question is about our application’s observability and monitoring.
If you don’t know that your API is failing you might continue to work on new features in blissful ignorance while your users are struggling with errors. We don’t want to be the last to know that something’s not working, asking users to reproduce it.
Logs are a window inside our application - they show us what happened, when, and what caused it.
Logging using the built-in console objects won’t be enough for a production application, though. We can use it to output an arbitrarily structured message containing the time, and an error code, but this format will only be useful if we want to manually read through the logs.
It would be better to use a structured logger that provides the logs in a format digestible by machines.
This way we can use tools to search through the logs, aggregate them, and visualize them. Logging libraries give us different log levels, so we can set alarms based on severity. To many people’s surprise, logging libraries can sometimes be more performant than the built-in console object because they may use async logic or buffering.
// Create a logger object
const logger = pino({})
// Attach it as a middleware BEFORE our routes
app.use(logger)
app.use('/v1', router)This will log every request that comes into our server, but we also need to use it inside our handler to log any unfortunate errors during the data retrieval
app.get('/prompts', (req, res) => {
try {
const prompt = promptService.getCurrentPrompt(req.user)
return res.status(200).json(prompt)
} catch (error) {
logger.error(error)
return res
.status(500)
.json({ message: 'Failed to retrieve the current prompt' })
}
})We can use the logger in our domain and database layers too.
We don’t want to log a successful message every time we fetch something from the database, though. That’s not a notable event - that’s how our application is supposed to work. But if we have an important edge case that we’re handling, we might want to log that it’s happened.
Another thing to note is that we need to be careful when logging sensitive data. Don’t log the whole user object to show who experienced an error, log the identifier and then look for any relevant information manually.
Hardcoding values
In the front-end chapter, we made sure to extract all the links and hardcoded values out into constants that we can import. The reasons behind that are simple - we want to give descriptive names to the values, avoid errors, and have a centralized place to manage them.
The URLs inside the REST API are no exception.
const endpoints = {
activePrompt: '/prompts',
}
router.get(endpoints.activePrompt, authenticate, (req, res) => {
// ...
})Extracting the endpoints is useful mostly if you’re using HATEOAS and you have to reference them in multiple places. If you’re only referencing the URL once, when you’re defining the handler, then feel free to hardcode it.
Error codes are something I’d extract, just to ensure myself that I don’t write the wrong status code by accident.
router.get(endpoints.activePrompt, authenticate, (req, res) => {
try {
const prompt = getPromptAndAnswersIfUserContributed(
req.user,
currentDate
)
res.status(http.OK).json(prompt)
} catch (error) {
logger.error(error)
res
.status(http.InternalServerError)
.json({ message: 'Failed to retrieve the current prompt' })
}
})Occasionally we may need to use a more “exotic” status code and it will be better to have a name attached to it to describe its purpose. For example, not everyone is familiar with the 422 status code, but returning http.UnprocessableEntity when validation fails describes the situation much better.
What’s next?
We’ve introduced the fundamental principles we will follow when building our application, both in the front-end and back-end. Next, we will see how they get applied in different scenarios, then we’ll slowly start zooming out, focusing on the application structure, and then on the infrastructure of the product.
