Begin with a monolith over microservices

I've recently joined a startup, and we've started a product from scratch. While designing how to implement the product (specifically the back-end part), our engineering team concluded that building a monolith (single service, instead of multiple services) was a good idea. In this piece, I will explain the reasoning behind the decision.

If you have worked for a startup in an early stage, you might have noticed that knowing what you need to build exactly is hard. Usually, a startup's product or even its business model can change significantly in a short period. Therefore, it is vital to keep in mind that a big concern of a startup, if not the biggest, is to validate its principal product or service as quickly as possible. Accordingly, timing and budget management are critical.

Hence we needed to act fast, and for the first couple of months, we only had a rough idea of our product vision, and from there, we started to enumerate which components were necessary to achieve it. So I won't go over all the details, but I'll summarize the architecture.

In short, we have multiple data sources, which are third-party applications that we need to fetch data from. Since we don't have control over those applications, we have to integrate by using a set of REST APIs they offer.

Each Agent is responsible then for fetching data, performing a few conversions from the data source's model to an internal model, and finally publishing the converted model to a message broker. From our requirements, we knew that eventually, we would need to replay messages sent by the agents, meaning the message broker must offer messages persistence, as Kafka provides.

With the messages in a Kafka topic, the Consumer component could read the messages, perform some data transformation, and finally save it to a database.

I've been working with distributed systems for a while. After discussing the architecture with the team, I immediately thought we would need to build several microservices and set up a Kafka cluster. It seemed a good idea since we already had a Kubernetes cluster up and running for another product.

But then we started to estimate how much time we would need to build the first components and soon realized it would take more time than initially expected. Then, a day later, a team member made the monolithic architecture suggestion. It was something like:

“What if instead of building multiple services and setting up a Kafka cluster, we used a multi-package monolith with an in-memory queue?”

It was an unusual approach given that we were used to microservices and using the pattern in another product. However, after playing with the idea, we decided to try it.

We are using GoLang to build our services, and the language offers some mechanisms to isolate code. Then, in summary, each Agent and consumer are contained in individual packages, and we used an in-memory queue to exchange data between them.

It's important to mention that the team must define a clear path to package communication to use such an approach. With that in mind, even though we're using an in-memory queue, we still have well-defined message schemas by using Apache Avro. With strict schemas, it's easier to change things later when we need to use Kafka.

Another crucial point to pay attention to is the code extensibility and aim to increase it as much as possible, which means taking advantage of many interfaces. For example, since we knew that Kafka would come into play later, we created an abstraction layer on top of the component responsible for sending messages to the in-memory queue. And know, to switch to the actual Kafka implementation, we would only need to implement the defined interface. Implementing this should be simple if you are used to applying the S.O.L.I.D principles in your projects.

One vital trade-off to consider is that we can't have multiple service instances because we're using an in-memory queue, which means we have a scaling limitation. Otherwise, the service would duplicate messages, and the consumer could save outdated data to the database.

Although we can't scale due to our engineering decisions, it's essential to remember that given the context (startup with few users), scalability is not a problem we need to worry about for now. We needed to maximize the team output.

We've been executing this strategy for six months, and the team aggress that the productivity is high due to the simplicity of using a single project (and service). And by defining interfaces and communication schemas, we're confident about converting the service to a microservice architecture later. Honestly, it took me a while to get used to this strategy, but after seeing how much software we were delivering every week, I now believe this was an excellent idea.

I hope this issue was insightful to you. Next time you start the next project, consider keeping everything simple rather than using what you are used to or using the pattern your company has already established.