From monolith to microservices - a journey
Reposted from InsurtechGuys (my new insurance and insurtech related blog).
This is part 2 in the series of articles about the microservices architecture for Insurance and insurtech companies.
In part 1, I gave a brief overview of microservices architecture. In this part I'll provide a rough guideline of how you can either build an application ground-up with microservices architecture or migrate a monolithic architecture to microservices.
Introspect a.k.a. do I really need a microservices architecture?
Although there are plenty of reasons to be enamoured by microservices architecture, it does have a high "initial setup" cost in terms of money, effort and collective cognitive load on the team. It is possible to build a monolithic application with clean modular components (that all sit within one process or platform) and scale it vertically or horizontally as you see fit. There is nothing inherently wrong with monolithic apps if the modules and interfaces are defined properly. It is also possible to to do a bad job of implementing microservices - the services are too granular or the interface is not well defined which leads to high operational overhead and creates a noisy distributed architecture.
It is possible to build a monolithic application with clean modular components (that all sit within one process or platform) and scale it vertically or horizontally as you see fit. There is nothing inherently wrong with monolithic apps if the modules and interfaces are defined properly.
A microservices architecture also requires you to have a higher organisational process maturity and a team experienced with maintaining medium to large systems or web applications. If you don't have a team with the correct skillset and/or mindset you might end up botching up the implementation/migration.
You will need to dedicate time and resources for the re-architecture which means you may have to pause or de-prioritise other initiatives until the migration is complete. You may choose to phase the re-architecture (see below) and incrementally create one service at a time but one shouldn't underestimate the impact of such endeavours on long term product roadmap. Make sure you understand the trade-offs involved early on before you commit to this journey.
It is also possible that microservices architecture might be an overkill for your particular application or system. You should review your current tech stack and understand the pain points and contrast them against the benefits of moving to microservices If the perceived gains are only 10% then maybe microservice is not the right choice at this point in time. You may revisit this decision in 12 - 18 months as your organisation and monolithic application grows and you can't deliver changes fast enough.
After considering the factors listed above, if you still think microservices is the right approach for you then read on.
Build a devops culture
One of the (unwritten) pre-requisites before you can start hacking your monolith into microservices is to define and setup a continuous integration and continuous delivery process. A minimum level of operational readiness maturity is required before you trot down the path of actualising your microservices dream. You will need to build new continuous delivery pipelines, a way to programmatically set-up up infrastructure (ideally in the cloud), and the ability to debug and monitor a distributed architecture.
You can choose to build out the above and related api management system(see below) with the first and second service that you build. You will have to re-configure your team around the service being built. This means creating a cross functional team or squad that is not only responsible for designing, building and deploying the service but are also responsible for its ongoing maintenance and enhancements. It also means developers and ops teams working together to design, build, test and deploy the service application a.k.a devops.
Define your services
Services or modules or components - whatever you call them - are the core of your microservices design. You want to think long and hard on what services you need and how you would like to design and build them - especially if you are migrating from a monolith.
The basic principles are:
The service wraps a business capability or group of related business capabilities. You can use the principles of domain driven design to define and capture the services raison d’etre. Ideally you want one service per business capability but you also don’t want your service to be too granular or you’ll end up with an operational overhead too big to manage. Designing the size of the service is more of an art than science.
Each service is self contained and has its own UI, business logic, and data store. Both UI and data store are optional, the only minimum requirement for a service is the domain logic that it encapsulates.
The order in which you create microservices from your monolith will determine the complexity of the approach. The recommended approach is to start with capabilities that are fairly decoupled from the monolith i.e. they don’t require changes to many client facing applications. Then you should tackle some of the core functionality of the monolith and extract them out into ms one service at a time.
When you move out a capability from monolith, make sure you also decouple its data. This allows you to create a service that you are able to release/deploy independently and this principle should guide every decision you make around how to perform the decoupling. This also means you need a robust data migration strategy that allows your incrementally move data away from your central data store (that relates to the monolith).
It’s important to reiterate that the design of the service and the order in which services are extracted and/or built will determine the success of the migration.
Asynchronous Communication and Infrastructure setup
You will need to set up infrastructure and operational processes that allow you to do service registration and discovery, monitoring and logging, resiliency (take corrective actions during failure), devops, and (optionally) an api gateway.
There are 3 main integration patterns for the services to interact with each other:
Point to Point - services and clients talking to each other directly
Message broker - lightweight asynchronous messaging
API gateway - most common design pattern used in ms. It has minimal routing capabilities and acts as a ‘dumb pipe’ with no business logic inside.
API Gateway design pattern is becoming a popular “middleware” option for ms. It acts as a single point of entry for clients and exposes microservices to the outside world as managed APIs. It also additionally supports the following functions - load balancing, authentication and authorisation, failure handling, auditing, protocol translation and routing. You can either build your own api gateway or use an off the shelf product from a vendor.
Hopefully that gives you a flavour of the scope and effort involved in migrating a monolith to a microservice.
Reach out to me on twitter (@anup) if you have any follow-up questions.