Achieving high scalability and availability for RideShare application’s database using Docker SDK, Zookeeper leader-election and Rabbit MQ
Note: This is not an in-depth explanation, just a brief explaination of all the points one must cover if they know what the terms in the title mean.
Imagine you want to build an Uber-like application, with a REST-based backend architecture, the major question that arises is —
HOW WILL YOU MANAGE THE TRAFFIC?
There are going to be thousands of incoming requests, for creating and deleting users/rides, querying user details/ride details, and so on. How do you build a backend which can keep up to such demands without breaking down? Especially when you’re just starting up and hosting your application on a low powered AWS server?
Step 1: Users and Rides Endpoints
So, we have two major query parts of the entire application, USERS, and RIDES. The first step is to build and serve all your respective REST endpoints to these services via two different servers. This will help you keep your code clean and clutter-free. Your endpoints might contain functions to create a user/ride, deleting user/ride, querying ride details, so on and so forth.
Users and Rides can be deployed on two different EC2 instances and can be connected via a simple legacy amazon load balancer service.
Step 2: Create an Orchestrator — A 3rd server on a different instance to handle all your database requests
Remember, do not directly connect your users and rides to your databases directly, always create a 3rd endpoint file to which both the above files forward the request. This will help you at a later stage with messaging queues.
Step 3: Database design
For simplicity, let’s assume a master-slave architecture for your database where the writes happen only to the master whereas the reads can happen from any of the slaves.
Step 4: Dockererize everything — Scalability
You need to learn to use docker. It will help you dockerize each instance of your database (i.e one master and multiple slaves). This will allow you to run multiple instances of the same database service on your machine without breaking a sweat or any of the process interfering or any unwanted port issues.
And as they are all running in their own respective shells, using the DOCKER SDK you will be able to dynamically scale up or down the number of database instances based on the number of incoming requests. This will require a container containing a watch script. This script basically checks the number of requests the orchestrator is being bombarded with, ever two minutes, and using the docker SDK either scales up or scales down the number of pairs of slave-workers and their respective DB containers.
Step 5: Rabbit MQ — A message queue service
You now have all your 3 endpoint scripts. So your Users/Rides endpoints receive the initial requests, which are then forwarded to your Orchestrator. Your orchestrator now needs to interact with the database. But there’s a catch. There are multiple copies of the database. (Let’s assume 2– a master and a slave) So which one will it send a request to? This is where message queues come into play.
Message queues are just FIFO queues in which your orchestrator can insert any message into a particular queue, and your working scripts — can pick up the messages from these queues and execute them. Once a worker script picks up a message from the queue, it’s no longer in the queue. Non-persistent.
An everyday example would be the chef ( orchestrator ) sliding different types of McDonald’s burgers into different trays, and the counter boy (worker) picking up a specific burger from its respective slide and giving it to the customer (database)
Hence you need to instantiate a docker container with RabbitMQ’s image and instantiate message queues in it — Writes, Reads, Response, and Sync
Hence from your orchestrator — All your write requests are forwarded into your Write Queue and Read requests into your read queue.
The master-worker (another docker container running an infinite looped script) only picks up messages from the Write queue and executes it into the master database. Whatever the master writes is also written into the Sync Queue.
The slave-worker (another docker container running an infinite looped script) only picks up messages from the Read queue and queries it from the master database and returns the result into the Response queue. The slave worker script also reads from the sync queue and executes those write statements to keep up with the state of the master database.
Note: Remember to either keep your sync queue persistent until all your slave workers have read the update or publish N copies of the message into the sync queue for N slaves to pick up one by one. As it will always follow a race condition — every slave will just pick one message.
Here’s a snipped of the core logic of the worker scripts — the role of the worker can be very easily switched using an interplay of environment variables and zookeeper’s node data. The switching of the node-data happens on line 44 & 52 in the zookeeper script :)
Step 6: Zookeeper Cluster coordination service
At this point, your entire backend is 100% functional. Zookeeper comes in, to manage any failures. i.e what if one of your database containers crash? Your entire system goes down :(
So that is why there needs to be a system in place which will automatically spawn new docker containers for your worker scripts and their respective databases. So now, you need two more containers to pull this off . In the first new container, you need to initialize the Zookeeper base image and start the service. In the second container, you need to have another script, a zookeeper watch decorator is used, which gets triggered every time a slave database or master database crashes.
But how will it know that? It is because, on boot-up, the master-worker script and the slave worker script register themselves with the Zookeeper service as its nodes. Essentially commanding zookeepers to babysit them in case they fail.
Hence, the zookeeper watch script can query a list of all the nodes registered with the zookeeper service and can tell if any of them have crashed based on a simple difference operation between its last query and current query. For example: If the last list of nodes (can simply be stored as a temporary variable) returned with [M, S1, S2, S3], and the now if the zookeeper service returned [M, S1, S2] — It knows that S3 has failed. And hence using the docker SDK, it will create a docker container containing a new slave worker and another containing the database.
Note: Master can crash too, your can either employ a leader-election algorithm to convert one of the slave workers into master — or just start up a whole new container (2 of them) one for the master script and one to act as the master database.
Hence, achieving high availability.
The following snippet describes the core logic of the script which connects and talks to the zookeeper service —
Here’s a picture denoting the entire set-up
The rmq and zoo are the RabbitMQ and Zookeeper services initialized using their respective docker images.
The zookeeper container talks to the zoo service to query a list of all the active nodes.
The master and slave refer to the worker nodes which register themselves with the zoo service on boot.
So what does the docker-compose to launch this entire backend finally look like?
If you have made it this far, Happy deployment! Thanks for reading!