Achieving high scalability and availability for RideShare application’s database using Docker SDK, Zookeeper leader-election and Rabbit MQ

Note: This is not an in-depth explanation, just a brief explaination of all the points one must cover if they know what the terms in the title mean.

Imagine you want to build an Uber-like application, with a REST-based backend architecture, the major question that arises is —


Step 1: Users and Rides Endpoints

Users and Rides can be deployed on two different EC2 instances and can be connected via a simple legacy amazon load balancer service.

Step 2: Create an Orchestrator — A 3rd server on a different instance to handle all your database requests

Step 3: Database design

Step 4: Dockererize everything — Scalability

Step 5: Rabbit MQ — A message queue service

Message queues are just FIFO queues in which your orchestrator can insert any message into a particular queue, and your working scripts — can pick up the messages from these queues and execute them. Once a worker script picks up a message from the queue, it’s no longer in the queue. Non-persistent.

An everyday example would be the chef ( orchestrator ) sliding different types of McDonald’s burgers into different trays, and the counter boy (worker) picking up a specific burger from its respective slide and giving it to the customer (database)

Hence you need to instantiate a docker container with RabbitMQ’s image and instantiate message queues in it — Writes, Reads, Response, and Sync

Hence from your orchestrator — All your write requests are forwarded into your Write Queue and Read requests into your read queue.

The master-worker (another docker container running an infinite looped script) only picks up messages from the Write queue and executes it into the master database. Whatever the master writes is also written into the Sync Queue.

The slave-worker (another docker container running an infinite looped script) only picks up messages from the Read queue and queries it from the master database and returns the result into the Response queue. The slave worker script also reads from the sync queue and executes those write statements to keep up with the state of the master database.

Note: Remember to either keep your sync queue persistent until all your slave workers have read the update or publish N copies of the message into the sync queue for N slaves to pick up one by one. As it will always follow a race condition — every slave will just pick one message.

Here’s a snipped of the core logic of the worker scripts — the role of the worker can be very easily switched using an interplay of environment variables and zookeeper’s node data. The switching of the node-data happens on line 44 & 52 in the zookeeper script :)

Step 6: Zookeeper Cluster coordination service

So that is why there needs to be a system in place which will automatically spawn new docker containers for your worker scripts and their respective databases. So now, you need two more containers to pull this off . In the first new container, you need to initialize the Zookeeper base image and start the service. In the second container, you need to have another script, a zookeeper watch decorator is used, which gets triggered every time a slave database or master database crashes.

But how will it know that? It is because, on boot-up, the master-worker script and the slave worker script register themselves with the Zookeeper service as its nodes. Essentially commanding zookeepers to babysit them in case they fail.

Hence, the zookeeper watch script can query a list of all the nodes registered with the zookeeper service and can tell if any of them have crashed based on a simple difference operation between its last query and current query. For example: If the last list of nodes (can simply be stored as a temporary variable) returned with [M, S1, S2, S3], and the now if the zookeeper service returned [M, S1, S2] — It knows that S3 has failed. And hence using the docker SDK, it will create a docker container containing a new slave worker and another containing the database.
Note: Master can crash too, your can either employ a leader-election algorithm to convert one of the slave workers into master — or just start up a whole new container (2 of them) one for the master script and one to act as the master database.

Hence, achieving high availability.

The following snippet describes the core logic of the script which connects and talks to the zookeeper service —

Here’s a picture denoting the entire set-up

The rmq and zoo are the RabbitMQ and Zookeeper services initialized using their respective docker images.

The zookeeper container talks to the zoo service to query a list of all the active nodes.

The master and slave refer to the worker nodes which register themselves with the zoo service on boot.

So what does the docker-compose to launch this entire backend finally look like?

If you have made it this far, Happy deployment! Thanks for reading!