An Efficient Disaster Recovery Mechanism for Multi-region Apache Kafka Clusters

Abstract

Today many internet messaging platforms are built based on publish-subscribe event delivery models that needs to deliver large amounts of messages to ubiquitous clients in a timely manner. To deal with the huge amount of messages reliably, fault tolerance and disaster recovery, e.g. power outages, network interruptions, and software failures, have become vital issues. This paper studies the issue of disaster recovery for the popular Apache Kafka message delivery system. We adopt the strategy of cooperative redundant among multi-region Kafka clusters. By using this approach, the internet messaging applications connect to a local Kafka consumer gateway which uses multi-threading to connect multiple Kafka clusters in different regions. The gateway automatically selects active cluster thereby achieves location transparency and fault tolerance.