Step-by-Step Guide to Kafka MirrorMaker 2 for Cross-Data Center Replication

Introduction

Apache Kafka has established itself as the backbone for real-time data streaming architectures, powering numerous critical applications. When organizations operate across multiple geographical regions or data centers, ensuring seamless and reliable data replication becomes paramount. This is where Kafka MirrorMaker 2 (MM2) shines.

MirrorMaker 2 is a robust tool introduced as part of Apache Kafka 2.4, designed to simplify and optimize the process of cross-data center replication. Unlike its predecessor, MirrorMaker 1, MM2 is built on Kafka Connect, giving it a more scalable, manageable, and extensible architecture.

Cross-data center replication plays a vital role in disaster recovery, geo-redundancy, data locality, and compliance. By replicating Kafka topics across multiple clusters geographically, organizations can maintain high availability, reduce latency for local consumers, and meet regulatory requirements.

In this comprehensive guide, we'll dive deep into Kafka MirrorMaker 2, from understanding its architecture to step-by-step implementation and optimization techniques. Whether you're just starting or looking to refine your replication strategy, this tutorial covers everything you need.

Understanding Kafka MirrorMaker 2 Architecture

Key Components and Workflow

Kafka MirrorMaker 2 is built on top of Kafka Connect, which acts as an extensible framework for streaming data between Kafka clusters. The core components include:

• Source Cluster: The Kafka cluster where your primary data resides.
• Target Cluster: The cluster where data gets replicated.
• MirrorMaker 2 Connectors: Specialized Kafka Connect source and sink connectors designed to replicate topic data and metadata.
• Offset Sync: Synchronizes consumer group offsets between clusters for seamless failover.

The workflow involves MirrorMaker 2 replicating topic data, consumer offsets, and ACL configurations from the source cluster to the target cluster continuously, enabling near real-time synchronization.

Differences Between MirrorMaker 1 and 2

Feature	MirrorMaker 1	MirrorMaker 2
Architecture	Custom, built directly on Kafka	Built on Kafka Connect framework
Manageability	Limited, manual scaling	Scalable, with connector management UI
Metadata Replication	No	Yes (consumer offsets, ACLs, configs)
Fault Tolerance	Basic	Enhanced fault tolerance and restartability
Configuration Flexibility	Minimal	Highly configurable and extendable

Key Considerations for Cross-Data Center Replication

• Latency and Bandwidth: Cross-region network latency can impact replication speed.
• Data Consistency: Eventual consistency is typical; design consumers accordingly.
• Security: Ensure encrypted communication between clusters.
• Topic Configurations: Topics must be pre-created or auto-created if enabled.
• Conflict Resolution: Multi-active active scenarios require careful topic and offset management.

Preparing Your Environment

Prerequisites and System Requirements

• Kafka clusters running version 2.4 or later (MirrorMaker 2 introduced in 2.4).
• Java 8 or higher installed on nodes running MirrorMaker 2.
• Network connectivity between source and target clusters.
• Appropriate authentication mechanisms in place (SSL/SASL).
• Sufficient system resources for Kafka Connect workers handling replication.

Setting up Kafka Clusters in Source and Target Data Centers

Ensure both Kafka clusters are fully operational with:

• Properly configured brokers with advertised listener configurations.
• Matching topic configurations (partition counts, replication factors).
• Access control configured if security is enabled.

Network and Security Considerations for Replication

• Firewall Rules: Allow Kafka broker ports and MirrorMaker 2 workers communication ports.
• Encryption: Use SSL/TLS between clusters.
• Authentication: SASL/PLAIN, SASL/SCRAM, or Kerberos as per organizational policy.
• Authorization: Replicating users require appropriate ACLs for topics, consumer groups, and configurations.

Step-by-Step Implementation of MirrorMaker 2

Installing MirrorMaker 2

MirrorMaker 2 is included in Apache Kafka distributions version 2.4+. Download and extract Kafka on the host where MM2 will be deployed:

wget https://downloads.apache.org/kafka/3.4.0/kafka_2.13-3.4.0.tgz
tar -xzf kafka_2.13-3.4.0.tgz
cd kafka_2.13-3.4.0

You can run MirrorMaker 2 using the packaged connect-mirror-maker.sh script.

Configuring Source and Target Kafka Clusters

Prepare configuration files for both clusters including bootstrap servers and security settings.

Example snippet for source cluster config (source-cluster.properties):

bootstrap.servers=source-kafka-broker1:9093,source-kafka-broker2:9093
security.protocol=SSL
ssl.truststore.location=/path/to/source/truststore.jks
ssl.truststore.password=truststore-password
ssl.keystore.location=/path/to/source/keystore.jks
ssl.keystore.password=keystore-password
ssl.key.password=key-password

Similarly for target cluster (target-cluster.properties):

bootstrap.servers=target-kafka-broker1:9093,target-kafka-broker2:9093
security.protocol=SSL
ssl.truststore.location=/path/to/target/truststore.jks
ssl.truststore.password=truststore-password
ssl.keystore.location=/path/to/target/keystore.jks
ssl.keystore.password=keystore-password
ssl.key.password=key-password

Creating and Configuring MirrorMaker 2 Properties Files

Create a MirrorMaker 2 configuration file (mirror-maker.properties) that specifies the clusters and replication parameters:

# Connector class
clusters=SOURCE, TARGET

# Source cluster config prefix
SOURCE.bootstrap.servers=source-kafka-broker1:9093,source-kafka-broker2:9093
SOURCE.security.protocol=SSL
SOURCE.ssl.truststore.location=/path/to/source/truststore.jks
SOURCE.ssl.truststore.password=truststore-password
SOURCE.ssl.keystore.location=/path/to/source/keystore.jks
SOURCE.ssl.keystore.password=keystore-password
SOURCE.ssl.key.password=key-password

# Target cluster config prefix
TARGET.bootstrap.servers=target-kafka-broker1:9093,target-kafka-broker2:9093
TARGET.security.protocol=SSL
TARGET.ssl.truststore.location=/path/to/target/truststore.jks
TARGET.ssl.truststore.password=truststore-password
TARGET.ssl.keystore.location=/path/to/target/keystore.jks
TARGET.ssl.keystore.password=keystore-password
TARGET.ssl.key.password=key-password

# Replication policies
replication.policy.class=org.apache.kafka.connect.mirror.DefaultReplicationPolicy
replication.factor=3

# Topics to replicate (can use regex)
topics=.*

# Enable offset sync
emit.heartbeats.enabled=true
sync.group.offsets.enabled=true

# Replicate consumer groups and configs
sync.topic.configs.enabled=true
sync.acls.enabled=true

# Number of tasks
tasks.max=4

Starting MirrorMaker 2 for Replication

Use the included script to launch MirrorMaker 2:

bin/connect-mirror-maker.sh mirror-maker.properties

This starts MirrorMaker 2, initializing connectors and beginning replication.

Monitoring and Troubleshooting Replication

• Check the logs in the Kafka Connect worker output for errors.
• Use Kafka Connect REST API to query connector and task status:

curl http://localhost:8083/connectors
curl http://localhost:8083/connectors/your-mm2-connector/status

• Monitor topic lag and consumer offsets to ensure replication health.
• Adjust logging verbosity as needed.

Code Example: MirrorMaker 2 Configuration and Deployment

Sample Properties Files for Source and Target Clusters

source-cluster.properties:

bootstrap.servers=source-kafka-1:9093,source-kafka-2:9093
security.protocol=SSL
ssl.truststore.location=/etc/kafka/source.truststore.jks
ssl.truststore.password=changeit
ssl.keystore.location=/etc/kafka/source.keystore.jks
ssl.keystore.password=changeit
ssl.key.password=changeit

target-cluster.properties:

bootstrap.servers=target-kafka-1:9093,target-kafka-2:9093
security.protocol=SSL
ssl.truststore.location=/etc/kafka/target.truststore.jks
ssl.truststore.password=changeit
ssl.keystore.location=/etc/kafka/target.keystore.jks
ssl.keystore.password=changeit
ssl.key.password=changeit

Example Command for Starting MirrorMaker 2

bin/connect-mirror-maker.sh 
  --clusters SOURCE,TARGET 
  --source-cluster-config source-cluster.properties 
  --target-cluster-config target-cluster.properties 
  --whitelist '.*' 
  --config mirror-maker.properties

Script Snippet for Automation and Management

A simple bash script to restart MM2 service:

#!/bin/bash

SERVICE_NAME=mirror-maker

stop_mm2() {
  pkill -f connect-mirror-maker.sh
  echo "Stopped MirrorMaker 2"
}

start_mm2() {
  ./bin/connect-mirror-maker.sh mirror-maker.properties > mm2.log 2>&1 &
  echo "Started MirrorMaker 2"
}

restart_mm2() {
  stop_mm2
  sleep 5
  start_mm2
}

case "$1" in
  start)
    start_mm2
    ;;
  stop)
    stop_mm2
    ;;
  restart)
    restart_mm2
    ;;
  *)
    echo "Usage: $0 {start|stop|restart}"
    exit 1
esac

Best Practices and Performance Optimization

Tuning Replication Performance

• Parallelism: Increase the number of tasks (tasks.max) for faster replication.
• Batch Size and Poll Intervals: Tune the producer and consumer batch sizes and intervals.
• Compression: Use compression (e.g., snappy) to reduce bandwidth usage.
• Network QoS: Prioritize replication traffic to reduce latency.

Handling Failover and Conflict Scenarios

• Synchronize consumer offsets to enable smooth failover.
• For active-active clusters, design topics and replication to prevent circular replication.
• Use replication policies to rename or filter topics appropriately.

Managing Quotas and Network Usage

• Configure Kafka quotas on producers and consumers to prevent replication overload.
• Monitor network bandwidth and adjust replication rate limits if needed.

Conclusion

Kafka MirrorMaker 2 is a powerful, flexible tool to achieve reliable cross-data center replication, ensuring your streaming data is highly available and resilient. Leveraging its Kafka Connect based architecture offers superior control, easier management, and enhanced features over MirrorMaker 1.

This guide has walked you through the architecture fundamentals, environment preparation, detailed deployment steps, and best practices to optimize and maintain your replication pipelines effectively.

By following these practices, you can confidently implement cross-data center Kafka replication that meets your availability, disaster recovery, and compliance goals.

Additional Resources

• Kafka MirrorMaker 2 official documentation
• Kafka Connect documentation
• Confluent's blog on MirrorMaker 2

---

FAQ

What is the main difference between MirrorMaker 1 and MirrorMaker 2?

MirrorMaker 2 is built on the Kafka Connect framework, offering better scalability, manageability, and replication of metadata like offsets and ACLs, unlike MirrorMaker 1 which is a simpler, standalone tool.

Can MirrorMaker 2 replicate consumer offsets and ACLs?

Yes, MirrorMaker 2 supports syncing consumer group offsets and ACLs between source and target clusters for seamless failover.

Is it possible to filter which topics get replicated?

Yes, by configuring whitelist or regex filters in the MirrorMaker 2 properties, you can selectively replicate topics.

How do I monitor the health of MirrorMaker 2 replication?

Use Kafka Connect's REST API for connector status, monitor logs, and track topic offsets and consumer group lags.

Can MirrorMaker 2 be used in cloud environments?

Absolutely. MirrorMaker 2 can run on any environment where Kafka and Java are supported, including public clouds, private clouds, and hybrid deployments.

What are common causes of replication lag?

High network latency, insufficient resources on Kafka Connect workers, and cluster load spikes often contribute to replication lag. Optimizing these parameters helps reduce lag.

Related reading

• Implementing Kafka Tiered Storage for Cost-Effective Long-Term Data Retention
• Implementing Kafka Connect for Streamlined Integration with Databases and Data Lakes
• Implementing Kafka Streams for Real-Time Data Processing with Fault Tolerance