Migrate third-party and self-managed Apache Kafka clusters to Amazon MSK Specific brokers with Amazon MSK Replicator

Migrating Apache Kafka workloads to the cloud usually entails managing advanced replication infrastructure, coordinating utility cutovers with prolonged downtime home windows, and sustaining deep experience in open-source instruments like Apache Kafka’s MirrorMaker 2 (MM2). These challenges decelerate migrations and improve operational threat. Amazon MSK Replicator addresses these challenges, enabling you emigrate your Kafka deployments (known as “exterior” Kafka clusters) to Amazon MSK Specific brokers with minimal operational overhead and lowered downtime. MSK Replicator helps information migration from Kafka deployments (model 2.8.1 or later) which have SASL/SCRAM authentication enabled – together with Kafka clusters operating on-premises, on AWS, or different cloud suppliers, in addition to Kafka-protocol-compatible companies like Confluent Platform, Avien, RedPanda, WarpStream, or AutoMQ when configured with SASL/SCRAM authentication.

On this submit, we stroll you thru the right way to replicate Apache Kafka information out of your exterior Apache Kafka deployments to Amazon MSK Specific brokers utilizing MSK Replicator. You’ll learn to configure authentication in your exterior cluster, set up community connectivity, arrange bidirectional replication, and monitor replication well being to realize a low-downtime migration.

The way it works

MSK Replicator is a completely managed serverless service that replicates matters, configurations, and offsets from cluster to cluster. It alleviates the necessity to handle advanced infrastructure or configure open-source instruments.

Earlier than MSK Replicator, clients used instruments like MM2 for migrations. These instruments lack bi-directional subject replication when utilizing the identical subject names, creating advanced utility architectures to eat completely different matters on completely different clusters. Customized replication insurance policies in MM2 can enable equivalent subject names, however MM2 nonetheless lacks bidirectional offset replication as a result of the MM2 structure requires producers and shoppers to run on the identical cluster to duplicate offsets. This created advanced migrations that required both migrating shoppers earlier than producers or big-bang migrations migrating all functions directly. When clients run into points in the course of the migration, the rollback course of is error-prone and introduces massive quantities of duplicate message processing because of the lack of shopper group offset synchronization. These approaches create threat and complexity for patrons that make migrations tough to handle.

MSK Replicator addresses these issues by supporting bidirectional replication of knowledge and enhanced shopper group offset synchronization. MSK Replicator copies matters and offsets from an exterior Kafka cluster to MSK, permitting you to protect the identical subject and shopper group names on each clusters. MSK Replicator additionally helps making a second Replicator occasion for bidirectional replication of each information and enhanced offset synchronization, permitting producers and shoppers to run independently on completely different Kafka clusters. Information revealed or consumed on the Amazon MSK cluster might be replicated again to the exterior cluster by the second Replicator. This characteristic works when producers and shoppers are migrated no matter order with out worrying about dependencies between functions.

As a result of MSK Replicator gives bidirectional information replication and enhanced shopper group offset synchronization, you’ll be able to transfer producers and shoppers at your individual tempo with out information loss. This reduces migration complexity, permitting you emigrate functions between your exterior Kafka cluster and Amazon MSK no matter order. In case you run into issues in the course of the migration, enhanced offset synchronization lets you roll again adjustments by transferring functions again to the exterior Kafka cluster, the place they restart from the newest checkpoint from the Amazon MSK cluster.

For instance, take into account three functions:

1. The “Orders” utility, which accepts incoming orders and writes them to the orders Kafka subject
2. The “Order standing” utility, which reads from the “orders” Kafka subject and writes standing updates to the order_status subject
3. The “Buyer notification” utility, which reads from the order_status subject and notifies clients when standing adjustments

MSK Replicator permits these functions to be migrated between an on-premises Apache Kafka cluster and an Amazon MSK Specific cluster with low downtime and no information loss, no matter order. The “Order standing” utility can migrate first, obtain orders from the on-premises “Orders” utility, and ship standing updates to the on-premises “Buyer notification” utility. If points come up in the course of the migration, the “Order standing” utility can roll again to the on-premises cluster and its shopper group offsets for the orders subject might be prepared for it to select up from the place it left off on the Amazon MSK cluster.

MSK Replicator helps information distribution throughout hybrid and multi-cloud environments for analytics, compliance, and enterprise continuity. It’s also configured for catastrophe restoration situations the place Amazon MSK Specific serves as a resilient goal to your exterior Kafka clusters.

In case you are at present utilizing MM2 for replication, see Amazon MSK Replicator and MirrorMaker2: Choosing the proper replication technique for Apache Kafka catastrophe restoration and migrations to know which resolution most closely fits your use case.

Answer overview

MSK Replicator helps Kafka deployments operating model 2.8.1 or later as a supply, together with third occasion managed Kafka companies, self-managed Kafka, and on-premises or third-party cloud-hosted Kafka. MSK Replicator routinely handles information switch, makes use of SASL/SCRAM authentication with SSL encryption, and maintains shopper group positions throughout each clusters. If you don’t use SASL/SCRAM right now, this may be configured as a brand new listener used for MSK Replicator permitting present purchasers to make use of their current authentication mechanisms alongside MSK Replicator.

Stipulations

To observe together with this walkthrough, you want the next sources in place:

Organising replication

Step 1: Configure community connectivity

You possibly can arrange community connectivity between your exterior Kafka cluster and your AWS VPC utilizing strategies corresponding to AWS Direct Join for devoted community connections, AWS Web site-to-Web site VPN for encrypted connections over the web, and AWS VPC peering or AWS Transit Gateway for connections between AWS VPCs. Confirm that IP routing and DNS decision are correctly configured between your exterior cluster and AWS.

To confirm IP routing and DNS decision, hook up with your exterior Kafka cluster from inside your VPC by utilizing the Kafka CLI to record matters on the exterior cluster. In case you can record matters out of your VPC utilizing the Kafka CLI, this implies DNS decision and IP routing are working efficiently. If it fails, work together with your community admins to troubleshoot community connectivity points.

Step 2: Configure exterior cluster

On this step, you’ll arrange authentication in your exterior Kafka cluster and retailer the credentials in AWS Secrets and techniques Supervisor in order that MSK Replicator can join securely.

Configure authentication

Utilizing the exterior cluster admin consumer, configure SASL/SCRAM authentication for MSK Replicator utilizing SHA-256 or 512 in your exterior Kafka cluster. Create a SASL/SCRAM consumer for MSK Replicator and provides the consumer the next ACL permissions:

• Subject operations – Alter, AlterConfigs, Create, Describe, DescribeConfigs, Learn, Write
• Group operations – Learn, Describe
• Cluster operations – Create, ClusterAction, Describe, DescribeConfigs

Configure SecretsManager

AWS Secrets and techniques Supervisor shops your SASL/SCRAM credentials securely in order that MSK Replicator can retrieve them at runtime. The key should use JSON format and have the next keys:

• username – The SCRAM username that you just configured within the authentication step above
• password – The SCRAM password that you just configured within the authentication step above
• certificates – The general public root CA certificates (the top-level certificates authority that issued your cluster’s TLS certificates) and the intermediate CA chain (intermediate certificates between the basis and your cluster’s certificates), used for SSL handshakes with the exterior cluster

Optionally, you could create separate secrets and techniques for SCRAM credentials and the SSL certificates. This method is beneficial when secrets and techniques for SCRAM credentials and certificates are provisioned in numerous phases, corresponding to in Infrastructure as Code (IaC) pipelines.

Retrieve the cluster ID

Because the admin consumer, use the Kafka CLI instruments to retrieve the cluster ID of your exterior cluster. Run the next command, changing your-broker-host:9096 with the deal with of one among your exterior cluster’s bootstrap servers:

bin/kafka-cluster.sh cluster-id --bootstrap-server your-broker-host:9096 --config admin.properties

The command returns a cluster ID string corresponding to lkc-abc123. Pay attention to this worth as a result of you have to it when creating the replicator in Step 4.

Step 3: Create your MSK Specific goal cluster

Together with your exterior cluster configured, now you can arrange the goal. Create an Amazon MSK Specific cluster with IAM authentication enabled. Ensure that the cluster is in subnets which have entry to AWS Secrets and techniques Supervisor endpoints. See Get began utilizing Amazon MSK for extra data on creating an MSK cluster.

Step 4: Create the replicator

Now that each clusters are prepared, you’ll be able to join them by establishing the MSK Replicator with the suitable IAM function and replication configuration.

Arrange an IAM function for MSK Replicator

MSK Replicator wants an IAM function to work together together with your MSK Specific cluster and retrieve secrets and techniques. Arrange a service execution IAM function with a belief coverage permitting kafka.amazonaws.com and fix the AWSMSKReplicatorExecutionRole permissions coverage. Pay attention to the function ARN for creating the replicator.

Create and fix a coverage for accessing your Secrets and techniques Supervisor secrets and techniques and studying/writing information in your MSK cluster. See Creating roles and attaching insurance policies (console) for extra data on creating IAM roles and insurance policies.

The next is an instance coverage for studying and writing information to your MSK cluster and studying KMS-encrypted Secrets and techniques Supervisor secrets and techniques:

{ 
    "Model": "2012-10-17", 
    "Assertion": [ 
        { 
            "Sid": "SecretsManagerAccess", 
            "Effect": "Allow", 
            "Action": [ 
                "secretsmanager:GetSecretValue", 
                "secretsmanager:DescribeSecret" 
            ], 
            "Useful resource": [ 
                "", 
                "" 
            ] 
        }, 
        { 
            "Sid": "KMSDecrypt", 
            "Impact": "Permit", 
            "Motion": "kms:Decrypt", 
            "Useful resource": "" 
        }, 
        { 
            "Sid": "TargetClusterAccess", 
            "Impact": "Permit", 
            "Motion": [ 
                "kafka-cluster:Connect", 
                "kafka-cluster:DescribeCluster", 
                "kafka-cluster:AlterCluster", 
                "kafka-cluster:DescribeClusterDynamicConfiguration", 
                "kafka-cluster:AlterClusterDynamicConfiguration", 
                "kafka-cluster:DescribeTopic", 
                "kafka-cluster:CreateTopic", 
                "kafka-cluster:AlterTopic", 
                "kafka-cluster:DescribeTopicDynamicConfiguration", 
                "kafka-cluster:AlterTopicDynamicConfiguration", 
                "kafka-cluster:WriteData", 
                "kafka-cluster:WriteDataIdempotently", 
                "kafka-cluster:ReadData", 
                "kafka-cluster:DescribeGroup", 
                "kafka-cluster:AlterGroup" 
            ], 
            "Useful resource": [ 
                "arn:aws:kafka:::cluster/*/*", 
                "arn:aws:kafka:::topic//*", 
                "arn:aws:kafka:::group/*/*" 
            ] 
        }, 
        { 
            "Sid": "CloudWatchLogsAccess", 
            "Impact": "Permit", 
            "Motion": [ 
                "logs:CreateLogStream", 
                "logs:PutLogEvents", 
                "logs:DescribeLogStreams" 
            ], 
            "Useful resource": "" 
        } 
    ] 
}

Create the replicator for exterior to MSK replication

Use the AWS CLI, API, or Console to create your replicator. Right here’s an instance utilizing the AWS CLI:

aws kafka create-replicator 
  --replicator-name external-to-msk 
  --service-execution-role-arn "arn:aws:iam::123456789012:function/MSKReplicatorRole" 
  --kafka-clusters file://./kafka-clusters.json 
  --replication-info-list file://./replication-info.json 
  --log-delivery file://./log-delivery.json 
  --region us-east-1

The kafka-clusters.json file defines the supply and goal Kafka cluster connection data, replication-info.json specifies which matters to duplicate and the right way to deal with shopper group offset synchronization, and log-delivery.json specifies the CloudWatch logging configuration. The next tables describe the required parameters:

CLI inputs:

Key kafka-clusters.json inputs:

Key replication-info.json inputs:

Word that startingPosition is ready to EARLIEST within the configuration beneath, which implies the replicator begins studying from the oldest obtainable offset on every subject. That is the really helpful setting for migrations to keep away from information loss.

Key log-delivery.json inputs:

Further log supply strategies for Amazon S3 and Amazon Information Firehose are supported. On this submit, we use CloudWatch logging.

The configs ought to appear to be the next for exterior to MSK replication.

kafka-clusters.json:

[ 
  { 
    "ApacheKafkaCluster": { 
      "ApacheKafkaClusterId": "lkc-abc123", 
      "BootstrapBrokerString": "broker1.example.com:9096" 
    }, 
    "ClientAuthentication": { 
      "SaslScram": { 
        "Mechanism": "SHA512", 
        "SecretArn": "arn:aws:secretsmanager:::secret:my-creds" 
      } 
    }, 
    "EncryptionInTransit": { 
      "EncryptionType": "TLS", 
      "RootCaCertificate": "arn:aws:secretsmanager:::secret:my-cert" 
    } 
  }, 
  { 
    "AmazonMskCluster": { 
      "MskClusterArn": "arn:aws:kafka:::cluster/my-cluster/abc-123" 
    }, 
    "VpcConfig": { 
      "SecurityGroupIds": ["sg-0123456789abcdef0"], 
      "SubnetIds": ["subnet-abc123", "subnet-abc124", "subnet-abc125"] 
    } 
  } 
] 

replication-info.json: 

[ 
  { 
    "SourceKafkaClusterId": "lkc-abc123", 
    "TargetKafkaClusterArn": "arn:aws:kafka:::cluster/my-cluster/abc-123", 
    "TargetCompressionType": "LZ4", 
    "TopicReplication": { 
      "TopicsToReplicate": ["my-topic"], 
      "CopyTopicConfigurations": true, 
      "CopyAccessControlListsForTopics": true, 
      "DetectAndCopyNewTopics": true, 
      "StartingPosition": {"Kind": "EARLIEST"}, 
      "TopicNameConfiguration": {"Kind": "IDENTICAL"} 
    }, 
    "ConsumerGroupReplication": { 
      "ConsumerGroupsToReplicate": ["my-group"], 
      "SynchroniseConsumerGroupOffsets": true, 
      "DetectAndCopyNewConsumerGroups": true, 
      "ConsumerGroupOffsetSyncMode": "ENHANCED" 
    } 
  } 
] 

log-delivery.json: 

{ 
  "ReplicatorLogDelivery": {
     "CloudWatchLogs": {
       "Enabled": true, 
       "LogGroup": ""
     }
  } 
}

Configure bidirectional replication from MSK to the exterior cluster

To allow bidirectional replication, create a second replicator that replicates in the other way. Use the identical IAM function and community configuration from Step 4, however swap the supply and goal. Substitute SourceKafkaClusterId with TargetKafkaClusterId and TargetKafkaClusterArn with SourceKafkaClusterArn in a brand new msk-to-external-replication-info.json file:

aws kafka create-replicator 
  --replicator-name msk-to-external 
  --service-execution-role-arn "arn:aws:iam::123456789012:function/MSKReplicatorRole" 
  --kafka-clusters file:///./kafka-clusters.json 
  --replication-info-list file:///./msk-to-external-replication-info.json 
  --log-delivery file:///./log-delivery.json 
  --region us-east-1

Monitoring replication well being

Monitor your replication utilizing Amazon CloudWatch metrics. Three key metrics to know are MessageLag, SumOffsetLag, and ReplicationLatency. MessageLag measures how far behind the replicator is from the exterior cluster when it comes to messages not but replicated, whereas SumOffsetLag measures how far behind a shopper group is from the newest message in a subject. ReplicationLatency is the quantity of latency between the supply and goal clusters in information replication. When the three attain a sustained low stage, your clusters are absolutely synchronized for each information and shopper group offsets.

To troubleshoot MSK Replicator replication or errors, use the CloudWatch logs to get extra particulars concerning the well being of the replicator. MSK Replicator logs standing and troubleshooting data which could be useful in diagnosing points like connectivity, authentication, and SSL errors.

Word that the replication is asynchronous, so there might be some lag throughout replication. The lag will attain zero as soon as a shopper is shut down throughout migration to the goal cluster. This takes about 30 seconds below regular operations, permitting a low downtime migration with out information loss. In case your lag is frequently rising or doesn’t attain a sustained low stage, this means that you’ve got inadequate partitions for high-throughput replication. Confer with Troubleshoot MSK Replicator for extra data on troubleshooting replication throughput and lag.

Key metrics embrace:

• MessageLag – Displays the sync between the MSK Replicator and the supply cluster. MessageLag signifies the lag between the messages produced to the supply cluster and messages consumed by the replicator. It isn’t the lag between the supply and goal cluster.
• ReplicationLatency – Time taken for information to duplicate from supply to focus on cluster (ms)
• ReplicatorThroughput – Common variety of bytes replicated per second
• ReplicatorFailure – Variety of failures the replicator is experiencing
• KafkaClusterPingSuccessCount – Connection well being indicator (1 = wholesome, 0 = unhealthy)
• ConsumerGroupCount – Complete shopper teams being synchronized
• ConsumerGroupOffsetSyncFailure – Failures throughout offset synchronization
• AuthError – Variety of connections with failed authentication per second, by cluster
• ThrottleTime – Common time in ms a request was throttled by brokers, by cluster
• SumOffsetLag – Aggregated offset lag throughout partitions for a shopper group on a subject (MSK cluster-level metric)

For extra particulars on these metrics, see the MSK Replicator metrics documentation.

Your functions are able to migrate when the next circumstances are met. For many workloads, it’s best to anticipate these metrics to stabilize inside just a few hours of beginning replication. Excessive-throughput clusters could take longer relying on subject quantity and partition rely.

• ReplicatorFailure = 0
• ConsumerGroupOffsetSyncFailure = 0
• KafkaClusterPingSuccessCount = 1 for each supply and goal clusters
• MessageLag
• Your sustained lag could also be decrease or larger relying in your throughput per partition, message dimension, and different components
• Sustained excessive message lag normally signifies inadequate partitions for high-throughput replication

Migrating your functions

With bidirectional shopper offset synchronization, you’ll be able to migrate your producers and shoppers no matter order. Begin by monitoring replication metrics till they attain the goal values described within the earlier part. Then migrate your functions (producers or shoppers) to make use of the MSK Specific cluster endpoints and confirm that they’re producing and consuming as anticipated. In case you encounter points, you’ll be able to roll again by switching functions again to the exterior cluster. The patron offset synchronization makes positive that your functions resume from their final dedicated place no matter which cluster they hook up with.

For a complete, hands-on walkthrough of the end-to-end migration course of, discover the MSK Migration Workshop, which gives step-by-step steerage for migrating your Kafka workloads to Amazon MSK.

Safety concerns

MSK Replicator makes use of SASL/SCRAM authentication with SSL encryption for safe information switch between your exterior cluster and AWS. The answer helps each publicly trusted certificates and personal or self-signed certificates. Credentials are saved securely in AWS Secrets and techniques Supervisor, and the goal MSK Specific cluster makes use of IAM authentication for entry management.

When configuring safety, hold the next in thoughts:

• Ensure that the IAM function you create in Step 4 follows the precept of least privileges. Solely connect AWSMSKReplicatorExecutionRole and an IAM coverage for Secrets and techniques Supervisor with least-privileges entry to learn secret values and keep away from including broader permissions.
• Confirm that your Secrets and techniques Supervisor secret is encrypted with an AWS KMS key that the MSK Replicator service execution function has permission to decrypt.
• Verify that the safety teams assigned to MSK Replicator enable outbound site visitors to your exterior cluster’s dealer ports (sometimes 9096 for SASL/SCRAM with TLS) and to the MSK Specific cluster.
• Rotate your SASL/SCRAM credentials periodically and replace the corresponding Secrets and techniques Supervisor secret. MSK Replicator picks up the brand new credentials routinely on the subsequent connection try.

Underneath the AWS shared accountability mannequin, AWS is answerable for securing the underlying infrastructure that runs MSK Replicator, together with the compute, storage, and networking sources. You’re answerable for configuring authentication mechanisms (SASL/SCRAM), managing credentials in AWS Secrets and techniques Supervisor, configuring community safety (safety teams and VPC settings), implementing IAM insurance policies following least privilege, and rotating credentials. For extra data, see Safety in Amazon MSK within the Amazon MSK Developer Information.

Cleanup

To keep away from ongoing fees, delete the sources you created throughout this walkthrough. Begin by deleting the replicators first, as a result of they depend upon the opposite sources:

aws kafka delete-replicator --replicator-arn

After each replicators are deleted, you’ll be able to take away the next sources in the event that they have been created solely for this walkthrough:

1. The MSK Specific cluster (deleting a cluster additionally removes its saved information, so confirm that your functions have absolutely migrated earlier than continuing)
2. The Secrets and techniques Supervisor secrets and techniques containing your SASL/SCRAM credentials and certificates
3. The IAM function and insurance policies created for MSK Replicator

You possibly can confirm {that a} replicator has been absolutely deleted by operating aws kafka list-replicators and confirming it now not seems within the output.

Conclusion

Amazon MSK Replicator simplifies the method of migrating to Amazon MSK Specific brokers and establishes hybrid Kafka architectures. The absolutely managed service alleviates the operational complexity of managing replication whereas bidirectional shopper offset synchronization permits versatile, low-risk utility migration.

Subsequent Steps

To get began utilizing MSK Replicator emigrate functions to MSK Specific brokers, use the MSK Migration Workshop for a hands-on, end-to-end migration walkthrough. The Amazon MSK Replicator documentation consists of detailed configuration particulars to assist configure MSK Replicator to your use case. From there, use MSK Replicator emigrate your Apache Kafka workloads to MSK Specific dealer.

As soon as your migration is full, take into account exploring multi-region replication patterns for catastrophe restoration, or integrating your MSK Specific cluster with AWS analytics companies corresponding to Amazon Information Firehose and Amazon Athena. In case you need assistance planning your migration, attain out to your AWS account group, AWS Help or AWS Skilled Companies.

Concerning the authors