Configure a customized area title on your Amazon MSK cluster enabled with IAM authentication

Most Amazon Managed Streaming for Apache Kafka (Amazon MSK) prospects are simplifying and standardizing entry management to Kafka sources utilizing AWS Id and Entry Administration (IAM) authentication. This adoption can also be accelerated as Amazon MSK now helps IAM authentication in standard languages together with Java, Python, Go, JavaScript, and .NET.

Within the first a part of Configure a customized area title on your Amazon MSK cluster, we mentioned about why customized domains are essential and offered particulars on find out how to configure a customized area title in Amazon MSK when utilizing SASL_SCRAM authentication. On this put up, we focus on find out how to configure a customized area title in Amazon MSK when utilizing IAM authentication. We advocate you learn the primary a part of this weblog because it captures resolution particulars implementation steps.

Resolution overview

IAM authentication for Amazon MSK makes use of TLS to encrypt the Kafka protocol site visitors between the shopper and Kafka dealer. To make use of a customized area title, the Kafka dealer must current a server certificates that matches the customized area title. To attain this, this resolution makes use of an Community Load Balancers (NLBs) with Amazon Certificates Supervisor to supply a customized certificates on behalf of the MSK brokers, and a Route 53 Non-public Hosted Zone to supply DNS for the customized area title.

The next diagram reveals all parts utilized by the answer.

Certificates administration

For shoppers to carry out TLS communication with the MSK cluster the cluster wants to supply a certificates with hostnames matching the customized area title. This resolution makes use of a certificates in AWS Certificates Supervisor (ACM) signed with a Non-public Certificates Authority (PCA) for TLS with the customized area title. This resolution makes use of a certificates with bootstrap.instance.com because the Frequent Identify (CN) in order that the certificates is legitimate for the bootstrap handle, and Topic Various Names (SANs) are set for all dealer DNS names (resembling b-1.instance.com). Since this resolution makes use of a non-public certificates authority, the CA chain should be imported into the shopper belief shops.

This resolution works with any server certificates, whether or not certificates are signed by a public or non-public Certificates Authority (CA). You’ll be able to import current certificates into ACM for use with this resolution. Certificates should present a typical title and/or topic various names that match the bootstrap DNS handle in addition to the person dealer DNS addresses. If the certificates is issued by a non-public CA, shoppers have to import the foundation and intermediate CA certificates to the shopper belief retailer. If the certificates is issued by a public CA, the foundation and intermediate CA certificates can be within the default belief retailer.

Community Load Balancer

The NLB offers the power to make use of a TLS listener. The ACM certificates is related to the listeners and allows TLS negotiation between the shopper and the NLB. The NLB performs a separate TLS negotiation between itself and the MSK brokers. Along with the above structure, this resolution additionally permits utilizing AWS Non-public Hyperlink to attach the cluster to exterior VPCs. This enables safe entry to MSK between VPCs whereas utilizing a customized area title.

The next diagram illustrates the NLB port and goal configuration. A TLS listener with port 9000 is used for bootstrap connections with all MSK brokers set as targets. IAM authentication is configured to run on port 9098 of the MSK brokers utilizing a TLS goal kind. A TLS listener port is used to symbolize every dealer within the MSK cluster. On this put up, there are three brokers within the MSK cluster beginning with port 9001, representing dealer 1 and as much as port 9003, representing dealer 3.

Area Identify System (DNS)

For the shopper to resolve DNS queries for the customized area, we use an Amazon Route 53 non-public hosted zone to host the DNS data, and affiliate it with the shopper’s VPC to allow DNS decision from the Route 53 VPC resolver. This resolution makes use of a non-public MSK cluster and personal DNS. For publicly accessible MSK clusters a public NLB and DNS supplier resembling a Route53 public hosted zone can be utilized.

Amazon MSK

Lastly, every dealer must have its marketed listeners configuration (marketed.listeners) up to date to match the customized area title and NLB ports. Marketed listeners is a configuration possibility utilized by Kafka shoppers to hook up with the brokers. By default, an marketed listener is just not set. As soon as set, Kafka shoppers use the marketed listener as a substitute of listeners to acquire the connection data for brokers. MSK brokers use the listener configuration to inform shoppers the DNS names and ports to make use of to hook up with the person brokers for every authentication kind enabled. Marketed listeners are distinctive to every dealer; and the cluster gained’t begin if a number of brokers have the identical marketed listener handle. For that reason, this resolution makes use of a singular customized DNS title for every dealer (resembling, b-1.instance.com).

Resolution Deployment

To deploy the answer, use the CloudFormation template from the GitHub repository.

This template deploys a VPC, NLB, PCA, ACM certificates, MSK cluster, and an Amazon EC2 occasion for cluster connectivity. The EC2 occasion features a script to deal with updating the dealer marketed.listeners settings to match the customized area title. For extra data on deploying a CloudFormation template, discuss with Create a stack from the CloudFormation console.

After deploying the CloudFormation template, run the script to replace marketed listeners as follows:

1. Retrieve the MSKClusterARN and CertificateAuthorityARN from the CloudFormation outputs on your stack as they are going to be utilized in subsequent steps.
   
   
2. Navigate to the EC2 console and establish the KafkaClientInstance. Select Join to hook up with the occasion utilizing AWS Techniques Supervisor Session Supervisor.
3. Session Supervisor begins a session in shell. Begin a bash session with the command:

   

4. The Kafka shopper SDKs have already been put in within the EC2 occasion. You’ll be able to replace the marketed.listeners configuration as follows, changing CLUSTER_ARN with the ARN of your MSK cluster retrieved from CloudFormation in step 1:

   ./update_advertised_listeners.sh --region us-east-1 --cluster-arn CLUSTER_ARN

   Be aware that after this script completes, the brokers may have new marketed listeners configurations. Connections utilizing the usual IAM handle for the MSK service is not going to work till we full the subsequent steps, because the brokers will redirect connections over this handle again to the customized area title and TLS will fail.

5. Subsequent, we have to create a truststore with the certificates for our AWS Non-public Certificates Authority (PCA) to permit TLS with the NLB. Within the following command, exchange PCA_ARN with the ARN of the PCA retrieved from CloudFormation in step 1:
   
   We’re utilizing the default Java truststore which makes use of the password changeit.When requested “Belief this certificates?” enter “sure”.

   export PCA_ARN=>
   export REGION=>
   
   cp /and so forth/pki/java/cacerts . && chmod 600 cacerts
   aws acm-pca get-certificate-authority-certificate --certificate-authority-arn $PCA_ARN --region $REGION | jq -r '.Certificates' > pca.pem
   keytool -import -file pca.pem -alias AWSPCA -keystore cacerts

6. Create a brand new properties file to permit IAM authentication with our customized truststore:

   cat > /dwelling/ssm-user/client-iam.properties
   ssl.truststore.location=/dwelling/ssm-user/cacerts
   ssl.truststore.password=changeit
   EOF

7. Confirm you may hook up with the cluster utilizing IAM authentication utilizing our new customized area title, changing bootstrap.instance.com with your individual customized area title for those who used a distinct one in CloudFormation:

   bin/kafka-topics.sh --list --command-config client-iam.properties --bootstrap-server bootstrap.instance.com:9000

   

Cleanup

To cease incurring prices navigate to CloudFormation and delete the CloudFormation stack to take away all sources provisioned by CloudFormation.

Regularly Requested Query about Customized Area Identify

Clients have requested a number of questions on implementing customized domains with MSK. You could find solutions to a number of the hottest questions right here.

Are there any limitations for this resolution on MSK?

The marketed.listeners setting was eliminated as a dynamic dealer in KRaft-based Kafka clusters. Subsequently, this resolution is just supported in Zookeeper-based MSK clusters. Moreover, this resolution is just relevant to SASL/SCRAM and IAM-authentication primarily based MSK clusters.

How the customized area title resolution scales after we add new brokers?

When utilizing the NLB for dealer connectivity (possibility 2 within the configure a customized area title on your Amazon MSK cluster weblog put up), you will want so as to add an extra listener for every further dealer created.

For TLS, if utilizing Topic Various Identify (SAN) to record particular person dealer DNS hostnames, you will want to create a brand new certificates that features the names of the extra brokers. One possibility is to create a certificates with SANs for extra brokers than wanted to permit for progress.If a wildcard certificates is used, you don’t want to change certificates when including brokers.

What modifications are required after we take away brokers?

Amazon MSK helps scale-in by eradicating brokers from the cluster. Brokers are faraway from every availability zones (AZ). So a 6 dealer Amazon MSK cluster deployed in 3 AZ will be decreased to three dealer cluster deployed in 3 AZ. When brokers are eliminated, you may take away the NLB listeners for the eliminated dealer together with the Route53 DNS endpoints. Nonetheless, you may as well depart them as is, or simply take away the goal IP from the dealer numbers goal group. The NLB will mark the targets as unhealthy and cease directing site visitors to them. When you ever plan to scale-out the variety of brokers, you may re-use the present NLB listeners and Route 53 DNS entries and would solely have to replace the goal IPs used within the dealer numbers goal group.

Is there any change in configuration required if there may be any dealer failure?

No. When a dealer fails, Amazon MSK replaces the failed dealer with a brand new dealer occasion protecting the configuration of the dealer precisely the identical. So, there could be no change within the marketed listener of the dealer. As soon as the dealer is wholesome, the dealer can settle for new connections and skim/write site visitors.

Can you employ Amazon MSK Replicator between MSK clusters in a number of AWS Areas when utilizing the customized area title resolution?

The Amazon MSK Replicator can be utilized when utilizing the customized area title resolution, both in an active-passive or active-active setup. The identical course of will be adopted to set the customized area title.

You then comply with construct multi-Area resilient Apache Kafka purposes with equivalent subject names utilizing Amazon MSK and Amazon MSK Replicator put up to configure MSK Replicator.

The next diagram reveals an active-active AWS multi-Area MSK setup utilizing the customized area title resolution:

Can I exploit a worldwide bootstrap DNS title to hook up with Amazon MSK clusters deployed throughout a number of AWS areas when IAM authentication is enabled?

No, it’s not attainable to make use of a worldwide bootstrap reference to symbolize MSK clusters deployed in a number of AWS Areas, until the shopper is conscious of the cluster’s area when connecting. To make use of IAM authentication, the right AWS Area should be included within the IAM authentication request for a given cluster. It is because the AWS Area is part of the Sigv4 authentication protocol utilized by IAM. This scope prevents the IAM authorization getting used to speak to a useful resource in one other AWS Area. You’ll be able to present the AWS Area in one in every of two methods– with region-specific bootstrap URLs or by explicitly configuring the area.

For instance, if the bootstrap string is bootstrap.us-east-1.instance.com, then msk-iam-auth library will to extract the AWS Area from the dealer connection string and use us-east-1 in its IAM requests. If the bootstrap string is solely bootstrap.instance.com, then the shopper should explicitly configure AWS_REGION=us-east-1 to hook up with the cluster whether it is in us-east-1, or us-west-2 whether it is in us-west-2.

Be aware that this can be a limitation for IAM authentication, however not for SASL/SCRAM authentication. With SASL/SCRAM authentication, if the shopper’s credentials are utilized to each clusters the worldwide endpoint can level to both cluster and the shopper will have the ability to join. The AWS Area is just not utilized in SASL/SCRAM authentication, so it doesn’t prohibit the authentication scope.

The way to enable public entry to a non-public MSK cluster utilizing the customized area title resolution?

To supply public entry to a MSK cluster utilizing the customized area resolution, you will want to do the next:

• Create an Web-facing NLB, and affiliate public subnets (subnets which have a path to the Web Gateway connected to the VPC).
• Create ingress guidelines in each the NLB and MSK safety teams allowing the required public addresses. Be aware: the port can be 9098 for the MSK safety group, and the ports you’re utilizing on the NLB listeners.
• Present public DNS decision for the Kafka shoppers, by utilizing a Route 53 public zone, or another public DNS resolver.
• The shopper wants have IAM credentials, with permission, to speak to the MSK brokers, utilizing an IAM position, IAM entry keys, IAM Roles Wherever, or one other mechanism that makes use of the AWS Safety Token Service (AWS STS) to create and supply trusted customers with non permanent safety credentials.

Within the first a part of the weblog, two patterns have been highlighted. The way to determine which sample to make use of and why?

Choice 1: Solely bootstrap connection by means of NLB

If the Kafka shoppers have direct entry to the dealer, then you need to use customized area title for the bootstrap connection whereas the shoppers can nonetheless hook up with the MSK Brokers with dealer DNS. That is the best possibility, because it doesn’t require customized TLS certificates or TLS listeners.Be aware that this feature is just not obligatory when utilizing MSK Specific brokers, as MSK Specific brokers already manages bootstrapping through a broker-agnostic connection string. For MSK Specific, this feature doesn’t add worth apart from configuring a customized area title for appearances / simplicity of shopper configuration. For MSK Commonplace brokers, this will enhance shopper connectivity by making connection strings dealer agnostic.

Choice 2: All connections by means of NLB

When Kafka shoppers don’t have direct entry to Amazon MSK Brokers, routing all connections by means of the NLB will be most popular. This could happen when a shopper is deployed in a distinct VPC than Amazon MSK VPC or the shopper is exterior, and when Amazon MSK Multi VPC Connectivity is just not an possibility. On the whole, Amazon MSK Multi VPC Connectivity is most popular as this can be a easier sample for many organizations to handle MSK Connectivity throughout accounts and VPCs.When Multi VPC Connectivity is just not an possibility, NLB can be utilized to supply connectivity with Transit Gateway or PrivateLink, and the answer talked about within the weblog needs to be used.

Right here is an instance structure how Kafka shopper and Amazon MSK cluster deployed in two separate VPCs however related through AWS Non-public Hyperlink.

Is Amazon Route 53 required to make use of a customized area title with Amazon MSK?

You should utilize another DNS resolver service, and don’t require Amazon Route 53 to make use of a customized area title with Amazon MSK. The one requirement is that your shoppers can resolve towards your DNS resolver service. The one change required, is to make use of a CNAME for the DNS data, referencing the NLBs DNS report, rather than the Alias data, as that is report kind is just out there in Amazon Route 53.

We don’t use Amazon Certificates Supervisor (ACM), can NLB combine with different third occasion certificates managers?

NLB solely helps ACM to bind a certificates to a TLS listener. You’ll be able to import a certificates created utilizing your third occasion certificates supervisor into ACM, and don’t have to create a certificates utilizing ACM.

Getting connection to node terminated throughout authentication after setting marketed.listeners , what might be the problem?

As the problem began to happen after altering the marketed.listeners configuration, the problem is unlikely to be associated to permissions. The next may cause this situation:

• The NLB and/or shopper’s Safety Group doesn’t allow entry to the listener ports on the NLB from the shopper.
• A firewall equipment between the NLB and shopper doesn’t allow the shopper to speak to the NLB utilizing the listener ports.
• The marketed.listeners configuration has an error inflicting the shopper to obtain invalid particulars, resembling a typo within the title. If so, use a shopper in the identical VPC because the MSK dealer that has IAM permissions to speak to the MSK dealer, and Safety Group guidelines allowing connectivity, you then use the next command to delete the marketed.listeners configuration.

/dwelling/ec2-user/kafka/bin/kafka-configs.sh --alter 
         --bootstrap-server  
         --entity-type brokers 
         --entity-name  
         --command-config ~/kafka/config/client_iam.properties 
         --delete-config marketed.listeners

BROKERS_AMAZON_DNS_NAME resembling b-1.clustername.xxxxxx.yy.kafka.area.amazonaws.com:9098.

Getting “surprising dealer id, anticipated 2 or empty string, however acquired 1”, what’s inflicting this error?

This error is usually offered when the marketed.listeners configuration for one of many brokers has the port utilized by one other dealer set. For instance dealer 2 has port 9001 set for IAM, however this port is used to hook up with dealer 1, so dealer 1 is responding with an error to say you offered dealer id 2, however I’m dealer 1.

To appropriate this, you will want to replace the dealer with the inaccurate marketed.listeners configuration to make use of the right port. To realize entry to the dealer to make the change, you will want to make use of the next command to delete the inaccurate configuration:

/dwelling/ec2-user/kafka/bin/kafka-configs.sh --alter 
         --bootstrap-server 
         --entity-type brokers 
         --entity-name  
         --command-config ~/kafka/config/client_iam.properties 
         --delete-config marketed.listeners

BROKERS_AMAZON_DNS_NAME resembling b-2.clustername.xxxxxx.yy.kafka.area.amazonaws.com:9098.

You then want to make use of the next command to set the marketed.listeners configuration for that dealer:

Be aware: The marketed.listeners configuration within the under assumes solely IAM is used for authentication. In case you are utilizing further authentication choices, you will want to incorporate them.

MSKDOMAIN=
broker_id=
Area=

/dwelling/ec2-user/kafka/bin/kafka-configs.sh --alter 
         --bootstrap-server  
         --entity-type brokers 
         --entity-name "$broker_id" 
         --command-config ~/kafka/config/client_iam.properties 
         --add-config "marketed.listeners=[CLIENT_IAM://b-$broker_id.$Domain:900$broker_id,REPLICATION://b-$broker_id-internal.$MSKDOMAIN:9093,REPLICATION_SECURE://b-$broker_id-internal.$MSKDOMAIN:9095]"

Abstract

On this put up, we defined how you need to use an NLB, Route 53, and the marketed listener configuration possibility in Amazon MSK to help customized domains with MSK clusters when utilizing IAM authentication. You should utilize this resolution to maintain your current Kafka bootstrap DNS title and cut back or take away the necessity to change shopper purposes due to a migration, restoration course of, or to make use of a DNS title in keeping with your group’s naming conference (for instance, msk.prod.instance.com).

Strive the answer out for your self, and depart your questions and suggestions within the feedback part.

Concerning the authors