Securely join Kafka purchasers working outdoors AWS to Amazon MSK with IAM Roles Wherever

Kafka purchasers which are working outdoors of AWS (on-premises atmosphere or different clouds) would require an IAM consumer with a long-lived entry key to be offered as a part of their codebase or of their server configuration. From a safety perspective, there may be an extra threat if anybody positive aspects entry to these long-term credentials, as they’d have entry to that AWS account.

On this put up, we show the right way to use AWS IAM Roles Wherever to request short-term AWS safety credentials, utilizing x.509 certificates for consumer functions which permits safe interactions with an Amazon Managed Streaming for Apache Kafka (Amazon MSK) cluster. The answer described on this put up is appropriate with each Amazon MSK Provisioned and Serverless clusters.

Introduction to AWS IAM Roles Wherever

AWS Identification and Entry Administration (IAM) Roles Wherever means that you can acquire short-term safety credentials in IAM for workloads working outdoors of AWS, reminiscent of servers, containers, and functions.

By utilizing IAM Roles Wherever, your workloads can make the most of the identical IAM insurance policies and roles utilized by AWS functions to entry AWS assets. This eliminates the necessity to handle long-term credentials for kafka purchasers working outdoors AWS. By associating a number of roles with a profile and enabling IAM Roles Wherever to imagine these roles, your functions can make use of the consumer certificates issued by your Certificates Authorities (CAs) to securely provoke requests to AWS. Consequently, your functions acquire short-term credentials, granting them entry to the AWS atmosphere.

IAM entry management for Amazon MSK means that you can handle each authentication and authorization in your Amazon MSK cluster at no further value. This eliminates the need of utilizing separate mechanisms for authentication and authorization. We suggest Amazon MSK prospects use IAM Entry Management except they’ve a selected want for utilizing mutual TLS or SASL/SCRAM authN/Z.

Within the following sections, we present you the right way to implement a safe Kafka consumer machine with an in depth step-by-step tutorial utilizing an AWS IAM Roles Wherever to attach with a MSK Cluster.

Resolution overview

The next diagram illustrates the answer structure.

The move of the structure is as follows:

1. The session token question out of your consumer machine is directed to an AWS IAM Roles Wherever endpoint, facilitated by the change of X.509 certificates.
2. IAM Roles Wherever validates the certificates and retrieves a brief session token from STS, which is then returned to the consumer machine.
3. In Amazon MSK Provisioned, the MSK consumer machine connects to the AWS Transit Gateway or AWS Community Load Balancer in your VPC over AWS VPN or AWS Direct Join. For extra data, discuss with Safe connectivity patterns to entry Amazon MSK.
4. In Amazon MSK Serverless, the MSK consumer machine connects to the interface VPC endpoint in your VPC over AWS VPN or AWS Direct Join. For extra data, discuss with Connect with Amazon MSK Serverless out of your on-premises community.
5. In Amazon MSK Serverless , the interface endpoint is a group of a number of elastic community interfaces with a non-public IP tackle inside your account. It serves because the entry level for visitors directed in direction of a MSK Serverless service.

Stipulations

The directions offered on this put up assume that you’re already acquainted with the method of making an MSK serverless cluster and a consumer machine. Moreover, it’s presumed that you’ve efficiently completed the next duties:

1. Create an Amazon MSK serverless cluster or Create an Amazon MSK Provisioned Cluster
2. Create a MSK consumer machine in your on-prem knowledge middle or a VPC from one other AWS account.
3. Set up community connectivity between on premises and the Amazon MSK Serverless Cluster or Set up community connectivity between on premises and the Amazon MSK Provisioned Cluster

Configure AWS IAM Roles Wherever

To allow IAM Roles Wherever in your on-premises Kafka consumer machine, you need to configure two important elements in AWS Roles Wherever: the belief anchor and the profile. The belief anchor establishes the belief relationship between Roles Wherever and your certificates authority. This belief is utilized for authenticating certificates to acquire credentials for an IAM function. Profiles are predefined units of permissions which are utilized as soon as profitable authentication with Roles Wherever has been achieved.

Step 1: Generate a CA

An X.509 certificates performs an necessary function in facilitating communication between the consumer machine and Roles Wherever. You should utilize Public Key Infrastructure (PKI) platform of your alternative to determine a certificates authority (CA).

In case you desire to generate your individual X.509 consumer certificates, you possibly can discuss with the directions outlined in IAM Roles Wherever with an exterior certificates authority to information you thru the method.

For simplicity of this instance, we use an AWS Non-public CA:

Navigate to the AWS Non-public CA console.

Create a Root CA

1. Select Root as CA kind choice and put your group title and group unit title.
2. Select default RSA 2048 key algorithm.
3. Select Create CA button to generate a non-public the CA and set up the certificates.

Create a Subordinate CA

1. Select Subordinate as CA kind choice.
2. Select default RSA 2048 key algorithm.
3. Select Create CA button.
   
   
4. Receive the CSR from the subordinate CA and have it signed by the foundation CA.
   
   

This CA can be used for issuing certificates to IAM Roles Wherever.

For producing a extra secured and auto-renewed AWS personal CA, discuss with Process for making a CA and The best way to construct a CA hierarchy.

Step 2: Configure anchor

1. Go to Roles Wherever console and open the Create a belief anchor web page.
2. Present a reputation in your belief anchor and choose the personal CA that we created in step 1. In case you desire to make use of your individual exterior CA, select the Exterior certificates bundle choice and supply the mandatory certificates bundle.
3. Select create a belief anchor button to complete the method.

Step 3: Create and configure a job that trusts IAM Roles Wherever

Now we create a job that you really want your on-premises Kafka consumer machine to imagine after authenticating to IAM Roles Wherever.

1. The belief coverage of the function ought to include the next:

   {
     "Model": "2012-10-17",
     "Assertion": [
       {
         "Effect": "Allow",
         "Principal": {
           "Service": "rolesanywhere.amazonaws.com"
         },
         "Action": [
           "sts:AssumeRole",
           "sts:SetSourceIdentity",
           "sts:TagSession"
         ],
         "Situation": {
           "StringEquals": {
             "aws:PrincipalTag/x509Subject/CN": "specific-certificate-common-name"
           }
         }
       }
     ]
   }

2. For this demo, create the next coverage and connect it to the function:

   {
       "Model":"2012-10-17",		 	 	 
       "Assertion": [
           {
               "Effect": "Allow",
               "Action": [
                   "kafka-cluster:Connect",
                   "kafka-cluster:AlterCluster",
                   "kafka-cluster:DescribeCluster"
               ],
               "Useful resource": [
                   "arn:aws:kafka:::cluster//"
               ]
           },
           {
               "Impact": "Permit",
               "Motion": [
                   "kafka-cluster:CreateTopic",
                   "kafka-cluster:DescribeTopic",
                   "kafka-cluster:WriteData",
                   "kafka-cluster:ReadData",
                   "kafka-cluster:AlterGroup",
                   "kafka-cluster:DescribeGroup"
               ],
               "Useful resource": [
                   "arn:aws:kafka:::cluster/ /",
                   "arn:aws:kafka:::topic/msk-// ",
                   "arn:aws:kafka:::group///"
               ]
           }
       ]
   }

Step 4: Setup profile

1. Navigate again to the Roles Wherever console.
2. Underneath Profiles, select Create a profile.
3. Enter a reputation for the profile.
4. Choose the function we created in Step 3 and create the Roles Wherever profile.

Step 5: Take a look at the consumer machine

Now that we’ve got efficiently arrange Roles Wherever by making a belief anchor and a profile, the subsequent step is to check the consumer machine’s communication with Roles Wherever. This entails retrieving a session token and establishing communication with the MSK dealer.

1. Request a non-public certificates from the CA we created in Step 1 and export the consumer certificates for use within the consumer machine.
2. Create a .pem file and replica all of the certificates contents into this .pem file(e.g. private_key. pem) and run beneath command to generate a decrypted model of certificates.

   openssl rsa -in private_key.pem -out decrypted_private_key.pem

   

3. Obtain the credential helper and use this signing helper instrument to check and make sure the performance out of your consumer machine. We provide the ARNs of the belief anchor and profile of Roles Wherever, and the function we created in IAM.

   ./aws_signing_helper credential-process 
   --certificate /path/to/certificates.pem 
   --private-key /path/to/decrypted_private_key.pem 
   --trust-anchor-arn  
   --profile-arn  
   --role-arn  
   --region 

   It is best to obtain the session credentials efficiently from IAM Roles Wherever.

   

4. After verifying the profitable setup, proceed to replace or create the ~/.aws/config file. Add the signing helper as a credential_process on this file to allow unattended entry for the on-premises server.
   
   [default]

   credential_process = ./aws_signing_helper credential-process 
   --certificate /path/to/certificates.pem 
   --private-key /path/to/decrypted_private_key.pem 
   --trust-anchor-arn  
   --profile-arn  
   --role-arn 
   --region 

As soon as all steps are finished, you need to have the ability to see the Kafka consumer speaking to the MSK dealer.

./kafka-topics.sh --create 
--bootstrap-server  
--command-config  
--replication-factor  
--partitions  
--topic 

Clear up

To cease incurring prices, it is strongly recommended to manually delete the IAM Position, Profile, Belief Anchor, Insurance policies, requested certificates in ACM and created certificates in AWS Non-public CA.

aws delete-role --role-name 

aws delete-profile --profile-id 

aws delete-trust-anchor --trust-anchor-id 

aws acm delete-certificate --certificate-arn 

aws acm-pca revoke-certificate --certificate-authority-arn  --certificate-serial  --revocation-reason  

aws acm-pca delete-certificate --certificate-authority-arn  --certificate-serial 

Conclusion

On this put up, we confirmed you the right way to make the most of AWS IAM Roles Wherever to generate short-term session tokens for accessing MSK brokers from consumer machines outdoors of AWS. By implementing this method, the safety posture of Kafka purchasers connecting to MSK from outdoors of AWS are enhanced, permitting prospects with stringent safety necessities to confidently undertake MSK.

When you’ve got any questions, you can begin a brand new thread on AWS re:Put up or attain out to AWS Assist.

In regards to the authors