Migrating TLS Purchasers managed by third-party Certificates Authorities from self-managed Apache Kafka to Amazon MSK

Amazon Managed Streaming for Apache Kafka (Amazon MSK) is a completely managed streaming information service that handles Apache Kafka infrastructure and operations, so builders and DevOps managers can run Apache Kafka functions on AWS. Migrating to Amazon MSK requires no software code adjustments as a result of Amazon MSK makes use of absolutely open supply Apache Kafka, permitting present functions and instruments to work seamlessly. Amazon MSK with Specific brokers streamlines Kafka administration by offering as much as 3x extra throughput, 20x sooner scaling, and 180x sooner restoration with nearly limitless storage, delivering resiliency and elasticity for mission-critical workloads.

Amazon MSK helps a number of authentication strategies to safe consumer connections to Kafka clusters. These strategies embrace:

When clients handle their very own Kafka clusters and undertake mTLS, they usually depend on a third-party managed certificates authority (CA) to signal and confirm each consumer and server certificates. This establishes a belief relationship the place the CA acts because the trusted middleman that validates the id of each events within the communication. When clients migrate their workloads to Amazon MSK, they have to make it possible for consumer certificates are signed by a CA that’s acknowledged and trusted by the MSK cluster. Amazon MSK recommends clients to make use of AWS Non-public Certificates Authority to create a non-public CA inside AWS that MSK trusts. The migration path usually requires clients to both:

1. Generate new consumer certificates signed by an AWS Non-public CA that Amazon MSK acknowledges, or
2. Set up a certificates chain the place their present third-party CA is subordinate to or trusted by an AWS-managed CA

On this submit, we offer an method to reuse your present consumer certificates with out reissuing them by AWS Certificates Supervisor (ACM) Non-public Certificates Authority. This resolution permits an accelerated migration path by utilizing your present third-party CA infrastructure. This removes the complexity and operational overhead of certificates re-issuance whereas sustaining the safety posture that you just’ve established along with your present mTLS implementation.

Resolution overview

This method includes 4 key steps to reuse your present consumer certificates when migrating to Amazon MSK:

1. Create an Intermediate Certificates Utilizing Your Third-Occasion CA

First, you generate an intermediate certificates authority (CA) certificates utilizing your present third-party CA infrastructure. This intermediate certificates acts as a bridge between your present certificates administration system and AWS.

2. Import the Intermediate Certificates into AWS Certificates Supervisor as a Non-public CA

Subsequent, you import this intermediate certificates into AWS Certificates Supervisor (ACM) as a Non-public Certificates Authority (PCA). This step establishes the intermediate CA inside the AWS setting, making it recognizable to AWS companies.

3. Combine Amazon MSK with the PCA created out of your Intermediate Certificates

You then configure your Amazon MSK cluster to make use of the ACM Non-public CA that accommodates your imported intermediate certificates. This integration permits Amazon MSK to acknowledge and belief certificates signed by your certificates authority.

4. Set up belief by widespread Certificates Authority

This method works as a result of each the AWS Non-public CA and your present consumer certificates share the identical root of belief—they’re each signed by your third-party CA. When Amazon MSK validates consumer certificates, it may possibly hint the certificates chain again by the intermediate certificates in AWS Non-public CA to your trusted third-party CA, establishing an entire chain of belief with out requiring certificates reissuance.This resolution maintains your present safety structure whereas enabling seamless migration to Amazon MSK, so your purchasers can proceed utilizing their present certificates with out interruption.

Implementation steps

In real-world eventualities, you have already got a certificates authority that has issued certificates in your purchasers. For the aim of this submit, we use a code pattern to create a self-signed certificates authority (utilizing OpenSSL) to exhibit the implementation steps. If you have already got an present certificates authority, you don’t have to create a root CA. You possibly can generate an intermediate CA (Step 2) utilizing your third-party CA and proceed following the steps from the place you import the intermediate CA certificates into AWS ACM as a Non-public Certificates Authority.

Step 1: Create a root Certificates Authority utilizing OpenSSL

Cloning the repository

To clone the repository, full the next steps:

1. Clone the repository utilizing the next command:

git clone https://github.com/aws-samples/msk-third-party-mtls

2. Change to the repository’s root listing:

cd ./msk-third-party-mtls/openssl

3. Run the setup script:

make the script executable first:

chmod +x *.sh
./setup-ca.sh

You’ll be prompted to arrange a password for the personal key and the certificates. Right here is an instance of an output

Step 2: Create an intermediate CA for AWS ACM

1. Within the AWS Non-public CA console, create a subordinate CA.

2. Enter distinguished identify info matching your group, Key algorithm and Create CA.
3. From the Actions menu, choose Set up CA certificates.
4. Obtain the Certificates Signing Request (CSR) file supplied by AWS Non-public CA.

5. Obtain the CSR file to your native listing (“certs”) as “CSR.pem”.

6. Signal the ACM PCA issued CSR along with your Root CA utilizing the supplied ./sign-acm-ca.sh within the code instance.

Be aware: AWS Non-public CA retains the personal key internally. You solely signal their CSR and import the ensuing certificates again to the AWS Non-public CA.

Step 3: Import signed certificates to AWS ACM Non-public CA

1. Return to the AWS ACM console.
2. Choose the CA that you just created and choose Set up CA certificates.

3. Choose Exterior personal CA as CA kind.

Importing the certificates into AWS Certificates Supervisor

Open each recordsdata in a textual content editor:

• acm-subordinate-ca-cert.pem
• acm-ca-chain.pem

Do the next within the Certificates physique subject in AWS ACM:

• Copy the whole content material from the acm-subordinate-ca-cert.pem file and paste it into the textual content field.
• Open the acm-ca-chain.pem file.
• This file accommodates one certificates (The basis CA certificates)
• Do the next within the Certificates chain subject in AWS ACM:
• Copy the basis CA certificates portion and paste it into the textual content field

Vital: The certificates chain shouldn’t embrace the subordinate CA certificates itself—solely the certificates above it within the chain (the basis CA).

• Select Affirm and set up to finish the method.

It is best to see the AWS Non-public CA turns into lively state.

Step 4: Configure your MSK cluster for Mutual TLS authentication

1. Choose your MSK cluster, go to Properties and edit the Safety settings.
2. Choose TLS consumer authentication by AWS Certificates Supervisor (ACM) because the entry management methodology and select the Subordinate CA that you just created earlier. Then select Save adjustments.

Step 5: Take a look at your consumer

Run the certificates era script

Execute the next command, changing with a descriptive identify in your consumer (this will probably be used within the certificates filename):./generate-client-cert.sh

Instance:

./generate-client-cert.sh kafka-admin

Enter distinguished identify info

When prompted, enter the distinguished identify (DN) choices. These ought to match your root CA settings apart from the Widespread Title (CN):

• Nation (C): Match your root CA (for instance, US)
• State (ST): Match your root CA (for instance, State)
• Group (O): Match your root CA (for instance, Anycompany)
• Organizational Unit (OU): Match your root CA (for instance, IT)
• Widespread Title (CN): Use a client-specific identifier (for instance, kafka-admin or consumer)

Confirm certificates recordsdata

After the certificates is generated, confirm that the recordsdata had been created efficiently by working:ls ~/ca/certsIt is best to see recordsdata along with your consumer identify, together with:

• .key (personal key)
• .crt (certificates)
• .p12 (PKCS12 keystore)

Create Kafka consumer properties file

Create a brand new properties file in your Kafka consumer (for instance, kafka-tls-client.properties) primarily based on the supplied kafka-admin-ssl.properties instance file. Replace the file paths to reference your newly generated consumer certificates recordsdata.

Instance configuration:

safety.protocol=SSL
ssl.keystore.location=/path/to/.p12
ssl.keystore.password=your-keystore-password
ssl.key.password=your-key-password #omit for those who didn’t set key password
ssl.keystore.alias=your-private-key-alias

Step 6: Testing the Kafka consumer connection

To check the Kafka consumer connection, do the next.

Set setting variables

First, set the required setting variables in your Kafka set up and MSK cluster:

export KAFKA_HOME=/residence/ec2-user/kafka
export BOOTSTRAP_SERVERS=

Be aware: Substitute along with your precise Amazon MSK cluster bootstrap server endpoints (for instance, b-1.mycluster.abc123.kafka.us-east-1.amazonaws.com:9094,b-2.mycluster.abc123.kafka.us-east-1.amazonaws.com:9094)

Run the Kafka listing subjects command

Execute the next command to confirm that your consumer can efficiently connect with Amazon MSK utilizing mutual TLS authentication:

$KAFKA_HOME/bin/kafka-topics.sh 
  --bootstrap-server $BOOTSTRAP_SERVERS 
  --list 
  --command-config kafka-tls-client.properties

What this check does:

• Connects to your Amazon MSK cluster utilizing the TLS configuration in your properties file
• Authenticates utilizing your consumer certificates
• Lists all out there Kafka subjects

Anticipated consequence: If profitable, it’s best to see a listing of subjects in your Kafka cluster (or an empty listing if no subjects exist but).

If the connection fails, examine:

• Your bootstrap server endpoints are right
• You imported the personal key, and certificates chain to your keystore
• The paths in your properties file level to the proper keystore and truststore recordsdata
• Your consumer certificates was correctly imported
• Your Amazon MSK cluster safety settings permit TLS consumer authentication
• Your Amazon MSK cluster references right PCA ARN in AWS ACM

Troubleshooting

Allow debug mode to confirm certificates handshake

To troubleshoot certificates points and confirm which certificates are concerned within the TLS handshake, allow Java SSL debug mode:

export KAFKA_OPTS="-Djavax.web.debug=ssl:handshake:verbose"
$KAFKA_HOME/bin/kafka-topics.sh 
  --bootstrap-server $BOOTSTRAP_SERVERS 
  --list 
  --command-config kafka-tls-client.properties

What this debug mode exhibits:

• The whole TLS handshake course of
• Which certificates are being introduced by each consumer and server
• The certificates chain validation steps
• Which certificates out of your truststore is getting used for authentication

When that is useful:

• When you’ve a number of certificates in your truststore and have to establish which one is getting used
• When troubleshooting certificates chain validation points
• When verifying that the proper consumer certificates is being introduced throughout authentication
• When diagnosing certificates mismatch or belief points

Studying the debug output:

Search for traces containing:

• ***Certificates chain – Exhibits the certificates being introduced
• Discovered trusted certificates – Signifies which certificates in your truststore matched
• Cert path validation – Exhibits the certificates chain validation course of

To disable debug mode after troubleshooting, merely unset the setting variable:

unset KAFKA_OPTS

Conclusion

This submit presents an answer for migrating TLS purchasers from self-managed Apache Kafka to Amazon MSK whereas reusing present third-party CA-signed certificates. The method removes the necessity for certificates reissuance by as an alternative creating an intermediate CA from the prevailing third-party CA, importing it into AWS Certificates Supervisor as a Non-public CA, and integrating it with Amazon MSK. This maintains the established chain of belief by the widespread certificates authority, enabling seamless migration with out operational disruption whereas preserving the prevailing safety structure and mTLS implementation. To learn extra in regards to the Amazon MSK safety mannequin, see Safety in Amazon MSK.

Concerning the authors