Migrate to Apache Flink 2.2 on Amazon Managed Service for Apache Flink

Migrating to Apache Flink 2.2 on Amazon Managed Service for Apache Flink provides you entry to Java 17 runtime, quicker checkpoints and restoration by RocksDB 8.10.0, and SQL-native synthetic intelligence and machine studying (AI/ML) inference. In the event you run Flink 1.x right this moment, you could be coping with an growing old Java 11 runtime that can not obtain customary assist by the top of this 12 months, slower state backend efficiency, and a fragmented API floor cut up throughout DataSet, DataStream, and legacy connector interfaces. Flink 2.2 addresses these gaps in a single main model improve.

Apache Flink is an open supply distributed processing engine for stream and batch knowledge, with first-class assist for stateful processing and event-time semantics. Amazon Managed Service for Apache Flink removes the operational overhead of operating Flink. You present your software code, and the service provisions, scales, checkpoints, and patches the infrastructure for you.

On this submit, we clarify what’s new in Amazon Managed Service for Apache Flink 2.2, present a guided migration utilizing CLI instructions, console directions, and code examples, and present you methods to monitor the improve and roll again if wanted.

Earlier than you improve: Flink 2.2 removes the DataSet API, drops Java 11 assist, and replaces legacy connector interfaces. We advocate reviewing the Upgrading to Flink 2.2: Full Information and the State Compatibility Information for Flink 2.2 Improves earlier than upgrading manufacturing purposes.

What’s new in Amazon Managed Service for Apache Flink 2.2

This launch spans runtime upgrades, SQL, and Desk API capabilities. The next sections break down every space.

Runtime and efficiency

These modifications enhance software efficiency and convey your runtime as much as present requirements.

• Java 17 runtime – Flink 2.2 requires Java 17. Construct your software code with JDK 17 for higher rubbish assortment, a safer runtime, and fashionable language options like sealed lessons and data. Java 11 is not supported.
• Python 3.12 – Flink 2.2 requires Python 3.9+, with Python 3.12 because the default. Python 3.8 is not supported.
• RocksDB 8.10.0 – Your stateful purposes profit from improved I/O efficiency with the upgraded state backend, leading to quicker checkpoints and restoration.
• Devoted assortment serializers – Improved serializers for Map, Record, and Set varieties scale back serialization overhead, which lowers checkpoint sizes for purposes that use these knowledge constructions incessantly.
• Kryo 5.6 – Kryo upgrades from model 2.24–5.6. This has state compatibility implications coated within the migration part.

SQL and Desk API highlights

With Flink 2.2, you may:

For particulars on these options, see the Apache Flink 2.2 launch documentation.

Migrating from Flink 1.x to 2.2

In-place model upgrades

You possibly can improve a operating Flink 1.x software to 2.2 utilizing the UpdateApplication API, the AWS Administration Console, AWS CloudFormation, the AWS SDK, and Terraform Modules. The improve preserves your software configuration, logs, metrics, tags, and, in case your state and binaries are appropriate.

Auto-rollback

With auto-rollback turned on, binary incompatibilities detected throughout job startup set off an computerized revert to the earlier Flink model inside minutes, with no guide intervention required. For state incompatibilities that floor as restart loops after a profitable improve, invoke the Rollback API to return to your earlier model and state.

Unsupported open supply options

The next Flink 2.2 options aren’t presently supported in Amazon Managed Service for Apache Flink as a result of they’re nonetheless thought of experimental: Materialized Tables, ForSt State Backend (disaggregated state storage), Java 21, and customized metric reporters/telemetry configurations. We proceed to guage these options as they mature within the Apache Flink undertaking and can share updates on availability. You possibly can have a better look to which options are supported in Apache Flink 2.2 options supported

Now that you understand what’s modified, the subsequent part walks by the migration course of.

Conditions

Earlier than beginning the migration, affirm that you’ve got the next in place:

• An current Apache Flink 1.x software operating on Amazon Managed Service for Apache Flink.
• JDK 17 put in in your native construct surroundings.
• The AWS Command Line Interface (AWS CLI) put in and configured with permissions to name the kinesisanalyticsv2 APIs (UpdateApplication, CreateApplicationSnapshot, DescribeApplication, RollbackApplication).
• An Amazon Easy Storage Service (Amazon S3) bucket to add your up to date software JAR.

We advocate testing every section on a non-production reproduction of your software earlier than making use of the identical steps to manufacturing.

Step 1: Replace your software code

Begin by updating your Flink dependencies to model 2.2.0 and changing deprecated APIs. The next sections present the most typical modifications.

Replace your pom.xml:

Substitute legacy Kinesis connectors:

Flink 2.2 removes the FlinkKinesisConsumer and FlinkKinesisProducer lessons. The next instance exhibits methods to migrate to the FLIP-27 primarily based KinesisStreamsSource.Earlier than (Flink 1.x):

FlinkKinesisConsumer client = new FlinkKinesisConsumer(
    "my-stream",
    new SimpleStringSchema(),
    consumerConfig);
env.addSource(client);

After (Flink 2.2):

KinesisStreamsSource supply = KinesisStreamsSource.builder()
    .setStreamArn("arn:aws:kinesis:us-east-1:123456789012:stream/my-stream")
    .setDeserializationSchema(new SimpleStringSchema())
    .construct();
env.fromSource(supply, WatermarkStrategy.noWatermarks(), "Kinesis Supply");

Replace connector dependencies:

The next AWS connectors have Flink 2.x-compatible releases:

Throughout writing, the JDBC, OpenSearch, and Prometheus connectors don’t but have Flink 2.x-compatible releases. For the newest variations, see the Amazon Managed Service for Apache Flink connector documentation.

Past connector updates, make the next code modifications:

• Substitute DataSet API utilization with the DataStream API or Desk API/SQL.
• Substitute Scala API utilization with the Java API.
• Confirm that your construct targets JDK 17.

Construct your up to date software JAR and add it to Amazon S3 with a special file identify than your present JAR (for instance, my-app-flink-2.2.jar).

Step 2: Test state compatibility

Earlier than upgrading, assess whether or not your software state is appropriate with Flink 2.2. The Kryo improve from model 2.24 to five.6 modifications the binary format of serialized state. Functions utilizing POJOs with Java collections (HashMap, ArrayList, HashSet) are the most typical supply of incompatibility.

Fast compatibility examine:

Test your logs for Kryo fallback:

Search your software logs for this sample, which signifies a sort is falling again to Kryo serialization:Class class can't be used as a POJO kind

Step 3: Activate auto-rollback and computerized snapshots

Activate auto-rollback so the service mechanically reverts to the earlier model if the improve fails. Additionally, confirm that computerized snapshots are turned on. The service takes a snapshot earlier than the improve that serves as your rollback level.

Test present settings:

aws kinesisanalyticsv2 describe-application 
    --application-name MyApplication 
    --query 'ApplicationDetail.ApplicationConfigurationDescription.{
        AutoRollback: ApplicationSystemRollbackConfigurationDescription.RollbackEnabled,
        AutoSnapshots: ApplicationSnapshotConfigurationDescription.SnapshotsEnabled
    }'

Activate each in the event that they’re not already energetic:

aws kinesisanalyticsv2 update-application 
    --application-name MyApplication 
    --current-application-version-id  
    --application-configuration-update '{
        "ApplicationSystemRollbackConfigurationUpdate": {
            "RollbackEnabledUpdate": true
        },
        "ApplicationSnapshotConfigurationUpdate": {
            "SnapshotsEnabledUpdate": true
        }
    }'

Step 4: Take a guide snapshot (beneficial)

Though the improve course of takes an computerized snapshot, taking a guide snapshot provides you a named restore level that you would be able to rapidly determine.

aws kinesisanalyticsv2 create-application-snapshot 
    --application-name MyApplication 
    --snapshot-name pre-flink-2.2-upgrade

Confirm that the snapshot is prepared earlier than continuing:

aws kinesisanalyticsv2 describe-application-snapshot 
    --application-name MyApplication 
    --snapshot-name pre-flink-2.2-upgrade

Wait till SnapshotStatus is READY.

Step 5: Run the improve

Run the improve whereas the applying is in RUNNING or READY (stopped) state. The next instance upgrades a operating software and factors to the brand new JAR.

AWS CLI:

aws kinesisanalyticsv2 update-application 
    --application-name MyApplication 
    --current-application-version-id  
    --runtime-environment-update FLINK-2_2 
    --application-configuration-update '{
        "ApplicationCodeConfigurationUpdate": {
            "CodeContentUpdate": {
                "S3ContentLocationUpdate": {
                    "FileKeyUpdate": "my-app-flink-2.2.jar"
                }
            }
        }
    }'

AWS Administration Console:

To improve from the console, comply with these steps:

1. Navigate to your software within the Amazon Managed Service for Apache Flink console.
2. Select Configure.
3. Choose the Flink 2.2 runtime.
4. Level to your new software JAR on Amazon S3.
5. Choose the snapshot to revive from (use Newest to begin from the latest snapshot).
6. Select Replace.

AWS CloudFormation:

Replace the RuntimeEnvironment area in your template. AWS CloudFormation now performs an in-place replace as an alternative of deleting and recreating the applying.

Terraform:

In the event you handle your Flink software with Terraform, you may carry out the identical in-place improve by updating the runtime_environment and code reference in your aws_kinesisanalyticsv2_application useful resource. Notice: Terraform assist for FLINK-2_2 requires AWS supplier model 6.40.0 or later (launched April 8, 2026). Earlier supplier variations don’t acknowledge this runtime worth. First, replace your supplier model constraint:

terraform {
  required_providers {
    aws = {
      supply  = "hashicorp/aws"
      model = ">= 6.40.0"
    }
  }
}

Then run terraform init -upgrade to drag the brand new supplier.Subsequent, replace your software useful resource. Change runtime_environment from “FLINK-1_20” to “FLINK-2_2” and level to your new JAR:

useful resource "aws_kinesisanalyticsv2_application" "my_app" {
  identify                   = "MyApplication"
  runtime_environment    = "FLINK-2_2"
  service_execution_role = aws_iam_role.flink.arn
  application_configuration {
    application_code_configuration {
      code_content_type = "ZIPFILE"
      code_content {
        s3_content_location {
          bucket_arn = aws_s3_bucket.app_code.arn
          file_key   = "my-app-flink-2.2.jar"
        }
      }
    }
    application_snapshot_configuration {
      snapshots_enabled = true
    }
    flink_application_configuration {
      checkpoint_configuration {
        configuration_type = "DEFAULT"
      }
      monitoring_configuration {
        configuration_type = "CUSTOM"
        log_level          = "INFO"
        metrics_level      = "APPLICATION"
      }
      parallelism_configuration {
        auto_scaling_enabled = true
        configuration_type   = "CUSTOM"
        parallelism          = 4
        parallelism_per_kpu  = 1
      }
    }
  }
}

Run the improve:

terraform plan    # Evaluate the in-place replace
terraform apply   # Apply the runtime change

Terraform will carry out an in-place replace of the applying, altering the runtime model and code location. The applying will restart with the brand new Flink 2.2 runtime. To roll again with Terraform, revert runtime_environment to “FLINK-1_20”, level file_key again to your unique JAR, and run terraform apply once more. Notice that you just can not restore a Flink 2.2 snapshot on Flink 1.x, so the rollback will begin from the final Flink 1.x snapshot.

Essential Terraform concerns:

• Auto-rollback and the RollbackApplication API aren’t immediately uncovered as Terraform useful resource attributes. In the event you want auto-rollback throughout the improve, allow it utilizing the AWS CLI (Step 3) earlier than operating terraform apply, or use a provisioner/null_resource to name the CLI.
• At all times take a guide snapshot (Step 4) earlier than operating terraform apply for the improve. Terraform doesn’t mechanically snapshot earlier than updating the runtime.

Step 6: Monitor the improve

After initiating the improve, monitor the applying to confirm that it completes efficiently.

Test software standing:

The applying ought to transition by RUNNING → UPDATING → RUNNING. Affirm the runtime model modified to 2.2:

aws kinesisanalyticsv2 describe-application 
    --application-name MyApplication 
    --query 'ApplicationDetail.RuntimeEnvironment'

What to look at for:

Key CloudWatch metrics to watch:

1. numRestarts: ought to be zero after improve
2. lastCheckpointDuration: ought to be just like pre-upgrade values
3. numberOfFailedCheckpoints: ought to stay at zero
4. uptime: ought to be steadily rising

Step 7: Validate software habits

After the applying is operating on Flink 2.2:

• Affirm that knowledge is being learn from sources and written to sinks.
• Examine the output along with your pre-upgrade baseline.
• Monitor latency, throughput, checkpoint period, and useful resource utilization.
• Run for at the very least 24 hours to verify secure habits: no reminiscence leaks, no surprising restarts, constant checkpoint sizes.

Step 8: Rollback (if wanted)

If the applying is operating however is unhealthy after the improve, invoke the Rollback API:

AWS CLI:

aws kinesisanalyticsv2 rollback-application 
    --application-name MyApplication 
    --current-application-version-id 

AWS Administration Console:

• Navigate to your software.
• Select Actions, Roll again.
• Affirm the rollback.

Throughout rollback, the applying stops, reverts to the earlier Flink model and software code, and restarts from the snapshot taken earlier than the improve.

Essential: You possibly can’t restore a Flink 2.2 snapshot on Flink 1.x. Rollback makes use of the snapshot taken earlier than the improve. That is why Steps 3 and 4 are important.

Subsequent steps

Your path relies on the place you might be right this moment:

1. In the event you’re new to Apache Flink: Begin with the information to choosing the proper API and language, the Amazon Managed Service for Apache Flink getting began information, and the Amazon Managed Service for Apache Flink workshop.
2. In the event you’re operating Flink 1.x in manufacturing: Observe the migration steps on this submit on a non-production reproduction first, then apply to manufacturing. For the whole reference, see the Upgrading to Flink 2.2: Full Information and the State Compatibility Information for Flink 2.2 Upgrades.
3. In the event you’re evaluating Flink 2.2 options: Launch a brand new software on the Flink 2.2 runtime to discover SQL/ML capabilities, the VARIANT knowledge kind, and the brand new be a part of operators. See the Amazon Managed Service for Apache Flink pattern purposes on GitHub for reference architectures.
4. In the event you need assistance along with your migration: Use the Kiro Energy and Agent Talent for Amazon Managed Service for Apache Flink to determine compatibility points in your current codebase and obtain steering on refactoring steps. You too can open a case by AWS Help, submit a query on AWS re:Put up for Amazon Managed Service for Apache Flink, or attain out by the Apache Flink neighborhood.

For the Apache Flink 2.2 documentation, see nightlies.apache.org/flink/flink-docs-release-2.2. For Amazon Managed Service for Apache Flink documentation, see the Developer Information. For pricing, see the pricing web page.

Conclusion

With Apache Flink 2.2 on Amazon Managed Service for Apache Flink, you get a contemporary Java 17 runtime, SQL-native AI/ML inference, improved state administration efficiency, and a streamlined API floor. In-place upgrades with state preservation and auto-rollback make the migration easy. Check on a duplicate, comply with the steps on this submit, and begin constructing on Flink 2.2.

In regards to the authors