Introducing Apache Airflow 3 on Amazon MWAA: New options and capabilities

October 5, 2025

88

At present, Amazon Net Companies (AWS) introduced the overall availability of Apache Airflow 3 on Amazon Managed Workflows for Apache Airflow (Amazon MWAA). This launch transforms how organizations use Apache Airflow to orchestrate information pipelines and enterprise processes within the cloud, bringing enhanced safety, improved efficiency, and trendy workflow orchestration capabilities to Amazon MWAA prospects.

Amazon MWAA introduces Airflow 3 options that modernize workflow administration for AWS prospects. Following the April 2025 launch of Airflow 3 by the Apache group, AWS has integrated these capabilities into Amazon MWAA. Airflow now includes a fully redesigned, intuitive UI that simplifies workflow orchestration for customers throughout expertise ranges. With the Job Execution Interface (Job API), duties can run each inside Airflow and as standalone Python scripts, enhancing code portability and testing. Scheduler-managed Backfill strikes operations from the CLI to the scheduler, offering centralized management and visibility by means of the Airflow UI. CLI safety enhancements substitute direct database entry with API calls, sustaining constant safety throughout interfaces. Airflow now helps event-driven workflows, enabling triggers from AWS companies and exterior sources. Amazon MWAA additionally provides help for Python 3.12, bringing the newest language capabilities to workflow improvement.

This publish explores the options of Airflow 3 on Amazon MWAA and descriptions enhancements that enhance your workflow orchestration capabilities. The service maintains the Amazon MWAA pay-as-you-go pricing mannequin with no upfront commitments. You’ll be able to start instantly by visiting the Amazon MWAA console, launching new Apache Airflow environments by means of the AWS Administration Console, AWS Command Line Interface (AWS CLI), AWS CloudFormation, or AWS SDK inside minutes.

Architectural developments in Airflow 3 on Amazon MWAA

Airflow 3 on Amazon MWAA introduces important architectural enhancements that improve safety, efficiency, and suppleness. These developments create a extra sturdy basis for workflow orchestration whereas sustaining backward compatibility with present workflows.

Enhanced safety

Amazon MWAA with Airflow 3 modifications the safety mannequin by making part isolation a normal observe moderately than non-compulsory. In Airflow 2, the DAG processor (the part that parses and processes DAG information) runs inside the scheduler course of by default, however can optionally be separated into its personal course of for higher scalability and safety isolation. Airflow 3 makes this separation normal, sustaining constant safety practices throughout deployments.

API server and Job API

Constructing on this safety basis, a brand new API server part is launched in Amazon MWAA with Airflow 3, which serves as an middleman between activity cases and the Airflow metadata database. This transformation improves your workflows’ safety posture by minimizing direct entry to the Airflow metadata database from duties. Duties now function with least privilege database entry, lowering the danger of 1 activity affecting others and enhancing total system stability by means of fewer direct database connections.

The standardized communication by means of well-defined API endpoints creates a basis for safer, scalable, and versatile workflow orchestration. The Job Execution Interface (Job API) helps duties run each inside Airflow and as standalone Python scripts, enhancing code portability and testing capabilities.

From data-aware to event-driven scheduling

Airflow’s evolution towards event-driven scheduling started with the introduction of data-aware scheduling in Airflow 2.4, so DAGs might be triggered based mostly on information availability moderately than time schedules alone. Amazon MWAA with Airflow 3 builds on this basis by means of a transition that features the renaming of datasets to belongings and introduces superior capabilities, together with asset partitions, exterior occasion integration, and asset-centric workflow design.

The transition from datasets to belongings represents greater than a easy rename. An information asset is a group of logically associated information that may characterize numerous information merchandise, together with database tables, continued ML fashions, embedded dashboards, or directories containing information.

Amazon MWAA with Airflow 3 introduces a brand new asset-centric syntax that represents an necessary shift in how workflows might be designed. The @asset decorator helps builders put information belongings on the middle of their workflow design, creating extra intuitive asset-driven pipelines.

The next code is an instance of asset-aware DAG scheduling:

from airflow.sdk import DAG, Asset
from airflow.suppliers.normal.operators.python import PythonOperator

# Outline the asset
customer_data_asset = Asset(title="customer_data", uri="s3://my-bucket/customer-data.csv")

def process_customer_data():
    """Course of buyer information..."""
    # Implementation right here

# Create the DAG and activity
with DAG(dag_id="process_customer_data", schedule="@every day"):
    PythonOperator(
        task_id="process_data", 
        shops=[customer_data_asset], 
        python_callable=process_customer_data
    )

The next code exhibits an asset-centric method with the @asset decorator:

from airflow.sdk import asset

@asset(uri="s3://my-bucket/customer-data.csv", schedule="@every day")
def customer_data():
    """Course of buyer information..."""
    # Implementation right here

The @asset decorator mechanically creates an asset with the operate title, a DAG with the identical identifier, and a activity that produces the asset. This reduces code complexity and facilitates automated DAG creation, the place every asset turns into a self-contained workflow unit.

Exterior event-driven scheduling with Asset Watchers

A major development in Amazon MWAA with Airflow 3 is the introduction of Asset Watchers, which assist Airflow react to occasions occurring outdoors of the Airflow system itself. Whereas earlier variations supported inside cross-DAG dependencies, Asset Watchers lengthen this functionality to exterior information techniques and message queues by means of the AssetWatcher class.

Amazon MWAA with Airflow 3 contains help for Amazon Easy Queue Service (Amazon SQS) by means of Asset Watchers. This enables your workflows to be triggered by exterior messages and facilitates extra event-driven scheduling. Airflow now helps event-driven workflows, enabling triggers from AWS companies and exterior sources. Asset Watchers monitor exterior techniques asynchronously and set off workflow execution when particular occasions happen, enabling workflows to reply to enterprise occasions, information updates, or system notifications with out the overhead of conventional sensor-based polling mechanisms.

Trendy React-based UI

Amazon MWAA with Airflow 3 includes a fully redesigned, intuitive UI constructed with React and FastAPI that simplifies workflow orchestration for customers throughout expertise ranges. The brand new interface offers extra intuitive navigation and workflow visualization, with an enhanced grid view that gives higher visibility into activity standing and historical past. Customers will recognize the addition of darkish mode help, which reduces eye pressure throughout prolonged use, and the general sooner efficiency that’s particularly noticeable when working with giant DAGs.

The brand new UI maintains acquainted workflows whereas offering a extra trendy and environment friendly expertise for DAG administration and monitoring, making every day operations extra productive for each builders and operators. The legacy UI has been fully eliminated, providing a cleaner, extra constant expertise throughout the system. The muse for the brand new UI is constructed on REST APIs and a set of inside APIs for UI operations, each of which at the moment are based mostly on FastAPI, making a extra cohesive and safe structure for each programmatic entry and UI operations.

Scheduler optimizations

Amazon MWAA with Airflow 3’s enhanced scheduler delivers efficiency enhancements for activity execution and workflow administration. The redesigned scheduling engine processes duties extra effectively, lowering the time between activity submissions and executions. This optimization advantages information pipeline operations that require speedy activity processing and well timed workflow completion.

The scheduler now manages computing assets extra successfully, enabling secure efficiency whilst workloads scale. When working a number of DAGs concurrently, the improved useful resource allocation system helps stop bottlenecks and maintains constant execution speeds. This development is especially helpful for organizations working advanced workflows with various useful resource necessities. The brand new scheduler additionally handles concurrent operations with elevated precision, so groups can run a number of DAG cases concurrently whereas sustaining system stability and predictable efficiency.

Enhanced scheduler backfill operations

Scheduler-managed backfill (the method of working DAGs for historic dates) strikes operations from the CLI to the scheduler, offering centralized management and visibility by means of the Airflow UI. Amazon MWAA with Airflow 3 delivers necessary upgrades to the scheduler’s backfill capabilities, serving to information groups course of historic information extra effectively. The backfill course of has been optimized for higher efficiency, lowering the database load throughout these operations and ensuring backfills might be accomplished extra shortly, minimizing the influence on close to real-time workflow execution.

Amazon MWAA with Airflow 3 additionally improves the administration of backfill operations, with the scheduler offering higher isolation between backfill jobs and supporting extra environment friendly processing of historic datasets. Operators now have higher monitoring instruments to trace the progress and standing of their backfill jobs, leading to simpler administration of those crucial information processing duties.

Developer-focused enhancements

Airflow 3 on Amazon MWAA delivers a number of enhancements designed to enhance the developer expertise, from simplified activity definition to raised workflow administration capabilities.

Job SDK

The Job SDK offers a extra intuitive option to outline duties and DAGs:

# Instance utilizing the Job SDK
from airflow.sdk import dag, activity
from datetime import datetime

@dag(
    start_date=datetime(2023, 1, 1),
    schedule="@every day",
    catchup=False
)
def modern_etl_workflow():
    
    @activity
    def extract():
        # Extract information from supply
        return {"information": [1, 2, 3, 4, 5]}
    
    @activity
    def rework(input_data):
        # Remodel the info
        return [x * 10 for x in input_data]
    
    @activity
    def load(transformed_data):
        # Load information to vacation spot
        print(f"Loading information: {transformed_data}")
    
    # Outline the workflow
    extracted_data = extract()
    transformed_data = rework(extracted_data["data"])
    load(transformed_data)

# Instantiate the DAG
etl_dag = modern_etl_workflow()

This method affords extra intuitive information move between duties, higher built-in improvement surroundings (IDE) help with improved kind hinting, and extra simple unit testing of activity logic. The result’s cleaner, extra maintainable code that higher represents the precise information move of your pipelines. Groups adopting this sample typically discover their DAGs change into extra readable and easier to keep up over time, particularly as workflows develop in complexity.

DAG versioning

Amazon MWAA with Airflow 3 contains fundamental DAG versioning capabilities that come by default with Airflow 3. Every time a DAG is modified and deployed, Airflow serializes and shops the DAG definition to protect historical past. This automated model monitoring minimizes the necessity for guide record-keeping and ensures each modification is documented.

By the Airflow UI, groups can entry and evaluate the historical past of their DAGs. This visible illustration exhibits model numbers (v1, v2, v3, and so forth.) and helps groups perceive how their workflows have advanced over time.

The DAG versioning supported in Amazon MWAA offers the aptitude to see totally different DAG variations that have been run within the Airflow UI, providing improved workflow visibility and enhanced collaboration for information engineering groups managing advanced, evolving information pipelines.

Python 3.12 help

Amazon MWAA provides help for Python 3.12, bringing the newest language capabilities to workflow improvement. This improve offers entry to the newest Python language enhancements, efficiency enhancements, and library updates, holding your information pipelines trendy and environment friendly.

Options not at the moment supported in Amazon MWAA

Though we’re launching many of the Airflow 3 options on Amazon MWAA on this launch, some options should not supported presently:

DAG versioning (AIP-63) – Superior versioning options past fundamental model monitoring
Exchange Flask AppBuilder (AIP-79) – Full alternative capabilities
Edge Executor and activity isolations (AIP-69) – Distant execution capabilities
Multi-language help (AIP-72) – Help for languages apart from Python

We plan to help these options in subsequent variations of Airflow on Amazon MWAA.

Conclusion

Airflow 3 on Amazon MWAA delivers enhanced workflow automation capabilities. The architectural enhancements, enhanced safety mannequin, and developer-friendly options present a strong basis for constructing extra dependable and maintainable information pipelines.The introduction of Asset Watchers modifications how workflows can reply to exterior occasions, enabling actually event-driven scheduling. This functionality, mixed with the brand new asset-centric workflow design, makes Airflow 3 a extra highly effective and versatile orchestration service.

The scheduler optimizations ship efficiency enhancements for activity execution and workflow administration, and the improved backfill capabilities make historic information processing extra environment friendly. The DAG versioning system improves workflow stability and collaboration, and Python 3.12 help retains your information pipelines trendy and environment friendly.

Organizations can now benefit from these new options and enhancements in Airflow 3 on Amazon MWAA to boost their workflow orchestration capabilities. To get began, go to the Amazon MWAA product web page.

Concerning the authors

Anurag Srivastava works as a Senior Huge Information Cloud Engineer at Amazon Net Companies (AWS), specializing in Amazon MWAA. He’s captivated with serving to prospects construct scalable information pipelines and workflow automation options on AWS.

Kamen Sharlandjiev is a Sr. Huge Information and ETL Options Architect, Amazon MWAA and AWS Glue ETL knowledgeable. He’s on a mission to make life simpler for patrons who’re dealing with advanced information integration and orchestration challenges. His secret weapon? Totally managed AWS companies that may get the job executed with minimal effort. Observe Kamen on LinkedIn to maintain updated with the newest Amazon MWAA and AWS Glue options and information!

Ankit Sahu brings over 18 years of experience in constructing progressive digital services. His numerous expertise spans product technique, go-to-market execution, and digital transformation initiatives. At the moment, Ankit serves as Senior Product Supervisor at Amazon Net Companies (AWS), the place he leads the Amazon MWAA service.

Mohammad Sabeel works as a Senior Cloud Help Engineer at Amazon Net Companies (AWS), specializing in AWS Analytics companies together with AWS Glue, Amazon MWAA, and Amazon Athena. With over 14 years of IT expertise, he’s captivated with serving to prospects construct scalable information processing pipelines and optimize their analytics options on AWS.

Satya Chikkala is a Options Architect at Amazon Net Companies. Based mostly in Melbourne, Australia, he works carefully with enterprise prospects to speed up their cloud journey. Past work, he’s very captivated with nature and images.

Sriharsh Adari is a Senior Options Architect at Amazon Net Companies (AWS), the place he helps prospects work backward from enterprise outcomes to develop progressive options on AWS. Through the years, he has helped a number of prospects on information system transformations throughout business verticals. His core space of experience embrace expertise technique, information analytics, and information science. In his spare time, he enjoys enjoying sports activities, binge-watching TV exhibits, and enjoying Tabla.