Introducing Amazon MWAA Serverless | AWS Huge Information Weblog

Immediately, AWS introduced Amazon Managed Workflows for Apache Airflow (MWAA) Serverless. It is a new deployment choice for MWAA that eliminates the operational overhead of managing Apache Airflow environments whereas optimizing prices by serverless scaling. This new providing addresses key challenges that knowledge engineers and DevOps groups face when orchestrating workflows: operational scalability, price optimization, and entry administration.

With MWAA Serverless you possibly can focus in your workflow logic fairly than monitoring for provisioned capability. Now you can submit your Airflow workflows for execution on a schedule or on demand, paying just for the precise compute time used throughout every activity’s execution. The service routinely handles all infrastructure scaling in order that your workflows run effectively no matter load.

Past simplified operations, MWAA Serverless introduces an up to date safety mannequin for granular management by AWS Identification and Entry Administration (IAM). Every workflow can now have its personal IAM permissions, working on a VPC of your selecting so you possibly can implement exact safety controls with out creating separate Airflow environments. This method considerably reduces safety administration overhead whereas strengthening your safety posture.

On this publish, we display tips on how to use MWAA Serverless to construct and deploy scalable workflow automation options. We stroll by sensible examples of making and deploying workflows, establishing observability by Amazon CloudWatch, and changing current Apache Airflow DAGs (Directed Acyclic Graphs) to the serverless format. We additionally discover finest practices for managing serverless workflows and present you tips on how to implement monitoring and logging.

How does MWAA Serverless work?

MWAA Serverless processes your workflow definitions and executes them effectively in service-managed Airflow environments, routinely scaling assets based mostly on workflow calls for. MWAA Serverless makes use of the Amazon Elastic Container Service (Amazon ECS) executor to run every particular person activity by itself ECS Fargate container, on both your VPC or a service-managed VPC. These containers then talk again to their assigned Airflow cluster utilizing the Airflow 3 Job API.

Determine 1: Amazon MWAA Structure

MWAA Serverless makes use of declarative YAML configuration recordsdata based mostly on the favored open supply DAG Manufacturing facility format to reinforce safety by activity isolation. You will have two choices for creating these workflow definitions:

This declarative method supplies two key advantages. First, since MWAA Serverless reads workflow definitions from YAML it may possibly decide activity scheduling with out working any workflow code. Second, this enables MWAA Serverless to grant execution permissions solely when duties run, fairly than requiring broad permissions on the workflow degree. The result’s a safer setting the place activity permissions are exactly scoped and time restricted.

Service issues for MWAA Serverless

MWAA Serverless has the next limitations that it’s best to take into account when deciding between serverless and provisioned MWAA deployments:

• Operator help
  • MWAA Serverless solely helps operators from the Amazon Supplier Package deal.
  • To execute customized code or scripts, you’ll want to make use of AWS providers, reminiscent of:
• Person interface
  • MWAA Serverless operates with out utilizing the Airflow net interface.
  • For workflow monitoring and administration, we offer integration with Amazon CloudWatch and AWS CloudTrail.

Working with MWAA Serverless

Full the next stipulations and steps to make use of MWAA Serverless.

Stipulations

Earlier than you start, confirm you have got the next necessities in place:

• Entry and permissions
  • An AWS account
  • AWS Command Line Interface (AWS CLI) model 2.31.38 or later put in and configured
  • The suitable permissions to create and modify IAM roles and insurance policies, together with the next required IAM permissions:
    • airflow-serverless:CreateWorkflow
    • airflow-serverless:DeleteWorkflow
    • airflow-serverless:GetTaskInstance
    • airflow-serverless:GetWorkflowRun
    • airflow-serverless:ListTaskInstances
    • airflow-serverless:ListWorkflowRuns
    • airflow-serverless:ListWorkflows
    • airflow-serverless:StartWorkflowRun
    • airflow-serverless:UpdateWorkflow
    • iam:CreateRole
    • iam:DeleteRole
    • iam:DeleteRolePolicy
    • iam:GetRole
    • iam:PutRolePolicy
    • iam:UpdateAssumeRolePolicy
    • logs:CreateLogGroup
    • logs:CreateLogStream
    • logs:PutLogEvents
    • airflow:GetEnvironment
    • airflow:ListEnvironments
    • s3:DeleteObject
    • s3:GetObject
    • s3:ListBucket
    • s3:PutObject
    • s3:Sync
  • Entry to an Amazon Digital Personal Cloud (VPC) with web connectivity
• Required AWS providers – Along with MWAA Serverless you will have entry to the next AWS providers:
  • Amazon MWAA to entry your current Airflow setting(s)
  • Amazon CloudWatch to view logs
  • Amazon S3 for DAG and YAML file administration
  • AWS IAM to manage permissions
• Growth setting
• Extra necessities
  • Fundamental familiarity with Apache Airflow ideas
  • Understanding of YAML syntax
  • Data of AWS CLI instructions

Word: All through this publish, we use instance values that you simply’ll want to interchange with your individual:

• Change amzn-s3-demo-bucket together with your S3 bucket title
• Change 111122223333 together with your AWS account quantity
• Change us-east-2 together with your AWS Area. MWAA Serverless is on the market in a number of AWS Areas. Verify the Record of AWS Companies Out there by Area for present availability.

Creating your first serverless workflow

Let’s begin by defining a easy workflow that will get a listing of S3 objects and writes that record to a file in the identical bucket. Create a brand new file known as simple_s3_test.yaml with the next content material:

simples3test:
  dag_id: simples3test
  schedule: 0 0 * * *
  duties:
    list_objects:
      operator: airflow.suppliers.amazon.aws.operators.s3.S3ListOperator
      bucket: 'amzn-s3-demo-bucket'
      prefix: ''
      retries: 0
    create_object_list:
      operator: airflow.suppliers.amazon.aws.operators.s3.S3CreateObjectOperator
      knowledge: '{{ ti.xcom_pull(task_ids="list_objects", key="return_value") }}'
      s3_bucket: 'amzn-s3-demo-bucket'
      s3_key: 'filelist.txt'
      dependencies: [list_objects]

For this workflow to run, you could create an Execution function that has permissions to record and write to the above bucket. The function additionally must be assumable from MWAA Serverless. The next CLI instructions create this function and its related coverage:

aws iam create-role 
--role-name mwaa-serverless-access-role 
--assume-role-policy-document '{
    "Model": "2012-10-17",
    "Assertion": [
      {
        "Effect": "Allow",
        "Principal": {
          "Service": [
            "airflow-serverless.amazonaws.com"
          ]
        },
        "Motion": "sts:AssumeRole"
      },
      {
        "Sid": "AllowAirflowServerlessAssumeRole",
        "Impact": "Enable",
        "Principal": {
          "Service": "airflow-serverless.amazonaws.com"
        },
        "Motion": "sts:AssumeRole",
        "Situation": {
          "StringEquals": {
            "aws:SourceAccount": "${aws:PrincipalAccount}"
          },
          "ArnLike": {
            "aws:SourceArn": "arn:aws:*:*:${aws:PrincipalAccount}:workflow/*"
          }
        }
      }
    ]
  }'

aws iam put-role-policy 
  --role-name mwaa-serverless-access-role 
  --policy-name mwaa-serverless-policy   
  --policy-document '{
	"Model": "2012-10-17",
	"Assertion": [
		{
			"Sid": "CloudWatchLogsAccess",
			"Effect": "Allow",
			"Action": [
				"logs:CreateLogGroup",
				"logs:CreateLogStream",
				"logs:PutLogEvents"
			],
			"Useful resource": "*"
		},
		{
			"Sid": "S3DataAccess",
			"Impact": "Enable",
			"Motion": [
				"s3:ListBucket",
				"s3:GetObject",
				"s3:PutObject"
			],
			"Useful resource": [
				"arn:aws:s3:::amzn-s3-demo-bucket",
				"arn:aws:s3:::amzn-s3-demo-bucket/*"
			]
		}
	]
}'

You then copy your YAML DAG to the identical S3 bucket, and create your workflow based mostly upon the Arn response from the above perform.

aws s3 cp "simple_s3_test.yaml" 
s3://amzn-s3-demo-bucket/yaml/simple_s3_test.yaml

aws mwaa-serverless create-workflow 
--name simple_s3_test 
--definition-s3-location '{ "Bucket": "amzn-s3-demo-bucket", "ObjectKey": "yaml/simple_s3_test.yaml" }' 
--role-arn arn:aws:iam::111122223333:function/mwaa-serverless-access-role 
--region us-east-2

The output of the final command returns a WorkflowARN worth, which you then use to run the workflow:

aws mwaa-serverless start-workflow-run 
--workflow-arn arn:aws:airflow-serverless:us-east-2:111122223333:workflow/simple_s3_test-abc1234def 
--region us-east-2

The output returns a RunId worth, which you then use to test the standing of the workflow run that you simply simply executed.

aws mwaa-serverless get-workflow-run 
--workflow-arn arn:aws:airflow-serverless:us-east-2:111122223333:workflow/simple_s3_test-abc1234def 
--run-id ABC123456789def 
--region us-east-2

If it’s worthwhile to make a change to your YAML, you possibly can copy again to S3 and run the update-workflow command.

aws s3 cp "simple_s3_test.yaml" 
s3://amzn-s3-demo-bucket/yaml/simple_s3_test.yaml

aws mwaa-serverless update-workflow 
--workflow-arn arn:aws:airflow-serverless:us-east-2:111122223333:workflow/simple_s3_test-abc1234def 
--definition-s3-location '{ "Bucket": "amzn-s3-demo-bucket", "ObjectKey": "yaml/simple_s3_test.yaml" }' 
--role-arn arn:aws:iam::111122223333:function/mwaa-serverless-access-role 
--region us-east-2

Changing Python DAGs to YAML format

AWS has revealed a conversion instrument that makes use of the open-source Airflow DAG processor to serialize Python DAGs into YAML DAG manufacturing facility format. To put in, you run the next:

pip3 set up python-to-yaml-dag-converter-mwaa-serverless
dag-converter convert source_dag.py --output output_yaml_folder

For instance, create the next DAG and title it create_s3_objects.py:

from datetime import datetime
from airflow import DAG
from airflow.fashions.param import Param
from airflow.suppliers.amazon.aws.operators.s3 import S3CreateObjectOperator

default_args = {
    'start_date': datetime(2024, 1, 1),
    'retries': 0,
}

dag = DAG(
    'create_s3_objects',
    default_args=default_args,
    description='Create a number of S3 objects in a loop',
    schedule=None
)

# Set variety of recordsdata to create
LOOP_COUNT = 3
s3_bucket="md-workflows-mwaa-bucket"
s3_prefix = 'test-files'

# Create a number of S3 objects utilizing loop
last_task=None
for i in vary(1, LOOP_COUNT + 1):  
    create_object = S3CreateObjectOperator(
        task_id=f'create_object_{i}',
        s3_bucket=s3_bucket,
        s3_key=f'{s3_prefix}/{i}.txt',
        knowledge="{{ ds_nodash }}-{ decrease }",
        substitute=True,
        dag=dag
    )
    if last_task:
        last_task >> create_object
    last_task = create_object

After getting put in python-to-yaml-dag-converter-mwaa-serverless, you run:

dag-converter convert "/path_to/create_s3_objects.py" --output "/path_to/yaml/"

The place the output will finish with:

YAML validation profitable, no errors discovered

YAML written to /path_to/yaml/create_s3_objects.yaml

And ensuing YAML will seem like:

create_s3_objects:
  dag_id: create_s3_objects
  params: {}
  default_args:
    start_date: '2024-01-01'
    retries: 0
  schedule: None
  duties:
    create_object_1:
      operator: airflow.suppliers.amazon.aws.operators.s3.S3CreateObjectOperator
      aws_conn_id: aws_default
      knowledge: '{{ ds_nodash }}-{ decrease }'
      encrypt: false
      shops: []
      params: {}
      priority_weight: 1
      substitute: true
      retries: 0
      retry_delay: 300.0
      retry_exponential_backoff: false
      s3_bucket: md-workflows-mwaa-bucket
      s3_key: test-files/1.txt
      task_id: create_object_1
      trigger_rule: all_success
      wait_for_downstream: false
      dependencies: []
    create_object_2:
      operator: airflow.suppliers.amazon.aws.operators.s3.S3CreateObjectOperator
      aws_conn_id: aws_default
      knowledge: '{{ ds_nodash }}-{ decrease }'
      encrypt: false
      shops: []
      params: {}
      priority_weight: 1
      substitute: true
      retries: 0
      retry_delay: 300.0
      retry_exponential_backoff: false
      s3_bucket: md-workflows-mwaa-bucket
      s3_key: test-files/2.txt
      task_id: create_object_2
      trigger_rule: all_success
      wait_for_downstream: false
      dependencies: [create_object_1]
    create_object_3:
      operator: airflow.suppliers.amazon.aws.operators.s3.S3CreateObjectOperator
      aws_conn_id: aws_default
      knowledge: '{{ ds_nodash }}-{ decrease }'
      encrypt: false
      shops: []
      params: {}
      priority_weight: 1
      substitute: true
      retries: 0
      retry_delay: 300.0
      retry_exponential_backoff: false
      s3_bucket: md-workflows-mwaa-bucket
      s3_key: test-files/3.txt
      task_id: create_object_3
      trigger_rule: all_success
      wait_for_downstream: false
      dependencies: [create_object_2]
  catchup: false
  description: Create a number of S3 objects in a loop
  max_active_runs: 16
  max_active_tasks: 16
  max_consecutive_failed_dag_runs: 0

Word that, as a result of the YAML conversion is finished after the DAG parsing, the loop that creates the duties is run first and the ensuing static record of duties is written to the YAML doc with their dependencies.

Migrating an MWAA setting’s DAGs to MWAA Serverless

You’ll be able to benefit from a provisioned MWAA setting to develop and check your workflows after which transfer them to serverless to run effectively at scale. Additional, in case your MWAA setting is utilizing suitable MWAA Serverless operators, then you possibly can convert the entire setting’s DAGs directly. Step one is to permit MWAA Serverless to imagine the MWAA Execution function through a belief relationship. It is a one-time operation for every MWAA Execution function, and could be carried out manually within the IAM console or utilizing an AWS CLI command as follows:

MWAA_ENVIRONMENT_NAME="MyAirflowEnvironment"
MWAA_REGION=us-east-2

MWAA_EXECUTION_ROLE_ARN=$(aws mwaa get-environment --region $MWAA_REGION --name $MWAA_ENVIRONMENT_NAME --query 'Surroundings.ExecutionRoleArn' --output textual content )
MWAA_EXECUTION_ROLE_NAME=$(echo $MWAA_EXECUTION_ROLE_ARN | xargs basename) 
MWAA_EXECUTION_ROLE_POLICY=$(aws iam get-role --role-name $MWAA_EXECUTION_ROLE_NAME --query 'Function.AssumeRolePolicyDocument' --output json | jq '.Assertion[0].Principal.Service += ["airflow-serverless.amazonaws.com"] | .Assertion[0].Principal.Service |= distinctive | .Assertion += [{"Sid": "AllowAirflowServerlessAssumeRole", "Effect": "Allow", "Principal": {"Service": "airflow-serverless.amazonaws.com"}, "Action": "sts:AssumeRole", "Condition": {"StringEquals": {"aws:SourceAccount": "${aws:PrincipalAccount}"}, "ArnLike": {"aws:SourceArn": "arn:aws:*:*:${aws:PrincipalAccount}:workflow/*"}}}]')

aws iam update-assume-role-policy --role-name $MWAA_EXECUTION_ROLE_NAME --policy-document "$MWAA_EXECUTION_ROLE_POLICY"

Now we are able to loop by every efficiently transformed DAG and create serverless workflows for every.

S3_BUCKET=$(aws mwaa get-environment --name $MWAA_ENVIRONMENT_NAME --query 'Surroundings.SourceBucketArn' --output textual content --region us-east-2 | minimize -d':' -f6)

for file in /tmp/yaml/*.yaml; do MWAA_WORKFLOW_NAME=$(basename "$file" .yaml); 
      aws s3 cp "$file" s3://$S3_BUCKET/yaml/$MWAA_WORKFLOW_NAME.yaml --region us-east-2; 
      aws mwaa-serverless create-workflow --name $MWAA_WORKFLOW_NAME 
      --definition-s3-location "{"Bucket": "$S3_BUCKET", "ObjectKey": "yaml/$MWAA_WORKFLOW_NAME.yaml"}" --role-arn $MWAA_EXECUTION_ROLE_ARN  
      --region us-east-2  
      executed

To see a listing of your created workflows, run:

aws mwaa-serverless list-workflows --region us-east-2

Monitoring and observability

MWAA Serverless workflow execution standing is returned through the GetWorkflowRun perform. The outcomes from that may return particulars for that specific run. If there are errors within the workflow definition, they’re returned underneath RunDetail within the ErrorMessage subject as within the following instance:

{
  "WorkflowVersion": "7bcd36ce4d42f5cf23bfee67a0f816c6",
  "RunId": "d58cxqdClpTVjeN",
  "RunType": "SCHEDULE",
  "RunDetail": {
    "ModifiedAt": "2025-11-03T08:02:47.625851+00:00",
    "ErrorMessage": "anticipated token ',', received 'create_test_table'",
    "TaskInstances": [],
    "RunState": "FAILED"
  }
}

Workflows which are correctly outlined, however whose duties fail, will return "ErrorMessage": "Workflow execution failed":

{
  "WorkflowVersion": "0ad517eb5e33deca45a2514c0569079d",
  "RunId": "ABC123456789def",
  "RunType": "SCHEDULE",
  "RunDetail": {
    "StartedOn": "2025-11-03T13:12:09.904466+00:00",
    "CompletedOn": "2025-11-03T13:13:57.620605+00:00",
    "ModifiedAt": "2025-11-03T13:16:08.888182+00:00",
    "Period": 107,
    "ErrorMessage": "Workflow execution failed",
    "TaskInstances": [
      "ex_5496697b-900d-4008-8d6f-5e43767d6e36_create_bucket_1"
    ],
    "RunState": "FAILED"
  },
}

MWAA Serverless activity logs are saved within the CloudWatch log group /aws/mwaa-serverless// (the place / is identical string because the distinctive workflow id within the ARN of the workflow). For particular activity log streams, you will have to record the duties for the workflow run after which get every activity’s info. You’ll be able to mix these operations right into a single CLI command.

aws mwaa-serverless list-task-instances 
  --workflow-arn arn:aws:airflow-serverless:us-east-2:111122223333:workflow/simple_s3_test-abc1234def 
  --run-id ABC123456789def 
  --region us-east-2 
  --query 'TaskInstances[].TaskInstanceId' 
  --output textual content | xargs -n 1 -I {} aws mwaa-serverless get-task-instance 
  --workflow-arn arn:aws:airflow-serverless:us-east-2:111122223333:workflow/simple_s3_test-abc1234def 
  --run-id ABC123456789def 
  --task-instance-id {} 
  --region us-east-2 
  --query '{Standing: Standing, StartedAt: StartedAt, LogStream: LogStream}'

Which might consequence within the following:

{
    "Standing": "SUCCESS",
    "StartedAt": "2025-10-28T21:21:31.753447+00:00",
    "LogStream": "//aws/mwaa-serverless/simple_s3_test_3-abc1234def//workflow_id=simple_s3_test-abc1234def/run_id=ABC123456789def/task_id=list_objects/try=1.log"
}
{
    "Standing": "FAILED",
    "StartedAt": "2025-10-28T21:23:13.446256+00:00",
    "LogStream": "//aws/mwaa-serverless/simple_s3_test_3-abc1234def//workflow_id=simple_s3_test-abc1234def/run_id=ABC123456789def/task_id=create_object_list/try=1.log"
}

At which level, you’ll use the CloudWatch LogStream output to debug your workflow.

It’s possible you’ll view and handle your workflows within the Amazon MWAA Serverless console:

For an instance that creates detailed metrics and monitoring dashboard utilizing AWS Lambda, Amazon CloudWatch, Amazon DynamoDB, and Amazon EventBridge, overview the instance in this GitHub repository.

Clear up assets

To keep away from incurring ongoing prices, observe these steps to wash up all assets created throughout this tutorial:

1. Delete MWAA Serverless workflows – Run this AWS CLI command to delete all workflows:

   aws mwaa-serverless list-workflows --query 'Workflows[*].WorkflowArn' --output textual content | whereas learn -r workflow; do aws mwaa-serverless delete-workflow --workflow-arn $workflow executed

2. Take away the IAM roles and insurance policies created for this tutorial:

   aws iam delete-role-policy --role-name mwaa-serverless-access-role --policy-name mwaa-serverless-policy

3. Take away the YAML workflow definitions out of your S3 bucket:

   aws s3 rm s3://amzn-s3-demo-bucket/yaml/ --recursive

After finishing these steps, confirm within the AWS Administration Console that every one assets have been correctly eliminated. Keep in mind that CloudWatch Logs are retained by default and will have to be deleted individually if you wish to take away all traces of your workflow executions.

If you happen to encounter any errors throughout cleanup, confirm you have got the required permissions and that assets exist earlier than trying to delete them. Some assets could have dependencies that require them to be deleted in a particular order.

Conclusion

On this publish, we explored Amazon MWAA Serverless, a brand new deployment choice that simplifies Apache Airflow workflow administration. We demonstrated tips on how to create workflows utilizing YAML definitions, convert current Python DAGs to the serverless format, and monitor your workflows.

MWAA Serverless provides a number of key benefits:

• No provisioning overhead
• Pay-per-use pricing mannequin
• Automated scaling based mostly on workflow calls for
• Enhanced safety by granular IAM permissions
• Simplified workflow definitions utilizing YAML

To study extra MWAA Serverless, overview the documentation.

In regards to the authors