Amazon Managed Workflows for Apache Airflow (Amazon MWAA) has turn into a cornerstone for organizations embracing data-driven decision-making. As a scalable resolution for managing complicated information pipelines, Amazon MWAA allows seamless orchestration throughout AWS companies and on-premises methods. Though AWS manages the underlying infrastructure, you should rigorously plan and execute your Amazon MWAA surroundings updates in accordance with the shared duty mannequin. Upgrading to the newest Amazon MWAA model can present vital benefits, together with enhanced safety by means of vital safety patches and potential enhancements in efficiency with sooner DAG parsing and lowered database load. You should utilize superior options whereas sustaining ecosystem compatibility and receiving prioritized AWS assist. The important thing to profitable upgrades lies in selecting the best resolution and following a methodical implementation strategy.
On this put up, we discover finest practices for upgrading your Amazon MWAA surroundings and supply a step-by-step information to seamlessly transition to the newest model.
Answer overview
Amazon MWAA gives two major improve options:
- In-place improve – This technique works finest when you may accommodate deliberate downtime. You deploy the brand new model straight in your current infrastructure. In-place model upgrades on Amazon MWAA are supported for environments operating Apache Airflow model 2.x and later. Nonetheless, if you happen to’re operating model 1.10.z or older variations, you should create a brand new surroundings and migrate your assets, as a result of these variations don’t assist in-place upgrades.
- Cutover improve – This technique helps reduce disruption to manufacturing environments. You create a brand new Amazon MWAA surroundings with the goal model after which transition out of your outdated surroundings to the brand new one.
Every resolution gives a special strategy that can assist you improve whereas working to take care of information integrity and system reliability.
In-place improve
In-place upgrades work nicely for environments the place you may schedule a upkeep window for the improve course of. Throughout this window, Amazon MWAA preserves your workflow historical past. This technique works finest when you may accommodate deliberate downtime. It helps preserve historic information, gives a simple improve course of, and consists of rollback capabilities if points happen throughout provisioning. You additionally use fewer assets since you don’t have to create a brand new surroundings.
You possibly can carry out in-place upgrades by means of the AWS Administration Console with a single operation. This course of helps cut back operational overhead by managing many improve steps for you.
In the course of the improve course of, your surroundings can’t schedule or run new duties. Amazon MWAA helps handle the improve course of and implements security measures—if points happen in the course of the provisioning part, the service makes an attempt to revert to the earlier steady model.
Earlier than you start an in-place improve, we advocate testing your DAGs for compatibility with the goal model, as a result of DAG compatibility points can have an effect on the improve course of. You should utilize the Amazon MWAA native runner to check DAG compatibility earlier than you begin the improve. You can begin the improve utilizing both the console and specifying the brand new model or the AWS Command Line Interface (AWS CLI). The next is an instance Amazon MWAA improve command utilizing the AWS CLI:
The next diagram reveals the Amazon MWAA in-place improve workflow and states.
Discuss with Introducing in-place model upgrades with Amazon MWAA for extra particulars.
Cutover improve
A cutover improve gives an alternate resolution when it’s good to reduce downtime, although it requires extra handbook steps and operational planning. With this strategy, you create a brand new Amazon MWAA surroundings, migrate your metadata, and handle the transition between environments. Though this technique gives extra management over the improve course of, it requires extra planning and execution effort in comparison with an in-place improve.
This technique can work nicely for environments with complicated workflows, significantly while you plan to make vital modifications alongside the model improve. The strategy gives a number of advantages: you may reduce manufacturing downtime, carry out complete testing earlier than switching environments, and preserve the flexibility to return to your authentic surroundings if wanted. You may also evaluate and replace your configurations in the course of the transition.
Contemplate the next elements of the cutover strategy. Whenever you run two environments concurrently, you pay for each environments. The pricing for every Amazon MWAA surroundings is determined by:
- Period of surroundings uptime (billed hourly with per-second decision)
- Setting dimension configuration
- Computerized scaling capability for employees
- Scheduler capability
AWS calculates the price of extra computerized scaled employees individually. You possibly can estimate prices in your particular configuration utilizing the AWS Pricing Calculator.
To assist stop information duplication or corruption throughout parallel operation, we advocate implementing idempotent DAGs. The Airflow scheduler robotically populates some metadata tables (dag, dag_tag, and dag_code) in your new surroundings. Nonetheless, it’s good to plan the migration of the next extra metadata parts:
- DAG historical past
- Variables
- Slot pool configurations
- SLA miss data
- XCom information
- Job data
- Log tables
You possibly can select this strategy when your necessities prioritize minimal downtime and you’ll handle the extra operational complexity.
The cutover improve course of entails three most important steps: creating a brand new surroundings, restoring it with the present information, and performing the improve. The next diagram illustrates the complete workflow.
Within the following sections, we stroll by means of the important thing steps to carry out a cutover improve.
Stipulations
Earlier than you start the improve course of, full the next steps:
Create a brand new surroundings
Full the next steps to create a brand new surroundings:
- Generate a template in your new surroundings configuration utilizing the AWS CLI:
- Modify the generated JSON file:
- Copy configurations out of your backup file
.json to .json. - Replace the surroundings identify.
- Preserve the AirflowVersion parameter worth out of your current surroundings.
- Evaluation and replace different configuration parameters as wanted.
- Copy configurations out of your backup file
- Create your new surroundings:
Restore the brand new surroundings
Full the next steps to revive the brand new surroundings:
- Use the mwaa-dr PyPI bundle to create and run the restore DAG.
- This course of copies metadata out of your S3 backup bucket to the brand new surroundings.
- Confirm that your new surroundings incorporates the anticipated metadata out of your authentic surroundings.
Carry out the model improve
Full the next steps to carry out the model improve:
- Improve your surroundings:
- Monitor the improve:
- Observe the surroundings standing on the console.
- Look ahead to error messages or warnings.
- Confirm the surroundings reaches the AVAILABLE
Plan your transition timing rigorously. When your authentic surroundings continues to course of workflows throughout this improve, the metadata between environments can change.
Clear up
After you confirm the steadiness of your upgraded surroundings by means of monitoring, you may start the cleanup course of:
- Take away your authentic Amazon MWAA surroundings utilizing the AWS CLI command:
- Clear up your related assets by eradicating unused backup information from S3 buckets, deleting momentary AWS Identification and Entry Administration (IAM) roles and insurance policies created for the improve, and updating your DNS or routing configurations.
Earlier than eradicating any assets, be sure to observe your group’s backup retention insurance policies, preserve mandatory backup information in your compliance necessities, and doc configuration modifications made in the course of the improve.
This strategy helps you carry out a managed improve with alternatives for testing and the flexibility to return to your authentic surroundings if wanted.
Monitoring and validation
You possibly can observe your improve progress utilizing Amazon CloudWatch metrics, with a give attention to DAG processing metrics and scheduler heartbeat. Your surroundings transitions by means of a number of states in the course of the improve course of, together with UPDATING and CREATING. When your surroundings reveals the AVAILABLE state, you may start validation testing. We advocate checking system accessibility, testing vital workflow operations, and verifying exterior connections. For detailed monitoring steering, see Monitoring and metrics for Amazon Managed Workflows for Apache Airflow.
Key concerns
Think about using infrastructure as code (IaC) practices to assist preserve constant surroundings administration and assist repeatable deployments. Schedule metadata backups utilizing mwaa-dr in periods of low exercise to assist shield your information. When designing your workflows, implement idempotent pipelines to assist handle potential interruptions, and preserve documentation of your configurations and dependencies.
Conclusion
A profitable Amazon MWAA improve begins with choosing an strategy that aligns together with your operational necessities. Whether or not you select an in-place or cutover improve, thorough preparation and testing assist assist a managed transition. Utilizing out there instruments, monitoring capabilities, and really useful practices can assist you improve to the newest Amazon MWAA options whereas working to take care of your workflow operations.
For extra particulars and code examples on Amazon MWAA, confer with the Amazon MWAA Person Information and Amazon MWAA examples GitHub repo.
Apache, Apache Airflow, and Airflow are both registered logos or logos of the Apache Software program Basis in america and/or different nations.
Concerning the Authors
Anurag Srivastava works as a Senior Large Knowledge Cloud Engineer at Amazon Net Companies (AWS), specializing in Amazon MWAA. He’s keen about serving to prospects construct scalable information pipelines and workflow automation options on AWS.
Sriharsh Adari is a Senior Options Architect at Amazon Net Companies (AWS), the place he helps prospects work backwards from enterprise outcomes to develop progressive options on AWS. Through the years, he has helped a number of prospects on information platform transformations throughout trade verticals. His core space of experience embrace Expertise Technique, Knowledge Analytics, and Knowledge Science. In his spare time, he enjoys enjoying sports activities, binge-watching TV reveals, and enjoying Tabla.
Venu Thangalapally is a Senior Options Architect at AWS, primarily based in Chicago, with deep experience in cloud structure, information and analytics, containers, and software modernization. He companions with Monetary Companies trade prospects to translate enterprise targets into safe, scalable, and compliant cloud options that ship measurable worth. Venu is keen about leveraging know-how to drive innovation and operational excellence. Outdoors of labor, he enjoys spending time together with his household, studying, and taking lengthy walks.
Chandan Rupakheti is a Senior Options Architect at AWS. His most important focus at AWS lies within the intersection of analytics, serverless, and AdTech companies. He’s a passionate technical chief, researcher, and mentor with a knack for constructing progressive options within the cloud. Outdoors of his skilled life, he loves spending time together with his household and associates, and listening to and enjoying music.
