Allow real-time mainframe analytics with Exactly Join and Amazon S3

This can be a visitor submit by Supreet Padhi, Expertise Architect, Strategic Applied sciences, and Rochelle Grubbs, Senior Director, Answer Architect at Exactly in partnership with AWS.

Enterprise leaders face a vital problem to allow real-time analytics. Their most useful knowledge sits in mainframe programs that reliably course of billions of transactions every day, however extracting worth for contemporary analytics and AI stays complicated and expensive. Conventional mainframe-to-cloud integration approaches require multi-step replication with middleman programs, creating operational overhead, latency, and knowledge integrity dangers. This complexity delays insights, will increase infrastructure prices, limits agility, and blocks organizations from utilizing AI and machine studying on their mainframe knowledge.

Exactly, a world chief in knowledge integrity with over 12,000 clients together with 95 of the Fortune 100, has introduced an growth of its collaboration with AWS via new enhancements to Exactly Join. Exactly is an AWS Knowledge and Analytics ISV Competency and AWS Migration and Modernization ISV Competency associate. Exactly has service specializations in Amazon Redshift and Amazon Relational Database Service (Amazon RDS).

In Stream mainframe knowledge to AWS in near-real time with Exactly and Amazon MSK, we confirmed you the way to arrange mainframe CDC and the AWS Mainframe Modernization – Knowledge Replication for IBM z/OS Amazon Machine Picture (AMI) out there in AWS Market. On this submit, we focus on how you should use Exactly Connect with allow real-time, direct replication of mainframe knowledge to Amazon Easy Storage Service (Amazon S3), and the way your group can prolong this basis utilizing Amazon S3 Tables for superior analytics.

Actual-time mainframe knowledge entry

Organizations that may join their mainframe environments with trendy cloud platforms can achieve benefits via improved agility, diminished operational prices, and enhanced analytics capabilities.For instance, shifting acceptable analytics and reporting workloads to the cloud can considerably cut back mainframe operational prices whereas sustaining efficiency and reliability. Actual-time knowledge entry makes insights out there inside seconds fairly than ready for batch processing cycles, enabling sooner responses to market modifications and buyer wants. Eliminating bulk knowledge extracts and middleman programs additionally reduces infrastructure and upkeep bills. This frees IT assets to give attention to higher-value initiatives.

Nonetheless, implementing mainframe-to-cloud integrations presents distinctive technical challenges that require specialised options. These embody changing mainframe character encoding (EBCDIC) to straightforward ASCII format and dealing with mainframe-specific knowledge sorts corresponding to packed decimal (COMP) fields. You additionally have to handle the complexity of VSAM (Digital Storage Entry Technique) recordsdata that may retailer a number of file sorts in a single file, and keep real-time synchronization with out impacting mainframe efficiency.

Change Knowledge Seize (CDC) expertise addresses these challenges via incremental knowledge motion that eliminates disruptive bulk extracts by streaming solely modified knowledge to cloud targets, minimizing system affect and guaranteeing knowledge foreign money. Actual-time synchronization retains cloud purposes in sync with mainframe programs, enabling speedy insights and responsive operations.

Exactly Join: Actual-time knowledge replication to Amazon S3

With Exactly Join, you may replicate knowledge straight from mainframes to Amazon S3 in actual time, eliminating the necessity for intermediaries and simplifying modernization.Knowledge flows straight from mainframe sources, together with Db2 z/OS, IMS, and VSAM, to Amazon S3, eliminating middleman steps and lowering each latency and operational complexity. You may transfer mainframe knowledge on to Amazon S3 knowledge lakes and analytics platforms with out managing complicated, multi-step replication processes.

The simplicity of this method reduces upkeep overhead and integration complexity by eradicating the necessity for staging servers, middleware, or batch processing programs. After knowledge lands in Amazon S3, it turns into instantly out there for downstream AWS workloads. You should utilize Amazon Athena for SQL queries, AWS Glue for ETL and knowledge cataloging, Amazon EMR for giant knowledge processing, Amazon SageMaker AI for machine studying, and Amazon Fast Sight for enterprise intelligence dashboards.

Answer overview

Right here we current an answer structure for streaming mainframe knowledge modifications from Db2z via AWS Mainframe Modernization – Knowledge Replication for IBM z/OS AMI on to Amazon S3 after which utilizing Amazon S3 Tables for superior analytics capabilities.

By introducing direct S3 replication and streamlining deployment via the pre-configured AWS Market AMI, you may deploy in minutes fairly than weeks. This creates new potentialities for knowledge distribution, transformation, and consumption. This structure gives a number of key advantages:

1. Simplified deployment – Speed up implementation utilizing the preconfigured AWS Market AMI
2. Direct replication – Eradicate middleman programs by streaming knowledge on to Amazon S3, lowering latency and operational overhead
3. Actual-time synchronization – Seize modifications as they happen on the mainframe, guaranteeing downstream purposes function on present knowledge
4. Versatile analytics choices – Use S3 Tables for Iceberg-compatible tabular knowledge storage
5. Complete AWS integration – Acquire speedy entry to Amazon EMR, Amazon Athena, AWS Glue, Amazon SageMaker AI, and Amazon Fast Sight
6. Pure language knowledge entry – By the MCP Server for Amazon S3 Tables, AI assistants can work together with structured knowledge utilizing conversational interfaces without having to put in writing SQL queries.

Conditions

To finish the answer, you want the next conditions:

Exactly elements

1. AWS Mainframe Modernization – Knowledge Replication for IBM z/OS – Deploy this Exactly Join AMI from AWS Market. This pre-configured picture comprises the Apply Engine and Controller Daemon elements required for replicating mainframe knowledge modifications to Amazon S3.
2. Exactly Join CDC Seize/Writer – Deploy the Exactly Join CDC Seize/Writer in your mainframe atmosphere. This element captures modifications from Db2z logs and streams them to the Apply Engine over TCP/IP.

For detailed setup and configuration steps for Exactly elements, confer with our earlier submit Stream mainframe knowledge to AWS in near-real time with Exactly and Amazon MSK.

Connectivity necessities

1. Have community connectivity established between your mainframe atmosphere and AWS utilizing your group’s permitted connectivity technique (corresponding to AWS Direct Join or VPN).
2. Confirm that firewall guidelines permit TCP/IP communication between the mainframe Seize/Writer and the Apply Engine.

AWS analytics elements (non-compulsory extension)

After mainframe knowledge lands in Amazon S3, your group can prolong its analytics capabilities utilizing AWS companies. One method is to make use of Amazon EMR streaming jobs to course of and write knowledge to Amazon S3 Tables. After the information is saved in S3 Tables, the information may be queried straight utilizing Amazon Athena for ad-hoc SQL evaluation. This extension is non-compulsory and represents one in every of a number of methods to eat and analyze mainframe knowledge after it reaches Amazon S3.

The next diagram illustrates the answer structure.

1. Seize/Writer – Join CDC Seize/Writer captures Db2 modifications from Db2 logs utilizing IFI 306 Learn and communicates captured knowledge modifications to a goal engine via TCP/IP.
2. Controller Daemon – The Controller Daemon authenticates all connection requests, managing safe communication between the supply and goal environments.
3. Apply Engine – The Apply Engine receives the modifications from the Writer agent and applies the modified knowledge to the goal Amazon S3.
4. Amazon S3 – Serves because the scalable knowledge lake basis the place replicated mainframe knowledge lands.
5. Amazon EMR streaming job – As knowledge arrives, an occasion of the Amazon EMR streaming job writes the information to focus on tables in Amazon S3 Tables.
6. Amazon Athena – Queries knowledge saved in Amazon S3 Tables utilizing normal SQL.

This structure offers a clear separation between the information seize course of and the information consumption course of, permitting every to scale independently. When CDC knowledge arrives in Amazon S3, you should use Amazon S3 Tables to retailer Db2 z/OS, VSAM, and IMS knowledge in an open desk format (Apache Iceberg) that’s prepared for analytics, offering a versatile path to mainframe modernization.

Quantifiable enterprise worth

Organizations implementing this answer usually see vital reductions in mainframe operational prices by offloading analytics and reporting workloads to the cloud. The elimination of middleman infrastructure reduces each capital and operational bills. The diminished upkeep burden frees IT assets to give attention to strategic initiatives fairly than managing complicated replication programs. Velocity and agility enhancements are equally vital. Close to real-time knowledge availability, measured in seconds to minutes fairly than hours to days, permits organizations to reply quickly to market modifications and operational occasions. The speedy deployment of latest analytics use circumstances with out requiring mainframe modifications accelerates innovation. Organizations achieve entry to the total breadth of AWS companies that can be utilized instantly after knowledge lands in Amazon S3.

From an analytics and AI perspective, the answer creates a unified knowledge platform that brings collectively mainframe, cloud-native, and third-party knowledge sources. This unified view permits superior machine studying on historic and present knowledge, delivering predictive insights that drive proactive decision-making throughout the group.

Buyer story

A number one international funds supplier put this into apply. The funds supplier was struggling to generate well timed analytics and insights from Level of Sale (POS) transaction knowledge. As one of many world’s largest fee suppliers, they course of lots of of 1000’s of transactions per second. Customers count on to swipe their card and have their transaction permitted in seconds. New structure was wanted to maintain up with buyer calls for and quantity. By streaming mission-critical mainframe knowledge on to AWS in actual time utilizing Exactly Join and touchdown it in Amazon S3 Tables, the corporate used storage constructed on the Apache Iceberg open normal. This method permits high-performance analytics straight on mainframe knowledge alongside cloud-native sources.

Conclusion

On this submit, we demonstrated how Exactly Join permits real-time, direct knowledge replication from mainframes to Amazon S3, eliminating intermediaries and simplifying mainframe modernization.

Your group can additional prolong this basis with Amazon S3 Tables, purpose-built storage for Apache Iceberg tables in S3, enabling analytical purposes to question essentially the most present mainframe knowledge utilizing instruments corresponding to Amazon Athena, Amazon EMR, and Amazon Redshift.

Get began by deploying AWS Mainframe Modernization – Knowledge Replication for IBM z/OS from AWS Market and use Amazon S3 as a goal to your mainframe use circumstances. Study extra about Exactly’s mainframe knowledge integration capabilities at exactly.com. Contact AWS and Exactly consultants to debate your particular modernization challenges and design a proof-of-concept that demonstrates enterprise worth shortly.

Concerning the authors