Are you modernizing your legacy batch processing techniques? At Vanguard, we confronted vital challenges with our legacy mainframe system that restricted our capability to ship trendy, customized buyer experiences. Our centralized database structure created efficiency bottlenecks and made it troublesome to scale providers independently for our thousands and thousands of non-public and institutional buyers.
On this submit, we present you ways we modernized our knowledge structure utilizing Amazon Redshift as our Operational Learn-only Knowledge Retailer (ORDS). You’ll learn the way we transitioned to a cloud-native, domain-driven structure whereas preserving essential batch processing capabilities. We present you ways this answer enabled us to create logically remoted knowledge domains whereas sustaining cross-domain analytics capabilities—all whereas adhering to the rules of bounded contexts and distributed knowledge possession.
Background and challenges
As monetary wants proceed to evolve, Vanguard is dedicated to delivering adaptable, top-notch experiences that foster long-lasting buyer relationships. This dedication spans from enhancing the private investor journey to bringing customized cell dashboards and connecting institutional purchasers with superior recommendation choices.
To raise buyer expertise and drive digital transformation, Vanguard has embraced domain-driven design rules. This strategy focuses on creating autonomous groups, fostering quicker innovation, and constructing knowledge mesh structure. Central to this transformation is the Private Investor group’s mainframe modernization effort, transitioning from a legacy system to a cloud-based, distributed knowledge structure organized round bounded contexts – distinct enterprise domains that handle their very own knowledge. As a part of this shift, every microservice now manages its personal native knowledge retailer utilizing Amazon Aurora PostgreSQL-Suitable Version or Amazon DynamoDB. This strategy permits domain-level knowledge possession and operational autonomy.
Vanguard’s current mainframe system, constructed on a centralized Db2 database, permits cross-domain knowledge entry and integration but in addition introduces a number of architectural challenges. Although batch processes can be a part of knowledge throughout a number of bounded contexts utilizing SQL joins and database operations to combine data from varied sources, this tight coupling creates vital dangers and operational points.
Challenges with the centralized database strategy embody:
- Useful resource Competition: Processes from one area can negatively affect different domains because of shared compute sources, resulting in efficiency degradation throughout the system.
- Lack of Area Isolation: Adjustments in a single bounded context can have unintended ripple results throughout different domains, rising the chance of system-wide failures.
- Scalability Constraints: The centralized structure creates bottlenecks as load will increase, making it troublesome to scale particular person parts independently.
- Excessive Coupling: Tight integration between domains makes it difficult to change or improve particular person parts with out affecting the complete system.
- Restricted Fault Tolerance: Points in a single area can cascade throughout the complete system because of shared infrastructure and knowledge dependencies.
To deal with these architectural challenges, we selected to make use of Amazon Redshift as our Operational Learn-only Knowledge Retailer (ORDS). The Amazon Redshift structure has compute and storage separation, which permits us to create multi-cluster architectures with a separate endpoint for every area with unbiased scaling of compute and storage sources. Our answer leverages the information sharing capabilities of Amazon Redshift to create logically remoted knowledge domains whereas sustaining the flexibility to carry out cross-domain analytics when wanted.
Key advantages of the Amazon Redshift answer embody:
- Useful resource Isolation: Every area will be assigned devoted Amazon Redshift compute sources, ensuring one area’s workload doesn’t affect others.
- Impartial Scaling: Domains can scale their compute sources independently based mostly on their particular wants.
- Managed Knowledge Sharing: Amazon Redshift’s knowledge sharing function permits safe and managed cross-domain knowledge entry with out tight coupling, sustaining clear area boundaries.
Let’s discover the totally different options we evaluated earlier than choosing ORDS with Amazon Redshift as our optimum strategy.
Options explored
We applied ORDS as our optimum answer after conducting a complete analysis of obtainable choices. This part outlines our decision-making course of and examines the alternate options we thought-about throughout our evaluation.
Operational Learn-only Knowledge Retailer (ORDS):
In our analysis, we discovered that utilizing Amazon Redshift for ORDS supplies a strong answer for dealing with knowledge throughout totally different enterprise areas. It excels at managing giant volumes of knowledge from a number of sources, offering quick entry to replicated knowledge for batch processes that require cross-bounded context knowledge, and mixing data utilizing acquainted SQL queries. The answer significantly shines in dealing with high-volume reads from our knowledge sources.
Benefits:
- Works nicely in a relational database
- Excels at real-time entry to knowledge from a number of enterprise areas
- Improves efficiency of batch jobs coping with giant knowledge volumes
- Shops knowledge in acquainted desk format, accessible through SQL
- Enforces clear knowledge possession, with every enterprise space chargeable for its knowledge
- Presents scalable structure that reduces the chance of single level of failure
Disadvantages:
- Requires further knowledge validation throughout loading processes to take care of knowledge uniqueness
- Wants cautious administration of major key constraints since Amazon Redshift optimizes for analytical efficiency
- Could require further monitoring and controls in comparison with conventional RDBMS techniques
Listed below are the opposite options we evaluated:
Bulk APIs:
We discovered that Bulk APIs supplies an strategy for dealing with giant volumes of knowledge.
Benefits:
- Close to actual time entry to bulk knowledge by means of a single request
- Autonomous groups have management over entry patterns
- Environment friendly batch processing of enormous datasets with multi-record retrieval
Disadvantages:
- Every product group must create their very own bulk API
- Should you want knowledge from totally different areas, it’s essential to mix it your self
- The group offering the API should make sure that it could possibly deal with giant quantities of requests
- You would possibly want to make use of a number of APIs to get all the information you need
- Should you’re getting knowledge in chunks (pagination), you would possibly miss some data if it modifications between requests
Whereas Bulk APIs supply highly effective capabilities, we discovered they require substantial group coordination and cautious implementation to be efficient.
Knowledge Lake:
Our analysis confirmed that knowledge lakes can successfully mix data from totally different components of our enterprise. They excel at processing giant quantities of knowledge directly, offering search capabilities by means of unified knowledge codecs, and managing giant volumes of various and sophisticated knowledge.
Benefits:
- Handles large knowledge volumes effectively
- Helps a number of knowledge codecs and constructions
- Permits advanced analytics and knowledge science workloads
- Gives cost-effective storage options
- Accommodates each structured and unstructured knowledge
Disadvantages:
- Could not present real-time, high-speed knowledge entry
- Requires further effort with advanced knowledge constructions, particularly these with many interconnected components
- Wants particular methods to arrange knowledge in a easy, flat construction
- Calls for vital knowledge governance and administration
- Requires specialised abilities for efficient implementation
Whereas knowledge lakes excel at big-picture evaluation of enormous datasets, they weren’t optimum for our real-time knowledge wants and sophisticated knowledge relationships.
S3 Export/Alternate:
In our evaluation, we discovered that S3 Export/Alternate supplies a way for sharing knowledge between totally different enterprise areas utilizing file storage. This strategy successfully handles giant volumes of knowledge and permits simple filtering of knowledge utilizing knowledge frames.
Benefits:
- Gives easy, cost-effective knowledge storage
- Helps high-volume knowledge transfers
- Permits simple knowledge filtering capabilities
- Presents versatile entry management
- Facilitates cross-region knowledge sharing
Disadvantages:
- Not appropriate for real-time knowledge wants
- Requires further processing to transform knowledge into usable desk format
- Calls for vital knowledge preparation effort
- Lacks fast knowledge consistency
- Wants further instruments for knowledge transformation
Whereas S3 Export/Alternate works nicely for sharing giant datasets between groups, it didn’t meet our necessities for fast, real-time entry or instantly usable knowledge codecs.
The next desk supplies a high-level comparability of the totally different knowledge integration options we thought-about for our modernization efforts. It outlines the place every answer is most acceptable to make use of and when it may not be your best option:
| Answer | Bulk APIs | Knowledge Lake | ORDS | S3 Export/Alternate |
| When to make use of | Actual-time operational knowledge is required
Fetching particular knowledge subsets |
Processing giant quantities of knowledge directly
Many bounded context |
Close to real-time entry throughout a number of bounded contexts
Massive quantity batch processing |
Few bounded contextsHandling giant volumes of knowledge
Level-in-time export is enough |
| When to not use | Many bounded contexts concerned | Actual-time knowledge entry wanted
Structured, transactional knowledge processing |
Inside a single bounded context | Actual-time knowledge wants
Many bounded contexts |
Desk 1: Knowledge Integration Options Comparability
Primarily based on our comparability, we discovered ORDS to be the optimum answer for our wants, significantly when our batch processes require entry to knowledge from a number of bounded contexts in real-time. Our implementation effectively handles giant volumes of knowledge, considerably enhancing the efficiency of our batch jobs. We selected ORDS as a result of it shops knowledge in a well-recognized desk format, accessible through SQL, making it easy and environment friendly for our groups to make use of.
The structure additionally aligns with our domain-driven design rules by implementing clear knowledge possession, the place every bounded context maintains duty for its personal knowledge administration. This strategy supplies us with each scalability and reliability, lowering the chance of a single level of failure.
Amazon Redshift: Powering Vanguard’s ORDS Answer
Amazon Redshift serves because the spine of our ORDS implementation, providing a number of essential options that assist our modernization objectives:
Knowledge Sharing
Our answer leveraged the sturdy knowledge sharing capabilities of Amazon Redshift, accessible on each Server-based Redshift RA3 situations and Redshift Serverless choices. This performance supplied us with instantaneous, safe, and reside knowledge entry with out copies, sustaining transactional consistency throughout our surroundings. The flexibleness of similar account, cross-account, and cross-Area knowledge sharing has been significantly precious for our distributed structure.
Excessive Efficiency
We’ve achieved vital efficiency enhancements by means of Amazon Redshift’s environment friendly question processing and knowledge retrieval capabilities. The system successfully handles our advanced knowledge wants whereas sustaining sturdy efficiency throughout varied workloads and knowledge volumes.
Multi-Availability Zone Help
Our implementation benefited from Amazon Redshift’s Multi-AZ assist, which maintains excessive availability and reliability for our essential operations. This function minimizes downtime with out requiring intensive setup and considerably reduces our danger of knowledge loss.
Acquainted Interface
The relational surroundings of Amazon Redshift, comparable conventional databases like Amazon RDS and IBM Db2, has enabled a clean transition for our groups. This familiarity has accelerated adoption and improved productiveness, as our groups can leverage their current SQL experience. By centralizing knowledge from a number of enterprise areas in ORDS utilizing Amazon Redshift, we preserve constant, environment friendly, and safe knowledge entry throughout our product groups. This setup is especially precious for our batch processing that requires knowledge from varied components of the enterprise, providing us a mix of efficiency, reliability, and ease of use.
Operational Learn-only Knowledge Retailer (ORDS) utilizing Amazon Redshift
Right here’s how our ORDS structure implements Amazon Redshift knowledge sharing to resolve these challenges:
Determine 1: Vanguard’s ORDS Structure utilizing Amazon Redshift Knowledge Sharing
Amazon Redshift Ingestion Sample:
We utilized Amazon Redshift’s zero-ETL performance to combine knowledge and allow real-time analytics straight on operational knowledge, which helped scale back complexity and upkeep overhead. To enrich this functionality and to meet our complete compliance necessities that necessitate full transaction replication, we applied further knowledge ingestion pipelines.
Our knowledge ingestion technique for Amazon Redshift employs totally different AWS providers relying on the supply. For Amazon Aurora PostgreSQL databases, we use AWS Database Migration Service (AWS DMS) to straight replicate knowledge into Amazon Redshift. For knowledge from Amazon DynamoDB, we leverage Amazon Kinesis to stream the information into Amazon Redshift, the place it lands in materialized views. These views are then additional processed to generate tables for end-users.
This strategy permits us to effectively ingest knowledge from our operational knowledge shops whereas assembly each analytical wants and compliance necessities.
Amazon Redshift Knowledge Sharing:
We used the Amazon Redshift’s knowledge sharing function to successfully decouple our knowledge producers from shoppers, permitting every group to function inside their very own boundaries whereas sustaining a unified and simplified ruled mechanism for knowledge sharing.
Our implementation adopted a transparent course of: as soon as knowledge is ingested and accessible in Amazon Redshift desk format, we created views for shoppers to entry the information. We then established knowledge shares and granted entry to those views to client Amazon Redshift knowledge warehouses for batch processing. In our surroundings with a number of bounded contexts, we’ve established a collaborative mannequin the place shoppers work with varied producer groups to entry knowledge from totally different knowledge shares, every created per bounded context.
This entry remained strictly read-only—when shoppers have to replace or write new knowledge that falls outdoors their bounded context, they have to use APIs or different designated mechanisms for such operations. This strategy has confirmed efficient for our group, selling clear knowledge possession and governance whereas enabling versatile knowledge entry throughout organizational boundaries. It simplified our knowledge administration and made positive every group can function independently whereas nonetheless sharing knowledge successfully.
Instance: VG couple of cross bounded context
Disclaimer: That is supplied for reference functions solely and doesn’t signify an actual instance.
Let’s have a look at a sensible instance: our brokerage account assertion era course of. This cross-bounded context batch course of requires integrating knowledge from a number of sources, accessing a whole bunch of tables and processing giant volumes of knowledge month-to-month. The problem was to create an environment friendly, cost-effective answer that minimizes knowledge replication whereas sustaining knowledge accessibility.ORDS proved superb for this use case, because it supplies knowledge from a number of bounded contexts with out replication, presents close to real-time entry, and permits simple knowledge aggregation utilizing SQL-like queries in Amazon Redshift.
The next diagram reveals how we applied this answer:
Determine 2: Cross-Bounded Context Instance for Brokerage Account Assertion Technology
We’d like the next bounded contexts to generate brokerage statements for thousands and thousands of our purchasers.
- Account:
- Particulars: Consists of details about the shopper’s brokerage accounts, comparable to account numbers, sorts, and statuses.
- Holdings and Positions: Gives present holdings and positions throughout the account, detailing the securities owned, their portions, and present market values.
- Stability Data: Incorporates the steadiness data of the account, together with money balances, margin balances, and whole account worth.
- Consumer Profile:
- Private Data: Details about the shopper, comparable to their title, date of start, and social safety quantity.
- Contact Data: Consists of the shopper’s e mail handle, bodily handle, and cellphone numbers.
- Transaction Historical past:
- Transaction Information: A complete file of transactions related to the account, together with buys, gross sales, transfers, and dividends.
- Transaction Particulars: Every transaction file consists of particulars comparable to transaction date, kind, amount, worth, and related charges.
- Historic Knowledge: Historic knowledge of transactions over time, offering an entire view of the account’s exercise.
By means of this structure, we effectively generate correct and complete brokerage account statements by consolidating knowledge from these bounded contexts, assembly each our purchasers’ wants and regulatory necessities.
Enterprise End result
Our journey with the Operational Learn-only Knowledge Retailer (ORDS) and Amazon Redshift has enhanced our shopper expertise (CX) by means of improved knowledge administration and accessibility. By transitioning from our mainframe system to a cloud-based, domain-driven structure, we have now empowered our autonomous groups and established a resilient batch structure.
This shift facilitates environment friendly cross-domain knowledge entry, maintains high-quality knowledge consistency, and supplies scalability. Our ORDS implementation, supported by Amazon Redshift, presents near-real-time entry to giant knowledge volumes, guaranteeing excessive efficiency, reliability, and cost-effectiveness. This modernization effort aligns with our mission to ship distinctive, customized shopper experiences and maintain long-lasting shopper relationships.
Name to Motion
In case you are going through comparable challenges together with your batch processing techniques, we encourage you to discover how an Operational Learn-only Knowledge Retailer (ORDS) can remodel your knowledge structure. Begin by assessing your present system’s limitations and figuring out alternatives for enchancment by means of domain-driven design and cloud-based options. Take into account how this strategy can assist you handle giant volumes of knowledge from a number of sources, present quick entry to replicated knowledge for batch processes, and assist high-volume reads from varied knowledge sources.
Take the subsequent step by conducting a proof of idea (POC) to judge ORDS effectiveness in reaching environment friendly cross-domain knowledge entry, enhancing the efficiency of batch jobs, and sustaining clear knowledge possession inside what you are promoting domains. By implementing this answer, you possibly can improve your knowledge administration capabilities, scale back operational dangers, and drive innovation inside your group. Embrace this chance to raise your knowledge structure and ship distinctive buyer experiences.
Conclusion
Our transition to a cloud-native, domain-driven structure with ORDS utilizing Amazon Redshift has efficiently remodeled our batch processing capabilities in AWS cloud. This modernization effort has considerably enhanced the efficiency, reliability, and scalability of our batch operations whereas sustaining seamless knowledge entry and integration throughout totally different enterprise domains.
The strategic adoption of ORDS has harnessed the potential of cross-domain knowledge entry in a distributed surroundings, offering us with a strong answer for real-time knowledge entry and environment friendly batch processing. This transformation has empowered us to higher meet the calls for of the digital age, delivering superior buyer experiences and reinforcing our dedication to innovation within the monetary providers business.
In regards to the authors
© 2025 The Vanguard Group, Inc. All rights reserved.
