This can be a visitor put up by Alex Rabinovich, Anindya Dasgupta, and Vijesh Chandran from Vanguard, Monetary Advisor Companies division, in partnership with AWS.
Vanguard stands as one of many world’s main funding corporations, serving greater than 50 million buyers globally. The corporate presents an intensive collection of low-cost mutual funds and ETFs with over 450 funds/ETFs together with complete funding recommendation and associated monetary providers. With a workforce of roughly 20,000 crew members, Vanguard has constructed its popularity on offering low-cost, high-quality funding options that assist buyers obtain their long-term monetary targets.
Inside this huge group, Vanguard’s Monetary Advisor Companies (FAS) division stands as one of the outstanding B2B operations within the monetary providers business. Working at a unprecedented scale, FAS oversees a broad vary and various vary of belongings via the middleman channel whereas supporting an enormous community of advisory companies and monetary advisors throughout the nation. This division delivers a full suite of funding merchandise, mannequin portfolios, analysis capabilities, and technology-driven help providers designed to assist monetary advisors serve their purchasers extra successfully.
Enterprise use circumstances and preliminary structure
The size and complexity of FAS operations generate huge quantities of knowledge that require subtle analytics capabilities to drive enterprise insights, regulatory compliance, and operational effectivity. To deal with this, Vanguard launched the FAS 360 initiative. This initiative goals to empower Monetary Advisor Companies (FAS) with a centralized cloud knowledge warehouse that integrates each inside and exterior knowledge sources right into a unified, clever system.
Key enterprise use circumstances:
- Enterprise operations – Permits gross sales aim setting, monitoring, and compensation administration to drive operational excellence. It delivers insights on product utilization patterns throughout monetary advisor purchasers.
- Information science – Powers buyer segmentation fashions and name transcription analytics to drive strategic insights. It additionally helps advertising marketing campaign preparation and buyer insights for gross sales name preparation.
- Exploratory analytics – Permits ad-hoc management questions, what-if situation evaluation, and gross sales development evaluation for channel managers competitor comparative evaluation.
By consolidating these use circumstances right into a centralized system, FAS 360 allows constant reporting and data-driven decision-making throughout Vanguard’s Monetary Advisor Companies division.
Centralized knowledge warehouse FAS 360:
Vanguard’s first wave of modernization established FAS 360 as a centralized enterprise knowledge warehouse, migrating from a fragmented “knowledge swamp” of Parquet recordsdata on Amazon Easy Storage Service (Amazon S3) to a structured, unified system.
The next structure diagram leverages Amazon S3 for uncooked knowledge storage with Amazon Redshift serving because the core processing engine, offering built-in entry for BI instruments, analyst exploration, and knowledge science workloads.
Listed below are the important thing advantages achieved with this structure:
- Single supply of fact – Consolidated fragmented knowledge sources right into a unified system, minimizing a number of variations of fact and establishing constant reporting practices throughout the group
- 10x sooner question efficiency – Dramatically improved question response instances in comparison with the earlier resolution, serving to improve analyst productiveness and enabling extra advanced analytical workloads
- Seamless knowledge lake integration – Maintained connectivity with the broader knowledge lake surroundings whereas offering structured warehouse capabilities
- Enhanced enterprise agility – Elevated belief in metrics and unlocked new use circumstances that have been beforehand untenable, directing the brand new migration efforts towards the FAS360 system
This centralized structure efficiently addressed the restrictions of Vanguard’s earlier method, the place knowledge was scattered throughout people with restricted governance, and established a basis for his or her subsequent architectural evolution.
Vital development and increasing use circumstances
Vanguard FAS skilled outstanding development of their knowledge analytics necessities over a two-year interval, demonstrating the fast evolution of contemporary knowledge wants:
Preliminary State:
- 20 AWS Glue ETL jobs processing every day knowledge hundreds
- Roughly 100 tables of their knowledge warehouse
- 20 Tableau dashboards serving enterprise customers
- Round 60 analysts accessing the system
Two Years Later:
- 20 TB in knowledge quantity in Amazon Redshift and one other 150 TB in S3 knowledge lake
- 600+ AWS Glue ETL jobs (a 30x enhance) dealing with advanced knowledge transformations
- 300+ tables (3x development) storing various enterprise knowledge
- 250+ Amazon Redshift materialized views optimizing question efficiency
- Over 500 Tableau dashboards (25x enlargement) serving varied enterprise capabilities
- 500,000+ consumer queries/months
This exponential development mirrored FAS’s rising reliance on data-driven resolution making throughout the enterprise capabilities, from threat administration and compliance to consumer service optimization and operational effectivity enhancements.
Useful resource competition and efficiency bottlenecks
As Vanguard FAS’s knowledge surroundings expanded, their preliminary structure, a single Amazon Redshift provisioned cluster with 2 nodes (ra3.4xlarge), started experiencing extreme efficiency challenges that threatened enterprise operations:
ETL efficiency points:
- Frequent ETL SLA failures disrupting essential enterprise processes
- Tableau extract failures leading to stale dashboard knowledge
- Useful resource conflicts between knowledge ingestion and transformation workloads
Finish-user expertise degradation:
- Poor question efficiency throughout peak utilization intervals
- Desk and object locking points stopping concurrent entry
- Pissed off analysts unable to carry out deep knowledge exploration
- Restricted capability to run long-running analytical queries
Operational challenges:
- Useful resource competition between ETL workloads and interactive analytics
- Lack of ability to scale compute sources independently for various workload varieties
- Single level of failure affecting the information operations
- Problem in workload prioritization and useful resource allocation
These challenges have been basically limiting FAS’s capability to leverage their knowledge belongings successfully, impacting all the things from every day operational reporting to strategic enterprise evaluation.
Resolution overview
To deal with these essential challenges, Vanguard FAS carried out following multi-warehouse structure that leverages the superior knowledge sharing capabilities of Amazon Redshift for workload isolation and unbiased scaling.
Producer – Amazon Redshift Provisioned Cluster
The central hub consists of the unique Amazon Redshift provisioned cluster with RA3 nodes, optimized for constant, predictable workloads:
- Devoted ETL processing: Handles knowledge ingestion, transformation, and loading operations
- Write workload optimization: Manages knowledge writes and updates with out interference
- Value optimization: Makes use of reserved situations for predictable, steady-state workloads
- Information governance: Serves as the one supply of fact for the enterprise knowledge
Client – Amazon Redshift Serverless Workgroups
A number of Amazon Redshift Serverless situations function specialised client endpoints which auto-scales compute sources based mostly on demand:
- Analyst Exploration: Devoted surroundings for analyst knowledge discovery and experimentation
- BI Instruments: Occasion optimized particularly for Tableau dashboard and visualization workloads
- Information Science: For advanced and lengthy operating machine studying workloads in utterly remoted surroundings
The answer leverages the native knowledge sharing capabilities of Amazon Redshift to allow safe connectivity between the producer and shoppers situations. Client clusters can entry dwell knowledge from the producer with out knowledge motion, offering real-time entry to probably the most present data out there. This zero-copy sharing method alleviates the necessity for knowledge duplication or advanced synchronization processes, serving to cut back each storage prices and operational complexity.
Outcomes
The implementation of the multi-warehouse structure delivered vital enhancements throughout the important thing efficiency indicators:
Predictable Efficiency
Nightly ETL cycles now persistently full earlier than the 9 AM SLA, eliminating the earlier SLA failures that disrupted enterprise operations and making certain recent knowledge is accessible for morning enterprise actions. Dashboards and experiences now replicate probably the most present knowledge out there, offering groups with up-to-date insights for decision-making.
Improved Analyst Productiveness and Expertise
The brand new structure eliminated the restrictive 10-minute question timeout that beforehand prevented deep advert hoc exploratory queries. Analysts can now run advanced analytical workloads exceeding half-hour in a completely remoted surroundings with out impacting different customers or ETL processes. This transformation, mixed with considerably sooner question response instances, has led to greater analyst satisfaction and productiveness throughout the group.
New Analytical Capabilities
The structure launched a devoted “Information Lab” surroundings the place analysts have write entry to experiment with knowledge utilizing CREATE TABLE AS SELECT (CTAS) instructions. Every workload kind can now scale independently based mostly on demand, with totally different client clusters optimized for particular use circumstances, enabling extra subtle analytical approaches.
Operational Excellence
The separation of workloads enabled environment friendly utilization of compute sources throughout totally different patterns, main to raised value management via acceptable sizing, serverless pay-as-you-go pricing, and reserved occasion utilization. The cleaner separation of issues between ETL and analytics workloads has simplified general administration of the information platform.
Ongoing modernization: Evolution towards knowledge mesh structure
As Vanguard’s knowledge surroundings matured and their success with the multi-warehouse structure enabled broader adoption throughout the group, they acknowledged a chance to evolve their structure to match their organizational development. The increasing portfolio of knowledge merchandise and rising variety of groups leveraging the system created new alternatives for innovation.
As Vanguard’s knowledge surroundings grew, three key challenges emerged:
- Centralized possession bottleneck – Single-team knowledge possession couldn’t scale with the rising variety of knowledge merchandise
- Write workload competition – Useful resource competition persevered for write operations on shared endpoints
- Cross-domain dependencies – Information object interdependencies throughout enterprise domains slowed knowledge product improvement
Rationale for Information Mesh
Vanguard’s resolution to undertake Information Mesh was pushed by the necessity to:
- Decentralize knowledge possession by establishing knowledge domains with devoted stewards
- Take away write competition by isolating every area’s knowledge hundreds to separate endpoints
- Allow autonomous improvement permitting stewards to personal the entire knowledge product lifecycle and governance
- Leverage trendy knowledge lake capabilities utilizing AWS Glue and Apache Iceberg format for knowledge product curation
This evolution helps Vanguard’s capability to scale organizationally whereas constructing on the technical basis and operational excellence achieved with their multi-warehouse structure. Constructing on the success of their Amazon Redshift multi-warehouse implementation, Vanguard FAS is now exploring on the subsequent part of their knowledge structure evolution, implementing following knowledge mesh method.
This new knowledge mesh structure has a number of key elements that work collectively to allow scalable, domain-oriented knowledge administration.
Area-Oriented Information Possession
Vanguard is establishing distinct knowledge domains aligned with enterprise capabilities and assigning devoted knowledge stewards to every area for clear possession and accountability. This technique shifts from centralized knowledge administration to a decentralized mannequin the place knowledge possession and duty might be distributed throughout enterprise domains, enabling groups nearer to the information to make knowledgeable selections about their domain-specific wants.
Distributed Information Structure
The brand new structure isolates domain-specific knowledge hundreds to separate compute endpoints and creates unbiased knowledge processing pipelines for every area. This method helps cut back cross-domain dependencies and conflicts that beforehand slowed improvement cycles, permitting groups to iterate and deploy adjustments with out ready for coordination throughout the complete group.
Information Product Strategy
Vanguard is curating knowledge merchandise on the information lake utilizing Apache Iceberg format and leveraging AWS Glue for metrics computation and knowledge lake integration. This method treats knowledge as merchandise with outlined SLAs and high quality metrics, serving to facilitate dependable, high-quality knowledge supply that downstream shoppers can rely on with confidence.
Self-Service Analytics
The implementation allows area groups to handle their full knowledge product lifecycle independently whereas sustaining enterprise governance requirements. Vanguard supplies complete instruments and techniques for unbiased knowledge administration, permitting groups to innovate rapidly with out compromising knowledge high quality or safety, finally accelerating time-to-insight throughout the group.This evolution represents a pure development from centralized knowledge warehouse to multi-warehouse structure, and at last to a completely distributed, domain-oriented knowledge mesh that may scale with Vanguard’s continued development.
Conclusion
Vanguard Monetary Advisor Companies’ journey demonstrates that scaling analytics is now not about scaling a single warehouse larger, however about architecting for workload isolation, unbiased scaling, and organizational development.
By evolving from a single 2-node RA3 provisioned cluster to a multi-warehouse structure utilizing Amazon Redshift Serverless and Provisioned, Vanguard achieved measurable, production-grade outcomes:
- 500,000+ month-to-month queries supported with out ETL or dashboard competition
- 100% ETL SLA adherence, with nightly pipelines finishing earlier than 9 AM
- 25x development in BI consumption (20 → 500+ Tableau dashboards) with out efficiency degradation
- 8x development in analyst inhabitants (60 → 500+) enabled via workload isolation
- 30x enhance in ETL pipelines (20 → 600+) with out re-architecting ingestion logic
- Zero-copy Amazon Redshift knowledge sharing throughout producer and client warehouses, minimizing knowledge duplication and synchronization prices
- Removing of 10-minute question limits, unlocking superior exploratory and long-running analytics
Critically, these good points weren’t achieved by over-provisioning compute, however by right-sizing and specializing compute per workload, reserving capability the place demand was predictable (ETL) and utilizing Amazon Redshift Serverless auto-scaling the place demand was bursty (BI and ad-hoc evaluation).
As Vanguard now progresses towards a domain-oriented knowledge mesh, their expertise reinforces a key lesson: Multi-warehouse structure is a foundational enabler for organizational scale, knowledge product possession, and autonomous analytics.For organizations experiencing thrilling development of their knowledge analytics necessities, Vanguard’s method showcases the super potentialities that await. With the fitting structure and the assistance of AWS providers, organizations can rework their knowledge infrastructure to realize outstanding enhancements in efficiency, vital value reductions, and unlock highly effective new analytical capabilities that speed up enterprise worth creation.
AWS encourages you to attach together with your AWS Account Crew to interact an AWS analytics specialist who can present skilled architectural steerage and tailor-made suggestions that can assist you obtain your knowledge transformation targets.
© 2026 The Vanguard Group, Inc. and Amazon Net Companies, Inc. All rights reserved. This materials is offered for informational functions solely and isn’t supposed to be funding recommendation or a advice to take any specific funding motion.
In regards to the authors
