When engineering groups at Slack, Reddit, Canva, Airbnb, Shopify, and Uber must ship code with confidence, they depend on Buildkite. As a CI/CD platform, Buildkite orchestrates advanced construct, take a look at, and deployment pipelines for a few of the most demanding engineering organizations on this planet. It handles every part from routine code commits to synthetic intelligence (AI) model-training workloads, processing over 50 billion requests per 30 days.
On the coronary heart of Buildkite’s take a look at orchestration portfolio is Check Engine, a specialised analytics product designed to assist engineering groups perceive and optimize their take a look at suites at scale. Check Engine aggregates outcomes throughout 1000’s of builds, flags flaky exams, runs parallel take a look at execution throughout machine fleets, and delivers interactive analytics on take a look at execution information. It helps arbitrary metadata tagging for dimensions like occasion kind, structure, language model, cloud supplier, and have flags.
The problem? Delivering all of this in actual time, throughout a number of enterprise tenants, at a quantity that may stress even probably the most sturdy information infrastructure. On this submit, we discover how Buildkite makes use of Amazon Managed Streaming for Apache Kafka (Amazon MSK) and Amazon Managed Service for Apache Flink to energy Check Engine’s streaming-first analytics structure at scale.
The issue: When scale breaks conventional architectures
Buildkite’s Check Engine should ingest and serve analytics on take a look at telemetry from 1000’s of distributed pipelines concurrently, for a number of enterprise clients. The dimensions is unforgiving: 50 billion take a look at executions per 30 days, 500K occasions per second at peak ingestion, and webhook payloads reaching 21 MB.
The architectural evolution and its limits
The unique Rails and PostgreSQL stack couldn’t maintain this progress. In 2024, the workforce re-architected round a distributed streaming layer, a stateful stream processor for pre-aggregations, and a number of specialised shops: a key-value retailer for quick lookups, a relational database for pre-computed aggregates, and an open desk format (Iceberg) with a distributed question engine (Trino) for versatile querying.
But the core pressure remained unsolved. Enterprise clients demanded interactive, arbitrary slicing of billions of information throughout high-cardinality dimensions, not canned experiences. The stream processor couldn’t deal with advert hoc aggregations at question time. The important thing-value retailer was blind to analytical queries. The distributed question engine supplied flexibility however was too gradual for interactive use.
The consequence was a system that was costly and operationally advanced. It included 9 relational database clusters, sprawling ETL pipelines, and 24/7 pre-aggregation jobs operating no matter demand. It nonetheless couldn’t ship the one factor clients wanted most: quick, versatile, interactive analytics at scale.
Structure and implementation: MSK and Amazon Managed Service for Apache Flink because the streaming spine
The answer Buildkite arrived at facilities on Amazon Managed Streaming for Apache Kafka (Amazon MSK) and Amazon Managed Service for Apache Flink because the real-time information streaming and processing layers, decoupling high-throughput ingestion from downstream analytics.
The information pipeline
The next diagram exhibits the end-to-end information move from CI/CD brokers by way of Amazon MSK and Amazon Managed Service for Apache Flink to the analytics layer.
Amazon MSK sits on the essential junction between information producers (the distributed CI/CD brokers and take a look at collectors operating throughout buyer infrastructure) and the downstream processing and analytics layers. Amazon Managed Service for Apache Flink then transforms these uncooked occasion streams into enriched, queryable information earlier than it reaches the analytics retailer.
Excessive-throughput ingestion from CI/CD pipelines
Amazon MSK’s function begins at ingestion. Check collectors embedded in CI/CD pipelines publish take a look at execution occasions on to Kafka subjects. The prevailing Amazon MSK cluster handles between 5 MB/sec and 100 MB/sec of inbound information below regular working situations. The structure is designed to soak up the numerous variance inherent in CI/CD workloads, the place pipeline exercise is bursty and correlated with engineering workforce working hours throughout international time zones.
When the Buildkite mission was initiated, MSK Categorical Brokers weren’t but obtainable, main the workforce to undertake MSK Tiered Storage as the first mechanism for scaling and restoration. With MSK Categorical Brokers now typically obtainable, the workforce is evaluating a migration of its most important log ingestion workload, which sustains as much as 1 GB/s at peak ingestion. MSK Categorical Brokers convey automated storage scaling with zero storage administration overhead, as much as 20x sooner scaling and 90% sooner dealer restoration, 3x greater per-broker throughput, 5x extra partitions per dealer, and built-in Clever Rebalancing.
Actual-time stream processing with Amazon Managed Service for Apache Flink
Sitting between Amazon MSK and the analytics layer, Amazon Managed Service for Apache Flink acts because the stateful stream processing engine that transforms uncooked occasion streams earlier than they attain downstream programs. Buildkite chosen Flink for its exactly-once processing, mature stateful computation mannequin, and deep Kafka integration. Dealing with sustained peaks of over 25,000 occasions per second, Amazon Managed Service for Apache Flink eliminates the operational overhead of cluster provisioning, model upgrades, checkpointing, and job restoration. This frees engineering groups to deal with software logic.
Amazon Managed Service for Apache Flink powers key stateful processing duties, together with flaky take a look at detection by way of time-windowed sample matching, enriching execution occasions with pipeline and buyer metadata, and routing processed information to downstream programs resembling ClickHouse for analytics, PostgreSQL for operational workloads, and Amazon Easy Storage Service (Amazon S3) for long-term archival.
Reliability and fault tolerance
Amazon MSK’s three-replica configuration ensures that no single dealer failure may cause information loss or ingestion interruption. Mixed with versatile information retention, the structure offers a significant replay window. If a downstream client (Amazon Managed Service for Apache Flink, ClickHouse, or one other service) experiences an outage, it will probably resume processing from its final dedicated offset with out information loss.
In the course of the migration to the present structure, Buildkite employed a dual-write technique: concurrently writing to each the present PostgreSQL pipeline and the brand new Amazon MSK/ClickHouse path. This strategy allowed the workforce to validate information consistency and regularly shift site visitors with out risking customer-facing disruption. This sample speaks to the operational maturity Amazon MSK offers.
Operational effectivity positive factors
The shift to a streaming-first structure, mixed with the downstream simplification of the analytics engine, produced important operational enhancements:
- Flink workloads decreased by 60%+: Eliminating pre-aggregation jobs that ran repeatedly no matter demand.
- Key/worth retailer fully retired: Amazon MSK’s buffering functionality, mixed with ClickHouse’s question efficiency, eradicated the necessity for a separate fast-lookup retailer.
- PostgreSQL capability reduce in half: 9 separate database clusters consolidated and right-sized.
- 1000’s of strains of software code deleted: Easier structure means much less ETL code, fewer failure modes, and sooner onboarding for brand new engineers.
Platform efficiency at a look
| Metric | Worth |
| Month-to-month take a look at executions (for take a look at engine platform) | 50 billion (4x progress from 3B) |
| Sustained peak ingestion | 500K occasions/second |
| Complete information in analytics retailer | 200 billion |
| Log ingestion requests | 70,000+ per second |
| Peak webhook throughput | 1.7 GB/second |
| MSK inbound throughput vary | 5 MB/sec – 100 MB/sec |
Enterprise and developer influence
The technical structure finally exists to serve one function: serving to builders ship higher software program sooner. The streaming-first structure constructed on Amazon MSK and Amazon Managed Service for Apache Flink delivers on that promise throughout 4 dimensions.
On-demand analytics changed pre-computed experiences. Prospects can now interactively slice and cube 70 billion information throughout arbitrary metadata dimensions. They get solutions to queries like “Present me P50 take a look at durations by occasion kind and structure for the final 30 days” in seconds, not hours. Actual-time log streaming by way of the “stay tail” function means builders not await a construct to finish earlier than diagnosing failures. At 25,000 occasions per second, this expertise scales throughout 1000’s of concurrent enterprise pipelines with out degradation.
Smarter take a look at intelligence comes from Amazon Managed Service for Apache Flink’s stateful flaky take a look at detection: when a take a look at begins exhibiting intermittent failure patterns, Amazon Managed Service for Apache Flink identifies it because it occurs, not after the very fact. That is what separates a proactive analytics platform from a reactive one. It requires publishing information to Kafka, processing with Flink, and letting ClickHouse deal with the advanced learn requests.
Conclusion: Streaming as a strategic basis
Buildkite’s journey from a Rails/Postgres monolith to a streaming-first analytics platform displays a sample more and more frequent amongst enterprise SaaS corporations: a dependable, high-throughput streaming and processing layer shouldn’t be an optimization. It’s a prerequisite for working at scale.
Amazon MSK and Amazon Managed Service for Apache Flink kind the spine that helps Buildkite ingest 50 billion take a look at executions per 30 days, serve real-time interactive analytics to enterprise clients, and accomplish that at decrease value than the extra advanced structure it changed. Amazon MSK handles sturdy, elastic occasion buffering. Amazon Managed Service for Apache Flink transforms uncooked streams into enriched, queryable information. Collectively they take up the operational complexity that may in any other case devour engineering capability.
For platform engineers evaluating streaming infrastructure for multi-tenant SaaS workloads, the sign is evident: put money into the streaming spine early, and let managed providers deal with the operational complexity.
To be taught extra about Amazon MSK and Amazon Managed Service for Apache Flink, go to aws.amazon.com/msk and aws.amazon.com/managed-service-apache-flink.
In regards to the authors
