How Slack achieved operational excellence for Spark on Amazon EMR utilizing generative AI

At Slack, our information platform processes terabytes of information every day utilizing Apache Spark on Amazon EMR on Amazon Elastic Compute Cloud (Amazon EC2), powering the insights that drive strategic decision-making throughout the group.

As our information quantity expanded, so did our efficiency challenges. With conventional monitoring instruments, we couldn’t successfully handle our techniques when Spark jobs slowed down or prices spiraled uncontrolled. We had been caught looking via cryptic logs, making educated guesses about useful resource allocation, and watching our engineering groups spend hours on handbook tuning that ought to have been automated. That’s why we constructed one thing higher: an in depth metrics framework designed particularly for Spark’s distinctive challenges. It is a visibility system that offers us granular insights into utility conduct, useful resource utilization, and job-level efficiency patterns we by no means had earlier than. We’ve achieved 30–50% value reductions and 40–60% sooner job completion occasions. That is actual operational effectivity that straight interprets to raised service for our customers and vital financial savings for our infrastructure price range. On this submit, we stroll you thru precisely how we constructed this framework, the important thing metrics that made the distinction, and the way your workforce can implement comparable monitoring to remodel your personal Spark operations.

Why complete Spark monitoring issues

In enterprise environments, poorly optimized Spark jobs can waste 1000’s of {dollars} in cloud compute prices, block essential information pipelines affecting downstream enterprise processes, create cascading failures throughout interconnected information workflows, and influence service stage settlement (SLA) compliance for time-sensitive analytics.

The monitoring framework we’re inspecting captures over 40 distinct metrics throughout 5 key classes, offering the granular insights wanted to stop these points.

How we ingest, course of, and act on Spark metrics

To handle the challenges of managing Spark at scale, we developed a customized monitoring and optimization pipeline—from metric assortment to AI-assisted tuning. It begins with our in-house Spark listener framework, which captures over 40 metrics in actual time throughout Spark functions, jobs, levels, and duties whereas pulling essential operational context from instruments similar to Apache Airflow and Apache Hadoop YARN.

An Apache Airflow-orchestrated Spark SQL pipeline transforms this information into actionable insights, surfacing efficiency bottlenecks and failure factors. To combine these metrics into the developer tuning workflow, we expose a metrics software and a customized immediate via our inner analytics mannequin context protocol (MCP) server. This allows seamless integration with AI-assisted coding instruments similar to Cursor or Claude Code.

The next is the listing of instruments used for our Spark monitoring resolution, which incorporates metric assortment to AI-assisted tuning:

The result’s quick, dependable, deterministic Spark tuning with out the guesswork. Builders get environment-aware suggestions, automated configuration updates, and ready-to-review pull requests.

Deep dive into Spark metrics assortment

On the heart of our real-time monitoring resolution lies a customized Spark listener framework that captures thorough telemetry throughout the Spark lifecycle. Spark’s built-in metrics are sometimes coarse, quick‑lived, and scattered throughout the person interface (UI) and logs, which leaves 4 essential gaps:

1. Constant historic report
2. Weak linkage from functions to jobs to levels to duties
3. Restricted context (person, cluster, workforce)
4. Poor visibility into patterns similar to skew, spill, and retries

Our expanded listener framework closes these gaps by unifying and enriching telemetry with setting and configuration tags, constructing a sturdy, queryable historical past, and correlating occasions throughout the execution graph. It explains why duties fail, pinpoints the place reminiscence or CPU strain happens, compares supposed configurations to precise utilization, and produces clear, repeatable tuning suggestions so groups can baseline conduct, reduce waste, and resolve points sooner. The next structure diagram illustrates the circulation of the Spark metrics assortment pipeline.

Spark listener

Our listener framework captures Spark metrics at 4 distinct ranges:

1. Utility metrics: Total utility success/failure charges, complete runtime, and useful resource allocation
2. Job-level metrics: Particular person job period and standing monitoring inside an utility
3. Stage-level metrics: Stage execution particulars, shuffle operations, and reminiscence utilization per stage
4. Activity-level metrics: Particular person process efficiency for deep debugging situations

The next Scala instance code reveals the SparkTaskListener extends the category SparkListener to seize detailed task-level metrics:

class SparkTaskListener(conf: SparkConf) extends SparkListener {
 val taskToStageId = new mutable.HashMap[Long, Int]()
 val stageToJobID = new mutable.HashMap[Int, Int]()
 non-public val emitter: Emitter = getEmitter(conf)
  override def onTaskStart(taskStart: SparkListenerTaskStart): Unit = {
   taskToStageId += taskStart.taskInfo.taskId -> taskStart.stageId 
 }
 override def onTaskEnd(taskEnd: SparkListenerTaskEnd): Unit = {
   val taskInfo = taskEnd.taskInfo
   val taskMetrics = taskEnd.taskMetrics
   val jobId = stageToJobID.apply(taskToStageId.apply(taskInfo.taskId))
   val metrics = Map[String, Any](
     "event_type" -> "task_metric",
     "job_id" -> jobId,
     "task_id" -> taskInfo.taskId,
     "period" -> taskInfo.period,
     "executor_run_time" -> taskMetrics.executorRunTime,
     "memory_bytes_spilled" -> taskMetrics.memoryBytesSpilled,
     "bytes_read" -> taskMetrics.inputMetrics.bytesRead,
     "records_read" -> taskMetrics.inputMetrics.recordsRead
     // further metrics.....
   )
   emitter.report(convertToJson(metrics))
 }
}

Actual-time streaming to Kafka

These metrics are streamed in actual time to Kafka as JSON-formatted telemetry utilizing a versatile emitter system:

class KafkaEmitter(conf: SparkConf) extends Emitter {
     non-public val dealer = conf.get("spark.customized.listener.kafkaBroker", "")
     non-public val subject = conf.get("spark.customized.listener.kafkaTopic", "")
     non-public var producer: Producer[String, Array[Byte]] = _
     override def report(str: String): Unit = {
         val message = str.getBytes(StandardCharsets.UTF_8)
         producer.ship(new ProducerRecord[String, Array[Byte]](subject, message))
     }
}

From Kafka, a downstream pipeline ingests these data into an Apache Iceberg desk.

Context-rich observability

Past normal Spark metrics, our framework captures important operational context:

• Airflow integration: DAG metadata, process IDs, and execution timestamps
• Useful resource monitoring: Configurable executor metrics (heap utilization, execution reminiscence)
• Atmosphere context: Cluster identification, person monitoring, and Spark configurations
• Failure evaluation: Detailed error messages and process failure root causes

The mix of thorough metrics assortment and real-time streaming has redefined Spark monitoring at scale, laying the groundwork for highly effective insights.

Deep dive into Spark metrics processing

When uncooked metrics—usually containing hundreds of thousands of data—are ingested from varied sources, a Spark SQL pipeline transforms this high-volume information into actionable insights. It aggregates the info right into a single row per utility ID, considerably lowering complexity whereas preserving key efficiency indicators.

For consistency in how groups interpret and act on this information, we apply the 5 Pillars of Spark Monitoring, a structured framework that turns uncooked telemetry into clear diagnostics and repeatable optimization methods, as proven within the following desk.

AI-powered Spark tuning

The next structure diagram illustrates the usage of agentic AI instruments to research the aggregated Spark metrics.

To combine these metrics right into a developer’s tuning workflow, we construct a customized Spark metrics software and a customized immediate that any agent can use. We use our present analytics service, a homegrown net utility that customers can question our information warehouse with, construct dashboards, and share insights. The backend is written in Python utilizing FastAPI, and we expose an MCP server from the identical service by utilizing FastMCP. By exposing the Spark metrics software and customized immediate via the MCP server, we make it attainable for builders to attach their most popular assisted coding instruments (Cursor, Claude Code, and extra) and use information to information their tuning.

As a result of the info uncovered by the analytics MCP server is perhaps delicate, we use Amazon Bedrock in our Amazon Net Companies (AWS) account to supply the muse fashions to our MCP purchasers. This retains our information safer and facilitates compliance as a result of it by no means leaves our AWS setting.

Customized immediate

To create our customized immediate for AI-driven Spark tuning, we design a structured, rule-based format that encourages extra deterministic and standardized output. The immediate defines the required sections (utility overview, present Spark configuration, job well being abstract, useful resource suggestions, and abstract) for consistency throughout analyses. We embody detailed formatting guidelines, similar to wrapping values in backticks, avoiding line breaks, and implementing strict desk constructions to keep up readability and machine readability. The immediate additionally embeds express steerage for deciphering Spark metrics and mapping them to really useful tuning actions primarily based on greatest practices, with clear standards for standing flags and influence explanations. The immediate signifies that the AI’s suggestions may be traced, reproduced, and actioned primarily based on the offered information by tightly controlling the input-output circulation and trying to stop hallucinations.

Ultimate outcomes

The screenshots on this part present how our software carried out the evaluation and offered suggestions. The next is a efficiency evaluation for an present utility.

The next is a suggestion to cut back useful resource waste.

The influence

Our AI-powered framework has essentially modified how Spark is monitored and managed at Slack. We’ve reworked Spark tuning from a high-expertise, trial-and-error course of into an automatic, data-backed normal by shifting past conventional log-diving and embracing a structured, AI-driven method. The outcomes communicate for themselves, as proven within the following desk.

Conclusion

At Slack, our expertise with Spark monitoring reveals that you simply don’t have to be a efficiency skilled to realize distinctive outcomes. We’ve shifted from reacting to efficiency points to stopping them by systematically making use of 5 key metric classes.

The numbers communicate for themselves: 30–50% value reductions and 40–60% sooner job completion occasions characterize operational effectivity that straight impacts our skill to serve hundreds of thousands of customers worldwide. These enhancements compound over time as groups construct confidence of their information infrastructure and may give attention to innovation reasonably than troubleshooting.

Your group can obtain comparable outcomes. Begin with the fundamentals: implement complete monitoring, set up baseline metrics, and decide to steady optimization. Spark efficiency doesn’t require experience in each parameter, but it surely does require a robust monitoring basis and a disciplined method to evaluation.

Acknowledgments

We need to give our due to all of the individuals who have contributed to this unbelievable journey: Johnny Cao, Nav Shergill, Yi Chen, Lakshmi Mohan, Apun Hiran, and Ricardo Bion.

In regards to the authors