How Socure achieved 50% value discount by migrating from self-managed Spark to Amazon EMR Serverless

Socure is likely one of the main suppliers of digital identification verification and fraud options. Its predictive analytics platform applies synthetic intelligence (AI) and machine studying (ML) methods to course of each on-line and offline intelligence, together with government-issued paperwork, contact data (e-mail, telephone, handle), private identifiers (DOB, SSN), and gadget or community information (IP, velocity) to confirm identities precisely and in actual time.

Socure ID+ is an identification verification platform that makes use of a number of Socure choices comparable to KYC, SIGMA, eCBSV. Telephone Threat and extra. It has two environments centered on proof of idea (POC) and stay clients. The Information Science (DS) surroundings is designed for the POC or proof of worth (POV) stage. On this surroundings, clients present datasets through SFTP, that are processed by Socure’s information scientists by means of an inner endpoint. The information undergoes ML-based scoring and different intelligence calculations relying on the chosen modules and processed outcomes are saved in Amazon Easy Storage Service (Amazon S3) in delta open desk format . Within the Manufacturing (Prod) surroundings, clients can confirm identities both in actual time by means of stay endpoints or through a batch processing interface.

Socure’s information science surroundings features a streaming pipeline known as Transaction ETL (TETL), constructed on OSS Apache Spark operating on Amazon EKS. TETL ingests and processes information volumes starting from small to giant datasets whereas sustaining high-throughput efficiency.

The first objective of this pipeline is to present information scientists a versatile surroundings to run POC workloads for purchasers.

Information scientists…

• set off ingestion of POC datasets, starting from small batches to large-scale volumes.
• devour the processed outputs written by the pipeline for evaluation and mannequin growth.
• share the outcomes with Socure’s clients.

The next diagram exhibits the Transaction ETL (TETL) structure.

This pipeline straight helps buyer POCs, guaranteeing that the best information is offered for experimentation, validation, and demonstration. As such, it’s a crucial hyperlink between uncooked information and customer-facing outcomes, making its reliability and efficiency important for delivering worth. On this put up, we present how Socure was in a position to obtain 50% value discount by migrating the TETL streaming pipeline from self-managed spark to Amazon EMR serverless.

Motivation

As information volumes have scaled by 10x, a number of challenges like latency and information reliability have emerged that straight affect the shopper expertise:

• Efficiency points attributable to inefficient autoscaling main to extend in latency as much as 5x
• Excessive operational value of sustaining an OSS Spark surroundings on EKS

Moreover, we now have recognized different necessary points:

• Useful resource constraints attributable to occasion provisioning limits, forcing using smaller nodes. This results in frequent spark executor out of reminiscence (OOM) failures underneath heavy masses, rising job latency and delaying information availability.
• Efficiency bottlenecks with Delta Lake, the place giant batch operations comparable to OPTIMIZE compete for sources and decelerate streaming workloads.

Throughout this migration, we additionally took the chance to transition to AWS Graviton, enabling further value efficiencies as defined on this put up.

With these two main drivers we started exploring different structure utilizing Amazon EMR. We already dd intensive benchmarking on a number of identification verification associated batch workloads on completely different EMR platforms and got here to the conclusion that Amazon EMR Serverless (EMR-S) presents a path to scale back operational value, enhance reliability, and higher deal with large-scale batch and streaming workloads; tackling each customer-facing points and platform-level inefficiencies.

The brand new pipeline structure

The information processing pipeline follows a two-stage structure the place streaming information from Amazon Kinesis Information Stream first flows into the uncooked layer, which parses incoming information into giant JSON blobs, applies encryption, and shops the leads to append-only Delta Tables. The processed layer consumes information from these uncooked Delta tables, performs decryption, transforms the information right into a flattened and vast construction with correct area parsing, applies particular person encryption to personally identifiable data (PII) fields, and writes the refined information to separate append-only Delta Tables for downstream consumption.

The next diagram exhibits the TETL earlier than/after structure we applied, transitioning from OSS Spark on EKS to Spark on EMR Serverless.

Benchmarking

We benchmarked end-to-end pipeline efficiency throughout OSS Spark on EKS and EMR Serverless. The analysis centered on latency and price underneath comparable useful resource configurations.

Useful resource Configuration

EKS (OSS Spark):

• Min 30 executors
• Max 90 executors
• 14 GB reminiscence / 2 cores per executor

EMR Serverless:

• Min 10 executors
• Max 30 executors
• 27 GB reminiscence / 4 cores per executor
• Successfully ~60 executors when normalized for 2x reminiscence and cores, designed to mitigate the OOM points described earlier.

Observations

• Autoscaling Effectivity: EMR Serverless scaled down successfully to twenty staff on common over the weekend (low site visitors day), leading to decrease prices as much as 12% in comparison with weekday.
• Executor Sizing: Bigger executors on EMR Serverless prevented OOM failures and improved stability underneath load.

Definitions

• Price: It’s the service value for each uncooked & processed jobs from the AWS Price Explorer.
• Latency: Finish-to-end latency measures the time from Socure ID+ occasion technology till information arrives within the processed delta desk, calculated as Inserted Date minus Occasion Date.

Outcomes

The values within the following desk characterize share enhancements noticed when operating on EMR in comparison with EKS.

Observe: Even with a conservative 40% value discount utilized to the EKS surroundings to account for Graviton, EMR-S stays roughly 15% cheaper.

The benchmarking outcomes clearly exhibit that EMR Serverless outperforms OSS Spark on EKS for our end-to-end pipeline workloads. By transferring to EMR Serverless, we achieved:

• Improved efficiency: Common latency diminished by greater than 50%, with persistently decrease min and max latencies.
• Price effectivity: General pipeline execution prices dropped by greater than half.
• Scalability: Autoscaling optimized useful resource utilization, additional reducing value throughout off-peak intervals.
• Operational overhead: EMR-S absolutely managed and serverless nature eliminates the necessity to keep EKS and OSS Spark.

Conclusion

On this put up, we confirmed how Socure transitioning to EMR Serverless not solely resolved crucial points round value, reliability, and latency, but additionally offered a extra scalable and sustainable structure for serving buyer POCs successfully, enabling us to ship outcomes to clients quicker and strengthen our place for potential customized contracts.

In regards to the authors