When Compute Sits and Waits: Fixing the Hidden Storage Bottleneck in EDA and HPC with Azure NetApp Recordsdata

Hi there Of us!

If in case you have ever stared at an HPC pipeline and puzzled why the queue depth retains climbing whereas each CPU graph seems to be lazy, this session from the Microsoft Azure Infrastructure Summit 2026 goes to really feel very acquainted. Ron Hogue from the Azure Storage group and Ranga Sankar from NetApp spend twenty-seven minutes diagnosing the issue that no person desires to confess out loud, that the bottleneck just isn’t compute, not the scheduler, and never licenses. It’s storage. After which they stroll by means of precisely how Azure NetApp Recordsdata (ANF) solves it.

📺 Watch the session:

In the event you assist a semiconductor, simulation, rendering, or scientific computing group, you’ve gotten lived this story. Engineers ask for extra cores. You purchase them. Wall-clock occasions barely transfer. Managers then ask for extra software licenses, as a result of certainly that have to be the constraint. Spoiler, it normally just isn’t.

Right here is why this issues to anybody operating shared infrastructure on Azure:

• EDA and HPC pipelines hammer storage with thousands and thousands of tiny reads and writes, metadata operations like file creation, rename, and unlink, all occurring concurrently throughout 1000’s of processes.
• That sample breaks generic cloud file storage that was tuned for giant sequential I/O.
• ANF brings the identical NetApp ONTAP knowledge companies that EDA groups have used on-premises for many years, delivered as a local Azure service.
• You may transfer design environments to Azure with out re-architecting instruments, schedulers, or scratch path conventions.
• The economics modified within the final twelve months with the Versatile service stage and Cool Entry, so the previous “ANF is simply too costly for scratch” argument deserves a recent look.

In brief, in case your job entails retaining costly engineers and costly licenses busy, the storage layer deserves your consideration.

ANF is a first-party Azure service operating NetApp ONTAP on bare-metal infrastructure inside Azure datacenters. It speaks NFS v3, NFS v4.1, and SMB, with dual-protocol choices, and it preserves the file semantics that EDA instruments assume. Issues like POSIX permissions, quick metadata operations, snapshots, and constant low latency.

Ron and Ranga framed the session in eight layers (Downside, Basis, Accelerator, Scale, AI Prepared, Guardrails, Optimize, Proof). Three items did many of the heavy lifting.

Basis, Migration Assistant with SnapMirror. That is the way you get the information into Azure with out rewriting your stock. SnapMirror replicates from on-premises ONTAP or Cloud Volumes ONTAP into ANF whereas preserving metadata, permissions, snapshots, and listing construction, with steady synchronization and minimal downtime. For EDA flows the place a lacking ACL can invalidate a whole run, that constancy just isn’t optionally available.

Accelerator, Cache Volumes. Constructed on NetApp FlexCache, these volumes entrance your authoritative dataset (whether or not it lives in on-prem ONTAP or in Cloud Volumes ONTAP) and pull scorching reads near your Azure compute. Device libraries, PDKs, shared reference knowledge, all served at sub-millisecond latency with out copying petabytes round. There’s nonetheless one supply of reality, which retains your knowledge governance story clear.

Scale, Giant Volumes with Breakthrough mode. A single ANF massive quantity in Breakthrough mode scales to 2 PiB and delivers throughput within the tens of GiB per second by fanning I/O throughout six storage endpoints. That allows you to collapse sharded namespaces (the basic /proj1, /proj2, /proj3 cut up that no person loves) into fewer high-throughput volumes that behave predictably beneath rivalry.

Cache Volumes are the FlexCache sample most ONTAP prospects already know. You peer a cluster, level a cache at an origin, and the cache propagates knowledge on demand. Ron demoed this reside: cluster peering with on-prem ONTAP, a cache quantity created in ANF, and reads served from cache whereas the authoritative copy stayed residence.

Giant Volumes in Breakthrough mode are the place the structure will get fascinating. As an alternative of a single mount level pinned to a single storage endpoint, Breakthrough mode exposes six storage endpoints for one logical quantity. Purchasers can mount, stability I/O, and mixture throughput throughout all six. Microsoft revealed Linux scale-out benchmarks displaying a single 50 TiB massive quantity in Breakthrough mode sustaining roughly 50,000 MiB/s of sequential reads and approaching 1.8 million 8 KiB random learn IOPS utilizing twelve VMs (see the Microsoft Study hyperlink in Assets).

For shared environments, ANF added consumer and group quotas with real-time consumption reporting and onerous limits. Ranga demoed quota guidelines that stopped a runaway simulation producing thousands and thousands of scratch recordsdata from ravenous the remainder of the group. If in case you have ever needed to ship the “who stuffed the scratch quantity” e-mail at 2am, this function alone would possibly justify the journey.

On the price facet, the Versatile service stage decouples capability from throughput. You purchase a capability pool, you choose throughput independently, with 128 MiB/s of baseline throughput included and a ceiling of as much as 640 MiB/s per provisioned TiB (which is roughly 5 occasions the Extremely service stage). Cool Entry transparently tiers chilly blocks to Azure Blob behind the identical file mount level.

In brief, you cease paying for throughput you don’t want on archival volumes, and also you cease overprovisioning capability to chase throughput on small scratch volumes.

The Proof part closed with SPEC Storage 2020 EDA Blended outcomes which can be price studying fastidiously:

• A single massive quantity in Breakthrough mode sustained 2,880 EDA jobsets at about 0.51 ms general response time.
• Six volumes scaled linearly to 17,280 jobsets at about 0.60 ms.
• FIO measurements approached 2 million IOPS.

The trustworthy tradeoff: these numbers come from a benchmark, not your setting, and the SPEC tables present latency climbing on the highest load factors. So deal with the end result as proof that the platform behaves predictably beneath EDA-style concurrency, not as a assure for each workflow. That predictability is the half that issues. EDA leads don’t lose sleep over peak throughput, they lose sleep over latency that drifts when concurrency rises.

Sensible eventualities the place this lands:

• Burst regression and verification runs to Azure throughout tape-out crunches, with Cache Volumes retaining your on-prem golden tree authoritative.
• Full migration of EDA environments to Azure for groups whose datacenters are out of capability or out of lease.
• HPC simulation workloads (CFD, climate, seismic, life sciences) that share the identical metadata-heavy I/O profile.
• AI-adjacent pipelines that must learn design knowledge with file semantics and floor the identical bytes as objects to Material, OneLake, or Databricks through the twin file and object entry sample Ron talked about within the AI Prepared part.

You do not want a multi-quarter undertaking to get a helpful pilot transferring.

1. Register the Azure NetApp Recordsdata useful resource supplier in your goal subscription and request quota within the areas you care about.
2. Arise a small capability pool, begin with the Versatile service stage so you’ll be able to dial throughput independently.
3. Create a check quantity, mount it from a consultant VM (HBv4 or Ev5 household are good beginning factors for HPC and EDA respectively).
4. Run your actual workload, not simply FIO. Use a consultant regression batch or simulation job and watch the metadata patterns within the ANF metrics blade.
5. If in case you have on-premises ONTAP, peer a Cache Quantity towards it to check FlexCache habits together with your precise datasets earlier than committing to a full migration.
6. Layer in consumer and group quotas earlier than you open the quantity to a wider group. Belief me on this one.

Catch the complete Microsoft Azure Infra Summit 2026 session playlist right here.

Cheers!

Pierre Roman