Whats up Of us!
Welcome again to the ITOpsTalk Weblog and the Microsoft Azure Infrastructure Summit 2026 collection. On this session James Baker and Sai Runtham, each from the Azure Information Lake Storage product staff, take us via what a contemporary Lakehouse truly is, how one can design one on Azure, after which they roll up their sleeves and construct one finish to finish. If in case you have been listening to “Lakehouse” thrown round in structure opinions and weren’t 100% positive what it modifications for you as an IT Professional, this one is for you.
📺 Watch the session:
You is likely to be considering, “I run infrastructure, not analytics.” Honest level. However right here is the factor. The lakehouse is more and more the platform your online business will run BI dashboards, AI brokers, and resolution help techniques on, and you’re the one who has to maintain the info secure, ruled, and reachable.
Here’s what is in it for you:
- It’s a platform dialog. James spends an enormous chunk of the session on horizontal platform capabilities (storage, catalog, identification, secrets and techniques, community, coverage) versus vertical pipeline considerations. That’s squarely an IT Professional downside.
- Information is the asset. Workspaces, question engines, and dashboards are transient. The info lives eternally, and defending it’s in your plate.
- Governance is what stops your knowledge lake from rotting into an information swamp.
- Scale is a virtuous cycle. Extra knowledge drives extra perception, which drives extra knowledge. Your platform can not turn out to be the ceiling.
- AI brokers are the brand new customers. They don’t simply learn dashboards, they question gold tables instantly. Your community, identification, and entry controls should sustain.
An information lakehouse is strictly what it appears like. You are taking a budget, versatile, schema-light scale of an information lake, and also you fuse it with the low-latency question efficiency, replace semantics, and governance of an information warehouse. One copy of the info. One place to manipulate it. No extra forking from the lake right into a warehouse simply to make BI instruments blissful.
Fast distinction:
- Information lake. Massive, low-cost, versatile. No schema enforced on write. Traditionally vulnerable to turning into a swamp.
- Information warehouse. Low-latency queries, updates, robust governance, structured. Hits a scale ceiling and prices extra.
- Information lakehouse. Lake-scale storage, with a high-performance question layer and warehouse-grade governance sitting excessive. No knowledge fork.
The large shift is that the info doesn’t transfer. Your BI dashboards, your AI brokers, your serverless SQL queries, all of them hit the identical ruled tables within the lake. That retains lineage clear and your safety mannequin sane.
James and Sai are clear that the structure is much less a hard and fast diagram and extra a listing of platform capabilities you compose. Right here is the form of it on Azure.
Storage layer (the asset).
- Azure Information Lake Storage Gen2 (ADLS) with hierarchical namespace turned on. That’s non-negotiable for analytics workloads. It offers you atomic listing operations, POSIX-style ACLs, and the efficiency Delta Lake depends on.
- OneLake in Microsoft Cloth if you would like a tenant-wide logical lake that’s constructed on ADLS Gen2 and shops every thing in open Delta Parquet by default.
Desk format and pipelines.
- Open desk codecs: Delta Lake (and Apache Iceberg because it converges) provide you with ACID transactions, time journey, schema evolution, and streaming on low-cost object storage.
- Azure Databricks Lakeflow Declarative Pipelines with Autoloader for incremental ingestion of each batch and streaming sources straight into Delta tables. Autoloader handles new file discovery, schema inference, and evolution for you.
- The medallion structure for stamping out repeatable pipelines:
o Bronze. Uncooked, append-only touchdown zone. Supply of reality.
o Silver. Cleansed, deduplicated, conformed, enriched.
o Gold. Enterprise-ready, aggregated, performance-optimized for consumption.
Governance and identification.
- Unity Catalog as the one supply of reality for catalog, lineage, and fine-grained entry management throughout bronze, silver, and gold.
- Entra ID for identification. Managed identities for compute. Key Vault for secrets and techniques.
- Community safety across the perimeter. The info is the crown jewel, so personal endpoints, firewalls, and VNet-attached compute are baseline.
Consumption layer.
- Energy BI Direct Question in opposition to a serverless SQL warehouse on the gold tables. No knowledge copies, governance flows via.
- AI brokers like Databricks Genie pointed at gold tables. Pure-language questions, dwell lineage, no knowledge motion.
The demo that ties it collectively. Sai walked via an actual pipeline: NYC TLC taxi journeys enriched with NOAA climate and ESPN/MLB sports activities occasions, ingested by Autoloader into bronze, remodeled via silver, aggregated into gold. A parallel streaming pipeline dealt with artificial dwell occasions for a real-time demand view. Energy BI dashboards hit gold through Direct Question. And Genie answered questions like “which zones are most delicate to sport occasions” by mapping demand round Madison Sq. Backyard, with the question and the chart generated for you. All in opposition to the identical lakehouse, no knowledge motion, full lineage.
That is the place loads of lakehouses go sideways. A couple of issues from the session and from the official steerage value pinning to your wall:
- Get hierarchical namespace proper. It’s the distinction between atomic listing operations and “copy then delete,” which is sluggish and costly at scale.
- Use storage tiers and lifecycle insurance policies. Sizzling for working knowledge, Cool or Chilly for older partitions, Archive for compliance retention. Lifecycle guidelines on ADLS do that mechanically.
- Partition and file-size matter. Numerous tiny information kill question efficiency. Use OPTIMIZE, Z-Order, or liquid clustering on Delta tables, and partition on the columns your queries truly filter on.
- Lean on vectorized reads. ADLS plus Delta plus trendy question engines push loads of work right down to columnar Parquet, which retains your compute invoice in test.
- Use serverless SQL warehouses the place it matches. Direct Question in opposition to a serverless endpoint scales compute to demand and allows you to preserve dashboards recent with out import refreshes.
- Observe knowledge, not simply techniques. “Is Databricks up” is important however not adequate. Watch knowledge freshness, row counts, pipeline blockages, and SLAs on the info itself.
- Govern every thing. A well-governed lakehouse drives belief, which drives use, which drives worth. Skipping governance early at all times prices extra later.
If you wish to put palms on a keyboard this week:
- Spin up an Azure Storage account with hierarchical namespace enabled. That’s your ADLS Gen2 basis.
- Rise up an Azure Databricks workspace, allow Unity Catalog, and level it at your ADLS account.
- Create a Lakeflow Declarative Pipeline. Use Autoloader to ingest a pattern dataset (the NYC taxi knowledge is a traditional start line) right into a bronze Delta desk.
- Add silver and gold notebooks or pipelines that clear and mixture the info.
- Join Energy BI to a serverless SQL warehouse in your gold tables with Direct Question.
- If you’re a Cloth tenant, mirror or shortcut knowledge into OneLake and check out the identical sample there, no infra to handle.
- Learn the Hitchhiker’s Information to ADLS earlier than you scale up. It’ll prevent future you loads of grief.
This session is one cease on an enormous tour. The total Microsoft Azure Infrastructure Summit 2026 playlist covers every thing from sovereign cloud and AKS networking to backup, storage, and AI-assisted operations. In case your job touches Azure, there’s something in right here for you.
Head over to the total playlist and binge what is helpful: https://www.youtube.com/playlist?checklist=PLjt5SKzX1iI8con7FJDB56G6hHqxGm7ki
Cheers!
Pierre Roman
