Breaking Down Information Silos: A Sensible Framework from the Subject – Atlan

Information silos are a serious roadblock to efficient decision-making, forcing groups to spend extra time discovering and getting ready information slightly than analyzing it. The truth is, IDC reviews that information professionals dedicate practically 80% of their time to data-related duties, leaving little room for precise insights. On this article, we discover an actionable framework designed to encourage a extra related strategy to information administration, highlighting real-world examples and sensible methods.

What are information silos and their influence?

At their core, information silos emerge when data is remoted inside two main mechanisms: folks and know-how. 

Departmental boundaries and organizational constructions naturally create obstacles to data sharing. Moreover, the rise of specialised instruments and platforms means vital enterprise information typically stays trapped in disconnected techniques.

And the influence of those silos? These are a number of the frequent points clients have shared with us. 

• Time and Effectivity Impacts: Information silos drive groups to spend days or perhaps weeks on easy information requests, and may trigger widespread duplication of labor as completely different teams unknowingly deal with the identical tasks. As information volumes develop, backlogs increase quicker than groups are in a position course of, creating mounting effectivity issues.
• Information High quality and Belief Points: A number of variations of the identical information emerged throughout completely different techniques, making it not possible to find out which model was correct and up-to-date. Eroding belief in information and KPIs all through the group. This creates extra effort to confirm information and hesitation to even use it. 
• Scale and Progress Issues: As corporations expanded, centralized groups couldn’t sustain with growing information calls for, resulting in a proliferation of uncoordinated instruments and techniques. Groups started working in isolation, and new applied sciences had been acquired with out correct adoption planning, creating an more and more fragmented information panorama.
• Discovery and Entry Challenges: Useful information belongings turned successfully invisible inside organizations as groups had no centralized approach to discover or entry current information. Customers remained unaware of related information that might assist them, whereas information house owners had no visibility into who was utilizing their information or the way it was being utilized.
• Useful resource and Value Implications: Organizations waste cash on duplicate information storage, redundant instruments, and underutilized techniques whereas inefficiently deploying engineering assets. The hidden prices of information silos manifested in each direct bills and alternative prices from inefficient useful resource allocation.

Regardless of these points, breaking down information obstacles is achievable with the appropriate technique. 

Fixing for information silos requires a twin technique with a 6-part framework

The identical two components that trigger information silos to emerge, folks and know-how, additionally illuminate the technique for dismantling them. Analyzing every space reveals a path ahead for organizations searching for to interrupt down information obstacles.

Let’s begin with the human factor. At its core, the answer begins with fostering the appropriate tradition. When groups perceive how information impacts enterprise selections and their position in proudly owning that information, a mindset emerges the place data is considered as a shared asset slightly than departmental property. 

Creating this tradition would require funding and organizational change. It means educating groups on information literacy, securing cross-functional dedication to collaboration, and gaining sturdy buy-in from management. 

Whereas these modifications are difficult and take time to implement, the end result is transformative: a tradition the place collaboration thrives as a result of everybody shares and has entry to the information they should make enterprise selections.

Addressing know-how is simply as vital as tackling organizational silos. Within the trendy information panorama, data belongings are sometimes scattered throughout varied techniques and platforms. Gross sales groups might depend on Salesforce, whereas advertising groups use HubSpot. Information might be unfold throughout a number of cloud suppliers like Azure and AWS, and even dispersed globally.

With out correct administration, this range of instruments can create chaos. Efficient information administration requires extra than simply connecting disparate techniques. It calls for a complete strategy that ensures information high quality, maintains consistency, and preserves very important context. This implies implementing processes and governance to make sure information is correct, up-to-date, and adheres to organizational requirements. It entails establishing clear possession and accountability for every dataset. Crucially, it additionally necessitates capturing and sharing vital metadata that gives perception into the information’s origins, which means, and lineage. When achieved proper, it permits groups throughout the group to grasp, belief, and successfully make the most of the knowledge they uncover. 

Whereas there’s no common guidelines for fixing information silos, we’ve developed a complete 6-part framework—a robust, adaptable roadmap that organizations can implement and grasp. The success tales of visionary corporations like Autodesk, ContentSquare, Kiwi.com, Nasdaq, Porto, and North assist this strategy and showcase how the framework can navigate distinctive challenges and democratize information throughout completely different organizational contexts.

1. Area Empowerment with a Information Middle of Excellence

Empower groups to be straight answerable for their information by way of a powerful centralized information crew that establishes the muse, requirements, and instruments for trusted information merchandise.

At Autodesk, by early 2021, their central Analytics Information Platform crew confronted an amazing backlog of information ingestion requests that exceeded all beforehand ingested information within the platform’s historical past. They solved this problem by implementing a collaborative area possession mannequin, one the place groups had been empowered to publish and handle their information merchandise whereas working inside a shared governance framework. This ensured interoperability and accessibility throughout the group. Their platform crew established frequent requirements, instruments, and processes that allowed 60 area groups to independently construct information merchandise whereas making them discoverable and usable by others by way of a central catalog. This balanced strategy led to the profitable supply of 45 new use instances in simply two years whereas guaranteeing information remained accessible and helpful throughout crew boundaries.

2. Clear Governance Construction

Present the framework for a way information is managed, documented, and shared throughout a corporation. 

Main corporations have applied numerous but efficient governance frameworks to handle their information belongings: 

• Contentsquare applied a hybrid possession mannequin the place their Info Methods Division (ISD) maintained system-level management whereas enterprise items retained information possession, complemented by information compliance ambassadors embedded throughout departments to make sure constant requirements throughout acquisitions.
• Porto established a tiered governance mannequin, designating belongings as both “Full Governance” (requiring full documentation, classification, high quality checks, and outlined possession) or “Simplified Governance” (needing solely fundamental lineage and cataloging), permitting their small five-person crew to effectively handle over 1 million information belongings. 
• At Nasdaq, they advanced from a centralized reporting construction to a federated mannequin with specialised groups. The Platform Staff maintained core know-how, an Financial Analysis crew centered on information science, and embedded analysts established inside enterprise items, all working inside outlined engagement protocols. 
• Kiwi.com took a product-centric strategy to governance, overlaying possession, documentation, high quality, structure, safety, and processes, which helped them scale back engineering workload by 53% whereas growing information consumer satisfaction by 20%. 

These diversified approaches showcase that efficient governance constructions can take completely different kinds whereas attaining the identical objective: creating clear, scalable frameworks for managing and sharing information throughout advanced organizations.

3. Constructing Belief Via Requirements

Set up constant practices for a way information is created, maintained, and shared inside a corporation. 

Kiwi.com’s huge information panorama of 100 Postgres databases and tens of 1000’s of tables resulted in overwhelming search outcomes for analysts, with easy phrases like “Vacation spot” returning over 200,000 entries. They tackled this by growing a governance framework for information merchandise and establishing requirements round possession, documentation, high quality, structure, safety, and processes. This reworked their strategy from merely storing information to curating 58 fastidiously managed, top-tier information merchandise that met strict standards for reliability and accessibility. Every information product now requires technical and product-level possession, complete documentation, high quality monitoring by way of their observability platform, and formal information contracts specifying SLAs and SLOs between producers and customers. Their central engineering crew noticed a 53% discount in workload as groups turned extra self-sufficient, information consumer satisfaction elevated by 20%, and so they efficiently onboarded over 20 groups to responsibly share and use information throughout the group. This standardized strategy not solely improved effectivity but additionally constructed confidence of their information, enabling groups to make quicker, extra knowledgeable selections with dependable information merchandise.

4. Unified Discovery Layer

Present a single place to seek out, perceive, and entry information throughout a corporation. 

At Nasdaq, groups would go to a number of teams concurrently to get information solutions, with Michael Weiss noting “If you happen to had been a enterprise unit, you may go to considered one of 4 groups to try to get a solution. As a matter of truth, they’d go to all 4 groups to see who would win first.” Their survey revealed that energy customers spent one-third of their time simply attempting to grasp the context round information they already had entry to. They reworked this by implementing Atlan as their unified metadata layer, which customers described as “having Google for our information.” This allowed groups to rapidly uncover current information belongings, perceive their context, and ask the appropriate questions, slightly than duplicating work throughout a number of groups.

5. Automated Governance

Use know-how to constantly apply information administration guidelines and insurance policies at scale. 

At Porto, a five-person information governance crew was manually managing over 1 million information belongings, spending vital time on tedious duties like updating classifications and possession in spreadsheets. They reworked this by implementing Atlan to automate their governance processes. The automation categorized belongings as both “Full” or “Simplified” governance based mostly on preset guidelines, robotically assigned possession based mostly on information domains, and recognized potential PII fields by way of sample matching. This resulted in a 40% discount in time spent on guide governance duties, permitting the crew to deal with extra strategic work. As Danrlei Alves famous, “With the time we saved, we’re growing the scope of labor we now have, and tasks we do. For the guide actions we nonetheless have, we’re discovering a approach to automate them, too.”

6. Related Instruments & Processes 

Create a seamless movement of information work throughout completely different platforms and groups. 

At North, their information crew was fighting inefficient difficulty decision and duplicate work throughout their Snowflake and Sigma environments. Staff members would obtain tickets with out clear references to particular information belongings, typically resulting in a number of engineers unknowingly engaged on the identical points. By integrating Atlan’s Chrome Extension with Jira and Slack, they enabled customers to boost points straight from Sigma with computerized hyperlinks to particular belongings. As Daniel Dowdy defined, “Eliminating duplicate work, or eliminating your engineers engaged on the identical mission with out understanding about it? These effectivity features are large, and the financial savings add up actually rapidly.” The combination additionally supplied engineers with a complete historical past of all tickets and conversations regarding particular information belongings, serving to them higher perceive and forestall comparable previous points.

Your Path Ahead: From Framework to Implementation

The challenges posed by information silos are multi-faceted, however solvable by way of a strategic, coordinated strategy. As we’ve seen, siloed information stems from two core points:

1. Fragmented possession and accountability
2. Lack of unified processes and tooling for discovery, governance and entry

To systematically break these obstacles down, organizations ought to deal with constructing six key capabilities:

1. Set up clear area possession to empower groups
2. Implement sturdy governance constructions and processes
3. Construct belief by way of constant requirements and information contracts
4. Create unified discovery and exploration layers
5. Automate governance duties to make sure scalability
6. Join instruments and workflows to allow seamless collaboration

The ability of this framework has been confirmed out by pioneering information organizations. North American Bancard struggled with scattered information and guide governance holding them again—making use of automated governance with Atlan drove a 700% enhance in managed information belongings. Autodesk handled information trapped in crew silos, however by rolling out standardized processes and a central discovery layer, they broke obstacles between domains. Contentsquare lacked dependable entry to trusted information, by way of structuring governance and possession, they unlocked fluid, safe information sharing.

Every of those successes validates how a scientific strategy to dismantling information silos can speed up a corporation’s progress towards changing into really data-driven. Whereas the journey takes dedication, the vacation spot is evident: empowered groups seamlessly accessing and enriching a shared pool of trusted information to drive innovation.

Able to discover how your group can operationalize this framework to attain data-driven transformation? We’d love that can assist you map out the appropriate path ahead. Guide a demo with our crew to see how Atlan will help you break down silos and democratize your information.