Earlier than making architectural choices, it’s value revisiting the broader migration technique. In our earlier put up, we launched Databricks Skilled Companies’ strategy to advanced knowledge warehouse migrations, highlighting the significance of early choices round technique and design. These foundational selections immediately affect the goal platform’s implementation and structure.
We additionally launched two sequencing methods: ETL-first and BI-first. The BI-first strategy delivers fast worth by modernizing the consumption layer, whereas the ETL-first strategy focuses on upstream pipelines. Every has its place, relying on priorities.
On this put up, we discover one of the essential design selections: selecting between a Raise-and-Shift or Modernization strategy. We clarify what every strategy entails, when to make use of it, and tips on how to merge them right into a hybrid strategy for long-term success on Databricks.
From technique to migration strategy: selecting the correct path
After you’ve aligned on the broader migration technique—ETL-first or BI-first—the subsequent main resolution is tips on how to construction the migration. Do you replicate what exists, or reimagine it for the long run?
This architectural resolution sometimes comes down to 2 core approaches:
- Raise-and-Shift: Transfer workloads as-is to speed up the migration
- Modernization: Redesign the platform to unlock long-term effectivity and scale
The suitable strategy relies on your objectives, constraints, and timeline. Under, we break down the tradeoffs of every and embrace a hybrid mannequin that many organizations use to mix the perfect of each.
Raise and shift migration
Raise-and-Shift includes shifting your present knowledge fashions and codebase to the brand new platform with minimal adjustments. You don’t introduce new use instances, and the structure stays unchanged.
This strategy is interesting as a result of it’s simpler to scope, plan, and automate. Instruments like profilers and code analyzers assist measure workload patterns, complexity, and value, making it simpler to judge and execute.
Key advantages embrace:
- Predictable timelines
- Automated tooling (e.g., code converters, reconciliation validators)
- Sooner migration when dealing with deadlines or expiring licenses
For instance, code converters can robotically deal with as much as 80% of scripts. Since performance stays the identical, validation and working queries on each programs and evaluating outputs are simpler.
On Databricks, Raise-and-Shift will get you off legacy platforms shortly whereas unlocking speedy efficiency good points utilizing options like z-ordering and liquid clustering. After you may have migrated, your group can start incrementally modernizing the platform.
Modernize the migration sample
Modernizing, in distinction to Raise-and-Shift, means constructing a brand new knowledge platform in your goal system with out being constrained by your legacy structure. The main target shifts from merely migrating present property to reimagining use instances and designing for future wants. As an alternative of mapping outdated optimizations, you implement greatest practices and the well-architected pillars of the lakehouse.
On an open lakehouse, this includes refactoring code and re-architecting knowledge buildings to fulfill your group’s present and future scalability, efficiency, price, and functionality necessities, free from legacy limitations.
Tooling stays helpful, however extra for discovery and planning:
- Profilers and code analyzers assist stock what you should migrate
- Code converters and reconciliation instruments play a minimal position, since this isn’t a direct code translation
This strategy is good when you may have versatile timelines and an outdated or overly advanced legacy system, typically with hundreds of tables and scripts. Whereas beginning contemporary can really feel sluggish and overwhelming, the long-term advantages are substantial: simplified structure, higher efficiency, and lowered upkeep overhead.
That mentioned, migrating hundreds of scripts typically means sustaining their upkeep complexity. If that appears daunting, contemplate partnering with Databricks Skilled Companies or licensed migration consultants to assist information the planning and design section and guarantee a smoother path.
A hybrid strategy: carry and shift, after which modernize
One other strategy is a hybrid migration technique that balances pace with long-term worth. You’ll start with the Raise-and-Shift strategy to eliminating their legacy platform as shortly as attainable, particularly when dealing with pressing constraints like expiring licenses. Automation and repeatable tooling assist speed up this preliminary section and scale back danger throughout execution.
You may transfer into the modernization section after you migrate your workloads to Databricks.
Within the hybrid strategy, you:
- Combine new and fashionable knowledge sources
- Implement an information product technique
- Allow superior analytics, AI, and new use instances that drive enterprise choices
This section typically requires architectural updates however lets you evolve steadily. With a hybrid technique, you don’t need to modernize every part on day one—you construct on a steady basis whereas aligning with future necessities.
In case you’re pursuing this strategy, Databricks Skilled Companies and authorized companions may also help information your roadmap, guaranteeing a easy transition and a future-ready structure.
Our standpoint
Choosing a migration strategy isn’t a one-size-fits-all. The most typical strategy is a hybrid migration:
- Create a migration manufacturing facility that leverages automation instruments.
- Raise-and-shift nearly all of the codebase.
- Allow out-of-the-box optimizations, corresponding to z-ordering and liquid clustering, to start out your modernization effort.
Databricks can act as your main knowledge warehouse. For instance, you possibly can migrate saved procedures to notebooks and use SQL Scripting for scalability and AI integration with out leaving the consolation of SQL. Migrating Transact-SQL to every other cloud knowledge warehouse requires an analogous effort to migrating that Transact-SQL to a pocket book with Python code wrapped round your SQL performance. The advantage of utilizing a pocket book is that you just additionally get flexibility and an important growth expertise.
What to do subsequent
Able to modernize your knowledge warehouse? Obtain our eBook, “Remodeling Legacy Knowledge Warehouses: A Strategic Migration Blueprint,” for detailed methods and greatest practices that guarantee a low-risk transition to the Databricks Knowledge Intelligence Platform.
If you’re a DBA, try this weblog on High 10 Suggestions for DBAs Migrating to Databricks.
