{"id":2099,"date":"2023-12-21T10:40:56","date_gmt":"2023-12-21T10:40:56","guid":{"rendered":"https:\/\/i-spark.nl\/?p=2099"},"modified":"2026-01-02T11:36:20","modified_gmt":"2026-01-02T11:36:20","slug":"the-migration-from-aws-with-redshift-to-databricks-for-e-commerce-company-wwl","status":"publish","type":"post","link":"https:\/\/i-spark.nl\/en\/blog\/the-migration-from-aws-with-redshift-to-databricks-for-e-commerce-company-wwl\/","title":{"rendered":"The migration from AWS with Redshift to Databricks for e-commerce company WWL"},"content":{"rendered":"

The i-spark team is actively engaged with World Wide Lighting [WWL], continuing their multi-year collaboration as a vital part of the data team using i-spark\u2019s Data Team-as-a-Service solution<\/a>. About a year ago, the e-commerce lighting company WWL reached a point where it had to choose between investing in its current data platform or switching to a modern cloud solution that could scale with the growth of the business. WWL aimed to enhance the reliability, stability, and speed of their data processes, including clickstream and ERP data, from their 24 diverse international webshops, while maintaining acceptable costs. This consultancy role and later the execution role around this issue was a perfect fit for i-spark.<\/span><\/p>\n

Their existing data platform was based on an Amazon Redshift data warehouse within AWS. However, they experienced challenges in reliability and resiliency, due to high demand and usage of the platform. Investing in the current platform would inevitably encounter the same challenges sooner or later, ultimately leading to possible large investments with an uncertain outcome. Therefore, the decision to migrate to a modern data platform was made.<\/span><\/p>\n

Tool Selection<\/h2>\n

I-spark explored multiple cloud solutions and Databricks<\/a><\/strong> turned out to be the best fit for WWL\u2019s requirements, as this platform was more comprehensive. <\/span>Databricks also had an advantage that it\u2019s accessible to a broader group of users such as analysts due to useful features such as the SQL editor. <\/span>
\n<\/span>The many processes that ran on the AWS platform, such as services like Lambda, Glue, and ECS that were used, were perfect candidates for migration to Databricks. Additionally, Databricks offered more solutions for Machine Learning, which was also one of WWL’s future aspirations. After setting up a business case as a Proof of Concept confirmed: Databricks was indeed a viable solution for enhancing reliability, stability, and speed of their data processes.<\/span><\/p>\n

Solution Architecture and Roadmap<\/h2>\n

A fair share of time was invested in this phase, because well begun is half done.<\/span>
\n<\/b>In the architecture<\/strong><\/a>, we laid the foundation of the future platform based on the learnings we had from the previous data platform. Therefore we gathered both functional and non-functional requirements from the business and translated these to a comprehensive plan for the architecture, including a design for the flow and storage of the large volumes of data. This resulted in a design based on the MACH-architecture principles using a medallion structure for the storage of data. MACH means that it is built around microservices, which are specific jobs for specific tasks, using API\u2019s where possible, being cloud-native and headless, which means that other tools, such as reporting or marketing automation, have (curated) access to the data in and the compute power of Databricks.<\/span><\/p>\n

As the next step, we determined the best approach for the delivery of the project, the allocation of hours amongst the team and divided the project in multiple phases.\u00a0<\/span>The result of this was a project roadmap where the realization of the architecture was mapped on a timeline using the phased approach.<\/span><\/p>\n

“i-sparks approach and good preparation ensured that the delivery of the final solution was done in under two days”<\/strong><\/p><\/blockquote>\n

Development<\/h2>\n

The roadmap guided the initial development. It began with migrating the processes that handled the most data or were the most complex in terms of processing and setup. While starting with a small team of Data Engineers for the first steps, this was gradually expanded to eventually a multidisciplinary team of Data Engineers<\/a><\/strong>, Analytics Engineers and Data Analysts<\/a><\/strong> when more and more data became available. The Data Engineers were assigned processes concerning data sources they were already familiar with, and worked in parallel on multiple flows to make this part of the project as efficient as possible. Analytics Engineers and Data Analysts worked on a more waterfall-based principle due to strong dependencies between data. The project was delivered through a ‘Big Bang’ release, following a few weeks of operating the new platform alongside the existing one. This parallel run was conducted to validate data and address initial issues.<\/span><\/p>\n

This approach and good preparation ensured that the delivery of the final solution was done in under two days. Any minor errors were immediately resolved, as the major ones had been prevented.<\/span><\/p>\n

Data History<\/h2>\n

A challenge during the engineering process was migrating some of the existing data from Redshift to Databricks. We did this by exporting the historical data as files and later importing them back into Databricks. This imported history was then merged with the new data. Both historical and newly collected data could be retrieved in the same way in the analysis tool, Looker.<\/span><\/p>\n

Looker<\/h2>\n

Since the source of data for Looker<\/a> <\/strong>was changed from Redshift to Databricks, a new project within Looker was necessary. Looker is not able to connect to multiple data sources in one project. Therefore, in Looker, all existing explores were first rebuilt and converted to a new project by our Data Analysts. We also took the opportunity to move logic from Looker to dbt, to make future development and data management easier and more resilient. During the migration we encountered a few challenges:<\/span><\/p>\n