Big Data Migration From Hadoop to Databricks

• The project is an unlimited subscription service that provides access to a wide range of books, audiobooks, news and magazine articles, documents, and more. Subscribers can get millions of titles through the platform on their iPhone, iPad, Android, Fire device, or web browser. This makes it easy to access titles wherever and whenever they want. Besides, users can save their favorite titles, create collections, bookmark titles and build the library of their dreams.

The platform uses analytic tools and recommendation engines to help people discover the millions of titles available on its platform, encouraging its users to read without limits by providing content according to their interests.

Data is a core of the recommendation engine, and the main issue here is the inability to process the platform’s massive data sets using legacy technologies. Due to their outdated on-premises Hadoop system, Data Team faced a lack of access control to production data, difficulties with platform scalability, high support requirements, and hefty maintenance costs.

Thus, the Data Team at the company had the vision to unify and democratize their data and move it all to the cloud. As a new solution required to implement an advanced tech stack, company sought high-skilled devs to enhance its Data Team.

The involvement of an experienced team would significantly increase the effectiveness of development, improve the quality of the product, and minimize the time for platform delivery. Since the previous project’s stack had hefty maintenance costs, it was also crucial to reduce development and maintenance costs.

The main challenges:

• Developing a cutting-edge, easy-to-use, and maintained platform for Data Scientists and Data Analysts.
• Providing smooth platform migration from Hadoop to Cloud solution without interrupting client’s business processes.
• Ensuring platform data security while hundreds of Data specialists are working at the same time.
• Achieving product conformity to the audit requirements.
• Reducing maintenance costs while enhancing operational efficiency.

Solutions:

• Databricks on AWS to unify all analytics and AI workloads in one place, benefitting both data warehouses and data lakes

• Delta Lake to provide the foundation for storing data and tables

• Apache Airflow to allow Data specialists to build, monitor and troubleshoot pipelines with ease

Integrating cost-effective infrastructure Databricks on AWS and the AirFlow orchestrator, which has replaced a previous custom solution on Ruby, saves thousands of dollars monthly. Sure, our team did not stop there and continued our partnership with client.

We are currently working together on the optimization of a delivered platform to make it even more cost-effective. Our main task is to reduce data warehouse costs as data amounts keep growing.