AI-Driven Data Engineering: Automating Data Quality, Lineage, And Transformation In Cloud-Scale Platforms

Abstract

In this book, we examine the need for automated data engineering. As organizations accelerate their adoption of cloud-scale analytics platforms, they increasingly rely on centralized core teams to deliver trusted data domains quickly and publish them for consumption by data analysts and business users. However, as the complex network of data domains increases, the cost and time to create and maintain them become unmanageable. Moreover, tensions grow between the core teams and the business teams who are most dependent on timely and relevant data, due to the long turnaround times for updates and the low tolerance for data quality issues.
Increasingly, organizations are recognizing the need for self-service data democratization to empower users close to business operations to curate their data domains. These business users best understand the specifics of their domain and know best how to enrich the quality of the domain. Accordingly, organizations need to ensure that the right tooling is provided to the business user so that they can publish relevant and trusted business data while ensuring central governance and security compliance. Technology must be used to ensure high data quality at scale and mitigate data usage and compliance risks. This need has recently become the focus of renewed efforts by cloud provider companies and third parties. These companies are investing in solutions that automate many parts of the data engineering process. Other new companies are accelerating the introduction of AI-led autonomous systems. Increasingly, solutions are moving closer to the business users, lowering the barrier for them to build trusted data domains. These market moves represent directions that data engineers should look to thoughtfully embrace as partners, not competitors.