What is Data Lakehouse and other frequently asked questions (2023)

problem index

What is a Data Lake House?
What is a data lake?
What is a data warehouse?
How is a Data Lakehouse different from a Data Warehouse?
How is a Data Lakehouse different from a data lake?
How easy is it for data analysts to use Data Lakehouse?
How does a Data Lakehouse system compare to a data warehouse in terms of performance and cost?
What data governance functions does the Data Lakehouse system support?
Must the Data Lakehouse be centralized or can it be decentralized into a data grid?
How is Data Mesh related to Data Lakehouse?

What is a Data Lake House?

In short, Data Lakehouse is an architecture that enables efficient and secure artificial intelligence (AI) and business intelligence (BI) directly on the massive amounts of data stored in the data lake.

Discover why lakehouses are the data architecture of the futureWith Bill Inmon, the father of data warehousing.

Today, the vast majority of enterprise data resides indata lake, a low-cost storage system that can manage any type of data (structured or unstructured), and has an open interface that any processing tool can run on. These data lakes are where most data transformation and advanced analytics workloads (such as AI) run to leverage the full set of data in an organization. Also, for business intelligence (BI) use cases, proprietary data warehouse systems are used to structure a small subset of data. These data warehouses primarily support BI, using SQL to answer historical analysis questions about the past (for example, what was my income last quarter), while data lakes store large amounts of data and support analysis using SQL and non-SQL interfaces, including predictive analysis and AI (e.g. which of my customers are likely to churn, or what coupons to offer my customers at what time). Historically, to achieve both AI and BI, you had to have multiple copies of data and move it between data lakes and data warehouses.

data lake houseYou can store all your data at once in a data lake and do AI and BI directly on that data. It has specific capabilities to efficiently enable AI and BI on large-scale data across all enterprises. Namely, it has the SQL and performance features (indexing, caching, MPP processing) that make BI run fast on the data lake. It also features direct file access and direct native support for Python, data science, and AI frameworks without being enforced through SQL-based data warehouses. The key technologies used to implement Data Lakehouses are open source, such asdelta lake, Woody And Iceberg. Vendors focused on Data Lakehouse include, but are not limited to, Databricks, AWS, Dremio, and Starburst. Vendors that provide data warehouses include, but are not limited to, Teradata, Snowflake, and Oracle.

More recently, Bill Inmon, widely considered the father of the data warehouse,The Evolution of Data LakehousesExplains lakehouse's unique ability to manage data in an open environment while combining the data science focus of a data lake with the end-user analytics of a data warehouse.

What is a data lake?

A data lake is a low-cost, open, durable storage system for any data type—tabular data, text, images, audio, video, JSON, and CSV. In the cloud, every major cloud provider leverages and promotes data lakes such as AWS S3, Azure Data Lake Storage (ADLS), Google Cloud Storage (GCS). As a result, the vast majority of data for most organizations is stored in cloud data lakes. Over time, most organizations store their data in an open standardized format, usually Apache Parquet format or ORC format. As a result, a large ecosystem of tools and applications can directly consume these open data formats. This approach to storing data in an open format at a very low cost enables organizations to amass large amounts of data in data lakes while avoiding vendor lock-in. At the same time, despite these advantages, the data lake still suffers from three main problems - security, quality and performance. Since all data is stored and managed in the form of files, it does not provide fine-grained access control to file content, but only coarse-grained access control, that is, who can access which files or directories. Query performance is poor because the format is not optimized for fast access and listing files is computationally expensive. In short, organizations end up moving data to other systems to take advantage of it unless the application can tolerate noise (i.e. machine learning). Finally, quality is a challenge because it is difficult to prevent data corruption and manage schema changes as more data is pulled into the data lake. Likewise, ensuring atomic operations when writing to a set of files is challenging, and there is no mechanism for rolling back changes. As a result, many believe that most data lakes end up as data "swamps." .So most organizations move a subset of this data into a data warehouse, which doesn't have these three problems, but suffers from others.

What is a data warehouse?

A data warehouse is a proprietary system dedicated to storing and managing structured or semi-structured (mostly in JSON format) data for SQL-based analytics and business intelligence. The most valuable business data is collated and uploaded to data warehouses that are optimized for high performance, concurrency, and reliability, but at a much higher cost because any data processing has to happen at the more expensive SQL rate , not cheap data lake access rates. Historically, data warehouses have been limited in capacity to support both ETL and BI queries; let alone live streaming. Since the data warehouse is mainly built for structured data and does not support unstructured data such as images, sensor data, documents, and videos, it has limited support for machine learning and cannot directly support popular open source libraries and tools (TensorFlow, PyTorch). and other Python-based libraries). As a result, most organizations end up keeping these datasets in data lakes, moving subsets into data warehouses for fast concurrent BI and SQL use cases

How Data Lakehouse is Different from Data Warehouse？

Lakehouse builds on existing data lakes, which typically contain more than 90% of the data in an enterprise. While most data warehouses support "external table" functionality to access that data, they have severe functional limitations (eg, only read operations are supported) and performance limitations in doing so. Lakehouse instead adds traditional data warehouse capabilities to an existing data lake, includingacid business, fine-grained data security, low-cost updates and deletes, first-class SQL support, optimized SQL query performance, and BI-style reporting. Lakehouse is built on the data lake to store and manage all existing data in the data lake, including various data such as text, audio and video, and structured data in tables. Unlike data warehouses, Lakehouse also natively supports data science and machine learning use cases by providing direct access to data using open APIs and supporting various ML and Python/R libraries such as PyTorch, Tensorflow or XGBoost. Therefore, Lakehouse provides a single system to manage all the data of the enterprise while supporting the analytical scope of BI and AI.

Data warehouses, on the other hand, are proprietary data systems built specifically for SQL-based structured data and certain types of semi-structured data analysis. Data warehouses have limited support for machine learning and cannot support running popular open source tools natively without first exporting the data (via ODBC/JDBC or data lakes). Today, no data warehouse system can natively support all the existing audio, image, and video data already stored in the data lake.

How is a Data Lakehouse different from a data lake?

The most common complaint about data lakes is that they can become data swamps. Anyone can dump any data into the data lake; data in the lake has no structure or governance. Performance is poor because data is not organized with performance in mind, resulting in limited analytics on the data lake. As a result, since data lakes use underlying low-cost object storage, most organizations use data lakes as a landing zone for the majority of their data, which is then moved to different downstream systems (such as data warehouses) to extract value.

Lakehouse solves the fundamental problem of getting data to emerge from a data lake. It adds ACID transactions to ensure consistency when multiple parties read or write data at the same time. It supports DW schema architectures such as star/snowflake schema and provides powerful governance and auditing mechanisms directly on the data lake. It also utilizes various performance optimization techniques such as caching, multidimensional clustering, and data skipping, using file statistics and data compression to resize files for fast analysis. It also adds fine-grained security and auditing capabilities to data governance. By adding data management and performance optimization to an open data lake, Lakehouse natively supports BI and ML applications.

How easy is it for data analysts to use Data Lakehouse?

Data Lakehouse systems implement the same SQL interfaces as traditional data warehouses, so analysts can connect to them in existing BI and SQL tools without changing their workflow. For example, leading BI products like Tableau, PowerBI, Qlik, and Looker can all connect to Data Lakehouse systems, data engineering tools like Fivetran and dbt can run against them, and analysts can export data to desktop tools like Microsoft Excel. Lakehouse's support for ANSI SQL, fine-grained access control, and ACID transactions enables administrators to manage them in the same way as data warehouse systems, but in one system covering all the data in their organization.

An important advantage of Lakehouse systems in terms of simplicity is that they manage all data in an organization, so data analysts can be granted access to process raw and historical data as it arrives, not just loaded into the A subset of data in a warehouse system. As a result, analysts can easily ask questions that span multiple historical datasets, or build a new pipeline to process new datasets, without preventing DBAs or data engineers from loading the appropriate data. Built-in support for AI also makes it easy for analysts to run AI models built by the machine learning team on any data.

How does a Data Lakehouse system compare to a data warehouse in terms of performance and cost?

Data Lakehouse systems are built around independent, elastically scalable compute and storage to minimize their operating costs and maximize performance. Recent systems use the same optimization techniques inside their engines (for example, query compilation and storage layout optimizations) to deliver comparable or even better performance for SQL workloads than traditional data warehouses. In addition, Lakehouse systems often take advantage of cloud provider cost-saving features such as spot instance pricing (which requires the system to tolerate losing worker nodes during queries) and reduced prices for infrequently accessed storage that traditional data warehouse engines typically do not have Features are designed to support.

What data governance functions does the Data Lakehouse system support?

By adding a management interface on top of data lake storage, the Lakehouse system provides a unified way to manage access control, data quality, and compliance for all of an organization's data, using a standard interface similar to those found in data warehouses. Modern Lakehouse systems support fine-grained (row, column, and view level) access control through SQL, query auditing, attribute-based access control, data versioning, and data quality constraints and monitoring. These capabilities are typically provided using standard interfaces familiar to database administrators (for example, the SQL GRANT command) to allow existing personnel to manage all data in an organization in a unified manner. Centralizing all data in a Lakehouse system using a single management interface also reduces the administrative burden and potential for errors that come with managing multiple independent systems.

Must the Data Lakehouse be centralized or can it be decentralized into a data grid?

No, organizations do not need to centralize all data in one Lakehouse. Many organizations using the Lakehouse architecture take a decentralized approach to storing and processing data, but a centralized approach to security, governance, and discovery. Depending on organizational structure and business needs, we see some common approaches:

• Each business unit builds its own Lakehouse to gain a complete view of its business—from product development to customer acquisition to customer service.
• Each functional area, such as product manufacturing, supply chain, sales and marketing, can build its own Lakehouse to optimize operations within its business domain.
• Some organizations also launch a new Lakehouse to address new cross-functional strategic initiatives, such as customer 360 or unexpected crises like the COVID pandemic, to drive fast, decisive action.

The unified nature of the Lakehouse architecture enables data architects to build simpler data architectures to match business needs without complexorchestrationData movement across siled data stacks for BI and ML. Additionally, the openness of Lakehouse's architecture enables organizations to leverage a growing ecosystem of open technologies without fear of being locked in to meet the unique needs of different business units or functional areas. Because lakehouse systems are typically built on independent, scalable cloud storage, it's also simple and efficient to give multiple teams access to each lakehouse. recent, delta shareWith the support of many different vendors, an open and standard mechanism for sharing data across Lakehouses is proposed.

How is Data Mesh related to Data Lakehouse?

Zhamak Dehghani outlines four basic organizing principles that embody any data grid implementation. Data Lakehouse architecture can be used to implement these organizing principles:

• • Domain-Oriented Decentralized Data Ownership and Architecture: As discussed in the previous section, the Lakehouse architecture takes a decentralized approach to data ownership. Organizations can create many different lakehouses to meet the individual needs of business groups. Depending on their needs, they can store and manage a variety of data—images, video, text, structured tabular data, and related data assets such as machine learning models and associated code to reproduce conversions and insights.

• • Data as product:The Lakehouse architecture helps organizations manage data as a product by providing complete control over the data lifecycle to different data team members within domain-specific teams. Data teams consisting of data owners, data engineers, analysts, and data scientists can manage data (structured, semi-structured, and unstructured data with appropriate lineage and security controls), code (ETL, data science notebooks, ML training and deployment) and supporting infrastructure (storage, compute, cluster strategies, and various analytics and ML engines). Lakehouse platform features such as ACID transactions, data versioning, and zero-copy cloning make it easy for these teams to release and maintain their data as production.

• • Self-service data infrastructure as a platform: The Lakehouse architecture provides an end-to-end data platform for data management, data engineering, analytics, data science, and machine learning, and integrates with a broad ecosystem of tools. Adding data management on top of an existing data lake simplifies data access and sharing—anyone can request access, the requester pays for cheap blob storage and gets secure access instantly. Additionally, using open data formats and enabling direct file access, data teams can use best-in-class analytics and ML frameworks on their data.

• • Federated Computing Governance:Governance in the Lakehouse architecture is achieved through a centralized catalog with fine-grained access control (row/column level), enabling easy discovery of data and other artifacts such as code and ML models. Organizations can assign different administrators to different parts of the directory to decentralize control and management of data assets. This hybrid approach of centralized catalog and federated control preserves the independence and agility of local domain-specific teams, while ensuring reuse of data assets across those teams and enforcing a common security and governance model globally.