Announcing Delta 2.0: Open Source Everything (2023)

This year's themeData + Artificial Intelligence SummitYes we are using lakehouse to build a modern data stack. A fundamental requirement of Data Lakehouse is the need to bring reliability to your data - an open, simple, production-ready and platform-agnostic data such asdelta lake.With this, we are happy to announce that with Delta Lake 2.0, we will open source all of Delta Lake!

What Makes Delta Lakes Different

Delta Lake enables organizations to build Data Lakehouse, enabling data warehousing and machine learning directly on the data lake. But Delta Lake doesn't stop there. Today, it is the most comprehensive Lakehouse format used by more than 7,000 organizations, processing exabytes of data every day. Besides the core functionality of being able to seamlessly ingest and consume streaming and batch data in a reliable and performant manner, one of the most important features of Delta Lake is Delta Sharing, which enables different companies to share datasets in a secure manner . Delta Lake also comes with standalone readers/writers, allowing any Python, Ruby, or Rust client to write data directly to Delta Lake without any big data engines like Apache Spark™. Finally, Delta Lake has been optimized over time and significantly outperforms all other lakehouse formats. Delta Lake comes with a rich set of open source connectors, including Apache Flink, Presto, and Trino. Today, we’re excited to announce our commitment to open source Delta Lake by open sourcing all of Delta Lake, including features that were only available in Databricks until now. We hope this democratizes the use and adoption of Data Lake House. But before we do that, we want to tell you a bit about the history of Delta.

Origin of Delta Lake

The genesis of this project started in a casual conversation between Spark Summit 2018Dominic Brezinski, a Distinguished Engineer at Apple, and our own Michael Armbrust (who originally created Delta Lake, Spark SQL, and Structured Streaming). Dominique, who works on Intrusion Monitoring and Threat Response, approached Michael with ideas on how to address the processing demands created by massive concurrent batch and streaming workloads (petabytes of log and telemetry data per day). They were unable to use data warehouses for this use case because (i) they were cost-prohibitive for the massive amount of event data they had, (ii) they did not support the real-time streaming use case essential for intrusion detection, and (iii ) lack support for the advanced machine learning needed to detect zero-day attacks and other suspicious patterns. So building it on a data lake was the only viable option at the time, but they were struggling with pipeline failures due to the high number of concurrent streaming and batch jobs, and couldn't ensure transactional consistency and data accessibility for all business data .

So the two of them got together to discuss the need for a unified data warehouse and AI, planting the seeds of what we now know as Delta Lake blossoming. Over the next few months, Michael and his team worked closely with Dominique's team to build this ingestion architecture designed to solve data problems at scale—allowing their team to easily and reliably handle low-latency stream processing and Interactive queries without job failure or reliability troubleshoot the underlying cloud object storage system, while enabling Apple's data scientists to process large volumes of data to detect anomalous patterns. We quickly realized that this issue was not unique to Apple, as many of our customers had experienced the same issue. Fast forward, and we're starting to see Databricks customers building reliable data lakes at scale effortlessly using Delta Lake. We started calling this approach to building a reliable data lake the data lake house pattern because it provides the reliability and performance of a data warehouse with the openness, data science, and real-time nature of a massive data lake.

Delta Lake becomes a Linux Foundation project

As more and more organizations start building lakehouses with Delta Lake, we hear that they want the data format on the data lake to be open source, avoiding vendor lock-in entirely. As a result, in2019 Spark+AI Summit, together withLinux Foundation, we announced the open sourcing of the Delta Lake format so that the larger community of data practitioners can make better use of their existing data lakes without sacrificing data quality. Since open sourcing Delta Lake (under the permissive Apache License v2, the same license we use for Apache Spark), we have seen massive adoption and growth of the Delta Lake developer community, as well as data sharing between practitioners and companies. Brigade's paradigm shift by unifying their data with machine learning and artificial intelligence use cases. That's why we're seeing such massive adoption and success.

Delta Lakes Community Grows

Today, the Delta Lake project is thriving with more than 190 contributors from more than 70 organizations, nearly two-thirds of which are external Databricks contributors from leading companies such as Apple, IBM, Microsoft, Disney, Amazon, and eBay, just to name Few cases are rare. In fact, we've seen a 633% increaseContributor strength (as defined by The Linux Foundation)in the past three years. This level of support is the heart and strength of this open source project.

Source: The Linux Foundation Contributor Strength:Growth in the total number of unique contributors analyzed over the past three years. A contributor is anyone who is associated with a project through any code activity (commits/PRs/changesets) or helps find and fix bugs.

Delta Lake: The fastest and most advanced multi-engine storage format

Delta Lake was built not just for one tech company's special use case, but for a wide range of use cases representing our customers and communities, from finance, healthcare, manufacturing, operations to the public sector. Delta Lake has been deployed and battle-tested in tens of thousands of the largest tables measured in exabytes. Therefore, Delta Lake is far ahead of other formats in real customer testing and third-party benchmarking time and time again¹About performance and ease of use.

With Delta Sharing, anyone can easily share data and read data shared from other Delta tables. We released Delta Sharing in 2021 to give the data community an option to escape vendor lock-in. As data sharing becomes more popular, most of you have expressed frustration with more data silos (now even outside your organization) due to proprietary data formats and proprietary computations required to read the data. Delta Sharing introduces an open protocol for secure real-time exchange of large data sets, enabling secure data sharing across products for the first time. Data users can now connect directly to shared data through Pandas, Tableau, Presto, Trino, or dozens of other systems that implement open protocols, without using any proprietary systems—including Databricks.

Delta Lake also has the richest ecosystem of direct connectors such asa lot,Presto, andTrinity, enabling you to read and write to Delta Lake directly from the most popular engines without Apache Spark. Thanks to contributors from Delta Lakegraffitiandbackground market, you can also usetriangle rust- The underlying Delta Lake library in Rust, enabling Python, Rust, and Ruby developers to read and write Delta without any big data frameworks. Today, Delta Lake is the most widely used storage tier in the world with over 7 million monthly downloads; a 10x increase in monthly downloads in just one year.

Announcing Delta 2.0: Open Source Everything

Delta Lake 2.0 is the latest version of Delta Lake and will further enable our large community to benefit from all Delta Lake innovations, all Delta Lake APIs are open source - especially the performance optimizations and features brought by Delta Engine, such asZ order,change data feed,dynamic partition coverageand discarded columns. Thanks to these new capabilities, Delta Lake continues to deliver unmatched out-of-the-box price/performance for all Lakehouse workloads, from streaming to batch—up to 4.3x faster than other storage tiers. Over the past six months, we have put in a huge effort to implement and contribute all of these performance enhancements to Delta Lake.Therefore, we will open source all of Delta Lake, and we are committed to ensuring that all functions of Delta Lake will be open source.

We're excited to see Delta Lake going from strength to strength. We look forward to working with you to continue the rapid innovation and adoption of Delta Lake in the years to come.

Interested in joining the open source Delta Lake community?
accessdelta lakeLearn more; you can join the Delta Lake community in the following waysrelaxationandgoogle group.

¹ https://databeans-blogs.medium.com/delta-vs-iceberg-performance-as-a-decisive-criteria-add7bcdde03d