Implementing a Data Lakehouse Architecture in AWS - Part 1 of 4 (2023)

Numerous applications in today's world accumulate vast amounts of data to build insight and knowledge. Adding value and improving functionality is essential, but at what cost? A key factor in the arrival of the "big data" era is the drastic reduction in data storage costs while increasing the computing power available in the cloud.

Since Apache Hadoop was first released in 2008, the extensibility of the framework has allowed it to continuously evolve and improve its ability to collect, store, analyze, and manage multiple sets of data. Several "big data" projects have been added to the data architect's toolbox to create more powerful, smarter solutions. One of these newer data structures is the so-calleddata lake house.

Data Lakehouse is a new form of data architecture that combines the positive aspects of Data Lakes and Data Warehouses. The advantages of a data lake include flexibility, low cost, and scale. When combined with strong data management andACID transactionalData Warehouse, which allows business intelligence, analytics and machine learning on all data in a fast and agile manner.

But it doesn't end there. A robust Data Lakehouse will connect the Data Lake, Data Warehouse, and purpose-built databases and services into a single fabric. This also includes a unified governance approach, tools and policies to move data inside-out and outside-in with minimal effort and cost.

Data Lakehouse is enabled by a new open system design that implements similar data structures and data management capabilities as in data warehouses directly onto low-cost storage for data lakes. Consolidating them into one system means data teams can move faster while working with data without accessing multiple systems. Data Lakehouses also ensure teams have the most complete and up-to-date data for data science, machine learning, and business analytics projects.

In this series of articles, we explore how to embrace thisbig data architectureand implemented in AWS.

The implementation of Data Lakehouse Architecture in AWS centers on a simple storage system or S3, providing object storage to build a Data Lake. Around the data lake, you can add other components such as:

• Use Elastic Map Reduce to process "big data";
• relational database with relational database service, in special Amazon Aurora;
• non-relational databases such as Amazon DynamoDB;
• Machine Learning for Business Analytics with Amazon SageMaker;
• Data warehouse using Amazon Redshift;
• Log analysis powered by OpenSearch.

As you can probably guess, connecting all of the above components into a Data Lakehouse requires setting up data stores, databases, and services and allowing for quick and easy transfer of data from the services used to the Data Lake (data moves from outside to inside), Extract data from data lakes to dedicated databases and services (data inside-out movement), and move data from one dedicated data store to another (perimeter data movement).

To get the most out of their data lakes and these purpose-built stores, customers need to easily move data between these systems. For example, clickstream data from a web application can be collected directly in a data lake. A portion of this data can be moved out to the data warehouse for daily reporting. We think of this concept as data movement from the inside out.

Likewise, clients move data in the other direction: from the outside to the inside. For example, copy query results for product sales in a given region from their data warehouse into their data lake to run a product recommendation algorithm using ML against a more important data set.

Finally, there are other situations where customers want to move data from one purpose-built data store to another: the perimeter. For example, they might replicate product catalog data stored in a database into their search service to make it easier to view their product catalog and offload search queries from the database.

As data in data lakes and other data stores increases,data gravity also grows, and it becomes more challenging to move data in any direction. The key service to make all data move agile while data gravity grows isamazon movement.In short, Amazon Kinesis allows you to process and analyze data as it is presented, responding quickly and in real time with no latency.

Let's take a deeper look at how this process works. Kinesis has three different services: Kinesis Data Stream, Kinesis Data Firehose, and Kinesis Data Analytics.

Kinesis is a serverless service that is elastic, durable, reliable, and available by default. With it, you can develop applications to ingest streams of data and react to them in real time, enabling you to build value from streaming data very quickly with minimal operational overhead.

Kinesis Data Stream is best for fast, continuous ingestion of data, which can include social media, application logs, market data feeds, or IT infrastructure log data. Each stream consists of a set of shards, which are units of read and write capabilities in Kinesis. A shard provides a stream that reads up to 1 MB per second and writes at a rate of 2 MB per second. When creating a Kinesis stream, you must specify the number of shards provisioned for the stream, as well as some other parameters. If the amount of data written to the stream increases, you can add more shards to scale the stream without downtime.

To use Kinesis Data Streams, a typical scenario is to have producers push data directly to a software system that writes Kinesis stream data. Producers can be EC2 instances, mobile applications, on-premises servers, or IoT devices. On the other hand, there are consumers that retrieve records from the data stream and process them accordingly. Consumers could be applications running on EC2 instances or AWS Lambda functions. If the consumer is on an Amazon EC2 instance, you can run it in an Auto-Scaling group to scale up. In this case, you only pay for the shards you allocate per hour.

There can be more than one application type processing the same data. An easy way to develop consumer applications is to use AWS Lambda, which scales up or down automatically because it allows you to run your consumer application's code without provisioning or managing servers.

Kinesis Data Streams allows preserving the client-side order of data and guarantees that consumers receive data in the order that producers sent it. Data can also be processed in parallel, so you can also consume the same data stream in parallel to multiple applications at the same time. For example, you can ingest application log data when a first consumer looks for a specific keyword, and have a second consumer use the same data to generate performance insights. This feature allows decoupling the collection and processing of data while providing it with persistent buffers. Also, it's up to you to process data at your own desired rate, so there's no need to worry about some processes being directly dependent on others.

Additionally, Kinesis Data Streams makes it easy to ingest and store streaming data while ensuring your data is durable and available, typically in less than 1 second after the data is written to the stream. For security purposes, server-side encryption can be used to protect your data to meet any compliance requirements.

Some use cases for big data streams are:

• E-commerce platform fraud detection (real-time monitoring logs);
• Analyze mobile or web applications with millions of users by ingesting clickstreams;
• Tracking and responding in real-time to IoT devices emitting massive amounts of sensor data;
• Perform sentiment analysis on social media posts in real time.

All of the above use cases have one thing in common, which is the need to process many small messages generated from various sources in near real time. Messages need to be ingested, stored and provided to different applications. Furthermore, it needs to be done in a reliable, secure and scalable manner. However, you don't want to keep multiple copies of this data, as one copy per application is sufficient.

In general, Kinesis stores streaming data for a limited amount of time. By default, it retains any sequence of events for 24 hours, but the retention period can be extended by up to 7 days. You need to write the processed results to persistent storage, such as Amazon S3, DynamoDB, or Redshift.

In this article, we want to illustrate how easy it is to start feeding data lakes built on top of Amazon S3 from streaming data. Applications may generate events, which we may wish to store to build insight and knowledge, while creating value from derived data.

A first initial idea for implementing a data lake based on real-time streaming data might be to implement a Kinesis consumer to store the consumed events into S3 using the AWS API or SDK. However, there is a more elegant option.

We will use aKinesis Data Firehose Delivery StreamFeed data to S3 seamlessly.

Among the several ways to send message streams to Kinesis, we will use a producer developed in Python with a libraryPorto 3Establish communication and messaging. Message producers will use the libraryfraud, widely used to create random fake data, and a parameter to define the number of messages created and sent to Kinesis. We will use Terraform to automatically create the necessary resources and ensure version control.

Below we list each resource used and what it does in that context:

• Data Producer - a Python program that generates data;
• Kinesis Data Streams - will receive the generated messages;
• Kinesis Data Firehose - passing received messages, converted into parquet files;
• Amazon S3 - buckets for storing generated data;
• Glue Data Catalog - we will use the Glue Data Catalog to have a unified metastore;
• Amazon Athena - Used to query data stored in S3 buckets.

The following figure illustrates the architectural design of the proposed solution.

AWS service creation

To start the deployment, we will verify the infrastructure code developed with Terraform. If you do not have Terraform installed, here we will see two methods, installing from the repository and downloading the standalone version.

Now, we need to initializelandformby runningTerrain initialization. Terraform will generate a file named.terraformand download atmain programdocument.

Following best practices, always run the commandterraform plan -out=kinesis-stack-planView output before starting, creating, or changing existing resources.

Once the plan has been validated, changes can be safely applied by runningterraform application "kinesis-stack-plan". Terraform will perform a final validation step and prompt for confirmation before applying.

In the video embedded below, you'll see the entire process of creating the environment.

data producer

The code snippet below shows how to randomly create a data dictionary and then convert it to JSON. We need to convert the dictionary to JSON in order to put the data into the Kinesis data stream.

Now is the time to start generating data, setting up a python virtual environment, and installing dependencies. The isolated environment will prevent breaking any other python libraries that may be in use.

Using the data generator script, we'll start sending 10 records to our Kinesis service.

An important thing to note is that it takes a few seconds to see the generated data reach Kinesis.

Check out the Kinesis Data Stream chart below to get an idea of ​​the rate at which data is being received and how it is being processed.

The same goes for the Kinesis Data Firehose statistics in the image below.

Can navigate within our S3 bucket to see how Kinesis Data Firehose delivers and organizes Parquet files and creates "subfolders" that will be recognized as partitions (Year/Month/Day/Hour - 2022/02/04/21) our table.

used byAWS Glue Service, we were able to see patterns automatically identified from messages processed by Kinesis Data Firehose.

Now that our data is available and its schema defined, we will be able to execute some SQL AdHoc queries using Amazon Athena as shown below.

Finally, we need to compromise our infrastructure by running the commandterrain destructionto avoid additional costs. Running the destroy command first asks for confirmation, and proceeds to delete the infrastructure after receiving an affirmative answer, as you can see in the short video below.

A new paradigm of Data Lakehouse architecture is coming, providing more opportunities for enterprises planning to start their businessData Driven Journey, the technologies, frameworks and cost ranges associated with cloud platforms are now more attractive than ever.

In the first blog post, we introduced a technology usage scenario where streaming data is ingested using Kinesis Data Streams, processed in real time using Kinesis Data Firehose, and delivered to object storage for further use by data analysts, ML Engineer, using Amazon Athena to run SQL AdHoc queries.

If you liked this article, here's our Lightning Talk on the topic.

In this series on Data Lakehouses, we'll discuss this architecture further. stay tuned!

Does your business require an efficient and robust data architecture to succeed?keep in touch, we can help you make it happen!

• example repo
• Building a Lake House Architecture on AWS
• Harness the Power of Data with AWS Analytics
• photographerStephen Dawsonexistno splash