Cloud Object Storage for Data Lakes (2023)

Data growth is exploding. A growing number of mobile devices, smart sensors, and smart endpoints are generating ever-increasing variety, volume, and velocity of data. IDC predicts that with the proliferation of connected devices and intelligent systems, the amount of data generated globally every year will grow from 33 ZB in 2018 to 175 ZB in 2025. (1 ZB = 1 trillion GB)

By turning vast amounts of raw data into meaningful and actionable insights, companies can accelerate the pace of business, improve employee productivity and streamline operations. Companies can optimize business processes and fine-tune sales, marketing and advertising campaigns. Municipalities and utilities can enhance public safety and services, optimize transportation and energy systems, and reduce expenses and waste. Researchers and scientists can improve our understanding of the universe, accelerate the treatment of diseases, and improve weather forecasting and climate modeling.

Big data has the potential to fundamentally change entire industries. But antiquated and expensive data storage solutions have hindered this process. In fact, most organizations cannot maintain large data sets over the long term using traditional on-premises storage solutions or first-generation cloud storage services from AWS, Microsoft Azure, or Google Cloud Platform. In fact, most enterprises store only the essential data needed to support primary business applications and regulatory requirements. Historical data containing valuable insights into customer behavior and market trends is often discarded.

But all that is about to change. A new generation of cloud storage has arrived, bringing utility pricing and simplicity. With Cloud Storage 2.0, you can cost-effectively store any type of data, for any purpose, for any length of time, in Wasabi's hot cloud storage. You no longer have to make painful decisions about what data to collect, where to store it, and how long to keep it.

This next-generation cloud storage is ideal for building data lakes -- large repositories where you can collect vast amounts of raw data for any purpose. In a TDWI survey of more than 250 data management professionals, nearly half of respondents said they already had a data lake in production (23%) or planned to have one within 12 months (24%) Data lake in production.

A data lake is an enterprise-wide system for securely storing different forms of data in a native format. Data lakes include a variety of data not found in traditional structured data stores (such as sensor data, clickstream data, social media data, location data, log data from servers and network devices) as well as traditional structured and semi-structured data. structured data. A data lake breaks down traditional enterprise information silos by bringing all of an enterprise's data into a single repository for analysis, without the historical constraints and hassles of schema or data transformation.

Data lakes provide the foundation for advanced analytics, machine learning, and new data-driven business practices. Data scientists, business analysts, and technical professionals can run analytics in-place using commercial or open source data analysis, visualization, and business intelligence tools of their choice. Dozens of vendors offer standards-based tools, from self-service data exploration tools for non-technical business users to advanced data mining platforms for data scientists, that help businesses monetize data lake investments and turn raw data into business value .

The diagram below depicts a data lake in an IoT implementation. Edge computing devices process and analyze local data before sending it to the data lake. For example, an edge server might perform real-time analytics, execute local business logic, and filter out data that has no inherent historical or global value.

The diagram below depicts a typical data lake technology stack. A data lake includes scalable storage and computing resources; data processing tools for managing data; analysis and reporting tools for data scientists, business users, and technical staff; and general data governance, security, and operations systems.

You can implement a data lake in an enterprise data center or in the cloud. Many early adopters deployed data lakes on-premises. As data lakes become more pervasive, many mainstream adopters are looking to cloud-based data lakes to accelerate time to value, lower TCO, and increase business agility.

You can implement a data lake in an enterprise data center using commodity servers and local (internal) storage. Today, most on-premises data lakes use commercial or open source versions ofHadoop, a popular high-performance computing framework, as a data platform. (insideTDWI Survey, 53% of respondents use Hadoop as their data platform, while only 6% use a relational database management system. )

You can combine hundreds or thousands of servers to create scalable and elastic Hadoop clusters capable of storing and processing massive data sets. The diagram below depicts the technology stack for an on-premises data lake on Apache Hadoop.

Technology stack includes:

Hadoop MapReduce
A software framework for easily writing applications that process large amounts of data in parallel on large clusters of commodity hardware in a reliable, fault-tolerant manner.
learn more

Hadoop Yarn
A framework for job scheduling and cluster resource management.
learn more

Hadoop Distributed File System (HDFS)
A high-performance file system designed to run on low-cost servers with cheap internal disk drives.
learn more

On-premises data lakes offer high performance and strong security, but they are notoriously expensive and complex to deploy, manage, maintain and scale. Disadvantages of on-premises data lakes include:

lengthy installation
Building your own data lake takes a lot of time, effort, and money. You must design and build systems; define and establish security and management systems and best practices; procure, build and test computing, storage and networking infrastructure; identify, install and configure all software components. It typically takes months (often more than a year) to get an on-premises data lake up and running in production.

high capital expenditure
Large upfront equipment expenditures can lead to unbalanced business models with low ROI and long payback periods. Servers, disks, and networking infrastructure are all over-engineered to meet peak traffic demands and future capacity needs, so you're always paying for idle computing resources and unused storage and networking capacity.

high operating costs
Recurring costs for power, cooling, and rack space; monthly hardware maintenance and software support costs; and ongoing hardware management costs all lead to high equipment operating expenses.

high risk
Ensuring business continuity (replicating live data to a secondary data center) is an expensive proposition out of the reach of most businesses. Many businesses back up data to tape or disk. In the event of a disaster, rebuilding systems and restoring operations can take days or even weeks.

Complex System Management
Running an on-premises data lake is a resource-intensive proposition that diverts valuable (and expensive) IT staff from more strategic work.

You can implement a data lake in the public cloud to avoid equipment expense and hassle and accelerate big data initiatives. General benefits of cloud-based data lakes include:

Quick time to value
You can reduce rollout time from months to weeks by eliminating infrastructure design work and hardware acquisition, installation, and startup tasks.

no capex
You can avoid upfront capital expenditures, better align spending with business needs, and free up capital budgets for other projects.

No equipment operating costs
You can eliminate ongoing equipment operating expenses (power, cooling, real estate), annual hardware maintenance expenses, and recurring system administration expenses.

Instant and unlimited scalability
You can add compute and storage capacity on demand to meet rapidly changing business needs and improve customer satisfaction (rapid response to line-of-business needs).

independent scaling
Unlike on-premises Hadoop implementations that rely on servers with internal storage, cloud implementations allow you to independently scale compute and storage capacity to optimize costs and maximize resource utilization.

reduce risk
You can replicate data across regions to increase resiliency and ensure continuous availability in the event of a disaster.

Simplify operation
You can free up your IT staff to focus on the strategic tasks of supporting your business (with the cloud provider managing the physical infrastructure).

Cloud-based data lakes are much easier and less expensive to deploy, scale, and operate than on-premises data lakes. That said, first-generation cloud object storage services such as AWS S3, Microsoft Azure Blob Storage, and Google Cloud Platform Storage were inherently expensive (in many cases as expensive as on-premises storage solutions) and complex. Many businesses are looking for simpler, more affordable storage services for their data lake initiatives. Limitations of first-generation cloud object storage services include:

Expensive and confusing service tiers
Traditional cloud providers sell several different types (tiers) of storage services. Each tier serves a different purpose, such as primary storage for active data, active archive storage for disaster recovery, or inactive archive storage for long-term data retention. Each has unique performance and resiliency characteristics, SLAs, and pricing tables. Complex fee structures with multiple pricing variables make it difficult to make informed choices, forecast costs and manage budgets.

supplier lock-in
Each service provider supports a unique API. Switching services is an expensive and time-consuming proposition—you must rewrite or replace existing storage management tools and applications. To make matters worse, traditional providers charge exorbitant data transfer (egress) fees to move data out of their clouds, making it costly to switch providers or leverage a mix of providers.

Wasabi Hot Cloud Storage is an extremely affordable, fast and reliable cloud object storage for any purpose. Unlike first-generation cloud storage services with confusing storage tiers and complicated pricing schemes, Wasabi is easy to understand and extremely cost-effective to scale. Wasabi is great for storing large amounts of raw data.

Key benefits of Wasabi for data lakes include:

commodity pricing
Wasabi Hot Cloud Storage costs $0.0059/GB/month. Compared to $.023/GB/month for Amazon S3 Standard, $.026/GB/month for Google Multi-Regional, and $.046/GB/month for Azure RA-GRS Hot.

Unlike AWS, Microsoft Azure, and Google Cloud Platform, we do not charge additional fees to retrieve data from storage (egress fees). We do not charge additional fees for PUT, GET, DELETE or other API calls.

Read our briefing on new economic technologies for cloud storage

outstanding performance
Wasabi's parallel system architecture provides faster read/write performance than first-generation cloud storage services, with significantly faster first-byte speed.

Download our performance benchmark report Wasabi Feature
Strong Data Persistence and Protection
Wasabi hot cloud storage is designed to provide extremely high data durability, integrity and security. Optional data immutability features protect against accidental deletions and administrative errors; protect against malware, errors, and viruses; and improve compliance.

Read our brief on the powerful security technology Wasabi Feature

Read our technical brief on data immutability Wasabi Feature

If you are running your data lake on Apache Hadoop, you can use Wasabi hot cloud storage as an economical alternative to HDFS, as shown in the diagram below. Wasabi hot cloud storage is fully compatible with AWS S3 API. You can use the Hadoop Amazon S3A connector (part of the open source Apache Hadoop distribution) to integrate S3 and S3-compatible storage clouds such as Wasabi into various MapReduce streams.

You can use Wasabi hot cloud storage as part of a multi-cloud data lake implementation to improve choice and avoid vendor lock-in. A multi-cloud approach lets you independently scale data lake compute and storage resources using best-of-breed providers.

Wasabi works through partnerships with leading colocation, carrier hotel and exchange providers such asEquinix,flexibleandspotlight network.These dedicated network connections avoid Internet delays and bottlenecks, providing fast and predictable performance. you can alsoConnect to your private cloudStraight to the mustard. Unlike first-generation cloud storage providers,WasabiYou never pay for data transfer (egress). In other words, you are free to move data out of Wasabi.

Wasabi is hosted in multiple geographically distributed data centers for resiliency and high availability. You can replicate data across Wasabi regions for business continuity, disaster recovery, and data protection as follows.

For example, you can replicate data across three different Wasabi data centers (regions) using the following method:

• Wasabi data center 1 is used for active data storage (primary storage).
• Wasabi Datacenter 2 acts as an active archive for backup and recovery (a hot backup in case Datacenter 1 becomes inaccessible).
• Wasabi Data Center 3 ASChangelessData storage (protecting data from mismanagement, accidental deletion, and ransomware). Immutable data objects cannot be deleted or modified by anyone, including Wasabi.

What if you could affordably store all your data in the cloud?

now you can.Wasabi is here to guide you through your migration to the enterprise cloud and work with you to determine which cloud storage strategy is right for your organization.