Microsoft Azure Data Lake

What is Microsoft Azure Data Lake?

Azure Data Lake is a cloud-based data repository service from Microsoft that enables organizations to store vast quantities of many types of data and perform data processing and analytics across multiple platforms and programming languages. This data lake is highly scalable, secure and supports massively parallel analytics -- all of which enables enterprise teams to unlock more insights from their unstructured, semistructured and structured data.

Azure Data Lake explained

Azure Data Lake is a centralized cloud repository that can store vast amounts of data in its original format. There's no need to convert unstructured or semistructured data into a structured format in order to run different types of analytics or to power intelligent actions.

Organizations can store data of any size and velocity in Azure Data Lake. They can then process and analyze the data on demand across many different platforms. They can also run parallel data transformation and processing programs over petabytes of data in many different programming languages, including U-SQL, R, Python and .NET. Because Azure Data Lake runs in the cloud, users aren't required to manage any hardware or software installations or upgrades.

In addition to its data storage and analytics capabilities, Azure Data Lake incorporates advanced features to simplify data management, governance and security. An organization can integrate it with its existing operational stores and data warehouses to extend current data applications. Azure Data Lake's analytics service is available on a pay-per-job basis.

Azure Data Lake is built on Yet Another Resource Negotiator technology and the open Hadoop Distributed File System standard. These architectural choices enable enterprise users -- including developers, data scientists and analysts -- to run massively parallel analytics on vast quantities of data. The service supports batch, interactive and streaming analytics and eliminates the complexities of data ingestion, conversion, storage, security and management common with on-premises data storage and analytics systems.

What can organizations do with Azure Data Lake?

Any organization can use Microsoft Azure Data Lake to store data and perform batch, streaming and interactive analytics on it. Azure Data Lake works with data of any size, including petabyte-size files and trillions of objects.

Azure Data Lake is also suitable for other data-related activities, such as the following:

• Debugging and optimizing big data programs.
• Developing and running massively parallel programs for data transformation and processing in different languages.
• Protecting data assets with enterprise-grade security and extending on-premises security and governance controls to the cloud.
• Encrypting sensitive data and safeguarding it from unauthorized and malicious use with SSL (for data in motion) and service or user-managed hardware security module (HSM)-backed keys in Azure Key Vault (for data at rest).
• Enabling role-based access controls (RBAC) to authorize users and groups with fine-grained POSIX-based access control lists (ACLs).
• Auditing access or configuration changes to the system to maintain security and regulatory compliance.

Key components of Azure Data Lake

Azure Data Lake includes three components that enable teams to build data lakes to their specific data analytics requirements and use cases. These components are Azure HDInsight, Azure Data Lake Analytics and Azure Data Lake Storage.

1. Azure HDInsight

Azure HDInsight is a fully managed Cloud Hadoop offering backed by a 99.9% service-level agreement. This Open Source analytics platform and enterprise-grade service enables organizations to manage big data needs and provision cloud Hadoop, Spark and HBase clusters. HDInsight provides analytics clusters and optimized components for Apache Hadoop, Spark, Hive, MapReduce, HBase, Storm, Kafka and R Server, so users can process massive amounts of any type of data in the cloud.

HDInsight does not require users to install hardware or manage infrastructure in order to quickly spin up open source projects and clusters. Teams can deploy all big data technologies and ISV applications as managed clusters and then secure and monitor them to protect data. After building a data lake, teams can integrate it with any number of Azure data storage tools and services, including Azure Synapse Analytics, Azure Cosmos DB and Azure Data Lake Storage.

2. Azure Data Lake Analytics

Azure Data Lake Analytics is a distributed analytics service to develop and run parallel transformation and processing programs on big data. Data Lake Analytics supports data transformation and processing programs in U-SQL, R, Python and .NET. U-SQL is particularly useful since it is a simple, expressive and extensible language that simplifies processing for diverse workload categories, including querying, machine learning, ETL and analytics.

Like HDInsight, Data Lake Analytics is a cloud-based service, which means enterprise teams don't have to manage or tune any infrastructure, such as servers, virtual machines or clusters. Instead, they can process data on demand within the cloud in just a few seconds. They can also instantly scale the processing power required for the job (measured in Azure Data Lake Analytics Units or AUs).

Data Lake Analytics charges organizations per job, which simplifies pricing and enables better control over cloud analytics costs. The service includes an execution environment that provides recommendations to improve the performance of big data programs, which can help organizations to reduce costs by as much as 95%. Virtualizing analytics -- moving processing close to the source data without data movement -- also improves performance and cuts costs.

3. Azure Data Lake Storage

Azure Data Lake Storage is a secure data lake that enables organizations to build a scalable foundation for their analytics needs. This single storage platform for ingestion, processing and visualization eliminates data silos and simplifies data analytics. It also supports the most common analytics frameworks and high-performance analytics workloads while ensuring consistent performance regardless of the scale of the analytics query.

Data Lake Storage offers limitless scale and automatic geo-replication for 16 9s of data durability. It provides features such as tiered storage and policy management to optimize costs, Azure Active Directory (Azure AD) and RBAC to authenticate users and data as well as data encryption, network-level control and advanced threat protection.

Benefits of Azure Data Lake

As a "no-limits" data lake, Azure Data Lake enables organizations to store and analyze any type of data at any time, at any scale and in a cost-effective manner. The service makes it easy to analyze petabyte-size files and trillions of objects across platforms and languages, and capture useful insights that support operations and business decision-making. Teams can do all of this in a single place, without artificial constraints and without having to worry about how to process and store large data sets.

The service simplifies data management and governance since it works with existing tools for identity, management and security, as well as operational stores and data warehouses. Companies can use their existing tech stack and data applications and strengthen them further with new data storage and analysis capabilities.

Azure Data Lake powers intelligent action from big data, provides optimized analytics clusters for numerous open source frameworks and runs massively parallel analytics on unstructured, semistructured and structured data. It also provides enterprise-grade security and auditing, as well as 24/7 support to protect data assets and mitigate challenges. Microsoft monitors every deployment of Azure Data Lake to guarantee that it runs continuously with the strongest security and governance controls in the cloud.

Data Lake seamlessly integrates with Visual Studio, Eclipse and IntelliJ, so enterprise teams can easily run, debug and tune their big data queries. They can visualize jobs to see how the code runs at scale and to identify performance and cost bottlenecks. The service also works with Azure Synapse Analytics, Power BI and Data Factory, making it easy for users to prepare data, perform interactive analytics on large-scale data sets and minimize data latency.

Read about who manages data lakes and what skills are needed. Check out 7 steps to a successful data lake implementation and how to build a strong data analytics platform architecture. Explore 5 principles of a well-designed data architecture.