PowerPoint Presentation

Big Data Made Easy with Azure Data LakeKanio Dimitrov,Tokyo Azure Meetup

https://www.getpostman.com/collections/2449c4125d7af478aed8http://azjobsdemo.azurewebsites.net/https://searchsamples.azurewebsites.net/#/1

Kanio Dimitrov (KD) - https://www.getpostman.com/collections/2449c4125d7af478aed8Kanio Dimitrov (KD) - https://searchsamples.azurewebsites.net/#/Kanio Dimitrov (KD) - http://azjobsdemo.azurewebsites.net/About Me

Azure Architect & Advisor

Tokyo Azure Meetup Host

twitter: @azurekanio

blog: https://azurekan.wordpress.com/

Big Data Made Easy WhenEasy to manage

Easy to debug

Easy to optimize

Key PointsAny Data

Enterprise

Developers

What is Azure Data Lake?

CosmosInternal Microsoft System

10 000 Developers

100 000-s interactive jobs/day

Exabytes of data

Microsoft Core System

From Cosmos to Azure Data Lake

Azure Data Lake StoreAzure Data Lake AnalyticsEase of use

Ability to Scale

Offered to the public

Azure Data Lake Based on Open Source

Easy to StartCreate ADL Store Account

Create ADL Analytics Account (90 seconds, free)

Write & Submit U-SQL script

U-SQL job executes

ADL Analytics

Distributed analysis service

Built on Apache YARN

Dynamic scaling

ADL Analytics

Pay per query

Scale per query

Federated query

ADL Analytics

Uses U-SQL - C# & SQL

No Scale limits

Optimized to work with ADL Store

Data SourceReadWriteADL StoreYesYesStorage BlobYesYesAzure SQLYesIn FutureAzure SQL Data WarehouseYesIn FutureAzure SQL DB in VMYesIn FutureOn Premise Data SourcesIn FutureIn Future

ADL Analytics AdministrationWeb-based management in Azure Portal

Automation with PowerShell

Role-Based Access Control with Azure Active Directory

Monitory Service Operation and Activity

DevelopmentAuthor, debug & optimize Big Data applications in Visual Studio

Languages: U-SQL & Hive (coming soon)

.Net integration with U-SQL

ADL Analytics SDK-sJAVAC++.NETNode.jsPythonU-SQL ExtensibilityYesManagement OperationsBy GAYesYesBy GA

U-SQLSQL

Support of familiar SQL clauses

Structured and Unstructured Data

Relational metadata objects.NET

U-SQL - full C# expressions

Reuse .NET code

Use C# for defining:Types, Functions, Joins, Aggregations, I/O (Extractors, Outputters)

Logical Plan -> Physical Plan

One node perspective

Physical plan created

Defines level of parallelism

HDInsightManaged Hadoop Cluster in the Cloud

Deploy Storm jobs from Visual Studio

Use C# to author event processing logic

Integrate existing packages & code

ADLA vs HDInsight

Azure Data Lake AnalyticsHDInsightAutomatically ScaleStart quickly with C#, SQL, Visual StudioJobs - Convenient, efficient, automatic scale Leverage open source tech Java, Eclipse, HiveManage clusters customization, control and flexibility

Azure Data Lake Store

Hyper Scale Web HDFS store in the cloud

Store any data in native format

Enterprise grade

No limits to Scale

Optimized for analytic workload performance

Azure Data Lake Store

Unlimited Storage (petabytes)

Optimized for AnalyticsParallel computing optimizedAuto optimization for any throughput

ReliableAutomatically replicates data (3 copies)Highly available

Integration

ADL Store

HDInsight

ADL StoreSDK-sJAVAC++.NETNode.jsPythonU-SQL ExtensibilityWebHDFS ClientLibWebHDFSYesYesBy GAManagement OperationsBy GAYesYesBy GA

Visual Studio Tools

ADL Analytics Billing ModelAccount is free

Pay for compute nodes for the duration of query

Formula (GA) = 5 cents + (minutes x parallelism x Analytics Unit Price)

Analytics Unit Price - $0.017 / minute

Preview 50% discount

ADL Store Billing ModelAccount is free

Pay for amount of data - $0.08 / GB per month

Pay for number of I/O operations - $0.14 / million transactions

Preview 50% discount

Security

Based on Azure Active Directory

Federate with Enterprise Active Directory

Two factor authentication

Security - AccessRole Based Access Control

Custom access with POSIX ACLs

Permissions for specific named users or groups

Security - RuntimeAll user code runs in VM

VM-s are locked down

Detailed audit records who, when, what, how long

Audit logs available out of the box

Security - EncryptionEncryption on the wire - Data uploaded via HTTPS

Encryption at rest after public preview

Integration with Azure Key Vault for keys

Encryption is optional

Security - Compliance

Certification

Azure compliance requirements

External auditing

DEMO

Tokyo Azure Meetup Learn | Share | Enjoy Cool Demos!