KOREA - maylife.co.krmaylife.co.kr/eDM/Dell_TechSummit/thanks/Day1... · revenue out of data High...

1 of Y Internal Use - Confidential

KOREATech Summit 2017

Software Defined Solutions

Han, Stanley

Internal Use - Confidential


ScaleIO

Project Nautilus


소프트웨어 정의 스토리지 매년 23% 이상 성장전망!

1 Wikibon Premium Report: Server SAN 2012-2026, David Floyer, 15 July 2015

The traditional storage model based on islands of proprietary code

supporting proprietary data services and proprietary management

tools will not be a significant part of this interconnected computing

beyond 2020.1

2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025 2026

Hyperscale Server SAN

Storage

Traditional Enterprise

Storage (SAN, NAS + DAS)

Enterprise Server SAN

and HCI

REVENUE PROJECTIONS


ScaleIO Ready Nodes

ScaleIO SoftwareVxRackFLEX

VxRail VxRackSDDC

vSAN Ready Nodes

vSANSoftware

ScaleIOvSAN

DellEMC / VMware View of SDS

Software-Defined Storage (SDS)

Server SAN Software (Compute + Storage + Single Pane

Storage Management Stack)

Hyper-Converged (HCI) Software (Compute + Storage + Network Virtualization Software +

Single Pane Management Stack)

Categories

Deployment

OptionsReady Nodes OEM ServersRack Scale HCI

Hyper-

Converged

(HCI)

Appliances*

Rack Scale

HCI

Ready

Nodes

OEM

Servers


클러스터 1

SSD SSDX86

X86

x86.

Fibre Channel

VM

SAN

SSD SSD SSD SSD HDD HDD

사용편의성이뛰어남, 가상머신과완벽하게통합

가상머신관리자가관리

단일플랫폼레벨에서리소스최적화

동종가상머신및이기종워크로드

이름만 SDS, SAN으로 가장

스케일아웃되지않음, 확장에따른비용지출모델이아님

하이퍼컨버지드구축불가

VM VM

vSphere

VM VM

vSphere

VM VM

다른 VM/운영 체제

유형 1: 재구성된 SDS

VM VM

vSphere

이더넷

유형 2 수직적으로통합

클러스터 2

SSD SSD

VM VM

vSphere

이더넷

클러스터 3

SSD SSD

VM VM


이더넷

이더넷

VM VM

vSphere

VM VM

vSphere

VM VM


세가지 유형의 SDS

유연성과성능이뛰어남, SAN 교체

스토리지/서버관리자가관리

데이터센터전반에걸쳐리소스최적화

이기종가상머신/OS, 이기종워크로드


ScaleIO• 추상화 - 풀링 - 자동화

100K IOPS

10TB

100K IOPS

10TB

100K IOPS

10TB

100K IOPS

10TB

100K IOPS

10TB

100K IOPS

10TB

100K IOPS

10TB

100K IOPS

10TB

100K IOPS

10TB

100K IOPS

10TB

ScaleIO는 HDD, SSD, 올플래시를 포함하여 각서버에서 로컬 스토리지추상화

1,000,000 IOPS

100TB

ScaleIO는 고립된 리소스없이 모든 스토리지리소스를 함께 풀로 구성

각 애플리케이션의 요구사항에 따라 리소스를자동으로 할당 및 조정

10TB

100K IOPS

30TB

50K IOPS

4TB

20K IOPS

2TB

35K IOPS

1TB

4K IOPS

17TB

20K IOPS

5TB

10K IOPS

10TB

10K IOPS

2TB

8K IOPS

5TB

5K IOPS

1

2

3


출시

190+ 레퍼런스전년대비성장률448%

–

400PB 이상판매

ScaleIO 1달러당하드웨어 6달러견인

60%의 고객이 180일이내에2세대 ScaleIO 구매

2016 회계연도에고객수173% 증가


세가지 활용 방법

VCE VxRack Systemwith FLEX

ScaleIO Ready

노드

탁월한스케일아웃 SDS

소프트웨어전용 완벽한유연성 최종사용자가서버제공 최종사용자가스위치제공 최종사용자가랙제공

스케일아웃블록스토리지

ScaleIO에대한튜닝, 최적화및검증을마친 Dell PowerEdge 서버

하이퍼컨버지드또는스토리지전용

올플래시구성

턴키소프트웨어정의IaaS

완벽하게제작되는플랫폼 사전통합되고논리적으로구성된 VCE

VCE 지원및수명주기보장

소프트웨어 정의최고의유연성

위험최소화, 최상의 가치,

최저수준의 TCO

0

1

0

1

0

1

1

0

1

1

0

1

1

0

1

1

0

0

1

1

0

1

1

ScaleIO소프트웨어 0

0

1

0

1

1

0

0

1

1

1

1

0

0

1

0

1

1

0

1

1


ScaleIO: 엔터프라이즈급 웹스케일 환경을 위해 설계

짧은지연시간으로높은입출력병렬처리량실현

최대 1,000만 IOPS의일관되게높은성능

스냅샷, 씬프로비저닝,

QoS, D@RE 등과같은주요기능

신속한복구를통한엔터프라이즈데이터보호

최대 99.999999%의가용성

3노드에서1,000노드이상으로원활하게확장가능

용량또는성능을독립적으로확장


KOREATech Summit 2017

Project Nautilus OverviewKim, YoungTaek(YT)

Consultant Corporate Systems Engineer

Elastic Cloud Stroage Engineering


Open Source

Community

Massive Data GrowthEmergence of

Real-Time Apps

Infrastructure

Commoditization

and Scale-Out

Rapid Ingest and

Dissemination of Data

to Apps

Monetize Data –

Analytics

Elastic, Re-playable,

Consistent, Durable,

Searchable

Market Drivers


Rising Needs for Big & Fast Data Solutions

Evolution and increase in IoT related

data growth

Explosion of data via social and

mobile apps

Desire of businesses to generate

revenue out of data

High speed ingestion of large volumes

of data

Continuous analysis of incoming data

−Real-time, interactive & batch

queries

Data driven decision making based

on analysis

Data Analytics solutions need to

deal with Data-At-Rest as well as

Data-In-Motion driven by:

Giving rise to the following

requirements:


A New Architecture Emerges: Streaming

• A new class of streaming systems is emerging to address the accidental

architecture’s problems and enable new applications not possible before

• Some of the unique characteristics of streaming applications

– Treat data as continuous and infinite

– Compute correct results in real-time with stateful, exactly-once processing

• These systems are applicable for real-time applications, batch applications,

and interactive applications

• Web-scale companies (Google, Twitter) are beginning to demonstrate the

disruptive value of streaming systems


What Analytics Stack to Target?

VMAX et al

Shared Block

Storage Array

SQL

Database

Data

Warehouse

Shared Block

Storage Array

SQL Databases?

• Began with simple SQL database + query

• Evolved to star schema, cubes, …

• Data’s final resting place: storage arrays

• Clear line of demarcation between data &

storage layers

• DELL EMC built storage arrays that implemented

many features and optimizations for DBs

Servers w/ DAS

MapReduce

HDFS

Hadoop?

• Emerged from web scale: Google …

• Design point: Massive scale, not low-

latency transactional semantics

• Data’s final resting place: DAS

• Shift in system design: storage now

integrated into the data platform

Project Nautilus

Cloud Scale Storage

Streaming

Engine

Streaming

Storage

Next Wave: Streaming• Emergence of streaming analytics driving a

new wave of technology

• Promises unified analytics across batch, real-

time, interactive

• New open source projects and startups

challenging Hadoop incumbents

• Proposed Nautilus Play: Target the next

analytics platform with streaming as its core

enabling us to operate at the data layer and

not just the storage layer


Stream Analytics in Open Source

Source: flink.apache.org

Community Momentum

Project Commercial

Vendor

Comments

dataArtisans True unification of batch

and stream

DataBricks

Cloudera, HWX

Micro-Batch

Confluent

MapR

Kafka Streams

MapR Streams

Hortonworks Hortonworks DataFlow

Google API supported by Spark

& Flink Runners

Flink

OSS Projects

Kafka

Beam

Source: Databricks Spark Users Survey


Streaming Analytics Opportunity

Market Snapshot by Verticals

Source:MarketsandMarkets, Stream Analytics Forecast

Algorithmic Trading

Real-time Customer Engagement

Location Intelligence

Operations Management

Supply Chain Optimization

Vehicle & Route Tracking

Network Monitoring

Real-time Patient Monitoring

Real-Time Call-Center Analysis

• Growing use cases across industry verticals

• Real-time monitoring

• Log Analysis

• Risk Analysis

• Location Intelligence

• Ability to handle data rate of millions of

messages/sec

• High speed ingestion of large volumes of data

• Continuous analysis of incoming data

• Real-time, interactive & batch queries

• Data driven decision making based on analysis


• Simplify real-time, batch, and interactive analytics

• Ingest data with low-latency Streaming Storage

• Process with an elastic, stateful Streaming Engine

• Land data in high-throughput Cloud Scale Storage

• Coherently orchestrate storage + compute resources

• Simplify the operations experience

• Provide enterprise security and robustness

Bring Streaming to the EnterpriseKEY FEATURES TO MAKE STREAMING WIDELY CONSUMABLE


Today’s Lamda Architecture

• IoT workflows are complex with

at least 4 different systems with

independent storage for each of

the components like ingestion,

stream, transfer and batch

• Several technology stacks are

required get to insights, securing

and protecting data is hard

• Managing this complex

ecosystem is costly and time

consuming

Accidental Architecture


Issues with Traditional Architecture

Traditional data platforms process data in finite and static batches - adding new realtime

analytics processing results in complex and inefficient data pipelines

• Infrastructure Admin

– Inefficient storage

– Deploy & maintain multiple clusters, complex capacity planning

– Disparate security models

– Individual DR for each cluster

– Complex analytics app testing and deployment workflows

• Data Scientist

– Separate batch and real-time programing models, hard to correlate output

– Disparate data sets with multiple ETL steps


A Case Study at Twitter

Problem Statement

For an incoming Tweet rate of 1.5 million Tweets/second, implement sentiment analysis

queries over live tweets as well as historical ones

Event Logger

Kafka Partition

Kafka Partition

Kafka Partition

TimeSeries

DB

HDFS

Pub-Sub Parse/Transform/Filter

K/V

Real-Time

Solution1. Replace STORM w/ Flink to get

correct real-time results

2. Persist correct hourly

aggregates into Time Series DB

directly from Flink eliminating

the need for the Hadoop batch

infrastructure

3. Utilize stateful properties of

Flink to eliminate large external

K/V store

4. Implement recent queries

directly against Flink’s internal

state

Batch

99.5% reduction in the hardware when

deploying the unified architecture


• Well designed streaming engines can produce correct, consistent & repeatable

results– Correctness via check-pointing application state, Exactly-Once delivery

– Repeatability via ability to replay events requires knowledge of event and processing time

Strongly Consistent storage layer a must to achieve the above

• Streaming engines can operate on both finite & infinite data sets– Batch is treated as a bounded stream!

• Beyond Batch: Programming model to reason about time– Given unbounded/unordered data sets of varying event-time skew

Key Learning from Success at TwitterSTATEFUL STREAM PROCESSING AT IN-MEMORY SPEEDS

Stream Processing in 2016 =

Hadoop in 2006Twitter POC has proven the value of stateful stream processing to

eliminate dual data processing architectures

Should not miss this opportunity to

take this new architectural paradigm

mainstream for enterprises.


Nautilus: A Unified Storage ArchitectureInterconnected layers each focused on a different aspect of the innovation space

Deep Storage LayerData’s Final Landing Place: Global Accessibility, Massive Capacity, Lowest Cost

Pravega Streams LayerRapid and Elastic Ingestion and Dissemination, Searchable,

Fluid App Data Types, Highest Performance

Storage Access LayerMulti-Protocol Accessibility, Higher Performance, Consistency and Durability Guarantees


Refactor the “Accidental Storage Stack”

Ingest

Buffer &

Pub/Sub

“Pravega Streams”

Scale-out Software Defined Storage

NoSQL DB SearchAnalytics

Engines

Using Logs as a Shared Storage Primitive

Ingest Buffer

& Pub/Sub

Proprietary

Log Storage

Local Files

DAS

Kafka

NoSQL DB

Proprietary

Log Storage

Local Files

DAS

Cassandra et al


Nautilus Vision

A Unified Platform For Big & Fast Data For The Enterprise

Enable Real-time Ingestion

• Support for low latency ingestion of continuous as well as trickle streams

• In-place analytics via Multi-protocol access: Kafka, HDFS & S3

• Multi-tenancy & Role Based access for streaming data

Unify Stream & Batch Processing

• Access to storage at memory speeds

• Fast checkpoint from streaming engine to storage

• Support for data science workflows

Reduce Storage & Operational costs

• Single cluster for storage (streaming, long-term) & analytics

• Seamless disaster recovery via geo-enabled storage

• Simplified architecture results in lower operational cost


Nautilus: A Unified Data PipelineUnified Analytics Exactly Once Processing Strongly Consistent Storage

Unified AnalyticsReal-Time, Batch, Interactive

Interactive exploration by Data Scientists

Real-time intelligence at the NOC

Sensors

Mobile Devices

App Logs

Pub/Sub Search

Pravega Streams

Unified Storage

S3… HDFS NFS

ECS/Isilon


Nautilus PlatformEnterprise grade platform unifying Nautilus services


Nautilus Use Cases

Connected Cars

Telematics, Remote

Diagnostics, Vehicle-X

Communications

Industrial

IoT

Energy,

Factory-As-A-Platform,

Manufacturing

Risk

Analytics

Fraud Detection,

Sensitivity Analysis,

Loss forecasting


Operational Scenario: TodayHow to test & deploy new version of an analytics business logic?

Archived DataHDFS or NFS or Object

Recent Data: Kafka Logs

x=5 z=6 y=2 x=4 a=7 y=5… …

older newer

HDFS API

ETL

StreamingApplicationBusiness Logic

StreamingApplicationBusiness Logic’

New version of app

deployed with different

data access methods

Challenges• Custom scripting and deployment required because historical

data is located in different storage system and accessed via

different data type (e.g. files vs. logs)

• Test run is not exactly like production due to mismatches

between log/file access and deployment differences – requires

more time, is error prone, and leads to inaccurate test results

• Often requires downtime if upgrading in place

• Upgrade “next to existing” requires complex workflow

Historical data

accessed via files

(not logs!) from

HDFS archive


Operational Scenario: With NautilusHow to test & deploy new version of a analytics business logic?

StreamingApplicationBusiness Logic

Streams

Tiering to/from ECS handled automatically by the Streaming Storage Subsystem

StreamingApplicationBusiness Logic’

New version of app

deployed exactly like

production1

Historical data accessed via

same stream as production

– just rewind the stream!

2

Once history is consumed,

seamlessly start reading real-

time data!

3

Once you are confident

things are working, turn off

old version and redirect

NOC consoles

4


Pravega StreamsA New Log Primitive Designed Specifically For Streaming Architectures

• Pravega is an open source distributed storage service offering a new storage

abstraction called a stream

• A stream is the foundation for building reliable streaming systems: a high-

performance, durable, elastic, and infinite append-only log with strict ordering and

consistency

• A stream is as lightweight as a file – you can create millions of them in a single

cluster

• Streams greatly simplify the development and operation of a variety of distributed

systems: messaging, databases, analytic engines, search engines, and so on


Pravega Architecture Goals

• All data is durable– Data is replicated and persisted to disk before being acknowledged

• Strict ordering guarantees and exactly once semantics– Across both tail and catch-up reads

– Client tracks read offset, Producers use transactions

• Lightweight, elastic, infinite, high performance– Support tens of millions of streams

– Dynamic partitioning of streams based on load and throughput SLO

– Size is not bounded by the capacity of a single node

– Low (<10ms) latency writes; throughput bounded by network bandwidth

– Read pattern (e.g. many catch-up reads) doesn’t affect write performance


Streaming Storage System

Pravega Architecture

Stream

Abstraction

Pravega Streaming Service

Cloud Scale Storage(Isilon or ECS)

• High-Throughput• High-Scale, Low-Cost

Low-Latency Storage

Apache Bookkeeper

Auto-Tiering

Messaging Apps

Real-Time / Batch / Interactive Predictive Analytics

Stream Processors: Spark, Flink, …

Other Apps & Middleware

Pravega Design Innovations

1. Zero-Touch Dynamic Scaling- Automatically scale

read/write parallelism

based on load and SLO

- No service interruptions

- No manual reconfiguration

of clients

- No manual reconfiguration

of service resources

2. Smart Workload Distribution

- No need to over-

provision servers for

peak load

3. I/O Path Isolation

- For tail writes

- For tail reads

- For catch-up reads

4. Tiering for “Infinite Streams”

5. Transactions For “Exactly

Once”


The Incumbent: Apache Kafka

• Distributed pub/sub messaging system

designed to be fast, scalable, durable

• Typical uses– High IOPs, low latency ingestion mechanism at the

start of a pipeline

– Pub/sub mechanism for downstream consumers

• Principle design innovation– Does not maintain server-side client state (e.g.

messages consumed)

– Clients keep their own state

– Messages kept on server for fixed time

Topic

Partitions

Topic

Pro

du

cers

Consumer

Topic

Pro

du

cers

Consumer

Kafka Broker

Messages


Why Not Extend Kafka?

Quality Pravega Goal Kafka Design Point

Data Durability Replicated and persisted to disk before ACK Replicated but not persisted to disk before ACK

Strict Ordering Consistent ordering on tail and catch-up reads Messages may get reordered

Exactly Once Producers can use transactions for atomicity Messages may get duplicated

Scale Tens of millions of streams per cluster Thousands of topics per cluster

Elastic Dynamic partitioning of streams based on load and SLO Statically configured partitions

Size

Log size is not bounded by the capacity of any single nodePartition size is bounded by capacity of filesystem

on its hosting node

Transparently migrate/retrieve data from Tier 2 storage for

older parts of the log

External ETL required to move data to Tier 2

storage; no access to data via Kafka once moved

Performance

Low (<10ms) latency durable writes; throughput bounded by

network bandwidth

Low-latency achieved only by reducing

replication/reliability parameters

Read pattern (e.g. many catch-up readers) does not affect

write performance

Read patterns adversely affects write performance

due to reliance on OS filesystem cache

✗✗✗

✗✗

✗

✗

✗

✗


StreamProcessor

App State

App Logic

Worker

Worker

Pravega Optimizations for Stream Processors

InputStream …

Worker

…Segment

Memory-Speed Storage

Dynamically split input stream into

parallel logs: infinite sequence,

low-latency, durable, re-playable

with auto-tiering from hot to cold

storage.

1

Coordinate via protocol

between streaming storage

and streaming engine to

systematically scale up and

down the number of logs and

source workers based on

load variance over time

2

Address requirements of

streaming apps with very

large state by connecting to

memory speed storage

3

Support streaming write COMMIT operation to

extend Exactly Once processing semantics across

multiple, chained applications4

So

cia

l, I

oT

Pro

du

cers

OutputStream

StreamProcessor

2nd App

LogSink

Segment

Segment


Pravega Benefits

• Low-Latency Tail + High-Throughput History

• Persistent and Durable

• Atomic, Batched Writes

• No Scalability Limits

• Dynamic Scaling Up and Down

• Infinite Retention

• Delivery Guarantee + Consistently Replay-able Reads


Flink in Nautilus

• Automatic Cluster Deployment

• Integrated Security

• Leverage PKI, Kerberos Infrastructure

• Single Sign-On for Flink Web UI

• Support for Scheduled Jobs & Long-Running

Jobs

• Savepoint Management

• Pravega Integration

• Connectors as data source/sink

• Exactly-Once Semantics

• End-to-end dynamic scaling

ECS

Pravega


Nautilus WorkspacesOrganizes the people, information and software related to a specific project or department

• A visual team or department space

• Security and resource-

management

• Gateway to ECS/Pravega storage

• Analytics job management

KOREA - maylife.co.krmaylife.co.kr/eDM/Dell_TechSummit/thanks/Day1... · revenue out of data High...

Documents

Transcript of KOREA - maylife.co.krmaylife.co.kr/eDM/Dell_TechSummit/thanks/Day1... · revenue out of data High...