x穩健的大數據平台創造互聯網競爭力曾... · 2015-11-20 · 2 Source: Analytics: A...

開放x穩健的大數據平台創造互聯網+競爭力

2015 Lenovo. All rights reserved.

曾文興聯想台灣商用業務部副總經理

2015/11/11 @twitterhandle


簡報大綱

資料所帶來的改變

管理、分析大數據所面臨的挑戰

邁向大數據分析的途徑

聯想-針對Hadoop設計的參考架構

聯想專長


重點摘要

資料是最新的資源– It takes more forms – at rest, in motion, structured, unstructured, internal, external. Analytics gives a more vivid

picture of your business and the forces that affect it. Outperformers are 3.6 times more likely to apply analytics to move more quickly with less time between insight and action.

決策層級大量增加 Decision-making extends from few to many

– See not just ‘what happened?’, but ‘why did it happen?’ and ‘what is likely to happen?’. Even ‘Tell me, what’s the best course of action based on what you’ve learned?’. Empower team members at every level of the organization, in every role, to make better decisions.

當資料的價值增加時，現有系統無法跟上腳步– As you recognize the revenue potential from analyzing and acting on data, the demand for business insight will

escalate. But your IT budget can’t keep up with the increased demand, volume of data, and complexity. Top performing organizations confront this reality by taking a fundamentally different approach to architecture, tools and practices.

Lenovo has a rich portfolio of Reference Architecture solutions for big data that meet such needs of progressive organizations.


大數據的成長


大數據／分析的巨量成長

More data + more devices = greater demand for data Insights

Most companies

analyze only

12%of their data2

50XYOY enterprise

data growth through 20203

Internet of Things

(IoT) devices will

grow from 11.4B to

28.1Bpredicted by 20204

90%Of data is less

than 2 years old1

1. US Chamber of Commerce, Forum for Innovation 2013

2. Forrester 2014


面臨的挑戰

Identify areas for business growth

– Attract, maintain, grow clients (e.g. mass personalization)

Improve efficiency (deliver insights faster)

– Cost: HW utilization, setup difficulty

– Speed: Faster setup, better response time

– Reliability/availability: extensive Predictive Failure Analysis (PFA)

Manage risk/fraud

– Identify questionable transactions

Of leaders cite growth as the key source of value

from analytics2

1 IDC, 2011

2 Source: Analytics: A blueprint for value – Converting big data and analytics into results, IBM Institute for Business Value, 2013.

80% of

data growth is

unstructured

8 zettabytes of

digital content

created by 2015,

up from 2.7 ZB

in 2012 Labor costs

will be

70% of IT spend

by 2013

71% of CEOs

identify technology

as the

most important external force

impacting their

organizations

Social Media data

is less certain and

time sensitive

In spite of challenges like these…

…IT must deliver business value:

New technologies

and algorithms demand

new skills & expertise


每一種產業皆可利用大數據分析而有所受益

Insurance

• 360˚ View of Domain or Subject

• Catastrophe Modeling

• Fraud & Abuse

Banking

• Optimizing Offers and Cross-sell

• Customer Service and Call Center Efficiency

Telco

• Pro-active Call Center

• Network Analytics

• Location Based Services

Energy & Utilities

• Smart Meter Analytics

• Distribution Load Forecasting/Scheduling

• Condition Based Maintenance

Media & Entertainment

• Business process transformation

• Audience & Marketing Optimization

Retail

• Actionable Customer

Insight

• Merchandise Optimization

• Dynamic Pricing

Travel & Transport

• Customer Analytics &

Loyalty Marketing

• Predictive Maintenance

Analytics

Consumer Products

• Shelf Availability

• Promotional Spend

Optimization

• Merchandising Compliance

Government

• Civilian Services

• Defense & Intelligence

• Tax & Treasury Services

Healthcare

• Measure & Act on Population Health Outcomes

• Engage Consumers in their Healthcare

Automotive

• Advanced Condition

Monitoring

• Data Warehouse

Optimization

Life Sciences

• Increase visibility into drug

safety and effectiveness

Chemical & Petroleum

• Operational Surveillance,

Analysis & Optimization

• Data Warehouse

Consolidation, Integration &

Augmentation

Aerospace & Defense

• Uniform Information Access

Platform

• Data Warehouse

Optimization

Electronics

• Customer/ Channel

Analytics

• Advanced Condition

Monitoring


使用 Hadoop 來解放資料的價值

Inspired by Google technologies

– Yahoo! adopted these technologies and open-sourced them into the Apache Hadoop project

Hugely scalable with thousands of nodes and petabytes of data

Handles both structured and unstructured data

Transaction and

application data

Data

Machine,

sensor data

Enterprise

content

Image,

geospatial,

video

Social data

Analytics is the key to

unlocking Business Insights


大數據應被提高為策略層次…not as science projects

“The Science Experiment”

General scenario Soon after

Start an “unbalanced configuration”

Leverage an open-source-only solution

Execute a “back of the envelope” project

Pile on more nodes to address the data

volume

Lack of cluster management degrades QoS

Workload bottlenecks occur due to

networking issues

Issues integrating with BI/DW resources

Users need more, but you can’t deliver

Our clients’ experiences suggest that considering the IT

architecture up-front results in fewer problems.


透過大數據分析來實現價值

PLATFORM

Source: Analytics: A blueprint for value – Converting big data and analytics into results, IBM Institute for Business Value, 2013

IBM Institute for Business Value Study identified key levers

Culture

Many are organizational

factors

One factor means

your solution providers

are critical

Integrate

Hardware & software

to manage Big Data


建立大數據架構所應考量因素…when you implement and optimize a Hadoop workload

Scaling?

Core-to-disk

Nodes/rack

Hadoop complexity

參考架構概述


132015 Lenovo Internal. All rights

reserved.

聯想-Cloudera大數據參考架構

– 整合優化並經過完整測試

– Cloudera 認證

– 幫助IT部門在大數據上可快速啟動和運行分析

– 平行運算環境下，可將巨量數據”區塊化”處理

– 規模靈活百變，助您隨工作量遞增而發展

Sample configuration of Lenovo Big Data Reference Architecture for Cloudera Distribution for Hadoop


reserved.

聯想-MapR大數據參考架構

– MAPR容器架構來存儲元數據( MetaData )

– 為集群管理提供可靠的服務

– 整合優化並經過完整測試

– MAPR部署在聯想的伺服器和網路模組，提供了優越的性能，可靠性和可擴展性.

Sample configuration of Lenovo Big Data Reference Architecture for MapR Distribution including Apache Hadoop


reserved.

聯想-InfoSphere大數據參考架構

– 發揮Hadoop Apache的力量

– 管理大量的結構化和非結構化數據。

– 強化Hadoop的技術符合企業工作負載的需求

– 結合IBM的InfoSphere BigInsights 以及InfoSphere的軟件，該架構使IT部門能夠快速部署經過驗證的設計。

Sample configuration of Lenovo Big Data Reference Architecture for IBM InfoSphere BigInsights


參考架構:Hadoop架構不僅是伺服器的集群(cluster); 他需要透過全方面的考量來加以設計並確保企業實現其目標

The Reference Architecture Advantage

專家定義的藍圖: 通過硬體，軟體，網路的專家設計

靈活: 以客戶特定工作負載要求優化組件選擇

動態:擴展納入新的Hadoop和基礎設施能力和功能

認證:確保功能，即使在重負載環境下確保性能

擴充性:從小規模並透過擴展至數千節點，可在一個一致的架構下實現相符的性能

彈性: 內建於架構中的備援以及高可用性，可針對客戶的需求加以客製化

整合:可以提供一個預先集成系統，已整合具有硬體/軟體和網路

Reference Architecture

Performance and reliability

Network, processor, and storage options

Hadoop software flexibility

Fast time-to-value with factory-integrated hardware/software

Lab Services pre/post sales skills`

聯想-Hadoop大數據參考架構


System x3550 M5 - 1U workhorse

Versatile storage configurations for diverse workloads (up to 46 TB) - Up to 10 + 2 x 2.5” or 4 x 3.5” HDDs/SSDs

#1 Reliability Built-in for exceptional uptime, reduced costs New proactive tools (e.g. Next Gen Light Path Diagnostic Panel)

Extreme Efficiency for energy & space savings 1,512 cores, 64.5TB memory, & 504 HDDs in 42U rack Titanium PSUs with Active/Standby Management Extended operating temperature Dual fan zones with N+1 design

Built-in, ground-breaking System x Trusted Platform Assurance security for superior hardware & firmware protection

Outstanding performance with Intel Xeon E5-2600v3 processors & TruDDR4 Memory for more VMs & workloads

Cloud Virtualization Virtual Desktop High Performance Computing

Web ServingBig Data & Analytics Email File & Print

Business Continuity Infrastructure & Security Data Management

50%More Cores

& Cache

2XMemory Capacity

39%Greater

ComputationalPerformance

61%Greater

VirtualizationPerformance

50%Increased Memory

Bandwidth

45%Memory Power

Savings

Approximately

2XMore VMs

Compact, powerful two-socket rack server, packed with the performance & reliability to fuel any workload

18

System x3650 M5 - Serving Diverse Workloads Worldwide

FLEXIBLE STORAGE CONFIGURATIONS for diverse workloads: Up to 26 x 2.5” or 14 x 3.5” + 2 x 2.5” HDDs/SSDs with up to 100TB

#1 RELIABILITY BUILT-IN for exceptional uptime, reduced costs

New proactive diagnostic tools (Next Gen Light Path Diagnostic Panel)

INNOVATIVE POWER & THERMAL MANAGEMENT DESIGN for smart energy savings

BUILT-IN SYSTEM x TRUSTED PLATFORM ASSURANCE SECURITY for superior hardware & firmware protection

OUTSTANDING PERFORMANCE for more VMs & workloads Intel Xeon E5-2600v3 processors TruDDR4 Memory Up to four optional GPUs 12Gbps end-to-end platform support with up to 4 RAID controllers

50%More Cores

& Cache

2XMemory Capacity

59%Faster DB

Performance

61%Greater

VirtualizationPerformance

50%Increased Memory

Bandwidth

45%Memory Power

Savings

62%More

HDD Density

Approximately

2XMore VMs

66%Greater SAP

performance

Cloud Virtualization Virtual Desktop High Performance Computing

Web ServingBig Data & Analytics Email File & Print

Business Continuity Infrastructure & Security Data Management

Commanding performance & versatility, integrated with leadership reliability and security for enterprise solutions #1

#1 Reliability



Hadoop 網路設計元素

- 1 Gb network switch– G8052 for administration network and IMM2 network

– 48 1Gb RJ45 ports with 4 SFP+ uplinks for 1/10Gb connectivity

– Designed with line-rate throughput and low latency (1.8 microseconds).

– Includes redundant and hot-swappable power supplies and fans.

- 10 Gb network switch– G8272 for performance Data network – well matched to server throughput

– 1U switch with 48 x SFP+ 10GbE ports plus 6 x QSFP+ 40GbE ports

– Support up to 72 x 10Gb connections using break-out cables

– 1.44 Tbps non-blocking throughput with very low latency (~ 600 ns)


網路設計對於成功的大數據基礎設施是至關重要

聯強專為 Hadoop設計的大數據參考架構基於網絡設計的最佳實踐→我們設計和建造交換機,所以我們有足夠的專業知識來設計理想解決方案我們提供聯想交換器並且成功部屬最佳網絡架構

Example of network topology diagram from Hadoop Reference Architecture

設計用來提供:–可擴展到數千個伺服器– Granular expansion with

minimum network reconfiguration (最小的網絡重構)

–高網路性能– 高可用性– 安全性–靈活地滿足客戶的網路策略

G8272

G8272


聯想針對Hadoop的集群設計

Lenovo Big Data Reference

Architecture for Hadoop

Up to 20 System x3650 M5 data nodes

– Up to 1,120TB storage

– Up to 1.5TB memory

Up to 6 System x3550 M5 management

nodes

Up to two Lenovo RackSwitch 10Gb

Ethernet switches

Edge nodes

Scalable to multirack configurations

Improved Hadoop application

performance to 9x depending on

workload1

Built on Lenovo M5 Server technology

faster

time to value !

Preassembled racks

Customized to your needs

Cluster is validated by architects

before order

Integrated and tested1

Supported as a solution

Tailored to your needs

Start small…and grow

Easy to order

Easy to manage

Server + networking + system software

Delivered Built

1. InfoSphere BigInsights or other Hadoop software installations not included; Lenovo Lab

Services engagements are available for installation/customization


為什麼聯想是針對Hadoop的最佳環境

RequirementsLenovo Big Data

Reference Architectures

Powerful, fast, and proven hardware platform

Reference architecture designed/optimized for Big Data

Networking elements and capabilities

Local resources and skills to design, implement and support

Scalable and can grow as the client grows

Proven, with large customer references

Leverages your existing partners and investments

Low cost, and the right total cost of ownership


聯想積極參與大數據的生態系統


Legal

© 2015 Lenovo. All rights reserved.

Availability: Offers, prices, specifications and availability may change without notice. Lenovo is not responsible for photographic or typographic errors or omissions.

Warranty: For a copy of applicable Lenovo warranties, write to: Warranty Information, 500 Park Offices Drive, RTP, NC, 27709, Attn: Dept. ZPYA/B600. Lenovo makes no representation or warranty regarding third-party products or services.

Trademarks: Lenovo, the Lenovo logo, and System x are trademarks or registered trademarks of Lenovo in the United States, other countries, or both. Other company, product, and service names may be trademarks or service marks of others. Visit http://www.lenovo.com/lenovo/us/en/safecomp.html periodically for the latest information on safe and effective computing.

http://www.lenovo.com/lenovo/us/en/safecomp.html

x穩健的大數據平台創造互聯網競爭力曾... · 2015-11-20 · 2 Source: Analytics: A...

Documents

Transcript of x穩健的大數據平台創造互聯網競爭力曾... · 2015-11-20 · 2 Source: Analytics: A...

x穩健的大數據平台 創造互聯網 競爭力 曾... · 2015-11-20 · 2 Source: Analytics: A...

Documents

Transcript of x穩健的大數據平台 創造互聯網 競爭力 曾... · 2015-11-20 · 2 Source: Analytics: A...

x穩健的大數據平台創造互聯網競爭力曾... · 2015-11-20 · 2 Source: Analytics: A...

Transcript of x穩健的大數據平台創造互聯網競爭力曾... · 2015-11-20 · 2 Source: Analytics: A...