開放x穩健的大數據平台創造互聯網+競爭力
2015 Lenovo. All rights reserved.
曾文興聯想台灣商用業務部副總經理
2015/11/11 @twitterhandle
22015 Lenovo. All rights reserved.
簡報大綱
資料所帶來的改變
管理、分析大數據所面臨的挑戰
邁向大數據分析的途徑
聯想-針對Hadoop設計的參考架構
聯想專長
32015 Lenovo. All rights reserved.
重點摘要
資料是最新的資源– It takes more forms – at rest, in motion, structured, unstructured, internal, external. Analytics gives a more vivid
picture of your business and the forces that affect it. Outperformers are 3.6 times more likely to apply analytics to move more quickly with less time between insight and action.
決策層級大量增加 Decision-making extends from few to many
– See not just ‘what happened?’, but ‘why did it happen?’ and ‘what is likely to happen?’. Even ‘Tell me, what’s the best course of action based on what you’ve learned?’. Empower team members at every level of the organization, in every role, to make better decisions.
當資料的價值增加時,現有系統無法跟上腳步– As you recognize the revenue potential from analyzing and acting on data, the demand for business insight will
escalate. But your IT budget can’t keep up with the increased demand, volume of data, and complexity. Top performing organizations confront this reality by taking a fundamentally different approach to architecture, tools and practices.
Lenovo has a rich portfolio of Reference Architecture solutions for big data that meet such needs of progressive organizations.
2015 Lenovo. All rights reserved.
大數據的成長
52015 Lenovo. All rights reserved.
大數據/分析的巨量成長
More data + more devices = greater demand for data Insights
Most companies
analyze only
12%of their data2
50XYOY enterprise
data growth through 20203
Internet of Things
(IoT) devices will
grow from 11.4B to
28.1Bpredicted by 20204
90%Of data is less
than 2 years old1
1. US Chamber of Commerce, Forum for Innovation 2013
2. Forrester 2014
62015 Lenovo. All rights reserved.
面臨的挑戰
Identify areas for business growth
– Attract, maintain, grow clients (e.g. mass personalization)
Improve efficiency (deliver insights faster)
– Cost: HW utilization, setup difficulty
– Speed: Faster setup, better response time
– Reliability/availability: extensive Predictive Failure Analysis (PFA)
Manage risk/fraud
– Identify questionable transactions
Of leaders cite growth as the key source of value
from analytics2
1 IDC, 2011
2 Source: Analytics: A blueprint for value – Converting big data and analytics into results, IBM Institute for Business Value, 2013.
80% of
data growth is
unstructured
8 zettabytes of
digital content
created by 2015,
up from 2.7 ZB
in 2012 Labor costs
will be
70% of IT spend
by 2013
71% of CEOs
identify technology
as the
most important external force
impacting their
organizations
Social Media data
is less certain and
time sensitive
In spite of challenges like these…
…IT must deliver business value:
New technologies
and algorithms demand
new skills & expertise
72015 Lenovo. All rights reserved.
每一種產業皆可利用大數據分析而有所受益
Insurance
• 360˚ View of Domain or Subject
• Catastrophe Modeling
• Fraud & Abuse
Banking
• Optimizing Offers and Cross-sell
• Customer Service and Call Center Efficiency
Telco
• Pro-active Call Center
• Network Analytics
• Location Based Services
Energy & Utilities
• Smart Meter Analytics
• Distribution Load Forecasting/Scheduling
• Condition Based Maintenance
Media & Entertainment
• Business process transformation
• Audience & Marketing Optimization
Retail
• Actionable Customer
Insight
• Merchandise Optimization
• Dynamic Pricing
Travel & Transport
• Customer Analytics &
Loyalty Marketing
• Predictive Maintenance
Analytics
Consumer Products
• Shelf Availability
• Promotional Spend
Optimization
• Merchandising Compliance
Government
• Civilian Services
• Defense & Intelligence
• Tax & Treasury Services
Healthcare
• Measure & Act on Population Health Outcomes
• Engage Consumers in their Healthcare
Automotive
• Advanced Condition
Monitoring
• Data Warehouse
Optimization
Life Sciences
• Increase visibility into drug
safety and effectiveness
Chemical & Petroleum
• Operational Surveillance,
Analysis & Optimization
• Data Warehouse
Consolidation, Integration &
Augmentation
Aerospace & Defense
• Uniform Information Access
Platform
• Data Warehouse
Optimization
Electronics
• Customer/ Channel
Analytics
• Advanced Condition
Monitoring
82015 Lenovo. All rights reserved.
使用 Hadoop 來解放資料的價值
Inspired by Google technologies
– Yahoo! adopted these technologies and open-sourced them into the Apache Hadoop project
Hugely scalable with thousands of nodes and petabytes of data
Handles both structured and unstructured data
Transaction and
application data
Data
Machine,
sensor data
Enterprise
content
Image,
geospatial,
video
Social data
Analytics is the key to
unlocking Business Insights
92015 Lenovo. All rights reserved.
大數據應被提高為策略層次…not as science projects
“The Science Experiment”
General scenario Soon after
Start an “unbalanced configuration”
Leverage an open-source-only solution
Execute a “back of the envelope” project
Pile on more nodes to address the data
volume
Lack of cluster management degrades QoS
Workload bottlenecks occur due to
networking issues
Issues integrating with BI/DW resources
Users need more, but you can’t deliver
Our clients’ experiences suggest that considering the IT
architecture up-front results in fewer problems.
102015 Lenovo. All rights reserved.
透過大數據分析來實現價值
PLATFORM
Source: Analytics: A blueprint for value – Converting big data and analytics into results, IBM Institute for Business Value, 2013
IBM Institute for Business Value Study identified key levers
Culture
Many are organizational
factors
One factor means
your solution providers
are critical
Integrate
Hardware & software
to manage Big Data
112015 Lenovo. All rights reserved.
建立大數據架構所應考量因素…when you implement and optimize a Hadoop workload
Scaling?
Core-to-disk
Nodes/rack
Hadoop complexity
參考架構概述
2015 Lenovo. All rights reserved.
132015 Lenovo Internal. All rights
reserved.
聯想-Cloudera大數據 參考架構
– 整合優化並經過完整測試
– Cloudera 認證
– 幫助IT部門在大數據上可快速啟動和運行分析
– 平行運算環境下,可將巨量數據”區塊化”處理
– 規模靈活百變,助您隨工作量遞增而發展
Sample configuration of Lenovo Big Data Reference Architecture for Cloudera Distribution for Hadoop
142015 Lenovo Internal. All rights
reserved.
聯想-MapR大數據 參考架構
– MAPR容器架構來存儲元數據( MetaData )
– 為集群管理提供可靠的服務
– 整合優化並經過完整測試
– MAPR部署在聯想的伺服器和網路模組,提供了優越的性能,可靠性和可擴展性.
Sample configuration of Lenovo Big Data Reference Architecture for MapR Distribution including Apache Hadoop
152015 Lenovo Internal. All rights
reserved.
聯想-InfoSphere大數據 參考架構
– 發揮Hadoop Apache的力量
– 管理大量的結構化和非結構化數據。
– 強化Hadoop的技術符合企業工作負載的需求
– 結合IBM的InfoSphere BigInsights 以及InfoSphere的軟件,該架構使IT部門能夠快速部署經過驗證的設計。
Sample configuration of Lenovo Big Data Reference Architecture for IBM InfoSphere BigInsights
162015 Lenovo. All rights reserved.
參考架構:Hadoop架構不僅是伺服器的集群(cluster); 他需要透過全方面的考量來加以設計並確保企業實現其目標
The Reference Architecture Advantage
專家定義的藍圖: 通過硬體,軟體,網路 的專家設計
靈活: 以客戶特定工作負載要求優化組件選擇
動態:擴展納入新的Hadoop和基礎設施能力和功能
認證:確保功能,即使在重負載環境下確保性能
擴充性:從小規模並透過擴展至數千節點,可在一個一致的架構下實現相符的性能
彈性: 內建於架構中的備援以及高可用性,可針對客戶的需求加以客製化
整合:可以提供一個預先集成系統,已整合具有硬體/軟體和網路
Reference Architecture
Performance and reliability
Network, processor, and storage options
Hadoop software flexibility
Fast time-to-value with factory-integrated hardware/software
Lab Services pre/post sales skills`
聯想-Hadoop大數據參考架構
172015 Lenovo. All rights reserved.
System x3550 M5 - 1U workhorse
Versatile storage configurations for diverse workloads (up to 46 TB) - Up to 10 + 2 x 2.5” or 4 x 3.5” HDDs/SSDs
#1 Reliability Built-in for exceptional uptime, reduced costs New proactive tools (e.g. Next Gen Light Path Diagnostic Panel)
Extreme Efficiency for energy & space savings 1,512 cores, 64.5TB memory, & 504 HDDs in 42U rack Titanium PSUs with Active/Standby Management Extended operating temperature Dual fan zones with N+1 design
Built-in, ground-breaking System x Trusted Platform Assurance security for superior hardware & firmware protection
Outstanding performance with Intel Xeon E5-2600v3 processors & TruDDR4 Memory for more VMs & workloads
Cloud Virtualization Virtual Desktop High Performance Computing
Web ServingBig Data & Analytics Email File & Print
Business Continuity Infrastructure & Security Data Management
50%More Cores
& Cache
2XMemory Capacity
39%Greater
ComputationalPerformance
61%Greater
VirtualizationPerformance
50%Increased Memory
Bandwidth
45%Memory Power
Savings
Approximately
2XMore VMs
Compact, powerful two-socket rack server, packed with the performance & reliability to fuel any workload
18
System x3650 M5 - Serving Diverse Workloads Worldwide
FLEXIBLE STORAGE CONFIGURATIONS for diverse workloads: Up to 26 x 2.5” or 14 x 3.5” + 2 x 2.5” HDDs/SSDs with up to 100TB
#1 RELIABILITY BUILT-IN for exceptional uptime, reduced costs
New proactive diagnostic tools (Next Gen Light Path Diagnostic Panel)
INNOVATIVE POWER & THERMAL MANAGEMENT DESIGN for smart energy savings
BUILT-IN SYSTEM x TRUSTED PLATFORM ASSURANCE SECURITY for superior hardware & firmware protection
OUTSTANDING PERFORMANCE for more VMs & workloads Intel Xeon E5-2600v3 processors TruDDR4 Memory Up to four optional GPUs 12Gbps end-to-end platform support with up to 4 RAID controllers
50%More Cores
& Cache
2XMemory Capacity
59%Faster DB
Performance
61%Greater
VirtualizationPerformance
50%Increased Memory
Bandwidth
45%Memory Power
Savings
62%More
HDD Density
Approximately
2XMore VMs
66%Greater SAP
performance
Cloud Virtualization Virtual Desktop High Performance Computing
Web ServingBig Data & Analytics Email File & Print
Business Continuity Infrastructure & Security Data Management
Commanding performance & versatility, integrated with leadership reliability and security for enterprise solutions #1
#1 Reliability
2015 Lenovo. All rights reserved.
192015 Lenovo. All rights reserved.
Hadoop 網路設計元素
- 1 Gb network switch– G8052 for administration network and IMM2 network
– 48 1Gb RJ45 ports with 4 SFP+ uplinks for 1/10Gb connectivity
– Designed with line-rate throughput and low latency (1.8 microseconds).
– Includes redundant and hot-swappable power supplies and fans.
- 10 Gb network switch– G8272 for performance Data network – well matched to server throughput
– 1U switch with 48 x SFP+ 10GbE ports plus 6 x QSFP+ 40GbE ports
– Support up to 72 x 10Gb connections using break-out cables
– 1.44 Tbps non-blocking throughput with very low latency (~ 600 ns)
202015 Lenovo. All rights reserved.
網路設計對於成功的大數據基礎設施是至關重要
聯強專為 Hadoop設計的大數據參考架構 基於網絡設計的最佳實踐→我們設計和建造交換機,所以我們有足夠的專業知識來設計理想解決方案 我們提供聯想交換器並且成功部屬最佳網絡架構
Example of network topology diagram from Hadoop Reference Architecture
設計用來提供:–可擴展到數千個伺服器– Granular expansion with
minimum network reconfiguration (最小的網絡重構)
–高網路性能– 高可用性– 安全性–靈活地滿足客戶的網路策略
G8272
G8272
212015 Lenovo. All rights reserved.
聯想針對Hadoop的集群設計
Lenovo Big Data Reference
Architecture for Hadoop
Up to 20 System x3650 M5 data nodes
– Up to 1,120TB storage
– Up to 1.5TB memory
Up to 6 System x3550 M5 management
nodes
Up to two Lenovo RackSwitch 10Gb
Ethernet switches
Edge nodes
Scalable to multirack configurations
Improved Hadoop application
performance to 9x depending on
workload1
Built on Lenovo M5 Server technology
faster
time to value !
Preassembled racks
Customized to your needs
Cluster is validated by architects
before order
Integrated and tested1
Supported as a solution
Tailored to your needs
Start small…and grow
Easy to order
Easy to manage
Server + networking + system software
Delivered Built
1. InfoSphere BigInsights or other Hadoop software installations not included; Lenovo Lab
Services engagements are available for installation/customization
222015 Lenovo. All rights reserved.
為什麼聯想是針對Hadoop的最佳環境
RequirementsLenovo Big Data
Reference Architectures
Powerful, fast, and proven hardware platform
Reference architecture designed/optimized for Big Data
Networking elements and capabilities
Local resources and skills to design, implement and support
Scalable and can grow as the client grows
Proven, with large customer references
Leverages your existing partners and investments
Low cost, and the right total cost of ownership
232015 Lenovo. All rights reserved.
聯想積極參與大數據的生態系統
252015 Lenovo. All rights reserved.
Legal
© 2015 Lenovo. All rights reserved.
Availability: Offers, prices, specifications and availability may change without notice. Lenovo is not responsible for photographic or typographic errors or omissions.
Warranty: For a copy of applicable Lenovo warranties, write to: Warranty Information, 500 Park Offices Drive, RTP, NC, 27709, Attn: Dept. ZPYA/B600. Lenovo makes no representation or warranty regarding third-party products or services.
Trademarks: Lenovo, the Lenovo logo, and System x are trademarks or registered trademarks of Lenovo in the United States, other countries, or both. Other company, product, and service names may be trademarks or service marks of others. Visit http://www.lenovo.com/lenovo/us/en/safecomp.html periodically for the latest information on safe and effective computing.
2015 Lenovo. All rights reserved.
Top Related