1

COSBench: A Benchmark Tool for Cloud Object Storage Services

Jiangang Duan (段建钢)

2012/8

2

Agenda

• Self introduction

• Cloud Storage in tomorrow’s Data Center

• COSBench Introduction

• Case Study to evaluate OpenStack* swift performance with COSBench

• Summary

3

Self introduction

• Jiangang Duan

• Working in Cloud Infrastructure Technology Team (CITT) of Intel APAC R&D Ltd.

• We are software team, good at performance

• Try to understand how to build an efficient/scale Cloud Solution with Open Source software (OpenStack*, Xen*, KVM*)

• All of work will be contributed to Community

• Today we will talk about some efforts we try to measure OpenStack* performance and know people who want to contribute to OpenStack and work together

4

Storage

Open Platforms

Network

Common Fabrics

Compute

Flexible Workloads

Virtualized Infrastructure

Data Centers are Evolving

• Data centers are built upon three fundamental pillars:

– Compute

– Storage

– Networking

• All three are critical for efficient data center operations

– Balanced in performance and utilization

A Balanced Data Center is Essential for Efficiency

5

IDC Storage Capacity Growth†

Structured data (23.6% CAGR) Traditional enterprise database

Replicated data (24.2% CAGR) Backups

Data warehouses

Unstructured data (54.8% CAGR) Archives

Content Depots (75.6% CAGR) Web

Email

Document sharing

Social network content (pictures/videos) 10

20

30

40

50

60

70

80

90

2009 2010 2011 2012 2013 2014

EB

†Source: IDC, Worldwide Enterprise Storage Systems 2010–2014 Forecast: Recovery, Efficiency, and Digitization Shaping Customer Requirements for Storage Systems, Doc

‡Source: Internal estimates based on the IDC Worldwide Enterprise Storage Systems Forecast # 223234., May 2011

2012 Deployment ~7.6 million drives Estimate: ~500,000 storage systems‡

6

Usage Models Dictate the Solutions

Key Storage Usage Models Have Differing Requirements

Thus Need New Benchmarks

Sto

rag

e P

erfo

rm

an

ce

Req

uir

em

en

t (O

bje

cts

per s

eco

nd

)

Storage Capacity Requirement

Application data store (e.g. e-mail, VM/Boot, Sharepoint*)

Large Relational DB (e.g. NoSQL, non ACID)

Content distribution network (CDN)

Business DB (OLTP, OLAP)

Ran

do

m s

mall

Seq

uen

tial

Larg

e

Gigabytes Terabytes Petabytes Exabytes

Performance Storage

Capacity Storage

Backup and archive (server and client)

Cloud Object storage (e.g. photos/videos)

Large analytics (e.g Hadoop*/HDFS)

High performance compute (e.g. pNFS, Luster*)

COSBench

7

COSBench Introduction

• COSBench is an Intel developed benchmark to measure Cloud Object Storage Service performance

• Cloud end user can use COSBench to compare different public Cloud Object Storage service performance

• Cloud provider can use it to

– Compare different Hardware/Software Stacks

– Identify bottleneck and make optimization

COSBench is the IOMeter for Cloud Object Storage service

8

COSBench Key Component

Config.xml:

– define workload with flexibility.

Controller:

– Control all drivers

– Collect and aggregate stats.

Driver:

– generate load w/ config.xml parameters.

– can run tests w/o controller.

Web Console:

– Manage controller

– Browse real-time stats

– Communication is based on HTTP (RESTful style)

Storage Cloud

COSBench

Controller

Driver Driver

Controller Node

Storage Node

Config.xml

Web Console

9

Web Console

Intuitive UI to get Overview.

Driver list

Workload List

History list

10

Workload Configuration

Flexible configuration parameters is capable of complex Cases

Workflow for complex stages

Read/Write Operations

Flexible load control

object size distribution

11

Performance Metrics

Throughput (Operations/s): the operations completed in one second

Response Time (in ms): the duration between operation initiation and completion.

Bandwidth (KB/s): the total data in KiB transferred in one second

Success Ratio (%): the ratio of successful operations

12

OpenStack* Swift overview

OpenStack* is open source software to build private and public clouds.

OpenStack Object Store (Swift): Create petabytes of reliable storage using standard servers

†Source: docs.openstack.org

13

Storage Node

Proxy Node

OpenStack* Swift Overview

Entities RING physical location (zone/device/partition/…)

Proxy Server

Account

Ring

Container

Ring Object

Ring

Account Server Container Server Object Server

Account

DB

Container

DB

Object

file

metadata

14

Test Configuration

15

Test OpenStack* Swift performance

Software and workloads used in performance tests may have been optimized for performance only on Intel® microprocessors. Performance tests, such as SYSmark* and MobileMark*, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated

purchases, including the performance of that product when combined with other products. For more information go to http://www.intel.com/performance.

16

Swift characterization

- Insufficient processing power throttles overall performance.

Software and workloads used in performance tests may have been optimized for performance only on Intel® microprocessors. Performance tests, such as SYSmark* and MobileMark*, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated

purchases, including the performance of that product when combined with other products. For more information go to http://www.intel.com/performance.

Op/s RT (ms) BW (KB/s)

5,644 91 361,220

Proxy Node 6 Node 7 Node 8 Node 9 Node 10Client

r/s 90.31 91.16 90.64 91.01 91.17

w/s 0.01 0.01 0.01 0.00 0.01

rKB/s 5,633 5,378 5,384 5,379 5,381

wKB/s 0.09 0.09 0.09 0.08 0.08

await 5.16 5.05 5.13 5.16 5.14

rxkB/s 356,225 1,813 1,826 1,844 1,757 1,710

txkB/s 8,356 71,910 73,393 74,265 74,559 72,641

rxkB/s 3,524

txkB/s 357,506

user% 79.70 1.42

system% 19.40 1.74

iowait% 0.00 0.13

cpu%

14.35

4.26

Baseline (ObjectSize=64KB, Concurrency=512, 12 Disks per node)

Performance

data disk

Internal

External

21.85

17

Summary

• New storage Usage model rises for Cloud Computing age, which need new benchmark

• COSBench is a new benchmark developed by Intel to measure Cloud Object Storage service performance

• COSBench is useful to analyze Cloud Object Service system performance, identify bottleneck and conduct optimization

18

Next Step and call for action

• We are WIP developing COSBench to support more Cloud Object Storage service

• Our final goal is to open source COSBench to make it available for industry and community use to make better Object Storage service design

• We will continue to use COSBench to analyze the optimize OpenStack* Swift performance and share back our finding to community

• Any question, feedback, please contact me at:

• Jiangang.duan@intel.com

19

backup

20

Storage Layout

How data is stored in each node ?

object

1025

1026

1027

DG3

G1J

12C

45A1…12C SFT3…12C

partition

hash suffix hash

/dev/swift/a

• accounts

• containers

• objects

• async_pending

/dev/swift/b

• accounts

• containers

• objects

• async_pending

/dev/swift/c

• accounts

• containers

• objects

• async_pending

21

GET/HEAD@Proxy Node

How proxy node cooperate with storage nodes to obtain object data ?

Retrieve container

information

Perform A&A using pre-hooked

facilities

Consult the Ring for

candidate nodes

GET/HEAD candidate server(s)

(R&H)

Return the result

200

314

404

412

507

5xx

0

unmodified

no response

precondition failed

file not found or not synchronized

disk error

server error ?

22

PUT@Proxy Node – Part I

How proxy cooperate with storage nodes to create an object ?

Retrieve container

information

Perform A&A using pre-hooked

facilities

Consult the Ring for

candidate nodes

Create the timestamp

header

Check various

constraints

413

Object Too Large

411

Length Required

404

Container not Found

400

Invalid Path

Invalid Object Name

23

PUT@Proxy Node – Part II

How proxy cooperate with storage nodes to create an object ?

Try making R conns to

storage servers

Assign each conn a container

server

Forward data to storage servers

Collect resps from

storage servers

Return the result

201

408

422 507

5xx

0

time out

no response

data corrupted disk error

server error

Time out: 86400 secs

3 Phased Workflow

24

Disclaimers

INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT.

A "Mission Critical Application" is any application in which failure of the Intel Product could result, directly or indirectly, in personal injury or death. SHOULD YOU PURCHASE OR USE INTEL'S PRODUCTS FOR ANY SUCH MISSION CRITICAL APPLICATION, YOU SHALL INDEMNIFY AND HOLD INTEL AND ITS SUBSIDIARIES, SUBCONTRACTORS AND AFFILIATES, AND THE DIRECTORS, OFFICERS, AND EMPLOYEES OF EACH, HARMLESS AGAINST ALL CLAIMS COSTS, DAMAGES, AND EXPENSES AND REASONABLE ATTORNEYS' FEES ARISING OUT OF, DIRECTLY OR INDIRECTLY, ANY CLAIM OF PRODUCT LIABILITY, PERSONAL INJURY, OR DEATH ARISING IN ANY WAY OUT OF SUCH MISSION CRITICAL APPLICATION, WHETHER OR NOT INTEL OR ITS SUBCONTRACTOR WAS NEGLIGENT IN THE DESIGN, MANUFACTURE, OR WARNING OF THE INTEL PRODUCT OR ANY OF ITS PARTS.

Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined". Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information.

The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.

Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order.

Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained by calling 1-800-548-4725, or go to: http://www.intel.com/design/literature.htm%20

This document contains information on products in the design phase of development.

Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries.

*Other names and brands may be claimed as the property of others.

Copyright © 2012 Intel Corporation. All rights reserved.