SODA: Service-On-Demand Architecture for

Application Service Hosting Utility Platforms

Dongyan Xu, Xuxian Jiang

Lab FRIENDS (For Research In Emerging Network and Distributed Services)

Department of Computer Sciences Center for Education and Research in Information Assurance

and Security (CERIAS) Purdue University

Outline

Motivations and goals Related work Research components of SODA Summary and on-going work

Motivations

Vision of utility computing Computation utility Storage utility

Application service hosting Conference management e-Campaign Digital government Serving the underserved communities IT function shadowing for disaster recovery

Virtual enterprise, collaboratory, and community

Our Goal

To build a value-added application service hosting platform based on shared infrastructure, achieving: On-demand creation and provisioning Virtualization Isolation Protection Accountability Privacy

Utility computing architectures VERITAS, HP UDC, IBM Oceano

Grid platforms Computation: Globus, Condor, Legion, NetSolve,

Harness, Cactus Storage and data: SRB, NeST, Data Grid,

OceanStore Shared infrastructure

PlanetLab, Emulab Active services

Active Service Grid, Berkeley Active Service Framework, CANS (NYU), Darwin, WebOS

Related Work

Resource isolation GARA, QLinux (UMass), Virtual service (UMich),

Resource Container, Cluster Reserves (Rice) Virtualization technologies

Virtual super computer (aggregation): NOW, HPVM

Virtual OS, isolation kernel (slicing): VMWare, Xen (Cambridge), Denali (UW), UML, UMLinux, Virtual Private Server (Ensim)

Grid computing on VM: Virtuoso (Northwestern), Entropia

Virtual cluster: Cluster-on-Demand (Duke)

Related Work

SODA Service-On-Demand Architecture for

application service hosting utility platforms

Research components of SODA General architecture Protection, intrusion detection, logging Confined and VM-based overlay Market-driven planning and management

Outline

Research components of SODA: General architecture Security and protection Confined VM-based overlay ‘Property’ planning and management

Detailed Information Xuxian Jiang, Dongyan Xu, "SODA: a Service-On-Demand

Architecture for Application Service Hosting Utility Platforms", Proceedings of The 12th IEEE International Symposium on High Performance Distributed Computing (HPDC-12), Seattle, WA, June 2003.

Overview of SODA

SODA Host (physical)

AS

AS’

Virtual service node

Virtualization: Key Technique

Two-level OS structure Host OS Guest OS

Strong isolation Administration isolation Installation isolation Fault / attack Isolation Recovery, migration, and

forensics Virtual service node

Application service (AS) Guest OS Internetworking enabled

One SODA host

Host OS

…Guest OS Guest OS

AS1 ASn

SODA Master SODA Agent

Host OS

Guest OS

Service SSODA

Daemon

Host OS

Guest OS

Service S

SODADaemon

Host OS

Guest OS

Service S’SODA

DaemonGuest OS

Service S’

Service Switch for S Service Switch for S’

Service Requests From ClientsService Requests From Clients

Service Creation Requests From ASP

Virtual servicenode

On the Same SODA Host

WWW service Honeypot

Host OS and Guest OS Guest OS: based on User-Mode Linux

(UML), an open-source virtual OS (different from UMLinux and VServer ) By Jeff Dike, http://user-mode-linux.sourceforge.net Running in user space of host OS Separate kernel address space Physical memory usage limit

Host OS: Linux (linux-2.4.19, enhanced) CPU fair share scheduler (for CPU isolation

between virtual service nodes)

Experiment: CPU Isolation

Original Linux Scheduler Enhanced Linux Scheduler

VM1: CPU-intensive VM2: IO-intensive VM3: Web

On-Demand Service Priming

Performed by SODA Daemon Customization of guest OS (“cook to

order” ) Active service image downloading Automatic bootstrapping of virtual

service node

Service Bootstrapping Time

Linux Configuration

Image sizeTime

(seattle)Time

(tacoma)

Rootfs_tomrtbt_1.7.205

15 MB 2.0 sec. 3.0 sec.

Rootfs_base_1.0 29.3 MB 3.0 sec. 4.0 sec.

Root_fs_lfs_4.0 400 MB 4.0 sec. 16.0 sec.

Root_fs.rh-7.2-server.pristine.2

0021012253 MB 22.0 sec. 42.0 sec.

Slow-Down (w/o optimization)

1,368 37,004gettimeofday

1,200 27,044munmap

1,208 27,864mmap

1,084 26,904dup2

1,064 26,648geteuid

1,208 27,276getpid

Linux UMLSystem call

System call level(clock cycles)

Application level

Outline

Research components of SODA: General architecture Security and protection Confined VM-based overlay ‘Property’ planning and management

Detailed Information Xuxian Jiang, Dongyan Xu, Rudolf Eigenmann, "Protection

Mechanisms for Application Service Hosting Platforms", Proceedings of IEEE/ACM Int'l Symposium on Cluster Computing and the Grid (CCGrid 2004), Chicago, IL, April 2004.

Xuxian Jiang, Dongyan Xu, "Collapsar: A VM-Based Architecture for Network Attack Detention Center", to appear in Proceedings of the 13th USENIX Security Symposium (Security '04), San

Diego, CA, August 2004.

Security and Protection

Virtual switching and firewalling

IDS in guest OS kernel

Untamperable logging (‘blackbox’-ing) Host OS

…Guest OS Guest OS

AS1 ASn

Virtual Switching and Firewalling

Virtualmachine(with IPaddr.)

SODA host(Invisible on

Internet)

Guest OS Guest OSGuest OS

Host OS

Firewall

Kernort: IDS in Guest OS Kernel

Problems with traditional IDS Encrypted traffic (e.g. ssh) makes NIDS less effective App-level IDS process will be “killed”, once a machine is

compromised Log may be tampered with Fail-open

Related projects Backtracker (Michigan) VMM-based retrospection (Stanford) Forensix (OHSU) ESP (Purdue CERIAS) Open-source projects: Snort, Saint Jude

Kernort

VM-based IDS Deployed in each VM

Inside guest OS kernel: a unique vista point Customizable without affecting host OS Clearer view Untamperable logging (saved to SODA host) Renewable signature (read from SODA host) Fail-close instead of fail-open

Kernort: IDS in Guest OS Kernel

Guest OS Guest OS

Kernort

Components Kernort sensor

Event-driven (system call and packet reception) Renewable signature set Matching against a small signature set (“Top 20

most wanted”) Kernort blackbox

Untamperable logging Privacy preservation of ASes

Analyzer Exhaustive signature matching Detection of complex attack patterns Session replay

Kernort

Virtual machine

Host OS

Kernort (shaded areas: logs)

Real-Time Alert

Session Re-play

Impact on Performance

Impact on Performance

Outline

Research components of SODA: General architecture Security and protection Confined VM-based overlay ‘Property’ planning and management

Detailed Information Xuxian Jiang, Dongyan Xu, "vBET: a VM-Based Emulation Testbed",

Proceedings of ACM Workshop on Models, Methods and Tools for Reproducible Network Research (MoMeTools, in conjunction with ACM SIGCOMM 2003), Karlsruhe, Germany, August 2003.

Xuxian Jiang, Dongyan Xu, "VIOLIN: Virtual Internetworking on OverLay INfrastructure", Department of Computer Sciences Technical Report CSD TR 03-027, Purdue University, July 2003.

Xuxian Jiang, Dongyan Xu, “A Middleware Architecture for Confined Virtual Machine Overlays", in preparation, March 2004.

Traditional Overlay Network Problems with traditional overlays:

Open for attacks Attacks from the outside (i.e. Internet) against

overlay nodes Attacks from an overlay node against the outside

Difficult to manage An overlay across multiple administration domains A host participate in multiple overlays Difficult to enforce overlay topology and traffic

volume

VPN does not solve the problems

Traditional Overlay Network

FirewallFirewall

Firewall

VM-based Overlay The case for VM-based overlay

Multiple overlays on shared infrastructure On-demand creation Confinement and isolation

VM introduces new network administration complexity “What is this new machine that has suddenly

appeared in my domain?” “Where is the machine that was in my domain

yesterday?” “How much network connectivity should a VM have?” “How many IP addresses for VMs?”

Confined VM-based Overlay In addition to VM, we need VN for VMs

VN: a highly overloaded term (VPN, X-bone…)

What is new: Confined and VM-based overlays

Applications Multi-institutional collaborations Philanthropic (volunteer) computing systems Network emulations

Confined VM-based Overlay

Firewall Firewall

Firewall

VMVM

VM

≤1Mbps

≤2Mbps ≤2MbpsVirtual

infrastructure

Key Properties

Confined overlay topology and traffic No attack possible from inside the

overlay to the outside world Virtual IP address space No need for application modification

and re-compilation

A More Generic Picture

VIOLIN:Virtual

Internetworkingon OverLay

INfrastructure

vBET: an Example of Confined Overlays on Demand

An education tool for network and distributed system emulation

Fidelity-preserving setup Maneuverable network entities Real-world network software

Strict confinement (network security experiment)

Flexible configuration Not constrained by device/port availability No manual cable re-wiring or hardware setup

Simultaneous experiments Cost-effective

vBETvBET Features Can be deployed in n ≥ 1 vBET servers Efficient startup and tear-down of

emulated entities Strong network virtualization

IP address space Virtual routers, switches, firewalls, end-hosts,

links Communications confined by virtual topology

Dynamic addition, deletion, migration, configuration of network entities

vBET GUI

Sample Emulation: OSPF Routing

Emulation of OSPF RoutingDemo video clip at:http://www.cs.purdue.edu/~jiangx/vBET/videos/vbet_ospf.avi

Sample Emulation: Distributed Firewalls

Screenshot

Sample Emulation: Chord P2P Network

Screenshot

Outline

Research components of SODA: General architecture Security and protection Confined VM-based overlay ‘Property’ planning and management

Property Planning and Management

Tenant selection: Among a set of potential tenants (ASes),

which ones to host? (for maximum revenue, resource utilization, security…)

SODA provider selection: Among a set of SODA providers, which one

should be chosen to host an AS?

Examples of bad planning: Many PDA transcoding ASes in an area with

a small PDA user population AS not requiring client registration and log-in

(potential DDoS attacks) Majority of ASes exhibiting similar demand

characteristics such as:

Property Planning and Management

Load

Time

Property Planning and Management AS profiling

Resource requirement Security/authentication Demand characteristics

Market analysis Competing ASes, market

size/growth/expected share ASes correlation (“80% of clients requesting

AS X also request AS Y” ) Trading/pricing of SODA machine slices

Property Planning and Management Forming alliance of SODA providers

Property Planning and Management

Forming alliance of SODA providers

Summary

Virtualization: a key enabling technology in realizing utility computing vision

Hosting utility is more complex than computation utility (host – tenants – clients)

SODA achieves: On-demand service creation Service virtualization, isolation and

confinement Protection, accountability, privacy Overlay isolation and confinement

Ongoing Work

VM/service migration, shadowing, recovery

Service profiling, accounting, auditing (resources, security)

Market-driven planning, provisioning, and management (SODA ecology)

Deployment and evaluation (Purdue Bindley Bioscience Center)

Thank you.

For more information:

{dxu, jiangx}@cs.purdue.eduhttp://www.cs.purdue.edu/~dxu

AOL keywords “Purdue SODA Friends”