Senlin Clustering Service Deep DiveEthan Lynn, IBM, IRC: elynnQiming Teng, IBM, IRC: QimingYanyan Hu, IBM, IRC: yanyanhu

Agenda

• Why did we start senlin project

• Senlin overview

• Senlin status and roadmap

Why did we start senlin

Background

• Why Senlin?• CNCP (IBM research project)• Heat AutoScaling Evaluation• Heat AutoScaling Rework• Customer Engagement

Gap Analysis

Heat AutoScaling ReworkTalk on Atlanta Summit

Evaluation Report

Background• Why Not Heat?

• Heat AutoScaling today is based on AWS design, with gaps

• Heat's mission• Defined in project governance• It's NOT about autoscaling, NOT about HA,

...• It's NOT Heat's job to provide new service(s)

To orchestrate composite cloud applications using a declarative template format through an OpenStack-native REST API.

Heat Mission Statement

https://github.com/openstack/governance/blob/master/reference/projects.yaml

• Proposal: https://wiki.openstack.org/wiki/Heat/AutoScaling

• API Design: http://docs.heatautoscale.apiary.io/• Targeting at many use cases not covered by Heat

today

AutoScaling: a new design

Heat as a CompilerYAML

Template

Heat

Nova

Cinder

Neutron

Keystone

• CREATE• GET• UPDATE• DELETE

Blueprints on Reworking Heat AutoScalingBP Priority Descriptionautoscaling-api-resources

high Heat resources invoking AS APIs

as-api-group-resource

high ScalingGroup resource wrapping AS API's group functionality

as-api-policy-resource

high ScalingPolicy resource wrapping AS API's policy functionality

as-api-webhook-res

high Webhook resource wrapping AS API's execution of webhooks

autoscaling-api-client

high A python client for Heat to interact with AS API

autoscaling-api

- A separate service for the implementation of autoscaling w/ Heat

as-engine - A separate engine/service for autoscaling support AS API

as-engine-db - A DB dedicated to autoscaling, using schema created in as-lib-db

as-lib - A separate module to be used by the AS service

as-lib-db - A DB for autoscaling bookkeeping

What Do We Really Need?

Scalable

Load-Balanced

Highly-Available

Manageable

......

of any (OpenStack) objects

- What is missing from OpenStack?A Clustering Service

- Auto-scaling?Just one of the usage scenario of a cluster.

- Auto-Healing (HA)?Just another usage scenario.

- We can address the concerns by making policies orthogonal

Senlin Overview

Senlin Architecture

Senlin Engine

Senlin API

Senlin Database

Senlin Client

REST RPC

Profiles

Policies

Senlin Features• Profiles: A specification for the objects to be managed• Policies: Rules to be checked/enforced before/after actions are performed

10

(others)

Senlin

Nova

Docker

Heat

Ironic BareMetal

VMs

Stacks

Containersplacement

deletion

scaling

health

load-balance

affinity

Policies as Plugins Profiles as Plugins Cluster/Nodes Managed

Senlin Server Architecture

openstacksdkidentity

compute

orchestration

network

...

engineengine lock

scheduler

actions

nodecluster

service

registry

receiverparser

drivers

openstackdummy

(others)

dbapi

rpc clientpolicies

placementdeletionscaling

healthload-balance

affinity

receiverwebhoookMsgQueue

extension points for external monitoring services

extension points facilitating a smarter cluster management

extension points to talk to different endpoints for object CRUD operations

extension points for interfacing with different services or clouds

profilesos.heat.stack

(others)

os.nova.server

senlin-apiWSGI

middleware

apiv1

Senlin Operations (Actions)• Cluster

• CREATE• DELETE• UPDATE• LIST• SHOW• ADD_NODES• DEL_NODES• SCALE_OUT• SCALE_IN• POLICY_ATTACH• POLICY_DETACH• POLICY_UPDATE

12

§Node• CREATE• DELETE• UPDATE• LIST• SHOW

§Policy• CREATE• UPDATE• DELETE• LIST• SHOW

§Profile• CREATE• UPDATE• DELETE• LIST• SHOW

§Action• LIST• SHOW

§Event• LIST• SHOW

§Receiver• CREATE• DELETE• LIST• SHOW

profile/cluster/node operations

13

$ senlin profile-create -s <spec> <name>$ senlin profile-list$ senlin profile-show <profile>$ senlin profile-delete <profile>$ senlin profile-update -s <spec> <profile>$ senlin cluster-create -p <profile> -n <size> <name>$ senlin cluster-delete <cluster>$ senlin cluster-list$ senlin cluster-show <cluster>$ senlin node-create -p <profile> -c <cluster> <name>$ senlin node-delete <node>$ senlin node-show <node>$ senlin node-join -c <cluster> <node>$ senlin node-leave <node>$ senlin cluster-node-add -n <nodes> <cluster>$ senlin cluster-node-del -n <nodes> <cluster>$ senlin cluster-node-list <cluster>

Profile CRUD

Cluster CRUD

Node CRUD

Cluster Membership

policies operations

14

$ senlin policy-create -s <spec> <name>$ senlin policy-list$ senlin policy-show <policy>$ senlin policy-delete <policy>$ senlin policy-update -n <name> ... <policy>

$ senlin cluster-policy-attach <cluster> <policy>$ senlin cluster-policy-detach <cluster> <policy>$ senlin cluster-policy-list ... <cluster>$ senlin cluster-policy-show <cluster> <policy>$ senlin cluster-policy-enable <cluster> <policy>$ senlin cluster-policy-disable <cluster> <policy>$ senlin cluster-policy-update ... <cluster> <policy>

Policy Management

Cluster-Policy Bindings

Other Useful Operations

15

$ senlin cluster-scale-in [-c <count>] <cluster>$ senlin cluster-scale-out [-c <count>] <cluster>$ senlin cluster-resize <options> <cluster>

$ senlin profile-type-list$ senlin profile-type-show <type>$ senlin policy-type-list$ senlin policy-type-show <type>

$ senlin action-list$ senlin action-show <action-id>

$ senlin event-list$ senlin event-show <type>

Cluster Resize

Profile Type

Policy Type

Action

Event

senlin cluster-resize

16

usage: senlin cluster-resize [-c <CAPACITY>] [-a <ADJUSTMENT>] [-p <PERCENTAGE>][-t <MIN_STEP>] [-s] [-n MIN] [-m MAX] <CLUSTER>

Positional arguments:<CLUSTER> Name or ID of cluster to operate on.

Optional arguments:-c <CAPACITY>, --capacity <CAPACITY> The desired capacity the cluster.-a <ADJUSTMENT>, --adjustment <ADJUSTMENT>

A positive integer meaning the number of nodes to add,or a negative integer indicating the number of nodesto remove.

-p <PERCENTAGE>, --percentage <PERCENTAGE>A value that is interpreted as the percentage of sizeadjustment. This value can be positive or negative.

-t <MIN_STEP>, --min-step <MIN_STEP>An integer specifying the number of nodes foradjustment when <PERCENTAGE> is specified.

-s, --strict A boolean specifying whether the resize should beperformed on a best-effort basis when the new capacitymay go beyond size constraints.

-n MIN, --min-size MIN New lower bound of cluster size.-m MAX, --max-size MAX New upper bound of cluster size. A value of -1

indicates no upper limit on cluster size.

Adjustment Types• EXACT CAPACITY• CAPACITY CHANGE• CAPACITY PERCENTAGE

Optional Modifiers• Strict vs. Best-Effort• Minimum Step

Dealing with Constraints• MIN_SIZE• MAX_SIZE

Senlin Roadmap

Senlin Journey

Talk on China OpenStack/Docker community joint meetup

2015-04-25

• IBM internal announce

• Email to OpenStack Heat Core team

2015-02-10

Project accepted to StackForge

2015-03-16

Community Announcement

2015-03-26

Git repo migrate to github.com

2014-12-25

Initial GitRepository inside CRL

2014-12-10

First weekly IRC meeting

2015-06-10

Talk on OPNFV meeting

2015-07-08

Start work with TOSCA std team

2015-07-09

Talk on 2nd China OpenStack Hackathon

2015-08-20

Senlin dashboard project initiated

2015-08-24

Talk on Tokyo Summit -- Scaling for Containers

2015-10-29

Senlin docs went online

2015-12-03

Project accepted into Big Tent

2015-11-11

MitakaMid-Cycle Meetup

2016-01-11

Vancouver Heat design summit

2015-05-20

2015Jan

2015Jul

2016Jan

2015Apr

2015Feb

2015Mar

2015May

2015Jun

2015Oct

2015Aug

2015Sep

2015Nov

2015Dec

2014Dec

2016Feb

2016Mar

2016Apr

Version 1.0 released

2016-04-07

Senlinclient0.4.0 released

2016-03-19

Mitaka Features• API Changes and API Reference

• OpenStackClient Support

• Heat Resources

• Health Management primitive

• User/Developer Documentation

• Senlin Dashboard

Newton Outlook

API micro-version

Health management

Container Cluster

Testing:API/Scenario/Performance...

Additional Clustering Operations

Queue (e.g. Zaqar) type Receiver

Notifications to other services

Batched Operations

User Defined Actions

Access Control

More communications, more collaborations

Relation To Other Projects

21

Senlin

Ceilometer

Heat

Nova Cinder Neutron Swift Keystone

Horizon

Primitive Data Types

Complex Data Types

struct person {int age;char name[0];

}

person team[10]; // Senlin cluster of Heat stacks

// Senlin cluster of nova servers

// Heat stack containing senlin clusters

Mistral MagnumSaharaMonasca Tacker Zaqar

Senlin provides the array data type for cloud programming

We Are Hiring, J

Standard

OpenStack

Customer

IBMTacker

Contributors

Zaqar

Questions?

Thank You!