Senlin Clustering Service Deep DiveEthan Lynn, IBM, IRC: elynnQiming Teng, IBM, IRC: QimingYanyan Hu, IBM, IRC: yanyanhu
Agenda
• Why did we start senlin project
• Senlin overview
• Senlin status and roadmap
Why did we start senlin
Background
• Why Senlin?• CNCP (IBM research project)• Heat AutoScaling Evaluation• Heat AutoScaling Rework• Customer Engagement
Gap Analysis
Heat AutoScaling ReworkTalk on Atlanta Summit
Evaluation Report
Background• Why Not Heat?
• Heat AutoScaling today is based on AWS design, with gaps
• Heat's mission• Defined in project governance• It's NOT about autoscaling, NOT about HA,
...• It's NOT Heat's job to provide new service(s)
To orchestrate composite cloud applications using a declarative template format through an OpenStack-native REST API.
Heat Mission Statement
https://github.com/openstack/governance/blob/master/reference/projects.yaml
• Proposal: https://wiki.openstack.org/wiki/Heat/AutoScaling
• API Design: http://docs.heatautoscale.apiary.io/• Targeting at many use cases not covered by Heat
today
AutoScaling: a new design
Heat as a CompilerYAML
Template
Heat
Nova
Cinder
Neutron
Keystone
• CREATE• GET• UPDATE• DELETE
Blueprints on Reworking Heat AutoScalingBP Priority Descriptionautoscaling-api-resources
high Heat resources invoking AS APIs
as-api-group-resource
high ScalingGroup resource wrapping AS API's group functionality
as-api-policy-resource
high ScalingPolicy resource wrapping AS API's policy functionality
as-api-webhook-res
high Webhook resource wrapping AS API's execution of webhooks
autoscaling-api-client
high A python client for Heat to interact with AS API
autoscaling-api
- A separate service for the implementation of autoscaling w/ Heat
as-engine - A separate engine/service for autoscaling support AS API
as-engine-db - A DB dedicated to autoscaling, using schema created in as-lib-db
as-lib - A separate module to be used by the AS service
as-lib-db - A DB for autoscaling bookkeeping
What Do We Really Need?
Scalable
Load-Balanced
Highly-Available
Manageable
......
of any (OpenStack) objects
- What is missing from OpenStack?A Clustering Service
- Auto-scaling?Just one of the usage scenario of a cluster.
- Auto-Healing (HA)?Just another usage scenario.
- We can address the concerns by making policies orthogonal
Senlin Overview
Senlin Architecture
Senlin Engine
Senlin API
Senlin Database
Senlin Client
REST RPC
Profiles
Policies
Senlin Features• Profiles: A specification for the objects to be managed• Policies: Rules to be checked/enforced before/after actions are performed
10
(others)
Senlin
Nova
Docker
Heat
Ironic BareMetal
VMs
Stacks
Containersplacement
deletion
scaling
health
load-balance
affinity
Policies as Plugins Profiles as Plugins Cluster/Nodes Managed
Senlin Server Architecture
openstacksdkidentity
compute
orchestration
network
...
engineengine lock
scheduler
actions
nodecluster
service
registry
receiverparser
drivers
openstackdummy
(others)
dbapi
rpc clientpolicies
placementdeletionscaling
healthload-balance
affinity
receiverwebhoookMsgQueue
extension points for external monitoring services
extension points facilitating a smarter cluster management
extension points to talk to different endpoints for object CRUD operations
extension points for interfacing with different services or clouds
profilesos.heat.stack
(others)
os.nova.server
senlin-apiWSGI
middleware
apiv1
Senlin Operations (Actions)• Cluster
• CREATE• DELETE• UPDATE• LIST• SHOW• ADD_NODES• DEL_NODES• SCALE_OUT• SCALE_IN• POLICY_ATTACH• POLICY_DETACH• POLICY_UPDATE
12
§Node• CREATE• DELETE• UPDATE• LIST• SHOW
§Policy• CREATE• UPDATE• DELETE• LIST• SHOW
§Profile• CREATE• UPDATE• DELETE• LIST• SHOW
§Action• LIST• SHOW
§Event• LIST• SHOW
§Receiver• CREATE• DELETE• LIST• SHOW
profile/cluster/node operations
13
$ senlin profile-create -s <spec> <name>$ senlin profile-list$ senlin profile-show <profile>$ senlin profile-delete <profile>$ senlin profile-update -s <spec> <profile>$ senlin cluster-create -p <profile> -n <size> <name>$ senlin cluster-delete <cluster>$ senlin cluster-list$ senlin cluster-show <cluster>$ senlin node-create -p <profile> -c <cluster> <name>$ senlin node-delete <node>$ senlin node-show <node>$ senlin node-join -c <cluster> <node>$ senlin node-leave <node>$ senlin cluster-node-add -n <nodes> <cluster>$ senlin cluster-node-del -n <nodes> <cluster>$ senlin cluster-node-list <cluster>
Profile CRUD
Cluster CRUD
Node CRUD
Cluster Membership
policies operations
14
$ senlin policy-create -s <spec> <name>$ senlin policy-list$ senlin policy-show <policy>$ senlin policy-delete <policy>$ senlin policy-update -n <name> ... <policy>
$ senlin cluster-policy-attach <cluster> <policy>$ senlin cluster-policy-detach <cluster> <policy>$ senlin cluster-policy-list ... <cluster>$ senlin cluster-policy-show <cluster> <policy>$ senlin cluster-policy-enable <cluster> <policy>$ senlin cluster-policy-disable <cluster> <policy>$ senlin cluster-policy-update ... <cluster> <policy>
Policy Management
Cluster-Policy Bindings
Other Useful Operations
15
$ senlin cluster-scale-in [-c <count>] <cluster>$ senlin cluster-scale-out [-c <count>] <cluster>$ senlin cluster-resize <options> <cluster>
$ senlin profile-type-list$ senlin profile-type-show <type>$ senlin policy-type-list$ senlin policy-type-show <type>
$ senlin action-list$ senlin action-show <action-id>
$ senlin event-list$ senlin event-show <type>
Cluster Resize
Profile Type
Policy Type
Action
Event
senlin cluster-resize
16
usage: senlin cluster-resize [-c <CAPACITY>] [-a <ADJUSTMENT>] [-p <PERCENTAGE>][-t <MIN_STEP>] [-s] [-n MIN] [-m MAX] <CLUSTER>
Positional arguments:<CLUSTER> Name or ID of cluster to operate on.
Optional arguments:-c <CAPACITY>, --capacity <CAPACITY> The desired capacity the cluster.-a <ADJUSTMENT>, --adjustment <ADJUSTMENT>
A positive integer meaning the number of nodes to add,or a negative integer indicating the number of nodesto remove.
-p <PERCENTAGE>, --percentage <PERCENTAGE>A value that is interpreted as the percentage of sizeadjustment. This value can be positive or negative.
-t <MIN_STEP>, --min-step <MIN_STEP>An integer specifying the number of nodes foradjustment when <PERCENTAGE> is specified.
-s, --strict A boolean specifying whether the resize should beperformed on a best-effort basis when the new capacitymay go beyond size constraints.
-n MIN, --min-size MIN New lower bound of cluster size.-m MAX, --max-size MAX New upper bound of cluster size. A value of -1
indicates no upper limit on cluster size.
Adjustment Types• EXACT CAPACITY• CAPACITY CHANGE• CAPACITY PERCENTAGE
Optional Modifiers• Strict vs. Best-Effort• Minimum Step
Dealing with Constraints• MIN_SIZE• MAX_SIZE
Senlin Roadmap
Senlin Journey
Talk on China OpenStack/Docker community joint meetup
2015-04-25
• IBM internal announce
• Email to OpenStack Heat Core team
2015-02-10
Project accepted to StackForge
2015-03-16
Community Announcement
2015-03-26
Git repo migrate to github.com
2014-12-25
Initial GitRepository inside CRL
2014-12-10
First weekly IRC meeting
2015-06-10
Talk on OPNFV meeting
2015-07-08
Start work with TOSCA std team
2015-07-09
Talk on 2nd China OpenStack Hackathon
2015-08-20
Senlin dashboard project initiated
2015-08-24
Talk on Tokyo Summit -- Scaling for Containers
2015-10-29
Senlin docs went online
2015-12-03
Project accepted into Big Tent
2015-11-11
MitakaMid-Cycle Meetup
2016-01-11
Vancouver Heat design summit
2015-05-20
2015Jan
2015Jul
2016Jan
2015Apr
2015Feb
2015Mar
2015May
2015Jun
2015Oct
2015Aug
2015Sep
2015Nov
2015Dec
2014Dec
2016Feb
2016Mar
2016Apr
Version 1.0 released
2016-04-07
Senlinclient0.4.0 released
2016-03-19
Mitaka Features• API Changes and API Reference
• OpenStackClient Support
• Heat Resources
• Health Management primitive
• User/Developer Documentation
• Senlin Dashboard
Newton Outlook
API micro-version
Health management
Container Cluster
Testing:API/Scenario/Performance...
Additional Clustering Operations
Queue (e.g. Zaqar) type Receiver
Notifications to other services
Batched Operations
User Defined Actions
Access Control
More communications, more collaborations
Relation To Other Projects
21
Senlin
Ceilometer
Heat
Nova Cinder Neutron Swift Keystone
Horizon
Primitive Data Types
Complex Data Types
struct person {int age;char name[0];
}
person team[10]; // Senlin cluster of Heat stacks
// Senlin cluster of nova servers
// Heat stack containing senlin clusters
Mistral MagnumSaharaMonasca Tacker Zaqar
Senlin provides the array data type for cloud programming
We Are Hiring, J
Standard
OpenStack
Customer
IBMTacker
Contributors
Zaqar
Questions?
Thank You!