Openstack taskflow果凍

簡介

● 任職於迎廣科技○ python○ openstack

● c++, java, scala● http://about.me/ya790206● http://blog.blackwhite.

tw/● https://github.

com/ya790206/call_seq

聯絡資料

● 個人信箱： http://www.blackwhite.tw/#/contact

● 公司信箱：jam.kao at in-win.com.tw

關注 openstack 以下技術

● cinder● glance● ceilometer● horizon● log● monitor and alert

What is it?

● TaskFlow is a Python library for OpenStack (and other projects) that helps make task execution easy, consistent, scalable and reliable.

What it help us to do?

1. Retry2. Revert3. Each independent task (or independent

subgraph) in the graph could be ran in parallel when its dependencies have been satisfied.

4. Concurrent / Parallel

前情提要

A 要找 B 和 C 去餐廳 X 進行祕密會談。只要有人無法到場，則取消該次會談

1. 聯絡 A 和 B2. 聯絡 X(餐廳)

def call(name, phone, msg):

if phone in ['123']:

print 'name: %s, phone: %s, msg: %s, status: failed' % (name, phone, msg)

return False

print 'name: %s, phone: %s, msg: %s, status: success' % (name, phone, msg)

return True

call('B', '123', 'invite B')call('C', '124', 'invite C')call('X', '124', 'make a reservation in X')

def try_call(name, phone_list, msg): ret = False for phone in phone_list: ret = call(name, phone, msg) if ret: break return ret

try_call('B', ['123'], 'invite B')try_call('C', ['124', '126'], 'invite C')try_call('X', ['224'], 'make a reservation in X')

try:

revert_list = []

successed = try_call('B', ['123'], 'invite B')

if not successed:

raise Exception()

revert_list.append(lambda: try_call('B', ['123'], 'cancel B'))

successed = try_call('C', ['124', '126'], 'invite C')

if not successed:

raise Exception()

revert_list.append(lambda: try_call('B', ['123'], 'cancel B'))

successed = try_call('X', ['224'], 'make a reservation in X')

if not successed:

raise Exception()

revert_list.append(lambda: try_call('B', ['123'], 'cancel B'))

except:

for action in revert_list:

action()

Q: Do we have a better way?A: Yes, we have openstack taskflow

Tasks

A task (derived from an atom) is the smallest possible unit of work that can have a execute & rollback sequence associated with it.

Flows

A flow is a structure that links one or more tasks together in an ordered sequence pattern. When a flow rolls back, it executes the rollback code for each of it's child tasks using whatever reverting mechanism the task has defined as applicable to reverting the logic it applied.

Patterns

1. Linear2. Unordered3. Graph

Engine

1. Engines are what really runs your atoms(tasks, flows).

2. Engines decide which atom to run and when.

3. Engine type:a. serial(default)b. parallelc. worker-based

store(dict)

get dependence

provides

Task

get dependence

provides

Task

provides: Any outputs this task produces.

empty store(dict)

After executing Provider task Dmsg

-> XXX

Dphone_list -> [XXX, YYY]

Dname -> ZZZ

Task dependence

rebind: An immutable input resource mapping dictionary that can be used to alter the inputs given to this task.provides: Any outputs this task produces.

Dphone_list -> [XXX, YYY]Dnam

e -> ZZZ

Dmsg -> XXX

get dependence from store for CallTask(Dname, Dphone, Dmsg)

Run CallTask

Provide`Ddone`

Dphone_list -> [XXX, YYY]

Dname -> ZZZ

Dmsg -> XXX

Ddone -> True

Provide -> CallStackQ: Why it is Dname istead of name A: because of rebind

Q: where is the Dphone?

flowprovide_flow call_flow

revert rules:

● 每次 task 執行失敗，則會 revert 。● 只要 flow 中有個 task 執行失敗且無法

retry，則已經執行過的 task 則會 revert。

Demo

https://gist.github.com/ya790206/14229922355077713ad9

http://goo.gl/gyd3K6

Another example

x2 = y3+y4x1 = y1+y2x5 = x1+x3x3 = x1+x2 x4 = x2+y5x6 = x5+x4x7 = x6+x6

x7

x6

x5 x4

x4

x2 y5

y3 y4

x1

y1 y2

x3

x1 x2

Graph pattern

A tasks dependents are guaranteed to be satisfied before the task will run. Each independent task (or independent subgraph) in the graph could be ran in parallel when its dependencies have been satisfied.

Code

https://github.com/openstack/taskflow/blob/master/taskflow/examples/graph_flow.py

Why Openstack need taskflow?

OpenStack code has grown organically, and does not have a standard and consistent way to perform sequences of code in a way that can be safely resumed or rolled back if the calling process is unexpectedly terminated while the code is busy doing something

Results

With widespread use of TaskFlow, OpenStack can become very predictable and reliable, even in situations where it's not deployed in high availability configurations.

Summary

TaskFlow is a Python library for OpenStack (and other projects) that helps make task execution easy, consistent, scalable and reliable.

迎廣雲端研發中心介紹

● 提供 openstack 相關服務○ 教育訓練○ 客製化○ 佈署

徵才

● web 前端工程師(angularjs, html5, css)● web 後端工程師(python)● 熟悉 linux 網路與 SDN● 熟悉 linux 分散式儲存與 SDS

Reference

https://wiki.openstack.org/wiki/TaskFlow

建議學習順序

1. simple_linear.py2. calculate_linear.py3. reverting_linear.py4. retry_flow.py5. graph_flow.py6. simple_linear_listening.py7. calculate_in_parallel.py8. pseudo_scoping.py