Event Driven with LibUV and ZeroMQ

Post on 16-Jul-2015

455 views 14 download

Transcript of Event Driven with LibUV and ZeroMQ

Event Driven with LibUV and ZeroMQ

Phase I Summary

Luke.jf.luo@gmail.com

Outline

● Algo Trading Business Flow Review● Order Module Design: Current Flaw● Distributed System: Theory Review● Event Driven System: Review● Event Driven Design In Algo Trading● Project Plan

Algo Trading Flow

I1 I2 I3 …... In

任务(下达 /取消 /停止 )

Instrument (Buy/Sell)

下单

撤单

下单

撤单

步长

步长

.

.

.

...

...

….........

...

...

….........

...

...

….........

End

…............

Current Design Flaw

● Bad Assumptions– Serial Execution Among Instructments

– Batch Job ( 1 Sec Interval)

– I/O Cost/ Latency: Neglectable

– Computation Cost/Latency: Neglectable

– No Uncertainty

Reality

● Parallel execution of Instruments● Market change in Ms, 1 Sec (or More) fixed reponse interval

is 1000X● I/O is SLOW comaring to CPU speed (10^9 order)● Communication Channel will Break● Only God have ALL TIMELY information for decision making.

We people live in UNCERTAINTY● Tradeoff: Best Effort Decision versus Perfect Decision

Distributed System: Our Reality

● Parallel: Everything is Moving● Error Prone (Fault): No 99.9999999....%

Gurantee● Uncertainty: Who/What/Where/When/Why● Concensus/Consistency with HIGH COST● Speed limit: Light Travels at (300,000km/Sec)

CAP: No Perfect World

Case:Two Generals' Problem

Case:Two Generals' Problem

● 1975 . E. A. Akkoyunlu, K. Ekanadham, and R. V. Huber "Some Constraints and Trade-offs in the Design of Network Communications"

● “A pragmatic approach to dealing with the Two Generals' Problem is to use schemes that accept the uncertainty of the communications channel and not attempt to eliminate it, but rather mitigate it to an acceptable degree.”

TCP: Connection Establishment

TCP: Sliding Window

Event Driven: History

● 1960s: Mainframe. Batch Job Oriented● 1970s: Unix. Interactive,Timesharing Terminal● GUI programming:

– Xerox : mouse

– Apple: Apple II

– Microsoft: Windows

– MIT: X windows

– …...

Event Driven: Now

● Libuv: Chrome browser, node.js...● Libev/libevent● Boost ASIO● ACE/ICE● Qt event loop● Glib/Gevent

Event Driven: Now

● Nginx,apache,Twisted,Comet● Node.js● Python Asyncio● Go channel● Scala Actor● C++ 11 Promise/Future,Coroutine,Closure● …........

I/O Modelling

● blocking I/O● nonblocking I/O● I/O multiplexing (select and poll)● signal driven I/O (SIGIO)● asynchronous I/O (the POSIX aio_functions)

Quoted from “UNIX Network Programming Volume 1”

Async/Event Driven Design: Reactor

Proactor

So we are event-driven ready. But...

● Components are mostly event driven● Interface Among components are BLOCKING

INTERFACE! (block/wait/timeout....)● One blocking will ruin the whole infrastructure!● “Public Land Tragic”

class StringQue : public MsgQueBase

StringQueue is everywhere

[luke@rmbp service_multichannel]$ find . -name StringQue.h./orderservice/OMSInfterfaceDemo/StringQue.h./optionhedgeservice/optionhedgeservice/StringQue.h./queryservice/queryservice/StringQue.h./queryservice/queryservice_auto_qt_profitchart/StringQue.h./queryservice/queryservice_auto_qt/StringQue.h./strategyservice_new/hedgefutureservice/StringQue.h./strategyservice_new/algoorderservice/StringQue.h./strategyservice_new/algoorderserviceFuture/StringQue.h./strategyservice/hedgefutureservice/StringQue.h./strategyservice/optionhedgeservice/optionhedgeservice/StringQue.h./strategyservice/algoorderservice/StringQue.h./strategyservice/algoorderserviceFuture/StringQue.h./quoteservice/quoteservicelevel2/StringQue.h./quoteservice/quoteservicelevel2datafeed/StringQue.h./quoteservice/quoteservicelevel2_new/StringQue.h./quoteservice/quoteservicelevel2datafeed_clb/StringQue.h

How we use StringQueue?

Vision

● A fault tolerant, responsive trading platorm– Respond to every marktet tick/heartbeat

● Maintain “fresh” market snapshot● React as every tick comes

– Async, Parallel order fullfillment ● Handle > 1000 orders one server instance without

pressure● Milisec level processing delay per order action

Vision: Unified Event Loop

Libuvevent Loop

ZeroMqsocket

Normal socket

timer

Other Async events

eventfd

fd

timer_t handle

async_t handle

Parallel Tasks req_t

Thread PoolAsync Tasks

Order Fullfill

Market Snapshotupdate

Other Event Loop

Other businessmodules

Async Programming

● Callback● Task(threading)● Future/Promise● Async I/O● Coroutine●

Trading Version 2

I1 I2 I3 …... In

下单

撤单

下单

撤单

timeout

timeout.

.

.

...

...

...

...

….........

...

...

….........

End

…............

tick

….........

EndEndEnd

tick

tick

tick

Project Plan

● Goal: A testable new framwork – Algotrading as typical business test case

– High throughput, low latency

– Fault Tolerant with sensible tradeoff strategy

Side by Side comparison with current order system

Task:

● Project Managment (Wei Song)● Event Loop, ZeroMQ (Tie Gang/Luke)● Market snapshot update (Wei Song/Ze Yu)● Order business module rewrite(Ze Yu/Wei Song)● Testing environment (Wei Song/Tie Gang)

– Simulation data/replay

– OMS simulator

● Testing– All

– + Calvin + other business staff

Schedule?

● What can we accomplish before Spring Festival?