Post on 08-May-2015
“RabbitMQpython and ruby staring
at the looking glass”- an asynchronous tale -
Paolo Negri @hungryblank
• http://www.slideshare.net/hungryblank
• http://github.com/hungryblank/rabbitmq_rupy_2009
Resources
• http://www.slideshare.net/hungryblank
• http://github.com/hungryblank/rabbitmq_rupy_2009
About meTime
GNU/Linux - Dbs“systems”
Perl
Python
Ruby
PHP
AMQP
• standard for high performance enterprise messaging
• totally open
• platform agnostic
• interoperable
Advanced Message Queuing Protocol
AMQP
“We use asynchronicity heavily in those cases where performance is an issue.
This is generally where we send content from one peer to another. We send off methods as fast as possible without waiting for confirmations”
AMQP Specs [0.8 June 2006]
Why is it interesting?
Summary
“AMQP as a case of asynchronous protocol implementation in python and ruby”
AMQP - Entities
ConsumerBroker
msgProducer
http://www.flickr.com/photos/17258892@N05/2588347668/
msg
AMQP - Entities
ConsumerBroker
msgProducer
http://www.flickr.com/photos/17258892@N05/2588347668/
msg
msgProducer
Consumer msg
msgProducer
AMQPImplementations
• Producer/Consumer: libs available for python and ruby
• Broker no python or ruby implementation
• RabbitMQ is an implementation of AMQP Broker
• It’s opensource
• Written in Erlang
Erlang?
• general-purpose concurrent programming language designed by Ericsson
• first version written by J. Armstrong in 1986
• distributed
• fault tolerant
• soft real time
• high availability
Erlang?Erlang's main strength is support for concurrency
“Process communication is done via a shared-nothing asynchronous message passing:
every process has a “mailbox”, a queue of messages sent by other processes, that are not yet consumed” (Wikipedia)
+ Erlang
It’s messages all the way down
Async Python libs
• “twisted AMQP”
• based on twisted python
• https://launchpad.net/txamqp
txAMQP
Async Python libs
• based on eventlet
• experimental, not ready for production
• http://bitbucket.org/edarc/amqpev
amqpev
Async Python libs
• based on asyncio
• young but mantained by member of RabbitMQ core team
• http://github.com/tonyg/pika
pika
Async Ruby libs
• amqp gem
• based on EventMachine
• mature and widely used
• http://github.com/tmm1/amqp/
amqp
Async?
• Twisted Python
• Asyncio (python stdlib)
• eventlet (python)
• EventMachine (ruby)
Reactor Pattern
• Non blocking IO and lightweight concurrency
• eliminate the complexities of high-performance threaded network programming
synchronous
code
networkoperation
use network operation
result
Free
code
networkoperation
use network operation
result
Callback
Free
Free
Timeasynchronous
process process
Not only reactor
Some of the libs provide more i.e. eventlet provides also coroutines
Python, pick one
• asyncio is part of python stdlib
• easy syntax
• developer member of RabbitMQ team
• fits on slides :-P
pika
Code! Producer (ruby)
Consumer (ruby)
Behind the scenes
msg Queue Consumer
Publisher
Exchange
What and where
RabbitMQ(Erlang)
TCP/IP
Producer(ruby/python)
Consumer(ruby/python)Queue Exchange
Producer (python)
Producer (python)
TCP/IP connection
Producer (python)
TCP/IP connectionchannel 1
Producer (python)
TCP/IP connectionchannel 1
channel 2
Producer (python)
TCP/IP connectionchannel 1
channel 2
channel n
Producer (python)
Producer (python)
Producer (python)
Producer (python)
Producer (python)
Producer (python)
Producer (python)
Producer (python)
Producer (python)
Producer (python)
Producer (python)
Producer (python)
msg Queue Consumer
Publisher
Exchange
Producer (python)
msg Queue Consumer
Publisher
Exchange
key based routing
Producer (python)
msg Queue Consumer
Publisher
Exchange
binding by key
key based routing
Consumer (python)
Things go wrong
If a consumer has a problem we might lose one or more messages
http://www.flickr.com/photos/danzen/2288625136/
ACKing consumer.rb
ACKing consumer.rb
ACKing consumer.rb
ACKing consumer.py
ACKing consumer.py
ACKing consumer.py
Achieved: decoupling
msg Queue Consumer
Publisher
Exchange
Achieved: easy work load distribution
msg A Queue
Consumer
Publisher
Exchange Consumer
Consumer
Achieved: easy work load distribution
Queue
Consumer
Publisher
Exchange Consumer
Consumer
msg A
Achieved: easy work load distribution
Queue
Consumer
Publisher
Exchange Consumer
Consumer
msg A
Achieved: easy work load distribution
Queue
Consumer
Publisher
Exchange Consumer
Consumer
msg A
prefetch?
http://www.flickr.com/photos/balakov/327431271
prefetch?
tcp ipbuffer
http://www.flickr.com/photos/balakov/327431271
prefetch - ruby
prefetch - ruby
prefetch - python
prefetch - python
More routing models
multicastingwith fanout exchange
Fanout producer.rb
Fanout producer.rb
Fanout consumer.rb
Fanout consumer.rb
Fanout consumer.rb
Fanout consumer.rb
Fanout
Queue1
Queue3
Queue2
Cons1
Cons2
Cons3
Producer
Exchangemsg A
Fanout
Queue1
Queue3
Queue2
Cons1
Cons2
Cons3
Producer
Exchangemsg A
Fanout
Queue1
Queue3
Queue2
Cons1
Cons2
Cons3
Producer
Exchange
msg A
msg A
msg A
Fanout producer.py
Fanout producer.py
Fanout producer.py
Fanout consumer.py
Exchanges
• Direct - based on binding routing key
• Fanout - always to all bound queues
• Topic - pattern matching on routing key
Clustering
RabbitMQ - node C
RabbitMQ - node A RabbitMQ - node B
TCP/IP
TCP/IPTCP/IP
Plugins
• introduced with v 1.7.0
• Add functionalities to the broker
• makes a interesting first erlang project
Ecosystem
• 65 RabbitMQ / AMQP projects on github
• + more on bitbucket, and others
Sync?
• easier to add to web apps
• suitable for non performance critical messaging
• useful to build cli tools
Sync?
• celldee / bunny - Ruby (github)
• famoseagle / carrot - Ruby (github)
• py-amqplib - Python (http://barryp.org/software/py-amqplib/)
• ...
Distributed systems
• ezmobious / nanite - Ruby (github)
• ask / carrot - Python (github)
• ask / celery - Python (github)
• auser / poolparty - Erlang (github)
Real time web
•RabbitHub PubSubHubBub
•dansimpson / amqp-js (github)
Thank you!
?