We don’t need consensus: all agreed?

tom@weave.works@tom_wilkie

μServices

μServices

“In computing, microservices is a software architecture style in which complex applications are composed of small,

independent processes communicating with each other using language-agnostic APIs.”

https://en.wikipedia.org/wiki/Microservices

ContainerizationMonoliths

Developer Freedom

Services & ReplicasDistributed Systems

Infrastructure μServices

Routing Service Discovery IP Address Management Load Balancing

Infrastructure Microservices

Fallacies of distributed computing• The network is reliable. • Latency is zero. • Bandwidth is infinite. • The network is secure. • Topology doesn't

change.

• There is one administrator.

• Transport cost is zero. • The network is

homogeneous.

Bill Joy, Peter Deutsch et al., 20 years ago

Routing Service Disco IPAM Load Bal.

Flannel, libnetwork Consul, SkyDNS

Flannel, libnetwork bakersteet, SmartStack,

kubeproxy

Infrastructure Microservices

Our industry’s addiction to this architecture is leading us to build fragile systems.

This is a dangerous precedent

We should be more concerned about ease of use and reliability.

Ease of use

FragilityPrivate DC Public Cloud

Where do I put my KV store?

Do we even need strong consistency?

Routing Service Disco IPAM Load Bal.

BGP DNS

ICANN …

Infrastructure Microservices

An alternative approach

Gossip• Consider each node

broadcasting updates:• O(n2) traffic, O(1)

propagation delay• Alternatively, each node picks

random set of neighbours and tells them:• O(log n) traffic, O(log n)

propagation delay on average

Convergent Replicated Data Type• maintains consistency

without consensus • nodes merge updates

from other nodes • merge function must be

commutative, associative and idempotent

https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type

Concrete Application of CRDTs

Routing: Weave Net

• Data structure is a mapping from container MAC address to host ID

• We also gossip the topology, who is connected to whom

IP Address Management• Data structure is a ring,

mapping IP address segments to hosts

• Each host only ever manipulates its own entries.

• When one host wants to grant some free space to another host, it splits one of its own ranges and gossips an update.

Service Discovery: WeaveDNS• Data structure is a mapping

from DNS name to IP address • Entries also contain the host

and container ID, so multiple hosts can have an entry for the same name

• This allows for round-robin DNS load-spreading

Results

A demo

@weaveworks github.com/weaveworks

Questions?http://weave.works/product/scope

tom@weave.works @tom_wilkie

Credits

Based on talk by Bryan Boreham, who cowrote the IPAM implementation with me.

https://www.youtube.com/watch?v=117gWVShcGU

Details• How do we keep from accumulating an ever-

increasing set of tombstoned entries? • Tombstoned entries time out after a while

• CRDTs are not something you can pull off-the-shelf • Data structure must be carefully designed

• How to bootstrap the IP allocator system? • We use Paxos