Meta Programming with Streams and Pipes

18

description

My slides from the Great British Node Conference 2013 at Shoreditch Village Hall in London.

Transcript of Meta Programming with Streams and Pipes

Page 1: Meta Programming with Streams and Pipes
Page 2: Meta Programming with Streams and Pipes

  Essentially, programs that can be created by themselves.

  We’re twisting the term a little for streams

  We will be defining streaming constructs using streaming constructs

  By streams I mean beams which are just streams and pipes for compute as opposed to IO

Page 3: Meta Programming with Streams and Pipes

Any info? Please contact #nodeland police at @cianomaidin on twitter…

Page 4: Meta Programming with Streams and Pipes

  Readable Streams   IO / throughput oriented

  Events – data, end, error, close

  Methods – pause, resume, end, destroy

  Writeable Streams   IO / throughput oriented

  Events - drain, pause, resume, error, close

  Methods – write, end, destroy

Page 5: Meta Programming with Streams and Pipes

  IO Oriented

  Finite, not infinite

  Asynchronous

  Lazy

  Assumes in transit data, not in memory forms

  Great for IO. Not efficient for compute

Page 6: Meta Programming with Streams and Pipes
Page 7: Meta Programming with Streams and Pipes
Page 8: Meta Programming with Streams and Pipes

  Sources   Where data pours in

  Sinks   Where results pop out

  Operators   Filters – Drop or pass on data based on a UDF   Transforms – Mutate data based on a UDF   Branch with Filters   Combine with Sinks

Page 9: Meta Programming with Streams and Pipes

  Compute Oriented

  Infinite, not finite

  Synchronous

  Eager

  Assumes in memory form, not encoded or wire data

  Convenient for compute. Not good for IO

Page 10: Meta Programming with Streams and Pipes
Page 11: Meta Programming with Streams and Pipes
Page 12: Meta Programming with Streams and Pipes
Page 13: Meta Programming with Streams and Pipes
Page 14: Meta Programming with Streams and Pipes
Page 15: Meta Programming with Streams and Pipes

  Eager: a.pipe(operation).pipe(z).pipe(a);

  Implies:

RangeError: Maximum call stack size exceeded

Page 16: Meta Programming with Streams and Pipes
Page 17: Meta Programming with Streams and Pipes
Page 18: Meta Programming with Streams and Pipes

  npm install beam   Compute oriented streams   Branch, Combine, Filter, Transform data   Extend with goto and modules.

  npm install eep   Beam will be integrated into eep soon   Eep provides aggregate windows   Slide, Tumble, Monotonic, Periodic   Stats Library   Ordering now supported

  npm install nodesapiens   Robosapiens V1 for node.js

  Code on github   https://github.com/darach