ERS downscale2016
-
Upload
victor-de-boer -
Category
Education
-
view
90 -
download
0
Transcript of ERS downscale2016
![Page 1: ERS downscale2016](https://reader036.fdocument.pub/reader036/viewer/2022062401/5873c6201a28abbc788b7a63/html5/thumbnails/1.jpg)
Entity Registry SystemFrom concept to deploymentMihai Gramada, Christophe Gueret(Victor de Boer)
![Page 2: ERS downscale2016](https://reader036.fdocument.pub/reader036/viewer/2022062401/5873c6201a28abbc788b7a63/html5/thumbnails/2.jpg)
Linked data- In computing, linked data (often capitalized as Linked
Data) describes a method of publishing structured data so that it can be interlinked and become more useful through semantic queries (Wikipedia)
- Tim Berners Lee- DBpedia alone 3.4 million concepts described by 1
billion triples
![Page 3: ERS downscale2016](https://reader036.fdocument.pub/reader036/viewer/2022062401/5873c6201a28abbc788b7a63/html5/thumbnails/3.jpg)
![Page 4: ERS downscale2016](https://reader036.fdocument.pub/reader036/viewer/2022062401/5873c6201a28abbc788b7a63/html5/thumbnails/4.jpg)
Example solutions - centralised- rdf4j- neo4j- gremlin (apache tinkerpop)- virtuoso- many others
![Page 5: ERS downscale2016](https://reader036.fdocument.pub/reader036/viewer/2022062401/5873c6201a28abbc788b7a63/html5/thumbnails/5.jpg)
Entity Registry System- decentralised- linked data in
document store(s, p1, o1), (s, p2, o2) -> {“id”: s, “p1”: o1, “p2”: o2, ..}- designed for poor
network connectivity
![Page 6: ERS downscale2016](https://reader036.fdocument.pub/reader036/viewer/2022062401/5873c6201a28abbc788b7a63/html5/thumbnails/6.jpg)
Example: OLPCs and RPi
![Page 7: ERS downscale2016](https://reader036.fdocument.pub/reader036/viewer/2022062401/5873c6201a28abbc788b7a63/html5/thumbnails/7.jpg)
Architecture- contributors, bridges, aggregator
- avahi(zeroconf) for peer discovery
- couchdb for heavy lifting
![Page 8: ERS downscale2016](https://reader036.fdocument.pub/reader036/viewer/2022062401/5873c6201a28abbc788b7a63/html5/thumbnails/8.jpg)
Contributor- read and write content- private, public and cache- private never shared- public can be searched by others and distributed
![Page 9: ERS downscale2016](https://reader036.fdocument.pub/reader036/viewer/2022062401/5873c6201a28abbc788b7a63/html5/thumbnails/9.jpg)
Bridge- can connect isolated parts of the system- similar to a cache (if network goes down, data can still be read
from the bridge)- reduce O(n^2) links down to O(n)
![Page 10: ERS downscale2016](https://reader036.fdocument.pub/reader036/viewer/2022062401/5873c6201a28abbc788b7a63/html5/thumbnails/10.jpg)
Aggregator- optional component- high-performance node/cluster- read-only entry retrieval of data- contributors and bridges push public data to it
![Page 11: ERS downscale2016](https://reader036.fdocument.pub/reader036/viewer/2022062401/5873c6201a28abbc788b7a63/html5/thumbnails/11.jpg)
Deployment example
![Page 12: ERS downscale2016](https://reader036.fdocument.pub/reader036/viewer/2022062401/5873c6201a28abbc788b7a63/html5/thumbnails/12.jpg)
![Page 13: ERS downscale2016](https://reader036.fdocument.pub/reader036/viewer/2022062401/5873c6201a28abbc788b7a63/html5/thumbnails/13.jpg)
Replication strategy
![Page 14: ERS downscale2016](https://reader036.fdocument.pub/reader036/viewer/2022062401/5873c6201a28abbc788b7a63/html5/thumbnails/14.jpg)
Similar solutionsNintendo streetpass and spotpass(3DS only, no aggregator)
Transparent Inter-process Communication- nodes, zones, cluster (only logical grouping)- ip layer, bridges reduce links
Sugar network (OLPC)- in sugar network, clients can communicate to nodes(equivalent of bridges)
or Master node(equivalent of Aggregator)- ers has more flexibility in the data format.
![Page 15: ERS downscale2016](https://reader036.fdocument.pub/reader036/viewer/2022062401/5873c6201a28abbc788b7a63/html5/thumbnails/15.jpg)
Initial status- not working,- very limited testing- a bit frustrating to install - not much investigation (real world tests)
Thus far, the highest number of concurrent users of ERS has been 4 XO laptops, with one bridge, all in the same geographical location
![Page 16: ERS downscale2016](https://reader036.fdocument.pub/reader036/viewer/2022062401/5873c6201a28abbc788b7a63/html5/thumbnails/16.jpg)
Research Questions- Can the Entity Registration system reliably perform in
a real-world scenario ? (i.e. provide the required functionality in a robust manner)?
- Does the ERS scale to a large number of users?
- How does the ERS cope with poor network connectivity?
![Page 17: ERS downscale2016](https://reader036.fdocument.pub/reader036/viewer/2022062401/5873c6201a28abbc788b7a63/html5/thumbnails/17.jpg)
“local” tests- unit tests for storage, daemon and api
communication- basic operations performance tests
- entity creation 5/s- property edits 20/s- value edits 20/s
![Page 18: ERS downscale2016](https://reader036.fdocument.pub/reader036/viewer/2022062401/5873c6201a28abbc788b7a63/html5/thumbnails/18.jpg)
Docker- much larger number of nodes- faster to start- docker hub image -> ers can be ran within
seconds on any x86 device (atm no docker image for arm)
![Page 19: ERS downscale2016](https://reader036.fdocument.pub/reader036/viewer/2022062401/5873c6201a28abbc788b7a63/html5/thumbnails/19.jpg)
“Real world” simulation- simulations so far have a more or less ideal
network- reality is a bit different
![Page 20: ERS downscale2016](https://reader036.fdocument.pub/reader036/viewer/2022062401/5873c6201a28abbc788b7a63/html5/thumbnails/20.jpg)
Simian army- http://techblog.netflix.com/2011/07/netflix-simian-army.html- december 2012 amazon employee launched a
maintenance process against the running production system which deleted the state information needed by load balancers- problems on Christmas Eve at 1:45 p.m. - lasted until 9:41 a.m. on Christmas Day,
an outage of about 20 hours- no Netflix on christmas - unhappy customers
![Page 21: ERS downscale2016](https://reader036.fdocument.pub/reader036/viewer/2022062401/5873c6201a28abbc788b7a63/html5/thumbnails/21.jpg)
Simian army - cont- clunky to integrate- only works on aws
![Page 22: ERS downscale2016](https://reader036.fdocument.pub/reader036/viewer/2022062401/5873c6201a28abbc788b7a63/html5/thumbnails/22.jpg)
Monkeybusiness- imitate simian army- works on vagrant and docker- work in progress :)
![Page 23: ERS downscale2016](https://reader036.fdocument.pub/reader036/viewer/2022062401/5873c6201a28abbc788b7a63/html5/thumbnails/23.jpg)
Latency- insert network issues- latency- loss- reordering- duplication- corruption
![Page 24: ERS downscale2016](https://reader036.fdocument.pub/reader036/viewer/2022062401/5873c6201a28abbc788b7a63/html5/thumbnails/24.jpg)
Real world case - Conference deployment- Simulate conference social network- Think LinkedIn without central server- Profile, skills- Endorsements- Fixed bridge, mobile contributors
![Page 25: ERS downscale2016](https://reader036.fdocument.pub/reader036/viewer/2022062401/5873c6201a28abbc788b7a63/html5/thumbnails/25.jpg)
Conference resultsTested with up to 30 nodes and a bridge.
- Can tolerate disconnects/poor network- It just works™
![Page 26: ERS downscale2016](https://reader036.fdocument.pub/reader036/viewer/2022062401/5873c6201a28abbc788b7a63/html5/thumbnails/26.jpg)
Remote merchants- vendors in different remote villages- no network connectivity- box on a truck that visits every
farmer, provides up to date information on the prices of the other villages, picks up any new information from current one
- Fixed contributors, mobile bridge
![Page 27: ERS downscale2016](https://reader036.fdocument.pub/reader036/viewer/2022062401/5873c6201a28abbc788b7a63/html5/thumbnails/27.jpg)
Behavior- As long as network isn’t too bad(will be
detailed) if the truck stops for a couple of seconds we achieve synchronization
![Page 28: ERS downscale2016](https://reader036.fdocument.pub/reader036/viewer/2022062401/5873c6201a28abbc788b7a63/html5/thumbnails/28.jpg)
Network toleranceFixed 5 seconds wait time, contributor writing as fast as it can- 100 ms each way- 15% loss/corruption each way- duplication is fine- mostly binary progress
![Page 29: ERS downscale2016](https://reader036.fdocument.pub/reader036/viewer/2022062401/5873c6201a28abbc788b7a63/html5/thumbnails/29.jpg)
Research Questions Revisited
Can the Entity Registration system reliably perform in a real-world scenario ? (i.e. provide the required functionality in a robust manner)? Suite deployed, tests indicate yes
- Does the ERS scale to a large number of users? 40 nodes
- How does the ERS cope with poor network connectivity?- If connection for .5 sec or longer: yes
![Page 30: ERS downscale2016](https://reader036.fdocument.pub/reader036/viewer/2022062401/5873c6201a28abbc788b7a63/html5/thumbnails/30.jpg)
Discussion