Post on 21-Mar-2017
Big Data zum AnfassenKai Voigt, Cloudera Inc.
AIShttps://en.wikipedia.org/wiki/Automatic_Identification_System
Sailing Yacht A ("White Pearl")Länge: 140+ m, Höhe: 100+ m, Kosten: 400+ Mio EUR
Dynamische Daten
{"msgid":1,"rot":0,"sub_message":806,"aisclient_id":2051,"slot_timeout":6,"spare":2,"userid":211704630,"pos_acc":1,"sentences":["!AIVDM,1,1,,A,139qK=R009PffhLO9MrU9@4v8H<V,0*38"],"nav_status":2,"sync_state":0,"time_received":1443099141551,"repeat":0,"pos":[10.20845,54.43495],"utc_sec":31,"sog":9,"time_captured":1443099141000,"cog":1317,"raim":0,"true_heading":2,"smi":0}
Statische Daten{"imo":1012141,"dim_port":15,"aisclient_id":2216,"pos_type":1,"userid":211704630,"draught":76,"sentences":["!AIVDM,2,1,2,A,539qK=T0?LJl@l4H001LPU@F10D58h000000000T:0w?:6J`hC4iA@PDP000,0*13","!AIVDM,2,2,2,A,00000000000,2*26"],"dim_starboard":10,"callsign":"DMAF@@@","repeat":0,"time_received":1443098166711,"version":1,"name":"WHITE PEARL@@@@@@@@@","dte":0,"eta_minute":48,"dim_bow":80,"msgid":5,"eta_hour":8,"ship_type":36,"spare":0,"eta_month":9,"dim_stern":63,"dest":"SEEBAR@@@@@@@@@@@@@@","eta_day":21,"time_captured":1443098166000}
http://www.vesseltracker.com/
Satellite AIS
Big Data?
• Nord- und Ostsee, 500 Records/Sekunde
• 250 KByte/Sekunde
• Weltweit: 10.000 Records/Sekunde
• 5 MByte/Sekunde = ~400 GByte/Tag
λApache Hadoop
AIS Data Stream
Flume/Kafka
Spark StreamingHBase/HDFS
Impala/Spark Near Real Time
Kollisionserkennung• 2 Schiffe zur gleichen Zeit am gleichen Ort
Illegaler Fischfang
Illegaler Fischfang
Illegaler Fischfang
Weitere Anwendungsfälle• Tracking
• Protokollierung von Unfällen
• Piraterie
• Routen-Optimierung
• Hafen-Logistik
• Flotten-Logistik
• Umweltschutz
• Search & Rescue
• Navigationshilfe
Danke!Kai Voigt, kai@cloudera.com