Transcript of Amazon...
- 1. Amazon Kinesis
- 2. ! ohtani@amazon.co.jp @shot6 ! !
- 3. ! : ( ) ! : Partner Solution Architect ! : ! 3 AWS ! AWS :
KinesisS3
- 4. Kinesis
- 5. AWS Kinesis DynamoDB RedshiftRDS EMR Map-Reduce) Kinesis ! !
! ! ! ! !
- 6. Amazon Kinesis Data Sources App.4 [Machine Learning] AWS
Endpoint App.1 [Aggregate & De-Duplicate] Data Sources Data
Sources Data Sources App.2 [Metric Extrac=on] S3 DynamoDB Redshift
App.3 [Sliding Window Analysis] Data Sources Availability Zone
Shard 1 Shard 2 Shard N Availability Zone Availability Zone
SDK
- 7. ! Management Console Stream Cloud Watch API
- 8. Amazon Kinesis
- 9. Kinesis Kinesis Kinesis Kinesis Kinesis
- 10.
- 11. MD5 Max: 256Byte Max: 50KB Kinesis
- 12. PutDC 24
- 13. READ : 5 TPS or 2MB/sec WRITE : 1000 TPS or 1MB/sec
- 14. 0 MD5 Range 2128 -1 (2128/2 - 2128) MD5 () -0 (0 -
2128/2)
- 15. (14) (15) (17) (19) (20)
- 16. Kinesis(KCL) Kinesis (KCL) Kinesis (KCL) AAA BBB 12345
98765 24680 (24680) (12345) (98765) AAA BBB DynamoDB
- 17. Kinesis(KCL) Kinesis (KCL) Kinesis (KCL) AAA BBB 12345
98765 24680 (24680) (12345) (98765) AAA BBB DynamoDB 1. KCL 2. ID
DynamoDB
- 18. Kinesis Kinesis S3 Kinesis S3 (24680) (12345) (98765) S3-1
S3-2 Kinesis DynamoDB Kinesis DynamoDB D-1 D-2
- 19. Amazon Kinesis
- 20. Raspberry pi +MQTT Kinesis App [Real=me ETL] Frontend [MQTT
Proxy] Frontend [MQTT Proxy]
- 21. Raspberry pi ! Raspberry pi + (ADT7410) 21 ! I2CRaspberry
pi ! 101 ! Raspberry piKinesisPut ! Kinesis
- 22. MQTT ! M2Mpub/sub ! !
http://public.dhe.ibm.com/software/dw/webservices/ws-
mqtt/mqtt-v3r1.html ! Mosquitto : MQTT http://mosquitto.org/ !
Raspberry pi MQTT 22
- 23. Raspberry pi +MQTT Kinesis App [Real=me ETL] Frontend [MQTT
Proxy] Frontend [MQTT Proxy] import paho.mqtt.publish as publish
publish.single("mqttkb/test", "boo", hostname=xxxx")
- 24. Raspberry pi +MQTT Kinesis App [Real=me ETL] Frontend [MQTT
Proxy] Frontend [MQTT Proxy] MQTT-Kinesis
- 25. Raspberry pi +MQTT Kinesis App [Real=me ETL] Frontend [MQTT
Proxy] Frontend [MQTT Proxy]
- 26. Raspberry pi +MQTT Kinesis App [Real=me ETL] Frontend [MQTT
Proxy] Frontend [MQTT Proxy] KinesisDynamoDB
- 27. Raspberry pi +MQTT Kinesis App [Real=me ETL] Frontend [MQTT
Proxy] Frontend [MQTT Proxy] Kinesis Client LibraryPython
- 28. 28
- 29. Amazon Kinesis
- 30. Kinesis ! Kinesis SDK SDK ! Kinesis Client Library(KCL)
https://github.com/awslabs/amazon-kinesis-client ! Kinesis
Connectors https://github.com/awslabs/amazon-kinesis-connectors !
Kinesis Storm Spout https://github.com/awslabs/kinesis-storm-spout
! EMR Connector EMRAMI 30
- 31. Kinesis Client Library(KCL) ! KinesisJava DynamoDB,
CloudWatch, AutoScaling + ! IRecordProcessorFactory
IRecordProcessor 31
- 32. 32
- 33. Kinesis WebHDFS ! KinesisEMRHDFS ! WebHDFSREST ! TO 33 REST
HTTP PUT http://xxx.9101/webhdfs/v1/user/ohtani/le
?op=create&user.name=hadoop Kinesis EMR Client
- 34. 34 Kinesis WebHDFS (EC2)
- 35. 35 Kinesis WebHDFS (EC2) EMR HDFS
- 36. Kinesis Connectors ! KinesisAWS 36 ITransformer Kinesis
IFilter IBuer IEmitter AWS S3 DynamoDB Redshift Kinesis
- 37. Amazon Kinesis
- 38. AWS AWS Endpoint Kinesis App.1 Kinesis App.2 Redshift
DynamoDB Kinesis App.3 Availabilit y Zone Shard 1 Shard 2 Shard N
Availabi lity Zone Availabi lity Zone RDS BI as a Service S3
Kinesis
- 39. ! Stream Data Sources Data Sources Data Sources Kinesis App
Kinesis App Kinesis App Data Sources Data Sources Data Sources
Kinesis App Kinesis App Kinesis App Kinesis App Data SourceA Data
SourceB Data SourceAETL Data SourceBETL
- 40. KinesisEMR DynamoDB RedshiftRDS EMR Map-Reduce) Kinesis ! !
! ! ! ! !
- 41. Kinesis Kinesis EMR EMR DynamoDB ElastiCache Redis
S3DynamoDB ElastiCache Redis Redshift S3 Redshift EMR
KinesisEMR
- 42. ! Kinesis ! Kinesis + AWS ! 42
- 43.