Streaming platform Kafka in SK planet
-
Upload
byeongsu-kang -
Category
Software
-
view
5.250 -
download
1
Transcript of Streaming platform Kafka in SK planet
![Page 1: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/1.jpg)
Streaming Platform Apache Kafka
Data Platform 본부
Data Infrastructure팀
강병수
![Page 2: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/2.jpg)
Intro
Data Infrastructure팀의 Kafka
Kafka Summit 2017 NY
적용기
![Page 3: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/3.jpg)
Intro
![Page 4: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/4.jpg)
![Page 5: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/5.jpg)
![Page 6: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/6.jpg)
![Page 7: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/7.jpg)
![Page 8: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/8.jpg)
![Page 9: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/9.jpg)
상품의구매전환율을실시간통계로뽑고싶어요
광고비용을실시간으로집계하고싶어요
현재접속해있는유저상태에따라마케팅을하고싶어요
현재상품의품절여부를알고싶어요
![Page 10: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/10.jpg)
사업적니즈들을기술적으로어떻게서포팅해야할지고민이늘어난다.
외래문물을수용하러...
![Page 11: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/11.jpg)
Systems Track Streams Track Pipelines Track
Data Processing at LinkedIn with Apache Kafka Portable Streaming Pipelines with Apache Beam Capture the Streams of Database Changes
Kafka in the Enterprise: What if it Fails? The Best Thing Since Partitioned Bread :
Rethinking Stream Processing with Apache
Kafka’s new Streams API
Billions of Messages a Day - Yelp’s Real-time Data
Pipeline
Apache Kafka Core Internals : A Deep Dive Microservices with Kafka : An Introduction to Kafka
Streams with a Real-Life Example
California Schemin! How the Schema Registry has
Ancestry Basking in Data
Simplifying Omni-Channel Retail at Scale Hanging Out with Your Past Self in VR: Time-
Shifted Avatar Replication Using Kafka Streams
Achieving Predictability And Compliance With The
Data Distribution Hub at Bank of New York Mellon
How to Lock Down Apache Kafka and Keep Your
Streams Safe
Building Advanced Streaming Applications using
the Latest from Apache Flink and Kafka
Every Message Counts: Kafka as Foundation for
Highly Reliable Logging at Airbnb
Running Hundreds of Kafka Clusters with 5 People The Data Dichotomy: Rethinking Data & Services
with Streams
The Source of Truth: Why the New York Times
Stores Every Piece of Content Ever Published in
Kafka
Hardening Kafka for New Use Cases with Venice Easy, Scalable, Fault-tolerant Stream Processing
with Kafka and Spark’s Structured Streaming
Single Message Transformations Are Not the
Transformations You’re Looking For
Introducing Exactly Once Semantics in Apache
Kafka
Scalable Real-time Complex Event Processing at
Uber
Cloud Native Data Streaming Microservices with
Spring Cloud and Kafka
![Page 12: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/12.jpg)
Systems Track Streams Track Pipelines Track
Data Processing at LinkedIn with Apache Kafka Portable Streaming Pipelines with Apache Beam Capture the Streams of Database Changes
Kafka in the Enterprise: What if it Fails? The Best Thing Since Partitioned Bread :
Rethinking Stream Processing with Apache
Kafka’s new Streams API
Billions of Messages a Day - Yelp’s Real-time Data
Pipeline
Apache Kafka Core Internals : A Deep Dive Microservices with Kafka : An Introduction to Kafka
Streams with a Real-Life Example
California Schemin! How the Schema Registry has
Ancestry Basking in Data
Simplifying Omni-Channel Retail at Scale Hanging Out with Your Past Self in VR: Time-
Shifted Avatar Replication Using Kafka Streams
Achieving Predictability And Compliance With The
Data Distribution Hub at Bank of New York Mellon
How to Lock Down Apache Kafka and Keep Your
Streams Safe
Building Advanced Streaming Applications using
the Latest from Apache Flink and Kafka
Every Message Counts: Kafka as Foundation for
Highly Reliable Logging at Airbnb
Running Hundreds of Kafka Clusters with 5 People The Data Dichotomy: Rethinking Data & Services
with Streams
The Source of Truth: Why the New York Times
Stores Every Piece of Content Ever Published in
Kafka
Hardening Kafka for New Use Cases with Venice Easy, Scalable, Fault-tolerant Stream Processing
with Kafka and Spark’s Structured Streaming
Single Message Transformations Are Not the
Transformations You’re Looking For
Introducing Exactly Once Semantics in Apache
Kafka
Scalable Real-time Complex Event Processing at
Uber
Cloud Native Data Streaming Microservices with
Spring Cloud and Kafka
서비스 Database를실시간입수하는방법들
Kafka를이용한 Microservice
Kafka Streams 소개
그외에도.. 안정적으로 System을운영하는사례들이나,
하둡데이터와실시간이벤트를함께활용하는방법들
![Page 13: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/13.jpg)
자세한내용과팀에어떻게적용하고있는지는뒤에서얘기하고,
먼저, Kafka를모르시는분들에위해서간단하게설명을하려고합니다.
![Page 14: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/14.jpg)
Apache Kafka
publish / subscribe
distributed system
stable store
low latency
high throughput
real-time processing
![Page 15: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/15.jpg)
Apache Kafka
Producer
BrokerProducer
Producer
Consumer
Consumer
Consumer
![Page 16: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/16.jpg)
Apache Kafka
BrokerProducertopic : syrup
data : 캣츠
![Page 17: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/17.jpg)
Apache Kafka
BrokerProducer
Disk
topic : syrup
data : 캣츠
![Page 18: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/18.jpg)
Apache Kafka
BrokerProducer Consumer
Disk
topic : syrup
data : 캣츠topic : syrup
data : 캣츠
![Page 19: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/19.jpg)
Apache Kafka
Producer
BrokerProducer Consumer
topic : 11st
data : 전자담배
topic : syrup
data : 캣츠topic : syrup
data : 캣츠
Disk
![Page 20: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/20.jpg)
Apache Kafka
Producer
BrokerProducer
Consumer
Consumer
Disk
topic : syrup
data : 캣츠topic : syrup
data : 캣츠
topic : 11st
data : 전자담배topic : 11st
data : 전자담배
![Page 21: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/21.jpg)
Apache Kafka
Producer
BrokerProducer
Consumer
Consumer
Consumer
topic : 11st
data : 전자담배
topic : 11st
data : 전자담배
Disk
topic : syrup
data : 캣츠topic : syrup
data : 캣츠
topic : 11st
data : 전자담배
![Page 22: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/22.jpg)
Apache Kafka
Producer
BrokerProducer
Consumer
Consumer
ConsumerProducer
topic : OCB
data : 500p
Disk
topic : syrup
data : 캣츠topic : syrup
data : 캣츠
topic : 11st
data : 전자담배topic : 11st
data : 전자담배
topic : 11st
data : 전자담배
![Page 23: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/23.jpg)
Apache Kafka
변경이불가능한이벤트로그를스트림으로처리한다.
Producer 와 Consumer는의존도가없다.
Disk를사용하기때문에안정적이고, 잘못됐을때돌아올수있다.
millisecond 단위의 low latency를보장한다.
![Page 24: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/24.jpg)
Disk를사용하는데어떻게빠르지?운영체제는 Disk I/O 효율을위하여 Page Cache라는것을사용한다.
![Page 25: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/25.jpg)
Apache Kafka
메모리사용상태를보면 190GB중 165GB를 Page Cache로사용
![Page 26: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/26.jpg)
Apache Kafka
CPU IO wait 역시거의발생하지않음
![Page 27: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/27.jpg)
ProcessingKafka Streams를이용하여 ETL이가능하다.
StoreDisk를이용하는안정적인저장소다.
Pub / Sub자유롭게데이터를넣고땡길수있다.
![Page 28: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/28.jpg)
Data Infrastructure팀의 Kafka
![Page 29: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/29.jpg)
App log
Web log
Server log
DB data
Data Play를하기위해서...
![Page 30: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/30.jpg)
App log
Web log
Server log
DB data
원시시대
FTP
![Page 31: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/31.jpg)
FTP로전송중에실패하면?하둡에전송된일부를지우고재전송한다.
제시간에 File이생성되지않으면?cron이동작하지않을테니손으로다시돌린다..
App이나 Web같은실시간로그는어떻게?무엇인가대책이필요하다.
![Page 32: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/32.jpg)
App log
Web log
Server log
DB data
2011년에 Kafka를도입
![Page 33: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/33.jpg)
전송중실패부분부터재전송가능
파일로뽑고읽어서보낼것없이이벤트가발생하면즉시전송
App, Web Client 로그도이벤트건 by 건적재가능
현재는대부분의로그를 Kafka를통해입수한다.
![Page 34: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/34.jpg)
38000건 / Sec 33억건 / day
IN : 92 MB / Sec 8TB / day
OUT : 310 MB / Sec 25TB / day
현재 Main Cluster는평균적으로
![Page 35: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/35.jpg)
Data Infrastructure팀의 Kafka
Recopick
3ea
Global
11st
5ea
11st
3ea
Main
10ea
Dev
5ea
Mirror
5ea
커머스
AWS
보라매
커머스
일산
MirrorMaker
MirrorMaker
MirrorMaker
![Page 36: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/36.jpg)
Data Infrastructure팀의 Kafka
Recopick
3ea
Global
11st
5ea
11st
3ea
Main
10ea
Dev
5ea
Mirror
5ea
커머스
AWS
보라매
커머스
일산
서비스로그
MirrorMaker
MirrorMaker
MirrorMaker
실시간추천
Hadoop
Monitoring 검색인덱싱
서비스로그
서비스로그
서비스로그
CEP
![Page 37: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/37.jpg)
클러스터들을혼자구성및운영하려니고충이많이생김
![Page 38: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/38.jpg)
IDC 및클러스터별로입수상태및현황을파악해야 했고,
문제가생겼을때어디쪽의문제인지추적이되어야했고,
장애시복구포인트나복구는잘되는지를알아야한다.
또한언제어떤 Topic이생겼는지등의이력관리가되어야하고,
문제제기가들어왔을때증명도해야한다...
![Page 39: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/39.jpg)
![Page 40: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/40.jpg)
Data Infrastructure팀의 Kafka
운영상편의를위하여.... IDC 통합모니터링개발
![Page 41: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/41.jpg)
Data Infrastructure팀의 Kafka
Kafka 상태모니터
![Page 42: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/42.jpg)
Data Infrastructure팀의 Kafka
Zookeeper 데이터수집
![Page 43: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/43.jpg)
Data Infrastructure팀의 Kafka
통합 Transfer
![Page 44: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/44.jpg)
운영이편해졌다. 하지만...
![Page 45: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/45.jpg)
서비스 DB를실시간으로어떻게입수하지?
real-time streaming 환경은되어있는데실제서비스는몇없다….
MicroService를 Kafka와함께쓰는게추세라던데?
실시간으로통계를뽑으려면 Hadoop으로는한계가있지..
![Page 46: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/46.jpg)
고민해결의꿈을안고 Kafka Summit 2017 NY
![Page 47: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/47.jpg)
Kafka Streams
![Page 48: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/48.jpg)
Kafka Streams
Kafka project의 streaming library
2016년에 release
기존의다른 Streaming Framework에비하여큰이점들이있다.
그러나, 나온지가얼마안되서증명이안됐다.
![Page 49: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/49.jpg)
Kafka Streams
눈부신성공사례로어느정도증명이...
Line corps. 내부파이프라인에 Kafka Streams적용하기https://engineering.linecorp.com/en/blog/detail/80
Peak time : 100만메시지 / sec
![Page 50: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/50.jpg)
Kafka Consumer
BrokerProducertopic : 11st
data : 전자담배data : 화분
![Page 51: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/51.jpg)
Kafka Consumer
Broker
Consumerconsumergroup : group-1
topic : 11st
data : 전자담배
Consumer
consumergroup : group-1
topic : 11st
data : 화분
Producertopic : 11st
data : 전자담배data : 화분
![Page 52: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/52.jpg)
Kafka Consumer
Broker
Consumerconsumergroup : group-1
topic : 11st
data : 전자담배
Consumer
consumergroup : group-1
topic : 11st
data : 화분
Producertopic : 11st
data : 전자담배data : 화분
Scalability
Load balancing
![Page 53: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/53.jpg)
Kafka Consumer
BrokerProducertopic : 11st
data : 자전거data : 전동킥보드
![Page 54: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/54.jpg)
Kafka Consumer
Broker
Consumerconsumergroup : group-1
topic : 11st
Consumer
consumergroup : group-1
topic : 11st
data : 자전거전동킥보드
Producertopic : 11st
data : 자전거data : 전동킥보드
![Page 55: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/55.jpg)
Kafka Consumer
Broker
Consumerconsumergroup : group-1
topic : 11st
Consumer
consumergroup : group-1
topic : 11st
data : 자전거전동킥보드
Producertopic : 11st
data : 자전거data : 전동킥보드
High Availability
![Page 56: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/56.jpg)
Kafka Streams
Consumer, Producer
State
1m
2m
3m
4m
Join
Time Window
![Page 57: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/57.jpg)
Kafka Streams
비슷한역할을수행하던것들
![Page 58: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/58.jpg)
Kafka Streams
Kafka Streams의장점은?
![Page 59: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/59.jpg)
Kafka Streams
Kafka Streams의장점은?
라이브러리이다.
Yarn과같은 resource manager가필요없다.
오직데몬을돌릴수있는서버만있으면된다.
소스가가볍다.
H/A, Scalability, Load Balancing 을구현할필요가없다. Kafka에서다해준다.
생산성이좋아진다.
디버깅이쉽다. 비즈니스로직에만집중해서개발을하면된다.
![Page 60: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/60.jpg)
Kafka Streams로쉽고좋은 Streaming Application을만들수있다.
![Page 61: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/61.jpg)
Microservice
![Page 62: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/62.jpg)
하나의레파지토리에 100만라인의코드였다.
모노리딕앱 : 옐프메인2011
2013
2017
Kafka를백본으로, 마이크로서비스로옮겼다.
70개이상의 production 서비스였다.
2013년보다 R&D에 100만달러를아끼게됐다.
코드의복잡성이줄어들고, downtime이줄어들었다.
![Page 63: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/63.jpg)
Monolithic service Micro service
![Page 64: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/64.jpg)
서비스끼리통신이발생한다.
![Page 65: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/65.jpg)
서비스는필연적으로복잡해진다.
![Page 66: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/66.jpg)
서비스하나의장애가다른서비스에도영향을준다
![Page 67: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/67.jpg)
시스템의복잡도를줄이고 Coupling을줄일수없을까?
![Page 68: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/68.jpg)
Kafka를 Backbone으로활용하면해결할수있다.
![Page 69: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/69.jpg)
Asynchronous하게 데이터를던지고받는다
A B
DC
E
B - getItem(1) A - 수영복C - 양말, 물안경, 수영복….
B - top10
![Page 70: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/70.jpg)
Asynchronous하게 데이터를던지고받는다
A B
DC
E
A - 수영복
C - 양말, 물안경, 수영복,
…...
![Page 71: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/71.jpg)
하나의서비스에장애가나도다른서비스는모른다
![Page 72: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/72.jpg)
또한복구가됐을때, 처리못한 request를모두불러올수있다.
![Page 73: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/73.jpg)
Message Queue는서비스의복잡도를확연하게줄여준다.
Publish / Subscribe 구조는 Coupling을줄여준다.
![Page 74: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/74.jpg)
Kafka를이용한 Microservice를어떻게만들수있을까?
![Page 75: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/75.jpg)
출처 : 넷플릭스 마이크로서비스 가이드https://www.youtube.com/watch?feature=youtu.be&v=OczG5FQIcXw&app=desktop
Microservice의단위는?
![Page 76: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/76.jpg)
Consumer, Producer
State
1m
2m
3m
4m
Join
Time Window
Kafka Streams를이용해서만들수있다.
![Page 77: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/77.jpg)
Kafka에서 Data 입출력이쉽고,
Scalability,
Load Balancing,
H/A 구성
![Page 78: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/78.jpg)
Spring Cloud의 Kafka binder를사용해서쉽게만들수있다.
![Page 79: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/79.jpg)
Kafka를사용하면 Microservice의복잡한문제들을꽤많이해결해준다.
![Page 80: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/80.jpg)
Change Data Capture
![Page 81: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/81.jpg)
11번가상품 DB
11번가주문 DB
Syrup
유저 DB
서비스 DB
![Page 82: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/82.jpg)
11번가상품 DB
11번가주문 DB
Syrup
유저 DB
서비스 DB
Data Application
Data Analysis
![Page 83: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/83.jpg)
서비스 DB는 Kafka를통해적재를못하고있다.
![Page 84: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/84.jpg)
PK Data
A 2
B 2
결과
실시간트랜잭션Update A 1 -> 2
Delete C
Insert B 1
Update B 2
DB
Event Log들과다르게 DB는최종변경내용이저장된다.
![Page 85: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/85.jpg)
/data/test
A 1
A 2
B 1
B 2
Hadoop
실시간트랜잭션Update A 1 -> 2
Delete C
Insert B 1
Update B 2
DB
Hadoop은상태변경을처리하기에적합하지않다.
![Page 86: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/86.jpg)
RDB Hadoop
매일또는매시간Full dump batch
그래서우리는
![Page 87: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/87.jpg)
RDB Hadoop
Data Analysis
Data Application
매일또는매시간Full dump batch
![Page 88: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/88.jpg)
RDB Hadoop
최신상태가필요해..
Full dump에만 6시간이걸려…
File dump가없으면연락후재입수
매일또는매시간Full dump batch
![Page 89: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/89.jpg)
실시간으로데이터베이스변경분을반영할수없을까?
![Page 90: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/90.jpg)
Database의실시간트랜잭션로그는어떻게뽑지?
뽑을수있다면데이터를어떻게순서보장을하면서전송을할까?
Hadoop이아닌어떤 Repository를이용해야할까?
실시간상태를 Hadoop에서도볼수있어야하는데..
넘어야할산들
![Page 91: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/91.jpg)
우리에게도움을줄회사들..
![Page 92: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/92.jpg)
Database의실시간트랜잭션로그는어떻게뽑지?
넘어야할산들
Redo Log, Undo Log parser - Log Miner, OGG, XStreams etc...
Bin log parser - Open-Replicator Link : https://github.com/whitesock/open-replicator
![Page 93: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/93.jpg)
뽑을수있다면데이터를어떻게순서보장을하면서전송을할까?
넘어야할산들
실시간로그를입수하면서
유실없이
메시지간순서보장을할수있다.
![Page 94: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/94.jpg)
Hadoop이아닌어떤 Repository를이용해야할까?
넘어야할산들
![Page 95: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/95.jpg)
![Page 96: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/96.jpg)
실시간상태를 Hadoop에서도볼수있어야하는데..
넘어야할산들
Snapshot to HDFS
또는 Hive와연동
![Page 97: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/97.jpg)
RDB Hadoop
Change
Data
Capture
![Page 98: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/98.jpg)
RDB Hadoop
Change
Data
Capture
Data Application
Data Analysis
![Page 99: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/99.jpg)
Data 분석시, 또는 Data Application에서
실시간으로서비스 DB의상태를사용할수있다.
![Page 100: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/100.jpg)
이제다시 DI Cluster로돌아가보면
![Page 101: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/101.jpg)
상품의구매전환율을실시간통계로뽑고싶어요
광고비용을실시간으로집계하고싶어요
현재접속해있는유저상태에따라마케팅을하고싶어요
현재상품의품절여부를알고싶어요
![Page 102: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/102.jpg)
현재상태로돌아가보면
Sentinel
Rake
Server
Kafka
Hadoop Hive
Log
Agent
RakeAPI
Mirror
Maker
Collector
Other
Streaming
Apps
![Page 103: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/103.jpg)
Router
위내용들을적용하여...
Sentinel
Rake
Server
Kafka
Hadoop
Druid
Hbase
Redis
Hive
Log
Agent
RakeAPI
Mirror
Maker
RDB
Collector
Other
Streaming
Apps
실시간 OLAP
CDC
OGG
유저상태
![Page 104: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/104.jpg)
![Page 105: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/105.jpg)
사업적니즈를함께풀어가기위한노력들
유저가여러번클릭한상품에대해서즉시구매까지전환되도록할수없을까?
실시간통계를보면서수익이많이나는광고를내보내면좋을것같은데..
현재접속해있는유저를대상으로마케팅을하면어떨까?
![Page 106: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/106.jpg)
사업적니즈를함께풀어가기위한노력들
사업적요건들을 Data Infrastructure Cluster안에서해결해나갈수있도록
고민과노력중!
![Page 107: Streaming platform Kafka in SK planet](https://reader034.fdocument.pub/reader034/viewer/2022050613/5a64a1a97f8b9a27568b7d75/html5/thumbnails/107.jpg)
함께데이터의활용가치를높여나가자