서비스 모니터링 구현 사례 공유 - Realtime log monitoring platform-PMon을 지탱하는 기술
AWS Summit - AWS를 통한 신뢰성 높은...
-
Upload
jongwon-han -
Category
Software
-
view
410 -
download
2
Transcript of AWS Summit - AWS를 통한 신뢰성 높은...
©�2017,�Amazon�Web�Services,�Inc.�or�its�Affiliates.�All�rights�reserved.
한종원,�‘HB�Smith’�CEO
AWS를�통한�신뢰성�높은�지속적�배포�및�통합(CD/CI)�사례�
1부
본�강연에서�다룰�내용�
•1부�(20분)�-�한종원�
• Introduction�
• 'Daily�Continuous�Deployment'�(이하�'daily�CD’)로의�여정�
• Play�with�AWS�
• 실제�운영�사례들�
• Wrap�Up�
•2부�(20분)�-�윤제상
2
Introduction
발표자�소개
•한종원�(�https://www.linkedin.com/in/addnull/�)�
•Python과�Cloud�Infra,�Lean/Agile�방법론�그리고�애플의�제품을�사랑�
•2012년�석사�학위를�마치고,�스타트업을�시작�(이때부터�AWS를�production�level에서�사용)�
•'의미가�있는�일을,�올바르게�하고�싶다.’�
•경력��
• (현)�DevOps�전문�스타트업�‘HB�Smith’�대표�(�https://hbsmith.io�)�
• (2016)�택시�O2O�서비스�스타트업�‘Kanizsa�Lab’의�backend�server�/�infra�devops�담당�
• (2015)�Cloud�computing�전문�스타트업�'A2�company'�co-founder�(‘KINX’에�인수합병)�
• (2010)�Samsung�Software�Membership�(수원,�대전)�
• (2007)�NEXON�'MapleStory�국내�Live�Team'에서�DBA,�SA로�근무�(산업�기능�요원)
4
Introduction
•PyCon�APAC�2016�(�https://www.pycon.kr/2016apac/program/15�)�
• “Daily�Continuous�Deployment를�위한Custom�CLI�개발�및�AWS�Elastic�Beanstalk에�적용하기”
5
Introduction
•오늘은�‘daily�CD’와�AWS�Elastic�Beanstalk에�집중�
• ‘daily�CD’란�무엇이고,�왜�필요한가?�
• 왜�AWS�를�선택했는가?�
• 왜�Elastic�Beanstalk를�선택했는가?
6
Daily�CD로의�여정
Continuous�Deployment란?
•from�Wikipedia"Continuous�delivery�is�sometimes�confused�with�continuous�deployment.�Continuous�deployment�means�that�every�change�is�automatically�deployed�to�production.”�
•정의:�자동으로�최신�버전의�소스�코드를�테스트,�패키징,�배포
이미지�출처:�http://evolutyz.in/devops.html
8
Daily�CD란?
•정의:�“매일”�자동으로�최신�버전의�소스�코드를�테스트,�패키징,�배포�
• 쉽게�적용�가능한�대상�
• REST�API�server�(WAS),�Worker�(async�job)�
• 주로�computing만�하고�data,�log�저장을�거의�하지�않는�서버군�
• 적용하기�어려운�대상�
• DB,�Storage�
• 대용량�data,�log�저장을�해야하는�서버군
9
Daily�CD로의�여정
•DevOps�외길�인생�
• 2006년�Nexon�MapleStory�개발팀의�SA,�DBA로�근무�
• 실제�물리머신�환경�(IDC)�
• non-stop�patch,�DR�등의�요구사항�
• DB�HA�구성을�고려�
• L4�switch�준비�
• PoC,�PoC�그리고�PoC�
• 그러나…
10
Daily�CD로의�여정
•DevOps�외길�인생�
• 2012년�Cloud�전문�startup,�‘A2�company’�창업�
• 외주�SI/SM으로�AWS을�처음�접함�
• OpenStack으로�AWS와�유사한�IaaS를�구축�(‘Juno’)�
• 2016년�DevOps�전문�startup,�‘HB�Smith’�창업
11
Daily�CD로의�여정
•약�10년간�이�바닥(devops)에서�굴러보니…�
• 자동화는�장기적으로�언제나�승리한다.(사람�손이�문제더라..)�
• 3R�(Repeatable,�Reproducible,�Reliable)을�추구하라.�
• 서버(물리머신이든,�VM이든)는�오래되면,�갑자기�맛이�간다.(memory�leak?)�
• 마지막으로�reboot한지�오래된�서버일�수록�위험하다.(모든�면에서)
12
이미지�출처:�https://pinterest.com/psychokwack/devops/
Daily�CD로의�여정
•소년은�늘�꿈을�꿉니다.�
• non-stop�자동�server�patch�
• HA와�DR�
• 항상�깨끗한�서버�(알�수�없는�계정,�쓰레기�임시�파일�찌거기들�없는�무공해�청정�서버)
13
Daily�CD로의�여정
•목표�
• 매일�서버(VM)를�새로�다시�띄워서(deployment)�이전�서버와�교체하자.�
• 즉,�서버를�‘하루살이’로�만들자.�
•요구사항�
• 3R�(Repeatable,�Reproducible,�Reliable)�
• non-stop�(zero�downtime,�seamless)�
• HA와�DR
14
Play�with�AWS
AWS와�첫만남
•“EC2�server”가�아니라�“EC2�instance”�
• VM�역시�bug가�있는�software�
• 언제든지�갑자기�망가질�수�있는�임시적�자원�
• Infra�장애를�100%�미리�예방할�수�없기에,“빠른�복구”가�가능하도록�system�architecture를�구성한다.
16
왜�AWS�인가?
•2012년�부터�production�level에서�사용�
•높은�신뢰성�
•풍부한�3rd�party�
•낮은�비용�!!!�
•AWS를�쓰면,�개발/배포�속도가�빨라지는�게�아니라,�느려지지�않음.
17
왜�AWS�인가?
•다양한�troubleshooting
18
왜�AWS�인가?
•지속적으로�확장되는�기능,�제품군
19
###�QUIZ�###
•첫번째�AWS�서비스는?�
•https://aws.amazon.com/blogs/aws/aws-blog-the-first-five-years/
20
Play�with�AWS�#1
•EC2(AMI)�+�ELB�조합으로�Daily�CD�도전�
• ELB�health�check�기능�사용�
• ELB�밑에�있는�EC2�중에�절반을�detach�함�
• detach된�EC2를�patch�(이때,�ASG를�위한�AMI�생성)�
• 다시�attach�하고�나머지�절반을�detach�
• detach된�EC2를�patch�
• 다시�attach
21
Play�with�AWS�#1
•EC2(AMI)�+�ELB�조합의�문제점�
• AMI�version�관리�어려움�(도대체�안에�뭐가�있는거지?)�
• OS�level의�security�patch�등을�직접해야함�
• AMI�안에�쌓이는�찌꺼기�(사용하지�않는�계정,�임시�파일들)�
• ELB�에서�detach,�attach�하는�번거로움�
• 짧은�순간이지만,�patch된�EC2와�그렇지�않은�EC2가�동시에�ELB에�attach된�상태가�존재함
22
Play�with�AWS�#2
•EB로�Daily�CD�도전�
• EB�environment(EC2�+�ELB)�단위로매일�새로�deployment�
• 새로운�EB�env를�생성�
• 이때�새로�생성한�EC2의OS(Amazon�Linux)는�최신�AMI로부터�생성�
• EC2�에�software�stack을�처음부터�설치�
• 새�EB�env가�green�status로�바뀌면,이전�EB�env와�CNAME�swapping으로�교체
23
Play�with�AWS�#2
•EB의�Daily�CD�단점�
• EB의�동작�원리를�이해해야함(예:�‘/opt/python/ondeck’�경로가�나중에�‘/opt/python/current’으로�변경)�
• EB의�기본�설정을�너무�건드리면,�알�수�없는�문제가�발생될�수�있음�
• 교체당한�EB�env로도�일정�시간�traffic이�흘러감�(client�DNS�caching)�
• 교체�후�1시간�정도�지난�후에�해당�EB�env�삭제�
• Log�유실의�우려�
• 기본�15분마다�S3에�복사하는�cron�job의�주기를�짧게�줄일�수�있음http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.logging.html
24
Play�with�AWS�#2
•EB의�Daily�CD�장점�
• 모든�요구사항을�다�만족함�
• 3R�(Repeatable,�Reproducible,�Reliable)�->�신입�개발자에게�빠르게�개발�환경�제공�가능�
• non-stop�(zero�downtime,�seamless)�
• HA와�DR�
• 그외�추가적인�장점들�
• 항상�깨끗한�서버�(알�수�없는�계정,�쓰레기�임시�파일�찌거기들�없는�무공해�청정�서버)�
• 오래된�서버가�가지는�문제점,�고민�해결�(memory�leak,�오랜만에�reboot�등)�
• 개발자들은�logic�code에�더욱�집중�
• 서버�tier를�작게�쪼갤�수�있음�(즉,�microservices�또는�nanoservices�가�가능)�
• T2�instance의�bonus�CPU�credit�!!!
25
실제�운영�사례들
실제�운영�사례�#1
•이미�AWS�Tokyo�region을�쓰고�있었음�
•2016년�AWS�Seoul�region�open�
•AWS�Tokyo�region에서�운영하던�infra�전체를�Seoul�region으로�migration�
•기존에�Daily�CD를�쓰고�있던�상태이므로�migration에�반나절�정도�걸림�
• Tokyo�region의�S3,�RDS�data를�Seoul�region으로�migration�(수작업)�
• REST�API�server�(WAS),�Worker�(async�job)는�Daily�CD�설정에서‘ap-northeast-1’을�‘ap-northeast-2’로�변경만�함�
•약�1년간�운영
27
실제�운영�사례�#2
•다른�public�cloud�서비스를�쓰고�있었음�
•1대의�VM에�all-in-one(DB,�WAS…)�상태�
•AS-IS�code�분석�후�WAS,�worker,�storage,�DB�tier�분리�및�AWS�로�migration�
• AWS�migration�하면서�동시에�WAS,�Worker�는�Daily�CD�설정�적용�
•AWS으로�전체�infra�migration�완료까지�작업에�4주�소요�
•현재�약�6개월�정도�운영�중
28
실제�운영�사례�#3
•저희�‘HB�Smith’�사례�(�https://hbsmith.io�)�
•현재�전체�AWS�infra를�Daily�CD로�운영�중�
• 단,�대용량�data,�log�저장을�해야하는�서버군(예:�RDS)은�제외�
•관련된�Python�code를�OSS�‘Tiamat’로�공개�
• https://github.com/HardBoiledSmith/tiamat�
• https://github.com/HardBoiledSmith/johanna
29
운영하면서�느낀�Daily�CD�단점
•Provisioning�code�관리에�공수가�아주�없지�않음�
•철학을�이해�못하면,�전체�운영이�어려워짐�
• 문제제기:�‘기능�구현,�bug�수정이�급한데,�왜�이걸로�시간과�돈을�들임?’�
• 답변:�‘이건�개발,�배포�속도를�빠르게�하는게�아니라,�느려지지�않게�하기�위함임.’
30
Wrap�Up
체크�포인트�
•발표�내용�wrap�up�
• 'daily�CD’로의�여정�
• ‘daily�CD’란�무엇이고,�왜�필요하다고�느꼈는지�
• Play�with�AWS�
• 왜�AWS를�선택했는지,�그리고�어떻게�‘daily�CD’를�구축했는지�
• 실제�운영�사례들�
• 3개의�법인에서�적용�사례와�장기간�운영하면서�느낀�‘단점’
32
본�강연이�끝난�후…
•AWS�Elastic�Beanstalk�시작하기�
• http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/GettingStarted.html�
•Python�+�AWS�EB�조합으로�구현한�‘daily�CD’의�자세한�정보�얻기�
• https://www.pycon.kr/2016apac/program/15�
•GitHub�‘Tiamat’�(AWS�CLI�기반)에�contributor�지원�
• https://github.com/HardBoiledSmith/tiamat�
•AWS�migration이나�EB�사용(세부적�tuning)�문의�보내기�
• https://hbsmith.io�
33
https://www.awssummit.kr�
AWS�Summit�모바일�앱을�통해��지금�세션�평가에��참여하시면,�행사�후�기념품을�드립니다.
#AWSSummitKR�해시태그로�소셜�미디어에�여러분의�행사�소감을�올려주세요.
발표�자료�및�녹화�동영상은�AWS�Korea�공식�소셜�채널로�공유될�예정입니다.
여러분의�피드백을�기다립니다!