Softwareson Large Scale Internet Services

2013/07/18 at Tokyo Univ.SATOSHI TAGOMORI (@tagomoris)

LINE Corp.

topics

About LINE Corp.

Service scales

Large is worse

Internet (Web) service architecture

Case studies on LINE Corp.

Q&A

TAGOMORI Satoshi (@tagomoris)LINE corp.

tagomoris: 田籠 聡

-2005.03:東京大学計数工学科システム情報工学コース新研究室

2005.04-2006.03: 朝日ネット2006.03-2010.06: NTTデータ先端技術2010.08-:ライブドア → NHN Japan → LINE

Service scales

small, middle, large and huge scale

Scale(1): Users, traffics, data amount, market share, ...

Scale(2): Servers, programmers, lines of code, ...

Service scales

small middle large huge

users ～100k ～1m ～10m 100m～

servers 1, 2 or 3 ... tens handreds thousandsor more

ex. … …livedoor blog

cookpadAmeba, 楽天, ...

Google, TwitterFacebook

Amazon, ...

More/less servers

Servers

less: computing power-- / management cost--

more: computing power++ / management cost++

Server types (complexity)

less: efficiency-- / management cost--

more: efficiency++ / management cost++

Large is worse:more traffics needs more computing power

less server types more server types

less servers

more servers

small middle

huge large

Small scale internet (web) serviceserver-side architecture

Users (PC, smartphone, mobilephone, ...)

Database(RDBMS, NoSQL)

Web Server + Application Server(Apache + PHP, nginx + passenger(ruby), ...)

External API

Middle scale internet (web) serviceserver-side architecture

Users (PC, smartphone, mobilephone, ...)

Database(RDBMS, NoSQL)External API

Application Server(Java, Perl, Ruby, ...)

Caches(memory, disk)

Web Server (Apache, nginx, ...)

Large scale internet (web) serviceserver-side architecture

Users (PC, smartphone, mobilephone, ...)

Database(RDBMS, NoSQL)External API

Application Server(Java, Perl, Ruby, ...)

Caches(memory, disk)

CDN

Storage Service

Web Server (Apache, nginx, ...)

Dataanalysis

Huge scale internet (web) serviceserver-side architecture

Users (PC, smartphone, mobilephone, ...)

Distributed datastore(GFS+BigTable, HDFS+HBase, ...)

Application server farm (IaaS, PaaS)

Web server cluster + CDN

Software engineering:for large scale services

Performance: application, database, storage, cpu

Stability: network, high availability, monitoring, logging

Low costs: deployment, continuous testing

Incomes: response time, data analysis, visualization

Our scope in server side

Users (PC, smartphone, mobilephone, ...)

Database(RDBMS, NoSQL)External API

Application Server(Java, Perl, Ruby, ...)

Caches(memory, disk)

CDN

Storage Service

Web Server (Apache, nginx, ...)

Dataanalysis

Our scope in server side

Users (PC, smartphone, mobilephone, ...)

Database(RDBMS, NoSQL)External API

Application Server(Java, Perl, Ruby, ...)

Caches(memory, disk)

CDN

Storage Service

Web Server (Apache, nginx, ...)

Dataanalysis

BREAK.

CASE STUDIES OFLINE CORP.

LINE secret sale:traffic spike

Heavy traffic on sale timetransactionsremaining updatesimage outbounds

SSDApplication tuning

transactionscache updates

LINE manga:content delivery

content (image) delivery

STF

distributed object storage

livedoor blog:heavy traffic

STF based image storage + cache

CDN

MySQL database cluster

Server Side Include (SSI)

Access analysis tools

system administration:server setup, deployment, monitoring, ...

server setup / deployment:automation(kickstart), testing(serverspec)itempotent deployment(chef, puppet)transactional operation

monitoring:server/switch metricsCDN API integration

Service monitoring:Hadoop, stream processing, ...

Daily service metrics (hadoop+hive)

Pageviews, Unique users, ...

(Semi-)Real-time service metrics (fluentd+norikra)

Req/sec, Error rate, Response time percentile, ...

+ system integration

autoscaling, service status for end users, ...

micro level tunings

Log archive serverlinux md raid-6 → hardware raid-1+0I/O performance dramatically decreasedI/O schedular: CFQ → deadline

Stream processing node (ruby, fluentd)Increasing memory usage (fragmentation)Memory allocator: glibc malloc → jemalloc

Our scope in server side

Users (PC, smartphone, mobilephone, ...)

Database(RDBMS, NoSQL)External API

Application Server(Java, Perl, Ruby, ...)

Caches(memory, disk)

CDN

Storage Service

Web Server (Apache, nginx, ...)

Dataanalysis

Our scope of computer stack

PC/smartphone browser iOS/Android App

web servers / application serversapplication code

language runtime object stores RDBMS / KVS

OSserver, NIC, memory, HDD/SSD, ...

network, firewall, load balancingauthentication(policies), audit

logger / watchdogs / cron / ...

HTML/CSS, Javascript ...

Softwareson large scale internet services

All of available things!

linux, lvs, haproxy, apache, nginx, squid, varnish, memcached, mysql, kyoto tycoon, mongodb, cassandra, redis, riak, perl, ruby, php, python, java, javascript, node.js, hadoop, hbase, hive, fluentd, ...

except for

global-scale distributed systems

processor design

ALL OPTIONSARE AVAILABLEINSERVICE PROVIDERS.