memcached proxy server development and operation

memcached proxy server development and operation

Tatsuhiko [email protected]

13年9月20日金曜日

About me

✦ Tatsuhiko Kubo([email protected])✦ @cubicdaiya(twitter, github)✦ Senior Software Engineer@pixiv Inc.


mailto:[email protected]

mailto:[email protected]

Recent Work

✦ mruby_nginx_module✦ Embedded mruby into Nginx

✦ ngx_small_light✦ Dynamic Image Transformation for Nginx

✦ Contributions/pull requests to many other projects on github


https://github.com/cubicdaiya/mruby_nginx_module

https://github.com/cubicdaiya/mruby_nginx_module

https://github.com/cubicdaiya/ngx_small_light

https://github.com/cubicdaiya/ngx_small_light

Recent WorkWEB+DB PRESS Vol.72

□detailed nginx

flexible configuration and

brilliant scalability

□collaborators

@harukasan@semind



～at pixiv～


Before that,


Background


or


About datastore & caching strategy

at pixiv


Previously(2 years ago...)

・・・・・・



There was a memcached on each application server.

・・・・・・



・・・・・・



AP connected through UNIX domain socket.

・・・・・・



Connects through TCP socket, too.(For data synchronization)


rough-and-ready patch for this


Problems

✦ data synchronization is straining✦ TCP connection overhead✦ processing time is directly proprtional the

number of APs

✦ data restoration is painful✦ memcached is not persistent✦ e.g, when AP server goes down.


Migration to KyotoTycoon

✦ KyotoTycoon supports

✦ data persistency.

✦ data expiration

✦ memcached ASCII protocol.✦ We could migrate without modifying application!


After migration


KyotoTycoon overwhelmed！


KyotoTycoon overwhelmed

✦ KyotoTycoon(using memcached protocol plugin) compared with memcached, ✦ Lower performance




Even so, persistency is some good!




Even so, persistency is some good!We wanted it at that time!


Other Problems

✦ We needed

✦ seamless fail-over mechanism

✦ convenient monitoring mechanism

✦ slow query log(like MySQL)

✦ statistics for whole queries


dealing with these problems

✦ Scale out(For example, add servers)✦ This was difficult for us at least at that time...

✦ Scale up(For example, buy expensive servers)

✦ This was more difficult for us.

✦ Use proxy server!

✦ Actually, I developed one in C.

✦ Twemproxy was not released yet.


✦ Scale out(For example, add servers)✦ This was difficult for us at least at that time...

✦ Scale up(For example, buy expensive servers)

✦ This was more difficult for us.

✦ Use proxy server!

✦ Actually, I developed one in C.

✦ Twemproxy was not released yet.

dealing with these problems


Currently


Currently

✦ Applications connect to neoagent by UNIX domain socket✦ reduced consumption of local-ports

✦ neoagent keeps persistent TCP connections to KyotoTycoon✦ bypass overhead of TCP connection


neoagent～A Yet Another Memcached Protocol Proxy Server～


neoagent✦ event-driven & multi-threaded server

✦ proxies a subset of memcached protocol

✦ get, set, add, delete, incr, decr, quit

✦ supports following features

✦ connection-pooling

✦ graceful-restart & online-update

✦ convenient monitoring

✦ etc...13年9月20日金曜日

neoagent architecture✦ core

✦ written in C (C99 style)✦ tools(CLI)

✦ written in Python✦ dependencies

✦ libev, json-c, pthread(core)✦ SCons, Sphinx(tools)


neoagent architecture

✦ Single-master and multiple workers✦ Configuration by JSON✦ Multi-threaded✦ Event-driven


Single Master and multiple workers


neoagentprocess 1

neogentprocess 2

neoagent process 3

KyotoTycoon

KyotoTycoon

KyotoTycoon

proxies

proxies

proxies

Apache(mod_php)

send request

Previously


Previously

✦ each neoagent-process was independent✦ neoagent-process was daemonized by

daemontools


neoagentprocess 1 KyotoTycoon

proxies

neoagent processes spawn gradually



proxies


proxies




proxies


proxies


proxies




proxies


proxies


proxies

・・・




proxies


proxies


proxies

neoagentprocess n KyotoTycoon

proxies

・・・



Previously


(ノ`Д´)ノ┻┻


neoagentworker

process 1

neogentworker

process 2

neoagentworker

process 3

KyotoTycoon

KyotoTycoon

KyotoTycoon

proxies

proxies

proxies

Currentlyneoagentmasterprocess

fork


neoagentworker

process 1

neogentworker

process 2

neoagentworker

process 3

KyotoTycoon

KyotoTycoon

KyotoTycoon

proxies

proxies

proxies

Apache(mod_php)

send request

Currentlyneoagentmasterprocess


single-master and multiple workers

✦ Master is responsible for✦ controlling worker processes

✦ Workers are responsible for✦ processing user’s requests

✦ Supervisor may control only Master✦ Now we use Monit instead of daemontools


Configuration by JSON


Configuration by JSON{ "ctl" : { "sockpath" : "/var/run/neoagent_ctl.sock", "logpath" : "/var/run/neoagent_ctl.log", }, "environments" : [ { "name" : "env1", "sockpath" : "/var/run/neoagent.sock", "target_server" : "127.0.0.1:11212", "backup_server" : "127.0.0.1:11213", (abbreviated) }, { "name" : "env2", "sockpath" : "/var/run/neoagent2.sock", "target_server" : "127.0.0.1:11214", "backup_server" : "127.0.0.1:11215", (abbreviated) }, ]}


“ctl” block

sockpath socket path for controlling neoagent

logpath log path for master process

"ctl" : { "sockpath" : "/var/run/neoagent_ctl.sock", "logpath" : "/var/run/neoagent_ctl.log",},

This block is for master13年9月20日金曜日

Configuration by JSON{ "ctl" : { "sockpath" : "/var/run/neoagent_ctl.sock", "logpath" : "/var/run/neoagent_ctl.log", }, "environments" : [ { "name" : "env1", "sockpath" : "/var/run/neoagent.sock", "target_server" : "127.0.0.1:11212", "backup_server" : "127.0.0.1:11213", (abbreviated) }, { "name" : "env2", "sockpath" : "/var/run/neoagent2.sock", "target_server" : "127.0.0.1:11214", "backup_server" : "127.0.0.1:11215", (abbreviated) }, ]}


“environments” block

✦ In neoagent, ✦ environment is the configuration for workers.

✦ Each entry in “environments” is the configuration for each worker.

✦ Master manages workers for each of their of environments.


“environments” block

name environment name

sockpath socket path for communicating neoagent’s worker

target_server primary memcached server

backup_server secondary memcached server

conn_max max connections(backlog size)

connpool_max connection pool size

etc...

document is herehttp://cubicdaiya.github.io/neoagent/


http://cubicdaiya.github.io/neoagent/

http://cubicdaiya.github.io/neoagent/

Muti-threaded


neoagent’s threads✦ Master-process

✦ sigwait-thread✦ ctl-thread

✦ Worker-process✦ sigwait-thread✦ event-threads✦ support-thread


Master’s threads

✦ sigwait-thread✦ waiting for a signal from ctl-

thread✦ ctl-thread

✦ receiving a instruction from neoctl through ctl-socket


Worker’s threads✦ sigwait-thread

✦ waiting for a signal from ctl-thread

✦ event-threads✦ processing client requests

✦ support-thread✦ health-checking & statictics


Event-driven


Event-threads’ architecture

✦ event-driven✦ non-blocking I/O✦ multi-threaded

✦ request-queuing & thread-pooling


Event processing phases


multithreaded with request-queuing & thread-pooling

event-thread

event-thread

event-thread

request1

request2

request3

empty

・・・

empty

dequeue

dequeue

dequeue

event-thread(accept)

request-queue

enqueue

processing requests


Usage at pixiv


Usage

✦ Building neoagent✦ Monitoring neoagent✦ Controlling neoagent


Building


Building neoagent


Building neoagent

with SCons！


SCons

✦ Simple & flexible & programable build tool✦ Auto anaylysis of dependencies✦ Build-configuration is written with Python

✦ Suitable for small or medium scale projects


Monitoring


Monitoring

✦ health-checking worker with Nagios & NRPE

✦ monitoring neostat & Munin


neostat$ neostat -s /var/run/neoagent_st.sock -o{ "datetime" : 2013-09-08 23:12:39 "version": "0.8.7", "host": "localhost", "port": "/var/run/neoagent_st.sock", "environment_name": "yapcasia2013", (abbreviated) "current_conn": 5, "available_conn": 20 (abbreviated)}


neostat -> Munin Graph

->

current connections

$ neostat -s /var/run/neoagent_st.sock -o{ "datetime" : 2013-09-08 23:12:39 "version": "0.8.7", "host": "localhost", "port": "/var/run/neoagent_st.sock", "environment_name": "yapcasia2013", (abbreviated) "current_conn": 5, "available_conn": 20 (abbreviated)}


neostat -> Munin Graph

->

neostat outputs current status of neoagent. (For example, current connection numbers)

current connections

$ neostat -s /var/run/neoagent_st.sock -o{ "datetime" : 2013-09-08 23:12:39 "version": "0.8.7", "host": "localhost", "port": "/var/run/neoagent_st.sock", "environment_name": "yapcasia2013", (abbreviated) "current_conn": 5, "available_conn": 20 (abbreviated)}


neostat internal

✦ neoagent workers have a suppport-thread.

✦ support-thread returns JSON as against request of neostat.


slowlog for neoagent■ configuration

■ output

"slow_query_sec" : 0.01, # milli-seconds"slow_query_log_path" : "/var/log/neoagent_slowlog.log","slow_query_log_format" : "json","slow_query_log_access_mask" : "0666",

{ ..., "na_to_ts": 0.000006, "na_from_ts": 0.000003, "na_to_client": 0.014320, "querytxt": "get key1" }{ ..., "na_to_ts": 0.000012, "na_from_ts": 0.000004, "na_to_client": 0.011587, "querytxt": "get key2" }{ ..., "na_to_ts": 0.000009, "na_from_ts": 0.000003, "na_to_client": 0.017788, "querytxt": "get key3" }{ ..., "na_to_ts": 0.000009, "na_from_ts": 0.000009, "na_to_client": 0.036082, "querytxt": "get key4" }{ ..., "na_to_ts": 0.000008, "na_from_ts": 0.000008, "na_to_client": 0.010506, "querytxt": "get key5" }{ ..., "na_to_ts": 0.000014, "na_from_ts": 0.000003, "na_to_client": 0.029464, "querytxt": "get key6" }{ ..., "na_to_ts": 0.000007, "na_from_ts": 0.000004, "na_to_client": 0.012440, "querytxt": "get key7" }{ ..., "na_to_ts": 0.000007, "na_from_ts": 0.000003, "na_to_client": 0.043834, "querytxt": "get key8" }{ ..., "na_to_ts": 0.000007, "na_from_ts": 0.050442, "na_to_client": 0.000015, "querytxt": "get key9" }


slowlog for neoagent

na_to_ts taken time for sending request to target server

na_from_ts taken time for receiving response from target server

na_to_client taken time for sending response to client

querytxt request body

{ ..., "na_to_ts": 0.000006, "na_from_ts": 0.000003, "na_to_client": 0.014320, "querytxt": "get key1" }{ ..., "na_to_ts": 0.000012, "na_from_ts": 0.000004, "na_to_client": 0.011587, "querytxt": "get key2" }{ ..., "na_to_ts": 0.000009, "na_from_ts": 0.000003, "na_to_client": 0.017788, "querytxt": "get key3" }{ ..., "na_to_ts": 0.000009, "na_from_ts": 0.000009, "na_to_client": 0.036082, "querytxt": "get key4" }{ ..., "na_to_ts": 0.000008, "na_from_ts": 0.000008, "na_to_client": 0.010506, "querytxt": "get key5" }{ ..., "na_to_ts": 0.000014, "na_from_ts": 0.000003, "na_to_client": 0.029464, "querytxt": "get key6" }{ ..., "na_to_ts": 0.000007, "na_from_ts": 0.000004, "na_to_client": 0.012440, "querytxt": "get key7" }{ ..., "na_to_ts": 0.000007, "na_from_ts": 0.000003, "na_to_client": 0.043834, "querytxt": "get key8" }{ ..., "na_to_ts": 0.000007, "na_from_ts": 0.050442, "na_to_client": 0.000015, "querytxt": "get key9" }



slowlog is sent to MongoDB by Fluentd！13年9月20日金曜日


Monitoring slow queries with Monitoring Web Application!


Controlling neoagent


Controlling with signals


Problem with signals

✦ complicated & easy to mistake✦ must send multiple signals to master

and workers at a time✦ must link worker’s PID to neoagent’s

environment


Controlling with neoctl


neoctl internal

✦ neoctl sends instructions to master

✦ by UNIX domain socket

✦ master manages worker’ PIDs

✦ and the link between each worker’s PID and environment name, too.

✦ master sends signals master-self and workers.


Graceful restart

neoctlneoagentmasterprocess

neoagentworker

env1

neoagentworker

env2

neoagentworker

env3


Graceful restart


graceful env1 neoagentworker

env1

neoagentworker

env2

neoagentworker

env3

through neoagent_ctl.sock


Graceful restart



env1

neoagentworker

env2

neoagentworker

env3


SIGUSR2


Graceful restart



env1

neoagentworker

env2

neoagentworker

env3


SIGUSR2

worker(env1) exits after processing remaining requests


Graceful restart



env1

neoagentworker

env2

neoagentworker

env3


SIGUSR2

SIGCONT


Graceful restart



env1

neoagentworker

env2

neoagentworker

env3


SIGUSR2

SIGCONT

neoagentworker

env1(new)

fork


Graceful restart


neoagentworker

env2

neoagentworker

env3

neoagentworker

env1(new)


Online version update


neoagentworker

env1

neoagentworker

env2

neoagentworker

env3




neoagentworker

env1

neoagentworker

env2

neoagentworker

env3

update





neoagentworker

env1

neoagentworker

env2

neoagentworker

env3

update


SIGUSR2




neoagentworker

env1

neoagentworker

env2

neoagentworker

env3

update


SIGUSR2

workers exits after processing remaining requests




neoagentworker

env1

neoagentworker

env2

neoagentworker

env3

update


SIGUSR2

SIGUSR1



neoctlneoagentmaster

process(old)

neoagentworker

env1(old)

neoagentworker

env2(old)

neoagentworker

env3(old)

update


SIGUSR2

SIGUSR1

neoagentworker

env1(new)

neoagentworker

env2(new)

neoagentworker

env3(new)

neoagentmaster

process(new)

exec

fork



neoctlneoagentworker

env1(old)

neoagentworker

env2(old)

neoagentworker

env3(old)

neoagentworker

env1(new)

neoagentworker

env2(new)

neoagentworker

env3(new)

neoagentmaster

process(new)



neoctl

neoagentworker

env1(new)

neoagentworker

env2(new)

neoagentworker

env3(new)

neoagentmaster

process(new)


neoctlneoagentmaster

process(new)

neoagentworker

env2(new)

neoagentworker

env3(new)

neoagentworker

env1(new)




with Capistrano！13年9月20日金曜日

Summary

✦ Why memcached protocol proxy server is required?

✦ reduced TCP connections

✦ enhance scalability of system

✦ Monitoring is important！

✦ Don’t guess！ Measure！

✦ Create tools for simplifying operation！


Thanks!


memcached proxy server development and operation

Technology

Transcript of memcached proxy server development and operation