C10K on Mongo's sharding
-
Upload
hiroaki-kubota -
Category
Documents
-
view
2.624 -
download
6
description
Transcript of C10K on Mongo's sharding
![Page 1: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/1.jpg)
MongoDB shardingC10Kとの戦い
![Page 2: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/2.jpg)
-2 -
Shardingおさらい
![Page 3: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/3.jpg)
-3 -
Simple case
shard1
mongod
shard2
client host
httpd httpd httpd httpd
mongosmongos
httpd httpd httpd
server
client
![Page 4: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/4.jpg)
-4 -
Simple case
shard1
mongod
shard2
client host
httpd httpd httpd httpd
mongosmongos
httpd httpd httpd
server
client
Give me DATA_A !
![Page 5: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/5.jpg)
-5 -
Simple case
shard1
mongod
shard2
client host
httpd httpd httpd httpd
mongosmongos
httpd httpd httpd
server
client
Where is the DATA_A ?
Give me DATA_A !
![Page 6: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/6.jpg)
-6 -
Simple case
shard1
mongod
shard2
client host
httpd httpd httpd httpd
mongosmongos
httpd httpd httpd
server
client
Give me DATA_A !
Shard1 is saving it !
![Page 7: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/7.jpg)
-7 -
Simple case
shard1
mongod
shard2
client host
httpd httpd httpd httpd
mongosmongos
httpd httpd httpd
server
client
Give me DATA_A !
Give me DATA_A !
![Page 8: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/8.jpg)
-8 -
Simple case
shard1
mongod
shard2
client host
httpd httpd httpd httpd
mongosmongos
httpd httpd httpd
server
client
Give me DATA_A !
This is DATA_A.
![Page 9: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/9.jpg)
-9 -
Simple case
shard1
mongod
shard2
client host
httpd httpd httpd httpd
mongosmongos
httpd httpd httpd
server
clientThis is DATA_A.
![Page 10: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/10.jpg)
-10 -
Connection pool
![Page 11: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/11.jpg)
-11 -
Connection pool
shard1
mongod
shard2
client host
httpd httpd httpd httpd
mongosmongos
httpd httpd httpd
server
client
Used connection is pooled !
![Page 12: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/12.jpg)
-12 -
Connection pool
shard1
mongod
shard2
client host
httpd httpd httpd httpd
mongosmongos
httpd httpd httpd
server
client
Give me DATA_A !
Used connection is pooled !
![Page 13: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/13.jpg)
-13 -
Connection pool
shard1
mongod
shard2
client host
httpd httpd httpd httpd
mongosmongos
httpd httpd httpd
server
client
Give me DATA_A !
shard1のコネクションが再利用できそう
![Page 14: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/14.jpg)
-14 -
Connection pool
shard1
mongod
shard2
client host
httpd httpd httpd httpd
mongosmongos
httpd httpd httpd
server
client
Give me DATA_A !
Give me DATA_A !
![Page 15: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/15.jpg)
-15 -
Connection pool
shard1
mongod
shard2
client host
httpd httpd httpd httpd
mongosmongos
httpd httpd httpd
server
client
Give me DATA_A !
This is DATA_A.
![Page 16: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/16.jpg)
-16 -
Connection pool
shard1
mongod
shard2
client host
httpd httpd httpd httpd
mongosmongos
httpd httpd httpd
server
client
This is DATA_A.
![Page 17: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/17.jpg)
-17 -
New connection
![Page 18: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/18.jpg)
-18 -
Busy case
shard1
mongod
shard2
client host
httpd httpd httpd httpd
mongosmongos
httpd httpd httpd
server
client
Give me DATA_A !
Give me DATA_A !
![Page 19: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/19.jpg)
-19 -
Busy case
shard1
mongod
shard2
client host
httpd httpd httpd httpd
mongosmongos
httpd httpd httpd
server
client
Give me DATA_A !
shard1 へのコネクションは既にあるが 今は利用中で使えない!
なので新たに1本コネクションを張る
![Page 20: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/20.jpg)
-20 -
Busy case
shard1
mongod
shard2
client host
httpd httpd httpd httpd
mongosmongos
httpd httpd httpd
server
client
![Page 21: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/21.jpg)
-21 -
recapitulation
■ mongosの役割● mongsはクエリーをプロキシし問題を解決してくれる。● スター型接続になるのでC10K対策にもなる。● mongos - mongod(shard)間のコネクションをプールし性能向上● プールするコネクション数は最大同時クエリー数(+α)
α = mongodの生死チェックの為のコネクションも別途張る
![Page 22: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/22.jpg)
-22 -
recapitulation
■ mongosの役割● mongsはクエリーをプロキシし問題を解決してくれる。● スター型接続になるのでC10K対策にもなる。● mongos - mongod(shard)間のコネクションをプールし性能向上● プールするコネクション数は最大同時クエリー数(+α)
α = mongodの生死チェックの為のコネクションも別途張る
いい感じジャン!
![Page 23: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/23.jpg)
-23 -
戦いが始まる...
![Page 24: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/24.jpg)
-24 -
お客が増えたら?
![Page 25: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/25.jpg)
-25 -
There are alot of clients
shard1
mongod
shard2
client host
httpd httpd httpd httpd
mongosmongos
httpd httpd httpd
server
client
![Page 26: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/26.jpg)
-26 -
レプリカ無いと怖いよね...
![Page 27: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/27.jpg)
-27 -
Keep availability !
shard1
mongod mongod mongod
shard2
client host
httpd httpd httpd httpd
mongosmongos
httpd httpd httpd
server
client
![Page 28: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/28.jpg)
-28 -
やっぱshard足んね...
![Page 29: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/29.jpg)
-29 -
shard3 shard4 shard5
Enough ?
shard1
mongod mongod mongod
shard2
client host
httpd httpd httpd httpd
mongosmongos
httpd httpd httpd
server
client
![Page 30: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/30.jpg)
-30 -
How to operate shardingそもそもshardingを組むという事はそれなりの規模のデータやトラフィックを扱っている
のでC10K対策は避けては通れない。
■Shardingシステムは
接続数の爆発との戦い!!!■mongod最大コネクション数 = 20000
● 制限値を超える接続は拒否されるので一発レッド(サービス停止)● 一見充分な値だが現在のWEBでは簡単に超え得る– PHP on apache(MAX_CLIENT=500) × 4台 がフル稼働で超える計算– 実際は5000クライアント程度でも簡単に超える。※後述
● 最近のLinuxのTCPバッファデフォルトサイズから考える read (rmem) = 16k , write (wmem) = 85k
101k × 20000 = 2G強妥当な設計ではあるが安心はできない
![Page 31: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/31.jpg)
-31 -
コネクション数爆発
![Page 32: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/32.jpg)
-32 -
Connection inflation
shard1
mongod
shard2
client host
httpd
mongosmongos
server
client
Give me DATA_A !
![Page 33: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/33.jpg)
-33 -
shard1
mongod
shard2
client host
httpd
mongosmongos
server
client
Give me DATA_A !
Connection inflation
Give me DATA_A !
![Page 34: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/34.jpg)
-34 -
shard1
mongod
shard2
client host
httpd
mongosmongos
server
client
Time out (TT;
Give me DATA_A !
Connection inflation
![Page 35: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/35.jpg)
-35 -
shard1
mongod
shard2
client host
httpd
mongosmongos
server
client
Give me DATA_A !!!
Give me DATA_A !
前回使ったshard1へのコネクションはまだ使えない なので新たに1本コネクションを張る必要がある
Connection inflation
![Page 36: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/36.jpg)
-36 -
shard1
mongod
shard2
client host
httpd
mongosmongos
server
client
Give me DATA_A !Give me DATA_A !
Connection inflation
Give me DATA_A !!!
![Page 37: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/37.jpg)
-37 -
shard1
mongod
shard2
client host
httpd
mongosmongos
server
client
Give me DATA_A !
Connection inflation
Time out (TT;
Give me DATA_A !
![Page 38: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/38.jpg)
-38 -
Connection inflation
shard1
mongod
shard2
client host
httpd
mongosmongos
server
client
Please give me DATA_A !!!!!!!
Give me DATA_A !Give me DATA_A !Give me DATA_A !
リトライ3回なら
3コネクション消費する
勿論mongodに掛ける負荷も3倍
![Page 39: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/39.jpg)
-39 -
Connection数が爆発1クライアントでもmongos-mongod間のコネクション数を大量に消費するケースがある。
どの分散システムにも言えるし接続数に限った話でもないが、
![Page 40: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/40.jpg)
-40 -
Connection数が爆発1クライアントでもmongos-mongod間のコネクション数を大量に消費するケースがある。
どの分散システムにも言えるし接続数に限った話でもないが、
エラー時のリトライ禁止! ちょっぴり負荷が掛かり始めたシステムを
みんなで袋叩きして殺してしまう・・・
リトライして解決なら最初からクライアントが対応してるって。。。
![Page 41: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/41.jpg)
-41 -
How to operate sharding■ 危険なケース
● ピーク時間帯に断続的にslow queryが起きている● クライアントが多く通常時のコネクション数がギリギリだ● リトライしている
これらのケースでは障害に至る可能性が特に高い
また明示的なリトライをせずとも、client - mongos間でTIMEOUTしたクエリーで利用したmongos - mongod間コネクションは即座に再利用できない。
接続数を完全に見積もり切る事は困難
![Page 42: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/42.jpg)
-42 -
How to operate sharding■ mongs - mongod コネクションプールの問題
● mongos - mongod間の接続数を制御する手段が無い--maxConnsオプションはclient - mongos間のコネクションを扱っている
● mongosは定期的に不必要なコネクションプールを回収するが 30分以上使われなかったコネクション限定(hardcode)
瞬間負荷で張ったコネクションを長時間保持し続けてしまう
結局、本来必要ない接続を張ってしまう。
最大接続数20000という厳しい制限の元この手の無駄はシステムのスケーラビリティを大きく下げる!!
![Page 43: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/43.jpg)
-43 -
How to operate sharding
■ 接続数不足の対策● 不必要なクライアントを排除
安易にapache MAX_CLIENT=1000とかやらない!(結局普通のアプリはそんなに食えない)
通常時mongodの接続数が7000位が安全ギリギリ
● リトライ処理をやめる システム保全の方が大事!そもそもリトライが成功する確率も低い。
● slow query抑止のチューニング– table scan禁止 (--notablescan)
– M/R 禁止– ピーク時の書き込みを抑止(各種ファイルpreallocateが一番怖い)– shard増やす– etc...
![Page 44: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/44.jpg)
-44 -
それでもダメなら、、、
奥の手だ!
![Page 45: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/45.jpg)
-45 -
hack
![Page 46: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/46.jpg)
-46 -
![Page 47: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/47.jpg)
-47 -
![Page 48: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/48.jpg)
-48 -
パッチ解説■ 不要コネクションを回収するタイミングを修正
● 1800秒固定→オプションで指定可能● コネクションプール数の最大値を指定可能 必要なコネクションは張るが、指定値以上はプールしなくなる
最小値も指定出来た方が良いんだろうか? とりあえず今のところ必要なさそう
![Page 49: C10K on Mongo's sharding](https://reader033.fdocument.pub/reader033/viewer/2022061218/54b7a1784a795993718b470a/html5/thumbnails/49.jpg)
-49 -
補足
■パッチ
https://github.com/crumbjp/mongo/commit/b4a424fe02d7b5410db364b2d71f11ce09fd3fc3
■C10K と スター型接続
http://www.slideshare.net/crumbjp/cockatoo(39p~)