Fluentd in Co-Work

Fluentdin

Fluentd meetup in Fukuoka2013/03/07@Spring_MT

Profiletwitter : @Spring_MTCompany : 10xLabEngineer

Agenda

Fluentdin Co-Work appin Co-Work infra

SNS for Companies (like Yammer)During Closed β

InfraArchitecture

10xlab InfraALL AWS!Use VPCRuby, Fluentd and Chef in all serversgem install fluentd

public

subnetCo-Work

subnetutility

APP

redis

worker

subnetstorage

subnetfluentd

deliver

worker

proxy

varnish+

sinatra

f f f

f f

f f

Watch

subnetnetwork

Gateway NATSSLの変換

f

f

f

f

f

f

f

f fluentd

DB(MySQL on EC2)

Co-WorkWAFRuby on Rails

DBMySQL on EC2

KVSredis

Fluentdin

Co-Work App

App

QueuingLogging

Queuing

QueuingResqueRuby based job queue liblary using githubMultiprocessingBackend is redis

https://github.com/defunkt/resque



fluent-plugin-resque

Fluentd output plugin for sending resque-queue to redis

https://github.com/hotchpotch/fluent-plugin-resque



https://github.com/hotchpotch








Why Fluentd?

app

app

app

workerworkerworkerworker



queueworker

queueworker

resque

resque

app

worker

worker

worker

worker X

You write a retry processing?

If queing fail, App should retry queuing.

app

app

appworkerworkerworkerworker



queueworker

queueworker

fl u ent-resque

resque

fl u ent-logger

appworkerworkerworkerworker X X

If queuing fail, Fluentd retry queuing !Fluentd(and Logger)Buffering internallyRetrying

Send Queue Data to Fluentd

Send Resque-Queue to Redis

BreaK

http://photo.v-colors.com/2111.html



Logging

LoggingAccess log(nginx)PV log(rails)Ref. WEB†DB PRESS vol.

App Log(rails)

Why Fluentd?

Log FileTransferForwarding server is full m9(^Д^)ﾌﾟｷﾞｬｰ

Operate transfer script?

Real-Time analysis ?

Delete log file

Log FileFormat and AnalysisDisjointed format (CSV? TSV? LTSV? JSON?)

Operate analysis script?

FluentdUnified Format(JSON)Real-Time transfer and High AvailabilityEasy Setup and Easy extensionWell Tested At Big services(○HN, ○ookpad)

Just Do It !

http://d.hatena.ne.jp/tagomoris/20121029/1351491111



Just Do It !


今でしょ



deliver

Web Server

Web Server

Web Server

Fluentd(

Fluentd(

Fluentd(

Deriver&Fluentd&Server�

Deriver&Fluentd&Server&

(standby)�

Worker(Fluentd(Server�



Archive(Fluentd(Server�

Worker(Fluentd(Server� Resque�

File�

out_forward�

out_forward�

in_tail�

out_forward�

proxyaccess log

in_tail

appPV log

in_tail

app log logger

GrowthForecast

S3

Analysis

copyout_route

fluent-plugin-rewrite-tag-filter

analysisfluent-plugin-datacounter

fluent-plugin-numeric-counter

fluent-plugin-numeric-monitor

fluent-plugin-amplifier-filter

Output

GrowthForecastfluent-plugin-growthforecast

For visualize data

GrowthForecast

Web API for drawing graph

Perl

Treasure Data

TreasureData

Hadoop-based data store and analysis service

fluent-plugin-td

For data analysis (mainly access log)

S3

S3

Data Store Service in AWS

Eleven Nine

fluent-plugin-s3

For backup logs

ikachan

ikachan

Web API for IRC

Perl

fluent-plugin-ikachan

IRC notification(Ex.fatal log)

Analysis in TDSQL Like500 GB ($299/M)Analysis using Ruby!

CollectLib

Fluentdin

Co-Work Infra

WatchFor resource watchFluentd collect data and push data each notification tools.

fluent-plugin-mysqlslowquery

Collect mysql slow query logNotify IRC through fluent-plugin-ikachan + ikachan

fluent-resque-stat

Collect Resque statuspending キューの残数

processed 取り出したキューの総数

queues 登録されてるキューの名前の総数

workers キューを取り出すworkerの数

working キューを処理中のworker数

failed 失敗したキュー数

servers resqueが動いてるサーバーのhostやportの情報

environmentresque_statが動いているサーバーの RAILS_ENV or

RACK_ENV(なければdevelopmentが入る)

fluent-resque-stat

Thank You !

FluentdSetup

Install and SetupInstallattion of Fluentd is AWS cloud-init(user_data)Setting Fluentd is chef-sole and capistrano

I Extend a little above system for specifying cookbook.$ cap -S subnet=fluentd -S hosts=xfdeliver001 chef

Launch and Monitoring Process

supervisor

Fluentd in Co-Work

Documents

Transcript of Fluentd in Co-Work