Fluentd Hacking Guide at RubyKaigi 2014

Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved.

Fluentd Hacking Guide (Fluentd ソースコード完全解説)

September 20th, 2014 !Naotoshi Seo @sonotsDeNA Co., Ltd.


2

Who am I?・Naotoshi Seo @sonots

・DeNA Co, Ltd.

・Infrastructure Engineer

・Fluentd Commiter


⁃ An application (or a framework) to process log streaming ⁃ Powerful plugin architecture (+250 plugins) ⁃ written in Ruby


Disclaimer

4

1. I do not explain what is Fluentd, how to use Fluentd 2. I assume audiences have ever used Fluentd, have ever

created Fluentd plugins, have interests inside Fluentd !

3. Source Codes are from v0.10 branch as of September 2014


Topics

5

1. The bootstrap sequence of Fluentd, and how Fluentd loads plugins

2. How an input plugin passes data to output plugins 3. How BufferedOutput plugin works 4. How Fluentd parses the config file 5. The event-driven programming using cool.io, and effects of

GVL


6

Let's Dive into Source Codes


7

vim


Topics

8


2. How an input plugin passes data to output plugins 3. How BufferedOutput plugin works


The Bootstrap Sequence

9

Fluent::Supervisor#run_configure 1. require 2. new 3. configure(conf ) Fluent::Supervisor#run_engine 4. start 5. shutdown (if signal received)

Input plugin creates threads on #startOutput plugin does nothing on #start (typically)


Topics

10




Dataflow from Input to Output

11

Input Engine Output

emit(tag, es)

emit(tag, es)

If an input thread receives data, call Engine.emit


CAUTION: Output Plugin

12

Input Engine Output

emit(tag, es)

emit(tag, es)

BLOCK!!!

Can not receive new input during blocking

ex) HTTP POST


13

Use BufferedOutput to avoid blocking


Topics

14




BufferedOutput Plugin

15

Input Engine BufferedOutput BasicBuffer

emit(tag,es)emit(tag,es)

emit(tag,data)enqueueimmediately return!!

Can receive new input. No blocking.


BufferedOutput Plugin

16

BufferedOutput BasicBuffer

try_flush(push)

OutputThread

pop

write(chunk)

do some EXPENSIVE things

Run heavy processing in OTHER threads


CAUTION: BufferedOutput

17

Stuck if enqueued over its capability !

HOW TO IMPROVE 1. Increase num_threads 2. Enlarge buffer_chunk_limit 3. Set smaller queued_chunk_flush_interval, try_flush_interval

(Secret parameters)


18

But,


19

Improving actual processing throughputs

is most important

fluent-plugin-elasticsearch gets stuck? Then, tune Elasticsearch!


Conclusion

20

1. Output plugin blocks 2. BufferedOutput does not block, but stuck if enqueued

over its capability • Tune with option parameters such as num_threads

3. In either case, improving actual processing capability itself is most important!!

4. I am happy if this talk helps your operation, your plugin development, and contributions

Fluentd Hacking Guide at RubyKaigi 2014

Engineering

Transcript of Fluentd Hacking Guide at RubyKaigi 2014