Running A Realtime Stats Service On My Sql

Running a Realtime Stats Service on MySQL

Cybozu Labs, Inc. Kazuho Oku

Background

Apr. 23 2009 Running Realtime Stats Service on MySQL 2

Who am I?

 Name: Kazuho Oku (奥一穂)  Original Developer of Palmscape / Xiino

 The oldest web browser for Palm OS

 Working at Cybozu Labs since 2005  Research subsidiary of Cybozu, Inc.  Cybozu is a leading groupware vendor in Japan  My weblog: tinyurl.com/kazuho


Introduction of Pathtraq


What is Pathtraq?

 Started in Aug. 2007  Web ranking service

 One of Japan’s largest  ~10,000 users submit access information  ~1,000,000 access infomation per day

 like Alexa, but semi-realtime, and per-page


What is Pathtraq? (cont'd)

 Automated Social News Service  find what's hot  like Google News + Digg  calculate relevance from access stats

 Search by...  no filtering (all the Internet)  by category  by keyword  by URL (per-domain, etc.)


How to Provide Real-time Analysis?

 Data Set (as of Apr. 23 2009)  # of URLs: 147,748,546  # of total accesses: 413,272,527

 Sharding is not a good option  since we need to join the tables and aggregate

 prefix-search by URL, search by keyword, then join with access data table

 core tables should be stored on RAM  not on HDD, due to lots of random access


Our Decision was to...

 Keep URL and access stats on RAM  compression for size and speed

 Create a new message queue  Limit Pre-computation Load  Create our own cache, with locks

 to minimize database access

 Fulltext-search database on SSD


Our Servers

 Main Server  Opteron 2218 x2, 64GB Mem  MySQL, Apache

 Fulltext Search Server  Opteron 240EE, 2GB Mem, Intel SSD  MySQL (w. Tritonn/Senna)

 Helper Servers  for Content Analysis  for Screenshot Generation


The Long Tail of the Internet

y=C・x-0.44

# of URLs with 1/10 hits: x2.75


Compressing URLs


Compressing URLs

 The Challenges:  URLs are too short for gzip, etc.  URLs should be prefix-searchable in compressed

form  How to run like 'http://www.mysql.com/%' on a

compressed URL?

 The Answer:  Static PPM + Range Coder


Static PPM

 PPM: Prediction by Partial Matching  What is the next character after ".co"?

 The answer is "m"!

 PPM is used by 7-zip, etc.

 Static PPM is PPM with static probabilistic model  Many URLs (or English words) have common

patterns  Suitable for short texts (like URLs)


Range Coder

 A fast variant of arithmetic compression  similar to huffmann encoding, but better  If probability of next character being "m" was

75%, it will be encoded into 0.42 bit

 Compressed strings preserve the sort order of uncompressed form


Create Compression Functions

 Build prediction table from stored URLs  Implement range coder

 took an open-source impl. and optimized it  original impl. added some bits unnecessary at the tail  use SSE instructions for faster operation  coderepos.org/share/browser/lang/cplusplus/range_coder

 Link the coder and the table to create MySQL UDFs


Rewriting the Server Logic

 Change schema url varchar(255) not null # with unique index ↓ urlc varbinary(767) not null # with unique index

 Change prefix-search form url like 'http://example.com/%' ↓ url_compress('http://example.com/')<=urlc and

urlc<url_compress('http://example.com0')

Note: "0" is next character of '/' Apr. 23 2009 Running Realtime Stats Service on MySQL 18

Compression Ratio

 Compression ratio: 37%  Size of prediction table: 4MB

 Benchmark of the compression functions  compression: 40MB/sec. (570k URLs/sec.)  decompression: 19.3MB/sec. (280k URLs/sec.)  fast enough since searchable in compressed form

 Prefix-search became faster  shorter indexes lead to faster operation



Re InnoDB Compression

 URL Compression can coexist with InnoDB compression

 though we aren't using InnoDB compression on our production environment

Compression Table Size N/A 100% URL compression 57% InnoDB compression 50% using both 33%

Compressing the Stats Table

 Used to have two int columns: at, cnt  it was waste of space, since...

 most cnt values are very small numbers  most accesses to each URL occur on a short period (ex.

the day the blog entry was written)  at field should be part of the indexes


at (hours since epoch) cnt (# of hits)

330168 1

330169 2

330173 1

330197 1

Compressing the Stats Table (cont'd)

 Merge the rows into a sparse array  example on the prev. page becomes: (offset=330197),1,0(repeated 23 times),1,2,1

 Then compress the array  the example becomes a blob of 8 bytes  originally was 8 bytes x 4 rows with index

 And store the array in a single column  fewer rows lead to smaller table, faster access


Compressing the Stats Table (cont'd)

 Write MySQL UDFs to access the sparse array

 cnt_add(column,at,cnt) -- adds cnt on given index (at)  cnt_between(column,from,to) -- returns # of hits between given hours  and more...

 We use int[N] arrays for vectorized calc.  especially when creating access charts


Create a new Message Queue


Q4M

 A simple, reliable, fast message queue  runs as a pluggable storage engine of MySQL  GPL License; q4m.31tools.com  presented yesterday at MySQL Conference :-p

 slides at tinyurl.com/q4m2009

 Used for relaying messages between our servers


Limiting Pre-computation Load


Limit # of CPU-intensive Pre-computations

 Use cron & setlock  setlock is part of daemontools by djb

 setlock  serializes processes by using flock  -n option: use trylock; if locked, do nothing

# use only one CPU core for pre-computation */2 * * * * setlock –n /tmp/tasks.lock precompute_hot_entries 5 0 * * * setlock /tmp/tasks.lock precompute_yesterday_data


Limit # of Disk-intensive Pre-computations

 Divide pre-computation to blocks and sleep depending on the elapsed time

my $LOAD = 0.25;

while (true) { my $start = time(); precompute_block(); sleep(min(time - $start, 0) * (1 - $LOAD) / $LOAD); }


Creating our own Cache System


The Problem

 Query cache is flushed on table update  access stats can be (should be) cached for a

certain period

 Memcached has a thundering-herd problem  all clients try to read the database when a

cached-entry expires  critical for us since our queries does joins,

aggregations, and sort operations


Swifty and KeyedMutex

 Swifty is a mmap-based cache  cached data shared between processes  lock-free on read, flock on write  notifies a single client that the accessed entry is

going to expire within few seconds  notified client can start updating a cache entry

before it expires

 KeyedMutex  a daemon used to block multiple clients issuing

same SQL queries Apr. 23 2009 Running Realtime Stats Service on MySQL 31

Swifty and KeyedMutexd (cont'd)

 Source codes are available:  coderepos.org/share/browser/lang/c/swifty  coderepos.org/share/browser/lang/perl/Cache-Swifty  coderepos.org/share/browser/lang/perl/KeyedMutex


Fulltext-search on SSD


Senna / Tritonn

 Senna is a FTS engine popular in Japan  might not work well with European languages

 Tritonn is a replacement of MyISAM FTS  uses Senna as backend  faster than MyISAM FTS

 Wrote patches to support SSD  during our transition from RAM to SSD  patches accepted in Senna 1.1.4 / Tritonn 1.0.12


FTS: RAM-based vs. SSD-based

 Size of FTS data: ~ 20GB  Downgraded hardware to see if SSD-

based FTS is feasible  Speed became ¼

 but latency of searches are well below one second


Old Hardware New Hardware

CPU Opteron 2218 (2.6GHz) x2 Opteron 240 (1.4GHz)

Memory 32GB 2GB

Storage 7200rpm SATA HDD SSD (Intel X25-M)

Summary


Summary

 Use UDFs for optimization  Sometime it is easier to scale UP

 esp. when you can estimate your data growth

 Use SSD for FTS  Baidu (China's leading search engine) uses SSD

 Most of the things introduced are OSS  We plan to open-source our URL compression

table as well


We are Looking for...

 If you are interested in localizing Pathtraq to your country, please contact us  we do not have resources outside of Japan

 to translate the web interface  to ask people to install our browser extension  to follow local regulations, etc.


Thank you for listening

tinyurl.com/kazuho


Running A Realtime Stats Service On My Sql

Technology

Transcript of Running A Realtime Stats Service On My Sql