[2B4]Live Broadcasting 추천시스템

Live Broadcasting

| 2014.9.30

() TV .

TV?

BJ

TV?

SNS

TV0255075100

100 K 3 M 20 M 20 M 120

TV

Touch the void

?

AB

!Watcha

http://watch.net

Touching the void

: http://www.elhablador.com/blog/wp-content/uploads/2012/05/touching-the-void.jpg

http://www.elhablador.com/blog/wp-content/uploads/2012/05/touching-the-void.jpg

1988

!

Touching the void

!!!

10 Into Thin Air !!!

!

Touching the

void

: www.amazon.com

http://www.amazon.com

0

25

50

75

100

0

250

500

750

1000

57% ! !

.

. !

!

! , !

.

BJ BJ `

BJ

BJ

1.0 () 0.0

[0.0 , 1.0]

0.0

[0.0 , 1.0]

uno !

BJID

10001 rani0224 10 9

10001 afkbo1 120 95

10001 afmlb1 32 93

uno !

BJID Y/N

10001 rani0224 T

10001 afkbo1 T

uno !

BJID

10001 rani0224 50

10001 afkbo1 10

uno !

BJID

10001 rani0224 0.57

10001 afkbo1 0.73

10001 afmlb1 0.21

uno !

BJID Y/N

10001 rani0224 1.0

10001 afkbo1 1.0

uno !

BJID

10001 rani0224 0.5

10001 afkbo1 0.1

Boolean Value !

[ F, T ] [ 0 , 1]

Scale !

[0 , 100] [0.0 1.0]

1.0 ()

0.0

[0.0 , 1.0]

0

7.5

15

22.5

30

D D -2 D -4 D -6 D -8 D -10 D -12 D -14 D -16 D -18 D -20 D -22 D -24 D -26 D -28

1.0 ()

0.0

[0.0 , 1.0]

0

7.5

15

22.5

30

D D -2 D -4 D -6 D -8 D -10 D -12 D -14 D -16 D -18 D -20 D -22 D -24 D -26 D -28

1.0 ()

[ Tips. ] = () = D/1 + D/2 + D/3 D/30 = ( () = ( log() )

0.0

[0.0 , 1.0]

0

0.125

0.25

0.375

0.5

0.1

0.20.2

0.5

= 1.0

1.0 ()

0.0

0.623573 0.57521 0.34565 0.21736

1.0 ()

?

0.0

0.623573 0.57521 0.34565 0.21736

1.0 ()

HDFS HQL

Redis Cluster

Sorted Set

Hive External Table

HQL

BJ HQL ( )

Redis Save (UDF) HQL

HQL

HQL

HQL

HQL

0.0

0.623573 0.57521 0.34565 0.21736

1.0 ()

HDFS HQL

Redis Cluster

Sorted Set

HIVE External Table

HQL


HQL

HQL

HQL

HQL

MR-Hive

Compatible

BJ HQL ( )

0.0

0.623573 0.57521 0.34565 0.21736

1.0 ()

HDFS HQL

Redis Cluster

Sorted Set

HIVE External Table

HQL

Merge HQL


HQL

HQL

HQL

HQL

MR-Hive

Compatible

Query Base Processing

0.0

0.623573 0.57521 0.34565 0.21736

1.0 ()

HDFS HQL

Redis Cluster

Sorted Set

HIVE External Table

HQL

Merge HQL


HQL

HQL

HQL

HQL

MR-Hive

Compatible

Query Base ProcessingIn

Memory Store

1.0

0.0

Clustering BJ Matrix

BJ

BJ

1.0

0.0


n=bj, m=cluster

:

Clustering

Graph Clustering

BJ Matrix

Clustering BJ Similarity

BJ

0.0

!! ( )

? !

Euclidean Distance Cosine Similarity Jaccard coefficient

manhattan distance, pearson

Jaccard : !

Euclidean Distance!

1

2

Jaccard, Euclidean, Cosine

Cosine Similarity!

1 2

Euclidean VS Cosine

! ! ! ! ! ! !! 191 ! 173 !79! ! 46! ! 124!! 129! !56! ! 34! ! 91! ! 59! ! 21! ! 24

?

Cosine Similarity - = 0.96966224 - = 0.98190527 ( WIN )

Euclidean Distance - = 85 ( Win ) - = 163

?

Cosine Similarity ! ?!Cosine Item !

Euclidean Similarity! ?!Euclidean Item !

Eucliean ?

Euclidean

Cosine

.

!

TV ?

BJ BJ ?

BJBJ

BJ

Euclidean distance : BJ

Cosine Similarity : BJ


TV !

Euclidean Distance Cosine Similarity Jaccard coefficient

manhattan distance, pearson


BJ


BJ1 BJ2 BJ3 BJ4 BJ5 BJ6 BJ7 BJ8 BJ9 BJ10BJ1 0.12 0.1009 0.1582BJ2 0.12 0.0008 0.029 0.0572 0.290BJ3 0.12 0.1391BJ4 0.0854 0.12BJ5 0.12 0.1391 0.0854 0.12 0.0854BJ6 0.290 0.290BJ7 0.12 0.1391 0.0854 0.12BJ8 0.290 0.0854 0.31BJ9 0.0854 0.0854BJ10 0.290

BJ Matrix ( Cosine )


?

HDFS Redis Cluster

Sorted Set

hdfs files

Data MR

BJ MR


BJ MR

BJ Matrix MySQL

White / Black List /

HDFS Redis Cluster

Sorted Set

hdfs files

Data MR

BJ MR


BJ MR

BJ Matrix MySQL


Hourly Batch Job

HDFS Redis Cluster

Sorted Set

hdfs files

Data MR

BJ MR


BJ MR

BJ Matrix MySQL


?


TV BJ

, .

!

?? BJ ?


Cowatching Counts

In/Out Stream Counting

70

45

23

5

89

14

2


Cowatching Counts

Message Queue

Redis Writer

Redis Writer

Redis Writer

Redis Cluster

Sorted Set

Aggregation Set

Async Processing

In-Memory


Message Queue

Redis Writer

Redis Writer

Redis Writer

Redis Cluster

Sorted Set

Async Processing

Cowatching Counts


Cowatching Counts

Message Queue

Redis Writer

Redis Writer

Redis Writer

Redis Cluster

Sorted Set

Async Processing

Aggreg

ation

Write


Cowatching Counts

Message Queue

Redis Writer

Redis Writer

Redis Writer

Redis Cluster

Sorted Set

Async Processing

Aggreg

ation

Write In

Memory Store


Cowatching Counts

Redis SortedSet

instream:{bjid}

outstream:{bjid}

BJ BJ Matrix .

0.0

0.623573 0.57521 0.34565 0.21736

1.0

BJ = X

0.0

0.623573 0.57521 0.34565 0.21736

1.0

1.0

0.0

BJ = X BJ = Y

BJ = X BJ = Y ??

0.0

0.623573 0.57521 0.34565 0.21736

1.0

1.0

0.0

0.0

0.623573 0.57521 0.34565 0.21736

(1.0,

1.0)

0.0

(0.62, 0.56)

(0.62, 0.41)

(0.62, 0.12

(0.62, 0.03)

(0.57, 0.74)

(0.57, 0.59)

(0.57, 0.48)

(0.57, 0.43)

(0.57, 0.17)

(0.34, 0.72)

(0.34, 0.50)

(0.34, 0.24)

(0.34, 0.09)

(0.21, 0.61)

(0.21, 0.48)

(0.21, 0.01)

0.0

0.623573 0.57521 0.34565 0.21736

0.0

(0.62, 0.56)

(0.62, 0.41)

(0.62, 0.12

(0.62, 0.03)

(0.57, 0.74)

(0.57, 0.59)

(0.57, 0.48)

(0.57, 0.43)

(0.57, 0.17)

(0.34, 0.72)

(0.34, 0.50)

(0.34, 0.24)

(0.34, 0.09)

(0.21, 0.61)

(0.21, 0.48)

(0.21, 0.01)

?

(1.0, 1.0)

0.0

0.623573 0.57521 0.34565 0.21736

0.0

(0.62, 0.56)

(0.62, 0.41)

(0.62, 0.12

(0.62, 0.03)

(0.57, 0.74)

(0.57, 0.59)

(0.57, 0.48)

(0.57, 0.43)

(0.57, 0.17)

(0.34, 0.72)

(0.34, 0.50)

(0.34, 0.24)

(0.34, 0.09)

(0.21, 0.61)

(0.21, 0.48)

(0.21, 0.01)

(1.0, 1.0)

0.58

0.700.95

0.500.72

0.590.67

0.0

0.623573 0.57521 0.34565 0.21736

0.0

(0.62, 0.56)

(0.62, 0.41)

(0.62, 0.12

(0.62, 0.03)

(0.57, 0.74)

(0.57, 0.48)

(0.57, 0.43)

(0.57, 0.17)

(0.34, 0.72)

(0.34, 0.50)

(0.34, 0.24)

(0.34, 0.09)

(0.21, 0.61)

(0.21, 0.48)

(0.21, 0.01)

(1.0, 1.0)

0.58

0.700.95

0.500.72

0.590.67(0.57, 0.59)

0.0

0.623573 0.57521 0.34565 0.21736

1.0

1.0

0.0

!

!

!

!

!

!

??

Offline? !

(0.5) .

!

Offline = *0.5

0.0

0.623573 0.57521 0.34565 0.21736

1.0

1.0

0.0

!

!

X X

Offline

!

() IRF(Inverse Relation Frequency)

!

BJ Ranking IFR(Inverse Frequency Rank)

24 6 67 12 23

0.298 0.201 0.14 0.11 0.0056

Inverse Relation Frequency

0.0335 0.0124 0.0091 0.002 0.00024

/ Theme Rank

Theme Rank

AB Test

AB Test

: 2.3%

165%

: 6.1%

165%

AB Test

: 5.0%

22%

: 6.1% 22%

? !

//

Data Stream

Data Routing

Cache Store

Batch Process

Realtime Process

BJ

Sys

Stat

Hadoop

HQL Jobs

MapReduce

Elastic Search

Workflow Jobs

Message Q

ueue Architecture

Shard Store

None Shard Store

Zookeeper Coordinator

Data Stream

Data Routing

Cache Store

Batch Process

Realtime Process

API

Hadoop

HQL Jobs

MapReduce

Elastic Search

Workflow Jobs

Redis Cluster

MySQL


UDP

SenderFile/M

ysql Tailer

Netw

ork Mirroring

ZeroMQ

Thrift

Data Stream

Data Routing

Cache Store

Batch Process

Realtime Process

API

Hadoop

HQL Jobs

MapReduce

Elastic Search

Workflow Jobs

Redis Cluster

MySQL


UDP

SenderFile/M

ysql Tailer

Netw

ork Mirroring

ZeroMQ

Thrift

( )

( , )

Data Stream

Data Routing

Cache Store

Batch Process

Realtime Process

API

Hadoop

HQL Jobs

MapReduce

Elastic Search

Workflow Jobs

Redis Cluster

MySQL


UDP

SenderFile/M

ysql Tailer

Netw

ork Mirroring

ZeroMQ

Thrift

-Dynamic Data Pipeline // - Workflow scale-out - Async Data

- Broker - ( )

Data Stream

Data Routing

Cache Store

Batch Process

Realtime Process

API

Hadoop

HQL Jobs

MapReduce

Elastic Search

Workflow Jobs

Redis Cluster

MySQL


UDP

SenderFile/M

ysql Tailer

Netw

ork Mirroring

ZeroMQ

Thrift

Data Stream

Data Routing

Cache Store

Batch Process

Realtime Process

API

Hadoop

HQL Jobs

MapReduce

Elastic Search

Workflow Jobs

Redis Cluster

MySQL


UDP

SenderFile/M

ysql Tailer

Netw

ork Mirroring

ZeroMQ

Thrift

Shard/None Shard Store Topology

/

Data Stream

Data Routing

Cache Store

Batch Process

Realtime Process

API

Hadoop

HQL Jobs

MapReduce

Elastic Search

Workflow Jobs

Redis Cluster

MySQL


UDP

SenderFile/M

ysql Tailer

?

Data Stream

Data Routing

Cache Store

Batch Process

Realtime Process

BJ

Hadoop

HQL

MapReduce

Theme Ranks (Elastic Search)

Cowatching Counts (Workflow Jobs)

Message Q

ueue Architecture

()

BJ Matrix ()


Sys

Stat

Theme Ranks

Data Stream

Data Routing

Cache Store

Batch Process

Realtime Process

BJ

Hadoop

HQL Jobs

MapReduce

Elastic Search

Workflow Jobs

Message Q

ueue Architecture

Shard Store

None Shard Store


Sys

Stat

Data Stream

Data Routing

Cache Store

Realtime Process

BJ

Elastic Search !

Message Q

ueue Architecture

None Shard Store


Gender/Age

?

// .

.

.

/ .

I know what you want. : http://blog.nemesys.co.kr/tt/entry/LEAP-Motion-%EB%A7%88%EC%9D%B4%EB%84%88%EB%A6%AC%ED%8B%B0-%EB%A6%AC%ED%8F%AC%ED%8A%B8%EC%9D%98-%EA%B7%B8%EB%82%A0%EC

%9D%B4-%EC%9D%B4%EC%A0%9C-%EB%A9%80%EC%A7%80-%EC%95%8A%EC%95%98%EB%8B%A4?category=0

. : http://blog.nemesys.co.kr/tt/entry/LEAP-Motion-%EB%A7%88%EC%9D%B4%EB%84%88%EB%A6%AC%ED%8B%B0-%EB%A6%AC%ED%8F%AC%ED%8A%B8%EC%9D%98-

%EA%B7%B8%EB%82%A0%EC%9D%B4-%EC%9D%B4%EC%A0%9C-%EB%A9%80%EC%A7%80-%EC%95%8A%EC%95%98%EB%8B%A4?category=0

: http://mary-ann.tistory.com/610 :

[2B4]Live Broadcasting 추천시스템

Technology

Transcript of [2B4]Live Broadcasting 추천시스템