Problem Solusion See what happens!cuckoo.js.ila.titech.ac.jp/~yamagen/JADH2018-slide01.pdf · A...
Transcript of Problem Solusion See what happens!cuckoo.js.ila.titech.ac.jp/~yamagen/JADH2018-slide01.pdf · A...
A Study on the Distribution of Cooccurrence Weight Patterns ofClassical Japanese PoetryHilofumi Yamamoto
Tokyo Institute of TechnologyBor HodoscekOsaka University
← more generic words more specific words →
Tall HeadTall HeadWord distributionby frequency-rank
Long Tail DinosaurLong Tail Dinosaur
x
σ0−σ
y
1
Function Tail Content Tail
C.F.Gauss(1777–1855)
ProblemProblemHairball Effect!!!
SolusionSolusionCooccurrence Weight⇒ Bell curve!
ring!ring!
It allows us to divide words into3 kinds:⇒ high, middle, and low.
Cooccurrence weight word patternslike two person’s weights
See what happens!See what happens!Content TailContent TailOver 1σ ⇒
cherry blossom (28/131,4.28): CT ctf.>0.00;non-dist=off; idf=off; pruned under U:1:
here way home1village.11
lodging1
scatter.1
1
forget1
confuse
1
do.1
1
(adv.neg)
1
(must)
1
while
2
�
2
�
1
11
1
1
1
1
1
1
2 21
1
1
1
1
1
1
1
22 1
1
1
1
1
1
1
2 21
1
21
104
2
8
15
8
perplexed
1
(adv.will)4
forcibly
1
�9
�6
even
3
see.310�11
��
2
get used
1
know
1say
2
(p.wish)
6
�7
bloom
1
ç�¡ã�˙5
(adv.would)
1
ã�˛
4break off
1
�
1
�
2
ã�`ã��
1
increase
1
love.1
1
will not
1
kurabu.pl
1
�2
cloud.vi
1
such as.1
1
��2
wait2
�8�3
borrow1
regrettable1
here we go1
like.2
1
blow
2
�佯�
1
regret
1
bear1
think
1
fast/early1
ã�˙ 1
(ku)
3
lovely
1
sorrowful.1
11
1
1
1
22 1
1
1
1
2
3
1
1
1
1
1
1
1
1 1
1
6
3 2
2
1
1
2
(aux.past)
1(aux.perf)1
2
1
8
4
118
41
2 13
1
1
2
113
3
6 1
1
33
(suffix.forbid)
11
painful.1
1high.1
2
(emph)
1
wish
1
1
(interj.)
2
Yoshino.PN1seemed
1
3
2
1
151
quiet
2
although
1
21
1
2
22 1
2 1
1
15
71
7
7
4
��
1
seem
1
1
5
2
1
1
11
5
14
2
ã�ˆã�˝
2
2
11
6
2
1
11
1
7
7
2
2
3
2
2
2
1 2
2
1
1
2
233
1
1
42
2
1
2
1
41
1
either way
1 till
1
person
1
road1
stop.vi.21
go
1
1
1
1
1
2
2
1
1
11
1
1
2
1
1
2 111
1
1
11 lodge.n
1
be.3
1
take1
1
1
1
1
4
2
727 8
2
1
1
1
1
6
1
3
34
73
mountain village
1
1
3
1
2
mind2
wind.n1
mountain2
desolate
1
come1
2
be lonely
1
prosper
1
1
1
1
1
stone
1
waterfall
1
hand
11
run
1
1
stand.vi
11
cross
1
go over
1
send.v2 word
1
hear
1 1
1
present
1
tell
1
2
add 1
get tired.1
12
1
name
1
1vain
1
rare.11
1
1
1
fall
1
1spring rain
1
tear
1
1
1
1
1
1
2
1
1
1
1
1
2
1
2
2
1
1
exchange
1
11
1
1
1
1
1
1
1
2
1112
1
1
2
1
2
2
11
1
see.1
1
1
1
1
1
12
65
4
1
3
2
4
1
1
1
3
1
2
1
1
1
1
1
2
4 8
1
45
2
3
2
1
2
1
1
1
2
1
12
1
1
13
2
3
1
12
1
what1
1
2
thing.2
1
between
1
1spring haze1
hide.vt1
1
1
1
1
1
2
1
11
1
11
1
21
1
24
11
2
11
1
12
1
2
1
myself1
2
1
number
1
only
1
1
1 1
1
1
11
1
1
1
2
1
2 1
1
11
2
1
12
3
11
11
2
1
2
1 1
1
1
1
1
11 1
1
11
111 1
1
snow
1
11
1
such as.2
1
11
1
1
1
11
1
12
1
1
1
2
1
23
1
2
1
1
1
1
trail2
1
change.1
1
transfer.1
11
1
2
1
1
1
1
71
1
7
56
62
2
1
10
1
3
5
23
11
3
1
trust.2
1
1
1
1
2
1
1 11
3
1
1
1
4
2
1
2
51
1
11
1
1 11
11
1
1
1
2
1
1
2
12
1 1
1
thing.1
11
1
spring
1
this year1
��
1
begin1
11
11
1
1
1
21
11
1
1
1
1
1
111
1
1
21
1
year
1
1
11
1
1
1
1
1
111
11
1
11
1
2
1
111
1
1
1
1
1
1
1
white cloud
1
foot
1
pull
1
1
1
1
1
11
1
1
11
1
1
11
1
1
1
1
1
1
11 1
2
1
1
11
11
1
2
2
1
22
1
1
2
1
4
1
2
1
1
1
1
1
3
21
1
2
1
out
1
12
1
111
after
11
11
11112
11
1
1
1
11
1
1
11
2
111
11
1
1
2 225
1
1
11
2
12
1
1
1 1
1
1
1
1
11
1
1
5 25
9
3
2
1
1
34
14
1
1
1
1
1
1
1
1
3
1
1
1
1
1
211
1
31
2
1
1
1
1
mi
221
1
1 1
121
1
1
1
day
111
2
1
cloud.1
1
3
1
1
1
1
1
1
1
1
1
3
1
11
1
1
11
11
1
1
1
1
1
1
1
1
1
13122
1 11
211
1 12
1
11
1
1 12
1
1
11
1
1
1
112
1 11
1
1
11
1
1
1
11
1
1
1
1
413
3
1
1
1 132
1
2
2
1
2
1
1
1
1
21
1 1
1
1
1
11
11
1
1
1
3
2
233
3
11
2
1
2
12
1
1
1
1
1
1 1
1
wave
1
sky
1
11
1
11
1
2
1
1
1
1
11
11
1
1
1
1
1
1
1
1 415
9
1
2
2
2
3
43
121
1
1
1
1
2
1
1
1
1
1
32
1 11
1
1
11
1
1
1
2
1
1
1 1
1
1
1
1
1
2
1
1
1
11
1
1
1
1
1
1
1
1
6
1
32
33
22
1
5
3
12
2
1
1
1
1
1
11
2
1
1
11
1
1
1
2
1
1
24
2
4
421
2
1
1
1
11
1
1
1
1
1
11
1
111
1
1
1
1
1
11
13
1 1
1
1
11
1
1
1
11
11
1
111
1
1
1
1
11
1
1
1
1
11
1
1
21
1
11
1
1
1
1 11
1
22
1
12
1 1
1
1
1
1
1
1
1
for
11
1
1 11
1
1
1
1
1
11 1
1
1
11
11
1
1
1
1
11 1
1
1
1 1
111
1
1
1
1 1
1
1
131 1
1
2
11 11
1
1
11
1
1
1
1
1
1
11
11
1
1
1
1
1
1 1
11
remnant
112
1
1
1
1
1
water1
1
1
11
1 121
1
11
1
1
1
11
1121
1
1
1
1
11 11
2 1
1
11
1
1
11
41
1
11
1
1
2
1
11
1
1
1
1
112
1
11
1
11
1
1
1
1
1
1
1
1
5
4
1
22
3
11
1
1
11
1
1
1
1
1
111
1
1
1
1
11
1
1
1
2
mistake
1
21
11
1
112
1
11
11
1
112
1
1
11
111 2
1
11
1
1 2
1
11
1
111
1
12
1
11
1
111
112
1
1
1 1
1
gorge12
1
1
1
1
1
1
1
1
1
12
1
1
1
1
1
1
1
12
1
1
1
1
1
121
1
1
1
every 1
3
1
12
11
1
1
1 1
1
1
3
1
21
1
1
1
1
31
2
1
1
1
11
old age
1
1
1
1
11
1
11
11
1
1
1
1
11
11
1
11
11
1
1
1
1
131
21
1
211
11
1
1
12
1
1
11
1
1
1
1
1 1
11
1
11
1
1
2
2
2
31
1
1
1
1
1
2
1
2
1
2
1
1
11
1
111
1
2
1
2
2
1
1
1
1
11
11
1
1
21 1
1
1
1
1
2
1
1
1
1
11
1
1
today2
1
1
21
1
11
1
1
1
a glance
1
you.3
1 12
1
1
2
1
1
1
1
1
1
1
11
21
1 1
1
1
1
33
1
1
11
11
11
1
1
1
1
1
34
1
1
2
2
2
1
1
mountain area
11
1
1
1
1
1
1
1
1
11
1
112
3
1
111
1
11
1
1
2
111
1
1 1
1
11
1
11
11
1
1
1 1
1
1
1
1
1
111
1
11
11
1
11
11
1
332
3
2
12
13
3
1
2
1
2
1
1
1
1
2
1
32
1
1
11
1
1
1
1
1
1
1
1
remain
1
be over
111
1
1
1
11
1
1
1
11
1
1
1
1
1
1111
11
1
1
1
1
111
1
11
1
1
1
11 1
11
1
1
1
1
1
1
1
2
1
1
11
1
1
1
1
1
1
1
1
2 12
2
2
1
1
1 121
1
1
1
2 12
31
2
11
1
1
1
1
2
1
1
1
112
1
1
1
12
1
11
11
1
2
111
1
1
1
1 11
1
11
1
1 11
1
1
1 1
1
1
1
1
111
1
1 11
1
21
1
1
2
1
1
1
1
21
1
1
2
11
1
1
1
2
1
person from old village32
1
11
1
1
1
1
1
1
1
world.2
1
11
1
1
1
1
11
11
111
1
1
1
1
11
11
1
1
1
1
11
111
1
1
111 1
colour1
1
1
1
1
1
1
1
1
1
1
11
1
1
1
1
1
1
1
1
11
Function TailFunction TailUnder −1σ ⇒
omou (think.1) B1
VERB a3 LK a1 a2 DN gamma b1 b2 END
omou(think)
suru(do.2)
1
haseru(run)
1
(ra)reru(PASS/POT)5
te/de(LK)
24
ni(LK)
1
ta(PERF)
5
no/koto(DN)38
node(because.2)
1
hodo(as much as..)
2
yo(FP)
7
wa(FP)
13
.
37
u(CJR)
1
torawareru(be captured)
1
1
1
2
11
iru(be-AS)23 kuru
(toward-AS)1 koi(love.2)
1
fuan(anxiety)
1
1
15
ka(Q)
1
2
5
5
1
5
da(ASSER)
34
4 sa(FP)
125
kara(because.1) 1
taisetsu(precious)
1
1 1
6
1
14
20
cherry blossom (28/131,4.28): CT ctf.>0.00;
non-dist=off; idf=off; pruned under U:1:
here
way home
1
lodging1
confuse
1
1
1
village.1
1
scatter.1
1
forget
1
(must)
1
1cloud.vi
1
such as.1
1
either way
road
1
stop.vi.2
1
perplexed1
forcibly1
1
1
1
1
2
exchange
11
1
1
1
what
hide.vt
1
out after1
mountain village
1 1
mi
cloud.1
1
mountain
3
desolate1
prosper
1
painful.1
1
high.13
seem
1
1
break off
1
1
3
trail
2
1
1
1
(suffix.forbid)
1
be lonely
1
(emph)
1
1
11
1
1
1
1
1
for
stone
1waterfall 1
hand
1
run 1
wish
1
1
1
1
1
1
1
1
1
1
1
1
present
1
tell
1
1 1
come
1
1
1
remnant
wave1
1
1
2
1kurabu.pl
1
1(p.wish)
6
see.310even3
(ku)3
lovely
1
fast/early
1
between
1
2
spring haze
1
send.v2 word
1
1
1
2
transfer.11
2
get tired.1
1Yoshino.PN
1
gorge
white cloud
1
foot 1
pull
1
bloom
1
1
seemed1
1
1
11
1
1
1
1
such as.2
1
quiet
1
1
1
1
1
every1
11
1
1
(adv.will)
2
old age
1
11
1
say
11
1
1
1
1 1
1
11
spring
add
1
1
(aux.past)
1
(aux.perf)
1
year
1
rare.1
1
day
1
1
this year
begin 1
get used
1
11
1
1
1
1
today
a glance
1
2
1
you.3
1
2
wait
1
mountain area
mistake
1
1
1
1
1
number
1
till
1
borrow
1
like.2
1
1
regrettable
1
here we go
1
1
1
1
1
only
1
1
myself
1
1
1
1
1
1 1
1
1
person from old village3
1
11
lodge.n
1
world.2
1
sorrowful.1
1
name
1
vain
1
1
wind.n
3
1
1
1
spring rain
regret1
snow1
6
remain
1
be over
1
take
1 1
1
11
1
1
1
1
1
1
1
1
change.11
cross
1
go over1
1
increase
1
bear
1
think
1
1
love.11
blow
1
trust.2
1
1
will not
1