Corpus Linguistics and NINJAL Japanese Language...2014/11/23 3 History of Japanese Corpus 1 Surveys...

20141123

1

Corpus Linguistics and

Japanese Language

Takehiko Maruyama

丸山岳彦National Institute for Japanese Language and Linguistics

University of Oxford

18 November 2014

SEMINAacuteŘ JAPONSKYacuteCH STUDIIacute

Masarykova Univerzita

コーパス言語学と日本語

NINJAL

National Institute of Japanese Language and

Linguistics (ldquoNINJALrdquo) 国立国語研究所

Established in 1948

Scientific surveys of Japanese language

Creation of Japanese corpora

Contents

(1050 ndash 1150)

Introduction Japan and Japanese Language

Japanese Corpus History

What is a ldquoCorpusrdquo

History of Japanese Corpus

Japanese Corpus Present situation

Spoken Corpus CSJ

Written Corpus BCCWJ

Introduction

Japan and

Japanese Language

Where is Japan Japan Nihon日本

Tokyo

Kyoto

Mt Fuji

Hokkaido

Japan Alps

Okinawa

20141123

2

Dialects in Japan

Dialect surveys by NINJAL since 1940s

Fukushima pref

1949

Hachijo island

1949

Tokyo

Iwate pref

1980

Tottori pref 1984

Okinawa pref 1978

Dialects in Japan

Linguistic Atlas of Japan (NINJAL 1966)

Japanese Writing System

Three types of characters

Kanji 教科書玉子

Hiragana ほんたまご

Katakana テキストタマゴ

Other types of characters

Punctuation mark 「（

Alphabet NINJAL

Arabic numeral 1234

Roman numeral I IV XIII





A ldquocorpusrdquo ishellip

an collection of language in ldquoreal worldrdquo

ldquoa collection of texts assumed to be representative of a

given language dialect or other subset of a language

to be used for linguistic analysisrdquo (Francis 1982)

Various corpora

Text (written) corpus Speech (spoken) corpus

Historical corpus Learner corpus Dialect corpushellip

ldquoCorpus linguisticsrdquo ishellip

a methodology of linguistic study using corpora

Corpus collectioncreation started in 1960s

1959 UK The Survey of English Usage (1 million words)

1964 US Brown Corpus (1 million wds)

1991 UK Bank of English (BOE) (500 million wds)

1994 UK British National Corpus (BNC) (100 million wds)

2000 CZ Czech National Corpus (CNC) (100 million wds)

2004 JP Corpus of Spontaneous Japanese (CSJ) (75 million wds)

2011 JP Balanced Corpus of Contemporary Written Japanese

(BCCWJ) (100 million wds)

Various corpora in the world

Where is the origin of Japanese corpus

20141123

3

History of Japanese Corpus 1

Surveys of daily vocabulary at NINJAL

1953 Research on vocabulary in womens magazines

1957-1958 Research in vocabulary in cultural reviews

1962-1964 Vocabulary and Chinese characters in

ninety magazines of today (I II III) 05 million words

Real text Sampling Vocabulary list

Origin of Japanese written Corpus


Surveys of colloquial speech at NINJAL

1955 Research in the colloquial Japanese

30 hours of colloquial speech were recorded 83620 words

1960 1963 A research for making sentence patterns

in colloquial Japanese (1 dialog) (2 monolog)







Origin of Japanese spoken Corpus

Vocabulary surveys using computers

1970-1973 Studies on the vocabulary of modern

newspapers 1-4

2 million words from three major newspapers in 1966




newspapers 1-4


History of Japanese Corpus 3Origin of Japanese electric Corpus

20141123

4

Japanese Corpora in 2000s

NINJAL started creating large sized corpora

Corpus of Spontaneous Japanese (CSJ) - 2004

651 hours 752 million words of spontaneous speech

Balanced Corpus of Contemporary Written Japanese

(BCCWJ) - 2011

100 million words of various written text (well balanced)

Corpus of Historical Japanese (CHJ) - 2013~

14 literary works with 079 million words in Heian period

Ultra Large-sized Corpus (ULC) - under construction

10 billion words of Japanese text extracted from web

Japanese Corpus

Present situation

Spoken Corpus CSJ


Keywords

Knowledge and Behavior

Znalosti a chovaacuteniacute

知識と行動

CSJ

Corpus of Spontaneous Japanese

『日本語話し言葉コーパス』

1 コミニケーション

2 コミュニケーション

3 コミニュケーション

4 コミュニュケーション

Question 1

Which one is a correct spell of ldquocommunicationrdquo in Japanese

Variable forms in speech

How do you read this word

自転車じてんしゃ

Question 2

ji ten syaYes

20141123

5

How do Japanese people pronounce the word ldquo自転車rdquo in real life

自転車じでんしゃ

Question 2-2

ji den sya

Guess the percentages of each pronunciation in real Japanese

コミニケーション（）

コミュニケーション（）

コミニュケーション（）

コミュニュケーション（）

じてんしゃ（）

じでんしゃ（）

Question 3

How should we get the answers

When you hesitate while speaking you might use FPs (filled pauses)

hm er uh What type of FP do you use most frequently in your daily Czech

How about in Japanese

Question 4



Think it in your head (intuition)

Your answer may be wrong

Who guarantee your answer

Ask the speech corpus (survey)

Everyone can get the same answer

Of course you need a reliable corpus

We have knowledge about (at least) a language

But we donrsquot know how we behave with it

CSJ

Corpus of Spontaneous Japanese (2004)

Japanese spontaneous speech (mainly monolog)

651 hours 752 million words

3302 lectures by 1418 different speakers

Rich annotations

18 DVDs

Aims

Automatic Speech Recognition (ASR) system

Linguistic study of spontaneous speech

20141123

6

Basic Form amp Pronounced FormFP

Frag-

ment

Repair

Variable

pronunciation

Elongation

Two Ways of Transcription Answer to ldquoCommunicationrdquo

How do Japanese pronounce ldquocommunicationrdquo

Corpus CSJ 651 hours 752 million words

Frequency of the word ldquocommunicationrdquo 601 times

コミニケーション 296

コミニュケーション 136

コミュニケーション 123

コミュニュケーション 36

misc 10

Total 601

49

23

20

6 2

コミニケーション

コミニュケーション

コミュニケーション

コミュニュケーション

その他

Answer to ldquo自転車rdquo

How do Japanese pronounce 自転車


Frequency of the word 自転車 483 times

ジテンシャ 349

ジデンシャ 116

misc 18

Total 483

72

24

4

ジテンシャ

ジデンシャ

misc

Answer to Filled Pauses (JP)

What FP do Japanese use most frequently


Frequency of Filled Pauses 430472 times

(F えー) (F e) 116772 271

(F え) (F e) 45665 106

(F ま) (F ma) 44549 104

(F あのー) (F ano) 40695 95

(F あの) (F ano) 33330 77(top 5)

Answer to the Filled Pauses (JP)




Male

(F e) 95359 302

(F e) 36078 114

(F ma) 34643 110

(F ma) 24369 77

(F ano) 21302 68

Female

(F e) 21413 187

(F ano) 19393 169

(F ano) 15954 139

(F ma) 9906 86

(F e) 9587 84

えー e あのー ano

Answer to the Filled Pauses (CZ)

What FP do Czech use most frequently

hellip and tell me the result

20141123

7

Annotations to speech signals

Two-way Transcription

Segment Labels

Intonation Labels

Morphological Analysis

Clause Boundary Labels

Dependency Structure

Discourse Structure

Impression Rating

Speaker Info

Phonetics Phonology

Morphology Lexicon

Syntax

Discourse analysis

Metadata bibliography


All transcriptions were segmented into words

(manuallyautomatically) with rich information

Transcription ID

Utterance time

Orthographic form

Pronunciation form

Part-of-Speech

Conjugation type

Conjugation form

XML Encoding

Various annotation were encoded into XML file

Concordancer ldquoHimawarirdquo

What CSJ offers

Variations in spontaneous speech

Pronunciation Accent Intonation Grammarhellip

Disfluency in spontaneous speech

FP Word Fragments Elongation Self-repairhellip

Resource to analyze behaviors in spontaneous JP

Future work to create a large dialog corpus

Linguistic knowledge never tells us our behavior

BCCWJ

Balanced Corpus of Contemporary

Written Japanese

『現代日本語書き言葉均衡コーパス』

20141123

8

BCCWJ

Contents balanced corpus for general purpose

Corpus Size 100 million words

Period 1976 - 2005 (-2009)

Media Books Magazines Newspapers

Whitepapers Textbooks Web Documents Law

Verse Diet minutes

Method Stratified random sampling

Aim Vocabulary survey Grammatical study

Lexicography Natural language processing

Structure of BCCWJ

Publication sub-corpus

Books Magazines

Newspapers

35 million words

2001-2005

Library sub-corpus

Books stored in many

public libraries

30 million words

1986-2005

Special-purpose sub-corpus

Whitepapers Textbooks Public Relation Best-Seller

book Web documents Verse Law Diet minutes

40 million words 1976-2005

Publication Sub-corpus

Population

All the books magazines and newspapers published

in the years 2001 to 2005

defined by the number of characters

Actual state of Publication

Population ( of chars)

Books

Magazines

NewspapersSample (35M words)

Definition of Population

Investigated number of chars in 2001- 2005

Titles Pages Chars

Books 317117 74911520 48539925351

Magazines 55779 10414955 10515681636

Newspapers 49625 1198189 6416070114

Powered by

National Diet Library

Japan Magazine Publishers Association

Japan Newspaper Publishers Association

Stratification and Each Ratio

chars ratio

Book 48539925351 74138

Magazines 10515681636 16063

Newspaper 6416070114 9800

TOTAL 65471677100 100

Genres

times11

times 6

times 3

Media Strata of chars Ratio Media Strata of chars Ratio

Book

0 General works 1636414548 250

Magazine

General 7421447806 1134

1 Philosophy 2597610813 397 Education 877875592 134

2 General history 4301204340 657 Politics 456459405 070

3 Social sciences 12408321943 1895 Industry 110640958 017

4 Natural sciences 5069594034 774 Technology 1468293360 224

5 Technology 4615929967 705 Medical 180964513 028

6 Industry 2196387437 335

Newspaper

National 2417622461 369

7 The arts 3258432447 498 Block 1296592154 198

8 Language 888800128 136 Local 2701855499 413

9 Literature 9341275486 1427 Total 65471677100 100

n Unclassified 2225954208 340

Distribution of chars = Compositional Ratio

Extracting sample

A character randomly

chosen in a page

Sample starts here

Figures old Japanese are

omitted

20141123

9

Compilation of BCCWJ

Sampling (as shown above)

Copyright solution

We identified almost 30000 copyright holders

70-80 of them approved to the request

Text digitalization and XML tagging

Logical structure of text

Annotation of Part of Speech information

98 accuracy with an electronic dictionary UniDic

999 with annotatorrsquos modification for 1 million wd


ltxml version=10 encoding=UTF-8gt

ltsample sampleID=LBe2_00005 version=10 type=fixedLengthgt

ltarticle articleID=LBe2_00005_F001gt

ltparagraphgt

ltsentencegtやがて後ltsampling type=start gt燕は漢人のltruby rubyText=

ひょうgt馮ltrubygtltruby rubyText=ばつgt跋ltrubygtに乗っ取られてしまいますltsentencegt

ltsentencegt西暦四〇九年のことですがこの翌年前記の南燕が東晋のltruby

rubyText=りゅうgt劉ltrubygtltruby rubyText=ゆうgt裕ltrubygtによってほろぼされてしまいましたltsentencegt

ltparagraphgt

ltparagraphgt

ltsentencegt 四〇九年にはいろいろなことがおこっていますltsentencegt

ltsentencegtさしもの拓跋珪もこの年思わぬことであろうことか息子の一人ltruby rubyText=ldquoたくrdquogt拓ltrubygtltruby rubyText=ldquoばつrdquogt跋ltrubygtltruby

rubyText=ldquoしょうrdquogt紹ltrubygtによって殺されましたltsentencegt

ltparagraphgt

Release of BCCWJ

In 2011 completed BCCWJ is released

少納言 Shonagonhttpwwwkotonohagrjpshonagon

Character-based Concordance on the web

Free max 500 examples (randomly chosen)

中納言 Chunagon

httpschunagonninjalacjp

Word-based Concordance on the web

Registration is needed all the examples downloadable

DVD

All the morphologically analyzed text bibliographic data

Academic Use 52500 YEN

Collocations in BCCWJ

NINJAL-LWP for BCCWJ

httpnlbninjalacjp

Shows collocation (common word combinations)

Question 5How do Japanese write in daily life

tamagokatakana

hiragana kanjikanji

Which is most frequent

What BCCWJ offers

The first balanced corpus of written Japanese

Actual situation of published spread written text

Various types of written text

Easy access to 100 million words corpus

Everybody can use a large-sized corpus

Objective tests for linguistic analyses

Infrastructure for Japanese corpus linguistics

20141123

10

Conclusion before Lunch

Japanese corpora

NINJAL stated creating a series of large corpora

rapidly since 2000

Infrastructures for Japanese corpus linguistics


There are many linguistic questions we can not

answer with our linguistic knowledge

Linguists need reliable corpora to investigate the

linguistic behavior in actual life

Use corpora

Workshop after Lunch

BCCWJ demonstrations

少納言 Shonagon

中納言 Chunagon


CSJ demonstration

ひまわり Himawari

Other resources

青空文庫 Aozora Bunko on ひまわり Himawari


Japanese Language (2)

Workshop

Takehiko Maruyama

National Institute for Japanese Language and Linguistics


18 November 2014




少納言 Shonagon

中納言 Chunagon


CSJ demonstration


Other resources


Contents

(1410 ndash 1545)

BCCWJ

demonstrations


What is this

すいか

西瓜スイカ

20141123

11

すいかスイカ西瓜

How do they write it



Question 6すいかスイカ西瓜

Which is the most frequent in Newspapers

Ask 少納言 Shonagon

httpwwwkotonohagrjpshonagon

Question 7

Give an example of writing variation like すいか

and ask 少納言 Shonagon

For examplehellip

バイオリンヴァイオリン

ダイヤモンドダイアモンド

買い物買物

打ち合わせ打合わせ打合せ

にんじんニンジン人参

ひふ科ヒフ科皮ふ科皮フ科皮膚科


tamagokatakana

hiragana kanji kanji


Question 5たまごタマゴ玉子卵

- Which is the most frequent in BCCWJ

Is it a good way to ask 少納言 Shonagon

Example of search result ldquo卵rdquo

「バター黒糖卵黄をよくすり混ぜる」

（Butter brown sugar yolk mix them well）

卵黄 (yolk)

らんおう (ran o )Itrsquos not the case of 卵

たまご

20141123

12

Ask 中納言 Chunagon in which Part-of-Speech

information can be used

httpschunagonninjalacjplogin

Registration is needed to log in

Question 5Settings for the corpus search

『語彙素』が『卵』 larr Lemma

AND 『語彙素読み』が『タマゴ』 larr Reading

Question 5

Question 8

Give an example of writing variation like たまご

and ask 中納言 Chunagon

For examplehellip

買い物買物

ねこネコ猫

いぬイヌ犬



httpnlbninjalacjp


Ask NLB about Japanese collocations

「 X を飲む」 (to drink X)

What is the most frequent word for X in BCCWJ

Question 9 Question 10Give an example of collocation like 「Xを飲む」

and ask NLB

For examplehellip

「 X を食べる」 eat X

「 X を聞く」 listen to X

「 X を読む」 read X

「 X を書く」 write X

「 X を話す」 speak X

20141123

13

CSJ



Distribution of CSJ

CSJ (with 18 DVDs) is distributed at the Center

for Corpus Development NINJAL

httpwwwninjalacjpcorpus_centercsj

Himawari

Himawari is a character-based concordance

system for Japanese linguistics

httpgooglnBcPO

Answer to ldquoCommunicationrdquo








misc 10

Total 601

49

23

20

6 2





その他





(F えー) (F e) 116772 271

(F え) (F e) 45665 106

(F ま) (F ma) 44549 104

(F あのー) (F ano) 40695 95

(F あの) (F ano) 33330 77(top 5)

Aozora Bunko

『青空文庫』

20141123

14

Aozora Bunko

Aozora Bunko (青空文庫) is a Japanese digital

library This online collection has several

thousands of works of Japanese-language fiction

and non-fiction Aozora Bunko has digital copies

of many out-of-copyright books

httpwwwaozoragrjp

Aozora Bunko on Himawari

Aozora Bunko Package can be downloaded

httpgooglRe73C

Instead of Conclusionhellip

ありがとう 7085

有難う 419ありがと 337有り難う 102

アリガト 26アリガトウ 24あリがとう 3アリガト 2あリがと 2

アリヽ（＾＾）ノガトゥ 1総計 8001

20141123

2

Dialects in Japan

Dialect surveys by NINJAL since 1940s

Fukushima pref

1949

Hachijo island

1949

Tokyo

Iwate pref

1980

Tottori pref 1984

Okinawa pref 1978

Dialects in Japan

Linguistic Atlas of Japan (NINJAL 1966)

Japanese Writing System

Three types of characters

Kanji 教科書玉子

Hiragana ほんたまご

Katakana テキストタマゴ

Other types of characters

Punctuation mark 「（

Alphabet NINJAL

Arabic numeral 1234

Roman numeral I IV XIII





A ldquocorpusrdquo ishellip

an collection of language in ldquoreal worldrdquo

ldquoa collection of texts assumed to be representative of a

given language dialect or other subset of a language

to be used for linguistic analysisrdquo (Francis 1982)

Various corpora

Text (written) corpus Speech (spoken) corpus

Historical corpus Learner corpus Dialect corpushellip

ldquoCorpus linguisticsrdquo ishellip

a methodology of linguistic study using corpora

Corpus collectioncreation started in 1960s

1959 UK The Survey of English Usage (1 million words)

1964 US Brown Corpus (1 million wds)

1991 UK Bank of English (BOE) (500 million wds)

1994 UK British National Corpus (BNC) (100 million wds)

2000 CZ Czech National Corpus (CNC) (100 million wds)

2004 JP Corpus of Spontaneous Japanese (CSJ) (75 million wds)

2011 JP Balanced Corpus of Contemporary Written Japanese

(BCCWJ) (100 million wds)

Various corpora in the world

Where is the origin of Japanese corpus

20141123

3
























newspapers 1-4





newspapers 1-4



20141123

4






(BCCWJ) - 2011






Japanese Corpus

Present situation

Spoken Corpus CSJ


Keywords



知識と行動

CSJ







Question 1





Question 2

ji ten syaYes

20141123

5



Question 2-2

ji den sya








Question 3





Question 4











CSJ





Rich annotations

18 DVDs

Aims



20141123

6


Frag-

ment

Repair

Variable

pronunciation

Elongation









misc 10

Total 601

49

23

20

6 2





その他





ジテンシャ 349

ジデンシャ 116

misc 18

Total 483

72

24

4

ジテンシャ

ジデンシャ

misc





(F えー) (F e) 116772 271

(F え) (F e) 45665 106

(F ま) (F ma) 44549 104

(F あのー) (F ano) 40695 95

(F あの) (F ano) 33330 77(top 5)





Male

(F e) 95359 302

(F e) 36078 114

(F ma) 34643 110

(F ma) 24369 77

(F ano) 21302 68

Female

(F e) 21413 187

(F ano) 19393 169

(F ano) 15954 139

(F ma) 9906 86

(F e) 9587 84





20141123

7



Segment Labels

Intonation Labels




Discourse Structure

Impression Rating

Speaker Info

Phonetics Phonology

Morphology Lexicon

Syntax

Discourse analysis





Transcription ID

Utterance time

Orthographic form

Pronunciation form

Part-of-Speech

Conjugation type

Conjugation form

XML Encoding



What CSJ offers








BCCWJ


Written Japanese


20141123

8

BCCWJ



Period 1976 - 2005 (-2009)



Verse Diet minutes




Structure of BCCWJ


Books Magazines

Newspapers

35 million words

2001-2005

Library sub-corpus


public libraries

30 million words

1986-2005






Population






Books

Magazines




Titles Pages Chars

Books 317117 74911520 48539925351

Magazines 55779 10414955 10515681636

Newspapers 49625 1198189 6416070114

Powered by





chars ratio

Book 48539925351 74138

Magazines 10515681636 16063

Newspaper 6416070114 9800

TOTAL 65471677100 100

Genres

times11

times 6

times 3


Book


Magazine

General 7421447806 1134






6 Industry 2196387437 335

Newspaper

National 2417622461 369

7 The arts 3258432447 498 Block 1296592154 198

8 Language 888800128 136 Local 2701855499 413




Extracting sample


chosen in a page

Sample starts here


omitted

20141123

9



Copyright solution












ltparagraphgt





ltparagraphgt

ltparagraphgt




ltparagraphgt

Release of BCCWJ





中納言 Chunagon




DVD





httpnlbninjalacjp



tamagokatakana

hiragana kanjikanji


What BCCWJ offers








20141123

10


Japanese corpora


rapidly since 2000







Use corpora



少納言 Shonagon

中納言 Chunagon


CSJ demonstration


Other resources




Workshop

Takehiko Maruyama



18 November 2014




少納言 Shonagon

中納言 Chunagon


CSJ demonstration


Other resources


Contents

(1410 ndash 1545)

BCCWJ

demonstrations


What is this

すいか

西瓜スイカ

20141123

11









Question 7



For examplehellip



買い物買物





tamagokatakana









卵黄 (yolk)


たまご

20141123

12








Question 5

Question 8



For examplehellip

買い物買物

ねこネコ猫

いぬイヌ犬



httpnlbninjalacjp






and ask NLB

For examplehellip






20141123

13

CSJ



Distribution of CSJ




Himawari



httpgooglnBcPO









misc 10

Total 601

49

23

20

6 2





その他





(F えー) (F e) 116772 271

(F え) (F e) 45665 106

(F ま) (F ma) 44549 104

(F あのー) (F ano) 40695 95

(F あの) (F ano) 33330 77(top 5)

Aozora Bunko

『青空文庫』

20141123

14

Aozora Bunko






httpwwwaozoragrjp



httpgooglRe73C






20141123

3
























newspapers 1-4





newspapers 1-4



20141123

4






(BCCWJ) - 2011






Japanese Corpus

Present situation

Spoken Corpus CSJ


Keywords



知識と行動

CSJ







Question 1





Question 2

ji ten syaYes

20141123

5



Question 2-2

ji den sya








Question 3





Question 4











CSJ





Rich annotations

18 DVDs

Aims



20141123

6


Frag-

ment

Repair

Variable

pronunciation

Elongation









misc 10

Total 601

49

23

20

6 2





その他





ジテンシャ 349

ジデンシャ 116

misc 18

Total 483

72

24

4

ジテンシャ

ジデンシャ

misc





(F えー) (F e) 116772 271

(F え) (F e) 45665 106

(F ま) (F ma) 44549 104

(F あのー) (F ano) 40695 95

(F あの) (F ano) 33330 77(top 5)





Male

(F e) 95359 302

(F e) 36078 114

(F ma) 34643 110

(F ma) 24369 77

(F ano) 21302 68

Female

(F e) 21413 187

(F ano) 19393 169

(F ano) 15954 139

(F ma) 9906 86

(F e) 9587 84





20141123

7



Segment Labels

Intonation Labels




Discourse Structure

Impression Rating

Speaker Info

Phonetics Phonology

Morphology Lexicon

Syntax

Discourse analysis





Transcription ID

Utterance time

Orthographic form

Pronunciation form

Part-of-Speech

Conjugation type

Conjugation form

XML Encoding



What CSJ offers








BCCWJ


Written Japanese


20141123

8

BCCWJ



Period 1976 - 2005 (-2009)



Verse Diet minutes




Structure of BCCWJ


Books Magazines

Newspapers

35 million words

2001-2005

Library sub-corpus


public libraries

30 million words

1986-2005






Population






Books

Magazines




Titles Pages Chars

Books 317117 74911520 48539925351

Magazines 55779 10414955 10515681636

Newspapers 49625 1198189 6416070114

Powered by





chars ratio

Book 48539925351 74138

Magazines 10515681636 16063

Newspaper 6416070114 9800

TOTAL 65471677100 100

Genres

times11

times 6

times 3


Book


Magazine

General 7421447806 1134






6 Industry 2196387437 335

Newspaper

National 2417622461 369

7 The arts 3258432447 498 Block 1296592154 198

8 Language 888800128 136 Local 2701855499 413




Extracting sample


chosen in a page

Sample starts here


omitted

20141123

9



Copyright solution












ltparagraphgt





ltparagraphgt

ltparagraphgt




ltparagraphgt

Release of BCCWJ





中納言 Chunagon




DVD





httpnlbninjalacjp



tamagokatakana

hiragana kanjikanji


What BCCWJ offers








20141123

10


Japanese corpora


rapidly since 2000







Use corpora



少納言 Shonagon

中納言 Chunagon


CSJ demonstration


Other resources




Workshop

Takehiko Maruyama



18 November 2014




少納言 Shonagon

中納言 Chunagon


CSJ demonstration


Other resources


Contents

(1410 ndash 1545)

BCCWJ

demonstrations


What is this

すいか

西瓜スイカ

20141123

11









Question 7



For examplehellip



買い物買物





tamagokatakana









卵黄 (yolk)


たまご

20141123

12








Question 5

Question 8



For examplehellip

買い物買物

ねこネコ猫

いぬイヌ犬



httpnlbninjalacjp






and ask NLB

For examplehellip






20141123

13

CSJ



Distribution of CSJ




Himawari



httpgooglnBcPO









misc 10

Total 601

49

23

20

6 2





その他





(F えー) (F e) 116772 271

(F え) (F e) 45665 106

(F ま) (F ma) 44549 104

(F あのー) (F ano) 40695 95

(F あの) (F ano) 33330 77(top 5)

Aozora Bunko

『青空文庫』

20141123

14

Aozora Bunko






httpwwwaozoragrjp



httpgooglRe73C






20141123

4






(BCCWJ) - 2011






Japanese Corpus

Present situation

Spoken Corpus CSJ


Keywords



知識と行動

CSJ







Question 1





Question 2

ji ten syaYes

20141123

5



Question 2-2

ji den sya








Question 3





Question 4











CSJ





Rich annotations

18 DVDs

Aims



20141123

6


Frag-

ment

Repair

Variable

pronunciation

Elongation









misc 10

Total 601

49

23

20

6 2





その他





ジテンシャ 349

ジデンシャ 116

misc 18

Total 483

72

24

4

ジテンシャ

ジデンシャ

misc





(F えー) (F e) 116772 271

(F え) (F e) 45665 106

(F ま) (F ma) 44549 104

(F あのー) (F ano) 40695 95

(F あの) (F ano) 33330 77(top 5)





Male

(F e) 95359 302

(F e) 36078 114

(F ma) 34643 110

(F ma) 24369 77

(F ano) 21302 68

Female

(F e) 21413 187

(F ano) 19393 169

(F ano) 15954 139

(F ma) 9906 86

(F e) 9587 84





20141123

7



Segment Labels

Intonation Labels




Discourse Structure

Impression Rating

Speaker Info

Phonetics Phonology

Morphology Lexicon

Syntax

Discourse analysis





Transcription ID

Utterance time

Orthographic form

Pronunciation form

Part-of-Speech

Conjugation type

Conjugation form

XML Encoding



What CSJ offers








BCCWJ


Written Japanese


20141123

8

BCCWJ



Period 1976 - 2005 (-2009)



Verse Diet minutes




Structure of BCCWJ


Books Magazines

Newspapers

35 million words

2001-2005

Library sub-corpus


public libraries

30 million words

1986-2005






Population






Books

Magazines




Titles Pages Chars

Books 317117 74911520 48539925351

Magazines 55779 10414955 10515681636

Newspapers 49625 1198189 6416070114

Powered by





chars ratio

Book 48539925351 74138

Magazines 10515681636 16063

Newspaper 6416070114 9800

TOTAL 65471677100 100

Genres

times11

times 6

times 3


Book


Magazine

General 7421447806 1134






6 Industry 2196387437 335

Newspaper

National 2417622461 369

7 The arts 3258432447 498 Block 1296592154 198

8 Language 888800128 136 Local 2701855499 413




Extracting sample


chosen in a page

Sample starts here


omitted

20141123

9



Copyright solution












ltparagraphgt





ltparagraphgt

ltparagraphgt




ltparagraphgt

Release of BCCWJ





中納言 Chunagon




DVD





httpnlbninjalacjp



tamagokatakana

hiragana kanjikanji


What BCCWJ offers








20141123

10


Japanese corpora


rapidly since 2000







Use corpora



少納言 Shonagon

中納言 Chunagon


CSJ demonstration


Other resources




Workshop

Takehiko Maruyama



18 November 2014




少納言 Shonagon

中納言 Chunagon


CSJ demonstration


Other resources


Contents

(1410 ndash 1545)

BCCWJ

demonstrations


What is this

すいか

西瓜スイカ

20141123

11









Question 7



For examplehellip



買い物買物





tamagokatakana









卵黄 (yolk)


たまご

20141123

12








Question 5

Question 8



For examplehellip

買い物買物

ねこネコ猫

いぬイヌ犬



httpnlbninjalacjp






and ask NLB

For examplehellip






20141123

13

CSJ



Distribution of CSJ




Himawari



httpgooglnBcPO









misc 10

Total 601

49

23

20

6 2





その他





(F えー) (F e) 116772 271

(F え) (F e) 45665 106

(F ま) (F ma) 44549 104

(F あのー) (F ano) 40695 95

(F あの) (F ano) 33330 77(top 5)

Aozora Bunko

『青空文庫』

20141123

14

Aozora Bunko






httpwwwaozoragrjp



httpgooglRe73C






20141123

5



Question 2-2

ji den sya








Question 3





Question 4











CSJ





Rich annotations

18 DVDs

Aims



20141123

6


Frag-

ment

Repair

Variable

pronunciation

Elongation









misc 10

Total 601

49

23

20

6 2





その他





ジテンシャ 349

ジデンシャ 116

misc 18

Total 483

72

24

4

ジテンシャ

ジデンシャ

misc





(F えー) (F e) 116772 271

(F え) (F e) 45665 106

(F ま) (F ma) 44549 104

(F あのー) (F ano) 40695 95

(F あの) (F ano) 33330 77(top 5)





Male

(F e) 95359 302

(F e) 36078 114

(F ma) 34643 110

(F ma) 24369 77

(F ano) 21302 68

Female

(F e) 21413 187

(F ano) 19393 169

(F ano) 15954 139

(F ma) 9906 86

(F e) 9587 84





20141123

7



Segment Labels

Intonation Labels




Discourse Structure

Impression Rating

Speaker Info

Phonetics Phonology

Morphology Lexicon

Syntax

Discourse analysis





Transcription ID

Utterance time

Orthographic form

Pronunciation form

Part-of-Speech

Conjugation type

Conjugation form

XML Encoding



What CSJ offers








BCCWJ


Written Japanese


20141123

8

BCCWJ



Period 1976 - 2005 (-2009)



Verse Diet minutes




Structure of BCCWJ


Books Magazines

Newspapers

35 million words

2001-2005

Library sub-corpus


public libraries

30 million words

1986-2005






Population






Books

Magazines




Titles Pages Chars

Books 317117 74911520 48539925351

Magazines 55779 10414955 10515681636

Newspapers 49625 1198189 6416070114

Powered by





chars ratio

Book 48539925351 74138

Magazines 10515681636 16063

Newspaper 6416070114 9800

TOTAL 65471677100 100

Genres

times11

times 6

times 3


Book


Magazine

General 7421447806 1134






6 Industry 2196387437 335

Newspaper

National 2417622461 369

7 The arts 3258432447 498 Block 1296592154 198

8 Language 888800128 136 Local 2701855499 413




Extracting sample


chosen in a page

Sample starts here


omitted

20141123

9



Copyright solution












ltparagraphgt





ltparagraphgt

ltparagraphgt




ltparagraphgt

Release of BCCWJ





中納言 Chunagon




DVD





httpnlbninjalacjp



tamagokatakana

hiragana kanjikanji


What BCCWJ offers








20141123

10


Japanese corpora


rapidly since 2000







Use corpora



少納言 Shonagon

中納言 Chunagon


CSJ demonstration


Other resources




Workshop

Takehiko Maruyama



18 November 2014




少納言 Shonagon

中納言 Chunagon


CSJ demonstration


Other resources


Contents

(1410 ndash 1545)

BCCWJ

demonstrations


What is this

すいか

西瓜スイカ

20141123

11









Question 7



For examplehellip



買い物買物





tamagokatakana









卵黄 (yolk)


たまご

20141123

12








Question 5

Question 8



For examplehellip

買い物買物

ねこネコ猫

いぬイヌ犬



httpnlbninjalacjp






and ask NLB

For examplehellip






20141123

13

CSJ



Distribution of CSJ




Himawari



httpgooglnBcPO









misc 10

Total 601

49

23

20

6 2





その他





(F えー) (F e) 116772 271

(F え) (F e) 45665 106

(F ま) (F ma) 44549 104

(F あのー) (F ano) 40695 95

(F あの) (F ano) 33330 77(top 5)

Aozora Bunko

『青空文庫』

20141123

14

Aozora Bunko






httpwwwaozoragrjp



httpgooglRe73C






20141123

6


Frag-

ment

Repair

Variable

pronunciation

Elongation









misc 10

Total 601

49

23

20

6 2





その他





ジテンシャ 349

ジデンシャ 116

misc 18

Total 483

72

24

4

ジテンシャ

ジデンシャ

misc





(F えー) (F e) 116772 271

(F え) (F e) 45665 106

(F ま) (F ma) 44549 104

(F あのー) (F ano) 40695 95

(F あの) (F ano) 33330 77(top 5)





Male

(F e) 95359 302

(F e) 36078 114

(F ma) 34643 110

(F ma) 24369 77

(F ano) 21302 68

Female

(F e) 21413 187

(F ano) 19393 169

(F ano) 15954 139

(F ma) 9906 86

(F e) 9587 84





20141123

7



Segment Labels

Intonation Labels




Discourse Structure

Impression Rating

Speaker Info

Phonetics Phonology

Morphology Lexicon

Syntax

Discourse analysis





Transcription ID

Utterance time

Orthographic form

Pronunciation form

Part-of-Speech

Conjugation type

Conjugation form

XML Encoding



What CSJ offers








BCCWJ


Written Japanese


20141123

8

BCCWJ



Period 1976 - 2005 (-2009)



Verse Diet minutes




Structure of BCCWJ


Books Magazines

Newspapers

35 million words

2001-2005

Library sub-corpus


public libraries

30 million words

1986-2005






Population






Books

Magazines




Titles Pages Chars

Books 317117 74911520 48539925351

Magazines 55779 10414955 10515681636

Newspapers 49625 1198189 6416070114

Powered by





chars ratio

Book 48539925351 74138

Magazines 10515681636 16063

Newspaper 6416070114 9800

TOTAL 65471677100 100

Genres

times11

times 6

times 3


Book


Magazine

General 7421447806 1134






6 Industry 2196387437 335

Newspaper

National 2417622461 369

7 The arts 3258432447 498 Block 1296592154 198

8 Language 888800128 136 Local 2701855499 413




Extracting sample


chosen in a page

Sample starts here


omitted

20141123

9



Copyright solution












ltparagraphgt





ltparagraphgt

ltparagraphgt




ltparagraphgt

Release of BCCWJ





中納言 Chunagon




DVD





httpnlbninjalacjp



tamagokatakana

hiragana kanjikanji


What BCCWJ offers








20141123

10


Japanese corpora


rapidly since 2000







Use corpora



少納言 Shonagon

中納言 Chunagon


CSJ demonstration


Other resources




Workshop

Takehiko Maruyama



18 November 2014




少納言 Shonagon

中納言 Chunagon


CSJ demonstration


Other resources


Contents

(1410 ndash 1545)

BCCWJ

demonstrations


What is this

すいか

西瓜スイカ

20141123

11









Question 7



For examplehellip



買い物買物





tamagokatakana









卵黄 (yolk)


たまご

20141123

12








Question 5

Question 8



For examplehellip

買い物買物

ねこネコ猫

いぬイヌ犬



httpnlbninjalacjp






and ask NLB

For examplehellip






20141123

13

CSJ



Distribution of CSJ




Himawari



httpgooglnBcPO









misc 10

Total 601

49

23

20

6 2





その他





(F えー) (F e) 116772 271

(F え) (F e) 45665 106

(F ま) (F ma) 44549 104

(F あのー) (F ano) 40695 95

(F あの) (F ano) 33330 77(top 5)

Aozora Bunko

『青空文庫』

20141123

14

Aozora Bunko






httpwwwaozoragrjp



httpgooglRe73C






20141123

7



Segment Labels

Intonation Labels




Discourse Structure

Impression Rating

Speaker Info

Phonetics Phonology

Morphology Lexicon

Syntax

Discourse analysis





Transcription ID

Utterance time

Orthographic form

Pronunciation form

Part-of-Speech

Conjugation type

Conjugation form

XML Encoding



What CSJ offers








BCCWJ


Written Japanese


20141123

8

BCCWJ



Period 1976 - 2005 (-2009)



Verse Diet minutes




Structure of BCCWJ


Books Magazines

Newspapers

35 million words

2001-2005

Library sub-corpus


public libraries

30 million words

1986-2005






Population






Books

Magazines




Titles Pages Chars

Books 317117 74911520 48539925351

Magazines 55779 10414955 10515681636

Newspapers 49625 1198189 6416070114

Powered by





chars ratio

Book 48539925351 74138

Magazines 10515681636 16063

Newspaper 6416070114 9800

TOTAL 65471677100 100

Genres

times11

times 6

times 3


Book


Magazine

General 7421447806 1134






6 Industry 2196387437 335

Newspaper

National 2417622461 369

7 The arts 3258432447 498 Block 1296592154 198

8 Language 888800128 136 Local 2701855499 413




Extracting sample


chosen in a page

Sample starts here


omitted

20141123

9



Copyright solution












ltparagraphgt





ltparagraphgt

ltparagraphgt




ltparagraphgt

Release of BCCWJ





中納言 Chunagon




DVD





httpnlbninjalacjp



tamagokatakana

hiragana kanjikanji


What BCCWJ offers








20141123

10


Japanese corpora


rapidly since 2000







Use corpora



少納言 Shonagon

中納言 Chunagon


CSJ demonstration


Other resources




Workshop

Takehiko Maruyama



18 November 2014




少納言 Shonagon

中納言 Chunagon


CSJ demonstration


Other resources


Contents

(1410 ndash 1545)

BCCWJ

demonstrations


What is this

すいか

西瓜スイカ

20141123

11









Question 7



For examplehellip



買い物買物





tamagokatakana









卵黄 (yolk)


たまご

20141123

12








Question 5

Question 8



For examplehellip

買い物買物

ねこネコ猫

いぬイヌ犬



httpnlbninjalacjp






and ask NLB

For examplehellip






20141123

13

CSJ



Distribution of CSJ




Himawari



httpgooglnBcPO









misc 10

Total 601

49

23

20

6 2





その他





(F えー) (F e) 116772 271

(F え) (F e) 45665 106

(F ま) (F ma) 44549 104

(F あのー) (F ano) 40695 95

(F あの) (F ano) 33330 77(top 5)

Aozora Bunko

『青空文庫』

20141123

14

Aozora Bunko






httpwwwaozoragrjp



httpgooglRe73C






20141123

8

BCCWJ



Period 1976 - 2005 (-2009)



Verse Diet minutes




Structure of BCCWJ


Books Magazines

Newspapers

35 million words

2001-2005

Library sub-corpus


public libraries

30 million words

1986-2005






Population






Books

Magazines




Titles Pages Chars

Books 317117 74911520 48539925351

Magazines 55779 10414955 10515681636

Newspapers 49625 1198189 6416070114

Powered by





chars ratio

Book 48539925351 74138

Magazines 10515681636 16063

Newspaper 6416070114 9800

TOTAL 65471677100 100

Genres

times11

times 6

times 3


Book


Magazine

General 7421447806 1134






6 Industry 2196387437 335

Newspaper

National 2417622461 369

7 The arts 3258432447 498 Block 1296592154 198

8 Language 888800128 136 Local 2701855499 413




Extracting sample


chosen in a page

Sample starts here


omitted

20141123

9



Copyright solution












ltparagraphgt





ltparagraphgt

ltparagraphgt




ltparagraphgt

Release of BCCWJ





中納言 Chunagon




DVD





httpnlbninjalacjp



tamagokatakana

hiragana kanjikanji


What BCCWJ offers








20141123

10


Japanese corpora


rapidly since 2000







Use corpora



少納言 Shonagon

中納言 Chunagon


CSJ demonstration


Other resources




Workshop

Takehiko Maruyama



18 November 2014




少納言 Shonagon

中納言 Chunagon


CSJ demonstration


Other resources


Contents

(1410 ndash 1545)

BCCWJ

demonstrations


What is this

すいか

西瓜スイカ

20141123

11









Question 7



For examplehellip



買い物買物





tamagokatakana









卵黄 (yolk)


たまご

20141123

12








Question 5

Question 8



For examplehellip

買い物買物

ねこネコ猫

いぬイヌ犬



httpnlbninjalacjp






and ask NLB

For examplehellip






20141123

13

CSJ



Distribution of CSJ




Himawari



httpgooglnBcPO









misc 10

Total 601

49

23

20

6 2





その他





(F えー) (F e) 116772 271

(F え) (F e) 45665 106

(F ま) (F ma) 44549 104

(F あのー) (F ano) 40695 95

(F あの) (F ano) 33330 77(top 5)

Aozora Bunko

『青空文庫』

20141123

14

Aozora Bunko






httpwwwaozoragrjp



httpgooglRe73C






20141123

9



Copyright solution












ltparagraphgt





ltparagraphgt

ltparagraphgt




ltparagraphgt

Release of BCCWJ





中納言 Chunagon




DVD





httpnlbninjalacjp



tamagokatakana

hiragana kanjikanji


What BCCWJ offers








20141123

10


Japanese corpora


rapidly since 2000







Use corpora



少納言 Shonagon

中納言 Chunagon


CSJ demonstration


Other resources




Workshop

Takehiko Maruyama



18 November 2014




少納言 Shonagon

中納言 Chunagon


CSJ demonstration


Other resources


Contents

(1410 ndash 1545)

BCCWJ

demonstrations


What is this

すいか

西瓜スイカ

20141123

11









Question 7



For examplehellip



買い物買物





tamagokatakana









卵黄 (yolk)


たまご

20141123

12








Question 5

Question 8



For examplehellip

買い物買物

ねこネコ猫

いぬイヌ犬



httpnlbninjalacjp






and ask NLB

For examplehellip






20141123

13

CSJ



Distribution of CSJ




Himawari



httpgooglnBcPO









misc 10

Total 601

49

23

20

6 2





その他





(F えー) (F e) 116772 271

(F え) (F e) 45665 106

(F ま) (F ma) 44549 104

(F あのー) (F ano) 40695 95

(F あの) (F ano) 33330 77(top 5)

Aozora Bunko

『青空文庫』

20141123

14

Aozora Bunko






httpwwwaozoragrjp



httpgooglRe73C






20141123

10


Japanese corpora


rapidly since 2000







Use corpora



少納言 Shonagon

中納言 Chunagon


CSJ demonstration


Other resources




Workshop

Takehiko Maruyama



18 November 2014




少納言 Shonagon

中納言 Chunagon


CSJ demonstration


Other resources


Contents

(1410 ndash 1545)

BCCWJ

demonstrations


What is this

すいか

西瓜スイカ

20141123

11









Question 7



For examplehellip



買い物買物





tamagokatakana









卵黄 (yolk)


たまご

20141123

12








Question 5

Question 8



For examplehellip

買い物買物

ねこネコ猫

いぬイヌ犬



httpnlbninjalacjp






and ask NLB

For examplehellip






20141123

13

CSJ



Distribution of CSJ




Himawari



httpgooglnBcPO









misc 10

Total 601

49

23

20

6 2





その他





(F えー) (F e) 116772 271

(F え) (F e) 45665 106

(F ま) (F ma) 44549 104

(F あのー) (F ano) 40695 95

(F あの) (F ano) 33330 77(top 5)

Aozora Bunko

『青空文庫』

20141123

14

Aozora Bunko






httpwwwaozoragrjp



httpgooglRe73C






20141123

11









Question 7



For examplehellip



買い物買物





tamagokatakana









卵黄 (yolk)


たまご

20141123

12








Question 5

Question 8



For examplehellip

買い物買物

ねこネコ猫

いぬイヌ犬



httpnlbninjalacjp






and ask NLB

For examplehellip






20141123

13

CSJ



Distribution of CSJ




Himawari



httpgooglnBcPO









misc 10

Total 601

49

23

20

6 2





その他





(F えー) (F e) 116772 271

(F え) (F e) 45665 106

(F ま) (F ma) 44549 104

(F あのー) (F ano) 40695 95

(F あの) (F ano) 33330 77(top 5)

Aozora Bunko

『青空文庫』

20141123

14

Aozora Bunko






httpwwwaozoragrjp



httpgooglRe73C






20141123

12








Question 5

Question 8



For examplehellip

買い物買物

ねこネコ猫

いぬイヌ犬



httpnlbninjalacjp






and ask NLB

For examplehellip






20141123

13

CSJ



Distribution of CSJ




Himawari



httpgooglnBcPO









misc 10

Total 601

49

23

20

6 2





その他





(F えー) (F e) 116772 271

(F え) (F e) 45665 106

(F ま) (F ma) 44549 104

(F あのー) (F ano) 40695 95

(F あの) (F ano) 33330 77(top 5)

Aozora Bunko

『青空文庫』

20141123

14

Aozora Bunko






httpwwwaozoragrjp



httpgooglRe73C






20141123

13

CSJ



Distribution of CSJ




Himawari



httpgooglnBcPO









misc 10

Total 601

49

23

20

6 2





その他





(F えー) (F e) 116772 271

(F え) (F e) 45665 106

(F ま) (F ma) 44549 104

(F あのー) (F ano) 40695 95

(F あの) (F ano) 33330 77(top 5)

Aozora Bunko

『青空文庫』

20141123

14

Aozora Bunko






httpwwwaozoragrjp



httpgooglRe73C






20141123

14

Aozora Bunko






httpwwwaozoragrjp



httpgooglRe73C






Corpus Linguistics and NINJAL Japanese Language...2014/11/23 3 History of Japanese Corpus 1 Surveys...

Documents

Transcript of Corpus Linguistics and NINJAL Japanese Language...2014/11/23 3 History of Japanese Corpus 1 Surveys...