文本信息检索 —— 文本操作

南京大学多媒体研究所Multimedia Computing Institute of NJU

文本信息检索——文本操作

武港山Tel : 83594243

Office: 蒙民伟楼 608B

Email : [email protected]

南京

大学

多媒

体研

究所

Mul

tim

edia

Com

puti

ng I

nsti

tute

of

NJU

23/4/19 Wu Gangshan: Modern Information Retrieval 2

信息检索系统的体系结构

文本数据库

数据库管理建索引

索引

提问处理

搜索

排序排序后的文档

用户反馈

文档处理

用户界面

检出的文档

用户需求

文档

提问

逻辑视图

倒排文档

查询语言和查询处理

索引和检索

文本处理

具体应用系统

(clir,QA,Web)

南京

大学

多媒

体研

究所

Mul

tim

edia

Com

puti

ng I

nsti

tute

of

NJU


内容提要文档预处理文档分类文档聚类文档摘要文档压缩

南京

大学

多媒

体研

究所

Mul

tim

edia

Com

puti

ng I

nsti

tute

of

NJU


文档预处理TokenizationStopword removalLemmatization[ 词性还原 ]Stemming [ 词干 ]Metadata and markup languages

南京

大学

多媒

体研

究所

Mul

tim

edia

Com

puti

ng I

nsti

tute

of

NJU


Simple Tokenization

Analyze text into a sequence of discrete tokens (words).

Sometimes punctuation (e-mail), numbers (1999), and case (Republican vs. republican) can be a meaningful part of a token. However, frequently they are not.

Simplest approach is to ignore all numbers and punctuation and use only case-insensitive unbroken strings of alphabetic characters as tokens.

More careful approach: Separate ? ! ; : “ ‘ [ ] ( ) < > Care with . - why? when? Care with …

南京

大学

多媒

体研

究所

Mul

tim

edia

Com

puti

ng I

nsti

tute

of

NJU


Punctuation（标点符号）Ne’er: use language-specific mappings to

normalizeState-of-the-art: break up hyphenated

sequence.U.S.A. vs. USA a.out

南京

大学

多媒

体研

究所

Mul

tim

edia

Com

puti

ng I

nsti

tute

of

NJU


Numbers

3/12/91Mar. 12, 199155 B.C.B-52100.2.86.144

Generally, don’t index as text

南京

大学

多媒

体研

究所

Mul

tim

edia

Com

puti

ng I

nsti

tute

of

NJU


Case folding

Reduce all letters to lower case exception: upper case in mid-sentence

e.g., General Motors Fed vs. fed SAIL vs. sail

南京

大学

多媒

体研

究所

Mul

tim

edia

Com

puti

ng I

nsti

tute

of

NJU


Tokenizing HTML

Should text in HTML commands not typically seen by the user be included as tokens? Words appearing in URLs. Words appearing in “meta text” of images.

Simplest approach is to exclude all HTML tag information (between “<“ and “>”) from tokenization.

南京

大学

多媒

体研

究所

Mul

tim

edia

Com

puti

ng I

nsti

tute

of

NJU


Stopwords It is typical to exclude high-frequency words (e.g. function

words: “a”, “the”, “in”, “to”; pronouns: “I”, “he”, “she”, “it”). Stopwords are language dependent For efficiency, store strings for stopwords in a hashtable to

recognize them in constant time. Simple Perl hashtable for Perl-based implementations

How to determine a list of stopwords? For English? – may use existing lists of stopwords

E.g. SMART’s commonword list (~ 400) WordNet stopword list

For Spanish? Bulgarian?

南京

大学

多媒

体研

究所

Mul

tim

edia

Com

puti

ng I

nsti

tute

of

NJU


Lemmatization（同义异形） Reduce inflectional/variant forms to base form Direct impact on VOCABULARY size E.g.,

am, are, is be car, cars, car's, cars' car

the boy's cars are different colors the boy car be different color

How to do this? Need a list of grammatical rules + a list of irregular words Children child, spoken speak … Practical implementation: use WordNet’s morphstr function

南京

大学

多媒

体研

究所

Mul

tim

edia

Com

puti

ng I

nsti

tute

of

NJU


Stemming Reduce tokens to “root” form of words to recognize

morphological variation. “computer”, “computational”, “computation” all reduced to

same token “compute” Correct morphological analysis is language specific

and can be complex. Stemming “blindly” strips off known affixes

(prefixes and suffixes) in an iterative fashion.for example compressed and compression are both accepted as equivalent to compress.

for exampl compres andcompres are both acceptas equival to compres.

南京

大学

多媒

体研

究所

Mul

tim

edia

Com

puti

ng I

nsti

tute

of

NJU


Porter Stemmer Simple procedure for removing known

affixes in English without using a dictionary. Can produce unusual stems that are not

English words: “computer”, “computational”, “computation” all

reduced to same token “comput” May conflate (reduce to the same token)

words that are actually distinct. Not recognize all morphological derivations.

南京

大学

多媒

体研

究所

Mul

tim

edia

Com

puti

ng I

nsti

tute

of

NJU


Typical rules in Porter

sses ss ies iational ate tional tion

See class website for link to “official” Porter stemmer site Provides Perl, C ready to use implementations

南京

大学

多媒

体研

究所

Mul

tim

edia

Com

puti

ng I

nsti

tute

of

NJU


Porter Stemmer Errors

Errors of “comission”: organization, organ organ police, policy polic arm, army arm

Errors of “omission”: cylinder, cylindrical create, creation Europe, European

南京

大学

多媒

体研

究所

Mul

tim

edia

Com

puti

ng I

nsti

tute

of

NJU


Other stemmers

Other stemmers exist, e.g., Lovins stemmer http://www.comp.lancs.ac.uk/computing/research/stemming/general/lovins.htm

Single-pass, longest suffix removal (about 250 rules)

Motivated by Linguistics as well as IRFull morphological analysis - modest benefits

for retrieval

南京

大学

多媒

体研

究所

Mul

tim

edia

Com

puti

ng I

nsti

tute

of

NJU


Metadata

On Metadata Often included in Web pages Hidden from the browser, but useful for indexing

Information about a document that may not be a part of the document itself (data about data).

Descriptive metadata is external to the meaning of the document:

Author Title Source (book, magazine, newspaper, journal) Date ISBN Publisher Length

南京

大学

多媒

体研

究所

Mul

tim

edia

Com

puti

ng I

nsti

tute

of

NJU


文本的特性Text properties

Distribution of words in language Why are they important?

Zipf’s LawHeap’s Law

南京

大学

多媒

体研

究所

Mul

tim

edia

Com

puti

ng I

nsti

tute

of

NJU


Statistical Properties of Text

How is the frequency of different words distributed?

How fast does vocabulary size grow with the size of a corpus?

Such factors affect the performance of information retrieval and can be used to select appropriate term weights and other aspects of an IR system.

南京

大学

多媒

体研

究所

Mul

tim

edia

Com

puti

ng I

nsti

tute

of

NJU


Word Frequency

A few words are very common. 2 most frequent words (e.g. “the”, “of”) can

account for about 10% of word occurrences.Most words are very rare.

Half the words in a corpus appear only once, called hapax legomena (Greek for “read only once”)

Called a “heavy tailed” distribution, since most of the probability mass is in the “tail”

南京

大学

多媒

体研

究所

Mul

tim

edia

Com

puti

ng I

nsti

tute

of

NJU


Sample Word Frequency Data(from B. Croft, UMass)

南京

大学

多媒

体研

究所

Mul

tim

edia

Com

puti

ng I

nsti

tute

of

NJU


Zipf’s Law

Rank (r): The numerical position of a word in a list sorted by decreasing frequency (f ).

Zipf (1949) “discovered” that:

If probability of word of rank r is pr and N is the total number of word occurrences:

)constant (for kkrf

1.0 const. indp. corpusfor Ar

A

N

fpr

南京

大学

多媒

体研

究所

Mul

tim

edia

Com

puti

ng I

nsti

tute

of

NJU


Zipf and Term Weighting

Luhn (1958) suggested that both extremely common and extremely uncommon words were not very useful for indexing.

南京

大学

多媒

体研

究所

Mul

tim

edia

Com

puti

ng I

nsti

tute

of

NJU


Predicting Occurrence Frequencies

By Zipf, a word appearing n times has rank rn=AN/n

Several words may occur n times, assume rank rn applies to the last of these.

Therefore, rn words occur n or more times and rn+1 words occur n+1 or more times.

So, the number of words appearing exactly n times is:

)1(11

nn

AN

n

AN

n

ANrrI nnn

南京

大学

多媒

体研

究所

Mul

tim

edia

Com

puti

ng I

nsti

tute

of

NJU


Predicting Word Frequencies (cont’d)

Assume highest ranking term occurs once and therefore has rank D = AN/1

Fraction of words with frequency n is:

Fraction of words appearing only once is therefore ½.

)1(

1

nnD

In

南京

大学

多媒

体研

究所

Mul

tim

edia

Com

puti

ng I

nsti

tute

of

NJU


Occurrence Frequency Data(from B. Croft, UMass)

南京

大学

多媒

体研

究所

Mul

tim

edia

Com

puti

ng I

nsti

tute

of

NJU


Does Real Data Fit Zipf’s Law?

A law of the form y = kxc is called a power law. Zipf’s law is a power law with c = –1 On a log-log plot, power laws give a straight line

with slope c.

Zipf is quite accurate except for very high and low rank.

)log(log)log()log( xckkxy c

南京

大学

多媒

体研

究所

Mul

tim

edia

Com

puti

ng I

nsti

tute

of

NJU


Fit to Zipf for Brown Corpus

k = 100,000

南京

大学

多媒

体研

究所

Mul

tim

edia

Com

puti

ng I

nsti

tute

of

NJU


Mandelbrot (1954) Correction

The following more general form gives a bit better fit:

,, constantsFor )( BPrPf B

南京

大学

多媒

体研

究所

Mul

tim

edia

Com

puti

ng I

nsti

tute

of

NJU


Mandelbrot Fit

P = 105.4, B = 1.15, = 100

南京

大学

多媒

体研

究所

Mul

tim

edia

Com

puti

ng I

nsti

tute

of

NJU


Explanations for Zipf’s Law

Zipf’s explanation was his “principle of least effort.” Balance between speaker’s desire for a small vocabulary and hearer’s desire for a large one.

Li (1992) shows that just random typing of letters including a space will generate “words” with a Zipfian distribution. http://linkage.rockefeller.edu/wli/zipf/

南京

大学

多媒

体研

究所

Mul

tim

edia

Com

puti

ng I

nsti

tute

of

NJU


Zipf’s Law Impact on IR

Good News: Stopwords will account for a large fraction of text so eliminating them greatly reduces inverted-index storage costs.

Bad News: For most words, gathering sufficient data for meaningful statistical analysis (e.g. for correlation analysis for query expansion) is difficult since they are extremely rare.

南京

大学

多媒

体研

究所

Mul

tim

edia

Com

puti

ng I

nsti

tute

of

NJU


Vocabulary Growth

How does the size of the overall vocabulary (number of unique words) grow with the size of the corpus?

This determines how the size of the inverted index will scale with the size of the corpus.

Vocabulary not really upper-bounded due to proper names, typos, etc. （没有上界的原因是名称、打字错等）

南京

大学

多媒

体研

究所

Mul

tim

edia

Com

puti

ng I

nsti

tute

of

NJU


Heaps’ Law

If V is the size of the vocabulary and the n is the length of the corpus in words:

Typical constants: K 10100 0.40.6 (approx. square-root)

10 , constants with KKnV

南京

大学

多媒

体研

究所

Mul

tim

edia

Com

puti

ng I

nsti

tute

of

NJU


Heaps’ Law Data

南京

大学

多媒

体研

究所

Mul

tim

edia

Com

puti

ng I

nsti

tute

of

NJU


Explanation for Heaps’ Law

Can be derived from Zipf’s law by assuming documents are generated by randomly sampling words from a Zipfian distribution.

Heap’s Law holds on distribution of other data Own experiments on types of questions asked by

users show a similar behavior

文本信息检索 —— 文本操作

Documents

Transcript of 文本信息检索 —— 文本操作