Chapter 5. Probabilistic Models of Pronunciation and Spelling

2007 년 05 월 04 일

부산대학교 인공지능연구실 김민호

Text : Speech and Language ProcessingPage. 141 ~ 189

Outline

Introduction 5.1 Dealing with Spelling Errors 5.2 Spelling Error Patterns 5.3 Detecting Non-Word Errors 5.4 Probabilistic Models 5.5 Applying the Bayesian Method to Spelling 5.6 Minimum Edit Distance 5.7 English Pronunciation Variation 5.8 The Bayesian Method for Pronunciation 5.9 Weighted Automata 5.10 Pronunciation in Humans

Introduction

Introduce the problems of detecting and correcting spelling errors

Summarize typical human spelling error patterns The essential probabilistic architecture:

Bayes Rule Noisy channel model

The essential algorithm Dynamic programming Viterbi algorithm Minimum edit distance algorithm Forword algorithm Weighted automaton

3 / 40

5.1 Dealing with Spelling Errors (1/2)

The detection and correction of spelling error integral part of modern word-processors

Applications in which even the individual letter aren’t guaranteed to be accurately identified Optical character recognition (OCR) On-line handwriting recognition

Detection and correction of spelling errors, mainly in typed text

OCR systems often misread “D” as “O” or “ri” as “n” producing ‘mis-spelled’ words like dension for derision

4 / 40

5.1 Dealing with Spelling Errors (2/2)

Kukich (1992) breaks the field down into three increasingly broader problems: non-word error detection (graffe for giraffe) isolated-word error correction (correcting graffe to giraffe) context-dependent error detection and correction

- there for three, dessert for desert, piece for peace

5 / 40

5.2 Spelling Error Patterns (1/2)

Single-error misspellings - Damerau (1964) insertion: mistyping the as ther deletion: mistyping the as th substitution: mistyping the as thw transposition: mistyping the as the

Kukich (1992) breaks down human typing error Typographic errors (spell as speel) Cognitive errors (separate as seperate)

6 / 40

5.2 Spelling Error Patterns (2/2)

OCR errors are usually grouped into five classes substitutions (e →c) multi-substitutions (m →rn, he →b) space deletions or insertions failures (u →~)

framing errors

7 / 40

5.3 Detecting Non-word Errors

Detecting non-word errors in text is done by the use of dictionary dictionaries would need to be kept small large dictionaries contain very rare words that resemble

misspellings of other words

8 / 40

5.4 Probabilistic Models (1/3)

The intuition of the noisy channel model is to treat the surface form as an instance of the lexical form

to build a model of the channel so that we can figure out how it modified this “true” word and recover it

source of noise variation in pronunciation, variation in the realization of phones,

acoustic variation due to the channel

9 / 40

5.4 Probabilistic Models (2/3)

string of phones (say [ni]) word corresponds to this string of phones consider all possible words P (word | observation) is highest

(5.1) : our estimate of the correct w O : the observation sequence [ni] function argmaxx f(x) : the x such that f(x) is maximized

10 / 40

5.4 Probabilistic Models (3/3)

(5.2) (5.3)

substituting (5.2) into (5.1) to get (5.3) we can ignore P(O). Why?

(5.4) P(w) is called the Prior probability P(O|w) is called the likelihood

11 / 40

5.5 Applying the Bayesian Method to Spelling (1/5)

12 / 40

5.5 Applying the Bayesian Method to Spelling (2/5)

13 / 40

5.5 Applying the Bayesian Method to Spelling (3/5)

p(acress|across) → number of times that e was substituted for 0 in some large corpus of error

confusion matrix a square 26 * 26 table number of times one letter was incorrectly used instead of

another [o,e] in a substitution confusion matrix

- count of times e was substitution for o

14 / 40

5.5 Applying the Bayesian Method to Spelling (4/5)

del[x,y] contains the number of times in the training set that the characters xy in the correct word were typed as x

ins[x,y] contains the number of times in the training set that the character x in the correct word was typed as xy

sub[x,y] the number of times that x was typed as y

trans[x,y] the number of times that xy was typed as yx

15 / 40

5.5 Applying the Bayesian Method to Spelling (5/5)

16 / 40

5.6 Minimum Edit Distance (1/6)

string distance - some metric of how alike two strings are to each other

minimum edit distance - the minimum number of editing operations needed to transform one string into another operation - insertion, deletion, substitution

For example the gap between intention and execution is five operation trace, alignment, operation list (Figure 5.4.)

17 / 40

5.6 Minimum Edit Distance (2/6)

18 / 40

5.6 Minimum Edit Distance (3/6)

Levenshtein distance assign a particular cost or weight to each of operations simplest weighting factor three operation has a cost of 1 Levenshtein distance between intention and execution is 5 alternate version - substitutions has a cost of 2 (why?)

The minimum edit distance is computed by dynamic programming

19 / 40

5.6 Minimum Edit Distance (4/6)

Dynamic programming large problem can be solved by properly combining the

solution to various subproblems minimum edit distance for spelling error correction Viterbi and the forward for speech recognition CYK and Earley for parsing

20 / 40

5.6 Minimum Edit Distance (5/6)

21/ 40

5.6 Minimum Edit Distance (6/6)

22 / 40

5.8 The Bayesian Method for Pronunciation (1/6)

Bayesian algorithm can be used to solve what is often called the pronunciation subproblem in speech recognition

when [ni] occurs after the word I at the beginning of a sentence investigation of the Switchboard corpus produces a total of

7 words the, neat, need, new, knee, to, you (Chapter 4 참고 )

two components candidate generation candidate scoring

23 / 40

5.8 The Bayesian Method for Pronunciation (2/6)

Speech recognizers often use an alternative architecture, trading off speech for storage

each pronunciation is expanded in advance with all possible variants, which are then pre-stored with their scores

Thus there is no need for candidate generation the word [ni] is simply stored with the list of words

that can generate it

24 / 40

5.8 The Bayesian Method for Pronunciation (3/6)

y represents the sequence of phones w represents the candidate word

it turns out that confusion matrices don't do as well for pronunciation the changes in pronunciation between a lexical and surface

form are much greater probabilistic models of pronunciation variation include a

lot more factors than a simple confusion matrix can include

One simple way to generate pronunciation likelihoods is via probabilistic rules

25 / 40

5.8 The Bayesian Method for Pronunciation (4/6)

a word-initial [δ] becomes [n] if the preceding word ended in [n] or sometimes [m]

ncout : number of times lexical [δ] is realized word initially by surface [n] when the previous word ends in a nasal

envcount : total number of times lexical [δ] occurs when the previous word ends in a nasal

26 / 40

5.8 The Bayesian Method for Pronunciation (5/6)

27/ 40

5.8 The Bayesian Method for Pronunciation (6/6)

Decision Tree Models of Pronunciation Variation

28 / 40

5.9 Weighted Automata (1/12)

Weighted Automata simple augmentation of the finite automaton each arc is associated with a probability the probability on all the arcs leaving a node must sum to 1

29/ 40

5.9 Weighted Automata (2/12)

30 / 40

5.9 Weighted Automata (3/12)

31 / 40

5.9 Weighted Automata (4/12)

3 2/ 40

5.9 Weighted Automata (5/12)

3 3/ 40

5.9 Weighted Automata (6/12)

3 4/ 40

5.9 Weighted Automata (7/12)

35 / 40

5.9 Weighted Automata (8/12)

36 / 40

5.9 Weighted Automata (9/12)

37 / 40

5.9 Weighted Automata (10/12)

38 / 40

5.9 Weighted Automata (11/12)

39 / 40

5.9 Weighted Automata (12/12)

40 / 40