(AOI)

Textbook R. Jain, R. Kasturi and B. G. Schunck, Machine Vision, McGraw-Hill Inc., 1995

References 1. M. Sonka, V. Hlavac & R. Boyle, Image Processing,

Analysis and Machine Vision, Brooks/Cole Pub., 1999.

2. E. R. Davies, Machine Vision- Theory, Algorithms,

Practicalities, Morgan Kaufmann Pub., 2005.

3. R. C. Gonzalez and R. E. Woods, Digital Image

Processing, 2nd Ed., Prentice Hall, 2002

4. L. G. Shapiro and G. C. Stockman, Computer Vision,

Prentice Hall, 2001

5. Christopher M. Bishop, Neural Networks for Pattern

Recognition, Oxford University Press, 1995

1

Topics

1.

2. Illumination and shading

3. morphological

operations

4. SMT

5. mura BGA

6. LCD

7. SMTscanner

8.

9.

10. 2.5

11.

2

Chap. 1: Introduction (case study and basic notions)

Chap. 2: Operations on Binary images

PCB Circuit Defects Detection

Chap. 3: Segmentation and Region Representation

Chap. 4: Filtering and Morphology operations on gray

images

Chap. 5: Edge detection

Chap. 6: Contour representations and description

Solder Joints Classification

Chap. 7: Colour

Chap. 8: Depth and 2.5D profilomotry 2.5

Chap. 9: Pattern classification and Object Recognition

Chap. 10: Optics, camera, grabber and calibration

* The above Chapter numbers are the numbers of lecture

note, not the textbook.

3

AOI and Image processing

There are three typical applications of image

processing: Medical image, Video, and AOI

Image

quality

Computation

time

Precision Reliability/

robustness

Medical

image

Poor Hours-days Extremely

high

Extremely

high

Video Poor 30sec/frame Low Low-medium

AOI Best 22sec/4,000

solders

High Extremely

high

Medical Image:

4

Industrial Images

5

6

Chap. 1

1.1A Touch on AOI (by case studies)

This section begins with basic definitions and notions

of CCD image

Followed by brief introduction to industrial application

of AOI

1.1.1Basic definitions and notions

(1) Image acquisition system

Single layer color CCD is not popular in industrial

7

applications due to (a)inferior resolution, (b)distortion

due to interpolation.

Three layer color CCD is still too expensive.

B/W CCD still dominant industrial applications. Hence,

unless stated explicitly, images addressed in this course

are gray images (B/W) and binary images (special

form of B/W images).

Industrial CCD is different from consumer CCD

mainly in: (a)low S/N ratio, (b)low distortion,

(c)external trigger, (d)faster image capturing and

transmission rate.

Machine vision consists of three key technology:

(1)Lighting design to highlight desirable features and

to suppress unwanted details, (2)image processing to

extract characteristic features contained in an image,

and (3)pattern classification to categorize objects in the

image.

(2) Coordinate and matrix associated with an image

CCD camera consists of an array of pixels (such as 8

1024*768), each pixel records the light intensity using

0~255 (28) gray level, with 0 corresponds to black

and 255 corresponds to white.

Hence it is natural to treat a gray image as a matrix:

9

In the above, the ith row and jth column element of a

matrix corresponds to the pixel at the coordinate (i, j),

and the value of the element corresponds to the

intensity of the (i, j) pixel.

Even when a linear camera (line scan camera) is used,

the image can still be transmitted and stored as an array,

hence an image can be treated as a matrix.

(3) Global and local coordinate in AOI measurement

The length/pixel resolution of an image depends on the

amplification factor of the lenses before the CCD chip.

Given the amplification factor and the size a CCD chip,

one can calculate the length/pixel resolution.

Given the length/pixel resolution and the number of

pixels corresponds to a line in an image, one can decide

the true length of this line.

The global coordinate of an AOI system is usually

fixed by linear encoders. The origin of the local

coordinate nof the CCD can be calibrated against the

global coordinate when the system is set up. 10

Hence a machine vision system can be used for:

(1)measuring the length and size of an object, (2)detect

defects of an inspected component, (3)classify the

defects for in process quality control.

1.1.2Illustrative examples of AOI

Three illustrative examples are presented to show some

of most prevailing AOI problems currently encountered

in local industry.

[A] LCD inspection Under proper lighting condition, the image of circuit

and fiducial mark on LCD panel generally has fairly

good contrast, hence binary image can easily be

obtained. This example illustrates some basic steps in

the inspection of binary mages.

STEP 1: fiducial mark and true position of LCD When a board (PCB or LCD panel ) is loaded, there is

11

usually translation and orientation dislocation. To know

these position and orientation error, fiducial marks are

first examined and analyzed to give true position and

orientation (w. r. t. the global coordinate).

Normally the fiducial mark occurs within a

predetermined area with extremely high contrast

against its background.

To determine the true position of a LCD panel,

(1)Move the CCD camera to the nominal position

(expected to be the center of a fiducial mark) and take

out the image of the fiducial mark using a window.

Window size depends on the

precision of the loading

mechanism.

12

(2)Use a threshold to convert the image into a binary

image.

(3)Find out the centroid of the first fiducial mark.

(4) Find out the centroid of the second fiducial mark.

(5)Determine position error and orientation error of

the LCD panel using center positions of two fiducial

marks.

This example is an extremely easy task in AOI because:

(a) the value of the threshold can easily be determined 13

since the fiducial mark is designed and manufactured to

give very good contrast in any lighting condition.

Beside, (2)the image processing is extremely simple

since the fiducial mark is carefully designed to a

regular shape.

STEP 2: measurement of corner length After some examinations, one may one to find out the

length of the corner on the LCD panel.

In this case we need to change lighting condition

and/or proceed some image pre-processing on the

original LCD image to result in good contrast between

the corner, the LCD panel and the background.

The we use edge detector (standard image processing

algorithms) to detect three edges on the corner: 14

Then the three edges (lines) are curve fitted to obtain

their mathematical equations.

Intersection points of three lines are solved from the

equations to give the length of the corner.

In this case, AOI algorithms are basically operated on

gray images, and pre-processing on the gray image

might be required to improve the contrast before edge

detection.

STEP 3: short circuit and open circuit There are printed circuits on IC, PCB and LCD panel.

15

For the purposes of either Qc or repairing, it is often

required to pin point the location of short circuit and

open circuit.

In such applications it is often the case that images with

good contrast are attainable, hence we can easily apply

a suitable threshold to turn the gray image into a binary

image as follows.

If the width of circuits is known, the binary close

operation can be applied to the B/W image in order to

erase all circuit from the image and leaves only black

holes as follows.

16

To detect the white hole (open circuit), we need a

golden template shown below

Comparing the golden template with the test image

given above, the difference is

17

Remove the defects which have been identified as

black holes, it results in white holes.

This case uses binary morphological operations and

golden template. The use of golden template

assumes that the true position of the LCD panel has

been determined before we start the process.

[B] Defect detection of passive R/C components Defects may appear as different shade as follows

(1) Take the component out of its

background use a threshold.

(2) Detect defects by thresholding

18

To take the image of the component out of the original

image, we need to (a)Use a threshold for segmentation,

(b)use morphological operations to fill cavities in the

binary image of the component, (c)define a mask of

the component, (d)take out the gray image of the

component from the original image.

To detect the location of defects (area with a gray level

different from surrounding area), we then apply an

optimal or adaptive thresholding technique.

In this example, we mainly work with gray images with

the aid of operations on binary images.

[C] Defects classification of solder joints In the above examples, we do nothing but detect and

locate defects, and measure their size. However, we did

nothing to classify the defects. To complete such

tasks which require no classification, we need only to

borrow standard algorithms from existing image

processing libraries and use them smartly. 19

However, if the AOI is to serve the purpose of quality

control, we need to go further to classify the defects we

find. For instance, the most challenging task in AOI of

PCB is the classification of solder joints.

20

The types of these defects can easily be distinguished

in color images, but very difficult to tell the differences

from gray images.

Type of

joints

Vertical image Side image

good

open

Tombstone

skew

missing

To solve such problems, (a)carefully define features

that reliably tell differences between types of defects,

21

(b)design lighting condition to highlight the features

in gray images with good contrast, (c)clever image

processing to extract features of the defects, (d)apply

or develop suitable classification method (such as

fuzzy-neural network).

1.1.3: An overview of AOI and machine vision

AOI is a technique that embraces several disciplines:

(a)optics and lighting, (b)image processing, (c)pattern

classification in addition to (d)precision machine and

(e)automation and control.

Precision

machine &

automation

Optics and Lighting Pattern classification

Image processing

AOI

Statistics & QC

22

In regard of a project of AOI, its usually consists of the

following main steps

Define the problem and key features of the image

Design suitable lighting condition to highlight features

Proper pre-processing on gray images for segmentation

Proper pre-processing on gray images for thresholding

Operations on binary image for defect

detection, measurement & feature extraction

Defects classification

Statistical analysis and QC report

Lighting should be used to highlight the features.

Failing in doing so, remaining tasks may be difficult, if

23

possible, to be accomplished by mere image processing

algorithms.

When take time (cycle time for examining a piece) is

critical, we need three more expertise: (f)image

acquisition (CCD camera and lens), (g)image

transmission (grabber card and data bus), (h)software

coding.

1.1.4: Organization of this course

(1) Introduction (case study and basic notions)

(2) Operations on Binary images

(3) Segmentation and Region Representation

(4) Filtering and morphological operations on gray images

(5) Edge detection

(6) Contour representations and texture

(15) Pattern classification and Object Recognition

(9) Lighting, luminance and Shading

(10) Color images

(11) Depth and 2.5D profilomotry

(12) Optics, camera, grabber and calibration 24

1.3Fundamental Concepts and Terminology 1.3.1Image geometry (1.4)

x

Chip object y x P(x, y z)

r

z

P f

Camera lens On the left is the CCD chip (blue lines), in the center

the coordinate (x, y) marks the position of the CCD

camera lens with a focal length f, and P is a point on

the surface of an object in the real world, while P on

the left be its image on the CCD chip.

It can be shown that ' 'f x y

z x y= = , hence

f xz

='x and f yz

='y

where f and z are given 25

1.3.2Sampling and Quantization (1.5)

Sample means true length/pixel, which is initially

dependent on the amplification factor of the lens.

One can perform down sampling to reduce the pixels

of an image. For instance, in the following diagram, the

original image is down sampled by a factor of 2, i.e.,

the gray level of four pixels in the original image is

averaged to give a pixel in the resultant image.

Down

original down sampled

Down sampling results in an image which is smoother

and fuzzier.

The quantization level of an image is the quantization

level of its intensity. For instance, the quantization

level of B/W camera is generally 256 = 28.

26

1.3.3Levels of Computation1.7

There several levels of image processing, beginning

with the lowest level pixel-by-pixel operations, local

operation such as convolution masks, and global

operations (such as histogram analysis).

The higher the level of operation, the more pixels

involves which usually implies higher demand on

memory resources and computation time.

APixel level operation

Pixel level operation involves only one pixel at a time.

Two of the most popular pixel level operations are

thresholding and pixel- to-pixel transformation.

P(x, y) PB(x, y)

T (x, y) (x, y)

TP(x, y) PB(x, y)

27

BLocal level operation

Local level operation involves several pixels in the

original image to determine the gray level of a pixel in

the resultant image.

P(x, y) PB(x, y)

(x, y) T

For instance, in the operation of a convolution mask it

usually involves 9 pixels in the original image to give

the gray level of a pixel in the resultant image, such as

28

3 3

1 1

11 12 13

21 22 23

31 32 33

1, 1 1, 1, 1

, 1 , , 1

1, 1 1, 1, 1

( , ) ( , ) ( , )

i j i j i j

i j i j i j

i j i j i j

Bk l

P i j g i k j l p k l

g g gg

p p pg g

g g g

p p p

p p p

+

= =

+

+ + +

=

= + +

+ + +

+ + +

+

The larger the size of a convolution mask, the more it

demands memory and computation. For instance, a

mask of size demands 25/95 5 3 times resources than a mask. Hence, unless necessary, we usually

uses masks of only the size

3 3

3 3 .

CGlobal level operation

If the output of a certain operations involves all pixels

in an image, such an operation is global. For instance,

calculating the histogram of an image is a global level

operation.

Fortunately, histogram analysis involves only addition

(no multiplication), hence it does not demand too much

computation effort. 29

DObject level operation

An image may contains several objects. To find out the

size, length or other attributes of an object, it may

requires several operations in the object level to draw

all the features required for complete analysis.

30

1.4: Lighting and illumination CCD

CCD camera

CCD camera

PCB after reflow

31

CCD

camera infra-red coating

22 /W 1000

27 2000

50~104 5000

LED

CCD camera

CCD camera

and/or

LED

noise 32

A specular surface

Down

normal

surface 45O+/2

cameranormal surface

45O+/2 camera

normal

surface

33

B

normal surface

C

34

A

B

35

C

36

D

37

E

F

38

GLighting and FOV

The following comments apply equally to a field camera as well as a line camera.

When the front lighting is used, the incident angle of light usually significantly affects the quality of an AOI. However, if the length of a line camera (or a filed camera) is too long (in terms of the true length of the FOV), it will be difficult to create uniform luminance of the object except when the incident angle of light is almost 90o from the horizon. That is, too long a FOV restricts the incident angle of light.

Besides, when the FOV is too long, it also causes serious image distortion.

Practice I: Try to illuminate the same object using

different illumination condition and see how features in the captured image change.

39

1.5Case study -- classifying solder types

This subsection presents a method for classification of

solder joints in order to illustrate how lighting, image

processing and pattern recognition techniques are

integrated to work for an industrial problem in PCB.

1.5.1The need for AOI

Solder joints defects occupies 55% of PCB failure (the

others are 25% wrong polarity, 10% mis-aligned and

10% missing components).

Consider a production line of PC board consisting of

250 SMDs/board. Even under the idea production

condition that defected solder be at a low rate of 100

ppm (parts per million), the first throughput rate is only

90%.

If there exists a economic full inspection method, the

failed 10% PCB can be reworked to significantly

increase the profit. The need for full inspection

To inspect a PCB of 2000 solder joints, human

inspector takes 60~90 mins (25sec by AOI) at a best 40

recognition rate of 8090, which lasts for 4 hours,

then starting to decline until around 60%. While a good

AOI maintains at a recognition rate of above 90% for

24 hrs/day. The need for AOI inspection

1.5.2: History of AOI in solder inspection

M. R. Driels (1990) from Phillips Corp. used diffusive

lighting to study 50 features and selected among them

11 features to arrive at a G/B recognition arte of around

90%, but classification rate is less than 60%.

U. Michigan, Ann Arbor (1988) used diffusive light

and 27 features to arrived at a G/B recognition arte of

around 92% and a classification rate around 88%.

However, both approaches are too time consuming.

Carnegie Mellon University (1990) used 127 point

source light, 4 cameras and 9 features to arrive at a

claimed 97% recognition rate. But it takes 10 mins to

examine 2000 joints (not including scanning time).

Kim (2000) used 3 tiered light and 4 features + neuro-

network to arrived at 99% recognition rate in 4 mins. 41

It is easier to perform G/B inspection

than classification of defect types

Definitions of features and lighting are

keys to success.

1.5.3: Revisit physics for lighting design (2001 summer)

Solder joint obeys specular reflection

42

Simulation for optimal incident angle Insufficient solder Acceptable solder Excessive

20

27

35

1.5.4: Feature definition

1

2

2

1 _ _

1 _ _

1 _ _

III

I

IV

I

IV

I

AIf insufficient solder imageA

AElse If acceptable solder ima

f

f geAAIf excessive solder imageA

>

>

<

f

43

1.5.5: Experimental results (2002 Winter)

Total: 62 joints, mis-classified: 3 joints (4.8%), all on

the boarder line

(a)insufficient or acceptable (b)acceptable or excessive

44

1.5.6: Compared with the best (2003 Spring)

All 62 joints are used

Features X1 and X2 are supposed to divide solder joints

into 2 groups, but in fact they do not.

Recognition rate is 57.62% (compare to 95.16%)

45

46

Take away ambiguous solders

1.5.7: More to think about

Recognition rate depends on testing samples.

Recognition rate repeatability Illumination (lighting) is important, usually leads to

patents.

Features design is key to success (and failure)

Feature definitions have to be obtained by interviewing

customer, who can be very mis-leading.

Physic law beneath the phenomena is important,

without it testing samples can be NOT sufficiently

representative (leading to ill-defined features and

classification criteria), and there will be no way to

double check interviewed information.

47

AOI

AOI

AOI 12

image acquisition and data

transmission3algorithms

patterm recognition4Coding

5

IC TFT-LCD AOI

handle

AOI

AOI

image grabber CodingMill

AOI

48

survey

image grabber

coding Mill

Mill

coding

telecentric lens

10m/pixel coherence

distorsion

Take time image grabber

coding efficiency 49

AOI

algorithms

operators

operators

algorithms

algorithms coding (computation

efficiency)

programming

50