An Image Mosaicking Method for Tunnel Lining

Inspection

Cuong Nguyen Kim Mien Truong University of Civil Engineering, Phu Yen, Vietnam

Email: nguyenkimcuong@muce.edu.vn

Kei Kawamura Graduate School of Science Technology Innovation of Yamaguchi University, Ube, Yamaguchi, Japan

Email: kay@yamaguchi-u.ac

Masando Shiozaki Sumitomo Mitsui Construction Co., Ltd., Tokyo, Japan

Email: MasandoShiozaki@smcon.co.jp

Amir Tarighat Department of Civil Engineering, Shahid Rajaee Teacher Training University, Iran

Email: tarighat@srttu.edu

Abstract—The paper proposes an image mosaicking method

for tunnel lining inspection. The conventional methods only

used the cost value of the pixel being processed based on

similarity metric to estimate an image-matching location. To

improve the image-mosaicking efficiency, the curvatures of

the cost curve at candidate matching points are adapted.

Moreover, experimental results for an actual tunnel

demonstrate that the curvature measurement can select the

precise matching points accurately for assisting defect

inspection.

Index Terms—tunnel lining inspection, image mosaicking,

curvature metrics

I. INTRODUCTION

Many concrete infrastructure structures, including

buildings, bridges, and especially tunnels, have been in

use for over 50 years now deteriorating. Therefore, to

improve the efficiency of tunnel inspection work and

obtain more accurate results, the authors propose a

mobile tunnel inspection system (MOTIS) that is capable

of image acquisition with reliable resolution and high

running speed. Its advantages are simple structure,

inexpensive cost, and quality images. Subsequently, an

image stitching technique is applied to generate an

overall view of the tunnel wall at high resolution.

Moreover, many researchers have considered

automated crack detection on the tunnel lining surface

such as Yu et al. [1] and Zhang et al. [2]. However, they

did not detail about tunnel-image mosaicking generation

to assist visual inspection.

Manuscript received March 7, 2019; revised January 8, 2020.

Many concrete infrastructure structures, including

buildings, bridges, and especially tunnels, have been in

use for over 50 years now deteriorating. Therefore, to

improve the efficiency of tunnel inspection work and

obtain more accurate results, the authors propose a

mobile tunnel inspection system (MOTIS) that is capable

of image acquisition with reliable resolution and high

running speed. Its advantages are simple structure,

inexpensive cost, and quality images. Subsequently, an

image stitching technique is applied to generate an

overall view of the tunnel wall at high resolution.

II. EXISTING METHOD

In our previous papers [3], [4], and [5], the authors

reported a video acquisition system and a procedure of

the image stitching for the entire tunnel lining as follows:

A. Video Acquisition System

The video images of the entire tunnel lining surface are

continuously scanned by a video acquisition system

assembled on a car. Fig. 1 shows the system consisting of

six digital video cameras (from V1 to V6), three

illuminators attached to a steel framework shaped to work

within a half of the cross-section of the tunnel, and a car.

This device is able to slide from the side to the top of the

car so that the full tunnel lining surface can be captured

by several passes through the tunnel.

Fig. 2 shows an image stitching procedure for the

entire tunnel lining which involves three steps. Captured

image data are first retrieved from the six video cameras,

each of which contains a number of images depended on

the length of the tunnel. In Step 1, the images for each

camera are stitched into a panoramic image in the

longitudinal direction of tunnel for each camera. In Step 2,

these panoramic images obtained in Step 1 are stitched in

101

International Journal of Structural and Civil Engineering Research Vol. 9, No. 1, February 2020

© 2020 Int. J. Struct. Civ. Eng. Res.doi: 10.18178/ijscer.9.1.101-105

the circumferential direction of tunnel for each region

corresponding to each pass. Subsequently, in Step 3, the

connected images of all regions in step 2 are stitched

together to make a layout panorama which consists of a

full view of the entire tunnel lining surface.

Video cameraSONY HDR-CX630V

Illuminator LED light Steel frameSliding from side to top

Inspection CarImage acquisition systemcamera angle of view between 5o- 6o

Figure 1. Image acquisition system.

Figure 2. Procedure of automatic image-mosaicking construction.

B. Case Study

The tunnel inspected in this experiment is an actual

single-core circular tube with the length of 230 m, the

width of 10.25m, and the height of 4.7m in Yamaguchi

Prefecture, Japan. The video digital cameras are SONY

HDR-CX630V, with LED light 600lx and camera angle

of view between 50- 6

0. The distance between the

cameras and the tunnel wall is 3m. The data set acquired

by each camera consists of 1,558 images with a

resolution of 1920 × 1080 pixels for each picture. It is

useful to use a photographic laser distance meter to

maintain the constant distance between the tunnel lining

and the inspection car to ensure the constant resolution.

The speed of the inspection car is maintained at 30 5

km/h corresponding to the desired image motion quantity

from 500 to 700 pixels. Further, a prototype software

written in C++ was developed to implement the full

image stitching algorithm.

III. PROPOSED METHOD

A. Image Matching Procedure

Fig. 3 shows an automatic image-matching process of

two consecutive images in the tunnel longitudinal

direction (X-axis). Here, the image 1 is the referenced

image, and the image 2 is the registered image of the

image matching process. A search area is set in advance.

Each movement step of the search point is with respect to

the location which enables the image 2 to be shifted on

image 1 to find the appreciate image-matching location

by measuring the similarity and sharpness in the

brightness of all pixel pairs in the overlapped region of

the two images. Furthermore, to accelerate the search and

measurement process, the search point and similarity

metric are skipped in the search area and overlapped

region with predefined values, respectively.

In this study, matching cost functions to measure the

most intuitive similarity of the pixel intensity values

within the overlapped region are used. Those are the most

popular functions such as the sum of square difference

(SSD), and the sum of absolute difference (SAD). More

complex measure about angle correlation such as the

normalized cross correlation (NCC) and Zero mean

normalized cross correlation (ZNCC) get score in the

interval [-1, 1] are used to measure intuitive similarity of

the color pixel-wise pair in terms of brightness in the

overlapped region, as in Eqs. (1) to (4):

2

1 2 ( , ){ , , }[( )]

( 1) ( 1)

M N

i jR G Bi m j n

I I

SSDN n M m

(1)

{ , , }1 2 ( , )| ( ) |

( 1) ( 1)

R G B

M N

i j

i m j n

I I

SADN n M m

(2)

1( , ) 2( , ){ , , }

2 2

1 ( , ) 2 ( , )

M N

i j i jR G Bi m j n

M N M N

i j i j

i m j n i m j n

I I

NCC

I I

(3)

1( , ) 1 2(i, j) 2

2 2

1(i, j) 1 2(i, ) 2

(I ).(I )

( ) . ( )

M N

i j

i m j n

M N M N

j

i m j n i m j n

I I

ZNCC

I I I I

(4)

In these equations, I1 and I2 are the intensity of pixels

at coordinates (i, j) of images 1 and 2 in the overlapped

region, respectively; SAD is the sum of difference

between the pixel values of images 1 and 2 for each color

channel (R, G, and B).

(Y-axis)

Running direction

(X-axis)

Image 1

Image 2

Search Point

(k, l )

-50

50

0810

Overlapped

region

Figure 3. Searching process of image-matching location

102

International Journal of Structural and Civil Engineering Research Vol. 9, No. 1, February 2020

© 2020 Int. J. Struct. Civ. Eng. Res.

However, the standard SAD and SSD functions have

poor performance because image data are attributable to

radiometric distortion and noise. For that the authors

propose modified SAD and SSD functions as follows: The

numerators of Eqs. (1) and (2) are divided by the area of

the overlapping region (the total number of pixels) to

normalize the overlapping region in each measurement.

Furthermore, (m, n) and (M, N) are as the lower left and

upper right coordinates (pixel) of the overlapped region

of the image pairs, respectively. In these functions, the

higher similarity of the overlapped region of two images

yields the smaller score of SAD, SSD.

Conversely, NCC, and ZNCC are determined from the

correlation between the pixel intensity values in the

overlapped region. The larger value of them yields the

higher similarity. These measure is robust than distance

metrics because they are invariant to linear brightness and

contrast variations. However, they have more complex

calculations of division, multiplication and square root.

Therefore, their computation time is more than SAD, SSD.

Advantage over NCC, ZNCC is immune to intensity

distortion. But it is computationally more expensive than

NCC.

The traditional similarity metrics are affected by

artifacts such as non-uniformed brightness, periodic

structures, featureless, and noises. These factors result in

the image-matching error. Therefore, in some cases, the

global minimum cost value of the similarity metric

doesn’t yield well corresponding to the ground truth (G-T

image mosaicking). Otherwise, the local minimum cost is

coincident with the G-T image mosaicking shown in Fig.

4. The authors consider to the local minimum cost.

Global minimum cost

local minimum cost

Figure 4. Cost space of SAD metric for a (31-312) pair in camera 1.

B. Curvature Metric with Full Search Method

The conventional similarity metrics only exploited the

information of the matching point of the pixel being

processed without considering the information of

neighbour pixels. Therefore, the results of similarity

metrics frequently cause to many error ratio. The authors

propose a direct curvature metric method based on

density gradient of the neighbor pixels on full search area

as the following equations:

4

(k,l)

0 1 0 (k , l)

1 4 1 (k, l ) (k, l) (k, l )

0 1 0 (k , l)

C

CUR C C C

C

(5)

(k ,l ) (k ,l) (k ,l )

8(k,l) (k ,l) (k,l) (k , )

(k ,l ) (k ,l) (k ,l )

1 1 1

1 8 1

1 1 1

l

l

C C C

CUR C C C

C C C

(6)

4

(k-1,l)

4 4 4 4(k,l) (k,l-1) (k,l) (k,l+1)

4(k+1,l)

1=

5

CUR

CNP CUR CUR CUR

CUR

(7)

(k 1,l 1) (k 1,l) (k 1,l 1)

8(k,l) (k 1,l) (k,l) (k 1, )

(k 1,l 1) (k 1,l) (k 1,l )

1

9l

CUR CUR CUR

CNP CUR CUR CUR

CUR CUR CUR

(8)

where C(k,l) is a cost value of the central pixel using cost

functions ; () is the skipped pixel number or a size of

search window; CUR4

(k,l) and CUR8(k,l) are the curvature

metric value refer to the 4-connected and 8-connected

neighborhood pixels at the (k,l) coordinates, respectively;

CNP4(k,l) and CNP

8(k,l) are the average values of the

curvature metrics between the central pixel and neighbor

pixels refer to the 4-connected and 8-connected

neighborhood pixels at the (k,l) coordinates, respectively.

The larger CUR and CNP value mean higher matching

possibility.

In most of image matching cases, the curvature metric

yields the correct image matching point result shown in

Fig. 5. However, in some special cases, the direct CUR is

also affected by the outliers shown in the following Fig. 6.

Therefore, to eliminate the outliers, the search-strip

method is adopted [6].

Figure 5. The correct matching point using direct curvature metric in the

cost space.

103

International Journal of Structural and Civil Engineering Research Vol. 9, No. 1, February 2020

© 2020 Int. J. Struct. Civ. Eng. Res.

TABLE I. ERROR RATE (%) OF THE SIMILARITY METRICS

Camera

SAD SSD NCC ZNCC

Error rate

Running

time Error rate

Running

time Error rate

Running

time Error rate

Running

time

V1 63.50 62 40.99 65 42.59 78 39.28 88

V2 45.38 60 32.39 63 29.63 77 27.45 139

V3 46.47 62 37.27 66 32.84 82 29.97 140

V4 33.18 64 20.78 68 20.85 83 18.99 138

V5 39.00 64 23.48 67 21.55 83 19.12 138

V6 37.66 64 41.23 63 41.94 82 41.71 139

Average 44.20 62.67 32.69 65.33 31.57 80.83 29.42 130.33

IV. PERFORMANCE EVALUATION

A Evaluation of Image Matching Accuracy Based on

the Parameters of the Video Acquisition System

The speed of the inspection car is set at about 30 km/h,

with possible variation in the range 25 to 35 km/h with

respect to the reliable range of image motion quantity of

[500 700] pixel. The error rate is computed from the

following equation (9):

1

1100%

0 if IMQ [500 700]

1

N

i

i

i

i

AME IMEN

IMEOtherwise

(9)

where N is a number of images in each camera, IMEi is

ith image matching error. IMQi is image motion quantity

at ith measurement; and AME is the accumulative error

rate for each camera. This equation indicates that an IMQ

value is out of range [500 700] pixels considered as an

IME.

Effected by outliers

Figure 6. The incorrect matching point using direct curvature metric in the cost space.

From results of the Table I, the average matching

errors of SAD, SSD, NCC, and ZNCC are respective

around 44.2%, 62.7%, 31.6% and 29.42%. Those

locations where the IMQ has below 500 pixels result from

lacking information variance of the tunnel wall.

Additionally, those locations where IMQ has over 700

pixels are out of search range.

Comparing among the four methods, the error ratios of

SSD, NCC, and ZNCC are almost same. The ZNCC

method is a little better than the others. However, its

running time is the longest of the others with 130.33

minutes, around time cost double.

The SAD method gets the best of time cost, but the

worst of accuracy image stitching in this case. The SSD

method results in higher accuracy than NCC and SAD

methods with average time cost is 65.33 minutes. This

time is faster than the one of NCC matching cost function

but slower than the running time of SAD matching cost

function. SAD metric is too sensitive with noise or outlier

than SSD metric. As shown in Fig. 7, the result of the

SAD metric chooses the location 1 to match. The location

1 is an example of the overlapped region of two images.

In contrast, the result of the SSD metric chooses the

location 2 to match.

Fig.3.1 Comparison of similarity measurement between SAD and SSD.

36 31 27

261 29 26 24

30 25 33

Location: 2

50 5 10

256 75 13 25

15 60 3

Location: 1

<

1296 961 729

7693 841 676 576

900 625 1089

2500 25 100

5625 169 625 12878

225 3600 9

>

SAD

SSD

Figure 7. Comparison of the similarity measurement between SAD and SSD.

104

International Journal of Structural and Civil Engineering Research Vol. 9, No. 1, February 2020

© 2020 Int. J. Struct. Civ. Eng. Res.

The subtraction values of the blocks in the overlapped

region at the location 2 have a less difference than the

ones at the location 1.

The Table II shows comparison results of the proposed

curvature metrics for six cameras before refinement is

applied. As can be seen, all of CUR and CNP metrics

have error ratio very small. Comparing among the results

of CUR metrics, CUR4 is the highest accuracy with the

average accuracy of six cameras of 1.95%. Otherwise,

CNPstrip has the best results of the error ratio of 0.52%

compared to the others [6].

TABLE II. ERROR RATE OF THE CURVATURE METRICS

Cam-era

CUR4

CNP4

CUR8

CNP8

CUR strip

[6]

CNPstrip

[6]

V1 1.09 0.90 0.96 0.83 0.77 0.45

V2 0.45 0.39 0.83 0.39 0.77 0.26

V3 0.71 0.64 2.12 0.90 1.67 0.26

V4 0.90 0.90 1.16 0.96 1.99 0.83

V5 0.45 0.45 0.77 0.51 1.60 0.52

V6 8.10 7.52 19.78 4.50 5.80 0.77

Aver

age 1.95 1.80 4.27 1.35 2.10 0.52

V. DISCUSSION

The methods of similarity metrics only exploited the

cost value of the pixel being processed, such that its

image matching accuracy wasn’t high. However, the

curvature metric at the nearest candidates and the

candidate being processed yielded the precise matching-

point. The main advantage of the curvature metric using

search strip was to eliminate the artifacts of the neighbor

pixels surrounding the candidate matching point before

measuring the sharpness at the peaks of the cost curve.

Furthermore, a huge number of the raw images of

tunnel lining surface were matched automatically with the

high accuracy no using pre-processing step.

VI. CONCLUSIONS

The main contributions of the paper comprised of

introducing an imaging device capable of scanning

pictures of the entire tunnel surface.

Comparing the image motion quantity between the

result of similarity metrics and curvature metrics, the

results of curvature metrics are improved significantly.

Indicating the cost space of the similarity and

curvature metrics, the authors propose the curvature

metric to find the precise-matching points for image

mosaicking and assisting tunnel lining inspection.

CONFLICT OF INTEREST

The authors declare no conflict of interest.

AUTHOR CONTRIBUTIONS

Cuong Nguyen Kim conducted research and wrote

paper; Kei Kawamura contributed methodology;

Masando Shiozaki supplied image acquisition device;

Amir Tarighat checked and wrote paper; all authors had

approved the final version.

ACKNOWLEDGMENT

The research in this article is supported by the financial

support of JSPS KAKENHI Grant Number 15K0618.

REFERENCES

[1] S. Yu, J. H. Jang, C. S. Han, “Auto inspection system using a

mobile robot for detecting concrete cracks in the tunnel,” Automation in Construction 16, pp. 255-261, 2007.

[2] W. Zhang, Z. Zhang, D. Qi, Y. Liu, “Automatic crack detection

and classification method for subway tunnel safety monitoring,” Sensors, vol. 14, pp. 19307-19328, 2014.

[3] J. Byrne, F. D. Laefer, “Maximum feature detection in aerial unmanned aerial vehicle datasets,” Journal of Applied Remote

Sensing, vol. 11, 2017.

[4] K. Kawamura, M. Koga, J. Matsumoto, M. Shiozaki, and S. Sawamura, “A study on panoramic images generation of tunnel

wall,” Japan Society of Civil Engineers, vol. 71, no. 2, pp. 142-151, 2015.

[5] K. Kawamura, M. Koga, J. Matsumoto, M. Shiozaki, and S.

Sawamura, “Tunnel wall panoramic images generation using a local search algorithm,” Japan Society of Civil Engineers, vol. 72,

no. 2, pp. 73-82, 2016. [6] C. N. Kim, K. Kawamura, A. Tarighat, M. Shiozaki, “An image

matching method based on the curvature of cost curve for

producing tunnel lining panorama,” journal of JSCE, vol. 6, no. 1, pp. 78-90, 2018.

Copyright © 2020 by the authors. This is an open access article

distributed under the Creative Commons Attribution License (CC BY-

NC-ND 4.0), which permits use, distribution and reproduction in any medium, provided that the article is properly cited, the use is non-

commercial and no modifications or adaptations are made.

Cuong Nguyen Kim is currently a PhD candidate in department of information science

and engineering at Yamaguchi University, Japan.

He has considered research on image processing, meta heuristics, and the assessment of concrete

structure conditions.

Kawamura Kei is an Associate Professor in department of information

science and engineering at Yamaguchi University, Japan. He received

his PhD degree in system engineering from Yamaguchi University, Japan, in 2000. His research areas are structural engineering, and

maintenance management engineering. For many years, he has developed methods for bridge management systems and the evaluation

of concrete structure conditions.

Amir Tarighat is an Associate Professor in department of civil

engineering at Shahid Rajaee Teacher Training University, Tehran, Iran.

His research interests consist of concrete technology, durability-based design, structural health monitoring and simulation/modeling of

cementitious materials structure from nano to macro scale.

Masando SHIOZAKI is currently a PhD candidate at Okayama

University and belonged to Sumitomo Mitsui Construction Company.

He has considered image processing, crack detection, inspection device of bridges and tunnels, and the assessment of concrete structure

conditions.

105

International Journal of Structural and Civil Engineering Research Vol. 9, No. 1, February 2020

© 2020 Int. J. Struct. Civ. Eng. Res.