imagenes coincidentes.pdf

8/11/2019 imagenes coincidentes.pdf

1/22

DEVELOPMENT AND STATUS OF IMAGE MATCHING

IN PHOTOGRAMMETRY

Armin Gruen ([email protected])

Institute of Conservation and Building Research, ETH Zurich, Switzerland

(Based on a presentation at the Ian Dowman Retirement Symposium, entitledProgress and Prospects for Photogrammetry and Remote Sensing in a Changing World,

held at University College London on 28th June 2010)

Abstract

Image and template matching is probably the most important function in digital

photogrammetry and also in automated modelling and mapping. Many approachesfor matching have evolved over the years, but the problem is still unsolved in general

terms. This paper describes the development of image matching techniques in

photogrammetry over the past 50 years, addresses the results of some empiricalaccuracy studies and also provides a critical account of some of the problems that

remain. Although automated approaches have quite a number of advantages, the

quality of the results is still not satisfactory and, in some cases, far from acceptable.

Even with the most advanced techniques, it is not yet possible to achieve the quality ofresults that a human operator can produce. There is an urgent need for further

improvements and innovations, be it through more powerful multi-sensor ap-

proaches, thereby enlarging the information spectrum, and/or through advancements

in image understanding algorithms, thus coming closer to human capabilities of

reading and understanding image content.

Keywords: DSM generation, empirical tests, image matching, least squares,multi-image, multiple image features

IntroductionProfessorIanDowman was among the firstto propose the use of fully digital systems inphotogrammetry (Dowman, 1984), in this case for topographic mapping from satellite data. Inany of those systems image matching is a crucial function, upon which many other follow-upproducts will depend.

Image matching is a key component of many tasks in photogrammetry, computer vision andimage analysis; it is also crucial to a wide range of applications such as navigation, guidance,automatic surveillance, robot vision, medical image analysis and to the modelling and mappingsciences. For more than 50 years, image matching has been an issue of research, developmentand practical implementation in software systems. Nevertheless, a critical assessment of thecurrent status of image matching shows that the problem has not yet been solved in general terms.

The Photogrammetric Record 27(137): 3657 (March 2012)

DOI: 10.1111/j.1477-9730.2011.00671.x

2012 The Author.The Photogrammetric Record 2012 The Remote Sensing and Photogrammetry Society and Blackwell Publishing Ltd.

Blackwell Publishing Ltd. 9600 Garsington Road, Oxford OX4 2DQ, UK and 350 Main Street Malden, MA 02148, USA.


2/22

This paper aims to describe the major lines of development and achievements as they canbe traced in the mapping community. With a topic so important to many disciplines and the

limited space provided, it is clear that not all developments can be described and given propercredit here.

There is a large body of publications on image matching in the computer vision literature;however, no particular developments, except some basic interest operators or other imageanalysis algorithms, have transferred successfully into photogrammetric systems. Lately,PhotoSynth (Snavely et al., 2006) has become well known for its simultaneous and automatedorientation of hundreds of non-calibrated (Internet) images and its associated derivation ofsparse point clouds. While this is a quite interesting development, the focus of this approach isnot on image matching for surface model generation; indeed no high-quality dense andcomplex surface models have been shown up to now. Some other recent developments,however, show some promising matching results (Hirschmueller, 2008; Vu et al., 2009), albeitthat strictly controlled tests are still missing.

This contribution aims at tracing the development in image matching in photogrammetryfrom the middle of the 1950s until the present day. Since the author and his group ofresearchers have worked in this field for about 30 years and contributed a great number ofpublications, there will be a certain focus on this work. The chronology in developments willbe structured into the following periods: the Early Years (1960s and 1970s); the NewApproaches (1980s); the Time of Consolidation and Extensions (1990s); and finally theTime of Acceptance (2000s).

The discussion on the development and status of image matching must also take into

consideration that this technique is used for a variety of different tasks, with differentprerequisites and expectations. The prime, though not exclusive, applications of imagematching in our fields are for surface generation, tie and control point measurement for

orientation and triangulation, industrial quality control (targeted and non-targeted points),feature (edge) extraction and feature/object tracking. The type, size and quality of images used(satellite, aerial, terrestrial) and the expected accuracies of the results vary greatly.Accordingly, a critical analysis will lead to different results. Therefore, the arguments willbe based on the achievable results and not necessarily on those required by a certainapplication. Research in this context is not only about developing some new methodology, butalso about providing a clear understanding of its properties, which means pushing thismethodology to its performance limits in order to gain an insight into its potential andlimitations.

Three Basic Matching Techniques

Image matching has been a major research issue in computer vision and digitalphotogrammetry for many years; accordingly, many different approaches have evolved. Threebasic matching techniques can be distinguished:

(a) intensity-based;(b) feature-based; and(c) relational.

In intensity-based matching the original, or slightly modified (enhanced), image data isused in the form of a matrix of grey values. The most prominent methods are cross-correlationand least squares matching (LS matching or LSM), which are also calledarea-basedmatching.They provide sub-pixel accuracy, in extreme cases 1/10 pixel and even better. LS matching is a

highly non-linear process and therefore requires very good approximate values.

The Photogrammetric Record

2012 The Author

The Photogrammetric Record 2012 The Remote Sensing and Photogrammetry Society and Blackwell Publishing Ltd 37


3/22

Feature-based matching requires, firstly, the extraction of basic image features, such aspatches, corners, junctions, edges and so on. In a second step, matching is performed between

these features. Features are sometimes, but not always, more stable with regard to reflectancecharacteristics. On the other hand, information that is lost during the feature extraction phasecan no longer be recovered. Some methods provide for sub-pixel accuracy, but not at the levelof the intensity-based methods.

Feature-based matching has been performed with:

(1) relaxation;(2) dynamic programming;(3) robust estimation;(4) cross-correlation; and(5) graph matching.

The solution space may be reduced by constraints such as:

(1) use of epipolar images;(2) use of more than two images;(3) limits on the magnitude of changes in parallax;(4) a priori modelling of objects (coarse description of object);(5) hierarchical coarse-to-fine strategies;(6) best-first strategies, using features sequentially, according to the relevance of their

information content;(7) thin-to-thick or thick-to-thin strategies (either starting with just a few saved

match points as a skeleton and densifying, or starting with a dense point field and

thinning out by blunder detection); and(8) observation of behaviour of parallaxes (inflections are not allowed).

Relational matching uses geometric or other relations between features and structures(combination of features). Correspondence is established by tree-search techniques. Thesemethods are not very accurate but are usually robust; they do not require good approximations.

Their use in digital photogrammetry for digital terrain model (DTM) generation is rather scarce.There are several more or less exhaustive descriptions of these various techniques

available (Lemmens, 1988; Baltsavias, 1991, Chap. 3).

The Early Years (1960s and 1970s)

Image matching was first introduced in the early 1950s. It started as an analogueprocedure using electrical circuits for solving the matching equations (see, for example,Hobrough, 1959). A good survey of the very early efforts of analogue cross-correlation isgiven in Hobrough (1965). It clearly shows that equipment manufacturers were the drivingforce behind the development, rather than university groups. A particularly successful andmuch discussed system was the Gestalt Photomapper GPM I and GPM II (Kelly et al., 1977;Alberich, 1985). In Mikhail et al. (1978), Fred Doyle reports the experiences of the USGS withthe GPM. They used it only with small scale photography (1:80 000) for orthophotoproduction and reported a height accuracy of 5 m. Hobrough (1965) already lists two attemptsat digital correlation (Williams, 1959; DeMeter, 1963). What is amazing is the great optimismthat accompanied these developments. The automated mapping problem was considered to bepractically solved; Hobrough (1965) states: Within the next three to five years many of thepresent automation programs should be completed and several of the conventional

photogrammetric operations will probably be automated on a more or less routine basis.

Gruen. Development and status of image matching in photogrammetry

2012 The Author

38 The Photogrammetric Record 2012 The Remote Sensing and Photogrammetry Society and Blackwell Publishing Ltd


4/22

By the middle of the 1970s, the increasing computational power of digital computersallowed for the fully digital treatment of the matching problem. Helava (1972) already

discussed the epipolar principle in matching and a little later (Helava, 1976, 1978) presented anin-depth study of auto- and cross-correlation based on linear system theory. He listed 11 cross-correlation principles and was critical of the traditional cross-correlation approach, which isinsufficient for photogrammetric stereo applications and requires image data shaping toaccount for the phase differences between different image frequencies and within eachfrequency. The image power spectrum also reveals cases of multiple correlation peaks,whereby the highest peak does not necessarily represent the correct solution. Helava alsoreferred to the new AS-11B-X system, an automated stereo mapper, which used parallelprocessing of neighbouring terrain profiles in order to perform the matching of one aerialmodel within 10 minutes, a speed that is dreamt of even today. For more technical details ofthe system, see Scarano and Brumm (1976). In an interesting variant, Masry (1974) used cross-correlation on an analytical plotter with epipolar constraints for change detection. Dowman andHaggag (1977) also worked on this method.

By the early 1980s, the literature on image analysis and matching had growntremendously. Therefore, only very few publications can be referenced here, most of whichpossess an overview character.

The approaches and achievements within the photogrammetric community of those earlyyears are described in Makarovic (1980), Konecny and Pape (1981) and Baltsavias (1984).Computer vision scientists also had an early and significant impact on matching techniques (asan example, see Baker and Binford, 1981). Overviews and summaries of the state-of-the-art

methods can be found in Aggarwal et al. (1977), Andrews (1978), Bernstein (1978) andChellappa and Sawchuk (1985).

It was obvious for those closely involved in matching problems that the existing

approaches, which were mostly based on cross-correlation, had significant deficiencies. Helava(1976) states that the human operator is far superior and addresses with this remark the lackof suitable image understanding algorithms.

The characteristics of cross-correlation were well understood by the early 1980s and itsdeficiencies were identified:

(1) discrepancy between conjugate images, caused by geometrical distortions (terrainslope, height differences, positional and attitude differences of sensors), radiometricproblems (illumination, reflectance, varying material properties) and imaging arte-facts;

(2) discretisation of trial step size; and(3) lack of good methods for the assessment of results, figure-of-merit (quality of match).

The shortcomings of this class of image matching methods finally, after early excitementand false predictions, caused a slow-down in the development of operational automatedcorrelation systems. This problem was addressed at a panel session at the annual ASPConvention in Denver, Colorado in March 1982. Five short articles in PhotogrammetricEngineering & Remote Sensing (PE&RS, 1983) reflect a part of this discussion.

Cross-correlation cannot respond appropriately to a number of facts that are inseparablyrelated to stereo-images of three-dimensional and sometimes even two-dimensional objects.The conjugate images created under the laws of perspective projection might differconsiderably from each other. Terrain slope, height differences, and positional and attitudedifferences of the sensors cause geometrical distortions. Illumination and reflectance conditionsmight distort the images radiometrically. Under certain circumstances, this may even trigger a

geometrical displacement in the matching. Noise from the electrical components and the


2012 The Author



5/22

sampling rate (pixel size) could also influence both the geometric and the radiometriccorrespondence of the images.

Cross-correlation is very simple conceptually, easy to implement and computationallyfast. The main problem with conventional cross-correlation is that it allows only for two shiftparameters between template and patch. Rotations, scale and other deformations betweentemplate and patch cannot be accommodated. Therefore, the following rules should beobserved when applying cross-correlation: use it only with epipolar images and use small patchsizes.

Cross-correlation works well and is fast if the patches to be matched contain sufficientsignal without too much high-frequency content, and if geometric and radiometric distortionsare kept to a minimum. Both conditions are not often encountered in aerial and terrestrialimages. On the other hand, in satellite images, due to their smaller image scale, these problemsare less influential and cross-correlation is more likely to succeed. Therefore, some efforts weremade very early on to design matching techniques that are more efficient than cross-correlation. Concepts that were suggested by the artificial intelligence community included:first- and second-order derivative matching; relaxation methods; segmentation and graphstructure matching; transform (Hough transform) matching; and feature (edge) matching.Thus, a tendency to switch from area-based to edge-based analysis could be observed.However, Rosenfeld (1984) remarked, in an excellent critical review, that these new methodsalso did not solve the problems referred to above.

It is important to note that the concept of epipolar line matching, for the purpose ofreduction of the matching solution space and thus the number of false matches, was already

proposed by Helava (1972). Panton (1978) clearly expressed the need for epipolar lineconstraints, patch shaping, algorithmic tuning, reliability monitoring and even parallelprocessing, by showing the first implementations of these concepts. He demonstrated that

parallel processing can result in amazing performance: 270 match points per second,35 minutes for a full stereomodel consisting of 562 500 points on a CDC 1700 minicomputer.Compared to the single processor mainframe computer CDC 6400 this gave an increase inspeed by a factor of 34.

In a much later study, Zheltov and Sibiryakov (1997) have shown that cross-correlationcan be modified into a version that takes care of all six parameters of an affine transformationbetween template and patch, albeit at higher computational expense. It also has been shownthat this modified version is equivalent to least squares matching.

New Approaches (1980s)

The 1980s saw a long period of very active development of new, more powerful,matching approaches. Originally, the driving force behind the new developments was theequipment industry. Now researchers from universities took charge. At the same time, thesystem manufacturers started to offer software solutions for digital matching that wereincorporated in photogrammetric equipment, firstly in analytical plotters and later in digitalstations.

Probably the most significant contribution, the least squares matching technique (LSmatching, LSM), was developed in the early 1980s. Due to its flexibility and accuracy, it hasturned out to have a major impact on image matching, with many extensions, and is currentlyused in many digital photogrammetric matching tasks.

Early investigations were reported by Forstner (1982), Ackermann (1984) and Pertl(1984). The author also investigated this technique in 1982 as adaptive least squares matching

(ALSM). The method was called adaptive because it can be executed in a self-tuning mode,


2012 The Author



6/22

meaning the parameter set to be estimated can be corrected automatically in order to obtain amost appropriate estimation model set-up with respect to the specific signal content of the

patches to be matched.Early concepts and software were developed in a project funded by Helava Associates,

Inc. Tests were performed with synthetic and real images. The method was found to be of greatpotential for a variety of image- and template-matching problems (Baltsavias, 1984; Grun,1984; Gruen and Baltsavias, 1984).

If properly used, least squares matching combines the advantages of area-based and edge-based matching. The basic equations are set up in the context of a statistical estimation model.The estimation itself is performed as least squares estimation. The familiar apparatus of theleast squares approach with respect to parameter estimation and hypothesis testing can befavourably utilised. Precision and reliability measures are readily available and allow anassessment of the quality of the match in a better way than is feasible with other matchingtechniques. Algorithmic, computational and numerical aspects can also be studied in a well-known environment.

ALSM has great potential in different respects, as recognised from the very beginning andreiterated here:

(a) high matching accuracy;(b) geometrical/stochastical constraints: stabilisation, reliability, speed;(c) multi-image matching (reliability);(d) simultaneous matching/point positioning;(e) multi-patch matching: neighbourhood conditions;(f) multispectral, multitemporal matching;

(g) monitoring of quality (precision, reliability);(h) simultaneous image reshaping, radiometric adjustment;

(i) combination of area-based and edge-based analysis;(j) usable in hierarchical mode (coarse-to-fine);(k) usable as derivative-operator-based matching procedure (first-order slope vari-

ables, second order);(l) rule-based matching: patch selection (good signal content);(m) incomplete data patches (for example, triggered by occlusions);(n) computational performance: parallel implementation possible;(o) usable for pattern recognition (template matching), feature extraction, image feature

measurement (fiducials, tie points, control points), change detection, line following;and

(p) general matching technique (beyond images): DTM/DSM analysis/co-registration,

image/map registration.

A comprehensive description of the basic algorithm and its multiphoto geometricallyconstrained (MPGC) extension and many results are given in Baltsavias (1991).

The quality of a matching procedure depends mainly on the type and content of the imagesignal. Given the images of an object, there is not much room for improvement of the signal.Very often there is, however, additional information available that could support the matching.Important categories of information are geometrical and radiometric conditions. They relate tothe imaging geometry of the sensor, orientation and positional data of the sensor, image featureradiometry and geometric constraints, and to object constraints. They have to be set up as linearor linearised observation equations in the least squares context and are, as such, added to theobservation equations for the grey values of the pixels. The resulting hybrid system is of the

combined adjustment type. It leads not only to a much improved matching procedure but, in


2012 The Author



7/22

addition, provides for a simultaneous matching/object point positioning technique. Thus, thetwo-stage process of image measurement object positioning is replaced by a one-stage

solution that is capable of utilising all available radiometric and geometric information at once.The MPGC technique offers considerable advantages with respect to precision and

reliability. Because of the use of all the geometric information available and the internalconsistency of the algorithm, the success rate increases and many problematic situations aresignalised:

(1) The use of the geometrical constraints increases the convergence radius and ratebecause the search is one-dimensional. The use of multiple scenes has the sameeffect. In addition, the search is constrained with respect to direction and step size, sothat image patches subject to small displacements can support those that are subjectto larger displacements.

(2) In many cases, occlusions do not prevent correct convergence. The less occludedpatches beneficially influence those that are more occluded. The quality measures of

the algorithm allow for the detection of occlusions.(3) Multiple solutions are drastically reduced because of the conditional one-

dimensional search. Mismatches can be detected unless all image patches hit falsemaxima along epipolar lines simultaneously, which would be a very rare case.

Other extensions are summarised in Gruen (1996b). In the following some modificationsand extensions are addressed, which make this approach so powerful and usable in a variety ofdifferent forms. In particular, the following stages of algorithmic development aredistinguished:

(1) Stereo (two-image) adaptive least squares matching (ALSM) (Gruen, 1985b).(2) Multiphoto geometrically constrained (MPGC) matching(Gruen, 1985a; Gruen and

Baltsavias, 1985, 1988b; Baltsavias, 1991):(a) collinearity constraint;

(b) forward intersection constraint (interior and exterior orientations known;X,Y, Zestimated simultaneously);

(c) epipolar constraint (interior and exterior orientations known;X,Y,Zderived in aseparate step); and

(d) bundle constraint (interior and exterior elements are simultaneously estimatedwith X, Y, Zcoordinates).

(3) Digital surface model (DSM) constraints:(a) XYconstraint (Gruen, 1985a; interior and exterior elements known; given X, Y,

only Zis estimated simultaneously; this also became known as the vertical line

locus method); and(b) Z(contour) constraint (Gruen, 1985a; Gruen and Baltsavias, 1986; interior and

exterior elements known; givenZ, onlyX,Yare estimated simultaneously; this isequivalent to drawing contours from a stereomodel or a multi-image arrangement).

(4) Image feature constraints:(a) edge constraint (Gruen and Stallmann, 1991).

(5) Globally enforced least squares matching:(a) multiple patch matching;(b) 2D patches (Gruen, 1985a);(c) 3D (volume element) patches (Maas et al., 1994);(d) linear feature extraction with LS template matching (Gruen and Agouris, 1994);

(e) linear feature extraction with LSB-snakes (Gruen and Li, 1997);


2012 The Author



8/22

(f) object-space-oriented LSM (Gruen and Zhang, 2002; Zhang, 2005); and(g) neighbourhood constraints by stochastic relaxation (Gruen and Zhang, 2002;

Zhang, 2005).(6) 3D surface matching:

(a) 3D surface and space curve matching (Gruen and Akca, 2005; Akca, 2007).

A particular type of image feature constraint is the image edge constraint. Whenever edgeshave to be measured, such as breaklines in DTMs, this may constitute an appropriate solution.Essentially MPGC is an area-based matching technique. For high accuracy edge matching, themethod is transformed into a combination of an area-based and feature-based technique. This isachieved by introducing, as a reference template, a synthetic (or real) edge pattern, which is tobe matched with the actual image edges. Compared to the conventional feature-based matchingtechniques, this method does not require the extraction of image edges, but matching is donedirectly by using the original grey value edges. For algorithmic details, see Gruen andStallmann (1991). An efficient automatic measurement procedure can be realised via

implementation of a tracking technique, which tracks the edges either in object or in imagespace.

If larger image regions have to be processed by matching, image patches with low or nosignal content pose a serious problem. For such cases the technique ofglobally enforced leastsquares matching has been developed. Here the basic idea is to establish geometricalneighbourhood conditions between adjacent patches in order to give stabilising support to theweak patches by the strong ones. Ideally, the weak regions would be bridged and a stableglobal solution would be obtained. The first such solutions were introduced as multipointmatchingormulti-patch matching techniques. The introduction of neighbourhood constraints

leads to a simultaneous solution for all patches. Thus, a full image format can be processed inone sweep. Early approaches to this concept have been presented by Gruen (1985a), Rauhala

(1986), Rosenholm (1986) and Li (1989).Another globally enforced technique is object space oriented least squares matching as

introduced by Wrobel (1987), Ebner and Heipke (1988) and Helava (1988b). It represents themost generalised approach to least squares matching. Due to its complexity with respect toimplementation and handling, it has been used only occasionally and under laboratoryconditions (Kempa and Schlueter, 1993).

In the 1980s, image matching was mostly used in close-range applications. Examplesinclude: camera calibration (Beyer, 1987, 1992); human face measurements (Gruen andBaltsavias, 1988a); object tracking (Baltsavias and Stallmann, 1990); and industrial qualitycontrol (Gruen and Stallmann, 1991). Often it was applied in the form of template matching(for example, Lue et al., 1987). Many more tests and applications are reported in Gruen (1988)

and Baltsavias (1991), the latter noting that scientific investigations with aerial imagery wererather scarce. Otto and Chau (1988) demonstrated an early application to SPOT satellite stereo-images using the techniques of region growing. For the results of an ISPRS WGIII/4 test onimage matching, see Guelch (1988).

The many activities in algorithmic development were occasionally also accompanied byaccuracy studies. Schewe and Forstner (1986) reported industrial car measurements on theanalytical plotter Planicomp, with an accuracy of 005 to 02 pixels. Gruen and Baltsavias(1986) derived DTMs from 1:5300 scale aerial images with a height error of 002 to 004%of flying height for natural points. This compares with an accuracy of manual measurementsof signalised points of 0003% of flying height (Trinder, 1986). Rosenholm (1986)processed aerial images at scales between 1:4000 and 1:50 000 with an accuracy of 0 4 to

06 pixels.


2012 The Author



9/22

In summary, the following accuracies have been attained with LSM:

(1) laboratory test, targeted points: 001 to 002 pixels;(2) close-range applications, project conditions, targeted points: 01 to 02 pixels; and

(3) aerial photogrammetry, natural points: 03 to 0

5 pixels.

In parallel to these many research and development efforts, commercial systems cameonto the market. Correlators were implemented on analytical plotters, for example, by Kern inthe form of the vertical line locus method (Bethel, 1986; Almroth and Hendriks, 1987) or leastsquares matching (Pertl, 1984) and on mono-comparators (Helava, 1988a, with least squaresmatching).

From its very beginnings, image matching was not isolated from practice, but alwaysdeveloped with the aim of integrating it into a photogrammetric processing system. Asexplained earlier, these efforts go back to the early 1950s and have been pursued ever since. Inaddition, the ISP (International Society of Photogrammetry) had established a working group onAutomated and Analytical Instruments in 1969, well before analytical plotters entered thecivilian market. In 1976, at the ISP Helsinki Congress, this working group was split into two anda new working group entitled Automated Instruments and Systems was formed. TheInternational Archives of Photogrammetry, Volume 24 (Proceedings of ISPRS TechnicalCommission II Symposium held in Ottawa in 1982) contains a number of papers addressing thisissue. Of particular interest is Case (1982). He describes the concept of a fully digital stationDSCC (digital comparator/correlator) and gives detailed numbers on expected performanceparameters. This is the time when university groups were also investigating fully digital systems

and a number of studies and implementations were revealed (Albertz and Koenig, 1984;Dowman, 1984; Gruen and Beyer, 1986, 1990; Haggren, 1986; Gruen, 1989), although not allwere supported by image matching functions. A system for the processing of SPOT images

was under development by Dowman et al. (1987). Four different image matching algorithmswere tested, with least squares matching finally implemented (Otto and Chau, 1988).

Many efforts in digital photogrammetry at that time were driven by the fact that the USDefence Mapping Agency had launched a large research and development programme with thegoal of executing its mapping operations fully automatically and, thus, fully digitally by theyear 1990. The first commercial photogrammetric digital station was presented as DSP1 byKern at the ISPRS Congress in Kyoto in 1988, but at this time without a matching module(Cogan et al., 1988).

Time of Consolidation and Extension (1990s)

The idea ofglobally enforced matchingwas later generalised such that 3D image data-sets(voxel cuboids) could be matched. The related method was used for the measurement of laser-induced fluorescence flow fields in a technical chemistry application (Maas et al., 1994). Afurther modification was suggested for the extraction of linear features (Gruen and Agouris,1994). This, again, was further developed into LSB-snakes, which combine the powerful toolsof least squares estimation with the determination of energy-minimising functions (Gruen andLi, 1997).

A particularly promising and successful implementation has been presented by Maas(1996). This approach is derived from the concept of MPGC matching. It is a multi-imagematching technique, whereby features are searched for along epipolar lines. The computationaleffort grows exponentially with the number of images, but specific search strategies help tokeep the computing times within reasonable bounds. Compared to MPGC matching, this is a

linear procedure requiring no approximate values and no iterations. On the other hand, it is


2012 The Author



10/22

essentially a searching technique for features whose locations are defined by epipolar lines.There is some room for improvement in the sense that feature attributes could be considered for

matching and/or least squares area-based matching could be done as a last step.

DTM Generation on Digital Workstations

A large number of commercial digital systems emerged in the 1990s, but not all of themsurvived for very long. Only a few have made a commercial impact. Automated DTMgeneration, in particular for ortho-image production, is a major function of digitalphotogrammetric stations. Most systems and approaches work hierarchically with imagepyramids. They use epipolar images rather than the original images and apply either cross-correlation or feature-based techniques. The sampling mode is often on a regular grid in eitherobject space or image space, or based on arbitrarily distributed points. In the last two cases, thedata is often transformed into a regular DTM grid before being presented to the user. In theauthors opinion, this is not an appropriate solution. DTM generation by image matching andDTM interpolation should be separated. Otherwise, the effects of both are inseparablycombined and the user has no indication of the quality of either procedure.

The most popular digital photogrammetric stations at that time had the following DTMsoftware and approaches installed:

(1) Leica/Helava DPW, Automated Terrain Extraction (AATE), cross-correlation(Miller and De Venecia, 1992; Zhang and Miller, 1997);

(2) Zeiss PHODIS ST, Topo SURF (MATCH-T), feature based;(3) Intergraph ImageStation, MATCH-T, feature based (Krzystek, 1995);

(4) PCI Geomatics, DEM extraction;(5) SOCET SET, BAE Systems digital DTM generation; and

(6) VirtuoZo, cross-correlation with reshaping, global matching with probabilisticrelaxation (Zhang et al., 1992, 1996).

The results of studies comparing the performance of different digital stations with respectto DTM generation are referenced in Baltsavias et al. (1996), Gruen (1996a) and Smith andSmith (1996).

In general, most high-end workstations delivered results of similar quality, although theunderlying algorithms, strategies, robustness and ease of use of the software varied. Mostsystems required the user to set a large number of input parameters (up to 28 in a particularsystem). Even small changes in seemingly harmless parameters often led to significantalterations in the results. There was no logical visible or predictable connection between thechange of parameters and the results. Major problems occurred in the case of homogeneoustexture, shadows, dense vegetation (trees), dark and very steep slopes, water features, urbanenvironments, and so on. In Gruen (1996a), the main problems were listed as:

(1) lack of recognition of object edges and geomorphologically important features;(2) no bridging of regions with poor signal content;(3) insufficient handling of occlusions and shadow areas;(4) unreliable reduction of DSM to DTM; and(5) missing quality assessment; no good internal quality control.

In another study (Gruen, 1999), it was shown by empirical tests that the accuracy resultsof automated DTM generation were worse by a factor of five or more compared with thoseobtained from analytical plotters. The results of yet another empirical accuracy study were

published in Gruen et al. (2000). The matching software of three commercial systems was


2012 The Author



11/22

tested with three different aerial image data-sets of different image scales. The rms errorsachieved were worse by factors of 16 to 16 than the theoretical expectations for manual

measurements. This was largely due to very large numbers of blunders. Gong et al. (2000)have also carried out an interesting assessment. The low-quality results were caused by the factthat the capabilities of existing matching algorithms had not been fully utilised. Significantimprovements could be expected through:

(1) use of more than two images for matching;(2) making use of all available geometrical constraints;(3) development of methods for internal quality control;(4) improvement of user interface;(5) explanation of functions of parameter settings;(6) integration of a priori knowledge (such as existing DTMs); and(7) integration of image understanding algorithms and use of multi-sensor data.

Empirical, controlled testing of matching procedures and software, when applied tosurface model generation, is not a simple task. The need to generate reference data-sets with anaccuracy at least three times better than the expected system performance (a conservativenumber) indeed causes some headaches. If a system performance of 03 pixels is assumed thenthis translates, in the case of medium scale aerial images (20 cm footprint), into a requiredreference point accuracy of 2 cm. In the case of close-range applications with a footprint of1 mm this value would be 01 mm. For satellite images there are not such stringentrequirements, although when considering the latest generation with a footprint of 50 cm,

problems with reference data generation are also faced. This situation is complicated by thefact that the matcher will generate hundreds of thousands, or even millions, of points perstereomodel, thus requiring a very large number of reference points. This is surely one of the

reasons why little suitable empirical test data is available, even up to the present time. As aresult it can be said that the matching procedures and software have developed not onlywithout a solid theoretical basis (company approaches are usually not even published), but alsowithout much empirical scientific control. It is up to the individual user to judge the suitabilityof his resultsnot a very comforting situation.

Time of Acceptance (2000 to Now)

Automatic DTM/DSM generation through image matching has gained much attention inrecent years. A wide variety of approaches have been developed and automatic DSMgeneration packages have, in the meanwhile, been commercially available on several digitalphotogrammetric workstations. At the turn of the century, it was still noted that commercialimage matching software did not have the required performance in terms of high-qualityresults. This applies in particular to the processing of aerial images, but also to the recentlyavailable high-resolution stereo satellite images.

Close-range applications are considered by most system manufacturers as niche markets.Therefore, no effort is made to provide suitable software and this area is left entirely to theactivities of academic research groups.

Although the algorithms and the matching strategies of commercial systems may differfrom one another, the accuracy performance and the problems encountered are very similarin the major systems. Furthermore, the performance of commercial image matchers doesnot live up, by far, to the standards set by manual measurements (Gruen, 1996a, 1999;Gruen et al., 2000). The main problems in automated DTM generation are encountered

with:


2012 The Author



12/22

(a) little or no texture;(b) distinct object discontinuities;

(c) local object patches are not sufficient approximations of planar faces;(d) repetitive objects;(e) occlusions;(f) moving objects, including shadows;(g) multi-layered and transparent objects;(h) radiometric artefacts such as specular reflections; and(i) reduction from DSM to DTM.

In the year 2000, the author started development cooperation with Starlabo Inc., Tokyo.As part of a larger software package, a new matching approach and module was developed foran aerial three-line scanner (TLS) system. It was later modified to also be able to handlesatellite images (SAT-PP) and terrestrial close-range cases (CLORAMA). The following refersto descriptions of the TLS system without, however, any restriction in the generality of the

chosen algorithms.The TLS matcher aims to generate DSMs by considering specifically the problems (a) to (f)

above. The matcher is described in detail in several publications (for example, Gruen andZhang, 2002; Zhang, 2005). The raw level TLS images were used together with the given, orpreviously triangulated, orientation elements. After the generation of image pyramids, thematcher uses three kinds of image features, namely, general feature points, edge points and gridpoints. A triangular irregular network (TIN) based DSM is constructed from the matched pointson each level of the pyramid, which in turn is used in the subsequent pyramid level forapproximation and adaptive computation of the matching parameters. Finally, the modified

MPGC matching is used optionally to achieve more accurate results for all the matched features.Among the usual matching techniques, area-based matching (ABM) and feature-based

matching (FBM) are the two main ones applied in automatic DSM generation, but additionallyrelational matching is sometimes used. All basic matching techniques have advantages anddisadvantages with respect to the problems presented above. The key to successful matching isan appropriate matching strategy, making use of all available and explicit knowledgeconcerning the sensor model, network structure and image content. However, even then thelack of an image understanding capability will lead to problems, whose impact must be judgedby the project specifications.

The matching approach is a hybrid method that combines ABM and relational matching.It uses a coarse-to-fine hierarchical strategy with a combination of several image matchingalgorithms and automatic quality control. ABM (both in the form of a modified cross-correlation and least squares matching) is employed to match feature points and grid points

(see also Hsia and Newton, 1999). Generally, the performance and success rate of ABMmainly depends on: the existence of sufficient image texture; the quality of the approximationsand a set of matching parameters, such as the matching window size; the search distance; andthe acceptance threshold for the correlation coefficient. How to select a set of correct matchingparameters is problematic, because the requirements for these parameter values are conflicting.These matching parameters are functions of many factors, including terrain type, imagetexture, image scale, disparity variations and image noise. The TLS matcher uses a set ofadaptively determined matching parameters. This is done by analysing the results of the higherlevel image pyramid matching and using them at the current pyramid level.

The performance of ABM is not good if there is insufficient image texture, in the case ofrepetitive patterns, and at surface discontinuities. Unfortunately, these problems are very

typical of large scale images as provided by TLS. In the first case, because of missing points,


2012 The Author



13/22

the ABM may lead to holes in the DSM. To overcome this problem, a global image matchingtechnique, based on probabilistic relaxation (Hancock and Kittler, 1990), is employed to match

grid points in order to bridge the poor texture areas. This relational matching uses localsmoothness constraints. In the last case, ABM generates smoothing effects at surfacediscontinuities. ABM is also employed to match the edges located on such discontinuities, butthe matched edges are used as breaklines to control the weights of the surface smoothnessconstraints in the global image matching procedure. As such, they prohibit the smoothnessconstraints from crossing the edges. The quasi-epipolar curves derived from the TLS sensormodel are used to restrict the search range to only one direction. The residual y parallax grid isused to compensate some of the errors in the raw image data.

In summary, the matching approach can be characterised by the following aspects:

(1) Multiple image matching and different matching algorithms. A new flexible and robustmatching algorithm, thegeometrically constrained cross-correlation (GC3) method,has been developed in order to take advantage of multiple images. The algorithm is

based on the concept of multi-image matching guided from the object space and allowsthe reconstruction of 3D objects by matching all available images simultaneously,without having to match all individual stereopairs separately and merge the results.Besides this special form of cross-correlation, LSM is also used as an option.

(2) Matching with multiple primitives. More robust hybrid image matching algorithmshave been developed by taking advantage of both ABM and FBM techniques andutilising both local and global image information. In particular, an edge matchingmethod is combined with a grid point matching method through a probability relax-ation-based relational matching process. The use of edges leads to the preservation of

surface discontinuities, while grid points bridge areas with little or no texture.(3) Self-tuning matching parameters. The adaptive determination of the matching

parameters results in a higher success rate and fewer mismatches. These parametersinclude the size of the correlation window, the search distance and the correlationthreshold values. This is done by analysing the matching results at the previousimage pyramid level and using them at the current level.

(4) High matching redundancy. With this matching approach, highly redundantmatching is achieved, so that points and edges can be generated. Highly redundantmatching results are suitable for representing very steep and rough terrain and allowthe terrain microstructures and surface discontinuities to be well preserved. More-over, this high redundancy also allows for better automatic blunder detection.

(5) Efficient surface modelling. The object surface is modelled by a TIN generated by aconstrained Delauney triangulation of the matched points and edges. A TIN is suitable

for surface modelling because it integrates all the original matching results, includingpoints and edge features, without any interpolation. It is adapted to describe complexterrain types that contain many surface microstructures and discontinuities.

(6) Coarse-to-fine hierarchical strategy. The algorithm works in a coarse-to-fine multi-resolution image pyramid structure, and obtains intermediate DSMs at multipleresolutions. Matches on low-resolution images serve as approximations to restrict thesearch space and to adaptively compute the matching parameters for the subsequentlevels.

Results of Controlled Accuracy Tests

Results, based on controlled tests, have been published at several times, such as in Gruen

and Zhang (2002) for aerial images. In Eisenbeiss et al. (2005) and Lambers et al. (2007) the


2012 The Author



14/22

results of a comparison of image matching from aerial UAV images (using SAT-PP) withterrestrial laser scan data showed no clear superiority of either technique. Image matching,

however, turned out to be of greater generality and more flexible use.Recently, the German Society of Photogrammetry and Remote Sensing (DGPF)

conducted a test on evaluation of digital photogrammetric aerial camera systems. As partof this test the accuracy of DSMs, derived automatically, was also investigated. Some of theresults have been reported in Wolff (2009). The key problem with such tests, which were donewith aerial images with footprints of 8 and 20 cm, is the generation of sufficiently goodreference data.

Tests in close range applications have been reported, for example, by Remondino etal. (2008, 2009) and Remondino and Menna (2008), although not always supported byaccurate reference data. In some cases, terrestrial laser scanning was compared with imagematching results, with no clear indication as to which technique would deliver the betterresults.

The performance of the matching software SAT-PP for DSM generation has been verifiedextensively with several high-resolution satellite imagery data-sets, such as SPOT-5, IKONOS,QuickBird, ALOS/PRISM, Cartosat-1 and WorldView-1, over different terrain types; theseinclude hilly and rugged mountainous areas, and rural, suburban and urban areas. A detailedanalysis of the results of IKONOS was presented in Gruen et al. (2005) and Zhang and Gruen(2006). Other processing and evaluation results of IKONOS and SPOT-5 HRS/HRG can befound in Zhang and Gruen (2004), Poli et al. (2004), Baltsavias et al. (2006), Poon et al. (2005)and Crespi et al. (2008). ALOS/PRISM results are described in Gruen and Wolff (2007) whilst

WorldView-1 results can be found in Poli et al. (2009). A general summary of satellite image-matching results is given in Wolff and Gruen (2008). In a special application, the performanceof image matching with respect to the generation of 3D tree canopy models from aerial images

was tested and compared to lidar data. Image matching results (computed with SAT-PP) turnedout to give better results (Baltsavias et al., 2008).

What has been observed in all the controlled tests is that an accuracy of 1 to 2 pixels canbe achieved in DSM generation, but there are many blunders in the results. These blunders areusually not single spikes (which could be easily detected), but rather groups of gross errors thatact like local systematic errors. Therefore, they are not easily detected automatically. In futureresearch and development activities emphasis should be on the understanding of the underlyingreasons for these blunders and on the development of techniques to avoid them. The mainsources of errors are:

(1) object features (edges, height differences, steepness of slopes, repetitive objects, roleof vegetation);

(2) illumination and reflectance properties (lack of image texture, shadows, specularreflections), image quality (signal-to-noise ratio, image artefacts); and

(3) network problems (insufficient design, partial occlusions).

Image matching must be seen in the context of automation. The potential advantages ofautomation are:

(1) increased accuracy;(2) reduced equipment costs;(3) increased throughput;(4) faster availability of results (online capability);(5) new kinds of products; and

(6) better quality products.


2012 The Author



15/22

The last promise, especially, has not yet been fulfilled. The quality of DSMs produced byimage matching still cannot compare with those derived from manual measurements.

Advancements in image understanding algorithms are therefore urgently needed.

Quality Control and Editing

Internal quality control should be a key feature of any supposedly automatic procedure. InDSM/DTM generation this quality control is either non-existent or comes with greatdeficiencies. In some systems the traffic light system is being used, which classifies matchedpoints in three colours, red, yellow and green, according to their quality. However, how theseclassifiers are computed remains largely unknown. In addition, in real data-sets, manymisclassifications are observed. Therefore, the only reliable quality control procedure currentlyavailable is visual checking, for example, by overlaying the generated point cloud and thestereomodel. This, however, turns the automated procedure into a semi-automated one.

In connection with these problems, the editing of data must be considered. Editing is donein the form of pre- and post-editing. Pre-editing includes the definition of dead areas such aswater surfaces, clouds or very steep (and possibly vegetation-covered) slopes. Procedures forpost-editing are mainly concerned with blunder and systematic error removal, but also withreduction from DSM to DTM. Although there are supposedly automated procedures offeredfor this latter task, there is a latent danger that serious errors are introduced, because thedecisions are based only on purely geometric considerations. Automated blunder detection isonly possible at very coarse levels, such as through the filtering of single spikes. Usually the

results returned need heavy manual editing.With commercial digital workstations, editing and quality control tools are available in

terms of:

(1) stereo-superimpositioning of point and/or contours;(2) correction of points, lines and areas;(3) geomorphological editing;(4) definition of dead areas (where no matching is performed);(5) analysis of wireframes and contours;(6) stereoscopic ortho-images;(7) colour marking of points (red, yellow, green for quality status); and(8) representation of parallaxes or heights in terms of grey values.

Relation to Laser Scanning

Laser scanning has generated much interest in recent years and is used on many projects.There is an ongoing discussion as to which method is more suitableimage matching or laserscanning. Despite the relevance of this topic, it is surprising how few comprehensiveinvestigations are available up to now. Gruen (2007) has addressed some problems whendealing with laser scan point clouds for 3D modelling and in Gruen (2009) the advantages ofphotogrammetry were emphasised and advertised. Lately, the superiority of laser scanning hasbeen increasingly questioned; Leberl et al. (2010) show how multipoint matching withUltracam images with 90% overlap can provide high-quality point clouds. Some test examplesfrom satellite, aerial and terrestrial images have been listed in the previous text. There has beena trend to consider laser scan data superior in terms of accuracy, and so used as reference datafor the checking of the image matching results; however, this assumption has been found to be

wrong in quite a few examples. Laser scan and image matching data have quite different error


2012 The Author



16/22

properties, which have never been compared comprehensively. This is still a useful task to beperformed in the future. In 3D city modelling, a particularly important application, a recent

survey by Haala and Kada (2010), considering the use of image as well as laser scan data,indicates that full automation in high-resolution building modelling is still a widely unsolvedproblem, no matter whether image or laser scan data is used.

Concluding Remarks

The development of image matching in photogrammetry has been described chrono-logically. While the original approaches to image matching were derived from signal processingand image transfer requirements, specific models trying to consider the photogrammetricimaging and network aspects were introduced later. The underlying models for matchingbecame more complex over time. With the advent of digital techniques, there was no longer anyactual limit to the refinement of models but computing time considerations became criticalissues. This is where the research stands today. Despite the fact that the matching problem hasnot yet been solved in general terms, there are quite refined models available, with millions ofmatch points being produced per stereomodel. Least squares matching, especially, can be usedin many different modes and variations. As such, it provides for an algorithmic framework thatis highly adaptive to various types of image content, network structures, processingrequirements and accuracy expectations. It has also been demonstrated in the past that leastsquares matching allows exploitation of the full accuracy potential of images and systems. Itprovides for a measurement accuracy that goes way beyond the capabilities of a human operator.

The advantages of LS matching lie in its inherent high accuracy. For DSM generation, anaccuracy limit of 01 pixels can be expected under favourable conditions. Furthermore,precision and reliability measures and statistical tests can readily be derived for internal quality

control. This comes at high computational costs. Therefore, researchers are currently looking attechniques for speeding up the computations. This may be either done by grid computing and/or integration of GPU (graphics processing unit) technology into the algorithmic flow (Ernstand Hirschmuller, 2008). This, however, does not solve the basic problems in image matching,as stated in this contribution. There are essentially two ways in which further progress can beachieved: (a) adding more information by using a multi-sensor approach; and (b) advancingimage understanding algorithms. While the realisation of the former is currently ongoing, thelatter is the harder option to fulfil.

In Haala (2009), the comeback of digital image matching has been proclaimed. In myunderstanding, image matching was always present and available, but up to now its fullpotential has not been fully realised.

In an outstanding article, Rosenberg (1955) remarks on the future of automated mapping,of which image matching is a crucial component: The engineering problems in electronic

photogrammetry are very considerable It will be a long time before completely automatic,electronic photogrammetry is actually at hand. Despite all the progress that has beenachieved and the availability of new technologies, more than 50 years later this statement isstill essentially correct.

references

Ackermann, F., 1984. Digital image correlation: performance and potential application in photogrammetry.Photogrammetric Record, 11(64): 429439.

Aggarwal, J. K., Duda, R. O. and Rosenfeld, A. (Eds.), 1977. Computer methods in image analysis. IEEE

Press, New York. 466 pages.


2012 The Author



17/22

Akca,D., 2007.Least squares 3D surface matching. Ph.D. thesis, Institute of Geodesy and Photogrammetry, ETHZurich, Switzerland, Mitteilungen 92. Dissertation No. 17136. 78 pages.

Alberich, L., 1985. Orthophotomapping with an automated system at 1:50 000 map scale. Proceedings ASPACSM Convention, Washington, DC, 1: 222228.

Albertz, J. and Koenig, G., 1984. A digital stereophotogrammetric system. International Archives of Photo-grammetry and Remote Sensing, 25(A2): 17.Almroth, U. andHendriks, L., 1987. High accuracy digital matching of close-range objects on the analytical

plotter Kern DSR-11. Proceedings of ISPRS Intercommission Conference on Fast Processing of Photo-grammetric Data, Interlaken, Switzerland. 437 pages: 193203.

Andrews, H. C. (Ed.), 1978. Digital image processing. IEEE Press, New York. 737 pages.Baker, H. H. and Binford, T. O., 1981. Depth from edge and intensity based stereo. Proceedings of the 7th

International Joint Conferences on Artificial Intelligence. 631636.Baltsavias, E., 1984. Digital image correlation in photogrammetry. M.Sc. technical report, Department of

Geodetic Science and Surveying, Ohio State University, Columbus, Ohio.Baltsavias, E., 1991. Multiphoto geometrically constrained matching. Ph.D. thesis, Institute of Geodesy and

Photogrammetry, ETH Zurich, Switzerland, Mitteilungen 49. 221 pages.Baltsavias,E. P.andStallmann,D., 1990. Trinocular vision for automatic and robust 3-D determination of the

trajectories of moving objects. International Archives of Photogrammetry and Remote Sensing, 28(5/1): 620

629.Baltsavias, E.,Li, H., Gaschen,S. and Sinning, M., 1996. DTM generation with the Leica/Helava DPW 770

and VirtuoZo digital photogrammetric systems. Paper presented at GeoInformatics 96, Wuhan, China, 16th to19th October. 1017.

Baltsavias, E., Zhang, L. and Eisenbeiss, H., 2006. DSM generation and interior orientation determination ofIKONOS images using a testfield in Switzerland. Photogrammetrie, Fernerkundung, Geoinformation, (1): 4154.

Baltsavias, E., Gruen, A., Eisenbeiss, H., Zhang, L. and Waser, L. T., 2008. High quality image matchingand automated generation of 3D tree models. International Journal of Remote Sensing, 29(5): 12431259.

Bernstein, R. (Ed.), 1978. Digital image processing for Remote Sensing. IEEE Press/Wiley, New York. 473pages.

Bethel, J., 1986. The DSR11 image correlator. American Congress on Surveying and Mapping & AmericanSociety of Photogrammetry and Remote Sensing. Annual Convention, IV: 4449.

Beyer, H. A., 1987. Some aspects of the geometric calibration of CCD-cameras. Proceedings of ISPRS Inter-commission Conference on Fast Processing of Photogrammetric Data, Interlaken, Switzerland. 437 pages:6881.

Beyer, H. A., 1992. Geometric and radiometric analysis of a CCD-camera based photogrammetric close-rangesystem. Ph.D. thesis, Institute of Geodesy and Photogrammetry, ETH Zurich, Switzerland. Dissertation No.9701. 186 pages.

Case, J. B., 1982. The digital stereo comparator/compiler (DSCC). International Archives of Photogrammetry,24(2): 2329.

Chellappa, R. and Sawchuk, A. A. (Ed.), 1985. Digital image processing and analysis, Volume 2. IEEEComputer Society Press, New York. 721 pages.

Cogan, L., Gugan, D., Hunter, D., Lutz, S. and Peny, C., 1988. Kern DSP1 digital stereo photogrammetricsystem. International Archives of Photogrammetry and Remote Sensing, 27(B2): 7183.

Crespi, M., De Vendictis, L., Poli, D., Wolff, K., Colosimo, G., Gruen, A. and Volpe, F., 2008.Radiometric quality and DSM generation analysis of Cartosat-1 stereo imagery. International Archives of

Photogrammetry, Remote Sensing and Spatial Information Sciences, 37(B13): 13491355.DeMeter,E. R., 1963. Latest advances in automated mapping.Photogrammetric Engineering, 29(6): 10271036.Dowman, I., 1984. Instrumentation for topographic mapping from satellite data. International Archives of

Photogrammetry, 25(2): 155163.Dowman, I. J. and Haggag, A. A. Y., 1977. Digital image correlation along epipolar lines. International

Symposium on Image Processing, Interactions with Photogrammetry and Remote Sensing, Graz, Austria.4749.

Dowman,I. J.,Gugan,D. J.,Muller,J.-P.,ONeill,M. andParamananda,V., 1987. Digital processing ofSPOT data.Proceedings of ISPRS Intercommission Conference on Fast Processing of Photogrammetric Data,Interlaken, Switzerland. 437 pages: 318330.

Ebner, H. and Heipke, C., 1988. Integration of digital image matching and object surface reconstruction. Inter-national Archives of Photogrammetry and Remote Sensing, 27(B11): 534545.

Eisenbeiss,H.,Sauerbier,M.,Zhang,L. andGrun,A., 2005. Mit dem Modellhelikopter uber Pinchango Alto.

Geomatik Schweiz, 2005(9): 510515 (in German).


2012 The Author



18/22

Ernst, I. and Hirschmuller, H., 2008. Mutual information based semi-global stereo matching on the GPU.Advances in Visual Computing. Proceedings 4th International Symposium on Visual Computing (ISVC08).Springer Lecture Notes in Computer Science, 5358: 228239.

Forstner,W., 1982. On the geometric precision of digital correlation.International Archives of Photogrammetry,

24(3): 176189.Gong, J., Li, Z., Zhu, Q., Sui, H. and Zhou, Y., 2000. Effects of various factors on the accuracy of DEMs: anintensive experimental investigation. Photogrammetric Engineering & Remote Sensing, 66(9): 11131117.

Gruen, A., 1985a. Adaptive kleinste Quadrate Korrelation und geometrische Zusatzinformationen. Vermessung,Photogrammetrie, Kulturtechnik, 83(9): 309312 (in German).

Gruen, A., 1985b. Adaptive least squares correlation: a powerful image matching technique. South AfricanJournal of Photogrammetry, Remote Sensing and Cartography, 14(3): 175187.

Gruen, A., 1988. Towards real-time photogrammetry. Photogrammetria, 42(56): 209244.Gruen,A. W., 1989. Digital photogrammetric processing systems: current status and prospects. Photogrammetric

Engineering & Remote Sensing, 55(5): 581586.Gruen, A., 1996a. Digital photogrammetric stations revisited. International Archives of Photogrammetry and

Remote Sensing, 31(B2): 127134.Gruen, A., 1996b. Least squares matching: a fundamental measurement algorithm. Chapter 8 in Close Range

Photogrammetry and Machine Vision (Ed. K. B. Atkinson). Whittles, Caithness, Scotland. 371 pages: 217255.

Gruen, A., 1999. Digital photogrammetric stations. Promises, performance, perspectives. GeoInformatics, 2(3):2227.

Gruen, A., 2007. Laser scanning: blackbox or braintwister? GeoInformatics, 10(7): 4951.Gruen, A., 2009. Aerial photogrammetryrevival of a discipline. GIS Development, 13(1): 3031.Gruen, A. andBaltsavias, E. P., 1984. Adaptive least squares correlation. Project Report for Helava Assoc.,

Inc., USA.Gruen, A. and Baltsavias, E. P., 1985. Adaptive least squares correlation with geometrical constraints. SPIE,

595: 7282.Gruen, A. and Baltsavias, E. P., 1986. High precision image matching for digital terrain model generation.

International Archives of Photogrammetry and Remote Sensing, 26(3/1): 284296.Gruen, A. and Beyer,H. A., 1986. Real-time photogrammetry at the digital photogrammetric station (DIPS) of

ETH Zurich. Canadian Surveyor, 41(2): 181199.Gruen,A. andBaltsavias,E. P., 1988a. Automatic 3-D measurement of human faces with CCD-cameras. SPIE,

1030: 106115.Gruen, A. and Baltsavias, E. P., 1988b. Geometrically constrained multiphoto matching. PhotogrammetricEngineering & Remote Sensing, 54(5): 633641.

Gruen, A. and Beyer, H. A., 1990. DIPS IIturning a standard computer workstation into a digital photo-grammetric station. International Archives of Photogrammetry and Remote Sensing, 28(2): 247255.

Gruen, A. and Stallmann, D., 1991. High accuracy edge matching with an extension of the MPGC-matchingalgorithm.SPIE, 1526: 4255.

Gruen, A. and Agouris, P., 1994. Linear feature extraction by least squares template matching constrained byinternal shape forces. International Archives of Photogrammetry and Remote Sensing, 30(3/1): 316323.

Gruen, A. and Li,H., 1997. Semi-automatic linear feature extraction by dynamic programming and LSB-snakes.Photogrammetric Engineering & Remote Sensing, 63(8): 985995.

Gruen, A., Bar, S. and Buehrer, Th., 2000. DTMs derived automatically from DIPSwhere do we stand?GeoInformatics, 3(5): 3639.

Gruen,A. andZhang,L., 2002. Automatic DTM generation from three-line-scanner (TLS) images. International

Archives of Photogrammetry and Remote Sensing, 34(2A): 131137.Gruen,A. andAkca,D., 2005. Least squares 3D surface and curve matching. ISPRS Journal of Photogrammetry

and Remote Sensing, 59(3): 151174.Gruen, A., Zhang, L. and Eisenbeiss, H., 2005. 3D precision processing of high-resolution satellite imagery.

Proceedings of the ASPRS 2005 Annual Conference, Baltimore, Maryland (on CD-ROM).Gruen, A. and Wolff, K., 2007. DSM generation with ALOS/PRISM data using SAT-PP. Proceedings of the

IEEE International Geoscience and Remote Sensing Symposium (IGARSS), Barcelona, Spain. 4 pages (on CD-ROM).

Grun, A., 1984. Adaptive least squares correlationconcept and first results. Intermediate Research ProjectReport to Helava Assoc., Inc., Ohio State University, Columbus, Ohio. 13 pages.

Guelch, E., 1988. Results of test on image matching of ISPRS WGIII/4. International Archives of Photogram-metry and Remote Sensing, 27(B3): 254271.

Haala, N., 2009. Comeback of digital image matching. In Photogrammetric Week 2009 (Ed. D. Fritsch).

Wichmann, Heidelberg, Germany. 289301.


2012 The Author



19/22

Haala, N. and Kada, M., 2010. An update on automatic 3D building reconstruction. ISPRS Journal of Photo-grammetry and Remote Sensing, 65(6): 570580.

Haggren,H., 1986. Real-time photogrammetry as used for machine vision applications. International Archives ofPhotogrammetry and Remote Sensing, 26(5): 374382.

Hancock, E. R. and Kittler, J., 1990. Discrete relaxation. Pattern Recognition, 23: 711733.Helava, U. V., 1972. Digital processing and analysis of image data. International Archives of Photogrammetry,20(1). 354 pages.

Helava, U. V., 1976. Digital correlation in photogrammetric instruments. International Archives of Photogram-metry, 21(2). 26 pages.

Helava, U. V., 1978. Digital correlation in photogrammetric instruments. Photogrammetria, 34(1): 1934.Helava, U. V., 1988a. Digital comparator correlator system. International Archives of Photogrammetry and

Remote Sensing, 27(B2): 160170.Helava, U. V., 1988b. Object-space least squares correlation. Photogrammetric Engineering & Remote Sensing,

54(6): 711714.Hirschmueller,H., 2008. Stereo processing by semiglobal matching and mutual information.IEEE Transactions

on Pattern Analysis and Machine Intelligence, 30(2): 328342.Hobrough, G. L., 1959. Automatic stereo plotting. Photogrammetric Engineering, 25(5): 763769.Hobrough,G. L., 1965. Automation in photogrammetric instruments. Photogrammetric Engineering, 31(4): 595

603.Hsia, J.-S. and Newton, I., 1999. A method for the automated production of digital terrain models using a

combination of feature points, grid points, and filling back points. Photogrammetric Engineering & RemoteSensing, 65(6): 713719.

Kelly, R. E., McConnell, P. R. H. and Mildenberger, S. J., 1977. The Gestalt photomapping system.Photogrammetric Engineering & Remote Sensing, 43(11): 14071417.

Kempa, M. and Schlueter, M., 1993. DEM evaluation by an operator and facets stereo vision: a comparisonbased on close range imagery. In Optical 3-D Measurement Techniques II(Eds. A. Gruen & H. Kahmen).Wichmann, Karlsruhe, Germany. 624 pages: 502509.

Konecny, G. and Pape, D., 1981. Correlation techniques and devices. Photogrammetric Engineering & RemoteSensing, 47(3): 323333.

Krzystek, P., 1995. Generation of digital elevation models. Chapter 7 in Second Course in Digital Photogram-metry. Landesvermessungsamt Nordrhein-Westfalen and Institut fur Photogrammetrie, Universitat Bonn.

Unpaginated.Lambers, K., Eisenbeiss, H., Sauerbier, M., Kupferschmidt, D., Gaisecker, T., Sotoodeh, S. andHanusch, T., 2007. Combining photogrammetry and laser scanning for the recording and modelling of thelate intermediate period site of Pinchango Alto, Palpa, Peru. Journal of Archaeological Science, 34(10): 17021712.

Leberl, F., Irschara, A., Pock, T., Meixner, P., Gruber, M., Scholz, S. andWiechert, A., 2010. Pointclouds: lidar versus 3D vision. Photogrammetric Engineering & Remote Sensing, 76(10): 11231134.

Lemmens,J. P. M., 1988. A survey on stereo matching techniques.International Archives of Photogrammetry andRemote Sensing, 27(B8): 1123.

Li,M. X., 1989.Hierarchical Multi-point Matching with simultaneous detection and location of breaklines. Ph.D.thesis, Department of Photogrammetry, Royal Institute of Technology, Stockholm, Sweden. 177 pages.

Lue, Y., Zhang, Z. and Zhang, J., 1987. Calibration of a drum scanner with least squares correlation. Pro-ceedings of ISPRS Intercommission Conference on Fast Processing of Photogrammetric Data, Interlaken,Switzerland. 437 pages: 109119.

Maas,H.-G., 1996. Automatic DEM generation by multi-image feature based matching. International Archives ofPhotogrammetry and Remote Sensing, 31(3): 484489.

Maas, H.-G., Stefanidis, A. and Gruen, A., 1994. From pixels to voxels: tracking volume elements insequences of 3-D digital images. International Archives of Photogrammetry and Remote Sensing, 30(3/2):539546.

Makarovic,B., 1980. Image correlation algorithms.International Archives of Photogrammetry, 23(B2): 139158.Masry, S. E., 1974. Digital correlation principles. Photogrammetric Engineering, 40(3): 303308.Mikhail, E. M., Helava, U. A., Jancaitis, J. R., Doyle, F. J., Helmering, R. J., Ackerman, F. and

Turner, A. K., 1978. Panel discussion: the future of DTM. Photogrammetric Engineering & RemoteSensing, 44(12): 14871497.

Miller, S. and De Venecia, K., 1992. Automatic elevation extraction and the digital photogrammetric work-station.ASPRSACSM Annual Convention, Albuquerque, New Mexico, 1: 572580.

Otto,G. P. andChau,T. K. W., 1988. A region-growing algorithm for matching of terrain images. Proceedings

4th Alvey Vision Conference, University of Manchester, UK. 123128.


2012 The Author



20/22

Panton, D. J., 1978. A flexible approach to digital stereo mapping. Photogrammetric Engineering & RemoteSensing, 44(12): 14991512.

PE&RS, 1983. Five short articles on automated correlation of stereo data. Photogrammetric Engineering & RemoteSensing, 49(4): 527544.

Pertl, A., 1984. Digital image correlation with the analytical plotter Planicomp C100. International Archives ofPhotogrammetry, 25(3b): 874882.Poli, D., Zhang, L. and Gruen, A., 2004. SPOT-5/HRS stereo images orientation and automated DSM

generation. International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences,35(B1): 421432.

Poli, D., Wolff, K. and Gruen, A., 2009. Evaluation of Worldview-1 stereo scenes. International Archives ofPhotogrammetry, Remote Sensing and Spatial Information Sciences, 38(1-4-7/W5). 6 pages (on CD-ROM).

Poon, J., Fraser, C. S., Zhang, C., Zhang, L. and Gruen, A., 2005. Quality assessment of digital surfacemodels generated from IKONOS imagery. Photogrammetric Record, 20(110): 162171.

Rauhala, U. H., 1986. Compiler positioning of array algebra technology. International Archives of Photo-grammetry and Remote Sensing, 26(3/3): 173198.

Remondino, F., el-Hakim, S. F., Gruen, A. and Zhang, L., 2008. Turning images into 3-D models. IEEESignal Processing Magazine, 25(4): 5565.

Remondino,F. andMenna,F., 2008. Image-based surface measurement for close-range heritage documentation.

International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences, 37(B5): 199206.Remondino, F., Gruen, A., von Schwerin, J., Eisenbeiss, H., Rizzi, A., Girardi, S., Sauerbier, M. and

Richards-Rissetto,H., 2009. Multi-sensor 3D documentation of the Maya site of Copan. Proceedings of theXXIIth CIPA International Symposium, Kyoto, Japan. 6 pages.

Rosenberg, P., 1955. Information theory and electronic photogrammetry. Photogrammetric Engineering, 21(4):543555.

Rosenfeld, A., 1984. Image analysis: problems, progress and prospects. Pattern Recognition, 17(1): 312.Rosenholm, D., 1986. Accuracy improvement of digital matching for evaluation of digital terrain models.

International Archives of Photogrammetry and Remote Sensing, 26(3/2): 573587.Scarano, F. A. and Brumm, G. A., 1976. A digital elevation data collection system. Photogrammetric

Engineering & Remote Sensing, 42(4): 489496.Schewe, H. and Forstner, W., 1986. The program PALM for automatic line and surface measurement using

image matching techniques. International Archives of Photogrammetry and Remote Sensing, 26(3/2): 608

622.Smith, M. J. and Smith,D. G., 1996. Operational experiences of digital photogrammetric systems. InternationalArchives of Photogrammetry and Remote Sensing, 31(B2): 357362.

Snavely, N., Seitz, S. M. and Szeliski, R., 2006. Photo tourism: exploring photo collections in 3D. ACMTransactions on Graphics, 25(3): 835846.

Trinder, J. C., 1986. Precision of stereoscopic height measurements. Photogrammetric Engineering & RemoteSensing, 52(1): 7579.

Vu, H. H., Keriven, R., Labatut, P. and Pons, J.-P., 2009. Towards high-resolution large-scale multi-viewstereo. Paper presented at CVPR09. http://imagine.enpc.fr/publications/publications1.html#CVPR09a [Ac-cessed: 8th January 2011].

Williams, R. E., 1959. The automatic map compilation system. Photogrammetric Engineering, 25(1): 103110.Wolff,K., 2009. DGPF project: evaluation of digital photogrammetric aerial based imaging systemsgeneration

of digital surface models.International Archives of Photogrammetry, Remote Sensing and Spatial InformationSciences, 38(1-4-7/W5). 5 pages (on CD-ROM).

Wolff, K. and Gruen, A., 2008. Up-to-date DSM generation using high resolution satellite image data. Inter-national Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences , 37(B13): 11031108.

Wrobel, B., 1987. Facets stereo vision (FAST vision)a new approach to computer stereo vision and to digitalphotogrammetry. Proceedings of ISPRS Intercommission Conference on Fast Processing of PhotogrammetricData, Interlaken, Switzerland. 437 pages: 231258.

Zhang, B. and Miller, S., 1997. Adaptive automatic terrain extraction. SPIE, 3072: 2736.Zhang, J., Zhang, Z., Shen, W. and Wang, Z., 1996. VirtuoZo digital photogrammetric system and its theo-

retical foundation and key algorithms.International Archives of Photogrammetry and Remote Sensing, 31(B2):424429.

Zhang, L., 2005. Automatic digital surface model (DSM) generation from linear array images. Institute ofGeodesy and Photogrammetry, ETH Zurich, Switzerland, Mitteilungen 88. Dissertation No. 16078. 199 pages.

Zhang, L. and Gruen, A., 2004. Automatic DSM generation from linear array imagery data. International

Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences, 35(B3): 128133.


2012 The Author



21/22

Zhang, L. and Gruen, A., 2006. Multi-image matching for DSM generation from IKONOS imagery. ISPRSJournal of Photogrammetry and Remote Sensing, 60(3): 195211.

Zhang, Z., Zhang, J., Wu, X. and Zhang, H., 1992. Global image matching with relaxation method.Proceedings of the International Colloquium on Photogrammetry, Remote Sensing and Geographic Infor-

mation Systems, Wuhan, China, 11th to 14th May. 175188.Zheltov, S. Y. and Sibiryakov, A. V., 1997. Adaptive subpixel cross-correlation in a point correspondence. InOptical 3D Measurement Techniques (Eds. A. Gruen & O. Ku hbler). Wichmann, Zurich, Switzerland. 8695.

Resume

Lappariement dimages et de formes est sans doute loperation la plus

importante en photogrammetrie numerique et en modelisation et cartographieautomatiques. De nombreuses approches se sont succededepuis des annees mais le

probleme nest toujours pas totalement resolu. Cet article decrit le developpement

des techniques dappariement dimages en photogrammetrie pendant les 50

dernieres annees, presente les resultats obtenus dans quelques etudes de precisionempiriques, et dresse un bilan critique des problemes qui subsistent. Bien que les

approches automatiques aient un grand nombre davantages, la qualitedes resultats

nest toujours pas satisfaisante, et meme loin detre acceptable dans certains cas.

Meme avec les techniques les plus avancees, nous sommes toujours dans lincapacite

datteindre la qualitedes resultats obtenus par un operateur humain. Il y a un besoin

urgent dameliorations et dinnovations, soit a travers des approches multi-capteurs

plus puissantes et consistant a elargir le spectre dinformation, soit a travers une

amelioration des algorithmes de comprehension dimages visant a les rendre plus

proches des possibilites humaines de lecture et de comprehension du contenu des

images.

Zusammenfassung

Bild- und Musterkorrelation gehoren zu den wichtigsten Grundfunktionen der

Digitalen Photogrammetrie und somit auch der automatischen 3D Modellierung und

Kartierung. Viele Ansatze zur Korrelation wurden uber die Jahre entwickelt, aber

das Problem gilt grundsatzlich noch immer als ungelost. Dieser Beitrag beschreibt

die Entwicklung der Verfahren der Bildkorrelation in der Photogrammetrie uber die

letzten 50 Jahre, verweist auf die Ergebnisse einiger empirischer Genauigkeitsstudienund diskutiert einige der immer noch bestehenden Probleme.Obwohl automatische

Verfahren eine ganze Reihe von Vorteilen aufweisen, ist doch die Qualitat der

Ergebnisse meist nicht ausreichend, teilweise ja sogar weit entfernt von jeglicherAkzeptanz. Selbst mit den hochstentwickelten Verfahren sind wir immer noch nicht in

der Lage, die Qualitat der Ergebnisse eines menschlichen Operateurs zu erreichen.

Wir benotigen dringend weitere Verbesserungen und Innovationen. Dazu gibt esgegenwartig zwei grundsatzlich gangbare Wege: (a) Nutzung von Multi-Sensor

Informationen und somit Erweiterung der Informationsgrundlagen und/oder (b) durch

Fortschritte bei den Algorithmen des Bildverstehens und somit besserer Modellierung

des menschlichen Prozesses des Bildverstehens.


2012 The Author



22/22

Resumen

La correspondencia de imagenes y muestras es, probablemente, la funcion masimportante en la fotogrametra digital, en el modelado 3D y en la cartografa

automatica. Muchos metodos de correspondencia han evolucionado a lo largo de losanos pero, en terminos generales, el problema se considera aun no resuelto

completamente. Este artculo describe la evolucion de las tecnicas de corre-

spondencia de imagenes en la fotogrametra a lo largo de los ultimos 50 anos,

analiza los resultados de algunos estudios empricos de la exactitud, y ofrece una

valoracion crtica de los problemas aun sin resolver. Aunque los metodos

automaticos poseen un gran numero de ventajas, la calidad de los resultados no

es todava satisfactoria y, en algunos casos, incluso esta lejos de ser aceptable.

Incluso con las mas avanzadas tecnicas no somos capaces de lograr la calidad de los

resultados que un operador humano puede conseguir. Hay una necesidad urgente de

continuar las mejoras e innovaciones, ya sea mediante la utilizacion de multiples

sensores que incrementen el espectro de la informacion, o por avances en losalgoritmos de comprension de la imagen que permitan acercarnos mas a l a

capacidad humana de lectura e interpretacion de su contenido.


2012 The Author


imagenes coincidentes.pdf

Documents

Transcript of imagenes coincidentes.pdf