A Hybrid Optimization ApproachA Hybrid Optimization Approachfor Global Explorationfor Global Exploration

20052005 年度年度 713713 番番日和　悟日和　悟

Satoru HIWASatoru HIWA

知的システムデザイン研究室知的システムデザイン研究室Intelligent Systems Design LaboratoryIntelligent Systems Design Laboratory

OptimizationOptimization

Optimization problem consists of:- Objective function: we want to minimize or maximize.- Design variables: affect the objective function value.- Constraints: allow the design variables to take on certain

values but exclude others.

Mathematical discipline that concerns the finding of minima or maxima of functions, subject to constraints

Real-world applications

Optimization techniques have been applied to various real-world problems.e.g.) Structural design

Electric device design

Problem Solving by OptimizationProblem Solving by Optimization

There are many good optimization algorithms. Each method has its own characteristics.

- It is difficult to choose the best method for the optimization problem.

It is important to select and apply the appropriate algorithms according to the complexities of the problems.

It is hard to solve the problem with only one algorithm when the problem is complicated.

Hybrid optimization approach, which combines plural optimization algorithms, should be necessary.

Purpose of the research:To develop an efficient hybrid optimization algorithm

Hybrid Optimization ApproachHybrid Optimization Approach

It provides the high performance which cannot be accomplished with only one algorithm.

Hybrid optimization algorithm

We have to determine what kinds of solutions are required. Desired solutions may vary depending on the user:

- One may require the better result within a reasonable time.- The other may want not only the optimum, but also the

information of the landscape. Optimization strategy

- First, how the optimization process is performed should be determined.

To develop an efficient hybrid optimization algorithm

Optimization StrategyOptimization Strategy

By this, we can obtain not only the optimum point, but also the information of the landscape.

Many optimization algorithms are designed only to derive an optimum.

To explore the search space uniformly and equally

Why is the strategy needed?

Why is the Strategy Needed?Why is the Strategy Needed?

When we solve real-world optimization problems;- Usually, the landscape and the optimum are unknown.- In this case, the obtained results should be reliable.

Genetic Algorithms (GAs) are powerful techniques to obtain the global optimum.- Probabilistic algorithm inspired by evolutionary biology

Example of optimization by GAs:

Problem GAs

Why is the Strategy Needed?Why is the Strategy Needed?

When we solve real-world optimization problems;- Usually, the landscape and the optimum are unknown.- In this case, the obtained results should be reliable.

Genetic Algorithms (GAs) are powerful techniques to obtain the global optimum.- Probabilistic algorithm inspired by evolutionary biology

Example of optimization by GAs:

Problem GAs

Why is the Strategy Needed?Why is the Strategy Needed?

When we solve real-world optimization problems;- Usually, the landscape and the optimum are unknown.- In this case, the obtained results should be reliable.

Genetic Algorithms (GAs) are powerful techniques to obtain the global optimum.- Probabilistic algorithm inspired by evolutionary biology

Example of optimization by GAs:

Problem GAs

Unknown

The result is not reliable.

Unexplored area exists.

Is real optimum in the area?

Why is the Strategy Needed?Why is the Strategy Needed?

When we solve real-world optimization problems;- Usually, the landscape and the optimum are unknown.- In this case, the obtained results should be reliable.

Genetic Algorithms (GAs) are powerful techniques to obtain the global optimum.- Probabilistic algorithm inspired by evolutionary biology

Example of optimization by GAs:

Problem GAs

Unknown

The result is not reliable.

Unexplored area exists.

Is real optimum in the area?

The strategy is not achieved only by GAs.

Why is the Strategy Needed?Why is the Strategy Needed?

When we solve real-world optimization problems;- Usually, the landscape and the optimum are unknown.- In this case, the obtained results should be reliable.

Genetic Algorithms (GAs) are powerful techniques to obtain the global optimum.- Probabilistic algorithm inspired by evolutionary biology

Example of optimization by GAs:

Problem GAs

Unknown

Reliability can be evaluated.

The strategy is achieved.

The landscape is grasped.

Optimization AlgorithmsOptimization Algorithms

The strategy is not achieved only by GAs. Other algorithm, which provides more global search, is

needed. However, the globally-intensified search converges slowly

compared to GAs or local search algorithms.- the much time is consumed in exploring the entire search space.

There are tradeoff between the search broadness and the convergence rate.

It is necessary to balance the global and local search.

GAs DIRECT: explores search space globally. SQP: is high-convergence local search method.

Both global and local search algorithms are hybridized.

DIRECTDIRECT

Deterministic, global optimization algorithm Its name comes from ‘DIviding RECTangles’.

- Search space is considered to be a hyper-rectangle (box).- Each box is trisected in each dimension.- Center point of each box is sampled as solution.

Boxes to be divided- are mathematically guaranteed to be promising.- are called ‘potentially optimal boxes.’

Characteristics of the DIRECT searchCharacteristics of the DIRECT search

Potentially optimal boxes potentially contain a better value than any other box.

DIRECT divides the potentially optimal boxes at each iteration.

Characteristics of the DIRECT searchCharacteristics of the DIRECT search

Example: 2-dimensional Schwefel Function- Some Local optima exist far from the global optimum.- DIRECT explores the search space uniformly and equally.- DIRECT also detects the promising area.

Global Optimum

Local Optima

Characteristics of the DIRECT searchCharacteristics of the DIRECT search

Example: 2-dimensional Schwefel Function- Some Local optima exist far from the global optimum.- DIRECT explores the search space uniformly and equally.- DIRECT also detects the promising area.

Global Optimum

Local Optima

Characteristics of the DIRECT searchCharacteristics of the DIRECT search

Example: 2-dimensional Schwefel Function- Some Local optima exist far from the global optimum.- DIRECT explores the search space uniformly and equally.- DIRECT also detects the promising area.

Global Optimum

Local Optima

Genetic Algorithms (GAs)Genetic Algorithms (GAs)

Heuristic algorithms inspired by evolutionary biology.- Solutions are called ‘individuals’, and genetic operators

(Crossover, Selection, Mutation) are applied. Real-coded GAs

- Individuals are represented by real number vector. Although GAs are global optimization algorithm, the search

broadness is inferior to DIRECT. GAs are used as more locally-intensified search than

DIRECT.

Parents

ChildrenIndividuals

Sequential Quadratic Programming Sequential Quadratic Programming (SQP)(SQP) Gradient-based local search algorithm

- The most efficient method in nonlinear programming- By using gradient information, SQP rapidly converges to

the optimum. Advantage

- High convergence Disadvantage

- SQP is often trapped to the local optima, for the problem which has many local optima.

Idea of the proposed hybrid optimization approach

Global explorationby DIRECT

Locally-intensifiedsearch by GAs

Fine tuningby SQP

Hybrid Optimization AlgorithmHybrid Optimization Algorithm

1. Perform the DIRECT search.2. Execute GAs.3. Improve the best solution obtained in GAs search by SQP.

Procedure of the proposed algorithm

Idea of the proposed hybrid optimization approach

Global explorationby DIRECT

Optimum

Locally-intensifiedsearch by GAs

Fine tuningby SQP

Hybrid Optimization AlgorithmHybrid Optimization Algorithm

1. Perform the DIRECT search.2. Execute GAs.3. Improve the best solution obtained in GAs search by SQP.

Procedure of the proposed algorithm

How to Combine DIRECT and GAsHow to Combine DIRECT and GAs

GAs utilize the center points of the potentially optimal boxes in DIRECT as their individuals.

Number of potentially optimal = number of individuals- Number of potentially optimal differs at each iteration.- Number of individuals are determined according to the

complexities of the problems.(e.g. In N-dim. space, N×10 individuals are recommended.)

DIRECT stopped. GAs start.

Number of potentially optimal = number of individuals- Number of potentially optimal differs at each iteration.- Number of individuals are determined according to the

complexities of the problems.(e.g. In N-dim. Space, N×10 individuals are recommended.)

How to Combine DIRECT and GAsHow to Combine DIRECT and GAs

GAs utilize the center points of the potentially optimal boxes in DIRECT as their individuals.

DIRECT stopped. GAs start.

Number of potentially optimal boxes should be adjusted according to the number of individuals.

How to Combine DIRECT and GAsHow to Combine DIRECT and GAs

If the number of potentially optimal is smaller than Ni,randomly generated individuals are added.

If the number of potentially optimal is larger than Ni,a certain number of potentially optimal boxes are selected.

Box selection rules are proposed and applied.

Ni: Number of individuals in GAs

Box Selection Rules for DIRECTBox Selection Rules for DIRECT

Idea of selecting the boxes to be divided

DIRECT sometimes performs an local improvement. In the hybrid optimization, it is not necessary for DIRECT

to perform locally-intensified search. Proposed rules reduce the crowded boxes.

- Distance from the box with best function value is calculated.- A certain number of boxes far from the best point are selected.- The rules are applied at each iteration in DIRECT search.

Box Selection Rules for DIRECTBox Selection Rules for DIRECT

Idea of selecting the boxes to be divided

The number of potentially optimal boxes is reducedwithout breaking the global search characteristics of DIRECT.

Potentially optimal boxes near the best point are discarded, and locally-biased search is prevented.

DIRECT sometimes performs an local improvement. In the hybrid optimization, it is not necessary for DIRECT

to perform locally-intensified search. Proposed rules reduce the crowded boxes.

- Distance from the box with best function value is calculated.- A certain number of boxes far from the best point are selected.- The rules are applied at each iteration in DIRECT search.

ExperimentsExperiments

10-dimensional Schwefel function- A lot of local optimum exist.- The function value of the global optimum is zero.

Target problem

Numerical example is shown- to verify whether the proposed method achieve the proposed

strategy − to explore the search space uniformly and equally. The proposed hybrid optimization algorithm

- is applied to the benchmark problem.- is compared to the search only by GAs.

Verification of effectiveness of the hybrid approach

Results and DiscussionsResults and Discussions

Searching ability

Average of 30 runs Hybrid GAs

Function value 9.07×10-8 5.58×102

Function evaluations 129,373 279,703

Average values of function value and the number of function evaluations are shown.

Proposed hybrid algorithm obtains better function value than that of GAs, with less function evaluations.

Results and DiscussionsResults and Discussions

To see whether the proposed strategy is achieved…

Search histories of DIRECT and GAs in the hybrid algorithm are checked.

History in 10-dimensional space is projected into 2-dimensional plane.

Although 45 plots exist, 4 typical examples are picked. (x1, x2, …, x10) → (x1, x2), (x1, x3), …

Search History of DIRECTSearch History of DIRECT

(x1, x2)

(x3, x6)

(x2, x5)

(x7, x9)

Search History of DIRECTSearch History of DIRECT

Search Histories of DIRECT and GAsSearch Histories of DIRECT and GAs

Search Histories of DIRECT and GAsSearch Histories of DIRECT and GAs

The proposed strategy is achieved.

ConclusionsConclusions

‘optimization strategy’ is proposed:- To explore the search space uniformly and equally

Optimization algorithms used for the strategy:- DIRECT, GAs, and SQP

Hybrid optimization approach is proposed.

Modification to DIRECT

Box selection rules are proposed and applied.

Hybrid optimization algorithm

It achieved the proposed strategy. It provided the efficient performance than the search only

by GAs.

Paper ListPaper List

Mitsunori Miki, Satoru Hiwa, Tomoyuki Hiroyasu“Simulated Annealing using an Adaptive Search Vector”Proceedings of IEEE International Conference on Cybernetics and Intelligent Systems 2006 (Bangkok, Thailand)

Proceeding of International Conference

The Science and Engineering Review of Doshisha University 三木光範，日和 悟，廣安知之

「 LED を用いた調色用照明システムの基礎的検討」同志社大学理工学研究報告 Vol.46 No.3 pp 9-18 ， 2005

Oral Presentation (in Japan) 日和 悟，廣安知之，三木光範

「大域的最適化のための複数最適化手法の動的制御法」日本機械学会 第 7 回最適化シンポジウム， 2006

日和 悟，廣安知之，三木光範「大域的最適化のための複数最適化手法の動的制御法」日本機械学会 第 6 回設計工学・システム部門講演会， 2006

三木光範，日和 悟，廣安知之「適応的探索ベクトルをもつシミュレーテッドアニーリング」日本機械学会 第 8 回計算力学講演会， 2005

Lipschitzian Optimization Lipschitzian Optimization [Shubert 1972][Shubert 1972]

It requires the user to specify the Lipschitz constant K

.

a bx1

Slope = −K Slope = +K

a bx1 x2 a bx1 x2x3

K is used as a prediction of the maximum possible slope of the objective function over the global domain.

– K

+K

DIRECT (one-dimensional)DIRECT (one-dimensional)

a bBox 1 Box 1Box 2 Box 3

Box 1Box 2 Box 3Box 4 Box 5

DIRECT (one-dimensional)DIRECT (one-dimensional)

a bBox 1 Box 1Box 2 Box 3

Box 1Box 2 Box 3Box 4 Box 5 Box 4

Box 1Box 5

Box 2

Slope = K1

Slope = K2

Slope = K

DIRECT (one-dimensional)DIRECT (one-dimensional)

a bBox 1 Box 1Box 2 Box 3

Box 1Box 2 Box 3Box 4 Box 5 Box 4

Box 1Box 5

Box 2

Slope = K1

Slope = K2

Slope = K

If box i is potentially optimal, then f(ci) <= f(cj) for all boxes that are of the same size as i.

In the largest boxes, the box with the best function value is potentially optimal.

DIRECT DIRECT ーー PPotentially Optimal Boxesotentially Optimal Boxes

DIRECT divides all potentially optimal boxes. Potentially optimal boxes are defined by:

Identification of potentially optimal boxes

A hyper box j is potentially optimal if there exists some such that

cj: center point of the box jdj: distance from the center point to vertices

DIRECT DIRECT ーー PPotentially Optimal Boxesotentially Optimal Boxes

DIRECT divides all potentially optimal boxes.

Identification of potentially optimal boxes

Search space

DIRECT DIRECT ーー PPotentially Optimal Boxesotentially Optimal Boxes

DIRECT divides all potentially optimal boxes.

dj

Search space

cj

Box j

Identification of potentially optimal boxes

DIRECT DIRECT ーー PPotentially Optimal Boxesotentially Optimal Boxes

DIRECT divides all potentially optimal boxes.

dj

Search space

Center - vertex distance (dj)

f (cj)cj

Box j

Identification of potentially optimal boxes

DIRECT DIRECT ーー PPotentially Optimal Boxesotentially Optimal Boxes

DIRECT divides all potentially optimal boxes.

dj

Center - vertex distance (dj)

f (cj)cj

Box j

( 0, fmin -ε| fmin | )

fmin

Identification of potentially optimal boxes

DIRECT DIRECT ーー PPotentially Optimal Boxesotentially Optimal Boxes

DIRECT divides all potentially optimal boxes.

dj

Center - vertex distance (dj)

f (cj)cj

Box j

( 0, fmin -ε| fmin | )

fmin

Make the convex hull which contains all points.

Identification of potentially optimal boxes

DIRECT DIRECT ーー PPotentially Optimal Boxesotentially Optimal Boxes

DIRECT divides all potentially optimal boxes.

dj

Center - vertex distance (dj)

f (cj)cj

Box j

( 0, fmin -ε| fmin | )

fmin : Potentially optimal

Boxes on the lower part of convex hull is selected as potentially optimal.

Identification of potentially optimal boxes

DIRECT DIRECT ーー PPotentially Optimal Boxesotentially Optimal Boxes

DIRECT divides all potentially optimal boxes.

dj

Center - vertex distance (dj)

f (cj)cj

Box j

: Potentially optimal

Boxes on the lower part of convex hull is selected as potentially optimal.

Search space

Identification of potentially optimal boxes

Genetic Algorithms (GAs)Genetic Algorithms (GAs)

Global search algorithm inspired by evolutionary biology.- Solutions are called ‘individuals’, and genetic operators

(Crossover, Selection, Mutation) are applied. Real-Coded GAs (RCGAs)

- Individuals are represented by real number vector.- Crossover operator significantly affects the searching ability.

Simplex Crossover (SPX)- One of the efficient crossover operator for RCGAs.- Generates offspring in a simplex, which is formed by n+1

individuals in n-dimensional space RCGAs using the SPX operator

- has both global and local searchcharacteristics.

RCGAs using the SPX operator are used.

GAs and SQPGAs and SQP

Gradient-based local search algorithm- By using gradient information, SQP rapidly converges to the

optimum.

GAs (Genetic Algorithms)

SQP (Sequential Quadratic Programming)

Heuristic algorithm inspired by evolutionary biology.- Solutions are called ‘individuals’, and genetic operators

(Crossover, Selection, Mutation) are applied.

Parents

ChildrenIndividuals

Stopping CriterionStopping Criterion

is terminated when the size of the best potentially optimal box is less than certain value prescribed.

A certain depth of search space exploration is obtained.

DIRECT

are terminated when their individuals converged. Spread of the individuals in design variable space:

xmax – xmin < threshold

GAs

SQP

continues its search until the improvement of solution becomes a minute value.

Stopping Criterion (DIRECT)Stopping Criterion (DIRECT)

is terminated when the longest side length of the best potentially optimal box is less than 10-3.

A certain depth of search space exploration is obtained.

DIRECT

Stopping Criterion (GAs)Stopping Criterion (GAs)

GAs

are terminated when their individuals converged. Spread of the individuals in design variable space:

Spreadi = xmax – xmin

xmax : the maximum value of i-th design variables in all individuals.

xmin : the minimum value of i-th design variables in all individuals.

If Spreadi is smaller than 10-3 × feasible range for all dimensions, GAs are terminated.

Spread1

Spread2

Population converged

Results (of each algorithm)Results (of each algorithm)

Function value

DIRECT(po 52)

GAs(Ind 100)

SQP HybridGAs only(Ind 100)

Average 3.52x10-2 1.23x10-4 9.07x10-8 9.07x10-8 5.58x102

St. Dev. 0.00 1.29x10-4 9.16x10-8 9.16x10-8 1.82x102

Num. of eval.

DIRECT GAs SQP Hybrid GAs only

Average 13529 115793 50 129373 279703

St. Dev. 0 9300 18 9307 22402

How to Combine DIRECT and GAsHow to Combine DIRECT and GAs

GAs utilize the center points of the potentially optimal boxes in DIRECT as their individuals.

If Npo > Nind

- Box selection rules are applied.

If Npo < Nind - Randomly generated individuals are added to GAs.

DIRECT stopped. GAs start.

1. Select two boxes, with the smallest size and with the largest from the set of potentially optimal boxes.

2. For each boxes, calculate the distance from two box.3. Sort the boxes by the distance in descending order,

and select N boxes from them.

Modification to DIRECTModification to DIRECT

Box selection rules

The number of potentially optimal boxes is reducedwithout breaking the global search characteristics of DIRECT.

Potentially optimal boxes near two boxes are discarded, and locally-biased search is prevented.

1. Select two boxes, with the smallest size and with the largest from the set of potentially optimal boxes.

2. For each boxes, calculate the distance from two box.3. Sort the boxes by the distance in descending order,

and select N boxes from them.

Modification to DIRECTModification to DIRECT

Box selection rules

The number of potentially optimal boxes is reducedwithout breaking the global search characteristics of DIRECT.

Potentially optimal boxes near two boxes are discarded, and locally-biased search is prevented.

Potentially optimal boxesPotentially optimal boxes(when DIRECT was terminated)(when DIRECT was terminated)

(x1, x2)

(x3, x6)

(x2, x5)

(x7, x9)

History of the search only by GAsHistory of the search only by GAs

(x1, x2)

(x3, x6)

(x2, x5)

(x7, x9)

History of the search only by GAsHistory of the search only by GAs

(x1, x2)

(x3, x6)

(x2, x5)

(x7, x9)

GAs were trapped to the local optima.