A Study of Priors and Algorithms for Signal Recovery

by Convex Optimization Techniques

Shunsuke Ono

Yamada Lab.

Dept. Communications and Integrated Systems

Tokyo Institute of Technology

2014/06/12

General Introduction

2

3

Signal Recovery Problem

Signal recovery is a fundamental problem in signal processing

• signal reconstruction

• image restoration

• compressed sensing

• tensor completion...

signal recovery problems = inverse problems of the form:

observation

Goal: estimate from and

noise contamination linear degradation

How to resolve this ill-posed/ill-conditioned problem?

unknown signal

4

Prior and Convex Optimization

Some a priori information on signal of interest, e.g.,• sparsity

• smoothness

• low-rankness

should be taken into consideration.

desired signal

convex function

convex set

1. D. P. Palomar and Y. C. Eldar, Eds., Convex Optimization in Signal Processing and Communications, Cambridge University Press, 2009.

2. J.-L. Starck et al., Sparse Image and Signal Processing: Wavelets, Curvelets,Morphological Diversity. Cambridge University Press, 2010.

3. H. H. Bauschke et al., Eds., Fixed-Point Algorithm for Inverse Problems in Science and Engineering, Springer-Verlag, 2011.

a-priori information convex function =: prior

• l1-norm [Donoho+ ‘03; Candes+ ‘06]

• total variation (TV) [Rudin+ ‘92; Chambolle ‘04]

• nuclear norm [Fazel ‘02; Recht et al. ‘11]

advantage: 1. local optimal = global optimal

2. flexible framework

A powerful approach: convex optimization [see, e.g., 1-3]

5

Optimization Algorithms for Signal Recovery

Optimization algorithms for signal recovery must deal with

• useful priors = nonsmooth convex function

• problem scale = often more than 10^4

proximal splitting methods [e.g., Gabay+ ‘76; Lions+ ‘79; Combettes+ ‘05; Condat ‘13]

• first-order (without Hessian)

• nonsmooth functions

• multiple constraints

Why do we need more?

Useful priorsEfficient

algorithms

Motivation & Goal

6

# prior: signal-specific properties are NOT fully exploited.

=> undesired results, e.g., texture degradation, color artifact, …

# algorithm: CANNOT deal with sophisticated constraints.

=> only the intersection of projectable convex sets.

Goal: design priors and algorithms to resolve them.

7

Structure of The Dissertation

Chap. 1 General Introduction

Chap. 2 Preliminaries

Chap. 3 Image restoration with component-wise

use of priors

Chap. 4 Blockwise low-rank prior for cartoon-texture

image decomposition and restoration

Chap. 5 Priors for color artifact reduction

in image restoration

Chap. 6 A hierarchical convex optimization algorithm

with primal-dual splitting

Chap. 7 An efficient algorithm for signal recovery with

sophisticated data-fidelity constraints

Chap. 8 General conclusion

Main

chapters

priors

algorithms

8

Structure of The Dissertation

Chap. 1 General Introduction

Chap. 2 Preliminaries

Chap. 3 Image restoration with component-wise

use of priors

Chap. 4 Blockwise low-rank prior for cartoon-texture

image decomposition and restoration

Chap. 5 Priors for color artifact reduction

in image restoration

Chap. 6 A hierarchical convex optimization algorithm

with primal-dual splitting

Chap. 7 An efficient algorithm for signal recovery with

sophisticated data-fidelity constraints

Chap. 8 General conclusion

priors

algorithms

Main

chapters

9

Chap. 4 Blockwise low-rank prior for cartoon-texture

image decomposition and restoration

Background

10

Cartoon-Texture Decomposition Model

11

image cartoon texture

Assumption: image = sum of two components

optimization problem [Meyer ‘01; Vese+ ‘03; Aujol+ ’05; ~ Schaeffer+ ‘13 ]

advantage: 1. prior suitable to each component

2. extraction of texture

priors for each component data-fidelity to

Cartoon-Texture Decomposition Model

12

image

optimization problem [Meyer ‘01; Vese+ ‘03; Aujol+ ‘05; ~ Schaeffer+ ‘13 ]

total variation (TV) [Rudin 92]: suitable for cartoon

cartoon texture

Cartoon-Texture Decomposition Model

13

image

optimization problem [Meyer ‘01; Vese+ ‘03; Aujol+ ‘05; ~ Schaeffer+ ‘13 ]

Texture is rather difficult to model…

cartoon texture

Existing Texture Priors

14

G norm frame2001 ~ 2005 ~ 2013

noise

fine pattern

local adaptivity

capability

prior

->

[Schaeffer+ ‘13][Meyer ‘01]

[Aujol+ ‘05]

[Ng+ ‘13]

[Daubechies+ ‘05]

[Elad+ ’05]

[Fadili+ ‘10]

Contribution

15

Propose a prior for a better interpretation of texture.

2001 ~ 2005 ~ 2013

Noise

Fine pattern

Local adaptivity

Proposed

prior

G norm frame[Schaeffer+ ‘13][Meyer ‘01]

[Aujol+ ‘05]

[Ng+ ‘13]

[Daubechies+ ‘05]

[Elad+ ’05]

[Fadili+ ‘10]

SVD, Rank and Nuclear Norm

16

• singular value decomposition (SVD)

nonzero singular values

• number of nonzero singular values

• nuclear norm

tightest convex relaxation of rank [Fazel 02]

* applications to robust PCA (sparse + low-rank)[Gandy & Yamada ‘10; Candes+ ‘11; Gandy, Recht, Yamada ‘11]

rank

nuclear norm

Proposed Method

17

How To Model Texture?

18

globally dissimilar but locally well-patterned

Any block is approximately low-rank after suitable shear.

Any block is approximately low-rank after suitable shear.

Proposed Prior: Block Nuclear Norm (1/2)

19

Definition: pre-Block-nuclear-norm (pre-BNN)

nuclear normpositive weight

Important property of pre-BNN

Pre-BNN is tightest convex relaxation of

weighted blockwise rank

* Generalization of [Fazel ‘02]

Proposed Prior: Block Nuclear Norm (2/2)

20

Definition: Block Nuclear Norm (BNN)

periodic expansion operator (overlap) shear operator

Any block is approximately low-rank after suitable shear.

BNN becomes small,

i.e., good texture prior.

Cartoon-Texture Decomposition Using BNN

21image cartoon texture

• Patterns running in different directions are separately extracted.

• Proximal splitting methods can solve the problem after reformulation.

proposed cartoon-texture decomposition model

texture (K=3) sub-texture 1 sub-texture 2 sub-texture 3

various shear angles

22

Experimental ResultsCASE 1: pure decomposition compared with a state-of-the-art decomposition [Schaeffer & Osher, 2013]

image

cartoon

texture

cartoon

texture

[Schaeffer & Osher 2013] “A low patch-

rank interpretation of texture,” SIAM J.

Imag. Sci. [Schaeffer & Osher 2013] proposed

23

Experimental ResultsCASE 2: blur+20%missing pixelscompared also with [Schaeffer & Osher 2013]

PSNR: 23.20

SSIM: 0.6613

PSNR: 23.75

SSIM: 0.6978

observation [Schaeffer & Osher 2013] proposed

24

Experimental ResultsCASE 2: blur+20%missing pixelsCompared with a state-of-the-art decomposition [Schaeffer & Osher, 2013]

PSNR: 23.20

SSIM: 0.6613

PSNR: 23.75

SSIM: 0.6978

observation [Schaeffer & Osher 2013] proposed

25

Chap. 5 Priors for color artifact reduction

in image restoration

Background

26

Color Artifact in Image Restoration

27

restored by an existing prior

original

observation

color artifact

Color-Line Property

28color-line

restored by

an existing priororiginal

corrupted

# color-line: RGB entries are linearly distributed in local regions.[Omer & Werman ‘04]

Contribution

29color-line

original

corrupted

Propose a prior for promoting color-line property.

reconstructed

existing +

proposed prior

restored by

an existing prior

Proposed Method

30

Mathematical Modeling of Color-Line

31

color image

B G R

-th local region

(e.g., block)

Vectorize

matrix for

-th local region

Define matrices for every local region of a color image.

color-line property low-rankness of

Proposed Prior: Local Color Nuclear Norm

32

number of local regions

key principle

rank( ) = 1exact cases

Local Color Nuclear Norm (LCNN)

Proposed Prior: Local Color Nuclear Norm

33

key principle

color-line property small singular values of

practical casesrank( ) ≠ 1

but is small

Suppressing LCNN promotes the color-line property.

Local Color Nuclear Norm (LCNN)

Application to Denoising

34

color-line

Proximal splitting methods are applicable after reformulation.

smoothness [VTV]

dynamic range

data-fidelity robust to

Impulsive noise

: color image contaminated by impulsive noise

optimization problem

[VTV] Bresson et al. “Fast dual minimization of the vectorial total variation norm and

applications to color image processing”, Inverse Probl. Img., 2008.

Experimental Results

35

observation VTV VTV+LCNN original

22.95, 7.59 24.30, 5.80

25.12, 3.07 27.22, 2.58

(PSNR, D2000) (PSNR, D2000)

36

Chap. 6 A hierarchical convex optimization algorithm

with primal-dual splitting

Background

37

NOT uniqueUnique

NOT strictly convexStrictly convex

.

38

contains infinitely many solutions non-strict convexity of .

Solutions of Convex Optimization Problems

Solution set of a convex optimization problem

Solutions could be considerably different in another criterion.

39

Hierarchical Convex Optimization

ideal strategy: hierarchical convex optimization:

highly involved (≠the intersection of projectable convex sets)

proximal splitting methods cannot solve the problem.

selector: smooth convex function

via fixed point set characterization[e.g., Yamada ‘01; Ogura & Yamada‘03; Yamada, Yukawa, Yamagishi ‘11]

Definition: nonexpansive mapping

computable nonexpansive mapping on a certain Hilbert space

40

Hierarchical Convex Optimization

fixed point set characterized problem

Hybrid Steepest Descent Method (HSDM) [e.g., Yamada ‘01; Ogura & Yamada ‘03]

nonexpansive mapping gradient of selector

Q. What kinds of are available?

41

• Forward-Backward Splitting (FBS) method [Passty ’79; Combettes+ ‘05]

• Douglas-Rachford Splitting (DRS) method [Lions+ ‘79; Combettes+ ‘07]

Two characterizations underlying proximal splitting methods

are given in [Yamada, Yukawa, Yamagishi ‘11].

Q. Can we deal with a more flexible formulation?

Nonexpansive Mappings for

Definition: proximity operator [Moreau ‘62]

42

• Forward-Backward Splitting (FBS) method [Passty ’79; Combettes+ ‘05]

• Douglas-Rachford Splitting (DRS) method [Lions+ ’79; Combettes+ ‘07]

• Primal-Dual Splitting (PDS) method [Condat ‘13; Vu ‘13]

Nonexpansive Mappings for

Two characterizations underlying proximal splitting methods

are given in [Yamada, Yukawa, Yamagishi ‘11].

43

• Primal-Dual Splitting (PDS) method [Condat ‘13; Vu ‘13]

Contribution

• reveal convergence properties

• modify gradient computation

• extract operator-theoretic idea from [Condat 13]

• reformulate in a certain product space

incorporate

hierarchical convex optimization by HSDM

Proposed Method

44

45

Outline

Reformulate in the canonical product space with dual problem

Extract & incorporate fixed point set characterization from [Condat ‘13]

Install another inner product for nonexpansivity of by [Condat ‘13]

Apply HSDM with modified gradient computation w.r.t.

46

Reformulation in The Canonical Product Space

solution set of the first stage problem (=primal problem)

solution set of the dual problem of the first stage problem

By letting

Note:

47

Incorporation of PDS Characterization

Extract the PDS fixed point characterization from [Condat ‘13]

48

Activation of Nonexpansivity

is nonexpansive NOT on the canonical product space

Definition: canonical inner product of

BUT on the following space with another inner product [Condat ‘13]

where

: strongly positive bounded linear operator

49

Solver via HSDM

NOTE:

We can apply HSDM [e.g., Yamada ‘01; Ogura & Yamada ‘03]

50

Convergence of HSDM with PDS

Assumptions:

Convergence 1:

Convergence 2:

Recall

Definition: distance function

51

Application to Signal Recoveryunknown signal

Gaussian noisedegradation

observation model:

first stage problem:

priornumerical rangedata-fidelity

hierarchical convex optimization problem:

non-strictly convex

another prior

to specify

a better solution

Definition: indicator function

52

Application to Signal Recoveryunknown signal

Gaussian noisedegradation

observation model:

first stage problem:

hierarchical convex optimization problem:

non-strictly convex

another prior

to specify

a better solution

Definition: indicator function

53

Experimental Results

original

observed

no-

hierarchical

proposed

54

General Conclusion

Chap. 3 Image restoration with component-wise

use of priors

Chap. 4 Blockwise low-rank prior for cartoon-texture

image decomposition and restoration

Chap. 5 Priors for color artifact reduction

in image restoration

Chap. 6 A hierarchical convex optimization algorithm

with primal-dual splitting

Chap. 7 An efficient algorithm for signal recovery with

sophisticated data-fidelity constraints

priors: to model signal-specific properties

algorithms: to deal with involved constraints

We have developed novel priors and algorithms for signal recovery.

Related Publications

55

# Journal Papers[J1] S. Ono, T. Miyata, I. Yamada, and K. Yamaoka, "Image Recovery by

Decomposition with Component-Wise Regularization,"

IEICE Trans. Fundamentals, vol. E95-A, no. 12, pp. 2470-2478, 2012.

(Best Paper Award from IEICE)

[J2] S. Ono, T. Miyata, and I. Yamada, "Cartoon-Texture Image Decomposition

Using Blockwise Low-Rank Texture Characterization,"

IEEE Trans. Image Process., vol. 23, no. 3, pp. 1028-1042, 2014.

[J3] S. Ono and I. Yamada, "Hierarchical Convex Optimization with Primal-Dual

Splitting,“ submitted to IEEE Trans. Signal Process (accepted conditionally

in May. 2014).

[J4] S. Ono and I. Yamada, "Signal Recovery Using Complicated Data-Fidelity

Constraints,“ in preparation.

Related Publications

56

# Articles in Proceedings of International Conferences (reviewed)[C1] S. Ono, T. Miyata, and K. Yamaoka, "Total Variation-Wavelet-Curvelet

Regularized Optimization for Image Restoration," IEEE ICIP 2011.

[C2] S. Ono, T. Miyata, I. Yamada, and K. Yamaoka, "Missing Region Recovery by

Promoting Blockwise Low-Rankness," IEEE ICASSP 2012.

[C3] S. Ono and I. Yamada, "A Hierarchical Convex Optimization Approach for High

Fidelity Solution Selection in Image Recovery,'' APSIPA ASC 2012, (Invited).

[C4] S. Ono and I. Yamada, "Poisson Image Restoration with Likelihood Constraint

via Hybrid Steepest Descent Method," IEEE ICASSP 2013.

[C5] S. Ono, M. Yamagishi, and I. Yamada, "A Sparse System Identification by Using

Adaptively-Weighted Total Variation via A Primal-Dual Splitting Approach,"

IEEE ICASSP 2013.

[C6] S. Ono and I. Yamada, "A Convex Regularizer for Reducing Color Artifact in

Color Image Recovery,“ IEEE Conf. CVPR 2013.

[C7] I. Yamada and S. Ono, "Signal Recovery by Minimizing The Moreau Envelope

over The Fixed Point Set of Nonexpansive Mappings," EUSIPCO 2013, (invited).

[C8] S. Ono and I. Yamada, “Second-Order Total Generalized Variation Constraint,”

IEEE ICASSP 2014.

[C9] S. Ono and I. Yamada, “Decorrelated Vectorial Total Variation,” IEEE Conf. CVPR

2014 (to appear).

Other Publications

57

# Journal Papers[J5] S. Ono, T. Miyata, and Y. Sakai, "Improvement of Colorization Based Coding by

Using Redundancy of The Color Assignment Information and Correct Color

Component," IEICE Trans. Information and Systems, vol. J93-D, no. 9, pp.

1638-1641, 2010 (in Japanese).

[J6] H. Kuroda, S. Ono, M. Yamagishi, and I. Yamada, "Exploiting Group Sparsity in

Nonlinear Acoustic Echo Cancellation by Adaptive Proximal Forward-Backward

Splitting," IEICE Trans. Fundamentals, vol.E96-A, no.10, pp.1918-1927, 2013.

[J7] T. Baba, R. Matsuoka, S. Ono, K. Shirai, and M. Okuda, "Image Composition Using

A Pair of Flash/No-Flash Images by Convex Optimization,“ IEICE Transactions on

Information and System, 2014 (in Japanese, to appear)

Other Publications

58

# Articles in Proceedings of International Conference (reviewed)[C10] S. Ono, T. Miyata, and Y. Sakai, "Colorization-Based Coding by Focusing on

Characteristics of Colorization Bases," PCS 2010.

[C11] M. Yamagishi, S. Ono, and I. Yamada, "Two Variants of Alternating Direction

Method of Multipliers without Inner Iterations and Their Application to Image

Super-Resolution,'' IEEE ICASSP 2012.

[C12] S. Ono and I. Yamada, "Optimized JPEG Image Decompression with Super-

Resolution Interpolation Using Multi-Order Total Variation," IEEE ICIP 2013

(top 10% of all accepted papers).

[C13] K. Toyokawa, S. Ono, M. Yamagishi, and I. Yamada, "Detecting Edges of

Reflections from a Single Image via Convex Optimization,“ IEEE ICASSP 2014.

[C14] T. Baba, R. Matsuoka, S. Ono, K. Shirai, and M. Okuda, "Flash/No-flash Image

Integration Using Convex Optimization,“ IEEE ICASSP 2014.

* Many other articles in proceedings of domestic conferences