Noisy Video Super-Resolution
description
Transcript of Noisy Video Super-Resolution
![Page 1: Noisy Video Super-Resolution](https://reader036.fdocument.pub/reader036/viewer/2022062501/56816207550346895dd23157/html5/thumbnails/1.jpg)
Noisy Video Super-Resolution
Feng Liu, JinjunWang,ShenghuoZhu (MM’08)University of Wisconsin-Madison, NEC Laboratories America, Inc.
第一組: 資訊四 B95902105 黃彥達 資訊碩一 R98922046 蔡旻光 網媒碩二 R97944012 鄒志鴻
![Page 2: Noisy Video Super-Resolution](https://reader036.fdocument.pub/reader036/viewer/2022062501/56816207550346895dd23157/html5/thumbnails/2.jpg)
2
Outline
Introduction Goal File Format
Noise Reduced Image Proposed Approach Motion Estimation & Estimated Super-
Resolution Result Implementation Result Conclusion
![Page 3: Noisy Video Super-Resolution](https://reader036.fdocument.pub/reader036/viewer/2022062501/56816207550346895dd23157/html5/thumbnails/3.jpg)
3
Introduction
Low-quality videos often not only have limited resolution but also suffer from noise In fact, the requirements of de-noising & super-
resolution is quite similar
This paper present a unified framework which achieves simultaneous video de-noising and super-resolution algorithm by some measurements of visual quality
![Page 4: Noisy Video Super-Resolution](https://reader036.fdocument.pub/reader036/viewer/2022062501/56816207550346895dd23157/html5/thumbnails/4.jpg)
Goal
Refine low-quality videos from YouTube, and make the video better effects, which has better quality by human eyes.
Input is low-quality and noise-included (block effects or somewhat noise) videos
![Page 5: Noisy Video Super-Resolution](https://reader036.fdocument.pub/reader036/viewer/2022062501/56816207550346895dd23157/html5/thumbnails/5.jpg)
Noise-Reduced Image
mv-SAD Gaussian-space
Gaussian-time
| p(I,j) – p(i’, j’) | > threshold
![Page 6: Noisy Video Super-Resolution](https://reader036.fdocument.pub/reader036/viewer/2022062501/56816207550346895dd23157/html5/thumbnails/6.jpg)
Gaussian Space
Frame t
Pixel(I,j)
Standard deviation
Set Mean = 0
![Page 7: Noisy Video Super-Resolution](https://reader036.fdocument.pub/reader036/viewer/2022062501/56816207550346895dd23157/html5/thumbnails/7.jpg)
Motion Vector
Frame tPixel ( i , j , t)
Frame t+1
Pixel ( i + mv_i , j + mv_j , t+1)
(mv_i , mv_j)
![Page 8: Noisy Video Super-Resolution](https://reader036.fdocument.pub/reader036/viewer/2022062501/56816207550346895dd23157/html5/thumbnails/8.jpg)
Gaussian Time
Fram
e t
- 2Fram
e t
- 1Fram
e t
Space Gaussian
Time Gaussian
Pixel(I,j)
Fram
e
t+1Fram
e
t+2Fram
e t
![Page 9: Noisy Video Super-Resolution](https://reader036.fdocument.pub/reader036/viewer/2022062501/56816207550346895dd23157/html5/thumbnails/9.jpg)
Noise-Reduced ImageBefore After
![Page 10: Noisy Video Super-Resolution](https://reader036.fdocument.pub/reader036/viewer/2022062501/56816207550346895dd23157/html5/thumbnails/10.jpg)
10
Proposed Approach – 1 / 4 Consider the visual quality with respect to
the following 3 aspects: Fidelity Preserving▪ To achieve similar high-resolution result
Detail Preserving▪ Enhanced details (edge)
Spatial-Temporal Smoothness▪ Remove undesirable high-frequency contents (e.g. jitter)
![Page 11: Noisy Video Super-Resolution](https://reader036.fdocument.pub/reader036/viewer/2022062501/56816207550346895dd23157/html5/thumbnails/11.jpg)
11
Proposed Approach – 2 / 4 Fidelity Preserving
Conventional metrics:▪ Measure fidelity by the difference between Ih & Il would
be problematic & waste useful time-space information in video
Proposed metrics:▪ Estimate an approximation of super-resolution results
from space-time neighboring pixels▪ The fidelity measurement:
see next page for details
noised
![Page 12: Noisy Video Super-Resolution](https://reader036.fdocument.pub/reader036/viewer/2022062501/56816207550346895dd23157/html5/thumbnails/12.jpg)
12
Proposed Approach – 3 / 4 Detail Preserving
Enhanced details (edge)
Contrast preserving▪ Human visual system is more sensitive to contrast
than pixel values▪ Gradient fields of Ih & should be close
,where Wk is one or zero if the patchk with/o edges (canny detector)
![Page 13: Noisy Video Super-Resolution](https://reader036.fdocument.pub/reader036/viewer/2022062501/56816207550346895dd23157/html5/thumbnails/13.jpg)
13
Proposed Approach – 4 / 4 (Spatial-Temporal) Smoothness
Smooth results are often favored by the human system Encourage to minimize:
A 2-D Laplace filter may be
Spatial-temporal Laplacian
OR
![Page 14: Noisy Video Super-Resolution](https://reader036.fdocument.pub/reader036/viewer/2022062501/56816207550346895dd23157/html5/thumbnails/14.jpg)
14
An Optimization Problem
Proposed Measurements
A quadratic minimization problem to solve (AX = b):
Contrast
Similarity
Detail Information(edge)
Spatial-Temporal Smoothness
![Page 15: Noisy Video Super-Resolution](https://reader036.fdocument.pub/reader036/viewer/2022062501/56816207550346895dd23157/html5/thumbnails/15.jpg)
15
Implementation – 1 / 2
InputlowhI~
hI~
hG~
hI~
)1,()1,(~~
tIwtIw hmv
hmv
= X
I
6 -1 … -1-1 6 -1 … -1 -1 6 -1 … -1
Laplacian
Gradient-1 0 1 … 1 -1 0 1 … 1 -1 0 1 … 1
EdgeMinimize
Motion Estimation
+
Result (X)
Fidelity
Bilateral filter
mvw
fidv
fidg
dt
sm
![Page 16: Noisy Video Super-Resolution](https://reader036.fdocument.pub/reader036/viewer/2022062501/56816207550346895dd23157/html5/thumbnails/16.jpg)
16
Implementation – 2 / 2
Adjustments for the weight terms The measurement term is more emphasized if
the weight is larger
By iteratively experiments for our test data, we took
However, we found that for different videos, the best weight sets may be also different
3.0,1.0,1,1 smdtfidgfidv
![Page 17: Noisy Video Super-Resolution](https://reader036.fdocument.pub/reader036/viewer/2022062501/56816207550346895dd23157/html5/thumbnails/17.jpg)
17
Result
352 x 288 Result
![Page 18: Noisy Video Super-Resolution](https://reader036.fdocument.pub/reader036/viewer/2022062501/56816207550346895dd23157/html5/thumbnails/18.jpg)
18
Conclusion
The proposed framework formulates noisy video super-resolution as an optimization problem, aiming to maximize the visual quality of the result
The measurements of fidelity-preserving, detail-preserving and smoothness are considered to maximize the visual quality results
![Page 19: Noisy Video Super-Resolution](https://reader036.fdocument.pub/reader036/viewer/2022062501/56816207550346895dd23157/html5/thumbnails/19.jpg)
19
Thank you!!