Image and Video Coding I: [0.5ex] Fundamentalsiphome.hhi.de/schwarz/assets/ivc1/Introduction.pdf ·...
-
Upload
truongminh -
Category
Documents
-
view
214 -
download
0
Transcript of Image and Video Coding I: [0.5ex] Fundamentalsiphome.hhi.de/schwarz/assets/ivc1/Introduction.pdf ·...
Image and Video Coding I:Fundamentals
Heiko Schwarz
Freie Universität BerlinFachbereich Mathematik und Informatik
Organization
Vorlesung: Montag 14:15-15:45Raum SR 006 / T9
Übung: Montag 12:30-14:00Raum SR 006 / T9
Material: http://iphome.hhi.de/schwarz/IVC1.htm
Literatur:Cover, T. M. and Thomas, J. A. (2006), “Elements of Information Theory,”John Wiley & Sons, New York.
Gersho, A. and Gray, R. M. (1992), “Vector Quantization and Signal Compression,”Kluwer Academic Publishers, Boston, Dordrecht, London.
Jayant, N. S. and Noll, P. (1994), “Digital Coding of Waveforms,”Prentice-Hall, Englewood Cliffs, NJ, USA.
Wiegand, T. and Schwarz, H. (2010),“Source Coding: Part I of Fundamentals of Source and Video Coding,”Foundations and Trends in Signal Processing, vol. 4, no. 1-2. (available on web page)
H. Schwarz (FU Berlin) — Image and Video Coding
Motivation
Motivation for Source Coding
Source Coding / Data Compression
Efficient transmission or storage of dataUse less throughput for given dataTransmit more data for given throughput
Source Coding: Enabling TechnologyEnables new applicationsMakes applications economically feasible
ExamplesDistribution of digital imagesDistribution of digital audio (first mobile audio players)Digitial television (DVB-T, DVB-T2, DVB-S, DVB-S2)Internet video streaming (YouTube, Netflix, Amazon, ...)
H. Schwarz (FU Berlin) — Image and Video Coding: Introduction 2 / 20
Motivation
Practical Source Coding Examples
File CompressionCompress large document archivesGZIP, WinZIP, WinRar, ...
Audio CompressionCompress audio for storage on mobile deviceFLAC, MP3, AAC, ...
Image CompressionCompress images for storage and distributionRaw camera formats, JPEG, JPEG-2000, JPEG-XR, ...
Video CompressionCompress video for streaming, broadcast, or storageMPEG-2, H.264 | AVC, VP9, H.265 | HEVC, ...
H. Schwarz (FU Berlin) — Image and Video Coding: Introduction 3 / 20
Motivation
Types of Compression
Lossless Compression
Invertible / reversible form of data compressionOriginal input data can be completely recoveredRequired for document compression
Examples: Lempel-Ziv coding (GZIP), FLAC, JPEG-LS
Lossy CompressionNot invertibleOnly approximation of original input data can be recoveredAchieves much higher compression ratiosDominant form of compression for media data (audio, images, video)
Examples: MP3, AAC for audioJPEG, JPEG-2000 for imagesMPEG-2, H.264 | AVC, H.265 | HEVC for video
H. Schwarz (FU Berlin) — Image and Video Coding: Introduction 4 / 20
Source Data Sampling and Quantization
Source Data
Analog Signals
Continuous-time/space and continuous-amplitude signalsTypically resulting from physical measurementsMost signals in reality (audio and visual signals)
Digital Signals
Discrete-time/space and discrete-amplitude signalsOften generated from analog signalSometimes directly measured
Input Data for Source CodingRequire digital signalsNeed to be stored and processed with a computerCompression methods are computer programs
H. Schwarz (FU Berlin) — Image and Video Coding: Introduction 5 / 20
Source Data Sampling and Quantization
Analog-to-Digital Conversion
Sampling
Maps continuous-time/space signal into discrete-time/space signal
QuantizationMaps continuous-amplitude signal into discrete-amplitude signalSimplest case: Rounding
H. Schwarz (FU Berlin) — Image and Video Coding: Introduction 6 / 20
Source Data Sampling and Quantization
Impact of Sampling (Spatial Resolution)
400×300 samples 200×150 samples
100×75 samples 50×38 samples
H. Schwarz (FU Berlin) — Image and Video Coding: Introduction 7 / 20
Source Data Sampling and Quantization
Impact of Quantization (Bit Depth)
8 bits per component 4 bits per component
3 bits per component 2 bits per component
H. Schwarz (FU Berlin) — Image and Video Coding: Introduction 8 / 20
Source Data Sampling and Quantization
Raw Images and Videos
Single-Component ImageMatrix of integer samples
s[x , y ]
Characterized byNumber of samples in horizontaland vertical direction W×H
Sample bit depth B
x
y
Color ImagesThree color components (typically RGB or YCbCr)Additionally characterized by color sampling format
VideosSequence of imagesAdditionally characterized by frame rate
H. Schwarz (FU Berlin) — Image and Video Coding: Introduction 9 / 20
Source Data Sampling and Quantization
Color Sampling Formats
RGB YCbCr 4:4:4 YCbCr 4:2:2 YCbCr 4:2:0
mostcommonformat
H. Schwarz (FU Berlin) — Image and Video Coding: Introduction 10 / 20
Source Data Sampling and Quantization
Raw Data Rate
Raw Data RateBit rate R of raw data format
R = (samples per time unit) · (bit depth per sample) (1)
Example: Full High Definition (HD) Video
1920×1080 luma samples, 50 frames per second (Europe)4:2:0 chroma format (chroma components with 1/4 luma resolution)8 bits per sampleraw data rate = 50Hz · 1920 · 1080 · (1+ 2/4) · 8 bits ≈ 1.24Gbit/s
Example: Ultra High Defintion (UHD) Video
3840×2160 luma samples, 60 frames per second (USA, Japan)4:2:0 chroma format, 10 bits per sampleraw data rate = 60Hz · 3840 · 2160 · (1+ 2/4) · 10 bits ≈ 7.5Gbit/s
H. Schwarz (FU Berlin) — Image and Video Coding: Introduction 11 / 20
The Source Coding Problem
Typical Video Communication Scenario
capture
raw inputvideo samples
raw inputvideo samples
pre-processing
videoencodervideo
encoder
transmission channel (can be replaced by storage)
channelencodermodulatorchanneldemodulatorchannel
decoder
videodecodervideo
decoderpost-
processing
raw outputvideo samplesraw output
video samples
display andperception
encodedbitstream
receivedbitstream
no transmission errorsbitstream
H. Schwarz (FU Berlin) — Image and Video Coding: Introduction 12 / 20
The Source Coding Problem
Typical Video Communication Scenario
capture
raw inputvideo samples
raw inputvideo samples
pre-processing
videoencodervideo
encoder
transmission channel (can be replaced by storage)
channelencodermodulatorchanneldemodulatorchannel
decoder
videodecodervideo
decoderpost-
processing
raw outputvideo samplesraw output
video samples
display andperception
encodedbitstream
receivedbitstream
no transmission errorsbitstream
H. Schwarz (FU Berlin) — Image and Video Coding: Introduction 12 / 20
The Source Coding Problem
Typical Video Communication Scenario
capture
raw inputvideo samples
raw inputvideo samples
pre-processing
videoencodervideo
encoder
transmission channel (can be replaced by storage)
channelencodermodulatorchanneldemodulatorchannel
decoder
videodecodervideo
decoderpost-
processing
raw outputvideo samplesraw output
video samples
display andperception
encodedbitstream
receivedbitstream
no transmission errorsbitstream
H. Schwarz (FU Berlin) — Image and Video Coding: Introduction 12 / 20
The Source Coding Problem
Application Examples
HD movie UHD broadcaston Blu-ray Disc over DVS-S2
raw video format
1920×1080 luma samples 3840×2160 luma samples4:2:0 chroma format 4:2:0 chroma format8 bits per sample 10 bits per sample24 frames per second 60 frames per second
raw data rate ca. 600 Mbit/s ca. 7.5 Gbit/s
channel bit rate 36 Mbit/s (read speed) 58 Mbit/s (8PSK 2/3)
video bit rate ca. 20 Mbit/s ca. 25 Mbit/s
required compression ca. 30:1 ca. 300:1
H. Schwarz (FU Berlin) — Image and Video Coding: Introduction 13 / 20
The Source Coding Problem
The Basic Communication / Source Coding Problem
Basic Source Coding ProblemTwo equivalent formulations:
Representing source data with highest fidelity possiblewithin an available bit rate
and
Representing source data using lowest bit rate possiblewhile maintaining a specified reproduction quality
Source CodecSource codec: System of encoder and decoder
encoder decoderbitstream
H. Schwarz (FU Berlin) — Image and Video Coding: Introduction 14 / 20
The Source Coding Problem
Practical Communication / Source Coding Problem
Characteristics of Source Codecs / Overall Communication
Bit rate: Throughput of the communication channelQuality: Fidelity of the reconstructed signalDelay: Start-up latency, end-to-end delayComplexity: Computation, memory, memory access
Practical Source Coding Problem
Given a maximum allowed complexity and a maximum delay,achieve an optimal trade-off between bit rate and reconstructionquality for the transmission problem in the targeted application
In this course:Will concentrate on source codec
Ignore aspects of transmission channel (e.g., transmission errors)
H. Schwarz (FU Berlin) — Image and Video Coding: Introduction 15 / 20
Coding Efficiency
Image Compression Example
100%
Original Image (1024×678 image points, 945 KB)Lossless Compressed: GZIP 67.03%Lossless Compressed: PNG 46.50%Lossy Compressed: JPEG (Quality 95) 12.03%Lossy Compressed: JPEG (Quality 75) 3.18%Lossy Compressed: JPEG (Quality 50) 1.94%Lossy Compressed: JPEG (Quality 25) 1.11%Lossy Compressed: JPEG (Quality 1) 0.24%
H. Schwarz (FU Berlin) — Image and Video Coding: Introduction 16 / 20
Coding Efficiency
Image Compression Example
100%
Original Image (1024×678 image points, 945 KB)Lossless Compressed: GZIP 67.03%Lossless Compressed: PNG 46.50%Lossy Compressed: JPEG (Quality 95) 12.03%Lossy Compressed: JPEG (Quality 75) 3.18%Lossy Compressed: JPEG (Quality 50) 1.94%Lossy Compressed: JPEG (Quality 25) 1.11%Lossy Compressed: JPEG (Quality 1) 0.24%
H. Schwarz (FU Berlin) — Image and Video Coding: Introduction 16 / 20
Coding Efficiency
Image Compression Example
100%
Original Image (1024×678 image points, 945 KB)Lossless Compressed: GZIP 67.03%Lossless Compressed: PNG 46.50%Lossy Compressed: JPEG (Quality 95) 12.03%Lossy Compressed: JPEG (Quality 75) 3.18%Lossy Compressed: JPEG (Quality 50) 1.94%Lossy Compressed: JPEG (Quality 25) 1.11%Lossy Compressed: JPEG (Quality 1) 0.24%
H. Schwarz (FU Berlin) — Image and Video Coding: Introduction 16 / 20
Coding Efficiency
Image Compression Example
100%
Original Image (1024×678 image points, 945 KB)Lossless Compressed: GZIP 67.03%Lossless Compressed: PNG 46.50%Lossy Compressed: JPEG (Quality 95) 12.03%Lossy Compressed: JPEG (Quality 75) 3.18%Lossy Compressed: JPEG (Quality 50) 1.94%Lossy Compressed: JPEG (Quality 25) 1.11%Lossy Compressed: JPEG (Quality 1) 0.24%
H. Schwarz (FU Berlin) — Image and Video Coding: Introduction 16 / 20
Coding Efficiency
Image Compression Example
100%
Original Image (1024×678 image points, 945 KB)Lossless Compressed: GZIP 67.03%Lossless Compressed: PNG 46.50%Lossy Compressed: JPEG (Quality 95) 12.03%Lossy Compressed: JPEG (Quality 75) 3.18%Lossy Compressed: JPEG (Quality 50) 1.94%Lossy Compressed: JPEG (Quality 25) 1.11%Lossy Compressed: JPEG (Quality 1) 0.24%
H. Schwarz (FU Berlin) — Image and Video Coding: Introduction 16 / 20
Coding Efficiency
Image Compression Example
100%
Original Image (1024×678 image points, 945 KB)Lossless Compressed: GZIP 67.03%Lossless Compressed: PNG 46.50%Lossy Compressed: JPEG (Quality 95) 12.03%Lossy Compressed: JPEG (Quality 75) 3.18%Lossy Compressed: JPEG (Quality 50) 1.94%Lossy Compressed: JPEG (Quality 25) 1.11%Lossy Compressed: JPEG (Quality 1) 0.24%
H. Schwarz (FU Berlin) — Image and Video Coding: Introduction 16 / 20
Coding Efficiency
Image Compression Example
100%
Original Image (1024×678 image points, 945 KB)Lossless Compressed: GZIP 67.03%Lossless Compressed: PNG 46.50%Lossy Compressed: JPEG (Quality 95) 12.03%Lossy Compressed: JPEG (Quality 75) 3.18%Lossy Compressed: JPEG (Quality 50) 1.94%Lossy Compressed: JPEG (Quality 25) 1.11%Lossy Compressed: JPEG (Quality 1) 0.24%
H. Schwarz (FU Berlin) — Image and Video Coding: Introduction 16 / 20
Coding Efficiency
Image Compression Example
100%
Original Image (1024×678 image points, 945 KB)Lossless Compressed: GZIP 67.03%Lossless Compressed: PNG 46.50%Lossy Compressed: JPEG (Quality 95) 12.03%Lossy Compressed: JPEG (Quality 75) 3.18%Lossy Compressed: JPEG (Quality 50) 1.94%Lossy Compressed: JPEG (Quality 25) 1.11%Lossy Compressed: JPEG (Quality 1) 0.24%
H. Schwarz (FU Berlin) — Image and Video Coding: Introduction 16 / 20
Coding Efficiency
Trade-Off between Quality and Compression Ratio
Coding EfficiencyAbility to trade-off bit rate and reconstruction qualityWant best reconstruction quality for a given bitrate (or vice versa)
codec A
bettercodec B
bit rate
quality
H. Schwarz (FU Berlin) — Image and Video Coding: Introduction 17 / 20
Coding Efficiency
Coding Efficiency Example
1:100 Compression: JPEG1:100 Compression: H.265 | HEVC
H. Schwarz (FU Berlin) — Image and Video Coding: Introduction 18 / 20
Coding Efficiency
Coding Efficiency Example
1:100 Compression: JPEG1:100 Compression: H.265 | HEVC
H. Schwarz (FU Berlin) — Image and Video Coding: Introduction 18 / 20
Coding Efficiency
Measuring Coding Efficiency
Average Bit Rate
R =number of bits in bitstream for a video sequence
nominal duration of the video sequence(2)
Quality / Distortion
Typically: Mean square error (MSE) and peak-signal-to-noise ratio (PSNR)For a W×H array s[x , y ] of original samples, the array s ′[x , y ] ofreconstructed samples, and a sample bit depth B
MSE =1
W · H∑∀(x,y)
(s[x , y ]− s ′[x , y ]
)2 (3)
PSNR = 10 · log10
((2B − 1)2
MSE
)(4)
For videos: Average PSNR over entire video sequence
H. Schwarz (FU Berlin) — Image and Video Coding: Introduction 19 / 20
Outline
Outline of Course
Image and Video Coding I: FundamentalsProbability, Random Variables, and Random ProcessesLossless CodingRate-Distortion TheoryQuantizationPredictive CodingTransform Coding
Image and Video Coding II: Algorithms and ApplicationsAcquisition, Representation, Display, and Perception of ImagesVideo Coding OverviewVideo Encoder ControlIntra-Picture CodingInter-Picture CodingVideo Coding Standards
H. Schwarz (FU Berlin) — Image and Video Coding: Introduction 20 / 20