Dept. of Mobile Systems Engineering Junghoon Kim.
-
Upload
bertha-daniels -
Category
Documents
-
view
214 -
download
0
Transcript of Dept. of Mobile Systems Engineering Junghoon Kim.
Metrics for Evaluating Video Streaming Qualityin Lossy IEEE 802.11 Wireless Networks
Dept. of Mobile Systems EngineeringJunghoon Kim
OutlinePaper InfoIntroductionBackgroundMotivationIdeaExperimentsEvaluationContribution
Paper InfoIEEE INFOCOM 2010
The 29th conference on computer communica-tions sponsored by IEEE communications society
March 15-19, 2010, San Diego, CA, USAAuthors
An (Jack) Chan, Kai Zeng, Prasant Mohapatra Dept. of Computer Science University of California, Davis
Sung-Ju Lee, Sujata Banerjee Multimedia Communication & Networking Lab Hewlett-Packard Labs
IntroductionImportant issue
Multimedia streaming is becoming one of the most popular applications recently
Video streaming over WLANs in very commonVideo quality can be measured objectively and
automatically by a computer program It is important to government and industries For specification of system performance require-
ments Comparison of competing service offerings
IntroductionPeak Signal-to-Noise Ratio (PSNR)
simplest and the most widely used video quality evaluation methodology
Problem of traditional PSNRFail to capture the packet loss characteristics of
wireless networksNon-linearity of the human visual system
MPSNR (Modification of PSNR)Retaining the simplicity of PSNR calculationHandles video frame losses
IntroductionDeriving two specific objective video quality
metricsPSNR-based Objective MOS (POMOS)Rates-based Objective MOS (ROMOS)Demonstrate high correlation with MOS
Our metrics evaluate video streaming quality in wireless networks with a much higher accu-racy
BackgroundMean Opinion Score (MOS)
Measured through each viewers giving a score ranging from one to five
Arithmetic mean of all these individual scoresPros
MOS is subjective metricCons
Expensive process Needs a large number of viewers Controlled evaluation environments
BackgroundPeak Signal-to-Noise Ratio (PSNR)
Most widely used objective video quality metric
MSE : Mean Squared Error
BackgroundPeak Signal-to-Noise Ratio (PSNR)
Problem A missing frame results in the latter frames in
shifted positions when compared with the reference video
MotivationInaccuracy in the existing PSNR calculation
Average PSNR value of the reference video : 100dB
Video streaming A : 38dBVideo streaming B : 40dB
(a) Reference video (b) Video streaming A
(c) Video streaming B
IdeaMPSNR
Modification of PSNRAdd matching process in the correct PSNR cal-
culationTwo ways
An optimized algorithm for matching corresponding frames
A heuristic algorithm for matching corresponding frames
IdeaAn optimized algorithm
Assumption The sum of PSNR of all frames in a streamed video is
the maximum when all the frames are correctly matched with the corresponding frames in the refer-ence video
Each frame in a streamed video must have a matched frame in the reference video
We consider a global maximization of the sum of PSNR
IdeaAn optimized algorithm (Cont’d)
Define Maximum total PSNR value achieved when a
streamed video with j frames is matched to the ref-erence video with i frames
Define PSNR value of frame x and frame y
If no match can be found for a frame in the ref-erence video, we ignore the frame in the calcu-lation of the total PSNR value
IdeaAn optimized algorithm (Cont’d)
Three possible cases for the last match in two videos But, Case 3 would never happen
Recurrence equation
IdeaAn optimized algorithm (Cont’d)
Use dynamic programming! Time complexity :
g : the total number of frames lost during streaming n : the number of frames in the streamed video
Given a streamed video of 40 seconds (1000 frames) with 20 frames lost (about 2% frame loss rate), a personal computer with 2.8GHz CPU and 1GB RAM Traditional PSNR : less than 2 seconds Optimized algorithm : about 20 seconds
We need a faster algorithm!!!
IdeaA heuristic algorithm
Define The PSNR value calculated for frame j in the
streamed video when it is compared with frame i in the reference video
Define The set containing the continuous frames in the ref-
erence video when frame j in the streamed video is processed
Define The PSNR value of the frame j in the streamed video
IdeaA heuristic algorithm (Cont’d)
A parameter called PSNR threshold, thresh To mitigate this problem
Frame j in the streamed video is distorted severely and has a larger similarity to a non-corresponding frame k than to the actual corresponding frame h
Take the maximum only if it is greater than thresh Otherwise, we will regard the first frame in as the
matched frame
IdeaA heuristic algorithm (Cont’d)
Time complexity : t : the number of different thresh tried w : window size n : the total number of frames in the streamed video
t and w are small constants. Therefore, time complexity is
Previous experiment Traditional PSNR : less than 2 seconds Heuristic algorithm : about 4 seconds
IdeaMeasuring other parameters
Distorted frame rate Averaged PSNR of distorted frames Frame loss rate
ExperimentsCollecting videos of dif-
ferent qualityA total of 40 streamed
videos with different qualities 30 video clips in the
training set 10 video clips in the vali-
dation set
(a) Streaming with intra-flow interfer-ence
(b) Streaming with inter-flow interfer-ence
(c) Streaming with background data flow
ExperimentsCollecting subjective evaluation for video
qualityEngaged 21 volunteers
Diversity was taken into account Age : from 20 to 45 Occupation : from university undergraduate students
to laboratory techniciansFor each video clip, average the quality scores
given by the subjects and obtain MOS
ExperimentsCollecting subjective evaluation for video
quality
MOS and 95% confidence intervals of videos in the train-ing set
ExperimentsDeriving metrics from subjective evaluation
and MPSNRPSNR-based Objective MOS (POMOS)
Define The average PSNR calculated from MPSNR
Define By setting the window size to one
ExperimentsDeriving metrics from subjective evaluation
and MPSNR (Cont’d)PSNR-based Objective MOS (POMOS)
Use the linear model package of the statistics tool R
ExperimentsDeriving metrics from subjective evaluation
and MPSNR (Cont’d)Rates-based Objective MOS (ROMOS)
To mitigate this problem Assigned a PSNR of 100dB for the perfect frames
ExperimentsDeriving metrics from subjective evaluation
and MPSNR (Cont’d)Rates-based Objective MOS (ROMOS)
Use the linear model package of the statistics tool R
EvaluationEvaluation of objective metrics
Pearson correlation (= correlation coefficient) A heuristic algorithm
: 0.8666 : 0.9346
An optimized algorithm : 0.8838 : 0.9509
EvaluationEvaluation of objective metrics
ContributionIdentify the detrimental impact of packet
losses during video streaming on video quality metric, such as PSNR
Propose a simple objective video quality eval-uation methodology, MPSNR, that alleviates the inaccuracy caused by packet loss
Derive two specific video quality metrics that provide a tool for evaluating video streaming over lossy wireless networks