Executive Summary
Global Internet video traffic is likely to surpass peer-to-peer traffic by the end of this year and is expected to account for 50% of the consumer Internet traffic. This exponential growth of video traffic is expected to reach 62% of the global Internet traffic by the end of 2015 [59]. Service providers, both locally and abroad, specify that they can guarantee specific quality of experience criteria. These providers employ networking technologies and measurement tools to ensure that the content is accurately delivered [3]. There are several objective image/video quality metrics that can be used to evaluate the perceptual quality perceived by the user. However, there is no standard video quality metric that dominates over the other metrics. Moreover, the results found in literature are not measured on the same databases, which make the comparison of different objective measures difficult.
In this work we have managed to identify the set of video databases appropriate for this research. From over 40 databases available, the Video Quality Experts Group (VQEG) HDTV Phase I database [37] and another from the Laboratory for Image and Video Engineering (LIVE) [7] were considered for this project. These datasets were considered since they contained a set of videos with impairments consisting of both coding-only and coding-with-transmission over both wireless and IP networks. These databases also contained the mean opinion scores (MOS) of each video, which is a metric that represents the average quality perceived by the end user. The data contained within the LIVE database, which was used for the Quality of Experience (QoE) evaluation tests, was validated using a subjective test within a laboratory environment, where the scores from 9 test subjects were considered.
The correlation between the different Quality of Service (QoS) parameters, which are commonly used by today’s network companies, and the MOS scores was computed using the VQEG HDTV database. These results indicate that although QoS parameters such as packet loss rates can be correlated with the mean opinion scores, their correlation has a large variance which indicates that QoS parameters are not enough to quantify the subjective quality perceived by the end user with a high degree of confidence. This work also investigates 27 QoE metrics including full reference, reduced reference and no reference metrics. The results indicate that QoE metrics such as the Motion based Video Integrity Evaluation (MOVIE) metric and the Temporal Reduced Reference Entropic Differences (TRRED) metric achieve excellent correlation with the MOS scores contained within the LIVE database. This was attributed to the fact that these metrics investigate the effect of temporal distortions. These results further demonstrate that colour information is not significantly correlated to the subjective scores.
This research also demonstrates that the MOVIE quality metric achieves the best correlation to the subjective MOS scores. The MOVIE is a full reference metric and therefore the original video is required at the decoder in order to be able to compute the metric. It was also found that the Spatio-Temporal Reduced Reference Entropic Differences (STRRED) metric and its derivatives can be used instead which achievesimilar correlations to the MOVIE metric with the advantage that significantly less information is required at the decoder, making it more practical for implementation. Another advantage of the STRRED metric is that it is less computationally intensive than the MOVIE metric. This research also demonstrates that the no reference quality metrics achieve very poor correlation with respect to the subjective scores and further research is required to make them more competitive.
Related:
An Objective No-Reference Video Quality Assessment Metric (Masters thesis)
Christian Galea 