Measuring Video Performance - Overview

The successful deployment of IPTV and videoconferencing services requires a range of tools for measuring and monitoring video quality. The impact of encoding and transmission impairments on the perceptual quality video streams is quite complex and depends heavily on the codec type and configuration, and on end system characteristics. There are a variety of algorithms for video quality estimation but relatively few standards and a fair amount of confusion.

There are essentially three “models” for video performance measurement.

Full Reference Algorithms

Full reference algorithms perform a detailed comparison of the input and output video stream. This is a computationally intensive process as it not only involves per-pixel processing but also time and spatial alignment of the input and output streams. Full reference algorithms can achieve good levels of correlation with subjective test data however can only be used in certain applications - for example in lab testing, pre-deployment test or troubleshooting.

One of the earliest full reference algorithms is PSNR (Peak Signal to Noise Ratio), which is literally a measurement of the mean error between input and output as a ratio of the peak signal level, expressed in dB. A typical “good” PSNR is around 35dB and it is generally accepted that PSNR values of less than 20dB are unacceptable. PSNR is the most widely used technique for image and video quality measurement.

The Video Quality Experts Group (VQEG) has been actively working on objective video quality assessment since 1997, and generally acts in cooperation with ITU. VQEG has conducted two phases of testing; in the first phase ten algorithms were tested and the conclusion reached was that most of the algorithms (including PSNR) were statistically equivalent. The second phase of testing, conducted several years later, involved a smaller number of algorithms and concluded that these did achieve good enough results to warrant recommendation for use, resulting in a recent ITU-T Recommendation J.144.

It is important to note that the test conditions for J.144 did not include packet loss and hence J.144 conformant algorithms do not necessarily perform well on IPTV systems.

ITU-T J.144 does not actually specify a single algorithm but “provides guidelines on the selection of appropriate” techniques. J.144 does contain descriptions and test results for four full reference algorithms, and also included PSNR as a reference. The VQM algorithm from the US Government’s NTIA ITS lab achieved slightly better performance than the other algorithms listed.

Zero Reference Algorithms

Zero reference algorithms are generally more suitable for in-service monitoring of video services as they can analyze live streams. This type of algorithm can consider fewer factors than a full reference algorithm however can be deployed in a much wider variety of scenarios.

Media stream based algorithms such as Telchemy’s VQmon® analyze the IP stream and video transport protocols, building up an assessment of video quality and expressing this as a perceptual quality score.

Telchemy’s algorithm is differentiated in its ability to analyze the time distribution of lost and discarded packets and to model the impact of transient IP problems on perceptual quality, based on their widely adopted VQmon technology.

This type of algorithm can be very computationally efficient, taking a fraction of a MIP for processing, and is suitable for integration into a wide range of network devices and high performance test equipment.

Some zero reference algorithms are based on analysis of the decoded video stream and the identification of visual impairments. This approach is more complex and does require access to the decoded video stream.

There is standardization activity within the industry related to the definition and reporting of zero reference performance metrics - this is at an early stage, but is providing some structure within which better definition can be achieved.

Some companies, such as Ineoquest, market technology that reports packet loss and jitter on the basis that if packet loss and jitter are low then quality must be ok. This approach would be fine in a scenario where quality was either perfect or terrible, but is likely to be less dependable when packet loss rates are “noticeable” and it becomes important to understand the impact of loss on the specific codec type and configuration.

Partial Reference Algorithms

Partial reference or reduced reference algorithms can also be used for in-service monitoring as they reduce the complexity of the real time analysis required.

Summary

Emerging video performance monitoring technology can be immensely helpful in testing video equipment performance, performing pre-deployment testing and in-service monitoring. There is considerable activity within the industry in the development of new tools and technology for both full and zero reference video performance measurement.

Back To Top

Conditions of use: The material on this site is copyright Telchemy and may be freely used but not copied or downloaded.  In making use of this site the user acknowledges that Telchemy or Contributor has no liability for any issues or problems that may arise directly or indirectly as a result of such use.  Telchemy and Contributor are providing this material as-is with no warranty as to correctness or completeness and do not accept any responsibility for any issues or problems of any nature whatsoever that may arise from the use of the material on this site.