Mike Jones Digital Basin
screen media rinse cycle

Perpetuated Myths - codecs, compression and transcoding

Compression is the great boogieman word of digital production and has a direct set of correlations in the minds of too many producers ?
Compression equals quality loss. Transcoding means quality loss. Mpeg is bad. Long GOP is bad. 4:2:0 is bad.

Right...?  Wrong and its about time the myths were dispensed with.

Compression is not of itself a bad thing. There are two key points here:
1. Compression technologies have advanced enormously over just the past five years and extraordinary reductions in file size are possible with no reductions in visible quality.
And
2. Compression is a fact of production. Unless you are George Lucas shooting 4:4:4 uncompressed then you will be dealing with compressed material from an increasing variety of compressed formats. AVCHD, HDV, DVCProHD, DV, XDCAM, XDCAMHD, IMX, Digibeta, HDCAM SR are ALL compressed in various ways. Anyway you slice it compression is a fact of production life and despite what seems to be a commonly held vibe, it is not the end of the world.

Any discussion of compression should really be about efficiency not bit-rate. Bit-rate is the great misnomer of compression quality. That one format has a higher bit-rate than another Does Not by default mean that it is better quality. DVCProHD is a good example of a proprietary format that trades its' market cache on the traditional compression/bit-rate phobia. I don't mean to necessarily pick on a particular developer like Panasonic, all have been guilty of dispensing myths about compression, but Panasonic have gone it alone with their own formats and billed these formats as 'Better' based on the 'compression=bad, high bit-rate=better' mantra.

Devotees will declare that DVCProHD is better quality than comparable formats, such as HDV because it has a higher bit-rate. But this is a selective comparison that doesn't take into account the efficiency of the codec which is in many was more significant than the bit-rate.

Panasonic has pushed hard the line that DVCProHD, from a camera like the HVX200, can generate 720p24fps at 40mbps and that this must be better than the 19-25mbps of 1080HDV. But this isn't telling the whole story. Mpeg codecs are designed for efficiency and, as anyone who has put DVCProHD and XDCAM Mpeg2  in the same project can attest, they can retain the same quality at a greatly reduced bit-rate. All other things being equal I defy anyone to tell them apart in quality. Both have a slightly different feel but that is largely dictated by frame rate, lens quality and personal preference. Broadly speaking Mpeg2 1080 HD is more than two and half times more efficient than DVCProHD to produce an image of the same perceptual quality. Judging the two simply by the bit-rate is a useless comparison.

This is just one of the reasons that Panasonic has remained very much on their own with the DVCProHD, with limited suppourt from the post-production sector, while all other developers have embraced the Mpeg frame work for acquisition, production and distribution. Do Panasonic know something ALL other developers don't? Or do they just have their head stuck in the sand over out-dated thinking about compression? Or worse, is it simply that Panasonic know that the advantage of higher bit-rate, for bit-rate sake, is of little worth but persist on pushing the line anyway because they perceive a market value in playing on those concerns?

Now there are of course other considerations in regard to DVCProHD, such as variable frame-rates and a particular cadence and feel of 720 over 1080 - and these are certainly valid - but the ongoing misnomer of bit-rate=quality is just not accurate and as codecs continue to get more and more efficient we need to seriously reconsider the benchmarks by which we consider technical quality. Moreover, we should persist in questioning the market spiels we are saturated with in regard to these formats... Golden rule ? never take advice from someone trying to sell you something.

In context of this is also the notion of 'editing native' pushed by a number of major NLE's and the often prescribed ills of Transcoding. The idea is that if you Transcode from one format to another you loose quality. Not only do official websites tout this idea but the myth is picked up in a great many reviews such as this from Macworld.com of Final Cut Pro 5.
http://www.macworld.com/2005/07/reviews/finalcutpro/index.php
"Intermediate Codec, which transcodes HDV?s hard-to-edit long-GOP MPEG-2 format into an easier format to edit at the expense of some quality. Final Cut Pro 5, by contrast, handles HDV natively?so it captures video in real time and doesn?t degrade the quality.?"

Absolute Rubbish..!! Transcoding HDV to a lossless HD intermediate codec does not, in any way loose quality. This idea is simply a myth, or at worst a lie, that has been perpetuated by a large number of developers in their promotion, most notably Apple.  4:2:0 HDV transcoded to a lossless 4:2:2 HD intermediate does nothing but gain image headroom and improve the results possible from heavy colour grading or effects, as well as improve timeline performance.

So, with this having been the company line for a number of years, it is most particularly interesting that Apple have released ProRes 422 a lossless HD intermediate at this most recent NAB. Effectively Apple's response to Cineform and Avid's DxNHD codecs; ProRes 422 is very much an about-face backflip to the hard line Apple have been touting for a number of years with Final Cut Pro..

Apple had been very dismissive of HD Intermediates with lossless compression, instead marketing hard on the "edit native" line and deliberately playing to the fears of old school concerns of compression and transcoding. Even the Final Cut Pro entry on Wikipedia declared that FCP "can deliver better quality than other NLE's because it edits in the native codec of the source material". What absolute horsesh@#t (Ive subsequently amended Wikipedia in the long standing process of peer-review that Wikipedia is built on). This sadly widespread perspective, and its appearance in a medium like Wikipedia, is clear evidence of a win for the marketing campaigns built on outmoded fears and ignorant concerns to do with transcoding and the myth of 'editing native' superiority.

Apple did release the Apple Intermediate Codec as a HD intermediate (a decidedly poor effort) but it was a very quiet release and pushed mainly as a low end I-movie and FCXpress solution to make up for poor native HDV real-time performance.

Now it seems Apple have finally decided to see the light, acknowledging a more accurate technical perspective with the introduction of a high-quality, low bit-rate, lossless codec solution for HD intermediates.  See this entry from David Newman CTO of Cineform ?
http://cineform.blogspot.com/
"Several years ago CineForm attempted to sell Apple on the idea of a digital intermediate codec for professional post. Unfortunately back then they still had their heads stuck firmly in the DV sand, and believed DVCPRO-HD would sufficient, which everyone today knows it isn't. With the announcement of Apple ProRes 422 it seems the CineForm message has finally taken some hold?"

So what do Apple now do with all those marketing graphics they have lying around declaring that virtues of 'editing native'??? They'll do with them the same thing Steve Jobs did with his comments that the I-pod will never get video capabilities - pretend he never said them. :)

Apple, of course, are not alone in propagating and marketing on these archaic and obsolete notions of the compression boogieman and the transcoding myth - Adobe and Avid are also guilty of compounding the myths in an effort to build a perception of superiority with users. But, both Premiere Pro and Avid have also long had a transcoding, HD intermediate solution available so perhaps they only marketed the idea of 'native superiority' because Apple did. Keeping up with the Jones' regardless of technical fact.

This brings me to AVCHD, the new High Definition Mpeg-based format using the H264 compression scheme. AVCHD is one of the most significant progressions in digital media technology in a long time for the simple reason that it is the first format in a very long time to be universally embraced by all major developers. Any time the big corporations agree it's a big deal and we should pay attention.

AVCHD is primarily designed for consumer camcorders, Sony already have two in the market, but there is certainly potential for AVCHD to be used in professional work, possibly with professional variants.

Even Panasonic (back flipping on their previous diatribes against Mpeg) have come to the AVCHD party - albeit they have done so with a caveat - intra-frame AVCHD for their pro HVX cameras and the P2 storage format. Panasonic obviously are still stuck on this idea of compression - specifically Long GOP - being so horrible they need to develop their own proprietary version of an otherwise universal format.

As there are not yet any NLE's that can work with AVCHD the test remains to be seen but I'll make a fair bet that once AVCHD is more widespread, possibly with a Pro variant, there will be absolutely Zero discernible quality difference or tangible performance issues between Long GOP AVCHD and Intra-frame AVCHD, simply that one has bigger files sizes. I have little doubt that Panasonic's fears over Long GOP will be proven baseless in real-world production and the market wont put up with bigger files sizes for no tangible benefit.

In the meantime critics of Mpeg-based Long GOP will continue to cite concerns over two main areas -
1. Motion artifacts and smearing in fast movements of either camera or subject. This again is a myth, all cameras can exhibit problems under different types of shooting conditions based on shutter, CCD/CMOS configuration and lighting. The pristine footage shot by people such as Douglas Spotted Eagle hurtling through the air in sky-diving free-fall with a HDV camera strapped to his head puts these myths firmly to bed.  
2. Poor editing system performance with Long-GOP formats. To this there are two responses: firstly that I'm currently on my laptop, have Vegas running and have 4 streams of native HDV in realtime with primary and secondary colour corrections. And secondly that we come back full circle to where I started with  ProRes422, Cineform and DxNHD; frame-based lossless HD intermediates.

Compression, transcoding, re-encoding, Mpeg, Long GOP - these are not scary digital boogiemen. They are not technical components to be feared or disparaged. They are, rather, core processes to be understood and worked with through a broad ranging knowledge of technical craft. Not to be obstinately avoided or decried in a corporate, marketing induced, and unfounded haze of concern.

Know your craft, know your camera, understand the technology properly and astounding images are more than possible from any format and any camera.

Posted at 12:00AM Apr 25, 2007 by Mike Jones in general  |  Comments[3]