H.264 encoder benchmark

The most widely used delivery codec today on the internet is h.264. There are at least 10 encoders out there from different vendors, but which one is the best? I tried MainConcept (via Vegas Pro 8.0c), SonyAVC (via Vegas Pro 8.0c), Apple’s h.264 (via the latest Quicktime Pro), and x264 (via the latest ffmpeg).

The maximum profile/level is used by the user interface offered. For example, if an encoder only supports the “main” profile I would use that, even if it goes against another encoder that supports the “high” profile. Same goes for CABAC and CAVLC — I used the best options an encoder or its UI can offer me. What I am comparing here is *end-user solutions*, not how the encoder itself could perform tweaked if the sun was black and the moon was red. The source file is a 4 second uncompressed Quicktime 720/30p progressive file. The machine used is my usual 630-P4 3Ghz with 3 GBs of RAM (more specs were given before in this blog). All encodings used 4 mbps CBR, 1 pass, at 720/30p. Quality is compared using a Perceptual Diff utility: I compared frames 1 and 41. Frame 1 is stationary, Frame 41 has lots of motion in it.

A note on Quicktime

I tested both the .mov and the .mp4 h.264 exporting options from Quicktime, just in case they are internally configured differently. Please note that a lot of the quality loss shown below for Quicktime, is because of the gamma change Quicktime applies to these files. In the past, you could go to the video track “properties” with Quicktime, and change the “blend” to 100%, in order to get the original color look back. I would have tested that kind of exporting too, but it seems that the latest Quicktime Pro has this feature broken (it used to work a few months ago with an older version).

So I had to do it manually: I played with a lot of gamma values on PaintShopPro and when I gave it a 0.75 gamma (making it darker, as it was supposed to be), the pixel difference went down to 18,826 and 31,794 points for frames 1 and 41 respectively (down from about 200,000 points). So if Apple stops adding that dithering crap by default, they can deliver a pretty good result by default. “Defaults matter” as users whine and whine.

A note on x264

I used the following ffmpeg switches for the x264 encoding: -me_method umh -subq 5 -coder 1 -trellis 1 -g 300 -qmin 10 -qmax 51 -qdiff 4 -level 41 -rc_eq "blurCplx^(1-qComp)"

I did try to further optimize the x264 encoding like this: -flags +loop -coder ac -refs 5 -loop 1 -deblockalpha 0 -deblockbeta 0 -parti4x4 1 -partp8x8 1 -me full -subq 6 -me_range 21 -chroma 1 -slice 2 -bf 3 -b_strategy 1 -level 41 -g 300 -keyint_min 30 -sc_threshold 40 -rc_eq blurCplx^(1-qComp) -qcomp 0.7 -qmax 51 -qdiff 4 -i_qfactor 0.71428572 -cmp 1 -maxrate 4000k -bufsize 4M, but the video file produced was not readable by Vegas’ MainConcept h.264 reader, and therefore I could not take PNG screenshots off specific frames in order to carry out the comparison. FFdshow and Quicktime could read the file btw. I don’t have all day trying to find needles in a haystack, so if anyone knows which switch creates the incompatibility, let me know and I will update the article.

Update: It’s the -g 300 that created the problem. I removed it, but the resulted video is not frame by frame identical to the original. Some of these “advanced” switches do something really nasty to the video in terms of timing. Because the frames I get at position 1 and 41 are not identical to the original, there can’t be pixel comparison.

Update 2: Now, get this. Using this pretty optimized command line: -me_method umh -subq 5 -coder 1 -trellis 1 -qmin 10 -qmax 51 -deblockalpha 0 -deblockbeta 0 -parti4x4 1 -partp8x8 1 -qdiff 4 -level 41 -rc_eq "blurCplx^(1-qComp)" I got over 55,000 pixels of difference. However, it seems that x264 uses a different gamma than the original file, just like Quicktime does so above. So when I changed its gamma using the “Color Corrector” Vegas plugin to 0.925, I was able to get the “true” quality that the encoder is capable of. And that comes out to 16085 for frame 1, and 19597 for frame 41. In other words, if both Quicktime and Ffmpeg were to fix their gamma problems, they are pretty much identical in quality (it’s just that x264 is way faster).

The tests

Conclusion

In terms of quality, it seems that Sony AVC and MainConcept are pretty good here. x264 would probably fair better too if some of their optimization switches are not so incompatible with some decoders or if it fixes its gamma problem. Nevertheless, x264 is the fastest encoder. Quicktime quallity suffers by default more than x264 because of its extreme gamma changes. If Apple wasn’t doing that gamma change by default, it would fair much better, as my gamma test shows. Download the OpenOffice .ods spreadsheet file here.

11 Comments »

Cesar wrote on October 23rd, 2008 at 9:29 PM PST:

In my personal experiments I often find Sony AVC (from Sony Vegas) an excellent encoder over the others (and faster). But I have to admit that x264 is just as great and can be tweaked even more (though it’s not as practical as Sony’s). It can become a real challenge to find x264’s optimum settings for a given type of video and the most advanced ones won’t be handled by some of the recent decoders (just a few months ago x264 could encode interlaced video but the libavcodec itself could not decode it, only CoreAVC).

I wish Sony released their encoder as a standalone software, so that it’s not only tied to Vegas. They have a professional solution to offer, but that’s way too expensive.

By the way, thanks for taking your time to test the perceptual differences between different popular encoders. It was interesting.


William Eggington wrote on October 24th, 2008 at 8:08 AM PST:

I always turn to Nero’s Recode for super fast high quality x264 encoding. I’d throw that one into your comparison list next time.


This is the admin speaking...
Eugenia wrote on October 24th, 2008 at 1:39 PM PST:

I don’t own Nero, so I can’t test it.


Ryan wrote on October 24th, 2008 at 10:15 PM PST:

Thanks for doing the comparison, I really appreciate it. Did you find that a gamma of around .75 worked for all videos, or do you think it’ll vary for each project?


This is the admin speaking...
Eugenia wrote on October 24th, 2008 at 10:33 PM PST:

I think it’s the same for all quicktime videos, yeah. It’s at around 0.75, although it might not be 100% that value, you will have to do tests yourself. The gamma on a Mac is different too.


Ryan wrote on October 24th, 2008 at 10:57 PM PST:

Hey – quick note on the Quicktime front.

After doing some research and testing, my personal preference is now most certainly the x264 for Quicktime codec.

It’s easy to use (handy sliders and checkboxes for the features) and is not only faster than the default Apple H.264 version, the results do not have the inherent gamma issue and to my eye, the compression just looks a bit better. (Less artifacting, etc.)

It can be used anywhere that Quicktime is used, so you can render straight out of FCP / FCE / etc… with the x264 codec. Seems pretty perfect in my opinion.

It’s available all over the place: http://www.google.com/search?hl=en&q=x264+os+x


This is the admin speaking...
Eugenia wrote on October 24th, 2008 at 11:10 PM PST:

I am not sure what you mean. Apple only has one h.264 encoder. Yes, there are two ways to export with it, via two different interfaces, but under the hood it’s the same encoder. And I used both interfaces btw, it’s the ones I called “quicktime h.264” and “quicktime mp4” (and then selecting the h.264 option).


Ryan wrote on October 24th, 2008 at 11:31 PM PST:

You can download an x264 encoder for Quicktime that is separate from the default Apple encoder, and produces better results faster.

http://www.macupdate.com/info.php/id/20273


This is the admin speaking...
Eugenia wrote on October 24th, 2008 at 11:37 PM PST:

Ah, you are talking about x264 for Quicktime. Unfortunately that’s an ancient version of the codec you linked. It hasn’t been updated for 2 years. Lots has changed to the x264 codec since then.

You can still use the Apple codec btw, but you might need to adjust the gamma before exporting.


Ryan wrote on October 25th, 2008 at 12:14 AM PST:

Well, as far as I can tell, the ancient x264 seems to do a marginally better job. It seems like there is less banding in solid colors (especially darks), and just generally smoother gradation where the Apple version has harsh lines.

I could be totally just seeing things, but even the old version just looks a little nicer to me. Here’s a comparison of the exact same frame. It’s from a dark video of a match lighting that I’ve been messing with, and I think it’s a useful comparison since there are areas of solid dark color, with noticeable compression. Exact same settings in both.

X264 H.264: http://i35.tinypic.com/291orxz.png
Apple H.264: http://i36.tinypic.com/140bvdg.png

Now, to my eyes, flipping back and forth between those two, the default Apple encoding is “rougher”, with more visible and chaotic artifacting.


Ryan wrote on October 25th, 2008 at 12:15 AM PST:

(By the way, that test excludes the gamma issues, since they’re viewed in FCP, which uses the proper color space and does not wash out the files. So, even ignoring the gamma problems, I still think the outdated X264 implementation is superior.)


Comments are closed as this blog post is now archived.

Lines, paragraphs break automatically. HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

The URI to TrackBack this blog entry is this. And here is the RSS 2.0 for comments on this post.