A really simple way to obtain very high quality per bit per second, given prior knowledge of the nature of the material (film or not, grainy or not, cartoon or not, etc) and the type of output desired (AVCHD, Blu-Ray, etc) is to install MeGUI, then pick one of the community-built encoding profiles for x264.
Choosing the right profile for the job is absolutely crucial. The combinations of x264 parameters can be pretty arcane, and they sometimes change from one x264 version to another. There's a pretty active community on forum.doom9.org maintaining collections of profiles for MeGUI, some of those are excellent.
E.g., it is totally within the realm of possibility to put two hours of 1080p content on a single-layer DVD (4.4GB), in a format compatible with any Blu-Ray player out there (AVCHD, a subset of the Blu-Ray standard that accepts DVD as the storage layer), while keeping video quality at a very high level - basically indistinguishable from commercial Blu-Ray discs. But using a good encoding profile, feeding the appropriate parameters to x264, is the single most important factor in achieving that goal.
Choosing the right profile for the job is absolutely crucial. The combinations of x264 parameters can be pretty arcane, and they sometimes change from one x264 version to another. There's a pretty active community on forum.doom9.org maintaining collections of profiles for MeGUI, some of those are excellent.
E.g., it is totally within the realm of possibility to put two hours of 1080p content on a single-layer DVD (4.4GB), in a format compatible with any Blu-Ray player out there (AVCHD, a subset of the Blu-Ray standard that accepts DVD as the storage layer), while keeping video quality at a very high level - basically indistinguishable from commercial Blu-Ray discs. But using a good encoding profile, feeding the appropriate parameters to x264, is the single most important factor in achieving that goal.