There are guys who have been encoding for years who do this stuff.
I'm sure it is much much easier and faster now, but in my day you had to know what you were doing and to rip a DVD down to 700MB took quite a while.
I don't even know what tools you use anymore.
We are at the same exact boat. I used to be really active in the scene when xvid was first released and I watched it grow over divx. Then I found a lady and now I'm just enjoy watching encodings

Remember vsloathe when those perfect bitrate calculators came out. It was awesome that you could put the length, resolution and wanted size and it would calculate everything ready for you. Oh boy that was cool back then. And I remember how cool it was when I got computer that was powerful enough to encode xvid in real-time (23.976fps) from DVD.
So this answers to Nutballs as well. I have no idea what you should use now. You should go to Doom9.org which is still leading forum in this topic. But my guess is that you have to get compiled version of x264. h264 is Apple's and I'm not sure what's the situation with that but x264 is like xvid but for HD.