The first question is the audio. How many channels are on the other file? Is it compressed? Compressed audio can sound basically identical to the raw Blu-ray audio, especially if you're testing on a 2-speaker setup and one of the audio tracks has been downmixed
to 2 channels. Anyway, the audio might be part of the difference here.
Then there's the video. There are two popular ways to do it:
1) Pick a file size or bitrate. The encoder does one pass to find out what quality to target. For simple, low-motion videos, you'll get high quality. For complex, high-motion videos, you'll get lower quality.
2) Pick a quality target. The encoder will use whatever data necessary to make the video meet the quality you specified. So every video you encode will have the same consistent quality, but the file sizes can vary.
I prefer #2, picking the quality target. That's why all of HandBrake and VidCoder's presets use it. It lets you save space on simple videos, rather than just throwing extra unneeded data at it, and have nice looking complex, high-motion videos. Plus, the
encode is faster since you only need one pass. The only advantage that #1 has is that you always know what the file size will be for a video of a given length. Unless you need to burn it to a CD or DVD (which is rarer and rarer these days), it's not very useful.
What I suggest you do is fiddle around with the quality target until you get a good quality/size mix. Good values are around 18 (high quality) to 25 (low quality). You can use the preview window or just encode 15-second segments or something. x264 is the
best video encoder in the world, so for any size, it should look the best, and any video quality it should have the lowest file size.
So in short, you may be able to target a lower quality without noticing the difference (and thus get a lower file size).
Also it helps to use the x264 settings from the High Profile preset: this uses all the bells and whistles in H.264 to efficiently encode good looking video.