Ya, having a HDDVD or Blu-ray in a HTPC is pretty cutting edge right now.
Have you been focusing your research efforts on Vista+8800 driver issues? That's probably where I'd be putting my efforts into. Do you have any other video cards in the house you could try for a while? I find my 7300GT works perfect for my HTPC. I've had a 6600 in there too which was flawless. And I watch/record HDTV on my box. Looks like a 7600GT is the min for next gen DVD playback thou.
One of the reason's I nested with MCE was the "just works" principle (of course the hardware has to be working). Years ago I played the game of running this software, and that software on my "HTPC" of the time. If you are running Vista Home Premium or Ultimate, you can simply fire up the MCE interface as it's built in.
Oh one other thing, how does the box do everything else? Put the Blu-ray on hold for a moment and make sure everything else works the way you want.
RE: CableCard. I can't stress it enough to anyone looking at those... be sure to check and see what your local cable company has planned for their networks. Time Warner for instance (not my cable co, but my friends) is putting everything new on 'switched video' which means cable card PC's and Tivo and TV's won't work with it. You don't want to drop 2-3 grand on a OEM HTPC to have it work for a year. At least I don't. I've been a bit of a broken record on that, but I'm pretty pissed about the whole cable card thing and want to make sure everyone that is looking at them knows about 'switched video'.
