I have found the loading times to be fine so far. There is a little bit of a lag/stutter when you enter battles but it's not too bad- nothing to turn me off of the game.
The load times in oblivion are WAY longer then on this game.
I think that a more valid complaint is that loading occurs fairly frequently. However, the load times are not that long.
I think the devs wanted to put stuff on the hard drive to reduce the ammount of loading required but microsoft wanted everyone with a 360 (even the ones with a crappy memory card instead of a hard drive) to be able to play the game the same way... go figure.
anyway- I would suggest that you rent the game or something and see for yourself... because I don't think the load times are too bad at all.
