With my Denon, the auto setup microphone does a very good job with dB levels and distance settings. I have verified the settings with my SPL Radio Shack meter and they are very close to being correct. I think your Yamaha microphone will be pretty accurate. If you have the sub on 9 o'clock, I still think your AVR is set to high. You want all the speakers to be at the same SPL, this is where the Radio Shack meter is nice, you can verify your Auto Setup results.
There are some other Yamaha/EP500/600 owners on this forum. Maybe they will jump in here and give you their settings. My guess is that your still running to hot on the sub volume.
I don't think this is causing the rattling your hearing, but it may be enhancing the problem. As ken mentioned, you might want to check the screws on the back/front of the sub to make sure they are all snug. Air escaping around seals on the driver/amp can cause funny sounds. Or it could be as simple as a wire vibrating on the inside. I had a problem with my first EP500 and I could trace the sound to the back amp plate, the rattle was on the inside, so they sent me a new amp.
If you leave the sub at 9 o'clock, you'll need to lower the AVR to -6dB or more.
If you put the sub at around 7 o'clock, your AVR will be around -4dB or so.
As wid mentioned, trust the yamaha setup, maybe bump up the AVR a few dB, no more.
If you can buy an SPL meter from Radio Shack or online, it is a great investment to verify your settings.
