You need to level match your system with a reference track of pink noise or whatever your AVR has (usually 1khz centered pink noise). Download an app with an spl meter at the very least. It wont be accurate, but all channels will be treated the same at least.
Dont use auto setup, dont set by ear. Turn the gain on your sub amp down and the sub preout level up on your yamaha. Again, set the level first with a reference tone and adjust to taste from there. You want to be around -3 to 0 on your sub trim setting. All your channel levels are supposed to be a reference to a test level benchmark. It is not a volume setting per se. Learning the relationship between channel trim level and system volume is important. If you need to max your trim levels to hit a reference of 75db your system might have problems if you crank the volume. Choose a lower reference tone, like 70db, to get your trim levels away from max.
