Having a couple of similar amps (NAD 20w and 40w) that I swap in and out of a midfi setup, I can tell you that there is less of a difference between brand mates, then amps of different manufacturers. The main differences are as mentioned previously, the impact of the frequency response. The larger amp will be able to utilize the increased power (current, etc) when called for, even in average listening.
The basic tenet of this is to compare amplifier sections of similar quality pieces. There is a couple of threads that deal with "quality of amplifier output". I would suggest you search for them.