OK, 'ruler flat' wasn't the right choice of words.
Thinking more on this, I remembered doing some reading on what spl changes we could hear and tracked it back to an audioholics article. It seems average joe listener can detect changes as low as ~1db. So, in theory, the differences we see between two speakers that both have a tolerance of +-3db over a given range could result in a noticable difference in sound.
I still wonder what the reality is.
Given the above, it is a reasonable proposition that when room effects combine with particular peaks/valleys it is likely to have a noticable effect on the sound of the speaker.
You know, if I ever win the lottery, I can keep myself amused for years on stuff like this.