Size can matter....but it is not an absolute.

I have heard towers with 8" drivers that didn't produce any more bass than others I've heard with 6.5" drivers. I would imagine there are other contributing factors besides driver size. For example, driver design/quality, cabinet design, crossover, etc.