The real danger lies not in having too much power, but in having too little power.

The speaker will take the power it needs to play a specific sound at a specific volume. If that power is not there, you will get clipping from your amp. That is a dirty signal that cause distortion and damage to your speaker.