It'll be fine. Remember that the 4 ohm/8 ohm switch actually limits current which could result in damage to the speaker (when set to 4 ohms). Always leave the switch on 8 ohms.

I am the Doctor, and THIS... is my SPOON!