You will not harm speaker nor receiver. The very worst thing that can happen is the receiver will shut down, but it shouldn't.
From an article at Audioholics.:
"Impedance Selector Switches
This so called feature, used by some manufacturers, is designed to prevent overheating of the receiver or damage to its output transistors because of excessive current flow. The manufacturer accomplishes this in one of 2 ways: 1) Stepping down rail voltage supplied to the power amp or 2) feeding half the signal strength to a voltage divider of power resistors. Both of these methods severely limit dynamics and current capability of the power amp. This results in an audible decrease in bass capability and dynamics transient sound because the 4 ohm setting effectively increases the receiver's output impedance. Unfortunately many manufacturers put these features on their products to ease customer concerns with driving low impedance loads and for safety reasons when getting UL approvals. Note: In order to meet UL requirements, a receiver cannot be rated down to 4 ohms without having this switch onboard. Receivers without this switch are usually rated down to 6 ohms. In most cases, well designed receivers can easily handle 4 ohm loads safely and efficiently. It is highly recommend to keep the impedance switch set to 8 ohms regardless of your speakers impedance and make sure your receiver has plenty of ventilation."
