Micah, afraid you still have a bit to learn.
Yes, the speaker is part of the circuit, but it doesn't "draw" anything, it only impedes the flow of current supplied by the amp.
An amp, with both terminals shorted together, i.e. no speaker, just wire, would flow as much current as it could move. Something would fail, probably the output transistors, but it could also over-heat the power supply, or if it was a seriously robust amp, that much flow through the piece of wire could melt it.
What happens when you put a speaker in place, it that its has a really, long, thin piece of wire (to simplify things), that keeps the current flow under control. Speakers of higher impedance slow the flow even more. But if there's enough current, it can still put so much through the speaker that it can cause heat to build up. If you get the wire hot enough you can melt the insulation off of it, and that long, thin piece of wire starts to look more like one big lump of metal, and then other things start to break. There's also the possibility of over-extending the cone and causing physical damage that way. You ears will be telling you to turn it down before you get to that point.
The real problem is under-powered amps. Either having a speaker which can flow current more rapidly than the amp can stably provide (which will cause it to go into protection), or the amp just running out of steam and not being able to provide the highest peaks (clipping).
