Why do you think it would be a poor connection? What would you prefer, soldering them in place so that you can never remove them if the speaker should be damaged? I don't think that would result in any demonstrable change in the speaker.

I have SMA connectors in my lab for evaluating 12GHz SiGe parts that add almost no jitter at all. Connectors aren't inherently bad.