Has anyone here had any experience with buying speakers, and then changing drivers in them, just to see what would happen?

I am tempted (definitely not with my Axioms!) to get a used pair of decent speakers for a couple hundred bucks, and then buying some higher end drivers that fit and putting them in, just to see what would happen. I have always been tempted to try a DIY project, and this might be a good start.

Thoughts? Comments?