I don't think it's crazy at all.
If I were in your position, I would almost certainly be the general contractor and chief-bottle-washer for that project. Especially since you're not adding plumbing.
I don't think buying a new house would be "cheaper". I don't know about you, but I think just the realtor and closing costs could just about get the addition done.
That being said, you have to seriously ask some important questions. Like, how long will you be in this house? If not for an extended period, are you likely to recoup the costs when you sell? It doesn't sound like you are overbuilding for your neighborhood, which is also really important.
I would be tempted to get get an experienced framer for the shell. Starting from nothing rather than doing remodel work is hard and dangerous and will take YOU forever, but somebody who does it for a living could do it in a couple days. It will make the project go much faster, and save your muscles and energy for the stuff you KNOW you can do. So, I guess I'd sub-out the foundation and the framing and probably do just about everything else myself. You'll probably have to get a licensed electrician to do at least part of the work, but hopefully you can find one that will let you run the wires to save some money.
I'm envious of your adventure and will look forward to frequent updates with pictures
