I've always loved flying and always had a problem with heights. Never knew exactly where the line was.
I discovered it while taking flying lessons. Groundschool no problem, flying no problem, but stalling even a Cessna 172 (maybe the most gentle stall of anything with wings) was enough to put me into a blind panic every time. And that was the end of my "flying career".
It might have been possible to tough my way through the stalls or go for a sport license where stall/spin testing is not covered (I think that's still the case) but I didn't feel that I would come out a safe pilot. If I couldn't count on my reactions in an emergency then IMO there was no way I should be flying...
On a positive note, once you've budgeted for supporting a private pilot habit, everything else seems really inexpensive by comparison