That sounds like us, minus the refis, most of the time.
We just bailed on our failed experiment in real estate by successfully negotiating a short-sale on our town home with both our lenders. We were told that most lenders won't give you the time of day unless you start missing payments. The payments weren't a problem for us -- it was the home repairs, space issues, and the fact that we bought at the peak and the market had fallen so far that were getting to us -- so we didn't want to play that game. Luckily our real estate agent was amazing and kept on top of the lenders until we found some mutually acceptable terms. We're still waiting for any negative credit rating fallout to occur, but it should be minimal.
"I wish I had documented more…" said nobody on their death bed, ever.