With interest rates at all time lows right now, how does one figure out if it is time to refinance?

We are 1 year into our current home ownership and have a special mortgage situation.

When we bought our house, instead of putting 20% down or whatever, we actually took out a second mortgage right from the start (called an 80/20 loan) to fund 100% of the house. Then we took the proceeds from the sale of our previous home and invested that money into this house. That way we increase the equity and get to enjoy the additional space right away.

Anyway, the 80% loan is at 4.25% and is due basically in 29 years.
The remaining 20% loan is at 5.25% and is due in 9 years (it was a 10 year loan with a balloon payment at the end).

Current re-financing rates at our bank are showing at 3.50% with zero points. I would have to pay for an appraisal and whatever the other closing costs are.

Does this make sense? I sounds like it would, but I am just not sure....
2-M60s, VP180, 8-M3s, SVS 20-39PCi, DIY Sub, 8-Shakers, JVC RS45, Anthem MRX-1120