Murray Rothbard lays it out here: http://mises.org/rothbard/ethics/nineteen.asp
The bank lent you money to buy a house and you signed a contract saying you would pay them back plus interest. Your failure to pay makes you a thief.
Libertarianism doesn't work if contracts are ignored. You signed a contract saying you would pay, so you should pay.
Content of posts and comments on the Daily P