You made a promise to pay the bank and you are not living up to your end of the deal.
This is a contract issue, not a fiat currency issue.
If you don't agree with fiat currency then you never should have taken out the mortgage in the first place. You should have gone to the seller and offered them a certain amount of silver or gold or goats or lollipops or whatever else they might want.
The fact is you borrowed something and made a promise to pay it back. If you don't pay it back then you have stolen that something. The fact that the "something" is a fiat currency makes no difference.