I have not assumed people will always be just. I do remember the bit about "because men are not angels," but Madison was wrong on that. It is precisely because men are not angels, that one ought never agree to be ruled over by them. It's as simple as that.
When you agree to the principle that other men have the right to rule over you, then you are in no position to "limit" how they rule. And because they are not angels, you end up being ruled by tyrants.
The answer is to remove your consent to be ruled. When the predominant recognition in society has come around to this unavoidable conclusion, then you have a general rejection of the notion of authority, that is, you have anarchy.
Natural laws exist providing a framework that spells out what "just" means. That has nothing to do with government. Other kinds of laws (and there are other kinds---most of them, in fact) simply spell out how those who are perceived as rulers will exploit those who are ruled. They spell out the particulars of enslavement.
I'm not sure it is easy to mete out justice via roaming gangs. Sounds like government to me.