FK, I fully comprehend the argument.
Rights derive from the axiom of self ownership. Property acquired through voluntary exchange, or 'homesteaded,' extends from that first axiomatic right. All other rights extend from the property acquired in those two ways.
As a corollary, a 'crime' or 'violation' is only that which interferes in another person's free exercise of control over their justly acquired property, as defined above. No force is justified that is not used to prevent the violation of those property based rights.
Therefore, if someone snaps a shot of a nude 8 yr old in their own home and sells it to perverts online, punishing them would be unjust.
Physically preventing someone from torturing an animal to death on their property would likewise be unjust.
If someone came onto one's rightfully owned property, and the owner then seized this person, arbitrarily sentenced them to life imprisonment in a private dungeon, this would be fully justified, and no force used against said person on their property would be just.
Likewise, if one lived on a lot surrounded by 4 stretches of private property, and they refused passage, that would be just. The sorry sob stuck in the middle has no right to trespass or use force., even if it means he dies.
Your philosophy has been tried before. It is feudalism.