True freedom is as much internal as it is external. In fact, I would argue, it begins inside. It begins with very small gestures in which you are true to yourself ("The right to be who you are."), as in your example of turning down that dinner invitation without any excuse at all. Simply because the answer is: "I don't want to." In a truly free society, that is justification enough. You don't owe anyone an explanation.
But instead, most people have this internal conversation: "I don't want to turn it down because then they'll never invite me again, they'll think I'm a snob and talk about me behind my back, they won't like me anymore." And/or some variation of that.
How many times do we let others dictate our actions in small ways like this? We just give away our self determination.
That is the beginning of opening oneself to larger forms of manipulation.
"I don't want to speak out against the war, because then they'll think I'm weak, that I'm disrespecting their son/daughter/husband/father who is a veteran. They'll think I'm a snob and talk about me behind my back, they won't like me anymore." Etc.
Begin by being who you are. Begin by being true to yourself. That is the beginning of freedom.
To be mean is never excusable, but there is some merit in knowing that one is; the most irreparable of vices is to do evil out of stupidity. - C.B.