I attended LayerOne this year and ToorCon last year. You'll see scruffy older dudes all around wearing Hawaiian shirts that look and act like they're outsiders. The talks they choose to attend definitely gives it away too--talk about hacking things like smart meters, or manners of accessing major infrastructural stuff, and they'll all be there.
I asked an organizer about em, and he told me they're essentially part of the community--that their status isn't really a secret. DefCon usually has a "spot the fuzz" event or moment or somesuch where they willingly raise their hands to out themselves.