I completely agree and this happens far too often. (Here come the downvotes :) ) - I personally have found gang-downvoting from the anti-semites to occur against those they perceive (rightly or wrongly) as pro-semitic or pro-Israel or anti-anti-zionist (very different from being pro-zionist).
They have a list and will gang-downvote anything from a person on that list, regardless of what it is.
I've counted approximately 12-14 people who belong to this group of gang downvoters. Perhaps there are more, perhaps there are less. Either way its very nasty.