Why do some men want to bring me down?

Is it because they are misunderstanding me and thinking I am being mischievous (when I'm not intending to) or is it simply because they plainly don't like me and want to do me harm? I have always thought the second explanation was true, but a doubt gnaws away at me that I might be unconsciously behaving in a rude manner. In any case I wish I could just tell people to go to hell and not feel worried about the rude things they might say to me as a result. I mean who cares if some drunken/high/psychotic stranger wants to verbally abuse you in the street; it's nothing personal and they could easily have done it to the next person on their way home from work - right?