I don't think you can separate ideology and morality entirely from this. Even the statement "it's wrong to harass people for entertainment" is ideological.
The statement "it's wrong to harass people for entertainment" is much less ideological to the extent that (i) you do not immediately jump to conclusions such as "you must be a pro-peace rather than a pro-lulz person then" or (ii) people do not make statements such as "any person who cares for [important thing X] must support the case of [entertainment rather than boredom || giving people peace of mind]"
Ideology starts when you automatically view an instance of a decision on a particular topic as a person's disposition towards that topic, and automatically view a person's disposition towards one topic as a reason to put a label on them and ascribe to them dispositions towards totally unrelated topics.
A non-ideological stance can consist in noting that, while many occasions have shown a co-occurrence of sexism-related idiocy and trolling, they are not the same thing (see Lennart Poettering being harassed for non-sexist reasons), and hence we have multiple angles of attack: combating the trolls through effective lawful action (which is a bit tedious, but still what OP thinks should be done when the guy in person has a well-known track record of harassing people in ways that frequently go beyond what's funny or harmless), combating people's willingness to act on behalf of trolls (e.g. reddit and 4chan's ban on personal-army type posts), as well as acting on the polarizing issues that make people more prone to get drawn into these astroturfing-initiated conflicts (which is a hard thing to do).
Do we have a moral common ground to do any of this? Absolutely. Even if we suspend contentious arguments and ideological blame-shifting.
