In addition to what Jobe hs stated.
This is actually an ircd issue. All modes are specificly handled by the ircd. Banning a vhost and bypassing a ban by issuing mode -x is all controlled by the ircd and can happen even in the abscence of Services.
2 quick solutions:
1> stop using vhosts.
2> Using an ircd that doesn't allow the unsetting of assigned usermodes.
Reference #2, Unrealircd has such a feature called set::restrict-usermodes.