Do you have any particular corals that you are concerned about? Do they show any signs of damage?
It sounds like you find these little critters interesting...I'm not so quick to condemn anything even if others tell me it is "bad".
It depends on what you want your aquarium to be. For me, there are a lot of "bad" things in my tank, but I find them equally interesting.