Yeah, the little buggers get a bad rap when they're actually doing a vitally important job. The only "dangerous" species are VERY recognizable as being different. Google "bearded fireworm" and "bobbit worm" to see the only two species I am aware of that I'd remove on sight.
The thing to remember about bristle worms, like any other small benthic organisms in our tanks, is that we have absolute control over the population. Add silicates to the water and you get diatoms and pineapple sponge. Add rotifers and you get more pods. Add too much food and you get more worms and mini brittle stars.
I have a bad habit of overfeeding, but I've worked hard to create a very stable food chain. I have many types of pods, a huge population of mysid, at least 3 species of bristle worms, mini brittle stars, and a breeding population of nassarius snails, so it stays pretty clean. I actually really enjoy watching the worms. Fascinating little critters.