When my Six Line Wrasse decided life was too much and he ended his life by jumping out of the tank, I started to notice a ton of things swimming around too.
It's been a few months, and they mostly stick by my rock and just scurry around the rock. They have not hurt my tank yet. ;-p
If you don't have a Six Line Wrasse and your tank can support one (30 gallons, tank mates being compatible) I would suggest one! They keep a lot of the population down of the things that swim in the night, plus they go after Bristle Worms. I like Bristle Worms as they clean up a lot, but at least the Six Line will stop them from becoming huge.
Fish Sticks, the best thing you can do with your dead fish.