Ususally the population will increase if you're overfeeding. How often and what do you feed? You should probably feed every other day if you don't already. As the food for them lessens the population will decrease.
But you shouldn't need to remove them. They are good janitors and turn the sandbed over and eat detritus, waste, etc.
I had a large one (about a foot or so stretched out in my little 14G tank) and I took out a rock one day to do something and he was on it, I took that opportunity to remove him. IMO they're fine unless they're huge for your tank, that's when I'd take some out if you can. I got lucky