How long has the tank been set up?
Has it been properly cycled?
What are your water parameters (ammonia, nitrite, nitrate)?
What is your water change schedule (how often and how much)?
The best thing to do would be to do a water change and gravel vacuum the bottom of the tank to suck up that stuff. If left in there it can cause a lot of problems for your fish (ammonia spike, etc). Either you are overfeeding or/and you aren't changing water regularly enough. Both of these, again, can cause problems.