Doesn't really matter.
If it's bugging you, take the stuff out an clean it.
If you're not that bothered, leave it.
Best bet is to get to the root of the problem so once you do get rid of it, it's not coming back.
How long do you have the lights on for?
How much natural daylight/direct sunlight does the tank get?