Do you have another filter in that tank? If so, then perhaps it is doing the bulk of cleaning.
Would you describe the sponge as coarse, medium, or fine? If it is coarse, then particulates will not get trapped in it.
What kind of powerhead are you using?
Pics would help.
You should pull the sponge and give it a squeeze in a bucket.Often my sponges will appear clean when in fact they are loaded with crap.I have run a power head on a sponge for many tanks and if you are feeding enough I would think it needs maintenance regularly.