Sponge filters, no matter how big, are not very powerful when it comes to actually removing debris from your tank. A sponge filters primary purpose is biological filtration, and they are pretty good at that. Sponge filters are slow and gentle, which is why most people use them in hospital, fry, and even shrimp tanks.
So even if you did get a sponge with finer holes it still might not be able to pick all that stuff up simply due to that lack of suction power.
I would just go with a regular gravity type of vacuum. Just some simple tubing