Yes you can use gravel vac. Just wave the wand over the sand and the detremis will be sucked up. I also deep gravel vac and some of the sand gets sucked into the tube but I raise it out of the sand before it reaches the top of the tube and it falls back out before beig sucked up into the hosing. Pool filter sand can be quite coarse so it doesn't get sucked up too easily.
Keep one finger ready at the end of the pipe in the bucket.
When you want to stop the suction just put your finger over the hole, let the sand fall back out and then release your finger and start again.