The bacteria should be in highest concentrations on surfaces that have water flowing near them. On top of that the high-quality, modern biological filter media has massive amounts of surface for the bacteria to cultivate. In theory, this means that more bacteria should be in the filter than the substrate.
Now, if you were using a UGF system the story would be different. Although, that probably wouldn't work with the sand anyway.
I have a feeling the whole "be careful not to disturb the gravel" thing is really a holdover from when most people were using UGFs.