Yes BB lives in the substrate too. I've seen very experienced and knowledgable fish keepers claim that the substrate holds the majority of bacteria, and others claim the majority is in the filter. In my opinion both are right and wrong - depends on the individual tank I think as to which holds more.
One of the reasons I remove the gravel over the span of a few days is to allow the bacteria colony in the filters to pick up the slack. Removing it all at once may leave the tank in deficit, though that would only be a temporary situation. Placing the gravel in media bags and leaving them in the tank is just another precaution one can take. Many roads to the same end.