It's a common misconception about the filter being the main area where bacteria live, but it's not the case. Unless you are cleaning your filter media 2-4 times per month, the media gets plugged up, and surface area drops. At this point, the filter's main job would be to circulate water.
Substrate is where the majority of bacteria would be actively cycling the tank.
What you have laid out would work, you would have to keep a close eye on water parameters in case you cause a mini-cycle.
A better option would be to aggressively gravel vac every water change, and to not over feed.
If it were easy we wouldn't be here