Having dirty substrate actually isn't a bad thing for plants because it generates more nitrogen in the form of ammonia that plants can utilize.
As others have said, I also just vacuum over the top of the gravel and it is fine.
In a non-planted tank, it is very important to keep the substrate clean because the ammonia generation in a non planted tank leads to increased nitrite and eventually nitrate, which will cause fish stress.
However, in a planted tank, plants utilize all 3 of those forms of nitrogen depending on the abundance, so they keep the water clean and safe from those compounds as well as get a boost from them.