I think it depends on how dirty and the size of the tank/filter.
Just rearranging could stir up a lot of crud, but if you do a partial water change right after, and maybe lightly rinse the filter media for a start, could be ok. Clean a few pieces of décor, but maybe not all at once.
I think...