if its just a few pieces? is it really bad? if its just one or two rocks maybe spot scrub if your concerned. i would still try to leave it alone if i could.
even if it was fishless there is still organic life on and in the rock.
die off from moving everything will start your cycle.