I would try and take care of all of it now. Get a really hard dish brush and like melosu said, go to town on it. You can use a hard toothbrush to get into the crevices as well. I introduced a small piece into my tank once that had this nasty algae on it, and it ended up ravaging my tank.
Like they say, and ounce of prevention is worth a pound of cure.