I've heard that a lot of people do this. To me, it depends on the condition of the environment in the immediate area. Too many pollutants like oil or chemicals can be absorbed by the porous portions of the rock or by the growth on the rocks. I often tend to lean on the conservative side of cautious but unless you can be absolutely 120% sure that it's safe, I wouldnt.
Why do I love fish? Because every time I stare into the little world I have created for them, every problem I have just floats away....