I switched gravel with my old fish once, and he seemed fine with the change. Not sure about sand though, I've heard it can trap gas pockets or something... But then again there's always some kind of downside. Just take the fish out when you do it, because he won't be able to see anything.