What is it going to harm? ph is basically the reading of whether your water is acidic, neutral or alkaline.
For the most part, most domestically bred fish can tolerate a very large range of Ph. What they MOST care about is a STEADY ph. Not one that swings wildly. When you get into "specialty" fish or imported wild caught fish is when you really need to start worrying about getting a more "homelike" ph (as well as hardness etc).
If your ph is consistent and you don't keep very delicate fish, I wouldn't worry about. I know of quite a few people who use Co2 injection in their planted tanks that buffer with crushed coral to keep from getting a ph bounce. Test your water at different times of the day just to see what kind of and if you have a ph swing and go from there.