The pH will not go down because of not adding buffer to the makeup water. The pH may go down for other reasons... but not because of the makeup water issue.
When you have a lot of biological activity in a tank... say... like algae growth, or cyanobacteria growth... pH will naturally go down. The more biological acticity, the more tendancy for pH to go down.
Buffer is to increase alkalinity anyway. It does provide a quick pH increase, but that's not it's purpose. Do you know what your alkalinity levels are?
Also... if you only have polyps (zoas, palythoas, etc), stop the phyto. They aren't utilizing it.