Think of it this way - even when your water chemistry is good, water changes mimic rainfall in nature.
In nature, if the water isn't to the fish's liking, it can swim off to another place. In a tank, they're trapped.
When you change your water, in addition to adding new water, you also super oxygenate the water with the splashing from the water changer or bucket.
So yes, I wouldn't let my tanks go for weeks at time without a water change, even if testing showed the chemistry of the water was perfect...