the best investment with your co2 investment is a drop checker. the drop checker changes color between blue (not enough co2), green (perfect co2), yellow (too much co2). as long as its not yellow, your fish will be fine. co2 doesnt replace o2 in the water. so as long as you have an o2 supply (plants during the day, turn your co2 off at night with a timer) you'll be fine.
what kind of filter do you have? is it a hob, or canister? the more surface turbulation that you have the more your water will lose its co2. but i wouldnt turn it off