remove what you can, but for the most part its just going to slowly break down into nitrate and phosphate. no serious health risk to the fish.
also, I don't consider carbon a necessary filter media. its great for removing tannins from new driftwood discoloring your water, and for stripping medications out after a treatment.
but a healthy FW
aquarium shouldn't need carbon on a regular basis.