Hi Debby,
The general rule for goldfish is one fancy for the first 20 gallons and 1 more for each additional 10 gallons, so your 40 has room for three. Plenty of goldie keepers do that, so don't be discouraged! As others have suggested, though, I would consider re-homing the other fish (including the goldfish you don't keep) before the water quality suffers too much.