Unfortunately 20L is a tiny tank and not suitable for goldfish. Your guys are pond fish and could easily reach 30-45cm. Do you own a test kit or know somewhere that tests? What are the ammonia, nitrite, nitrate and pH of the tank? My guess so far is that the toxins in the water have built up and he's having issues with it.
The very best thing you can do is rehome them to your LFS
or a pond. 20L is a good fit for a Betta if you get a heater.
Please read this.