Well, sounds like a compatibility issue for sure...
What size tank? Have you tested the water parameters? Are there any hides/caves/plants in the tank? I've heard of gouramis being territorial. What kind of catfish do you have? Don't think the swordtails or tetras would be an issue normally, but I could be wrong or they could be lashing out since I believe they both are schooling fish that should be in groups of 6+. Still new to the hobby so pardon me if I'm incorrect on anything. Just some things to look into.
*Also, are all your guppies male OR female? If you have more males than females, they'd fight but I don't think they'd end up with holes like that...