I have had similar bad luck with neons and rummys. I have, in the past, had great success with cardinals. I think it has a lot to do with the source of the fish and our particular tank conditions.
Some of the smaller, delicate tetras like neons have been said to be overbred and may be weak even though they are tank bred. In other cases, some fish may be wild caught and suffer from both the stress of being caught, shipping, and also adjusting to the many different water conditions our tanks present.
Frankly though I love them, because I love them, I'm not getting any more rummys. I just have far more losses than I find acceptable. And believe me, I transport and acclimate very carefully. I do everything I can to ensure a proper environment. Those that survive do have a long life and seem quite hardy. But not enough survive. I have no such losses or problems with my other fishes.
So yes, some of us just have trouble with certain fishes from certain sources and probably should look to keeping other fishes or fishes from different sources if possible. Live, happy fish are so much more fun.