Once your tank is cycled and properly colonized, it's highly stable and resistant to invasive bacteria. Overfeeding and high bioload can allow blooms of other organisms, but apart from that, the bacteria that live on your skin and equipment are not going to cause problems to your fish.
Antibacterial soap is not good either, it just makes them angry. Unless you work with pathogens, just rinse off oils and crap as jratuszn said, and make sure jou rinse off any soap, moisturizer, etc.
If you're still concerned, you can try using long plastic gloves, and rinse and make sure that your equipment dries well between uses. That should minimize communication between your tank and the outside world.