The folks at the LFS
I frequent never use gloves. However, I use gloves on ALL my corals when handling them. If nothing else, there's a ton of not-so-friendly bacteria that find the coral's mucous coat a nice home. They don't bother the corals, but there is always the possibility that they could bother YOU if they find a nick or cut in your hand.
Of the three you named, like melosu58 commented, I've heard/read of people having issues with zoanthids. They do have some fairly nasty neurotoxins associated with them, but having never handled mine without gloves, I can't speak from personal experience.