I've made a breakthrough that may lead to some interesting conclusions about fish psychology. (I'm being serious)
During the time when my fish hated me, I was living with my girlfriend at the time. Certain fish would change color when they were angry, or act differently, and gradually I noticed this behavior more and more often until it was that way every time I saw them. This applies to fish in both my freshwater and saltwater tanks.
Well I recently found out she was cheating on me and I left her and moved myself and my fish into my house. It's just me and the fish living here and I've noticed that they don't hate me anymore! The color is back to normal, the behavior is a lot better across the board, but more noticeably in the freshwater tank. The freshwater tank happens to be the one that used to be next to her desk, where she (supposedly) spent a lot of time studying.
My conclusion: she must have been brainwashing the fish, trying to convince them (successfully) to hate me. Now that they're free from her corrupt influences, they don't hate me anymore.