I agree that you probably do not spend enough time in that room for them to be accustomed to you. I have two tanks in my computer room, which I spend about six hours a day in, give or take. The fish became used to my presence quickly--within two weeks, the friendlier types (like my female bettas, peacock gudgeons, dojo loaches and balloon mollies) came right up to the glass, eager to see me, not the least bit shy. I decided some time ago that I prefer the calmer, more "intelligent" acting fish that seem to be interested in me.
I do know that some tetras, sharks and swords can be a lot more skittish. I have Rainbowfish and even though they don't hide or huddle on one side of the tank, they are not the personable, friendly type of fish that my others are. They are very quick, and move away if I get super close to the glass, looking in. From now on, I am only going to keep friendly types, as I enjoy them so much more. But to each his own... I've had swords in the past, and didn't care for their personalities at all. Not only were many of them downright nasty to other fish, but they never seemed to "bond" with me at all, as the other types I have do. They remained fairly "wild" acting. I have two Honey Gouramis. One in my 20, and one in my 55. The one in the 55 is far friendlier and calmer than the other one, so sometimes its also just the individual fish, not necessarily the species as a whole that are shy/skittish. I find I also enjoy endlers a lot; even though they are pretty hyper, they don't show any fear of me.
I also agree that if you get more fish, they may very well feel more comfortable/safe swimming all over the tank, and not just staying in that corner. Remember, safety in numbers!