The frogs have different ph requirements than some fish, prefer calmer water than some filters will allow for, differing water temps, water hardness requirements, are slow moving and usually non-aggressive. In a tank where frogs are mixed with fish there's the possibility of frogs nipping on fins(just like they nip other frogs because they don't see well) and if fish are aggressive frogs lose hands, feet or limbs.
Of course they can adapt but they do so much better in their own tank. They're nocturnal, lights go out and they'll swim and play together all night.
It's great that you take time to feed them the worms by hand. Some people just dump their food in when the fish eat and the frogs end up going hungry.
My male Betta loves the frogs too but after watching them nip each other accidentally, I moved Rocky in with the other fish and the frogs have homes to themselves.
I belong to-two great Yahoo groups one is AfricanDwarfFrogs and the other froggyfriends. The moderators are awesome and if you have any issues they'll work night and day with you!! They've taught me so much.