Baby fish are very fragile for the first bit of their lives. So moving them very well can be the problem. I know when we move our fry (egg layers) we're always very nervous that something will happen to them. What we do is drain out most of the water in the grow out tank and fill it with water from the main tank so that they are being moved into water they are already accustomed to. This helps limit the shock.
Also make sure you are letting them acclimate like you would when you get new fish. I keep the bags from the
LFS when I get new fish. That way if I'm moving older fry I can acclimate them easier.