What kind of fish ? Little crays hide, it's instinctive with them, if there are hiding places. If not, provide some. Rock, caves, wood, plants, whatever. Full size cray or the mexican dwarf orange ones ?
Fish may indeed eat some, but probably not all of them. Unless you are really anxious to save every baby cray, you are likely ok leaving her in the tank. And fish are not likely to eat so many that it causes trouble. They have a lot of calcium in the shells, so the protein content is not as high as it would be if it ate a fish of similar weight.