The same way babies recognize human faces. Instinct! Fish are highly reliant on instinct, as knowledge isn't usually passed down through generations. They would naturally be able to sense characteristics of their own species, perhaps color, shape, or pheromones. It's not always perfect though. For instances, bettas can often confuse and become aggressive towards fish with large flowing fins, probably mistaking them for another betta.
--Adeeb