I just learned the hard way that aquaponicpaw is completely correct. I hadn't had a fish die for approaching a year (48 litre tank, with 4 x corydoras, 6 x guppies, 1 x Kuli loach and 5 x neon tetras, which was a little overstocked and everyone was doing just fine!).
Last week I did a complete substrate change and swapped a couple of ornaments. By the next morning ALL of my tetras were dead. The corydoras, loach and guppies all made it, but they weren't happy at all (coming up for air all the time, the loach was going crazy). I realised what the problem was (total lack of bacteria) and took an old ornament, old plant and some of the old substrate out of the bag I'd put in the outside bin (which overnight would have reached almost freezing temperatures!) back into the tank. After a few hours, everything was looking better and now 5 days on I am beginning to see bacteria building up on the glass again (never been so happy to see dirty brown marks on the tank!).
I guess the moral of the story is A) don't take too much "dirty" stuff out of your tank at the same time, and B) bacteria lives for a LONG time outside of the water, so if you make a mistake just stick something covered in it back in the tank and hope for the best!
Strange how ALL the tetras died, but none of the guppies, corydoras or the loach. I guess they aren't as hardy to new water conditions!