So you now have 0 ammonia, 0 nitrites, and 10-20 ppm nitrates?
If so, I think you're good. You have fish in there, even though just two small ones, but they are producing ammonia. If there's no ammonia in the water, and no nitrites, your bacteria colonies are established. Don't toss a bunch...