I would really advise against taking the fish out to do water changes. That is extraordinarily stressful to them, and is not necessary. You might be doing 1) too drastic of a water change, so that the new water is very different from the tank water (i.e., suddenly very low nitrate compared to the high level before the water change) or 2) not matching the temperature properly so there is a sudden change in temp with the addition of the new water.
If your water has chlorine and chloramines (most municipal systems do) then your dechlorinator needs to handle both of these.
This is aside from the fact that you may have a cycling tank (read the article here at AA about the nitrogen cycle) and this is very stressful for fish. During this time you don't "clean" the tank at all, because you will destroy beneficial bacteria that will make the water safe and sound for the fish.
The brown stuff is diatom algae that you need to ignore for now - it is harmless - and it may just go away on its own. At any rate, it is not hurting anything and efforts to rid the tank of this are not going to help things at this point. Let's put that on the back burner just for the time being.
I echo the idea that you need to get yourself an ammonia, nitrite and nitrate test kit, liquid reagent form, and test the tank yourself.
Do not add any more fish. This will put a strain on your bacterial colonies and we need to get a handle on the problem you are having before any new fish go in there.
Good luck!