Do not remove the gravel!

I imagine part of your problem is that you are having a massive ammonia and / or nitrite spike. (This can easily be confirmed with a test kit.) Here's the basics of the nitrogen cycle as it applies to aquariums:
Fish create waste which is basically ammonia and toxic to the fish. In an established and properly stocked aqaurium, beneficial bacteria which live on solid surfaces (like gravel) establish themselves and break down the ammonia into less toxic nitrite. Now, it's still toxic, just not quite as bad. A different kind of bacteria also establish themselves and break down the nitrite into less toxic nitrate. Without advanced filtration methods, nitrate has to be removed via partial water changes.
The problem is that if you remove the gravel, you're going to lose a LOT of your beneficial bacteria, and you have to start the tank over from square one and this same scenario will play out again. Think of the beneficial bacteria as the "probiotics" people eat yogurt to get.
Without a history of pH, ammonia, nitrite, and nitrate readings, this is my best guess as to what's going on. I've always found your best bet is to start making partial water changes when things start looking off with your fish.