I experimented with dirt mixes & it was very messy when disturbed. I end up redoing the entire tank as the goldies keep digging & I never get the water clear.
Short of a complete tear down, the only suggestion I have is to gradually mix in some more granular stuff (if you are sure that it is root compaction rather than nutrient depletion .... if you have enough vermiculite to start with, the dirt should not be too compacted ... ). I would suggest something like Shultz's aqua soil (basically 1/8" clay pellets) or very small gravel so there will be no future compaction. Put a layer on top & gently work it into the sand & substrate. It will be VERY messy, so you might want to remove all your inhabitants (with maybe half the water). After you mix everything together, you will end up with a single layer, so you will have to put an additional layer of sand on top if you want the sand look. Do a 100% water change to get rid of most of the suspended particulates, (or 2 or 3 100% changes until the water is clear enough). Refill half way the last time & return your fish & the saved tank water (so the fishies will only see a 50% pwc to reduce shock).