I'd change all the gravel and decor all at once unless you have an under gravel filter. Something like 98% of the bacterial population is in your filter and you're also removing all of the hidden gross stuff in the gravel that contributes to production of ammonia. Everyone thinks they do a good job vacuuming the gravel but you might be shocked by how unclean it actually is. I've moved tanks a number of times and have never caused a mini cycle so long as the filter was preserved.
It's a huge job but well worth the trouble when you see the results. Just make sure you have some time and some place to keep the fish for a while. I like ice chests for that. Just put a big, black trash bag in a good sized ice chest and siphon as much tank water in there as you can. Put the fish and any live plants in there too along with a little water conditioner. I close the top of the bag with a bubble and shut the lid to keep it all warm. Depending on your filter type, you may need to put that in plastic too so it stays moist. Drying out is no good at all for bacteria. Then you can go nuts cleaning and changing out your substrate and decor without worry. When it's done, fill the tank halfway with new water and dechlorinate. Install any plants first and then start returning the old tank water. I wait to move the fish until the tank is full and the filter is back on.
Be sure to snap before and after photos!
