If it's your typical rock then yes. Granite or quartz will be fine. You may wish to boil it though in water or wash it in bleach and rinse very well. Now some folks will tell you never to do that and go to a pet store, but you can put garden rocks in your aquarium.
Driftwood is trickier. I would say yes, but closely watch your water chemistry. Make sure the wood has been dried and you can put it boiling water for a few minutes and then put it in your tank. Now some will say never and only get stuff from a pet store.
My hunch is that those who tell you only pet stores own or work at pet stores.