Hi Anna,
You have a lot of options in choosing a cave. Just make sure it's not painted (if you want colors, it should be dyed in such a way that it cannot leach out) and doesn't have adhesive stickers. Avoid metal, especially copper.
You could use a glass bottle, jar, or toothbrush holder, as long as it doesn't have sharp edges. Glassware is a great choice, as bottles and such have survived centuries underwater after shipwrecks.
You could also use an unglazed terra cotta pot turned on its side, which looks very lovely in my opinion.
You could try a ceramic one, too, as long as it was food safe. Mugs and other similar items, especially dishwasher safe ones, would be great.
If you want plastic, some types are safe and some aren't. Food grade plastics like #2, #4, and #5 work great, just look for the number inside the little recycling sign. Lego bricks and PVC pipe are safe, too, and you can do really cool things with those. Avoid any plastics that smell strongly or have paint or stickers that flake off, these are not safe.