You can certainly keep mollies in a reef. Go with the black or Dalmatian mollies though, those are the easiest to acclimate. As for the process, drip acclimate them very very slowly in a dark(ish) room. Some sources will say that total acclimation can be achieved in 8hrs. Others say two weeks. I'd personally go in the middle somewhere, it's really up to you.
Mollies are great for algal control, and I think they'd do well in your setup. Just make sure your mandarin gets enough food, because mollies looove to eat and swim much faster and are more competitive.