You need atleast a 20 gallon for mollies. A 20 gallon long is nice.
Mollies don't school, but they are more active when other fish are around. Females are quite peaceful. Males aren't too bad. In a 20 gallon, they won't attack another male usually - but if they are sailfins they'll flare their fins a bit and then swim to another part of the tank, maybe with some of chasing. I wouldn't do more than two males in a 20 gallon. Make sure you have females to balance them out.
Male fish pester female fish. One male to two females is good. You will get babies. Mollies are livebearers, which mean they do not lay eggs. The eggs are "incubated" in the pregnant mollie until she is ready to give birth. A lot of female mollies are pregnant at the pet store, so you may end up with some fry even if you don't have the intention of having little fish. The mollies will eat their young. If you're not okay with that, lots of plants make nice hiding places. Java Moss is especially good. If you want, you can always use the 5 gallon for the fry until they're big enough to not be swallowed. When the female is looking particularly large, place her in the tank (make sure it's cycled and heated), and she'll eventually get around to the birth process.
You also have the option of getting a tank divider. Leave a decent amount of space for the mollie - ideally she'll be able to swim around. The advantage to this is that you don't need a separate tank that needs to cycle. The disadvantage is that it will take away space from the other fish (Though this is temporary). I do not suggest the tank divider if you have more than the one male to two female fish ratio. There will not be enough space for the fish.
If you decide to get eight fish or so (two males and six females), you will need to clean the tank and change water more frequently. Everyone is right when they say mollies are messy. You can do eight mollies in a 20 gallon in my experience and they will be happy provided the tank is healthy and scrupulously maintained. However, if you do that you will not be able to add more fish, and you will not be able to keep any young (Which you probably won't want to. Give the young fish to a friend or a pet store). I say this because people people have a tendency of cramming as many fish as possible into a tank out of lack of knowledge. Eight is the max for mollies in a 20 gallon. There are species of fish that you would not want to put eight of in a 20 gallon even if they are smaller than mollies. Temperment, if they school/shoal, adult size, activity level, your willingness to keep up on extra tank maintenance, a fish's ability to play well with others, and bioload are all things to consider when stocking a tank. One inch per fish per gallon does not work. By that logic, you'd have one mollie in a four gallon tank, and three mollies in a 12ish gallon tank, and so on. It's not healthy for one mollie to be in a 4 gallon tank, but it does get to a point where you can have more than enough room for a larger number of fish.
Fry need very small food. You can buy fry food at the pet store, which is what I've used in the past. You could probably also pulverise flake food or give them some very young brine shrimp. Brine shrimp are easy to grow, and it would be another use for that 5 gallon tank!
Mollies are omnivores, but they're closer to the vegetarian spectrum. They need the veggie stuff. Buy a flake food that has spirulina in it, or you can use "regular food" and add spirulina. They do eat green algae, but it won't be enough to meet their veggie needs (or clean your tank!). Do not buy the "food for all tropical fish." It doesn't actually meet the needs of any tropical fish.
Mollies are tropical fish, speaking of. They can be found near Florida and other various parts of warmer climate areas. They naturally live in brackish water, and some even make their way to the ocean. They're very adaptable fish. It isn't necessary to keep mollies in brackish water, however. It is unlikely that the mollies you purchased were bred and raised in brackish water. Should you have taken a fish finding expedition or purchased from a breeder who uses brackish water, adding salt wouldn't be a bad idea. Even still, mollies adapt quickly to varying salinities.
Speaking of tropical, mollies do pretty well in the high seventies to low eighties, atleast in my experience.
If you decide to get snails and they end up spawning, you can feed the young snails - maybe twoish days old, you'll have an idea as the shells look nearly translucent and the snails are very small - to the mollies by flicking them off the glass. The mollies will make a strong effort to eat them. If the snails are really bothering you, you can put a slice of cucumber in the tank overnight. Use something to weigh it down (pet stores typically sell clips). In the morning, you should have a decent number of snails on the cucumber. Either throw the snails in the garbage or give them away. Absolutely do not throw them outside, flush them down the toilet/drain, or "rehome" them to a nearby body of water. There is a chance this could screw up your local ecosystem.
Snails will find their way into your tank via live plants or you introducing them purposefully. Rinse live plants in a bucket prior to putting them in your tank. If you choose to purchase snails, buy something that won't breed or get something you can live with. I had some prolific-breeding snails that were a nice redish-amberish color. They weren't particularly big, they didn't make a mess, and they ate green algae. We'd flick the babies off of the glass for "fish treats," but they weren't very bothersome. There were a lot of them. I'm kind of one of those people who feels that if it isn't killing my tank, my fish, or my equipment, I don't care. I'm a bit of an oddball like that, though.
Speaking of, do not ever ever ever use chemical snail killer. Not good for your fish at all. Be wary of snail eating fish. One type of fish, loaches, are often advertised as snail eaters. They do eat snails, but they need additional food as well. Oh, and they get huge.
Hope that helps. Definitely do some poking around. I tried to cover as much as possible, but I wouldn't be surprised if I missed something.
. We just purchased a 5 gallon tank with filter, cover, rocks, plants, etc. we used tap water to fill it, conditioned the water twice in a week's time. We picked up a gold dust Molly on Tuesday. I acclimated it in water in bag for 15 minutes and let him out. 
