I think at this point, you just have to observe him for awhile. He may simply be a bit of an oddball. If he had swim bladder problems, he would be struggling to swim, possibly on his side, or at the bottom and not able to swim up or stay level.
Some fish can be picky eaters. Goldfish usually don't really fall into this category, but it's possible. You might try some different types of fish food. Or you might try feeding him some fresh veggies. Cooked, shelled peas, broken into small pieces, may be accepted.. if so, they'd be good for his digestion. Or small pieces of lettuce or other leafy veggie.
So long as he IS eating, and swims normally, I don't think there is any need to worry overmuch just yet. But do try to see if he produces, more or less, the same sort and amount of poop as the other fish do. If yes, chances are he's eating enough for his needs. If not, it's possible he's a bit constipated, and the peas are the usual suggestion for that.
If you see any other symptom or strange behaviour, let us know.