Pay back, hmmmmmm not a good idea. You can continue to ask the neighbor, but they could very well call for harassment eventually. I would document (write out and date) the events and take pictures of any future incidents including cigarette butts, gum, etc. on your side of the fence. You may even be able to get a picture of your dog trying to eat one if you're patient. Any pictures will provide visual evidence along with the littering of your yard. I don't suppose you belong to a community where you could complain to the board?