If you've had your betta for a week, he may not be settled down yet. It may take another week before he feels comfortable. I had a betta that did seem to take awhile to settle into his new home.
The tank is 2 gallons, right? This is really too small for a betta. It's a shame that they are sold as bowl-type fish. They really do need heat and filtration just like any other fish. Do you have any heat or filtration in this tank?
There are some other possibilities I can think of. One of my bettas swam very erratically when the temperature rose (the heater stuck). If you don't have a heater, then I'm thinking that the temperature is fluctuating. Bettas are very sensitive to temperature fluctuation.
Do you have a pH test? A stable pH is more important than artifically lowering or raising the pH. Another time when I felt my betta was swimming erratically, I did a pH test and discovered that the pH had fallen a little too much (this was an error on my part as I mixed tap and RO
water). I gave some specific examples, but in general, water parameters that fluctuate quickly could be to blame.
When you say he seems to calm down when you turn off the light -- are you talking about the tank light or room light? Bettas don't like bright light. Maybe you could move him to a less bright spot of the house or room, or don't use the tank light if it's too bright. The tank may not even need a light. The room light is probably enough.
What kind of decorations do you have in the tank? The betta would like some silk plants to hide and rest in.
If you're able to get him a bigger tank, that would be good for him. I have pictures in my gallery of bettas in 5 gallon tanks. There are some small, 3 to 5 gallon tank kits available, with filtration included, like the Eclipse or Minibow systems. All you add is a heater, gravel, and some silk plants.
Hope this helps a bit!