1st of all a solenoid is electrical, and controls a valve for on or off.
a regulator is usually purely mechanical and requires mechanical adjustment to reduce the pressure from tank PSI to a lower more usable pressure (PSI)
SO if I understand you correctly you remove power completely from the solenoid and its still runs as if it was powered on.
I would say the valve itself, that is what the solenoid connects to is bad (not the regulator) and needs to be replaced.
Now if a solenoid is rated for low power and you apply the wrong power to it you can cause it to fry.. OS make sure the power ratings are 100% correct
So the only part that would be bad in the case of the photo above and IN MY OPINION is the aluminum bodied valve sandwich between all that brass