a update around 3:30pm est
<fishfreek> - The building where our datacenter is located lost power.
<fishfreek> - The UPS systems did function properly and carried the power load.
<fishfreek> - The generators did start, however the transfer switch that moves load from commercial power to generator power did not function properly.
<fishfreek> - The UPS batteries were drained because the generator power could not be transferred.
<fishfreek> - The UPS system does not power the HVAC units, so the temperature inside the datacenter climbed very quickly, forcing us to shut down ALL servers to prevent heat from damaging hardware and data.
<fishfreek>
<fishfreek> The generators and UPS systems are the responsibility of the building owners which we are leasing space from, and they were tested and expected to work properly. It has not yet been determined why the generator power was not transferred properly.