Date: 08.06.2008

Time: 08:30pm EST - ???? EDT

Location: Tampa, FL

Summary: As a follow up to the unexpected power outage on Sunday August 5th Sago presents this followup: Around 7pm on August 3rd there was what can only be described as a power surge, during an intense lightning storm in the area of Sago’s Tampa data-center. When the power incoming from the power company was interrupted by the surge, the load was automatically failed over to the UPS battery backup system. At this time, a failure occurred (probably as a direct result of the power surge) which disabled both the primary and secondary UPS battery bank systems. This failure caused permanent and unrecoverable damage to several of the batteries in the battery bank system. On Monday August 4th, when electricians were able to identify and isolate the problematic batteries that were the weakest links, maintenance persons were tasked with replacing these large 300lb batteries. The replacement of these faulty batteries is now completed and has restored the full capacity of the UPS backup systems. Follow up checks to all the UPS batteries are part of Sago's regular maintenance routine and load tests on the UPS system and generators will be performed as normal. During the 10-20 second power outage, the generator would have usually picked up where the local power company failed, however, the interruption to power which did affect DC1 and some of DC2 had already occurred before the generator had time to do its job. The power from the local power company did come right back up and we have remained on the local power company with the generators in standby mode in case of further failure. This power failure by act of nature was very unfortunate with 3 systems failing including local power, primary UPS, and redundant UPS battery banks simultaneously. We have had regular tests of the batteries in place and load tests of the UPS system, but this was unpredicted due to the power surge being the cause.

Date: 08.03.2008

Time: 08:30pm EST - ???? EDT

Location: Tampa, FL

Summary: At approximately 8:30pm EST, a segment of the Tampa network has become unreachable. Our IP Engineers are actively working to bring back up all services as quickly as possible. More information is forthcoming.

TTonight the Allmanaged.com domain is being transferred, Which may affect shared hosting customers who registered their domains  with allmanaged.com as the name servers. The outage will only be temporary, when it is transferred all services will be brought back. This transfer will occur at night to avoid any major problems.

At 2:43pm EST our local power company experienced a brief interruption of service lasting two seconds. All automated protection measures enacted, UPS/Battery power kicked in, and a few seconds later the generator automatically started. Facility power was switched to generator. There were no interruptions in power service to equipment anywhere in the facility. The only unexpected event was the breakage of an air conditioning belt on a backup unit. The belt was immediately replaced.