Description

The above video from Renesys shows how Internet outages spread across New York and New Jersey.



Update 10/29/2012 21:00 PST



1. 8x8 outage/111 8th Ave NYC



There are widespread power outages downtown, with much of Manhattan south of 39th st going dark starting at 2200 Eastern.



2. Toronto, Primus Outage - PRI's and DC -



Possibly related to Sandy forecast track as she travels through Pennsylvania, Rochester, Toronto, Kingston, Kanata, Ottawa and upwards.



Toronto-10 Bay Data Centre Network Service Outage

On Monday, October 29, 2012 at 9:10PM EDT, Black Iron Data (Primus Telecommunications) experienced a network service outage that impacted customers at the Toronto-10 Bay Data Centre. Services should be back up as of 10:35PM EDT. We are still investigating the cause of failure. We apologize for the inconvenience this has caused and we thank you for your patience.



Reference Number MT12671.



Recovered: 29-Oct-12 - 22:35



3. Internap also has big issues, likely to go dead ~3:30-5:30a EDT 10/30/2012.



4. Init7 reports issues, but still has connectivity to NYIIX or Equinix NYC: http://www.init7.net/en/status/&ticket=6088



5. Voxel/Internap are down at 111 8th Avenue http://www.webhostingtalk.com/showthread.php?t=1205042



Received this update by one of Zayo's customer. In preparation for this potential power event, here's the readout by site of Zayos generator backup capacity is as follows:



Hopefully this helps if you are located in one of these locations and /or floors:




* 60 Hudson (NY)



-- 8th Floor MMR (Meet Me Room) 18 Hours of generator backup, with additional fuel onsite to replenish the generator



-- 12th Floor has 30+ hours or generator back up



-- 19th and 20th Colocation Floors 30+ hours or generator back up which is provided by the building




* 111 8th Ave (NY)



-- Has 90,000 gallons of fuel, which is able to provide up to 4 days of power



* 165 Halsey (NJ)



-- 5th Floor has 17+ hours of generator backup



-- 10th Floor has 30+ hours of generator backup



* 401 N Broad (PA)



-- 2nd Floor has 30+ hours of generator backup



-- 3rd Floor has 50+ hours of generator backup



* 1 Summer St (CT)



-- The building has 16+ hours of generator backup



* 111 Market Place (MD)



-- Has 36+ hours of generator backup



* 2100 M St (DC)



-- The building has 20+ hours of generator backup.




The local power provider in the NY Market, ConEd, has intentionally de-energized power systems in the New York metropolitan area to reduce system damage as salt water has entered the underground infrastructure.
Additionally, power outages have been observed in the Newark metropolitan area affecting facility at 165 Halsey.



At this point, there has been multiple fiber cuts as a result of the storm which has resulted in service interruptions to transport services in the Pennsylvania, New Jersey and New York markets.



Update 10/30/2012 7:00AM PST:


#NYS #outages 6am: Cent Hudson 82,489 Con Ed 647,988 LIPA 932,836 Natl Grid 20,935 NYSEG 106,013 Orange+Rockland 130,024 RG+E 23,287



National Grid Power Status Map:

https://www1.nationalgridus.com/PowerOutageMap-NY-RES



Latest alerts, NYC Emergency Mgmt

https://twitter.com/#!/notifynyc



WNYC Transit Tracker

http://project.wnyc.org/transit-tracker/embed.html



City of New York on Twitter

http://twitter.com/#!/nycgov



Update 10/30/2012 9:40AM PST:

Covad/Megapath lines are experiencing service interruption due to equipment failure at the NYCMNYBS central office in New York, NY.



75 Broad Street in Manhattan, where both Internap and Peer1 Hosting are shutting down operations.



Internap notification to customers via pastebin



peer1 status page:

http://forums.peer1.com/viewtopic.php?f=37&t=7532&p=9463%23p9463



Web hosting forum WebHostingTalk, members reported multiple network and data center outages including Cogent, InterServer, HoneLive, ColoCrossing, XO Communications, Atlantic Metro and SteadFast.



Verizon said that not only was its cell service struggling, its fiber and home broadband services were also experiencing extreme difficulties, thanks to underground tunnels being flooded by the coastal surge and high tides during Monday's storm. The largest U.S. cellular network by subscribers said in a public statement today at 12:17 a.m. EDT.



Sprint is experiencing service impacts in the states affected by Hurricane Sandy and the concurrent winter weather conditions, particularly in the New York tri-state area, parts of Pennsylvania, and parts of New England. These impacts are due to loss of commercial power, flooding, loss of cell site backhaul connections, site access and damaging debris.



T-Mobile USA, Severe high winds and weather conditions continue to affect many areas. Due to the impacts of Sandy, T-Mobile customers may be experiencing service disruptions or an inability to access service in some areas, especially those that were hardest hit by the storm.



AT&T, said it is monitoring the situation and is aware of "some issues" in areas impacted directly by the storm.



Update 10/30/2012 13:50 PST:

Google Voice ‏@googlevoice
Due to Hurricane Sandy, some of our carriers are experiencing disruptions which are affecting both inbound and outbound calls.



From the web:
https://productforums.google.com/forum/#!category-topic/voice/O1zHgpbimTc The east coast power outages are affecting some coverage including inbound calls.



10/31/2012 10:00AM PST:

AC2 Transatlantic cable update - The backhaul in Long Island went down (29th, 22.40 UTC) but has been repaired (31st, 02.40 UTC) .. this was only one backhaul path so not everyone was affected - Stephen Wilcox.



Equinix Internet Exchange, Metro Connect



LOCATION: NY METRO AREA

PRODUCTS: Equinix Internet Exchange, Metro Connect

SR#:6445883

Date and Time of Occurrence: 31-OCT-2012 05:34 Site Local Time



INCIDENT SUMMARY: Network Service Issue



UPDATE:

Network services have been restored. Backup generator repairs were completed and power has been restored to customer loads.



This issue is now resolved and service returned to normal as of 31-OCT-2012 07:55 Site Local Time



RIPE Analysis on Sandy's trip:

https://labs.ripe.net/Members/emileaben/ripe-atlas-superstorm-sandy