OpenDNS System

Chicago NTT Issues

George Patterson — Mon, 29 Jun 2009 21:01:12 +0000

Hello,

Today at Mon Jun 29 19:18:33 UTC 2009 NTT Communications started experiencing issues with one of their routers near our Chicago facility. As a result, we’ve temporarily withdrawn our announcement from NTT and users are using alternate carriers or going to alternate sites. NTT is currently investigating this issue. There may have been a brief period where users going across NTT to our Chicago site, may have experienced suboptimal routing and minor packet loss.

We’ll follow up with NTT’s response to what the issue was and why it occurred.

Update: NTT has stated this issue is resolved. We’re still waiting to hear from them regarding specifics on what the issue was. Traffic patterns should routed normally now.

Scheduled Maintenance at Chicago facility

George Patterson — Mon, 15 Jun 2009 21:41:52 +0000

We’ll be performing scheduled maintenance at our Chicago site between 22:00 and 23:00 UTC. The nature of this maintenance is non critical and is fairly routine. No DNS services will be affected by this maintenance. If you encounter any issues during this time frame, please contact our support department at http://www.opendns.com/support/

Update: This maintenance has been completed. Everything went smoothly and no unexpected issues occured.

Scheduled Maintenance at Washington DC site

George Patterson — Fri, 05 Jun 2009 19:46:35 +0000

We’ll be performing scheduled maintenance at our Washington DC site between 20:00 and 22:00 UTC. The nature of this maintenance is non critical and is fairly routine. During this time, we will be disabling the shortcuts feature for Washington DC users only. This is strictly a precautionary measure. No DNS services will be affected by this maintenance.

Update: This maintenance is now completed and the shortcuts features has been re-enabled. No other services were affected by this maintenance.

Shortcuts disabled in Palo Alto and Washington DC

George Patterson — Mon, 11 May 2009 18:13:07 +0000

We’ve been having issues with our shortcuts feature in Palo Alto and Washington DC today. We’ve disabled the shortcuts feature in Palo Alto at approximately 15:30 UTC and at about 17:05 UTC in Washington DC.

We’re currently evaluating the situation and will re-enable the feature when the issue is resolved. DNS is not affected by this issue.

UPDATE: The shortcuts feature has been re-enabled in Palo Alto.

Shortcuts

George Patterson — Tue, 24 Mar 2009 20:02:32 +0000

We’ve been experiencing issues with our shortcuts feature at our Washington DC, Chicago and London locations. We’re currently investigating the issue and working on a resolution. For the time being, we have disabled shortcuts at these locations. We expect the issue to be fully resolved within one hour. No DNS services are affected by this issue.

Service latency and packet loss in Chicago

David Ulevitch — Wed, 11 Mar 2009 18:40:31 +0000

We had a routing issue internal to our network in Chicago this morning. It’s currently resolved and we’re sorry it happened. We’re currently investigating the cause. For the routing geeks out there, we use iBGP multipath to direct traffic to various servers in our clusters. For some reason, one route became preferred and all traffic was directed to just one portion of our DNS serving infrastructure, rather than being load balanced across the entire cluster. This caused increased load and latency. Again, we’re very sorry and are looking into the cause as well as improving our monitoring to better detect this type of failure in the future.

New York Maintenance

Noah Treuhaft — Fri, 20 Feb 2009 08:30:42 +0000

We will be performing maintenance at our New York location from 2009-02-20 6:00 AM UTC until 2009-02-20 7:00 AM UTC. During this window, customers will be routed to one of our other locations.

Update: Equipment failure is delaying completion of this maintenance.

Update: Maintenance was completed at 2009-02-20 9:30 AM UTC.

Chicago Maintenance

Noah Treuhaft — Wed, 18 Feb 2009 06:40:03 +0000

We will be performing maintenance at our Chicago location from 2009-02-18 6:00 AM UTC until 2009-02-18 7:00 AM UTC. During this window, customers will be routed to one of our other locations.

Update: Maintenance was completed at 2009-02-18 6:40 AM UTC.

Washington, DC, Maintenance

Noah Treuhaft — Thu, 12 Feb 2009 07:05:49 +0000

We will be performing maintenance at our Washington, DC, location from 2009-02-12 6:00 AM UTC until 2009-02-12 7:00 AM UTC. During this window, customers will be routed to one of our other locations.

Update: Maintenance was completed at 2009-02-12 7:05 AM UTC.

WorldNic (Network Solutions) resolution issue

George Patterson — Fri, 23 Jan 2009 21:49:06 +0000

Domains hosted with Worldnic (Network Solutions) are currently
experiencing issues with their name servers. Network Solutions is working to resolve the issue on their end. See http://forums.networksolutions.com/announcements-f4-dns-issues-possible-impact-to-your-ecommerce-site-t3433.html#entry14382 for more information on their forums page.

If you are able to resolve the site with another DNS provider this is
merely due to them having the site cached locally.

Hopefully, Network Solutions will have this resolved soon. As a result of this issue, you may experience the OpenDNS ServFail page explaining that the sites Name Servers are failing.