OpenDNS System

Washington DC Maintenance

George Patterson — Thu, 24 Jul 2008 23:09:14 +0000

We’ll be performing routine maintenance at our Washington DC site between 23:00 and 01:00 UTC. During this time period, traffic normally routed to this site will be sent to the closest alternate sites. Services will continue to operate as normal during this maintenance.

UPDATE: Site maintenance completed.

Palo Alto issue

bill fumerola — Thu, 17 Jul 2008 09:25:40 +0000

We’re currently experiencing an issue at our Palo Alto site. We’ve re-routed all DNS related traffic and services to the closest sites available. For most users this will be Seattle. We’ll update this notice as soon as we have additional details.

UPDATE: we’ve put traffic back onto the site and will monitor for any hiccups. if the router crashes again, the site will come up hosting only web traffic and DNS traffic will be re-routed. meanwhile, we’ll investigate how a Cisco router can spew errors from the MPLS subsystem when we don’t use MPLS at all at OpenDNS. curious, that.

Emergency Router Maintenance in Palo Alto

David Ulevitch — Wed, 07 May 2008 07:49:13 +0000

We had a serious router malfunction in Palo Alto tonight. In short, for Cisco folks out there, the “IP RIB Update” process wedged somehow and saturated all CPU.

This isn’t as bad as it sounds, but it’s pretty bad. It meant some packets got dropped on the floor and some routes didn’t seem to get updated. We’ve been aware of some bugs in the IOS code we had running in Palo Alto but had not yet scheduled an upgrade. All of the other OpenDNS datacenters had already had their code updated.

During the course of this issue we decided to do the router code update on the spot. Everything came up okay and things appear stable in Palo Alto.

Stats issues in New York

David Ulevitch — Fri, 25 Apr 2008 15:16:11 +0000

We’re investigating some stats processing issues in New York that are causing statistics for New York DNS servers to be delayed. DNS is completely unaffected by this issue. We’ll update this post when we know more about what’s going on and when we expect to have it resolved.

LON - Router Maintenance: 09 April 2008 - 00:15 BST

bill fumerola — Wed, 09 Apr 2008 00:15:11 +0000

As part of a planned, staggered, system-wide software upgrade of the routers running the OpenDNS network, we will be reloading the router serving our London point of presence. This upgrade will take place on April 9th approximately between the hours of 00:15 and 01:15 (BST +0100). No service loss will occur for any users. The vast majority of traffic will be re-routed to our NYC and Washington D.C. sites. Users will notice an approximate 70ms increase in latency to our service beyond whatever latency between the user and OpenDNS existed during this time. The precise amount of latency increase and which alternative site they reach will be dependent on both the user’s location and their Internet Service Provider.

EDIT: work is complete with no problems encountered. (more…)

NYC: Scheduled Maintenance - 03 April 2008 - 08:00 EDT

bill fumerola — Thu, 03 Apr 2008 04:03:58 +0000

As part of a planned, staggered, system-wide software upgrade of the routers running the OpenDNS network, we will be reloading the router serving our NYC point of presence. This upgrade will take place on April 3rd approximately between the hours of 08:00 and 09:15 (EDT -0400). No service loss will occur for any users. The vast majority of traffic will be re-routed to our Washington D.C. site. Some users may notice a negligible change in latency to our service. This will be dependent on the user’s location and their Internet Service Provider. (more…)

Performance in the Washington Area

George Patterson — Tue, 01 Apr 2008 18:24:35 +0000

We’ve identified a DNS performance issue in the Washington DC area and have implemented a fix to resolve the issue. The issue was related to the domain tagging system, to which there are still on-going intermittent issues with domain tagging in specific that are being investigated by our engineering team. All DNS performance related issues have been resolved.

Seattle scheduled maintenance

George Patterson — Sat, 22 Mar 2008 01:03:35 +0000

Our Seattle facility will be performing scheduled maintenance over the weekend. Accordingly, to avoid any potential issues, we will be routing traffic normally sent to Seattle to our Palo Alto facility. Users may see a slight latency increase between Saturday March 22 03:00:00 UTC and Monday March 24 17:00:00 UTC. All services and DNS resolution will continue to function during these time frames.

RESOLVED: Seattle is back in production use as of Monday March 24 22:10:00 UTC

Temporary shortcut errors

John Roberts — Fri, 14 Mar 2008 14:53:55 +0000

Shortcuts are working intermittently. DNS and all other features are unaffected. We’re looking into it.

Update: resolved completely on March 18, at 17:40 UTC.

Washington, DC slowdown

John Roberts — Fri, 14 Mar 2008 01:20:51 +0000

One server in the Washington, DC area cluster was delivering slower responses from ~21:00 UTC on March 13 through 00:30 UTC on March 14. Customers whose queries hit that server may have seen error messages if it did not respond quickly enough. The server was taken out of rotation, and the problem is resolved.