We had a serious router malfunction in Palo Alto tonight. In short, for Cisco folks out there, the “IP RIB Update” process wedged somehow and saturated all CPU.

This isn’t as bad as it sounds, but it’s pretty bad. It meant some packets got dropped on the floor and some routes didn’t seem to get updated. We’ve been aware of some bugs in the IOS code we had running in Palo Alto but had not yet scheduled an upgrade. All of the other OpenDNS datacenters had already had their code updated.

During the course of this issue we decided to do the router code update on the spot. Everything came up okay and things appear stable in Palo Alto.