DNS - The Weakest Link

We’ve spent a good deal of time and money making all of the FeedLounge systems redundant, with scheduled backups, redundant power supplies, RAID 5 disks, etc. so that if anything does go wrong it shouldn’t take down the system and we can replace it quickly and easily. Today though, we learned that we overlooked something.

This morning austin.kingdesign.net, the server that has hosted Scott and my web sites since late 2003, died a tragic death as the CPU fan failed and the CPU fried in the motherboard. I’ve requested a photo to put up on Flickr.

We’d set up redundant systems for FeedLounge in our co-lo facility, but we’d completely overlooked what would happen if the primary DNS server for feedlounge.com went down. Hadn’t ever even discussed it. Oops.

While both the FeedLounge web site and the FeedLounge service were up and running throughout the day, the DNS server failure made them rather difficult to access as the DNS caches expired. Apologies for the inconvenience to anyone affected by this.

Once we realized the problem (and thanks to the users that let us know), we quickly changed the master DNS records for feedlounge.com and by now I hope that information has propagated and everyone can access both this web site and the FeedLounge service at my.feedlounge.com.

The lesson? Don’t take anything for granted - no matter how well it has performed in the past or how little it seems. Make sure you take a good look at everything that could possible interrupt service and make sure they all have backup systems.