We’ve put a patch in place to avoid the cached data that has shown itself to be problematic in our debugging. This should resolve the “funny” behavior we’ve been seeing today, if you’re still seeing problems please let us know in the forums.

UPDATE: *sigh* - seems like this fix doesn’t work for a lot of folks, we’re still on it.

This morning I discovered that the web server for the ‘Lounge (my.feedlounge.com) had gone down and not restarted properly. This is the first time this has happened since January, and the hits have just kept on coming all day long.

As soon as the ‘Lounge was brought back online, we started getting reports FeedLounge acting “funny”: feed items not loading properly, non-matching unread counts for feeds and strangeness with the items returned for some feeds (read/unread status being off, etc.).

After a day of debugging, Scott traced the problem to (what looks like) corrupt memcached data.

The good news (if you can call it that) is that the “real” data in the databases seems to be fine - it’s only the cached data that is out of whack.

We’re trying to find a way to fix this without requiring a clearing of the cached data, because if we have to clear the cache performance will suffer for a few days while the cache repopulates.

More news as we have it.

We are very pleased to announce the “Avi Flax” release of FeedLounge.

This release adds two major features:

1. Mark Page Read - Many people have asked for this, we’re happy to make it available. There is now a small drop down arrow next to the “Mark All Read” button, when you click this you can choose to have either a Mark All Read button or a Mark Page Read button.
2. “Big Red Button” - from time to time people want to be able to completely reset their account, deleting all their feeds and tags. We’ve now given each user the ability to do this (though we’ve also added several layers of “are you sure”s to make sure no one does it by mistake).

We’ve also fixed a few bugs:

• When “paging” to a page that no longer exists due to hidden read items, the feed properly shows the last available page. (forum thread)
• After adding an authenticated feed, the Private checkbox is properly available when adding additional feeds. (forum thread)

And we’re continuing to do lots of behind the scenes work on the back-end systems so that we can continue to meet the needs of our customers as we grow.

We are honoring Avi Flax with this release. Avi has been an active member of the FeedLounge community, providing great feedback for us in the forums and helping spread the word about FeedLounge.

Mike Marusin would like to hear some opinions about FeedLounge. Perhaps some of you could drop by and share your thoughts?

After our DNS fiasco last month, we’ve taken a number of steps to ensure that we can better handle an unexpected situation like that in the future. One of the things we choose to do was to move the System Status information for FeedLounge to a different domain entirely: feedlounge.info.

This change gives us 2 things:

1. A historical view of any issues that arise (you can see if there was a problem recently and when it was fixed).
2. A location for this information that is not affected by anything that happens to the feedlounge.com domain.

We’ve transitioned system status posts from this blog over to the new log site.

Many thanks to Geof Morris for both the “.info” suggestion and for hosting the feedlounge.info domain on his server (to give us yet another geographic location). Also, a nod to FastMail¹ for having a good example of how this can work.

Also, to address potential DNS problems specifically, we’re now set up with DNS servers in 4 geographic locations in the US:

• Austin, TX
• Houston, TX
• Silverspring, MD
• Englishtown, NJ

We’ve also got a 5th server in San Francisco ready to bring up should we need to at any point.

1. Disclosure: that link has a referral ID in it. [back]

I will be speaking in a PostgreSQL Migration Webinar, hosted by GreenPlum. I know it is late notice, but if you are interested in PostgreSQL in your enterprise, come on over and check it out, because when Stephen O’Grady speaks, nearly everyone listens.

1. Most of the invalid characters in XML are now taken care of. (forum thread)
2. Feedburner feeds now properly subscribe/unsubscribe. (forum thread, forum thread, forum thread)
3. Flagged items from older subscriptions now clear correctly. (forum thread)

Alex talked earlier this morning about how little one thinks about DNS until it is gone/has failed you in some way.

This post will explain the problems we encountered, and cover the basics of DNS. Feel free to skip the tutorial if you are already familiar.

Basics of DNS

Everything on the internet today has an IP address. This is simply a street sign that tells you where you can find some resource, be it a server (like the server hosting this blog), a service endpoint (like email or a web site). When you sign onto the internet you want to get to these places, but you don’t want to remember addresses like 65.90.218.228. You want to use easy names like feedlounge.com, and alexking.org. DNS (Domain Name System) helps you do that.

You type feedlounge.com into the browser, the browser asks the DNS system who knows where feedlounge.com lives? Your DNS then queries that server (the authoritative server) for feedlounge.com’s address. Once your browser has that address, it then initiates a conversation with the feedlounge.com web server, and you finally get that pretty page you’ve always wanted.

What went wrong

What happened in our case is that the authoritative server went away (burned up), and the cached information on your DNS server expired. At this point, even though feedlounge.com is in the exact same place it was, your DNS doesn’t know where feedlounge.com is - and furthermore when it checks to find out, it gets no response.

The Catch-22, or why we were stuck

When we started FeedLounge, things were going fast and furious. We were in a constant change mode, and at the time we couldn’t live with the timing limits of DNS changes provided by our registrar, so we took over DNS responsibilities on our shared server, austin.kingdesign.net (RIP). At the time, it was just to make sure that changes could be made as timely (fast) as possible - to keep everyone happy. And DNS hummed along¹ without ever having a problem… until it disappeared along with all our other websites, email, etc. this weekend.

Generally a DNS record has a TTL (time-to-live, the expiration date on the carton of a DNS record) of 24 hours.

The problem is that when your TTL of DNS records is turned down so low, bringing up another DNS server to serve new records can take what seems like FOREVER (24-48 hours). 24-48 hours isn’t bad when your TTL is also in the same range, and your current DNS server is alive. You just make the transition, and everyone eventually forgets about the old server, and then move on to the new server.

However, when your old server dies you are stuck.

So you turned down the TTL to be responsive, but when the DNS server went away, everyone forgot where you were! All for something that we had previously taken for granted because it “Just Worked”.

Why couldn’t we just set up a redirect?

Redirection is something that happens once you have found a resource to connect to. If you could get to feedlounge.com to receive a redirect, you would have been able to get to the real feedlounge.com. The DNS is the redirect, and if that fails, you’ve got bupkis. The fastest solution for this scenario is to register new authoritative name servers, and that is exactly what we did, it just takes a long time to fully transition.

Conclusion

As Alex said, don’t forget about the little things, like DNS.

If you are changing server addresses often, turn down your TTL to make sure the transition happens as fast as possible. To change DNS servers, turn it up, so that it happens as slow as possible in case your server melts down, and you need some breathing room to bring it back.

And ALWAYS have a backup.

1. Like it had on that server since late 2003). [back]

We’ve spent a good deal of time and money making all of the FeedLounge systems redundant, with scheduled backups, redundant power supplies, RAID 5 disks, etc. so that if anything does go wrong it shouldn’t take down the system and we can replace it quickly and easily. Today though, we learned that we overlooked something.

This morning austin.kingdesign.net, the server that has hosted Scott and my web sites since late 2003, died a tragic death as the CPU fan failed and the CPU fried in the motherboard. I’ve requested a photo to put up on Flickr.

We’d set up redundant systems for FeedLounge in our co-lo facility, but we’d completely overlooked what would happen if the primary DNS server for feedlounge.com went down. Hadn’t ever even discussed it. Oops.

While both the FeedLounge web site and the FeedLounge service were up and running throughout the day, the DNS server failure made them rather difficult to access as the DNS caches expired. Apologies for the inconvenience to anyone affected by this.

Once we realized the problem (and thanks to the users that let us know), we quickly changed the master DNS records for feedlounge.com and by now I hope that information has propagated and everyone can access both this web site and the FeedLounge service at my.feedlounge.com.

The lesson? Don’t take anything for granted - no matter how well it has performed in the past or how little it seems. Make sure you take a good look at everything that could possible interrupt service and make sure they all have backup systems.

1. More invalid character fixes. (forum thread)