Posts in: General

Mike Marusin would like to hear some opinions about FeedLounge. Perhaps some of you could drop by and share your thoughts?

After our DNS fiasco last month, we’ve taken a number of steps to ensure that we can better handle an unexpected situation like that in the future. One of the things we choose to do was to move the System Status information for FeedLounge to a different domain entirely: feedlounge.info.

This change gives us 2 things:

1. A historical view of any issues that arise (you can see if there was a problem recently and when it was fixed).
2. A location for this information that is not affected by anything that happens to the feedlounge.com domain.

We’ve transitioned system status posts from this blog over to the new log site.

Many thanks to Geof Morris for both the “.info” suggestion and for hosting the feedlounge.info domain on his server (to give us yet another geographic location). Also, a nod to FastMail¹ for having a good example of how this can work.

Also, to address potential DNS problems specifically, we’re now set up with DNS servers in 4 geographic locations in the US:

• Austin, TX
• Houston, TX
• Silverspring, MD
• Englishtown, NJ

We’ve also got a 5th server in San Francisco ready to bring up should we need to at any point.

1. Disclosure: that link has a referral ID in it. [back]

I will be speaking in a PostgreSQL Migration Webinar, hosted by GreenPlum. I know it is late notice, but if you are interested in PostgreSQL in your enterprise, come on over and check it out, because when Stephen O’Grady speaks, nearly everyone listens.

Alex talked earlier this morning about how little one thinks about DNS until it is gone/has failed you in some way.

This post will explain the problems we encountered, and cover the basics of DNS. Feel free to skip the tutorial if you are already familiar.

Basics of DNS

Everything on the internet today has an IP address. This is simply a street sign that tells you where you can find some resource, be it a server (like the server hosting this blog), a service endpoint (like email or a web site). When you sign onto the internet you want to get to these places, but you don’t want to remember addresses like 65.90.218.228. You want to use easy names like feedlounge.com, and alexking.org. DNS (Domain Name System) helps you do that.

You type feedlounge.com into the browser, the browser asks the DNS system who knows where feedlounge.com lives? Your DNS then queries that server (the authoritative server) for feedlounge.com’s address. Once your browser has that address, it then initiates a conversation with the feedlounge.com web server, and you finally get that pretty page you’ve always wanted.

What went wrong

What happened in our case is that the authoritative server went away (burned up), and the cached information on your DNS server expired. At this point, even though feedlounge.com is in the exact same place it was, your DNS doesn’t know where feedlounge.com is - and furthermore when it checks to find out, it gets no response.

The Catch-22, or why we were stuck

When we started FeedLounge, things were going fast and furious. We were in a constant change mode, and at the time we couldn’t live with the timing limits of DNS changes provided by our registrar, so we took over DNS responsibilities on our shared server, austin.kingdesign.net (RIP). At the time, it was just to make sure that changes could be made as timely (fast) as possible - to keep everyone happy. And DNS hummed along¹ without ever having a problem… until it disappeared along with all our other websites, email, etc. this weekend.

Generally a DNS record has a TTL (time-to-live, the expiration date on the carton of a DNS record) of 24 hours.

The problem is that when your TTL of DNS records is turned down so low, bringing up another DNS server to serve new records can take what seems like FOREVER (24-48 hours). 24-48 hours isn’t bad when your TTL is also in the same range, and your current DNS server is alive. You just make the transition, and everyone eventually forgets about the old server, and then move on to the new server.

However, when your old server dies you are stuck.

So you turned down the TTL to be responsive, but when the DNS server went away, everyone forgot where you were! All for something that we had previously taken for granted because it “Just Worked”.

Why couldn’t we just set up a redirect?

Redirection is something that happens once you have found a resource to connect to. If you could get to feedlounge.com to receive a redirect, you would have been able to get to the real feedlounge.com. The DNS is the redirect, and if that fails, you’ve got bupkis. The fastest solution for this scenario is to register new authoritative name servers, and that is exactly what we did, it just takes a long time to fully transition.

Conclusion

As Alex said, don’t forget about the little things, like DNS.

If you are changing server addresses often, turn down your TTL to make sure the transition happens as fast as possible. To change DNS servers, turn it up, so that it happens as slow as possible in case your server melts down, and you need some breathing room to bring it back.

And ALWAYS have a backup.

1. Like it had on that server since late 2003). [back]

We’ve spent a good deal of time and money making all of the FeedLounge systems redundant, with scheduled backups, redundant power supplies, RAID 5 disks, etc. so that if anything does go wrong it shouldn’t take down the system and we can replace it quickly and easily. Today though, we learned that we overlooked something.

This morning austin.kingdesign.net, the server that has hosted Scott and my web sites since late 2003, died a tragic death as the CPU fan failed and the CPU fried in the motherboard. I’ve requested a photo to put up on Flickr.

We’d set up redundant systems for FeedLounge in our co-lo facility, but we’d completely overlooked what would happen if the primary DNS server for feedlounge.com went down. Hadn’t ever even discussed it. Oops.

While both the FeedLounge web site and the FeedLounge service were up and running throughout the day, the DNS server failure made them rather difficult to access as the DNS caches expired. Apologies for the inconvenience to anyone affected by this.

Once we realized the problem (and thanks to the users that let us know), we quickly changed the master DNS records for feedlounge.com and by now I hope that information has propagated and everyone can access both this web site and the FeedLounge service at my.feedlounge.com.

The lesson? Don’t take anything for granted - no matter how well it has performed in the past or how little it seems. Make sure you take a good look at everything that could possible interrupt service and make sure they all have backup systems.

We are very pleased to announce the “Matt Walters” release of FeedLounge. This releases introduces the first FeedLounge APIs.

We’ve had OPML export in FeedLounge for a long time, but now we’ve added the ability for users to publish their reading list from a new API tab in the Settings window. If you choose to do so, your reading list (including your tags) can be made public as:

• OPML
• RSS
• HTML

Pretty cool, right? But wait, that’s not all!

We’ve also added an API for a FeedLounge notifier! This doesn’t just return the number of unread items, it also gives you the number of unread items in each tag. This way, notifier developers can add the kinds of smart features our users are asking for.

We’ve had a number of folks volunteer to create notifiers¹ for FeedLounge, and we are happy we can now take them up on their offers. We just might consider giving a little reward to folks that create really great notifiers as well. What will it be? Build one and find out!

We’ve introduced an API token for each user as part of the API feature set. The notifier API supports this token, so notifier developers can ask for either the user’s API token (highly recommended) or for their username and password and use 401 authentication.

There is now an API section on our Support page to cover the new APIs and how to use them.

These are the first APIs we’re making available, not the last. We’ve got some others in the pipeline that we’re excited about and look forward to making public.

We are honoring Matt Walters with this release. Matt is the first non-alpha user we’ve honored in our release naming. He signed up on the first day FeedLounge became publicly available, and has been a wonderful source of ideas and help for other users in the forums. Matt is also the one who added the “notifier” feature to the voting list - glad we’ve got that in for you now Matt, and thanks for your continued contributions.

1. You know, a little app that runs in your menu bar/system tray and tells you how many new items you have. [back]

Basement.org has posted an e-mail interview with me about FeedLounge and our business model. Rich asked some interesting questions.

I participated in this week’s TalkCrunch podcast discussion about feed readers. Towards the end I talk a little about some of the things we have coming soon (for those who want a sneak preview).

Scott has posted some of his thoughts and follow-up over here.

In the “SOG” release, we are introducing the first features that allow FeedLounge to work with other services.

We’ve introduced TagThru™ in this release. TagThru™ takes any item tags you add in FeedLounge and also adds them to your account in a tagging/bookmarking service. In talking with a number of users about this feature, we believe that being able to use the smooth integration of tagging in the FeedLounge interface with the ability for you to also have these tags in your primary tagging/bookmarking service.

The first implementation of TagThru™ is an integration with del.icio.us. If you have a del.icio.us account, you can activate TagThru™ in FeedLounge and whenever you tag an item in FeedLounge, we’ll also add that tag to your del.icio.us account¹.

We plan to add support to other services like furl, Ma.gnolia, Google Bookmarks, Yahoo! MyWeb, etc. in the future. At the moment these services don’t have APIs for us to use, though most of them have announced that they have an API is planned.

As they create their API’s, we strongly encourage these services to have an API that simply allows the adding of a tag/label to an URL with a single API call. For our del.icio.us TagThru™ implementation, we have to first ask for any tags for the item, do our own merge, then update the item. We do this to make sure we don’t stomp any existing tags you may have given the item in del.icio.us prior to tagging the item in FeedLounge. Sure it’s an edge case, but folks generally aren’t very happy when they lose data.

We’ve also added the ability to sort items “oldest first”. Our users told us that this was important to them on the Voting Page (it was #3 on the list today), we’re glad to be able to deliver it to them. One more dot on the TechCrunch chart for FeedLounge.

In addition to the new features, we’ve also expanded our 50 person, 3 Hour Tour to a 500 person, 24 Hour Tour (restarting at midnight PST). Now you can take FeedLounge for a spin for a full day, so you can really get the full FeedLounge experience.

Stephen O’Grady is honored with this release. Steve has been with us since our early alpha days and has generously provided invaluable feedback to us in a number of areas. Thanks Steve, hope you like being able to tag things in del.icio.us right from within FeedLounge!

1. The call to tag in del.icio.us is placed in the Queue, so the tag may be slightly delayed in appearing in del.icio.us depending on the size of the Queue. [back]

TechCrunch’s comparison of web-based feed readers includes a feature chart that has been copied and included in a number of posts - unfortunately, many of these posts grabbed the original version of the chart which missed a couple of features we have here at FeedLounge.

• Export Reading Lists (OPML) - FeedLounge has supported OPML export since July 2005.¹
• Shortcut Keys - FeedLounge has been built with extensive shortcut keys, including support for arrow keys and the spacebar. There is an FAQ with a full list.

I wanted to mention these here since some folks will be seeing incorrect information on the sites that had copied the original version of the chart. We may be able to add a few more dots to our column soon as well.

Thanks to Frank for including us in his review and for updating the chart on TechCrunch.

1. And our customers notice. [back]