Rewarding 304s (Faster Feed Refreshing)
Since FeedLounge is a bootstrapped operation, we wanted to make sure we could deliver what we promised. This meant being conservative to start, in several areas. One of these was our feed refresh schedule.
Though we are already trying a faster schedule, we are also trying to think of other ways to be bandwidth friendly, yet attempt to provide the best possible service to our users. Oh, the conflict. 
One option we’re looking at is to reward content that is naturally bandwidth friendly.
If a feed serves up 304 Not Modified HTTP response when there is nothing new (saving bandwidth on both ends and processing time for FeedLounge), FeedLounge could reward the subscribers of that feed by updating that feed more often.
Some examples:
- Let’s assume that this theoretical feed is about 20KB, and FeedLounge has decided to update it every 8 hours. In any given day, this feed may max out at about 62KB of bandwidth if it changed every time FeedLounge checked. If the feed does not respond with a 304, then FeedLounge has spent (wasted, really) 62KB of bandwidth for nothing. If the feed does report a 304, then the bandwidth per check is only ~1K.
- Assuming that a non-changing feed reports a 304 when asked by FeedLounge for content, we then have some room to spend bandwidth and gain faster content refreshes. In this latter example, let’s say that FeedLounge updated the schedule to check every 30 minutes. We are now talking about 50K per day in bandwidth, assuming no changes, and only 70KB if the feed changed once per day.
The result is that we could well be refreshing about ~60% of our feeds every 30 minutes. This is something we’ll likely be testing over the next couple of weeks - any initial thoughts?
How many sites do serve up 304s? Do you need anything special to make yours do so?
In our experience, about 60% of feeds serve up a 304 when being checked. Most versions of WordPress, FeedBurner feeds, Movable Type, etc. Did you have a specific feed you were wondering about?
I think that’s a great idea, Scott. If more aggregators start doing this, it will hopefullly push publishers towards adopting 304s, which seems like a positive feedback loop to me.
Respect for HTTP 304: Positive Feedback Loop?
The FeedLounge crew are considering rewarding feeds that publish feeds that are capable of sending HTTP 304 responses. Here’s how this is a win for most everyone:
I, as a FeedLounge user, get fresher feed data. This is where the rubber really …
Geof,
You see the adoption of 304s as an overriding benefit, even if FeedLounge were to end up using the same amount of bandwidth? I was wondering if doing this makes FeedLounge the bad citizen, for being somewhat obsessive in our checking. Should we split the difference?
Well, I don’t think you’re obsessive if you’re checking 304-aware feeds only every 30 minutes; the industry seems to have accepted that as a standard. If that seems OCD, maybe you just develop a second scheduling algorithm and have it be IF(304-AWARE=TRUE). :shrug:
You would be a bad citizen if you pounded non-304-aware feeds every 30 minutes. That just seems needless, and there’s a bandwidth and computational penalty to you for doing it—every received, unmodified feed not only has to be grabbed but then parsed [or maybe you MD5 sum those feeds for newness before parsing? I dunno] as well. Seems to me that you have a disincentive to checking those 304 laggards so often.
Fast feed refresh and “conditional GET” support
When a feed has not changed since the last fetch, a server can answer the request with a “304 HTTP code”. Not all servers have this feature enabled, and the result is bandwidth loss.
Here’s a good idea from Scott (Feedlounge). Server…
I think it’s an interesting idea.
If it increases the adoption by CMS’s of 304 messages for feeds, it would be ‘a good thing’. Well, and as a FL user I get fresher feeds!
Hey Scott,
I assume you guys did some basic math to decide how much bandwidth you’d be saving if everyone were to support 304 responses. Care to let us in on the #s? I’m always curious about statistics for stuff like this, and since I haven’t seen any stats about the number of feeds you guys are supporting, I can’t run the numbers myself.
And just in case you didn’t get that, I was referring to overall total bandwidth. You gave per feed stats, but that doesn’t really tell me anything real-world.
Chris, given that we would decrease our check interval, we would only save about 20% bandwidth if everyone were to support 304s. If we did not change the check interval, our bandwidth would decrease about 3 times. The major savings to FeedLounge would actually be in CPU cycles wasted on non-updates.
I think it’s a great idea! It’s a very creative solution which awards feeds which follow standards, but for good practical reasons; not just standards for their own sake. It helps FL be more successful as a business and makes FL customers happier.
Honestly I don’t even see any potential downsides, so I’m confused by the question. Am I missing something?