Posts in: September, 2005

Scott and I had a good time meeting folks this evening. I hope you new alpha users are ‘lounging happily.

Besides the pleasure meeting some interesting folks, we got some good feedback and got to watch people using the software for the first time. That is always an interesting event for an interaction designer (users will always find a way to surprise the designer). I was interested that no one tried to use the keyboard until I suggested it; we need to make that functionality more obvious somehow.

A few people there were taking photos, so hopefully we’ll be able to link to them soon.

UPDATE: Adam Tow’s photos from the evening are up on Flickr (and on his blog).

When: Wednesday, September 21, 2005, starting at 6:30pm PDT
Where: Borders Bookstore in Palo Alto on University Drive

We will be hanging out on Wednesday evening, happy to chat to anyone who wants to drop by. We will also gladly give demos of FeedLounge. Stop on by and say hello.

You will find us out in the courtyard of the University Avenue entrance, drinking various beverages. You will know it is us by the over average number of laptops on the table (PC and Mac). If you see a concentration of laptops greater than 1:1, that is us

Scott and I will be in the Bay Area next week, taking a few meetings and catching up with some old friends. We might even squeeze in a little development time¹. Get in touch if you’d like to meet up.

We were also talking over the idea of spending an hour or three at a coffee shop on Wednesday evening or Thursday ( in Palo Alto area?) to meet with local alpha testers and anyone else who’d like to come by and chit chat. We could have a little FeedLounge demo set-up too if there are people who’d like to play with it. Would this be of interest?

1. It might be more fun to get a couple developers together and try to keep them from working. [back]

I guess I should add Google Blog Search to the available search feeds in FeedLounge for the next release.

We are back online, on the new hardware, and all network issues have been resolved.

In the forums, we are asking for feedback on load times, so we can prioritize where to spend our time improving performance (now that it can be improved).

When you login to the new FeedLounge hardware, please keep track of some timing information and post them here:

The 4 times I am looking for are:

1. Home Page Loading time (time after login)
2. Feed Item List Loading (clicking on a feed in the list, time to load the item list)
3. Item to Item transition time (classic spacebar, next unread time)
4. Time to load a tag (please note how many feeds in the tag)

Some times that I am seeing, in testing:

Due to some database development that was delayed due to other factors, the home page loading will still be fairly slow (I have seen between 5 seconds and 2 minutes, based on the number of feeds and tags).

The feed item list loading should be in the 4-5 second range right now.

The item to item transition should be ~1 second, usually less.

The time to load a tag currently depends on the number of feeds in the tag. We will be refactoring the code to attempt to bring that time down to a constant.

Given enough data points, we will prioritize what needs to be tackled from a performance perspective after the home page load (that is priority number one right now).

Scott

George, Justin and I spent most of Saturday and part of Sunday either traveling to/from the data center, or inside setting things up.

My access badges were still not ready so Otto (a sysadmin from our provider) met us at the facility to let us into the server room. Our rack was ready and waiting, with a Cat5 network drop from the top. The space was quite large (5000 square feet), raised floor space with separate NOC offices.

As Justin and I started putting the machines on the shelves, George started configuring the switch. Justin and I made good time installing all the servers into the racks, but slowed down on the cabling and labeling. Without proper labeling, your future data center visits are much less fun.

After we had the switch configured, we plugged it in to test. Everything was fine, so we plugged in the Airport Express so all the laptops had internet access, for IM and search. The rest of the configuration went slowly but smoothly, configuring the web servers, the feed parsing boxes, the database server, and finally the load balancers.

One load balancer would not come up correctly in the failover position, so it headed back home to be repaired.

All the cables were tied into place with “FeedLounge colored” orange cable ties and blue cable wraps. Alex and his organizational skills must be rubbing off of me. I even logged everything we had to do in Tasks Pro. This organizational thing is still new to me.

One of the interesting experiences was the experience of that much electricity for our intern. Justin was amazed that you can feel the electricity.

We finished up about 11pm, and headed out to find some dinner. I want to personally thank George and Justin for their help in the setup process. Without them, I would have spend all 3 weekend days in the colo by myself. I also want to thank Otto for his gracious help in giving us everything we needed, when we needed it.

Installed:

• 1 managed gigabit switch
• 1 load balancer (will be 2 soon)
• 2 web servers, serving the FeedLounge web interface
• 2 feed downloading and parsing servers
• 1 database server

Since we are trying to bootstrap this operation, some of the essential system administration equipment was left for the next upgrade. Things like Power Distribution Units (for remote power control), Terminal Server (for remote configuration of machines before/while they boot up), etc.

Once we can get the alpha speed back to an acceptable level (with the associated scalability features in place), we can prep the feature set for beta and we can also upgrade the infrastructure. The key components: rack, power, and connectivity, are there, and upgradable very quickly.

If you see a a FeedLounge User Agent in your server logs that happened to hit your server a few hundred times within the span of a minute or two, we do apologize - it was completely unintentional.

The problem was moving from one box to multiple servers, and not using ntp to synchronize the time (yet). The feed downloading box would change the time for the next download to some time it thought was in the future, but which was less than the current time on the database box.

We have synchronized the server times, and will be setting up ntp on all the boxes first thing in the morning.

Oh yeah, the new servers are installed in the data center - more on that tomorrow.

It’s been a rough past couple of weeks here in the ‘Lounge. There’s been too much radio silence here in the blog lately, so I thought I’d at least give a status update.

Problem #1 - Scott and I miscalculated how long we’d be able to survive on the temporary alpha box. This was due to a number of factors, probably the largest being that the alpha users each have an average of 74 unique feeds. The current box can barely handle refreshing all these feeds, so it doesn’t serve the web interface well at all right now.

Problem #2 - We hit database scalability issues a little sooner than we thought. The only good news here is that we’re also addressing them sooner than we thought and have a number of solutions planned.

Problem #3 - An unfortunate series of events led to a reduced amount of available time for development over the last month.

Problem #4 - We didn’t get access to the data center to install our real servers when we expected. We should be in there this weekend finally.

Problem #5 - Taking care of business. A number of opportunities and proposals have come our way since we’ve announced the alpha. These are all great things, but they do take time.

While we didn’t expect all of these problems, we knew we’d have some and that was the reason we decided to make our initial release a private alpha. Our alpha users have been amazingly patient during the process, and we are extremely grateful for that.

Note to alpha users: if you need to grab your OPML for use somewhere else at the moment, we completely understand. You can grab it here:

http://my.feedlounge.com/export/opml

That was the ugly bad news - now for the good news.

There is a lot of cause for optimism moving forward. It’s been painful how it’s happened, but we’ve identified the main areas we need to optimize so that FeedLounge can scale well as a service - and we’re making those changes.

The new hardware will also solve a lot of problems. We did a re-load of our database on the current alpha server about a week ago and it took 16+ hours. A full load of the same database on the new database server took 20 minutes. Throwing hardware at a problem is not always a good solution, but having suitable hardware for your application is certainly a good idea.

Here is a rough schedule of things to come:

• Friday night we’ll be taking down the current FeedLounge alpha server. Poor little box, it’s earned a vacation.
• By early next week, we’ll be up on our new servers in the new data center.
• Once we’re comfortable that everything is running smoothly with the new hardware and location, we’ll be doing some serious QA on the new version of FeedLounge we currently have in-house and we’ll be pushing that out when it’s ready.
• More releases to follow, including great new features, many of the optimizations we’ve got planned and the additional service infrastructure we need to support going to a wider release.
• The long awaited beta release!

Good things are coming, stay tuned.