Posts in: Development

Intro

This post is the first in a series documenting our performance testing of the Sun Fire T2000 server against the servers we currently use for the FeedLounge production site.

Sign Up Process

3 easy steps to a server:

1. Signed up for the 60 day free trial. Qualification is easy: (a) you have multi-threaded apps, and (b) they will run on Solaris 10. This one is easy for anyone standing on the shoulders of the open source giants.
2. Received a call about being a business - This one seems to be incorrect, but I am just telling you my experience. According to Jonathan:
   

   And to answer a few of the comment questions - the offer applies to anyone interested - not just corporate customers. We don’t care if you’re an educator or a park ranger or a blogger or a physicist or a CIO - so long as you’re in the market for the fastest/most efficient server on earth.

   This is directly in conflict with the fact that the registration page still notes that company is a required field, and that I was required by the salesperson to have a Federal EIN.

3. Received server 4 days later - This was much faster than most people have blogged about, but it may be because I chose the configuration that most closely matched our existing servers, instead of trying for the maxxed out configuration.

Before I received the server, I received the quote for the server:

• 6 Core 1.0 GHz UltraSPARC T1 processor with 6 cores (24 threads)
• 8 GB DDR2 memory
• 2*73GB Hard Drives
• Redundant power supplies
• 4 GBE ports
• Quote total: $10,895.00 US

The server showed up at the office as 2 boxes : the largest is the server (including rack mounting hardware, 2 network cables, warranty, and a sheet of URLs pointing you to the documentation). The other, seemingly oversized box was just 2 power cords, presumably to keep international shipments easier to process.

The server itself is beautiful, almost like Sun hired Apple to design and build it. Maybe there is a marriage there someday. I would also note that Sun’s experience in the server market shows in every piece of the design. Everything is easily removable, marked well, and generally pleasing to any eyeball.

Setup Disappoinment

I unboxed everything, tried to find a quiet corner, and started the setup. Since they publish all documentation for the server in one place, it is easy to get started.

• Setup server on table, with a piece of cardboard underneath to not damage the server or the table).
• Plug in network cable and power cable. Wait a second! As soon as you plug in the first power cable, it comes to life, and I must say it is quite rowdy (where rowdy is loud). Pull the plug until we are ready.
• Get down to about here in the docs, and it says:
  

  1. If you have not already done so, connect a terminal or a terminal emulator (PC or workstation) to the SC serial management port. Configure the terminal or terminal emulator with these settings:

  * 9600 baud

  * 8 bits

  * No parity

  * 1 Stop bit

  * No handshaking

  Note - When you power on the server for the first time and you do not have a terminal or terminal emulator (PC or workstation) connected to the SC serial management port, you will not see system messages. The display disappears after about 60 seconds.

  2. Turn on the terminal or terminal emulator, if it is not already turned on.

  3. Connect the AC power cables to Power Supply 0 and Power Supply 1, and watch the terminal for system messages.

  Great, where is the magic serial cable? It didn’t come in the box. How do you expect me to set up this server? Where is my great OOB experience!?!

  Not to worry, I ran out to Radio Shack (Fry’s is way too far away these days), and picked up the parts and soldering iron to build myself a serial cable. Good times, I hadn’t soldered for about 10 years, and it is only partially like riding a bicycle, so my joints look like crap, but do work. I bought the Radio Shack 276-1538 DB9 Female Serial connector, a $7.99 soldering kit (wow), and hacked up an old cat5 cable.

  Just take a normal cat 5 network cable, hack off one end, and solder up pin 2 to white/green, pin 3 to green, and pin 5 to both blue and blue/white. Works like a charm. For those using windows, it is trivial to use HyperTerminal to see all the output of the management processor. That is, after you surrender and tell it which area code to use.

  Update: Sue Tobin of Sun called me to follow up after a couple of weeks (excellent), informed me that the serial cable is now included in the box and apologized for any inconvenience. I told her that was excellent news, as it was the single largest barrier to my testing of the server.

  Once I had a serial cable, following the manual to set up the server and get it on the network took just a shade under 1 hour.

System management processor is always on, giving you great remote control…

Assistance From Sun

A Denver area Sun employee has graciously offered to give his time gratis to us to bring up the FeedLounge services on the server (configuring SMF, etc).

As mentioned previously, I also received a follow up call by Sue to check in on me and make sure everything was going well. She responded well to my anger constructive criticism.

Running

The server is now up and running, and boy is it loud! Not really coffee table material. Over the next couple of weeks, we will set up the full FeedLounge architecture on the machine, and benchmark it to a similar setup of our current Opteron based system.

Our current machine is:

• Dual Opteron (Single Core)
• 8GB RAM
• 6x 250GB SATA drives
• Redundant Power Supplies
• 2x Gigabit Ethernet
• Cost: $4900

As you can see, the hard drives don’t match up, so I will keep my testing as drive agnostic as possible.

Why are we doing it?

As with almost any hosted service, FeedLounge pays for hosting in 3 ways (most to least important): By the megabit, by the watt, and by floor space. Since power is an extremely significant part of the equation, anything that drives that number down will significantly drop the overall hosting cost. We are already working on driving down the megabit cost by using gzip, and being smart about bandwidth.

The second reason is that FeedLounge caters extremely well to the massive multi-threading this server is capable of. We use heavy database, web serving, and feed parsing. Almost all of that code is threaded heavily (and where it isn’t, it is large numbers of single threaded processes), and is also lacking in the floating point. If we can get better performance out of less electricity, it is a double win for FeedLounge. Throw in the onboard crypto accelerator, and we are talking about a pretty compelling offering.

Props to Sun

I have to give Sun credit for trying to do something progressive and decidedly “non-corporate” (and also admitting to and fixing any warts in an open manner). The biggest issue, lack of the serial adapter cable, is a big win for the evaluation experience.

Now if only Sun was giving free trials on large amounts of storage. Maybe we should just look at using Amazon S3.

Next up: fitting a round peg into a square hole. Or rather, a Linux sysadmin learns the Solaris ropes.

FeedLounge has reached alpha 6!

As planned, this release is smaller in scale than the previous alpha releases. There are three main features added in this version:

1. Browser Compatibility - FeedLounge now works on IE 6 and Safari, as well as gecko browsers. All alpha users are encouraged to use FeedLounge in IE and Safari for bug hunting. This completes the stated browser compatibility goals we have for our beta release. We will explore supporting additional browsers in the future, but likely after our beta release.
2. Removal of Explicit Save - Items are now saved through tagging, not through clicking the ’save’ icon for each item. This is how our alpha users assumed it worked anyway.
3. Bug Fixes - in particular, we cleaned up some keyboard navigation issues in the “river of news” view and the Tags and History screens properly show items again.

Please let us know in the forums about anything that doesn’t seem right.

This post contains some technical information that will be very interesting to other developers, but may not be so interesting to those that don’t work with databases.

For those of you that just want the nutshell:

We’ve switched to new database server software because we think the new software will be faster and store the same data while using less disk space (making our storage costs lower).

Now for the detailed explanation:

Back in August, we switched the FeedLounge database from MySQL’s MyISAM tables to InnoDB tables. This solved a number of problems that we were having with the MyISAM tables, but also introduced a number of other issues including slower performance on a number of queries (counting in particular). As we did more and more research into how to solve these problems, we decided that one promising solution would be to switch to a PostgreSQL database instead of a MySQL database altogether.

That is what we were testing this weekend, and the alpha server is now running live with a PostgreSQL database instead of the MySQL database. This is still in “testing” mode - if we run into any serious issues we might switch back (temporarily or permanently) to the MySQL database (which will have data current through last Friday evening).

Some of the reasons we decided to make the switch to PostgreSQL:

• Database size - when we switched from MyISAM tables to InnoDB tables in MySQL, the size of our database grew from ~1GB to 10+GB! When we made the switch this weekend, the MySQL InnoDB database was using 34 GB, and the same data in a PostgreSQL database is only 9.6 GB - this should keep our hardware costs down a bit.
• Load time - The current MySQL setup takes over a day to restore the current database, the load of the data into the PostgreSQL database took just over 4 hours.

I was once given a simplistic summary of the MySQL and PostgreSQL databases:

MySQL was built with speed as the primary feature, PostgreSQL was built with more robust features like triggers, but lacked the speed of MySQL. As both are maturing, they are moving towards the other. MySQL 5 adds triggers and stored procedures, while PostgreSQL is focused on improving performance.

For an application like FeedLounge, the faster counting and smaller row size of PostgreSQL are compelling reasons to use it instead of MySQL with InnoDB tables. Add that it runs in 1/3 the time, using 1/3 the memory (making it essentially 10x faster for us) and that we can start taking advantage of other Postgres features, like data partitioning and partial indices in the near future, and you have the reason we decided to make the switch.

This week should be an interesting one, we’re looking forward to getting feedback from our alpha users on how FeedLounge is performing on the new database.

Finally! We’re live with our alpha 5 release.

Some new features of note:

• A “River of News” view
• Atom 1.0 support
• Auto-discovery of feeds (can enter http://feedlounge.com/blog/ instead of http://feedlounge.com/blog/feed/)
• Revamped tagging interface for easier addition and removal of tags
• Revamped item header display (includes author information if available)
• Enclosures are shown as download links in the item content area
• Links to subscribe to comment feeds for items that offer them
• Creation of the feeds list on the server - speeds up initial load
• A new read/unread icon
• Ability to edit feed tags
• Ability to delete tags in the feeds list
• Ability to set the color of a tag
• Back end optimizations
• The page title now prepends “Loading…” when busy, so you can be in another tab and see when it is done loading

Also, I really need to update the screenshots here at some point.

Back when we released alpha 3, I had this to say about our development process:

…our plan moving forward is to push out more feature focused releases on a faster timeframe. Having more bite size releases rather than cooking up a whole meal for each release should help us get specific testing and feedback on these features from our alpha users faster, and should make the releases easier for us to manage.

As you might expect, alpha 5 is (instead of an incremental feature release) a massive rewrite of major parts of the system. Sure, we’re adding a bunch of new features, but we’ve also made some fundamental changes.

Some of these changes are part of our initial plans to helps scalability, some are streamlining the code for easier maintenence, and some are due to things we’ve learned about browser JavaScript performance.

We are planning to push Alpha 5 to the alpha servers soon - it’s almost ready! We’re quite excited about the new features we are rolling out over the next few releases, and quite optimistic about our scalability plans. Right now, it looks like we’ll have another handful of alpha releases where we make a few more major changes/additions, then we’ll be working towards our beta release.

One of the changes I’d like to include in the next alpha release of FeedLounge is a better read/unread icon. Right now we use a page with writing on it or blank.

This is pretty weak in several ways, mainly:

• It looks like a checkbox in Windows XP (depending on your theme)
• Its meaning isn’t immediately recognizable

My first version of the icon was an eye - open or closed. The meaning may have been a little better, but Scott thought it looked too freaky. Here is one of the early drafts of it - he’s probably right.

E-mail programs generally use open and closed envelopes to indicate read/unread status, but there isn’t an equivilent “wrapper” for a feed item. Perhaps we should use a rolled/unrolled newspaper or something like that?

Maybe you folks have some ideas for a great read/unread icon we could use? Feel free to post your ideas and links to your icon designs (16×16 pixels please) in the comments.

A few people e-mailed us or blogged¹ their extreme displeasure at our “Firefox-only-religious-zealot-Microsoft-hating-open-source-crazies” browser policy for the alpha.

We tried to clear this up once already², but we seem to have done a poor job of it.

So here it is one more time - I hope this is more clear:

1. When developing a web application, supporting only a single browser makes it much easier to test and support your application.
2. From experience, we know that it is easier to make a web application that already works in a Gecko browser (like Firefox) work in IE or Safari than vice versa.
3. Firefox has some great development tools (the Web Developer extension, JavaScript console, Venkman, etc.) which save us time and make development easier.

Once we have our core functionality working in Firefox, we will take the time to bridge whatever delta may exist between our Firefox tested code and what is needed to work on additional browsers.

Thanks for understanding.

1. Not picking on anyone, just a recent example. [back]
2. Twice actually. [back]

Since FeedLounge is using so many modern web application techniques to deliver the user experience, we occasionally run across issues that look suspiciouly like browser bugs. We want to make sure these get reported and fixed (either in our code or in the browser’s code), so we’d like to invite a developer or two from all browser teams to join our alpha test.

We feel that this will be a mutually beneficial arrangement:

1. We push the limits of the browser and potentially find bugs that don’t normally get surfaced, report them to you (the browser developer) and provide test and use cases.
2. Bugs that affect FeedLounge get fixed (or we get work-arounds for them) which is a great thing for our users.

If you are a browser developer and are a feed reading fan or are just curious to see what we’ve got going on here, please get in touch - we’d love to work with you.

Right now we use AJAX to store each user’s preferences (current feed/tag/item, which tags are open in the feeds list, etc.) on the server, maintaining a single consistent experience across any browsers and computers someone is using. There are a lot of benefits to this, it’s often great to be able to pick up right where you left off.

However, we’ve also considered taking a different route: storing preferences (for things liks open/closed feed tags, selected item, etc.) in a browser cookie. This would allow you to have a “work” profile and a “home” profile of sorts.

How do you read? Which would you prefer? We want your feedback…

One of our #1 goals for the alpha release was to guage how much hardware we needed to have in place for our beta release. Over the last few weeks, we feel like we’ve gotten the information we were looking for; and it’s a fairly sizeable bit of infrastructure.

We’ve got a lot of things in motion right now:

• We’re in the process of setting up an interim dedicated server for FeedLounge so that we can bring on a few more alpha users and get some final performance metrics without the other box activity cluttering up the numbers.
• We’re finishing our business incorporation, setting up bank accounts, etc. etc.
• We’re getting ready to purchase a number of servers which will all need to be configured.
• We’re deciding on a datacenter to put our hardware into and getting that set up.
• Oh yeah, and we’ve got a bunch more software development on the list too.

While we’re committed to making a free version of FeedLounge available, we’re also committed to making sure the service can sustain itself. We have a few creative ideas in mind regarding revenue that we’ll be talking about more as we get closer to a beta launch. Luckily, there are people out there who are willing to pay for online services (well, at least one).

Getting a service like this ready to go is expensive, both in time and money. Unfortunately, there aren’t shortcuts for either.

While we know that many of you are eagerly awaiting the beta release; we also know that a premature release won’t be good for anyone. Your patience is greatly appreciated.