Unscheduled Downtime 2005-01-27

FeedLounge was offline today from approximately 5 PM PST to 9 PM PST due to the database server becoming non-responsive to write operations.

We brought down the database server as gently as we could under the circumstances, and had the problem cleared by 7 PM PST. Once the problem was cleared, we took the time to investigate all of the server logs in some depth to make sure there wasn’t an underlying hardware problem that would require us to move the main database to one of our backup machines.

Everything checked out fine, and the Queue was brought back online to begin catching up just after 8 PM PST. The FeedLounge web interface was brought back online around 9 PM PST.

This was the first database problem we’ve had since we moved to PostgreSQL and we’re working to make sure we can recover more quickly from this kind of problem in the future.