StandoutJobs is live: tales of a harrowing launch

We lived through crunch mode and the site is live. Since we’re no longer in stealth mode,  I can now tell people what I do. Well, after I kvetch a bit.
We found out the issue we were having had been reported 4 days ago as Bug #10926. We have only managed to replicate it in our production environment, but not in staging. The consequence for us was a 404 (page not found) error when logged in users tried applying for a job. While we did work around it, let’s say that error pretty much sucks for an app that’s supposed to help companies hire people.

It was a horrible type of bug to track down. The only difference we could see? More Mongrel instances, and a 64-bit version of the OS.

• Is it in Rails’ incredibly complex routing code?
• Our own subdomain support?
• Is Capistrano doing a clean restart and getting all the Mongrels running the same version?
• Is caching an issue?

Unable to replicate the bug on staging and given it only appears randomly (1/3 to 1/2 of the refreshes, both hard and soft),  it’s the kind of experience you want to avoid on launch day.Anyhow, the site is live and the inflow of bugs has slowed. We’re still madly fixing any issues as they get reported, and we’ll have refinements over the next few days before we tackle new functionality.

Overall this is a great success. I’m happy to be working with this ‘A team’, and will have more stuff to share as soon as things calm down.