Entries Tagged 'tech' ↓

I took a week vacation from Standoutjobs, which of course means spending some time hacking on personal projects.

One task was to scrape the SAQ site to get the entire database of wines. Downloading with Ruby and wget / curl is child’s play. Building a scraper to retrieve data from those files should be just as easy - see e.g. Ruby Screen-Scraper in 60 Seconds. Basically, copy the XPath for the data you want from firebug, and paste inside your script - and you’re pretty much done.

This is where I hit gotcha #1. Firebug uses Firefox’s normalized html, which results in an extra tbody in the XPath, even if it’s not there in the HTML received by curl. “Oh, no problem, I’ll just remove the tbody from the Xpath”. That was wishful thinking: sometimes nested tables on saq.com have a tbody on only one of the tables, so you really have to check.

So now I’m happily going through the 7195 files that were downloaded. Some throw nil exceptions for elements that aren’t present. I fix the script, re-run the import. Rinse, repeat.

Until of course, I hit gotcha #2. Unlike missing data or XPath issues, I get this befuddling error:

TypeError: can't convert nil into String from /usr/local/lib/ruby/gems/1.8/gems/hpricot-0.6/lib/hpricot/parse.rb:51:in `scan'

A bit of google-fu reveals it is a known bug: Hpricot can’t handle some files that are multiples of 16384 bytes. The fix is easy once you know:

echo ' ' >> my_specially_sized_file

Through the process I added links to del.icio.us with ‘gotcha’ and ‘hpricot’ as keywords, and noticed others had done the same thing for other projects. That could be a very handy resource when starting a project with a new set of tools.

This thread is worth following. In the past couple days:

-Jamais Cascio posted an interesting essay on “The Big Picture“, looking at the important drivers for the next 20 years.

-Prag Programmers published “Augmented Reality: A Practical Guide” (See O’Reilly’s post)

-Schneier posts about using cell phones to detect nuclear radiation. Those in power will want this to detect terrorists, but it will also shine a light on the transport of nuclear fuel and waste.

The futurists are theorizing, the alpha geeks are playing and writing software. It won’t be long until we get a political crisis because of data that citizens weren’t supposed to collect.

We lived through crunch mode and the site is live. Since we’re no longer in stealth mode,  I can now tell people what I do. Well, after I kvetch a bit.
We found out the issue we were having had been reported 4 days ago as Bug #10926. We have only managed to replicate it in our production environment, but not in staging. The consequence for us was a 404 (page not found) error when logged in users tried applying for a job. While we did work around it, let’s say that error pretty much sucks for an app that’s supposed to help companies hire people.

It was a horrible type of bug to track down. The only difference we could see? More Mongrel instances, and a 64-bit version of the OS.

• Is it in Rails’ incredibly complex routing code?
• Our own subdomain support?
• Is Capistrano doing a clean restart and getting all the Mongrels running the same version?
• Is caching an issue?

Unable to replicate the bug on staging and given it only appears randomly (1/3 to 1/2 of the refreshes, both hard and soft),  it’s the kind of experience you want to avoid on launch day.Anyhow, the site is live and the inflow of bugs has slowed. We’re still madly fixing any issues as they get reported, and we’ll have refinements over the next few days before we tackle new functionality.

Overall this is a great success. I’m happy to be working with this ‘A team’, and will have more stuff to share as soon as things calm down.

I was told Bell sucks so much that I have to go with Videotron. After I ordered a package online, they said they would call within 24 hours.

A call came from Synergy, one of their subcontractors known to call with dead air and hang up. I called back, gave some info to a person who could not speak clearly. They put me on hold 4 minutes, and disconnected me. The next person also disconnected me, after a total of 20 minutes spent on the phone.

Whiskey Tango Foxtrot. Before agreeing to pay $136 deposits, I’d like to know when a company will deign connecting their service. Is that really too much to ask?

Recommendations of ISPs that don’t suck - or just suck less - would be heartily appreciated.

Should I buy a Meraki Mini, sign up with FON, or is there a more effective way to speed up free, ubiquitous wifi?

Mob Rules inspired me. If you haven’t yet, I recommend doing so. In that article, Mark Pesce mentions the Meraki Mini, and how that could be game-changing for telcos.

There’s a lot to like about the Minis. They’re not completely open-source, but they could run CUWiN software and are cheap. FON seems to have better backing (Sequoia, Google, Skype), but their marketing bugs me (Is that a for-profit trying too hard to be hip, calling itself a ‘movimiento’?).

My understanding is that Ile-sans-Fil isn’t trying to blanket the city with free wireless. Someone please correct me if I have that wrong. While I like what they are doing, I’m attracted to the idea of free wireless everywhere.

We don’t know who will use these technologies, or how. Pesce uses the example of Kerala fishermen, and there’s the example of homeless people using a phone number to improve their condition. The applications I have in mind for ubiquitous wireless - location-based services and augmented reality - might not be the ‘killer-app’. If there are any social aplications for these, I think there’s a good chance they’ll be built in a city like Montreal.

The thread on Montreal Tech Watch’s Capazoo Update got increasingly surreal this week, first with their Director of Communications heralding their strategic partnership with the National Lampoon. To contact him, anyone in the press should just do so through the service he is promoting (!).

Capazoo is a multi-level marketing scheme. Like Amway and countless other MLM schemes, it turns people into money-grubbing zombies that erode our social capital.

Today one Jean-Christophe decided to comment as a satisfied customer, pleading for us to see for ourselves how fantastic and lucrative Capazoo can be. Naturally, he finishes by giving us his profile page. If we sign up after visiting it, he earns zoops. So far, he’s earned 806 zoops - or $8.06.

So I looked at the referral scheme:

For every friend you invite who upgrades to Privileged Membership, you’ll earn 100 Zoops!

For every friend you invite who upgrades to VIP Membership, you’ll earn 130 Zoops!

The Capazoo.com Referral Program lets you earn Zoops when your friends refer friends – up to four generations!

Ick. Capazoo will quickly fill up with amateur spammers looking to make a single dollar of each of their social connections. Maybe one day they’ll have Amway-style rallies and conferences. Unlike other MLM organizations that manage to stick around for years however, this socio-economic virus should peter out once they burn through their cash reserves and exhaust their possible partnerships.

I hope Capazoo goes out with a bang rather than a whimper, that it serve as a reminder to other idiots that would try similar ideas.

The day Google announces it is creating a new wikipedia-like social network that pays authors, Amazon launches SimpleDB.

This morning I attended the December MTEB.

A couple of venture capitalists were in attendance, who graciously answered my questions about what is going on with the VC industry. I basically think the industry is big, slow, and doesn’t understand new technology. They’re closely watching the efforts of Montreal Startup, but weary several incubators, seed funds and other schemes have foundered before.

The real surprise was talking with Darrell and the engineers at Stanox. We come from different technology backgrounds: micro-chips, ERP and web. Yet we all have similar attitudes: KISS. YAGNI. Think. Simplify. We’re all fans of Saint-Exupery’s quote on perfection:

Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.

When I griped about a horrible API I might have to integrate (the variable names are inconsistent and ugly) one VC said “well, it can be ugly, but it’s ok if you make money”. The dividing line was obvious. For the engineers, if it’s ugly and verbose it’s probably buggy. For the investors, it’s bearable if it’s profitable.

The folks at Stanox look like they will be successful, and mentioned they may become angel investors in the future. I’m hoping they succeed; we need more angels like that.

After solving Word Numbers, maybe I should solve ITA’s other current puzzle, Sling Blade Runner. I wasn’t sure how to solve it, so I made a pretty picture:

ITA provides you a list of over 6,500 movies, and the objective is to make as long a chain as possible of overlapping titles: Sling Blade Runner, License to kill a Mockingbird. Every dot in the graph is a movie, every arrow connecting them indicating their titles overlap. In comments on Reddit, a few people found solutions of around 230 titles.

Puzzles are useful to learn. Since I’m not an engineer, they force me to learn about computer science theory I never learned. Looking up graph problems of the sort, I found out that this class of problem is defined as a ‘hard’, and it is far too computationally intensive to try every possibility (’brute force’). The only way to solve it elegantly is to use shortcuts, and those depend on the type of graph in question - which is why I decided it might be interesting to have an actual picture.

Tools like graphviz are slow for very large graphs, so this picture doesn’t represent the whole graph. Still pretty, no?

I don’t usually worry about performance. Today, when a simple AJAX request sending back an empty body took 2 seconds to complete, I started digging.

Oddly, Firebug indicated over 2000 ms, while my log showed under a half-second. Since it’s on the same machine, there’s no way network issues can account for 1.5 seconds. I tried several avenues, and Marc-André suggested several possibilities before suggesting running it in production mode.

We were both suprised at the difference. Before / after:
Firebug: 1755 ms

Log: Completed in 0.47638 (2 reqs/sec) | Rendering: 0.00010 (0%) | DB: 0.15976 (33%) | 200 OK [http://standoutjobs.dev.com/****]

Firebug: 121 ms

Log: Completed in 0.11076 (9 reqs/sec) | Rendering: 0.00007 (0%) | DB: 0.06037 (54%) | 200 OK [http://standoutjobs.dev.com/****]]

A full order of magnitude! I’m used to logging and other debug overhead to have about a 10% performance hit, so this was very surprising.

If you’re like me, you have some personal web apps running in development mode. My first instinct was always to look at the logs, and see if there are some unnecessary requests being done, or anything that is taking far too much time. There’s also Marc-André’s tips for improving rails app performance. Still, those are a lot of work compared to the simple step of setting that production flag.