Company blog

Usability testing – a sobering experience

Based on all the positive feedback we received about our online quoting tool, we thought that we had done a pretty good job! But one thing we have learned over the years is that it’s always wise to test things out. So, we recently subscribed to so that we could have a few people do a usability test of our site. We’d actually been urged to do this over a year ago, by friend and usability guru Paul Hibbits. Sadly, it took us longer to do it because we thought we had more important things to do than watch people use our site – boy were we wrong!

For those of you not familiar with what a usability test is, the concept is that you get a bunch of people that fit your target user profile (but haven’t used your website before). You give them a goal, then sit back and see how they do. Ideally, you want them to talk about what they’re thinking every step of the way. That way you can see exactly where things are going off the rails. And in our case, did they go off the rails quickly! :-)

We opted to get 3 different people to each perform a test that takes at most 15 minutes. These users aren’t full-time “professional” testers, but rather average people who are willing to do a test from time-to–time. The users earn a small fee for each test they take. Here’s an example of one of the usability tests done for our site (hopefully you don’t find it as cringe-worthy as we did):

You can choose the characteristics of the testers you want. As the testers proceed, a video of their screen is recorded along with a running audio commentary of what they’re thinking. Pretty simple in concept, and you wouldn’t think it would result in any earth-shattering revelations. From reviewing the results of just those 3 tests, we came up with over 30 changes we wanted to make to our site. Some examples of the changes we’ve recently made:

  • We rewrote the copy for every sentence in the quoting flow, trying to be more succinct and clear.
  • Error and help messages have been moved in-line with the questions, and out of their pretty pop up “bubbles” on the right side of the screen. We thought they looked cool, but users weren’t noticing them. The old approach was frustrating users.
  • We’ve switched from using Google’s autocomplete address entry tool, to a new one offered by Canada Post. We’re now paying 7 cents per lookup (whereas Google’s is free), but Canada Post’s works better for entering residential addresses, and supports apartment unit numbers. We think 7 cents is a small price to pay to make it easier for customers to get a quote.
  • For questions that users were having difficulty answering, we’ve tried our best to find another way of asking that question, recognizing that we still needed to collect that information. Examples of this are asking for the number of bedrooms, instead of square footage, if you live in an apartment, which allows us to come up with a close enough estimate of the size of the home. Another example is that we now ask for how many cars fit in your garage, instead of its square footage.
  • If you’ve answered something incorrectly, we’re now performing real-time validation so you don’t have to submit the form before being notified of an error.

Now that we’ve gone live with the “fixes” from the last round of usability testing, we plan to go through the exercise again. Previously, I wouldn’t have thought this next round would come up with anything much, but now I’m not nearly so confident.

Do you have any feedback on our site? While it’s nice to regularly hear praise, we’re actually surprised we don’t hear more gripes. Clearly, there are gripes, based on those testing results. So this is your invitation – if you find anything confusing or frustrating on our site, please send me an email or call me at 1.855.331.6933 ext 103.

Send an email to Jason


  1. Darius says:

    Hi Jason,
    congratulations on your new office and your online quoting system; I like the system a lot and the way it evolves;
    for the CP look-up suggest to limit the number of clicks per IP (per day or per h) to avoid somebody inflating your costs; 7 cents adds up quite quickly;

    good luck

  2. Jason Vander Zalm says:

    Hi Darius,

    Thanks for the feedback. While we’ve restricted our Canada Post API key so that it can only be used on our site, at this point we haven’t setup any usage restrictions beyond that. I’m not really sure why anyone would choose to do address lookups outside of a regular usage scenario, and I hate to make a regular usage scenario more difficult for anyone by restricting their ability to do a lookup. The other challenge is that we’ve also implemented a version of the lookup tool on the system that our agents use, so obviously we’d need to bypass the restriction for them. Canada Post does allow IP address filters, but our staff work from home from time to time, and I don’t really want the headache of having to keep track of all their IP addresses ;-)

    Unfortunately the reporting tools that Canada Post provides are very limited, so I can’t actually see if any one IP address has been doing an inordinate number of lookups.

Leave a Reply