Revisionist HistoryJuly 1, 2004
What is the easiest way to determine whether or not you can trust a product review? Does it begin with a description of how terrible the previous version was? You find this time after time within technology reviews. Here's a sample:
At last, Cool Product 2.0 is here! As those of you who suffered through Cool Product 1.0 know, the previous version was nearly unusable. Cool Product 1.0 was incredibly slow, difficult to master, tended to crash unexpectantly, and was so frustrating to use most simply gave up. The good news is that Cool Company has addressed these many issues in the latest version, Cool Product 2.0. It is faster, smarter, more reliable and has many new features. Highly recommended!
So far, so good. The problem is discovered when you happen to come across the review of Cool Product 1.0. More often than anyone would care to admit, the original review was full of praise, perhaps emphasizing that the product breaks new ground or takes a competitor's product one step further. There may be a brief hesitation mentioned, but the overall tone will be enthusiastically positive and forward looking. In other words, the only time you hear the truth about a product is when it's replacement has arrived.
So, how did we go from Cool Product 1.0 is the Best to Cool Product 1.0 was a Bust? Why has each version of Apples OS X been celebrated as a great achievement and major advance, and yet, we now hear that not until 10.1 or 10.2 or (amazingly) 10.3 was the operating system stable and actually usable? What are you not telling me today about this camera/software/stereo/automobile that you will be telling me 12 months from now when the next version is released?
I fully understand that I'm painting with a broad brush, and many publications do a fine job of producing honest, hard-hitting reviews. I also understand that much of this is driven, unfortunately, by the demands of advertising.
Here are my two proposals for cleaning up this process so we, the audience, can trust our sources.
First, every significant product should receive a three month follow-up review (for example, Living with 10.3: Our Three Months with Panther). I will grant that initial reviews are difficult to write due to the drive to be first to print and the initial cool factor of any new product. We ourselves are guilty of this. How often do we purchase a product (iPod, headphones, CD, book, bike, shoes) and absolutely rave about it to friends and family. Then, about three months later, we're asked, "Why don't you use/wear/read that cool product anymore?" And the answer is, "Well, I loved it at first, but after awhile I discovered that...the lyrics were hateful/the ending was horrible/it fell apart in the wash/the batteries don't last." Fair enough, we all live and learn. Why can't our industry sources do the same? A great model for this is Edmunds.com where you'll find First Look and Long Term Road Test reviews of most cars.
Second, a review of a new version of a previously existing product should include quotes and ratings from, and links to, previous reviews. A publication must be willing to admit mistakes and talk about what was learned from them.
In our day and age, we are each faced with unending choices and options, both personally and professionally. More than ever, we seek out advice from those who know. If we grow to trust our sources, they earn unbelievable loyalty and goodwill. Sometimes the source is a national publication, sometimes it is a friend, and more and more, it is a dedicated expert on his or her weblog. We'll keep searching for that trusted voice no matter where we find it. The problem for established publications is that there is an amazing equality within Google search results, where a link to Macworld is indistinguishable from a link to Daring Fireball.