A huge grain of salt, or why we can’t trust any Apple Watch reviews — yet

April 14, 2015

It hasn’t even been a week since the Apple Watch embargo was lifted, and in that time we’ve had about a kazillion words written and published about the device. I’ve read a few of them, most notably John Gruber’s excellent take on the philosophy behind the product, Nilay Patel’s flashy review with lots of pictures and videos over at The Verge, and David Pogue’s classic column.

Perhaps unsurprisingly, these and many other reviews are all over the place. While some journalists seem genuinely excited about the device, others are more skeptical and point to several things that could prove to be important shortcomings, like its apparently lackluster performance or the bands, some of which could be slightly awkward and uncomfortable for certain people.

Some reviewers, like the ones I mentioned above, have been around long enough to know that it’s better to be responsible when writing about unreleased products, and have measured their words accordingly. Others, unfortunately, have not been as sensible.

These early reviews have been interesting to read because they gave us the first glimpses of what it’s like to actually use an Apple Watch, as opposed to just playing with a demo unit. However at this point, I don’t think any of them really matter, or have any long-term significance. Whether a review has been positively glowing or downright scorching, there’s simply not enough real context and data for it to offer a reliable and accurate representation of the future of the Apple Watch.

For the moment, the only honest, responsible thing to say about the future of the Apple Watch is this: time will tell.

Photo credit: Ryan Ozawa.

The way gadget reviews work in the tech community is simple: when a new product is announced, several media outlets get review units in advance, so that they can work on their reviews. These reviews can’t be published before a certain, previously stipulated date — hence the embargo, which is usually lifted just hours or days before the product officially becomes available for purchase or preorder.

After that and on day one, all the other tech writers who didn’t get a review unit rush out to buy one so that they can work on their own reviews, which are usually published a few days later.

This well-established process creates two categories of reviews: the early ones, and all the rest. Usually, the biggest knock against early reviews is that they tend to be overly gracious, and sometimes avoid criticizing the product too much, perhaps to ensure their authors will stay in the company’s good graces and will keep receiving review units in the future. This is not necessarily true, but it’s what many people think, and so these authors are routinely accused of being biased. Though I don’t personally believe this to be the case, it never hurts to maintain a healthy dose of skepticism when reading a gadget review.

However, in the case of the Apple Watch, it’s not so much a question of bias — although that’s still present — as it is a question of implausibility. I simply don’t think it’s possible to present a coherent, informed opinion about the Apple Watch based on mere days of use, no matter how intensive that use may have been.1 And that has everything to do with the very nature of the Apple Watch.

With a new version of an existing product category, the criteria by which the product is rated are usually clear and understood by all. If it’s a laptop, you take a look at the combination of hardware design, display, performance, portability and battery life. That’s it. A laptop is a laptop, and we’re familiar enough with them that it’s entirely feasible to review one after a week of regular use.

The Apple Watch, on the other hand, lacks any such criteria by which to be judged. Practically all of its functionality is so radically new that we simply lack a framework for understanding how — and if — it will fit into our lives. Not even Apple knows that yet, no matter how confident they may seem today.

With the Apple Watch, we’re all in uncharted waters, Apple itself included. Not to mention the inherently social aspect of the device: features like digital touch and sending your heartbeat to another person are entirely predicated on said person owning an Apple Watch, too. So how can these reviewers test and review those features? Social habits take time to develop, and intimacy much more so. There’s just no way you can guess today if a year from now people are going to love being remotely touched of if it’s going to be the most annoying feature ever invented.

Apple hopes these features will prove to be compelling enough to be accepted into our lives and become established. They’re betting big on the Apple Watch to become their most personal device yet but the truth is, there are still too many unanswered questions, many of which are not Apple’s to answer.

And then there’s the question of 3rd-party apps, which we still know almost nothing about, and which could extend the functionality of the device in previously unimagined ways, much like the App Store did for the iPhone and iPad. Would the iPhone today be as compelling without the phenomenal iOS app ecosystem backing it? Not even close.

At the end of the day, it doesn’t matter if your favorite reviewer loved or hated the Apple Watch. If you want to find out if it’s the next big thing, you’ll have to wait a little more — or just buy one and find out for yourself. It’s still early days, and the ultimate fate of the product is yet to be decided. Apple has certainly done everything they could to give the Apple Watch the best possible chance at succeeding, but the final word is ours to say.

  1. It’s actually worse, because they only had a few days to use the device and write the reviews.