My thoughts on bias and 'pay-offs'

Well, The Inquirer certainly stirred up a hornets nest with their rant about the online hardware reviewing world - It's spawned threads here and there, and caused the likes of Anand from Anandtech no less to indirectly respond to some of Charlie Demerjian's thoughts in his Blog.

As someone who has reviewed hardware for three sites ranging from small to (albeit briefly) huge, I obviously have a fair few thoughts and some experience in the matter myself, so I figured I may as well put finger to keyboard and say a few words.

First up, Charlie mentions the dreaded NDA, something which, to be honest, nobody who wants to be on the forefront of the industry can go without signing. I've been under a few of these myself, and they can be a frustrating experience. Take ATI's HyperMemory launch, where a date was given for the NDA to expire upon its signature, which was then moved at the very last moment. Or 3DMark05, where the two major IHVs both released new 3DMark05 compliant drivers less than 24 hours before the NDA lifted for the benchmark. It's all about playing the PR game on the side of the companies involved (Although I'm not blaming FutureMark for the 3DMark05 issue, it was all down to ATI and NVIDIA), but at the end of the day it not only makes life difficult for us reviewers, it makes informing our readers as accurately as possible nigh-on impossible too. Carrying on with my 3DMark05 scenario, I had a massive number of benchmark results already recorded, and recreating all of these with two new drver sets would have been an impossible mission even had I been working full-time on said article. Thus, I was left with a dilemma as to whether to publish the article as is with 'old' results, or wait until I had the latest results available for the video cards I was using. In the end, I decided the article was more about 3DMark itself than the results, so I ran with it. Was I right to do so? I'm still not sure.

We witnessed a similar crisis around a number of sites this week with Intel's dual core Pentium Extreme Edition launch, which was pushed forward at the last minute. This left a lot of sites with half-baked, incomplete articles and no time to draw out the full picture either for themselves or their readers. Is that really helping anyone? Intel may feel it was a PR victory to introduce the first dual core parts to the community, but the subsequent rushed articles may well not have done the part the justice and fanfare it deserved. But who is to blame? Intel for moving the NDA expiry date, or the various web sites for rushing to comply with the change and release content early, rather than simply doing so 'when it's done'? Again, that's a question for others to answer, because I honestly can't say what I'd do in the same situation.

Which brings us to the more serious allegations, of bias and 'pay-offs' from vendors to certain sites, be they physical or psychological. I may be naive, but I find it very hard to believe that any online hardware sites are currently accepting money in return for positive reviews - In the past, maybe, but I can't imagine this kind of pratice going on in 2005.

A more 'psychological' form of being paid off however, I can believe. PR people are there to make the likes of us hardware reviewers feel like kings, while at the same time trying to ensure their products get positive press. This heady mix can often lead to journalists unwittingly becoming mouthpieces for a particular company, and end up with them being manipulated by a PR department without them even knowing it. These kinds of 'special relationship' are usually pretty easy to spot, and I'm sure even I don't need to name names for some of you to know the kind of thing I'm referring to. This is the reason why HEXUS tried to keep the vendors and reviewers as far away from one another as possible at all times, with all arrangements and information regarding hardware going through a central point - It stops these kind of relationships from forming, and thus helps to guarantee the impartiality of the individual reviewer. Of course, you can't separate the two entities completely, but the more the link is shortened the less of a problem it can be.

For me personally, the biggest issue in the reviewing community isn't bias, nor is it being 'paid off', but is in fact something which can often lead to accusations of either of these. No, the big problem is simply - Lack of knowledge. Think of the number of times you've read a review with a massive factual error around which its conclusions are based or, even worse, reviews which read in a horrifyingly similar fashion to the products press release. I often hear the 'f' word (fanboy) thrown at such reviews, but it seems to me that a lack of understanding is usually the key. And to be honest, it's perfectly understandable - While working at HEXUS I looked at a massive variety of products, and had to adapt to a lot of new things, and it really isn't easy - Indeed, I remember having to sit down for a good half hour one afternoon talking with another staff member over the basics of... Cooling fans. Not CPUs or video cards.. Cooling fans. If you really want to write great reviews, you need to know everything about everything, and however much you strive for it that status is, quite frankly, impossible.

There are no simple answers to these dilemmas for us as hardware reviewers, but things are much simpler for you readers out there - If you want to read about a particular subject, take in as many different articles from different sites about it as you can, and form your own opinions from what you've read. When you throw everyones ideas into the melting pot, chances are you'll be left with a far more complete picture of what you were reading about. It's why I link to every review I can find on Elite Bastards' front page when a product is launched - It isn't so people can click on the link for their favourite site, it's so they can take their fill of information about the product.