The truth about Amazon reviewsCan you always trust online user reviews?

19 October 2015

Amazon-five-stars rating

How useful are online reviews if almost everything gets rated five stars?

Did you know that when you look at the overall star rating for a product on Amazon, it isn’t always the average of all the ratings left by reviewers? And that a surprisingly high number of reviews are given four or five stars?

Which? analysed 3.6 million Amazon reviews to find out the truth behind the star ratings. Our investigation found that there may be some things about Amazon’s user reviews that you didn’t realise. 

Read on to find out what we discovered, and how you can analyse online reviews yourself to help you pick the best products and avoid the worst.

Want to know where to shop online? Our survey reveals the UK's best and worst online shops.

The average isn't the average

When you’re looking at online reviews to help you buy a product, it’s really useful to save time and narrow down your choice by looking at the overall headline rating. 

When you do this, you probably assume it is the average of all the ratings left by reviewers. While most online sites will simply calculate the average (mean average of all the ratings), Amazon doesn’t do this any more. 

Amazon changed the way it manages reviews in June. A product’s overall score is now based on an algorithm that considers several factors. This includes the age of a review, the number of helpful votes received and whether the reviews are from verified purchasers. 

This means that some reviews count more towards the overall score than others. 

Amazon reviews star ratings distribution

Almost everything gets five stars

We analysed a huge set of data from reviews left on Amazon.com ('Inferring networks of substitutable and complementary products, J McAuley, R Pandey, J Leskovec; Knowledge Discovery and Data Mining, 2015'; reviews taken from Amazon.com’s Home and Kitchen section, 2011 to July 2014).

We didn’t have access to an equivalent for Amazon.co.uk, but our analysis gives us an insight into how reviews on Amazon work. 

We found that over the time period we focused on, more than three quarters of the Amazon.com reviews we looked at gave five-star (60%) or four-star (17%) ratings (as the chart opposite shows). Hardly anyone ever leaves two- or three-star ratings (6% and 8% respectively).

However, as we know from our years of testing, there are some terrible products out there, as well as plenty of mediocre ones. So we’d expect to see more of a range of scores. And if everything scores well, just how useful are online reviews?

Freebies for reviews

We've also found that some reviewers get products for free from sellers (often on Amazon Marketplace) in exchange for writing a review. 

Whilst Amazon asks for these freebies to be disclosed, we also found instances when reviews were classified as a 'verified purchase'. This was even though the reviewer had said they'd been provided with a free sample for review. Remember, when a review is classified as a verified purchase, it counts more towards the overall score. Our analysis also found that Amazon Top Reviewers - most of whom get free products to review - tended to rate products higher than other reviewers.

We got these finding by looking at a huge set of data from reviews left on Amazon.com (Inferring networks of substitutable and complementary products, J McAuley, R Pandey, J Leskovec; Knowledge Discovery and Data Mining, 2015; Reviews taken from Amazon.com’s Home and Kitchen section, 2011 to July 2014). We didn’t have access to an equivalent for Amazon.co.uk, but our analysis gives us an insight into how reviews on Amazon work.

Despite the fact that we’re lifting the lid on Amazon’s customer reviews, we’re not telling you to avoid it. Quite the opposite – Amazon scores very well in our survey of the best and worst online shops. But we think the site could be clearer about how it calculates overall scores, and could do more to ensure reviewers declare whether they’ve reviewed free products.

Can you trust online reviews?

Website user reviews can provide helpful details about a product. A user might be able to tell you that the coffee machine you’re considering isn’t exactly the shade of grey shown in the pictures. 

But it’s important that you look at more than one person’s experience, and user reviews alone are unlikely to give you the full picture. Users normally don’t compare several models and can’t test products to the extent that Which? does. 

So use our test scores and expert reviews to help you narrow down your selection, then read user reviews to add to the overall picture. And take the time to post reviews yourself – the more accurate reviews there are, the better it is for everyone.

More on this...

Cookies at Which? We use cookies to help improve our sites. If you continue, we'll assume that you're happy to accept our cookies. Find out more about cookies