Can you trust reviews on Amazon?

Yahoo Premium Partners

On Amazon, the world’s largest retailer, user-generated product ratings matter. A lot. Consumers make billions of dollars of purchasing decisions that are influenced by the ratings other consumers give to products.

But on Amazon, it can appear that everyone likes everything.

We looked at 360,000 user ratings across 488 products in various categories [1], and found what is clearly an unrealistic proportion of 5-star ratings. Not all consumer products can possibly be this good:

Looking at 360,000 user ratings across 488 products shows a surprising proportion of 5-star reviews.

Other consumer reviews sites have more varied and realistic distributions of user scores. The closest site to Amazon in its distribution of overwhelmingly positive reviews is probably the ticket-selling site Fandango, where a FiveThirtyEight story from October 2015 shows a proportion of 4.5- and 5-star reviews of about 50%. That’s high, but it’s still less than Amazon, where our data shows that 66.3% of user-generated ratings are 5-star (Amazon doesn’t allow users to award half-star gradations, although it does report the weighted average ratings in fractions).

Amazon’s lopsided rating curve compared to other prominent consumer-generated review systems: Yelp, IMDB, Expedia, Rotten Tomatoes (user reviews), TripAdvisor, Fandango. [2]

How Amazon Gets Inflated Reviews

When a new product is released on Amazon, it has no user reviews. This creates a chicken-and egg-problem for Amazon and the product’s manufacturers. Users will generally not buy a product when it doesn’t have reviews, and if users aren’t buying the product, there are no reviews being written.

So product manufacturers pay people to review products, either when a product is new or is just languishing in obscurity and in need of a boost. This practice generates a lot of reviews and kick-starts the buying-reviewing cycle that’s necessary for a product to succeed on Amazon.

The pay that people get for these initial reviews is not direct cash money. Instead, manufacturers and marketers pay by giving reviewers their products for free or at a discount. They have to pay people this way, because Amazon disallows people from demanding cash payment for reviews. The company is in fact suing more than 1,000 individuals who did just this. Tom Cook, an Amazon spokesperson, confirmed with BestReviews that Amazon, in general, “does not allow compensation or incentive for reviews,” with the exception being, “when sellers provide a free copy of the product, in advance, in exchange for an unbiased review.”

Existing within Amazon’s rules, there is a robust market that materially rewards people for writing product reviews. Services like HonestFew and Snagshout coordinate the distribution of products to reviewers. Amazon even has its own program, Amazon Vine, to distribute early versions of products to top reviewers. There are also more informal groups, mostly organized on Facebook, that distribute sample products and coupons on behalf of manufacturers. We’ll look more deeply into these networks in a future story.

Regardless of how a review comes to Amazon, if it’s not for a product that was paid for by a regular shopper at a typical price, Amazon demands transparency for how it was acquired. These reviews generally contain text with the words “honest review” in them, and Amazon tells us that the company “removes reviews that don’t include the required disclosure.”

When you see “honest review” in a user rating, it almost always means the product was provided to the writer at a deep discount, or for free, with the proviso that the recipient write a review on Amazon.

Likewise, Amazon also flags reviews written by people who have bought the product “organically,” that is, on Amazon, at a typical price, and without an inducement.

Reviews only carry the “Verified Purchase” flag if the person writing the review bought the product through Amazon itself.

For the time being, some reviews can be both “honest” and “verified.” This devalues Amazon’s “verified” tag markedly. Users of the Snagshot system, for example, can “snag” promotional deals for deeply discounted products while they are logged into their Amazon account. We found very few of these reviews in in our research, fortunately. Amazon told us, “We are currently in the process of removing the AVP badge from products that were purchased at a deep discount, so customers can trust any review with the AVP badge was made by a customer who purchase the product… at a price reflective of a typical purchase price.”

Regardless of how they are flagged, are these sort-of-paid-for reviews reliable? Can they ever be? While all the companies running promotional programs that put products in the hands of users say that they do not require positive reviews, it follows logically that some of their reviewers may be more inclined to leave positive reviews than people who put up their own money for products. Gifts are sweeter than purchases. Our data shows that clearly: People who paid normal price for products with their own money are more critical.

Additionally, induced reviewers are may not have the same service-based mindset as “organic” reviewers, who may be motivated to write reviews specifically to warn people away from products. A paid reviewer, sitting at home with a bad product, may simply return the product, without the emotional connection to the bad experience that someone who put up their own money would have.

Amazon itself clearly understands the difference between paid and organic reviews, or it would not put so many restrictions on reviews based on induced purchases or freebies.

Yet Amazon tacitly encourages programs that improve the number of reviews for new products, and for a solid reason. Tom Cook of Amazon told us, “We believe that all reviews, positive and negative, help customers make informed purchasing decisions. The fact that customers received the product at a deep discount or for free does not preclude them from having an opinion on the product that can be helpful to other customers. Customers indicate that the content of many of these reviews are incredibly helpful. These reviews often provide additional factual information, videos and photos of the actual product in use, and the reviewers often answer follow-up questions.“

Sill, Amazon recognizes that not all reviews are equally valuable. The company wants to reflect the higher value of “organic” reviews. Cook says the company is working to improve how reviews are displayed. As he said, “We continue to work on additional improvements to help the customer. For instance, we are currently working on methods to improve the relevancy of review rankings to ensure that reviews that could most help shoppers are seen first. We hope to roll out these changes shortly.”

The Paid Review Bias

And not a moment too soon. Our analysis of 360,000 Amazon reviews on hundreds of products confirmed our hypothesis that “paid” reviews are disproportionately positive, and also much less likely to get the lowest possible rating.

Paid reviews are more likely to get 5-star ratings from Amazon users. Unpaid reviews are more likely to get 1-star ratings. (We determined if a review was paid by looking for the words “honest review” in the writeup, which is what the product distribution agencies require. This is not a perfect filter, but the results are stark enough that we believe it tells the story accurately.)

Products that are new on Amazon show these trends in the extreme. From February 7 to 9, 2016, we analyzed all the products in the new releases section on Amazon. Nearly three-quarters of the reviews for these products were marked as paid reviews, showing that the reviews jump-starting program was well underway for them.

These new products also had very positive reviews.

As we were working on this story, Amazon reached out to us to reinforce its view that many early reviews, even when they are paid or induced, can provide valuable information for customers. These early reviews can include videos and photos of products, for example.

We agree with Amazon on this. Paid reviews can be very useful. Yet we still feel that it is critical that reviews that are paid-for (through giveaways or discounts) are clearly labeled as such. The reason is in the data: User ratings from paid reviews vs. “organic” reviews are not comparable. Paid-for reviews score higher.

This chart represents 10,000 paid reviews for new products from Amazon’s “Hot New Releases” page. We consider a “positive” review one with a 4- or 5-star rating.

Amazon Is Making an Effort to Downplay Paid Reviews

Amazon, clearly aware that not all reviews scores are created equal, attempts to correct the overall score each product receives using an algorithm. That is, the reported star rating is not just a straight average of all the reviews a product has. It’s calculated to weight more reliable reviews more heavily. The word from Amazon is, “We are substantially down-weighting the star rating from these [paid] reviews. Last year we updated our algorithm for the star rating so that older reviews and reviews that came from free or heavily discounted products are weighted less in the star rating.”

However, the company doesn’t disclose more about how the number is calculated than saying this:

Amazon’s effort to factor in ratings is laudable. It should be a better method for judging a product’s quality than simply looking at the number of 5-star reviews a product has in comparison to lower-scoring reviews. However, we found only a 1 to 3% difference in Amazon-computed ratings compared to the purely mathematical averages we calculated from the ratings distribution on Amazon reviews.

The lack of complete transparency about how this score is calculated means that buyers should still look at user reviews scores on Amazon with a wary eye.

Conclusion: Buyer, Be Aware

Amazon’s review system is a valuable resource. However, it’s been gamed by market forces, especially on newer products. The incentives in place to increase the number of reviewers of new products demonstrably drives them to create inflated evaluations. Amazon has attempted to reduce the impact of paid reviews, but not yet to great effect.

Fortunately, Amazon itself allows buyers to filter out paid reviews. Our contact at Amazon reminds us, “Customers have the ability to filter reviews based on the Amazon Verified Purchase (AVP) if they’d like to only read reviews from customers who purchased the item on Amazon at a price that is reflective of the price that they will be paying.” (To access this feature, go to the dedicated Customer Reviews page for a product, scroll down past the Top Positive and Top Critical review, and click on the “Filter By” drop-down list.)

We recommend that buyers take all reviews into account when making a purchase decision, but that they pay much more attention to Amazon Verified Purchases than any other reviews. Our analysis shows that the paid, so-called “honest” reviews you see may be the least honest there are.

BestReviews is committed to providing comprehensive and trusted reviews for products that matter to consumers. We do the research to help you save time and money. See some of our latest product comparisons: Air Mattresses, Home Theater Projectors, and Electric Razors.

Notes:

1: Product categories included in our dataset of 360,000 user reviews:

Home, Office Products, Beauty, Health & Personal Care, Kitchen & Housewares, Outdoors, Toys & Games, Grocery & Gourmet Food, Automotive, Computers, Tools & Hardware, Shoes, Sports & Outdoors, Apparel & Accessories, Camera, Mobile Electronics, Baby, Tools, Books, Kindle Tablet accessories, Pet Supplies, Video Games, Electronics, Home & Garden, Cell Phones & Accessories, Industrial & Scientific, Musical Instruments, Music, DVD, Kindle Tablet devices, Luxury, Jewelry, Pantry, Miscellaneous, Handmade, Watches, Software

2: Sources for chart, “User Rating Distributons at Different Sites:”