Friday, February 3, 2012

Yelp sucks: By the numbers

Honestly, most of the time I'm not obsessing about Yelp, but part of my daily duties is to monitor social media about MTB, of which Yelp is a big part. I had no intention of writing about Yelp today, but we just had yet another REAL 5-star review from a REAL person go "behind the veil," and it inspired me to run a report.

We've been tracking the lifespan of reviews for about a month now. We have about 200 other reviews yet to archive, but this snapshot is particularly interesting because it came right after we canceled our advertising contract with Yelp.

Summary of the data:
  • With one notable exception, none of our 5-star reviews stayed visible longer than five days.
  • The one that did stay up for 29 days was taken down three days after I cancelled the contract.
  • The average number of days 5-star reviews stayed up WITH the one outlier is 5.3 days.
  • The average number of days 5-star reviews stayed up WITHOUT the one outlier is 3.5 days.

Granted, this is a very small slice of data, but it's interesting when analyzed in light of our cancellation. I can't wait to dig in to the numbers when we can figure out how to track down the archive dates for our 200-some other archived reviews.

First pic: the cancellation email. I thought it was pretty friendly and to the point -- nothing antagonistic and straightforward. I wrote our Yelp rep on December 29th, 2011, and told her we wouldn't be renewing our contract. Since then, all but one positive review has remained, and that's only because, as of this writing, there's one recent five-star review since we declined to renew. Given the pattern, I expect it to be gone in the next day or two.

Thanks, buh-bye...

Now, here are the numbers. I plugged in all the reviews along with other information and ran a report.

Almost all of our 5-star reviews were archived within a week of being posted. The one exception is particularly interesting -- Justin S.'s review was "disappeared" three days after I canceled the contract. We only have somewhat-educated guesses as to why his review survived as long as it had -- other, more active reviewers' posts have disappeared more quickly, and about-as-active reviewers' posts have stayed longer -- but it seems just a little TOO coincidental that his review would be taken down right after I cancelled the contract. And since he posted his review, no other 5-star reviews lasted longer than five days.

Yelp is going to have to do some 'splainin' sometime soon. Even if their algorithm works correctly and they DON'T futz with the numbers or review placement, it APPEARS to be very shady. The backlash is growing, and I'm not the only business owner paying very close attention to the numbers. My guess is that they don't want anyone probing their servers too closely, even if everything's legitimate.


Anonymous said...

Without naming companies I have watched another company in the industry lose about 5 reviews to the yelp filter over the past couple weeks and actually fall behind a competitor in terms of the number of reviews. The company they fell behind has over 100 more total reviews if you count the filtered which makes it somewhat senseless that they ever wound up with more reviews than them in the first place but i actually know what it feels like emotionally and business-wise to fall behind a competitor due to the whims of the yelp algorithms.

If you really want to get upset try searching the following terms on yelp using washington dc as the location and see what comes up.

"movers" "house movers" "furniture movers" "apartment movers" "moving"

I'm not going to sit here and name companies but there is only one company that shows up in the top 5 using these search terms that has any right to be there based on the number of reviews they have. There is at least one company that has a perfect 5 star rating and more reviews than the others that doesn't make the top 5 using any of these search terms. You should be particularly upset given the fact that you don't appear in the top ten on any of these searches despite the fact you have 4.5 stars and more non-filtered reviews than anyone else in the industry. There are a couple companies that show up in slots 6-10 in the search results that belong on the page but should actually be rated much higher based on # of reviews and stars but aren't

It isn't until I tried "apartment moving" that MTB cracked the top ten, coming in at #6. The top company is has zero reviews, #2 is reputable but has fewer stars and reviews than #5, #3 only makes sense alphabetically but otherwise shouldn't be above #7 by any other rational criteria and #4 is an apartment building.

Now here is where this gets potentially sinister. Using the "apartment moving" search again company # obviously fits as the first company alphabetically but has no reviews.

The second company definitely belongs in the top 5 but shouldn't be # 2 amongst movers by any rational measure, including alphabetically, however they happen to advertise on yelp. Perhaps that is the criteria that makes a company pop up first in the rankings.

At any rate, if you feel like ruining your blood pressure I recommend you continue playing with search terms involving moving to see what comes up while contemplating whether your company ranking based on those search terms would change if you went back to advertising with them.

All I know is that yelp is a destructive force that takes food of the plates of good people who work hard without any apparent rhyme or reason and zero accountability.

Mark D Worthen PsyD said...

I agree with you. I just posted my Yelp experience on my blog--from a consumer's perspective. I wrote 10 thoughtful, honest reviews about a wide variety of businesses and *all* of them have been filtered.

No more Yelp for me!

Christopher said...

Mark -- glad to know this post is "findable!" Yeah, they're going to have to open up a bit or fix what I believe is at least a broken algorithm. The only thing I hate more than Yelp is that I still need Yelp...