Be wary of Google Analytics case studies

Is it ethical to keep online case studies of clients that are not using your solution anymore? Or to present case studies in a way that could mislead your potential clients?

What I found out

I'm getting closer to the next version of my WASP Firefox extension, and to ease the testing process, I figured I could build a quality assurance test where I would visit client sites referred by each vendor. A list of high-profile clients shown with emphasis on the home page and case studies is a proven way to increase confidence and credibility. But here's what I found out...

Google Analytics case studies

I was in for a surprise! Take the case of Google, which highlights a number of case studies:

• BuildDirect Google Analytics
• Discount Tire: Google Analytics
• RE/MAX: Google Analytics
• Roche: Google Analytics
• Vueling: Google Analytics
• CKE Restaurants, for the Spicy Paris campaign: Google Analytics
• Financial Times: WebTrends live, Google Analytics is used for campaign landing pages
• Agency.com: WebSideStory HBX v1.4... no traces of Google Analytics
• Career Builder: Omniture SiteCatalyst, even CBCampus.com, the site referred in the case study, uses Omniture and as no traces of Google.
  
• Deckers: none identified (might be log based).

Could it be that 4 clients out of of the 10 Google Analytics case studies are not even using it? They might have been in the past, they are not anymore? Saying the Financial Times uses Google Analytics is a sure way to bring credibility, but when digging the case study, we find out GA is used for landing pages only. While the core site, FT.com, relies on WebTrends.

I'm sure Google Analytics is not the only solution provider in this situation. In fact, I've even found a number of vendors that are not using their own solution on their corporate web site...

What about others?

I took a look at a couple of the Omniture Customer Showcase and a random sample of 10 of those sites resulted in 100% Omniture usage (along with some double-tagging with Google Analytics). Same thing for VisualSciences WebSideStory Customers... I can stop there.

My two-cents

1. Can you put a date on the case studies? Being in the web analytics field, you know that what makes a good story is the context. Without any time reference, we loose a valuable contextual element that can easily lead to assumptions and misjudgment.
2. Take their claims with a grain of salt (and pepper). Everybody is a "market leader", and an "industry first" at something. Cut the crap and do your homework: does the solution fit your needs, not those of others!
   
3. As they say at Microsoft: "Eat your own dog found": if a vendor is not even using its own tool (and especially if using Google Analytics!), how can you believe their solution is anything but useless? (This is especially true of smaller players, but even some of the major vendors are using both their own and Google Analytics!).

(shameless plug)
WASP can help you detect which web analytics solution is being used by top sites and competitors. And it can help you insure the quality of your own site tagging.