“The Web Analytics Report 2008″ is a long report. No, it is a “very long report,” writes its principle analyst, Phil Kemelor, who is vice president of strategic consulting for Semphonic, in the report’s opening words. “That is intentional.” It is 343 pages of user guide, thorough backgrounders on web analytics applications, technology and acquisition — and the meat of the report, in-depth reviews of 15 web analytics products and services. Long reports are bad when they are poorly organized. This is not a bad report. Nobody should have to read the whole thing, but that is only because nobody should have to read everything about every product.
There is no perfect technology, so good reviewers, particularly in emerging markets, must go beyond capabilities and reveal what’s missing, what doesn’t work and how to avoid known mistakes. It is thrilling to know what one could do, but it is profitable also to know what not to do. This report has no shortage of reasonable skepticism and warnings. The healthy doubt shows up early, in an introductory section about standards. Referring to the WAA’s standards, the authors observe, “It will be interesting to see whether consensus can be built around these standards.” Indeed.
Disappointingly however, the authors seem to have made no effort to discern vendors’ current or planned compliance with the WAA standards [PDF], leaving that homework to the reader: “While we can expect some vendors to tout that they are compliant with WAA standards and perhaps even change some existing reporting terms, you may end up with more investigation work to validate these claims.” This seems to say that standards are important, but not important enough to matter when choosing among vendors and products. Or perhaps it was just too difficult a task to include in this sort of report — but it would have been good to know why the authors decided to leave the further investigation to us. It seems as though they didn’t even ask the vendors the obvious question — are you now or do you plan to follow the WAA standards? Given the industry’s frustration with comparing analytics numbers between systems, greater attention to this issue would have raised the report’s value considerably. Perhaps that will be the subject of the next report.
The report does a good job of identifying the industry’s dirty little secrets that make some metrics — unique visitors comes to mind immediately — less than solid. A “data Accuracy” section describes the strengths and weaknesses of data capture technologies. Like every honest practitioner, the report acknowledges that many numbers are best used to identify trends, not absolute quantities. Caveat emptor. Or perhaps caveat counters, since no product or service is immune to the vagueness of certain Internet technologies.
Part of choosing a web analytics solution is to understand the business process it fits into. The report does a fine job of walking the reader from having no analytics to evaluating, choosing and implementing the solution. This is not merely a technical description, but a description of how analytics can meet management objectives, relate to or be part of organizational units (marketing? IT? finance?) and how it can interact with the people who build and maintain the web site.
In an emerging market, web analytics constantly faces new challenges as new web technologies are developed. The authors identify blogs, Flash, RSS, user-generated content and the use of qualitative analysis (surveys and such) as areas where vendors are racing to keep up with how data is delivered and gathered. At the same time, they propose that we are on the cusp of a fourth generation of web analytics (where do the years go?), in which “analytics tools begin to mature into the ‘brains’ behind website marketing and content generation.” If true, this is not entirely good news except for those of us who secretly long to become the center of attention. The same data we generate for reporting can and will be used for personalization, but since when was it a good idea to use the same system for production and reporting? Am I really going to tie data from Omniture, etc., back into my production system? Probably not. The good news is that this sort of bull… hyperbole only shows up here. The other 342 pages are practical and down-to-earth.
The report’s greatest strength is its scenario-based comparison of web analytics solutions. The rest could have been written by any reasonably bright person from information found on the web. The comparisons clearly came from in-depth discussions with actual users of each of the products and services. Each offering is rated many ways, based on types and purposes of the site being measured, who will manage and use the system and its reports, what kind of offerings — software, services, methods — the vendor offers. The authors and editors obviously put great effort into organizing the data into tables that make it easy to look up and compare the various offerings. If they hadn’t, the report would probably be 700 pages and nearly useless.
The result is a report that overcomes the vendors’ marketing noise to truly differentiate the products and services being reviewed. For each scenario, the offerings are rated as “Likely,” “Possible” or “Unlikely” to meet the needs. Screenshots and descriptions back up the conclusions. For example, to choose a random offering, IndexTools is ranked as “Likely” to suit multi-site analysis and interactive marketing analysis, but unlikely to be suitable for an application-driven web site. More important, the authors tell you why they came to those conclusions, which makes it easy for you to decide if you agree or not. Often, the greater value in these kinds of reviews is in the thinking behind the conclusion, not the conclusion itself, so this report earns high marks for explaining itself.
There isn’t much missing. Robots and spiders are treated as a problem to be filtered or blocked, rather than being essential to anybody who wants to be found via search. Open-source analytics packages are described very briefly at the end, which isn’t as serious an omission as the lack of any explanation as to why they are described so briefly. Are they that useless?
Bottom line? This report is well worth the price. While staying grounded, it paints a picture of what web analytics can be and how to get there. It makes you want to go out and measure something.
About Nick Arnett
Nick Arnett is Director of Business Intelligence Services at Liveworld Inc.
About CMS Watch’s Web Analytics Review
View a sample chapter of the Web Analytics Review from CMS Watch (registration required). The list of vendors and the table of contents are available without registration.
13-Feb-08 10:00 AM
See original here:
Review: The Web Analytics Report 2008
Popularity: 16% [?]
John Lovett from JupiterResearch finished his report: Web Analytics Buyer’s Guide, Assessing Vendors’ Competencies and Value - which concluded (..and John is smart, so listen up!) that:
