Part VIII of a Series on Functionalism and Web Analytics – (You may want to visit http://www.semphonic.com/resources/whitepapers.asp and download the White Paper on Functionalism as a detailed technical companion piece to this series).
“Epiphenomena” is a name that Paul Legutko (our VP of Analytics) came up with to describe a set of problems in web analytics that are becoming increasingly central and are especially important to understanding the analysis of ad-based content. I loved the name because it captures perfectly the underlying issue. But, of course, almost nobody will know what it means.
Epiphenomena are effects or symptoms which are associated with a deeper effect or condition, but are not the primary cause of this condition. For example, rich people drive more BMW’s than poor people. But driving a BMW isn’t what makes you rich (quite the contrary)! Catholics are more likely to have a Christmas tree in their living room during the winter holidays. This doesn’t mean that setting up a Christmas tree in somebody’s living room will make them a Catholic.
Web site behaviors are loaded with epiphenomena – effects created as products of either visitor self-selection or site navigational structure. Epiphenomena are most easily mis-understood in the context of visitor segmentation.
The basic idea behind most segmentation is simple: let’s define a particular kind of visit or visitor, and then look at the traffic and page-consumption patterns for these particular segments, comparing them against each other and/or looking for particular site features. Findings are very often of the form “a visitor of feature X is Y% more likely to do Z on my site.”
Let’s start with an extreme and obvious example of how epiphenomenal effects might be misleading in this context.
For a website that has a membership component, marketers are nearly always interested in what kinds of pages or tools are most likely to get visitors to become members. By creating a segment of “new members” and looking at what pages they used on the site, it’s possible to answer the basic “what were Members most likely to do.”
Naturally, the membership sign-up page is going to be used much more often by this segment than by non-members. But from this simple fact one shouldn’t, of course, infer that the membership sign-up page is therefore the “cause” of memberships or the appropriate landing page for membership drives.
This example is childish, but there are many less obvious cases where it’s much easier to fall into the error of believing that epiphenomena are actually causal. Suppose I have a particular page on which Brad Pitt gives a testimonial about how wonderful it is to be a Member of my site. I want to see how effective this page is in driving membership signups, so I create a segment of “people who saw the Brad Pitt testimonial.” Again to my pleasant surprise, this segment contains a higher percentage of members than the overall unsegmented totals.
This might be actionable, since my visitor segment says nothing about whether someone’s a member or not. However, this effect might also be epiphenomenal, albeit less directly. A visitor has to be somewhat engaged on my site in order to see the testimonial page, and likely members are also, by definition, more engaged. By asking for people who saw that page, I’m in effect asking for visitors who are more engaged than the general population, which will also include unengaged single-access visitors. So naturally, this segment contains a greater percentage of members.
In general, the more requirements are placed on the visitor to be included in the segment, the more “engaged” the visitor is and the more likely they are to exhibit a particular success event. When we study advertising supported content sites, one of the things that we observe with high-consistency is that the visitors to less-used (non-entry) sections of a site are almost always much more-engaged than average visitors. This isn’t because the less-used areas build loyalty – it’s because by definition the likely finders of the area are highly engaged.
So how do you measure billboard type pages using Functionalism without measuring epiphenomena?
The Functional KPIs focus on three different ways a content area might be successful in generating additional traffic: holding visitors on the site, bringing visitors back to view more pages within the content area, and bringing visitors back (to other site areas) by boosting visit satisfaction.
Why three different types of measure? Some content is inherently single-use – it applies to a visitor at a special point in time or with a one-time specific need; while other types of content are only effective insofar as they bring visitors back for more (a blog is a good example of the latter).
Subsequent page consumption in a session is one of the basic measures to see if a content area is “holding” visitors and routing them effectively. Note, however, that this measure is vulnerable to the type of epiphenomenal misinterpretation I’ve been talking about.
To combat that, we have additional measures that are less likely to give you a mis-reading.
For single-use content, the type of success we’d expect to see is measured most effectively by subsequent visitor returns without viewing the target content.
On the other hand, many kinds of content area are only working well when they are gathering repeat viewers. For this kind of content, the type of success we’d expect to measure is subsequent visitor returns to the original area.
Let’s tackle this second metric first. In a sense, it’s a measure of the mindshare that a particular content area grabs from total site visits. And to measure it, we track how many of a visitor’s visits (and pages) include the target area. By measuring “mindshare” over time, we were able to screen off the vast majority of epiphenomenal effects – and establish a much better measure of how influential a content area actually is on return behavior.
But what about tools (and content) that are inherently single use? For this type of content, we’ve found that the most effective analysis is essentially to construct an artificial A/B test. We construct a population segment of first-time content area users and then create a matching set (across key variables like pages viewed and prior sessions) of visitors who didn’t see the content. By measuring the difference in subsequent over-time return behavior, we have a measure of whether or not the content actually improved (or degraded) visitor loyalty.
Billboard pages (and advertising based sites in general) present particular problems for web analytics. With an amorphous measure of success and a whole train-load of epiphenomenal baggage attached to almost every view, it’s particularly essential to find the right KPIs to measure. And because web analytics vendors have been driven more by traditional e-commerce and large corporate sites, the measurement needs of content-based ad-supported sites have been neglected in the most popular tools. The functional approach provides a good way to think clearly about the issues involved with measuring these types of sites and pages – but it remains a formidable measurement challenge.
Tag:
Add to 
Bookmark Murdok:
Gary Angel is the author of the “SEMAngel blog – Web Analytics and Search Engine Marketing practices and perspectives from a 10-year experienced guru.
