Sunday, March 11, 2012

Measuring Site Engagement: Pages or Sessions

One of our clients is a large media website that faced a simple question: What is the best way to find the most engaged users on the web site? The goal was to focus a marketing effort on these users.

A media web site is challenging, because there is no simple definition of engagement or customer worth. The idea is that engagement can either lead to more advertising views or to longer subscriptions, depending on the business model for the site. On the other hand, for a retailing site, the question is simpler, because there is a simple method to see who the best customers are. Namely, the amount of money they spend.

Engagement is a nice marketing concept, but how can it be defined in the real world? One way is to simply look at the number of page views during some period of time. Another is to look at the number of sessions (or alternatively days of activity if sessions are not available) during a specified period of time. Yet another is to measure breadth of usage of the site over a period of time: Does the user only go to one page? Is the user only coming in on referrals from Google?

The first analysis used one month of data to define engagement. The top users for one month were determined based on pages and sessions. Of course, there is a lot of overlap between the two groups -- about 60% of the top deciles overlapped.

Which group seems better for defining engagement, the top users by page views or by sessions? To answer this, let's borrow an idea from survival and measure how many users are still around nine months later. (Nine months is arbitrary in this case). In this case, the return rate for the top decile for sessions was 74.4% but for the top decile for pages was lower at 73.8%. Not a big difference, but one that suggests that sessions are better.

Actually, the results are even more striking for visitors who are not in both top deciles. For the non-overlapping group, the session return rate is69.6% versus 67.9% for the page deciles.

For defining engagement, we then extended these results to three months instead of one to find the top one million most engaged users. The three measures are:

  1. Visitors that have the most page views over three months.
  2. Visitors that have the most sessions over three months.
  3. Visitors in the top tercile of sessions (third) in each month, then take the highest terciles.

Three months was chosen as a rather arbitrary length of time, because the data was available. Holding it constant also lets us understand the difference between sessions and page views.

These three methods all produced about the same number of visitors -- the goal was to find the top one million most engaged users.

By these measures, the top one million visitors chosen by the three methods had the following "return" rates, nine months later:

  1. Page views in three months: 65.4%
  2. Sessions in three months: 65.9%
  3. Sessions over three months: 66.9%

The nine-month survival suggests that the sessions over three months is the better approach for measuring engagement.

5 comments:

  1. 65.4
    65.9
    66.9

    Seems like a wash! ;-)

    ReplyDelete
  2. Nice post..
    Engagement is a hard metric to pin down on..
    I liked the idea of return rates..I suppose one can expand this concept of return rates and service usage to areas other than web analytics as well.
    For example - Customer return rate in the services industry - hotels, banks..

    ReplyDelete
  3. Gotta agree with anonymous, even though it may be statistically significant, realistically it seems like these are more or less equivalent metrics.

    ReplyDelete
  4. I agree with Priya that engagement seems like a difficult metric to not only define but also to measure (hence, hard). Engagement seems to be an aggregate of measures of individual user activity (or at least it should be in order to be more accurate). One other thought on the metric is to also consider the ever growing pool of users as time passes (hopefully there are more users as time passes). It may be useful to define active users versus one-time or “dead” users and compare engagement of active users versus the not-so-active users.

    ReplyDelete
  5. This is where it helps to have qualitative data (surveys) to add context.

    ReplyDelete

Your comment will appear when it has been reviewed by the moderators.