Measurements of engagement aren’t necessarily measurements of interest.

Tipped off by a post on RWW, I  installed the Postrank extension for Firefox on Google reader yesterday. Postrank adds a number next to each post which is a composite of  “how engaging” the post has been, e.g whether it’s been clicked, bookmarked, commented on, or even inspired new posts. This number can be between 1 and 10, where 10 is the most engaging post and 1 is the least.

What’s was immediately interesting was how few of my starred stories qualify on Postrank  as “great” posts. I started looking into this and the trend that I began to see was that the top posts tend to be ones that are about the immediate present and are about the web, or social media or web tools.

picture 21 Measurements of engagement arent necessarily measurements of interest.

Among the least “engaging” stories are many of the larger stories that are, to me, more interesting but less immediately practical

picture 12 Measurements of engagement arent necessarily measurements of interest.

This story for example on Intel’s innovation which scored a fairly poor 1.6 (although I have just theoretically improved that) was one of the more interesting things I’ve read lately. Unlike almost everyone else, Intel has decided not to retrench and instead wants to use the recession to actually move ahead faster. In doing so they’re going to obsolete themselves. This press release goes into more detail with some great quotes from Craig Barrett:

“In the current global economic climate, thinking long-term is more important than ever,” Barrett said. “Today, we can lay the groundwork for growth. Many nations and businesses try to save their way out of a recession. It is much better to invest our way out.”

Barrett believes that investing in new ideas and inventions not only stimulates job creation, but also increases productivity, leads to new forms of energy and smarter energy consumption, improves health care and medicines, and gives consumers more benefits for less cost.

The thing is I don’t actually think I’m that different, I don’t think it’s that I’m just interested in different things than most people. I think a lot of people found this news interesting,  it’s that these types of posts aren’t things that we’re going to interact with physically, because they’re designed to make us think.

This is a cousin to the popularity problem that I wrote about before and which Noah wrote about this weekend. There are structures that the we use on the Web that are designed to surface the most interesting information that often end up actually hiding it.

I think that this problem also extends to other kinds of engagement measurements, particularly those that try to assess the effectiveness of marketing communications. When people click on virals, pass them on, or bookmark them, it does not immediately also mean that they find them interesting. They are clearly finding them useful, but that use may simply be as an object to continue a relationship with their friends rather than as a tool for making or forming an impression about a brand or product.

Technorati Tags: , ,

Random Posts

Tags: , ,

  • Popularity is fleeting. Interesting is forever.
  • Hi again,

    No worries, it's pretty hard for me to take offense, and really, I LOVE when people pick PostRank apart. It's from those folks that you often learn the most useful ways to improve what you're doing.

    I hadn't seen the research project; thanks for the link. It's a funny balance -- we want the credibility that comes with "just the facts" -- measurement by algorithm. But at the same time we want the nuance and recommendations that come from social connection, which is purely human. (Though, of course, the folks who create the algorithms are human, too...)

    And, of course, the same people who want to have the control over how we weight/measure things are usually the first ones to holler about compromised credibility of results, ironically. :)
  • @Melanie, thanks for stopping by and clarifying. Just so you know, I wasn't trying to pick on Postrank unfairly, I've actually found the ranking service to be pretty useful (in my short experience) and it'll be a permanent fixture on my FF extensions. I think this is a bigger problem (as you obviously know). I did a bunch of research on this a while back, not sure if you guys saw this: http://oak.cs.ucla.edu/~cho/research/bias.html

    There need to be system in place to balance out the bias that's being created by popularity systems. Glad to hear you're working on it, I'll give the discovery functionality a go.
  • The discussion about how to best measure "interestingness" has gone on for some time, and 100 people will give you 10 different answers. (I know, cuz I'm the one they tell...) :) The approach we've taken is three-pronged:

    - gather as many engagement sources as possible to capture as many expressions of interest as possible
    - gather engagement metrics over time to capture stories' real interest arc (some stories get all the attention immediately, others trickle in over time)
    - account for the human aspect - the individual interests and random finds.

    We've had a lot of feedback about needing to account for that third item, which was the big catalyst behind the Discovery functionality we just launched.

    I.e. it's about what YOU find interesting and what YOU read and making it easy to share that with others. Especially since there's way too much info online for any one person to keep on top of.

    Cuz if I've learned one thing from perusing the feeds in our system, it's that there are communities online around EVERYTHING, so there's always someone else into what you're into.
  • Seems to me that there are 3 different issues with being able to measure real interest.

    1. Only you can really say how interested you are, your actions and behaviour aren't always affected by interesting information, so research as you suggest (Mark) is probably required.
    2. Interesting is relative so what's interesting to you may not be interesting to others, so you would want a metric based on your personal interests (Lee, you'll be laughing as we've talked about this endlessly)
    3. Interesting information may take time to process and absorb before it's acted upon. Unfortunately because we have created, and are moving further towards a real-time web where information is tailored to being acted upon immediately, this problem will probably get worse before it gets better.
  • This may be a bad shortcut in my thinking right now, but it seems the only way to measure true engagement is to implement tools beyond the web, to get to a measure of 'interesting.' Maybe I'm talking about qualitative research, like attitudes, preferences and behaviors, somehow all mapped together, or some other ethnographic angle. I guess its not a simple answer to a very complex question. There is a water-skimming behavior that has become the typical information consumption experience online. Current analytic measures are simply not sophisticated enough to go much deeper than that at the moment.
  • Lee
    Completely agree! So what would be better way to measure engagement? Time on page? Clicks within the site? Likelihood to leave a comment?
blog comments powered by Disqus