Tags:
productivity1Add my vote for this tag statistics1Add my vote for this tag create new tag
, view all tags

How to Measure the Productivity of Contributors

A wiki in a corporate setting brings value because it increases the productivity of the users. But how can you measure productivity? This is a hard question.

TWiki's WebStatistics has a top contributor list, wich can be a proxy for productivity. It counts the number of topic saves. Which can be confusing as we have seen by the discussions in TWikiOrgStatistics. TWiki's log keeps track of topic saves and follow-up saves. The follow-up saves are primarily used to fix typos and to check-point save large text. Follow-up save log entries contain a repRev. So, with the data currently logged we can:

  • Count topic saves, including follow-up saves: (what TWiki's statistics currently does)
    • Measures each topic save, typically multiple per edit session
    • Does not measure time spent editing
    • Favors people who tend to re-edit a topic several times
    • Neglects people who mainly post once, e.g. in comment boxes

  • Count topic saves, excluding follow-up saves:
    • Measures each topic edit session once
    • Does not measure time spent editing
    • Neglects people who tend to re-edit a page several times, and people who work on a long essay
    • Favors people who mainly post once, e.g. in comment boxes

So, a better proxy might be something in between, e.g. formula:

  • top contributor = (topic saves excluding repRev) + (only repRev saves) / 2

This probably comes closer to measuring the productivity, but it does not account for the actual time spent editing content. This could be addressed by integrating the time between edits and saves, which might be challenging to do because how do you determine a edit/save session (http is stateless; user might hit the back button to edit more).

Even if we measure the time spend editing a topic it does not account for the quality of the contribution, which is ultimately a measure of productivity.

Measuring the productivity of contributors is not easy. May be that is a good thing, we do not want big brother watching us.

-- PeterThoeny - 02 Sep 2005

I think you have to consider how repRev is used. 9 times out of 10, re-edits within the window are simple corrections; spelling, grammar etc. This is exacerbated by checkpoint saves. Sometimes these changes add more content, but they are usually refinements on the original contribution. So I would advocate ignoring the repRevs altogether. In fact, I don't really think counting edits is a very good way to measure productivity. Some people can be observed contributing comments that are long, carefully spelt, and detailed, but that don't actually add any value.

On assessing the weight of a contribution; historically, the only really effective way to do that in a community is through peer review. As long as the numbers aren't taken too seriously, this could be done quite simply. For example we could do something similar to twikibot's karma feature - i.e. if I write in a topic "PeterThoeny++" it increases your karma; if I write "PeterThoeny--" it decreases it. This might be further refined; if my personal karma from last month is high, the impact on your karma could be that much greater. At the end of a month, total contribution count could be weighted by karma earnt in the month.

However like any peer review system, this requires the understanding and cooperation of all contributors to work, and is open to abuse if too much weight is put on the statistics (I'm sure there are a few academics out there who would have something to say about the way academic peer review works!)

In a corporate setting, TWiki could be part of the peer review system used in assessing technical contributors (most big companies have some sort of technical ladder, and IME it is really difficult to assess promotion candidates; this sort of support could be invaluable).

-- CrawfordCurrie - 02 Sep 2005

Karma is interesting, but can be abused as well. For example, an individual can be pushed up or down based on what vocal people say about the person. A quiet consistent contributor wo does not get along with a vocal person can get negative karma, just because of peer influence.

Interesting and scary, assessing contributions for performance reviews.

Here is another twist: What increase in productivity can you expect when introducing a wiki successfully? Managers who are looking into this "new thing called wiki" are interested to know an answer to this question. The question is collective productivity, not individual productivity for performance review. I do not know how this can be answered. It needs to be compared to how work has been done before introducing the wiki.

-- PeterThoeny - 02 Sep 2005

even counting repRevs as half a save wouldn't have changed the order of most results. Compare the following (Obfuscated to protect the guilty, but I assure you this is based on real data. I've decided it's unfair to point fingers directly.):

grep "| Main.SomeOne
 142  Codev.SomeTopic
  42  Codev.SomeTopic
   7  Codev.SomeTopic
   6  Sandbox.SomeTopic
   1  Sandbox.SomeTopic
   1  TWiki.SomeTopic
   1  Trash.SomeTopic
with
grep "| Main.SomeOneElse
   1  Codev.SomeTopic
   1  Codev.SomeTopic
   2  Codev.SomeTopic
   1  Codev.SomeTopic
   4  Codev.SomeTopic
   1  Codev.SomeTopic
   1  Codev.SomeTopic
   6  Main.SomeTopic
   1  Plugins.SomeTopic
   2  Plugins.SomeTopic
   1  Plugins.SomeTopic
   3  Plugins.SomeTopic
  10  Plugins.SomeTopic
   6  Plugins.SomeTopic
   1  Plugins.SomeTopic
   1  Plugins.SomeTopic
   2  Plugins.SomeTopic
   1  Plugins.SomeTopic
   2  Plugins.SomeTopic
   8  Plugins.SomeTopic
   2  Plugins.SomeTopic
   2  Sandbox.SomeTopic
   3  Support.SomeTopic
   2  Support.SomeTopic
   1  Support.SomeTopic
   2  Support.SomeTopic
   2  Support.SomeTopic
   3  Support.SomeTopic
   4  Support.SomeTopic
   4  Support.SomeTopic
   2  Support.SomeTopic

It's worth pointing out that that topic with 142 saves only has 6 revisions. Even with the repRevs=1/2 formula the first person above will be in the results and not the second. But it's clear that the second person was contributing far more.

-- SamHasler - 03 Sep 2005

I think it would also help if we ignored the Main and Sandbox webs as these seem to be rife with repRev and there are people in the top contributor lists who have only been saving (and resaving) in those webs.

-- SamHasler - 03 Sep 2005

Peter wrote Karma is interesting, but can be abused as well. For example, an individual can be pushed up or down based on what vocal people say about the person. A quiet consistent contributor wo does not get along with a vocal person can get negative karma, just because of peer influence.

Very true; but it is also easy to see when personal attacks are being made; the attacker will be seen to be adding loads of 'JohnDoe--' to topics, something which can be picked up on quickly by other members of the community, resulting in the attacker's karma itself coming under attack. Similarly someone who tries to elevate their own karma can be quickly identified and dealt with.

Highly public peer review like this is a self-levelling system. If you wanted to, you could easily disable the '--' side of karma, which would stop attacks dead in their tracks, as it would only allow positive feedback. Rather like TWikiHearts, but far more accessible for everyday usage.

Yes, peer review in a corporate setting is scary. But surprisingly enough it actually works pretty well. I was a member of a technical ladder review board for a number of years; one of our functions was to audit the peer feedback process, and in this respect we had very little to do. The right people do end up being identified and recognised, even in the face of the inevitable antagonistic management and personal grudges.

The point is that contribution is about far more than simple volume of changes. It's about the quality of those changes, the value they bring to the community, and no automatic system can ever assess that. You need the human element to give value to the statistics.

-- CrawfordCurrie - 03 Sep 2005

I don't think that kind of Karma system is appropriate for the asyncronous communication in a wiki. It would be too easy to cause offense with just a single -- when there's no chance for a quick short exchange for explanation before people start wrting long posts attacking eash other. I don't think it works all that well in IRC either. It's just a bit of fun which works well in IRC but I don't think the fun factor would translate so well into the wiki.

Plus I think it would polute the topics somewhat. Particularly when people start posting just a 'JohnDoe++' and you have to wade through dozens of them in the WebChanges.

-- SamHasler - 03 Sep 2005

As I said, you can drop --. We are talking about a tool that can be used in a corporate setting, here.

Karma works; look at eBay (where it is called "feedback"). It's an invaluable, human, guide to the people you are dealing with.

-- CrawfordCurrie - 03 Sep 2005

now there's a good example to use!

ebay karma works - and amazon lists are important for finding out other things you'd be interested in

-- SvenDowideit - 03 Sep 2005

Here's another possible angle: what if we had a way to quickly rate specific revisions. So one could rate a particular addition to content or refractoring. Then, conceivably, this one measure could be used to generate a score both for topics and contributors.

-- LynnwoodBrown - 03 Sep 2005

With tag voting you get a collective rating of content. More interesting topics get more votes, thus bubble up to the top.

-- PeterThoeny - 01 Mar 2006

Edit | Attach | Watch | Print version | History: r10 < r9 < r8 < r7 < r6 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r10 - 2006-03-01 - PeterThoeny
 
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2017 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.