Tags:
bugs1Add my vote for this tag cairo1Add my vote for this tag dakar1Add my vote for this tag documentation1Add my vote for this tag quality1Add my vote for this tag statistics1Add my vote for this tag create new tag
, view all tags

Statistics of TWiki Production Releases

Statistics comparing TWiki 01-Dec2001, TWiki 01-Feb-2003, TWiki 04-Sep2004, and TWiki 4.0. Production releases are 12 month or more apart.

Source Code Documentation Major Features & Bugs

Release SFiles Subs SLOC Topics Words Features Bugs 30 Bugs 90 Comments
Athens 18 284 4356 137 51241 46 1 3 1949
Beijing 23 412 5998 167 72743 21 2 3 3038
Cairo 50 691 12970 181 121915 47 1 1 7118
Dakar 97 1042 18954 301 160170 84 9 9 9732

Legend:

  • Production releases:
  • SFiles: Number of source code files (in bin, lib and tools)
    • See attached script
  • Subs: Number of Perl sub routines
    • See attached script
  • SLOC: Number of lines of source code
    • See attached script
  • Topics: Number of topics in Main web and TWiki web
    • find data/TWiki/ data/Main -name '*.txt' | wc -l
  • Words: Number of words in topics in Main web and TWiki web
    • find data/TWiki/ data/Main -name '*.txt' -exec cat {} \; | head | wc -w
  • Comment: Lines of comments (pod and #) including banner licenses.
  • Features: Number of major new features added
    • Note: This is a somewhat subjective measure, taken from the TWikiHistory
    • Athens: FormattedSearch 8; URLPARAM 1; Plugin API 7; web-based change/reset/install passwords 5; attachment under revision control 1; rename/move topics; headings & TOC 4; verbatim text 4; selective include with STARTINCLUDE/STOPINCLUDE 1; TWiki skins 6; ACLs with groups 9. (includes 01-Sep-2001 release)
    • Beijing: Internationalization for WikiWords 3; enhanced Plugin API and more Plugin handlers 3; URLENCODE 1; template topic based registration 2; web-based new web creation 1; RCS in Perl (RcsLite) 9; AND search 2
    • Cairo: Upgrade script 7; new CSS bases skin (PatternSkin) 9; skin browser 1; many SEARCH enhancements (keyword search, multiple="on", expand variables, separator, sort by creation date) 5; internationalisation enhancements (UTF-8 URLs) 1; 7 preinstalled Plugins 1; improved Plugins API and more Plugin handlers 3; support for different authentication methods 1; use TWiki Forms to set user preferences 3; FORMFIELD 1; tool-tip topic info 1; many user interface and usability improvements 7
    • Dakar: Simpler install and configuration 9; integrated session support 3; webserver-independent login/logout 4; security sandbox 5; edit conflict resolution handling 8; multilingual UI 10; e-mail confirmations for registration 9; WYSIWYG editor 10; hierarchical sub-webs 5; change notification on page level and parent/child relationship 4; INCLUDE improvements (named include sections, parameterized includes) 2; many SEARCH enhancements 3; many new variables 7; REST interface 2; improved Plugins APIs 3
  • Bugs 30: Number of major bugs discovered 30 days after production release
  • Bugs 90: Number of major bugs discovered 90 days after production release (includes the 30 days)

-- Contributors: PeterThoeny, CrawfordCurrie

Discussion

I compiled these statistics to give some food for thought. I listed bugs 30 and bugs 90 to give a measure on code quality of any given release (the TWikiMission's goal is to have "few or no bugs for production releases".)

Things I read:

  • Source code: Exponential growth in the number of files and subroutines vs. linear set of new features per release -- typical pattern in growing complexity of software
  • Docs: Linear growth in word count
  • Bugs: Few number of major bugs (and constant number) in earlier releases vs. 3 times the number of major bugs in Dakar

We should try hard to refine the PostDakarDevelopmentModel so that we get the quality back to the goal of the TWikiMission.

-- PeterThoeny - 05 Mar 2006

Please be very consious that the way bugs are measured before Dakar has changes dramatically, making it statistically inappropriate to compare the stats before and after.

pre Dakar, there was already some contention about the issues that we chose to list as Bugs, and those that we classified otherwise.

In Dakar this is a little more transparent.

-- SvenDowideit - 06 Mar 2006

If you look through the feature points, then something becomes very clear; not all features are equal. By counting each feature point as "1" you have equated "tool-tip topic info" with "multilingual UI". Sorry, but these are not equivalent; multilingual UI has included hundreds of code lines, many man-hours of development and testing, translations into 8 different languages.

Clearly those feature points have to be assigned a weighting, if only to take account of the fact that most involve many "sub features". Since it's basically a subjective measure, I have added subjective weightings above.

-- CrawfordCurrie - 06 Mar 2006

Where's the performance comparison chart (PCC)? wink

-- FranzJosefSilli - 06 Mar 2006

Note that SLOC was counting pod comments. The attached script generates source code counts correctly. I have corrected the counts above.

PeterThoeny wrote:

We should try hard to refine the PostDakarDevelopmentModel so that we get the quality back to the goal of the TWikiMission

I don't think you quite realise how very, very irritating I find statements like that, and the fact that the graphs originally posted in the topic were clearly selected to try and show Dakar in a bad light. The statement implies that somehow the development model is at fault. Crap. The problem is that only a subset of the community delivered on the commitments they made, in terms of input to the process. On the negative side:

  1. Despite repeated appeals during Dakar development, there are still only a small number of unit tests and testcases. The ones that do exist have been invaluable. If compatibility is so important, why are there still no tests to verify the spec?
  2. Despite repeated appeals during the development, the only non-developer to give regular testing feedback during most of the development was Anton. There was a sudden rush of interest during the last couple of months, but by then it was late in the game.
  3. Do I have to remind you that you repeatedly identified yourself to me as "the technical author"? By insisting on treating TWiki.org as the documentation masters, we ended up with documentation being pushed later and later in the process, eventually having an appalling manual merge, which you wanted to do twice for some reason that still escapes me, and which could easily have been avoided by direct updating of the docs in subversion.
  4. There were promises of refactoring in twiki.org from several people; hardly any of it came to anything. As a result Codev remained constipated and largely useless as a discussion forum.
  5. There were several "lob it over the wall" contributions, that ended up creating a lot of work as the original authors made themselves scarce for most of the development.
On the positive side:
  1. First Anton, and then Kenneth, ran live sites, at considerable risk to themselves, on the development code. The feedback from this was fantastic, as was the use of ~develop as a testbed.
  2. Several pure users got involved in the dev, which is a clear sign of the way to go. Early feedback from users preferred.
  3. Several companies (including you, Peter) came forward to fund developments that have been of significant value to TWiki-4 - the mailer, wysiwyg, template login, all would have been impossible without that sponsorship.
  4. We (mainly Sven, Will, and myself) put a huge amount of effort into infrastructure, including automation for the whole build process for core and plugins, and an excellent bugs web. This infrastructure has been a godsend, enabling not only developers, but testers as well.

OK, so what lessons can we learn from this?

  1. Test, test, test, test, test
  2. Live sites are critical as testbeds.
  3. Commitments have to be followed through. We need to be a lot more hard-nosed about demanding commitment to support, as well as commitment to contribute. And, importantly, following up on that commitment.
  4. Documentation needs to be everybodies responsibility, and continually updated.
  5. Feature sponsorship is a strong positive for the future of TWiki.

Finally, we currently have an active developer community, which is great. IMHO for it to remain active requires a lot less navel-gazing, and a lot more just getting on with things.

-- CrawfordCurrie - 07 Mar 2006

I find the tables totally useless because, from my point of view, there are more bugs in Dakar discovered the first 30 days because the testing of Dakar has been a lot more effective than the testing of Cairo the first 30 days.

How many "bugs" where fixed in Cairo last year? How many Cairo "bugs" where fixed in Dakar? How Many security advisories where issued because security bugs in Cairo?

Also... bugs against Skins have never been counted as Releases bugs, but Skin bugs.

-- RafaelAlvarez - 07 Mar 2006

Edit | Attach | Watch | Print version | History: r9 < r8 < r7 < r6 < r5 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r9 - 2006-06-09 - PeterThoeny
 
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2017 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.