Tags:
create new tag
, view all tags
Peter said in BeijingRelease:

Only code that has been committed back into CVS is considered 100% complete by the CoreTeam. The 100% by the contributor means that the code is ready.

I have added a column into the status for Nice To Haves in BeijingRelease showing status.

I propose that the core team designates an individual from their group who proactively takes responsibility for the integration of people's patches. That person should come up with a checklist that the patcher should test their patch for and then guide the patcher with what they need to do to make it acceptable.

-- MartinCleaver - 09 Jan 2003

Do most people submitting patches have the skill and resources available to FULLY test it under all circumstances? That's what a QA team is designed to do. As a general rule, having been involved in quite a few products of different types, my experience says that coders don't make very good QA people. No offense intended to anyone! smile

-- GrantBow - 09 Jan 2003

Why should that offend anyone? Its been common knowledge in the industry for 30 years that I know of personally. Anyone whose ego gets in the way of a tiger-team trashing his code or thinks his code is "unbreakable" shouldn't be programming.

Perhaps the tables in BeijingRelease should have "test-team" and "% tested" colums too :-)
Or better still, each item should have a test defintion page that lists the test plan, results and perhaps resulting revisions to the code, interface or spec. People can add tests they've done or suggest tests. See also "Fuzz" and "Fuzz Revisited": http://www.suffritti.it/informatica/tco/fuzz-revisited.pdf

-- AntonAylward - 09 Jan 2003

A very tricky part about running a volunteer open source project is balancing what you would like to have done and well-intended great ideas with who will take responsibility for actually making it happen.

The PDF you mentioned is interesting. However I do have experience with automated testing as a QA engineer. The effort to build the automation often is equal to or greater than the effort to do the testing. The exception I have found is using specific automation tools (perhaps like fuzz if hooked to a web interface testing tool) in specific instances. Regression testing lends itself well to automation. That PDF also talks about white box testing which might be needed for many of the items but I am more concerned with a more basic black box type of testing.

-- GrantBow - 09 Jan 2003

One thing that would help a lot is the Wiki:UnitTest approach - this has been adopted for some modules within TWiki, e.g. RcsLite, but there are several different approaches used. The CPAN Test::More module family is well worth looking at - some Perl development at my company has used this, and it is also compatible with the Perl module testing approach used for CPAN modules. Such tests aren't a substitute for system tests but are very good at detecting problems, and Wiki:TestFirstProgramming is an even better approach.

It would also be helpful for developers, within or outside the CoreTeam, to have access to a group of testers who install the latest alpha on their own platforms - ideally covering various flavours of Linux, Unix, Windows, MacOS X, and so on, with and without ModPerl. Finding a sufficiently varied set of test platforms, with people willing to test, has been a key constraint on getting the InternationalisationEnhancements done - there is only one I18N tester apart from me, so I had to find a Perl 5.6 box with I18N enabled, as well as my main Perl 5.005 server and Windows laptop.

-- RichardDonkin - 10 Jan 2003

I think Grant and I are talking about something else.

There are realy two kinds of tests.

  1. Testing that it does do what the spec says.
    Problems that arrise here are:
    • Lack of or inadequate specification
    • The spec being writen in such a way that the implementaiton can't be tested
  2. Testing that it doesn't do other things.
    • "Fuzz" is one example of this
    • Most "Internet attacks" such as buffer over-runs, XSS etc., are examples of this
    • Boundary value testing deals with this.

The latter is important becuase of what is called the "failure of imagination" situation. Some wits talk about "idiot proofing" and God building better idiots. The reality is that the idiots are the spec writers and programmers who never imagined the end users would do what they did. The users usually insiste its what the expected the aplication to do. Often its even what the manual said it does do! (I was reprimanded once for suggesting to a manager that he hire a vacation student to go through and verify the examples in the manual. What a "Dilbert-eqsue" moment that was!)

Sadly, such testing is difficult becuase the imagination of the testers is often limited too wink

-- AntonAylward - 10 Jan 2003

Patches in some part of TWiki can be very difficult to evaluate as small changes can have big effects e.g. to certain RegularExpressions. We can certainly improve on the testing and I think unit tests for the non-WEB functions are the place to concentrate. Anyone could write some http client tests - these would also be useful.

-- JohnTalintyre - 12 Jan 2003

Can we just use a separate CVS repository as a patch management system for the working alpha. After all, is that not what CVS is designed for?

Patches too easily get lost in the Wiki and, as John points out, when separate to a working system can be impossible to test.

With TWikiUnixInstaller now allowing us to quickly install a TWikiAlpha (minus revision history) we could get much better participation in testing new revisions. Frankly it costs far too much to upgrade to alphas and I would risk even more if I had to backtrack to the release + patches version I had installed previously. No wonder only the core team tests and no wonder our release cycle is at least four times slower than many other open sources projects.

I posit, therefore, that we should PleaseCreateWorkingCvsForUnstableCore. It would involve more people in the testing, encourage more code submissions, allow non-core members to workably proposition code refactors and get releases out sooner. Furthermore, as a separate code base it would not risk the stablity of the delivered version.

-- MartinCleaver - 27 Jul 2003

Edit | Attach | Watch | Print version | History: r9 < r8 < r7 < r6 < r5 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r9 - 2003-07-29 - RichardDonkin
 
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2017 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.