TWiki Developer Environment
Automates build and packaging process, including installer generation, for TWiki extension modules.
Summary of Contents
The BuildContrib can be used to create a build script for your TWiki extension. It is inspired by the Java ANT build tool developed by the Apache project, but is targeted specifically at building TWiki extensions. It is also used for TWiki release builds. The advantage of using BuildContrib is that it dictates a standard structure and build procedure, which makes your extension easier for you, and others, to build and maintain.
- not just for building code modules, can also be used to package TWikiApplications, and even pure documentation packages.
- automatically generates an installer script that can help simplify end-user installation.
- includes a wizard script for creating a new extension.
- supports creating collections of extensions.
- This is a build and packaging module for use by developers, not an install module for end users (though it does build an installer script, among other things).
- The module has only been tested on Linux, but should work with Cygwin OK.
- BuildContrib has only been tested with TWiki-4. It might work with earlier TWiki versions.
If you don't like reading documentation, and just want a fast route to creating a new TWiki extension, then:
- Install the contrib using the instructions below
cd to the root of your twiki installation
perl create_new_extension.pl extension_name then modify the extension files as required (including MANIFEST). Then when you are ready to create archives:
perl build.pl extension_name release
- Archives (.zip, .tgz) will be created in
The build module assumes:
- two kinds of extension modules; 'Plugins' (as in TWikiPlugins) and 'Contribs' (everything else e.g Skins, TWikiApplications etc),
- you have some passing familiarity with build tools such as
- you are not developing your extension in a 'production' TWiki installation (something that is usually a really bad idea), but are instead doing the sensible thing and developing in a separate directory tree
- usually - but not always - a subversion checkout area.
Standard directory structure
BuildContrib is used to build the TWiki core, as well as most extensions. This document will focus on its use for building extensions. See the
file in a subversion checkout for information on building the core.
Extensions are developed in a subdirectory below SVN trunk. For example, BathPlugin will be developed in
. This directory is called the root directory
for the extension.
The standard directory structure under a root directory mirrors a TWiki installation tree. Every plugin has some key files
name.pm - code file for the plugin, usually derived from EmptyPlugin
name/ - directory containing sub-modules used by your plugin, and for storing your
build.pl script and other support files. It is referred to as the module directory
build.pl - build script for this extension
MANIFEST - list of files to be installed
DEPENDENCIES - list of modules this extension depends on
configure setup for this extension
name.txt - your plugin/contrib topic
name/ - directory containing unit tests for the extension
lib/TWiki/Contrib directory instead of
lib/TWiki/Plugins but otherwise in exactly work the same way.
Other directories normally found in a TWiki installation may also exist under a root directory e.g.
Setting up for Development
The first thing to do is to either
- check out a TWiki development environment from subversion, or
- create a separate TWiki install. Configure this install so it's a running TWiki; we'll refer to this as your development TWiki.
Now install the BuildContrib. In a subversion checkout,
to the TWiki root and
perl pseudo-install.pl BuildContrib
. In a non-subversion environment, install the
package from TWiki.org.
Your build script has to know how to find the TWiki libraries, so it can pick up the components of the build system. Set
(which is a path, same as
) to point to your
directory in your development TWiki.
is used to extend @INC for the duration of the build only
, so it won't mask problems during testing.
The approach we recommend
is to set
in your login script (e.g.
depending on what shell you prefer).
build.pl does not read
bin/setlib.cfg. It uses
$TWIKI_LIBS only to find the modules for the BuildContrib.
Each individual extension has its own build script, called
, in its module directory. A build script is a perl script that takes a number of targets
as its parameters. For example,
perl build.pl test
will run unit tests, and
perl build.pl release
will build a new release.
The build script also accepts the following options:
| Do nothing; just print what you would have done
| Be verbose
| with target
upload, only upload the topic (not the archives)
Build targets are Perl functions, which operate on various data defined in control files to build the various targets. Perl is used rather than
for portability reasons.
The targets you will normally use are:
| perform basic build steps
| Run Perl::Tidy on all perl modules listed in the MANIFEST
| run unit tests
pod and package a release zip
release and upload
| print a guess at the MANIFEST
| Generates a list of svn checkins with comments suitable for use in the history section of the plugin/contrib topic.
| Find and print a best-guess dependencies list (for DEPENDENCIES)
The default target is
. The BuildContrib is designed so that most common behaviour is catered for. It is also easy to override
any of the default targets in your
and add extra behaviours.
Note that there is a shortcut script,
, that can be very helpful when you are building several extensions in one session. Run the script without parameters for documentation.
Does nothing by default. This is the first target executed, and can be overridden by your build.pl to do something unusual - for example, executing an ANT file to build some Java.
files listed in MANIFEST that have a corresponding
in the directory structure. When it finds one, it automatically compresses the
file to create/refresh
. If you are using Subversion, the generated file should then by checked in.
Also works on CSS files, for the file extensions
to perform the compression
This target runs Perl::Tidy (with default formatting options) over your source code. This reformats the code consistently with the TWiki coding standards.
The test target is designed for use with extensions that have unit tests written using the UnitTestContrib. It automatically runs the unit tests found in the
The results of the
- a Zip format archive,
- a gzipped tar archive,
- a md5 checksum,
- the extension topic,
- an installer script
The archives will each contain the following:
- All the files listed in the
- Another copy of the install/uninstall scripts
This target builds a release, and then tries to upload it to a target repository. The target uploads all the files in the release, and also tries to upload any attachments to the extension topic (as found by scanning the topic for META:FILEATTACHMENT).
You can control what server the upload is done to. This lets you - for example - set up your own corporate extensions server.
These are used when you are unsure of the correct contents of MANIFEST and DEPENDENCIES. They make a best-guess at the required contents for these files.
file contains a list of all the files that are wanted in the package. Each line is a file path, relative to the root of the installation. Wildcards may NOT be used. If the path contains spaces it must be enclosed in double-quotes.
Each file path has an optional octal permissions mask and a description. For example,
data/TWiki/BathPlugin.txt 0664 Plugin description topic
lib/TWiki/Plugins/BathPlugin.pm 0444 Plugin code module
If no permissions are given, permissions are guessed from the permissions on the file in the source tree. These permissions are used by the installer script to set file permissions in the installation.
The following permissions are recommended, and will be applied by default if you don't specify anything different:
| File type
|| Anyone can read, but cannot write or execute
|| Anyone can read, user and group can also execute
|| Anyone can read, only owner can write
| File in
| File in
|| Anyone can read or execute, but not write
| Anything other file
|| Anyone can read, but cannot write or execute
|| default directories to traversable
,v files. If you include a
,v file it will overwrite any existing
,v file when an extension is upgraded, potentially wiping out local changes on the end users installation.
MANIFEST, or any other side file used by the build process.
- unit tests
MANIFESTs can also include other extensions that have been packages using
BuildContrib. For example,
This will include the WysiwygPlugin in the release package.
Note that there is a script in the TWiki
directory (TWiki 4.2 and later) called
that can be run at any time to check the contents of your MANIFEST against what is checked into Subversion.
file specifies dependencies on other extensions and perl modules. Each line of the file is a single dependency:
name, version, type, description
- name is the name of the module,
- version is the version constraint (e.g. ">1.5"),
- type is its type (cpan, perl, C etc) and
- description is a short description of the module and where to get it.
The installer script written by the build process uses the dependency type to decide how to install dependant modules. 'cpan' means 'get the module from CPAN
' and 'perl' means 'get the module from the Plugins web on TWiki.org' (or whatever other repositories the admin has specified using
When your module (the depender
) depends on another module (a dependee
), it is important to think carefully about what version of the dependee your module requires.
When you are working with TWiki modules (such as contribs and plugins) you should list the version number of the module that you tested with. Normally you will want to use a
condition, so that more recent versions will also work. If a dependency on a TWiki module fails (because the module isn't installed, for example) then the installer script will pull the latest version
of the module from TWiki.org, whether that is the required version or not. This is a limitation of the way plugins are stored on TWiki.org.
When you are working with CPAN
modules, you need to take account of the fact that there are two types
are perl modules that are pre-installed in the perl distribution. Since these modules are usually very stable, it is generally safe to express the version dependency as ">0" (i.e. "any version of the module will do").
Note however that the list of built-in modules is constantly growing with each new release of perl. So your module may be installed with a perl version that doesn't have the required module pre-installed. In this case, CPAN
will automatically try to upgrade the perl version
! There is no way around this, other than for the admin on the target system to manually
install the module (download frm CPAN
and build locally). You can help out the admin by expressing the dependency clearly, thus:
File::Find,>0,cpan,This module is shipped as part of standard perl from perl 5.8.0 onwards. If your perl installation is older than this, you should either upgrade perl, or manually install this module. If you allow this installer to continue, it will automatically upgrade your perl installation which is probably not what you want!
A dependency may optionally be preceded by a condition that limits the cases where the dependency applies. The condition is specified using a line that contans
ONLYIF ( condition )
, where condition
is a Perl conditional. This is most useful for enabling dependencies only for certain versions of TWiki. For example,
TWiki::Rhinos,>=1.000,perl,Required. Download from TWiki:Plugins/RhinosContrib and install.
ONLYIF ($TWiki::Plugins::VERSION < 1.025)
TWiki::Plugins::CairoContrib, >=1.000, perl, Optional, only required if the plugin is to be run with versions of TWiki before Cairo. Available from the TWiki:Plugins/CairoContrib repository.
Thus CairoContrib is only a dependency if the installation is being done on a TWiki where the
is less than 1.025. The
only applies to the next dependency in the file.
Writing a build script
The easiest way to write a new build script is to use the
script, which is part of the BuildContrib.
- Create your plugin source tree under
perl create-new-extension.pl BathPlugin
lib/TWiki/Plugins/BathPlugin.pm as required to create your plugin functionality
lib/TWiki/Plugins/BathPlugin/MANIFEST and make sure it lists all the files you want to include in the release package
During development we recommend you use the
script to soft-link your plugin code into your development TWiki. This script uses the MANIFEST you write and creates softlinks in your development TWiki that allow you to run your test code without having to do a full re-install each time you make a change.
If you have a pre-existing extension, and you want to package it for use with BuildContrib, then you need to create the module directory and write the
files. The easiest way to do this is to copy those files from an existing extension in subversion, and modify
them for your extension.
Preparing the Installer
The installer script generated by the builder when target
is used is based on a template. This template is populated with lists of files and dependencies needed to make the extension-specific installer script.
You can extend this script by providing
files in the module directory. These optional files are embedded into the template install script at the appropriate stage of the installation. Read
(in the BuildContrib) to see how they fit in. The POD comments in that module indicate the functions that are most likely to be useful to anyone writing a script extension.
You are stongly recommended to develop a unit test suite for your extension. Unit tests are kept in the
directory for each extension.
To run the unit tests you will need to set up the test environment described in TWiki:Codev/TestCasesTutorial#SettingUpATestEnvironment
. That page also includes extensive information on writing testcases.
The easiest way to generate tests for your extension is to copy the approach taken in another extension. See for example ActionTrackerPlugin and CommentPlugin, which both have extensive test suites.
Tests are run using
Building a release
When you are almost ready to release, you should
- Build a release package and installer
perl build.pl release
- Remove the softlinked version from your development TWiki
cd development TWiki
perl pseudo-install.pl -uninstall BathPlugin
- Install the release package you just built:
cd development TWiki
target automatically expands certain tokens in
files and in the installer script. The following tokens are supported:
%$MANIFEST% - TWiki table of files in MANIFEST
%$FILES% - hash keyed on file name mapping to permissions i.e. 'data/TWiki/ThsiTopic.txt' => 0664, 'lib/TWiki/Plugins/BlahPlugin.pm' => 0775
%$DEPENDENCIES% - list of dependencies from DEPENDENCIES
%$VERSION% value of $VERSION
%$DATE% - local date
%$POD% - POD documentation for the package, excluding test modules.
%$PREINSTALL% - contents of PREINSTALL
%$POSTINSTALL% - contents of POSTINSTALL
%$PREUNINSTALL% - contents of PREUNINSTALL
%$POSTUNINSTALL% - contents of POSTINSTALL
%$BUGSURL% - URL of bugs web
%$INSTALL_INSTRUCTIONS% - basic instructions for installing
You can also create other tokens to be expanded by adding them in your
. For example,
TODO: add code sample here?
When you are happy the release package is built correctly, you can upload it.
perl build.pl upload
By default the
target will upload to TWiki.org. You will be prompted to enter an alternate upload target, should you require it (e.g. to upload to private corporate repository). The upload updates the topic and any associated Var topics published by the extension, and uploads zip, tgz, md5 and installer files.
Installer scripts build by BuildContrib are important for the full functioning of the extensions installer in
The installer script shipped with the package is very simple. By default all it does is to check the dependencies you list, and if necessary download and install any missing TWiki and CPAN
modules. Other dependencies are simply checked. In TWiki-4.0 and later releases, TWiki topics shipped with the module are automatically merged into any existing local copies, ensuring histories are preserved.
If you want your installer to do anything else then you will need to write a POSTINSTALL script
You are strongly recommended to use this Contrib to help split your code development away from your live TWiki environment, as described above
- For an automated installation, run the configure script and follow "Find More Extensions" in the in the Extensions section.
- Or, follow these manual installation steps:
- Download the ZIP file from the extension home on twiki.org (see below).
BuildContrib.zip in your twiki installation directory.
- Set the ownership of the extracted directories and files to the webserver user.
- Install the dependencies (if any).
Related Topics: TWikiPreferences
|| TWiki:Main.CrawfordCurrie, TWiki:Main.PeterThoeny
|| © 2004-2008, Crawford Currie wikiring.com
© 2004-2013 TWiki:Main.PeterThoeny
© 2004-2013 TWiki:TWiki.TWikiContributor
| Change History:
|| TWikibug:Item7151: Ask all questions upfront on upload
|| TWikibug:Item7123: Admin note in installation instructions not added automatically -- TWiki:Main.PeterThoeny
|| TWikibug:Item6913: Better installation instructions; conditional text for plugin enabling in case extension is a plugin -- TWiki:Main.PeterThoeny
|| TWikibug:Item6800: Fix for using qw(...) as parentheses, which is deprecated in Perl 5.14 -- TWiki:Main.PeterThoeny
|| TWikibug:Item6433: Doc improvements; replacing TWIKIWEB with SYSTEMWEB
|| TWikibug:Item6434: Use ISO dates for builds; doc fixes
|| TWikibug:Item5971 BuildContrib broken upload on 4.2.2 with template login. Also adding query to optionally skip topic attachment upload.
|| TWikibug:Item5556 BuildContrib defaults all unspecified permissions to 664 && TWikibug:Item5455 BuildContrib doesn't cope with larger numbers of files - TWiki:Main.SvenDowideit
|| TWikibug:Item4751 support spaces in MANIFEST TWikibug:Item4990 removed dependency on external md5sum program TWikibug:Item5309 added tidy target
|| TWikibug:Item4600 upload VarXXX topics to Plugins web, to support pluggable documentation architecture. Minor doc improvements. TWikibug:Item3839 add build script at the root to help with building several plugins in one session TWikibug:Item4601 remove duplicate installer script (.pl)
|| TWikibug:Item2006 fixed default permissions to allow group write, and rewrote the main (this) doc to be more useful. TWikibug:Item3624 added a pause between uploads (20s)
|| TWikibug:Item3118 Remember last upload target for each extension you upload. Handle upload to sites that use Template login. TWikibug:Item3445 Carry attributes over for files that are uploaded with the topic, so they remain hidden. Also added date to default version string.
|| TWikibug:Item3597 Duplicated _installer in _installer.pl, so that 4.1 extensions installer can use it
|| Fixed attachment upload. Will now upload attachments attached to the main topic
|| Install made easier to use for developers and end users; will now download an archive if it can't find one locally. Added dependencies target.
|| TWikibug:Item1718 discovered that svn log doesn't work recursively, so more code required to find changes. Added new inline tokens for various URLs. Updated history to 8894.
|| TWikibug:Item1718 added rel=nofollow to generated links to ~develop
|| TWikibug:Item1718 Added
history target to support extraction of history from SVN logs.
|| TWikibug:Item1527 BuildContrib: Remember original filedates for pub files
|| TWikibug:Item1527 BuildContrib: Oops, some (very few) plugins are not ActionTrackerPlugin
|| TWikibug:Item1527 Generated install scripts now really attaches files in pub to topics (means less of these couldn't update history error messages)
|| TWikibug:Item1527 BuildContrib: Execute topic saves in the TWiki web as TWikiAdminGroup by default
|| TWikibug:Item1347 made build.pl tidy up it's mess; and switched buildcontrib over to using push and pop on dirs instead of cd, so we don;t get lost so easily. Oh, and corrected the permissions on the generated viewauth and rdiffauth files
|| TWikibug:Item1393 For release topic name, removed upper case change of suffix. E.g. 'TWiki-4.0.0-beta6' becomes 'TWikiRelease04x00x00beta06'
|| TWikibug:Item1393 added final parameter to build.pl (release name) and added processing to derive topic name from release id.
|| TWikibug:Item663 fixing typos, etc
|| TWikibug:Item1374 BuildContrib wasn't deleting its temporary files staging area
|| TWikibug:Item663 fixing formatting/typos
|| TWikibug:Item1347 changes to support arbitrary release naming, to better support TWiki builds. Should not affect extension builds.
|| TWikibug:Item1285 Removed TWiki_installer.pl from release package
|| TWikibug:Item956 skins have their MANIFEST in lib/TWiki/Contrib/...Skin/
|| TWikibug:Item437 Build.pm reverted - sorry people, it seems this broke the build
|| TWikibug:Item437 Revised BuildContrib to build MD5s. Crawford - feel free to ditch anything you don't like. (I sent you email about this a couple of days ago). MD5 files are generated for each package and then aggregated during a hands-off install to collect all dependent MD5s into package/DEPS.md5
|| TWikibug:Item569 added default RELEASE to everything that had a version, and removed a load of dead code that was getting in the way
|| TWikibug:Item598 removed comment that was getting added to txt files outside data
|| TWikibug:Item569 computed version numbers for plugins from the repository rev they were built from.
|| TWikibug:Item559 fixed permissions
|| TWikibug:Item562 added ability to update VERSION number to current repository head
|| TWikibug:Item561 added a script to build and upload all the plugins, so we can have releases of them from SVN. Presently uploaded to the same topic as the beta releases.
|| TWikibug:Item432 applied the final polish (I hope) before beta release
|| TWikibug:Item437 generated md5 sums for packages built using build.pl, and added them to the upload
|| TWikibug:Item436 incremented vernos of all changed plugins
|| TWikibug:Item421 polishing up installation, trying to catch gotchas, improving docs
|| TWikibug:Item143 more apache warnings; and a silly error in comment plugin tests eliminated
|| TWikibug:Item404 removed CHANGELOG from MANIFEST and added build scripts for all plugins and contribs, so they can be used with the main build.
|| TWikibug:Item384 As excellent as Antonio's solution is, it breaks all the save script tests and is incompatible with previous releases. Need to fall back on the old 'action' parameter if the new parameters are not available - even though it is really bad news (it blocks the use of 'action' as a form-field name :-(. Also updated the script documentation for Antonio's changes.
|| TWikibug:Item380 do as the man says; make all $/ local
|| TWikibug:Item196 Extracted the Manifest File processing from Build.pm to a module that can be used by other Build implementations. Needed for a CommandSet of TWikiShellContrib
|| TWikibug:Item196 more plugin and contrib fixes for develop; mainly just moving tests around and making sure they all pass.
|| TWikibug:Item243 simplified gendocs to stop it generating ,v files, since Sven is going to solve that. Coorected MANIFEST for source code documents.
|| TWikibug:Item243 added build scripts for all plugins shipped with TWikiForGenericUse
|| TWikibug:Item244 extracted target_stage and target_archive from target_release to allow me to add functionality to target_stage
|| TWikibug:Item237 removed requirement to set TWIKI_LIBS for core build
|| TWikibug:Item237 now supports automatic instantiation of plugins and contribs in the release tree. Install scripts are not run.
|| TWikibug:Item237 MANIFEST build based on BuildContrib. Very few extensions to BuildContrib were required to do it. Things that are missing are: 1. generation of ,v files 2. packaging up and inclusion of plugins.
|| TWikibug:Item229 added support for sub-headings in
configure. Added support for adding comments to LocalSite.cfg to BuildContrib. Can now create sections in
configure for individual plugins.
|| TWikibug:Item181 new version, supports DEVELOP and Cairo, and adds support for attachments (you can now mail attachments to topics)