TWiki Developer Environment
Automates build and packaging process, including installer generation, for TWiki extension modules.
Summary of Contents
The BuildContrib can be used to create a build script for your TWiki extension. It is inspired by the Java ANT build tool developed by the Apache project, but is targeted specifically at building TWiki extensions. It is also used for TWiki release builds. The advantage of using BuildContrib is that it dictates a standard structure and build procedure, which makes your extension easier for you, and others, to build and maintain.
Features:
- not just for building code modules, can also be used to package TWikiApplications, and even pure documentation packages.
- automatically generates an installer script that can help simplify end-user installation.
- includes a wizard script for creating a new extension.
- supports creating collections of extensions.
Notes
- This is a build and packaging module for use by developers, not an install module for end users (though it does build an installer script, among other things).
- The module has only been tested on Linux, but should work with Cygwin OK.
- BuildContrib has only been tested with TWiki-4. It might work with earlier TWiki versions.
Fast Start
If you don't like reading documentation, and just want a fast route to creating a new TWiki extension, then:
- Install the contrib using the instructions below
-
cd
to the root of your twiki installation
-
perl create_new_extension.pl
extension_name then modify the extension files as required (including MANIFEST). Then when you are ready to create archives:
-
perl build.pl extension_name release
- Archives (.zip, .tgz) will be created in
extension_name
Development Model
The build module assumes:
- two kinds of extension modules; 'Plugins' (as in TWikiPlugins) and 'Contribs' (everything else e.g Skins, TWikiApplications etc),
- you have some passing familiarity with build tools such as
make
,
- you are not developing your extension in a 'production' TWiki installation (something that is usually a really bad idea), but are instead doing the sensible thing and developing in a separate directory tree
- usually - but not always - a subversion checkout area.
Standard directory structure
BuildContrib is used to build the TWiki core, as well as most extensions. This document will focus on its use for building extensions. See the
tools/build.pl
file in a subversion checkout for information on building the core.
Extensions are developed in a subdirectory below SVN trunk. For example, BathPlugin will be developed in
trunk/BathPlugin
. This directory is called the
root directory for the extension.
The standard directory structure under a root directory mirrors a TWiki installation tree. Every plugin has some
key files:
-
lib/
-
TWiki/
-
Plugins/
-
name.pm
- code file for the plugin, usually derived from EmptyPlugin
-
name/
- directory containing sub-modules used by your plugin, and for storing your build.pl
script and other support files. It is referred to as the module directory
-
build.pl
- build script for this extension
-
MANIFEST
- list of files to be installed
-
DEPENDENCIES
- list of modules this extension depends on
-
Config.spec
- configure
setup for this extension
-
data/
-
TWiki/
-
name.txt
- your plugin/contrib topic
-
test/
-
unit/
-
name/
- directory containing unit tests for the extension
-
pub/
-
TWiki/
-
name/
- directory where all your images, css, and Javascript files should go Contribs are held in the lib/TWiki/Contrib
directory instead of lib/TWiki/Plugins
but otherwise in exactly work the same way.
Other directories normally found in a TWiki installation may also exist under a root directory e.g.
bin
,
templates
etc.
Setting up for Development
The first thing to do is to either
- check out a TWiki development environment from subversion, or
- create a separate TWiki install. Configure this install so it's a running TWiki; we'll refer to this as your development TWiki.
Now install the BuildContrib. In a subversion checkout,
cd
to the TWiki root and
perl pseudo-install.pl BuildContrib
. In a non-subversion environment, install the
BuildContrib
package from TWiki.org.
Environment Variables
Your build script has to know how to find the TWiki libraries, so it can pick up the components of the build system. Set
TWIKI_LIBS
(which is a path, same as
PERL5LIB
) to point to your
lib
directory in your development TWiki.
$TWIKI_LIBS
is used to extend
@INC for the duration of the build only, so it won't mask problems during testing.
The approach we
recommend is to set
TWIKI_LIBS
in your login script (e.g.
.login
,
.csh
,
.profile
depending on what shell you prefer).
EXPERTS build.pl
does not read bin/setlib.cfg
. It uses $TWIKI_LIBS
only to find the modules for the BuildContrib.
Build script
Each individual extension has its own build script, called
build.pl
, in its module directory. A build script is a perl script that takes a number of
targets as its parameters. For example,
perl build.pl test
will run unit tests, and
perl build.pl release
will build a new release.
The build script also accepts the following options:
-n |
Do nothing; just print what you would have done |
-v |
Be verbose |
-topiconly |
with target upload , only upload the topic (not the archives) |
Build targets are Perl functions, which operate on various data defined in control files to build the various targets. Perl is used rather than
make
for portability reasons.
The targets you will normally use are:
build |
perform basic build steps |
compress |
Generate compressed versions of JavaScript and CSS files |
tidy |
Run Perl::Tidy on all perl modules listed in the MANIFEST |
test |
run unit tests |
release |
build , pod and package a release zip |
upload |
build , pod , release and upload |
manifest |
print a guess at the MANIFEST |
history |
Generates a list of svn checkins with comments suitable for use in the history section of the plugin/contrib topic. |
dependencies |
Find and print a best-guess dependencies list (for DEPENDENCIES) |
The default target is
test
. The BuildContrib is designed so that most common behaviour is catered for. It is also easy to
override any of the default targets in your
build.pl
and add extra behaviours.
Note that there is a shortcut script,
core/build.pl
, that can be very helpful when you are building several extensions in one session. Run the script without parameters for documentation.
The build
target
Does nothing by default. This is the first target executed, and can be overridden by your build.pl to do something unusual - for example, executing an ANT file to build some Java.
The compress
target
Usually only used if your extension includes Javascript or CSS, this target looks for
XXX.js
files listed in MANIFEST that have a corresponding
XXX_src.js
in the directory structure. When it finds one, it automatically compresses the
XXX_src.js
file to create/refresh
XXX.js
. If you are using Subversion, the generated file should then by checked in.
Also works on CSS files, for the file extensions
.css
. Compression improves the performance of Javascript and CSS.
Uses
CPAN:JavaScript::Minifier and
CPAN:CSS::Minifier to perform the compression
The tidy
target
This target runs Perl::Tidy (with default formatting options) over your source code. This reformats the code consistently with the TWiki coding standards.
The test
target
The test target is designed for use with extensions that have unit tests written using the UnitTestContrib. It automatically runs the unit tests found in the
test/unit/extension_name
directory.
The release
target
The results of the
release
target are:
- a Zip format archive,
- a gzipped tar archive,
- a md5 checksum,
- the extension topic,
- an installer script
The archives will each contain the following:
- All the files listed in the
MANIFEST
- Another copy of the install/uninstall scripts
The upload
target
This target builds a release, and then tries to upload it to a target repository. The target uploads all the files in the release, and also tries to upload any attachments to the extension topic (as found by scanning the topic for META:FILEATTACHMENT).
You can control what server the upload is done to. This lets you - for example - set up your own corporate extensions server.
The manifest
and dependencies
targets
These are used when you are unsure of the correct contents of MANIFEST and DEPENDENCIES. They make a best-guess at the required contents for these files.
MANIFEST
The
MANIFEST
file contains a list of all the files that are wanted in the package. Each line is a file path, relative to the root of the installation. Wildcards may NOT be used. If the path contains spaces it must be enclosed in double-quotes.
Each file path has an optional octal permissions mask and a description. For example,
data/TWiki/BathPlugin.txt 0664 Plugin description topic
lib/TWiki/Plugins/BathPlugin.pm 0444 Plugin code module
If no permissions are given, permissions are guessed from the permissions on the file in the source tree. These permissions are used by the installer script to set file permissions in the installation.
The following permissions are recommended, and will be applied by default if you don't specify anything different:
File type |
Permissions |
Meaning |
.pm file |
0444 |
Anyone can read, but cannot write or execute |
.pl file |
0554 |
Anyone can read, user and group can also execute |
data/....txt file |
0664 |
Anyone can read, only owner can write |
File in pub/ |
0644 |
ditto |
File in bin/ |
0555 |
Anyone can read or execute, but not write |
Anything other file |
0444 |
Anyone can read, but cannot write or execute |
directories |
0775 |
default directories to traversable |
Do
not include:
-
,v
files. If you include a ,v
file it will overwrite any existing ,v
file when an extension is upgraded, potentially wiping out local changes on the end users installation.
-
build.pl
, MANIFEST
, or any other side file used by the build process.
- unit tests
MANIFESTs can also include other extensions that have been packages using
BuildContrib. For example,
!include ../WysiwygPlugin/lib/TWiki/Plugins/WysiwygPlugin
This will include the WysiwygPlugin in the release package.
Note that there is a script in the TWiki
tools
directory (TWiki 4.2 and later) called
check_manifest.pl
that can be run at any time to check the contents of your MANIFEST against what is checked into Subversion.
DEPENDENCIES
The
DEPENDENCIES
file specifies dependencies on other extensions and perl modules. Each line of the file is a single dependency:
name, version, type, description
where
- name is the name of the module,
- version is the version constraint (e.g. ">1.5"),
- type is its type (cpan, perl, C etc) and
- description is a short description of the module and where to get it.
The installer script written by the build process uses the dependency type to decide how to install dependant modules. 'cpan' means 'get the module from
CPAN' and 'perl' means 'get the module from the Plugins web on TWiki.org' (or whatever other repositories the admin has specified using
$TWIKI_PACKAGES
or
$PLUGINS_URL
).
Calculating DEPENDENCIES
When your module (the
depender) depends on another module (a
dependee), it is important to think carefully about what version of the dependee your module requires.
When you are working with TWiki modules (such as contribs and plugins) you should list the version number of the module that you tested with. Normally you will want to use a
>
condition, so that more recent versions will also work. If a dependency on a TWiki module fails (because the module isn't installed, for example) then the installer script will pull
the latest version of the module from TWiki.org, whether that is the required version or not. This is a limitation of the way plugins are stored on TWiki.org.
When you are working with
CPAN modules, you need to take account of the fact that there are
two types of
CPAN modules;
built-ins and
add-ons.
Built-ins are perl modules that are pre-installed in the perl distribution. Since these modules are usually very stable, it is generally safe to express the version dependency as ">0" (i.e. "any version of the module will do").
Note however that the list of built-in modules is constantly growing with each new release of perl. So your module may be installed with a perl version that doesn't have the required module pre-installed. In this case,
CPAN will
automatically try to upgrade the perl version! There is no way around this, other than for the admin on the target system to
manually install the module (download frm
CPAN and build locally). You can help out the admin by expressing the dependency clearly, thus:
File::Find,>0,cpan,This module is shipped as part of standard perl from perl 5.8.0 onwards. If your perl installation is older than this, you should either upgrade perl, or manually install this module. If you allow this installer to continue, it will automatically upgrade your perl installation which is probably not what you want!
ONLYIF
A dependency may optionally be preceded by a condition that limits the cases where the dependency applies. The condition is specified using a line that contans
ONLYIF ( condition )
, where
condition is a Perl conditional. This is most useful for enabling dependencies only for certain versions of TWiki. For example,
TWiki::Rhinos,>=1.000,perl,Required. Download from TWiki:Plugins/RhinosContrib and install.
ONLYIF ($TWiki::Plugins::VERSION < 1.025)
TWiki::Plugins::CairoContrib, >=1.000, perl, Optional, only required if the plugin is to be run with versions of TWiki before Cairo. Available from the TWiki:Plugins/CairoContrib repository.
Thus CairoContrib is only a dependency if the installation is being done on a TWiki where the
TWiki::Plugins::VERSION
is less than 1.025. The
ONLYIF
only applies to the next dependency in the file.
Writing a build script
The easiest way to write a new build script is to use the
create_new_extension.pl
script, which is part of the BuildContrib.
- Create your plugin source tree under
trunk
:
-
cd core
-
perl create-new-extension.pl BathPlugin
- Edit
lib/TWiki/Plugins/BathPlugin.pm
as required to create your plugin functionality
- Edit
lib/TWiki/Plugins/BathPlugin/MANIFEST
and make sure it lists all the files you want to include in the release package
During development we recommend you use the
pseudo-install.pl
script to soft-link your plugin code into your development TWiki. This script uses the MANIFEST you write and creates softlinks in your development TWiki that allow you to run your test code without having to do a full re-install each time you make a change.
If you have a pre-existing extension, and you want to package it for use with BuildContrib, then you need to create the module directory and write the
build.pl
,
MANIFEST
and
DEPENDENCIES
files. The easiest way to do this is to copy those files from an existing extension in subversion, and modify
them for your extension.
Preparing the Installer
The installer script generated by the builder when target
release
is used is based on a template. This template is populated with lists of files and dependencies needed to make the extension-specific installer script.
You can extend this script by providing
PREINSTALL
,
POSTINSTALL
,
PREUNINSTALL
, and/or
POSTUNINSTALL
files in the module directory. These optional files are embedded into the template install script at the appropriate stage of the installation. Read
lib/TWiki/Contrib/BuildContrib/TEMPLATE_installer.pl
(in the BuildContrib) to see how they fit in. The POD comments in that module indicate the functions that are most likely to be useful to anyone writing a script extension.
Testing
You are stongly recommended to develop a unit test suite for your extension. Unit tests are kept in the
test/unit/<name>
directory for each extension.
To run the unit tests you will need to set up the test environment described in
TWiki:Codev/TestCasesTutorial#SettingUpATestEnvironment. That page also includes extensive information on writing testcases.
The easiest way to generate tests for your extension is to copy the approach taken in another extension. See for example ActionTrackerPlugin and CommentPlugin, which both have extensive test suites.
Tests are run using
Building a release
When you are almost ready to release, you should
- Build a release package and installer
-
cd ../BathPlugin/lib/TWiki/Plugins/BathPlugin
-
perl build.pl release
- Remove the softlinked version from your development TWiki
-
cd
development TWiki
-
perl pseudo-install.pl -uninstall BathPlugin
- Install the release package you just built:
-
cd
development TWiki
-
perl BathPlugin/BathPlugin_installer
- Test.
Token expansion
The
release
target automatically expands certain tokens in
.txt
files and in the installer script. The following tokens are supported:
-
%$MANIFEST%
- TWiki table of files in MANIFEST
-
%$FILES%
- hash keyed on file name mapping to permissions i.e. 'data/TWiki/ThsiTopic.txt' => 0664, 'lib/TWiki/Plugins/BlahPlugin.pm' => 0775
-
%$DEPENDENCIES%
- list of dependencies from DEPENDENCIES
-
%$VERSION%
value of $VERSION
-
%$DATE%
- local date
-
%$POD%
- POD documentation for the package, excluding test modules.
-
%$PREINSTALL%
- contents of PREINSTALL
-
%$POSTINSTALL%
- contents of POSTINSTALL
-
%$PREUNINSTALL%
- contents of PREUNINSTALL
-
%$POSTUNINSTALL%
- contents of POSTINSTALL
-
%$BUGSURL%
- URL of bugs web
-
%$INSTALL_INSTRUCTIONS%
- basic instructions for installing
You can also create other tokens to be expanded by adding them in your
build.pl
. For example,
TODO: add code sample here?
Upload
When you are happy the release package is built correctly, you can upload it.
-
cd BathPlugin/lib/TWiki/Plugins/BathPlugin
-
perl build.pl upload
By default the
upload
target will upload to TWiki.org. You will be prompted to enter an alternate upload target, should you require it (e.g. to upload to private corporate repository). The upload updates the topic and any associated Var topics published by the extension, and uploads zip, tgz, md5 and installer files.
Install support
Installer scripts build by BuildContrib are important for the full functioning of the extensions installer in
configure
.
The installer script shipped with the package is very simple. By default all it does is to check the dependencies you list, and if necessary download and install any missing TWiki and
CPAN modules. Other dependencies are simply checked. In TWiki-4.0 and later releases, TWiki topics shipped with the module are automatically merged into any existing local copies, ensuring histories are preserved.
If you want your installer to do anything else then you will need to write a
POSTINSTALL script.
Installation Instructions
You are strongly recommended to use this Contrib to help split your code development away from your live TWiki environment, as described
above.
- For an automated installation, run the configure script and follow "Find More Extensions" in the in the Extensions section.
- Or, follow these manual installation steps:
- Download the ZIP file from the extension home on twiki.org (see below).
- Unzip
BuildContrib.zip
in your twiki installation directory.
- Set the ownership of the extracted directories and files to the webserver user.
- Install the dependencies (if any).
Contrib Info
Authors: |
TWiki:Main.CrawfordCurrie, TWiki:Main.PeterThoeny |
Copyright: |
© 2004-2008, Crawford Currie wikiring.com © 2004-2021 TWiki:Main.PeterThoeny © 2004-2021 TWiki:TWiki.TWikiContributor |
License: |
GPL |
Version: |
2021-03-25 |
Change History: |
|
2021-03-25: |
TWikibug:Item7927: Copyright update to 2021 |
2019-05-04: |
TWikibug:Item7862: Fix build upload using https instead of http |
2018-07-08: |
TWikibug:Item7841: Copyright update to 2018 |
2016-01-14: |
TWikibug:Item7708: Copyright update to 2016 |
2013-03-22: |
TWikibug:Item7151: Ask all questions upfront on upload |
2013-01-09: |
TWikibug:Item7123: Admin note in installation instructions not added automatically -- TWiki:Main.PeterThoeny |
2012-10-06: |
TWikibug:Item6913: Better installation instructions; conditional text for plugin enabling in case extension is a plugin -- TWiki:Main.PeterThoeny |
2011-09-11: |
TWikibug:Item6800: Fix for using qw(...) as parentheses, which is deprecated in Perl 5.14 -- TWiki:Main.PeterThoeny |
2010-05-15: |
TWikibug:Item6433: Doc improvements; replacing TWIKIWEB with SYSTEMWEB |
2010-04-25: |
TWikibug:Item6434: Use ISO dates for builds; doc fixes |
2008-08-31: |
TWikibug:Item5971 BuildContrib broken upload on 4.2.2 with template login. Also adding query to optionally skip topic attachment upload. |
2008-04-22: |
TWikibug:Item5556 BuildContrib defaults all unspecified permissions to 664 && TWikibug:Item5455 BuildContrib doesn't cope with larger numbers of files - TWiki:Main.SvenDowideit |
2008-01-31: |
TWikibug:Item4751 support spaces in MANIFEST TWikibug:Item4990 removed dependency on external md5sum program TWikibug:Item5309 added tidy target |
2007-09-10: |
TWikibug:Item4600 upload VarXXX topics to Plugins web, to support pluggable documentation architecture. Minor doc improvements. TWikibug:Item3839 add build script at the root to help with building several plugins in one session TWikibug:Item4601 remove duplicate installer script (.pl) |
13599 |
TWikibug:Item2006 fixed default permissions to allow group write, and rewrote the main (this) doc to be more useful. TWikibug:Item3624 added a pause between uploads (20s) |
12953 |
TWikibug:Item3118 Remember last upload target for each extension you upload. Handle upload to sites that use Template login. TWikibug:Item3445 Carry attributes over for files that are uploaded with the topic, so they remain hidden. Also added date to default version string. |
12821 |
TWikibug:Item3597 Duplicated _installer in _installer.pl, so that 4.1 extensions installer can use it |
11505 |
Added twikiplugins/create_new_extension.pl script |
11492 |
Fixed attachment upload. Will now upload attachments attached to the main topic |
11478 |
Install made easier to use for developers and end users; will now download an archive if it can't find one locally. Added dependencies target. |
8947 |
TWikibug:Item1718 discovered that svn log doesn't work recursively, so more code required to find changes. Added new inline tokens for various URLs. Updated history to 8894. |
8894 |
TWikibug:Item1718 added rel=nofollow to generated links to ~develop |
8892 |
TWikibug:Item1718 Added history target to support extraction of history from SVN logs. |
8585 |
TWikibug:Item1527 BuildContrib: Remember original filedates for pub files |
8584 |
TWikibug:Item1527 BuildContrib: Oops, some (very few) plugins are not ActionTrackerPlugin |
8583 |
TWikibug:Item1527 Generated install scripts now really attaches files in pub to topics (means less of these couldn't update history error messages) |
8581 |
TWikibug:Item1527 BuildContrib: Execute topic saves in the TWiki web as TWikiAdminGroup by default |
8365 |
TWikibug:Item1347 made build.pl tidy up it's mess; and switched buildcontrib over to using push and pop on dirs instead of cd, so we don;t get lost so easily. Oh, and corrected the permissions on the generated viewauth and rdiffauth files |
8359 |
TWikibug:Item1393 For release topic name, removed upper case change of suffix. E.g. 'TWiki-4.0.0-beta6' becomes 'TWikiRelease04x00x00beta06' |
8346 |
TWikibug:Item1393 added final parameter to build.pl (release name) and added processing to derive topic name from release id. |
8263 |
TWikibug:Item663 fixing typos, etc |
8217 |
TWikibug:Item1374 BuildContrib wasn't deleting its temporary files staging area |
8216 |
TWikibug:Item663 fixing formatting/typos |
8209 |
TWikibug:Item1347 changes to support arbitrary release naming, to better support TWiki builds. Should not affect extension builds. |
8061 |
TWikibug:Item1285 Removed TWiki_installer.pl from release package |
7531 |
TWikibug:Item956 skins have their MANIFEST in lib/TWiki/Contrib/...Skin/ |
7215 |
TWikibug:Item437 Build.pm reverted - sorry people, it seems this broke the build |
7207 |
TWikibug:Item437 Revised BuildContrib to build MD5s. Crawford - feel free to ditch anything you don't like. (I sent you email about this a couple of days ago). MD5 files are generated for each package and then aggregated during a hands-off install to collect all dependent MD5s into package/DEPS.md5 |
6827 |
TWikibug:Item569 added default RELEASE to everything that had a version, and removed a load of dead code that was getting in the way |
6791 |
TWikibug:Item598 removed comment that was getting added to txt files outside data |
6759 |
TWikibug:Item569 computed version numbers for plugins from the repository rev they were built from. |
6757 |
TWikibug:Item559 fixed permissions |
6743 |
TWikibug:Item562 added ability to update VERSION number to current repository head |
6742 |
TWikibug:Item561 added a script to build and upload all the plugins, so we can have releases of them from SVN. Presently uploaded to the same topic as the beta releases. |
6515 |
TWikibug:Item432 applied the final polish (I hope) before beta release |
6512 |
TWikibug:Item437 generated md5 sums for packages built using build.pl, and added them to the upload |
6504 |
TWikibug:Item436 incremented vernos of all changed plugins |
6470 |
TWikibug:Item421 polishing up installation, trying to catch gotchas, improving docs |
6450 |
TWikibug:Item143 more apache warnings; and a silly error in comment plugin tests eliminated |
6426 |
TWikibug:Item404 removed CHANGELOG from MANIFEST and added build scripts for all plugins and contribs, so they can be used with the main build. |
6369 |
TWikibug:Item384 As excellent as Antonio's solution is, it breaks all the save script tests and is incompatible with previous releases. Need to fall back on the old 'action' parameter if the new parameters are not available - even though it is really bad news (it blocks the use of 'action' as a form-field name :-(. Also updated the script documentation for Antonio's changes. |
6353 |
TWikibug:Item380 do as the man says; make all $/ local |
6240 |
TWikibug:Item196 Extracted the Manifest File processing from Build.pm to a module that can be used by other Build implementations. Needed for a CommandSet of TWikiShellContrib |
6092 |
TWikibug:Item196 more plugin and contrib fixes for develop; mainly just moving tests around and making sure they all pass. |
6077 |
TWikibug:Item243 simplified gendocs to stop it generating ,v files, since Sven is going to solve that. Coorected MANIFEST for source code documents. |
6076 |
TWikibug:Item243 added build scripts for all plugins shipped with TWikiForGenericUse |
6068 |
TWikibug:Item244 extracted target_stage and target_archive from target_release to allow me to add functionality to target_stage |
6051 |
TWikibug:Item237 removed requirement to set TWIKI_LIBS for core build |
6040 |
TWikibug:Item237 now supports automatic instantiation of plugins and contribs in the release tree. Install scripts are not run. |
6039 |
TWikibug:Item237 MANIFEST build based on BuildContrib. Very few extensions to BuildContrib were required to do it. Things that are missing are: 1. generation of ,v files 2. packaging up and inclusion of plugins. |
6026 |
TWikibug:Item229 added support for sub-headings in configure . Added support for adding comments to LocalSite.cfg to BuildContrib. Can now create sections in configure for individual plugins. |
5960 |
TWikibug:Item181 new version, supports DEVELOP and Cairo, and adds support for attachments (you can now mail attachments to topics) |
Related Topics: TWikiPreferences