Feedback on BuildContrib

1. I have checked in a change that adds a target 'manifest'; this prints to STDOUT a likely result for your manifest file
2. Would it be a good idea to make the files installed by target_install read only? A couple of times now I have lost a change through accidently editing the live copy rather than the plugin development directory version.

-- MartinCleaver - 20 Aug 2004

We could, but that's a "working practices" kind of thing; you have been used to editing in the live code. I've never worked that way, so it makes very little difference to me, though I do occasionally do a quick hack in the live code to check something.

I'd rather you adjusted your working practices

-- CrawfordCurrie - 23 Aug 2004

I wasn't on an operational server, Crawford!

Also... we could do with a bin directory for BuildContrib, so that we can put commands in the user's PATH. BuildContrib/contrib/bin? BuildContrib/tools/bin? or in BuildContrib/bin

-- MartinCleaver - 24 Aug 2004

Then it wasn't "live" then, was it? >:-)

contrib/bin is fine, though I don't see it in my CVS checkout?? I try to did start the argument that there should be a top level bin directory for non-CGI scripts, but it fell on deaf ears. This is a good compromise.

I know it's only experimental, but in order to get quality feedback TWikiCLI needs more documentation;

• what do all the environment variables mean?
• Is "extension install" the onlt command it supports?
• What does "extension install" actually do?

I tidied it up a bit, but it needs you to add some semantic content.

Someone needs to make a proposal how it can check and satisfy (or at least warn of) dependencies.

-- CrawfordCurrie - 28 Aug 2004

BuildContrib now automatically generates an install script - or to be more accurate, a post install script. This script, when run after the unzip, will detect dependencies, and for those that are on twiki.org, offer to download and install them.

It's quite limited at the moment, but it has these advantages:

1. Automatically resolves dependencies on TWiki contribs
2. Driven from the dependencies table declared to checkDependencies (if it follows the BuildContrib standard) and the DEPENDENCIES
3. Detects dependencies on perl modules from CPAN and reports them
4. Automatically generated for any Plugin/Contrib that is built using BuildContrib
5. Has a built-in uninstall, though it doesn't uninstall dependencies
6. Can be run outside the TWiki installation (i.e. in a "safe" area) if the unzip-copy install methodology is required.

See CommentPlugin for an example of the documentation for use of it.

Some ideas for improvement:

• Installer
  • Invoke CPAN shell to install those dependencies -- CrawfordCurrie
  • If TWIKI_HOME is pointed at an installation, then add it to the search path for dependencies. -- CrawfordCurrie
  • Keep track of dependencies: Plugins dump a file in each contrib module's dependents directory (done automatically with unzip operation). -- PeterThoeny
  • Alternatively, record a count of number of times a dependency is used. -- PeterThoeny
  • Check in .txt and files in pub, retaining installation version history -- PeterThoeny
• Uninstaller
  • Use dependency info/count to offer to eliminate dead code -- PeterThoeny
  • Rveert histories in .txt and files in pub -- PeterThoeny

-- CrawfordCurrie - 02 Sep 2004

After putting all that effort into porting to LWP, I now find that perl is commonly distributed without LWP! Certainly, a fresh install of SuSE 9.1 Personal does not have it. So I guess a strategy for falling back on curl or wget is needed, and if those fail, falling back to prompt/confirm.

Grrr.

-- CrawfordCurrie - 18 Sep 2004

The TWikiShell / TWikiCLI stuff I built into BuildContrib is now in its own TWikiShellContrib. I will remove this functionality from BuildContrib soon.

-- MartinCleaver - 22 Oct 2004

I have now removed this from the CVS version.

-- MartinCleaver - 04 Dec 2004

See my note on UsingCVSForPluginsDevelopment - I think we ought embed this in EmptyPlugin.

-- MartinCleaver - 04 Dec 2004

I've been told on #perl that http://module-build.sourceforge.net/META-spec-new.html e.g. http://www.joshr.com/src/sman/Sman-0.9.1/META.yml is the standard way to write our DEPENDENCIES file

-- MartinCleaver - 08 Jan 2005

I tried to deploy this Contrib to package up plugins I have recently developed. Here is my experience so far:

1. Installed contrib and dependencies.
2. Created a build.pl file in the appropriate directory.
3. Set TWIKI_LIBS environment variable. However, documentation was unclear, so I did not get it right.
4. Emailed CrawfordCurrie who set me straight: use <path-to-twiki>/lib
5. cd into build directory and type perl build.pl manifest in the hope this would create the manifest.
6. Build aborts with uncaught exception, complains about <path-to-twiki>/MANIFEST missing
7. Create empty <path-to-twiki>/MANIFEST file
8. Building manifest does nothing (visibly), in particular, does not generate the expected file list
9. Manually populate the manifest with the needed filtes.
10. Now the build script fails with the message COULD NOT LOCATE TEMPLATE_installer.pl - required for install script generation at...
11. Emailed CrawfordCurrie again, who sent the attached template which was missing in the distribution. This template must go into the contrib/ directory.
12. Now the build runs until almost the very end, when the script tries to remove temporary directories it had created. Somehow that removal fails, claiming the device is busy.
13. Looking into the code, I find that the script cd's into the directory to be removed before...
14. Edited Build.pm to cd out of the tmp directory before removal (see attached patch).
15. Now everything runs through and the release is generated.

-- ThomasWeigert - 16 Mar 2005

One more comment... I think we should remove the files left behind in the TWiki directory other (e.g., the generated documentation file).

-- ThomasWeigert - 15 Mar 2005

Thanks Thomas. I've been guilty of not keeping the release package up to date. Those cd's are not in my copy of Build.pm, nor in CVS.

OK, I fixed the documentation I hope, and the manifest code (which was frankly very dim) does something useful now. I also added the template installer to the package and while I was at it, I made it a lot smarter while installing into DEVELOP code; it can now check in the files it installs in pub and data (it does this by creating a stand-alone instance of the "TWiki" object and using it to saveTopic the files. This raises the interesting point that the beforeSaveHandler (beforeAttachmentSaveHandler) will be called on any files so checked in).

-- CrawfordCurrie - 15 Mar 2005

Crawford, may I suggest that you take the MANIFEST file from $buildpldir, rather than $basedir? I.e., use

    my $manifest = "$buildpldir/MANIFEST"; 

Or, alternatively, move build.pl also into $basedir (maybe that is the better solution). But at this point this is a confusing inconsistency.

-- ThomasWeigert - 20 Mar 2005

Further, when I use this scheme on Plugins, rather than Contribs, I get an error message during the run:

WARNING: lib/TWiki/Plugins/Build.pm not found; cannot extract VERSION or in-code dependencies 

It seem odd that Build.pm has to be installed in both the Contrib and Plugins directories.

-- ThomasWeigert - 20 Mar 2005

Crawford, I made some enhancements to the NewContribTemplate and created supporting Contrib topics. With this I changed the BuildContrib topic text:

• Heading on top, followed by short description
• Added SHORTDESCRIPTION in "Settings" section
• Added a note to the "Installation Instructions" section
• Added link to GPL in "License" table row
• Added "Appraisal" table row
• Fixed heading levels (use level one only once on top)

Cold you take that into the next release of the package?

-- PeterThoeny - 22 Mar 2005

Crawford, could we add something into the install script so that it sets the executable bit on all files put into the bin/ directory?

-- ThomasWeigert - 25 Mar 2005

Thomas, it looks in the build.pl dir and also in the root dir now. Build.pm should only be in the Contrib directory. Are you sure you are not trying to "use TWiki::Plugins::Build"?

Peter, Darnit, I just uploaded a minor rev and probably overwrote your changes; I'll have a look.

Thomas, It's a right PITA, I agree, but I haven't found a way to do it yet.

-- CrawfordCurrie - 25 Mar 2005

Crawford, the cd problem pointed out earlier is back. The attached patch from lasttime is probably still good.

Also, now you require a DEPENDENCIES file, whether there are dependencies or not.

-- ThomasWeigert - 27 Mar 2005

The BuildContrib* (home, *Dev and *Appraisal) table row entries are missing.

-- PeterThoeny - 28 Feb 2006

The heading levels are off (only first one on top should be level 1)

The left column of the Info Table has right & left justified cells. Should all be right justified.

-- PeterThoeny - 28 Feb 2006

How are dependencies installed? For instance, TwistyPlugin has the dependency of TwistyContrib. How to let the installer know that TwistyContrib should be installed along with TwistyPlugin?

-- ArthurClemens - 04 Jun 2006

I moved above comment from the Contrib topic to this Dev topic.

Arthur added this to the Environment Variables section:

Linux: set these variables from the root of your twiki installation with:

export TWIKI_HOME="$(pwd)"; export TWIKI_LIBS="$(pwd)/lib"

This should be taken into the release.

-- PeterThoeny - 22 Jul 2006

The setConfig function of the TEMPLATE_installer undefs the input record separator, $/, and this causes subsequent calls to prompt wait for ever (or ctrl + d, on linux). I fixed this problem here saving the $/ variable and backing up it after each call to setConfig. A better way to fix it is to modify the setConfig function to do this. Another possible solution is to localize $/ at the beggining of setConfig. I didn't test te later. See patch attached.

-- GilmarSantosJr - 26 Jul 2006

Crawford, File::Find attempts to install perl 5.9.4 which CPAN says is an unstable development version.

Can we make sure that this plugin does not make such unreasonable requirements?

-- ThomasWeigert - 19 Aug 2006

Not sure how, since the plugin installation is just installing a CPAN module (it doesn't express any version constraints when it does the install from CPAN). Please raise this with the developer of File::Find.

Note that any module we use from CPAN might be changed by the developer to require a later perl version. Because CPAN is OSS, the only things we can do are (1) to stop using any CPAN modules (which is just silly) or (2) to work with the module developer to flag this sort of error.

-- CrawfordCurrie - 19 Aug 2006

There is a middle ground: Use CPAN modules as needed, and avoid where feasible.

-- PeterThoeny - 19 Aug 2006

Crawford, do you know what this module is needed for? What can a user do when they want to install BuildContrib but they get a frightening message such as that their perl is going to be overwritten with an "experimental version" and that they should not do that? Not being able to install BuildContrib is probably not a great situation?

-- ThomasWeigert - 20 Aug 2006

File::Find is used by the manifest target to find files for inclusion in the MANIFEST.

There is something wierd going on here. File::Find is a standard perl module - AFAICT it has been included in the standard perl distribution since 5.6. You shouldn't need to install it, unless you are using a pre-5.6 perl ( not recommended, we do not test on perl earlier than 5.8.4 ) The module itself does a require 5.006 which suggests it's author doesn't expect it to require a later perl either. Yet when I do a perl -MCPAN -e shell and install File::Find sure enough, it tries to upgrade my perl to 5.9.4.

I assume this is a packaging problem on CPAN, which might affect any perl module.

Later: I discussed this with the people on #perl, and the answer was:

08:30:49) CDot: presumably there is some logic in there that says "if this is a built in, and it was built for a later perl, then try to upgrade the perl"
(08:31:05) CDot: because other modules don't try to force-upgrade the perl version
(08:31:27) buu: CDot: Not really..  not even that complicated. It's a distribution issue.
(08:31:43) hobbs: CDot: it's just that it sees that the distribution that provides the given module is the one named "perl"
(08:31:49) hobbs: CDot: so it tries to install latest perl
(08:31:51) buu: CDot: File::Find for example is only distributed as part of the core, so it finds File::Find in the distro named perl.
This is mapped to perl5.9.4 (the latest release).

This is unlikely to be fixed in CPAN any time soon.

So the question in your case has to be; why is the installer trying to upgrade your File::Find? The dependency in BuildContrib is on version 1.05 of File::Find, which is pretty old, but the installer things that is newer than your version (can you check please? perl -e 'use FIle::Find; print File::Find::VERSION,"\n"').

The logic goes something like this. Thomas has asked to install BuildContrib, which uses the built-in module File::Find. We know that his existing version of File::Find is older than the required version, 1.05, because the installer script is trying to upgrade it. Since 1.05 was shipped with perl 5.8.1, his perl version must be older than this.

This points to a systemic problem. Any dependency on a standard module will, when the dependency is upgraded, require an upgrade to the latest release of perl. The only option is to install exactly the version listed in DEPENDENCIES, and the CPAN interface doesn't support that.

-- CrawfordCurrie - 20 Aug 2006

The diagnosis sounded reasonable, until it told me the wrong version of perl I am running. Based on perl -V I am running 5.8.7. Also, my File::Find version is 1.09, again defeating this diagnosis (by the way, you are missing the dollar sign when looking up the File::Find version).

So the long and short is that I am running a version of File::Find that is higher than BuildContrib is looking for, but nevertheless, BuildContrib is trying to update File::Find.

Let me see whether something went wrong in the information I gave you that led us down the wrong route....

Yesterday, I tried to install using BuildContrib_installer.pl. This failed by trying to update perl. However, I did not record which module BuildContrib was trying to install when that happened. Then I tried to install the modules separately, in the order that they were listed on BuildContrib. So the first one I tried was File::Find and again observed that it tried to update perl.

After your analysis above I looked into BuildContrib_installer.pl and found that it states that version 1000000 was needed for File::Path. And sure enough, when I was running BuildContrib_installer.pl again, it told me that I only had version 1.07 of File::Path and not the gigantic version required (which I take is just a way of saying, give me the latest). Now the logic that Crawford described kicks in....

So the real question is: What version of File::Path do we depend on? Then we should upgrade BuildContrib_installer.pl to that version. That would solve this problem without having to reprogram the CPAN interface. For now I changed my BuildContrib_installer.pl dependency on File::Path to 1.07 and that allowed the install to proceed.

-- ThomasWeigert - 20 Aug 2006

Makes sense. yes, 1000000 just means the latest, and this will trigger the upgrade of perl for a built-in :-(. Here's how I usually guesstimate dependency versions:

1. I use the version installed on my test system as the baseline
2. I visit CPAN and cruise back through versions to find the earliest that I think "makes sense" (usually the version shipped with 5.8.0, though it can be hard to tell; CPAN authors are lazy about maintaining accurate histories).

Cruising CPAN is easy; for example, for File::Path, visit

http://search.cpan.org/dist/perl-5.8.4/lib/File/Path.pm

Then manually change the version number in the browser bar to review what was shipped with different dists of perl.

The big problem is that I test with 5.8.4 (my installed version) so while I can use this approach to calculate a dependency on an earlier rev of the required modules, I can't actually test it. Once again we are back at the place where we can't claim to have tested earlier perl releases.

The only strategy I can think of for built-in modules is to list the dependency as ">0" - i.e. test that there is a version (any version) of the module installed. If the module is included in your perl distribution, then this will pass, and you won;t be asked to upgrade perl. If you are running an earlier perl without the module, then you will still be prompted to upgrade your perl; I can't see any way to avoid this.

Later: I worked out how to use CPAN to determine if a module is a standard perl module or not. I have coded this into the next release of this contrib.

-- CrawfordCurrie - 21 Aug 2006

Great. Note that this problem does not exist with most CPAN modules. It is only in the various File modules you include in BuildContrib that these surfaced....

-- ThomasWeigert - 21 Aug 2006

Crawford, I am terribly sorry to trouble you with this, but...

I cannot get the installers to work. I managed to use the release and upload functions of BuildContrib, but when I try to use the generated installer to actually install a plugin into a copy of my TWiki, it keeps failing on the "chmod" calls. I must also be not understanding something right, as I cannot see the code where the archive is actually unzipped or the files are being otherwise copied into the installation. Please advise...

-- ThomasWeigert - 22 Aug 2006

The archive is not unzipped by the installer, that is a manual step. The installer works after the archive has been unzipped. All the installer does is to verify that dependencies are satisfied, and to check in new versions of topics.

the installation instructions usually read:

• Download the archive (tgz or zip) from TWiki.org
• Unpack the archive in the root of your TWiki installation
• Run perl BuildContribDev_installer.pl to automatically resolve any dependencies
  • Alternatively, manually satisfy the dependenices listed below

The new code in configure (the extension installer) automates these steps.

-- CrawfordCurrie - 22 Aug 2006

This is very confusing. When I look at the installer code, there are unpack functions. It also tries to install dependencies, e.g., another TWiki plugin, albeit it is not successful in changing the permissions.

So why is the Plugin archive unzipped if it is listed as a dependency, but not, if it is the plugin that we want to install?

Further, as what user does the installer run (this is on Windows). I have a file in the pub archive (no RCS, just the file), and I am getting an error from RcsFile that it cannot open that file (it was created with 0440 but the installer is trying to change to 0660 eventually). I have manually set the directory where this file goes to 0777.

-- ThomasWeigert - 22 Aug 2006

sigh The installer does not unzip the zip, because I have been pandering to the belief that the installation process should be entirely manual. I have wanted to shift to self-extracting archives for a while now, but just haven't found the time to do it, sorry. You know what it's like.

If you use the extension installer interface in configure, all this is hidden from you.

I can't advise on use of permissions on Windows, I'm afraid; I don't have a Windows box that supports them. On Unix, creating a file with 0440 does't stop the owner of the file changing the permissions, though. As long as the installer script is run by the same person who unpacked the archive, it all works fine. Does Windows do something different? If so, how are you supposed to get a file back once you change your write permissions away?

-- CrawfordCurrie - 23 Aug 2006

On the unzipping... when something is a dependendcy, it is being unzipped (actually, the tar file is used, apparently). Why would we automatically do that for dependencies, but not for the real thing that we are trying to install? What is the difference?

-- ThomasWeigert - 23 Aug 2006

If you recall, the original approach was to install by unzipping over your TWiki installation, end of story. I wanted an installer, but met resistance to the idea of requiring an installer script (self extracting executable, whatever). So I compromised, and shipped a script in the zip that would resolve further dependencies on other zips. That script has grown over time until it is capable of doing a full install of other modules. I maintained the orginal manual install in support of people trapped behind corporate firewalls.

The full manual install has a number of serious problems.

1. every time you unzip, you unwittingly risk overwriting something in your install tree.
2. there is no way to check in new versions of files that have a ,v in the existing install (if you ship a ,v that's even worse, as you overwrite the local history).
3. there is no automatic checking, much less resolution, of dependencies.
4. there is no way to re-route installed files to other directories if a local install renames, for example, the Main or TWiki webs.

So a scripted install is really the only sensible option. What we really need is to continue to ship "unwrap and go" style zips, however brainless this approach is, but also ship a smart install script that can either be told to get components from CPAN/twiki.org, or to use local zips if available. This script could then be shipped outside the extension zip. So for someone not hidden by a firewall, the install would look something like this:

$ cd /usr/local/twiki
$ wget http://twiki.org/p/plugins/VrmlPlugin/VrmlPlugin_installer.pl
$ perl VrmlPlugin_installer.pl
.....found an existing archive ./VrmlPlugin.tgz. I can install this archive, or I can download a fresh one.
Use existing archive? [y/n] n
.....downloading http://twiki.org/p/plugins/VrmlPlugin/VrmlPlugin.tgz
.....unpacking the archive
.....resolving dependencies
VrmlPlugin needs to know the path to your X11 installation.
Path to the root of your X11 (hit return to accept /usr/lib/X11):
VrmlPlugin installed. You will have to enable this plugin in =configure= before you can use it.

Obviously we could support options to get the zips from another directory, so an admin behind a firewall could download en masse.

I have been working towards this slowly, but without much help (or even positive vibes) it's a slow process.

-- CrawfordCurrie - 24 Aug 2006

I guess the biggest concern about the installer is that we need to be sure that it actually works. For example, I have not been able to get the installs working on a windows machine (see above: it downloads dependencies, but fails on chmod those and it cannot chmod the component being installed). I see comments by Ken also to that effect on a unix box. The manual approach has the advantage of being certain to work, albeit it is more tedious.

Another thing that would help getting trust in the installer would be to be able to see what it is going to do. In case of problems, one can then recreate. The current -n option does not really work like -n in make in that once there are directories that should be created (but are not due to the -n, it will fail; there may be other such places also).

By the way, on more recent version of CPAN (or our firewall) I can install CPAN modules from behind the firewall....

-- ThomasWeigert - 24 Aug 2006

moved from TimelinePluginDev:

I manually set the hide flag of the attachments on the plugin topic that are not needed when downloading the plugin (images etc.). This is to avoid confusion for users. How about enhancing the BuildContrib to have control over the hide flag?

-- PeterThoeny - 03 Sep 2006

Yes, would be nice. It is already in the bug base as Bugs:Item1739.

-- SteffenPoulsen - 04 Sep 2006

Today I've tried, for the first time, to create a contrib with BuildContrib, on a plain SVN installation, trying to follow the steps in SVN's six-step recipe. I guess that all of you know this, but for first-timers there are a couple of gotchas:

1. In a SVN area, Remember that you need to perl pseudo-install.pl -link BuildContrib at least once.
2. SVN's BuildContrib topic says %$POD% where the interesting part of the documentation begins. I needed to consult TWiki:TWiki.BuildContrib, to find out about the environment variables TWIKI_LIBS and TWIKI_HOME. Can the doc be "built" somehow?
3. Testing on a web server where I am not root, and where the server runs under another user id, is a nightmare because of permissions. It is almost impossible when using .tgz archives which preserve ownership: Normal users are not permitted to chown on (SUSE) Linux.
4. It is recommended to develop in a SVN area? Well - if so, then at least at some point in the process, a developer should svn add his stuff. When doing perl build.pl release it complained
   svn: '/home/haj/public_html/TWikiRelease04x00/twikiplugins/BenchmarkContrib/bin' is no working copy
   Which files and directories must be svn add ed?
5. I see this has already been observed: The installer tries to download my zip from twiki.org (where it isn't), but fails to check success and just continues.

Perhaps I'll manage to svn commit tomorrow...

-- HaraldJoerg - 06 Sep 2006

1. Thanks, Harald, I'll update the docs accordingly.
2. Hmmmm. Need to think about that. Though you can always perl build.pl release and then read the topic it creates. Or install the zip.
3. Any ideas?
4. Yes. All files involved in the extension must be added.
5. Yeah, I know. Working on it.

-- CrawfordCurrie - 07 Sep 2006

About perl build.pl release for BuildContrib: Yes, that would earn both the current SVN information, and the expansion of the variables - at the expense of having to install BuildContrib on the test server. Good idea.

About the permissions: Sorry, No, I don't have an idea. I only had to find out with regard to my contrib that the idea of bringing its own web with it must fail in that respect. Usually you want the code to be owned by the installer (e.g. root) and readonly for the rest, and the data owned by the userid running the web server. Come to think of it - the plugin description topic has the same problem if it isn't readonly (and it seldom is, due to plugin preferences).

I came across some ideas of writing an 'import' CGI which would copy, under the web server's control, all files I provide in some installation directory. The installer could fire off the install CGI using LWP, if he finds TWiki's configuration and if the host name is correctly set, and if it doesn't conflict with TWiki permissions in the data directory. Too many ifs to be production grade.... So for the moment I'll fall back to the old habit of using root and doing chmod and chown as needed.

About svn add: Maybe build.pl could have a target to fire off the SVN commands for all files in the MANIFEST, plus those automatically created by create_new_plugin.pl?

-- HaraldJoerg - 07 Sep 2006

I've found 2 problems with BuildContrib on BSD.

In prot() function, $perms needs to be an octal number or a string, otherwise chmod complains. To fix this we could just add the line:

$perms = sprintf '0%o', $perms;

Also, there is no md5sum command, it's just called md5 on BSD.

-- VernonLyon - 14 Sep 2006

Thanks Vernon. Tracking your report in Bugs:Item2864

Harald, I'm reluctant to add those commands, just from a KISS perspective. Build.pm is already way too complicated....

-- CrawfordCurrie - 15 Sep 2006

It looks quite complicated indeed I have been trying to use BuildContrib to release TreePlugin and I keep getting errors like:

Can't locate Error.pm in @INC (@INC contains: /srv/www/slion/twiki/lib /srv/www/slion/twiki/lib /usr/lib/perl5/5.8.7/i586-linux-thread-multi /usr/lib/perl5/5.8.7 /usr/lib/perl5/site_perl/5.8.7/i586-linux-thread-multi /usr/lib/perl5/site_perl/5.8.7 /usr/lib/perl5/site_perl /usr/lib/perl5/vendor_perl/5.8.7/i586-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.7 /usr/lib/perl5/vendor_perl .) at /srv/www/slion/twiki/lib/TWiki/Contrib/Build.pm line 19.
BEGIN failed--compilation aborted at /srv/www/slion/twiki/lib/TWiki/Contrib/Build.pm line 19.
Compilation failed in require at build.pl line 11.
BEGIN failed--compilation aborted at build.pl line 11.

Can you give examples of what TWIKI_LIBS and PERL5LIB should contain?

-- StephaneLenclud - 16 Sep 2006

Ok I got the release target working by having something like that:

export TWIKI_LIBS=/srv/www/twiki/lib:/srv/www/twiki/lib/CPAN/lib
export TWIKI_HOME=/srv/www/twiki

Maybe we should have that as an example in the documentation. Now when I do perl build.pl install I got error messages saying it can't find TreePlugin_installer. So I moved build.pl, MANIFEST and DEPENDENCIEs from /twikiplugins/TreePlugin/lib/TWiki/Plugins/TreePlugin to /twikiplugins/TreePlugin. Then it finds the TreePlugin_installer but complains saying installer should be run from the root of TWiki. Is BuildContrib not supposed to be copying the installer to TWIKI_HOME and run it from there?

I'm having a hard time understanding the environement BuildContrib expects. I also find the documentation too talkative and not concret enough. I'm a very pratical guy I like to learn by looking at examples, like monkeys do

BTW BuildContrib is great. I can't wait until it's integrated in configure as I understand it will be in 4.1. The upload target is just amazing!!!

However when using installers I always have issues with the owner and group properties of files. Is there a way to tell the installer what's my apache owner and group so that the installed files have the correct properties? (wwwrun and www in my case)

-- StephaneLenclud - 16 Sep 2006

The problem is that I have always been the main user, as well as the developer, of buildContrib so the documentation has never been a priority. I confess I haven't tested the install target for ages, and nearly removed it last week. I just started BuildContribCookbook, because the doc is too hard for me alone to maintain.

-- CrawfordCurrie - 17 Sep 2006

Please don't why would you remove install? I'm sure we could fix it at some point. It seems to me like it could be useful if you develop in one place and want to validate your installation package by installing it to another place.
Now I understand why the build.pl for each plugin is buried down in the directory tree, it is so that one can checkout multiple plugins directly in a TWiki installation.
upload worked fine for TreePlugin but I always get blacklisted from TWiki.org whenever I try to upload TreeBrowserPlugin. Probably because TreeBrowserPlugin has quite a few files to upload. I sent a mail to PeterThoeny about that. For the first time I was happy to be on dynamic IP so I only had to reboot my router to change my IP and get access to TWiki.org again. Anyway I wonder what can be done for that overzealous balcklist. Can we ease the blacklist settings so that it's more permissive or can we introduce some temporization in BuildContrib so that it takes more time to upload but at least it does it without being caught by the watchdog? BTW thanks for pointing out BuildContribCookbook.

I have one concern with the upload of Plugins documentation topic. It seems to me like the master document is in SVN so if someone is kind enough to improve the documentation of a plugin on TWiki.org chances are that this work will be lost next time a developer upload a release.

-- StephaneLenclud - 17 Sep 2006

I was going to remove install because it just doesn't work that way any more.

Recently I changed the generated installer scripts so that they can download and install a zip from a different source. So the install target is as a simple as "cd to another directory and run the installer". I think that running the script externally is a better test, and with pseudo-install already in place, the install target is just redundant. See BuildContribCookbook for more on this.

On the blacklist, all I can suggest is that you once again get in touch with Peter. The blacklist seems to do a pretty good job of controlling wikispam, and I think we'd all hate to lose it.

-- CrawfordCurrie - 18 Sep 2006

I replied to Stephane after putting his current IP address on the WHITELIST.

-- PeterThoeny - 19 Sep 2006

Is there a way to automatically set the user and/or group id of the file after a plugin installation? I set the data and pub tree to nobody:nobody, but the template tree to root:nobody, on my install. And I find I have to manually change the uid/gid of each file after expanding the tar-ball.

In my mind, this would be a LocalSite.cfg setting that could be specified for each tree. Thoughts?

-- ScottHoge - 19 Sep 2006

You could always write a side script that does it, but it seems pretty pointless, since chown -R apache:apache * is all you usually need. The scripts can't do it, because it requires an external command, and usually requires privilege. Fortunately if you install from configure you get the right user anyway.

-- CrawfordCurrie - 20 Sep 2006

Hi Crawford, I loved your idea on 15 Dec, 2006 in BatchUploadPluginDev for using BuildContrib to generate an installer and automatically install dependencies in the chroot jail that my twiki runs in. If I could get this to work, then I would use it not only for installing the dependencies for BatchUploadPlugin, but also others such as the ImageGalleryPlugin.

So I would like to give this a try, but as a non-programmer, using BuildContrib for the first time in my live twiki site is a bit a scary. While there's already loads of documentation, could I ask for some instructions on how to repackage an existing PluginPackage to use BuildContrib ?

-- KeithHelfrich - 26 Dec 2006

There is something weird going on with the scripted uploads of plugins to twiki.org: TwistyContrib, TwistyPlugin and PreferencesPlugin had rev 1 or 2, but they actually have over 10 revs each. I did an edit/no-change/save on those three topics to fix the problem.

-- PeterThoeny - 13 Jan 2007

Two more: TipsContrib and PatternSkin showed r2 instead of a much higer rev. I brute-force fixed them an edit/save cycle.

I filed Bugs:Item3454.

-- PeterThoeny - 15 Jan 2007

For the upload target I suggest we ask before uploading the attachments in the same way we ask before uploading zip/tgz. The reason for doing that is that I get blacklisted by twiki.org whenever I upload TreeBrowserPlugin because it has too many attachments. I don't have a fixed IP so I can't be added to the whitelist. Moreover there is very little interest in having all the attachments uploaded since that plug-in is not functional on twiki.org.

• bcupload.patch.txt: Ask before uploading attachments.

I have now tested that patch it works fine.

-- StephaneLenclud - 09 Mar 2007

Hi

I am running the latest release version 4.1.2 of twiki on Ubuntu-server Fiesty 7.04 beta version with apache 2.2.3 server and perl 5.8.8. I have given full access to all twiki folders to apache user. On installing LdapContrib (any i guess) using configure page i get the error

Fetching http://twiki.org/p/pub/Plugins/LdapContrib/LdapContrib.tgz...

Warning: I can't install http://twiki.org/p/pub/Plugins/LdapContrib/LdapContrib.tgz because I don't recognise the download as a gzip file. 
Warning: Extension may not have been packaged correctly. Trying for a .zip file instead. 

Fetching http://twiki.org/p/pub/Plugins/LdapContrib/LdapContrib.zip...
Unpacking...
data/
data/TWiki/
data/TWiki/LdapContrib.txt
LdapContrib_installer
lib/
lib/TWiki/
lib/TWiki/Users/
lib/TWiki/Users/LdapUserMapping.pm
lib/TWiki/Users/LdapUser.pm
lib/TWiki/Contrib/
lib/TWiki/Contrib/LdapContrib/
lib/TWiki/Contrib/LdapContrib/Config.spec
lib/TWiki/Contrib/LdapContrib.pm
lib/ldap.cfg

Error: Failed to move file 'lib/TWiki/Contrib/LdapContrib/' to /home/twiki/lib/TWiki/Contrib/LdapContrib: Is a directory
Software error:
Installation terminated at /home/twiki/lib/TWiki/Configure/UIs/EXTEND.pm line 149.

For help, please send mail to the webmaster (webmaster@localhost), giving this error message and the time and date of the error. 

I had made the report at LdapContribDev thinking it to be ldapcontribs error

-- SibiJoseph - 03 Apr 2007

Hi I tried running the install script from command prompt the downloaded file has a "..tgz" extension and the error says "Could not open tar file " .

once i rename the file to .tgz script runs ok at command prompt

-- SibiJoseph - 03 Apr 2007

Hi folks. I'm having a crack at plugin maintenance - as a (hopefully) gentle introduction, I'm enhancing BeautifierPlugin to allow a default language and adding use strict. I've got most of the build process sorted out but am struggling to understand how the plugin topic history & displayed revision number is maintained.

I have a subversion working copy, and have made some edits to the topic page. I understand that build.pl will upload it to TWiki.org for me (and presumably manage the topic history there), but how is the history maintained for installed versions of the plugin? The zip/tgz files don't contain the ,v files so how can the history be updated?

If someone can give me a few pointers I'd be happy to write it up (perhaps in BuildContribCookbook?).

Thanks

-- MartinRothbaum - 27 Apr 2007

Regarding SibiJoseph's issue from 03 Apr 2007, see: Support.ConfigureDoesNotRecognizePluginFiles

-- CrisRhea - 13 Jul 2007

Rev 86 of WysiwygPlugin has a chopped off PackageForm, probably caused by the build contrib: https://www.twiki.org/cgi-bin/rdiff/Plugins/WysiwygPlugin?rev1=86;rev2=85

-- PeterThoeny - 09 May 2008

Problems observed with install so far:

• Requires Error but this is not checked for
• Requires to be run in a svn repository. This seems bad, as the user should not be forced to develop in an svn repository. Also, ther is no mention of this in the documentation. Worse, a newly developed plugin is forced to be checked into the svn repository before one can even build it. Error below:

Running: svn info /home/httpd/twiki.420/ControlsPlugin/data/TWiki/ControlsPlugin.txt /home/httpd/twiki.420/ControlsPlugin/lib/TWiki/Plugins/ControlsPlugin.pm /home/httpd/twiki.420/ControlsPlugin/lib/TWiki/Plugins/ControlsPlugin/menuexample.zip  svn: '/home/httpd/twiki.420/ControlsPlugin/data/TWiki' is not a working copy  svn: Can't read file '/home/httpd/twiki.420/ControlsPlugin/data/TWiki/.svn/entries'  

-- ThomasWeigert - 03 Jul 2008

Inexplicably, some installers produced with BuildContrib work great, but others fail with error "Odd number of elements in hash assignment at tools/extender.pl line 971." in web download. However, when run from the command line, also these installers complete without any error.

There is no obvious difference between the installers. Neither uses any special pre and post ambles, or dependencies, or the like.

For example, the ExplicitNumberingPlugin does not work, but EditHiddenTablePlugin works.

-- ThomasWeigert - 03 Jul 2008

Can't upload plug-in to upgraded twiki.org I'm getting "302 Moved" errors when trying to save the topic content.

-- StephaneLenclud - 28 Aug 2008

Perhaps the DNS record have not fully propagated yet. Retry from time to time until you succeed.

-- RafaelAlvarez - 29 Aug 2008

I've updated my BuildContrib cause I did not do it for ages. I'm not getting that 302 error anymore and it looks like it is working but it isn't really. Stuff did not update on twiki.org! Now I have a very little patch on my BuildContrib so that it is asking "Do you want to upload the attachments? [y/n]". But with or without that patch I get the very same result, or rather no result

Here are the output from BuildContrib:

Preparing to upload to:
Web:     Plugins
PubDir:  http://twiki.org/p/pub
Scripts: http://twiki.org/cgi-bin
Suffix:  
Is that correct? Answer 'n' to change? [y/n] y
Building a release for Version 17463 (29 Aug 2008) of TreePlugin
mkdir /tmp/7pBgWucfYQ/data
mkdir /tmp/7pBgWucfYQ/data/Sandbox/
mkdir /tmp/7pBgWucfYQ/data/TWiki/
mkdir /tmp/7pBgWucfYQ/lib
mkdir /tmp/7pBgWucfYQ/lib/TWiki
mkdir /tmp/7pBgWucfYQ/lib/TWiki/Plugins/
mkdir /tmp/7pBgWucfYQ/lib/TWiki/Plugins/TreePlugin/
mkdir /tmp/7pBgWucfYQ/pub
mkdir /tmp/7pBgWucfYQ/pub/TWiki
mkdir /tmp/7pBgWucfYQ/pub/TWiki/TreePlugin/
MD5 checksums in /home/sl/dev/twiki/trunk/TreePlugin/TreePlugin.md5
.tgz in /home/sl/dev/twiki/trunk/TreePlugin/TreePlugin.tgz
.zip in /home/sl/dev/twiki/trunk/TreePlugin/TreePlugin.zip
.txt in /home/sl/dev/twiki/trunk/TreePlugin/TreePlugin.txt
_installer in /home/sl/dev/twiki/trunk/TreePlugin/TreePlugin_installer
Enter username for http://twiki.org/cgi-bin: StephaneLenclud
Password: 
Config saved in /root/.buildcontrib
Downloading TreePlugin to recover form
Basing new topic on /home/sl/dev/twiki/trunk/TreePlugin/TreePlugin.txt
Saving TreePlugin
Taking a deep breath after the upload.....
Do you want to upload the attachments? [y/n] n
Do you want to upload the archives and installers? [y/n] y
Uploading Plugins/TreePlugin/TreePlugin.zip
Taking a deep breath after the upload.....
Uploading Plugins/TreePlugin/TreePlugin.tgz
Taking a deep breath after the upload.....
Uploading Plugins/TreePlugin/TreePlugin_installer
Taking a deep breath after the upload.....
Uploading Plugins/TreePlugin/TreePlugin.md5
Taking a deep breath after the upload.....

-- StephaneLenclud - 29 Aug 2008

Hmm, may be due to the fact that twiki.org uses now template login instead of Apache login?

-- PeterThoeny - 31 Aug 2008

That could the issue. I remember having problems with template login when I implemented my own perl TWiki client. With apache login you only need to set the credentials whereas template login will need an extra HTTP request.

-- StephaneLenclud - 31 Aug 2008

The upload should work OK with template login. Without seeing the error log on twiki.org it's impossible to guess.

-- CrawfordCurrie - 31 Aug 2008

That was indeed the problem. Thanks for the pointer Peter. Fixed in TWikibug:Item5971.

-- StephaneLenclud - 31 Aug 2008

BTW that's nothing to do with BuildContrib but twiki.org asks me to re-login very often it seems.

-- StephaneLenclud - 31 Aug 2008

Hi: The method of discovering whether a pwd command exists does not work for me. The code looks for pwd by testing /bin/pwd using -x. If I do this in a perl interactive debugger, I get this:

  DB<4> p (-x "/bin/pwd")?1:0
1
 

which is the test the code uses. However, if you try to use it this way, you get this

$ /bin/pwd
/bin/pwd: cannot open directory `../..': Permission denied
$

however, simple use of pwd is fine:

$ pwd
/home/copswiki

I think this is the problem I have with the automatic installer, as I get errors like this:

/tmp/XKuooGR91c
/bin/pwd: cannot open directory `../..': Permission denied
    data/
/bin/pwd: cannot open directory `../..': Permission denied
    data/TWiki/
/bin/pwd: cannot open directory `../..': Permission denied
    data/TWiki/MailInContrib.txt
/bin/pwd: cannot open directory `../..': Permission denied
    data/TWiki/MailInContribTemplate.txt
/bin/pwd: cannot open directory `../..': Permission denied
    lib/
/bin/pwd: cannot open directory `../..': Permission denied
    lib/TWiki/
/bin/pwd: cannot open directory `../..': Permission denied
    lib/TWiki/Contrib/
/bin/pwd: cannot open directory `../..': Permission denied
    lib/TWiki/Contrib/MailInContrib.pm
/bin/pwd: cannot open directory `../..': Permission denied
    lib/TWiki/Contrib/MailInContrib/
/bin/pwd: cannot open directory `../..': Permission denied
    lib/TWiki/Contrib/MailInContrib/Config.spec
/bin/pwd: cannot open directory `../..': Permission denied
    lib/CPAN/
/bin/pwd: cannot open directory `../..': Permission denied
    lib/CPAN/lib/
/bin/pwd: cannot open directory `../..': Permission denied
    lib/CPAN/lib/Email/
/bin/pwd: cannot open directory `../..': Permission denied
    lib/CPAN/lib/Email/Delete/
/bin/pwd: cannot open directory `../..': Permission denied
    lib/CPAN/lib/Email/Delete/IMAP.pm
/bin/pwd: cannot open directory `../..': Permission denied
    lib/CPAN/lib/Email/Delete/POP3.pm
/bin/pwd: cannot open directory `../..': Permission denied
Could not open file '/MailInContrib_installer': Permission denied at tools/extender.pl lin  e 694
Failed to extract MailInContrib_installer from tar file Archive::Tar=HASH(0x9516d44). Arch  ive may be corrupt.
Failed to unpack archive /home/copswiki/public_html/twiki/MailInContrib.tgz
Archive unpacked

This installer facility is very important to get general acceptance of TWiki and to enable ease of use by administrators. I've been using TWiki for years but I've never been able to get this install facility to work. I imagine it might be because I install TWiki on a shared server, but this should be possible.

-- RaymondLutz - 06 Dec 2008

I'm working behind a proxy, and I had to add the following to Build.pm in the target_upload subroutine:

after:

$userAgent->agent( 'TWikiContribBuild/' . $VERSION . ' ' );

I added:

$userAgent->env_proxy( 'TRUE' );

So that it would accept my http_proxy environmental variable.

-- AaronLWalker - 2010-12-31