Refactoring Proposal: TWiki.Cfg Needs Gardening

TWiki.cfg is, well, a bit of a mess.

Here are some of the things wrong with it:

1. The installer usually only configures a few things ($defaultUrlHost, $scriptUrlPath, $pubUrlPath, $pubDir, $templateDir, $dataDir, $logDir)
2. OS discovery (and module implementation selection) is done on every run, instead of once at install time
3. There are more comments than content, making it hard work to read and easy to miss something
4. There is duplication of settings (e.g. $dataDir and $storeSettngs{dataDir}
5. It's all global variables, making re-use of the configuration hard (a script that re-uses the configuration has to duplicate all those nasty global variables)

LocalSite.cfg has helped a bit, but there is still a need to "sort out" TWiki.cfg.

Here's my opinion (and it's just opinion, others can correct me on anything)

1. The "default configuration" should be moved into TWiki.pm.
2. All configuration variables should be in a hash, instead of dozens of globals
   • This is important for anyone who wants to manipulate a twiki configuration as an entity - for example, when moving a TWiki configuration from one site to another, or subclassing a configuration for testing purposes, or re-using the configuration in another application.
3. LocalSite.cfg should be removed. TWiki.cfg should become the "new TWiki.cfg"
4. TWiki.cfg should not be shipped with the release.
5. A template TWiki.cfg.txt can demonstrate setting variables in the configration hash.
6. Duplicate configuration variables should be merged
7. Unused configuration variables should be removed.
8. UpgradeTWiki should be made able to convert an "old" TWiki.cfg to the "new" form.

-- CrawfordCurrie - 22 Jan 2005

I originally added a Yes, Please to all the points :), the only other remaining comment is:

• I think there should be no difference between config vars, TWiki/Web/User/Topic - Preferences & FormFields, other than scope - and their setting should be totally gettable&settable from one perl module
  • For security I do not want to have sensitive config settings in the TWiki preferences, such as security filter. A compromised admin account currently cannot change the TWiki.cfg settings. Possible solution is to have a final preference setting in TWiki.cfg similar to the one in TWiki preferences -- PTh

-- SvenDowideit - 22 Jan 2005

To keep confusion levels to a minimum, I would just deprecate and remove TWiki.cfg, keeping the new LocalSite.cfg. It would still function exactly as you describe. If not I foresee a slew of Support Qs from people upgrading from old versions and copying their old configs over, or something like that. And then there's the hundreds of pages on twiki.org which say "in TWiki.cfg change...".

-- MattWilkie - 23 Jan 2005

I agree with all the points. I would add that plugin authors should be encouraged to do the same (not distributing editable configs, setting defaults in the .pm code, and encourage people to add their local setting of the plugin to the localsite/twiki.cfg new-style file

-- ColasNahaboo - 24 Jan 2005

OK, here's what I have done:

1. gardened TWiki.cfg to remove redundant configs
2. moved computed configs into TWiki.pm ($OS principally)
3. moved some of the constants into hashes, but left most in the $TWiki hash.
4. Renamed config constants to eliminate confusion e.g. $TWiki::dataDir is now $TWiki::cfgDataDir, $TWiki::egrepCmd is now $TWiki::cfgEgrepCmd.
5. Rewritten UpgradeTWiki to look at an existing setlib.cfg and TWiki.cfg, and generate new LocalLib.cfg and LocalSite.cfg from them

So, the idea is that TWiki.cfg and setlib.cfg contain the defaults, but they are treated as uneditable by the installer. They remain valuable as documentation topics. Local installer configuration is done by overriding the TWiki.cfg settings in LocalSite.cfg

Open questions:

1. Should we move LocalLib.cfg and LoclSite.cfg to a "conf" directory?
2. Is someone going to write a portable installer script?
3. Can someone please help out with testing UpgradeTWiki?

-- CrawfordCurrie - 25 Jan 2005

regarding DEVELOP r3576 LocalLib.cfg:

1. is it safe to leave out all settings from LocalSite.cfg which are not changed from the default example TWiki.cfg? Yes
2. $cfg{HtpasswdFileName} should be ='cfg{$datadir}/.htpasswd' instead of =''home/.../.htpasswd'
   • Default should be reverted to 'cfg{$datadir}/.htpasswd' since this directory works by default, e.g. is writable by the Apache user. It can be changed for non-standard installations -- PTh
3. why is '$cfg{MapUserToWikiName}' set to 1 when the default twiki auth system is htpasswd?
   • Because the out of the box TWiki is for behind firewall use, without TWiki managed htpasswd. All docs are done this way -- PTh
4. why is Auth Realm in both .htacecss and in TWiki.cfg (LocalLib.cfg) ? I've never bothered to make them match before (never noticed the twiki.cfg one TBH)
5. $cfg{UploadFilter} doesn't match .asp or .shtml (dunno if needs to either)
6. $cfg{NoSpamPadding} should not be blank by default (and probably be changed from time to time too)
   • No, should remain blank because TWiki focus is behind firewall -- PTh
7. please add UseTWikiRootInTWikiDotCfg so we can do things like: $cfg{twikiRoot}/logs; $cfg{twikiRoot}/data; $cfg{twikiRoot}/lib; etc... I know Peter doesn't seem to want it in MAIN, but that shouldn't stop putting it in DEVELOP? votes are 4:1 in favour PTh can reject that change for MAIN if hee feels strongly about it.

-- MattWilkie - 26 Jan 2005

"leave out all settings" - Yes, and I would even remove all comments from LocalSite.cfg to leave only local-admin comments, to reduce clutter

-- ColasNahaboo - 27 Jan 2005

Matt

1. Yes; in fact, that's what UpgradeTWiki now does (only adds changed settings to LocalSite.cfg, that is)
2. No; I frequently point it to odd places, often outside the TWiki tree altogether
3. erm.....
4. No idea - I didn;t read the code
5. Raise that as a bug, if you think it is a problem
6. The blank default assumes an intranet installation, which is reasonable
7. I don't want it either.

-- CrawfordCurrie - 27 Jan 2005

7. why not? it won't change the number of edits non-flat installs need to make; please answer in UseTWikiRootInTWikiDotCfg

2. retracted (solved by 7).

-- MattWilkie - 27 Jan 2005

Crawford, with the default configuration moved into TWiki.pm, does that mean that the admin is tempted to change TWiki.pm at installation time? It is fine to pre-initialize config in TWiki.pm, but any custom config should be done in a special config file; also for settings that are changed rarely such as doPluralToSingular. It is essential that we continue to be able to ship drop in patches, e.g. a TWiki.pm replacement for a security fix.

To mitigate the risk of touching TWiki.pm, how about not documenting the settings in TWiki.pm and have verbose comments in TWiki.cfg?

A twiki/conf directory seems to be too limiting for installs with non-standard directory layout? It should always be possible to detect all other directories by starting in the bin directory, not assuming a twiki root.

-- PeterThoeny - 28 Jan 2005

No, in the end I didn't move the default configuration. TWiki.cfg is still the default configuration. The idea is that the installer creates and edits LocalSite.cfg, which overrides the default settings, rather than editing TWiki.cfg. However they can still edit TWiki.cfg, they are just recommended not to.

On 2. the default is $cfg{DataDir/.htpasswd. I generated the new TWiki.cfg by simply writing a script that read in the old TWiki.cfg and wrote the new one using the new variable names. The reason I didn't use the $cfg{DataDir} symbol in the htpasswd value is that this symbol might be changed in the LocalSite.cfg, which is not read until after TWiki.cfg (the defaults). It would be easy for an installer to think that setting $cfg{DataDir} in LocalSite.cfg would have a knock-on effect on the htpasswd setting, but it wouldn't, because the full default path would already have been expanded. I felt it was better to remove this ambiguity altogether. I can be persuaded otherwise, as always - note that an alternative would be to delay expansion of such symbols until the entire config has been read, e.g. something like this: TWiki.cfg:

$cfg{DataDir} = "/home/httpd/twiki/data";
$cfg{HtpasswdFileName} = '{DataDir}/.htpasswd';

LocalSite.cfg

$cfg{DataDir} = "/green/eggs/and/ham"; # override setting in TWiki.cfg

Then in TWiki.pm:

map { $cfg{$_} = s/([^\\]){([A-Za-z]+)}/$1$cfg{$2}/g; } keys %cfg;

7. I don't want because I strategically place directories depending on the needs of the installation. I have in the past placed pub and data dirs on separate discs, in order to optimise disc usage. On Windows, which has no soft links (or very slow soft links, like cygwin). It would be even better if I could make individual webs point to different places, but the current compromise is a good one, IMHO.

-- CrawfordCurrie - 28 Jan 2005

Using the LocalSite.cfg file you can configure locally your own $twikiroot variable to help you configure other settings, so there is no need to have it as a default.

An example LocalSite.cfg could be shipped with that variable on it.

-- RafaelAlvarez - 28 Jan 2005

Crawford, on 2 I defer to your judgement. I didn't use the upgrade script, rather I copied the example TWiki.cfg to LocalSite.cfg and then changed as necessary. So I didn't realise it already does default to $dataDir/.htpasswd.

On 7, I really fail to understand your objection. The proposed change doesn't affect distributed directories. Perhaps you haven't had time to read UseTWikiRootInTWikiDotCfg yet?

-- MattWilkie - 28 Jan 2005

I read it. I understand your pro argument, you are not understanding my con - which I will summarise again:

The reason I don't use the symbols defined earlier in TWiki.cfg is that these symbols might be changed in the LocalSite.cfg, which is not read until after TWiki.cfg. It would be easy for an installer without perl experience to think that setting (for example) $cfg{RootDir} in LocalSite.cfg would have an effect on TWiki.cfg settings, such as $cfg{DataDir} = "$cfg{RootDir}/data". But it wouldn't, because the default value for $cfg{RootDir} would already have been expanded as soon as the DataDir setting in TWiki.cfg was first encountered.

The obvious fix, of reading LocalSite.cfg first, would have necessitated every default setting including a conditional:

$cfg{RootDir} = "/home/httpd/twiki" unless ( defined $cfg{RootDir} );

I felt it was better to remove this ambiguity altogether.

Note that an alternative would be to have a mechanism that allowed us to delay expansion of such symbols until the entire config has been read, e.g. something like this: TWiki.cfg:

$cfg{RootDir} = "/home/httpd/twiki";
$cfg{DataDir} = '{RootDir}/data';

LocalSite.cfg

$cfg{RootDir} = "/green/eggs/and/ham"; # override setting in TWiki.cfg

Then in TWiki.pm:

map { $cfg{$_} = s/([^\\]){([A-Za-z]+)}/$1$cfg{$2}/g; } keys %cfg;

I also considered a third alternative, viz. something like:

set "DataDir', '$cfg{DataDir}/data';

but I felt that both these approaches were over-engineered.

-- CrawfordCurrie - 29 Jan 2005

Okay, I understand now. Thank you for your patience and willingness to go over it again.

-- MattWilkie - 31 Jan 2005

I think we're pretty much done with this now.

In DevelopBranch, of course.

-- CrawfordCurrie - 13 Feb 2005

In DEVELOP 3709 I ran into some of the problem Matt hypothesisised.

The code in TWiki.pm notwithstanding, there seems to be some odd dependencies. Yes, LocalSite.cfg gets read in with a do before TWiki.cfg, but some things seem to be undefined and I get server error messges telling me things like %MAINWEB% can't be expanded.

Values that are in LocalSite.cfg are noted in TWiki.cfg. If I actually copy in the values from LocalSite.cfg into TWiki.cfg everything is fine.

I must admit, looking at the code in Twki.pm(3709) this makes no sense. No error is rasied so the "do" must have done.

I wish I I had some definitive traces on this, but I'm still trying to get the thing as a whole to work. Yea, I know, pre-Alpha.

-- AntonAylward - 28 Feb 2005

What perl version are you using? I wonder....

-- CrawfordCurrie - 28 Feb 2005