Tags:
performance1Add my vote for this tag scalability1Add my vote for this tag create new tag
, view all tags

Feature Proposal: TWiki as a Standalone Server

Motivation

TWiki runs as CGI script by default, what leads to some problems like:

  • Big overhead of loading, compiling and initializing for every request
  • As many copies in RAM as there are simultaneous requests

Alternatives such as ModPerl or SpeedyCGI helps a lot with the first point, but their use is not simple. Also there is no native support to FastCGI, an interesting execution mechanism that is available on many hosting services, web servers and operating systems.

ModPerl helps with performance, but increase memory use: it keeps a perl interpreter loaded with each Apache's child and possibly all TWiki modules, depending on how it was configured. Some problems of this setup are:

  • Forking a child is expensive (using worker or event MPM's reduces the impact of this problem)
  • Every apache child has a copy of TWiki with it in memory, even if it's not serving a TWiki request
  • This way is not possible to distinguish TWiki requests from static files requests
  • Memory usage grows a lot with many simultaneous requests

Depending on the RAM available and the concurrency level of requests, the Operating System may be forced to use swap, what degrades performance a lot. It can be explored in DoS attacks. To avoid that, we need to put hard limits on the number of simultaneous requests the webserver can handle. But, as pointed, it's not possible to distinguish twiki requests from static file requests, so users can face very big latency times to load a complete page (with images, css, etc).

SpeedyCGI and FastCGI keeps TWiki in memory, as ModPerl does, but separated from the webserver process, so it's possible to configure TWiki independently. To achieve better performance results it's often needed to keep more than one backend process (again: many copies of TWiki in RAM).

Also, it's not possible to use TWiki without a web server, what turns complicated things like TWikiOnMemoryStick.

Description and Documentation

Refer to the following blog posts for an overall summary:

What I wanted before start coding (Original Idea)

Performance raise when keeping TWiki in memory is noticeable. A brief look at source code shows that many operations are common for all requests (Users loading, preferences parsing, etc) and the results could potentially be reused among requests.

If TWiki could run as a Standalone Server it's performance would be comparable with achieved by ModPerl, SpeedyCGI or FastCGI, since the overhead of loading and compiling would be eliminated and it would also be possible to avoid performing many initialization operations repeatedly. Memory usage would be improved as well cause it would be possible to share memory used to store preferences (and whatever is common) between requests. That would be necessary some mechanism to forward requests from webserver to TWiki and to receive responses.

Besides, TWiki could run in some jail/chroot environment, reducing the impact of possible security problems.

What I Achieved

When I realized how flexible and easy to use Catalyst framework is, I decided that TWiki should be like that. So I started to modify TWiki core to be independent of execution mode: created all stuff needed, updated all necessary core code. Then I wrote some engine mechanisms, and it's done!

Now it's possible to run TWiki as CGI, under FastCGI, ModPerl and as an HTTP standalone server. And it's very easy! smile ( at least I think it is wink )

No dependencies were added to run TWiki as CGI and no configuration changes are needed.

Many hosting services provide FastCGI or ModPerl support or even let users run their own processes. There are many Codev topics with hacks to use these features. Most of them require source code changes, are not updated (or there is no feedback of successful use with newer versions) and can break TWiki somehow (I had a very bad experience with edit + SpeedyCGI...). With this new mechanism it's possible to fully take advantage of what a hosting service provides without fears.

FastCGI and HTTP engines lets administrators to control how many copies to run simultaneously, making it easier to balance performance and memory usage, independent of webserver limits.

Current Status

There are engines to support CGI, CLI, FastCGI, ModPerl, and a HTTP stand alone.

CGI implements the default TWiki behavior. CLI engine is automatic loaded when TWiki is used from command line.

FastCGI pre-compile and loads how many copies of TWiki as specified and forwards requests from webserver to them. This leads to a sensible performance increase, that is about 140% for simple topics, like Main.WebHome (on a fresh install, using pattern skin).

HTTP engine turns TWiki into a HTTP server itself, so it's possible to use it without a "real" webserver. It's very useful for TWiki development and also for TWikiOnMemoryStick and TWikiForWindowsPersonal (I didn't test it on windows, neither I tried to install in a memory stick, but I think HTTP engine make theses things much easier). It also has the same benefits FastCGI has and very similar performance (actually they are based on the same idea: keep things in memory, and differ in the protocol used to communicate).

ModPerl engine makes use of TWiki with mod_perl very easy, and actually doesn't suffer from historical problems with global variables (thanks to pos TWiki-4 code). It's a lot faster than CGI, but not as fast as FastCGI or HTTP. Each apache child (or thread, if you're using some multithreaded mpm) has it's own copy of TWiki, so be careful with MaxClients config option to avoid sending webserver to swap. If you can, use FastCGI or HTTP engines instead (see observations above)

In summary:

Engine Memory Usage Speed Support CPAN Dependencies
CGI As many copies of TWiki as there are simultaneous requests. Very Slow Very good. Almost all hosting services and webservers support CGI None
ModPerl As many copies of TWiki as there are apache children (or threads) Fast It only works with Apache webserver HTTP::Body
FastCGI Reasonably constant for each backend Very Fast Good. Many hosting services provide support for it FCGI and maybe FCGI::ProcManager
HTTP Reasonably constant for each instance Very Fast It probably runs on many operating systems and can be used if hosting service let users run their own processes HTTP::Headers, HTTP::Request, HTTP::Response, LWP::MediaTypes, HTTP::Daemon (all those are part of libwww-perl), HTTP::Body and maybe File::Monitor, File::Spec and File::Find
Shortcut: Go directly to Next Steps or Try it out if you're not interested on technical details.

Impact

WhatDoesItAffect: Install, Performance, Refactoring, Security

Implementation Details UPDATED

An engine framework, similar to catalyst's, is suggested. There is a TWiki::Request object, that gets filled by engine's methods. Then TWiki fills a TWiki::Response object, instead of writing answer to STDOUT directly. This object is passed to engine, that finally sends answer to client.

In this scenario CGI is simply one more engine, that uses %ENV and a CGI query object to populate TWiki::Request and that takes TWiki::Response to print its info to STDOUT.

Dynamic is as follows:

Engine's corresponding script gets executed, adjusts $TWiki::cfg{Engine} and loads TWiki
$TWiki::engine->run. The engine performs its specific setup tasks and waits for connections
$TWiki::engine->prepare fills a TWiki::Request object
prepareConnection fills remoteAddr, method and secure request fields
prepareQueryParameters fills request's queryParam field with data passed within query string, if any
prepareHeaders fills request object with connection headers available/pertinent and remoteUser field
prepareCookies fills cookies field with data available
preparePath fills pathInfo and uri fields
prepareBody performs any task related to process request body, if present
prepareBodyParameters fills request object with parameters passed within request body
prepareUploads fill request object with info about uploaded files, if any. Info is encapsulated in TWiki::Request::Upload objects
TWiki::UI::handleRequest: similar to old TWiki::UI::run. Gives TWiki::Request to TWiki and gets a TWiki::Response
$TWiki::engine->finalize: Sends to client data within TWiki::Response object and performs cleanup tasks
finalizeUploads deletes any temporary files created in prepare phase.
finalizeHeaders sends response headers to client
finalizeBody sends response body

Both TWiki::Request and TWiki::Response are reasonably compatible with CGI, but refer to documentation in code.

No dependencies were added to run TWiki with CGI engine. I won't promise that for other's engines wink (Actually IO::Handle is used, but since it's part of core perl distribution I think that's ok)

Changes ALERT! UPDATED

There were a lot of code changes and there must be a few about programming style. Main code changes were:

  • Created TWiki::Request, TWiki::Request::Upload, TWiki::Response, TWiki::Engine::.* and TWiki::LoginManager::Session classes
  • Modified TWiki::UI, based on bin/twiki and catalyst framework.
  • Updated TWiki::UI::View and TWiki::UI::Statistics to work well with engine mechanism
  • Created TWiki::UI::Rest class, adapted from bin/rest.
  • Eliminated almost all direct accesses to %ENV
    • $ENV{SCRIPT_NAME} replaced by $request->action(). They are not strictly the same, but as far as I could notice TWiki needs only $request->action()
  • Replaced print calls with TWiki::Response calls
  • cgiQuery field of TWiki class renamed to request (and it's a TWiki::Request object instead of a CGI)
  • Added response field to TWiki class
  • raw_cookie() not implemented and replaced by $request->headers('Cookie') in TWiki::LoginTemplate
  • TWiki::generateHTTPHeaders() fills $twiki->{response} object instead of returning HTTP header string
  • Added $TWiki::cfg{BehindProxy} config option
  • Unit tests:
    • new CGI replaced by new TWiki::Request
    • $twiki->{cgiQuery} replaced by $twiki->{request}
    • $query->path_info("/Web/Topic") replaced by $query->path_info("/script/Web/Topic")
    • Added $TWiki::cfg{Engine} = 'TWiki::Engine::CGI' to BEGIN block of test/bin/testRunner.pl
    • Some tests needed some little specific updates

Now that since TWiki can run persistently and is not necessarily a CGI script, extra care must be taken: ( Very Important! )
  • Never call exit
  • Avoid to call die (call it only if you're sure that it will be caught by exception mechanism)
  • Call TWiki::Response methods instead of print
  • Call TWiki::Request methods instead of reading %ENV or from stdin
  • Much care with circular references
  • Don't assume that some particular engine is been used
  • Always close file handles (I recommend the use of IO::Handle, so handles are closed when out of scope. Be sure they get out of scope wink )
  • Always be sure that forked applications terminate

How to Configure

Configuration examples assumes TWiki ins installed at /var/www/twiki. You have to configure an alias to TWiki pub if you have installed on another directory.

CGI Engine

It replaces traditional execution mode.

Apache

Alias /twiki/bin /var/www/twiki/bin

<Directory "/var/www/twiki/bin">
    Options +ExecCGI +FollowSymLinks
    SetHandler cgi-script
</Directory>

Lighttpd

    alias.url += ( "/twiki/bin" => "/var/www/twiki/bin" )
    cgi.assign = ( "" => "" )

And don't forget to load mod_cgi.

FastCGI Engine

It depends on FCGI CPAN perl module and webserver must support FastCGI. I used libapache2-mod-fastcgi and libapache2-mod-fcgid from debian to test.

Apache + mod_fastcgi

There are two ways to use FastCGI engine with mod_fastcgi. First one is to let twiki_fastcgi.pl to control servers and second to let apache.

First mode

This mode requires also FCGI::ProcManager CPAN perl module.

$ cd /var/www/twiki/bin
$ ./twiki_fastcgi.pl --listen /var/run/twiki/fastcgi.socket --daemon --nproc 1

  • --daemon is optional.
  • Adjust --nproc to satisfy your needs

And within config files:

FastCgiExternalServer /var/www/twiki/bin/twiki_fastcgi.pl -socket '/var/run/twiki/fastcgi.socket'
Alias /twiki/bin/configure /var/www/twiki/bin/configure
Alias /twiki/bin /var/www/twiki/bin/twiki_fastcgi.pl

<Directory "/var/www/twiki/bin">
    Options +ExecCGI
    SetHandler cgi-script
</Directory>

Second mode
FastCgiServer /var/www/twiki/bin/twiki_fastcgi.pl -processes 1
Alias /twiki/bin/configure /var/www/twiki/bin/configure
Alias /twiki/bin /var/www/twiki/bin/twiki_fastcgi.pl

<Directory "/var/www/twiki/bin">
    Options +ExecCGI
    SetHandler cgi-script
</Directory>

Again: adjust -processes to satisfy your needs

Apache + mod_fcgid

Alias /twiki/bin/configure /var/www/twiki/bin/configure
Alias /twiki/bin /var/www/twiki/bin/twiki_fastcgi.pl
DefaultMinClassProcessCount 1
DefaultMaxClassProcessCount 1

<Directory "/var/www/twiki/bin">
    Options +ExecCGI
    SetHandler fcgid-script
    <Files "configure">
        SetHandler cgi-script
    </Files>
</Directory>

Adjust DefaultMinClassProcessCount and DefaultMaxClassProcessCount to satisfy your needs. These parameters are global: apply to all FastCGI applications.

Lighttpd

$HTTP["url"] =~ "^/twiki/bin/configure" {
    alias.url += ( "/twiki/bin" => "/var/www/twiki/bin" )
    cgi.assign = ( "" => "" )
}
else $HTTP["url"] =~ "^/twiki/bin/" {
    alias.url += ( "/twiki/bin" => "/var/www/twiki/bin/twiki_fastcgi.pl" )
    fastcgi.server = ( "/twiki_fastcgi.pl" =>
        (
            (   "bin-path"  => "/var/www/twiki/bin/twiki_fastcgi.pl",
                "max-procs" => 1,
                "socket"    => "/var/www/twiki/working/tmp/twiki.socket"
            )
        )
    )
}

And don't forget to load mod_cgi and mod_fastcgi.

ModPerl Engine

There is support for versions 1.24+, 1.9 and 2.0+.

It depends on HTTP::Body CPAN perl module.

In order to use it, create a file named mod_perl_startup.pl in tools directory with the following contents: ( Adjust your paths wink )

$ENV{MOD_PERL} =~ /mod_perl/ or die "mod_perl_startup called, but mod_perl not used!";
use lib qw( /var/www/twiki/lib );
require '/var/www/twiki/bin/setlib.cfg';
1;

It's also cool to load most TWiki modules, so that some memory is saved.

Apache 1.x + ModPerl 1.24+

Alias /twiki/bin/configure /var/www/twiki/bin/configure
PerlTaintCheck On
PerlRequire /var/www/twiki/tools/mod_perl_startup.pl
PerlModule TWiki::Engine::Apache
<Location "/twiki/bin">
   SetHandler perl-script
   PerlHandler TWiki::Engine::Apache
</Location>
<Location "/twiki/bin/configure">
    SetHandler cgi-script
    Options +ExecCGI
</Location>

Apache 2.x + ModPerl 1.9+

Alias /twiki/bin/configure /var/www/twiki/bin/configure
PerlSwitches -T
PerlRequire /var/www/twiki/tools/mod_perl_startup.pl
PerlModule TWiki::Engine::Apache
<Location "/twiki/bin">
   SetHandler perl-script
   PerlResponseHandler TWiki::Engine::Apache
</Location>
<Location "/twiki/bin/configure">
    SetHandler cgi-script
    Options +ExecCGI
</Location>

HTTP Engine

I think this one is the most interesting smile

It depends on HTTP::Headers, HTTP::Request, HTTP::Response, HTTP::Daemon and HTTP::Body CPAN perl modules. If you want to use the auto-restart feature, then you also need File::Monitor, File::Spec and File::Find

Run:

$ cd /var/www/twiki/bin
$ ../tools/twiki_http.pl --host localhost --port 3000 --detach --restart --rdelay 1

And enjoy: http://localhost:3000/bin/view/Main/WebHome

  • --binurl and --puburl defaults to $TWiki::cfg{ScriptUrlPath} and $TWiki::cfg{PubUrlPath}, so it's safe to omit them. (Remember to run configure script to adjust Path Settings)
  • --host defaults to all (listen on all interfaces available)
  • --port defaults to 3000
  • --detach may not work on windows.
  • --restart re-executes itself if any file that is depended on changes, or if files are created or deleted under lib/ (feature specially designed for developers)
  • NEW --rdelay delay between check for changes (default: 1 second)
  • --help show usage information and exit

There is primitive support for static files, mainly added to help twiki development. In production, it's really recommended to run HTTP stand alone server behind a proxy. Some reasons are:

  • Proxy webserver handles static files more efficiently
  • Proxy webserver can be configured to force mime types of some dangerous files (.php, .html, ...) to text/plain
  • Many instances of twiki can be executed, with proxy webserver acting as a load balancer

If, and only if, you are running behind a proxy, check {BehindProxy} option within configure script (Expert mode => Security Setup => Miscellaneous). If you run http stand alone behind a proxy and don't check {BehindProxy}, then %ENV{"REMOTE_ADDR"}% will always return proxy's IP address and {Sessions}{UseIPMatching} option would be useless. On the other hand, if you check {BehindProxy} without a proxy, then an attacker could easily perform an IP spoofing attack by sending a fake X-Forwarded-For HTTP header. So, adjust {BehindProxy} according to your setup (unless you are a TWiki developer, there is no reason to not use a proxy).

The following configuration assumes you are running two twiki_http.pl instances, on ports 3000 and 3001.

Apache

Alias /twiki/bin/configure /var/www/twiki/bin/configure

<Directory "/var/www/twiki/bin">
    Options +ExecCGI
    SetHandler cgi-script
</Directory>

ProxyRequests Off
ProxyPreserveHost On
ProxyPass /twiki/bin/configure !
ProxyPass /twiki/bin balancer://twiki

<Proxy balancer://twiki>
    BalancerMember  http://127.0.0.1:3000/twiki/bin
    BalancerMember  http://127.0.0.1:3001/twiki/bin
</Proxy>

If you are running only one twiki_http.pl instance then you can do something like:

Alias /twiki/bin/configure /var/www/twiki/bin/configure

<Directory "/var/www/twiki/bin">
    Options +ExecCGI
    SetHandler cgi-script
</Directory>

ProxyRequests Off
ProxyPreserveHost On
ProxyPass /twiki/bin/configure !
ProxyPass /twiki/bin http://127.0.0.1:3000/twiki/bin

Don't forget to load mod_proxy, mod_proxy_http and mod_proxy_balancer (this one can be ignored if the last example is used).

Lighttpd

proxy.balance = "fair"

$HTTP["url"] =~ "^/twiki/bin/configure" {
    alias.url += ( "/twiki/bin" => "/var/www/twiki/bin" )
    cgi.assign = ( "" => "" )
}
else $HTTP["url"] =~ "^/twiki/bin/" {
    proxy.server = (
       "/twiki/bin" => ( ( "host" => "127.0.0.1", "port" => 3000 ),
                         ( "host" => "127.0.0.1", "port" => 3001 ) )
    )
}

Don't forget to load mod_cgi and mod_proxy.

Refer to mod_proxy documentation. Adjust proxy.balance for your needs.

Other configuration tips

{StoreImpl} and {RCS}{SearchAlgoritm}

If you're using some persistent engine (FastCGI, ModPerl or HTTP stand alone), it's better to set {StoreImpl} to RcsLite and {RCS}{SearchAlgorithm} to TWiki::Store::SearchAlgoritm::PurePerl or TWiki::Store::SearchAlgorithm::Native. The last one is a distributed as a Contrib: NativeSearchContrib.

ApacheLogin

In order to use ApacheLogin as login manager there are some extra configurations needed.

Note:: Please, consider switching to TemplateLogin.

Apache

<LocationMatch "/twiki/bin/(attach|edit|manage|rename|save|upload|mail|logon|.*auth).*">
   AuthUserFile /var/www/twiki/data/.htpasswd
   AuthName 'Enter your WikiName: (First name and last name, no space, no dots, capitalized, e.g. JohnSmith). Cancel to register if you do not have one.'
   AuthType Basic
   require valid-user
</LocationMatch>

Lighttpd

Refer to observation at TWikiOnLighttpd#User_Authentication.

$HTTP["url"] =~ "^/twiki/bin/(attach|edit|manage|rename|save|upload|mail|logon|.*auth).*" {
    auth.backend = "htpasswd"
    auth.backend.htpasswd.userfile = "/var/www/twiki/data/.htpasswd"
    auth.require += ( "" => (   "method" => "basic",
                                "realm"  => "Enter your WikiName: (First name and last name, no space, no dots, capitalized, e.g. JohnSmith). Cancel to register if you do not have one.",
                                "require" => "valid-user"
                            )
    )
}

Don't forget to load mod_auth.

Known Issues

  • ALERT! configure script doesn't work under engine mechanism (but it can be configured as default CGI and it's good to do that way, since it is a very special case)
  • ALERT! OopsExceptions leads to Access Denied error with HTTP engine, cause http://localhost:3000 doesn't match $TWiki::cfg{DefaultUrlHost}. It's not really a bug, but I think it's cool to register this. It doesn't show up when running behind a proxy. If you want to use it without a proxy (to develop, for example), then configure $TWiki::cfg{DefaultUrlHost} to the correct url wink
  • ALERT! HTTP engine doesn't support yet https ( Really needed? Probably not, since it's recommended to run behind a proxy. Then configure https on proxy)

Legend:

DONE Solved
Work in progress, under construction Work in progress
ALERT! No action needed

Pending tasks

As soon as I can I'll work on the following tasks:

  • Write unit test cases (and evaluate them with Devel::Cover)
  • Request Dependet/Independent tags: What is common between requests? And what is specific?
    • Memory usage analysis
    • Singletons rethink (Potentially they will be split between request and execution singletons)
  • To implement classes that take advantage of persistent execution (Put preferences in a DBM and loads it in a shared memory block, so reducing processing time and memory usage. Maybe something similar to users, forms...)
  • Plugins port guide, possibly suggestions of hooks
  • Analise performance
  • ... ( suggestions welcome! smile )

Try it out!

Update: Current documentation refers to last svn version. There were changes to ModPerl and CGI. Since it will be merged to core, probably I won't make another release. Refer to revision 64 of this topic to see documentation that applies to the release below (4.3.0-auto15428).

-- GilmarSantosJr - 28 Feb 2008

Note: Use it for test purposes only. All existing unit tests are OK and as far as I could test (not too far...) things work fine, but probably there are some bugs around. If you find any, please report them at Bugs:Item4662 (add bug description as a comment).

Version Description Download
First release, from svn rev 15428 Independence from execution environment, support for CGI, CLI, FastCGI, ModPerl and HTTP stand alone http://www.twiki.org/p/pub/Codev/TWikiStandAlone/TWikiStandAlone-4.3.0-auto15428.tgz
Refer to TWikiInstallationGuide for general tips and to above configuration examples, run the configure script and have fun wink

ALERT!: If you're using some persistent engine (FastCGI, ModPerl or HTTP without --restart), you need to restart webserver (or HTTP instances) if you update configuration.

This is a work in progress, you can always test lattest version from SVN:

svn co http://svn.twiki.org/svn/twiki/scratch/TWikiStandAlone
cd TWikiStandAlone
perl pseudo-install.pl -link default

and you can update an existing working copy:

for dir in lib data locale logs pub test tools twikiplugins working templates; do find $dir -type l -exec unlink {} \; ; done
svn update
perl pseudo-install.pl -link default

If you want to run unit tests, refer to TestCasesTutorial wink

-- Contributors: GilmarSantosJr

Discussion

Actually I'm already working on this as my Undergraduation Final Project. Currently, I'm analyzing what needs to be changed, what is common for all requests and how could it be possible to handle more than one request at a time, sharing common objects. The next step is to implement a mechanism similar to Apache's MPM, define how TWiki and webserver would communicate and finally implement, test and compare multithread and multiprocess with shared memory approaches.

Maybe multithread is not a good idea, since Perl Threads doesn't share data between threads by default, but I think it's worth a try.

My goal is to achieve similar performance results to ModPerl, SpeedyCGI and FastCGI, but without the problem of excessive memory usage. smile

-- GilmarSantosJr - 21 Aug 2007

Which CPAN web server / web app are you thinking of using as the base?

-- SvenDowideit - 22 Aug 2007

I'm taking a look at Catalyst::Engine::* to figure out how it works. I want something as flexible, extensible and easy as that!

-- GilmarSantosJr - 23 Aug 2007

I've just created a branch to start playing smile

-- GilmarSantosJr - 23 Aug 2007

First commited version show blank pages with CGI or FastCGI engines. Corrected and commited.

-- GilmarSantosJr - 16 Sep 2007

Just to let you know it is great you are pursuing this!

-- ArthurClemens - 18 Sep 2007

I would like to point a very nice possibility this opens to us: a "real" webserver is not needed anymore to develop TWiki. We just need to do a checkout, run the HTTP standalone engine and browse to e.g. http://localhost:3000/. smile

-- AntonioTerceiro - 19 Sep 2007

Thanks, Arthur!

I've just finished the "restart" feature on HTTP engine: it's no longer necessary to restart twiki_http.pl when a change is made to any file. It helps development a lot smile

-- GilmarSantosJr - 15 Oct 2007

For the non-techies among us: What would be a typical use case for this proposal?

Could someone explain in "real words" what this is all about? Thanks, I'm quiet shure this is useful wink

-- CarloSchulz - 19 Oct 2007

Hi Carlo,

This proposal increases twiki's execution mode flexibility. So it would be easier to run twiki in many hosting services (If the service provides FastCGI, ModPerl or lets users to run their own processes) and take more advantage.

Execution under FastCGI or as HTTP stand alone raises performance a lot, and it's possible to better control memory usage. It also permits some more code improvements to reduce resource usage even more (Less resource usage => cheaper hosting services => more twiki installations) .

HTTP stand alone could be run under a chroot jail, reducing impact of possible security holes.

HTTP stand alone (or some other persistent engine [but I think HTTP stand alone is the "best"]) together with DatabaseStore can be used to build very scalabe TWiki-based sites using more than one machine to load balance.

There are some more advantages/usages for developers too. wink

-- GilmarSantosJr - 19 Oct 2007

Thanks for claryfiing smile

-- CarloSchulz - 22 Oct 2007

This looks very useful indeed - would be great to make this really easy to install (i.e. a TWikiFor distribution, as in TWikiForWindowsPersonal) so you can simply unzip the TWikiStandalone distribution into a directory, start the HTTP server, and go.

Performance and WYSIWYG are the last two hurdles to TWiki adoption in my view. and this helps enormously with performance. The FastCGI support is really interesting for web hosting, where TWiki is quite popular, and for any Linux distros that already support FastCGI in their Apache versions.

-- RichardDonkin - 30 Oct 2007

Just wanting to let you know that I am still interested in this:

MichaelDaum - 27 Oct 2007:

This is great work and I am looking forward to give it a run.

Still, I wonder why you had problems with SpeedyCGI. It may be worse the effort to fix that too as the very same problem could bite you in the stand-alone version. I am using speedy on a regular base and have a high interest to see standard twiki be fine with speedy. Btw. TWiki.org is speedified too.

-- MichaelDaum - 14 Nov 2007

I've been using MediaWiki quite a lot recently, and it really is blindingly fast - they do a lot of caching, and of course PHP in most web servers runs as a module. TWikiStandalone is important to catching up with this, as are both FastCGI and SpeedyCGI support.

-- RichardDonkin - 14 Nov 2007

I will have to have a play with this - I have TWikiOnUbuntu installed via the twiki package for Ubuntu, but its speed on my old PCs is quite leisurely. FastCGI seems the way to go for me. Also, it would be great to have this up on Dreamhost (see DreamhostSetupNotes), who do support FastCGI.

-- RichardDonkin - 22 Nov 2007

Anyone get fastcgi working on Sun Web server?

-- DwayneLee - 04 Dec 2007

This sounds very interesting, unpack and run directly without apache or other web server? I tried to download but could not make it work. Am I doing it wrong or what is missing? I'm running Ubuntu 7.10.

  • download the tgz from this page (15428)
  • install cpan modules, HTTP::Headers..etc as specified above
  • unpack to /var/www/twiki
  • cd to /var/www/twiki/bin
  • run ../tools/twiki_http.pl arguments...as specified above
I get something like:

Content-type: text/plain
Perl error when reading LocalSite.cfg: 
Please inform the site admin.
BEGIN failed--compilation aborted at /var/www/twiki/lib/TWiki.pm line 484.
Compilation failed in require at ../tools/twiki_http.pl line 53.
BEGIN failed--compilation aborted at ../tools/twiki_http.pl line 53.

-- LarsEik - 14 Feb 2008

I haven't tried it yet, but perhaps you need to create a LocalSite.cfg in lib/ ? I just tried to "echo "1;" > ../lib/LocalSite.cfg" and it eliminates the error you get.

-- KoenMartens - 15 Feb 2008

Thanks Koen, after copying LocalSite.cfg from a debian server with the same paths it started and greets me with http://localhost/twiki/bin/view/Main/WebHome when I use port 80. but I cannot get configure it just goes "Not found". The configure script is there with same props as view. Docs say it dont work default but can be configured?

-- LarsEik - 15 Feb 2008

Hi Lars,

configure script is very special, since it must work even if TWiki is not configured yet, so it doesn't make sense to get it working under this mechanism. The solution is to configure TWiki manually (copy lib/TWiki.spec to lib/LocalSite.cfg and change the desired settings) or to set up apache and run configure script as a CGI.

Thanks for the feedback. I'll work on a better out-of-the-box experience wink

-- GilmarSantosJr - 17 Feb 2008

Are there plans to incorporate this work into the upstream release?

-- MichaelDaum - 17 Feb 2008

I hope so and I'm working with that in mind smile

-- GilmarSantosJr - 17 Feb 2008

So basically one could copy a LocalSite.cfg from a server with the same paths. I also copied plugins but suddenly I copied too much.... It's really great and I look forward to test updated version, hope this is something that will be standard in TWiki.

I hope the core developers are following this one.

-- LarsEik - 17 Feb 2008

GilmarSantosJr, please file a new proposal so that your work can enter the roadmap for TWiki-5.x

Great work! Go ahead!

-- MichaelDaum - 18 Feb 2008

Thanks, Michael!

Actually this topic is a feature request, proposed for GeorgetownRelease (I think this one will be 5.x, wont it?) and UnderInvestigation. According to diagram at CurrentState we need to reach consensus or decide about it on some release meeting to get this accepted and then figure out how to merge with trunk.

I'll sync TSA scratch branch with trunk, update docs and wait for more feedback (maybe we got a consensus smile ), or the next release meeting.

-- GilmarSantosJr - 18 Feb 2008

This is all kinds of awesome! Finally TWiki can run as a different user than Apache and it's fast to boot! I hope it gets merged soon... Can you outline what the big changes were that you needed to do for this?

-- WoutMertens - 27 Feb 2008

I cannot see this one listed under TWikiFeatureProposals, am I missing it? Maybe it is just because DateOfCommitment is missing?? I just afraid it could be missed on the agenda since it's not listed with the other feature requests.

-- LarsEik - 27 Feb 2008

lets find out smile I've made Gilmar the commited developer - hope thats true smile

-- SvenDowideit - 28 Feb 2008

Yeah, that's true smile

Actually I was waiting to finish some architecture enhancements and update docs (especially for developers). I'm working on this right now, so I think it's OK to have 2008-02-28 as DateOfCommitment wink

-- GilmarSantosJr - 28 Feb 2008

I've just finished enhancements and docs update that I mentioned previously. Now TWiki::Engine and TWiki::UI are more independent from each other, the former is more simple and, besides to the lack of additional dependencies, no configuration change is needed to update from the traditional method (cgi script) to CGI engine.

Thanks Wout! Take a look at implementation details section wink

-- GilmarSantosJr - 28 Feb 2008

Date of Commitment is the date that you as a developer commit to be the driver and implementer of the feature proposed and it is what makes the 14 day clock tick.

The community now has 14 days to accept or reject or raise concern.

I have a question. You have now checked in a lot of code in a unique branch. How to you suggest merging this into the trunk?

-- KennethLavrsen - 28 Feb 2008

I could keep using SVK the same way I use it to get changes from trunk. But I think it's better to merge incrementally: one commit for each logical change (there could be one bug corresponding to each one). The big picture is:

  • Add TWiki::Request, TWiki::Response and TWiki::LoginManager::Session (abstraction layer created)
  • Add TWiki::Engine*, tools/twiki_http.pl, bin/twiki_cgi.pl, bin/twiki_fastcgi.pl and update lib/TWiki.spec
  • Merge changes to TWiki.pm
    • Add response field
    • Rename cgiQuery field to request (and update it everywhere it's used within TWiki.pm as well other classes)
    • Update some methods: UTF82SiteCharset, generateHTTPHeaders, etc
    • Eliminate direct %ENV accesses
  • Update TWiki::UI::* as needed (replace print 's and %ENV reads)
  • Update TWiki::Func
  • Update TWiki::LoginManager
  • Update TWiki::Sandbox
  • Update lib/MANIFEST
  • Update UnitTestContrib and default plugins unit tests (refer to implementation details above)
  • Merge above documentation about configuration to official docs

Some of these changes also fix minor bugs that apply to 4.2 (I didn't have the time to describe them at bugs web, but I will)

After these first steps trunk will have engine mechanism merged, but I'd like to discuss about some changes to make architecture simpler:

  • Split TWiki class into TWiki::Util, TWiki::DefaultHandlers and TWiki (much small and simple)
  • Let TWiki::generateHTTPHeaders responsibility with engine mechanism
  • Rethink about classes relationships and singleton design (the idea is to avoid performing common-to-all-requests tasks when persistent engines are used [and it must work for non-persistents too])
  • Add plugin hooks to give them the chance to take advantage of persistent execution (something like a globalInit and globalFinish)
  • We will need to hunt memory leak sources (Yeah, they still exist... frown ), or add options/doc to restart persistent engines regularly.

Also I have some more enhancements in mind (these ones will direct affect performance and resource consumption):

  • Implement a prefs mechanism that uses less RAM, that can be shared among many processes and that doesn't need to parse topics for every request (If what I'm thinking about works, then it would be a drop-in replacement for the actual TWiki::Prefs, but would have some more dependencies, so it could exist a cfg{PrefsMechanism} to let users choose the more indicated for their needs).
  • Try to make something similar to Forms and Users...

-- GilmarSantosJr - 28 Feb 2008

I am absolutely happy to see someone new that wants to address the core code.

Naturally we will need to create proposal topics for each of your ideas beyond the stand alone changes but it is encouraging to see new ideas related to the core.

Gilmar have you reviewed the TWiki:Codev.TWikiRoadMap? Do you agree with it. DO you want to add new things (anyone is encouraged to maintain and evolve the roadmap)?

-- KennethLavrsen - 03 Mar 2008

I'm monitoring not only TWikiRoadMap, but also TWikiRoadMapDiscussionFiveDotZero, DatabaseStore and TopicObjectModel. I'm happy to see that many ideas I had are already in progress!

I'll mature the ideas beyond changes already done and file new feature/refactoring proposals as soon as I fell I'm ready. wink

-- GilmarSantosJr - 11 Mar 2008

I am having some troubles:

  1. The version in svn stops at a symlink bin/oops - it just does not update svn any further
  2. I can't get the downloadable version to work when following the instructions in revision 64. There is no ScriptAlias, just 2 times Alias, and that does not make the bin folder executable. Could you add a complete httpd.conf example?

-- ArthurClemens - 12 Mar 2008

I don't know what caused the problem with svn, but I removed all links and added them again, it's ok now.

Note that there is an Alias to the configure script, and other to the twiki_cgi.pl script (instead of the bin directory. This was changed in svn, and a simple ScriptAlias is enough, like the standard way to run as CGI)

Alias /twiki/bin/configure /var/www/twiki/bin/configure
Alias /twiki/bin /var/www/twiki/bin/twiki_cgi.pl  # <--- Note this line makes "/twiki/bin" path point to twiki_cgi.pl script!!

There are also some options to the /var/www/twiki/bin directory, that make it executable:

<Directory "/var/www/twiki/bin">
    Options +ExecCGI
    SetHandler cgi-script
</Directory>

It works well with Apache 2.2.8-1 from Debian (I have this version now, but I remember it always worked at least with 2.x)

This way you'll run TWiki as a CGI script. Try to use other engines as well wink (other engines may have CPAN dependencies. All of them are supposed to work!)

-- GilmarSantosJr - 13 Mar 2008

We have passed the 14 day timeout and I have only seen positive responses to I am happy to flip this into an accepted proposal.

-- KennethLavrsen - 15 Mar 2008

  • Can you please add a logging facility in the httpd mode? It took me a while to figure out how to fix it just giving me 404 errors
  • In Engine/HTTP.pm, please change all references to binUrl to scriptUrl. I couldn't make it work otherwise.
  • For some reason, I had a lot of references to $cfg{} in my LocalSite.cfg file. I had to change all of them to $TWiki::cfg{} before the standalone would work.
  • Looks like the standalone httpd doesn't support ShorterUrlCookbook . I'm investigating if I can make the apache proxy do all the legwork.

-- WoutMertens - 22 Apr 2008

Ok got it working - wow that's fast, I love it! TagMePlugin is failing though, at this code:

    my $query = $TWiki::Plugins::SESSION->{cgiQuery};
    my $tagMode = $query->param('tagmode') || '';

The query object in there is undef which makes the param() call fail. Does it not get filled in?

I also discovered that it won't handle subwebs exactly like plain TWiki: if you go to twikihost/Web/Subweb it will say it can't find the topic "Subweb" while plain TWiki will instead show twikihost/Web/Subweb/WebHome.

-- WoutMertens - 22 Apr 2008

Hi Wout, thanks for your feedback!

I'm pretty busy now (and that's why it's not merged yet), but I have many enhancements to HTTP engine in mind. Some kind of logging is one of them wink

binUrl is set to scriptUrl unless you specify them at command line, so it should work. Can you give more details about your configuration and problem? (Please, post the details at Bugs:Item4662)

ShorterUrlCookbook is something that I'll review latter. Probably, there must be explicit support for it at HTTP engine.

The problem with TagMePlugin is that it doesn't use plugins API, so it was expected to break sometime. frown Please, file a bug report to TagMePlugin: it should use TWiki::Func::getCgiQuery instead of taking it directly from core.

I'll look at subwebs issue carefully and fix it if something is broken.

-- GilmarSantosJr - 24 Apr 2008

Actually I'm not sure if the binUrl/scriptUrl thing is a bug. I hadn't properly configured the twiki install because I thought copying localsite.cfg was enough; however my current install is 4.0.5 and that one had a twiki.cfg with defaults. Your tarball doesn't and therefore it was missing the default value for binUrl. Running configure fixed it.

Aside from TagMePlugin failing: If I "fix" the $query problem by removing the call and use PatternSkin, server response is insanely slow. Setting skin=plain works fine. When trying to run perl profiling on the code, I noticed that for loading 1 page, TWiki::Users::TWikiUserMapping::isGroup gets called 55000 times! Skin=plain has normal behaviour. Does PatternSkin work for you? Should I use the svn version instead of the tarball? Should I wait until you merged?

-- WoutMertens - 24 Apr 2008

I just fixed TagMePlugin and I didn't notice any problem with PatternSkin. I made many enhancements to svn version since I released the tarball, so I recommend to use the svn version. I'll add support to run configure script from HTTP engine before merge, cause it's a important feature. I'll come back to work on this as soon as I have the time for it. wink

I also tested the subweb issue with both TWikiStandAlone and trunk latest svn revision and they have the same behavior: I created a Main/Test web and requested /view/Main/Test and /view/Main/Test/. I got a "this topic doesn't exist yet" for the first from by both TSA and trunk, and I got Main.Test.WebHome for the second request. By looking at TWiki.pm:1348, I see:

    if( $pathInfo =~ /\/((?:.*[\.\/])+)(.*)/ ) {
        # is 'bin/script/Webname/SomeTopic' or 'bin/script/Webname/'
        $web   = $1 unless $web;
        $topic = $2 unless $topic;
        $web =~ s/\./\//go;
        $web =~ s/\/$//o;

This regex always interpret Main/Test as $web = "Main" and $topic = "Test". Maybe it should verify if hierarchical webs are enabled and if the last name exists as a sub web...

-- GilmarSantosJr - 27 Apr 2008

Argh, in 4.0.5 that code is immediately followed by

    # Check to see if we just dissected a web path missing its WebHome
    if($topic ne "") {
      if(!$this->{store}->topicExists($web,$topic)) {
        if($this->{store}->webExists("$web/$topic")) {
          $web .= '/'.$topic;
          $topic = "";
        }
      }
    }

Which checks if the topic is actually a subweb, provided the topic doesn't exist. This code works even if hierarchical webs are disabled, for instance if someone tries to load "Main".

Looks like someone didn't like that feature? I can't see anything in the svn version doing the same thing...

-- WoutMertens - 28 Apr 2008

I searched for this change and found Bugs:Item2998. Since 4.1.0 it's needed to access subwebs by, at least, appending a trailing '/' to URL.

-- GilmarSantosJr - 30 Apr 2008

Heya Gilmar - do you have any info about scheduling bringing this work into the trunk? I've played with it a little, and really would like to see it go in before we do big work on the trunk, so that we can be sure that it gets testing. I'm even somewhat tempted to just get it merged now, in one big line, just because delaying isn't getting it done.

-- SvenDowideit - 19 Jun 2008

Hi Sven,

I will work on this as much as I can from now on (except this weekend, cause of So Joo's party). I hope to have it almost finished within the next two weeks. Almost finished because the merge will be relatively easy and fast, but it may cause some issues that were not caught by unit tests, neither by my use experience. And there is still some work to do: new unit tests, high level documentation and updates of the official documentation.

I didn't merge it before because it would (and will) probably break some things and I couldn't deal with that before (I didn't have free time for a while). But now I can smile

-- GilmarSantosJr - 20 Jun 2008

One thing we need to work out, is how to make sure that Plugins work for both trunk and TWikiRelease04x02 - It shouldn't be hard, but pretty much all the Plugins will need work.

Because the trunk is used to publish Plugins for released versions - this migration needs to be done extremely carefully.

great to see progress Gilmar smile

-- SvenDowideit - 29 Jun 2008

Which plugins do need work and what kind of work?

-- MichaelDaum - 30 Jun 2008

Well behaviored plugins doesn't need work (perhaps only in its unit tests, since the test framework had a few changes).

Plugins not well-behaviored are the ones that:

  • print data directly
  • read from %ENV directly
  • call die or exit
  • accesses core variables without TWiki::Func interface

I didn't test yet many plugins other than default ones.

-- GilmarSantosJr - 30 Jun 2008

Gilmar, did you modify any plugins in your branch? If so could you please make a list of them so that we know which changes and possible merges of your code and code in development environments not yet checked in are ahead.

-- MichaelDaum - 01 Jul 2008

I updated unit tests of:

And the only one I had to modify was WysiwygPlugin, but there is no consensus yet about this (I hope to have it today). I also changed TagMePlugin, but I already fixed it in trunk some time ago. By now, I tested only core plugins and TagMePlugin. I didn't tested others, cause I have no idea about plugins "popularity" and I couldn't test all of them.

-- GilmarSantosJr - 01 Jul 2008

That's okay (only 194 plugins remaining). Hey, this is five dot oh stuff, so things are allowed to break wink

-- MichaelDaum - 01 Jul 2008

I merged it to core today. bugs web is already running with the changes (using CGI engine). However some work is still needed:

  • Write unit tests to the new code
  • Figure out and implement something to make plugins (and tests) to work with trunk and older releases (currently, at least tests, work only with trunk)
  • Add support to run configure script from HTTP engine
  • Release FastCGIEngineContrib, ModPerlEngineContrib and HTTPEngineContrib (see EnginesAsContribs)

Now let's play with it! smile

-- GilmarSantosJr - 21 Jul 2008

Has anyone got WebChanges working with FastCGI? I keep getting "Could not perform search. Error was: Not a GLOB reference" - I suspect this has something to do with forking grep (on Debian linux).

-- PeterPayne - 25 Jul 2008

Hi PeterPayne,

Thanks for the feedback! There are still many small issues, including extensive testing, to be solved and I'm working on them. I didn't merged engines yet (and they won't be merged to core, but as contribs). After that, I'll post news here. Fell free (and invited!) to report bugs. Please, include:

  • Svn revision you are using
  • Which web server, version, fastcgi implementation...
  • Operating system (and version)

-- GilmarSantosJr - 25 Jul 2008

Hi GilmarSantosJr, I was using a merge between

Using the command su www-data -c "perl -T twiki_fastcgi.pl -l /tmp/twiki.socket" to start FastCGI (using FCGI::ProcManager).

Using Linux version 2.6.11.12 (root@colinux) (gcc version 3.4.4 20050314 (prerelease) (Debian 3.4.3-13)) #2 Mon Apr 3 09:12:20 EST 2006

-- PeterPayne - 01 Aug 2008

I've tracked down what is preventing the FastCGI version of TWikiStandAlone to display the WebChanges page. In file lib/TWiki/Sandbox.pm there is a function called sysCommand(). Inside this function there is a block that goes:
if ( $this->{REAL_SAFE_PIPE_OPEN} ) {
...
my $pid = open($handle, '-|');
if ( $pid ) {
# parent ...
} else {
# child
open (STDERR, '>'.File::Spec->devnull()) || die "Can't kill STDERR: '$!'";
...
}
...
}

The line in bold causes the child to die which results in the parent die-ing when it attempts to read $handle with the message "Not a GLOB reference". By simply commenting out the line the search works as expected. But of course what if the process did try to write to STDERR, then we'd have another issue.. so anyway there's the root cause of the problem for me.

-- PeterPayne - 08 Aug 2008

Sorry for the delay, Peter. I'm still working on getting TSA stable on trunk. After solving Bugs:Item5911 and Bugs:Item5846, I'll work on other engines: fix, update and release them (I mean, put them at svn trunk).

-- GilmarSantosJr - 11 Aug 2008

Thank you GilmarSantosJr, I wasn't expecting a quick response, so thank you. If I wasn't lazy I would work out how to redirect STDERR to nul under FastCGI myself. Maybe I'll figure it out and post the answer here. The main thing is knowing where the problem is (and now I know, as posted on 08 Aug 2008).

-- PeterPayne - 13 Aug 2008

I've just tested this issue and I could confirm this behavior. Peter, your trace was perfect! The problem is that FCGI ties STDERR and doesn't seem to handle open correctly. Anyway, within children process it doesn't make sense to STDERR be tied, so the solution is to untie STDERR before redirecting it do /dev/null. I'll fix this before FastCGI engine is released.

-- GilmarSantosJr - 14 Aug 2008

Any updates on this?

-- WoutMertens - 09 Mar 2009

Well, on re-reading this topic after a long time, I think it's about time for an update. As far as I can see, the "real" standalone thing never made it into the core. The script twiki_http.pl is still only in a scratch branch created by Gilmar. However, the core changes he described here have been merged. Based on these changes, there's a current approach for a standalone TWiki as PSGI application. This PSGI engine does pretty much the same thing as Gilmar's HTTPEngineContrib, which seems to be lost from SVN.

So yes, we do have a standalone TWiki today (well, to be released with the next patch version due to a few core changes), and it is mostly based on Gilmar's work, but it took a different implementation path: The PSGI specification (CPAN:PSGI) and the corresponding runtime modules weren't available back in 2008, and PSGI has a slightly different level where the abstraction layer kicks in. A PSGI application can be run from Perl webservers (that's very similar to Gilmar's approach featuring HTTP::Daemon), but also from Apache and various other frameworks, with small glue modules. One could make the PSGI engine the only one, and describe how to deploy TWiki into different web servers, but there's little benefit over the current CGI engine, so I don't think this is going to happen soon. But a standalone TWiki needs a new, pure Perl engine, so this is where PSGI and Gilmar's work meet. The main tangible benefit of PSGI over HTTPEngineContrib is that it works without any changes with a high-performance, multi-threading web server like CPAN:starman.

So, though it took some years until the seed grew: Thanks, GilmarSantosJr!

-- Harald Jörg - 2015-06-22

Topic attachments
I Attachment History Action Size Date Who Comment
Unknown file formatmd5 TWikiStandAlone-4.3.0-auto15428.md5 r1 manage 0.1 K 2007-10-29 - 05:20 GilmarSantosJr MD5 signature for first release pack
Compressed Zip archivetgz TWikiStandAlone-4.3.0-auto15428.tgz r1 manage 4017.6 K 2007-10-29 - 05:29 GilmarSantosJr First release of TWikiStandAlone!
Edit | Attach | Watch | Print version | History: r105 < r104 < r103 < r102 < r101 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r105 - 2015-06-22 - HaraldJoerg
 
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2016 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.