create new tag
, view all tags

TWiki Stand Alone Project: Do You Want Flexibility?

2007-10-26 - 20:33:57 by GilmarSantosJr in Development
I'm a undergraduate computer science student from Federal University of Bahia and I worked with TWiki administration at Computer Science Department and at SaferNet Brasil. I like TWiki a lot, but I'm a little frustrated with its performance and resource consumption. Many times I had to kill apache (sometimes I had also to reboot server) cause of "out of memory" errors. I started to test alternatives, such as SpeedyCGI and ModPerl. I tried also FastCGI but I didn't get it to work. Performance raise with ModPerl is fantastic, but it uses tons of memory, so it didn't solve my problem. SpeedyCGI is very fast too and I thought it was the solution to all problems! But I started to face some instability: user X tries to edit topic A, but contents of topic B, being edited by user Y, appears; Topic contents was lost after save; etc. Then I restricted use of SpeedyCGI to view script only. As my Undergraduate Conclusion Job, I'm playing with TWiki: my goal is to develop some way of increase TWiki performance and allow administrators to control memory usage. This way TWikiStandAlone was born wink

There are many topics within Codev web like "How can I run TWiki with Foo technology?" (e.g: FastCGI, AddFastCGISupport, PersistentPerl, ModPerlUnix, TWikiOnLighttpdUsingFastCgi), and most of them are hacks to get things working. In order to make it possible to run TWiki as a stand alone server and make it easy to switch from CGI to new mechanism I made some changes to TWiki core (inspired on Catalyst framework) and the result is:

  • Do you want to keep running TWiki as CGI? You can.
  • Do you use TWiki from command line? It's still possible.
  • Do you want to run TWiki under FastCGI? You can.
  • Do you want to run TWiki under ModPerl? You can.
  • Do you want to run TWiki as a stand alone server? Let me guess... You can!
  • Whatever you want to, do you want it easy? It is. wink

FastCGI and HTTP stand alone execution are very stable in respect to memory usage, no matter how many simultaneous requests are performed. The biggest usage I have seen along development and tests was about 70 MB in each instance of FastCGI or HTTP processes. Typically, use is 17 MB ~ 25 MB (or 30 MB ~ 40 MB after rendering TWiki.TWikiDocumentation [it is the biggest topic I know]). They are very fast too. CrawfordCurrie showed that compilation time is very big, then any solution that eliminates or minimizes this phase tends to be fast.

ModPerl is also fast, not as fast as FastCGI or HTTP stand alone, but much better than plain CGI. The "problem" with ModPerl is that each apache child (or thread, if you use some multithreaded mpm, like worker) has its own copy of TWiki in memory (believe me: that's a lot of memory) and there is no way of distinguish between static file requests and TWiki requests, so in order to avoid webserver to be sent to swap, administrators need to put hard limits on MaxClients directive, that can leads to wasted resources (many static requests that take longer) or open possibility to DoS attacks (if MaxClients is set to a high value).

And how about develop TWiki without configuring a webserver? I think it is very nice! HTTP execution has an option to restart itself whenever files are changed, so you change something and just browse to see the results (errors are displayed at terminal, instead of webserver error.log...).

Some more improvements can be made to TWiki core, in order to take advantage of persistent execution: to perform initialization tasks only once and share memory (preferences, users, forms...) between requests. I know it's not that simple, but I think some improvements are possible.

I'm very excited with one more possibility: DatabaseStore can be used with HTTP stand alone or FastCGI to easily build a cluster of TWiki installations that leads to very, very scalable sites.

This is a work in progress. Monitor status information for updates. I'm about to release the first "stable" version (probably when you read this post I'll have already done it). Update: Done. Try it out!

See complete details at TWikiStandAlone and feel free to comment, make suggestions, questions and contributions. Development is taking place at scratch area of TWiki svn. Contributions welcome wink


MichaelDaum - 27 Oct 2007:

This is great work and I am looking forward to give it a run.

Still, I wonder why you had problems with SpeedyCGI. It may be worse the effort to fix that too as the very same problem could bite you in the stand-alone version. I am using speedy on a regular base and have a high interest to see standard twiki be fine with speedy. Btw. TWiki.org is speedified too.

CrawfordCurrie - 29 Oct 2007:

I'm also excited about this work - I'm planning to give it a serious look later in the week. I gave up using Speedy a long time ago, due to problems with it hanging on heavily loaded servers, and use mod_perl almost exclusively. As you have observed, that has it's own suite of problems.

MartinCleaver - 29 Oct 2007:

Very interesting. I tend to use FastCGI on all servers with many servers, but its not ideal, and I have to use a weekly reset to eliminate accumulated memory issues.


Edit | Attach | Watch | Print version | History: r5 < r4 < r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r5 - 2007-11-10 - GilmarSantosJr

Twitter Delicious Facebook Digg Google Bookmarks E-mail LinkedIn Reddit StumbleUpon    
  • Help
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2017 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.