Tags:
create new tag
view all tags

What does compatibility mean in terms of future releases?

These are the (very broad) possible models than I can think of:

  1. Pre-Edinburgh installs have to stay pre-Edinburgh
  2. Edinburgh doesn't change the existing topic structure, etc., and only adds things that don't break existing installs
  3. Pre-Edinburgh installs get upgraded to the Edinburgh model via scripting
  4. Edinburgh supports both old and new store, etc., models simultaneously.

Model 1 implies that pre-Edin installs will continue to run like molasses, but Edin+ installs (in theory) are quick quick.

Model 2 implies that all TWiki installs will run like molasses.

Model 3 is enormously difficult to get right and will inevitably require some manual effort. It means, however, that previous installs will run quick quick after the conversion.

Model 4 allows Edin+ new installs to run quick quick but allows sites to upgrade. Whether upgrading sites without converting to a new model is beneficial is unclear.

-- Contributors: MeredithLesly

Discussion

We have a dilemma here: Existing sites shouldn't be abandoned, yet getting TWiki right may require some degree of incompatibility. Ironically, given the (previous?) focus on Intranet, companies are reaping the benefits of of our unpaid labour. Or, in some cases, people are being paid to create/maintain installs and they are also reaping the benefits of our unpaid labour.

Newcomers like me and at least some of those who maintain a handful of sites at most have no intrinsic need to maintain compatibility, other than the abstract notion of "backwards compatibility is a Good Thing." If compatibility weren't an issue, then we would be free to rip things apart and make them right. But that's not an option. So perhaps there's a need for a TWikiFoundation funded by these companies so that there's an alternative motivation for those doing unfun work to maintain compatibility.

-- MeredithLesly

Compatibility is enshrined in the TWikiMission. There are good reasons for this. It is forcefully mandated, to the point where you have to look carefully at each change and decide if it is "TWiki" or "Not TWiki". There is no option to release "TWiki" unless it is compatible. If it is not compatible, it is "Not TWiki".

Unfortunately there is no "TWiki spec", and there are many inconsistencies, so it is hard to know when that "Not TWiki" point is reached. The only solution to this is to take Peter's approach and be ultra-conservative about the definition of "TWiki" (throughout Dakar development it was "Cairo", even to the extent of reproducing Cairo bugs in Dakar). Peter is the final arbiter of "Not TWikiness" (though he is quite prepared to break his own rules if he thinks it is appropriate).

  • Can you give some examples of "reproducing Cairo Bugs in Dakar"? Not that I do not believe you but I seem to have missed that. -- Kenneth

While I understand the drive for stability, I personally believe that TWiki is a long way behind the curve, and to make real progress, we have to push beyond the "Not TWiki" point. I am specifically thinking of the architectural decisions made early in TWiki the project which were right at the time, but are now insuperable barriers to progress. These are what I call the P's; protections, preferences, plugins and perl. You are already touching on one of these, plugins.

We can decide now; is Edinburgh release a clone of Dakar (100% compatibility)? Or do we take on the P's in the hope of arriving at something so significantly better, that it blows questions of compatibility out of the water? Are you willing to take the risk of developing something on DEVELOP branch that is subsequently branded "Not TWiki"? The last four months of Dakar release involved reeling DEVELOP back in to be "compatible"; and I can't recommend it as a life experience.

Note that while I personally do get some benefit from all that unpaid labour during Dakar development, it is not enough to pay the bills. I'm not sure that a TWikiFoundation would do any better.

-- CrawfordCurrie - 10 Apr 2006

Thank you, Crawford, as you are one of the people who best understands what compatibility vis a vis TWiki means.

I think it's important to decide the main question you raise as soon as possible. As you note, it has critical implications as to what people are willing to do in DEVELOP.

Correct me if I'm wrong, but doesn't 100% compatible with Dakar imply minimal performance improvements for existing installs? Even if the refactorers manage to organise the code enough to allow for pluggable stores, etc, that drastically improve performance, that would only benefit new installs and those older ones who are able and willing to convert. At the very least this would create some awkward marketing: "50% faster, unless you run in compatibility mode".

Or, in brief, will "TWiki" and "not TWiki", as you defined them, be allowed to co-exist in Edinburgh? As Peter is the final arbitor of non-TWiki, it would seem essential to hear what he has to say about this.

-- MeredithLesly - 10 Apr 2006

I'd like to see if its possible to have a twiki-core thats agnostic about wiki compaitbility, with the ability to add the twiki-compatible personality - or a mediawiki personality etc

-- SvenDowideit - 11 Apr 2006

Out there in the real life there are probably a couple of 1000 TWiki installations that are in serious use. Ie. many users making many topics every day. Their TWikis grow in size (number of topics and number of users). Sure they have wished for all sorts of features but the really important one are.

  • Make TWiki faster so it can keep up with the growing number of topics and users.
  • Make TWiki easier to use for non-techies (better Wysiwyg)
  • Please protect the huge investment we have made in information contained within the millions of topics.

I firmly believe these are the fundamental requirements from our users. Our admins are also crying for easier upgrades and less trouble with dependencies but the real requirements are the 3 above.

And this is where the compatibility becomes important. And also where the TWikiMission may need a slight adjustment to be able to meet the challenges to come.

  • Today TWiki is coded in a way we all know could be more effective. Viewing even the most simple page takes too long on a loaded server.
  • Today TWiki keeps data in flat text files, history in RCS files and metadata is inline. This makes searches take forever when the number of topics grows. And the search feature is probably one of the strongest selling points in TWiki. The ability to create topics that are dynamic in nature built from content of other topics is a unique and great TWiki feature.

So what does this mean.

  • The TWiki code can be completely refactored. Maybe a different language than Perl. And doing that will demand huge effort. It will create a compatibility issue with respect to plugins but not necessarily to topic content. With an effort on recoding plugins I do not see this as a show stopper. But how more effective would another language than Perl be? I assume you do not think of writing TWiki in C or assembler.
  • The data can be stored in indexed databases with meta data like access rights stored seperately so it can be searched magnitudes of times faster. Here topic compatibility will be really difficult to maintain. But I do not believe that it is a "mission impossible" to do it.

I see no reason to throw the towel in the ring. But we also have to accept that is takes resources that we do not have available. We have to accept that the work has to be planned and done in small manageable chunks. And this means planning the work and devide it up in small manageable chunkc. You can eat an elephant if you it it in small bites.

Crawford hits an important point above.

  • We do not have a "spec" or even a full set of test cases. There is work to be done there both for coders, testers and doc people. And it will allow a less conservative approach than status quo.

On the performance side there are two major tasks.

  • Profile the code and find out where the slow performance comes from in basic topic view without searches. We only need to half the execution time. And that should be possible - even on the perl base.
  • We need to get an alternative storage of topics than 100000s of text files. TWiki can never scale with the current model. And it is the current customer base in the large corporations that have this issue. Small private sites and small departments live well with the current model. It is also the large corporations that needs compatibility. So this is where we have a chance to proof how brilliant we are. But it will not be easy and we should try and find some corporations that will sponser some of the work either as money or as human resources.

But corporations will need to believe that TWiki will continue to exist as TWiki before they even consider putting resources in it. Otherwise they will put their money elsewhere. And we will need to know what we ask for. Ie. we need a plan and a well defined work package breakdown. Volunteer non-payed contributors can work ad-hoc but if you want corporations to invest they will want to know what they get and what to do to get it. I am sure I could get my own company to sponser university students to do chunks of work if it was well defined. We have requests from universities all the time but have trouble finding suitable projects. TWiki could leverage a lot from these potential resources.

And finally. If we change storage model once and for all - the "no upgrade scripts" statement in the TWikiMission will need to be waved in this case. And the admins would most likely gladly accept it if the gain is future proofing TWiki and give the scaleability and vitality. But I agree that upgradescripts should not be the solution to choose when there are good runtime alternatives.

-- KennethLavrsen - 11 Apr 2006

Kenneth said:

_ We need to get an alternative storage of topics than 100000s of text files. TWiki can never scale with the current model._
We must think if the current data representation - the topic model - as a "proof of concept" that ahs persisted, persisted so long that it is now a liability!

-- AntonAylward - 11 Apr 2006

On-the-fly form update is what happens when you do something like running a new version of Word. It is definitely a friendlier solution than upgrade scripts. The problem is that we're not talking about the same sort of interaction. A single Word document does not refer to hundreds or thousands of other Word documents. And therein lies the difference between application-style on-the-fly updating and what TWiki needs to do.

As Kenneth notes above, "the search feature is probably one of the strongest selling points in TWiki." Many, perhaps most, topics refer in some way to other topics, most significantly via SEARCH. Updating the format of a topic that contains a SEARCH on the fly is fairly meaningless unless the topics within the SEARCH scope have been updated as well. That clearly means that all the updating of topics has to be done before they are referenced by other topics. And that means doing via an update script.

The only other alternative that i can think of is to rewrite SEARCH to be hugely faster using the current document format. I could be wrong, but I don't see that happening.

-- MeredithLesly - 11 Apr 2006

I think we all agree that future versions of TWiki must be compatible by some definition of compatible. That's why I created this topic in the first place, after all: compatibility is key, no matter what other whizzy features we can imagine adding to TWiki.

The key lies in what Kenneth says above: "We need to get an alternative storage of topics than [tens of thousands] of text files. TWiki can never scale with the current model." That means that TWiki has to have a different model, period, because it's vital to the usability of the existing TWiki base.

Yes, we can have a "classic" store which allows companies to continue to use their thousands of documents as is. That provides 100% compatibility. It also is and will be unacceptably slow for far too many users.

Assuming that what I've just said is generally agreed with, an important philosophical step must be taken: modifying the TWikiMission statement. I don't believe the current one is compatible with a changed store model and what's required for existing installations to use it. Doing this will help to clarify what Peter will deem "TWiki" for Edinburgh.

-- MeredithLesly - 12 Apr 2006

Edit | Attach | Watch | Print version | History: r11 < r10 < r9 < r8 < r7 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r11 - 2006-04-12 - MeredithLesly
 
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.