twiki_func1Add my vote for this tag create new tag
, view all tags
NOTE: Below doc does not represent the actual implementation. A new TWikiFuncEachChangeSince topic should be created documenting the functional spec

The Store module provides APIs for accessing core web meta-data, such as "changes" etc. These are conventionally stored in '.changes', '.mailnotify' etc, but that might change with a different store impl (which is why the store exports those APIs). Plugins also need to access that meta-data - for example, for processing changes. This is different to plugin work areas; this is access to meta-data that the core generates and stores alongside topics.

Proposed API:


readWebMeta($web, $name) -> $text

Read a web meta-data file. Web meta-data is data relating to the web that TWiki stores along with the web. Examples of web meta-data include the list of changes. Note that extensions should not attempt to store web meta-data; it is for the cores use only. Extensions can use work areas to store their own specific web-related data.


-- Contributors: CrawfordCurrie - 03 Mar 2007


If I understand correctly, a readWebMeta( "Main", "changes" ) returns the content of the .changes file? If this is the case I think we risk to expose too much implementation detail. What use cases do you have for this function?

-- PeterThoeny - 05 Mar 2007

The mailer needs to get the changes. That's the only one, at the moment. However the alternative is to short-circuit the store and read a file from disc, which is a really, really bad idea (who says changes have to be logged in a file?)

-- CrawfordCurrie - 05 Mar 2007

I find it problematic if the API offers a readWebMeta( "Main", "anything" ) because it is not a clearly defined spec. We could document what the "anything" can be, but then it needs to be enumarated (changes, mailnotify) and documented in details for every item. However, if we document the format of the .changes file (or database equiv) in details, it would be officially exposed and we can't change it later easily.

A more solid approach is to define an API that just returns the .changes content in a clearly defined format, such as an array of hashes. If needed add another function to set/add content to it.

-- PeterThoeny - 16 Mar 2007

Yes, I see your point and it's a good one. .changes is in fact the only web meta-data source at the moment; my concern was simply that that may not be the case forever.

Let's just take a step back for a moment, and consider what the .changes file really is.

.changes records a set of topic-user-time-rev tuples that record the history of recent changes in the web. .changes duplicates information about the web that already exists in the revision histories of the individual topics. It's a potted history of the web. So, let's consider whether .changes is really necessary.

What is a change? Well, a change is a record of an event; a save event to be precise. If we had an eventing model, then the mailer would be quite happy to receive and process that event when it happens. This would allow nice stuff like on-change notification. In a cron setting the mailer could record its own history of changes at the time they happened, and using it's own time horizon, instead of having to try to reconstruct history from .changes (an imperfect process, as data is lost between changes e.g. when a topic is renamed). If you think of Changes.pm in the same way you can see that it could also add an event listener and maintain its own records - in its own appropriate format - for use in changes reports.

That's an ideal. We don't have a proper eventing model right now, so we can't do that without a lot of pain. But it paints the right architectural direction for solving this problem.

OK, so we have the idea that change listeners like Mailer and Changes need to be able to process change events. We also have the situation that change events are not dispatched at the time they are raised. So we need some sort of compromise that lets us replay the change event history. Taking this approach would allow us to define what look like event listeners in the Changes and Mailer code, so that code could easily be ported when Dispatcher pattern eventually does appear.

I hope you all followed my logic here. If you did, then you will understand the power of this proposed API:


TWiki::Func::addEventListener($eventName, \&listener($clientData, \%event), $clientData)

  • $eventName - currently this must be SAVE. Only save events are recorded.
  • \&listener - reference to an event listener function (see below).
  • $clientData - data that will be passed on to the listener when it is called.
Register an event listener. An event listener is a function that is called when an event happens. There can be many event listeners registered for a single event. Event listeners take two parameters; first, the $clientData that was passed when the listener was registered, and second, a reference to a hash that describes the event. This hash will contain at least the following fields:
  • change - currently always 'SAVE'.
  • web - the web the event occurred in.
  • topic - the topic affected by the event.
  • time - epoch time the event occurred at.
  • extras - attributes on the change, as a string - e.g. "minor".
  • wikiname - wikiname of the user who caused the event.




In the absence of a true dispatcher in the TWiki core, this function causes the dispatch of all currently queued events.


"All currently queued events" at this point means "all changes". So to recover the content of .changes, a client does this:

TWiki::Func::addEventListener('SAVE', \&saveEventHandler, $data);
TWiki::Func::replayEvents(); # will probably be a NOP at some point in the future
sub saveEventHandler {
    my ($data, $event) = @_;
    print STDERR "$event->{topic} was saved at $event->{time} by $event->{wikiname}\n";

With this API in place we have an immensely general and powerful way to handle events, as well as a clean approach to transitioning TWiki to a true eventing model. The plugin/contrib author is cleanly isolated from the specific store implementation. We can even start tucking other events away in .changes, such as topic renaming events.

On the downside it requires some recoding of Changes.pm and Mailer.pm, but the changes are limited and well isolated.

-- CrawfordCurrie - 17 Mar 2007

Actually, most of the info in .changes is in the topic revisions. The only additional element (or lack thereof) is the "minor changes" flag. If that is set during save, no record is added in .changes. So, a logical solution is to add the "minor changes" state to the revision history. That way, the mailer could simply do a recent changes SEARCH, and for each of the topics look for the minor changes flag. This should be done in all revs since last notification.

An alternative is to pass the "minor changes" flag to the after save handler. That way, the extension can build its own changes history.

The event model is a nice idea. I think it needs to be fleshed out in anticipation of other scenarios.

-- PeterThoeny - 17 Mar 2007

The reason the mailer can't "simply do" a recent changes search is the same reason I suspect you invented .changes in the first place viz. it would be horribly slow. You have already complained about mailer performance, and that certainly wouldn't help frown BTW minor changes do go into .changes, they are tagged as "minor".

A better approach is provide an API that comes back with an enumeration over the available changes. That way the Store can choose whether to implement the changes in terms of a separate file, or embedded in the topic - i.e. it finalises the Store encapsulation, and not before time!

For Func, I propose to add the following to Func:


eachChangeSince($web, $time) -> $iterator

Get an iterator over the list of all the changes in the given web between $time and now. $time is a time in seconds since 1st Jan 1970. Changes are returned in most-recent-first order.

Use it as follows:

    my $iterator = TWiki::Func::eachChangeSince($web, time() - 7 * 24 * 60 * 60); # the last 7 days
    while ($it->hasNext()) {
        my $change = $it->next();
        # $change is a hash that contains the following fields:
        # topic => topic name
        # user => wikiname - wikiname of user who made the change
        # time => time of the change
        # revision => revision number *after* the change



Has Peter and Crawford agreed on the specs by now?

-- KennethLavrsen - 19 Apr 2007

I am fine with Crawford's latest proposal, it is much more specific than the first one. The name ProvideAccessToWebMetaData does no longer reflect the purpose of this proposal, better to track in a new Codev topic (and have it accepted immediately based on this topic history.)

-- PeterThoeny - 20 Apr 2007

Consensus reached. Proposal implemented by Crawford.

-- KennethLavrsen - 25 Apr 2007

The topic name ProvideAccessToWebMetaData does no longer reflect the purpose of this proposal. A new TWikiFuncEachChangeSince topic should be created documenting the functional spec of the actual implemenation.

I am setting this to RejectedProposal since the original proposal was not implemented. Please create the TWikiFuncEachChangeSince topic and set that one to MergedToCore.

-- PeterThoeny - 08 May 2007

A long forgotten topic, code is implemented, but TWikiFuncEachChangeSince topic is missing.

-- PeterThoeny - 28 May 2008

Edit | Attach | Watch | Print version | History: r13 < r12 < r11 < r10 < r9 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r13 - 2008-05-28 - PeterThoeny
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2017 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.