create new tag
, view all tags
The current meta data handling can be simplified so that issues like TWikiFormsDiffRendering and IncludeImageInSearchResult can be solved.

TWiki::Store::readTopic() currently returns ( $meta, $text ). All processing of text and meta data needs to be done separately. This can bite as we have seen in TWikiFormsDiffRendering.

The idea is to combine $meta and $text into the $text, e.g. $text will have the same format as it is on disk. That way we have no difference in meta data rendering and text rendering, that is, handleCommonTags() renders also meta data. Diffs will show changes to form data and file attachments nicely because each meta data entry occupies one line.

TWiki::Meta can be changed to operate on $text instead of $meta.

-- PeterThoeny - 25 Sep 2001

I am restating this. The current spec of separate handling of text and meta data complicates things because TWiki::handleCommonTags() is not aware of the meta data. For example:

  • The view and preview scripts need to call TWiki::handleCommonTags() and TWiki::handleMetaTags() separately
  • The rdiff scripot does not render meta data at all
  • It is not so simple to add new variables to TWiki::handleCommonTags() that depend on meta data. For example, how do you add a %FORMFIELD{"FieldName"}% variable as requested in FormattedTWikiFormDataInTopicText.
  • The TWiki::Func::* functions for the Plugins do not yet expose the meta data. Doing so with the current Store interface would complicate the Plugin API.

Ideally, all topic data should be handled as one text stream, and that stream gets changed along the way until it reaches the browser.

In procedural speak we could keep the text stream in memory as it is on disk, e.g. text and meta data combined. A meta data object is built when we need to manipulate the meta data (add/remove/change a record), but a fast regex could be used for simple references (e.g. when expanding a meta data variable).

In TWikiOO an object would hold text data (text and meta data), the object would get passed to TWiki::handleCommonTags() or a render object.


-- PeterThoeny - 18 Jul 2002

I agree that it would be much nicer to have meta and text merged into one, rather than treating them seperately like they are now. It seemed to me that at various stages throughout the processing of a topic that the meta and text data are available at times, and not at others and it makes sense to me that access to these data should be uniform throughout the code (where it's supposed to, that is). Perhaps somebody who knows more about it than I do would like to start suggesting an outline for a good way to do this? Or will it all just be re-done anyway in TWikiOO...? Again, if I'm missing something obvious or am misunderstanding things, let me know.

-- DavidSachitano - 19 Jul 2002

Um - I'm confused - if the metadata is separate why does the problem I described in AccidentalMetaDeclInTopicTextIsNasty occur? In any case it seems necessary to somehow separate metadata from text if metadata is to remain "meta" and not get accidentally (or otherwise) edited as part of the text. Is there some sparator that can be used to delineate text/meta? The freeform nature of the wiki text makes this messy - it would perhaps be easiest to define something that preceeds the text that has a more restricted format and allow freeform after it. The metadata could go in the "restricted" bit? This makes me want to suggest that the restricted bit should be XML and the metadata should be too - big change though.... Doesn't sound much like "simplifying"....

-- DarrylGreen - 22 Jul 2002

A solution would be to put metadata in the body text like now, but before a separator that the user cannot enter, which would separate the metadata from the text.

Conceptually, it is close to an email, which is comprised of:

  • header
  • blank line
  • body
Thus a TWiki page could be
  • metadata (in the current form: %META:...{...}% so that current code would still work
  • separator: I suggest a control character: ^@ (null) if perl can handle it, or ^L (formfeed), ^K (vertical tab), ^^, ^_ ...
  • body text, as before, where occurences of %META: could be ignored
By putting the metadata at the head we can accelerate metadata handling for big pages.

It could thus be eqasily implemented: just:

  • fix metadata code to add metadata always at the head, and add the separator if not there
  • fix text retreieving code to skip to the separator if found. We can decide to specify that the separator should be present before and after the metadata to speed up recognition.

-- ColasNahaboo - 09 Sep 2002

I think meta data would be better at the top of the file. In fact it used to be there, was split in half after a previous request. Also I agree that meta and text should be passed as one object. How about adding the text to the meta object?

-- JohnTalintyre - 09 Sep 2002

Separators should be chosen carefully if that's the way we go - e.g. the $TranslationToken broke Korean and Polish character sets (see NationalCharTokenClash). Null (\0) is now used internally as the translation token - something like Ctrl/A (\001) should be OK though, since i18n character sets (hopefully) don't use this. See InternationalisationEnhancements for more.

-- RichardDonkin - 26 Nov 2002

If the meta data were grouped at top of the file with one line per meta element, there would be no need to use a separator token. This avoids any potential problems with the choice of token. If the meta data were stored this way, The start of the text is easily found by searching the first line which does not start with '%META'. If stored in this fashion, the start of the meta data should also be obvious. smile

-- TomKagan - 27 Nov 2002

Right! this is the simplest solution. One should process in batch all the pages of existing wiki sites to move metadata of existing pages at top before installing the new version, however. (otherwise old metadata will be ignored).

-- ColasNahaboo - 27 Nov 2002

The current file format of meta data is flexible, fast and powerful. I do not recommend to change it unless there are compelling reasons. The meta data is split in two parts because of performance, META:TOPICINFO (user, date, version number) and META:TOPICPARENT are on top so that it can be retrieved quickly, other meta data is below the topic text. It is line oriented so that it can be processed quickly with regular expressions.

The new PluginApiForTopicHandling hides the meta data class from the Plugins. My idea is to rework the internal meta data handling (not in BeijingRelease smile ) to work on the serialized data (e.g. the file format) instead of the ( $meta, $text ) duality. A set of functions or a class can provide the same functionality like the current TWiki::Meta class, but working on the serialized data.

-- PeterThoeny - 01 Jan 2003

I was recently able to add a form to a topic by typing in the %META{"blah"} in the topic. I think it should be added to TWiki.TWikiVariables. It's very useful for when you don't have the rights to change (for example) WebPreferences to add a form to it.

-- GrantBow - 17 Jan 2003

Heaven forbid! Please, no, not that.

The underlying problem with TWiki is that it is too oriented to programmers and people working "under the hood". We already have that enough of problem with the access control being in-band. We don't want more.

The users are not programmers, nor should they be, nor should they be faces with having to know or manipulate such implementation-specific details. Even if they are not required to use it, presneting them with this is just another psychological discouragement.

Using "tools" such as the present ones and the ones in MetaTWiki hides the implementation details from the user, and more to the point, gives the developers the freedom to vary and refine the underlying mechanisms. This is such a fundamental aspect of good system design that I have to attribute the pervasiveness of implementation detials being visible to the user to either this being a poorly refined (i.e. not production quality) product or to it being too "programmer oriented".

-- AntonAylward - 17 Jan 2003

The mention of a bent toward creeping "in band" META style settings when it should be "out of band" is worth exploring, in my opinion. As I understand it, the META data is meant to contain information about the page. The page itself is meant to hold the information contained within the page. If this is the case, then, I agree, the method created to provide access control to a page is inherently a kludge.

I think a lot of uncertainty in deciding the most appropriate place to put information (META vs page text) could be alleviated by writing a generic META editor for TWiki. (This might work similarly to the 'edit' script already provided for the actual page text.) It would also be a good idea to finally nail down the incomplete META API for plugins. Done well, Such an API might make it possible to extend a generic META editor control to allow for customization by a plugin for its own purpose. (Maybe some ideas in Squirrelmail's "plugin expandable" user options editor might be insightful.)

I suspect the page access control directives would rightly have ended up within the META data if the META API/edit issues were already worked out before the need for access control arose. Oh well. I guess nothing is stopping anyone from fixing it before it gets too far out of hand (other than a lack of time, skill, or familiarity with the code smile ).

-- TomKagan - 17 Jan 2003

I don't know if there is anyone else still dreaming of it, but I still hope one day to have the ability to have my META data in a database. This is still part of the WorkFlow idea, which is partly lost in the depth of TWikiIdeas

-- SvenDowideit - 18 Jan 2003

I would also like to see the option of database storage. The existing meta code is OO based and it would be relatively easy to have one implementation for databases and one for the file format. That said it wouldn't be difficult to convert from database form to file format and back again. It's also worth pointing out the meta code does have some unit tests.

-- JohnTalintyre - 18 Jan 2003

That is mainly a design question. Perl is good for string manipulation, data in serialized string format is very portable. A database storage backend can convert internal data into serialized format for TWiki.

External meta data handling for Plugins is discussed in PluginApiForHandlingMetaData.

-- PeterThoeny - 28 Mar 2004

Indeed, this is a design question that I've wanted to discuss for a while now. While I understand your arguments about Perl being a good string language and serialization being useful, I disagree about the implication that the best way to store data is in a serialized format regardless of the actual structure of that data. I cannot believe that reparsing and recreating the meta object from the text every time it's needed is more efficient than creating it once and passing it along is parallel with the topic itself. Sure, regexes could do simple changes, but no matter how good Perl is at regexes, a hash write to make those same changes is always going to be faster still.

The whole idea of metadata is that it is information that does not quite fit inside the structure of the entity it describes. I completely disagree with keeping the topic in interleaved format anywhere. If we need the serialized format, for storage or communication, it can be generated. The conversion from structured to serialized data is a lot faster and simpler than the reverse anyway.

You're looking at things backwards. Yes, Perl is great at text manipulation and regex matching. No, that doesn't automatically make it the right decision to try and shove every problem into that paradigm. What needs to happen is not the interleaving of metadata, but rather the logical attachment of metadata to the topic text. Everywhere the text is passed, the meta object needs to go, too. That way, many of your concerns about the difficulty of getting at metadata vanish, because it's right there in a hash, waiting to be accessed.

-- WalterMundt - 28 Mar 2004

i agree with Walter - i think our current implementation is less flexible and exposes an implementation detail (the fact that at the moment everthing is in one file)

-- SvenDowideit - 28 Mar 2004

I also agree with Walter. But, I would say that the current implementation does expose the detail that much, given that almost all access is via the Meta object. However, certain parts of TWiki e.g. searching, do assume the existing implementation.

-- JohnTalintyre - 30 Mar 2004

Edit | Attach | Watch | Print version | History: r23 < r22 < r21 < r20 < r19 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r23 - 2004-03-30 - JohnTalintyre
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2017 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.