The topic file format in extended BNF notation is as follows:

    topic-file ::= header-meta-lines topic-text-lines footer-meta-lines
    header-meta-lines ::= meta-line { meta-line }
    footer-meta-lines ::= meta-line { meta-line }
    meta-line ::= meta-identifier upper-alpha-chars "{" meta-parameters "}%" end-of-line
    meta-identifier ::= "%META:"
    topic-text-lines ::= topic-text-line { topic-text-line }
    topic-text-line ::= printable-chars-not-starting-with-meta-identifier end-of-line
    end-of-line ::= "\n" | "\r\n"

This is a simplified picture, an exact BNF description of TWiki's topic file format would be much longer.

Notice that the end of line can be platform dependent, the parser should support any type of line ending.

Using meta data in the topic text is not supported. To document TWiki meta data in topic text it should be escaped the same way like other TWiki variables. It is recommended to insert a <nop> after the percent sign, e.g. type %<nop>META:TOPICPARENT{name="WebHome"}% to get %META:TOPICPARENT{name="WebHome"}%.

-- PeterThoeny - 07 Feb 2004

Discussion

Recent changes to TWiki's store mechanism have radically changed the ondisk topic format.

This appears to have been done without realising it.

Prior to the change on MetaDataHandlerCantProcessCrLfLineEndings (and the change that led to that change) the structure of topic storage was as follows: (Most general form follows - I'm not doing specific META lines.)

topictext := header-meta topic-lines trailer-meta
header-meta:= meta-lines
trailer-meta:= meta-lines
meta-lines:= meta-line meta-line*
meta-line:= "%META" anychars unix-end-of-line
topic-lines:= topic-line*
topic-line:= anychars network-end-of-line
network-end-of-line := "\r\n"
unixendofline := "\n"

In practice there are further constraints on which META lines appear at the beginning/end of the file.

Loosely speaking this means the structure is:

• META lines with normal termination
• topic lines terminating with "^M" (or "\r\n")
• META lines with normal termination

This means that you can clearly delineate topic text from meta data. It also means that when reading/writing from/to disk lines get stored in the "right" place.

The recent change made on MetaDataHandlerCantProcessCrLfLineEndings breaks this semantics dramatically. The definition now (6/Feb/4) for topic interpretation/storage is:

topictext := lines
lines := line line*
line := meta-line | topic-line
meta-line:= "%META" anychars end-of-line
topic-line:= anychars end-of-line

Loosely speaking this means a topic text consists of

• A sequence of any number of lines
• Whereby all lines matching metadata format are treated as metadat
• All other lines are topic lines.

This definition of intermixing of metadata and topic text throughout the topic text, in a potentially non-edittable format is a huge semantic change for such a small lexical change. It breaks many, many concepts of wrappers, and is a huge mistake IMO. The ability to misuse this to lock topics from edit (even from the Admin unless they jump through hoops or do so on the server) is a huge issue.

Of course it also breaks any plugins that are forced to parse topics (through lack of a published metadata interface) and have assumed the old structure.

Contributors

I inserted the file format description on top. Please note that this is based on the existing TWikiMetaData documentation. The recent change in line endings is in line with the documentation.

It is also in line with the code since TWikiRelease01Feb2003, that is, any unknown meta data found anywhere in the topic text is moved to the end of the file on topic save. See MetaDotPm, CVS:lib/TWiki/Meta.pm, sub read and sub writeEnd of revision 1.2, 29 Dec 2002.

Crawford, could you explain what kind of issues you have with Plugins? Is it something else besides making the Plugin agnostic to "\n" and "\r\n" line endings?

Regarding Plugins, we will add functions to manage meta data. In the mean time, Plugin authors should assume the recommended meta data sequence as documented when manipulating text and meta data.

This is not documented yet, but the TWiki code already handles additional meta data introduced by Plugins transparently. A Plugin can introduce new meta data by storing it after TWiki's footer meta data; TWiki will preserve any unknown meta data.

-- PeterThoeny - 07 Feb 2004

[...] my description wasn't based on documentation, it was based on how the code actually works, and has done for a very long time.

For comparison how other fields of standardisation that affect large number of people handle this sort of thing...

The HTTP spec defines a "Referer" (sic) field. By the time they noticed their error it was too late to correct it, and chose not to go around breaking things on people, even though it would've been more correct to the intent of what was written rather than what was actually written. [...]

-- MS - 07 Feb 2004

What I see on my file systems, linux, unix, and windows, is the following

    topic-file ::= header-meta-lines topic-text-lines footer-meta-lines
    header-meta-lines ::= meta-line*
    footer-meta-lines ::= meta-line*
    meta-line ::= meta-identifier upper-alpha-char* "{" meta-parameters "}%" end-of-line
    meta-identifier ::= "%META:"
    topic-text-lines ::= topic-text-line+
    topic-text-line ::= printable-chars-not-starting-with-meta-identifier "\r" end-of-line
    end-of-line ::= "\n"

-- ThomasWeigert - 08 Feb 2004

This observation is correct. It is in line with the "\n" | "\r\n" line ending stated on top.

-- PeterThoeny - 08 Feb 2004

Peter, the problem I alluded to above occured in the Action Tracker, where the action editor is invoked to edit an action in a topic. To do this, the beforeEditHandler has to remember the text before the action, the text after the action, and the META fields, so it can restore them when the action edit is complete. The first two (text before and after) are recoverable from $_[0], but the meta doesn't get passed to the beforeEditHandler so I had to re-read the topic and extract it from there. Very nasty. I have previously reported this problem several times over the last couple of years. See ActionTrackerPlugin.pm, search for "Oh how I wish" and "If only we had"....

-- CrawfordCurrie - 08 Feb 2004

Crawford, I too have often bemoaned that meta data is not being passed into the APIs accessible to Plugins. In InconsistentApiCallsInRendering, I have requested that the meta data be passed into various API calls to allow plugins the manipulation of meta data. While this has been rejected as a bug report, PeterThoeny stated that "A future TWiki::Func API will have functions to manipulate meta data."

-- ThomasWeigert - 08 Feb 2004