create new tag
, view all tags

TWiki Data Handling

TWiki needs better data handling for key=value pairs, like poll results, preferences, forms, attachment information and so on.

There is potential for EditTablePlugin, SimpleTableEntryUsingForms, PollPlugin, FeedbackPlugin and all preferences to use the same functions.

They all have a need for:

  • Allowing a user to enter data in a user friendly way
  • Allowing a user to define entry forms/display forms in a user friendly way
  • Storing the data in a correct manner (locking, handling any type of data, retrieving, deleting just that item, and so on)

Thoughts on how to unify functions like that in TWiki? I think this should eventually end up in TWiki::Store

We could develop a plugin that has one binary that does the work, so that only one binary would be required for the above plugins, and some functions to handle the data. In a later stage we could incorporate it in the TWiki code base.

Data concept:

  • Rows of key=value pairs in tables
    • A table is a named abstract entity that contains rows consisting of columns. Please don't think "TWiki table", "MySQL table" or "HTML table".
    • You can think of the keys as column names.
    • Each row can contain different keys, depending on situation. No effort is done to enforce keys being the same across rows.
    • Example: table "foo" has three rows:
      1. user="WoutMertens" comment="hello"
      2. user="WoutMertens" comment="there" extra_info="more"
      3. user="WoutMertens" extra_info="even more"
    • Traditionally, this would be represented as
      Table foo
      user comment extra_info
      WoutMertens hello empty
      WoutMertens there more
      WoutMertens empty even more
  • Multiple tables per topic
    • needed for saving topic preferences and a poll, for example.
  • Can contain textarea fields, which means embedded newlines I think?
  • Data must be identified with a unique identifier. E.g.: Row number of the current data.

Storage concept:

  • Store it in a topic
    • Data can be stored in metadata
    • Data can perhaps also be added to a TWiki table. See the discussion at SimpleTableEntryUsingForms.
      • This means that the keys must be fixed, or that the table headers must be extended for each new key. Thoughts welcome.
  • Store it in a separate data object not accessible from TWiki directly
    • Could be a database, or a flat file
    • Allows speedier access and better security for sensitive data

Tasks of the binary:

  • Useable as a url target for forms and so on
    • Perhaps allow a url parameter to define the plugin it is working for, so that callbacks can be defined
  • Storing/removing/changing the data
    • Must do proper locking

Tasks of the plugin functions:

  • Listing available tables
  • Retrieving table data for a certain topic
    • Allow a regular expression match on key values
    • Probably easiest as an array of hashes
  • Generating urls that do a certain function with the binary

-- WoutMertens - 22 Aug 2002


I think the data should be stored in topic so that it is searcheable as metadata is now. My reasons for rejecting the database option are many but the core issue is that if this data is "special" it limits its flexibility. You end up with either a lot of duplication or putting the whole wiki in the database. Btter to make it work and optimise later (if there is actually an optimisation to be made by building a cunning backend). "Search" = "query" and filesystem + text = freeform database = wiki concept. There are also complications added by versioning.

I think it is only a relatively small conceptual stretch to modify the existing form metadata handling to do this. I don't think this has anything directly to do with HTML tables, though it does have a lot to do with database tables. Please think database not HTML in the following:

The only things missing in the existing metadata form forms is that you can't have multiple forms per topic and you can't have multiple occurrences of the same field within a form.

I think that one table (database/metadata remember, not query/search result) per topic is sufficient. Nobody (so far) seems to be proposing "transactions" that modify more than one row of one table so there is no need for "writes" to multiple tables (or multiple rows). queries I'll get to later.

If we consider the form definition not as the definition of a form but as the definition of a row of a table all that is required is to allow multiple rows in the topic. This could be done with an extra (row) attribute in the current META:FORM data. There would then be a "form" per row in the metadata.

To use this data one would need to use (nested) formatted searches (with and). Ideally a specialisation of formatted search to do metadata queries in a friendly way would be nice. As this can easily use data from multiple topics, one table per topic doesn't seem too bad. Provide new variables that can be used in the format (and similar ones for elsewhere perhaps) which expand to the url and params to edit the row, add a row, delete a row. If these operations don't go through edit as such and return back to the originating topic, the result should be a system that would allow something a lot like EditTablePlugin or SimpleTableEntryUsingForms but with a lot more potential.

-- DarrylGreen - 23 Aug 2002

I agree on the database not being totally the Wiki way. But on the other hand, META: tags are already invisible for the user (except for searching), and therefore excluded from Wiki interaction.

So if TWiki::Store were adapted so that it stores META tags in a more efficient file format separately from the topic, also in RCS perhaps, I don't think that would hurt, and it would increase TWiki performance. Searching for metadata can only be improved by such a move.

Also note that I would also like the possibility to add to a TWiki table, so that, in cases where it makes sense, the data is readily available in a Wiki fashion.

Also, there seems to be confusion about what I mean by table, since you mention HTML tables. I adapted the text above to reflect what I mean, please reread the text.

I don't agree about one table per topic being enough. Having to use multiple topics for metadata that is linked to a certain topic clutters the WebIndex, makes reporting WebChanges difficult, and is less easy to work with. Sure, you can do it with one table per topic. But it would be a lot easier if you could have more than one.

And I'm afraid you lost me on the form row attribute. Currently we need a table to define a form, and you want to put it in one row to have more than one form (Note that this is just one thing of the many that would benefit of better DataHandling). Or am I wrong? Could you elaborate?

-- WoutMertens - 23 Aug 2002

Hm, I am a bit confused about this discussion.

TWiki allready has implemented a way to represent what you call a table based data structre. TWiki::Meta is a kind of database containing tables (the META:) which are represented as columns (the key) and their rows (the values). Indeed (as you mensioned) there are some limitations when using ::Meta

  • Your own types are not stored automatically
  • It does not support multiple occurences of the same key/value pair within one type.

In addition I also support Darryls opinion about using a database to store meta data. One of the biggest advantages of TWiki is that each topic is just ONE file. The syntax of all data is human readable and searchable by grep. In such way it is perfect to be handeld by a Perl engine.

Imho all your needs could be implemented when rewriting the ::Meta class. For instance you could think of the following implementation.

Pseudo Code:

    Data { 
      @<Entries> ( %{ key, value } )

    void ->put (TYPE, Data)
    @<Entries> ->get (TYPE)
    @<String> -> types()
    void ->store/read/...

-- MarkusKling - 24 Aug 2002

Marcus, could you elaborate? The pseudocode isn't very enlightening frown

For those who don't know, TWiki::Meta currently works as follows: you can define metadata of certain types, and some of those types support multiple definitions keyed on unique keys.

This is what I think should be added to TWiki::Meta: The ability for a TABLE tag defined with a unique name to have multiple entries, and each of these entries is a row in the table. This should probably be done by filling an array.

I don't know how ugly this makes the code, though...

-- WoutMertens - 25 Aug 2002

Oh well its just a blown up hash-hash structure:

  • I used -> for member functions
  • @ for representing an array
  • % for representing a hash
  • for representing the type of @/%
  • return type -> function_name (parameters)

  • ::Meta provides an object called Data which contains lists of key/value pairs (a hash). In addition it could provide convinient methods for accessing the keys or find entries within its managed structure.
  • ::Meta itself has getters/setters for accessing its data structures based on a TYPE (a given name).

Neither were these thoughts enought, nor should they provide a full blown Meta object. It were just some thoughts to make the Meta API more usable.

You could also think of using the DB interfaces and provide a custom MLDBM Serializer to render the meta-content.

-- MarkusKling - 26 Aug 2002

There are some interesting ideas above. However, just extending the meta data to store table data would not be enough, you would also have to make sure that a standard TWiki (without plugins) could edit this data. As alluded to above there would also be problems for searching if meta data wasn't kept in the topic e.g. rename can deal with a parent topic being renamed because the meta data is searchable just like other data.

I guess the key question is what is the advantage in doing the above? I can see that allowing existing meta to reside in a database would have speed advantages, but I've also reckoned the cost of implementing this not worth the effort given the reasonable speed already possible.

-- JohnTalintyre - 26 Aug 2002

Wout, Far be it from me to try to dissuade someone from doing something - if you want to implement some versioned freeform database system integrated into twiki, feel free - I'll be happy to use the result when/if it does what I need.

On the other hand, one could get something up and running pretty quickly with just some small(ish) tweaks to the existing metadata handling and core twiki handling of updates (edits). Various extensions to actually use this could be done incrementally/as/if required. At the end of it one has (presumably) something that does what (some at least) users need/want. That may be enough. There may be a list of additional features, problems, etc that need fixing - the fix may be to go to a "database". The fix may be for the user to use Zope instead of twiki smile .

The above para probably contains wishful thinking - I'm not claiming the knowledge of the core TWiki code to dive in and make some sort of "update metafield" handler work. Anyone who does know the code care to comment?

As to specific issues of what the data is, I'm not seeing the value in multiple tables comprised of multiple rows where the columns making up each row within a table are different! If the columns (keys) in a row can be anything the "table" is meaningless and the "row" is everything. At the extreme we come back to the "key=value" pair as the only entity being dealt with. My suggestion that we restrict the model to having a single table per topic is intended to simplify the data handling side of things without adversly impacting on its utility if the right tools are there. I am quite prepared to concede that the idea isn't perfect though - maybe we should indeed forget database tables and stick with attribute/value pairs, making everything else a presentation - not storage - issue. However, as I think you will see below, this all decomposes into much the same thing ( no, not compost wink ).

To answer your question re. the row attribute, you are once again mixing your use of the word "table" - the "table" that defines a form is a Twiki/HTML table in a different topic (maybe even in a different web). It is a "metadata definition table" that uses a row per attribute to define properties for that metadata attribute. Any syntax would have done - there is nothing special about it being in the form of a table, except that it renders nicely "for free". It is a list (ie 1 dimension) of attribute definitions. I view that set of attributes as the definitions of the columns in a database table. The actual metadata in the topic that uses that TWiki form currently has a single META:FORM element and multiple (1 per database table column) META:FIELD attributes, but has no concept of repitition of the same field (ie rows). I'm suggesting that it could if we considered the form element in the metadata to declare the start of a row, the following field elements to be the columns for that row, etc. This potentially allows the multiple tables per topic that you want, though I'm not convinced that a single table with null (ie. attribute not present) values for columns not present in a given row isn't sufficient (your twiki table at the start of this topic showed just that). Another option in that case is to go to an <attribute, row, value> tuple instead of the <attribute, value> currently used for META:FIELD elements. Anyway, this is detailed stuff, but the important part imho is the last para of my original comment ie. to implement all the get/set/add/delete machinery so that it can be generated anywhere it is needed eg:

%QUERY{topic=%TOPIC%,name="freddy", age="100", format="| $field(name) | $field(address) | $edit(TemplateTopic), $add(TemplateTopic), $copydown, $delete |}%

might produce a table of freddies aged 100 showing name and address, with buttons/links for edit, add, copy and delete. Edit and add use TemplateTopic as the form definition for making the changes. Off the top of my head. Needs more thought....

-- DarrylGreen - 27 Aug 2002

My feelings is that the wiki paradigm works well if everything is in the topic. For instance, for metadata I think the most "wikiesque" way would be to store them in "email" format, like:

   * Author: Colas Nahaboo
   * Date: 20 Aug 2002
   * Subject: A long subject
     Continuated on the next line
This could be hidden inside an HTML comment, and we could decide to use # (or other marker) instead of * to denote metadata

-- ColasNahaboo - 28 Aug 2002

I think we need to separate the discussion here into two threads:

  1. How to allow for more flexible handling of data in TWiki
  2. How to allow for metadata to be represented in a manner that allows for standard TWiki mechanisms to operate on them

Issue (1) deals with the concern that it is still quite cumbersome in TWiki to leverage tables and forms to represent data and analyze that data. Issue (2) is concerned with that we currently have to mechanisms of storing data which requires to different mechanisms for accessing that data.

-- ThomasWeigert - 28 Aug 2002

Edit | Attach | Watch | Print version | History: r11 < r10 < r9 < r8 < r7 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r11 - 2008-09-17 - TWikiJanitor
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2018 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.