database_store1Add my vote for this tag meta_data1Add my vote for this tag create new tag
, view all tags
Related topics: DatabaseForPreferences PrefsDotPm AddValuesToPreferenceVariable MetadataSucks MetaDataSucksMeasurement StepByStepRenderingOrder DatabaseForAccessControl SimplifyInternalMetaDataHandling MetaDataDefinition DatabaseForMETA MegaTWiki

There is no reason that a database similar to that described in DatabaseForAccessControl could not contain all a topic's META-data.

If the database is a HASH:

  • Search ~twiki/data/%WEB%/.metadata.db, key %TOPIC%
  • Returns all the metadata just as if it had been extraced from the curent style topic using a 'grep'.
  • The order of the lines is preserved becuase it is one object.

Alternatively, something othr than a hash can be used.

  • Berkeley DB_TREE allows more than one use of a key.
    I beleive that order is not preserved, however.
  • We could use a flat text file with the topic as a prefix key. A grep-like oepration would extract all the fields in the correct order.
  • Since ultimately the information gets stored in a perl data strcture, we could use one of the perl tools to store that data structure.
    • Data::Dumper - dumps the structure in a form that can easily read in by perl becuase it is a perl statement.
    • 'Storable' module -
      • This is fast!
      • The downside to 'fast" is that its a C extension and uses a binary format.

-- AntonAylward - 04 May 2003

I would certainly like to see the ability to store meta data in a database, but there's a lot of hard slog in such a move, especially in maintaining backwards compatibility. At present you can just move a topic file from one TWiki to another - we wouldn't want to lose this. Additionally meta data is under full version control - again something we don't want to lose.

For this to fly someone has to do a lot of coding and testing. It took me a fair amount of time to produce the current meta data code and a lot of difficult testing to make sure that old topics were upgraded automatically. There was a general reluctance to insist that people run an upgrade tool, so the upgrade code is still in place and could be called upon when someone views or edits and old topic.

As it happens the current code is OO based which would help when extending to a database. However, as Peter has pointed out the OO code doesn't sit terribly well with the rest of TWiki code base. Please note that there are a few unit tests for this code under tools.

-- JohnTalintyre - 28 May 2003

As is bemoaned and discusses elsewhere, if we had a proper installation and upgrade mechanism such s RPM, then the "upgrade tool" would run automatically and invisibly to the user.

The current coder is most certainly NOT OO, as I'll be showing shortly in a series of topic devoted specifically to that subject. The metadata code has been developed along OO lines but John's point about things like revision and renaming and moving make it clear that this limited OO context is not enough. A Topic as an Object rather than a set of files - body, revision history, settings DB, metadata DB, access control DB - that is managed by a Storage Object in the contxt of a Web Object would make all those impediment (I nearly said objections!) vanish.

I've been giving this a lot of thought, a lot of white-boards and a lot of effort with design tools. It has many ramifications and cannot be described piecemeal as the above observation makes clear. Not least of all those that are needed to maintain compatibility with the existing topic files and alow an upgrade path.

In the mean time, any and all who contribute, by partial enhancements and experimentation, are valuable. TWiki development has slowed and blocked. I hope my OO relevations will shatter the mental log-jam that has arrisen.

-- AntonAylward - 29 May 2003

Moving whole page with all metadata is valid point. Will it work seamlessly if Twikis are different releases? If not, we need upgrade tool. IMHO going without upgrade tool was not good idea - upgrade is major pain and slows Twiki acceptance. I am not OO guru but I can imagine creating program which will puul page with matadata ie in XML format and will import it into another Twiki. Big part of it is probably just using proper CPAN modules.

-- PeterMasiar - 29 May 2003

Some of the TWiki is OO based. Above I referred to the meta handling code, which is certainly OO and always has been. Much of the storage backend has already seen significant recoding to an OO form. The choice of not having an upgrade tool was very deliberate. Having to run such a tool is an extra step in upgrades and would cause problems when moving files between different versions - in what way is this viewed as a mistake? To be honest analysing how the code could be refactored to OO is not that hard, relative to actually changing the code base. I certainly don't find Perl a great language for OO and it's much worse when it comes to support for threads - we've really got to be clear where the benefit of such a refactor is. I changed much of the backend so a switch between RCS, RcsLite and I hoped in the future other backends was clean. There's a lot to be said for treating a topic as an Object, but it's also very powerful just treating it as a text sequence - OO is not the answer to every design problem. To make progress here we need a specific objective that requires refactoring e.g. allowing meta data to work in it's current form or a database, and we're really then in coding land. Whilst I'd be interested to see such a feature and don't have any need for it myself.

I'm not sure I'd want to implement renaming/moving along OO lines. However it would be nice to use exactly the same for find links to rename that is used when rendering text; I'm not too clear how this would be done though and I'm not sure that OO approaches have the best answers.

PPM is not a full answer to installs as it is not to my knowledge available to all the platforms that TWiki supports.

-- JohnTalintyre - 29 May 2003

Having written many applications in C, having been a kernel hacker on UNIX V6, V7, BSD2.7, 2,8, 4.0, 4.1 and 4.2 as well as using C for embedded applications and on other non-UNIX platforms and having my introduction to OO via Modula and SmallTalk, I find Perl a very nice OO langugae and find C++ a montrosity. YMMV. That's not what we're discussing, though.

The Meta dcode is OO and good OO at that.

The storage handfling is not OO. That it can switch from real RCS to the lite is doen entirely with that "eval" and not with an OO approach.

An OO approach would define a base set of methods then have an over-ride. Thus other storeage handlers could be pluged in without changing that code. This design assumption is the basis of my own OO redesign of TWiki.

  • A Topic Object thinks it knows its storage type.
  • If it doens't, it asks Web Object
  • If that doesn't it asks its Twiki Object

But the application programmer and end user doens't see that.

So in effect we have:

  • The topic is polymorphic - it knows its "type" and hence what it takes to render it.
  • The Topic object hides how metadata is implemented and stored
    • This means they can all be manipulated as one entity even if they are stored very differntly
    • It also means they can be manipulated seperately. Revising the attachements doens't mean revising the topic body; ditto revising the access control
      • Unless you didn't notice, doing a UNIX chmod() or chown() doesn't affect the contents of the file
      • The same applies for the corresponding functions in MS-Windows, VMS, MVS, VM-CMS ....
  • The topic may not have a store, it may be generated on the fly
    • RSS or WebServices are now much simpler.
  • A web may know how to store a number of topics
    • This makes the location and media of the individual web transparent to the application
    • The store doens't have to be a file system.
      • Actually parts of it can be and parts of it can be something else ...
    • Not all webs in a TWiki have to be stored the same
    • Making the storage or any part of it, a database, a FTP or any kind of addressable "feed" is now straight forward
      • You don't believe me about Pidgeons? Look here
    • A Web might be stored remotely
      • which instantly and simply solves problems to do with remote TWiki, mirroring etc.
    • A Web might be on a CD or DVD
    • In a corporate environment, some Webs might be shared between departments.
  • Store now looks like Storage::Topic Storage::Meta Storage::Preferences Storage::Access Storage::History
    • Actually its more like Storage::Meta::TopicInfo Storage::Meta::Parentage Storage::Meta::Forms Storage::Meta::Attachments
    • This makes mapping to CVS trivial, even for a revision history of changes to access control settings
      • ... or not. You can configure your Twiki to, for exampple, only track the history of changes to the topic and the attachements, if that's what your business needs are.
        • That's configuration not code change.
    • Did you notice the capability for multiple forms there?

A real and systemic Object Oriented approach makes all this very straight forward.

The pidgeons will not be part of the core distribution of the Object Oriented genration of TWiki.

  • Nor will the cats.

-- AntonAylward - 29 May 2003

You will find that the two revision control implementation are behind an OO interface, adding CVS would be relatively easy if all the methods could be implemented. However, I'm not sure if the replace revision capability exists in CVS. I don't understand the referance to eval above - the OO aspect I'm refering to is that the two mechanisms offer the same interface. I agree with much of what you say above, there has always such discussion about TWiki. One could start a Wiki from scratch that is fully OO, I'm sure there are already some out there, with TWiki I assume we are talking about refactoring that offers full backward compatibility. The existing OO work goes part way to that. To offer more storage mediums we need to extend the storage OO interface to also cover items like searching.

-- JohnTalintyre - 30 May 2003


That, and reviewing the design assumptions and architecture, is what I'm working on with my OO modeling.

The "eval" I'm talking about is in in =~twiki/lib/TWiki/Store.pm, the subroutine initialize (some would prefer it to be 'initialise') - see Fowler, where it says eval "use TWiki::Store::$$TWiki::storeTopicImpl;";

I don't call that OO. Its a hack. It's a very neat and creative hack and demonstrates ingenuity. What under there is OO code, but Store.pm itsef is not OO. That's not inheritance, that's not polymorphism.

To make storage OO, the Topic object and the web object need to "inherit" storage. If the topic is not exceptional - RSS for example - it will use its web's storage mechanism.

Whatever the revision control, the storage layer has to be polymorphic. We can't assume its a local file ssytem.

If we don't, if we make the storage layer polymorphic, we inzstantly have a free ride to

  • Distributed TWiki
  • Mirroring of individual webs
  • Webs that are not file systems but are actually
    • WebServices that hide the details
    • Remote FTP or HTTP services
    • Read only, e.g. CD-ROM
    • File systems that overlay a baseline CD-ROM with the changes to the baseline
  • Corporate webs that implement revision control - or not - of atachements, settings, access control and other metadata according to the business requirements

Currently, the storage is coded procedurally. What we want is a OO abstraction where $topic->doRender() sorts out the details according to the type of the topic.

  • The Topic Object is polymorphic, it might be RSS, WebObject, PDF, TWiki-Markup, Cunningham-markup, RTF, whatever
  • In each case the $topic->topicstore->readTopicBody() "does the right thing"
    • The right thing might actually be the equivlent of $topic->_web->readTopicBody($topic)
    • ... or perhaps the equivilent of $topic->_web->_web->web->twiki->readTopicBody($topic)
      • depending on how the Twiki and its heirarchy of webs have been configured
        • ... and how the Topic Object is created in context

Some of what we're arguing about here is static vs running code. Store.pm is a collection of stuff not a Storage Object that supplies a set of services and hides the details.

Making proper Storage Object mandates a Topic Object and a Web Object to communicate with it. That then mandates a Render Onbject and a Skin Object, and of course the TWiki Object to manage it all.

I've got a few hundred K of design notes on all this, as well as the UML models I'm working on. I'm sorry my UML tools can't generate Perl frown

-- AntonAylward - 30 May 2003

It would be useful if we could be a bit more specific and consice here.

  1. The eval is actually a Factory, it creates an object of type specified by the config file.
  2. There are a least two separate concepts here. Generallising how the data is physically stored and allowing for different types of topics (and items within them) that render in different ways. It is the first of these that has already had significant OO work done on it.

Assuming you agree with my statement above, then we're talking about refactoring TWiki not forking it. So, what are the first steps in this refactoring and how would they benefit users? It's easy to imagine a fully OO environment that is great, it's getting there that's tough.

-- JohnTalintyre - 30 May 2003

See ADatabaseCalledTWiki for an example of what can be done if you put just a teeny, gossamer thin layer over the store implementation.

-- CrawfordCurrie - 25 Jun 2003

Edit | Attach | Watch | Print version | History: r13 < r12 < r11 < r10 < r9 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r13 - 2008-09-17 - TWikiJanitor
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2017 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.