Logically Nested Webs

Introduction

Over time there has been a lot of discussion about HierarchicallyNestedTwikiWebs. One aspect that has sometimes been touched upon is the difference between a physical and logical hierarchy.

A summary of the various options is presented below, for a full discussion, scroll down.

1. Each topic in a web can be a subweb with a logical hierarchy:
   • Storage Options:
     • The subweb pages are stored in the same directory as the parent web pages.
     • The subweb pages are stored in a separate top level web.
   • Logical organisation options:
     • The children of a topic are defined in the topic, and the children form a logical subweb. The relation of parent to child is determined by the parent to child relationship.
     • The parent of a topic is defined in the topic, and the children form a logical subweb. (TWiki currently supports this model directly) The relation of parent to child is determined by the child to parent relationship.
2. Each topic in a web can be a subweb with a physical hierarchy. The subweb pages are stored in a subdirectory of the parent web pages.
   • MegaTWiki implements this (?) for an old release of TWiki
   • MultiLevelWikiWebs also provides an implementation, for a very old release of TWiki
3. Allow webs to define subwebs - either in WebPreferences or in TWikiPreferences (or similar). This would merge subweb's namespaces with the parent. Conflicts of names would need to be dealt with. This would be directly akin to:
   • use Web qw(all); or in perl terms
   • from Web import * in python terms
   • open Web in SML terms
4. Simply look at TWiki as effectively currently comprising a relational store, where:
   • The tuples structure and content are defined by random users (and hence non-normalised almost by definition).
   • TWiki webs are sets of related topics, in the same way a relation is a collection of related tuples
   • Subwebs are then just projects, selects, unions and intersections of relations. In practical terms this means merging subwebs/namespaces, filtering out topics based on TWiki form category values, or slicing out named sections of text.

Physical Hierarchy

This is the simplest in many respects to understand - since it matches the way people deal with files and directories. In this scenario a web is treated like a directory and can contain other webs - just like directories can contain other directories.

Logical Hierarchy

Logical Hierarchies relate one set of topics in one web with another set of topics in either the same web or a different web. If the topics are in a different web, topics can be named the same since they form a different namespace .

Thus you can either have a logical hierarchy where names...

• Can be repeated - due to the topics actually being stored in different webs. (Logical hierarchy of webs - Virtually Nested Webs?)
• Cannot be repeated - due to the topics actually being stored in the same web. (Logical hierarchy of topics in a web)

The intersection of these two ideas is a logical heirarchy of webs and topics. These will have name clashes occasionally that need resolving.

Logical hierarchies are thus all about namespaces - how groups of named pieces of information relate to each other. You can choose to :

• merge namespaces (FindElsewherePlugin, TopicNotFoundInThisWeb)
• Reference other namespaces explicitly based on implementation/physical aspects. In concrete terms this means using the notation Otherweb.SomeTopic where Otherweb actually exists.

On a different note however, there's no reason why a user should even know if they are using a Wiki with logically nested names or physically nested ones. Some of the discussion assumes two hierarchies:

• One flat that matches the file system
• One nested that doesn't

There are many variations on this that can be done. It's important to allow name duplication from the users point of view, even though it's not done that way physically.

How to refactor this suitable for this section? Whilst an opinion it strikes me worth being here? In principle I don't care how it is mapped to an underlying store as long as the users cross-web references look sensible and we can have conceptually HeirarchicallyNestedTWikiWebs.

URLs Following Logical Hierarchy

This is things like MegaTWiki, MultiLevelWikiWebs

URLs Not Following Logical Hierarchy

In this siuation, all URLs are still of the form Otherweb.OtherTopic Advantages:

• Simple cross referencing other webs - format is never more complicated than at present regardless of the hierarchy.
• Allows for hierarchies without increaing storage complexity - keeps backend things simple. (KISS)

URLs able to follow Logical Hierarchy

Existing Hierarchical Implementations

• MegaTWiki implements a physical hierarchy
• MultiLevelWikiWebs also contains another implementation of a physical hierarchy

• KoalaSkin implements a logical hierarchy where changing the logical heirarchy does not change URLs.

  On the uniqueness of names, my solution was to allow underscores in web names to allow having more descriptive names, and have a way to show abbrevs for the names in the navigation links. Users must refer to the true name when referncing topics across webs, but this solution seems "good enough". ColasNahaboo

Logical vs Physical - Pros & Cons

Logical hierarchy

• Pros
  • Very simple to implement in current TWiki e.g. see TigerSkin
  • You can have multiple hierarchies for different purposes
    • different ways of limiting searches
    • different ways of organising security
      • Is this a benefit ? Depending on a hierarchy to organise security and having the same web in more than one web is a surefire way of leaving holes in your security
      • This isn't necessarily the case - under Unix this exact case exists.
      • The issue does need thinking about and documenting
  • You can change the hierarchy without moving the content
    • why is it so bad to move the content?
    • ...

• • Shoud work well if content is moved to an RDBMS
    • why would this not work with physically nested webs?
    • RDBMs use relations to store content - this was invented after the network model which followed the hierarchical model. Essentially RDBMs use logical linkages between collections of data in namespaces. The namespaces (tables) are therefore stored physically independent (often all in a single directory) but implement many views - hierarchical or otherwise. Storing a physically nested web in an RDBMs would require a logical mapping of the hierarchy - and at that point you're implementing logical hierarchies. If you start with a logical hierarchy you have a closer direct mapping of data to storage.
    • If you use a database to store topics, you would almost certainly not use a physical hierarchy, given that RDMSes don't support this directly.
  • You can also add topics (pages) as terminal nodes. These are things like WebChanges and WebSearch - TigerSkin works this way. I don't understand, please explain
  • Hierarchy can span separate Wikis, they don't even have to be TWiki installation. Interwiki could potentially be another hierarchy, or just a part of the hierarchy. - you could do this with physically nested webs

• Cons
  • Web names have to be globally unique. Although note these could become keys with the user using names attached to the logical hierarchy

Physical Hierarchy

• Pros
  • Provided free by the operating system (note implemented in MegaTWiki)
  • Searching a web and sub-webs comes pretty much for free
• Cons
  • physically nested webs raise the complexity considerably / With logical webs we do not have that problem - from a users point of view? See my example above where a logically nested web is more complex

Use Cases & Requirements

Random Requirements

• logical is the way to go, as the hierarchy has a tendency to change - we need good tools to allow this
• moving webs in the hierarchy thus do not break URLs - we need good tools to allow this too

Usecase: Organisation Hierarchy

Usecase: Logical Webs with Overlapping Page Names

Re (from above):

• Cons
  • Web names have to be globally unique. Although note these could become keys with the user using names attached to the logical hierarchy

If not, can you give me an example of how to name the following two pages in a logical hierarchy:

• IfSyntax for C++
• IfSyntax for Pascal

In a physical hierarchy I'd name them Cpp.IfSyntax and Pascal.IfSyntax, but within their own webs (or elsewhere, depending on how I set up the link), they'd appear as just IfSyntax.

I probably am missing something, but I assume that in the logical hierarchy I'd have to include a prefix, suffix, or additional key word in the page name, like CppIfSyntax and PascalIfSyntax?

Just to clarify, I wasn't specifically thinking of CppIfSyntax and PascalIfSyntax as include sections, but instead as (perhaps very small) individual pages. (But, I have no objection to named include sections, or the concept of fine grained addressibility (or whatever) that's been discussed on a few pages recently.)

-- RandyKramer - 07 Jun 2003

Usecase: Complex Hierarchy, Flat Storage Space

This usecase puts forward a use case for a hierarchy that users might want, and a storage hierarchy that a sys-admin might prefer this system to have. The physical layout represents the sysadmin's preferance, the logical represents the user's preferance.

.
|-- Config
|   | OnePlugin.txt
|   | TwoPlugin.txt
|   | ThreePlugin.txt
|   | SitePreferences.txt
|   ` WebPreferences.txt
|-- LocalConfig
|   | OnePlugin.txt
|   | TwoPlugin.txt
|   | ThreePlugin.txt
|   | SitePreferences.txt
|   ` WebPreferences.txt
|-- CPP
|   | WebPreferences.txt
|   | IfThenElse.txt
|   ` WhileDo.txt
|-- Pascal
|   | IfThenElse.txt
|   | RepeatUntil.txt
|   ` WebPreferences.txt
|-- Prolog
|   |-- DataUnification.txt
|   `-- WebPreferences.txt
|-- Languages
|   |-- Concepts.txt
|   |-- Iteration.txt
|   |-- PatternMatching.txt
|   |-- CPP.txt
|   |-- Pascal.txt
|   |-- Prolog.txt
|   `  WebPreferences.txt
`-- Programming
    | LanguagesAvailable.txt
    | Programming.Examples.txt
    | IntroductionToProgramming.txt
    ` WebPreferences.txt

.
|-- Plugins
|   |--Config
|   |    `---Global
|   `--Config
|         `--Local
+-- Config
|   |--Local/
|   |    `--Plugins
|   `--Global/
|         `--Plugins
`-- Programming
      `-- Languages/
          |-- Concepts
          |   |-- Choice
          |   |   `-- Examples
          |   |       |-- Pascal.IfThenElse
          |   |       `-- CPP.IfThenElse
          |   |-- PatternMatching
          |   |   `-- Examples
          |   |       `-- DataUnification
          |   |-- Iteration
          |   |   `-- Examples
          |   |       |-- Pascal.RepeatUntil
          |   |       `-- CPP.WhileDo
          |   `-- Examples
          |       |-- Choice
          |       |   |-- Pascal.IfThenElse
          |       |   `-- CPP.IfThenElse
          |       |-- PatternMatching
          |       |   `-- DataUnification
          |       `-- Iteration
          |           |-- Pascal.RepeatUntil
          |           `-- CPP.WhileDo
          |-- Pascal
          |   |-- IfThenElse
          |   `-- RepeatUntil
          |-- CPP
          |   |-- IfThenElse
          |   `-- WhileDo
          `-- Prolog
              `-- DataUnification

Access to these topics/webs could be via the following URLs

%SCRIPTURL%/view/Plugins/Config/Global/
%SCRIPTURL%/view/Plugins/Config/Local/
%SCRIPTURL%/view/Config/Local/Plugins/
%SCRIPTURL%/view/Config/Global/Plugins/

%SCRIPTURL%/view/Programming/Languages/Concepts/Choice/Examples/Pascal.IfThenElse
%SCRIPTURL%/view/Programming/Languages/Concepts/Choice/Examples/CPP.IfThenElse
%SCRIPTURL%/view/Programming/Languages/Concepts/PatternMatching/Examples/DataUnification
%SCRIPTURL%/view/Programming/Languages/Concepts/Iteration/Examples/Pascal.RepeatUntil
%SCRIPTURL%/view/Programming/Languages/Concepts/Choice/Examples/CPP.WhileDo

%SCRIPTURL%/view/Programning/Languages/Pascal/IfThenElse
%SCRIPTURL%/view/Programning/Languages/Pascal/RepeatUntil
%SCRIPTURL%/view/Programning/Languages/CPP/IfThenElse
%SCRIPTURL%/view/Programning/Languages/CPP/WhileDo

Usecase: Multiple Teams Working on Overlapping Projects

Example: At the university I am studying at we have multiple teams all working on projects with exactly the same names. They are working on the same project in their own teams. As far as they are concerned, there is only one project with any given name. As far as the administrator is concerned, the projects are sub-projects of each team.

If then we have 16 teams (Ft04t2a1..Ft04t2a16) each with four projects (BusinessStrategy, ManagingProcesses, Economics and FinancialManagement), the teams want webs that reflect their work.

They want:

   *Ft04t2a1.BusinessStrategy*.WebHome etc
   *Ft04t2a2.BusinessStrategy*.WebHome etc
   *Ft04t2a3.BusinessStrategy*.WebHome etc

   *Ft04t2a1.ManagingProcesses*.WebHome etc
   *Ft04t2a2.ManagingProcesses*.WebHome etc
   *Ft04t2a3.ManagingProcesses*.WebHome etc

With the current system of webs this would have to be:

   *Ft04t2a1BusinessStrategy*.WebHome etc
   *Ft04t2a2BusinessStrategy*.WebHome etc
   *Ft04t2a3BusinessStrategy*.WebHome etc
   *Ft04t2a1ManagingProcesses*.WebHome etc
   *Ft04t2a2ManagingProcesses*.WebHome etc
   *Ft04t2a3ManagingProcesses*.WebHome etc

Such that a reference from a group own project's BusinesStrategy notes to their own ManagingProcesses notes include the prefixes (giving references such as Ft04t2a3ManagingProcesses.WebHome). This seems convoluted, unfriendly and just plain daft.

Paraphrasing Main.MartinCleaver from here :
I can see how to do this with a physical hierarchy using MegaTWiki. I can't see how to do this using a logical hierarchy

Implementation Possibilities

Store Parent Webs in Web Preferences Or Central Sitemap

-- Main.PeterThoeny

Yup, that's what I did with KoalaSkin, but the other way around: the logical hierarchy is computed from a central site map (a bullet list in a Topic), as I found that for a big number of webs, having all customisations in one place is really practical. For instance, to debate webs reorganisation with team leaders who were not twiki-savyy I could just mail them the topic text and have them edit it to show me their proposed reorg.

See: http://koala.ilog.fr/wiki/bin/view/Main/WebList from this topic is computed the sitemap itself: http://koala.ilog.fr/wiki/bin/oops/Main?template=sitemap and the navigation bar at the top of the skin.

I also provide a "search in group" facility by passing the (logical) list of webs in the current group to the serach routine. The selct menu on search defaults to search the group for webs having sub-groups, and to search the web only for web "leaves".

-- Main.ColasNahaboo

Consolidate Support from TigerSkin and KoalaSkin to achieve this

There is already some support for logical nesting in the TigerSkin and KoalaSkin. These don't require TWiki to alter. However, [To make this cleaner] the first step is a data structure in TWiki that can be used by multiple skins - this is also mentioned in ConsolidateFunctionalityFromSkins

Need for a more Powerful Mechanism However

We can also think about a more powerful mechanism that would allow duplicate Web names and hide the current 1 tier structure from users. Such a proposal should I feel have no impact on sites not using it - also for this to make it into the core we would want to keep it reasonably simple.

TWiki as a Non-Normalised Relational Store

This is being refactored out into a separate page which is why all the text has disappeared, older discourse here and here

The key points of this proposal:

• TWiki effectively currently comprises a relational store, where
  • The tuples structure and content are defined by random users (and hence non-normalised almost by definition).
  • TWiki webs are sets of related topics, in the same way a relation is a collection of related tuples
  • Subwebs are then just projects, selects, unions and intersections of relations. In practical terms this means merging subwebs/namespaces, filtering out topics based on TWiki form category values, or slicing out named sections of text.

• Allow webs to define subwebs - either in WebPreferances or in TWikiPreferences (or similar). This would merge subweb's namespaces with the parent. Conflicts of names would need to be dealt with. This would be directly akin to:
  • use Web qw(all); or in perl terms
  • from Web import * in python terms
  • open Web in SML terms

• This is an overlay that recognises on the existing system's inherent structure rather than imposing a new structure - can potentially be implemented as a plugin - hence low impact for users.

Putting this a different way:

1. Slices across a web's named sections is a way of getting at a subset of data.
2. Using categories logically denotes subsets of data.
3. Directories for physical divisions of data, which are subsets of the whole.
4. Hierarchies can be viewed as relationships between sets of information - most often a subsetting arrangement. I'm deliberately ignoring unions here. (Though in TWikiStructure when I reach that point I might lay out these ideas more formally. I intend to put out some code to play with this at some point)

Topics as tuples

Topics currently form a tuple of data (parts of these attributes overlap)

• • (topicname, section1,section2,section3,section4,...,text,metadatavar1,metadatavar2,metadatavar3...)

In database terms these are relations, certainly not normalised, but certainly relations. An example relation could be:

• Pascal(TOPICNAME, Summary, Introduction, Syntax, Example, Discussion, RawText, StatementType, PageType, Keywords)
• Statement type { Choice | Iteration ... }
• Page type { Definition | Discussion | Example ... }

An example tuple would be:

• IfThenElse, text of Summary , text of Introduction, text of Syntax, text of Example, text of Discussion, text of RawText, "Choice", "Definition"

Webs as Relations

If we think of a TWiki web as a relation, and topics as tuples, then it's logical to start thinking about what that gives us - we can start using relational algebra to help understand our data better, and come up with a better interface. (NB, I'm not speaking implementation, just analysis!) In this case, subwebs simply become subsets of the data stored. In relational terms they are simply the results of select/project/joins on the data. In this context, what would this mean ?

• select (not sql select) data - this means essentially filtering rows. In Twiki terms, this is eliminating topics based on either text, metadata, or topics names. In the above sections I am suggesting that selection of topics is based on categories.
  • The results of a relational algebra select is a relation, which in this case is a subweb. Therefore categories essentially determine potential subwebs. Physically separate webs are therefore also "just" an optimisation.

• project data (as in the verb, not noun). In relational terms this essentially means choosing attributes, or in table terms, columns. In TWiki terms the attributes are either slices through the topic (sections) or metadata. As a result the data resulting would again be a subset of the original data - and hence logically a subweb of the original data.

Worked Example

Taking a more complex example, what would this mean: * Programming/Languages/Concepts/Choice/Examples/Pascal.IfThenElse

Again, we know of the existance of the following physical webs:

• Programming
• Languages
• Pascal

Therefore one of these is going to be used as the Relation to search for the topic. As before we can simply pick the Pascal web since it is a child of Languages, which is a child of Programming and therefore the URL mentioning all 3 is delving into a subset of Programming .

In this scenario, what do the other 3 words in the path do?

• Concepts
• Choice
• Examples

These form subsetting - either slices across topics (which is the relational project operation in this case), or subsetting of topics based on category (which is the relational select operation).

If we choose the semantics that we prefer non-empty sets to empty sets (we want to find pages), and non-empty topic results to empty topic results we can choose this:

1. Examples isn't a category value in the Pascal web, so we treat it as a slice - we want named sections called Examples.
2. Concepts isn't a category value in the Pascal web either - we treat that as a slice. (Again we want named sections call Concepts
3. Choice is a category value in the Pascal web so we treat that as a topic filter, a relational select - we only match pages with that value.

Since 3 denotes an operation to choose subsets of data, the topics we wish to look at in this set are reduced from this:

|-- Pascal
|   | IfThenElse.txt
|   | RepeatUntil.txt
|   ` WebPreferences.txt

To this:

|-- Pascal
|   | IfThenElse.txt

This matches our topic IfThenElse.txt, so we then choose to include the named sections in this topic of Examples AND Concepts as noted above. (Not the intersection - named sections cannot at present overlap)

Requirements for implementation

• Hooks into reading a file from disk (given we're putting extra terms in the file name for filtering)
• Child webs defined in WebPreferences

Otherwise, TWiki stays as is.

As noted above, a full proposal along these lines will be factored out into a separate page shortly.

Configure Everything Using an On-Server Config File

I found this comment too wooley and often off topic to refactor in better - much of the content is best put in a separate topic IMO or refactored by the original author with a more concise summary left here. (Sub headings are not the original authors, and probably need changing) I would only have left in these two sections which struck me as too large a cut. Hence the lack of change

John has it right!

]]

The approach I'm currently working on is to seperate the physical representation from the logical.

Treat a web as only a physical container for topics
If we accept that a web is only a container for topics, then things become a lot easier to deal with. It doens't matter if the web is a directory, a database or remote FTP site. What matters is how we map the logical view - what is in the topic text as the logical path name - to the physical.

In the past some version of UNIX had a /proc filesystem type thing
Some experimental versions of UNIX had things like the /proc file ssytem under Linux. The "directory" was actually a handler in much the same way that Apache and TWiki handles the components of a URL. The URL may look like a path, but the "Web/Topic" are actually parameters to the handler that is "cgi-bin/view".

Two Possibilities using a config
So we have two possibilities that can be handled with a config file.

I'm playing with YAML these days to great effect. First, the logical web is "anynumberofsegments". This is mapped to the physical directory. Many ways of implementing that. I'm playing with YAML these days to great effect. If the logical follows the directory tree, that's fine, but it doens't have to. Here's some pseudo-YAML to illustrate:

   webs:
     main:
       label:    "Sys"
       location: /var/www/twiki/data/main/
     twiki:
       label:    "Config"
       location: /var/www/twiki/data/twiki/
    projects:
       label:    "Projects"
       location: /var/www/twiki/data/projects/
       ProjectOne:
          label:    "Database"
          location: /home/data/projects/proj1/
       ProjectTwo:
          label:    "Webserver"
          location: /var2/data/projects/proj2/

(In reality this would be a perl data structure, I've just written it that way for intelligibility.)

silly argument about re-naming webs.
You'll also note of course I've soaked up the silly argument about re-naming webs. Updates from new releases still go into the directory. (Of course the embedded references will still have to be deal with.)

You might gather from this and from my other postings that I'm a very table-drive sort of designer
You might gather from this and from my other postings that I'm a very table-drive sort of designer, putting prefs into hash DB's and so on. Since tables can equally well be in a database, MySQL, SQLLite (or if the fancy takes you DB2 for Linux), we're already half way there.

I did mention that the webs don't have to be in one place, didn't I

I did mention that the webs don't have to be in one place, didn't I. Well if you think in terms of URI and are a little loose on the syntax, we can have:

    projects:
       label:    "Projects"
       location: file:/var/www/twiki/data/projects/
       ProjectOne:
          label:    "Database"
          location: ftp://projectserver.corporate.tld/home/data/projects/proj1/
       ProjectTwo:
          label:    "Webserver"
          location: ldap://ldap.corp.tld/dc=projects,dc=proj2/

All just a matter of drivers. Of course CPAN:URI can take care of much of this.

Refactorer's comment - whilst this comment goes on about there being two cases, I don't see two cases presented in the original text. If you can't see it here, that's why.

Similar Ideas in Other Environments

Usage Styles

Opinions on Logical webs

If these appear out of context, please edit context back in. I've placed links to the original content using the date of opinion.

Multiple Webs is a feature much loved in TWiki, but it can lead to small Webs that don't have critical mass. Wikis are supposed to work with rich cross referencing, offering hierarchical orginisation might just make things worse. However, on balance I think Web nesting would greatly benefit the appeal of TWiki. I think logical would be better than physical - would do other people think?

-- JohnTalintyre - 06 Jun 2003

I agree with you, logical is the way to go, as the hierarchy has a tendency to change, moving webs in the hierarchy thus do not break URLs. I used Logical hierarchy for the KoalaSkin.

-- ColasNahaboo - 06 Jun 2003

Logical nested webs are a very good idea! I never liked physical nested webs because it raises the complexity considerably (not KISS) while not bringing much gain. As John points out, it complicates cross-referencing. With logical webs we do not have that problem, it is still Otherweb.OtherTopic, regardless of the hierarchy.

-- PeterThoeny - 07 Jun 2003

Anyway, that aside. I agree wholeheartedly that an administrator be very careful about how deeply nested the structure becomes but I disagree that we should not be able to make such deep structures if that reflects reality.
... Caveat: I'm definitely not happy with 2 levels, and I can't see why the design of TWiki should impose a limit at all, but I would be more happy with 4 levels than the one we have now. Remember CPM?...

-- MartinCleaver - 07 Jun 2003

Anton's suggestions look interesting for the long run (Will they make code by DakarRelease?).

-- MartinCleaver - 08 Jun 2003

Contributors -- PeterThoeny - 07, Jun 2003
-- JohnTalintyre - 06, 07, 08 Jun 2003
-- ColasNahaboo - 06, 07 Jun 2003
-- AntonAylward - 08, 22 Jun 2003
-- RandyKramer - 07, 19, 22 Jun 2003
-- TWikiGuest - 09, 20, 21, 22 Jun 2003 -- MartinCleaver - 07, 08 Jun 2003

=========================================================

Older Discussion

I've been refactoring this for several hours now and haven't had a chance to factor this into the above. The above also needs summarising better. But since it's late, I'll "release edit lock" on this topic now... Hopefully it's clearer...

TopicChildren is another way of looking at Nested Webs

Going through older topics I came across the idea of AreTopicsWebs, which is related to TopicChildren. The former discusses Topics as webs in the context of attachments, the latter can be viewed as "attachments" to a topic. (Why must an attachment be a file?) This allows nesting a Twiki web or collection of toics (same difference?) essentially inside a topic.

-- TWikiGuest - 21 Jun 2003

Aging and Archiving and Categorizing

TopicChildren leads to another way of utilizing "subwebs".

If a parent web becomes "overloaded" - the definition of that being whatever the business rules are - tools like the already exisiting ones to rename or move a topic can be used to automate a harvesting process and move topics to the newly created child.

The important thing to note is that links are preserved.

The business criteria determine the what and the when.

For example, here in Codev there are topics that are very clearly:

• Documentation of what is or has been implemented
• Discussion of things that led to that, which are of historic interest only
• Bugs that were fixed
• Ideas that were proposed and discarded
• Issues that relate purely to old releases or old code

So we can draw a disctinction betwee what is current and what is no longer pertinent.

My point here isn't that they should be "refactored on mass" to an archival "sub-web", there is no "management policy" for that here. Rather, the TopicChildren approach offers the opportunity of having and using such a policy.

The automated tool I described might create a portal topic as part of the harvesting process.

An idea for such a tool:
1. Select by any or all of
   • Topic Classification
   • Date of last change
   • Keywords
2. Display list with check-boxes
3. Create Portal topic
4. Move the selected topics

-- AntonAylward - 22 Jun 2003

Yes, such a tool would be useful to me. The Wikilearn web is overloaded, and I am using kludges such as prefixes in page names (Cpp..., Nedit..., ) to work around the problem.

-- RandyKramer - 22 Jun 2003

I'm in the process of refactoring this page but Anton's suggestion this would be possible to happen dynamically, and exist automatically upon creation of a category if we treat webs as relations. This will be clearer after the refactor. (I hope) Unlike Anton's other point however, the parent web could still appear to contain all the topics, as well as those partitioned off, except with an alternate naming technique. (Physical separation would be considered an automated optimisation)

-- TWikiGuest - 22 Jun 2003

Thanks, Michael, for your refactoring efforts!! It's looking quite good (i.e., some of it is sinking in), but I will have to read it more than once

I was going to make a few observations here, but I'm not sure I'll remember them all.

• There is a concern in some of the above discussions about keeping the URL the same even if a page is moved. I want just the opposite -- I want the URL to change if the page is moved. Well, maybe not, but I do want this:
  • I want to be able to enter a URL into the browser's address bar from memory. Going back to my earlier example, I want to be able to type in (or revise a bookmark, or revise an address currently in the address bar), I want to be able to type in something like http://twiki.org/....../Cpp/IfSyntax, if I want to see the IfSyntax for C++. If I then want to see the IfSyntax for Pascal, I want to change only the "Cpp" to "Pascal". _Maybe this particular example is unlikely (switching between viewing the syntax for two different languages), but it is the kind of thing I want to do. (I would not object if there were more than one URL to get to the same page, as long as one of them was of the "human understandable and modifiable form" of this example.)
  • I want to be able to limit searches to a specific [web | namespace] — if I know I'm working in C++, I want to be able to (easily) limit a search to just the Cpp namespace. (Of course, there are other times when I don't want to limit the search to one namespace — see next item.)
  • In a multilevel hierarchy, I can imagine wanting to limit the search to a specific "branch". (For example, if the web hierarchy includes Basket_weaving (trying to think of another "major category" ) and Programming with subwebs Cpp and Pascal, there are times when I will want to limit my search to Cpp, other times to Pascal, and other times to Programming including Cpp and Pascal
  • (will come back when I remember some of the others).

Having brought up that example, maybe we should list ways people navigate a TWiki?

• When I'm on a TWiki page and see a link for another page that I think will help me, I'll click on that link.
• Whether I'm on a TWiki page or not, if I want a TWiki page and don't see a link, I'll try typing the address into the address bar (or, really, call up a page like WebHome or WebSearch from a bookmark, then modify the URL to what I think I remember is the web and pagename.
• If I don't have any better clue, I'll call up search and start searching. I'll then choose to search in Google or one of the TWiki searches depending on my (imperfect) knowledge of what each is good for (the following is only a first attempt, I'll try to refine it as I think about it (and perform actual searches)):
  • If I can remember the web name I'll probably choose one of the TWiki searches
  • <Hmm, I have to think more about what makes me choose between Google and a TWiki search — even though (at least one) TWiki search now does implicit "and" instead of phrase searching, Google presents results so much faster that I'm developing a bias toward Google. But, Google doesn't do "stemming" (can't search for a part of a word), so if I'm unsure of the form of the word I might be searching for (hierachical, hierarchically) I'll probably tend toward using a TWiki search.>

Maybe my view of things is biased because I view TWiki as more of a knowledge base than anything else? But, that knowledge base could include knowledge of, for example, projects, and like programming languages, there are times when I'd want to search a specific project for some information, other times when I'd want to search all projects, or all projects in a certain category.

So (talking to myself) are all my desires for hierarchical organization of TWiki driven by ease of search as a criteria?

I know I've diverged from the main topic of this page, but I think this is a related topic that is relevant to the subject. If someone wants to propose moving this kind of stuff to a different page, linked to this page, suggest a page name.

Finally, there's an idea percolating in my head (that I'm sure has implicitly, if not explicitly, been suggested by others, on this page and or other pages):

I can imagine a TWiki organized as physically nested webs (based on the underlying file structure), with an overlaying relational data based "thingie".

The actual data and pages would be stored as today, as .txt and .txt,v files in webs that are (usually) directories in the underlying file structure of the OS running the TWiki. The relational data base would not store content, but I guess the right word would be metacontent, including addresses of the real pages.

I can see two "reasons" for this:

• the relational data base layer could support any kind of addressing anybody wanted, while the physical layer could still be addressed with the current TWiki URLs (and, I would expect to have the ability to move pages at either the physical level (to a different web/directory, while keeping the logical address (maintained in the RDB) the same), or to a different logical location, while keeping the physical address the same. (Or, of course, move it both physically and logically.)
• access via the RDB layer could presumably support faster searches and the like

(There's a specific reason for suggesting the following, which I can't recall at the moment. Oh, OK, now I sort of remember, and I may be mixing up a few things. But, I want a Google-like indexed search of TWiki (that does stemming), and, if we want things like both a Parent -> Child relation and the Child -> Parent relation, we can "store" just the smaller relation (Parent -> Child) in each topic, and derive the Child -> Parent relation from that and store it in the RDB (for, among other things, fast access). (The Parent -> Child relation would also be stored in the RDB, but the canonical source would be the .txt file "metadata", and there would be a background task that occasionally checked that the Parent -> Child relation in the RDB was in sync with the (canonical) one in the .txt file — read on.)

And, thinking about some of the concerns expressed elsewhere (IIRC), I would make the physical files (.txt, .txt,v) be the authoritative (canonical) source for all data in the relational data base, and include some background tasks that updated the relational data base from the physical files on at least two occasions (actually, I mean, put them on a schedule to be updated on two occasions — a save should not wait for the relational database to be updated, and, as an aside, each TWiki page should contain a visible field to indicate when the RDB last updated it's information about this page):

• when a page is changed
• periodically (the rdb would periodically check each physical page and confirm and update any metadata just because the physical page is the canonical source)

Is this overkill? (Probably) Is it the Wiki way? (Don't think so, but I will say that I came to TWiki looking for a replacement (in Linux) for askSam (available only in Windows) which is a free-format "database" with networking capabilities. Did it have the URL style addressing I'm so eager to maintain on TWiki? (I'll have to go back and look, but I suspect not quite the same. (I did maintain different databases, by project, and could search across databases using external tools like ZyIndex, so there are some similarities.))

Anyway, I'm not sure where I'm going with this, so I think I'll just sign off for now.

-- RandyKramer - 23 Jun 2003

See Also:

Other Refs: HierarchicalNavigation, MultiLevelWikiWebs, NavigationByTopicContext ( "The idea is to have a logical hierarchy of topics"), DataStorageForMultiLevelWikiWebs ("The basic concept I'm chasing at the moment is: "A Web is a Topic with Childen. TWiki is a Tree of Topics or Tree full of Webs.") TopicFrames covers ground similar to the ideas laid out for slice&dice/project&select - dynamic sub-webbing. IncludeWeb is a topic that covers the idea of including a separate physical web into the current web - logical subsetting again.

-- Anonymous