This is moving towards Twiki being a transparent filesystem interface.

Along these lines, there will probably arise the need for several different ways of forming names.

(BTW, the prospect of fancier names should not delay the incorporation of MegaTWiki into the core code.)

• Relative pathnames
  • in the current web - done
  • in subwebs of the current web
  • in parents of the current web (../ParentTopic)
  • in siblings of the current web (../Lib/SiblingTopic)

• Absolute pathnames

• "Paths" of directories/webs/pathnames for unresolved names
  • e.g. User.AndyGlew -> /Admin.User.AndyGlew
  • paths that search .;..;../..;../../.. etc. up to the root of the wiki, if a topic page is not found locally. (Years ago I modified the Bourne shell to do this for UNIX commands. Mixed. Main problem, security, just like having . in the path.)

I'm not sure if the following are exactly related, but anyway

• symlinks - might just expose the base OS functionality, or might be implemented within TWiki (which may give some convenient semantics)
  • symlinking web to web
  • symlinking topic to a web in another web
• directory continuation
  • a la David Korn - one web, which is the logical concatenation of several other webs. This would have to be implemented in TWiki, since few UNIX systems support it.

-- AndyGlew - 15 Apr 2003

I strongly support ColasNahaboos argument about reorgs and renames in RenameMainWebToHome today. I repeat one of my pet peeves Never break existing URLs! Really.

• Coding organisations into names is a recipe for constant renames.
• The file system analogy is flawed IMHO. The way how you need to organise your folders is an artifact of your navigational tools.
  • Use a DB, then you don't need to organise records into hierarchies.
  • Use a wiki, then you don't need to create zillions of micro compartents alias folders.
• Actually, deep hierarchies hurt a (t)wiki. Linking, searching, administration, almost everything gets harder. My recommendation:
  • use as little folders as you can
  • use parents / breadcrumbs / KoalaSkin to (re-)organise your hierarchies without breaking a single URL.

I agree about not breaking URLs, but think it needs a more fundamental approach. See KeepingReferenceNamesForever.

It gets ridiculous, eventually: if you rename a function from Foo to Bar, are you still going to keep the wikiname FunctionFoo, and not create FunctionBar? Well, no, you will create FunctionBar, and perhaps create an entry for FunctionFoo that says "renamed on date DDD to FunctionBar".

And if function Foo is recreated, perhaps with a different behaviour... then FunctionFoo gets reused, perhaps with a historical note that it used to do something else.

This is part of XP, eXtreme Programming: never be afraid to name things what they should be named. XP is a philosophy strongly associated with wiki.

But this is a red herring. What does placing things in a filesystem hierarchy have to do with preserving names forever? The very same issue applies to the filesystem. You are just afraid because, in your experience, filesystems get reorganized more.

Now, I confess an ulterior motive. Want Wiki in Filesystem for Literate Programming

Apart from that rant, let me also say: my wiki server is already being used by 4 groups and projects, of differing jurisdictions and security consciousness. It will be used in more. As I explained in UsingUnixGroupsForTWikiSecurity, I base my security system on UNIX groups. I use the group stick bit on directories to get correct group ownership. I basically just plain cannot do that if the groups share the same web, since a web is a single UNIX directory. So the question for me, using the best security arrangement I have been able to come up with and have seen recommended on TWiki, is not whether I put the different projects and groups into the same web, but whether I use lots of top level webs, or can arrange the webs in a nested hierarchical fashion.

Names collide. Multiple namespaces are needed. Namespaces nest.

You statement "Use a DB, then you don't need to organise records into hierarchies." is inconsistent with my experience. I have extensive database experience. I have used Zwiki, the wiki based on Zope, which is on a database. And, y'know, Zope/Zwiki do provide what is essentially a nested, hierarchical, view. I have nested one Zwiki within another, and so on.

What the database did do, however, was provide a system so that links within the database were always maintained consistently (more so than TWiki does). And the appropriate breadcrumbs were left behind for links from outside to inside.

(By the way, the reason I decided to go with TWiki rather than Zwiki was that TWiki stores its files as ordinary UNIX files, so that ordinary UNIX tools like grep, CVS, etc., can be used.)

Your statement "deep hierarchies hurt a (t)wiki. Linking, searching, administration, almost everything gets harder." is exactly what I used to hear from OS vendors (like Microsoft, Digital RSX) trying to excuse their lousy filesystems before hierarchical filesystems took over. IT's a tool problem. The tools get fixed.

Yes, database techniques will revolutionize filesystems. But hierarchy is one of the most powerful tools we have, and even in databases is a useful view.

Enough ranting. As is plain, I consider your objection to hierachy short-sighted.

But, you make two valid points above:

• It is highly desirable to have a system that preserves link names, URLs, etc.
• Tools to manipulate a nested hierarchy are helpful
  • Don't delay implementing a nested hierarchy until they are available, but expect the need.

Preserving lnk names is an orthogonal issue. even present non-hierarchical twiki breaks link names, when topics are moved. (At least, if you do a checkout of an old version of the RCS database, they are broken because the RCS file was moved.) The right thing is to figure out how to solve that problem properly.

By the way, PeterKlausner, you have pulled this topic off-topic. It has become an exchange of rants about HierarchicallyNestedTwikiWebs, not a discussion of naming syntaxes for such a web. I think we need top pull the rants into a topic of their own.

-- AndyGlew - 16 Apr 2003

Sorry, I didn't want to rant or sound impolite. I'll never, ever again use that yellow marker
Still, I think it is not too off-topic, because I am not at opposed to hierarchical organisation, I just don't like to have them fully visible in the namespace:

• Most wikis map their topic name into the URL;
  consequently, you propose to map hierarchy names into the URL
• Your vision of implementing a kind of LiterateProgramming with a wiki by overlaying the project code tree is great, but note:
  • In practice, the code hierarchy depends on the way we all use fs hierarchies
  • Therefore, the wiki namespace is based on fs needs
  • fs hierarchies tend to change constantly -- isn't this your experience as well? Granted, CVS trees are an exception to this rule because CVS makes it really hard to move anything. But then, this is the most requested feature.
• Besides this cool new idea (are there other special cases?), a wiki web needs not be constrained by the underlying fs: while it is unpractical, to have just 100 files in one directory, a wiki easily handles 1000 pages in a web
• My point is to use this fact and have the hierarchy as meta information outside of the path, thus keeping the visible URLs unaffected by any hierarchy reorganisation.
• Of course, there are many possible solutions to this problem. But a TWiki with a KoalaSkin and applied parent/child linking works right now.

Overall, your ulterior motive is indeed a very tempting reason to implement the directory nesting anyway. I might actually overcome my (irrational?) fear of "corporate weenies", who could abuse this feature to force the orgchart of the day into the URL namespace.

-- PeterKlausner - 16 Apr 2003

Urg... Started responding, went away, open edit window gone. Sigh. Shorter response as a result, probably better.

(Aside: collided with your lock. CVS could merge, but merging has problems. Zwiki has an "append mode" for comments, that does not require locking.)

(Aside: I think I'll refactor the above. Does TWiki use to OriginalTopic/DiscussionOfOriginalTopic idiom?)

Responding to PeterKlausner's poins that can be moved back to the HierarchicallyNestedTwikiWebsNaming topic:

• I am not at opposed to hierarchical organisation, I just don't like to have them fully visible in the namespace:

Well, having the hierarchy at least partially visible in the namespace is one of the reasons for hierarchy. Names collide. WikiNames collide. Separate namespaces mean that you do not have to use ComplexWikiNamesJustToAvoidCollidingWithOtherPages.

I agree that we want to flatten the namesapce as much as possible. Too much hierarchy is just as bad as too little.

I am also interested - this is why I started this topic - in naming syntax that make hierarchy less likely to complexify names. E.g. (I should have mentioned this originally), paths that search .;..;../..;../../.. etc. up to the root of the wiki, if a topic page is not found locally. (Years ago I modified the Bourne shell to do this for UNIX commands. Mixed. Main problem, security, just like having . in the path.)

• Most wikis map their topic name into the URL;
  consequently, you propose to map hierarchy names into the URL

Yep, probably.

But also note: that doesn't mean that there cannot be other URL to filesystem mappings. I gotta writte KeepingReferenceNamesForever.

By the way, one of the advantages of TWiki over collaboration tools, including some other wikis, that put some uneadable number in the URL, is that TWiki URLs are moderately informative in and of themselves.

• Your vision of implementing a kind of LiterateProgramming with a wiki by overlaying the project code tree is great, but note:
  • In practice, the code hierarchy depends on the way we all use fs hierarchies
  • Therefore, the wiki namespace is based on fs needs
  • fs hierarchies tend to change constantly -- isn't this your experience as well?

I've certainly worked with companies where this was the case. But I managed to persuade them to see the error of their ways.

In other words, I've had some modest success in persuading companies to make filenames durable, in much the same way we want URLs to be durable.

The biggest obstacle is the act that poorly managed filesystem hierarchies mix up logical naming and physical disk partitions and fileserver assignments. I've found that I can nearly always hide that with symlimks; and I can often do better when I have cooperative sysadmins who will create alternate mount points.

The biggest problem is when a single large logical directory needs to be split across two physical disks. Not a problem when it contains subdirectories, but a problem when it is flat, with a lot of large files. There's a decades old technological fix to this - Stephen Korn's directory continuations (or, in a database, a view which is the concatenation of several tables). I expect to run into this problem with wiki webs within 2 years.

Granted, CVS trees are an exception to this rule because CVS makes it really hard to move anything. But then, this is the most requested feature.

Yep, with me being one of the people requesting CVS rename. Look to KeepingReferenceNamesForever; the same issue applies here.

By the way, renaming in TWiki is broken with regards to version control.

• Besides this cool new idea (are there other special cases?), a wiki web needs not be constrained by the underlying fs: while it is unpractical, to have just 100 files in one directory, a wiki easily handles 1000 pages in a web

Well, I have worked on projects that had 64,000 files in a directory (running into some filesystem limits preventing going larger). typically, with a database frontend rather like twiki.

Other special cases? Like I said (1) my ulterior motive, mapping wiki onto the project source code tree; (2) making wiki security match UNIX security; (3) name collision.

Frustrating: like I said, all of the same arguments that made filesystems go hierarchical apply.

• My point is to use this fact and have the hierarchy as meta information outside of the path, thus keeping the visible URLs unaffected by any hierarchy reorganisation.

Flip side:

(1) One of the things that is most awkwad about Apache is that so much remapping can go on. It's good to be able to map; it's bad to have to essentially run Apache to figure out where the file for an URL lives. Most sites adopt conventions that allow big subtrees of the URL space and the file space to match, modulo a prefix.

(2) It sould be possible to map filenames to URLs. But it should also be possible to handle URLs which are no longer in the filesystem. If I can only have one, I'll take mapping, but eventually both.

(3) One of the big reasons why I chose to adopt TWiki rather than Zwiki/Zope was that it is trivial to map the URLs to the filesystem. I've got guys in my project who insist on using UNIX tools like find and grep to search for stuff - who insist on working outside wiki.

How about: I want there always to be an URL that maps to the filesystem. But I also want URLs that are guaranteed not to change, if the fuilesystem gets reorganized. And I want people to use wiki names as much as possible, which similarly do not change.

• Of course, there are many possible solutions to this problem. But a TWiki with a KoalaSkin and applied parent/child linking works right now.

Helps. And we are not going to lose that.

But it doesn't help me with the fact that I have to coexist with UNIX filesytem security, some of which is most conveniently done at directory granularity.

• Overall, your ulterior motive is indeed a very tempting reason to implement the directory nesting anyway.

Glad to have tempted you - or are you just saying that to appease me?

• I might actually overcome my (irrational?) fear of "corporate weenies", who could abuse this feature to force the orgchart of the day into the URL namespace.

Just say no. Change your organization, or change your organization. Actually, we probably want the ability to map the orgchart of the day, but we do not want to invalidate existing URLs as a side effect.

Maybe I am just lucky to have worked in companies that are not so short sighted. But, if so, why should you make MY job harder, because your corporate weenies are stupid.

-- AndyGlew - 17 Apr 2003

Interesting discussion. I agree with PeterKlausner, better to use as few webs as possible, and to depend on parents / breadcrumbs within a web. This has worked very well for Wind River where we have now over 20K pages. I have not found a reason yet why we would need hiearchical webs. Breadcrumbs, easy cross-linking, and search are important for a web site, and that does not map well into a file system hierarchy.

And yes, we had many reorgs and have been tempted to rename webs. We decided not to do so, but to introduce a "hierarchical map" of all TWiki webs, reflecting the org chart of the company. This is in lieu of the automatically generated SiteMap. The map mimics a hierarchy, but all webs are flat as we know. Here is a non-functional example for illustration purposes: (links and background color missing)

• Start here:
  • Main web: TWiki Users, Groups, Offices
  • TWiki web: TWiki documentation
  • Term web: Glossary of terms
• Sales -- John Smith
  • SAtech web: Technical sales -- Tom Miller
  • SAcall web: Call center -- Mr Beans
• Engineering -- Foo Bar
  • Dev web: Development -- Albert Einstein
    • Bread web: Bread slicer -- Aaa Bbbb
    • Toast web: Toasters -- Ccc Dddd
  • QA web: Quality Assurance -- Eeee Fffff
    • QAtest web: Test result database -- Eeee Fffff

and so on, you get the idea. Since this hierarchy does not reflect the underlying TWiki web structure it can be changed easily with each reorg. Problem solved.

-- PeterThoeny - 17 Apr 2003

Yes, an interesting discussion. I assume that your "hierarchical map" is based on the bread crumb trail?

I guess some of us (not me) use the breadcrumb trail far more than others — my intent was to use multiple webs to organize my private LAN and eventually WikiLearn. But, maybe I can / need to rethink that. Here are the problems that I see with the breadcrumb trail approach (correct me if I misunderstand / am missing something):

• The bread crumb trail approach doesn't avoid the problem of name space pollution -- I can't have, for example, a page named IfSyntax for both C++ (Cpp) and Pascal unless I actually prefix the page name with Cpp and Pascal (like CppIfSyntax and PascalIfSyntax).

• If, on the other hand, the bread crumb trail let me have two pages, something like this, that would be helpful:
  • TWiki > Wikilearn > C > IfSyntax
  • TWiki > Wikilearn > Cpp > IfSyntax.

• Another advantage of separate webs is limiting the scope of a search. If I want to find the "else" in C++, I search in the Cpp web, if I want to find "else" for Pascal I search in the Pascal web. If I was to switch to the bread crumb trail approach I'd want some simple way to similarly limit the scope of a search (and without having prefixes like Cpp, Pascal, etc. for each page).

• I assume you (who are using the bread crumb trail approach) are not using prefixes -- I'd see little value in having to do this:
  • TWiki > Wikilearn > Cpp > CppIfSyntax
  • TWiki > Wikilearn > Pascal > PascalIfSyntax

• Tools (of various sorts) don't exist to allow you to easily see and adjust the bread crumb trail. (Maybe some of these problems are just due to my mental habits of observation because I started using TWiki before the bread crumb trail existed and just haven't integrated it into my thought processes — I should be able to change that if it makes overall sense.):
  • In the default skin there is nothing like a "tree view" to show what pages are children of a particular parent. (For example, if I wanted to see a list of all Cpp or Pascal pages. You can't even find them by a "structured search", unless you use prefixes as mentioned above.)
  • Setting the parent of a page is [implicit | somewhat invisible] — if you simply put a link on a page that page becomes a child of the page you put it on. If you want to assign it to a different parent, you must adjust the parent after saving the page (using the "more" menu). Contrast that to the ease of creating a link to a page in another web using the <Web>.<PageName> syntax, with or without the double square brackets. (I am aware that you can use the parenttopic parameter in a URL to explicitly parent the topic as you desire, but that is not the level of convenience I'd like to have — first of all, how many people are going to remember that exists, and how many of those will bother to use it?)

Then, if you have both webs and the bread crumb trail, and you create a page in a different web than the parent page, you can have the interesting situation of a bread crumb trail like this:

TWiki > <Web> > <PageInThisWeb> > <PageInADifferentWeb>

... which seems to create a lot of potential for confusion.

• And, when people don't pay attention to the bread crumb trail and the length of the names of parent pages, you can get extremely long bread crumb trails:

TWiki > Codev > HierarchicallyNestedTwikiWebs > HierarchicallyNestedTwikiWebsNaming > KeepingReferenceNamesForever

... this is not the worst example I've seen, it was just convenient and, fortunately, it is sensible — mnay times the bread crumb trails that I get (because I'm not conscious of the bread crumb trail or trying to keep it sane) do not make much sense.

For WikiLearn, I had serious thoughts about figuring out how to turn off the bread crumb trail (or reallly, the parent / child relationship portion of it) and use only webs for organization. But, I would like to have the ability to have hierarchical webs.

So why did I write this? I guess first to learn how others deal with the problems I see using the bread crumb trail.

-- RandyKramer - 17 Apr 2003

Just a response to PeterThoeny's comments about virtual hierarchy, and a general comment on how even smart customers can seem pretty dumb.

This is going to sound harsh, but only the people who understand the code or spend all their time thinking about web sites could ever understand or appreciate the beauty of virtual hierarchy, unless extra-ordinary measures are taken by the site owner. I personally think it's great and I have absolutely no problem getting around, because I'm used to looking things up and figuring out things for myself. The average users of most corporate web sites, however, have extremely poor library skills, and don't care what's behind the interface they're using as long as they can quickly and intuitively get to the data they need or publish data to others without any fuss.

Arrr, Tharr Bee Yer Problem... Intuition. Most people don't have it (yes, even smart people) when it comes to using web sites, unless they're presented with something familiar.

Before I wrote MegaTWiki, the folks in my organization (somewhat average to brilliant chip designers) were thouroughly confused by the fact that the URL didn't match hierarchically to the place they had gone to in the web site. Folks generally seemed confused as to where they were and where they wanted to go, because people pay much more attention to URL's than you would think. Implementing hierarchy in the URL that matched what the UI was showing them in MegaTWiki cleared up most of the problem (although it took several weeks of coding && debug). You have to watch out for what these little inconsistencies will do to your average user, and my average users are not necessarily going to react the same way as your average users.

This little book, Don't Make Me Think by Stephen Krug, discusses all the little things that can make a website completely unusable to your customers, and how you can go about making it work. I used it to make make minor improvements to the MegaTWiki skin, but you can only help so many people.

I'd tell you all about the book and how cool it is, but the link should be sufficient

While I agree that minimal hierarchy is best, compartmentalization of data is sometimes a necessary evil, if only to keep the unwashed users from stepping on one another.

-- PeterNixon - 09 Jun 2003

Peter, I don't think anyone discussing logical/virtual hierarchies is saying that the URL wouldn't look nested to the end user - it's just the server side implementation people seem are discussing AFAICT. That way the hierarchical view of the website would look the same as the URL. The mapping of logical to physical would happen behind the scenes. (Or at least the way I read it). In this scenario, direct mapping of URL to filesystem is one idea, but a logical mapping allows greater flexibility. (At a trade off for a systems admin performing backups!)

-- TWikiGuest - 11 Jun 2003