persona1Remove my vote on this tag create new tag
, view all tags

A catalog system on top or below


Very soon most topics on our Wiki evolved, such that at the end of it there is a Links section. Also the start page contains a section topics, which lists some interesting start points to get into.

The handish catalog soon got outdated, most of our (5) users didn't keep it up to date, they just use the Recent changes to see what's new. But this keeps older stuff behind, even if it is interesting, the better a topic gets, the less it changes.

For that reason, I wanted to have some kind of catalog, found http://dmoz.org and downloaded their stuff. I used the Catalog package from CPAN to create MySQL-Tables.

I assembled a package and associated some topics which represent a catalog category with the corresponding dmoz category. Now I have a Category Browser which displays DMOZ-Cats. If a Cat. is a topic, it just displays that. The topic may include %CATALOG{...}% variables, which are replaced by corresponding queries, f.g. links, subcategories or the category path.

This works quite fine so far, but now I could do with some feedback:

  • I want to replace the Web.Topic title by a catalog path for a seemless integration, but this would obsolete the WEB.TOPIC scheme, so a user will probably not understand one or each system.
  • I want to let the user decide whether he wants to browse category oriented or the wiki way.
  • I want to let a topic decide whether it wants dmoz links and sub categories displayed expanded or just display a button or something.

But I do not have a concept to make these ideas sound and easy to use and understand. I'ld be glad for any comments and suggestions on this.

  • Ah yes, this sounds good. I'd like to play with that! -- MartinCleaver - 15 Nov 2001
    • If you want so online, just visit my site. If you'ld like to install it on your wiki, send me a mail then i'll send you the plugin. The time consuming part is to download and install the DMOZ data and the Catalog package (CPAN), so if you start right there, I guess you'll receive my code in time wink Btw. the package is not finished, you can display categories, include dmoz data, but you can not yet associate topics with categories or create new links or categories yourself (have to use sql for now) -- MichaelUtech

(Comments about this plugin using MySQL, partly moved to PackageTWikiStore)

I guess, if I continue this road, this will soon have several confusing effects, because there are two incopatible ways of doing it (f.g. using metadata or database). Using both is fine now, but if I'm going to use the databases more, I'm departing from TWiki which I would prefer not to do.

Performance Using mod_perl boosted the overall performance a lot (a factor between 5-10). My innovations make it slow down again (1.7 or so) which is partly because I can't cache much data between calls.

+--- Some applications are:

  • Better and more general forms (using databases to stored and display data and also a form layout). I would like to have a way for users to easily define databases themselves and associate records f.g. with wiki topics or display reports in topics. This is quite some work, but I need such a thing for my intranet applications anyway.

-- MichaelUtech - 14 Nov 2001 (Refactored from SeveralIdeas, hopefully acceptably!) (sure, thanks for taking the time)

  • Michael - Could you package it as a plugin and upload it to TWikiDotOrg?
    • Sure, but the package is not yet finished, and as a general purpose package it would be better not to install dmoz locally, but to access the dmoz site. I have not much time right now, so this will not be ready very soon. -- MichaelUtech - 17 Nov 2001

-- MartinCleaver - 17 Nov 2001

Micheal, a dmoz plugin is great idea. I have a few thoughts from a user perspective to throw in to the discussion.

I have been a dmoz editor for 6 months or so. Participating in dmoz has been both a wonderful and painful experience. One of the biggest holes in the way dmoz operates is that everything (meaning all data entry type activity) must be done on their server, which is often overloaded and unresponsive. Another lack is that adding extra attribute fields, like keywords or opinions/reviews, is next to impossible. (the dmoz governing bureaucracy is wary of, and slow to act on, adding complexity to the database.)

A TWiki<->dmoz could help rectify these lacks. For example, I could export the dmoz categories I edit to my local twiki server, I and other users add and update entries, and periodically synchronize the changes with dmoz using my editor account. Secondly, the local twiki-dmoz database could have fields the official dmoz directory does not have (such as opinions, reviews, keywords, & thumbnailed screenshots).

-- MattWilkie - 18 Nov 2001

I just got dmoz editor, so that idea sounds very interesting! wink

But then I'll have to relay that, until the part that motivated my in first place (adding hierarchie to wiki and then links to topics) is somewhat closer to usable. My first approach was to download the dmoz rdfs. This is because I'm kinda collector, but for a general mechanism this is impractical and unnecessary. As soon as I switch to this part, I'ld be happy to look at this perspective. Would dmoz like the idea of having an "offline" interface to their DB?

If they do, they may help us by providing a stable interface, if they don't I'm not sure if working to this direction is a good idea. Would you mind asking them (if what I understood is what you ment anyway).

-- MichaelUtech - 18 Nov 2001

>Would dmoz like the idea of having an "offline" interface to their DB?

I don't know. The idea has surfaced a few times in the Fora but no action has been taken by the "powers that be". I don't think they are too interested in the idea probably because of security concerns and the effort required. However if somebody were to build one that didn't require any changes (eg. effort) on their part and it didn't appear to present any security risks...

Dmoz is a wonderful idea and project but it is currently suffering from too many volunteers, who have no power to affect fundamental change, and too few developers who do have the power to change things. If it were an open source project, or Netscape dedicated more staff, this could be addressed but it isn't so here we are.

Disclaimer: I haven't followed any of the fora for about 2 months so I'm a little out of touch with the current state of affairs.

-- MattWilkie - 20 Nov 2001

That was all very interesting! I downloaded all 150MB or so of DMOZ data a while ago, just to have it, but it's too much data to use in any public way unless you're going into business. Google did a fine job of lifting it wholesale, and building around it, but they're spending a fortune.

I do have two tiny (200-300 links) databases that I manually conform to DMOZ classification scheme - I look up the sites as I add them, and create the category paths as required. At first I though that in the future I could incorporate huge chunks of DMOZ. But their classification system gets pretty arcane at times. On the top level, it's similar to Yahoo, but the subcategories often seem overcomplicated,Yahoo is a lot more logical, and gets more done without going as deep into subcats. So I'm about to start going my own way.

I'm curious about the idea of combining Wiki with a search engine. I kinda tried to get comments on ScoopMeetsTWiki, where a CMS/weblog system wanted to incorporate a wiki. PostNuke is quietly it seems from the CVS looking at Wiki. More CMS-type platforms want a freeform - wiki - addition. It's getting trendy. But I don't so much see the other way, Wiki to database. 'Cause there's no way to make a search directory not a database, unless you make it the wiki, like Wikipedia. TWiki has pretty much pushed the edge of wikiness - it's still fast and freeform, but go to usemod.com or c2.com and I find those clean, simple Wikis refreshing, although I love TWiki. Does that make sense to you?

The CMS/Weblog type set-up: Slash, Scoop, TWIG, Greymatter, PHPNuke/PostNuke, so many, are fun. Their moderation schemes, karma points and all, are cool, the structured version of expecting Wiki-ers to clean up, refactor, edit on their own. They integrate with web links directories, reviews bases, calendars, forums, anything. PostNuke - PHP/MySQL - is fun, but Scoop and Slash (perl) seem like the best for now. But basically, whatever the brand, it makes sense to me to optimize wiki for the basics: revision control, autolinking, supersimple slick HTML shorthand, searches, and a cool layer of meta data structure for classification, but beyond that, just wrapping it in a CMS-type setup.

I haven't quite nailed it, but I feel TWiki,specifically, is on the verge of being something amazing. Wikis are getting more popular, but TWiki's on another level. It's doomed if it loses the wiki core. But it could become an alternate CMW environment that funnels content out into other systems. Something like that.

This isn't the corporate collaboration mode, it's from c2 original Wiki, which gets quite rarified on wikiness - but is a software dev base too:

"wiki is not WYSIWYG. It's an intelligence test of sorts to be able to edit a wiki page. It's not rocket science, but it doesn't appeal to the Video Addicts. If it doesn't appeal, they don't participate, which leaves those of us who read and write to get on with rational discourse."

In that context, I don't see direct integration - as in, coded - with search engines and that sort of hybrid as the coolest course. More like, find the links using the many other tools for that - including lots of free and commercial frontends and live homes for DMOZ data, like ODP++ - then bring the carefully selected to T/Wiki and incorporate them with ease, as straight autolinking URLs, with square bracket rules, or added to Interlinks for regulars. Optimize those features to improve link access. Etc...

You can always delete this at will!

-- MikeMannix - 22 Nov 2001

Matt, I've heard that DMOZ is a nightmare of feudal states and off-the-hook politics. And now it's owned by AOL Time Warner, too. Is it more fun than painful?

-- MM

The more I think of it, the more uncertain I get about this plugin.

What I really want is:

An easy to use and flexible hierachy for topics. The reason to choose a database and not the meta-parent was to be flexible so as to make it easy to create links and also to do fast and powerfull searches on structural level.

Using DMOZ had two motivations. One was to choose a default hierachy, which people are used to (dunno how popular DMOZ really is) and then to have external links (both of local and DMOZ origin) in a database, to have a way to easily check them for availabilty and to offer ratings to external links. In short to be flexible so that new ideas can be implemented on the fly and removed just as fast if they prove unattractive. As off now, a topic does not display any catalog information (structure or links) if it's not using any %CATALOG{}% directives. So the whole thing is quite transparent. What I call the CategoryBrowser is simply a wiki page with two or three directive to show the contents.

Beside these two points, the hierarchy is just a view on the pool of topics which are available. Since the topics are self contained (they do not have any db-references) there could be several hierarchies on top, or none at all.

I think that this will not really influence the nature of TWiki, it should just add some neat features which are quite invisible if not used.

I don't know on the other hand how usefull this is anyway. wink

-- MichaelUtech - 23 Nov 2001

> ...DMOZ. But their classification system gets pretty arcane at times.

A result of the system having grown organically. There has been a lot of discussion of revamping the whole db and establishing some system wide standards . It will take some time for 1) a standard classification system to be decided on, and 2) to be implemented (there are a lot of people to retrain).

> ...On the top level, it's similar to Yahoo...

Not surprising since way back when Yahoo was based on dmoz. : )

> I've heard that DMOZ is a nightmare of feudal states and off-the-hook politics. ... Is it more fun than painful?

Now that I have settled into my little corner and am quietly refining it, yes it is more fun than painful. When I was still fired up with initial newbie enthusiasm it was painful to realize how hard it is to implement changes. In every other volunteer or open source type community I've participated in it has been relatively easy to get your voice heard and see your contributions make a difference. It took several weeks for the sheer scale to sink in: there are ~7,000 active* editors, which is at least an order of magnitude, maybe two, larger than anything else I have been a part of.

(*As of September 2001; 'Active' means has logged in within last 30 days. There are many categories which don't need such frequent updates.)

As for feudal states and politics, dmoz pretty much seems to encapsulate the full range of human behaviour.

> ...dunno how popular DMOZ really is

Depends on how you want to define popularity. I think there are many thousands (millions?) more people using downstream dmoz data than at the source. There are over 200 known sites using dmoz RDF dumps in one form or another (directory.google, about.com, thebrain.com, ask jeeves, etc, etc.)

In the context of your question I would use just the top two or three levels of the heirarchy since that does appear pretty standard.

Getting back to twikifying dmoz or dmozifying twiki, there are at least two distinct ideas mentioned so far:

  1. using TWiki to create a better dmoz-type bookmark collection
  2. using the dmoz hierarchy and a database as a template for a better within-twiki navigation system

I think the issue #2 is meant to address is probably better covered by some other alternate navigation schemes being discussed

-- MattWilkie - 26 Nov 2001

This is maybe on topic for this thread, and maybe not... That said, it's worth bringing up I think...

When I was at the Janet Web Cache Service I developed a very nice database abstraction layer that I've continued towork on, and added TWiki style request/response templates to as front ends, and I'm currently in the process of making the whole system embeddable inside a Twiki page as a TWiki plugin. The upshot of this is that you can create canned SQL queries on a page, and then reference those canned SQL queries on a standard TWiki page in a manner not dissimilar to the standard query syntax for a twiki search.

The results are then inserted into templates - which are skinnable - and then finally these are inserted into the Twiki Topic text, which are then parsed as if they were normal twiki text. (The ironic thing here is it made any WikiWord syntax words from inside the database into WikiWord links, which inadvertantly became Very Useful smile

The code is pretty much at a pre-alpha stage for generic use, and alpha stage for the intended use, but when it's of a good alpha quality, I'll add the code to the Plugins web - strictly speaking, it'd be a TWikiAddOnProduct rather than "just" a plugin, but it's hopefully "a start". It's designed as a relatively generic web query engine internally, with the aim of being fully configurable for certain uses from TWiki.

Given current workloads, I'd expect to make the alpha release sometime before Christmas, but maybe not much before then smile (My way of trying to give something back to the TWiki community!)

-- TWikiGuest - 27 Nov 2001

There has been some discusion on Open Directory project known as DMOZ (abbrevieted) and the lack of "free" inplementation of the software. As DMOZ is giving away data (collected by volunteering editors) in form of big XML dump from their database someon should try to use this for complettly FREE implementation. Guys at FreeMOZ.sf.net project are starting to work on this from scratch, but I am interested in option of using existing TWiki features (user managment, link system and XML support) to actually see if this is possible.

Similary it would be interesting to replicate Everything2 feature set in TwikiAsE2 (everything2.com) or TwikiAsSlash (Slashcode.com). It is interesting that Everything2 code was already used for Slashcode-like website on Clustering (part of VA Linux network).

Can anyone elaborate (who was experiance in these applications) how hard/easy would it be easy to achieve this... heaving in mind that not all things have to be the same/compatible to the original ... it should just be feature full implementation.

-- ZeljkoBlace - 28 Jul 2002

Edit | Attach | Watch | Print version | History: r14 < r13 < r12 < r11 < r10 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r14 - 2006-02-15 - PeterThoeny
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2018 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.