Tags:
attachments1Add my vote for this tag extract_idea1Add my vote for this tag create new tag
, view all tags

Automatic Attachments

Purpose

With Automatic Attachments enabled, all files in a topic's attachment directory are shown as attachments to the topic - even if they were directly copied to the directory and never attached by using an 'Attach' link.

Summary

The changes I have made (r6296) alter the behaviour of the FILEATTACHMENT meta data, so that TWiki queries the disk instead of looking in the meta data. This logic is switched off by default.

Motivation

Having TWiki autoattach makes a lot of sense. Having autoattach plus ReplicationUsingUnison a viable means to publish sets of photo just by dumping them in the correct directory.

I have worked around what I saw as the main drawback: that TWiki usefully stores comment data about each attachment.

Implementation Notes

My implementation refreshes the in-memory metadata structure but does not save that metadata structure back to TWiki�s Store.

It alters Meta::find and Meta::get. These now check for $TWiki::cfg{AutoAttachPubFiles} (off by default) and, if set, delegate to TWiki::Attach::findAttachments and TWiki::Attach::getAttachmentAttributes respectively. In turn, these use TWiki::Store::getAttachmentList and TWiki::Store::getAttachmentAttributes, which find implementations in RcsFile.

There exists logic that associates attributes from the stored metadata if the file mentioned in the metadata has the same name as a file in pub.

In the future I would expect Meta to be type-overridable such that a subclass could exist to explicitly override FILEATTACHMENT meta data information. This would avoid the need for conditional logic in Meta and create a new class TWiki::Meta::FILEATTACHMENT.

What is autoattached

Any file that:

  • does not end in ,v
  • does not start with . or _ (this allows for the convention that folders dedicated to plugin data start with _)
  • All folders

Writing back to the meta data

Metadata changes are recalculated and held only in memory. They do not get written to disk. This means they incur extra disk operations and that the topic never gets updated with the correct information.

I didn�t want to do this until someone with a fuller appreciation of how it should work (in particular the timing of such write backs).

  • I am aware that TWiki�s coupling of topic and meta data into the same underlying file would mean that last-file-write dates would be altered by periodic saves
  • If we don�t save it back then all accesses to FILEATTACHMENT will be slowed to the speed that TWiki can get to the disk.
  • This same code should be called from the WebDavPlugin where it refreshes its data set.

Some options include:

  • Add a button to the topic attach page to allow the user to refresh the document attachments. This would call TWiki::Attach::findAttachments() and write this back to the meta data
  • Have a CommandSet that sweeps a particular web or topic calling findAttachments and writing it back to the metadata.

I�ve always been a little wary of TWiki's notion that attachment data is metadata but as this structure is used by many other plugins (in particular the FQP) I guess it is too late to change it to a separate logical structure.

Perhaps TWiki should store meta data in a separate file (e.g. a .TMD, for TWiki Meta Data)

Future Enhancements

  • Why does TWiki disallow spaces in filenames?
    • This breaks attachments of filenames already existing on the disk
    • Is it a security risk to allow spaces?
  • Autoattach could perform automatic check-in of files it auto-attaches
    • This would likely have to be an off-line or progressive background tool as
  • Why does TWiki retain separate data and pub directories?
    • We could unify the data and pub hierarchies
    • Have a folder for every topic
      • Index.TML to store the topic content
      • Index.TMD to store meta data
      • Index.TMA to store attachment data (or MANIFEST.TMF)

-- MartinCleaver - 12 Sep 2005

How you can help

  • Does it work for you, on your platform?
  • Does it work with your favourite plugin?
  • Does it impair the performance of TWiki?
  • Should we and when should we cache the auto-attachments as persistent normal attachments?

-- MartinCleaver - 13 Sep 2005

-- MartinCleaver - 13 Sep 2005

Martin, thanks for this, it opens another can of worms, but one that needed opening wink

Autoattach is a great idea when the store implementation is files on disc. For some other store implementations it will work pretty well as well.

This sort of feature seems to me to be another strong case for store encapsulation, and the removal of attachment (and topicinfo) meta-data from topics (something I've been wanting to do for ages).

  • But it won't work for any store that elects to use another store medium (e.g. a database). So shouldn't this be something that is implemented by the Store? As such, why does the rest of TWiki have to know about it at all? Changing meta-data management to acknowledge this feature layers yet more complexity on an already over-complex implementation.

-- CrawfordCurrie - 13 Sep 2005 (addressed issued removed, 1 Oct 2005)

I'm happy to see this Martin, thanks! For some previous ruminations on the folder-for-every-topic idea see AreWebsTopics

-- MattWilkie - 22 Sep 2005

I'd moved some documentation into the release but Develop:Bugs.Item481 / SVN 6643 removed it again so I've restored it here.

-- MartinCleaver - 01 Oct 2005

What is the status of this feature? Bugs:Item204 lists this as "Discarded".

-- PeterThoeny - 27 Oct 2005

Thanks for the question. This feature is considered complete. Plugins that write into pub/ without following the conventions of using directories such as TopicName/_PluginName/ will have their data exposed.

-- MartinCleaver - 27 Oct 2005

Under What is autoattached you mention "all folders". I'm assuming that this means you ignore all folders.

I've been looking at this from the context of HelpUpgradingCustomizedTwiki, and it seems to have most of what we implemented, except:

  • revision control (which should be straightforward to add) and
  • support for unlimited directory hierarchy.

The second is a make-or-break requirement for our users, since many of them use it just like a Windows share drive. What would it take to support this? Technically, it should be feasible (although the attachment table templates will need to change), but is there any major part of TWiki that would break?

-- PankajPant - 26 Jun 2006

I'm wondering if folders can't be transparently matched to HierarchicallyNestedTwikiWebs.

-- FranzJosefSilli - 26 Jun 2006

The simples attachment table interface change would be to prepend the folder path names to the filenames:
myfile
myfolder/mynestedfile

How to update nested files will be a pesky one though.

-- ArthurClemens - 26 Jun 2006

Here are some images to illustrate what I'm talking about.

This is the initial view of the attachment table, collapsed by default.
Attach1.png

Here the attachment table has been expanded, but folders are still closed.
Attach2.png

Now the sub-folder has also been expanded.
Attach3.png

And finally, the Windows Explorer view of the topic's attachment folder.
Attach4.png

  • The Manage button is a direct link to the share drive in a Windows machine (and doesn't show up in Unix machines).
  • Per-folder open/close buttons use simple twisty-like Javascript and are very simple to support.
  • As with any "feature", unlimited folder hierarchies have been abused by some of our users with more than 300 files over 50 directories in one case. We had to add logic to turn off the rendering of the table in cases like this since IE would essentially hang for a few minutes trying to render the table.
  • Note that these images are from a pre-production version where user name to TWiki name mapping wasn't working correctly and the "delete" button was disabled.

One of the best examples of where automatic attachments is a big productivity win is for documentation images. A number of our doc topics contain quite a few images. Typically, these are contained in a single Visio file. Each "page" of the Visio doc is appropriately named and upon save, a VBA macro automatically exports each page to an image of the same name. Now, updating the inlined images in the topic is as simple as: open the Visio file from the share, edit it in place and save. All the images are regenerated and the topic immediately reflects the changes.

Hope this helps.

-- PankajPant - 27 Jun 2006

What is the current status of this feature?

-- BrianGupta - 14 Mar 2007

One interesting issue about automatic attachments is I18N - you need to make sure that the attachment filenames are converted into the server charset (e.g. from UTF-8 on the client to ISO-8859-1 on the server. Bugs:Item3652 has flagged up a regression here, but the basic model is that you can't use UTF-8 on the server if you want character set (content) I18N of WikiWords, WebIndex sorting, searching, etc. See InstallationWithI18N for recommendations on setup, UTF-8 should be used only for Chinese or Japanese sites where English-only WikiWords are generally fine.

-- RichardDonkin - 15 Mar 2007

Brian: It's implemented and available in TWiki4.x.y. Enable it in "Store Settings" in configure.

It's a shame that no-one has ever followed through with an Apache 2 version of WebDAVPlugin (funding has been hinted at several times but never delivered) as that is a better solution IMHO.

-- CrawfordCurrie - 16 Mar 2007

I just tried this out on a fresh install (4.1.2) and it seems to have updated the topic's metadata. As a test, I uploaded a file via the normal attach method and copied another directly into the file system, but both are listed in the FILEATTACHMENT metadata.

I thought this feature did not write update the metadata? What gives? Can someone educate me?

Here's what I see in the topic.txt file:

%META:FILEATTACHMENT{name="file2.jpg" attr="" autoattached="1" comment="" date="1178824606" path="file2.jpg" size="53178" user="UnknownUser" version=""}%
%META:FILEATTACHMENT{name="file1.jpg" attr="" autoattached="1" comment="Uploaded via TWiki attach." date="1178823975" path="file1.jpg" size="52408" user="Main.TWikiGuest" version="1"}%

-- PankajPant - 10 May 2007

And on an unrelated note, when I first tried to add the example metadata lines inside a verbatim block, TWiki seems to have interpreted them to be real metadata and as a result there are now 4 ghost files in the attachment table: {file1, file2, getfile, caltob02}.jpg.

-- PankajPant - 10 May 2007

Yes, that is a known limitation, TWiki interprets meta data also inside verbatim blocks.

-- PeterThoeny - 12 May 2007

I think I understand the behavior a bit more now. The attachment folder and metadata are generated on the fly, and as soon as the topic is edited / saved, the generated metadata gets written permanently to the topic. A couple of comments:

  • I'm not sure how this would play with a situation where most attachments are added/edited via an SMB interface (via Windows Explorer) and frequently out of synch with TWiki metadata. See my comments above from June 2006 and HelpUpgradingCustomizedTwiki to see where we are coming from.
  • Some of our topics have a large number of attachments and we really didn't care that they were not in the topic's metadata (in fact, if they were it would create a significant performance hit).
  • I tried creating a sub-folder under pub/Web/Topic, but it gets listed in the attachment table as an ordinary file.
% META:FILEATTACHMENT{name="more_files" attr="" autoattached="1" comment="" date="1179239573" path="more_files" size="4096" user="UnknownUser" version=""}%

Still continuing my investigation. We would really like to merge our changes with this feature.

-- PankajPant - 15 May 2007

very cool smile We'd love more code to make this feature even better - especially in a windows server environment (most developers spend too much time in linux - though I just bought a Mac, so that's even worse smile )

Please note that automated unit tests are one thing we desperatly need in this area - especially as we move to a more distributed & scalable server environment.

-- SvenDowideit - 15 May 2007

Aha. Store::saveTopic() calls _removeAutoAttachmentsFromMeta(), but this is currently just an empty stub. I guess some logic needs to go in there for this to prevent save from writing the metadata to disk.

Also, RcsFile::getAttachmentList() needs to be updated to skip (or otherwise handle) any subfolders in the attachment dir.

I'm sorry if all this is obvious information, but I'm just documenting things as I find them.

-- PankajPant - 16 May 2007

Pankaj - did you get any further with this? A new client's case might take me to add functionality in this area.

-- MartinCleaver - 08 Oct 2007

Sigh. Unfortunately, we haven't had the resources to do anything. Our management is not ready to allocate any dedicated support for this. Given the current situation, it doesn't look like we are going to be able to move forward from Cairo anytime soon (yikes!). We (the TWiki admins) all have real responsibilities and just support the system as a hobby.

I still believe this is very important and will attract many corporate users. I'm glad you've found a client to support this. Hope you manage to get something going.

-- PankajPant - 09 Oct 2007

I created UpdateAttachmentsPlugin to provide an alternative to using the built in autoattach feature. It can be run from a cron job, and updates topics with the new attachments in pub.

-- SvenDowideit - 29 Dec 2007

Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng Attach1.png r1 manage 1.1 K 2006-06-27 - 01:48 PankajPant Hierarchical attachment folders
PNGpng Attach2.png r1 manage 9.4 K 2006-06-27 - 01:48 PankajPant Hierarchical attachment folders
PNGpng Attach3.png r1 manage 12.3 K 2006-06-27 - 01:49 PankajPant Hierarchical attachment folders
PNGpng Attach4.png r1 manage 24.5 K 2006-06-27 - 01:49 PankajPant Hierarchical attachment folders
Edit | Attach | Watch | Print version | History: r29 < r28 < r27 < r26 < r25 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r29 - 2007-12-29 - SvenDowideit
 
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2015 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.