Tags:
development1Add my vote for this tag editing1Add my vote for this tag email1Add my vote for this tag create new tag
, view all tags

MailToTWikiAddOnDev Discussion: Page for developer collaboration, enhancement requests, patches and improved versions on MailToTWikiAddOn contributed by the TWikiCommunity.
• Please let us know what you think of this extension.
• For support, check the existing questions, or ask a new support question in the Support web!
• Please report bugs below

Feedback on MailToTWikiAddOn

Headlines

27 Jan 2003 First beta (v0.03) released, available here. Topic refactored.

Requirement definition

I have a requirement to be able to easily append emails to a web, the intended application being bug triage. Rather than cut and paste, I want to be able to forward mails to a mail bot which will then insert them into a web. There has already been some work in this area (see SubmitTopicByEmail, MailInAddOn, MailInAddOnDev), but they don't fit my exact requirements.

Detailed requirements

  • It must handle Content-Type: multipart/mixed mails.
  • Attachments must be extracted from the mail and stored as TWiki attachments.
  • Emails must be rendered in a similar fashion to a mail agent.
  • The header fields to be displayed must be configurable.
  • It must runnable from the command-line, via a sendmail alias and procmail.
  • It must adhere to TWiki permissions mechanisms.
  • It must use the senders email address to locate the TWiki user id via the TWikiUsers topic.
  • Errors should as far as possible be reported to the sender as email replies.
  • It must handle locked topics gracefully.
  • It should be restricted to a single web by default.
  • By default it should not create new topics, although this may be configurable.
  • It should allow guest usage for unregistered users, if required.
  • The topic name should be specified in the email body, via a tag.
  • The insertion point of new mails in a topic should be controllable.
  • A minimal number additional modules should be required for usage.

Prerequisites (see also TWikiSystemRequirements)

  • Perl 5.6.1
  • MIME::Tools and all it's friends.

Status

27 Jan 2003

First beta released.

Bugs

  • None currently known.

Issues

  • Possible TWiki bug found - see SaveAttachmentBroken
  • Assumes Unix pathname semantics, e.g. / as the dir seperator. Therefore it probably won't work on Win32.
  • There is no TWiki API for doing attachments, the code will have to follow bin/upload and hope nothing changes.
  • The published API (TWiki::Func) assumes that it is being used from a PlugIn. This has several consequences:
    • Non-API functions need to be used (e.g. TWiki::initialise). These may be subject to change.
    • The TWiki initialisation routines assume a CGI context, and in this case there isn't one. It isn't clear from the documentation what parameters should be passed, mainlu because these functions aren't part of the published interface ;-).

Feedback

Alan, this looks like a good start. Some comments:

I renamed the topic to ...Dev since it is under construction. Please create the MailToTWikiAddOn topic once you have a package.

You can combine a script with a Plugin. Is useful to read configurations from the Plugins topic.

The latest TWikiBetaRelease (soon to be release 01-Feb-2003 production release) does have a Plugin API to manipulate topics (including permissions).

There is no API yet to upload attachments. As a workaround you could check for the topic permissions first, then "attach" the file without version control to the pub directory of the corresponding topic (create directories if needed). Read the upload script for more.

The TWiki libs and Plugin API does work with and without cgi. Read the mailnotify script, it works without cgi. You can also create a script that runs under both modes, e.g. see statistics script.

-- PeterThoeny - 23 Jan 2003

Thanks for the feedback Peter smile

When you say 'create a Plugin, I guess you mean to create the Plugin topic without the Plugin.pm, and then use TWiki::Func::getPreferencesXXX to read the preferences out of the associated topic in the script - Duh, didn't I think of that!

Here is how I'm doing the locking/authentication:

  • Lock the topic with TWiki::Func::setTopicEditLock, fail if can't lock.
  • If new topic creation is disabled, check the topic exists with TWiki::Func::topicExists.
  • Read the topic text with TWiki::Func::readTopicText, ignoring permissions.
  • Check permissions with TWiki::Func::checkAccessPermission, passing in the topic text.
  • Insert email text, add attachments.
  • Unlock the topic with TWiki::Func::setTopicEditLock.

I'm reading in the topic then checking permissions to avoid reading in the topic text twice, as TWiki::Func::checkAccessPermission needs to look at the topic text.

For doing the attachments, I was intending to follow the lead of the upload script and use TWiki::Store::saveAttachment and TWiki::Attach::UpdateAttachment, I guess this will do versioning properly anyway, so I shouldn't have to worry about it.

Is the latest beta available for download yet? I can't spot any likely looking topic manipulation functions in TWiki::Func in the version I have (30=Dec-2002). I guessed the path of the 01-Feb-2003 version, but couldn't find it.

I based what I did on mailnotify and statistics script, so I'm doing TWiki::basicInitialize(); first so I can use bits of the API to check the existence of webs, topics and users, then once I have all that info I'm calling TWiki::initialize("/$web/$topic", $username, $topic, '', undef);, leaving the url and query parameters empty.

-- AlanBurlison - 23 Jan 2003

This is excellent. IMHO this kind of adding is closely related to CommentPlugin. It handles many identical issues:

  • check page locking and save request into pool if failed
  • identify into which part of document email/comment text s/b inserted

Instead of HTML comment you proposed, you might want to add plugin tag, like %EMAILCOMMENT {params}%, which will get expanded to empty string on view, but will hold the pointer where to add emailed comments. CommentPlugin can have multiple instances and can distinguish which one was used.

I understand you have your own goals and priorities, and integrating CommentPlugin is not high on the list wink but I believe this will be the ultimate commenting scheme, good for both commenting using email, and generating form to interactive comment (without edit):

%COMMENT % tag is expanded to

  • a form in view mode, as CommentPlugin does
  • mailnotify script is expanded, so it generated correct insert-here pointers. If I want to comment via email, I'll reply to mailnotify message, delete pointers to pages I am not interested except one, and add my text.
  • both mailnotify and savecomment will generate text file into spool using common format, from which both flavors of comments are inserted (properly handling page locking) into the same place marked by COMMENT variable, and properly lockin/unlocking pages during batch commenting.

My dream coming true. And RuleOne -- rules wink

-- PeterMasiar - 23 Jan 2003

So an unrecognised plugin tag just gets ignored? Good, I'll use that instead.

I'll take a look at CommentPlugin as well - thanks for the pointer. The idea of being able to reply to mailnotify messages is interesting. It would be relatively easy to accept a single message with multiple comments, split it up and insert into multiple topics, e.g.

%{APPEND_TO_TWIKI_TOPIC FirstTopic}%

My first comment

%{APPEND_TO_TWIKI_TOPIC SecondTopic}%

My second comment

The question is, if the reply has attachments, which topic do they belong to? One of the major design goals is to allow files to be attached to topics via email - I'm going to be using it for bug triage, and I get lots of big email attachments that I need to organise and save. I'm not sure how these two things can be reconciled.

-- AlanBurlison - 24 Jan 2003

Sorry, but unrecognised plugin tag is printed out with % around and is NOT ignored. Good idea, maybe it should. If somebody knows where changes in Plugin API should be discussed, please add link and remove this blahblah. wink Thank you.

KISS - append to first page commented, and if they need to go to other page, just comment in anouther email.

Your syntax for target page doesn't have to be in TWiki Plugin syntax, because mailsorter (??) script will sort and parse it into comment pools before hitting Twiki. Humans will see it a plain text, so it s/b something nice and easy to read/understand, like [Comment to page %SITENAME. %WEBNAME. %TWIKITOPIC ]. Text (wrapping around % variables) could be even plugin parameter.

--

I created a stub plugin anyway, just so I could store configuration variables for the addon and to make it easy to recognise it was installed. I put in a simple commonTagsHandler that strips the tag when TWiki renders the topic, so you can see it on the edit view, but it doesn't show up on the screen.

I've changed the tags, the marker tag for the topic is now %MAILTOTWIKI_INSERT_HERE%, and the command you nedd to put in a mail is %{MAILTOTWIKI_APPEND_TO SomeTopic}% I think for consistency, TWiki variable syntax should be used as it will be familiar to existing users, and is unlikely to be matched by any normal email content. I deliberately haven't allowed the site or web name to be specified in the email command, as I feel this is far to open to abuse. At the moment the web name is passed on the command-line. I'm thinking about allowing web.topic to be specified in the email body, and then adding a configuration item to the Plugin page that will contain a list of webs that the plugin will accept mail for, I think that is a reasonable compromise.

I've also changed the insertion logic slightly. If you send a mail with attachments where the body consists of just a %{MAILTOTWIKI_APPEND_TO SomeTopic}% command, the 'outer' mail will be discarded. This is useful to forward other people's mails into a topic, and for attaching files to a topic via email rather than upload.

There is another more important issue I've bumped up against as well - if you are running the script via a sendmail alias, it is run as the sendmail user - who probably won't have permission to write into the TWiki directories. I've made the necessary changes so the script can run setuid. In my install I've made it setuid to the user who wons the TWiki files and it works fine, but of course setuid scripts are always a little scary. If you have a seperate TWiki Unix account you could use a procmail rule to run the mailtotwiki script, which would then run with the correct uid anyway.

-- AlanBurlison - 24 Jan 2003

I am way too swamped right now to test it, but I like it! I did not think about user authorisation of email submission before. Tricky - it's easy to send email pretending to be from somebody else. So, IMHO, we need some defenses:

  1. User should enable accepting commnents via email from his account (making herself vulnerable to email spoofing - opt in). Reject contribution via email if not enabled - with explanation how to setup user's homepage to enable it, and a link to the page with explanation of the risks.
  2. User should get confirmation of email being inserted (and inserted text) - so if anybody did spoof her address, user will be aware of it, and could clean the mess.

-- PeterMasiar - 28 Jan 2003

All good suggestions - I think adding two config vars to the users topic would be the best bet, say MAILTOTWIKI_ENABLED and MAILTOTWIKI_CONFIRM, with the defaults if not specified being off and on respectively. I think however it should also be possible to specify these on a site-wide basis, for intranet use on a big site it might be a bit onerous to go through every user topic and modify them all. How about having the defaults set in the plugin topic, but with per-user overrides?

I've made a few more useability mods, I kept mistyping the goop you had to put into a mail to trigger the processing. Rather than the TWiki syntax the latest version just uses MAILTOTWIKI_APPEND_TO "Web.Topic" (quotes are optional). I've also tweaked the mail formatting a bit. I'll release the next version when I have tested it on the new TWiki release.

-- AlanBurlison - 31 Jan 2003

Installation report: Host system: debian linux 3.0, twiki beta 03 Aug 2002.

  • installed MIME::Tools via 'apt-get install libmime-perl'
  • edited $twikiroot/bin/mailtotwiki to adjust for local settings:
    • changed #!/bin/perl to #!/user/bin/perl
    • changed twiki lib to /var/local/www/twiki/lib
  • set permissions MailToTWikiAddOnDev in to wide open (map unkown to guest, allow create topic, etc.) and the allowed web to Sandbox

When I run mailtotwiki from the commandline I get:

# ../bin/mailtotwiki < /tmp/test-email.txt
Name "TWiki::mixedAlphaNum" used only once: possible typo at ../bin/mailtotwiki line 392.
Undefined subroutine &TWiki::basicInitialize called at ../bin/mailtotwiki line 666.

-- MattWilkie - 29 Jan 2003

UPDATE: upgrading to TWikiReleases 2003-Jan-20 fixed it. It works like a charm now (from the commandline anwyway. I still have to figure out how to set the server up to accept mail). Thanks a lot Alan!

-- MattWilkie - 30 Jan 2003

You are welcome smile I have the /etc/aliases setup explained in the documentation working on my machine, as long as you are prepared to make the mailtotwiki script setuid to the web user, it works fine.

-- AlanBurlison - 31 Jan 2003

Our organization is interested in providing email interface to Twiki. In this regards we are trying out MailToTwiki plug-in. We found the plug-in very usefull . We encountered a peculiar problem due to the way in which the users have configured their email clients.

Problem Most of the users configure their email clients to send "multipart/alternative" MIME mails with text/plain and text/html body parts (many webmail clients like Yahoo also do the same). In this case the plug-in was giving the follwing error .

Can't call method "path" on an undefined value at mailtotwiki line 397, line 129.

The plug-in encounters "multipart/alternative" entity and tries to get path for the bodyHandle of the same. As bodyHandle is not defined for "multipart/alternative" entity, we see the error.

We wanted to handle "multipart/alternative" mails. In case of such mails we wanted to skip the "text/html" part and append "text/plain" part to the topic (otherwise the "text/html" part will get attached to the topic). I have done the follwoing modifications to the plug-in to achieve the same.

Changes

  • Instead of using MIME::Entity.parts_DFS(), I wrote a function (getParts()) which will give all parts, except if it finds "multipart/alternative" it should give only one of the alternative (preferably the one with type 'text/plain').
  • Skip "multipart/alternative" in the 'IF' structure for mime goof just like it is done for "multipart/mixed" and "message/rfc822".

The modified file is attached below as 'mailtotwiki1'. To locate my changes search for '### Changed'.

-- RakeshJain - 21 Feb 2003

  • mailtotwiki1: Changed code to incorporate multipart/alternative

Can someone please attach an example of a mail file that works?

I get:

Can't call method "path" on an undefined value at bin/mailtotwiki line 395, <STDIN> line 177.

When I run it.

Thanks, M.

-- MartinCleaver - 05 Apr 2003

Martin,

The problem is that u r sending a "multipart/alternative" type of mail, and the script cant find a body for that part. I am attaching two files for the same mail, with and without multipart/alternative. Alan's script will fail in the latter case.

I have incorporated the changes needed for handling such mails. Replace the mailtotwiki script by mailtotwiki1 that I have attached to this topic. (Read my previous comment for an elaborate description)

-- RakeshJain - 15th April

Has anyone looked into why the error message quoted by MartinCleaver above causes all subsequent text to appear in bold face? I'm afraid that is beyond my current knowledge of the TWiki code.

-- DavidBright - 15 Apr 2003

The embedded set of angle brackets had something to do with it. I replaced the *'s with the verbatim tag and the problem went away. It also goes away if you use &lt; instead of <.

Logged as a bug at ImproperRendering

-- MattWilkie - 18 Apr 2003

Many thanks RakeshJain, I hope to have a chance to install this in the next few days. Regards. M.

-- MartinCleaver - 28 Apr 2003

Is there a reason, why this requires Perl 5.6? I cannot upgrade the platform and wonder how much effort it would be, to downgrade to 5.005?

-- PeterKlausner - 02 Jul 2003

The parent page to this implies that this is installed on TWiki.org - is this really the case? If so where/how/when (why?) smile I suspect it's a misclassification though wink

-- MichaelSparks - 13 Jul 2003

Allthough the parent page to this says that this add-on works under unix/linux only, I would like to have it working within my Windows/cygwin based installation (it's a request from a project team which is going to use TWiki as it's communication platform). Is there anybody who has integrated this add-on into a Windows/cygwin environment or can someone give any hints what I have to do to achieve the mail-connection?

My current status is as follows: the manual execution of the script bin/mailtotwiki with a mail saved as "text" from MS Outlook works as expected. But how can I obtain the correct mail format (multipart/mime) and feed it into the script? Is there a chance to find a suitable mail client running within cygwin?

-- MichaelSchmidt - 28 Feb 2005

At first blush, it seems that http://www.syntergy.com/sharepoint/products/current/livemail/index.html provides similar functionality for SharePoint.

-- MartinCleaver - 28 Feb 2005

That seems to be so. Perhaps we will consider using SharePoint services later and in addition. However, for now I would prefer to use a simple and independent solution, which simply transforms the content of a mail into a TWiki page.

-- MichaelSchmidt - 01 Mar 2005

smile What I'd neglected to say was that we ought to ensure we talk about the features and benefits of our solution in as much detail as they do.

-- MartinCleaver - 01 Mar 2005

After some searching and experimenting I have successfully installed the MailToTWikiAddOn within our Windows environment. I used a slightly modified version from RakeshJain and provided it as version 0.05. This version is able to handle all mail formats I came across in our Outlook environment.

The following restrictions apply:

  • Attachments which are not mails are handled as if they were uploaded manually. If some mails have attachments with identical filenames, these attachments are versioned and all links on the page refer to the last version of these attachments.
  • Attachments which are mails and may contain other mails, are properly expanded.
  • Recovery of spooled mail files does not work yet.

Here is the way I did it:

  1. Installed a local mail server with POP3 functionality (in our case the light version of mailenable, which is free).
  2. Installed fetchmail, procmail and cygrunsrv from cygwin.
  3. Set up the appropriate config files .fetchmailrc and .procmailrc
  4. Installed fetchmail as a cygwin service.

When the cygwin service is started, an error message appears (error 1053) complaining that the service did not respond in time. However, despite the error message, the service is running properly.

When the cygwin service is going to be stopped, a "Quit"-Signal needs to be sent to the process-id of /usr/bin/fetchmail and a "Kill"-Signal needs to be sent to the process-id of /usr/bin/cygrunsrv, since cygrunsrv -E does not work correctly (for this application).

-- MichaelSchmidt - 18 Mar 2005

As I struggled a little bit to get it working, here are the additional steps I followed. Please note that my Twiki is istalled as /home/httpd/twiki.

My configuration : Fedora Core 3 + Apache 2.0.52 + Perl 5.8.5 + Twiki 2 Sep 2004

  • Comment line in /home/httpd/twiki/bin/mailtotwiki (Yes, it works with version 5.8)
        use 5.6.1 
  • Edited line in /home/httpd/twiki/bin/mailtotwikito point to the correct lib location
        use lib qw(/home/httpd/twiki/lib);
  • Edited first line in /home/httpd/twiki/bin/mailtotwiki to remove the -wT option and replace then by -U to avoid the taint check as we run suid
        #!/usr/bin/perl -U
  • Made /home/httpd/twiki/bin/mailtotwiki suid
        chomd 7755 /home/httpd/twiki/bin/mailtotwiki
  • Created mailspool directory with the correct options. This directory belongs to the Twiki user
        mkdir /var/tmp/twiki.mailspool
        chmod 777 /var/tmp/twiki.mailspool
  • Edited /etc/aliases to add line
        twiki:  "|/home/httpd/twiki/bin/mailtotwiki"
  • Edited variables in TWiki.MailToTWikiPlugin to match the setup

-- JeanNoelSimonnet - 23 Mar 2005

This is a followup configuration related question to the notes from RakeshJain and MichaelSchmidt. We run an Exchange Server in our environment (on Win2003); We already have "well-known" mailboxes to which people post tips and "how-tos". I want to pull all this content into our Twiki environment, so its all Browser accessible (& hopefully, this will encourage people to summarize email content more often). 2 questions here:

  1. In this case, I don't think I really need a local Wiki-specific mail server (the mailenable server Michael talks about). I should be able to configure my fetchmail client to directly go pull mail from my Exchange Server on some scheduled basis, right? Any issue with this approach?
  2. Are there other variants to avoid the duplicated files/content between the Twiki site and the Email server? This plugin makes a local copy of the email content for storing in the Twiki content repository. I realize this is standard Wiki practice, but this could be an issue for very active mail forums. It may be useful to be able to just refer to the files on Exchange instead of making a local copy (I have no idea if Exchange allows this or whether it stores its content files in a readable format).

-- AshokNatesan - 26 Apr 2005

checked http://twiki.org/p/pub/Plugins/MailToTWikiAddOn/MailToTwikiPkugin_0.05.zip into CVS

-- WillNorris - 19 Jul 2005

To the add-on maintainer: This add-on does not work in TWiki 4.0 as reported in Support.MailToTWikiInDakar. Please consider upgrading this add-on so that it runs on Cairo and Dakar codebase. HandlingCairoDakarPluginDifferences has more.

-- PeterThoeny - 24 Feb 2006

We would like to have emails archived to the Twiki as threaded discussions. There is a threaded discussion plug-in. Has any thought been given to enabling this plug-in to output the topic markup in a form suitable for the threaded discussion plug-in?

-- RobertBlumen - 07 Apr 2006

You can also use MailInContrib,which works with TWiki-4 and meets all the requirements at the top of this page.

-- CrawfordCurrie - 10 Apr 2006

I'd like to take over maintanence of this plugin. I prefer the instant gratification of client push over the server pull of the MailInContrib. It looks like I've got this one up and running on TWiki-4 will much hassle (the biggest issue is taint checking on the spoolled data).

-- JadeCravy - 17 Jun 2006

Some time passed since you posted your offer. By all means, PleaseFeelFreeToModify this add-on as indicated in the package form.

-- PeterThoeny - 27 Jun 2006

This plugin looks very close to what I want, though I have two additional needs that I do not see covered. The insertion point is covered in the documentation, but what about total replacment? I want to send reports from a program into TWiki, I want the previous report replaced with the new report and I want the topic archived to create a new revision. I'm not charged by the revision number, so I don't care if it grows into the hundreds or more.

Have I missed something about the insertion, is there a way to configure the replacement of emailed content and is there a way already to archive the complete topic?

-- MikeEggleston - 25 Aug 2006

Is this plugin stable enough for Dakar? Has it been updated lately? I think it has a lot of potential!

-- MiloValenzuela - 08 Feb 2007

Hi, I get the following error when running the mailtotwiki script.

Name "TWiki::userName" used only once: possible typo at ./bin/mailtotwiki line 128. Name "TWiki::doLogTopicAttach" used only once: possible typo at ./bin/mailtotwiki line 495. Name "TWiki::uploadFilter" used only once: possible typo at ./bin/mailtotwiki line 405.

I've also noticed that there is no reference to any of these functions anywhere in the twiki code. Is there some other plugin that I need to install in order for mailtotwiki to work?

-- JosephThomas - 09 Jul 2007

Hi, when I run the following command, nothing happens at all. I get absolutely no output. Please help.

perl -I bin tools/mailincron debug=1

or

perl -I bin tools/mailincron

or

./mailincron

Get's nothing...

or

-- JosephThomas - 09 Jul 2007

This add-on is outdated and not maintained. You can try the MailInContrib.

-- PeterThoeny - 14 Jul 2007

Topic attachments
I Attachment History Action Size Date Who Comment
Unknown file formatEXT alternateMail r1 manage 1.1 K 2003-04-15 - 07:39 RakeshJain multipart/aternative type of mail.
Unknown file formatEXT mailtotwiki1 r1 manage 22.0 K 2003-02-21 - 11:47 RakeshJain Changed code to incorporate multipart/alternative
Unknown file formatEXT plainMail r1 manage 1.0 K 2003-04-15 - 07:34 RakeshJain Plain mail
Edit | Attach | Watch | Print version | History: r43 < r42 < r41 < r40 < r39 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r43 - 2007-07-14 - PeterThoeny
 
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2017 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.