Tags:
automation1Add my vote for this tag email2Add my vote for this tag import2Add my vote for this tag create new tag
, view all tags

Mail In to TWiki

E-mail content into TWiki topics

Introduction

This extension allows users to mail content into TWiki topics. Users send e-mail to a dedicated mailbox, TWiki grabs the e-mails in regular intervals and adds them to TWiki topics indicated in the Subject, To, CC or BCC field.

For example, your wayward child might send a mail like this from an internet cafe in Thailand:

To: twiki@mum_and_dad.org
From: gapper@desperate_son.com
Subject: Travels.DailyDiary: Cashless

*Phuket*: I've run out of money!

The message lands in your email folder at your ISP. Once an hour, a cron job runs the mailincron script, which scans the folder. If it finds any messages that have nothing but a correctly formatted Web.TopicName in the subject line, that topic is appended to (created) with the plain text of the email. The Web must exist, though the topic will be created if necessary.

In our example, the web exists, and so does the topic, so the following text gets appended to Travels.DailyDiary:

  • Subject: Cashless
  • To: twiki@mum_and_dad.org
  • From: gapper@desperate_son.com

Phuket: I've run out of money!

-- Prodigal Son <gapper@isp.co.uk> - 2015-04-01 - 08:43

Attachments to the mail get treated as attachments by TWiki, and attached to the target topic.

How it works

Mails is pulled out of mail folders using POP3, IMAP, or anything else supported by CPAN:Email::Folder.

A script called mailincron is run every so often (usually by cron or an equivalent offline job scheduler). The script trawls the mail folders you specify and grabs messages that it recognizes as being for TWiki.

Note that mailincron will only process messages that have arrived since the last time it ran. So if there was an error adding a mail, it won't attempt to add it again even if it is still in the inbox.

Where the mail gets added

Which topic

By default the target topic for mail is specified in the Subject: of the mail. You just put Web.Topic as the first thing on the subject line, optionally followed by a colon and other subject text. If there is no web name specified, then you can configure a default web to accept those mails.

  • A mail with a Subject: of Schedule.MyClass will be saved to Schedule.MyClass
  • Subject: Schedule.MyClass: timetable will be saved to Schedule.MyClass
  • Subject: MyClass will be saved to <default web>.MyClass.
  • Subject: MyClass: timetable will be saved to <default web>.MyClass, if the default web is set as Schedule.
  • If a topic specified on the Subject: line doesn't exist, it will be created.

You can also configure the module to examine the To:, CC: and BCC: fields in the mail, and get the name of an existing topic from there instead.

  • If a non-existing topic is specified in To:, CC: or BCC: then it is ignored.
  • The valid, existing topic found in the To:, CC: or BCC: fields is taken, and overrides the Subject:.
  • A mail addressed to <MyTopic@example.com> will be saved in <default web>.MyTopic
  • A mail addressed to <Schedule.MyTopic@example.com> will be saved in Schedule.MyTopic
  • A mail addressed to "Mylene Classe" <Schedule.MyClass@example.com> will be saved in Schedule.MyTopic
  • A mail addressed to "Mylene Classe" <MyClass@example.com> will be saved in <default web>.MyTopic
  • A mail addressed to <Schedule+MyTopic@example.com> will be saved in Schedule.MyTopic
  • A mail addressed to <twiki+Schedule.MyTopic@example.com> will be saved in Schedule.MyTopic

The web name and topic name can be redefined with configure settings. For example, it is possible to store mail in sub-webs, and have auto-numbered mail topics based on the indicated topic name.

You can also define a 'spambox' for each mail folder. A spambox is a topic that will take all messages that do not have a valid topic.

Where in the topic

Incoming mail can be added at the top or bottom of a topic, or above/below a marker in the topic. The marker can be a CommentPlugin %COMMENT% or a <!--MAIL--> HTML comment marker. The default location is specified by the where configure option of {MailInContrib} set to 'top', 'above', 'below', 'bottom', respectively.

The <!--MAIL--> accepts parameters like a TWiki Variable, such as <!--MAIL{where="below"}-->. The following options are available:

  • where - can be above, below (relative to the HTML comment), or top, bottom (of the topic)
  • template set to the name of one of the templates (see below) to change the formatting of comments.

If there is more than one %COMMENT% or <!--MAIL--> HTML comment marker, only the first is recognized. %COMMENT% is ignored if <!--MAIL--> exists.

Any attachments in the mail get added to the target topic as attachments. Attachments are also listed immediately below the mail body in the topic.

How the contributor is identified

The user identity is used for access control checks on the target topic, so you can use TWiki access controls to protect target topics.

You can configure the module to look at the From: entry in the email, and if a registered user has set that email (see ChangeEmailAddress), then they are identified as the contributor. Note that there is a security risk here, as the From: address in e-mail can easily be spoofed.

You can also optionally set a default user for an inbox, so if the user can't be identified from the mail, it will fall back to the default.

In what format the mail is added

The format of the mail added to the topic is defined by a template.

The templates are loaded from MailInContribTemplate, and the recommended way to change the look and feel of the output pages is to copy MailInContribTemplate as MailInContribUserTemplate and customize it.

The format of an e-mail is defined by a three template definitions named MAILIN:<name>, MAILIN:<name>:IMAGE, and MAILIN:<name>:ATTACHMENT, respectively. The first one defines the format of the mail, the second one the format of an image attachment, and the last one the format of other attachment types. The default template is called MAILIN:default, MAILIN:default:IMAGE, and MAILIN:default:ATTACHMENT, respectively.

The following variables are supported in the mail format definition template:

  • %MAILSUBJECT% - mail header Subject field
  • %MAILFROM% - mail header From field
  • %MAILTO% - mail header To field
  • %MAILCC% - mail header CC field, if any
  • %MAILDATE% - mail header Date field
  • %MAILISODATE% - mail header Date field converted to ISO date, such as 2015-04-28
  • %MAILBODY% - mail body
  • %MAILSUMMARY% - summary of mail body, 500 characters
  • %MAILSUMMARY{256}% - summary of mail body, 256 characters
  • %MAILATTACHMENTS% - attachments
  • %MAILWEB% - name of web where mail is posted, can be different from %WEB% if redefined in configure
  • %MAILPREVWEB% - use in second post (post2) to refer to name of web where mail was first posted
  • %MAILTOPIC% - name of topic where mail is posted, can be different from %TOPIC% if redefined in configure
  • %MAILPREVTOPIC% - use in second post (post2) to refer to name of topic where mail was first posted

The following variables are supported in the attachment format definition templates (:IMAGE and :ATTACHMENT) :

  • %A_FILE% - file name
  • %MAILWEB% - name of web where mail is posted
  • %MAILPREVWEB% - use in second post (post2) to refer to name of web where mail was first posted
  • %MAILTOPIC% - name of topic where mail is posted
  • %MAILPREVTOPIC% - use in second post (post2) to refer to name of topic where mail was first posted
  • any TWiki Variable, such as %ATTACHURL%

Example:

%TMPL:DEF{MAILIN:default}%
   * Subject: %MAILSUBJECT%
   * From: %MAILFROM%
   * To: %MAILTO%
   * Date: %MAILDATE%

%MAILBODY%
%MAILATTACHMENTS%
-- %WIKIUSERNAME% - %SERVERTIME%
%TMPL:END%

%TMPL:DEF{MAILIN:default:ATTACHMENT}%   * [[%ATTACHURL%/%A_FILE%][%A_FILE%]]
%TMPL:END%

%TMPL:DEF{MAILIN:default:IMAGE}%   * <img src='%ATTACHURL%/%A_FILE%' style='max-width:800px' alt='' />
%TMPL:END%

Consult MailInContribTemplate and TWikiTemplates for details.

Which part of the mail is added

The module defaults to extracts the plain-text portion of the mail; the HTML portion (if present) is discarded. When the module extracts the plain-text portion, it also discards inline images associated with the HTML (because, whilst they are part of the mail, they are not attachments).

ALERT! Experimental: You can configure the module to extract the HTML portion instead, in which case the plain-text portion is discarded. The module then also extracts and attaches inline images referenced in the mail (MailInContrib discards inline images if they are not referenced).

Content processing

You can also configure the module to process the mail content before adding to the target topic.

This is particularly important for html email, which could potentially contain web bugs and/or dangerous javascript. The default processing for html email tries to remove scripts, references to resources that are not part of the mail and references to resources that are not on the same server as TWiki. See the WARNING, below.

Available content processors

  • TWiki::Contrib::MailInContrib::NoScript
    Removes script tags and attributes like onclick, onload, onmouseover etc

  • TWiki::Contrib::MailInContrib::NoInlineContent
    Removes tags that reference content (e.g. images) contained in the message itself. Note: This does not remove attachments.

  • TWiki::Contrib::MailInContrib::FilterExternalResources
    Removes tags (e.g. img, script and style) tags that reference content on external servers. By default, only inline content (i.e. content attached to the message) and content that might be attached to the wiki is allowed. This processor is configurable via two options (pick one):
    • safedomains - a comma-separated list of domains which it is safe to reference, e.g. 'example.com, intranet.example.com'. Prefix a domain with *. to allow all sub-domains e.g. '*.example.com, services.local'. Do not include the http:// portion of the URL, that will be added automatically.
    • safeurlpattern - a regular expression that matches URLs that are safe to reference. That must include the http:// portion, if any.

Creating custom content processors

Content processors must be implemented as a perl Package that sub-classes TWiki::Contrib::MailInContrib::Filter. Content processors must provide the process method:

sub process {
    my $this = shift;
    # my ($content) = @_;
    #
    # You can work on $content in place by using the special perl
    # variable $_[0]. These allow you to operate on $content
    # as if it was passed by reference; for example:
    # $_[0] =~ s/RandomDigit/int(rand(10))/ge;
}

TWiki::Contrib::MailInContrib::Filter provides the following methods:

  • $this->options - returns a reference to the options hash for this processor, from $TWiki::cfg{MailInContrib}->[...]->{content}->{processors}->[...]

  • $this->mime - returns the MIME object for the content being processed

  • $this->topLevelMime - returns the MIME object for the mail that contains this content - useful for accessing the mail headers

  • $this->box - returns a reference to the hash for this mailbox, from $TWiki::cfg{MailInContrib}->[]

  • $this->processTag($content, $filter, $handler) - Processes $content as HTML, and calls a handler on each tag that matches the filter criteria. The tag is replaced with the handler's return value. The $content is modified in-place.
    • $filter is a hash reference. Keys:
      • tag - value is a reference to an array of tag names to match on. If the array is empty, then the handler is called for all tags.
    • $handler is a reference to the callback method. It is called as $this->$handler($tagAndContent, $tagName);
    • Example: To remove all table tags: $this->processTag( $_[0], { tag => ['table'] }, sub { return ''; } );

  • $this->processAttribute($content, $filter, $handler) - Processes $content as HTML, and calls a handler on each attribute that matches the filter criteria. The attribute is replaced with the handler's return value. The $content is modified in-place.
    • $filter is a hash reference. Keys:
      • tag - value is a reference to an array of tag names to match on. If the array is empty, then the handler is called for all tags.
      • attr - value is a reference to an array of attribute names to match on. If the array is empty, then the handler is called for all attributes.
    • $handler is a reference to the callback method. It is called as $this->$handler($attributeAndQuotedValue, $tagName, $attrName, $attrValue, $quote);
    • Example: To remove all class attributes from div and span tags: $this->processAttribute( $_[0], { tag => ['div', 'span'], attr => ['class'] }, sub { return ''; } );

Sample configuration

$TWiki::cfg{MailInContrib} = [
 {
   folder => 'pop://example_user:password@example.com/Inbox',
   onError => 'log',
   onNoTopic => 'error',
   onSuccess => 'log delete',
   topicPath => 'to subject',
   post => {
       template => 'archive',
       templateTopic => 'WebTopicEditTemplate',
       web => '$web/Emails',
       topic => '$topic',
       where => 'top',
       hideQuotes => 0,
       attachments => 'attach, link',
   },
   post2 => {
       template => 'summary',
       where => 'below',
   },
   content => {
     type => 'html',
     processors => [
       { pkg => 'TWiki::Contrib::MailInContrib::NoScript },
       { pkg => 'TWiki::Contrib::MailInContrib::FilterExternalResources', 
         safedomains => '*.example.com' },
     ],
   },
 },
];

Note: The format with options is documented in the Extensions section of configure.

WARNING

This software is provided in the hope that it may be useful. The authors make no warranty, implied or otherwise, about the suitability of this software for safety or security purposes.

The authors shall not in any case be liable for special, incidental, consequential, indirect or other similar damages arising from the use of this software.

If in any doubt, do not use it.

Installation

You do not need to install anything on the browser to use this extension. These instructions are for the administrator who installs the plugin on the TWiki server.

  • For an automated installation, run the configure script and follow "Find More Extensions" in the in the Extensions section.

  • Or, follow these manual installation steps:
    • Download the ZIP file from the extension home on twiki.org (see below).
    • Unzip MailInContrib.zip in your twiki installation directory.
    • Set the ownership of the extracted directories and files to the webserver user.
    • Install the dependencies (if any).

Note: The CPAN Email modules uses many pluggable modules to support different folder types. It's impossible to cover all the options here; all we can suggest is that you try running the script from the command-line and resolve missing modules as you find them.

  • Set up cron (or equivalent) jobs to run mailincron. You must cd to the bin directory, and invoke the script from there (this is so mailincron can read setlib.cfg)

Because of the security issues involved (passwords for the mailboxes etc.) configuration uses variables set in your LocalSite.cfg. The easiest way to set them is using configure.

The mailincron script takes one optional parameter, debug, which takes a boolean value e.g. debug=1. If you pass anything other than 0 or the empty string in debug, the script will scan the mail folders, describe what it would have done, and exit, without modifying any folders, Wiki content, or sending any mails.

To run the script you need to set up a cron job. For example, to transfer mail into TWiki once every hour you might write:

0 * * * * cd /var/www/twiki/bin && ../tools/mailincron 2>&1 >> /var/www/twiki/data/mailincronlog.txt
You must run the script from the bin directory. Make sure that the cron is run by a user with the permissions needed to read and write the data directory.

Contrib Info

  • Set SHORTDESCRIPTION = E-mail content into TWiki topics

Author: TWiki:Main.CrawfordCurrie (http://c-dot.co.uk), TWiki:Main.PeterThoeny
Sponsor: Evolved Media, Alba Power Quality Solutions
Copyright: © 2005-2015 TWiki:TWiki/TWikiContributor, 2008-2010 Foswiki Contributors
License: GPL v3 (GNU General Public License)
Version: 2015-04-23
2015-04-23 TWikibug:Item7490: Fix bug where user+TopicName@example.com did not work; change from GPL v2 to v3
2014-11-04 TWikibug:Item7490: Typo fix in docs
2014-06-02 TWikibug:Item7490: Fix small bug in date format of MAILISODATE -- TWiki:Main.PeterThoeny
2014-05-27 TWikibug:Item7490: Separate template for image attachments and regular attachments so that images can be embedded; properly filter illegal characters from file name -- TWiki:Main.PeterThoeny
2014-05-26 TWikibug:Item7490: Add "Attachments:" header to default template; fix template bug where Plugins expanded to Main web instead of actual web; fix bug where topic template was lost if mail contains attachments -- TWiki:Main.PeterThoeny
2014-05-24 TWikibug:Item7490: Support template topic in case mail topic does not exist (templateTopic field in configure); optionally hide quoted text in a TWISTY; better log of actions; fix time stamp to avoid race condition -- TWiki:Main.PeterThoeny
2014-05-23 TWikibug:Item7490: Support auto-numbered topic names using '$autoinc(0001)' token in configure; add attachments flags options in configure; possible to hide attachments; possible to link only to attachment without attaching the file; rename %ATTACHMENTS% to %MAILATTACHMENTS% and keep the former as an undocumented feature; fix attachment bug where file size and file date was missing -- TWiki:Main.PeterThoeny
2014-05-22 TWikibug:Item7490: Support %MAILISODATE%, %MAILSUMMARY{}%, %MAILWEB%, %MAILPREVWEB%, %MAILTOPIC% and %MAILPREVTOPIC% in template; rename %TEXT% to %MAILBODY% and keep the former as an undocumented feature; new 'post' and 'post2' hashes in configure with 'template', 'web', 'topic' and 'where' fields -- TWiki:Main.PeterThoeny
2014-05-21 TWikibug:Item7490: Possible to mail above/below %COMMENT% in topic; new 'template' and 'where' options in {MailInContrib} configuration -- TWiki:Main.PeterThoeny
2014-05-20 TWikibug:Item7490: Possible to use Web+TopicName@example.com and mailbox+Web.TopicName@examplePLEASENOSPAM.com e-mail format; support BCC field in addition to To and CC fields; allow topic names with dashes and underlines such as Bug-1234 and PID-0016_Summary -- TWiki:Main.PeterThoeny
2014-05-07 TWikibug:Item7490: Support %MAILSUBJECT%, %MAILFROM%, %MAILTO%, %MAILCC%, %MAILDATE% in template -- TWiki:Main.PeterThoeny
2014-05-06 TWikibug:Item7490: MailInContrib enhancements: Config options moved to Extensions section; force new revision on each submission; extract target topic from CC: field; general improvement to rules for extracting the target topic -- Crawford Currie
2010-05-26 TWikibug:Item6433: Doc improvements -- TWiki:Main.PeterThoeny
1 Aug 2008 Bugs:Item5450: minor doc fix
8 Sep 2007 Bugs:Item4575 various ideas from Dev topic, tested against 4.2.0
12412 Fixed Bugs:Item3334 by removing dependency on FuncUsersContrib
10239 Added support for listing attachments in-line with the mail
10182 Fixed parser failures on multipart MIME messages
10111 Fixed problem with spaces in attachment names
9427 Added Email::Delete::POP3 and Email::Delete::IMAP implementations, and solved a curious problem with IMAP folders, where the MIME parser couldn't parse messages coming from them.
8969 TWikibug:Item1665 - TWikibug:Item1666: Now configurable to accept mail only from registered users. Automatically maps the user and signs the contribution. Also configurable to parse the topic from the To: line. Also improved error reporting and recovery.
7205 TWikibug:Item788 - updated CPAN dependencies for MailInContrib
7196 TWikibug:Item784 - Added POP3 support for delete. Don't understand why it was working before; though I suspect it must have been using MailDir
6993 TWikibug:Item143 - Cairo compatibility in MailInContrib
5980 TWikibug:Item181 - added dependendcy on Error for Cairo installs; split contributions using a blank line to stop them crushing together
5961 TWikibug:Item181 - minor formatting issue fixed
5960 TWikibug:Item181 - new version, supports DEVELOP and Cairo, and adds support for attachments (you can now mail attachments to topics)
2005-07-31 1.001 Back-ported to Cairo, added spambox (work generously supported by the Evolved Media)
2005-03-10 1.000 Initial version
Dependencies:
NameVersionDescription
TWiki::Plugins>=1.1This module requires TWiki-4
Time::ParseDate>=2003.0211Required. Available from the CPAN:Time::ParseDate archive.
Error>=0.15Required. Available from the CPAN:Error archive.
Email::Folder>=0.84Required. Available from the CPAN:Email::Folder archive.
Email::FolderType::Net>=1.02Required. Available from the CPAN:Email::FolderType::Net archive.
Email::MIME>=1.82Required. Available from the CPAN:Email::MIME archive.
Email::Delete>=1.02Optional, recommended. Available from the CPAN:Email::Delete archive.
Home: http://TWiki.org/cgi-bin/view/Plugins/MailInContrib
Feedback: http://TWiki.org/cgi-bin/view/Plugins/MailInContribDev
Appraisal: http://TWiki.org/cgi-bin/view/Plugins/MailInContribAppraisal

Related Topics: MailInContribTemplate, TWikiPreferences, TWikiPreferences

Topic attachments
I Attachment History Action Size Date Who Comment
Unknown file formatmd5 MailInContrib.md5 r41 r40 r39 r38 r37 manage 0.2 K 2015-04-24 - 00:51 PeterThoeny  
Compressed Zip archivetgz MailInContrib.tgz r46 r45 r44 r43 r42 manage 64.0 K 2015-04-24 - 00:51 PeterThoeny  
Compressed Zip archivezip MailInContrib.zip r46 r45 r44 r43 r42 manage 71.7 K 2015-04-24 - 00:51 PeterThoeny  
Unknown file formatEXT MailInContrib_installer r7 r6 r5 r4 r3 manage 4.7 K 2015-04-24 - 00:51 PeterThoeny  
Edit | Attach | Watch | Print version | History: r44 < r43 < r42 < r41 < r40 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r44 - 2015-04-24 - PeterThoeny
 
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2015 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.