Mail In to TWiki
E-mail content into TWiki topics
Introduction
This extension allows users to mail content into TWiki topics. Users send e-mail to a dedicated mailbox, TWiki grabs the e-mails in regular intervals and adds them to TWiki topics indicated in the Subject, To, CC or BCC field.
For example, your wayward child might send a mail like this from an internet cafe in Thailand:
To: twiki@mum_and_dad.org
From: gapper@desperate_son.com
Subject: Travels.DailyDiary: Cashless
*Phuket*: I've run out of money!
The message lands in your email folder at your ISP. Once an hour, a cron job runs the
mailincron script, which scans the folder. If it finds any messages that have nothing but a correctly formatted Web.TopicName in the subject line, that topic is appended to (created) with the plain text of the email. The Web must exist, though the topic will be created if necessary.
In our example, the web exists, and so does the topic, so the following text gets appended to Travels.DailyDiary:
- Subject: Cashless
- To: twiki@mum_and_dad.org
- From: gapper@desperate_son.com
Phuket: I've run out of money!
-- Prodigal Son <gapper@isp.co.uk> - 2026-02-01 - 08:43
Attachments to the mail get treated as attachments by TWiki, and attached to the target topic.
How it works
Mails is pulled out of mail folders using
POP3
,
IMAP, or anything else supported by
CPAN:Email::Folder.
A script called
mailincron is run every so often (usually by
cron or an equivalent offline job scheduler). The script trawls the mail folders you specify and grabs messages that it recognizes as being for TWiki.
Note that
mailincron will only process messages that have arrived since the last time it ran. So if there was an error adding a mail, it won't attempt to add it again even if it is still in the inbox.
Where the mail gets added
Which topic
By default the target topic for mail is specified in the
Subject: of the mail. You just put Web.Topic as the first thing on the subject line, optionally followed by a colon and other subject text. If there is no web name specified, then you can configure a default web to accept those mails.
- A mail with a
Subject: of Schedule.MyClass will be saved to Schedule.MyClass
-
Subject: Schedule.MyClass: timetable will be saved to Schedule.MyClass
-
Subject: MyClass will be saved to <default web>.MyClass.
-
Subject: MyClass: timetable will be saved to <default web>.MyClass, if the default web is set as Schedule.
- If a topic specified on the
Subject: line doesn't exist, it will be created.
You can also configure the module to examine the
To:,
CC: and
BCC: fields in the mail, and get the name of an existing topic from there instead.
- If a non-existing topic is specified in
To:, CC: or BCC: then it is ignored.
- The valid, existing topic found in the
To:, CC: or BCC: fields is taken, and overrides the Subject:.
- A mail addressed to
<MyTopic@example.com> will be saved in <default web>.MyTopic
- A mail addressed to
<Schedule.MyTopic@example.com> will be saved in Schedule.MyTopic
- A mail addressed to
"Mylene Classe" <Schedule.MyClass@example.com> will be saved in Schedule.MyTopic
- A mail addressed to
"Mylene Classe" <MyClass@example.com> will be saved in <default web>.MyTopic
- A mail addressed to
<Schedule+MyTopic@example.com> will be saved in Schedule.MyTopic
- A mail addressed to
<twiki+Schedule.MyTopic@example.com> will be saved in Schedule.MyTopic
The web name and topic name can be redefined with configure settings. For example, it is possible to store mail in sub-webs, and have auto-numbered mail topics based on the indicated topic name.
You can also define a 'spambox' for each mail folder. A spambox is a topic that will take all messages that do
not have a valid topic.
Where in the topic
Incoming mail can be added at the top or bottom of a topic, or above/below a marker in the topic. The marker can be a
CommentPlugin %COMMENT% or a
<!--MAIL--> HTML comment marker. The default location is specified by the
where configure option of
{MailInContrib} set to
'top',
'above',
'below',
'bottom', respectively.
The
<!--MAIL--> accepts parameters like a
TWiki Variable, such as
<!--MAIL{where="below"}-->. The following options are available:
-
where - can be above, below (relative to the HTML comment), or top, bottom (of the topic)
-
template set to the name of one of the templates (see below) to change the formatting of comments.
If there is more than one
%COMMENT% or
<!--MAIL--> HTML comment marker, only the first is recognized.
%COMMENT% is ignored if
<!--MAIL--> exists.
Any attachments in the mail get added to the target topic as attachments. Attachments are also listed immediately below the mail body in the topic.
How the contributor is identified
The user identity is used for access control checks on the target topic, so you can use TWiki access controls to protect target topics.
You can configure the module to look at the
From: entry in the email, and if a registered user has set that email (see
ChangeEmailAddress), then they are identified as the contributor. Note that there is a security risk here, as the
From: address in e-mail can easily be spoofed.
You can also optionally set a default user for an inbox, so if the user can't be identified from the mail, it will fall back to the default.
In what format the mail is added
The format of the mail added to the topic is defined by a template.
The templates are loaded from
MailInContribTemplate, and the recommended way to change the look and feel of the output pages is to copy MailInContribTemplate as MailInContribUserTemplate and customize it.
The format of an e-mail is defined by a three template definitions named
MAILIN:<name>,
MAILIN:<name>:IMAGE, and
MAILIN:<name>:ATTACHMENT, respectively. The first one defines the format of the mail, the second one the format of an image attachment, and the last one the format of other attachment types. The default template is called
MAILIN:default,
MAILIN:default:IMAGE, and
MAILIN:default:ATTACHMENT, respectively.
The following variables are supported in the mail format definition template:
-
%MAILSUBJECT% - mail header Subject field
-
%MAILFROM% - mail header From field
-
%MAILTO% - mail header To field
-
%MAILCC% - mail header CC field, if any
-
%MAILDATE% - mail header Date field
-
%MAILISODATE% - mail header Date field converted to ISO date, such as 2026-02-10
-
%MAILBODY% - mail body
-
%MAILSUMMARY% - summary of mail body, 500 characters
-
%MAILSUMMARY{256}% - summary of mail body, 256 characters
-
%MAILATTACHMENTS% - attachments
-
%MAILWEB% - name of web where mail is posted, can be different from %WEB% if redefined in configure
-
%MAILPREVWEB% - use in second post (post2) to refer to name of web where mail was first posted
-
%MAILTOPIC% - name of topic where mail is posted, can be different from %TOPIC% if redefined in configure
-
%MAILPREVTOPIC% - use in second post (post2) to refer to name of topic where mail was first posted
The following variables are supported in the attachment format definition templates (:IMAGE and :ATTACHMENT) :
-
%A_FILE% - file name
-
%MAILWEB% - name of web where mail is posted
-
%MAILPREVWEB% - use in second post (post2) to refer to name of web where mail was first posted
-
%MAILTOPIC% - name of topic where mail is posted
-
%MAILPREVTOPIC% - use in second post (post2) to refer to name of topic where mail was first posted
- any TWiki Variable, such as
%ATTACHURL%
Example:
%TMPL:DEF{MAILIN:default}%
* Subject: %MAILSUBJECT%
* From: %MAILFROM%
* To: %MAILTO%
* Date: %MAILDATE%
%MAILBODY%
%MAILATTACHMENTS%
-- %WIKIUSERNAME% - %SERVERTIME%
%TMPL:END%
%TMPL:DEF{MAILIN:default:ATTACHMENT}% * [[%ATTACHURL%/%A_FILE%][%A_FILE%]]
%TMPL:END%
%TMPL:DEF{MAILIN:default:IMAGE}% * <img src='%ATTACHURL%/%A_FILE%' style='max-width:800px' alt='' />
%TMPL:END%
Consult
MailInContribTemplate and
TWikiTemplates for details.
Which part of the mail is added
The module defaults to extracts the plain-text portion of the mail; the HTML portion (if present) is discarded.
When the module extracts the plain-text portion, it also discards inline images associated with the HTML (because, whilst they are part of the mail, they are not
attachments).
Experimental:
You can configure the module to extract the HTML portion instead, in which case the plain-text portion is discarded. The module then also extracts and attaches inline images referenced in the mail (
MailInContrib discards inline images if they are not referenced).
Content processing
You can also configure the module to process the mail content before adding to the target topic.
This is particularly important for html email, which could potentially contain web bugs and/or dangerous javascript. The default processing for html email tries to remove scripts, references to resources that are not part of the mail and references to resources that are not on the same server as TWiki.
See the
WARNING, below.
Available content processors
- TWiki::Contrib::MailInContrib::NoScript
Removes script tags and attributes like onclick, onload, onmouseover etc
- TWiki::Contrib::MailInContrib::NoInlineContent
Removes tags that reference content (e.g. images) contained in the message itself. Note: This does not remove attachments.
- TWiki::Contrib::MailInContrib::FilterExternalResources
Removes tags (e.g. img, script and style) tags that reference content on external servers. By default, only inline content (i.e. content attached to the message) and content that might be attached to the wiki is allowed. This processor is configurable via two options (pick one):
-
safedomains - a comma-separated list of domains which it is safe to reference, e.g. 'example.com, intranet.example.com'. Prefix a domain with *. to allow all sub-domains e.g. '*.example.com, services.local'. Do not include the http:// portion of the URL, that will be added automatically.
-
safeurlpattern - a regular expression that matches URLs that are safe to reference. That must include the http:// portion, if any.
- TWiki::Contrib::MailInContrib::CharSetFilter
Changes common iso-8859-1, windows-1252, and utf-8 characters into HTML entities, and preserves newlines as newlines using BR tag.
Creating custom content processors
Content processors must be implemented as a perl Package that sub-classes
TWiki::Contrib::MailInContrib::Filter.
Content processors must provide the
process method:
sub process {
my $this = shift;
# my ($content) = @_;
#
# You can work on $content in place by using the special perl
# variable $_[0]. These allow you to operate on $content
# as if it was passed by reference; for example:
# $_[0] =~ s/RandomDigit/int(rand(10))/ge;
}
TWiki::Contrib::MailInContrib::Filter provides the following methods:
-
$this->options - returns a reference to the options hash for this processor, from $TWiki::cfg{MailInContrib}->[...]->{content}->{processors}->[...]
-
$this->mime - returns the MIME object for the content being processed
-
$this->topLevelMime - returns the MIME object for the mail that contains this content - useful for accessing the mail headers
-
$this->box - returns a reference to the hash for this mailbox, from $TWiki::cfg{MailInContrib}->[]
-
$this->processTag($content, $filter, $handler) - Processes $content as HTML, and calls a handler on each tag that matches the filter criteria. The tag is replaced with the handler's return value. The $content is modified in-place.
-
$filter is a hash reference. Keys:
-
tag - value is a reference to an array of tag names to match on. If the array is empty, then the handler is called for all tags.
-
$handler is a reference to the callback method. It is called as $this->$handler($tagAndContent, $tagName);
- Example: To remove all
table tags: $this->processTag( $_[0], { tag => ['table'] }, sub { return ''; } );
-
$this->processAttribute($content, $filter, $handler) - Processes $content as HTML, and calls a handler on each attribute that matches the filter criteria. The attribute is replaced with the handler's return value. The $content is modified in-place.
-
$filter is a hash reference. Keys:
-
tag - value is a reference to an array of tag names to match on. If the array is empty, then the handler is called for all tags.
-
attr - value is a reference to an array of attribute names to match on. If the array is empty, then the handler is called for all attributes.
-
$handler is a reference to the callback method. It is called as $this->$handler($attributeAndQuotedValue, $tagName, $attrName, $attrValue, $quote);
- Example: To remove all
class attributes from div and span tags: $this->processAttribute( $_[0], { tag => ['div', 'span'], attr => ['class'] }, sub { return ''; } );
Sample configuration
$TWiki::cfg{MailInContrib} = [
{
folder => 'pop://example_user:password@example.com/Inbox',
onError => 'log',
onNoTopic => 'error',
onSuccess => 'log delete',
topicPath => 'to subject',
post => {
template => 'archive',
templateTopic => 'WebTopicEditTemplate',
web => '$web/Emails',
topic => '$topic',
where => 'top',
hideQuotes => 0,
attachments => 'attach, link',
},
post2 => {
template => 'summary',
where => 'below',
},
content => {
type => 'html',
processors => [
{ pkg => 'TWiki::Contrib::MailInContrib::NoScript },
{ pkg => 'TWiki::Contrib::MailInContrib::FilterExternalResources',
safedomains => '*.example.com' },
{ pkg => 'TWiki::Contrib::MailInContrib::CharSetFilter' },
],
},
},
];
Note: The format with options is documented in the Extensions section of configure.
WARNING
This software is provided in the hope that it may be useful. The authors make no warranty, implied or otherwise, about the suitability of this software for safety or security purposes.
The authors shall not in any case be liable for special, incidental, consequential, indirect or other similar damages arising from the use of this software.
If in
any doubt, do not use it.
Installation
You do not need to install anything on the browser to use this extension. These instructions are for the administrator who installs the plugin on the TWiki server.
- For an automated installation, run the configure script and follow "Find More Extensions" in the in the Extensions section.
- Or, follow these manual installation steps:
- Download the ZIP file from the extension home on twiki.org (see below).
- Unzip
MailInContrib.zip in your twiki installation directory.
- Set the ownership of the extracted directories and files to the webserver user.
- Install the dependencies (if any).
Note: The
CPAN Email modules uses many pluggable modules to support different folder types. It's impossible to cover all the options here; all we can suggest is that you try running the script from the command-line and resolve missing modules as you find them.
- Set up cron (or equivalent) jobs to run
mailincron. You must cd to the bin directory, and invoke the script from there (this is so mailincron can read setlib.cfg)
Because of the security issues involved (passwords for the mailboxes etc.) configuration uses variables set in your
LocalSite.cfg. The easiest way to set them is using
configure.
The
mailincron script takes one optional parameter,
debug, which takes a boolean value e.g.
debug=1. If you pass anything other than 0 or the empty string in
debug, the script will scan the mail folders, describe what it would have done, and exit, without modifying any folders, Wiki content, or sending any mails.
To run the script you need to set up a cron job. For example, to transfer mail into TWiki once every hour you might write:
0 * * * * cd /var/www/twiki/bin && ../tools/mailincron 2>&1 >> /var/www/twiki/data/mailincronlog.txt
You
must run the script from the bin directory. Make sure that the cron is run by a user with the permissions needed to read and write the
data directory.
Contrib Info
- Set SHORTDESCRIPTION = E-mail content into TWiki topics
| 2018-07-10: |
TWikibug:Item7841: Copyright update to 2018 |
| 2016-01-14 |
TWikibug:Item7708: Copyright update to 2016 |
| 2015-05-08 |
TWikibug:Item7490: Defuse TWiki variables with %<nop>VAR{}% |
| 2015-05-08 |
TWikibug:Item7490: Add TWiki::Contrib::MailInContrib::CharSetFilter to change iso-8859-1, windows-1252, and utf-8 characters into HTML entities, and to preserve newlines as BR-newlines |
| 2015-05-05 |
TWikibug:Item7490: Better display of image attachments |
| 2015-04-23 |
TWikibug:Item7490: Fix bug where user+TopicName@example.com did not work; change from GPL v2 to v3 |
| 2014-11-04 |
TWikibug:Item7490: Typo fix in docs |
| 2014-06-02 |
TWikibug:Item7490: Fix small bug in date format of MAILISODATE -- TWiki:Main.PeterThoeny |
| 2014-05-27 |
TWikibug:Item7490: Separate template for image attachments and regular attachments so that images can be embedded; properly filter illegal characters from file name -- TWiki:Main.PeterThoeny |
| 2014-05-26 |
TWikibug:Item7490: Add "Attachments:" header to default template; fix template bug where Plugins expanded to Main web instead of actual web; fix bug where topic template was lost if mail contains attachments -- TWiki:Main.PeterThoeny |
| 2014-05-24 |
TWikibug:Item7490: Support template topic in case mail topic does not exist (templateTopic field in configure); optionally hide quoted text in a TWISTY; better log of actions; fix time stamp to avoid race condition -- TWiki:Main.PeterThoeny |
| 2014-05-23 |
TWikibug:Item7490: Support auto-numbered topic names using '$autoinc(0001)' token in configure; add attachments flags options in configure; possible to hide attachments; possible to link only to attachment without attaching the file; rename %ATTACHMENTS% to %MAILATTACHMENTS% and keep the former as an undocumented feature; fix attachment bug where file size and file date was missing -- TWiki:Main.PeterThoeny |
| 2014-05-22 |
TWikibug:Item7490: Support %MAILISODATE%, %MAILSUMMARY{}%, %MAILWEB%, %MAILPREVWEB%, %MAILTOPIC% and %MAILPREVTOPIC% in template; rename %TEXT% to %MAILBODY% and keep the former as an undocumented feature; new 'post' and 'post2' hashes in configure with 'template', 'web', 'topic' and 'where' fields -- TWiki:Main.PeterThoeny |
| 2014-05-21 |
TWikibug:Item7490: Possible to mail above/below %COMMENT% in topic; new 'template' and 'where' options in {MailInContrib} configuration -- TWiki:Main.PeterThoeny |
| 2014-05-20 |
TWikibug:Item7490: Possible to use Web+TopicName@example.com and mailbox+Web.TopicName@examplePLEASENOSPAM.com e-mail format; support BCC field in addition to To and CC fields; allow topic names with dashes and underlines such as Bug-1234 and PID-0016_Summary -- TWiki:Main.PeterThoeny |
| 2014-05-07 |
TWikibug:Item7490: Support %MAILSUBJECT%, %MAILFROM%, %MAILTO%, %MAILCC%, %MAILDATE% in template -- TWiki:Main.PeterThoeny |
| 2014-05-06 |
TWikibug:Item7490: MailInContrib enhancements: Config options moved to Extensions section; force new revision on each submission; extract target topic from CC: field; general improvement to rules for extracting the target topic -- Crawford Currie |
| 2010-05-26 |
TWikibug:Item6433: Doc improvements -- TWiki:Main.PeterThoeny |
| 1 Aug 2008 |
Bugs:Item5450: minor doc fix |
| 8 Sep 2007 |
Bugs:Item4575 various ideas from Dev topic, tested against 4.2.0 |
| 12412 |
Fixed Bugs:Item3334 by removing dependency on FuncUsersContrib |
| 10239 |
Added support for listing attachments in-line with the mail |
| 10182 |
Fixed parser failures on multipart MIME messages |
| 10111 |
Fixed problem with spaces in attachment names |
| 9427 |
Added Email::Delete::POP3 and Email::Delete::IMAP implementations, and solved a curious problem with IMAP folders, where the MIME parser couldn't parse messages coming from them. |
| 8969 |
TWikibug:Item1665 - TWikibug:Item1666: Now configurable to accept mail only from registered users. Automatically maps the user and signs the contribution. Also configurable to parse the topic from the To: line. Also improved error reporting and recovery. |
| 7205 |
TWikibug:Item788 - updated CPAN dependencies for MailInContrib |
| 7196 |
TWikibug:Item784 - Added POP3 support for delete. Don't understand why it was working before; though I suspect it must have been using MailDir |
| 6993 |
TWikibug:Item143 - Cairo compatibility in MailInContrib |
| 5980 |
TWikibug:Item181 - added dependendcy on Error for Cairo installs; split contributions using a blank line to stop them crushing together |
| 5961 |
TWikibug:Item181 - minor formatting issue fixed |
| 5960 |
TWikibug:Item181 - new version, supports DEVELOP and Cairo, and adds support for attachments (you can now mail attachments to topics) |
| 2005-07-31 |
1.001 Back-ported to Cairo, added spambox (work generously supported by the Evolved Media) |
| 2005-03-10 |
1.000 Initial version |
,
TWikiPreferences,
TWikiPreferences
md5 
tgz 
Site map
Blog web
Index
Search
Search extensions
Search by tags
Changes
Notifications
RSS Feed
Statistics
Preferences
Community 
Readme first!
Rate extensions!
Add-Ons 
Development topics
Idea topics
Plugins 
Development topics
Idea topics
Skins 
Development topics
Idea topics
Code Contribs 
Development topics
Idea topics
Account 
Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.