Simpler Mail Notification Formatting

Related topic: MozillaBrowser I had a few things I didn't like about the default mail notification, which I've addressed soewhat in a patch to templates/changes.tmpl.

Rationale

Spam Flagging

At a threshold of four, SpamAssassin flags TWiki emails as spam about 2/3 of the time. I know about whitelists, but not every user does; I also have been told 4 is too low, but the only other messages I can recall getting incorrectly getting flagged were to other mailing lists. (Only one or two spams slip through) Some users may have filters setup that either sends "spam" to a folder they check very infrequently, or outright deletes the message (foolish). I think it's better to address the problem than tell users workarounds.

I know there are different spam filtering programs out there, but the only one I've seen in the four or five mailing services I use is spamassassin. I assume these issues apply to other checking programs.

Specifically, spamassassin complains about:

• a high percentage of the message being HTML (partially addressed in patch)
• large font sizes (maybe fixable?)
• red font coloring (addressed in patch)
• using html at all (not fixable)
• a ?subject= in a mailto link (can be addressed by chaning settings on the TWikiPreferences page)
• no name in the from field (maybe fixable?)

Email Client

I primarily use mozilla-mail, which has a really nice feature to only use "simple" html. Basically this just strips out the images (wiki does this already), color formatting, and what mozilla deems excessive positioning. I can't find a spec on what exactly is done, and the html it creates is odd-looking to me, but it works well most of the time.

There is one major issue that "simple" html creates (or highlights in the twiki changes template), that each row is its own table. The only thing mozilla changes in the tables is removing the width attributes of cells and tables. This makes the email notifications look somewhat disorganized, which is due to the way the tables are made.

Smaller Emails

This wasn't really a goal of mine, but it saves approximately 70 bytes * number of updated topics - 70 bytes.

What I've Changed So Far

• Moved table code outside of the loop that prints topics, this fixes table garbling in mozilla with simple html, with no visual changes
• Created a simpler header to keep the mail more-to-the-point, showing changes and providing quick links to the notifying web's pages, this is duplicated at the bottom as the standardfooter
• Removed the "locked" stuff. I've never actually seen this work. It's responsible for the using red fonts, and extra bloat.

What could Be Fixed Elsewhere

• No from name: this would require some change in the mailnotify script of how to parse the mailnotify.tmpl file. I tried at one point but failed. I don't think the SMTP library support using a from name, but I could have missed it.

See an example

Questions or comments welcome.

-- MikeMaurer - 12 Jul 2003 Regarding these problems:

Specifically, spamassassin complains about:
• a high percentage of the message being HTML (partially addressed in patch)
• large font sizes (maybe fixable?)
• red font coloring (addressed in patch)
• using html at all (not fixable)
• a ?subject= in a mailto link (can be addressed by chaning settings on the TWikiPreferences page)
• no name in the from field (maybe fixable?)

all of these (bar the last) can be resolved by disposing of the HTML portion of the email. You can do this by changing mailnotify.tmpl appropriately. I've attached a 'minimal' non-HTML mailnotify.tmpl to this page to simplify things.

The last problem is probably due to WIKIWEBMASTER not being set. (Or perhaps your setting doesn't included an email address?) You could alternatively introduce a new variable WEBNOTIFYFROM in TWikiPreferences and use that instead. (That might be nice since it allows this to be overridden on a per web basis)

-- MichaelSparks - 12 Jul 2003

Thanks, I'm aware of the ability to change the mailnotify.tmpl file, but this wasn't what I wanted. I like the html formatting to some degree more than the text-only, but wanted simpler.

The "no name in the from field" item isn't complaining about the email address being missing, but no "Real Name", as in "Peter Thoeny <Pet er@Th oeny.com>" instead of just "Pet er@Th oeny.com".

• (Spaces added to confuse spambots MS)

-- MikeMaurer - 12 Jul 2003

It's interesting (read irritating ) that spam assassin views the format of X@YPLEASENOSPAM.Z bad... However this still boils down to just setting WIKIWEBMASTER to the "right" thing. Eg:

• • Set WIKIWEBMASTER = "Peter Thoeny" <P eter@Th oeny. com>
• (Spaces added to confuse spambots MS)

What this really signifies though is that mail notification needs to be separately skinnable.

-- MichaelSparks - 12 Jul 2003

I agree about mailnotification needing to be skinable, which it is on a web-level basis, but not a user-level basis.

I tried the setting of wikiwebmaster like that several months ago, and several variations with no luck, However, revisiting it, it looks like TWiki:Net nees to be changed to support a name:address pair. I'm not great with the finer points of perl, but it looks like a change in the grep logic on line 137 needs to change.

-- MikeMaurer - 13 Jul 2003

Odd. AFAICT that code on first glance should be correct:

• Lines 137-141 look for all lines starting with From: and copy the entire MIME header field value associated with the first From: header line into $from
• This is then used by _sendEmailByNetSMTP in this snippet here:
  • $smtp->mail( $from )
• In network terms this will be sending something along the lines of:

  HELO 192.168.0.3
  250 pingu.thwackety.com Hello zathras at 3.private.thwackety.com [192.168.0.3]
  MAIL FROM: "Michael Sparks" <zathras@thwackety.com>
  250 <zathras@thwackety.com> is syntactically correct
  

As you can see this format of mail address is syntactically valid (indeed many other highly wacky RFC822 formats are which is why TWiki is doing the right thing AFAICT in lines 137-141).

Just based on code inspection I can't see where the cause of your error is unfortunately - it's possible your mail server doesn't like what you're putting there. I'll try and replicate it instead as some point - I have to mess around with mail notification at some point anyway. (If you get chance and can replicate it could you create an appropriate bug report?)

-- MichaelSparks - 13 Jul 2003

Well, here's my error message when setting the wikiwebmaster variable as you suggest (I've gotten similar errors by not using quotes or brackets).

Checking TWiki.Sandbox
- Changed topics since 13 Jul 2003 - 01:22: MehMeh
- Sending mail notification to: mike@mikemaurer.net
* ERROR: Can't send mail using Net::SMTP. 5.0.0 <"Wiki Webmaster" <webmaster&#64;somewhere.mrs.umn.edu>... Unbalanced '<'

- End TWiki.Sandbox

This appears to be a perl issue and not even getting to the mail software, but if it helps, the server is using ssmtp to provide sendmail functionality, which is working fine for everything else.

-- MikeMaurer - 14 Jul 2003

I've done a simple patch to CVS:templates/mailnotify.tmpl - this uses a separate %WIKIWEBMASTERNAME% variable in TWikiPreferences. It would be good if people can test this, and post details of their Net::SMTP versions (provided by testenv) - should work with CPAN:Net::SMTP but the format may cause problems, in which case a more intelligent grep of the template is needed. Tested on Debian Linux using sendmail and Net::SMTP, sending to SpamAssassin 2.55 with mainly default setup (for these rules anyway) so I think the basic approach is OK.

-- RichardDonkin - 07 Aug 2003

Hmm, well, I upgraded my Net::SMTP to the latest version in CPAN (1.16), and now all the formats I was trying (one of which is in the cvs patch) work. So something changed recently in the library to be more friendly about the from line. Now, that combined with the other template changes I proposed, has brought the spamassassin score down to just 0.5, using the default settings.

Debian Woody's libnet-perl (version 1.09 which provides net::smtp) doesn't support the extended from syntax, I don't know if Debian Sarge/Sid's 1.12 does or not.

I still think the table coding should be re-worked, as no one has given a good reason why the tables are opened and closed for each entry; if there is no good reason, then it's just bloat, and affects those using limited (or intentionally crippled) html-viewing mail programs. Additionally, if the "locked" flag is never displayed, it should be removed, as it is also bloat, and triggers more spamassassin flags. Both issues can reduce generated data, with no visible changes to the user.

-- MikeMaurer - 08 Aug 2003

Thanks for testing this. Interesting about the table coding etc - will have to look at your patch a bit more. I'm off on holiday soon until near end of August, so it may take a while.

Could you post the old version number of Net::SMTP, and your Perl version? How old is Debian Woody if that's the default Perl + Net::SMTP? This will help in putting in a testenv check for that module's version.

Perhaps we should support the old module versions, i.e. a code change to grep would avoid people having to upgrade Net::SMTP, though it would be nicer to keep the TWiki code simple rather than work around module misfeatures.

-- RichardDonkin - 08 Aug 2003

Debian woody's default CPAN:Net::SMTP is version 2.19 and currently sarge/sid's version is 2.24 (which is also the current CPAN version, but it is actually part of the libnet package, in debian, and apparently CPAN). Base perl is 5.6.1 on woody, while 5.8.0 is the version on sarge/sid. Woody is a little over a year now, but is the preferred debian distribution for production-level servers. Sarge isn't en-route to becoming the "stable" release yet, so expect everyone that's only willing to use woody to be using it for at least another 6 months, as debian is very thourough at preping a stable release.

-- MikeMaurer - 09 Aug 2003

---

• changes.patch: Patch against Feb '03 TWiki Release