Tags:
create new tag
, view all tags
There are a variety of strategies to fight email spam.

This page refactored, deleted material moved to FightingSpamProposal for further refactoring. That page should be used for discussion of that proposal, but it is briefly summarized below. — Originally this page listed and sometimes described ways of fighting spam I found — then, a few weeks ago, I came up with what might be my own proposal for fighting spam. I tried to describe it on this page, but didn't do a real good job, and made the whole page rather confusing, so now I'm trying to refactor the page. I think the refactoring will primarily involve restoring this page to its original condition and moving the proposal to a different page (and improving the description). I may leave a brief description of "my" proposal on this page.

Aside: Disclaimers re "my": I cover these disclaimers a few times, on this page, other pages on WikiLearn, and other of my writings, either public or private (and I should move these to a "definitive" WikiLearn page to cover the subject thoroughly, once and for all, but) — to hit the high spots briefly, I make no claim that these are necessarily my original ideas (some of them might be, but that's not very important, especially at the moment), and further, if I can find some other proposal that incorporates the "critical factors" of "my" proposal, I will (probably) be happy to support that proposal instead of pushing "my" proposal.

To state my proposal in the simple terms (thanks to one of the guys on the LVLUG list for pointing out that my proposal was not very clear — I'm not at my email machine right now and don't recall his name at the moment):

  • I propose to run a program on my computer which deletes spam from my email account on my ISP before downloading it.

  • I want to encourage my ISP (and eventually all ISPs) to do the same thing (for at least readily identifiable spam) before he accepts email from his upstream SMTP relays.

  • Clearly, some more details are necessary, like what is readily identifiable spam, should there be bounce messages (yes, at least initially), and how to deal with something that might be identified as spam but that I do want to receive.

Clearly, the first part of the proposal is totally under my control, and can be done without anybody else's cooperation. The second part requires my ISPs cooperation, and then the cooperation of many other ISPs. The advantage is that it does not require any kind of top down impetus (like law, fiat, or change to Internet standards), it can be a grass roots movement.

For discussion of the details, see FightingSpamProposal.

See AboutThesePages.

Contents

Report It

  • To your ISP

  • Maybe to the FBI as well, if it contains threats

Stop Spammers' "Crawlers"

  • wpoison — Prevent spammers from "crawling" your web site for email addresses — either by setting up your own wpoison "trap" (honeypot — not quite, I guess) or linking to theirs:

From http://www.monkeys.com/wpoison/linking.html:

If you want to create a link from your own pages to our installed copy of Wpoison located here on our server, you may do so simply by incorporating the following snippet of HTML code into as many of your own web pages as you like:

<A HREF="http://www.monkeys.com/spammers-are-leeches/"> </A>

Kill Spam before Downloading

Hmm, I forgot this was here in my recent discovery and enthusiasm for SvenDeleter.

Procmail filter at your ISP

I'm going to write to my ISP and to shadie or dds or whatever, and suggest that my ISP do the same thing dds has. This is an especially good idea where ISP customers pay for connect time, but also good for anybody.

From: http://slashdot.org/article.pl?sid=02/02/15/2224229&mode=thread&tid=111, or more specifically: http://slashdot.org/comments.pl?sid=28055&cid=3016487

We (dds, a dutch isp) had a spam problem, and being a free email provider for such a long time did contribute to that. When we went out to solve this problem we did it in three steps:

- Implement RBL+ on our mailservers (got the load down a bit though)

- Created a global "spam filter" (weight system a la junkfilter) wich was opt-in for our users..

- We installed procmail, gave each user it's own .procmailrc and made a web interface to create procmail recipes in an "outlook" style.

This recipe maker could then be accessed by each user on their own user pages, or they could just make receipts through their shell access

Our end users didn't really notice much about our use of RBL. And most of them don't know what rbl is annyway.

But giving them the possibility of filtering email on the serverside themseve did make a difference! It gave them a feeling we are fighting spam, and that THEY are also in control !

And last but not least... Giving your users info on how to avoid spam is important!. We did this by writing clear faqs on avoiding spam, and pointing each new user to these faqs

Aside: I have some followup correspondence to add on this subject. The person that wrote this comment on slashdot seems concerned that his boss doesn't appreciate it, so I need to tread lightly.

A partial quote from one of his followup emails ("Re: A Contact at _ ..." dated 2/25/02 but stored 5/18/02, with some typos fixed):

Thanks for your comments. But I sadly have to dissapoint you. The piece of software our programmers have written (i'm also a sysyop at _) is not open sourced... And our boss is'n t planning on giving the source away (sigh..) I got a lot of requests on this!! But it's really not that hard to program in php. Maybe you can find someone who can.

I can tell you however how it works globally:

1: A user logs in to our member admin pages, in his homedir he has got an empty procmail (as all our users have) or has made rules allready, if so these are read, and displayed

2: On a page (php) he can add rules like:

When the from: address contains <input>spammer@hotmail.com</input> then <pulldown>deletee -send to other imap folder -forward to address</pulldown> etc etc (user chooses delete)

The rules he has submitted are then fed to a php script, which:

  1. checks for existing recipes
  2. adds the new recipe to the total
  3. reads and adds the user's own (not made by the web interface) recipes if any, and writes the whole to a new .procmailrc for the user which looks like this:

##### Do NOT edit this part below !
##### last generated www recipe: 12-1-02
:0
* ^From:.*spammer@hotmail.com
/dev/null
##### You can edit your own recipes below:

That's it actually.... I even think you can find working code around somewhere (cgi/php) that does this for you!

Please don't bother to contact annyone else at dds, because I've allready asked about this before (and my boss might think i shouldn't even have discussed this idea at /. frown

kshowmail

KShowmail is a KDE utility to show headers or the complete mails on pop3 servers without transfering them to the local mail client. Unpleasant mails can be deleted on the server. Filters can be assigned to allow automatic detection and deletion of spam. The information can be refreshed via timers.

pef

pef is software for an isp who runs a mail server on a Unix/Linux host, and who wishes to provide clients with a spam filter that ensures 100% success rate in stopping spam before it is delivered to clients' mail boxes. Using client's white and black lists of email addresses it prevents spam that has been accepted by the mail server from being delivered to the client mail box, and provides the client with a web interface for maintenance of the lists and switching the filter ON and OFF. It also provides a mechanism for senders who are on neither white nor black list to put their addresses on the recipient's white list. The mechanism requires such senders to use certain special alias of the recipient just once, i.e. something very simple for a real person, and practically impossible for software used by spammers.

SvenDeleter

SwenDeleter is a Perl script that can run on my local email server or workstation to examine email headers at my ISP (via POP3) and automatically (or manually) delete emails there (at my ISP) before downloading my mail. (It didn't work for me on Knoppix 3.2, because it needed a specific CPAN module (like ::SMTP or something) and my efforts to download the module came to nought.)

Killing Spam by Blacklisting Open Relays

Many spammers use "open relays" to pass their spam around. All mail on the Internet is passed from one MTA to another relay fashion to get to its destination. An open relay is an MTA that will accept mail from anyone and relay it. I don't know all the details, but there is no real need for open relays, even though some people have special needs and think they need open relays.

By learning to read the EnvelopeAddress of an email, you can learn to spot a potential open relay. You can then submit the IP address of an open relay to an organization like ordb.org who will check the open relay and add the IP address to a list if indeed it checks out as an open relay. Caring ISPs:

  • make sure they are not running an open relay
  • set up their MTAs to check this list and refuse to relay mail from those IP addresses

If you run an MTA on the Internet, you should make sure that you are not running an open relay.

The approach to determining an open relay from the envelope addresses on an email involves a little bit of magic, AFAICT. What I've done is look in the first three received from headers (starting from the bottom). If I see an "Unknown", I record the IP address and send it to ordb.org, in a format like Relay: 170.0.0.1. They will respond saying that the potential relay is queued for testing (the first time you send a potential relay in, they will ask you to confirm your request). Later they will do the actual testing and inform you of the results.

A good and bad thing about ordb.org is that they will retest an IP anytime it is resubmitted, so if your IP is an open relay, you can get removed from the blacklist within a few hours of fixing the problem without any human intervention. The (potential) bad thing about this is that they will remove you from the blacklist if they can't find your IP, which might allow a loophole (but maybe not).

ordb.org

From some correspondence with Pierre Fortin:

Pierre Fortin wrote: > If you have your own domain/IP, you can use PostFix' anti-spam features to greatly refuse spam before it gets delivered... Most spam comes through "open relays" and by blocking mail from any known open relay, we can virtually shutdown anonymous spam. This will eventually force the spammers to use their own resources and might make prosecution more likely; but that's orthoganal and argumentative... :^)

So, I think you're saying that if I'm running my own email server on the Internet, I can make sure it's not an open relay, thus minimizing spam for others (and myself).

However, even if I'm not running my own email server, I can still submit apparent open relay sites to ordb for testing and possible blacklisting. (Right?)

> I've noticed that since I've begun submitting spam relay hosts to http://ordb.org/submit/, spam attempts have dropped off to a trickle. In fact, the spam attempts (blocked by postfix) have dropped from ~50-70/day to a few every couple of days...

Ok, now I understand -- the number of attempts to use your machine as a relay have dropped to a few every couple of days.

> It's fun to be a spam fighter... :^) For more info, see my postfix page at http://pfortin.com/Linux/PostFix -- also, easy to miss but potentially useful for the mail-header-challenged is http://pfortin.com/Linux/PostFix/ORDBing.html

I've looked at both these pages, and the first one should be useful to me as I try to configure postfix for my (local) server.

I've reviewed the second, but I really don't understand what the key characteristic is that let's you decide which header represents the open relay. Or, do you more or less "assume" it is one of the first three and (worst case) submit the IPs from the first three received headers for testing?

If that's the case, I understand. If there is something more to look for, how about looking at the headers below and tell me which (if any) of the headers represent an open relay, and how you determined that. (Aside, I don't know that this came from an open relay, I just wanted an example we could talk about.)

regards, Randy Kramer

Return-Path: <1fp4@email1.qves.com>
Delivered-To: rhkramer@fast.net
Received: (qmail 25565 invoked from network); 1 Feb 2002 20:10:18 -0000
Received: from newmx2.fast.net ([209.92.1.32]) (envelope-sender <>)
          by mailstore1.fast.net (qmail-ldap-1.03) with QMQP
          for <>; 1 Feb 2002 20:10:18 -0000
Delivered-To: CLUSTERHOST newmx2.fast.net rhkramer@fast.net
Received: (qmail 18378 invoked from network); 1 Feb 2002 20:10:16 -0000
Received: from unknown (HELO email.qves.com) ([209.63.151.19]) (envelope-sender <1fp4@email1.qves.com>)
          by newmx2.fast.net (qmail-ldap-1.03) with SMTP
          for <rhkramer@fast.net>; 1 Feb 2002 20:10:16 -0000
Received: from qvp0002 ([209.63.151.3]) by email.qves.com with Microsoft SMTPSVC(5.0.2195.2966);
         Fri, 1 Feb 2002 13:07:31 -0700
From: "Become Wealthy!" <1fp4@email1.qves.com>
To: <rhkramer@fast.net>
Subject: Learn the Secrets of Becoming Wealthy! 
Date: Fri, 1 Feb 2002 13:07:31 -0700
Message-ID: <794d0201c1ab5c$147fdf30$0902fea9@freeyankeedom.com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
        boundary="----=_NextPart_000_794D03_01C1AB21.68210730"
X-Mailer: Microsoft CDO for Windows 2000
Thread-Index: AcGrXBR9+75qW/pGTbmEbQQCz8fEhA==
Content-Class: urn:content-classes:message
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2462.0000
Return-Path: 1fp4@email1.qves.com
X-OriginalArrivalTime: 01 Feb 2002 20:07:31.0245 (UTC) FILETIME=[148165D0:01C1AB5C]
X-Mozilla-Status: 0001
Content-Length: 3006

Killing Spam by Filtering

Tagged Message Delivery Agent (TMDA)

The way TMDA thwarts incoming junk-mail is simple yet extremely effective. You maintain a "whitelist" of trusted contacts which are allowed directly into your mailbox. Messages from unknown senders are held in a pending queue until they respond to a confirmation request sent by TMDA. Once they respond to the confirmation, their original message is deemed legitimate and is delivered to you. Updating your whitelist insures they won't have to confirm future messages. TMDA can even be configured to automatically whitelist confirmed senders.

This methodology has the advantage of being very selective about what it allows in, while at the same time permitting legitimate, but previously unknown senders to reach you.

TMDA is what I've been using for several months, and it works very well. I've recently been testing the Bayesian filtering techniques, see below. -- TomHoover - 17 Sep 2002

filtering using Bayesian techniques

spamprobe

After reading 'A Plan for Spam' (listed in Resources below), I set out to find a program that uses this technique so that I could test it. Spamprobe is the first one that I tried, and I found that it is very effective. After trying it for the past few days, my results are:

Date Total SPAMs
Rec'd Missed Caught Total SPAMs Percentage caught
9/13/02 39 14 53 26%
9/14/02 19 27 46 59%
9/15/02 4 8 12 67%
9/16/02 6 47 53 89%
9/17/02 3 28 31 90%
9/18/02 12 46 58 79%
9/19/02 6 60 66 91%

The percentage of SPAM caught increases as spamprobe "learns". Needless to say, I'm impressed that in just 5 days of "learning" spamprobe is catching 90% of the spam coming into my system. I'm anxious to see how well it's performing in a few weeks...I'll be sure to report back with the results.

-- TomHoover - 17 Sep 2002

I don't know what happened on the 18th, but we'll keep watching it. -- TomHoover - 19 Sep 2002

bogofilter

  • (rhk) SpamShield: A Perl-Based Spam Filter for sendmail; Glenn Graham; 09/12/2002 -- I'm not sure I like / agree with the approach here -- basically spamshield pays attention to how many mails have come from a source recently, and if that exceeds a threshold, it filters further emails from that source. Might be a good thing for an ISP or similar, not sure how well it would work for an individual. But, this page included the following two paragraphs, which would help to explain to somebody (like the guy on the Leo mail list, IIRC) why some mail lists and so forth filter out mail if a reverse lookup fails:

The Open Relay Server

The most common method used to send spam is through an "open" relay host. An open relay is simply an SMTP server that allows any domain to connect on port 25, and relay through to another domain. The engineers at sendmail.org have worked for several years to find ways to reject relaying, using filtering methods such as the access database.

Newer versions of sendmail do a reverse domain lookup before allowing mail to pass. If the incoming domain doesn't exist, sendmail will typically reject the message. This prevents spam from sources that use nonexistent domains in their return header.

Charging the Spammer for your Time

I've seen sigs with an approach similar to this -- things like any spam sent to me will be reviewed, there is a charge ...

Here is a more serious / workable approach, which does not depend on a law, but simply a grass roots movement. (An individual could even implement this for himself (at least for email -- the proposal addresses telephone calls and email)). The idea is basically to accept email (or phone calls) under only three conditions (IIRC):

  • From senders on a whitelist
  • From senders that have acquired a one time or multiple use token from you (by various means)
  • If they pay (or are willing to pay) a fee for your time, which you set

There is apparently a patent (discussed below) that could apply to some of these approaches, (or to an email stamp)

See:

Charge for Email

Mentioned above and elsewhere, added here just to make it more discernible.

Adopt Spam Resistant Email Systems

From a Slashdot thread, specifically proposing changes to SMTP:

Consider the following changes: Make the sender store the outgoing messages on their system until picked up, with only the header being sent as the initial message. Make sure the sender provides a real and valid reply to or other contact address (as the above system does), and maybe even tie that into geographic locations (preventing US spammers from sending through China or Argentina, or at least giving recipents an easy way to block such spam). Pass some laws against spam like we have against junk faxes (which have been very effective), don't just have those who hate spam stick their heads in the sand with filters and let the spammers continue to prey on the masses.

There are other good ideas in the thread, including this multipronged approach, and may be more as I haven't read the entire thread.

Make the Sender Responsible for Storage

Discussed above, and D. J. Bernstein's Internet Mail 2000 (proposal) is based on the concept.

Challenge Response Systems

Also mentioned are challenge response systems like tdma — if something comes in that your spam filter judges to be spam, let it send an email to the originator asking if they really want it delivered (i.e., do they exist).

SMTP Tarpit

TarProxy: a Statistically-Driven SMTP Tarpit

Resources

  • A Plan for Spam; Paul Graham; August 2002 -- using a Bayesian filter to identify spam -- "This article describes the spam-filtering techniques used in the new spamproof web-based mail reader we're building to exercise Arc."
  • Filtering Spam with LMAILER in Perl, Ricardo Ciria, April 02, 2002
  • ToDo: Check out a program named mailwasher at http://mailwasher.net -- (it's probably for windows) -- let's you check mail on your ISP before downloading it and bounce it from there, IIUC
  • STOP THE PRESSES!; Steve Outing; July 31, 2002 -- another perspective "I'm Sick and Tired Of Spam (Filters) They're Blocking Legitimate E-mail From Publishers"

Revision Comments

  • 29 Aug 2003 — Added spam resistant email systems, made charging for email more explicit
  • %DATE% —

Contributors

Page Ratings

Edit | Attach | Watch | Print version | History: r19 < r18 < r17 < r16 < r15 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r19 - 2003-10-27 - RandyKramer
 
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by PerlCopyright 1999-2017 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding WikiLearn? WebBottomBar">Send feedback
See TWiki's New Look