Tags:
spam1Add my vote for this tag create new tag
, view all tags

BlackListPluginDev Discussion: Page for developer collaboration, enhancement requests, patches and improved versions on BlackListPlugin contributed by the TWikiCommunity.
• Please let us know what you think of this extension.
• For support, check the existing questions, or ask a new support question in the Support web!
• Please report bugs below
• See BlackListPluginDevArchive for older discussions.

Feedback on BlackListPlugin

-- PeterThoeny - 21 Mar 2004

Possible Enhancements

  • Whitelist based on a TWiki group (in addition to IP address)
  • Scan for spam in edit, and warn user in case there is spam
  • Standalone cron script to scan for spam, sending an e-mail to admin list of web.topic - spam.site
  • Add a "How to remove topic spam and HtmlAttachmentSpam" help section in Plugin topic
  • Send e-mail to wiki admin on suspicious activity, such as adding 10 or more http links to a topic
  • Add new BANEMAIL list to prevent registration by disposable e-mail addresses, such as those from http://dodgeit.com/ or http://oneoffmail.com/
  • Bugs:BlackListPlugin

Releases

New Plugin version is released and installed at TWiki.org:

  • Added a BANLIST that gets updated automatically
  • Added form to add/remove IP addresses from the BANLIST
  • Added WHITELIST to exclude IP addresses from scrutiny
  • Added configurations for fine tuning

With the current configuration anyone is allowed to view topics once every 2 seconds, and save topics once every 10 seconds. This is in average, measured over a period of 5 minutes. We can fine tune the numbers over time.

Let me know if you want to add your IP address to the WHITELIST.

-- PeterThoeny - 04 Apr 2004

New Plugin version is released and installed at TWiki.org:

  • Fixed bug where the event log file got corrupted by infrequent cleanups

All users of the 04 Apr 2004 version should upgrade to the latest Plugin version.

-- PeterThoeny - 05 Apr 2004

New Plugin version with support for the rel="nofollow" parameter, proposed in SpamDefeatingViaNofollowAttribute. The latest Plugin version is installed on TWiki.org. Any topic that is one day old will have the "nofollow" parameter added to external URLs. Look at the HTML source of a recently edited topic / older topic to see.

-- PeterThoeny - 22 Jan 2005

New Plugin version is released and installed on TWiki.org:

  • Added multiple IP add/remove feature contributed by MichaelDaum (thanks Michael!)
  • Added a spam filtering on topic save, based on public and local regex list.

This version is supposed to work on Dakar, but it does not due to some rendering incompatibilities in Dakar. Follow-up in Bugs:Item797

-- PeterThoeny - 29 Oct 2005

New Plugin version is released and installed on TWiki.org:

  • Dakar release compatibility by working around Dakar preferences bug Bugs:Item797

-- PeterThoeny - 30 Oct 2005

New Plugin version released and installed on TWiki.org:

  • Added registration protection with magic number

-- PeterThoeny - 05 Nov 2005

New Plugin version released and installed on TWiki.org:

  • Doc fixes; code warning fixes; allow empty local SPAMLIST and public spam list

-- PeterThoeny - 08 Nov 2005

New Plugin version posted in BlackListPlugin topic:

  • Filter lines with space from spam list
  • Fixed bug that inproperly filtered HTML from spam list
  • Added Crawford's fix for end/postRenderingHandler spec change issue of Dakar Release

-- PeterThoeny - 03 Jan 2006

New Plugin release 07 Feb 2006 posted, adding TWiki Release 4.0 fix to allow registration with e-mail verification, reset password and approval. (Boud's fix is pending.)

-- PeterThoeny - 07 Feb 2006

New Plugin release 29 Apr 2006 posted, adding %BLACKLISTPLUGIN{ action="spam_show_n" }% that shows the local spam list with newline separator, useful to feed the spam list back to the merge list. Example: http://twiki.org/cgi-bin/view/TWiki/BlackListLocalSpam?skin=text&contenttype=text/plain

-- PeterThoeny - 29 Apr 2006

New Plugin release 02 Jun 2006 posted and installed on TWiki.org: Added wiki-spam filtering for HTML attachments to combat HtmlAttachmentSpam.

-- PeterThoeny - 02 Jun 2006

New Plugin release 01 Jul 2006 posted and installed on TWiki.org: Added EXCLUDELIST; scan for evil script eval in attachments; scan also .js and .css attachments; fixed writeLog error on Cairo

-- PeterThoeny - 01 Jul 2006

New Plugin release 27 Dec 2006 posted: Support for TWiki 4.1.

-- PeterThoeny - 27 Dec 2006

New Plugin release 28 Dec 2006 posted and installed on TWiki.org: Fixed bug where EXCLUDELIST pattern was removing only part of a wiki-spam pattern. (This was the cause of the '.' spam bounce.)

-- PeterThoeny - 28 Dec 2006

New Plugin release 18 Mar 2007 posted and installed on TWiki.org: Scan for evil script eval() and escape() in topic text and attachments; support for TWiki 4.2 (using new TWiki::Func::getExternalResource)

-- PeterThoeny - 18 Mar 2007

New Plugin release 29 Mar 2007, posted on TWiki.org: Doc fixes; change view=raw penalty from 20 to 5

-- PeterThoeny - 29 Mar 2007

Discussions

need to check for > in sub postRenderingHandler

i had a problem with the the version downloaded today 01 Feb 2006 on twiki version 02 Sep 2004 hotfixed $Rev: 1742 $ (i guess that's Cairo?)

sub postRenderingHandler incorrectly assumes that spammers will either be polite enough to put " " around the URL in an a href= tag, or else, at least have the decency to put a space after the >. As it happens, my "guests" were not quite so polite, so the nofollow occurred in the displayed text of the link instead of inside the tag itself. In fact, it might have a good psychological/deterrent effect to show the nofollow link to spammers, but AFAIK it should be present in the tag as well.

Here is my patch (debian directory hierarchy):

--- /usr/share/perl5/TWiki/Plugins/BlackListPlugin.pm~  2006-01-03 23:35:21.000000000 +0100
+++ /usr/share/perl5/TWiki/Plugins/BlackListPlugin.pm   2006-02-01 15:32:58.077809870 +0100
@@ -236,7 +236,9 @@
 ### my ( $text ) = @_;   # do not uncomment, use $_[0] instead

     return unless( $noFollowAge );
-    $_[0] =~ s/(<a .*?href=[\"\']?)([^\"\'\s]+[\"\']?)(\s*[a-z]*)/_handleNofollowLink( $1, $2, $3 )/geoi;
+    # $_[0] =~ s/(<a .*?href=[\"\']?)([^\"\'\s]+[\"\']?)(\s*[a-z]*)/_handleNofollowLink( $1, $2, $3 )/geoi;
+    # 01.02.2006 Boud Roukema: added  >  for URLs unprotected by " or '
+    $_[0] =~ s/(<a .*?href=[\"\']?)([^\"\'\s>]+[\"\']?)(\s*[a-z]*)/_handleNofollowLink( $1, $2, $3 )/geoi;
 }

original spam

<a href=h_ttp://www.gotspamdom.spam/banner4/?phenterspam/>phenterspam loss</a>

html sent to browser, 3 Jan version: \s]

<a href=http://www.gotspamdom.spam/banner4/?phenterspam/>phenterspam rel="nofollow" loss</a>

html sent to browser, with patch \s>]

<a href=http://www.gotspamdom.spam/banner4/?phenterspam/ rel="nofollow">phenterspam loss</a>

(BTW: This plugin looks quite good smile Congratulations to the people working on it!)

-- BoudRoukema - 01 Feb 2006

Note: This plugin current does not fully work in TWiki 4.0.0.

If you are using registration with verification code activated then the plugin's registration protection feature has to be disabled by setting Set REGEXPIRE = 0

The issue is raised on the Bugs web as Bugs:Item1586

-- KennethLavrsen - 05 Feb 2006

Posting feedback received via e-mail:

I'm fascinated by the social problems spam and revert wars have caused to public wikis):

BlackListPlugin

If you have not done so already, you might want to look at FiltersetG:

http://www.pierceive.com/

It's a curated collection of regexps for use with the mozilla/firefox extension adblock (awesome extension if you're an FF fan). It might be a little harsh as-is for use in a wiki, but it has an extensive list of regexps that might be useful as rules in your blacklist.

-- PeterThoeny - 01 Mar 2006

Thanks AndyPryke for adding:

  • Set ALLOWTOPICRENAME = %MAINWEB%.TWikiAdminGroup
to the Plugin topic.

All administrators using this Plugin: Please do the same to your BlackListPlugin to secure your settings!

-- PeterThoeny - 19 Mar 2006

Looking at the plugin topic, isn't the problem rather, that the ACL setting is commented out? CHANGE should include RENAME and hence setting the explicit RENAME right shouldn't be nescessary?

-- SteffenPoulsen - 19 Mar 2006

Plugins.BlackListPlugin is commented-out (not used for settings), TWiki.BlackListPlugin is locked down on TWiki.org.

  • And now I see it too smile - thanks for clearing this up. -- SteffenPoulsen - 19 Mar 2006

-- PeterThoeny - 19 Mar 2006

A Google search for the black list message reveals that some sites are blocking the world's favourite search engine. This may well result in those sites disappearing from google searches for which they are relevant.

I believe that GoogleBots are normally "nice" in that they leave a reasonable time gap between page requests. I don't know if this is the same for GoogleBots used for GoogleAdsense, but I'd imagine it is.

I had a quick look to see if there was a declared IP range for googlebots. There are some pages with lists of IPs, but other sites suggest that there's a wide variety of IPs.

Does anyone have suggestions to get around this problem?

P.S. I added a level 3 heading as I find it helps me navigate these long pages. Feel free to remove if you don't like it.

-- AndyPryke - 22 Mar 2006

Google is usually behaving OK, but there are times when their spiders are overly active. At times I have seen them accessing TWiki.org in less then one second intervals. I contacted Google several times asking to lower the rate of access, they asked me to create http://twiki.org/forgoogl.html. Nevertheless, I put the Google bots on the whitelist, 66.249.6, 66.249.7 so that users find the TWiki.org topics.

See Google info at http://www.google.com/support/bin/answer.py?answer=8708&topic=365

P.S. I archived older discussions to BlackListPluginDevArchive.

-- PeterThoeny - 23 Mar 2006

Tonight, while discussing the BlackListPlugin at IRC, it became clear that not everyone is aware of the very effective Apache modules that can be used in conjunction with BlackListPlugin.

One problem that BlackListPlugin doesn't solve is the prevent-one-ip-taking-up-all-apache-threads problem. This is a problem you will feel if someone is trying to DOS you or is using a download manager to retrieve one of your files. As an example, some of the pirated copies of Windows are now distributed with a default download manager that will open up hundreds of connections against your site (trying to optimize own throughput), effectively preventing any TWiki service to other users (as no apache threads will be available for servicing the view script) - and this is even with BlackListPlugin in place. Another example is a mirroring utility also taking up too many concurrent threads, leaving none for other people wanting to execute the view script.

While there are several mods to choose from for dealing with this problem, mod_cband is amongst the easiest to get running - i.e. on the Linux Debian distribution, you can simply do a aptitude install libapache2-mod-cband.

After the module is enabled in the Apache configuration, you just need to enable it for the TWiki virtual host configuration, doing something like:

<IfModule mod_cband.c>
        # Max 5Mb/s speed for this virtualhost
        # Max 10 requests per second for this virtualhost
        # Max 30 open connections for this virtualhost
        CBandSpeed 5Mb/s 10 30

        # Max 500kB/s speed, 2 requests/s and 2 open connections for any remote client
        CBandRemoteSpeed 500kb/s 2 2
</IfModule>

What is important here, is the "2 open connections for any remote client"-thing. If any one IP is trying to have more simultaneous connections than stated in the config, he will get a "503 Service Unavailable". He will not get permanently blacklisted, just the current connection is refused. You should give this number some thought, 2 open connections from one IP might be too restrictive for your environment. Do your own testing to make sure you actually achieved what you wanted after cband is installed, for instance using tools like ApacheBenchmark.

Btw: mod_cband adds a monitoring URL to your Apache installation, you probably want to protect that using normal Apache httpd.conf/.htaccess rules (i.e. allow only password protected access / access from your own known IPs).

For more info, visit cbands homepage: http://cband.linux.pl/.

As seen in the IRC discussion, SvenDowideit has hinted that something even more clever could be applied - but this might be helpful to some.

-- SteffenPoulsen - 26 Mar 2006

This is useful information, thanks for posting here. How about starting a SupplementalDocument for public TWiki site administrators?

-- PeterThoeny - 27 Mar 2006

Supplemental doc sounds like a good idea, I'll try to get such a page started asap. Just wanted to mention a few other modules that should go in, so I'm not forgetting:

-- SteffenPoulsen - 27 Mar 2006

Additional info for supplemental doc that is useful: Throtteling techniques to slow down requests by IP address

-- PeterThoeny - 27 Mar 2006

On a rare occation, the external spam file has a line with an invalid regex format. You get an error on topic save if this happens. Example message shown in the browser window:

Quantifier follows nothing in regex; marked by <-- HERE in
m/http://[\w\.\-:\@/]*?(([\w\-_.]+\.)?(l(so|os)tr)\.[a-z]{2,}|
.....|? <-- HERE bahhbahhbahhfoo\.monkey|.....)/
at /home/twiki/lib/TWiki/Plugins/BlackListPlugin.pm line 268.

With this, TWiki has a dependency on an external source. A proper fix is to get the regex fixed upstream.

Workaround until this happens:

  • Edit BlackListPlugin topic and set the SPAMLISTREFRESH to a high value, such as 6000. This will prevents a cache update for about 4 days.
  • On shell level, edit pub/twiki/BlackListPlugin/_spam_merge.txt and remove the offending line
  • On shell level, remove pub/twiki/BlackListPlugin/_spam_regex.txt to force a refresh of the regex cache
  • Restore the SPAMLISTREFRESH setting once the invalid regex has been fixed

Another fix would be to test for valid regexes. This would be very expensive performance-wise since several thousand regexes would need to be tested on topic save if there is a cache refresh.

-- PeterThoeny - 28 Apr 2006

Are all of those regex new every time? Couldn't you only check the ones that are changed/new?

-- SamHasler - 28 Apr 2006

Possibly. One would need to make a diff of the new and previous one. And keep a third list of invalid regexes. Messy.

-- PeterThoeny - 28 Apr 2006

My debug.txt file is reporting: | 29 May 2006 - 03:16 | - BlackListPlugin WARNING: Content of http://arch.thinkmo.de/cgi-bin/spam-merge is too short, using old cache

It looks like the last time there was a valid spam-merge file was May 18. - From the head of the _spam_merge.txt file...

# 5021 anti-spam patterns as of 2006-05-18

When I point my browser at that URL, I was getting an empty file. Now it just seems to hang.

-- GeorgeClark - 30 May 2006

Indeed, same thing here on TWiki.org, and spam regexes I added 12 hours ago did not get merged. I sent an e-mail to the maintainer of the spam-merge list.

-- PeterThoeny - 30 May 2006

Cannot add signatures here on TWiki.org even as a core member

You forgot to reenter white list when you updated plugin.

You forgot to set ALLOWTOPICCHANGE. THAT should be set by default. Noone will ever have BlackListPlugin installed and leave the topic open for editing.

-- KennethLavrsen - 30 May 2006

The Plugin takes only action in the %TWIKIWEB%, currently TWiki06x00. That is, it is safe to keep the setting commented out (I prefer that on the Plugin topic in the Plugins web so that people can change it.)

The Plugin installation instructions state to remove the #hash sign (write protect the topic), I prefer to keep it that way.

George: The maintainer of the spam-merge list fixed the issue, it is up-to-date again.

-- PeterThoeny - 30 May 2006

I had forgotten that the TWIKIWEB also sets where the plugin home topics live. Good.

-- KennethLavrsen - 30 May 2006

Bugs:Item2097, Bugs:Item2424, Bugs:Item2390, and LatestBlackListPluginCausesFileAttachmentToErasePageContent all point to a number of problems related to the current version of BlackListPlugin and its compatibility with different versions of TWiki.

Peter, you need to clarify this to our users and provide the needed patches for the released version of TWiki4 so people can use this plugin without danger of loosing information. Even I am confused now and do not know for sure if I should do any patching of my TWiki4.0.2

-- KennethLavrsen - 05 Jun 2006

All those issues are related to one and the same issue, TWiki core bug Bugs:Item2390.

The known issues section has this: "With TWiki 4.0.2 on some platforms, notably Solaris, attached files are uploaded with a zero file size. This is because there is a bug in how TWiki 4.0.2 handles the beforeAttachmentSaveHandler. If affected, upgrade TWiki or apply bug fix Bugs:Item2390". That bugs topic has a fix that can be applied. Any suggestions on how this can be stated more obviously?

-- PeterThoeny - 05 Jun 2006

Item2097 and Item2390 have different fixes, so it's not surprising Kenneth might be confused. In addition, the Summary of Item2390 says "beforeAttachmentSaveHandler is broken on Solaris", so if Kenneth isn't using Solaris, it's not clear that it applies to him.

-- MeredithLesly - 06 Jun 2006

Here is the fix again for reference: Change the the beforeAttachmentSaveHandler "if" section of sub saveAttachment in file twiki/lib/TWiki/Store.pm as follows:

            if( $plugins->haveHandlerFor( 'beforeAttachmentSaveHandler' )) {
                # SMELL: legacy spec of beforeAttachmentSaveHandler requires
                # a local copy of the stream. This could be a problem for
                # very big data files.
                use File::Temp;
                my $fh;
                # Note: do *not* rely on UNLINK => 1, because in a mod_perl
                # context the destructor may not be called for a *long* time.
                # Call tempfile in a list context so that file does not get
                # deleted when closed.
                ( $fh, $tmpFile ) = File::Temp::tempfile();
                binmode( $fh );
                # transfer 512KB blocks
                my $transfer;
                my $r;
                while( $r = sysread( $opts->{stream}, $transfer, 0x80000 )) {
                    syswrite( $fh, $transfer, $r );
                }
                close( $fh );
                $attrs->{tmpFilename} = $tmpFile;
                $plugins->beforeAttachmentSaveHandler( $attrs, $topic, $web );
                open( $opts->{stream}, "<$tmpFile" );
                binmode( $opts->{stream} );
            }

-- PeterThoeny - 06 Jun 2006

Thanks. So the short simple message is to apply the above fix no matter which version of TWiki4 and no matter what platform - then you should be safe. Meredith explained well what it was that was confusing.

-- KennethLavrsen - 06 Jun 2006

I just got this message in my apache log

[Fri Jun 16 16:41:57 2006] [error] [client 200.126.MyIP.191] [Fri Jun 16 16:41:57 2006] save: Can't call method "writeLog" on unblessed reference at /path/to/twiki/lib/TWiki/Plugins/BlackListPlugin.pm line 700., referer: http://my.twiki.URL/twiki/bin/edit/Web/TopicName?time=224049

A similar message was displayed in the browser, but I don't recall if it was exactly the same.

Thereafter, I received the %BLACKLISTMESSAGE and was no longer allowed to edit pages. My IP, however, was not added to the BANLIST.

To solve the problem, I added the BlackListPlugin to my TWikiPreferences#DisabledPlugins with a manual edit at the OS level. That's because I don't have a second IP to test from, but my guess is that all IPs would receive the same problem ?

With the BlackListPlugin disabled, at least I can work for now with a cron job scheduled to check for HtmlAttachmentSpam

Any idea what would cause the error message above ? I'm running CairoRelease

-- KeithHelfrich - 16 Jun 2006

It looks like for some reason the Dakar version writeLog function is called instead of the Ciaro version. In line 700, try to replace the TWiki version test from:

$TWiki::Plugins::SESSION

to:

$TWiki::Plugins::VERSION >= 1.1

Here is the procedure if you get blacklisted and and you have only one IP address: Login to your server and remove the IP address from file pub/TWiki/BlackListPlugin/_ban_list.txt

-- PeterThoeny - 16 Jun 2006

Thanks Peter,

I've made the change to line 700, removed my IP from pub/TWiki/BlackListPlugin/_ban_list.txt, and removed the BlackListPlugin from the %DISABLEDPLUGINS in TWikiPreferences.

A quick test shows I'm able to edit pages without a problem. But the curious part is that I've been running the BlackListPlugin for a week or two now without problems. I wonder why that writeLog bug would pop up now ?

-- KeithHelfrich - 17 Jun 2006

I may have locked myself out by editing so much and run into error message above upon the first time being redirected to the %BLACKLISTMESSAGE ?

-- KeithHelfrich - 17 Jun 2006

Here's a funny new idiosyncracy: I get the impression that my BlackListPlugin is not counting time. It seems errily as though I got black listed after hitting the configured score, no matter how many days it took me to get there.

When I did get locked out, I was working on a page that I wanted to save and was stuck in a loop: Remove my IP from file pub/TWiki/BlackListPlugin/_ban_list.txt, save the topic, get blacklisted immediately, and repeat.

The only escape was to edit TWikiPreferences.txt from the OS and add BlackListPlugin to the disabled plugins. My %BANLISTCONFIG is 20, 5, 1, 20, 200, 300. My OS & config is described pretty well in in JailMail and NoDiffsInJail

-- KeithHelfrich - 23 Jun 2006

Not sure what is going on. Are you using mod_perl? If so, could you try without mod_perl?

-- PeterThoeny - 27 Jun 2006

from TWiki-Dev discussion - enhanced further

You are blacklisted both when you save a topic with spam that was already there and when you score too many points. It has been proposed to not use the latter feature.

Using the point system and banning the IP address is a necessary protection.

It happens OFTEN that some idiot tries to mirror a TWiki. They start some program that follow ALL LINKS and download and download. and put a permanent load on the site for hours and hours.

The most popular can be blocked using apache config protection but there are always new "site sucking" software that comes out that we do not know or does not leave an "agent type fingerprint" we can use to protect ourselves.

There are FIVE things we can do to make the BlackListPlugin a little less pain in the butt.

  1. Anyone with a fixed IP address who is a regular developer of TWiki can go on the whitelist. Some of you that have fallen into the trap already have access to do it yourself. And others can have their IPs added as long as you are a known regular user and participate very actively in maintaining the site. If you have dynamic IP this solution will not work naturally.
  2. The banning of IP addresses is efficient in stopping a spammer or site sucker the minute he is doing something wrong. But I have one observation about spammers in general.
    • They change IP address all the time.
    • The IP address they use is never the IP that the spam points to.
    • They probably use various relays (anonymizers which change IP all the time) and compromised computers.
    • So I do not think there is any need to permanently blacklist an IP address.
    • The blacklist plugin could expire a banned IP address after maybe 2 hours or 12 hours without practical loss of any protection. We still need to ban the IP used for spamming for a period to avoid rapid experients trying to get around the spam patterns.
  3. The TRAP people fall into when saving a topic that contains a string that already contains spam added before this particular word got added to the signature file is very annoying. You can open a topic containing spam - add a "hello" - save it - and get blacklisted. The way to avoid this situation is to let the Blacklist plugin scan the topic also when you hit the EDIT button and then warn the user that the topic contains spam and which word is the spam word. Then you can either cancel out or make sure you remove the offending URL before you save. This will make the EDIT function a little slower but normal view will not be affected.
  4. I would like to see an enhancement of BlackListPlugin which can scan the entire site. It would be a script in the tools directory which scans through all topics looking for spam URLs. Each topic found with spam is listed with web.topic name and the spam pattern found. The result is emailed to the admin.
    This script can be run by cron only once per 24 hours. The admin can then remove spam that was added before spam pattern was added to the list. This will both help removing spam and prevent some of the incidents where people are blacklisted from saving a topic that somebody else added a spamming URL to. The scan cannot be run from the normal plugin code because it will take too long for the poor guy that happens to view a topic at the wrong time.
  5. The blacklist plugin could have an additional white list feature. WikiName based white list. It could look in a WhiteListGroup and never ban any members of this group. It should be allowed that this group can consist of both individuals and groups. This whitelist feature will only be used when you save topics so normal view performance should not be affected.

-- KennethLavrsen - 01 Jul 2006

Thanks Kenneth, good feedback. Just released a new version that solves the alphaworks.ibm.com issue.

-- PeterThoeny - 01 Jul 2006

Please consider adding the use strict; pragma to this plugin. Its use is important to ensuring the quality of TWiki plugins and avoiding unpleasant surprises. See UseStrict for more.

-- MeredithLesly - 02 Jul 2006

I noticed that spam entered into a comment box on twiki.org is not protected by the BlackListPlugin, even though the spam websites are in the shared list. After some investigation I found out that the CommentPlugin and BlackListPlugin use the beforeSaveHandler, but the BlackListPlugin was first in line, thus missed the text entered in a comment box. To fix this, run configure, and in {PluginsOrder} add CommentPlugin, such as: SpreadSheetPlugin, CommentPlugin

-- PeterThoeny - 03 Sep 2006

Something has changed in the spam regex file and it is blocking articles on our web that contain the word "top". That's a bit aggressive and I've ended up having to disable the spam part of the plugin. I kept banning myself even though my network is on the whitelist. It looks like the regex file was updated on September 6th, but it could have been changes anytime in the past week or two, as I don't do a lot of editing. Also, shouldn't the whitelist bypass banning of IP addresses due to the spam check?

-- GeorgeClark - 06 Sep 2006

Yes someone fell into this trap also on twiki.org today. The problematic entry in the _spam_merge.txt file is:

effects|top|pharm|pill|discount|discount-|deal|price|order|now|best|cheap|cheap- # 2006-09-01:WM

It originates from Mediawiki. I asked upstream to remove this line, it is excessive.

-- PeterThoeny - 06 Sep 2006

peter, how about the ability to ban users that post bad words in-addition to urls?

-- SteveStark - 10 Dec 2006

You mean, to scan for a list of banned words not in URLs?

-- PeterThoeny - 10 Dec 2006

yes, peter

-- SteveStark - 10 Dec 2006

I am considering this. It requires though an extension of the shared list; basically a flag on a spam signature indicating that this is for the body text, not URLs.

-- PeterThoeny - 27 Dec 2006

Someone found quite a simple and clever way of bypassing my installed version of the plugin (i think it's up-to-date) and posted about 2600 porno messages during about 13 hours before i found a simple (one-line) hack in BlackListPlugin.pm to block him/her/it. (i had to actually understand a bit about how the plugin works. smile ) A developer would probably do something more systematic (maintainable) than my one line hack, but my hack might at least motivate something better.

i didn't try playing with the rapid-fire registration parameters, because the spammer kept changing IPs and at 200 spams/hour, this is something which is at the borderline of what could reasonably happen when a new class of students joins - i.e. 3 registrations a minute, though it will at most happen for a few minutes...

Now that the spammer has been stopped, i've noticed that he/she/it used only 9 different IPs. So someone could in principle handle this "by hand" by checking the twiki log and adding all 9 IPs instead of by the hack i used. However, it's not so easy looking at the log by eye (without doing grep/awk/sed/sort/uniq/wc) to tell whether it's 9 IPs or 39 IPs. And even 9 IPs is too much to have to do by hand unless it's really a one-in-a-lifetime one-off event.

My question: should i post the details here or is there a security group i should email in private so as not to alert spammers? i will happily email details to anybody with reasonable credibility as a twiki developer or other person unlikely to be a spammer, or post it here on the twiki if the general feeling is that spammers are too stupid to read twiki development pages.

-- BoudRoukema - 16 Jan 2007

Update a few hours later: the spammer is continuing to (unsuccessfully) publish on my twiki - the one-line patch is holding fine. He/she/it has tried a bunch of new IP numbers - now the count is 19 different IPs which are now blacklisted.

-- BoudRoukema - 16:35 UTC 16 Jan 2007

Thanks Boud for the report. Could you send the suggested change by e-mail to the TWikiSecurityMailingList?

-- PeterThoeny - 16 Jan 2007

I removed the additional comment that was added to "write protect" : "See the path to this Plugin topic in the table above, and write protect this .txt file by removing all write permissions (using chmod etc)." The idea is actually to use TWiki's access control with an ALLOWTOPICHANGE setting.

-- PeterThoeny - 02 Feb 2007

I made the change to "write protect". The reason I made the change was that I got confused when I first tried to follow those instructions - they were ambiguous, so I wanted to make them clearer. It looks like I made the wrong assumption (because they were ambiguous). So if my comment is wrong, then by all means remove it, but that still leaves the original ambiguity.

So the two ambiguities were:

  1. What is the "Plugin Topic" - I (wrongly) assumed this meant "Refer to the colum of the table above called Plugin Topic to find the relevant file".
  2. How do I write protect it.

My suggestion would be "Ensure the page you are curently reading is write protected (the version on your wiki - e.g: twiki/bin/view/TWiki/BlackListPlugin). Do this by editing the page and modifying (especially uncommenting) the ALLOWTOPICHANGE setting near the top of the page."

-- EricWoods - 09 Feb 2007

Eric, thanks for the feedback. I made the doc a bit more explicit.

-- PeterThoeny - 30 Mar 2007

Just upgraded to TWiki 4.2.0 and BlackListPlugin appears to break file attachments. I get a zero byte file uploaded and binmode() on closed filehandle at lib/TWiki/Store.pm line 976 and read() on closed filehandle at lib/TWiki/Store/RcsFile.pm line 730 in the apache error log. Disabling the plugin in configure fixes uploads, but isn't very comforting frown

-- MartinRowe - 31 Jan 2008

Thanks for the report. This is tracked in TWikibug:Item5307. And it was early on reported at TWikibug:Item4611, but without a resolution. Public websites cannot upgrade to 4.2 untl this bug is fixed.

-- PeterThoeny - 04 Feb 2008

Just installed BlackListPlugin into a new TWiki Version 4.2. I ran into the following minor problem.

I could not save the topic Twiki.BlackListPlugin due to an RCS error. I checked the RCS ",v" file and found it was locked by user "nobody". My CGI scripts are not running as this user and hence failed.This lock can be removed by the following command:

 rcs -u -M twiki/data/TWiki/BlackListPlugin.txt,v

I will add this to the installation instructions. The next version of the plugin could remove this lock in the file itself.

-- AndyPryke - 15 Feb 2008

Edit | Attach | Watch | Print version | History: r121 < r120 < r119 < r118 < r117 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r121 - 2008-02-15 - AndyPryke
 
  • Learn about TWiki  
  • Download TWiki
This site is powered by the TWiki collaboration platform Powered by Perl Hosted by OICcam.com Ideas, requests, problems regarding TWiki? Send feedback. Ask community in the support forum.
Copyright © 1999-2016 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.