Importing an XML file of custom spam filters for MWES

Well, it's up and running, but I'm not sure if it's doing what it should.
User avatar
CorporateBackup
Student Sheep
Posts: 15
Joined: Fri Oct 17, 2014 2:55 pm

Importing an XML file of custom spam filters for MWES

Fri Oct 17, 2014 10:04 pm

Just a quick question - I've just set up MWES (trial account, 29 days to expiry) and when looking at the web interface under http://localhost:4044/FilterBlackListing.srv I'm having some problems trying to import an XML list of filters (http://www.wizcrafts.net/Filters.xml). I can only assume I'm doing something wrong. As it stands, I try this:

Click on the "Browse" button;
Put in either "http://www.wizcrafts.net/Filters.xml" or "c:\temp\Filters.xml" as the file name (the latter being a local copy of the former);
Click Open;
See what I just typed appear in the field immediately under the "Import filters from XML" label on the webpage;
Click the "Import All" button directly under the field;
See green text "Successfully imported xml filter file" appear next to the "Add filter" button a little further down the page; then
Reload the page and have absolutely no indication that any filters are loaded, or in use, or being pulled from a file.

Is there supposed to be an indication that the imported filters are in place?
-Steve
Corporate Backup
User avatar
nick.bolton
The Big Cheese
Posts: 1822
Joined: Thu Aug 28, 2008 4:02 pm

Re: Importing an XML file of custom spam filters for MWES

Fri Oct 17, 2014 11:54 pm

Those Wizcrafts filters will only work in MailWasher Pro, although we have adjusted some of them to work in mwes (attached). We're intending to re-work both Mailwasher Pro and MWES so they can use a common filter format, because at the moment you can write much more advanced filters in MailWasher Pro. So they're reasonably similar but MWES lacks some of the custom filter features.
Attachments
DrugsMaleEnhancement.xml
(33.04 KiB) Downloaded 146 times
ChineseAndScams.xml
(13.51 KiB) Downloaded 159 times
User avatar
CorporateBackup
Student Sheep
Posts: 15
Joined: Fri Oct 17, 2014 2:55 pm

Re: Importing an XML file of custom spam filters for MWES

Mon Oct 20, 2014 1:50 pm

Thanks for that, Nick. The DrugsMaleEnhancement.xml file imports OK. The ChineseAndScams.xml doesn't appear to import at all. I can't tell why, as MWES does not give any error, or even an indication that the import attempt was tried but failed. Relatedly, does the MWES service need to be restarted to pick up the changes, or do they instantly apply as per the next incoming email?

Relatedly, there appears to be no way to delete all the filters in an imported file in one go - each filter has to have 'Delete' manually clicked for it. Definitely an interface problem, as there could be hundreds of filters in a single XML file, and mail admins testing sets of spam filters generally like to be able to swap those sets in and out quickly.

Additionally, when loading filter files, the confirmation message does not say where the most recent filters were loaded from, and it will stick around onscreen if a filter file is attempted to be loaded but fails silently, thus giving the impression that it did load - unless the list of filters is checked manually. Altering the message to include the source file (and perhaps a timestamp) would help.
-Steve
Corporate Backup
User avatar
nick.bolton
The Big Cheese
Posts: 1822
Joined: Thu Aug 28, 2008 4:02 pm

Re: Importing an XML file of custom spam filters for MWES

Mon Oct 20, 2014 2:04 pm

That's odd, I just tried to import each file and they worked for me.

Yes, the GUI definitely needs work on these filters. Even for testing to be able to enable/disable a filter. We're working on new filters, here's a demo of where we're going with them if you're interested http://209.213.221.169:8080/
User avatar
CorporateBackup
Student Sheep
Posts: 15
Joined: Fri Oct 17, 2014 2:55 pm

Re: Importing an XML file of custom spam filters for MWES

Mon Oct 20, 2014 10:26 pm

Fair enough. In the meantime, is there a current set of best-practice MWES filters floating around? Even ones uploaded by current users?
-Steve
Corporate Backup
User avatar
nick.bolton
The Big Cheese
Posts: 1822
Joined: Thu Aug 28, 2008 4:02 pm

Re: Importing an XML file of custom spam filters for MWES

Tue Oct 21, 2014 12:33 pm

Sorry there's not, but here's some tips. They're pretty basic filters to match either plain text or regular expressions.

The regular expressions system we're using is http://www.regexlab.com/en/deelx/

All the filters are OR filters, but if you separate items in a filter by a carriage return then each item in the filter is used as AND operator
User avatar
nick.bolton
The Big Cheese
Posts: 1822
Joined: Thu Aug 28, 2008 4:02 pm

Re: Importing an XML file of custom spam filters for MWES

Tue Oct 21, 2014 2:22 pm

Actually you shouldn't really need to use the custom filters too much, they're really for exceptional cases. MWES should learn from what it blocks and should get better as time passes.

I'd add in this RBL as we've found it's quite good and accurate. Add to Filters(Black)>>RBLs

bl.score.senderscore.com as an IP type
Also quite a few users use bl.spamcop.net (IP type).
User avatar
CorporateBackup
Student Sheep
Posts: 15
Joined: Fri Oct 17, 2014 2:55 pm

Re: Importing an XML file of custom spam filters for MWES

Tue Oct 21, 2014 2:41 pm

Mmm. The problem we're running into is that the test case of MWES we've installed doesn't appear to be blocking much in the way of spam. Heuristic learning is a great tool, but until it comes up to speed, a set of predefined filters could be a useful initial starting point. Of course, it could also be that I installed it wrong...?
-Steve
Corporate Backup
User avatar
nick.bolton
The Big Cheese
Posts: 1822
Joined: Thu Aug 28, 2008 4:02 pm

Re: Importing an XML file of custom spam filters for MWES

Tue Oct 21, 2014 4:19 pm

Ok, have you got greylisting enabled? (Settings>>Action).

If you look in the headers of the spam which gets through, do you see the MWES X-headers?
User avatar
CorporateBackup
Student Sheep
Posts: 15
Joined: Fri Oct 17, 2014 2:55 pm

Re: Importing an XML file of custom spam filters for MWES

Tue Oct 21, 2014 4:45 pm

Greylisting is enabled. Interestingly, there doesn't appear to be a way to read the quarantined email from the web interface.

From the headers of one spam message:
Date: Mon, 20 Oct 2014 15:51:37 +0000
From: "MRS. JESSICA ESEH" <officefile490@yahoo.com.tw>
Reply-To: "MRS. JESSICA ESEH" <westerntransfer.office@yahoo.com>
Message-ID: <1926192686.164650.1413820297398.JavaMail.yahoo@jws11009.mail.tw1.yahoo.com>
Subject: YOUR ATTENTION IS NEEDED:
MIME-Version: 1.0
Content-Type: multipart/alternative;
boundary="----=_Part_164649_1833718912.1413820297394"
Content-Length: 6476
X-MWES-scanned: Checked by MailWasher Enterprise Server 2.8.1.4 {W}(2014-10-20 23:55:55)
X-MWES-status: Clean
X-MWES-reason: Grey Listed ok
X-MWES-sourceip: 203.188.200.183
X-MWES-smtp-from: <officefile490@yahoo.com.tw>
To: Undisclosed recipients:;
Return-Path: officefile490@yahoo.com.tw
-Steve
Corporate Backup
User avatar
nick.bolton
The Big Cheese
Posts: 1822
Joined: Thu Aug 28, 2008 4:02 pm

Re: Importing an XML file of custom spam filters for MWES

Tue Oct 21, 2014 5:17 pm

Ok, that one got through greylisting because it was sent from a compromised server.

If you go to Quarantine menu item you can search for quarantined items, or greylisted items. If you move your mouse over the subject text it will show a popup of the body text (first 200 characters).

If you go to System>>Tracking, and move the mouse over the 'Status' column emails you'll see a tooltip of why the email was let through. Yes we'll integrate all these views in together.

Are most of the delivered spam coming via greylisting?
User avatar
CorporateBackup
Student Sheep
Posts: 15
Joined: Fri Oct 17, 2014 2:55 pm

Re: Importing an XML file of custom spam filters for MWES

Tue Oct 21, 2014 5:37 pm

Here's some which got through:

Subject: Workplace Health and Safety Act <adv>
X-PHP-Originating-Script: 506:email.php
Message-ID: <2c82a61341909cda1838429ce592ec8b@ispire614.crownleadership-events.com>
Date: Sun, 19 Oct 2014 22:00:45 +0000
From: Mal Shepherd <no-reply@crownleadership-events.com>
Reply-To: <no-reply@crownleadership-events.com>
MIME-Version: 1.0
X-Mailer-LID: 30
List-Unsubscribe: <http://ispire614.crownleadership-events ... L=30&N=743>
X-Mailer-RecptId: 1709959
X-Mailer-SID: 743
X-Mailer-Sent-By: 1
Content-Type: multipart/alternative; charset="UTF-8";
boundary="b1_84e1a8b36c9a063e2489c0005551a1d1"
Content-Transfer-Encoding: 8bit
X-MWES-scanned: Checked by MailWasher Enterprise Server 2.8.1.4 {W}(2014-10-21 10:19:17)
X-MWES-status: Clean
X-MWES-reason: Grey Listed Friend
X-MWES-sourceip: 107.167.8.164
X-MWES-smtp-from: <no-reply@crownleadership-events.com>
Return-Path: bounce@crowntraining-events.com

Received: from corrientes.gov.ar (corrientes.gov.ar [10.0.0.25]) by
corrientes.gov.ar (Postfix) with ESMTP id 28ADBCE58D4; Mon, 20 Oct 2014
12:03:45 -0300 (ART)
Date: Mon, 20 Oct 2014 12:03:45 -0300
From: Mr Belo <tecnologiaeducativa@corrientes.gov.ar>
Message-ID: <2046585630.1341623.1413817425142.JavaMail.zimbra@corrientes.gov.ar>
Subject: hello
MIME-Version: 1.0
Content-Type: multipart/alternative;
boundary="----=_Part_1341622_1757002840.1413817425140"
X-Originating-IP: [10.0.0.5]
X-Mailer: Zimbra 8.0.7_GA_6021 (zclient/8.0.7_GA_6021)
Thread-Topic: hello
Thread-Index: D3ssnGfsxB/brU+ZVSiNRhLyk8LONw==
X-MWES-scanned: Checked by MailWasher Enterprise Server 2.8.1.4 {W}(2014-10-20 23:12:47)
X-MWES-status: Clean
X-MWES-reason: Grey Listed ok
X-MWES-sourceip: 181.14.206.139
X-MWES-smtp-from: <tecnologiaeducativa@corrientes.gov.ar>
To: Undisclosed recipients:;
Return-Path: tecnologiaeducativa@corrientes.gov.ar


Received: from unknown (HELO webmail.nso.go.th) ([172.19.2.20]) by
mailgw.nso.go.th with ESMTP; 20 Oct 2014 14:36:51 +0700
Received: from 41.203.67.138 by webmail.nso.go.th with HTTP;
Mon, 20 Oct 2014 14:36:51 +0700 (ICT)
Message-ID: <14310.41.203.67.138.1413790611.squirrel@webmail.nso.go.th>
Date: Mon, 20 Oct 2014 14:36:51 +0700
Subject: Treat As Immediate
From: Wang Lei <kcnburi@nso.go.th>
Reply-To: <leiwang112@qq.com>
User-Agent: SquirrelMail/1.4.13
MIME-Version: 1.0
Content-Type: text/plain; charset="tis-620"
Content-Transfer-Encoding: 8bit
X-Priority: 3 (Normal)
Importance: Normal
X-MWES-scanned: Checked by MailWasher Enterprise Server 2.8.1.4 {W}(2014-10-20 15:44:10)
X-MWES-status: Clean
X-MWES-reason: Grey Listed ok
X-MWES-sourceip: 123.242.133.195
X-MWES-smtp-from: <kcnburi@nso.go.th>
To: Undisclosed recipients:;
Return-Path: kcnburi@nso.go.th
-Steve
Corporate Backup
User avatar
nick.bolton
The Big Cheese
Posts: 1822
Joined: Thu Aug 28, 2008 4:02 pm

Re: Importing an XML file of custom spam filters for MWES

Tue Oct 21, 2014 8:39 pm

They're all getting through greylisting, which also means they're not in our database or any RBL database yet. Did you add those other RBL's?
User avatar
CorporateBackup
Student Sheep
Posts: 15
Joined: Fri Oct 17, 2014 2:55 pm

Re: Importing an XML file of custom spam filters for MWES

Tue Oct 21, 2014 8:49 pm

I've added bl.score.senderscore.com - we already had zen.spamhaus.org, bl.spamcop.net, cbl.abuseat.org, combined.njabl.org, spam.spamrats.com, b.barracudacentral.org, and multi.surbl.org.
-Steve
Corporate Backup
User avatar
nick.bolton
The Big Cheese
Posts: 1822
Joined: Thu Aug 28, 2008 4:02 pm

Re: Importing an XML file of custom spam filters for MWES

Tue Oct 21, 2014 9:33 pm

Ok, that's a lot :)

How many spam email are slipping through out of total or what % estimated?

Return to “Post Installation”