2019-06-18
fixed in assp 2.6.3 *SPAM-Evaporator* build 19169:
changed:
hidden parameter 'DKIMpassAction' - the default value is set to 0
2019-06-06
fixed in assp 2.6.3 *SPAM-Evaporator* build 19157:
- the file upload feature in the ASSP-filecommander has sometimes destroyed the uploaded files (+ was replaced by space)
- VirusTotalAPIKey was rejected in every case, because of a validation check bug
2019-05-31
fixed in assp 2.6.3 *SPAM-Evaporator* build 19151:
- 'fillUpImportDBDir' was not working on some systems
- a good rule '.*' in UserAttach was ignored
added:
- queries for viruses and bad URL's to
www.virustotal.com are now supported
virus checks require ASSP_AFC.pm (version 5.10)
lib/ASSP_VirusTotal_API.pm (version 1.01) and the changed ASSP_AFC.pm (version 5.11) and
'VirusTotalAPIKey','The Privat API-Key for VirusTotal'
'To query
www.VirusTotal.com for URIs and/or viruses (ASSP_AFC.pm), a valid API-Key is required. An API-Key is provided by VirusTotal for free, after your registration at
www.virustotal.com.
Such a free API-Key is limited to four queries at VirusTotal per minute. API-Keys for a higher query volume are also provided by VirusTotal.
Systems that are part of the ASSP-Global-PenalyBox network can leave this value empty. They are getting an API-Key with a much higher query volume from the GPB-Server automatically,
without any additionally costs. This API-Key is not shown here!'
'ASSP_AFCDoVirusTotalVirusScan','Enable VirusTotal Virus Scan'
'If a VirusTotalAPIKey is provided and this option is enabled, all MIME-parts will be (in addition to ClamAV and/or FileScan) checked by
www.virustotal.com.'
- DBD::MariaDB is now supported
changed:
'enhancedOriginIPDetect','Do an Enhanced Origin IP Address Detection in the Mail Header'
Local and private IP's, IP's assigned by IANA to the Shared Address Space (100.64.0.0/10 RFC6598) and IP's listed in ispip, acceptAllMail, whiteListedIPs, noProcessingIPs, noDelay and noPB
will be ignored.
'RBLServiceProvider','RBL Service Providers*'
references to combined.njabl.org are removed from the GUI
'URIBLServiceProvider','URIBL Service Providers*'
...
If VirusTotalAPIKey is configured, assp is able to query URIs on
www.virustotal.com . The API answers are in the range 127.0.0.2-127.0.0.253 (or none for OK), where the last digits represents HITS + 1.
Queries to VirusTotal are using HTTPS connections (
https://www.virustotal.com/...) instead of DNS!
example:
virustotal=>127.0.0.2=>1 # one hit
virustotal=>127.0.0.3=>0.5 # two hits
virustotal=>127.0.0.4=>0.33 # three hits
virustotal=>127.0.0.*=>0.25 # more than three hits'
2019-04-25
fixed in assp 2.6.3 *SPAM-Evaporator* build 19115:
- the post virusscan for the stored corpus files, scored for the already finished mail - this was confusing for some users and id removed
- HTML-comments are now removed from resend request emails, because there content may has affected the resend processing
2019-03-27
fixed in assp 2.6.3 *SPAM-Evaporator* build 19086:
- The ClamAV-engine now uses the modern INSTREAM clamav-API. It uses less system resources and is faster than the "old" STREAM-API.
changed:
- The default value for 'ClamAVtimeout' is changed to 30 seconds.
2019-03-26
fixed in assp 2.6.3 *SPAM-Evaporator* build 19085:
- Several domains provide their SPF-record (and possibly other DNS-records) as wildcard records (for each possible subdomain).
This caused the DKIM-preCheck to detect a (possible) provided DKIM-DNS-configuration, because it got a TXT record (the wildcard-record) for _domainkey.domain.tld and/or _adsp._domainkey.domain.tld.
Not DKIM related DNS TXT answers are now ignored by the DKIM-preCheck to prevent false positives.
2019-03-25
fixed in assp 2.6.3 *SPAM-Evaporator* build 19084:
- the resend from block report using the right button failed, if the subject of the mail contained 'x' followed by two digits (eg: x30)
- using the unix socket for the ClamAV communication failed on some systems
- assp has thrown an error if the ClamAV, configuration was anyway invalid or not working, but UseAvClamd was disabled
- the rebuildspamdb task crashed, if the HMMdb contained only one record
- ASSP_AFC.pm version 5.04 is released
ASSP_AFC.pm is now able to tell a local mail server or andvanced thread analyzer, if the attached files may need some further investigation or analysis
This is done by adding a special (hiddenly configurable) MIME header tag.
# advanced thread analyzing or deep thread inspection for incoming mails
$ASSP_AFC::enableATA = 0; # 1- check ATA if an attachment failed, 2- check if any attachment is found, 3- check every mail
$ASSP_AFC::ATAHeaderTag = "X-ASSP-Require-ATA: YES; RESENDLINK;SHOWMAIL;SHOWLOG\r\n"; # the literal RESENDLINK will be replaced by a mailto resendlink, which may be shown by an ATA report mail
# SHOWMAIL offers the link to open the file in the assp file editor
# SHOWLOG offers the link to show the log for the mail in maillogtail (an optional trailing number defines the days in the past e.g. SHOWLOG2 for example - two days is default and used if no number is given)
# every link is preceeded by \r\n\t
2019-01-19
fixed in assp 2.6.3 *SPAM-Evaporator* build 19019:
- the analyzer got changes to fully support ASSP_AFC 5.02
changed:
- ASSP_AFC 5.02 is released - it contains fixes and extensions for 'ASSP_AFCKnownGoodEXE','Well Known Good Executable Files'
[ASSP_AFCKnownGoodEXE,'Well Known Good Executable Files'
'Put the SHA256_HEX hash of all well known good executables in to this file (one per line). If the SHA256_HEX hash (not case sensitive) of an attachment or a part of a compressed attachment
(e.g. exe, *.bin MS-Macro or OLE) is equal to a line in this file, the attachment passes the attachment check for all mails (regardless its extension and the settings in UserAttach).
The same applies to the following ojects in a PDF file: Certificate, Signature, JavaScript . If the SHA256_HEX hash of any of these PDF objects matches, the PDF will pass the attachment check.
Comments are allowed after the hash and at the begin of a line (recommended).
If configured, the analyzer and the maillog.txt will show the SHA256_HEX hash and the optional defined comment for all detected executables and PDF objects.
For security reasons, virus scanning is not skipped.
Notice: this feature is mainly created for executable files, but it will work for every attachment and every part of a compressed attachment.
For example - this can be usefull, if clients regular sending or receiving documents or excel sheets, which contains every time the same MS-Macro/MS-OLE (e.g. executable).
In this case, decompress the doc[xm] and calculate the SHA256_HEX hash for the vbaProject.bin or the vbaProjectSignature.bin file and register the hash here.
examples:
# sales documents
a704ebf55efa5bb8079bb2ea1de54bfd5e9a0f7ed3a38867759b81bfc7b2cc9c # sales price_list.pdf - contains well known good Java-Script
96c4e6976d16b424ff02d7ef3fdabf41262d3ffc6a191431dc77176a814c1256 # sales sales_report.pdf - contains known Certificate
08d5518ef129ba1a992f5eb5c25e497cf886556710ffebe7cfb6aedf9d5727c9 # VBA Macro signature vbaProjectSignature.bin in sales info.docm
In addition to the SHA256_HEX hash, you can define at which compression level the hash should be valid. Compression levels are comma separated numerical values or ranges
- like 0,1,2 or 0-2 or 0...8 or 0-2,4...6 or 1 .
The compression level zero is the not decompressed attachment itself. To include all compression levels, define a single asterix * or no level definition.
examples:
# sales documents
a704ebf55efa5bb8079bb2ea1de54bfd5e9a0f7ed3a38867759b81bfc7b2cc9c 0,1 # sales price_list.pdf - contains well known good Java-Script - valid at zip level 0 and 1
96c4e6976d16b424ff02d7ef3fdabf41262d3ffc6a191431dc77176a814c1256 * # sales sales_report.pdf - contains known Certificate - valid at any zip level
08d5518ef129ba1a992f5eb5c25e497cf886556710ffebe7cfb6aedf9d5727c9 1 # VBA Macro signature vbaProjectSignature.bin in sales info.docm - only valid in the .docm itself (which is a zip) - .docm in a zip is not valid
08d5518ef129ba1a992f5eb5c25e497cf886556710ffebe7cfb6aedf9d5727c9 0 # VBA Macro signature vbaProjectSignature.bin in sales info.docm - this will not work, because a .docm is a compressed file
To show the SHA256_HEX value for a file at the command line, execute :>shasum -a 256 -b the_file_name
To show the SHA256_HEX values for all relevant PDF-objects in a PDF file, change in to the assp folder and execute :>perl getpdfsha.pl the_PDF_file_name .
You may also compose and send a mail with the files in question attached to the analyze email-interface - EmailAnalyze .
The log output of the analyzer will show all SHA256_HEX hashes (if AttachmentLog is enabled).
Notice: different PDF creator applications may store the same PDF-object (Cert, Sig, JS) in different ways, which will result in different SHA256_HEX hashes for the same PDF-object!
If this happens, you need to calculate the SHA256_HEX hash for each different occurence of the PDF-object.'
2019-01-15
fixed in assp 2.6.3 *SPAM-Evaporator* build 19015:
added:
- ASSP_AFC 5.01 is released - it includes a new extension
'ASSP_AFCKnownGoodEXE','Well Known Good Executable Files'
'Put the SHA256_HEX hash of all well known good executables in to this file (one per line). If the SHA256_HEX hash (not case sensitive) of an attachment or a part of a compressed attachment
(e.g. exe, *.bin MS-Macro or OLE) is equal to a line in this file, the attachment passes the attachment check for all mails (regardless its extension and the settings in UserAttach).
Comments are allowed after the hash and at the begin of a line.
If configured, the analyzer and the maillog.txt will show the SHA256_HEX hash and the optional defined comment for all detected executables.
For security reasons, virus scanning is not skipped.
Notice: this feature is mainly created for executable files, but it will work for every attachment and every part of a compressed attachment.
For example - this can be usefull, if clients regular sending or receiving documents or excel sheets, which contains every time the same MS-Macro/MS-OLE (e.g. executable).
In this case, decompress the doc[xm] and calculate the SHA256_HEX hash for the vbaProject.bin or the vbaProjectSignature.bin file and register the hash here.
examples:
# sales documents
a704ebf55efa5bb8079bb2ea1de54bfd5e9a0f7ed3a38867759b81bfc7b2cc9c # sales price_list.pdf - contains Java-Script
08d5518ef129ba1a992f5eb5c25e497cf886556710ffebe7cfb6aedf9d5727c9 # VBA Macrco vbaProject.bin in sales info.docm
To show the SHA256_HEX value for a file at the command line, execute :>shasum -a 256 the_file_name'
changed:
- the default value for 'DoNoFromSelect' is changed from 63 to 59
option 4 - multiple from: addresses or from: header tags found (potential 2x score if option 2 is also enabled) - caused too many false positives
2019-01-04
fixed in assp 2.6.1 *Fortress* build 19004:
- a new ASSP-MIB file is available and required for this version if SNMP is used
- specific unicode regular expressions like \p{Yi} and others - were not working for the MIME header under certain conditions (spam bomb definitions were not affected by this issue)
- improved domain name parsing - the length restiction (63 bytes) for each label is now checked
- assp_pop3.pl version 1.22 is released
- the SSL mode workaround for old Net::POP versions is removed - at least version 3.07 of Net::POP3 is now required
- in some exceptional cases it was possible, that an email was retrieved and delivered multiple times
2018-12-27
fixed in assp 2.6.1 *Fortress* build 18361:
added:
- using the command line switch 'checkLinuxENV:=n' or setting $main::checkLinuxENV=n; in 'lib/CorrectASSPcfg.pm', assp will the ulimit settings and the selinux state on nix systems
in case any settings seems to be too less, warnings or errors are shown at startup
our $checkLinuxENV = 0; # (0/1/2) check ulimit (1) on nix and selinux (2) on linux systems
added:
'DoNoFromRemovesNPWL','DoNoFrom Removes NP, WL Flag','0:disabled|1:whitelisted|2:noprocessing|3:both'
'If the combination of DoNoFrom , DoNoFromSelect , DoNoFromWL and DoNoFromNP gives more than one hit, the whitelisted and/or the noprocessing flag will be removed from the message.
For example: if the FROM: and /or SENDER: address fakes a whitelisted and/or noprocessing address or domain.
Default setting is both.
The noprocessing by size flag ( npSize ) will be keeped.'
2018-12-17
fixed in assp 2.6.1 *Fortress* build 18351:
- If the daily amount of collected .eml files in one folder exceeded the value of 'MaxFiles', new files were removed by the daily
file cleanup processing - which caused failing resend requests. New files are now keeped for at least five days, even the file count
exceeds the value of 'MaxFiles'. Set 'MaxFiles' high enough, to keep files for a longer periode.
- ASSP_AFC.pm 4.89 fixes a BUG where a missdetection of MIME-file-types prevented the decompression of zip files
- The file edit dialog got an additionally option, to resend a blocked mail and to copy the blocked spam file to correctednotspam at the same time.
2018-12-05
fixed in assp 2.6.1 *Fortress* build 18339:
- BerkeleyDB engine version 18.1 was detected as too old
2018-12-03
fixed in assp 2.6.1 *Fortress* build 18337:
- DoNoFrom detected email addresses in the text part of the header text - like: "do not detect this address user@domain.com but the next one" <other.user@other-domain.org>
- under rare conditions the file name in a blocked mail resend request was wrong parsed, the file was'nt found and the resend failed
added:
- 'DoNoFromSelect','Select Checks for From: and Sender: Header'
Select which check should be done in DoNoFrom .
1 - from: and sender: header tag are both missing
2 - different domains found in from: and sender: email addresses
4 - multiple from: addresses or from: header tags found
8 - multiple sender: addresses or sender: header tags found
16 - no or an invalid email address found in from: header tag
32 - no or an invalid email address found in sender: header tag
Simply form the sum of the numbers in front of the checks you want to select (0...63). Default vaule is 63 (1+2+4+8+16+32) - all checks are selected.'
2018-11-24
fixed in assp 2.6.1 *Fortress* build 18328:
- it was possible that bomb.. checks were matching with an empty string result - now only bombSubjectRe can match on an empty string
2018-11-22
fixed in assp 2.6.1 *Fortress* build 18326:
- using the search option in MaillogTail has show expected results
- faster search in MaillogTail
- an entry like "user@domain.com" <user@domain.com> in any header tag was missinterpreted as tow email addresses
2018-11-13
fixed in assp 2.6.1 *Fortress* build 18317:
- bad MIME encoded multiline headers were some times wrong decoded
(in some mails, header lines were broken in to multiple MIME encodings at any byte instead at a character)
2018-11-12
fixed in assp 2.6.1 *Fortress* build 18316:
changed:
The .eml file editor dialog got some new options
- the action pulldown menu got an option to force the resend of the email including ALL attachments 'copy file to resendmail and force attachments' (shown in red color !)
- an action button "show email in browser sandbox" is added
Using the left mouse button at "show email in browser sandbox" will show the email in a secured browser sandbox "
https://en.wikipedia.org/wiki/Content_Security_Policy" (Content Security Policy),
using the right mouse button, images will be show in addition. Showing images can be a risk, if they contain malicious code!
- If the email contains attachments or includes, a hint is given and the attachments are listed as links. Clicking on such a link will download the attachment to the local machine.
This may be used to check attachments for malicious content before a resend is requested.
2018-11-09
fixed in assp 2.6.1 *Fortress* build 18313:
- reduced memory footprint for GUI request handling
- a SMTP worker was in rare cases dieing, because syswrite was unable to process wrong encoded emails or attachments
changed:
- 'DoNoFrom' now also checks for multiple FROM: or SENDER: email addresses in a single header tag
- the default value for 'DoNoFromWL' and 'DoNoFromNP' is change to 1 (checked)
- the internal attachment blocking feature (without using the ASSP_AFC plugin) now allows to detect extended file extensions, like .... .tar.gz or .... .tar.gz.aes
(ASSP_AFC was and is able to handle those file extensions)
2018-10-31
fixed in assp 2.6.1 *Fortress* build 18304:
- after upgrading the perl module Win32::Daemon to version 20181025 assp was no longer starting as a windoes service
this assp version contains a workaround for the buggy Win32::Daemon module
- a small memory leak is solved in unicode processing for perl 5.22 to 5.28
- the detection of incoming DMARC-reports is improved
- the not RFC conform DMARC-reports from "Amazon SES" are now correctly detected
- perl module load errors in ASSP_AFC were not shown in the file moduleLoadErrors.txt - ASSP_AFC.pm is updated to version 4.87 to get this fix working
- some of the X-ASSP-... headers were some times too long (RFC822, RFC 1522)
changed:
- backscatter checks are skipped for regular incoming mails (not matching redRe) for local postmaster@ and webmaster@ addresses, even these addresses are listed in BounceSenders
2018-10-22
fixed in assp 2.6.1 *Fortress* build 18295:
- The DMARC check now follows the RFC7489 for the blocking rules. The DMARC-check is OK, when SPF or DKIM passes there check.
- If an email was blocked by the SPF-check, no DMARC-report was generated for this email.
- The DKIM-precheck is improved to detect, if a domain supports DKIM or not.
added:
- ASSP supports now the SMTP extension 'Require-TLS - REQUIRETLS'
https://tools.ietf.org/html/draft-ietf-uta-smtp-require-tls-04 This is just a Draft in version 04 - for this reason, the feature is still experimental.
The following hidden configuration parameters are used by this feature.
our $enableREQUIRETLS = 0; # (0/1) enable testing of the REQUIRETLS implementation
our $provideREQUIRETLS = 0; # (0/1) include REQUIRETLS in to the EHLO reply if not already provided
our $forceREQUIRETLS = 0; # (0/1) include REQUIRETLS in to the MAIL FROM: command if not provided by the MTA
changed:
- If DMARC is enabled and a NDR is received for a sent DMARC-report for any reason and 'noDMARCReportDomain' is configured using the 'file:...' option,
the foreign report recipient address and/or the report domain are automatically added to 'noDMARCReportDomain'.
- If a domain provides an explicite _adsp policy with the value 'unknown', the domain is no longer added to the DKIMCache and does no longer require to sign all mail using DKIM,
if 'DKIMCacheStrict' is not set.
The behavior is not changed for the case where the _adsp policy with the value 'unknown' is not explicite defined and falls per default to 'unknown' or 'DKIMCacheStrict' is set.
- The mail analyzer now shows results with more details for the DMARC-check.
- The maillog.txt file list in the 'MaillogTail view' is now shown permanent (unless closed) and contains four columns to provide more files to be shown.
2018-10-15
fixed in assp 2.6.1 *Fortress* build 18288:
- the included rebuildspamdb.pm inreases the rebuild performance by 10 to 20%
- the correction of the spamDB and HMMdb in case of reported spam or ham was too weak
changed:
- a new ASSP-MIB file is available and required for this version if SNMP is used
- the default value for 'backupDBInterval','backup database Interval' is changed from 2 to 12 hours
- the description for 'SNMP' is changed
...
The following OIDs (relative to the SNMPBaseOID) are available for SNMP-queries. The configuration values are changeable via snmp. The published file mib/ASSP-MIB,
which contains all possible OID\'s, could be used in SNMP browsers to get a human readable view of the OID's (copy it to the net-snmp MIB file location - eg: [C:]/usr/share/snmp/mibs
and the MIB location of your SNMP browser). Please keep in mind, that an extensive usage of SNMP queries will slow down assp.
Because the OID numbers can change in different assp versions, it is recommended to query the OID's by its consistent name (not by its number). This requires the usage of the assp version
compatible mib/ASSP-MIB file!
If you want to query or set any of the following configuration parameters: LocalAddresses_Flat, LocalAddresses_Flat_Domains, noBayesian_local, Bayesian_localOnly, SSL_version, SSL_cipher_list -
remove all underscores from the config name to build the OID-name, because underscores ar not allowed in SNMP queries. The MIB file already contains the corrected names.
If you get unexpected SNMP-query results or you've lost the version compatible MIB file, rename the perl scripts lib/SNMPmakeMIB.p_ and lib/SNMPmakeMRTG.p_ to *.pl and restart assp.
This will create the mib/ASSP-MIB and mib/assp-mrtg.cfg files, based on your installation and configuration. It is recommended to rename both scripts back, after the new MIB files are created.
NOTICE: If you install or uninstall any plugin or you enable or disable the configuration synchronization and you use such a custom MIB file, the mib/ASSP-MIB file needs to be recreated
to implement the new OID\'s and (at least) to correct the new OID order!
To prevent permantly copying the changed mib/ASSP-MIB file to your net-snmp daemons MIB-folder - (e.g.) create a link there to the mib/ASSP-MIB file.
...
- the check (and ignore) whitelisted and redlisted mails in the rebuildspamdb task is now disabled per default
to return to the old behavior set the hidden parameters 'DoRBWhite' and/or 'DoRBRed' to 1.
added:
'spfValencePB' is no longer scored in case DMARC failed - instead 'dmarcValencePB' is used
'dmarcValencePB','DMARC Failed, default=10 +'
2018-10-08
fixed in assp 2.6.1 *Fortress* build 18281:
- the analyzer has wrong shown nonascii characters in the analyzed file name
- depending on the modes set for 'ValidateSPF' and 'DoDKIM', the DMARC-check was unexpected using monitoring-mode instead of scoring-mode
2018-10-05
fixed in assp 2.6.1 *Fortress* build 18278:
- the (SPF) statistic counter was not working for failed DMARC checks
changed:
- 'DoNoSpoofing4ReplyTo','Do NoSpoofing for Reply-To:' now also processes 'Return-Path:' and 'Disposition-Notification-To:' addresses.
- If a malformed address is found in any of the following header tags,
from, sender, reply-to, errors-to, returnreceipt, return-receipt-to, return-path, disposition-notification-to
the mail and IP gets a score of'nofromValencePB' for each found malformed address and if 'DoDomainCheck' is enabled, this check failes/scores for each found malformed address.
2018-10-04
fixed in assp 2.6.1 *Fortress* build 18277:
- the DMARC check ignored the SPF alignment, if 'DoSPFinHeader' was not enabled
added:
notice: the default behavior of assp is changed for whitelisted and noprocessing envelope sender addresses, domains and IP's!
It was often the case, that mails from known good external senders were blocked, because they sent mails to a list of envelope recipients - but over the time, some of the recipient were no longer valid.
ASSP detected this 'invalid recipient' attempt and the known good mail was blocked by the recipient check or the mail/IP got a high penalty and was blocked by the penalty-box.
The hidden configuration parameter 'ignoreInvalidAddressNPWL' is used to ignore the defined 'invalid recipient action', if a known good sender uses unknown envelope recipents in a sequence
of multiple envelope recipents. The mail is only blocked, if no valid envelope recipient is left over at the DATA command. If an unknown envelope recipient is used, the sender gets no penalty score,
but the invalid 'RCPT TO:' command is replied with the permanent error '550 5.1.1 User <xxxxx> unknown'. The connection is not dropped in this case. Such an 'invalid recipient' attempt will also
not be counted for 'MaxErros'.
consequence: the mail is delivered to all left over valid envelope recipients and the sender will be informed about each invalid recipient (if NDR is supported by the sending server)
$ignoreInvalidAddressNPWL = 3; # (0/1/2/3) ignore invalid envelope recipients for whitelisted (2) or noprocessing (1) or both (3) senders and IP's (no score, no connection drop, no error count)
Until now, the default action of assp was like 'ignoreInvalidAddressNPWL = 0' - this is now changed to 'ignoreInvalidAddressNPWL = 3'
To change back to the old behavior or to change the default behavior, you have two options:
1. start assp with the commandline switch --ignoreInvalidAddressNPWL:=X
2. add the line below to the sub set in 'lib/CorrectASSPcfg.pm'.
$main::ignoreInvalidAddressNPWL = X;
In both cases, X is the configuration value of your choice (0...2).