eFloodLimiter

[Arf]

With the spat of spammers that are agressively using botnets to send so much spam to our servers, I felt something a little agressive is needed. So, I wrote a new program (big surprise there). Because the problem is so wide spread, I'm planning on giving away the first release for free. But before I do I wanted some input on the reasoning behind the program to see if my logic is going to cause problems...

How does this sound...This program checks the maillog to see how many times a particular IP has sent your server email in a given hour. If it's sent more than X times, it blocks the IP from any access to your server for Y hours and then releases it to allow email from that IP again.

This is different from a checking for dictionary attacks in that it doesn't check where the email is going to. I just limits the amount of emails that can come to the server from a particular IP. Even if an IP is dynamically shared over the period of an hour (like at AOL) it's not likely that there's going to be over 60 emails send from it per hour to one or more people on a given server.

I'd really like your input on this so I can post it publicly after a few days of testing on my test subjects...err...clients.

[Okidoki]

Arf,

Thank you.

An idea. I would like the IP that doesn't have resolution Domain Name to be blocked by more time (days or weeks)

[flaguy]

Arf would we have the option to un-block any IP's that get blocked...or white list an IP.

Thanks,

Mike

[Arf]

Arf would we have the option to un-block any IP's that get blocked...or white list an IP.

Yes, of course.

[Arf]

Arf,
Thank you.
An idea. I would like the IP that doesn't have resolution Domain Name to be blocked by more time (days or weeks)

So, for example if the IP was this and the test was as follows we would not allow this one for a longer time?

[root ~/cmd]# host 61.254.127.109
Host not found.

Is there a better test command than "host"?

[Okidoki]

yes Arf, but the command maybe is better

nslookp 61.254.127.109

*** ns.myserver.com can't find 61.254.127.109: Non-existent host/domain

[Okidoki]

or control if domain has MX Record

nslookup -q=mx spamcop.net

Non-authoritative answer:
spamcop.net preference = 5, mail exchanger = mx.spamcop.net
spamcop.net preference = 10, mail exchanger = mx2.spamcop.net

Authoritative answers can be found from:
spamcop.net nameserver = asia3.akam.net
spamcop.net nameserver = ns1-109.akam.net
spamcop.net nameserver = ns1-11.akam.net
spamcop.net nameserver = ns1-117.akam.net
spamcop.net nameserver = ns1-73.akam.net
spamcop.net nameserver = ns1-90.akam.net
spamcop.net nameserver = ns1-93.akam.net
spamcop.net nameserver = use1.akam.net
mx.spamcop.net internet address = 216.154.195.53
mx.spamcop.net internet address = 216.154.195.36
mx2.spamcop.net internet address = 216.154.195.36
mx2.spamcop.net internet address = 216.154.195.53
asia3.akam.net internet address = 61.200.81.105
use1.akam.net internet address = 63.209.170.136

nslookup -q=mx 61.254.127.109

*** ns.mysever.com can't find 61.254.127.109: Non-existent host/domain

[datazen]

Wow, everyone beat me to the questions

Arf, your logic sounds good. Let's give it a try!

[Arf]

Let's give it a try!

Well, I kind of did and will be running the program for the next few days on my server. So far so good. Blocked a few IPs that have sent more than 60 emails. Releasing them after two hours. I don't know how far to ratchet it down the quantity of emails per hour to allow. What if the guy has a mailing list to his company. I'm going to figure out a way to count the mailing list membership (dist file I guess) and maybe give the command the the group to see what we can do about those.

[Okidoki]

What if the guy has a mailing list to his company

It could verify before blocking the IP, if the IP pop3.log is, if is there not block.

[khimaira2]

Sounds a little like qmail's tarpitting feature which adds a delay to the SMTP process when a set email rate is exceeded (Anyone know a sendmail equivalent?). I do like the idea, but here's a scenario we've seen: An email address on our server forwards to their local ISP (quite common). Their address gets a flood of spam and triggers tarpitting at the local ISP. Now the backup is on our server as well as the added load of the repeated delivery attemps. OK, so your script might stop that original flood. That's good! But let's say a client has an outside address that legitimately forwards to an address on our server. They could complain if their email were completely blocked versus slowed. How could you distinguish all the email coming from yahoo, aol, msn, etc. versus something from xyz.ru or xyz.kr? Our primary market might be considered a niche market and there are a lot of subscribers to the same newsgroups. I wouldn't be surprised if hundreds of messages per hour don't come in from yahoogroups. Anyway, that's just something to consider.

On a philosophical note, I'm beginning to cringe when I see spam blockers installed on individual PC's. They might protect the user, and because they are so well protected, they can now ignore the problem. They have effectively just passed the problem back up the line. We're all in that boat. Look at how hard it can be to get users to set their default address to trash their mail. They want everything to go through and I assume rely on their spam blockers to delete the trash. In a way, we do the same thing when we try to block spam with all our techniques. They might be necessary and they might be justified, but the traffic and load on the internet is still there until the sources are shut down. That includes intentional spammers and irresponsibly infected PC's.

[Arf]

Good news - bad news.

The good news is that the program appears to be working very well. I have it stop all connections from IPs that send more than 50 emails in a given hour. After two hours, it releases the IP again. That's the good news.

The bad news is that there's only been about 30 IPs blocked in the last 24 hours. I was hoping to nail more. They're just using so many IP addresses through botnets. I guess I could lower the threshold to say 30 but it makes me wonder how much is too much.

On the other hand, the program is keeping a list of all the IPs it once blocked. I will add a repeat offender switch to it some time and then two strikes and you're out for a long time.

[datazen]

Here's an idea ... How about we all just block Ip's from China

75% problem solved

[Arf]

Here's an idea ... How about we all just block Ip's from China

75% problem solved

Can you be more specific? What range?

Actually, this probably will be useless. These are botnets (my guess). So the IPs are all across the board.

[Arf]

here's something I've found that would explain some of the server spikes. I got 100+ emails from on IP in 1 minute 11 seconds. Add this to a few other IPs doing the same thing and you've got trouble. This was a classic dictionary attack except that they didn't CC/BCC more than two addresses in any given email.

I had one send 1500 emails in 770 mailings. All in 2 minutes. This sent the server in upwards of a load average of 5.xx. Which isn't bad considering.

So far, my observation has been that they don't yet repeat the same from IP address and they usually only a few mass mailings though that most of which are a total of 50 emails with 100 recipients. Very hard to catch.

[jady]

Thomas - please go for it. You already have full access to one of our boxes - we will help you beta test! (Just dont format any hard drives if you get mad at me)

SOUNDS LIKE ANOTHER WINNER!

[rldev]

Any word on this?

[Arf]

This program has been working fine since July 12. It did block some legitimate IPs (5) at first but I whitelisted them and all has been fine. Overall, the program does what it says it's going to do but fails to make any headway in lowering the server load or solving any real problems. So, it's been a good test, but useless in results. Sorry.

Any suggestions on improving the logic would be appreciated. Here's the code for anyone who might wish to play with it.

Code: Select all

#!/bin/bash 
# BETA VERSION 0.5B
# eFloodLimiter (Email Flood Limiter) by Thomas Leo
# this program counts how many emails were sent from 
# a given IP address in the last hour. If they are over
# THRESHOLD the IP is blocked for X hours.
#
# INSTALLATION:
# Edit two variables below. Chmod program 700. 
# Run as a root cron each hour. 
# Be sure to change to the directory of the program in the cron.
# 1 1 * * * cd /PathToProgram/; ./eFloodLimiter.sh

THRESHOLD=60 # Max number of emails an IP can send per hour. 
JAILTIME=2   # Hours before accepting email from the IP again.

################# DO NOT EDIT BELOW THIS LINE ##############################
DEBUG=1 #0=off or 1=on  # On prevent the adding of new IPs.
SAFEMODE=0
let JAILTIME=$JAILTIME*3600
# Eliminiate the file not found errors on first run. 
touch floodlimiter.IPLIST floodlimiter.SYNFLOOD.tmp floodlimiter.WHITELIST

NOW=`date +%s`

spinner()
{
  let SPINNER=$SPINNER+1
  case $SPINNER in
  1) echo -ne "-\b" ;;
  2) echo -ne "/\b" ;;
  3) echo -ne "-\b" ;;
  4) echo -n '\';echo -ne "\b"  ;;
  5) echo -ne "|\b" ;;
  6) echo -ne "/\b" ;;
  *) SPINNER="0"  ;;
  esac
}

debug()
{ 
        if [ "$DEBUG" = "1" ]; then
                echo "$*"
        fi
}

# get a count of IPs that sent email in the last hour.
LASTHOUR=`date -d "1 hour ago" +%b" "%e" "%H":"`
debug LASTHOUR=$LASTHOUR
grep "$LASTHOUR" /var/log/maillog | grep "from=" \
  | cut -d'[' -f3 | cut -d']' -f1 \
  | grep -v "127.0.0.1" \
        | sed '/^$/d' \
  | sort -n | uniq -c | sort -n > floodlimiter.CURRENTLIST

# Add abusers to floodlimiter.IPLIST
while read ITEM; do
        spinner
        COUNT=`echo $ITEM | awk '{print $1}'`
        debug COUNT=$COUNT THRESHOLD=$THRESHOLD
        if [ $COUNT -gt $THRESHOLD ]; then
                IP=`echo $ITEM | awk '{print $2}'`
                debug IP=$IP
                if [ "$IP" != "" ]; then
                        if [ `grep -c "$IP$" floodlimiter.IPLIST` = 0 ]; then
                                if [ `grep -c $IP floodlimiter.WHITELIST` -gt 0 ];then
                                        echo "$IP has sent over $THRESHOLD emails but is whitelisted."
                                else
                                        echo -e "BLOCKING\t\t[$COUNT]\t\t$IP"
                                        echo "$NOW      $IP" >> floodlimiter.IPLIST
                                        debug IP=$IP   COUNT=$COUNT   LASTHOUR=$LASTHOUR
                                        echo "$IP [$COUNT] $LASTHOUR" >> floodlimiter.BLOCKLOG

                                fi
                        else
                                echo "Looks like we're already blocking $IP"
                        fi
                fi
        fi
done < floodlimiter.CURRENTLIST

# release any IPs blocked longer than JAILTIME
let LOCKEXPIRED=$NOW-$JAILTIME
LOCKEXPIREDHUMAN=`date -u --date "Jan 1, 1970 00:00:00 +0600 + $LOCKEXPIRED seconds"`
echo "Releasing IPs that are older than $LOCKEXPIREDHUMAN"
> floodlimiter.RELEASED.tmp
> floodlimiter.IPLIST.tmp
while read ITEM; do
        LOCKTIME=`echo $ITEM | awk '{print $1}'`
        if [ $LOCKTIME -lt $LOCKEXPIRED ]; then
                IP=`echo $ITEM | awk '{print $2}'`
                echo "Releasing $IP"
                echo "$IP" >> floodlimiter.RELEASED.tmp
        else
                echo "$ITEM" >> floodlimiter.IPLIST.tmp
        fi
done < floodlimiter.IPLIST

mv -f floodlimiter.IPLIST.tmp floodlimiter.IPLIST

#add or remove the IPs from the synflood list.
# Get existing synfloodlist
cat /usr/local/custom/etc/.synFloodList \
 | sort -n > floodlimiter.SYNFLOOD.tmp

#Add new IPs
cat floodlimiter.IPLIST | awk '{print $2}' >> floodlimiter.SYNFLOOD.tmp

#Return synfloodlist after removing released IPs and duplicates. 
# also making sure we don't include any white listed IPs. 
sort -n floodlimiter.SYNFLOOD.tmp | uniq \
  | grep -v -f floodlimiter.RELEASED.tmp \
        | grep -v -f floodlimiter.WHITELIST > floodlimiter.SYNFLOOD

#Overwrite existing synfloodlist file and restart:
echo "One moment please..."
if [ $SAFEMODE = 0 ]; then
        cat floodlimiter.SYNFLOOD > /usr/local/custom/etc/.synFloodList
        /usr/local/custom/sbin/checkip -S
else
        echo "SAFEMODE is ON so I won't add new IPs."
fi

echo -e "DONE!\nThank you for using my software --Thomas"

[ajai]

Hello everybody

Is there a way to limit the number of conections from any IP to any port at the level of the ipchains without specifing an concrete IP ?

I think in newer versions of iptables there is a way of doing this but i am not sure with our current navite version of ipchains

thx in advance