15,000+ spam attempts an hour...

[RJM62]

...all from Taiwan, all bearing spoofed Yahoo addresses, and all aimed at my server. Why? I have no idea. I have no open relays or other vulnerabilities that I know of, nor can the hosting company find any.

Apparently, however, the attempts actually started before I took possession of the IP addresses last week, and have been increasing steadily. I hadn't noticed because the machine has plenty of resources and was running pretty well.

Today, however, it reached a crescendo; and the server was so busy sending bounce messages (using 105 percent of CPU capacity and almost 2 GB of RAM) that services started crashing on both the master and the slave servers.

I tweaked a few settings in Exim, blocked the IP range, and disabled bouncing, and things settled down. What an annoyance, though.

-Rich

 

[Paul Allen]

This is unreal. I once had 2500 junk emails in an hour. I had no idea what to do so I just deleted all of them.

 

[rpadula]

Yeah, my inbox had about 30 "delivery failure" messages this morning at work. Usually our IT dept. is quite good at filtering this stuff, so I can only imagine how many they had already rejected. All had a link to a Google Docs page, which I presume is their real spam message. Hopefully the GOOG will clamp down on that.

It was my free entertainment of the morning, though. Best content: "Gain the full control over your drilling machine."

Now where is Bill's "spammers must DIE!" thread...

 

[wsuffa]

You mean the thread where the 10,000+ bounce messages per hour from other servers to mine (after some botnet spoofed my email address) brought my server to it's knees?

Rich - (RJM Rich) - I discovered the hard way that certain versions of Exim are not very good at storing excessive numbers of inbound emails.

 

[mantakos]

My mail server has about 1000 of these in the logs from the past hour:

Jan 7 08:54:33 dirty sm-mta[2739]: ruleset=check_relay, arg1=[189.24.3.122],arg2=127.0.0.4,relay=18924003122.user.veloxzone.com.br [189.24.3.122] (may be forged), reject=550 5.7.1 Rejected: 189.24.3.122 listed at zen.spamhaus.org
​

There's nothing unusual going on, this happens continually.

I'm using DNS "blocklisting" from Spamhaus, a free service. It requires very little CPU to reject a connection by IP address.

Some spam still leaks through, of course, most of which gets filtered out by spamassassin, but setting this up was the difference between my server getting killed regularly and a quiet serene existence.
-harry

 

[wbarnhill]

This is why we've got one of these...

Spammers seem to love .edu addresses. Guess they figure there are a lot of gullible students and/or professors out there..

 

[jmaynard]

I'm using DNS "blocklisting" from Spamhaus, a free service. It requires very little CPU to reject a connection by IP address.

Indeed. Every email server should use the Spamhaus blocklist.

 

[RJM62]

I implemented the Spamhaus service, Harry. Thanks. I had been using Spamcop, which apparently is lacking....

I had banned the entire IP ranges of TW and CN from connecting to SMTP early this morning because all the IPs traced back to those countries. I may just leave it that way, as none of my clients, to my knowledge, would ever send mail from Taiwan or China.

-Rich

 

[mantakos]

By the way, about the bounces, back when I was getting inundated with spam, I generally got the impression that I was being used, effectively, as an open mail relay. Not in the usual "open mail relay" way, which my server was properly configured to refuse, but by simply forwarding spam by generating bounce messages. So the spammer sends me a spam message with a forged sender address and a bogus recipient address, my server bounces that because of the bogus recipient, and sends the bounce to the forged sender address, including a snippet of the body of the original message.

The net result is that I end up, via bouncing, relaying the spam message from the spammer to the targeted recipient.
-harry

 

[ReverendSlappy]

My mail server has about 1000 of these in the logs from the past hour:

Jan 7 08:54:33 dirty sm-mta[2739]: ruleset=check_relay, arg1=[189.24.3.122],arg2=127.0.0.4,relay=18924003122.user.veloxzone.com.br [189.24.3.122] (may be forged), reject=550 5.7.1 Rejected: 189.24.3.122 listed at zen.spamhaus.org
​

There's nothing unusual going on, this happens continually.

I'm using DNS "blocklisting" from Spamhaus, a free service. It requires very little CPU to reject a connection by IP address.

Some spam still leaks through, of course, most of which gets filtered out by spamassassin, but setting this up was the difference between my server getting killed regularly and a quiet serene existence.
-harry

Indeed. Every email server should use the Spamhaus blocklist.

This. God bless Spamhaus.

 

[mikea]

By the way, about the bounces, back when I was getting inundated with spam, I generally got the impression that I was being used, effectively, as an open mail relay. Not in the usual "open mail relay" way, which my server was properly configured to refuse, but by simply forwarding spam by generating bounce messages. So the spammer sends me a spam message with a forged sender address and a bogus recipient address, my server bounces that because of the bogus recipient, and sends the bounce to the forged sender address, including a snippet of the body of the original message.

The net result is that I end up, via bouncing, relaying the spam message from the spammer to the targeted recipient.
-harry

ARGGHHH!!! Your server should NEVER, EVER reply to the sender address. IT IS NOT THE REAL SENDER. These clueless spam filters make up 75% of the message traffic.

I know it's the standard configuration. Consider this: Symantec and such are just using you to SEND SPAM advertising their spam server.

Just toss the messages.

 

[mantakos]

These clueless spam filters make up 75% of the message traffic.

I agree that bouncing a message because a spam filter identified it as spam is a very bad idea.

In my case, though, this is just the usual bounce mechanism resulting from sending to an unknown address, it has nothing to do with spam filtering.

Maybe I'm old-school, but completely eliminating the concept of a bounce message is something I'm hesitant to do. Too much baby going out with that bath water. Somebody mistypes an email address, they should get a bounce.
-harry

 

[jmaynard]

I agree that bouncing a message because a spam filter identified it as spam is a very bad idea.

There's a difference between sending a separate reply and rejecting during the SMTP transaction. The former is spam; the latter is the way things should be done: never, ever just drop a message on the floor, because if it's wrongly deleted, there's no way the real sender can know that his message never got delivered.

The only messages I drop on the floor are sent by viruses.

 

[wsuffa]

I'm using DNS "blocklisting" from Spamhaus, a free service. It requires very little CPU to reject a connection by IP address.

Some spam still leaks through, of course, most of which gets filtered out by spamassassin, but setting this up was the difference between my server getting killed regularly and a quiet serene existence.
-harry

And that's why spammers are using bounce messages to attack.... (I use SpamAssassin).

 

[mantakos]

There's a difference between sending a separate reply and rejecting during the SMTP transaction.

You're right, unknown addresses are rejected in the smtp transaction. Now I don't remember what was happening, before, or why I was generating bounces, or if I really was.

I'm remembering there being a lot of messages in my sendmail queue, though now I don't recall why. I do run an email list with 'mailman', which may have been the generating source of the bounce messages.
-harry

 

[wsuffa]

You're right, unknown addresses are rejected in the smtp transaction. Now I don't remember what was happening, before, or why I was generating bounces, or if I really was.

I'm remembering there being a lot of messages in my sendmail queue, though now I don't recall why. I do run an email list with 'mailman', which may have been the generating source of the bounce messages.
-harry

SOME of the SMTP products will check the IP list and generate a 5.5.0 or 5.3.x bounce message. This really should be turned off. Just blackhole the spam and be done with it.

If you use an IP filter in front of the SMTP server, no bounce will be generated, the connection will be refused.

 

[wsuffa]

At the office, I'm running Untangle as a 2nd level firewall (behind a hardware firewall). It uses Spamhaus and another one or two DNSBLs, as well as a phish filter and spyware blocker (nice little piece of freeware).

In the past 24 hours, we had 4282 emails hit the Untangle box, 3920 of them were rejected based on the DNSBL, another 78 were flagged and quarantined through SpamAssassin, leaving 304 "legit" messages to come through.

 

[mantakos]

SOME of the SMTP products will check the IP list and generate a 5.5.0 or 5.3.x bounce message.

If the receiving mail server rejects the message with a 5.x.x code, then it isn't generating a bounce message at all, it is simply telling the sending server "this failed, and don't bother retrying". The sending mail server may choose to react to this rejection by originating a bounce message towards the sender.
-harry

 

[RJM62]

I just found out that (at least) the two users whose accounts seemed to be involved had changed their default addresses for undeliverable mail from :fail: to :blackhole:, which also eats resources and bandwidth. I changed them. Now I have to check the rest.

Rich

 

[wsuffa]

If the receiving mail server rejects the message with a 5.x.x code, then it isn't generating a bounce message at all, it is simply telling the sending server "this failed, and don't bother retrying". The sending mail server may choose to react to this rejection by originating a bounce message towards the sender.
-harry

I meant generically that it was a bounce message as the terms are lumped together. In 95+% of the cases the sender gets a bounce from a 5.x.x.

 

[Dave Siciliano]

...all from Taiwan, all bearing spoofed Yahoo addresses, and all aimed at my server. Why? I have no idea. I have no open relays or other vulnerabilities that I know of, nor can the hosting company find any.

Apparently, however, the attempts actually started before I took possession of the IP addresses last week, and have been increasing steadily. I hadn't noticed because the machine has plenty of resources and was running pretty well.

Today, however, it reached a crescendo; and the server was so busy sending bounce messages (using 105 percent of CPU capacity and almost 2 GB of RAM) that services started crashing on both the master and the slave servers.

I tweaked a few settings in Exim, blocked the IP range, and disabled bouncing, and things settled down. What an annoyance, though.

-Rich

Rich: I know a couple deserving folks that could use those if you can redirect them

Best,

Dave

 

[mantakos]

...the server was so busy sending bounce messages (using 105 percent of CPU capacity...

I think your operating system was made by the same people who made Spinal Tap's amps. Or else, maybe, by professional athletes.
-harry

 

[RJM62]

I think your operating system was made by the same people who made Spinal Tap's amps. Or else, maybe, by professional athletes.
-harry

I thought that was a bit odd, too -- and I wrote it. Lack of sleep, most likely.

(CentOS 5.2, by the way.)

-Rich

 

[jesse]

I think your operating system was made by the same people who made Spinal Tap's amps. Or else, maybe, by professional athletes.
-harry

I thought that was a bit odd, too -- and I wrote it. Lack of sleep, most likely.

(CentOS 5.2, by the way.)

-Rich

Is your server single processor single core? If you have multiple cores/processors it is possible to have a number higher than 100%.

If you run top and press "1" it will break out the CPUs.

 

[RJM62]

Is your server single processor single core? If you have multiple cores/processors it is possible to have a number higher than 100%.

If you run top and press "1" it will break out the CPUs.

Yes and no, Jesse. It's a VPS with two Quad Xeons and 16 GB RAM. The machine hosts eight containers. The VPS is also mirrored to an identical machine on another node with automatic failover, but the machines are not load-balanced. (I'm not even sure load balancing is possible on a VPS.)

I chose this peculiar arrangement because I'm working on a couple of projects that need 100 percent uptime plus the ability to tweak the server settings and install software, but dual dedicateds are a bit out of my price range at the moment. But the host guarantees 100 percent uptime with this setup.

-Rich

 

[jesse]

Yes and no, Jesse. It's a VPS with two Quad Xeons and 16 GB RAM. The machine hosts eight containers. The VPS is also mirrored to an identical machine on another node with automatic failover, but the machines are not load-balanced. (I'm not even sure load balancing is possible on a VPS.)

I chose this peculiar arrangement because I'm working on a couple of projects that need 100 percent uptime plus the ability to tweak the server settings and install software, but dual dedicateds are a bit out of my price range at the moment. But the host guarantees 100 percent uptime with this setup.

-Rich

Sure. How does it appear to the VPS kernel though? That is what I was trying to get at.

run:
cat /proc/cpuinfo

paste contents in reply

 

[RJM62]

Sure. How does it appear to the VPS kernel though? That is what I was trying to get at.

run:
cat /proc/cpuinfo

paste contents in reply

root@vps1 [~]# cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Core(TM)2 Quad CPU           @ 2.40GHz
stepping        : 7
cpu MHz         : 1795.537
cache size      : 4096 KB
physical id     : 0
siblings        : 4
core id         : 0
cpu cores       : 4
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm constant_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr lahf_lm
bogomips        : 4790.00
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Core(TM)2 Quad CPU           @ 2.40GHz
stepping        : 7
cpu MHz         : 1795.537
cache size      : 4096 KB
physical id     : 0
siblings        : 4
core id         : 1
cpu cores       : 4
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm constant_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr lahf_lm
bogomips        : 4691.58
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 2
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Core(TM)2 Quad CPU           @ 2.40GHz
stepping        : 7
cpu MHz         : 1795.537
cache size      : 4096 KB
physical id     : 0
siblings        : 4
core id         : 2
cpu cores       : 4
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm constant_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr lahf_lm
bogomips        : 4787.31
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 3
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Core(TM)2 Quad CPU           @ 2.40GHz
stepping        : 7
cpu MHz         : 1795.537
cache size      : 4096 KB
physical id     : 0
siblings        : 4
core id         : 3
cpu cores       : 4
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm constant_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr lahf_lm
bogomips        : 4787.32
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

 

[jesse]

root@vps1 [~]# cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Core(TM)2 Quad CPU           @ 2.40GHz
stepping        : 7
cpu MHz         : 1795.537
cache size      : 4096 KB
physical id     : 0
siblings        : 4
core id         : 0
cpu cores       : 4
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm constant_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr lahf_lm
bogomips        : 4790.00
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Core(TM)2 Quad CPU           @ 2.40GHz
stepping        : 7
cpu MHz         : 1795.537
cache size      : 4096 KB
physical id     : 0
siblings        : 4
core id         : 1
cpu cores       : 4
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm constant_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr lahf_lm
bogomips        : 4691.58
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 2
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Core(TM)2 Quad CPU           @ 2.40GHz
stepping        : 7
cpu MHz         : 1795.537
cache size      : 4096 KB
physical id     : 0
siblings        : 4
core id         : 2
cpu cores       : 4
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm constant_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr lahf_lm
bogomips        : 4787.31
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 3
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Core(TM)2 Quad CPU           @ 2.40GHz
stepping        : 7
cpu MHz         : 1795.537
cache size      : 4096 KB
physical id     : 0
siblings        : 4
core id         : 3
cpu cores       : 4
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm constant_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr lahf_lm
bogomips        : 4787.32
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

That explains why you were seeing CPU percentages greater than 100%

 

[RJM62]

DOH!

I guess that explains it...

 

[RJM62]

Good Lord, what a mess that was. By the end of the day, I'd deleted about 400,000 frozen spams from the Exim queue. I found a nice little WHM plugin to do it, too: http://www.configserver.com/cp/cmq.html .

I don't know exactly what stopped the flood. I installed CSF, but that didn't seem to do anything special, so I uninstalled it. Tried switching from Courier to Dovecot (slow and cranky, IMO), but no change there, either; so I switched back to Courier. I also tightened up the security on all the services (should've done that, anyway; some of the cPanel defaults are atrocious), but found no joy there, either. The spam kept a-coming.

Finally, a little after midnight, I shut down Exim so I could do some needed updates to MySQL and Apache. I was still running MySQL 4.1, but I need MySQL 5 for a site I'm migrating over in the next few days. I had to shut down Exim because it was so busy stopping spam that the Apache rebuild would have taken forever had I left it running.

So I shut down the PHP/MySQL-driven sites, made backups, proceeded with the MySQL upgrade and the Apache rebuild, and waddya know -- the problem disappeared.

Maybe there was something corrupt that was replaced with the rebuild. I dunno. In fact, what I don't know about server administration could fill volumes. I was originally trained in electronics, not IT; and my greatest asset is the humility to listen to people smarter than I am.

Whatever the case, I'm sure glad it's over. Thanks for all your help.

-Rich

 

[jesse]

Tried switching from Courier to Dovecot (slow and cranky, IMO)

What problem did you have with Dovecot? I'm in the works of migrating a 2,000 domain, 10,000-ish user installation to Dovecot.

 

[RJM62]

What problem did you have with Dovecot? I'm in the works of migrating a 2,000 domain, 10,000-ish user installation to Dovecot.

It was a lot slower to connect than Courier and timed out on maybe 10 percent of my POP requests. Of course, the VPS wasn't in the best shape at the moment, so it probably wasn't a fair test.

-Rich

 

[jesse]

It was a lot slower to connect than Courier and timed out on maybe 10 percent of my POP requests. Of course, the VPS wasn't in the best shape at the moment, so it probably wasn't a fair test.

-Rich

Yeah. Sounds like something wasn't right. Dovecot *should* be quite fast if everything is working properly.

 

[jesse]

And that's why spammers are using bounce messages to attack.... (I use SpamAssassin).

Spamhaus is a great *first* line of defense. In large e-mail installations your almost a fool for not using it. Spam processing with something like SpamAssassin tends to be very CPU intensive.

After the first layer, Spamhaus, it then makes sense to use something like SpamAssassin. I can tell you that false-positives on Spamhaus are *extremely* rare.

SOME of the SMTP products will check the IP list and generate a 5.5.0 or 5.3.x bounce message. This really should be turned off. Just blackhole the spam and be done with it.

If you use an IP filter in front of the SMTP server, no bounce will be generated, the connection will be refused.

What do you mean by 'blackhole' the spam? The best method is to die out within the SMTP connection with an appropriate error at a good moment in time (right after the RCPT TO, for example). I am *not* a fan of those that just drop the entire network connection like there was a network error.

If the receiving mail server rejects the message with a 5.x.x code, then it isn't generating a bounce message at all, it is simply telling the sending server "this failed, and don't bother retrying". The sending mail server may choose to react to this rejection by originating a bounce message towards the sender.
-harry

Correct

I meant generically that it was a bounce message as the terms are lumped together. In 95+% of the cases the sender gets a bounce from a 5.x.x.

That is up to the sending mail server. Back-scatter can be a problem but so can just randomly dropping SMTP connections. In fact, that act in itself, will generate a bounce on many sending mail servers.

You really need to die within the bounds of the SMTP transaction. No reason to let the connection get to the DATA portion though. If you just drop the connection it becomes a major PITA for all parties to troubleshoot when someone starts to complain that they didn't receive an e-mail.

 

[wsuffa]

Spamhaus is a great *first* line of defense. In large e-mail installations your almost a fool for not using it. Spam processing with something like SpamAssassin tends to be very CPU intensive.

After the first layer, Spamhaus, it then makes sense to use something like SpamAssassin. I can tell you that false-positives on Spamhaus are *extremely* rare.

SpamAssassin incorporates Spamhouse RBL. A good second-level firewall (a box unto itself) can run SpamAssassin just fine with little degradation.

What do you mean by 'blackhole' the spam? The best method is to die out within the SMTP connection with an appropriate error at a good moment in time (right after the RCPT TO, for example). I am *not* a fan of those that just drop the entire network connection like there was a network error.

Accept and send the message to the equivalent of "/dev/null" with no error generated.

I do like the tarpit programs, though.

That is up to the sending mail server. Back-scatter can be a problem but so can just randomly dropping SMTP connections. In fact, that act in itself, will generate a bounce on many sending mail servers.

You really need to die within the bounds of the SMTP transaction. No reason to let the connection get to the DATA portion though. If you just drop the connection it becomes a major PITA for all parties to troubleshoot when someone starts to complain that they didn't receive an e-mail.

By dropping the IP, I mean outright refusing connections from certain IP/IP blocks. An SMTP connection never occurs. No backscatter issue. And yes, I've implemented that function on a couple of really "bad" IP blocks. Sorta doing an IDP within your own domain.

 

[mantakos]

Accept and send the message to the equivalent of "/dev/null" with no error generated.

The downside, of course, is that you have to eat the bandwidth to swallow the message. The other is that identification of "spam" is imperfect, and to disappear an email message completely leaves no evidence for the sender (who may simply have the misfortune of being named by his parents "Spamuel Bukkake Viagra Jr.", back when those words didn't mean anything) that his message was discarded.

Those who arrived to the internet in the past 12-14 years or so, in the era of spam filtering and Microsoft email servers, view email as an unreliable communication medium. You send an email, then call the person to see if they got it. It never used to be like that, it used to be viewed as completely unacceptable for an email message to just "get lost" silently, and that's how things were really implemented.

That notion seems to have gone by the wayside, but I think it's not one to be discarded lightly.
-harry

 

[ReverendSlappy]

Good Lord, what a mess that was. By the end of the day, I'd deleted about 400,000 frozen spams from the Exim queue. I found a nice little WHM plugin to do it, too: http://www.configserver.com/cp/cmq.html .

I don't know exactly what stopped the flood. I installed CSF, but that didn't seem to do anything special, so I uninstalled it. Tried switching from Courier to Dovecot (slow and cranky, IMO), but no change there, either; so I switched back to Courier. I also tightened up the security on all the services (should've done that, anyway; some of the cPanel defaults are atrocious), but found no joy there, either. The spam kept a-coming.

Finally, a little after midnight, I shut down Exim so I could do some needed updates to MySQL and Apache. I was still running MySQL 4.1, but I need MySQL 5 for a site I'm migrating over in the next few days. I had to shut down Exim because it was so busy stopping spam that the Apache rebuild would have taken forever had I left it running.

So I shut down the PHP/MySQL-driven sites, made backups, proceeded with the MySQL upgrade and the Apache rebuild, and waddya know -- the problem disappeared.

Maybe there was something corrupt that was replaced with the rebuild. I dunno. In fact, what I don't know about server administration could fill volumes. I was originally trained in electronics, not IT; and my greatest asset is the humility to listen to people smarter than I am.

Whatever the case, I'm sure glad it's over. Thanks for all your help.

-Rich

I run my personal email at home. A while back (I think it was early last year), I noticed an apparent drop in download speeds, and after looking at a few things I noticed that my Exchange server was grinding a little more than normal. Sure enough, somebody had apparently decided to try to use me as a relay, to the tune of a couple hundred thousand messages a day. But I didn't really pay too much attention to it; obviously I don't relay and I don't respond on rejected messages, so bounce-back spam wasn't happening. And a check of my queues made it clear I wasn't sending anything anywhere. But I couldn't figure out two things: 1) Why try to use me? It should've been immediately clear to whomever/whatever application was trying that it was pointless... I mean an SMTP 550 or whatever message is pretty unambiguous. And 2) what the hell could I do to actually stop it anyway? Shunning IPs at my firewall is pointless.

And so it went for 3 or 4 days... I was consistently losing a few KB/sec of download speed (and I only have a 1.5M DSL so every little bit counts) while watching my Exchange server reject relay attempts at a rate of 2-3 per second. So then one day the power was out in my part of the building for about 5 or 6 hours while I was at work. I got home, powered everything back up and... No more relay attempts. They completely stopped: Right before the power failure the 550s were still streaming into the log file at 2-3 per second, and then -- wham, nothing. And I haven't really had a problem since. So maybe shunning might've been a good idea.

So anyway, what I'm getting at is asking a question, out of curiosity: Could it be that your downtime caused it to stop? Or is it still coming in and it's just getting filtered better now?

 

[jesse]

SpamAssassin incorporates Spamhouse RBL. A good second-level firewall (a box unto itself) can run SpamAssassin just fine with little degradation.

Once you start getting into *very* large quantities of mail it takes significant CPU to process spam (think ISP). It'll generally require a cluster of SpamAssassin machines.

Accept and send the message to the equivalent of "/dev/null" with no error generated.

Sigh. It would probably work in small installations but people doing that kind of crap makes e-mail extremely hard to troubleshoot.

If you accept the message you need to have *something* happen with that message. If it is determined to be spam after you accept you should move it into some sort of quarantine and notify the receiving user in a report. Don't just /dev/null a message like it never happened.

Just because spam is a problem doesn't mean that you should completely cripple e-mail and remove all the things that were built into it to ensure reliable delivery. I can ensure you that doing this sort of stuff is completely unacceptable in large e-mail installations. You would cause major support headaches and **** off those of us that support large e-mail installations because we have no idea what happened when you just report success and /dev/null it.

A support department would get overwhelmed with calls by people wanting to know why they never received a message. The senders IT department would think their message was successful because you didn't report otherwise.

The first layer of your spam protection, Spamhaus, should report failure if it rejects the address. On the next layer (SpamAssassin) you may have to accept the message so that you can process it. If you determine the message to be spam it should be noted in a report that goes to the receving user so that they will know why the message wasn't received.

Any sort of check that you do where it is possible to report error and die in the SMTP transaction should be done instead of dropping it.

By dropping the IP, I mean outright refusing connections from certain IP/IP blocks. An SMTP connection never occurs. No backscatter issue. And yes, I've implemented that function on a couple of really "bad" IP blocks. Sorta doing an IDP within your own domain.

This will still generate a bounce on the sending mail server which can flood users if they forged the from address. The sending mail server knows that you dropped its network connection and knows that it never delivered the message. It will send a report of that event to the from address.

Basically, all I'm saying, is keep a paper trail and make sure that the receiving user is capable of knowing why a message was declined and *IF* you accepted that message they should be able to retrieve it.

 

[wsuffa]

Ya know, Jesse, I understand the implications.

Frankly, I'm tired of the 30,000 (and counting) backscatter messages that took down a server. The only reasonable way to deal with something like that is to /dev/null the error messages.

Likewise, I have zero, zip, nada compassion for spam servers in China and elsewhere that send enormous amounts of spam. If one handles little or no legit traffic from those servers, I have no issue with "blackholing" or /dev/nulling the incoming *from those servers*. It may not be a legit technique for *all* addresses (especially the dynblocks), but it is for something that has 99.9999999% of the messages as spam.

Don't even start about troubleshooting email. The likes of AT&T (Prodigy) and SBC have irritated me by their unwillingness to remove my business email server from their private block lists. The business email server in question is under our full control sends about 200 legit emails a day, and is outright incapable (bandwidth limitation) of being a spam source. Yet AT&T had the IP blocked privately, which required days of aggravation to remove (and nearly cost me a business consulting client) and SBC has been outright unresponsive to 6 months of requests... SBC outright won't respond to our requests, not even an acknowledgement, despite the fact that one of our lead subcontractors (who is an ex-Microsoft forensics employee) is on their system and has similarly made requests.

For all intents and purposes, there is no "troubleshooting" - just outright arrogance. YOU may be responsive - but others aren't.

SpamAssassin may not be the right solution for a Yahoo, Gmail, or even a university. It is one of many solutions for others.

 

[jesse]

...all from Taiwan, all bearing spoofed Yahoo addresses, and all aimed at my server. Why? I have no idea. I have no open relays or other vulnerabilities that I know of, nor can the hosting company find any.

Apparently, however, the attempts actually started before I took possession of the IP addresses last week, and have been increasing steadily. I hadn't noticed because the machine has plenty of resources and was running pretty well.

Today, however, it reached a crescendo; and the server was so busy sending bounce messages (using 105 percent of CPU capacity and almost 2 GB of RAM) that services started crashing on both the master and the slave servers.

I tweaked a few settings in Exim, blocked the IP range, and disabled bouncing, and things settled down. What an annoyance, though.

-Rich

Back to the original subject...

Richard, What was the RCPT TO: user/domain on these e-mails? Was it a domain and user that you host e-mail for? I am confused about you sending bounces. Why were the messages bouncing? If you don't host for that user/domain you should have never accepted the message in the first place.