HTTP URL filtering?

Yes, I'd recommend pf. If you've never worked with it before,
the PF section of the FAQ is an excellent starting point.

Xavier Mertens wrote:
Hi *,

I've a problem with an Apache web server hit by f*cking spammers
I would like to filter some URLs (unused but still used by the bots) *BEFRE* they reach the httpd processes. What could be the best method? pf? something else?

Thanks!

Xavier

Tue, Feb 06, 2007 at 10:50:36PM +0100, Xavier Mertens wrote:
Hi *,

I've a problem with an Apache web server hit by f*cking spammers
I would like to filter some URLs (unused but still used by the bots) *BEFRE* they reach the httpd processes. What could be the best method? pf? something else?

You need an application layer filter, not an IP layer.

There are several ways to do this, try squid as a reverse proxy:

06/02/07, Xavier Mertens <xavier (AT) rootshell (DOT) bewrote:
Hi *,

I've a problem with an Apache web server hit by f*cking spammers
I would like to filter some URLs (unused but still used by the bots) *BEFRE* they reach the httpd processes. What could be the best method? pf? something else?

Thanks!

I haven't played with it myself, and don't know if it even compiles on
BSD as it was written for FreeBSD and then ported to Linux, but I
think nginx would be something that would be more than appropriate in
such an occasion.

http://nginx.net/

To my knowledge, nearly half of Russian high-traffic web-sites utilise
this thing for various http traffic routing purposes

Cheers,
Constantine.

Xavier Mertens wrote:
I've a problem with an Apache web server hit by f*cking spammers
I would like to filter some URLs (unused but still used by the bots) *BEFRE* they reach the httpd processes. What could be the best method? pf? something else?

PF doesn't look at URL content, so can't be use to filter that type of
stuff.

What I do use extensively however that does a great job is the rewrite
module for virus attack and the like and I also put two files for the
bot that crawler sites with the proper directory not to scan and all
that do not respect that get block in PF after they access that file as
I log that in sql and pass the data to all servers.

I also put URL in the copyright section of my sites to point to the
forbidden directory of the bots and as such each low life that import my
site without rights, get also block via PF!

I do catch many bad bots that do not act properly and it's nice to see.

So, it's a combination of things.

Here is some extract for example of httpd.conf for the rewrite module.
Then you add and removed them as you see fit and new one coming up.

You can also limit the number of connections from the same IP address
with pf as well for very abusive crawlers, or bad DDoS guys, etc.

Hope this give you some ideas anyway.

Best,

Daniel

Special section for stopping bad traffic and DDoS attack type, etc
# This list are all global rewrite rules that can be turn on per virtual
server.
RewriteCond %{HTTP_REFERER} ^$
RewriteCond %{HTTP_USER_AGENT} ^$
RewriteCond %{REQUEST_URI} ^/$
RewriteRule ^/.* http://%{REMTE_ADDR}/
[L,E=nolog:1]
RewriteRule (.*)cmd.exe(.*)$ http://%{REMTE_ADDR}/
[L,E=nolog:1]
RewriteRule (.*)root.exe(.*)$ http://%{REMTE_ADDR}/
[L,E=nolog:1]
RewriteRule (.*)shell.exe(.*)$ http://%{REMTE_ADDR}/
[L,E=nolog:1]
RewriteRule (.*)\/_vti_bin\/(.*)$ http://%{REMTE_ADDR}/
[L,E=nolog:1]
RewriteRule (.*)\/_vti_cnf\/(.*)$ http://%{REMTE_ADDR}/
[L,E=nolog:1]
RewriteRule (.*)\/_vti_inf\/(.*)$ http://%{REMTE_ADDR}/
[L,E=nolog:1]
RewriteRule (.*)\/_vti_inf.html http://%{REMTE_ADDR}/
[L,E=nolog:1]
RewriteRule (.*)\/scripts\/\.\.(.*)$ http://%{REMTE_ADDR}/
[L,E=nolog:1]
RewriteRule (.*)\/_mem_bin\/(.*)$ http://%{REMTE_ADDR}/
[L,E=nolog:1]
RewriteRule (.*)\/msadc\/(.*)$ http://%{REMTE_ADDR}/
[NC,L,E=nolog:1]
RewriteRule (.*)\/MS/cltreq.asp(.*)$ http://%{REMTE_ADDR}/
[L,E=nolog:1]
RewriteRule (.*)\/c\/winnt\/(.*)$ http://%{REMTE_ADDR}/
[L,E=nolog:1]
RewriteRule (.*)\/d\/winnt\/(.*)$ http://%{REMTE_ADDR}/
[L,E=nolog:1]
RewriteRule (.*)\/x80\/(.*)$ http://%{REMTE_ADDR}/
[L,E=nolog:1]
RewriteRule (.*)\/x90\/(.*)$ http://%{REMTE_ADDR}/
[L,E=nolog:1]

2/6/07, Xavier Mertens <xavier (AT) rootshell (DOT) bewrote:
Hi *,

I've a problem with an Apache web server hit by f*cking spammers
I would like to filter some URLs (unused but still used by the bots)
*BEFRE* they reach the httpd processes. What could be the
best method? pf? something else?

I used snort to filter before httpd to build simple IP address lists
to feed into a pf table. It was kinda clunky. Second time
around I'd just parse my httpd log files and do the same thing.
With apache configured right and a cron running every minute
you'll get by with minimal work needed. I'd imagine.

Sorry I should have read the original more carefully before replying.
For some reason I thought you wanted to filter by source address. PF is
not the way you'd want to go for filtering based on the HTTP GET request.

Jonathan Whiteman wrote:
Yes, I'd recommend pf. If you've never worked with it before,
the PF section of the FAQ is an excellent starting point.

Xavier Mertens wrote:
>Hi *,
>>
>I've a problem with an Apache web server hit by f*cking spammers
>I would like to filter some URLs (unused but still used by the bots)
>*BEFRE* they reach the httpd processes. What could be the best
>method? pf? something else?
>>
>Thanks!
>>
>>
>Xavier

I use mod_security for filtering. Take a look at http://www.modsecurity.org/

You can find it in the ports system undes www/mod_security

Good luck

Andrei GUDIU

Xavier Mertens wrote:

Hi *,

I've a problem with an Apache web server hit by f*cking spammers
I would like to filter some URLs (unused but still used by the bots) *BEFRE* they reach the httpd processes. What could be the best method? pf? something else?

Thanks!

Xavier

Hello Xavier,

Tue, 06.02.2007 at 22:50:36 +0100, Xavier Mertens <xavier (AT) rootshell (DOT) bewrote:
I've a problem with an Apache web server hit by f*cking spammers
I would like to filter some URLs (unused but still used by the bots)
*BEFRE* they reach the httpd processes. What could be the best
method? pf? something else?

I guess that you want to keep the load off your Apache, right?

I'd also vote for a lightweight front-end reverse proxy like nginx
(already mentioned) or lighttpd to do this.

Best,
++

Hej there,

Xavier Mertens schrieb:
Hi *,

I've a problem with an Apache web server hit by f*cking spammers
I would like to filter some URLs (unused but still used by the bots) *BEFRE* they reach the httpd processes. What could be the best method? pf? something else?

I had the same problem with botnets, attacking a specific URL. Even
sending out 404 errors didn't help at all.
I wouldn't recommend the pf overload feature, as this depends on the
number of tcp connections to your webserver.
Say you have a webpage with 50 images, this would be 50 connections.
Another webpage may only have 2 images, this would lead to only 2
connections.
Here is what I did.
Install mod_security for apache.
Define rules like those:
<IfModule security2_module>

# Maximum request body size we will
# accept for buffering
SecRequestBodyAccess
#SecRequestBodyLimit 131072
# Store up to 128 KB in memory
#SecRequestBodyInMemoryLimit 131072

# Buffer response bodies of up to
# 512 KB in length
SecResponseBodyAccess
SecResponseBodyLimit 524288

# Debug log
SecDebugLog /
SecDebugLogLevel 0

# The audit engine works independently and
# can be turned of on the per-server or
# on the per-directory basis
#SecAuditEngine
SecAuditEngine R
SecAuditLogRelevantStatus ^5
#SecAuditLogParts ABIFHZ
SecAuditLogParts A
SecAuditLogType Serial

# The name of the audit log file
SecAuditLog /

# Default action set
#SecDefaultAction "deny,log,auditlog,status:403"

# Turn on Rule Engine
SecRuleEngine

# Refuse to accept PST requests that do
# not specify request body length
# SecRule REQUEST_METHD ^PST$ chain
# SecRule REQUEST_HEADER:Content-Length ^$
#
# Metal District Rules
#SecRule REQUEST_URI "/phpbb2/posting\.php\(.*\)"
"deny,phase:1,"
#SecRule ARGS /phpbb2/posting.php
"deny,phase:1,"
SecRule REQUEST_FILENAME /phpbb2/posting.php
"deny,phase:1,"
SecRule REQUEST_FILENAME /phpBB2/posting.php
"deny,phase:1,"

</IfModule>

Anytime someone is accessing /phpbb2/posting.php the script
fill-blacklist.sh is run:

([rabauke@beast] <~$ cat /root/bin/fill-blacklist.sh
#!/bin/sh
#
sudo pfctl -T add -t www-spammers $(echo ${REMTE_ADDR})
echo "${REMTE_ADDR} added to blacklist"

The ip gets added to the table www-spammers.
My pf rules look like that:
# www-spammers table
table <www-spammerspersist file "/etc/www-spammers"
block in quick on $ext_if proto tcp from <www-spammersto $ext_if port 80

Drawback: I need sudo to use pfctl as the user www (which apache runs
under).
Pro: Every bot can access the url exactly one time, afterwards its
blacklisted.
Use expire-table to free the pf table occassionally and of course make
sure that you don't block yourself - whitelist ip addresses like your
standard gateway, otherwise you may DoS yourself ;)

course this is just a hack, but it works in my case.
Any suggestions to improve this setup are welcome :)

best regards,
Marian

Hi,

Karsten McMinn schrieb:
2/6/07, Xavier Mertens <xavier (AT) rootshell (DOT) bewrote:
>Hi *,
>>
>I've a problem with an Apache web server hit by f*cking spammers
>I would like to filter some URLs (unused but still used by the bots)
>*BEFRE* they reach the httpd processes. What could be the
>best method? pf? something else?

I used snort to filter before httpd to build simple IP address lists
to feed into a pf table. It was kinda clunky. Second time
around I'd just parse my httpd log files and do the same thing.
With apache configured right and a cron running every minute
you'll get by with minimal work needed. I'd imagine.

I tried the very same when a webserver of mine was hitted by some
botnet. Unluckily, cron can only ran every minute as the fastest
interval and within 1 minute I already had around 1000 connections from
different IP addresses.
Ergo: A one minute interval didn't help at all

/Marian

Marian Hettwer wrote:
I tried the very same when a webserver of mine was hitted by some
botnet. Unluckily, cron can only ran every minute as the fastest
interval and within 1 minute I already had around 1000 connections from
different IP addresses.
Ergo: A one minute interval didn't help at all

I had and time to time still have attack like this and put together a
series of effective measure to take care of this. Some I explain and put
together on misc@ under the title:

Feedback wanted on gethttpd graylisting ideas included

So you can search marc for:

I also posted a few more things, but it is possible to control that.

I added many more things as well and here if you have URL not use what
you can do is actually may be very simple and effective right away as
well using PF and redirect if the connections are from source that
either will redirect or not.

What I did for example for source that do not redirect, or follow the
standard. If you connect to let say a URL

a.b.c/test.html

and that test.html is a huge page that many bots actually love to attack
to make you waist bandwidth and put your server to a crawl. What I did
is simply to have that page send a redirect right away and then close
the connection. So, any valid users that access that page will be
redirected to the valid page and the bot will simply have it;s
connection close. So, yes you still process all the connection, but the
handling from the server is pretty small. Just a few bytes. Also, that
same connection is logs into SQL server that I query from cron and add
to PF each minutes. Yes I need to handle all the connections for that
minute like you said, but the traffic is very minimal and before you
know it, the source is block. Then I also have built my scripts to
refresh the block IP's with time out, meaning that I wanted to be nice
and the source IP's where block in incremental time each time they were
process. So, if the source go away and was from a valid proxy from AL
for example, I wasn't going to loose the traffic for ever, but the
traffic for the time of the attack. And in the end, all the connections
that were following the redirect were process normally. That's because
DDoS bot attack so far call URL via GET and doesn't check for the return
code, so they were never going to the redirect new location and were
block later on.

Now for crawlers that follow bad URL or attack bad URL, you can here as
well put redirect to a different port.

Like a.b.c/follow.html would redirect to a.b.c:81/follow.html and then
you simply use PF to add right away all source trying to connect to
tcp/81 to your table and be done with them. That's also quick and simple
to do as well.

Anyway, that's just some idea that are fast efficient and proved to work
very well thank you.

I have more in place as well, ut if you do just these you will see light
at the end of the tunnel.

Best,

Daniel

Hi Daniel,

Wed, 07.02.2007 at 16:26:55 -0500, Daniel <daniel (AT) presscom (DOT) netwrote:
the connection. So, any valid users that access that page will be
redirected to the valid page and the bot will simply have it;s
connection close.

this requires that you control the URLs being served (ie, static
content & cooperation with the site owner), right?

Best,
++

Hi,

Wed, 07.02.2007 at 19:08:46 +0100, Marian Hettwer <MH (AT) kernel32 (DOT) dewrote:
I had the same problem with botnets, attacking a specific URL. Even
sending out 404 errors didn't help at all.
I wouldn't recommend the pf overload feature, as this depends on the
number of tcp connections to your webserver.

[ mod_security ]

Anytime someone is accessing /phpbb2/posting.php the script
fill-blacklist.sh is run:

([rabauke@beast] <~$ cat /root/bin/fill-blacklist.sh

and this doesn't dos the server? I guess in the case you mentioned,
this script must be run _very_ often.

Pro: Every bot can access the url exactly one time, afterwards its
blacklisted.
Use expire-table to free the pf table occassionally and of course make
sure that you don't block yourself - whitelist ip addresses like your
standard gateway, otherwise you may DoS yourself ;)

I'm researching the same problem and so far have arrived at the
following conclusions (feedback & improvement desired!):

* Blacklisting individual IPs is a sharp edged knife, and cumbersome
to handle.
* Some request storms appear to be triggered by a unlucky interaction
between the server sending PDF files, and the client using Internet
Exploder (which often breaks, see the discussion around
range-requests).
* Use a non-forking server.
* Rate limiting, or at least rate limiting per network (eg. per /16),
would "solve" the problem for me, and is maintenance-free.
* Use it with connection rate limiting in pf

Any comments on this are welcome!

obvious downside is that one apparently cannot make this work (eg
specifically denying range-requests from IE-users) with the stock
Apache.

Best,
++

Toni Mueller wrote:
>Pro: Every bot can access the url exactly one time, afterwards its
>blacklisted.
>Use expire-table to free the pf table occassionally and of course make
>sure that you don't block yourself - whitelist ip addresses like your
>standard gateway, otherwise you may DoS yourself ;)

I'm researching the same problem and so far have arrived at the
following conclusions (feedback & improvement desired!):

* Blacklisting individual IPs is a sharp edged knife, and cumbersome
to handle.

Not really when done automatically. I use incremental time per offense.
First time you do it, you are block for a period of time, then remove
from the lists later on. You do it again, you are block for more time
then clear again, etc. Works very well for me and I can share the same
SQL data between all servers.

* Some request storms appear to be triggered by a unlucky interaction
between the server sending PDF files, and the client using Internet
Exploder (which often breaks, see the discussion around
range-requests).
* Use a non-forking server.

?

* Rate limiting, or at least rate limiting per network (eg. per /16),
would "solve" the problem for me, and is maintenance-free.
* Use it with connection rate limiting in pf

PF can handle rate limit pretty well, just increase your table size if
you reach the limit of them and be aggressive optimization:

Start in PF with :

set optimization aggressive

Any comments on this are welcome!

obvious downside is that one apparently cannot make this work (eg
specifically denying range-requests from IE-users) with the stock
Apache.

You can deny request based on IE versions if need be from the stock
apache. All my previously describe time limiting and redirect is only
affecting the IE version and anything NT IE pass without delay or redirect.

Daniel

Hi,

Tue, 20.02.2007 at 12:33:17 -0500, Daniel <daniel (AT) presscom (DOT) netwrote:
* Use a non-forking server.
?

I've been hit by guys who simply exhausted the maximum number of
processes I configured with Apache. What limits do you usually have?

This was effectively a DoS against all other users. Whether
intentionally or by accident, I can't say right now, but all
non-forking servers I'm aware of have a much slower memory consumption
rate.

PF can handle rate limit pretty well, just increase your table size if
you reach the limit of them and be aggressive optimization:

Yes. I was already using it with good success.

You can deny request based on IE versions if need be from the stock
apache. All my previously describe time limiting and redirect is only
affecting the IE version and anything NT IE pass without delay or redirect.

I don't want to generally deny, or slow down, IE users of the site (I
can't), but only want to deny them range requests. I didn't find a knob
in Apache to do this. If anyone else does, I'm still interested.

Best,
++

Toni Mueller wrote:
Hi,

Tue, 20.02.2007 at 12:33:17 -0500, Daniel <daniel (AT) presscom (DOT) netwrote:
* Use a non-forking server.
>?

I've been hit by guys who simply exhausted the maximum number of
processes I configured with Apache. What limits do you usually have?

I am still very conservative and could increase it more, but choose not to.

# sysctl kern.maxproc
kern.maxproc=1024

Never needed to increase more then that after all protections are in
place. I use 768 limit for apache, need to change the default and
recompile, but real easy to do.

This was effectively a DoS against all other users. Whether
intentionally or by accident, I can't say right now, but all
non-forking servers I'm aware of have a much slower memory consumption
rate.

Memory consumption is not really max process, nor forking limits, unless
you run very limited memory servers.

Also, one situation describe before as well is that DDoS send lots of
connections to your server, but they do not process the content of the
reply, meaning they don't call all the images, etc that a regular
requests would do. Example one request from IE on a site can results in
20 stream back from the server as one attack still get one stream in
reply. So, with proper limits on connection and blocking in place, you
are good to go.

May be your KeepAliveTimeout in httpd.conf is to high. I reduce mine
from the default one. This free more forking process sooner.

You can also change the value of:

MinSpareServers
MaxSpareServers

To fit your server needs and reduce the time spend in start/stop new
one. All depend on your traffic.

>PF can handle rate limit pretty well, just increase your table size if
>you reach the limit of them and be aggressive optimization:

Yes. I was already using it with good success.

May be looking and spending some time to optimize your PF rules under
heavy load might be well spend time.

Daniel put a very good guideline on this:

Also for interest, may be these two might be interesting to read as well
for informations:


>You can deny request based on IE versions if need be from the stock
>apache. All my previously describe time limiting and redirect is only
>affecting the IE version and anything NT IE pass without delay or redirect.

I don't want to generally deny, or slow down, IE users of the site (I
can't), but only want to deny them range requests. I didn't find a knob
in Apache to do this. If anyone else does, I'm still interested.

May be I am thick here, I still don't understand what you are after
here. You can block access with:

Allow,Deny
Allow from all
Deny from w.x.y.z/xx

And with something like:
BrowserMatchNoCase "\MSIE 5.5" browser=MSIE_5.5
and SetEnvIf, you can do clever things.

I didn't put a lots of thoughts in the last piece here as I use that for
blocking some specific traffic and the like, but I don't see why it
woudln't be possible.

Then I use something like:

RewriteRule <Your rule herehttp://%{REMTE_ADDR}/ [L,E=nolog:1]

But that needs to be specific for each setup you want to use.

I would need to spend time thinking about it, but I would think it's
possible to do. Not easy, but possible. Some reading is needed.

Best,

Daniel

Toni Mueller schrieb:
Hi,

Wed, 07.02.2007 at 19:08:46 +0100, Marian Hettwer <MH (AT) kernel32 (DOT) dewrote:
>I had the same problem with botnets, attacking a specific URL. Even
>sending out 404 errors didn't help at all.
>I wouldn't recommend the pf overload feature, as this depends on the
>number of tcp connections to your webserver.

>[ mod_security ]

>Anytime someone is accessing /phpbb2/posting.php the script
>fill-blacklist.sh is run:
>>
>([rabauke@beast] <~$ cat /root/bin/fill-blacklist.sh

and this doesn't dos the server? I guess in the case you mentioned,
this script must be run _very_ often.

Nope, it doesn't. In my case, luckily, the script gets triggered (and
it's fast hej, it's just a pfctl run) and every next call from the
offending IP gets blocked by pf. Works like a charm.


>Pro: Every bot can access the url exactly one time, afterwards its
>blacklisted.
>Use expire-table to free the pf table occassionally and of course make
>sure that you don't block yourself - whitelist ip addresses like your
>standard gateway, otherwise you may DoS yourself ;)

I'm researching the same problem and so far have arrived at the
following conclusions (feedback & improvement desired!):

* Blacklisting individual IPs is a sharp edged knife, and cumbersome
to handle.
This won't help you against infected windows boxes around the world.

* Some request storms appear to be triggered by a unlucky interaction
between the server sending PDF files, and the client using Internet
Exploder (which often breaks, see the discussion around
range-requests).
* Use a non-forking server.
Well, if using Apache, I tend to use apache 2.2 with mpm-worker or
mpm-event (experimental).

* Rate limiting, or at least rate limiting per network (eg. per /16),
would "solve" the problem for me, and is maintenance-free.
Really? botnets doesn't use much bandwith. did you mean number of
connections by netblock by time?
Even that wouldn't help at all. Botnets filled up the rate you defined
and no one else can connect from the same netblock.
I wouldn't do this.

* Use it with connection rate limiting in pf

Be aware with rate limiting http connections. Mease the number of tcp
connections before you do that.
HTTP GET isn't equal to one TCP connection to port 80.

Cheers,
Marian

Daniel wrote:
Toni Mueller wrote:
>I don't want to generally deny, or slow down, IE users of the site (I
>can't), but only want to deny them range requests. I didn't find a knob
>in Apache to do this. If anyone else does, I'm still interested.

May be I am thick here, I still don't understand what you are after
here. You can block access with:

Allow,Deny
Allow from all
Deny from w.x.y.z/xx

And with something like:
BrowserMatchNoCase "\MSIE 5.5" browser=MSIE_5.5
and SetEnvIf, you can do clever things.

I didn't put a lots of thoughts in the last piece here as I use that for
blocking some specific traffic and the like, but I don't see why it
woudln't be possible.

Then I use something like:

RewriteRule <Your rule herehttp://%{REMTE_ADDR}/ [L,E=nolog:1]

But that needs to be specific for each setup you want to use.

I would need to spend time thinking about it, but I would think it's
possible to do. Not easy, but possible. Some reading is needed.

Todays post on Undeadly about the Layer-7 SSL load balancer almost
looked as if it could do just this, at least if you could identify
win-machines with the S fingerprinting, and send all IE (hence all
winders users) to a relay that would change so that the server doesn't
announce range capabilities.

Perhaps a long-winded way of "solving" this, if there is no other simple
knob in apache to turn ranges off.