Hi
Tue, 9 May 2006, Ketil Froyn wrote:
Jim Holland wrote:
I have been running dnscache for over 2 years now with no problems at all
(thanks DJB!) on a mail server. Recently however there are times when
suddenly dnscache seems to generate huge amounts of traffic that block our
64K connection completely. this starts it continues indefinitely.
Perhaps you are running some log analysis programs?
No.
How regular does this happen? a week? a day? Have you
installed some programs recently or changed some configurations
recently?
It has gone from very occasionally to once a week, to almost all the time
now. In fact I have had to abandon dnscache completely and now run no
caching nameservice at all. The only new program installed recently was
chkrootkit-0.46a downloaded from www.chkrootkit.org. It was just run a
couple of times to check the system (nothing was in fact found). I also
updated recently to ClamAV version 0.88.2 from an older version. Each
time I checked the signatures before installing.
Also, who (what IP) sends the queries to dnscache? You should find that
in your logs too, and it should help track down the offender:
http://dqd.com/~#query
As indicated previously, the problem continues even if all other machines
are disconnected from the network. dnscache is listening only on a
private IP address, and this was confirmed by the logs. All queries came
from the server itself - nothing from outside. I couldn't see anything
unusual about them - they were basically queries for PTR records, A
records or MX records for systems that were sending mail to us or which we
were sending mail to. The system is a mail server that is configured to do
quite a lot of DNS queries - checking that connecting systems have PTR
records, that these PTR records point to valid hostnames, as well as doing
DNS lookups on three different DNS spam blacklists for all received mail
before final delivery. The server handles 15 to 20k of Internet messages
per day.
What is curious is that when I stop running dnscache and the system
reverts to forwarding queries to the caching nameservers listed in
/etc/resolv.conf then the total DNS traffic load drops considerably - both
in normal times and especially when this particular problem arises. This
opinion is based on using iptraf to monitor traffic on our very narrow
(64K) pipe.
CACHESIZE:10000000
DATALIMIT:10485760
If you run log analysis software that does lots of DNS queries you might
want to experiment with bigger cache size:
Thanks for the tip.
FRWARDNLY:0
Beware that FRWARDNLY is active even if it is set to 0. You haven't
said whether you want dnscache to forward queries to another cache or
not. What's in your /etc/dnscache/root/servers/@ file? From
Versions 1.03 and above: If $FRWARDNLY is set, dnscache treats
servers/@ as a list of IP addresses for other caches, not root
servers.
The @ file contains the current IP addresses of a.root-servers.net through
to m.root-servers.net and nothing else. @.forwarding contains a single IP
address for a nearby caching nameserver, but I would presume that with
FRWARDNLY set to 0 this will be ignored anyway.
Thanks for your thoughtful suggestions, but it still seems to be a bit of
a mystery. I was using dnscache not only for improving bandwidth
utilisation but also to ensure that I did not have to rely on neighboring
caching nameservers that I didn't necessarily trust. I would have been
happy to use it just for the security benefit. However if it dramatically
worsens bandwidth utilisation then clearly it is not usable any more.
Regards
Jim Holland
System Administrator
MANG - Zimbabwe's non-profit e-mail service

Jim Holland wrote:
Hi

Tue, 9 May 2006, Ketil Froyn wrote:


>How regular does this happen? a week? a day? Have you
>installed some programs recently or changed some configurations
>recently?
>
>
It has gone from very occasionally to once a week, to almost all the time
now. In fact I have had to abandon dnscache completely and now run no
caching nameservice at all. The only new program installed recently was
chkrootkit-0.46a downloaded from www.chkrootkit.org. It was just run a
couple of times to check the system (nothing was in fact found). I also
updated recently to ClamAV version 0.88.2 from an older version. Each
time I checked the signatures before installing.


>Also, who (what IP) sends the queries to dnscache? You should find that
>in your logs too, and it should help track down the offender:
>>
>http://dqd.com/~#query
>
>
As indicated previously, the problem continues even if all other machines
are disconnected from the network. dnscache is listening only on a
private IP address, and this was confirmed by the logs. All queries came
from the server itself - nothing from outside. I couldn't see anything
unusual about them - they were basically queries for PTR records, A
records or MX records for systems that were sending mail to us or which we
were sending mail to. The system is a mail server that is configured to do
quite a lot of DNS queries - checking that connecting systems have PTR
records, that these PTR records point to valid hostnames, as well as doing
DNS lookups on three different DNS spam blacklists for all received mail
before final delivery. The server handles 15 to 20k of Internet messages
per day.

Ah, so it's just reverse-lookups of the mail-server IPs that send you mail.
The IPs will probably correspond.

What is curious is that when I stop running dnscache and the system
reverts to forwarding queries to the caching nameservers listed in
/etc/resolv.conf then the total DNS traffic load drops considerably - both
in normal times and especially when this particular problem arises. This
opinion is based on using iptraf to monitor traffic on our very narrow
(64K) pipe.

If the caching nameserver is BIND, that's no surprise.
BIND is a lot slower. Might not be significant with BIND9, but BIND8 is
really several orders of magnitude slower.

FRWARDNLY:0

>Beware that FRWARDNLY is active even if it is set to 0. You haven't
>said whether you want dnscache to forward queries to another cache or
>not. What's in your /etc/dnscache/root/servers/@ file? From
>
>>
>Versions 1.03 and above: If $FRWARDNLY is set, dnscache treats
>servers/@ as a list of IP addresses for other caches, not root
>servers.
>
>
The @ file contains the current IP addresses of a.root-servers.net through
to m.root-servers.net and nothing else. @.forwarding contains a single IP
address for a nearby caching nameserver, but I would presume that with
FRWARDNLY set to 0 this will be ignored anyway.

No, it won't.
Please believe us.
Put the nearest forwarding-servers in the @ file.

Thanks for your thoughtful suggestions, but it still seems to be a bit of
a mystery. I was using dnscache not only for improving bandwidth
utilisation but also to ensure that I did not have to rely on neighboring
caching nameservers that I didn't necessarily trust. I would have been
happy to use it just for the security benefit. However if it dramatically
worsens bandwidth utilisation then clearly it is not usable any more.

The "problem" is that by using DNSCACHE, you will be able to answer a
lot more queries per second and thus accept a lot more SMTP-connections
per second.
, other bottlenecks in the system start to show once BIND out of
the loop.

I acknowledge the fact that Zimbabwe is not the world's
bandwidth-capital, but 64k is a pretty thin line, for todays standards -
and that's mostly the fault of people who send large mail-attachements
and spammers.

- and you should run a firewall that can prioritize DNS-traffic.

cheers,
Rainer

Jim Holland wrote:
As indicated previously, the problem continues even if all other machines
are disconnected from the network. dnscache is listening only on a
private IP address, and this was confirmed by the logs. All queries came
from the server itself - nothing from outside.

I think we've established that dnscache is not at fault here. Something
else is creating the queries. A dnscache that forwards all queries, or
just sending queries to a resolver outside your network, will by nature
use less bandwith than having a dnscache that actually does all the
resolution, and I think this is the root of your problem.

I suggest you get to the bottom of the cause of all the queries, and
stop them. In addition you could increase your cache size. If records
are being expired earlier than necessary because the cache is full,
extra work is being done by dnscache.

Good luck.

Cheers,
Ketil Froyn
ketil (AT) froyn (DOT) name
http://ketil.froyn.name/