From dhcp-server -- To unsubscribe, see the end of this message.
PGP SIGNED MESSAGE
Hash: SHA1
There are several devices on my network that will occassionaly
misbehave. There are three different manifestations of this bad
behaviour that I have found so far (see below).
I'm wondering whether there is any way that I can detect this behaviour
and then either ignore any further requests from the device, rate limit
it or perhaps try something like sending a NAK to the device? of
these options, I like the idea of sending a NAK as it may be enough to
bring the client back to its senses.
I found the section on "lease events", in the DHCP handbook, and was
wondering whether I could use the commit event to check how recently a
lease was granted for a given client and then, if the lease was only a
couple seconds old, send a NAK to the current request.
Another option is to tail the log file using a perl script that watches
for bad behavior and then resets the customer's modem via SNMP. This
would probably be less impactful (on server performance) than the
previous idea, but it seems a bit kludgy.
Any thoughts?
Bad Behaviour
-
1 - phoebe octopus and Aopen AR401 home routers.
The device will send a discover, receive the offer, then suddenly start
sending requests like mad (many times a second) with the dhcp server
acking each one. This will go on for several minutes until the device
finally accepts the IP and stops requesting - I have no idea why it
stops asking. The logs show one phoebe doing this for 5 min solid and
another one for 14 min solid. The Aopen went for 8 min solid.
2 - belkin home routers
The router requests an IP, gets the offer, then for the next several
hours, it does one request per minute and the dhcp server ACKs each one.
Finally after hours of this, the router will either send a release or
not, but it will definitely do a normal discover, offer, request, ack.
At this point, the router seems sure of itself and doesn't request again
until half way through it's lease period (typical behaviour). However,
when it requests at the halfway point, the problem starts up all over again.
This isn't entirely consistent though. It appears that the device may
decide to arbitrarily release as it did today 9 hours after it's last
request ack cycle started up.
3 - docsis cable modems
Every once in a while, a docsis modem will go squirrelly and start
making several requests per minute until it is reset, then it behaves
itself.
I, unfortunately, still haven't figured out what triggers these events
- --
Mason Schmitt
Systems Administrator
Sunwave Cable Internet / Shuswap Internet Junction
ph: (250) 832-9711
www.sunwave.net
PGP SIGNATURE
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
XI3jutGKyY+lRc=
=Pati
PGP SIGNATURE
List Archives : http://www.isc.org/ops/lists/
Unsubscribe :
-or- : mailto:dhcp-server-request (AT) isc (DOT) org?Subject=unsubscribe

From dhcp-server -- To unsubscribe, see the end of this message.

Mason Schmitt wrote:

>I'm wondering whether there is any way that I can detect this behaviour
>and then either ignore any further requests from the device, rate limit
>it or perhaps try something like sending a NAK to the device? of
>these options, I like the idea of sending a NAK as it may be enough to
>bring the client back to its senses.

Is any use to you ?

It looks like it could be configured to look for the relevant log
messages and block requests from that device for a while.

List Archives : http://www.isc.org/ops/lists/
Unsubscribe :
-or- : mailto:dhcp-server-request (AT) isc (DOT) org?Subject=unsubscribe

From dhcp-server -- To unsubscribe, see the end of this message.

these devices (phoebe, aopen, belkin) can you duplicate the behaviour in
the lab?
Meaning, can you duplicate this by attaching the device directly to the
dhcpd server, (crossover cable).

I would try that. I have had something like this happen to me, and it was
because of a weak link and packets were getting lost.
in my case, a device with a good transmitter, but a bad receiver.

Maybe there is a possibility that the DISCVER comes in, but your FFERs
and/or ACKs are getting lost in the network.

Good Luck,
Duane Cox

Message
From: "Mason Schmitt" <mason.schmitt (AT) sunwave (DOT) net>
To: <dhcp-server (AT) isc (DOT) org>
Sent: Sunday, January 15, 2006 9:04 AM
Subject: possible server responses to bad clients (crappy home routers)

From dhcp-server -- To unsubscribe, see the end of this message.

PGP SIGNED MESSAGE
Hash: SHA1

There are several devices on my network that will occassionaly
misbehave. There are three different manifestations of this bad
behaviour that I have found so far (see below).

I'm wondering whether there is any way that I can detect this behaviour
and then either ignore any further requests from the device, rate limit
it or perhaps try something like sending a NAK to the device? of
these options, I like the idea of sending a NAK as it may be enough to
bring the client back to its senses.

I found the section on "lease events", in the DHCP handbook, and was
wondering whether I could use the commit event to check how recently a
lease was granted for a given client and then, if the lease was only a
couple seconds old, send a NAK to the current request.

Another option is to tail the log file using a perl script that watches
for bad behavior and then resets the customer's modem via SNMP. This
would probably be less impactful (on server performance) than the
previous idea, but it seems a bit kludgy.

Any thoughts?
--
Bad Behaviour
-
1 - phoebe octopus and Aopen AR401 home routers.
The device will send a discover, receive the offer, then suddenly start
sending requests like mad (many times a second) with the dhcp server
acking each one. This will go on for several minutes until the device
finally accepts the IP and stops requesting - I have no idea why it
stops asking. The logs show one phoebe doing this for 5 min solid and
another one for 14 min solid. The Aopen went for 8 min solid.

2 - belkin home routers
The router requests an IP, gets the offer, then for the next several
hours, it does one request per minute and the dhcp server ACKs each one.
Finally after hours of this, the router will either send a release or
not, but it will definitely do a normal discover, offer, request, ack.
At this point, the router seems sure of itself and doesn't request again
until half way through it's lease period (typical behaviour). However,
when it requests at the halfway point, the problem starts up all over
again.

This isn't entirely consistent though. It appears that the device may
decide to arbitrarily release as it did today 9 hours after it's last
request ack cycle started up.

3 - docsis cable modems
Every once in a while, a docsis modem will go squirrelly and start
making several requests per minute until it is reset, then it behaves
itself.

I, unfortunately, still haven't figured out what triggers these events

- --
Mason Schmitt
Systems Administrator
Sunwave Cable Internet / Shuswap Internet Junction
ph: (250) 832-9711
www.sunwave.net
PGP SIGNATURE
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

XI3jutGKyY+lRc=
=Pati
PGP SIGNATURE

List Archives : http://www.isc.org/ops/lists/
Unsubscribe :
-or- : mailto:dhcp-server-request (AT) isc (DOT) org?Subject=unsubscribe

--

List Archives : http://www.isc.org/ops/lists/
Unsubscribe :
-or- : mailto:dhcp-server-request (AT) isc (DOT) org?Subject=unsubscribe

From dhcp-server -- To unsubscribe, see the end of this message.

PGP SIGNED MESSAGE
Hash: SHA1

these devices (phoebe, aopen, belkin) can you duplicate the behaviour in
the lab?
Meaning, can you duplicate this by attaching the device directly to the
dhcpd server, (crossover cable).

If I can get a hold of one, I may try that.

I would try that. I have had something like this happen to me, and it was
because of a weak link and packets were getting lost.
in my case, a device with a good transmitter, but a bad receiver.

Very good point. Never forget layer1 especially when using UDP. I have
encountered many devices on our network that just don't have enough
drive to work with a few of our older modems. I'll look into this.

Maybe there is a possibility that the DISCVER comes in, but your FFERs
and/or ACKs are getting lost in the network.

<rant>
Even if the problem is as a result of a weak nic, or a flakey firewall
that blocks the incoming DHCP requests (this is a constant headache for
us, but mostly with software firewalls on the customer pc), I think the
vendor should be shot for creating a client that behaves in such an
atrocious fashion. DHCP clients are supposed to have a progressive
backoff algorithm, not a machine gun approach to getting an address

The cheap garbage that passes for home network devices is truely
appalling at times. of the most common things that our support
techs have to do is ask the customer to power cycle their router
</rant>

Good Luck,

Thanks :P
And thanks for your feedback, it gives me something new to look into to
resolve the root cause. I still want to find some way of protecting
myself from these kinds of events in the future, so I'll keep looking
for a way to send a NAK or go around the dhcp server and just parse logs
and issue a modem reset (this usually snaps errant clients out of their
maniacal behaviour). I'll just buy a 3Ghz server with a nice speedy
SCSI disc and be done with it (yeah right).
- --
Mason Schmitt
Systems Administrator
Sunwave Cable Internet / Shuswap Internet Junction
ph: (250) 832-9711
www.sunwave.net
PGP SIGNATURE
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

xkFwh1j2AtWbHi2P6JDaNAc=
=
PGP SIGNATURE

List Archives : http://www.isc.org/ops/lists/
Unsubscribe :
-or- : mailto:dhcp-server-request (AT) isc (DOT) org?Subject=unsubscribe

From dhcp-server -- To unsubscribe, see the end of this message.

Question: Are you using failover? Sometimes this kind of thing happens
if the server that is responsible for answering isnt always receiving
the discover, but the other one is receiving it. This will go on till
the other server decides to answer the request based on settings such as
max-unacked-updates

Possibility: I have seen many devices that firewall themselves right
out of receiving DHCP. It could be that some of these devices have
default enabled firewalls that block UDP 67/68. These firewalls will go
up after the offer when the interface is brought up on the offered ip.
The times that the device doesnt 'go crazy' are the times when the
firewall was a bit slow in coming up and the ACK made it through.

Possibility: If the server is somewhat slow in answering requests, some
devices may exhibit this behavior due to short timeouts set by
developers. They will continually request until the server manages to
get an answer to them in an amount of time that they find acceptable :)

Mason Schmitt wrote:

From dhcp-server -- To unsubscribe, see the end of this message.

PGP SIGNED MESSAGE
>Hash: SHA1
>
>There are several devices on my network that will occassionaly
>misbehave. There are three different manifestations of this bad
>behaviour that I have found so far (see below).
>
>I'm wondering whether there is any way that I can detect this behaviour
>and then either ignore any further requests from the device, rate limit
>it or perhaps try something like sending a NAK to the device? of
>these options, I like the idea of sending a NAK as it may be enough to
>bring the client back to its senses.
>
>I found the section on "lease events", in the DHCP handbook, and was
>wondering whether I could use the commit event to check how recently a
>lease was granted for a given client and then, if the lease was only a
>couple seconds old, send a NAK to the current request.
>
>Another option is to tail the log file using a perl script that watches
>for bad behavior and then resets the customer's modem via SNMP. This
>would probably be less impactful (on server performance) than the
>previous idea, but it seems a bit kludgy.
>
>Any thoughts?
>
>
>Bad Behaviour
>-
>1 - phoebe octopus and Aopen AR401 home routers.
>The device will send a discover, receive the offer, then suddenly start
>sending requests like mad (many times a second) with the dhcp server
>acking each one. This will go on for several minutes until the device
>finally accepts the IP and stops requesting - I have no idea why it
>stops asking. The logs show one phoebe doing this for 5 min solid and
>another one for 14 min solid. The Aopen went for 8 min solid.
>
>2 - belkin home routers
>The router requests an IP, gets the offer, then for the next several
>hours, it does one request per minute and the dhcp server ACKs each one.
>Finally after hours of this, the router will either send a release or
>not, but it will definitely do a normal discover, offer, request, ack.
>At this point, the router seems sure of itself and doesn't request again
>until half way through it's lease period (typical behaviour). However,
>when it requests at the halfway point, the problem starts up all over again.
>
>This isn't entirely consistent though. It appears that the device may
>decide to arbitrarily release as it did today 9 hours after it's last
>request ack cycle started up.
>
>3 - docsis cable modems
>Every once in a while, a docsis modem will go squirrelly and start
>making several requests per minute until it is reset, then it behaves
>itself.
>
>I, unfortunately, still haven't figured out what triggers these events
>
>- --
>Mason Schmitt
>Systems Administrator
>Sunwave Cable Internet / Shuswap Internet Junction
>ph: (250) 832-9711
>www.sunwave.net
PGP SIGNATURE
>Version: GnuPG v1.2.4 (GNU/Linux)
>Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
>
>
>XI3jutGKyY+lRc=
>=Pati
PGP SIGNATURE
>
>
>List Archives : http://www.isc.org/ops/lists/
>Unsubscribe :
>-or- : mailto:dhcp-server-request (AT) isc (DOT) org?Subject=unsubscribe
>

List Archives : http://www.isc.org/ops/lists/
Unsubscribe :
-or- : mailto:dhcp-server-request (AT) isc (DOT) org?Subject=unsubscribe

From dhcp-server -- To unsubscribe, see the end of this message.

PGP SIGNED MESSAGE
Hash: SHA1

Question: Are you using failover?

Nope.

Possibility: I have seen many devices that firewall themselves right
out of receiving DHCP. It could be that some of these devices have
default enabled firewalls that block UDP 67/68. These firewalls will go
up after the offer when the interface is brought up on the offered ip.
The times that the device doesnt 'go crazy' are the times when the
firewall was a bit slow in coming up and the ACK made it through.

Yeah, I've seen personal firewalls do many stupid things, but, more
often than not, the problem is a pebkac - zonealarm pops up a message
asking if the customer wants to allow a connection from the net and they
answer no and save that decision Then we get a support call with the
customer complaining that their internet connection is down *sigh*

Possibility: If the server is somewhat slow in answering requests, some
devices may exhibit this behavior due to short timeouts set by
developers. They will continually request until the server manages to
get an answer to them in an amount of time that they find acceptable :)

Thus the rant in my last post. I really wonder about the developers of
some of these home routers

Crappy home routers aside, I do think that this may be part of what is
happening. I have found that the server will be fine, sitting almost
100% idle answering the usual requests, then, I'm guessing, an event
drives the cpu utilization up just long enough for the dhcp server to
get a bit sluggish and then one of these stupid clients will decide to
just start hammering the server which further slows it down and it
snowballs from there to the point that the cpu is 100% utilized and many
requests are not being answered. Dhcpd will stay in this state until
I restart it. I'm going to look for possible events today to see
whether this hypothesis makes any sense or not.

The issue is somewhat complicated because I'm using an LDAP patch to
dhcpd and thus every request results in a lookup against ldap. If I
don't see any obvious events, then I may have to result to generating a
new static config every so often from ldap and see if that solves the
problem. I really hope I don't have to do this, as being able to make
changes without having to restart dhcpd is wonderful.
- --
Mason Schmitt
Systems Administrator
Sunwave Cable Internet / Shuswap Internet Junction
ph: (250) 832-9711
www.sunwave.net
PGP SIGNATURE
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

Vj1xBH7hc+ZexfKNac8o=
=Wc+K
PGP SIGNATURE

List Archives : http://www.isc.org/ops/lists/
Unsubscribe :
-or- : mailto:dhcp-server-request (AT) isc (DOT) org?Subject=unsubscribe

From dhcp-server -- To unsubscribe, see the end of this message.

Perhaps the LDAP server is not responding quickly enough at times? Is
it a separate machine from your DHCP server?
Mason Schmitt wrote:

PGP SIGNED MESSAGE
>Hash: SHA1
>

>
>>Question: Are you using failover?
>
>>
>
>Nope.
>

>
>>Possibility: I have seen many devices that firewall themselves right
>>out of receiving DHCP. It could be that some of these devices have
>>default enabled firewalls that block UDP 67/68. These firewalls will go
>>up after the offer when the interface is brought up on the offered ip.
>>The times that the device doesnt 'go crazy' are the times when the
>>firewall was a bit slow in coming up and the ACK made it through.
>
>>
>
>Yeah, I've seen personal firewalls do many stupid things, but, more
>often than not, the problem is a pebkac - zonealarm pops up a message
>asking if the customer wants to allow a connection from the net and they
>answer no and save that decision Then we get a support call with the
>customer complaining that their internet connection is down *sigh*
>

>
>>Possibility: If the server is somewhat slow in answering requests, some
>>devices may exhibit this behavior due to short timeouts set by
>>developers. They will continually request until the server manages to
>>get an answer to them in an amount of time that they find acceptable :)
>
>>
>
>Thus the rant in my last post. I really wonder about the developers of
>some of these home routers
>
>Crappy home routers aside, I do think that this may be part of what is
>happening. I have found that the server will be fine, sitting almost
>100% idle answering the usual requests, then, I'm guessing, an event
>drives the cpu utilization up just long enough for the dhcp server to
>get a bit sluggish and then one of these stupid clients will decide to
>just start hammering the server which further slows it down and it
>snowballs from there to the point that the cpu is 100% utilized and many
>requests are not being answered. Dhcpd will stay in this state until
>I restart it. I'm going to look for possible events today to see
>whether this hypothesis makes any sense or not.
>
>The issue is somewhat complicated because I'm using an LDAP patch to
>dhcpd and thus every request results in a lookup against ldap. If I
>don't see any obvious events, then I may have to result to generating a
>new static config every so often from ldap and see if that solves the
>problem. I really hope I don't have to do this, as being able to make
>changes without having to restart dhcpd is wonderful.
>
>- --
>Mason Schmitt
>Systems Administrator
>Sunwave Cable Internet / Shuswap Internet Junction
>ph: (250) 832-9711
>www.sunwave.net
PGP SIGNATURE
>Version: GnuPG v1.2.4 (GNU/Linux)
>Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
>
>
>Vj1xBH7hc+ZexfKNac8o=
>=Wc+K
PGP SIGNATURE

List Archives : http://www.isc.org/ops/lists/
Unsubscribe :
-or- : mailto:dhcp-server-request (AT) isc (DOT) org?Subject=unsubscribe

From dhcp-server -- To unsubscribe, see the end of this message.

PGP SIGNED MESSAGE
Hash: SHA1

Perhaps the LDAP server is not responding quickly enough at times? Is
it a separate machine from your DHCP server?

It's not on a separate machine. It has been my concern that the ldap
server isn't responding fast enough, but I haven't sorted out a good
test for determining whether that's the case or not. Given that I plan
to move to a central ldap repository for much more than just dhcp, if
there is an issue with timeliness of responses to the dhcp server, I
need to know soon. Anyone have any suggestions for how I might isolate
the ldap lookup part of the dhcp process so that I can see what the
latency is like? I'm guessing that syslog isn't going to be accurate
enough.

I think I might need to build a couple test boxes any play with this in
a test network. I may want to transition from ldbm to bdb or hdb as the
backend for openldap just to avoid the possibility of a lock during a
write operation preventing a read from the dhcp server.

As for the test client, does anyone have any suggestions for a client
that I could configure to send requests like crazy, but ignore the acks?
I was thinking that a nagios dhcp service plugin might do the trick.
- --
Mason Schmitt
Systems Administrator
Sunwave Cable Internet / Shuswap Internet Junction
ph: (250) 832-9711
www.sunwave.net
PGP SIGNATURE
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

QsifWK/SansMFkQc/svtD/A=
=wzcI
PGP SIGNATURE

List Archives : http://www.isc.org/ops/lists/
Unsubscribe :
-or- : mailto:dhcp-server-request (AT) isc (DOT) org?Subject=unsubscribe

From dhcp-server -- To unsubscribe, see the end of this message.

Mason Schmitt wrote:

>>Possibility: If the server is somewhat slow in answering requests, some
>>devices may exhibit this behavior due to short timeouts set by
>>developers. They will continually request until the server manages to
>>get an answer to them in an amount of time that they find acceptable :)

Also, UDP packets can get lost in the network.

The solution in the RADIUS space is to cache requests and responses:

receive:
if (cached(packet)) {
if (reply) {
send_reply
} else {
discard packet
}
stop
} else {
cache packet, with NULL reply
}

Duplicate requests can therefore be responded to very quickly. You
need timeouts in the cache of course, but a 5-30 second timeout is
usually sufficient.

It also means that duplicate request isn't processed through the slow
path again.

The issue is somewhat complicated because I'm using an LDAP patch to
dhcpd and thus every request results in a lookup against ldap.

The caching method described above is even more important then. DB
lookups are *expensive*.

Alan DeKok.

List Archives : http://www.isc.org/ops/lists/
Unsubscribe :
-or- : mailto:dhcp-server-request (AT) isc (DOT) org?Subject=unsubscribe

From dhcp-server -- To unsubscribe, see the end of this message.

PGP SIGNED MESSAGE
Hash: SHA1

Alan DeKok wrote:
The solution in the RADIUS space is to cache requests and responses:

receive:
if (cached(packet)) {
if (reply) {
send_reply
} else {
discard packet
}
stop
} else {
cache packet, with NULL reply
}

Duplicate requests can therefore be responded to very quickly. You
need timeouts in the cache of course, but a 5-30 second timeout is
usually sufficient.

It also means that duplicate request isn't processed through the slow
path again.

I agree that caching would go a long way towards alleviating the
symptoms of this problem. Are you aware of any way of instructing dhcpd
to cache like this? I'm not much of a programmer (yet?), so
unfortunately I can't poke my head into the code if that's what needs to
happen.
- --
Mason Schmitt
Systems Administrator
Sunwave Cable Internet / Shuswap Internet Junction
ph: (250) 832-9711
www.sunwave.net
PGP SIGNATURE
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

3eYmhVl8dI8vQFoAESdLKrY=
=MrCJ
PGP SIGNATURE

List Archives : http://www.isc.org/ops/lists/
Unsubscribe :
-or- : mailto:dhcp-server-request (AT) isc (DOT) org?Subject=unsubscribe

From dhcp-server -- To unsubscribe, see the end of this message.

Mason Schmitt wrote:
I agree that caching would go a long way towards alleviating the
symptoms of this problem. Are you aware of any way of instructing dhcpd
to cache like this?

Source code modifications.

It's a rare enough requirement that few people need it, which means
that it's less likely to be implemented.

Alan DeKok.

List Archives : http://www.isc.org/ops/lists/
Unsubscribe :
-or- : mailto:dhcp-server-request (AT) isc (DOT) org?Subject=unsubscribe