it seems that anycasting was quite insufficient to protect
netsol's service from being severely damaged (udp dead, tcp
worked) for a considerable length of time by a ddos [0] last
week [1]. it would be very helpful to other folk concerned
with service deployment to understand how the service in
question was/is anycast, and what might be done differently
to mitigate exposure of similar services.
anyone have clues or is this ostrich city? maybe a preso at
nanog would be educational.
randy
[0] - as it seems that the ddos sources were ip address
spoofed (which is why the service still worked for
tcp), i owe paul an apology for downplaying the
immediacy of the need for source address filtering.
[1] - netsol is not admitting anything happened, of course
<sigh>. but we all saw the big splash as it hit the
water, the bubbles as it sank, and the symptoms made
the cause pretty clear.

Fri, 6 May 2005, Randy Bush wrote:

it seems that anycasting was quite insufficient to protect
netsol's service from being severely damaged (udp dead, tcp
worked) for a considerable length of time by a ddos [0] last
week [1]. it would be very helpful to other folk concerned
with service deployment to understand how the service in
question was/is anycast, and what might be done differently
to mitigate exposure of similar services.

was the service in question anycast'ed? I got the impression that the
worldnic servers were all NN-anycast I only see the /21 covering these
servers through 10515 (which is verisign as I recall?)

Judging by latency I even think they are in the northern virginia area
I also noted:

worldnic.com. 86400 IN NS ns1.netsol.com.
worldnic.com. 86400 IN NS ns2.netsol.com.
worldnic.com. 86400 IN NS ns3.netsol.com.

;; ADDITINAL SECTIN:
ns1.netsol.com. 86400 IN A 216.168.229.228
ns2.netsol.com. 86400 IN A 216.168.229.229
ns3.netsol.com. 86400 IN A 216.168.229.229

why have 3 records and 2 ips? odd. You'd think they would have more ips in
that /21 or other /24's to allocate from, just in case they had to
jettison 1 address which was getting pounded :( (not that these were
getting attacked per-say, but still)

[0] - as it seems that the ddos sources were ip address
spoofed (which is why the service still worked for
tcp), i owe paul an apology for downplaying the
immediacy of the need for source address filtering.

It's also not clear that the sources were spoofed, if as Patrick says they
put in a riverhead(s) (which isn't too far fetched) the normal mode for
'protection' of DNS is to:
1) truncate
2) rate-limit - and cache (I think it caches atleast, I know it will go
into proxy mode and rate-limit)

truncate forces TCP which allows RHG to verify the source address is
really asking to chat, rate-limit function keeps 'bad actors' from
beatting the hell out of the protected resource.

So, without more info from NetSol (seems not to be forthcoming?) about the
mix of attack traffic (which the RHG will provide) it's hard to state
definitively that the attack was 'mostly spoofed' :(

was the service in question anycast'ed? I got the impression that the
worldnic servers were all NN-anycast.
i not believe you are correct, they are not anycasted. i was silly
enough to believe one (seemingly false) report. apologies.
randy

<sighlong day

>was the service in question anycast'ed? I got the impression that the
>worldnic servers were all NN-anycast.
i not believe you are correct, they are not anycasted. i was silly

s/not//

enough to believe one (seemingly false) report. apologies.

Fri, 6 May 2005, Randy Bush wrote:

was the service in question anycast'ed? I got the impression that the
worldnic servers were all NN-anycast.

i not believe you are correct, they are not anycasted. i was silly
enough to believe one (seemingly false) report. apologies.

no apologies to me required, but it'd still be interesting to hear what
happend, eh? :)

[ i figure if i keep asking poking and naive questions i'll keep
learning more about this, which may help me and others learn from
the mistakes of others. ]

no apologies to me required, but it'd still be interesting to
hear what happend, eh? :)

i suspect that we don't hear from the horse's mouth is a symptom of
one of the causes, "we know well enough to go it alone, and we can
pretend that we're perfect." well, a day+ long wipeout should make
it pretty clear that the bunker mentality is as much a fallacy as
the technology of the deployment. it failed, and badly.

but you are correct, those of us more responsible for network
engineering are as much concerned by the technological aspect(s).

and folk seem to think that it is a bunker mentality centralized
deployment, i.e., a small number of server clusters ripe for the
picking, that fell to a simple, though likely intense, ddos attack.

and one that we do not know was spoofed (i unapologize, paul:-) and
did not really need to be because of the weaknesses of the service
deployment.

and the above combined with problems of riverhead configuration and
limitations, and lack of cooperation with upstreams to mitigate the
attack, turned a fairly normal ddos into a day+ serious mess?

randy

At 01:38 AM 07-05-05 +0000, Christopher L. Morrow wrote:

I scanned my Telescope report of 3,382 spoofed DDS attacks last week (May
1-7) and could not find any listed for 216.168.229.0/24, worldnic.com,
netsol.com or AS6245.
-Hank


>worldnic.com. 86400 IN NS ns1.netsol.com.
>worldnic.com. 86400 IN NS ns2.netsol.com.
>worldnic.com. 86400 IN NS ns3.netsol.com.
>
>;; ADDITINAL SECTIN:
>ns1.netsol.com. 86400 IN A 216.168.229.228
>ns2.netsol.com. 86400 IN A 216.168.229.229
>ns3.netsol.com. 86400 IN A 216.168.229.229
>
>why have 3 records and 2 ips? odd. You'd think they would have more ips in
>that /21 or other /24's to allocate from, just in case they had to
>jettison 1 address which was getting pounded :( (not that these were
>getting attacked per-say, but still)
>
[0] - as it seems that the ddos sources were ip address
spoofed (which is why the service still worked for
tcp), i owe paul an apology for downplaying the
immediacy of the need for source address filtering.
>
>
>It's also not clear that the sources were spoofed, if as Patrick says they
>put in a riverhead(s) (which isn't too far fetched) the normal mode for
>'protection' of DNS is to:
>1) truncate
>2) rate-limit - and cache (I think it caches atleast, I know it will go
>into proxy mode and rate-limit)
>
>truncate forces TCP which allows RHG to verify the source address is
>really asking to chat, rate-limit function keeps 'bad actors' from
>beatting the hell out of the protected resource.
>
>So, without more info from NetSol (seems not to be forthcoming?) about the
>mix of attack traffic (which the RHG will provide) it's hard to state
>definitively that the attack was 'mostly spoofed' :(