Stephen Jones wrote:
An
interesting thing to note is that the system seems stable when it is
freshly booted and barely using swap, but
once it gets deeper into swap the dead lock becomes eminent.
Perhaps the backing store is broken as well?
Cheers,

Thu, Jun 22, 2006 at 09:59:30AM -0700, Stephen Jones wrote:
I have a file server that used to operate fairly reliably with 1GB of
RAM. A few weeks ago the memory tested bad and the system would not
boot, however the only known spares available were two 128MB modules
giving me 256MB of ram. Since bringing the server up with this
amount of ram, I am able to achieve multiple vnlock-deadlocks
scenarios on a daily basis. These cannot be recovered from and the
system must be hardware reset. The kernel is 2.1 and the a single
file system is being served via NFS with imapd, teapop and sendmail
processes accessing it. It appears that the nfsd processes are the
first to vnlock deadlock, which then cascade and are unstoppable. An
interesting thing to note is that the system seems stable when it is
freshly booted and barely using swap, but
once it gets deeper into swap the dead lock becomes eminent. Since
this problem only occurred every 2 to 3 weeks
with 1GB of ram and now is happening 2 to 3 times a day with 256MB of
ram, I thought it might be a good time to start having a kernel
expert investigate.

Did you do some tuning ? Large buffer cache, or change uvm settings ?
A too high kern.maxvnodes can cause this, for example.

Thu, Jun 22, 2006 at 01:36:14PM -0700, Stephen Jones wrote:

Jun 22, 2006, at 12:35 PM, Manuel Bouyer wrote:
>Did you do some tuning ? Large buffer cache, or change uvm settings ?
>A too high kern.maxvnodes can cause this, for example.

What is considered two high?

Depends on RAM available, and other things I guess

Would that really cause a deadlock?

in my experience, yes.

Is
there
a formula for determining what BUFCACHE or kern.maxvnodes set to?
What would
be too low? If the concern is that the BUFCACHE is too large or
kern.maxvnodes
then how is that determined? The default is 10055, is that too low?
why is it
10055?

If you didn't change the default value then it's probably K.

BUFCACHE I have set to 6 (%) in the kernel, can that be set
via sysctl as well?

Yes, it's vm.bufcache.

I should also note I have vfs.nfs.iothreads set to 0 and have had it
that way for
a couple of years now.

If it's a NFS server it doens't matter; this is for clients.

Thu, Jun 22, 2006 at 02:57:30PM -0700, Stephen Jones wrote:

Jun 22, 2006, at 2:15 PM, Manuel Bouyer wrote:
>
>Depends on RAM available, and other things I guess
>
>>Would that really cause a deadlock?
>
>in my experience, yes.

What sort of 'other things'

Network activity for example, and more generally all things that want
to allocate memory in kernel.

and can you please tell me more about
your experiences?

Well, I don't have much. I just noticed that if maxvnode is set too high,
the kernel ends up deadlocking, with free memory near 0.

>
>If you didn't change the default value then it's probably K.

They usually all show up as allocated in systat within a few hours of
the
system running no matter what the setting is set to. Is that
probably K too?

Yes, this is K.


>>BUFCACHE I have set to 6 (%) in the kernel, can that be set
>>via sysctl as well?
>
>Yes, it's vm.bufcache.

What I really meant to ask is it dynamic though? Can it only be
increased without a reboot
or drop to single user mode?

I think it can be decreased as well. But 6% isn't that much


>If it's a NFS server it doens't matter; this is for clients.

This particular server is an NFS client as well. I typically see
vnlock deadlocks
on clients and rarely on a server. However, this case is a bit of
exception as it only
has one served file system and its nfsd are the first to deadlock.

Hum, I didn't notice this on clients (although I've seen other issues for
which I have PR open).