A customer's server has run out of inodes, but has a tiny amount of files:
mailserver 3# find /usr | wc -l
112200
mailserver 4# df -i /usr
Filesystem 1K-blocks Used Avail Capacity iused ifree %iused
/dev/raid2a 153018716 17104340 128263444 11% 9539859 17131 99%
That's nearly 100 times as many inodes as files/directories! Now, I know
an inode != file/directory, but it should be the same order of magnitude!
Compared to a server here:
ugly 421# find /usr | wc -l
1434411
ugly 422# df -i /usr
Filesystem 1K-blocks Used Avail Capacity iused ifree %iused
/dev/raid2a 241151544 134683608 94410356 58% 1416711 13621495 9%
What's going on on our customer's server? What tools can I run (in
multi-user mode) to track this down a bit more?

Thu, Jul 06, 2006 at 01:39:13PM +0000, Christos Zoulas wrote:
In article <@ugly.precedence.co.uk>,
Stephen Borrill <netbsd (AT) precedence (DOT) co.ukwrote:
>A customer's server has run out of inodes, but has a tiny amount of files:
>
>mailserver 3# find /usr | wc -l
112200
>mailserver 4# df -i /usr
>Filesystem 1K-blocks Used Avail Capacity iused ifree %iused
>/dev/raid2a 153018716 17104340 128263444 11% 9539859 17131 99%
>
>That's nearly 100 times as many inodes as files/directories! Now, I know
>an inode != file/directory, but it should be the same order of magnitude!

Is it corrupted? is there a process holding open tons of files? Hard
to believe.

I remember having similar trouble on a very large file-system. I think
it was corrupted. I couldn't investigate, so I just newfs'd it after
moving away the data.

Thu, 6 Jul 2006, Quentin Garnier wrote:
Thu, Jul 06, 2006 at 01:39:13PM +0000, Christos Zoulas wrote:
>In article <@ugly.precedence.co.uk>,
>Stephen Borrill <netbsd (AT) precedence (DOT) co.ukwrote:
A customer's server has run out of inodes, but has a tiny amount of files:

mailserver 3# find /usr | wc -l
112200
mailserver 4# df -i /usr
Filesystem 1K-blocks Used Avail Capacity iused ifree %iused
/dev/raid2a 153018716 17104340 128263444 11% 9539859 17131 99%

That's nearly 100 times as many inodes as files/directories! Now, I know
an inode != file/directory, but it should be the same order of magnitude!
>>
>Is it corrupted?

That's all I can think.

>is there a process holding open tons of files?

No, this is immediately after a reboot.

I remember having similar trouble on a very large file-system. I think
it was corrupted. I couldn't investigate, so I just newfs'd it after
moving away the data.

It's a production server that I only have remote ssh access to, so I'd
rather avoid doing that (but it is possible: I could break the RAID1 array
and repartition one HD to hold the data temporarily).

Might a forced fsck from single-user help (if I can talk them through
it!)?

Thu, 6 Jul 2006, dieter roelants wrote:
07/06/06 15:50:51, Stephen Borrill wrote:

is there a process holding open tons of files?
>
>No, this is immediately after a reboot.
>
Was it an unclean shutdown?

Not at this time, but there may well have been one at some unspecified
time in the past.

I once had the same issue after a power failure and fsck. I went to
single user and ran a 2nd fsck and that cleaned it up.

Just got the customer to run fsck -f from single user and after rebooting,
df -i shows:

Filesystem 1K-blocks Used Avail Capacity iused ifree %iused
/dev/raid2a 153018716 17260336 128107448 11% 116342 9440648 1%

was:
/dev/raid2a 153018716 17104340 128263444 11% 9539859 17131 99%

:-)

Thanks for the discussion.