[vserver] guest inodes full => kernel panic !?!

From: Jarry <mr.jarry_at_gmail.com>
Date: Mon 06 Jul 2009 - 18:46:07 BST
Message-ID: <4A52385F.9020403@gmail.com>

Hi,

I just suffered the worst nightmare of any server-admin:
kernel panic! This is the way it happened:

I tried to update gcc on all my guests. Because they run gentoo
(as well as guest), updating takes rather long time (everything
is compilled from sources). After a couple of hours I found out,
upgrade process failed on some guests. After a little search
I discovered (with vdlimit) those guests were short of inodes:
...
inodes_used=99996
inodes_total=100000
...

I increased limits in /etc/vservers/<vs>/dlimits/root/inodes_total
and tried to stop & start all vservers:
/etc/init.d/vserver.default stop
Stopping default vservers...

(maybe I should have first stop vserver-guests, then increase
  inodes_total limits)

And nothing more. Network connection broken and I could not connect
to my server any more. So I rushed to server-room, attached screen
just to see "Kernel panic..."!

I did hard reset and everything seems to work for now. But I'm very
surprised, full inodes-limit in vserver-guest could bring the whole
host-server down! The only strange entry I found in messages is:

Jul 6 16:50:56 obel vxW: [xid #0] !!! limit: ffff8101373b7078[VM,9] = 33
on exit.

Otherwise no trace. Kernel 2.6.22, vserver 2.2.0.7. All guest and
host are gentoo, stable and updated.

Disk space was definitelly not a problem; all guests had ~10GB free.
I also have "10" in /etc/vservers/<vs>/dlimits/root/reserved ,
but this is probably related only to disk-space, not inodes number
(can someone from dev confirm?). If it is so, there should definitelly
be some reserved-limit for inodes too...

I'm not going to test it again, but something like this should
really *never* happen...

Jarry

-- 
_______________________________________________________________
This mailbox accepts e-mails only from selected mailing-lists!
Everything else is considered to be spam and therefore deleted.
Received on Mon Jul 6 18:46:45 2009
[Next/Previous Months] [Main vserver Project Homepage] [Howto Subscribe/Unsubscribe] [Paul Sladen's vserver stuff]
Generated on Mon 06 Jul 2009 - 18:46:47 BST by hypermail 2.1.8