Re: [vserver] vservers and inodes

From: Tor Rune Skoglund <trs_at_sg.no>
Date: Tue 08 Jun 2010 - 10:04:07 BST
Message-ID: <AANLkTilIcvELmfl_nu3Woed68HDSiZW7tbQ62kpT95UI@mail.gmail.com>

Thanks Herbert, I really appreciate your support.
It reminds me that there is time to support the
project financially.

I really hope we can fix this, and make the "management"
happy - as they seem to be starting pulling rank to get debian
and vmware instead.

2010/6/7 Herbert Poetzl <herbert@13thfloor.at>:
> On Mon, Jun 07, 2010 at 12:02:16PM +0200, Tor Rune Skoglund wrote:
>> I've ran into a strange (possibly) inode problem within
>> one vserver. Althought df -i on the host shows that the
>> filesystem on which the vserser lives has IUse of only 1%,
>> we get "cannot create" errors on new files.
>
> cannot create is not necessarily an indication to running
> out of inodes ... could you please strace -fF one of those
> and provide the trace via some pastebin?
>
>> Having seen that there is a configuration option
>> for the maximum number of inodes dedicated to a vserver,
>> I have the following questions:
>
>> - What is the default limit when this option is not set?
>
> when the dlimit is unset, the number of inodes is unlimited
>
>> - What mechanism is used when checking and enforcing vserver
>>   inode limits?
>
> the kernel takes care of that
>
>> - The total number of inodes on the filesystem on which the
>>   vserver lives is 324350 at the moment.
>>   Could someone comment if is so high that special actions
>>   need to be taken?
>
> nope, I have several millions of inode in use, and except
> for a certain memory overhead (for the caches) there is no
> problem to be expected ....
>
>> - Could any other limit be causing the error? E.g. number of
>>   files limit or whatever?
>
> could be, strace will probably shed some light on that

At the bottom, there an extract of the important parts of
the strace result.

We have also done some more testing of the
problem. I would like to add this to describe the setup:

* We have a vserver as a fileserver, running
samba 3.4.6. The host and fileserver are both
gentoo-based.

* The client, which in this case is a debian vserver on the
same host, mounts the fileserver share. Unzip/move operations
with large number of files fail as described.

* Another vserver client, but Gentoo based, on the same host
experiences the same problems.

* The client, when mounting another share, on another
computer, on which there is no vservers and special stuff,
gets the same problem with that share. I.e., it seems to
be related to the vserver client, and not the fileserver vserver.

* However, on another plain host, Debian based (without
vserver patches) the unzip operation works fine with the
fileserver share (!)

>> Here are some server facts:
>> * Gentoo vserver kernel, 2.6.32-vs2.3.0.36.28
>> * util-vserver 0.30.216_pre2883
>> * Filesystem ext3, default settings when formatting

Best regards,
Tor Rune Skoglund

execve("/usr/bin/unzip", ["unzip", "../stern_2007_all.zip"], [/* 18
vars */]) = 0
brk(0) = 0x2049000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0x7f295c980000
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0x7f295c97e000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/tls/x86_64/libc.so.6", O_RDONLY) = -1 ENOENT (No
such file or directory)
stat("/usr/local/lib/tls/x86_64", 0x7fffc2d2b560) = -1 ENOENT (No such
file or directory)
open("/usr/local/lib/tls/libc.so.6", O_RDONLY) = -1 ENOENT (No such
file or directory)
stat("/usr/local/lib/tls", 0x7fffc2d2b560) = -1 ENOENT (No such file
or directory)
open("/usr/local/lib/x86_64/libc.so.6", O_RDONLY) = -1 ENOENT (No such
file or directory)
stat("/usr/local/lib/x86_64", 0x7fffc2d2b560) = -1 ENOENT (No such
file or directory)
open("/usr/local/lib/libc.so.6", O_RDONLY) = -1 ENOENT (No such file
or directory)
stat("/usr/local/lib", {st_mode=S_IFDIR|S_ISGID|0775, st_size=4096, ...}) = 0
open("/etc/ld.so.cache", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=21183, ...}) = 0
mmap(NULL, 21183, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f295c978000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/libc.so.6", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300\342\1\0\0\0\0\0@"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1375536, ...}) = 0
mmap(NULL, 3482232, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
0) = 0x7f295c413000
mprotect(0x7f295c55d000, 2093056, PROT_NONE) = 0
mmap(0x7f295c75c000, 20480, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x149000) = 0x7f295c75c000
mmap(0x7f295c761000, 17016, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f295c761000
close(3) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0x7f295c977000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0x7f295c976000
arch_prctl(ARCH_SET_FS, 0x7f295c9766e0) = 0
mprotect(0x7f295c75c000, 12288, PROT_READ) = 0
munmap(0x7f295c978000, 21183) = 0
rt_sigaction(SIGINT, {0x409670, [INT], SA_RESTORER|SA_RESTART,
0x7f295c444f60}, {SIG_DFL}, 8) = 0
rt_sigaction(SIGTERM, {0x409670, [TERM], SA_RESTORER|SA_RESTART,
0x7f295c444f60}, {SIG_DFL}, 8) = 0
rt_sigaction(SIGBUS, {0x409670, [BUS], SA_RESTORER|SA_RESTART,
0x7f295c444f60}, {SIG_DFL}, 8) = 0
rt_sigaction(SIGSEGV, {0x409670, [SEGV], SA_RESTORER|SA_RESTART,
0x7f295c444f60}, {SIG_DFL}, 8) = 0
brk(0) = 0x2049000
brk(0x206c000) = 0x206c000
open("/etc/localtime", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
fstat(3, {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0x7f295c97d000
read(3, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\1\0\0\0\0\0"...,
4096) = 118
lseek(3, -62, SEEK_CUR) = 56
read(3, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\1\0\0\0\0\0"...,
4096) = 62
close(3) = 0
munmap(0x7f295c97d000, 4096) = 0
stat("../stern_2007_all.zip", {st_mode=S_IFREG|0755, st_size=53229057, ...}) = 0
open("../stern_2007_all.zip", O_RDONLY) = 3
ioctl(1, TIOCGWINSZ, {ws_row=47, ws_col=80, ws_xpixel=0, ws_ypixel=0}) = 0
write(1, "Archive: ../stern_2007_all.zip\n"..., 32) = 32
lseek(3, 53223424, SEEK_SET) = 53223424
read(3, "SiemensKurssprungBF6rsePiererRFCc"..., 5633) = 5633
lseek(3, 51617792, SEEK_SET) = 51617792
read(3, "\300\275\305a\344:W\331}\332\22\330\273\34Q\360$\306xq1\273@K`}@Nl;p%*"...,
8192) = 8192
umask(0) = 022
umask(022) = 0
umask(0) = 022
umask(022) = 0

***** lots more of these....

umask(0) = 022
umask(022) = 0
umask(0) = 022
umask(022) = 0
read(3, "\23X\"6\332|\262\204\10\26\0\0\363C\0\0t\0\0\0\0\0\0\0\0\0
\0\0\0\231\250\2"..., 8192) = 8192
umask(0) = 022
umask(022) = 0
umask(0) = 022
umask(022) = 0

***** some more of them....

umask(0) = 022
umask(022) = 0
umask(0) = 022
umask(022) = 0
lseek(3, 0, SEEK_SET) = 0
read(3, "PK\3\4\n\0\0\0\0\0rc\303<\0\0\0\0\0\0\0\0\0\0\0\0\5\0\0\000200"...,
8192) = 8192
stat("2007", 0x62ed08) = -1 ENOENT (No such file or directory)
mkdir("2007", 0777) = 0
ioctl(1, TIOCGWINSZ, {ws_row=47, ws_col=80, ws_xpixel=0, ws_ypixel=0}) = 0
write(1, " creating: 2007/\n"..., 19) = 19
chmod("2007/", 0755) = 0
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
stat("2007", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
stat("2007/01", 0x62ed08) = -1 ENOENT (No such file or directory)
mkdir("2007/01", 0777) = -1 ENOENT (No such file or directory)
ioctl(1, TIOCGWINSZ, {ws_row=47, ws_col=80, ws_xpixel=0, ws_ypixel=0}) = 0
write(2, "checkdir error: cannot create 20"..., 84checkdir error:
cannot create 2007/01
                 unable to process 2007/01/.
) = 84
stat("2007", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
stat("2007/01", 0x62ed08) = -1 ENOENT (No such file or directory)
mkdir("2007/01", 0777) = -1 ENOENT (No such file or directory)
ioctl(1, TIOCGWINSZ, {ws_row=47, ws_col=80, ws_xpixel=0, ws_ypixel=0}) = 0
write(2, "checkdir error: cannot create 20"..., 87checkdir error:
cannot create 2007/01
                 unable to process 2007/01/01/.
) = 87
stat("2007", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
stat("2007/01", 0x62ed08) = -1 ENOENT (No such file or directory)
mkdir("2007/01", 0777) = -1 ENOENT (No such file or directory)
ioctl(1, TIOCGWINSZ, {ws_row=47, ws_col=80, ws_xpixel=0, ws_ypixel=0}) = 0
write(2, "checkdir error: cannot create 20"..., 144checkdir error:
cannot create 2007/01
                 unable to process
2007/01/01/http__www_stern_de_lifestyle_leute_579484_htmlnvprpr1.raw.
) = 144
stat("2007", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
stat("2007/01", 0x62ed08) = -1 ENOENT (No such file or directory)
mkdir("2007/01", 0777) = -1 ENOENT (No such file or directory)
ioctl(1, TIOCGWINSZ, {ws_row=47, ws_col=80, ws_xpixel=0, ws_ypixel=0}) = 0
write(2, "checkdir error: cannot create 20"..., 144checkdir error:
cannot create 2007/01
                 unable to process
2007/01/01/http__www_stern_de_lifestyle_leute_579484_htmlnvprpr1.xml.
) = 144

******* ... and then lots of these, until it in the end ends with:

stat("2007", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
stat("2007/04", 0x62ed08) = -1 ENOENT (No such file or directory)
mkdir("2007/04", 0777) = -1 ENOENT (No such file or directory)
ioctl(1, TIOCGWINSZ, {ws_row=47, ws_col=80, ws_xpixel=0, ws_ypixel=0}) = 0
write(2, "checkdir error: cannot create 20"..., 175checkdir error:
cannot create 2007/04
                 unable to process
2007/04/21/http__www_stern_de_wissenschaft_ernaehrung_RisikoDieseFetteHE4rte_586102_htmlnvprpr1.raw.
) = 175
lseek(3, 51609600, SEEK_SET) = 51609600
read(3, "\214@\376\242\16\245\tj\270\317\335\306\245\247bG\251D\223g\22A\256\300=H\fx\va\212\322N"...,
8192) = 8192
stat("2007", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
stat("2007/04", 0x62ed08) = -1 ENOENT (No such file or directory)
mkdir("2007/04", 0777) = -1 ENOENT (No such file or directory)
ioctl(1, TIOCGWINSZ, {ws_row=47, ws_col=80, ws_xpixel=0, ws_ypixel=0}) = 0
write(2, "checkdir error: cannot create 20"..., 175checkdir error:
cannot create 2007/04
                 unable to process
2007/04/21/http__www_stern_de_wissenschaft_ernaehrung_RisikoDieseFetteHE4rte_586102_htmlnvprpr1.xml.
) = 175
lseek(3, 53223424, SEEK_SET) = 53223424
read(3, "SiemensKurssprungBF6rsePiererRFCc"..., 8192) = 5633
utime("2007/", [2010/06/03-12:27:36, 2010/06/03-12:27:36]) = 0
chmod("2007/", 0755) = 0
close(3) = 0
exit_group(2) = ?
Received on Tue Jun 8 22:20:09 2010

[Next/Previous Months] [Main vserver Project Homepage] [Howto Subscribe/Unsubscribe] [Paul Sladen's vserver stuff]
Generated on Tue 08 Jun 2010 - 22:20:12 BST by hypermail 2.1.8