-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 01/22/10 05:32, Claus Herwig wrote:
>>
>> I would be very interested in seeing the events and actual activity of
>> the oomkiller because a co-worker and I have been chasing something
>> very similar.
>>
>
> This is getting interesting.
>
> At the moment I try to hunt down some kind of memory leak myself. The
> problem started about two weeks ago on one of my vserver guests. Some
> days later I got almost the same symptoms with a second vserver guest on
> another host system.
>
> Both host systems are AMD Opterons, kernel x86_64 2.6.20.18 with
> vserver-patch 2.2.0.3. Yes, I know this is not really bleeding edge, but
> it was working w/o any problem until now. Software besides kernel is
> plain debian lenny on host and guests, security updates applied.
>
> My first impression of the symptoms was almost identical to what
> "cryptronic" described: Memory and load on the host are normal for days.
> Then suddenly memory usage is exploding, kernel starts swapping. Then
> swap is full and system load starts exploding (mainly because of kswapd
> going mad). I got a load of 600+ within minutes and wasn't able to do
> anything besides a restart.
>
> To investigate this (on one machine with about a dozen guests) I set rss
> limits and the VIRT_MEM flag for the guest. After that I started seeing
> that one guest (in contrast to all others) has a steadily growing RSS
> usage. This guest is a typical lamp server: apache2, mysql5, lots of
> wordpress blogs and an openx installation.
>
> I restarted this guest several times and found the following: RSS usage
> is constant after startup. Then after some (varying) time I get this in
> apache error log:
>> Inconsistency detected by ld.so: dl-open.c: 260: dl_open_worker:
>> Assertion `_dl_debug_initialize (0, args->nsid)->r_state ==
>> RT_CONSISTENT' failed!
> After this error RSS usage steadily grows until the RSS hardlimit is hit
> and I got lots of
> (12)Cannot allocate memory: fork: Unable to fork new process
> in apache error log.
>
>
> I admit, it's quite possible, that my problem is totally unrelated to
> yours (no 2.3 vserver patch involved in my setup).
>
> But on the other hand I think perhaps you're looking in the wrong place
> and linux-vserver isn't the culprit in your case. Perhaps we should
> start to search for some shiny new apache bug in the latest debian
> apache security update?
We have the exploding load problem now for more then a year in v2.3.
Until now we thought this is a hardware problem, but now there are 2
more hardware types so we wrote to the list. But it could be.
We have also debian lenny guests on our systems but not only. Because
i cannot say which apache process of which vserver caused the
exploding load i coulnd not say whether this is a apache bug oder
uv/lv bug.
>
>
> Any comments and ideas appreciated, of course...
> Claus
>
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.14 (GNU/Linux)
iEYEARECAAYFAktZceYACgkQOBdlVlcPuhz4iACgv235IFINzyFnYvbwsxBk6QBf
+BMAoLPo+h3OhMw9+GisHG83b4t431oa
=TR3b
-----END PGP SIGNATURE-----
Received on Fri Jan 22 09:38:08 2010