Re: [vserver] Exploding Load in v2.3

From: Cryptronic <>
Date: Fri 22 Jan 2010 - 09:37:42 GMT
Message-ID: <>

Hash: SHA1

On 01/22/10 05:32, Claus Herwig wrote:
>> I would be very interested in seeing the events and actual activity of
>> the oomkiller because a co-worker and I have been chasing something
>> very similar.
> This is getting interesting.
> At the moment I try to hunt down some kind of memory leak myself. The
> problem started about two weeks ago on one of my vserver guests. Some
> days later I got almost the same symptoms with a second vserver guest on
> another host system.
> Both host systems are AMD Opterons, kernel x86_64 with
> vserver-patch Yes, I know this is not really bleeding edge, but
> it was working w/o any problem until now. Software besides kernel is
> plain debian lenny on host and guests, security updates applied.
> My first impression of the symptoms was almost identical to what
> "cryptronic" described: Memory and load on the host are normal for days.
> Then suddenly memory usage is exploding, kernel starts swapping. Then
> swap is full and system load starts exploding (mainly because of kswapd
> going mad). I got a load of 600+ within minutes and wasn't able to do
> anything besides a restart.
> To investigate this (on one machine with about a dozen guests) I set rss
> limits and the VIRT_MEM flag for the guest. After that I started seeing
> that one guest (in contrast to all others) has a steadily growing RSS
> usage. This guest is a typical lamp server: apache2, mysql5, lots of
> wordpress blogs and an openx installation.
> I restarted this guest several times and found the following: RSS usage
> is constant after startup. Then after some (varying) time I get this in
> apache error log:
>> Inconsistency detected by dl-open.c: 260: dl_open_worker:
>> Assertion `_dl_debug_initialize (0, args->nsid)->r_state ==
>> RT_CONSISTENT' failed!
> After this error RSS usage steadily grows until the RSS hardlimit is hit
> and I got lots of
> (12)Cannot allocate memory: fork: Unable to fork new process
> in apache error log.
> I admit, it's quite possible, that my problem is totally unrelated to
> yours (no 2.3 vserver patch involved in my setup).
> But on the other hand I think perhaps you're looking in the wrong place
> and linux-vserver isn't the culprit in your case. Perhaps we should
> start to search for some shiny new apache bug in the latest debian
> apache security update?

We have the exploding load problem now for more then a year in v2.3.
Until now we thought this is a hardware problem, but now there are 2
more hardware types so we wrote to the list. But it could be.

We have also debian lenny guests on our systems but not only. Because
i cannot say which apache process of which vserver caused the
exploding load i coulnd not say whether this is a apache bug oder
uv/lv bug.

> Any comments and ideas appreciated, of course...
> Claus

Version: GnuPG v2.0.14 (GNU/Linux)

Received on Fri Jan 22 09:38:08 2010

[Next/Previous Months] [Main vserver Project Homepage] [Howto Subscribe/Unsubscribe] [Paul Sladen's vserver stuff]
Generated on Fri 22 Jan 2010 - 09:38:09 GMT by hypermail 2.1.8