[00:00] np [00:01] frz (~frz@213.235.213.90) joined #vserver. [00:03] hi frz! [00:10] ok 2 maschines, one with 1.22 and one with 1.23 runned sync [00:10] the same [00:10] 1.22 running, 1.23 lower then 1min and total crash [00:11] okay ... that is something we can work on ;) both UP? [00:12] jep [00:14] okay, we do that in a binary tree way ... [00:14] I cut out about half the changes made in 1.23 (make sure it works) and we test that one, okay? [00:14] stupidawy (foo@198.77.239.131) joined #vserver. [00:15] hi stupidawy! [00:15] Nick change: stupidawy -> stupidnic [00:16] atlanta, you.wish.you.were.pimp.olicio.us? [00:16] ? [00:16] come on... don't pick on my hostmask - it is 1337 :) [00:16] besides it is run off a vserver... that should count for something [00:17] ok bertl [00:17] v3ry l33t ... [00:17] can u send me the new diff? [00:17] sec [00:18] JonB (~NoSuchUse@129.142.112.33.ip.tele2adsl.dk) left irc: Remote host closed the connection [00:18] okay stupidnic, how is your vserver today? ;) [00:18] Good actually... just upgraded to 2.4.24-1.3.4 fix01 [00:19] hmm, better than 1.3.5? [00:19] have one minor issue... and I mean it is rather minor [00:19] Bertl: well considering that 1.3.5 was released the day after I finished my upgrade to 1.3.4 I decided I would let things settle down first :) [00:19] ah okay ... ic [00:19] what is your issue? [00:20] Sh[a]de: status (mine): currently testing the patch ... should be done in a few ... [00:20] when I start a vserver... I am getting this: /usr/local/sbin/vserver: ulimit: cannot modify max user processes limit: Invalid argument [00:20] bertl np, we stress only kernel's and not bertl's ;) [00:21] but only if ULIMIT is defined (obviously) [00:21] stupidnic: change the -H to -HS or check if you use -n with values above 1024 [00:21] Okay... I had added the S, but not the above 1024 [00:21] currently set to 1000 [00:22] (which is what it was set to ogirinally when I started runing ctx17 [00:23] let me have a look at your ULIMIT= line in the config please ... [00:24] sure... [00:24] ULIMIT="-HS -u 1000" [00:24] and give http://vserver.13thfloor.at/Stuff/testme.sh a try just to see everything is working fine ... [00:25] do I run that inside a vserver? or just the root server? [00:25] on the host as root ... [00:26] bertl... [00:26] thats interessting [00:26] -s [00:26] sry i have it loaded too ;) [00:26] all clear but one failed [00:26] [201]# failed. [00:27] on stable 201 failing is natural ... [00:27] ok *relief* [00:28] Bertl: on the root server all tests pass with succeeded [00:28] Bertl, i have to go short outside [00:29] paste me the link in querry [00:29] i do the tests [00:29] im back in aprox 30min [00:29] http://vserver.13thfloor.at/Experimental/delta-vs1.23-rev01.diff [00:30] what do the first 5 lines show? [00:31] Doener (~doener@pD9588138.dip.t-dialin.net) joined #vserver. [00:31] stupidnic: what do the first 5 lines show? [00:31] hi Doener! [00:31] Linux-VServer Test [V0.06] (C) 2003-2004 H.Poetzl [00:31] chcontext is working. [00:31] chbind is working. [00:31] Linux 2.4.24-vs1.3.4 i686/chcontext 0.27/chbind 0.27 [E] [00:31] --- [00:32] okay, shutting down and restarting that vserver gives you what messages? [00:32] come to think of it... I get it on enter too: /usr/local/sbin/vserver: ulimit: cannot modify max user processes limit: Invalid argument [00:33] granted... I haven't made the modifications you suggested about putting the limit number over 1024 [00:33] well I didn't suggest that ... [00:33] do you use nproc as flags? [00:33] affirmative [00:34] is this unmodified 1.3.4? on vanilla 2.4.24 or with some other patches? [00:35] totally vanilla 2.4.24 and the only patch to 1.3.4 was the fix01 (to fix the context numbering problem [00:36] hmm, which patch was that? [00:36] hio i've a problem with lsof in 2.4.23/vs 1.23 [00:36] It has to do with a bug in 1.3.4 where the context number won't increment the next time it was invoked. [00:36] hi [00:36] lsof brings up some warnings which contain the filestructure of the host system [00:36] for example: [00:37] stupidnic: hmm, interesting could you make that one available? [00:37] Sure... hold one [00:37] test-1:/var/www# lsof |grep mysql [00:37] lsof: WARNING: can't stat() ext3 file system /vservers/tooltime [00:37] Output information may be incomplete. [00:37] test-1 is a context below /vservers/test-1 [00:37] rmoriz: okay what is in your /porc/mounts on that server? [00:37] host or ctx? [00:38] s/porc/proc/ [00:38] vserver context [00:38] -> /dev/vserver_vol/tooltime /vservers/tooltime ext3 rw 0 0 [00:38] [00:39] want to have ssh access? :) [00:39] not required, I assume that is the reason for lsof complaining ... [00:39] is there a way to hide that mounts to vserver contexts? [00:39] as you are using vs1.23, you could use the vproc tool to disable the /proc/mounts entirely ... [00:40] oh. didn't try that ;) [00:40] vproc -d /proc/mounts on the host should do the trick ... [00:42] thanks! :) [00:42] Bertl: http://n0rp.floridachicks.com/patch-vs1.3.4-fix01.diff [00:42] don't mind the domain name :) [00:47] adding the S to the ULIMIT seems to sort out the problem [00:50] ah okay that is the nproc patch ... [00:53] Sorry... I didn't bother to read it... (I just did) [00:57] which package contains vproc? [00:57] i don't seem to have it with util-server-0.27 [00:58] Nick change: ccooke -> cc-away [01:09] Nick change: stupidnic -> stupidawy [01:23] stupidawy: it's an addon, available at the download site [01:36] any progress yet? [01:38] hmm, regarding 1.23, I assume? [01:39] yes [01:39] and/or other security/stability related issues [01:40] it would be interesting to test with 1.22 as it seems, that Sh[a]de was able to reproduce this with 1.23 on UP ... [01:40] i saw a patch vs 1.3.4 [01:40] ok [01:40] and it would also be very interesting to test with 1.3.5 [01:40] so i schedule tests with 1.22 and 1.3.5 for tomorrow? [01:40] yeah, preferable 1.3.5 first ... [01:42] ok [01:42] Last message repeated 1 time(s). [01:42] ill do one kernel right now [01:43] re [01:44] oki, patch is available ... [01:44] http://vserver.13thfloor.at/Experimental/delta-vs1.23-rev01.diff [01:44] so bertl [01:44] im ready for test ;) [01:45] this rips out the part, which actually was intended to fix the race issues once and for all ... [01:45] (so it's a good candidate for looking at) [01:49] bertl [01:49] sounds not good [01:50] see here [01:50] wwipvmdev-01:/install/1.23rev01-kernel/linux-2.4.24# patch -p1 < delta-vs1.23-rev01.diff [01:50] patching file Makefile [01:50] Hunk #1 FAILED at 1. [01:50] 1 out of 1 hunk FAILED -- saving rejects to file Makefile.rej [01:50] patching file fs/proc/array.c [01:50] Reversed (or previously applied) patch detected! Assume -R? [n] [01:50] you are patching against what? [01:50] a fresh kernel [01:50] 2.4.24 [01:51] with ur patch [01:51] it's agains vs1.23 ... so you have to apply that first ;) sorry [01:51] oh ok [01:51] np [02:08] patch 1.22, 1.23 to kernels 2.4.23 and 2.4.24 is giving socket.c1111 when attempting dig or nslookup on the host server [02:09] please provide strace of dig/nslookup in such case ... [02:09] I ust compiled 2.4.24 without the patch and dig works [02:09] Hi Bertl: [02:10] I did few days back remember [02:10] building the kernel [02:10] okay, that was the modloader, right? [02:10] 2.4.24-vs1.3.5-295 [02:11] serving: I have an idea why it works/doesn't work ... [02:11] ? [02:11] but I need another strace of that command ... [02:11] ok . [02:12] udp.c: In function `udp6_get_info': [02:12] udp.c:985: structure has no member named `vx_id' [02:12] okay, change the vx_id to xid [02:13] atd [02:13] you have ipv6 enabled, by the way? [02:13] wrong window [02:13] Bertl: you need a new strace out put or you wana look at the old ones ? [02:13] both should do ... [02:14] bertl [02:14] i have run the killer-03 on my remote maschine [02:14] right now . I am running th box with 2.4.24 clean. I will reboot to run 2.4.24-vs1.23 [02:14] it run without problems but only one thing is interesting [02:14] fs/fs.o(.text+0x2d544): In function `proc_virtual_readdir': [02:14] /home/raoul/test/src-vs.1.3.5-2.95/linux-2.4.24/fs/proc/virtual.c:415: undefined reference to `__cmpdi2' [02:14] mhm [02:15] ? [02:15] killer-03 hang for few seconds and running again [02:15] sry, not wanted to interrupt you! [02:16] Bertl: 213.186.190.24/yahoo4 [02:17] Bertl: sorry . http://213.186.190.24/vserver/yahoo4 [02:18] this is within the vserver, right? [02:21] Bertl: Killer-03 is running [02:21] bertl: sorry that everybode wants something from you [02:22] fs/fs.o(.text+0x2d544): In function `proc_virtual_readdir': [02:22] undefined reference to `__cmpdi2' [02:22] Bertl: here is the new starce http://213.186.190.24/vserver/yahoo5 [02:22] maharaja: hmm that is interesting ... [02:22] Bertl: no they are done in the host server [02:22] i already am sorry for him. [02:23] serving: hmm interesting too ... [02:23] the last lines of dmesg are? [02:24] hmm i think thats was a "bugfix" for the stressing problem, killer-03 still running without probs [02:25] Bertl: request_module[net-pf-10]: fork failed, errno 1 [02:25] shade: which patch? [02:25] shade: if its for 1.23, i would like to try [02:25] http://vserver.13thfloor.at/Experimental/delta-vs1.23-rev01.diff [02:25] yes 1.23 [02:26] with the patch from bertl [02:26] thnx [02:26] right, this one [02:26] without this patch my system was only < 1min stressed and then crashed [02:27] serving: hmm does 1.22 give the same issues? [02:27] yes [02:27] but on 2.4.24 vanilla, no problem, no dmesg, right? [02:27] shade: does a "make bzImage" suffice? or do i need a make clean too [02:28] I did not try dmesg but nslookup works fine [02:28] no make clean, but a make dep [02:28] in 2.4.24 vailla [02:28] maharaja: i always make "fresh"! kernels [02:28] right bertl [02:28] ok [02:28] and im happy, very happy :) [02:28] killer running and running [02:29] .oO(i should be studing *g*) [02:29] Sh[a]de: hmm, well, we just _removed_ the race protections in that version ;) [02:29] u guys, specialy bertl, do great things !! [02:29] k [02:30] so that should prove kind of unhealthy, or am i wrong [02:30] maharaja: you mean studying right ? [02:30] good question [02:30] serving: http://www.ussg.iu.edu/hypermail/linux/kernel/0206.1/0009.html [02:30] is 1.23 now on 1.22 level, no or? [02:30] i m there [02:31] please touch /var/log/ksymoops [02:32] rebooting [02:32] serving: try again (on vs1.23) and look what's written there ... [02:33] serving: you have ipv6 enabled? or compiled in? [02:33] Bertl: I read this few days back a dcould not figure out a solution for it. How come th eproblem only manifest itseld when running patched kernles ? [02:33] shade: you run killer-03 1/0/? [02:33] bertl: is killer-03 still ident? [02:33] maharaja: killer-03 [02:34] hmm, it was updated twice, so reload ... [02:34] ok [02:34] DaemonDazz (~spam@WYPP-p-144-134-3-182.prem.tmns.net.au) joined #vserver. [02:34] started "./killer-03" [02:34] hi DaemonDazz! [02:34] Bertl: ipv6 is compiled as modules I think . I am using config file from ftp.soluccorp.. [02:34] allo all [02:34] Thumb squeeze for maharaja ;) [02:35] hi Daemon [02:35] *reading some qm stuff [02:35] Bertl: I just replied to your email requesting an strace of host (Subject Socket.c Invalid Arguement) [02:35] ah, good we are working on that right now ... [02:36] ./killer-03 is still running [02:36] bb in 2mins or so [02:36] me too maharaja [02:37] DaemonDazz, serving: could you please try to insert the ipv6 module manually .. via modprobe [02:37] and then try again with the dig/host/nslookup/whatever ... [02:37] not sure if i built it, hang on i'll check [02:38] well, I'm sure ;) [02:38] Bertl: line in modprobe ipv6 ? [02:38] yep, i did ;) [02:39] don't know how it's called, what is in /lib/modules//kernel/driver/net ... [02:40] Bertl: I did modprobe ipv6 the nslookup yahoo.com and I got socket.c:1111: internal_send: 192.168.0.35#53: Invalid argument [02:40] is the ipv6 modules loaded? [02:40] or do you still get the request_module[net-pf-10]: fork failed, errno 1 message? [02:40] hmm killer now running > 20 min [02:40] Bertl: lsmod > ipv6 175220 -1 [02:41] seems to be ok [02:41] -1 okay? [02:41] the ipv6 loadded ok, the strace is at http://fuzzitech.net/host.strace.ipv6 and there is no more messages in dmesg after loading the module [02:42] [12:42am] hmm killer now running > 20 min <- seems to be ok [02:42] if u think this was my comment to Bertl: lsmod > ipv6 175220 -1 [02:42] ,) [02:43] Bertl: no more error request_module[net-pf-10]: fork failed, errno 1 [02:43] but the dig still fails? [02:43] yep [02:43] yep [02:43] stereo [02:43] same error [02:43] okay, I would ask you to compile the kernel without ipv6 support (just for a test) and try again ... [02:43] bertl: "./killer-03" is happily up and running [02:44] bertl: ill try "./killer-03 1" [02:44] with me also [02:44] Bertl: ok. i am off to recompile [02:44] maharaja: no, not necessary/useful ... [02:44] ok [02:45] what is usefull? [02:45] maharaja: you could start some loops, doing a cat /proc/*/status >/dev/null; endlessly on the host, while running killer-03 [02:45] ok [02:45] something like 'while true; do cat /proc/*/status >/dev/null; done ... [02:45] ill do that via ssh, to keep the console for debugging [02:45] yup [02:47] ok, running happily [02:48] i get a lot of: cat: /proc/20652/status: No such file or directory [02:48] but i guess thast from the fork [02:48] the second instance of that while loop produces no output of missing files/dirs [02:49] when i remove the /dev/null, i get the status as expected [02:49] yeah, you could try '(cat /proc/*/status >/dev/null &) inside the while loop, to push a little harder [02:53] ok running 2x cat loop and one killer-03 [02:53] without probs [02:53] same [02:53] CPU states: 26.6% user, 73.4% system, 0.0% nice, 0.0% idle [02:53] *g [02:53] no fork yet [02:54] 02:55:29 up 20 min, 3 users, load average: 173.37, 147.43, 89.87 [02:54] yeah, on UP this should not be a problem, on SMP this should hang after a while ... [02:54] ill do that & stuff [02:54] and run another killer-03 [02:54] maharaja: 2x killer & 2 cat loops? [02:55] setting it up [02:55] more than one killer doesn't make too much sense, more bashing on the procfs makes sense ... [02:55] ok [02:55] 2x with &, 1x the simple loop [02:56] serving: there is no line like the following one, in dmesg? [02:56] "kmod: failed to exec %s -s -k %s, errno = %d\n", [02:57] 4x bashing the procfs [02:57] 97,3% load and all running fine [02:57] 02:58:51 up 24 min, 5 users, load average: 68.73, 116.99, 89.99 [02:57] 2 loops, 1 loop & loop and 1 killer [02:57] how do you get that cpu load stuff? [02:57] top? [02:57] jep [02:57] Cpu(s): 14.2% user, 85.6% system, 0.0% nice, 0.2% idle [02:58] i'm suprised, that the system is very reactive [02:58] jey i can open more and more ssh windows without probs [02:58] login and other :) very nice [02:58] cause i thought that after a certain load, it gets really hard to interact [02:59] did you implement some schedule patches? [02:59] ok [02:59] i get debug output [02:59] <1>Unable to handle kernel NULL pointer dereference at virtual address 00000000 [03:00] interested in more? [03:00] printing eip: [03:00] c01686ce [03:00] *pde = 00000000 [03:00] Oops: 0000 [03:00] CPU: 1 [03:00] and going on with the register/stack/.. [03:00] yeah, that looks more like it ... [03:00] save it somewhere ... [03:00] we'll need to put it through ksymoops ... [03:01] Bertl: no there isnt such aline [03:01] okay, just checking ... [03:01] damn putty! [03:02] i increased the history size [03:02] dang p3 takes a long time to compile a 2.4 kernel [03:02] and my screan went black :) [03:02] so i'm waiting for more of that stuff? [03:02] :) [03:02] or is there any way the geht the output of minicom back? [03:02] well, probably not ... [03:03] 1x winscp data transfer (2gigs), a dig loop to web.de, 2 double cat, 2 normal cat and one killer-03 -> still stable [03:03] Sh[a]de: vs1.22 will not break on UP ;) [03:03] what I don't understand, why does vs1.23 break ... [03:04] cat_stress.sh: line 3: 11202 Killed sh cat_stress.sh [03:04] mhm [03:04] damn [03:04] will the oops reappear? :) [03:04] uhhhm [03:04] ok my system hangs [03:04] on start apache [03:04] too [03:05] sure? [03:05] ctx: 65258 [03:05] 2631 [03:05] thts the output from killer [03:05] +a [03:05] and he count [03:06] ctx: 65258 [03:06] 2683 [03:06] thats the only thing... [03:06] now i have killed all cat stress tests [03:06] but no change on killer [03:07] well, maybe killer was killed, that might be possible ... [03:07] but the system is responsive? [03:07] no [03:07] now killer and all terminals stand still [03:07] one sec [03:07] ah [03:07] the stuff is saved in dmesg [03:07] :) [03:08] i have last top cpu at 100% on system [03:08] killer killed himself [03:08] yes i have access for now [03:09] all works [03:09] well, it was probably killed by the OOM killer ... [03:09] no crash [03:09] killed all other tests and system is stable [03:09] okay, I'll modify the patch and we test again, okay? [03:09] sure [03:09] np [03:10] you want my oops? [03:10] i'm going to be building a new host server in a week or so, what kernel version and patch version would you recommend? [03:11] hmm im not sure [03:11] 1.22 works fine for our server [03:11] DaemonDazz: atm, 2.4.23+vs1.22 ... [03:11] :) [03:11] bertl... why not 2.4.24? [03:11] with 1.22 [03:12] how much difference is there between 1.21 and 1.22 ? [03:13] i like the kernel ppl: Hey! Please login now. You have one minute left [03:13] :) [03:13] or the login ppl [03:13] h��? [03:14] ;) [03:14] i simply like the tiny gimmicks they add [03:14] *g [03:14] DaemonDazz: in a week or so, I probably recommend 2.4.25 + vs1.24 ... [03:14] is that going to fix what we're looking at now? ;) [03:14] yeah, for sure ... [03:15] if I can track it down ... [03:15] it's now recompiling modules [03:16] ipv6 isn't supported yet, but it never gave problems before, maybe the tools changed a little, and now that missing support breaks them ... [03:17] ahh. one other thing i just remembered. i've added fakeinit to a RH9 vserver so i can use the inittab and now i get the following message whenever i start/stop/enter [03:17] /usr/lib/vserver/printconf.sh: G: command not found [03:17] any ideas? [03:18] [root@hornet root]# rpm -qa | grep vserver [03:18] vserver-0.29-1 [03:19]