[00:12] nathan_ (~nathan@207.44.202.162) joined #vserver. [00:12] hi nathan! [00:12] hey herbert [00:12] can you link me to 1.3.2 again? [00:12] i tested and 1.3.1 and it crashed the same but i noticed it didnt have the same locking as .2 [00:13] when do you plan to test? [00:13] right now unless you advise not to [00:13] okay, I'll upload the latest patches ... [00:14] get Con's patches for 2.4.23 too, if possible .. [00:15] ok ill apply ck1 [00:15] and there is a fix ... available from Con [00:15] against ck1? [00:16] yeah, he has it in the fix dir .. [00:16] fix1, i see it [00:16] k patching it up [00:16] http://vserver.13thfloor.at/Experimental/patch-2.4.23-ck1-vs1.3.2.diff [00:17] http://vserver.13thfloor.at/Experimental/patch-2.4.23-vs1.3.2.diff [00:18] and if possible, enable magic sysreq, in the case that it locks up, I would be glad to get some dumps ... [00:18] maybe compiling with -g as kernel CC options would also help later on ... [00:18] Bertl, yea i still have that problem with the remote box though :/ [00:18] i have no access to this box physically or serial [00:18] and lkcd is disabling smp [00:18] so its kinda like, yea it crashed let me go look at the code and hunt blindly right now [00:18] it is a SMP machine, so enable the nmi_watchdog ... [00:19] this will at least get you a clean reboot, if it's a locking issue ... [00:20] ok all patched clean [00:20] what do you think of me ;) [00:21] you are the awesome :) [00:24] hehe [00:24] assuming this fixes the vx_info crashing issues ill hack up some tests for the ip code too [00:24] the awesome! [00:24] it will be nice to finally see this box stay up [00:25] yeah, I totally agree, and I appreciate your testing ... [00:25] I really appreciate it, thank you .. [00:26] hi dan! how was your xmas? [00:26] it was good :) how was yours? [00:27] thanks fine ... [00:49] ok ck1 vs1.3.2 is booted, running same tests again. [00:57] hmm i think i just lost the box [00:57] my tests were running and then i threw jonathans killer.c into the mix [00:57] hmm, not good ... [00:57] lost it after about 2s of output on killer.c [00:57] gonna give it a boot and do it just with killer.c [00:59] hmm i wish i could get lkcd working without killing the smp [01:00] what about network console? [01:00] do you have a second machine in the same network? [01:00] network console requires same subnet right? [01:00] IIRC yes [01:00] hmm i dont, i wonder if someone i know does though [01:01] it almost seems like lkcd is no longer maintained [01:02] lkcd isn't/wasn't a very good approach ... [01:03] has anything else come along for dump analysis akin to sun? [01:05] http://iven.home.cern.ch/iven/linux_debug/linux_crash/tools.html [01:06] hm checking out mclinux [01:06] never tried, but might be an option ;) [01:07] my google skills suck i guess, i couldnt find any reference to mclinux [01:09] You don't have permission to access /downloads/patches/ on this server. [01:09] Action: nathan_ scratches head [01:09] their approach maybe not so hot either? :) [01:11] http://oss.mclx.com/projects/mcore/ [01:11] http://oss.mclx.com/projects/crash/ [01:11] yep and then download->kernel patches->forbidden [01:12] maybe they turned off indexing by accident [01:12] user tools are still downloadable [01:13] based on what im reading on their mlist archives it looks like mcore and lkcd may have become one [01:13] in that case, the result is a regression ;) [01:15] i wonder if my friend will give me back my dual i donated to him [01:22] dead in 2s with killer.c [01:22] just tried it from a fresh boot [01:23] hmm, sounds good, in that case, I 'just' missed something important ;) [01:24] box is still alive but nothing is running in user space [01:24] and the nmi watchdog hasnt panic'd the box yet [01:25] what do you mean with still alive? [01:25] network is responding [01:25] ping coming back, tcp negotiating [01:25] okay, let me suggest, to go back to 2.4.23-vs1.3.2 to limit the possibilities ... [01:26] ok [01:26] have you tried killer.c on your smp? [01:26] nope, not yet ... [01:27] Bertl, the fact that it is responding on the network would lead me to believe that it is a semaphore that is being deadlocked on and not spinlocking? [01:28] not that it makes a difference or anything, just something im wondering about [01:28] could be, but vserver doesn't use semaphores, or am I wrong? [01:28] netrose (~john877@CC3-24.171.21.47.charter-stl.com) left irc: Ping timeout: 480 seconds [01:28] Action: Bertl is searching the code ... [01:29] how is task_lock and such implemented? [01:29] Action: nathan_ looks as well [01:29] yeah, task_lock is semaphore, right ... [01:30] it is still running something because the nmi watchdog should have it panic'd and rebooted by now [01:31] well, it is a fork bomb ... so maybe it is working as expected ... [01:31] hmm [01:31] Action: nathan_ looks at that code a little more closely [01:31] ill punch myself if ive just wasted your time on a trivial case [01:32] hey, no problem, if so I'm glad .. if not we find the issue and fix it ... [01:32] sigh i missed an extra fork() in there, you are definitely right. [01:33] maybe a recursion/fork limit would be a good idea ;) [01:33] yes id say so :) [01:34] lets pretend this never happened and ill go back to my original tests that were crashing 1.3.1 and 1.2.22 [01:34] okay, but try with 2.4.23-vs1.3.2 first ... [01:34] so just vanilla 2.4.23 with 1.3.2? [01:34] that code is based on the 'solid' scheduler [01:35] the ck1 has the O(1) scheduler ... [01:35] yea i saw. ok will do it vanilla + the 1.3.2 [01:36] http://vserver.13thfloor.at/Experimental/patch-2.4.23-vs1.3.2.diff [01:36] k [01:45] http://vserver.13thfloor.at/Experimental/killer-fix.diff [01:45] (untested, but might work ;) [03:15] morning [03:15] morning! [03:20] netrose (~john877@CC3-24.171.21.47.charter-stl.com) joined #vserver. [03:20] Bertl: where are the already existing chctx/lsctx tools? [03:21] those are patches to the ext2 tools .. sec I'll get you the url .. [03:21] http://vserver.13thfloor.at/Experimental/patch-e2fsprogs-1.34-cti0.01.diff.bz2 [03:22] the name of the IOCTL has changed since ... have a look at the kernel patches ... [03:35] Bertl, got a null pointer oops with /proc/pid/status [03:35] hmm .. ksymoops? [03:35] yep running now [03:36] >>EIP; c017c53d <===== [03:36] are there any 'suspicious' error messages in the logfile? [03:37] Bertl, nope nothing abnormal [03:37] hmm, I thought one of the vx_verify checks could have hit ... [03:37] Bertl, was .1 more verbose? [03:38] might have been extraneous i just remember some debugging [03:38] hmm, yes, you can enable debugging with VX_DEBUG ... [03:38] but the vx_verify hits will be reported regardless of that .. [03:39] ah [03:39] let me look how the error message shiould look like .. [03:40] 'vx bad assumption' ... [03:40] negative [03:40] well, I take that as a good sign, so my assumptions where correct ;) [03:51] nathan: did you compile ther kernel with debug info, as suggested? [03:58] I am trying to minimize the size of a vserver. Do I need /lib/modules/* inside a vserve ? [03:58] nope [03:58] Hi Bertl. then they are gone :) [03:59] Also modutils package. Destroy it if you still didn't. :) [04:00] where are they ? [04:01] What packagin system is that? RPM? [04:01] yes [04:02] rpm -e modutils :) [04:03] i am deleting dirs from outside the vserver. DO I need /usr/share/locale/* ? can I keep just en? [04:03] just to give you an idea of the lower bound ... [04:03] # du -skx /vservers/NAME/ [04:03] 81428 /vservers/NAME [04:03] (this is mandrake 8.2 based) [04:04] He-he. I had one around 40mb or so. [04:04] rpm? cool! [04:04] Simon (~sgarner@apollo.quattro.net.nz) joined #vserver. [04:04] No-no. With debian for sure. [04:05] ah, yes with debian you can go lower ... [04:05] a standard install or did you remove dirs by hand ? [04:05] Not dirs but packages. [04:05] By removing modutils, console-tools, manpages etc. [04:06] i c [04:06] ah yes, this is without any manual removal, just package selection ... [04:06] This package selection ended with absent /etc/protocols. That was fun. :) [04:07] nathan_: still around? [04:15] kestrel (~athomas@dialup51.optus.net.au) left irc: Ping timeout: 480 seconds [04:25] Bertl, yea [04:25] did you compile with -g? [04:26] doing it now [04:26] forgot to do a make modules [04:26] okay, you didn't change the config did you? [04:26] nope [04:27] good, use addr2line on the new built kernel to get me the file/line for the oops above, please ... [04:27] yep [04:46] hmm well VX_DEBUG wasnt the hottest idea [04:46] that's why it is off by default ;) [04:47] rebuilding :-/ [04:52] Tamama (~Tamama@a62-216-20-152.adsl.cistron.nl) joined #vserver. [05:14] netrose (~john877@CC3-24.171.21.47.charter-stl.com) left irc: Ping timeout: 480 seconds [05:21] Tamama (~Tamama@a62-216-20-152.adsl.cistron.nl) left irc: Quit: one little two little three little piggies OINK! OINK! OINK! [05:36] ok well killer.c with a loop to 100 around start_ctx() killed the box [05:36] appears to be panic'd this time [05:37] could you ran the 'old' oops through ksymoops and use addr2line to fix the line? [05:37] yep lemme wait till the box comes back up [05:47] hmm what am i doing wrong here, addr2line is not giving me anything useful for any address [05:47] cmdline? [05:48] what are you typing? [05:48] addr2line -e /usr/src/linux/vmlinux c0205e98 [05:48] should be an acpi function [05:48] just giving me ??:0 [05:49] i pulled out that symbol address with nm [05:49] CFLAGS_KERNEL = [05:49] CFLAGS_KERNEL = -g ? [05:49] hmm I did it with HOSTCFLAGS [05:49] needs to be at CFLAGS_KERNEL? [05:50] host is where you compile it ... [05:51] ok im rebuilding it with -g on the CFLAGS_KERNEL [05:55] much better [05:56] >>EIP; c017c53d <===== [05:57] /usr/src/linux-2.4.23/fs/proc/array.c:155 [05:57] first one was array.c:135 [05:58] much better ;) [06:01] the first is a false positive, you seem to have changed something .. the second at 155 is valid .. [06:07] kestrel (~athomas@dialup51.optus.net.au) joined #vserver. [06:16] hmm not sure what ive changed between the builds [06:16] ill try to produce it again on the live kernel [06:25] nathan? [06:25] try this 'trivial' patch ... [06:25] --- fs/proc/array.c.origSun Dec 28 01:47:15 2003 [06:25] +++ fs/proc/array.cTue Dec 30 04:26:00 2003 [06:25] @@ -152,10 +152,12 @@ static inline char * task_state(struct t [06:25] pid_t ppid; [06:25] read_lock(&tasklist_lock); [06:25] ppid = p->p_opptr->pid; [06:26] +task_lock(p); [06:26] if (ppid != 0 [06:26] && current->vx_info [06:26] && current->vx_info->vx_initpid == ppid) [06:26] ppid = 1; [06:26] +task_unlock(p); [06:26] buffer += sprintf(buffer, [06:26] "State:\t%s\n" [06:26] "Tgid:\t%d\n" [06:26] g'night folks [06:26] night .. [06:26] Nick change: doener_aw -> doener_zzz [06:30] okay nathan, I have to go to sleep now .. please let me know if this 'stabilizes' vs1.3.2 for you, if not, send me the oopses you get, (ksymoopsed and addr2lined) via email ... (herbert@13thfloor.at) .. thanks for your time and testing ... [06:30] night everyone! [06:30] Nick change: Bertl -> Bertl_zZ [07:06] netrose (~john877@CC3-24.171.21.47.charter-stl.com) joined #vserver. [08:35] Simon (~sgarner@apollo.quattro.net.nz) left irc: Read error: Connection reset by peer [08:38] Simon (~sgarner@apollo.quattro.net.nz) joined #vserver. [10:33] noel_ (~noel@pD9FFA7E6.dip.t-dialin.net) joined #vserver. [10:40] noel- (~noel@p50859BE0.dip.t-dialin.net) left irc: Ping timeout: 493 seconds [11:45] Nick change: Bertl_zZ -> Bertl [11:45] morning everyone ... [11:45] nathan_: are you here? [11:47] <_Zoiah> Off-topic question: How do I change the ordering of the loading of kernel drivers? I have an ATARAID driver that depends on a SCSI driver (libata, for sata support) but IDE is initialized before SCSI if I load both in the kernel so it can't access the RAID array then yet. If I build the ATARAID as a module and load it later, it does wor. [11:48] hmm, why is that a problem? [11:48] I mean the ordering? [12:27] hmm ... _Zoiah? [12:47] Simon (~sgarner@apollo.quattro.net.nz) left irc: Quit: so long, and thanks for all the fish [13:20] serving (~serving@213.186.190.99) left irc: Ping timeout: 480 seconds [13:36] LL0rd (~dr@pD9507ED1.dip0.t-ipconnect.de) left irc: Read error: Connection reset by peer [14:16] _MedivhWrk (ck@netops.multimedia-centrum.de) joined #vserver. [14:16] MedivhWrk (ck@netops.multimedia-centrum.de) left irc: Read error: Connection reset by peer [14:16] Nick change: _MedivhWrk -> MedivhWrk [14:29] infowolfe (~infowolfe@pcp04891550pcs.frnkmd01.md.comcast.net) joined #vserver. [14:29] Bertl: ping? [14:29] pong! (or was it echo-reply?) [14:29] lol [14:29] pong works [14:29] did you have a nice holiday? [14:30] holiday? [14:30] ahh .. you mean xmas? [14:30] yah [14:30] lol [14:30] that non-denominational wintertime thing [14:30] yeah, had a good one ... [14:30] wonderful [14:30] and you? [14:30] ... i worked :-p [14:31] well, I worked too ... on vservers and it is still fun! [14:31] oooh, anyway, why i'm here... [14:31] Action: infowolfe is lamebrained today [14:31] could you point me in the right direction for some copy on write stuff? [14:32] as you probably know, i'm running gentoo vservers... [14:32] hmm, what do you ahve in mind? [14:33] I'd like to make a single skeleton that I keep updated with most system stuff (gcc, glibc, apache, etc etc etc) and then make configs and whatnot , things that are individual per vserver be part of each individual [14:34] how does gentoo packet management work? [14:34] s/packet/package/ [14:34] so i have one big boy that is the only one i need to keep updated and as long as i don't replace gcc/glibc/apache/php/postfix on the vservers and i restart them after the update, i don't use the same diskspace 5 times (yah, i'm up to 5 now :-p) [14:35] well, the backend is python... (/var would be one of the things that is non-replicating) [14:35] hmm, the question is, are you the sole maintainer of those vservers? [14:35] yes [14:36] okay, so whatever you do/plan, it won't interfere with other administrators ... [14:36] nope [14:36] i'm the only one with root on my machine [14:37] i give redhat vservers to people i don't care about and keep gentoo to myself (debian is the next new distro and that's for my boss) [14:37] in that case, I would suggest to use unification and to do updates in all servers at once ... [14:37] umm [14:37] but that doesn't help the disk space problem [14:37] sure it does ... [14:37] because i'm keeping about 2 GB of data for each vserver [14:37] (afaik) [14:37] magic word was 'unification' [14:37] ok [14:38] ... so how does that work :-p [14:38] I don't know how the gentoo package system works, but basically all you ahve to do is: install a apckage in all vserver, then (somehow) get a list of 'installed' files (for that package) ... [14:39] easy enough [14:39] and then do the unification for the files, which are not config or log files ... [14:39] or to be more precise, the files that won't change till the next update [14:40] (and wont differ between the vservers ;) [14:40] ok [14:40] what do i do with that list? [14:40] you replace each file, in all vservers, with a hardlink to the first one ... [14:41] lol, ouch [14:41] and modify the flags, setting immutable and ili ... [14:41] that is called unification ... [14:41] what's illi? [14:41] or ili [14:41] ILI = Immutable Link(age) Invert [14:42] ah [14:42] so it's time for me to read the vuinfy script? :-p [14:42] Immutable files are not modify/deletable in vservers [14:42] but the ILI flag weakens that, so it can be deleted. [14:42] ok [14:43] how do i set these flags? [14:43] and is there a way for me to hardlink across filesystems? or no [14:43] there is a tool which does that (in the (util-)vserver package) [14:43] no hardlinks won't work across filesystems ... [14:43] ooooh [14:43] hrm... [14:44] here's a scenario [14:44] I update apache due to some flaw allowing it to be used as a bouncing board to fry cpus or something ludicrous... [14:44] inside the skeleton [14:45] yet the old version that is still vulnerable is still running in memory on the vservers... how do i update the vservers to allow me to restart apache without taking the vservers down [14:45] I have to leave in a few minutes, (for lunch), are you here in, let's say 20minutes? [14:46] yup [14:46] okay, we discuss this then, right? [14:46] ok [14:46] thanks [14:47] Nick change: Bertl -> Bertl_oO [14:51] noel_ (~noel@pD9FFA7E6.dip.t-dialin.net) got netsplit. [15:02] noel_ (~noel@pD9FFA7E6.dip.t-dialin.net) got lost in the net-split. [15:12] serving (~serving@213.186.190.138) joined #vserver. [15:22] Madkiss (madkiss@madkiss.org) left irc: Remote host closed the connection [15:24] Nick change: Bertl_oO -> Bertl [15:25] infowolfe: okay, I'm back ... [15:27] noel_ (~noel@pD9FFA7E6.dip.t-dialin.net) joined #vserver. [15:27] Topic changed on #vserver by !unununium.oftc.net: http://linux-vserver.org/ || latest stable 1.22, devel 1.3.1 "Revolutions". [15:32] yay [15:32] sorry, didn't see ya blinkin [15:32] np [15:32] ok, as we speak i'm building the skeleton [15:33] which is turning out to be very easy (i love gentoo :-p) [15:34] good ... [15:34] how do you update sshd for example? [15:34] it's pretty easy to set and forget [15:34] update sshd? :-p [15:34] just an example .. [15:34] emerge -u openssh [15:34] and what does emerge -u openssh do, in detail? [15:34] to install it'd be just plain emerge openssh and to uninstall emerge unmerge openssh [15:35] the way gentoo' [15:35] fine, what does it do? [15:36] the way gentoo's portage system works... in a nutshell, it's got a python backend that parses an ebuild (plain text) file [15:36] the text file name defines the package's version and name [15:36] okay, okay, spare me the bloody gentoo details ;) what happens with the files? configs? etc? [15:37] the ebuild file allows portage to build the package from source with on the fly dependancy checking/satisfaction, configuration options and more [15:38] so it is built where? [15:38] within the skeleton server? [15:38] portage does the builds in /var/tmp/portage/packagename-version/ installs all files there and then syncs that with the live system and then removes any files that weren't replaced [15:39] (but are not needed anymore, like /usr/share entries) [15:39] i'm going to be doing the builds in a skeleton that i'm setting up right now [15:39] the places i will NOT need to unify: /etc /home /var [15:39] okay, so basically you have a 'before' update, and an 'after' update version of the skeleton, right? [15:39] nothing that is per-vserver specific goes anywhere else [15:40] yah [15:40] i think [15:40] the skeleton is the one i'd like to unify everything else against [15:41] good .. so what I would do, in your case (if I don't have a good package system like RPM ;) ... [15:41] when the skeleton is updated, (let's say for glibc update or gcc update) there will be a ton of files changed [15:41] good? rpm? same sentence? you're insane! [15:41] Action: infowolfe laughs [15:41] I would run a comparison between 'pre' and 'post' versions ... [15:42] alright... [15:42] and try to figure out, which files are new and not server specific ... [15:42] let's say I make a skeleton for redhat 9 [15:42] sorry RPM provides all this information in each package ... [15:43] there is no need to do that for RH9 [15:43] i can get the same information via qpkg -i [15:43] except my host doesn't have rpm installed NOR will I install it [15:43] mhepp (~mhepp@r72s22p13.home.nbox.cz) joined #vserver. [15:43] what does qpkg give you? [15:43] give me a moment [15:44] i'll give you a link of apache for example [15:44] http://thor.hardcore-linux.net/apache.txt [15:45] (it'd be best to wget that and less it) [15:45] hmm, might be colorful, but what is what? [15:45] it's a listing of every file installed by the apache package by default [15:45] and full path :-D [15:45] the top 2 lines can be snipped [15:45] yeah, but no info about the file type ... means (is a config file, is a executable binary, etc) [15:45] the first line is packagename [15:46] well, if it's in /etc it's config [15:46] and if it's in /var then it's ? [15:46] if it's /var/ it's data [15:46] unifyable or not? [15:46] not [15:47] okay, then you have to setup your own 'rules' to decide that ... [15:47] the stuff in /var is like /var/lib/mysql /var/www//htdocs, etc etc [15:47] if you can do that, fine ... [15:47] do you know of any documentation that will allow me to use a find to do the unification? [15:47] noel_ (~noel@pD9FFA7E6.dip.t-dialin.net) got netsplit. [15:47] no, but it is quite simple to do the unification ... [15:48] like find {/usr,/bin,/sbin,/lib} and then unify everything that matches? [15:48] I wouldn't do that ... [15:48] so it'd be easy for me to setup via bash [15:48] why not? I won't be updating anything *inside* the vservers except configuration files and data [15:49] sure, and then you decide not to update server 3 and the next time you 'unify' bad things happen ;) [15:49] it wouldn't happen [15:49] I would advice the following procedere ... [15:49] i'm looking to script it [15:49] and run via cron once a week [15:50] not useful .. as it is best 'event' triggered .. [15:50] ok [15:50] a) you update the skeleton [15:50] b) you get a list of _all_ changed files [15:51] c) you get a list of all unifyable files (subset) [15:51] ok [15:51] so what would i use to compare changed vs unchanged? [15:51] 2 skeletons? [15:51] d) you remove one file after the other and either replace it by a copy or hardlink to the skeleton [15:51] mhepp (~mhepp@r72s22p13.home.nbox.cz) left irc: Quit: Tak ja padaaaaM [15:52] (for each vserver) [15:52] e) for all unified files after all hardlinks are done, the IMMUTABLE and ILI flags are set [15:52] f) you are done. [15:52] alright... [15:53] the question stands, how do i get a "snapshot" of what i had before the update? [15:53] make a copy ... if possible (with gentoo) a shallow one ... [15:53] shallow? [15:53] cp -la [15:54] if gentoo's build system f** up the shallow copy, make a real one ... [15:54] that would all be done outside the chroot though... only the actual update would be done inside the chroot [15:55] does a shallow copy keep track of filesizes and modified times? [15:55] you could also put the skeleton on a lvm volume, and make a snapshot, but then you have to keep it ununified ... [15:55] man cp [15:56] beautiful [15:56] so cp -la skeleton snapshot [15:56] when the skeleton is happy... [15:57] and btw, i doubt gentoo's build system would f*** up the shallow copy :-D [15:57] you have to verify that the gentoo build system doesn't modify the files ... [15:57] doesn't modify which files, the shallow copy? [15:57] any files [15:57] it can remove and replace them, but not modify! [15:57] oh... lol [15:57] i can say honestly that it doesn't [15:58] noel_ (~noel@pD9FFA7E6.dip.t-dialin.net) got lost in the net-split. [15:58] RPM does this and other package systems (tar based) too ... [15:58] (not modifying, only replacing) [15:58] like i said, it doesn't modify anything... [15:58] it'll remove if outdated [15:59] let's say i emerge -u apache and it WAS 2.0.45 and is going now to 2.0.48 [15:59] okay, fine .. if unsure, I would check with a glibc update, and a deep copy to compare with ;) [15:59] how would you verify? simple ... [15:59] it'll REPLACE every file that both packages have in common and REMOVE any files unused by the new package [15:59]