Re: [Vserver] linux-vserver as a "poor man's warm standby solution"

From: Rúben Leote Mendes <ruben_at_nocturno.org>
Date: Sun 19 Mar 2006 - 17:45:38 GMT
Message-ID: <20060319174538.GH5416@innuendo>

Hello Marcel,

I don't know if this is also of interest to you, but you could use DRBD [0]
to perform a kind of network RAID1 that would keep both servers mirrored
in real time. Using heartbeat [1] you can detect if the other server is down
and start the remote vservers in the local server.

See also [2].

[0] http://www.drbd.org
[1] http://www.linux-ha.org/
[2] http://linux-vserver.org/Vserver+DRBD

Regards,
Rúben

On Sun, Mar 19, 2006 at 04:42:22PM +0100, Marcel Gsteiger wrote:
> Hi all
>
> I successfully installed two similar boxes running FC4 w/linux-vserver, implementing a hot-standby of each vserver on the other box. All VServers are simply installed on both boxes in the same directory structure; half of the servers are active on box A and standby on box B, the other half is active on box B and standby on box A. This is what I call a "poor man's warm standby solution", i.e. without the need for a SAN infrastructure. This setup does "load balancing" by having the active vserver on the CPU of my choice statically. On both boxes a cronjob runs once every hour (of course not both at the same time) that takes a LVM snapshot of the vserver partition and then uses rsync to mirror the active server(s) to the standby box. This setup works very well now for several weeks. Depending on the data changing and overall load, the hourly script needs about 1-5 minutes to sync 8 vservers in my case, together occupying about 23 GB of disk space (running on a x86_64 2.8GHz Dual Xeon box w/software RAID 1 SATA disks).
>
> For anyone interested, find my scripts below that do the work of syncing the servers and migrate a vserver from one mirror to the other (switchover).. As a prerequisite, each of the two boxes should be able to connect to the other one via SSH without using passwords, i.e. by public key authentication. The same scripts are installed on both servers, they automatically find out where they are running and what to do. I also have a .lan domain on my DNS to avoid having IP addresses in the script, but of course this can be adapted easily. In my own setup, I have a few additional site-specific scripts that take care of networking issues (firewalling, VLAN setup, static routes etc), this is necessary in my case since my vservers have RFC1918 IPs and are accessible from the outside ony via NAT.
>
> Syncing the /etc/vserver directory has to be done manually, my script just rsyncs the /vservers partition.
>
> Of course, the scripts come without any warranty whatsoever, all bugs have been added by me ;)
>
> Regards
> --Marcel
>
> <file rsync_vservers>
> #!/bin/bash
> # synchronizes all running vservers to mirror and checks the status.
> # Does not automatically launch vservers; this has to be done manually.
> # stdout can typically be piped into logger from within a cronjob, e.g.
> # /etc/rsync_vservers | /usr/bin/logger -t rsync_vservers
> # 3-MAR-2006 milprog/mg
> # 10-MAR-2006 milprog/mg : adapted to retry snapshot creation and wait 1 second before mounting
>
> if [ "$HOSTNAME" = "milprogmail.ch" ]
> then
> THISHOST=milprogmail.milnet.lan
> else
> THISHOST=$HOSTNAME
> fi
>
> MIRROR1=lepus.milnet.lan
> MIRROR2=milprogmail.milnet.lan
> # maximum 1 MB/s bandwidth please during rsync
> BWLIMIT=1024
> SNAPSHOTSIZE=4096M
>
> VSERVERVG=/dev/vg00
> VSERVERLV=lv00_vservers
> VSERVERSNAPSHOT=vservers_snapshot
> SNAPSHOTMOUNT=/mnt/vservers-snapshot
> VSERVERBASE=/vservers
>
> # no need to change anything below (hopefully)
>
> VSERVERCMD=/usr/sbin/vserver
> RSYNCCMD=/usr/bin/rsync
> LVCREATECMD=/usr/sbin/lvcreate
> LVREMOVECMD=/usr/sbin/lvremove
> MOUNTCMD=/bin/mount
> UMOUNTCMD=/bin/umount
> SSHCMD=/usr/bin/ssh
> LVSCMD=/usr/sbin/lvs
> AWKCMD=/bin/awk
> BASENAMECMD=/bin/basename
> DATECMD=/bin/date
> SYNCCMD=/bin/sync
> SLEEPCMD=/bin/sleep
>
> if [ "$THISHOST" = "$MIRROR1" ]
> then
> MIRROR=$MIRROR2
> fi
> if [ "$THISHOST" = "$MIRROR2" ]
> then
> MIRROR=$MIRROR1
> fi
> if [ "$MIRROR" = "" ]
> then
> echo "ERROR: VServer resync must be installed either on $MIRROR1 or $MIRROR2, not here ($THISHOST)"
> exit 1
> fi
> TIMESTART=`$DATECMD +%s`
> RETRIES=0
> MAXRETRIES=5
> $SYNCCMD
> if $LVSCMD --noheadings ${VSERVERVG}/${VSERVERSNAPSHOT} >/dev/null 2>&1
> then
> echo "ERROR: Snapshot overlap; cannot rsync from $THISHOST to $MIRROR"
> else
> # echo "creating LV..."
> # $LVSCMD
> if $LVCREATECMD -L${SNAPSHOTSIZE} -s --chunksize 4k -n ${VSERVERVG}/${VSERVERSNAPSHOT} ${VSERVERVG}/${VSERVERLV} >/dev/null 2>&1 && $MOUNTCMD -t ext2 -ni -o noatime,ro ${VSERVERVG}/${VSERVERSNAPSHOT} ${SNAPSHOTMOUNT} >/dev/null 2>&1
> then
> :
> else
> # echo "retrying to create snapshot..."
> while [ $RETRIES -lt "$MAXRETRIES" ]
> do
> RETRIES=`expr $RETRIES + 1`
> $SYNCCMD
> $SLEEPCMD 1
> $UMOUNTCMD -ni $SNAPSHOTMOUNT
> $SLEEPCMD 1
> $LVREMOVECMD -f ${VSERVERVG}/${VSERVERSNAPSHOT} >/dev/null 2>&1
> $SLEEPCMD 1
> $SYNCCMD
> $LVREMOVECMD -f ${VSERVERVG}/${VSERVERSNAPSHOT} >/dev/null 2>&1
> $SLEEPCMD 1
> if $LVCREATECMD -L${SNAPSHOTSIZE} -s --chunksize 4k -n ${VSERVERVG}/${VSERVERSNAPSHOT} ${VSERVERVG}/${VSERVERLV} >/dev/null 2>&1 && $MOUNTCMD -t ext2 -ni -o noatime,ro ${VSERVERVG}/${VSERVERSNAPSHOT} ${SNAPSHOTMOUNT} >/dev/null 2>&1
> then
> break
> fi
> done
> if [ $RETRIES -eq "$MAXRETRIES"]
> then
> echo "ERROR: cannot create snapshot after `expr $MAXRETRIES-1` retries"
> $LVREMOVECMD -f ${VSERVERVG}/${VSERVERSNAPSHOT} >/dev/null 2>&1
> exit 2
> fi
> fi
> NACTIVE=0
> NSTANDBY=0
> NINACTIVE=0
> NERROR=0
> for i in /etc/vservers/*; do
> VSERVER=`$BASENAMECMD $i`
> if $VSERVERCMD $VSERVER running
> then
> # echo "vserver $VSERVER running on $THISHOST"
> if $SSHCMD $MIRROR $VSERVERCMD $VSERVER running
> then
> NERROR=`expr $NERROR + 1`
> echo "ERROR: vserver $VSERVER running both on $THISHOST and $MIRROR"
> else
> NACTIVE=`expr $NACTIVE + 1`
> # echo "rsyncing vserver $VSERVER from $THISHOST to $MIRROR..."
> if $RSYNCCMD -aqzpotSx4 --numeric-ids --bwlimit $BWLIMIT --recursive --safe-links --delete --force -e ssh ${SNAPSHOTMOUNT}/${VSERVER}/ $MIRROR:${VSERVERBASE}/${VSERVER}/
> then
> # echo "rsync vserver $VSERVER from $THISHOST to $MIRROR done"
> :
> else
> echo "ERROR: cannot rsync $VSERVER from $THISHOST to $MIRROR"
> fi
> fi
> else
> if $SSHCMD $MIRROR $VSERVERCMD $VSERVER running
> then
> # echo "vserver $VSERVER running on $MIRROR"
> NSTANDBY=`expr $NSTANDBY + 1`
> else
> NINACTIVE=`expr $NSTANDBY + 1`
> echo "WARNING: vserver $VSERVER inactive both on $THISHOST and $MIRROR"
> fi
> fi;
> done
> $SYNCCMD
> if $UMOUNTCMD -ni $SNAPSHOTMOUNT
> then
> # echo "unmount of $SNAPSHOTMOUNT done"
> :
> else
> echo "ERROR: cannot unmount LV snapshot"
> fi
> ## else
> ## echo -n `$DATECMD +"%x %X" `
> ## echo " ERROR: cannot mount LV snapshot"
> ## fi
> SNAPPERCENT=`$LVSCMD --noheadings ${VSERVERVG}/${VSERVERSNAPSHOT} | $AWKCMD '{print $6}'`
> $SYNCCMD
> if $LVREMOVECMD -f ${VSERVERVG}/${VSERVERSNAPSHOT} >/dev/null 2>&1
> then
> TIMEEND=`$DATECMD +%s`
> echo "NOTICE: rsync_vservers complete; elapsedsecs=`expr $TIMEEND - $TIMESTART`; snapshotsize=$SNAPSHOTSIZE; snap%=$SNAPPERCENT; active=$NACTIVE; standby=$NSTANDBY; inactive=$NINACTIVE; error=$NERROR; snapshotretries=$RETRIES"
> # $LVSCMD
> else
> echo "ERROR: cannot remove LV snapshot"
> fi
> fi
> </file>
>
>
>
> <file migrate_vserver>
> #!/bin/bash
> # gets 2 parameters: vserver_name source_server
> # ensures that vserver_name is running on source_server and standby on the corresponding mirror
> # stops the vserver on source_server, does rsync of the remaining data and restarts on mirror
> # if only the vserver_name is given, the script assumes that the server it's running on should become the new active server
> #
> # 19-MAR-2006 milprog/mg
>
> if [ "$HOSTNAME" = "milprogmail.ch" ]
> then
> THISHOST=milprogmail.milnet.lan
> else
> THISHOST=$HOSTNAME
> fi
>
> MIRROR1=lepus.milnet.lan
> MIRROR2=milprogmail.milnet.lan
> SNAPSHOTSIZE=4096M
> VSERVERVG=/dev/vg00
> VSERVERLV=lv00_vservers
> VSERVERSNAPSHOT=vservers_snapshot
> SNAPSHOTMOUNT=/mnt/vservers-snapshot
> # parameters for taking the snapshot: number of retries, interval in seconds
> MAXSNAPSHOTRETRIES=3
> RETRYINTERVAL=250
>
> # no need to change anything below (hopefully)
>
> LVCREATECMD=/usr/sbin/lvcreate
> LVREMOVECMD=/usr/sbin/lvremove
> LVSCMD=/usr/sbin/lvs
> VSERVERCMD=/usr/sbin/vserver
> VSERVERROOT=/vservers
> RSYNCCMD=/usr/bin/rsync
> SSHCMD=/usr/bin/ssh
> AWKCMD=/bin/awk
> BASENAMECMD=/bin/basename
> DATECMD=/bin/date
> SYNCCMD=/bin/sync
> SLEEPCMD=/bin/sleep
>
> if [ "$THISHOST" = "$MIRROR1" ]
> then
> MIRROR=$MIRROR2
> fi
> if [ "$THISHOST" = "$MIRROR2" ]
> then
> MIRROR=$MIRROR1
> fi
> if [ "$MIRROR" = "" ]
> then
> echo "ERROR: $0 must be installed either on $MIRROR1 or $MIRROR2, not here ($THISHOST)"
> exit 1
> fi
>
> VSERVER=$1
> if [ "$VSERVER" == "" ]
> then
> echo "ERROR: VServer name missing"
> exit 2
> fi
> MIGRATEFROM=$2
> if [ "$MIGRATEFROM" == "" ]
> then
> MIGRATEFROM=$MIRROR
> MIGRATETO=$THISHOST
> else
> if [ $MIGRATEFROM == "$THISHOST" ]
> then
> MIGRATETO=$MIRROR
> else
> if [ $MIGRATEFROM == "$MIRROR" ]
> then
> MIGRATETO=$THISHOST
> else
> echo "ERROR: $MIGRATEFROM is not a valid host for this script (use $THISHOST or $MIRROR)"
> exit 3
> fi
> fi
> fi
> if [ $MIGRATEFROM == "$THISHOST" ]
> then
> RUNFROM=""
> RUNTO="$SSHCMD $MIGRATETO "
> else
> RUNFROM="$SSHCMD $MIGRATEFROM "
> RUNTO=""
> fi
>
> if $RUNFROM $VSERVERCMD $VSERVER running >/dev/null 2>&1
> then
> if $RUNTO $VSERVERCMD $VSERVER running >/dev/null 2>&1
> then
> echo "ERROR: vserver $VSERVER active on both $MIGRATEFROM and $MIGRATETO"
> exit 5
> else
> if [ $? -gt 1 ]
> then
> echo "ERROR: vserver $VSERVER unknown on $MIGRATETO as a standby server"
> exit 6
> else
> if $RUNFROM $LVCREATECMD -L${SNAPSHOTSIZE} -s --chunksize 4k -n ${VSERVERVG}/${VSERVERSNAPSHOT} ${VSERVERVG}/${VSERVERLV} >/dev/null 2>&1
> then
> TIMESTART=`$DATECMD +%s`
> if $RUNFROM $VSERVERCMD $VSERVER stop >/dev/null 2>&1
> then
> if $RUNFROM $RSYNCCMD -aqzpotSx4 --numeric-ids --recursive --safe-links --delete --force -e ssh ${VSERVERROOT}/${VSERVER}/ $MIGRATETO:${VSERVERROOT}/${VSERVER}/
> then
> if $RUNTO $VSERVERCMD $VSERVER start >/dev/null 2>1
> then
> TIMEEND=`$DATECMD +%s`
> echo "NOTICE: vserver $VSERVER migrated from $MIGRATEFROM to $MIGRATETO; elapsedsecs=`expr $TIMEEND - $TIMESTART`"
> else
> if $RUNFROM $VSERVERCMD $VSERVER start >/dev/null 2>&1
> then
> echo "ERROR: cannot start $VSERVER after migration from $MIGRATEFROM to $MIGRATETO; server was restarted again on $MIGRATEFROM"
> exit 8
> else
> echo "ERROR: cannot start $VSERVER after migration from $MIGRATEFROM to $MIGRATETO; server could not be restarted again on $MIGRATEFROM and is now dead on both mirrors"
> exit 9
> fi
> fi
> else
> if $RUNFROM $VSERVERCMD $VSERVER start >/dev/null 2>&1
> then
> echo "ERROR: cannot rsync $VSERVER for migration from $MIGRATEFROM to $MIGRATETO; server was restarted again on $MIGRATEFROM"
> exit 10
> else
> echo "ERROR: cannot rsync $VSERVER for migration from $MIGRATEFROM to $MIGRATETO; server could not be restarted again on $MIGRATEFROM and is now dead on both mirrors"
> exit 11
> fi
> fi
> else
> echo "ERROR: vserver $VSERVER cannot be stopped on $MIGRATEFROM"
> exit 7
> fi
> if $RUNFROM $LVREMOVECMD -f ${VSERVERVG}/${VSERVERSNAPSHOT} >/dev/null 2>&1
> then
> :
> else
> echo "ERROR: snapshot ${VSERVERVG}/${VSERVERSNAPSHOT} cannot be removed on $MIGRATEFROM after successful migration"
> exit 13
> fi
> else
> echo "ERROR: cannot migrate $VSERVER from $MIGRATEFROM to $MIGRATETO because snapshot ${VSERVERVG}/${VSERVERSNAPSHOT} cannot be established (possible overlap with backup/cronjob)"
> exit 12
> fi
> fi
> fi
> else
> echo "ERROR: vserver $VSERVER not running on $MIGRATEFROM"
> exit 4
> fi
> </file>
>
> _______________________________________________
> Vserver mailing list
> Vserver@list.linux-vserver.org
> http://list.linux-vserver.org/mailman/listinfo/vserver

-- 
Rúben Leote Mendes - ruben@nocturno.org
_______________________________________________
Vserver mailing list
Vserver@list.linux-vserver.org
http://list.linux-vserver.org/mailman/listinfo/vserver
Received on Sun Mar 19 17:46:30 2006
[Next/Previous Months] [Main vserver Project Homepage] [Howto Subscribe/Unsubscribe] [Paul Sladen's vserver stuff]
Generated on Sun 19 Mar 2006 - 17:46:34 GMT by hypermail 2.1.8