From: Tristan Donaldson (tristan.donaldson_at_skinkers.com)
Date: Mon 04 Oct 2004 - 14:09:29 BST
Hi David,
I designed and implemented a setup like this for our company. We run
two datastorage servers (primary and backup) which replicate to each
other using drbd. We then run 4 servers in front of those which are
diskless and boot via PXE and mount there root filesystems via NFSv3 to
the backend datastorage servers. On top of these front end servers we
run vservers which then allow us to seperate all of our services and
move them between front end servers to deal with hardware failures and load.
You have to be careful in what applications you run on the front end as
these need to be nfs nice. But most things work. But we did have
problems with mail queues in postfix, which initially caused lots of
corruption, but this was fixed by running the mail queue inside a ram
disk (initially), and then switched to using a loopback device.
I did have issues with NFS stale handlers when I tried to mount each
vserver as a NFS mount. But we just worked around this by just placing
everything inside the rootfs of the front server.
We did have a number of issues with performance of IO. Since we
actually have a firewall between the NFS servers and the front end
servers, we had performance problems with all of the udp traffic
creating states on the firewall. We have changed to using NFS over TCP.
We also use NFSv3 rather than NFSv2 as it is a lot faster when running
under the sync option (which you have to run).
Here are the options we use for our NFS mounts:
nfsvers=3,hard,nointr,tcp,timeo=600,retrans=2,rsize=32768,wsize=32768
Another thing to note is we don't run any of our major databases across
NFS. We run them inside vservers on the datastorage servers.
We run everything across gigabit ethernet.
For statistics, we are currently running 17 vservers on 3 servers (1
spare) all mounted across NFS. Our NFS server is running 5 vservers
contain different databases. Our bandwidth to our nfs server sits on
pretty much 2.5Mbit most of the time (peaks to 25Mbit at times). Most
of our vservers are not under heavy use, but we have peaked our out
going internet traffic from this (http) hosting to about 20Mbit without
any major performance issues.
If your hosting is high io intensive. Then you will probably have
issues, if its just static html files you should be ok as the file
should only be loaded over nfs once and cached locally. For our
environment all of the high io stuff is inside the database which
doesn't run over nfs.
Good luck.
Tristan.
David MacKinnon wrote:
> Hey. I know I vaguely recall someone talking about running vservers over
> nfs (ie the vserver directory is hosted on an nfs server, run on a
> front-end). I was just wondering if anyone who has set up something like
> this would care to comment on general performance, and any tuning they did?
>
> We're looking at seting up something like this for our hosting facility,
> to provide greater flexibility, but I didn't realise quite how slow nfs
> could be :( We are/were hoping to run over a dozen vservers over this
> setup.
>
> Any comments/suggestions?
>
> -David
>
> _______________________________________________
> Vserver mailing list
> Vserver_at_list.linux-vserver.org
> http://list.linux-vserver.org/mailman/listinfo/vserver
_______________________________________________
Vserver mailing list
Vserver_at_list.linux-vserver.org
http://list.linux-vserver.org/mailman/listinfo/vserver