vserver development mailing list: [Vserver] Re: Postrouting not available in directors ?

[Vserver] Re: Postrouting not available in directors ?:
vserver development mailing list
[Next/Previous Months] [Main vserver Project Homepage] [Howto Subscribe/Unsubscribe] [Paul Sladen's vserver stuff]

I cross post to linux-vserver mailing list where I submitted my problem
last week.
and I posted in text + HTML because of the ascii art...

>>I there a way to use the postrouting nat rule on a director ??
>>
>>
>
>No, I do not believe that this is possible without modifying lvs,
>which incidently should be easy enough.
>
>
>
Just a matter of time :-)

>>what I want to do is to dynamically change the source of the packet
>>before it reaches the realserver.
>>I use a masquerading topology.
>>
>>Long explanation (don't read this if you're not interested)
>>
>>I have setup 2 directors ;
>>I have thoses services on the director : Webmail, Ldap, imap/pop ... All
>>thoses services are served by 4 realservers:
>>4 for webmail, 4 for ldap, 4 for imap.
>>
>>All my realservers have 2 network interface : 1 for the service of
>>requests coming from ultramonkey, with an ip rule where default gw is
>>LVS, 1 for administrative (ssh) and requests to other services provided
>>on my intranet (default rule).
>>
>>The problem is coming from the fact that my realservers, are, in
>>reality, vservers (http://linux-vserver.org/), all hosted on 4 physical
>>machines.
>>
>>This configuration works very well, except when I re-enter in the
>>ultramonkey.
>>
>>Supose this case:
>>
>>I have connection to Webmail, which in turn needs to ask an ldap server.
>>
>>1) connection to webmail via ultramonkey, realserver Webmail-A is
>>choosen. connexion go to eth0. (Webmail-A is a vserver on host A). reply
>>go through the LVS. OK.
>>2) webmail-A need to ask the ldap. Request go from eth1, and reach the
>>VIP of the director. OK
>>case 3a) director choose Ldap-B or C or D. Ldap request go from eth0 on
>>vserver, and is answerd there, via LVS. All is fine.
>>case 3b) no luck, director choose Ldap-A. Ldap-A is hosted on the same
>>physical server. Ldap-A see packets from Webmail-A to Ldap-A. Ldap-A has
>>a local route (rule 0, local table). He will reply DIRECTLY to
>>Webmail-A, WITHOUT passing by director...
>>
>>This is my problem.
>>
>>
...

>>
>>
>This problem has cropped up a few times recently, and I don't think that
>anyone has a good solution. That is, if you have a VIP, then the
>real server's servicing that VIP will always connect to themselves
>when accessing the VIP.
>
>
In my case directors are separate machines.
I have a total of 6 machines :
2 for directors,
4 for real servers.
This is not exactly the same case as previous post posted recently, or,
at least it's not what I understood. I wasn't clear about this sorry.

All the vservers have two IP : 192.168.11.80 & 172.20.9.80 for webmail-A
for exemple

The general case is working well.
The only issue is when a vserver Hosted on A (Webmail-A) make a request
to 172.20.11.240:389 (VIP of the director)
and the director send the request on Ldap-A (wich is also located on host A)
The request seems local and didn't get by director, so the packet is
dropped.

>One way around this, is to have separate VIP's and separate real-servers
>for each service. So Webmail-A and Webmail-B would be different machines
>to Ldap-A and Ldap-B. With LVS-NAT this works as log as Webmail-X and
>Ldab-X are on separate subnets, although they can use the
>
>
I fail to see how this can solve the problem...

>same linux director. For LVS-DR Webmail-X and Ldab-X can be on the
>same subnet and share the same linux director.
>
>
DR is not an option I consider for the moment... Some directors have
public adresses and provide public services,
Direct replies won't be routed

>If you don't want to add extra machines, you could contrive some setup
>that works with virtual machines, like qemu, xen, user-mode linux,
>vmware, etc...
>
>
Well adding more machines isn't a problem,
BTW, my directors ARE already hosted on xen hosts... :-)

I have a total of 2x4 directors (1 for each network served), all packed
on 2 physical
machines with wen :)

And my realservers (the vservers) are already capable of reaching the
good director.
The only case to treat is when a request come from the realservers...

For the moment I've tried lots of things...
The answer for the moment is :
1) Doing a SNAT on PREROUTING on host A when the packet comes from src
172.20.9.xx -> change it to, say, src 192.168.11.xx
1b) Doing a DNAT on POSTROUTING on host A to dst 192.168.11.xx ->
172.20.9.xx
This way local tables are bypassed.
Unfortunately, PREROUTING only do DNAT and POSTROUTING only do SNAT :(
Maybe it's time to post on netfilter too...

2) Doing SNAT on POSTROUTING on the director,to handle this case. You
just replied to this...

3) Putting a XEN guest somewhere
    a) 1 on HOST A, 1 on HOSTB, just to do handle the case and doing
NAT; But this forces to ALSO use XEN for the linux host that
        handle the vservers, this is not something I want to do for
performance...
    b) Putting one more Xen dom_U on the physical machines that hosts
the directors (they already have 4 guest running). Yes this is probably the
        best choice.

4) Maybe investigate more time in NGNET features of vservers, where
networking is now virtualized . This could probably solve my problem
but this feature is still experimental if I remember well.

So I'll probably go for 3) Now. But this seems a litle bit overkill. I
thought there was a better & simpler way to do this.