Configure HAProxy with TPROXY kernel for full transparent proxy

Standard Kernel builds don’t support TPROXY ( 2.6.28 does now!).
For example if you use HaProxy as the load balancer then all of the backend servers see the traffic coming from the IP address of the load balancer. TPROXY allows you to make sure the backend servers see the true client IP address in the logs.

Ps. An easier alternative is inserting the clients ip in the x-forwarded-for header (option forwardfor).

For TPROXY to work you need three things:

1) TPROXY compiled into the linux kernel
2) TPROXY / Socket compiled into netfilter / iptables (due in v1.4.3?)
3) HaProxy compiled with the USE_LINUX_TPROXY option

The TPROXY patch for Linux Kernel 2.6.25.11 is here:
http://www.balabit.com/downloads/files/tproxy/tproxy-kernel-2.6.25-20080519-165031-1211208631.tar.bz2

The following is a guide how to install on Centos 5.1:

Heavily borrowed from: http://howtoforge.com/kernel_compilation_centos

Download The Kernel Sources

First we download our desired kernel to /usr/src. Go to www.kernel.org and select the kernel you want to install, e.g. linux-2.6.25.11.tar.bz2 (you can find all 2.6 kernels here: http://www.kernel.org/pub/linux/kernel/v2.6/). Then you can download it to /usr/src like this:

cd /usr/src
wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.25.11.tar.bz2

Then we unpack the kernel sources and create a symlink linux to the kernel sources directory:

tar xjf linux-2.6.25.11.tar.bz2
ln -s linux-2.6.25.11 linux
cd /usr/src/
wget http://www.balabit.com/downloads/files/tproxy/tproxy-kernel-2.6.25-20080519-165031-1211208631.tar.bz2
tar -xjf tproxy-kernel-2.6.25-20080519-165031-1211208631.tar.bz2
cd linux
cat ../tproxy-kernel-2.6.25-20080519-165031-1211208631/00*.patch | patch -p1 --dry-run
cat ../tproxy-kernel-2.6.25-20080519-165031-1211208631/00*.patch | patch -p1

Configure The Kernel

It’s a good idea to use the configuration of your current working kernel as a basis for your new kernel. Therefore we copy the existing configuration to /usr/src/linux:

make clean && make mrproper
cp /boot/config-`uname -r` ./.config

I needed to do a:

yum install ncurses-devel gcc gcc-c++ make rpm-build

Then we run

make menuconfig

which brings up the kernel configuration menu. Go to Load an Alternate Configuration File and choose .config (which contains the configuration of your current working kernel) as the configuration file:

Then browse through the kernel configuration menu and make your choices.

Make sure you enable tproxy support, `socket' and `TPROXY' modules (with optional conntrack support if you need SNAT)

Make sure you specify a kernel version identification string under General Setup —> () Local version – append to kernel release. I use CS3 so our kernel rpm package will be named kernel-2.6.25.11CS3.x86_64.rpm. You can leave the string empty or specify a different one which helps you identify the kernel (e.g. -custom or whatever you like).

Please note: After you have installed kernel-2.6.25.11CS3.x86_64.rpm and decide to compile another 2.6.25 kernel rpm package, it is important to use a different version string, e.g. -default1, -default2, etc., because otherwise you can’t install your new kernel because rpm complains that kernel-2.6.25.11CS3.x86_64.rpm is already installed!

Once you are happy with the kernel configuration, save & exit menuconfig then simply:

make rpm

This may take quite a long time….
Once it has finished:

Source RPM is here:

ls -l /usr/src/redhat/SRPMS/

Binary RPM is here:

ls -l /usr/src/redhat/RPMS/x86_64/

now install the new kernel:

cd /usr/src/redhat/RPMS/x86_64/
rpm -ivh --nodeps kernel-2.6.25CS-1.x86_64.rpm

Now you can either run the following command:

/sbin/new-kernel-pkg --package kernel --mkinitrd --depmod --install 2.6.25CS

Or you can do the usual manual steps i.e.

Make sure you create a new initrd file:

mkinitrd /boot/initrd-2.6.25.11CS3.img 2.6.25.11CS3

Now configure the boot loader:

vi /boot/grub/menu.lst
default=0
timeout=5
splashimage=(hd0,0)/boot/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.25.11CS3)
root (hd0,0)
kernel /boot/vmlinuz-2.6.25.11CS3 ro root=LABEL=/
initrd /boot/initrd-2.6.25.11CS3.img

That’s it, so reboot and do a:

uname -a

To check that we are using the new kernel:

Linux lbmaster 2.6.25.11CS3 #6 SMP Mon Jul 28 13:10:43 GMT 2008 x86_64 x86_64 x86_64 GNU/Linux

Compiling iptables with TPROXY support:

First download the current iptables source code:

cd /usr/src/
wget http://www.netfilter.org/projects/iptables/files/iptables-1.4.0.tar.bz2
tar -xjf iptables-1.4.0.tar.bz2

Then download the tproxy patch:

wget http://www.balabit.com/downloads/files/tproxy/tproxy-iptables-1.4.0-20080521-113954-1211362794.patch
cd /usr/src/iptables-1.4.0/
cat ../tproxy-iptables*.patch | patch -p1
make
make install

Compiling HAProxy with TPROXY support:

Download the latest version of the HAProxy source code:

wget http://haproxy.1wt.eu/download/1.3/src/haproxy-1.3.15.7.tar.gz
tar -xvf haproxy-1.3.15.7.tar.gz
cd haproxy-1.3.15.7/

Then compile making sure to enable TPROXY

make TARGET=linux26 CPU=x86_64 USE_STATIC_PCRE=1 USE_LINUX_TPROXY=1
make install target=linux26

If you have got this far then great, thats the hard part done!

Now before Haproxy can utilise TPROXY we need to set up some firewall marks:
You can put this script in a start up file such as rc.local etc.

#!/bin/bash
iptables -t mangle -N DIVERT
iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
iptables -t mangle -A DIVERT -j MARK --set-mark 111
iptables -t mangle -A DIVERT -j ACCEPT
ip rule add fwmark 111 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100

We also need to ensure that we have the correct architecture for the TPROXY trick to work. Using the normal HAProxy you can have real servers anywhere on the internet because the source address always points back at the HAProxy units IP address. However if the clients source IP address is going to be used then the HAProxy server MUST BE IN THE PATH of the return traffic.
The easiest way to do this is to put the backend servers in a different subnet to the front end clients and make sure that the default gateway points back at the HAProxy load balancer.

NB. With clever routing this should be possible on the same subnet but I haven’t tried that yet!

So here is an example configuration that I used for HAProxy:

# HAProxy configuration file
global
#	uid 99
#	gid 99
	daemon
	stats socket /var/run/haproxy.stat mode 600
	log 127.0.0.1 local4
	maxconn 40000
	ulimit-n 80013
	pidfile /var/run/haproxy.pid
defaults
	log global
	mode	http
	contimeout	4000
	clitimeout	42000
	srvtimeout	43000
	balance	roundrobin
listen	VIP_Name 192.168.2.87:80
	mode	http
	option	forwardfor
	source 0.0.0.0 usesrc clientip
	cookie	SERVERID insert nocache indirect
	server server1 10.0.0.60:80 weight 1 cookie server1 check
	server server2 10.0.0.61:80 weight 1 cookie server2 check
	server	backup 127.0.0.1:80 backup
	option redispatch

The most important line is this one:

	source 0.0.0.0 usesrc clientip

If your test setup doesn’t work then remove this line to check if a standard configuration does work.

Check your backend server logs to ensure that the client source IP address is correctly showing.

NB. One gotcha (of the many) is that you can no long use any local (i.e. 127.0.0.1) backup servers due to routing issues.
To resolve this change the backup server definition as follows:

server    backup 127.0.0.1:80 backup source 0.0.0.0


Ps. Many thanks to
John Lauro for his help with the firewall marks stuff

Oh and forgot to say change your sysctrls to allow redirects.. i.e.

echo 1 > /proc/sys/net/ipv4/conf/all/forwarding
echo 1 > /proc/sys/net/ipv4/conf/all/send_redirects
echo 1 > /proc/sys/net/ipv4/conf/eth0/send_redirects

71 thoughts on “Configure HAProxy with TPROXY kernel for full transparent proxy

  1. Hello, when I try to compile iptables, after “make”, I run “make install”: I have an error when it build dependencies:

    Unable to resolve dependency on asm/swab.h. Try ‘make clean’

    Have you got ideas?

  2. I’ve solved my problem by compiling iptables 1.4.3 that have tproxy patch inside (problem was introduced by balabit patch).

    But now I’ve another problem: line “source 0.0.0.0 usesrc clientip” doesn’t work: I obtain “503 Service Unavailable”.
    Please note that haproxy bind on a public IP and servers (two) are in a private network (haproxy’s server have two eth with public and private ip).

    Whitout “source 0.0.0.0 usesrc clientip” it work.

    Have you got ideas?

  3. Carlo,

    Yes, patches only work against the exact version….
    Are you sure you have set the default gateway on the backend servers to point at the internal interface on the haproxy instance. Traffic must pass through the proxy both ways.
    Does your setup work without that line?

  4. Firstly I’ve followeb your howto but iptables patch fails. So I have compiled iptables 1.4.3

    Haproxy configuration is the same that you have posted here.
    Can you provide me an example please?

    Without “source” line my setup work.

    Thanks.

  5. Solved. :)
    There was a problem in haproxy.cfg iptables:

    for haproxy use “source HAPROXY_IP usesrc clientip” (HAPROXY_IP can be public o private ip)

    for iptables:

    /usr/local/sbin/iptables -t mangle -N DIVERT
    /usr/local/sbin/iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
    /usr/local/sbin/iptables -t mangle -A DIVERT -j MARK –set-mark 1
    /usr/local/sbin/iptables -t mangle -A DIVERT -j ACCEPT

    ip rule add fwmark 1 lookup 100
    ip route add local 0.0.0.0/0 dev lo table 100

    Thanks!

  6. Hello Malcolm,

    Can you please provide details on what specific module to enable for the kernel? I am having a problem with my iptables, please see below;

    [root@SE-Caching ~]# service iptables status
    Table: mangle
    Chain PREROUTING (policy ACCEPT)
    num target prot opt source destination
    1 DIVERT tcp — 0.0.0.0/0 0.0.0.0/0 UNKNOWN match `socket’
    2 TPROXY tcp — 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 [16 bytes of unknown target data]

    What causes this one (UNKNOWN match `socket’ )?

  7. Hi Joseph,

    The modules load on demand so…

    Yes you’ve probably missed one of the required options while doing make menuconfig….
    Its really annoying that make menuconfig only shows the nice names of modules and not the precise name as in the .config file. As I forgot to take an exact copy of the text from each option at the time I can’t help much on the nice description side….

    Looking at my .config file I think the important ones are:
    CONFIG_NETFILTER_XT_TARGET_TPROXY=m
    CONFIG_NETFILTER_XT_MATCH_SOCKET=m
    CONFIG_NETFILTER_XT_MATCH_MARK=m
    CONFIG_NETFILTER_XT_TARGET_MARK=m
    CONFIG_NETFILTER_TPROXY=m
    CONFIG_NETFILTER_XTABLES=m
    CONFIG_NF_CONNTRACK=m
    CONFIG_NF_CT_ACCT=y
    CONFIG_NF_CONNTRACK_MARK=y

    IF in doubt select them ALL!

    Contact support@loadbalancer.org by email if you want the entire .config which works and may well help.
    I can even give you a full Centos5 kernel rpm if you want.

  8. Hi I have the same issue as Joseph:

    Table: mangle
    Chain PREROUTING (policy ACCEPT)
    num target prot opt source destination
    1 DIVERT tcp — 0.0.0.0/0 0.0.0.0/0 UNKNOWN match `socket’

    I recompiled the kernel a couple of times making sure I had at least all the modules you mention.

    Any pointers as to why I’m getting this error would be greatly appreciated.
    Thanks.

  9. Thanks for you response Jim, it looks like the relevant modules are there, below is the lsmod output.
    The weird thing is it’s working fine but that UNKNOWN match `socket’ still shows when I do /etc/init.d/iptables status.

    Module Size Used by
    xt_MARK 1892 1
    xt_socket 2212 1
    nf_conntrack 59436 1 xt_socket
    nf_defrag_ipv4 1764 1 xt_socket
    nf_tproxy_core 2404 1 xt_socket,[permanent]
    iptable_mangle 2372 1
    ip_tables 10452 1 iptable_mangle
    x_tables 13864 3 xt_MARK,xt_socket,ip_tables
    ipv6 214388 12
    autofs4 20648 2
    hidp 12196 2
    rfcomm 28692 0
    l2cap 18020 10 hidp,rfcomm
    bluetooth 48900 5 hidp,rfcomm,l2cap
    sunrpc 160704 1
    dm_multipath 12880 0
    sbs 10956 0
    sbshc 5124 1 sbs
    battery 10088 0
    lp 8804 0
    sg 25208 0
    floppy 46980 0
    ide_cd_mod 25996 0
    cdrom 30176 1 ide_cd_mod
    serio_raw 4712 0
    parport_pc 23044 1
    ac 4072 0
    button 6004 0
    parport 30796 2 lp,parport_pc
    pcnet32 29188 0
    mii 4740 1 pcnet32
    i2c_piix4 8564 0
    i2c_core 20824 1 i2c_piix4
    pcspkr 2276 0
    dm_snapshot 16040 0
    dm_zero 1348 0
    dm_mirror 11912 0
    dm_region_hash 9828 1 dm_mirror
    dm_log 8360 2 dm_mirror,dm_region_hash
    dm_mod 47720 11 dm_multipath,dm_snapshot,dm_zero,dm_mirror,dm_l og
    ata_piix 21384 0
    libata 147852 1 ata_piix
    mptspi 14988 2
    mptscsih 28996 1 mptspi
    mptbase 71364 2 mptspi,mptscsih
    scsi_transport_spi 18980 1 mptspi
    sd_mod 23456 3
    scsi_mod 134356 6 sg,libata,mptspi,mptscsih,scsi_transport_spi,sd_mod
    ext3 107308 2
    jbd 40280 1 ext3
    uhci_hcd 18516 0
    ohci_hcd 19444 0
    ehci_hcd 29264 0

  10. Pingback: Loadbalancer.org Blog » Blog Archive » Transparent proxy of SSL traffic using Pound to HAProxy backend patch and howto

  11. Hi friends! I used haproxy in my project. But I have one problem. What I can switch between two backends servers (me need used rule url_sub) if I used haproxy as frontend

    global
    uid 1004
    gid 1002
    daemon
    #stats socket /var/run/haproxy.stat mode 600
    log 127.0.0.1 local4
    maxconn 40000
    ulimit-n 80013
    pidfile /var/run/haproxy.pid
    defaults
    log global
    mode http
    contimeout 4000
    clitimeout 42000
    srvtimeout 43000
    balance roundrobin
    listen XXX.COM x.x.x.x:80
    mode http
    option forwardfor
    stats enable
    stats auth bvv2001:la-la-la
    cookie XXX.COM insert
    source 0.0.0.0 usesrc clientip
    server server1 x.x.x.x:10000 cookie XXX.COM check
    server server2 x.x.x.x:10001 cookie XXX.COM check
    dst_port 10000 if url_sub !sms
    dst_port 10001 if url_sub sms

    this config NOT WORKING
    Help me please!

    • You need something more like:

      frontend blah
      acl notsms url_sub !sms use_backend backend1
      acl sms url_sub sms use_backend backend2

      backend backend1
      source 0.0.0.0 usesrc clientip
      server server1 x.x.x.x:10000 cookie XXX.COM check

      NB. I haven’t tested this just off the top of my head…..
      Check out the HAProxy manual and mailing list for more specific help on ACLs…

  12. Pingback: Twitter Trackbacks for Loadbalancer.org Blog » Blog Archive » Configure HAProxy with TPROXY kernel for full transparent proxy [loadbalancer.org] on Topsy.com

  13. Hi Malcolm and thanks for your brilliant article!

    I have wasted a couple of days trying to debug the kernel which was throwing SOFT LOCKUP bugs. After investigation it was down to having 2 vCPU in the VM (from VMWare) I was using. Moving back to only one vCPU did the trick. Hope this can help someone else!

    In your post, you have commented the following line:
    # uid 99
    # gid 99

    Therefore, haproxy is running as root.

    I know it produces the following error if haproxy is run with another user then root:
    “Starting haproxy: [ALERT] 259/184020 (3497) : [/usr/sbin/haproxy.main()] Some configuration options require full privileges, so global.uid cannot be changed.”

    So it seems compulsory to run haproxy as root.
    I am now wondering if this can cause any problem related to using root to run haproxy.

    Thanks in advance for your answer!

    Gael

    • Willy suggests starting HAProxy as root because it can then jail itself in a chroot and drop all of its privileges before starting the instances. This is not possible if it is not started as root because only root can execute chroot()

  14. Hello,I have got a probelm like this:
    haproxy bind on a public IP,the apache server is on the same machine with haproxy’server (haproxy’s server have two eth with public and private ip)
    Whitout “source 0.0.0.0 usesrc clientip” it work.
    could they at the same machine?

    (please note: the other apache server is not on the same machine wich haproxy’server,and I have set the default gateway on the backend server to point at the internal interface on the haproxy instance,it can work)

    Have you got ideas?
    thanks

  15. Dear Malcolm,

    I know its very stupid to ask such newbie questions but that is all what i have right now.

    1) How is Haproxy different from squid.
    2) Transparent squid does not works with gtalk, is that a problem with Haproxy too, if it runs with tproxy.
    3) Does Haproxy support acl (based on time,based on url regex).
    4) Can we also use acl on HTTPS or in other words how does Haproxy handles HTTPS.

    Your reply would indeed be a lot of help to me.
    Expecting your positive reply.
    Thanks,
    Anand Phulwani

  16. Anand,

    1) I guess squid is an outbound proxy (standard) and HAProxy is an inbound proxy (reverse), like Pound.
    2) Why would you be running Google talk servers? (this is a reverse proxy not outbound)
    3) Yes, but I’m not sure exactly what you are after…
    4) Not directly, but you just terminate the HTTPS with stunnel or Pound first…so yes.

  17. Dear Malcolm,

    After Reading The Documentation I Had An Hint That This Is An Inbound Proxy And Works For Basically For Web Server Managed In A Kind Of Cluster.
    So Actually This Is Just Different For What I Am Trying To Do With Squid.

    Thanks,
    Anand

  18. Here is my question. I have a simple setup with HAProxy and 2 backend servers. Works like a charm except that on these backend servers I look at the HTTP_HOST with RewriteCond to do various things. When I turn logging on it looks like the “Host” that I am getting is either the IP address or nothing. I have looked at the documentation, but any idea how I can get the backend server to “see” what the URL is?

  19. Hello Malcolm,
    I try to implement haproxy in transparent mode like in your how-to.
    I add rules in iptables, but when I try to access to the service, the result is “503 Service Unavailable”.
    Without the line “source 0.0.0.0 usesrc clientip” all works correctly, obviously without transparency.
    Have you got any ideas? Where I’m wrong?
    Thanks! Daniel

  20. Pingback: Compile a (CentOS) Kernel And IPTables With TPROXY Support ~ Mattias Geniar

  21. now i account a problem about how to deal with the https request!
    In my config file:
    global
    maxconn 10000 # Total Max Connections.
    log 127.0.0.1 local0
    log 127.0.0.1 local1 notice
    daemon
    nbproc 1 # Number of processes
    uid 0
    gid 0

    defaults
    log global
    mode tcp
    clitimeout 60000
    srvtimeout 30000
    contimeout 4000
    retries 3
    option httpclose

    backend USpaceDefine
    timeout connect 10s
    timeout server 30s
    balance roundrobin
    server uspace-125 192.168.0.11 weight 1 maxconn 512
    server uspace-128 192.168.0.12 weight 1 maxconn 512

    backend NginxDefine
    timeout connect 5s
    timeout server 5m
    balance roundrobin

    server web-client 192.168.0.13 weight 1 maxconn 10000
    frontend http_proxy
    bind :80,:443
    mode tcp
    timeout client 5m
    option forwardfor
    default_backend NginxDefine
    acl req_pubsub_path path_beg /uspace #this is a context
    use_backend USpaceDefine if req_pubsub_path

    that is all!but when i input the website:http:// 192.168.0.11 ,it will redirect the correct website,but when i input https:// 192.168.0.11,it does not work! so i would like your help,thanks!!

  22. Allen,
    You have bind :80,:443 ?
    You cant do content rules or option forward for with encrypted traffic (HTTPS 443).
    You either need to decrypt it first with stunnel or pound then pass o HAProxy:80 OR
    split your rules so that 443 has its own simple mode=tcp backend.

  23. Thanks for your help last time!I haved sloved the problem with the below,but i meet other problem:
    for example:
    the server haproxy ip :192.168.0.12
    when i visit the site:https://192.168.0.12 it can correctly enter the NginxDefine(it is a backend,and it contains a nginx server),but try to visit https://192.168.0.12/space,it cant redispatch TomcatDefine (it is also a backend,and it contains a tomcat server)

    frontend https_proxy
    bind :443
    mode tcp
    acl is_ssl req_ssl_ver 2:3.1
    tcp-request content accept if is_ssl
    timeout client 5m
    option forwardfor
    default_backend NginxDefine
    acl req_pubsub_path path /space
    use_backend TomcatDefine if req_pubsub_path

  24. Hi, this HowTo is great. I am working through it at the moment.

    I am assuming that this will work for SMTP traffic load balancing.

    I am trying to use this method to avoid SPF and RBL issues when load balancing email content filtering servers. The standard HAProxy config would cause all sorts of problems.

  25. Mitchy,

    Yes it should be fine for SMTP servers, I would use a TCP front end with source IP table persistence (Recent feature of HAProxy if you need it). Their is also a dedicated SMTP health check as well….

  26. Hi Malcolm,

    Thanks for the info. it is very useful. I have a question about using tproxy to obtain ip transparency in haproxy setups.

    If I setup my webservers gateways to the loadbalancer, then any traffic from the webservers (lets say part of my application fetches a file externally) will need to be routed from the load balancer, right? so I need to setup some sort of forwarding on the LB?

    Further, if I use heartbeat for failover and redundant load balancers, what IP should I use for the server gateways? (The virtual ip cant be it, right? it would need to be one of the LBs real ips, which would change depending on active load balancer…)

    Is there anyway to go arouund this? Is there anyway of setting conditional gateways, or some other sort of workaround?

    Thanks!

  27. Roy,

    With a pair of HAProxy units under heartbeat you would want an external Floating IP for the VIP and and internal Floating IP for the default gateway. For the internal servers if they need separate Internet service access through the gateway then you will need an iptables rule to masquerade internal traffic on the external IP something like:

    $INT_SUBNET=”192.168.1.0/255.255.255.0”
    iptables -t nat -A POSTROUTING -s INT_SUBNET -j MASQUERADE

  28. Use rpaf module in apache/apache2 for a much easier way to have apache provide the x-forwarded-for ip to php and other apache modules. Just install, enable and that is it. No need to mess with firewall rules or anything else.

  29. if I want to use the HAproxy as a reverse proxy not a load balncer, could I make one transparent front end that is invisble and do all needed work then forward the traffic to one back-end? so if some one requested the ip of the back-end, it should first pass by the front end then forward the traffic, without the client knows that he passed by something at the middle

    • In theory you could make traffic route transparently, but I think you mean forward proxy like squid not reverse proxy like HAProxy. HAProxy would need to repond to ANY IP address (not just its own) with LVS you can use firewall rules an IP rule tables to achieve this (useful for load balancing proxies in transparent mode). Try the HAProxy mailing list somone may know their.

  30. Hi,

    I am working on configuring transparent proxy for https traffic. On the client side (IE or mozilla) without the proxy settings, URL given was https:// and the traffic was forced to go through the proxy m/c with the help of IP routes set.

    The problem I am facing is that in this manner (described above), the client sends the SSL message as “Client Hello” than a HTTP message “CONNECT”. If I specify the proxy settings in the IE (or mozilla) it works fine as it sends the HTTP message “CONNECT” and the transparency is successful.

    Thanks in advance.

  31. Pingback: Remote IP’s with HAProxy Drija

  32. Pingback: haproxy forwardfor ignored while in tcp mode - Admins Goodies

  33. Hi,

    I successfully setup haproxy with tproxy using this guide. However I have some question the backend server (real server) have no internet connection.

    Also can I ask if this setup are possible on public server assuming my server are on the same datacenter/network.

  34. Den,
    Um… Yes and yes. As long as HAProxy has one IP that can either be a public IP or be NAT’ed through a firewall / router to a public IP then it will work fine. Their is no difference between a public IP and a non-public IP as far as any networking equipment is concerned.

  35. Hi Malcolm,

    Thanks for the reply. My problem are on real server machine have no internet connection. But can ping gateway (haproxy server that have virtual IP as gateway of real server).

  36. Ah I see, you interneral server want Internet access? THen you need a simple firewall rule on the HAProxy machine so that all traffic from inside pretends to be directly from the HAProxy units public IP address:
    iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE
    (assuming eth1 is your external public interface)

  37. Hi Malcolm

    thanks, backend server have internet connection now. I have another problem. When I use source 0.0.0.0 usesrc clientip I cannot access the backend server from client. But the haproxy server itself can access the backend server. I test it using curl.

    It’s like problem on fetch all request from client then forward to backend server. And I think it is weird because sometimes its working. All I have to do is to wait around 10 minutes then viola its work like a charm.

    By the way I have only one nic interface for this setup.

  38. Hi,

    Update on my setup.. Whenever to flush iptables rules iptables -F then restart iptables services then manually added all firewalls rules. My setup are working now. But when I reboot my server (haproxy server) I cannot connect again to my web server from my pc client.

  39. With this setup is it possible to use both the tproxy and the regular method?

    We have many webservers behind haproxy, I dont want to change their gateways. But we wanted to add haproxy balancing for smtp servers, and need to have the correct source ip for spf, etc… Can I choose in haproxy conf when to use regular haproxy proxying and when to use tproxy or does it take over?

    Any ideas?

    thanks!

  40. To all the posters above – thank you. We now have a fully transparent Tproxy+HAproxy and our back end webservers have the correct (original) client IPs being logged. In addition, we can access the internet from the webservers using the MASQUARADE function detailed above.

    Since we’re running Debian X64 v6.0.3 we did not have to recompile the kernel or recompile IPTABLES, but we did have to recompile HA Proxy with T Proxy support.

    In case it helps someone else, here is out /etc/rc.local -
    iptables -t mangle -N DIVERT
    iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
    iptables -t mangle -A DIVERT -j MARK –set-mark 111
    iptables -t mangle -A DIVERT -j ACCEPT
    ip rule add fwmark 111 lookup 100
    ip route add local 0.0.0.0/0 dev lo table 100
    ip rule add dev eth0 fwmark 111 lookup 100
    ip rule add dev eth1 fwmark 111 lookup 100
    ip rule add dev eth2 fwmark 111 lookup 100
    ip rule add dev eth3 fwmark 111 lookup 100
    iptables -t nat -A POSTROUTING -s 10.20.192.0/255.255.240.0 -j MASQUERADE

  41. Hi all, thanks for this great post.

    I’ve got couple of questions:

    1. It’s 100% mandatory to change default gateway of real servers to point to 1 of the haproxy IP addresses?

    2. If yes, which one? The VIP one?

    Regards

  42. Generally the HAProxy box will be configured as a NAT gateway between your external network and internal network, you can use any physical IP (of the HAProxy unit) on the internal side for your default gateway. If you have a High-availability pair of HA-Proxy units this would be the internal floating IP, if you just have a single unit then you would probably just have one IP address. The VIP or bind address would usualy be on the external network (so you definately would not use that one).

  43. hey,

    great article.

    followed through this article by the letter and still my web server see the HAPROXY server ip.

    Anybody any idea?

  44. Hey I’m using centos 6.3 and did pretty much what you wrote and still my web server sees the haproxy up. What am I doing wrong?

  45. Pingback: Load Balancing – What are the supported load balancing methods? « Govardhan Gunnala

  46. Hello All,
    I’ve installated HAproxy on Debian with kernel Linux 2.6.32+.
    When HAproxy starts, I’ve the following error:
    Starting haproxy: haproxy

    [ALERT] 024/152707 (1802) : parsing [/etc/haproxy/hapr xy.cfg:54] : ‘usesrc’ not allowed here because support for TPROXY was not compiled in.

    [ALERT] 024/152707 (1802) : parsing [/etc/haproxy/haproxy.cfg:92] : error detected while parsing a ‘monitor fail’ condition.

    [ALERT] 024/152707 (1802) : parsing [/etc/haproxy/haproxy.cfg:104] : ‘usesrc’ not allowed here because support for TPROXY was not compiled in.

    [ALERT] 024/152707 (1802) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg

    [ALERT] 024/152707 (1802) : Fatal errors found in configuration.

    May you help me? Have any suggestions?
    I’m following this article:
    http://blog.exceliance.fr/2012/08/25/haproxy-varnish-and-the-single-hostname-website/

    Best regards.

    • Hello Matteo, Can I ask if you have compiled ha proxy from source or just installed from one of the Debian repository? as the error indicates that the build you are running was not compiled with TPROXY support.

  47. I have already installed squid+tproxy with a single instance an am using it.now i want to install multi instances of squid. my config has a iptable rule as follow that tells iptable where to forward requests:
    iptables -t mangle -A PREROUTING -p tcp –dport 80 -j TPROXY –tproxy-mark 0×1/0×1 –on-port 3129
    this line is missing in your config.is this mandatory?
    I couldn’t get it running…this is my haproxy config:
    please correct me where is my fault

    # HAProxy configuration file
    global
    # uid 99
    # gid 99
    daemon
    stats socket /var/run/haproxy.stat mode 600
    log 127.0.0.1 local4
    maxconn 40000
    ulimit-n 80013
    pidfile /var/run/haproxy.pid
    defaults
    log global
    mode http
    contimeout 4000
    clitimeout 42000
    srvtimeout 43000
    balance roundrobin
    listen VIP_Name 0.0.0.0:3129
    mode http
    option forwardfor
    # source 0.0.0.0 usesrc clientip
    cookie SERVERID insert nocache indirect
    server server1 127.0.0.1:4001 weight 1 cookie server1 check
    server server2 127.0.0.1:4002 weight 1 cookie server2 check
    server backup 127.0.0.1:80 backup
    option redispatch

  48. Why are you load balancing local instances of Squid?
    Or is this a firewall/router that is TPROXYing web requests from port 80 -> 4001 etc ?
    iptables based TPROXY with SQUID usually just means a transparent local TCP port REDIRECT (as you have shown with your iptables command)
    This has NO RELATIONSHIP WHATSOEVER to HaProxy TPROXY which is used to ensure that servers in the cluster behind HaProxy see the same source IP address of the client traffic even though they are on a different subnet.
    If your SQUID boxes are not on the same unit then you can use haproxy in TPROXY mode to transparently load balance the web filters.
    You can also use LVS in DR mode for transparent load balancing of SQUID.

  49. Hi Malcolm, excellent tutorial, trying it in my lab right now, do you have guide if the server gateway is not to the HAProxy?

  50. Pingback: HAProxy - Instalación, configuración, actualización. - Roberto Martinez Martin - Web Personal | Roberto Martinez Martin – Web Personal

  51. Pingback: HAProxy - Instalación, configuración, actualización.... - ROBIT Sistemas Informaticos y Comunicaciones, S.L | ROBIT Sistemas Informaticos y Comunicaciones, S.L

Leave a Reply

Login with your Social ID

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Powered by sweet Captcha