Load balancing Windows Terminal Server – HAProxy and RDP Cookies or Microsoft Connection Broker
by Nick Chalk
When you have users depending on Windows Terminal Services for their main desktop, it’s a good idea to have more than one Terminal Server. RDP, however, is not an easy protocol to load balance; sessions are long-lived and need to be persistent to a particular server, and users may connect from different source addresses during one session.
The current development version of HAProxy has made an important step forward in making this possible. Thanks to work by Exceliance, it now supports RDP Cookies, offering a solution to the persistence problem.
We have been testing the latest development release of HAProxy, 1.4-dev4, on a loadbalancer.org Enterprise R16 device. The real servers were two Windows Server 2008 machines, with identical test users set up on both.
We settled upon the following HAProxy configuration (RDP Cookies):
defaults
clitimeout 1h
srvtimeout 1h
listen VIP1 192.168.0.10:3389
mode tcp
tcp-request inspect-delay 5s
tcp-request content accept if RDP_COOKIE
persist rdp-cookie
balance rdp-cookie
option tcpka
option tcplog
server Win2k8-1 192.168.0.11:3389 weight 1 check inter 2000 rise 2 fall 3
server Win2k8-2 192.168.0.12:3389 weight 1 check inter 2000 rise 2 fall 3
option redispatch
Note that this is only a fragment of the haproxy.cfg file, showing the relevant options.
The load balancer’s Virtual IP is set to 192.168.0.10, listening on port 3389 for RDP. The two real servers are on 192.168.0.11 and 192.168.0.12, in the same subnet as the Virtual IP.
The two new configuration directives are persist rdp-cookie and balance rdp-cookie. These instruct HAProxy to inspect the incoming RDP connection for a cookie; if one is found, it is used to persistently direct the connection to the correct real server. The two tcp-request lines help to ensure that HAProxy sees the cookie on the initial request.
The only other tweak needed is to increase the clitimeout and srvtimeout values to one hour. In testing, this was found to be necessary to keep idle RDP sessions established.
Testing involved making multiple connections with different usernames, from varying IP addresses, using both Windows XP Professional and Linux clients. Sessions were disconnected and reconnected, and real servers removed from the cluster and re-inserted.
We found that, once a user had established a session with a particular real server, that user consistently reconnected to the correct server if it was available. When we removed and re-inserted servers, existing sessions were unaffected. After a simulated server failure, users could start a session on the remaining server.
When a failed server was brought back on-line, users that had been connected to that server would reconnect to it again – even if they had started a new session on the other server in the meantime. This may not be what you want, and requires further testing.
With client and server time-outs set to one hour, we were able to leave idle sessions running for 16 hours without problems.
For more information on the new configuration options, see the development version of HAProxy’s Configuration Manual.
NB. For some daft reason Microsoft restricted the login cookie in RDP to 9 characters! Now as the domain is usually listed first (mydomain/myusername) the first 9 characters may always be the same and RDP cookie session persistence wont work. Two work arounds for this are either reduce the length of your domain name (ouch!) OR use the myusername@mydomain format when you log in….
So what about Microsoft Connection Broker (session directory or whatever they call it) ?
A simple one line change in your HAProxy configuration (RDP Connection Broker):
#Balance rdp-cookie -> balance leastconn i.e.
defaults
clitimeout 1h
srvtimeout 1h
listen VIP1 192.168.0.10:3389
mode tcp
tcp-request inspect-delay 5s
tcp-request content accept if RDP_COOKIE
persist rdp-cookie
balance leastconn
option tcpka
option tcplog
server Win2k8-1 192.168.0.11:3389 weight 1 check inter 2000 rise 2 fall 3
server Win2k8-2 192.168.0.12:3389 weight 1 check inter 2000 rise 2 fall 3
option redispatch
Note that this is only a fragment of the haproxy.cfg file, showing the relevant options.
Its about time we updated this post for the juicy new features in HAProxy – Development 1.5-dev7
Their were a couple of the problems with the hash method used with RDP cookie load balancing (as described above):
- Lots of people would like to use least connection load balancing with WTS/RDP clusters (this is not possible with a HASH based persistence method).
- When you add or remove servers the HASh table gets re-configured i.e. users hit the wrong server.
So Loadbalancer.org took the decission to sponsor the development of a stick-table based RDP persistence (we sponsored the origional source IP stick table work as well). When we looked at it in more detail we decided that what we needed was:
- Flexible stick tables that could be used for multiple future requirements i.e. SSL Session ID persistence.
- RDP stick table support in order to enable least connection based scheduling.
- Some way of restoring stick tables on session restart (and also replication to other HAProxy instances).
- Ensuring that TCP connections are properly closed on server failure (especially important on long connections).
- Ensuring that the stick table is cleared out on server failure.
- And finaly making sure that the fallback server can be made non-sticky! (really irritating if you get stuck on the sorry site down page).
To cut a long story short lets just dive in with a full configuration file and explain it as we go:
#HAProxy configuration file generated by LB Cloud appliance global #uid 99 #gid 99 daemon stats socket /var/run/haproxy.stat mode 600 level admin log 127.0.0.1 local4 maxconn 40000 ulimit-n 81001 pidfile /var/run/haproxy.pid defaults log global mode http timeout connect 4000 timeout client 42000 timeout server 43000 balance roundrobin peers localpeer peer loadbalancer localhost:8888 listen stats :7777 stats enable stats uri / stats hide-version option httpclose frontend F1 bind *:3389 maxconn 40000 default_backend B1 mode tcp option tcplog backend B1 mode tcp option tcpka balance leastconn tcp-request inspect-delay 5s tcp-request content accept if RDP_COOKIE persist rdp-cookie stick-table type string size 204800 expire 120m stick on rdp_cookie(mstshash) server R1 www.loadbalancer.org:3389 weight 1 check port 3389 inter 2000 rise 2 fall 3 on-marked-down shutdown-sessions server R2 www.clusterscale.com:3389 weight 1 check port 3389 inter 2000 rise 2 fall 3 on-marked-down shutdown-sessions server backup us.loadbalancer.org backup non-stick option redispatch option abortonclose
An important new section is the peers section:
peers localpeer peer loadbalancer localhost:8888
In this configuration we are syncronising all of the stick table information with localhost:8888 (it could be with another HAProxy instance for session table high-availability).
When HAProxy restarts it will run existing sessions on the old process until they expire, only new sessions will run on the new HAProxy instance (this can get quite confusing as the stats socket or page will only show the new sessions (not the old ones)
You will need to change your HAProxy start up scripts:
start() {
/usr/local/sbin/$BASENAME -L loadbalancer -c -q -f /etc/$BASENAME/$BASENAME.cfg
if [ $? -ne 0 ]; then
echo "Errors found in configuration file."
return 1
fi
echo -n "Starting $BASENAME: "
daemon /usr/local/sbin/$BASENAME -D -f /etc/$BASENAME/$BASENAME.cfg -p /var/run/$BASENAME.pid -L loadbalancer
RETVAL=$?
echo
[ $RETVAL -eq 0 ] && touch /var/lock/subsys/$BASENAME
return $RETVAL
}
reload() {
/usr/local/sbin/$BASENAME -L loadbalancer -c -q -f /etc/$BASENAME/$BASENAME.cfg
if [ $? -ne 0 ]; then
echo "Errors found in configuration file."
return 1
fi
/usr/local/sbin/$BASENAME -D -L loadbalancer -f /etc/$BASENAME/$BASENAME.cfg -p /var/run/$BASENAME.pid -sf $(cat /var/run/$BASENAME.pid)
}
The important thing is that the peers definition “loadbalancer” must be prsent in both the start up scripts and the haproxy.cfg file.
Now we have the new section to make the stick table use RDP cookies and the least connection scheduler:
balance leastconn tcp-request inspect-delay 5s tcp-request content accept if RDP_COOKIE persist rdp-cookie stick-table type string size 204800 expire 120m stick on rdp_cookie(mstshash)
And the new clean and quick session kill options + making the backup server not go in the stick table:
server R2 www.clusterscale.com:3389 weight 1 check port 3389 inter 2000 rise 2 fall 3 on-marked-down shutdown-sessions server backup us.loadbalancer.org backup non-stick
I probably havent explained all that very well…
but feel free to ask questions 
.
November 9th, 2009 at 1:11 pm
It is great news that HAProxy now supports RDP Cookies! Now all it really needs for full functionality is some kind of CPU idle feedback method from the real servers. Enabling HAProxy to distribute new clients to the least loaded servers, this has been talked about on the HAProxy mailing list quite a lot so no doubt it will be on the way soon.
We have a Windows feedback service installer that responds to telnet on port 6666 with the CPU idle figure (0-100), I’d be happy to get the programmer to open source it under the GPL if anyone wants to use it?
January 17th, 2010 at 5:42 am
Hi Malcom
It is Terry from New Zealand, (It’s been a few years since we last talked).
I think it might be of interest to evaluate using this type of perpetual trial model (1Mb) and then pay as you grow for your excellent VSA product.
It would gain traction with certain cloud providers.
http://jariangibson.com/2009/09/18/netscaler-vpx-express-is-a-must-for-all-xenserver-environments/
http://jariangibson.com/2009/12/01/choosing-a-netscaler-hardware-appliance-or-virtual-appliance/
Regards
Terry Vercoe
January 17th, 2010 at 8:42 pm
Terry,
Yes long time no talk.. I’ve just got back from a cruise round New Zealand with the family (It was fantastic).
At the moment XEN proprietary or open source has very little traction with paying customers. But yes long term we are planning a whole host of virtualized load balancers like our current VMWare appliance (similar idea to Netscalers) i.e. an exact copy of our hardware OS/Software Stack runing on XEN/Microsoft etc. We have an Amazon EC2 platform close to release which relies heavily on HAProxy (as a lot of the layer 4 technology won’t work in the Amazon cloud). We also have several projects with ISPs integrating our load balancer to their various control panels.
Long term I think it would be great for the market if Google and Amazon came together to offer an open source cloud standard that would provide a platform with enough users to make commercial development of products a worthwhile effort; production cost wise (we’ve only had 3 serious XEN enquires but loads of VMWare sales…).
Our EC2 cloud development has taken over almost a year and that starts getting expensive! Maybe we should look at outsourcing all of our development to India?
February 5th, 2010 at 2:47 pm
Hi Guys
I have tried using haproxy for rdp with Win Xp. As you know win xp allows only one connection. I am using HAproxy and above configuration. when i try connecting to xp through my haproxy server it does the connection perfectly, but the problems are
1) No persistence
2) Existing connection gets disconnected if someone else tries to get connected to the same machine.[XP problem]
How do i overcome these issues.
All help greatly appreciated.
Thanks
Shyam
July 9th, 2010 at 5:21 am
Shyam,
Set the maxconn 1 attribute for each server.
Here’s an example from our webserver haproxy config:
listen webfarm *:8080
mode http
stats enable
stats uri /admin?stats
stats auth admin:secretpass
balance roundrobin
log global
cookie JSESSIONID prefix
option httpclose
option forwardfor
server web01 192.168.2.101:8000 maxconn 1 check
server web02 192.168.2.102:8000 maxconn 1 check
server web03 192.168.2.103:8000 maxconn 1 check
May 18th, 2011 at 1:30 pm
hi guys
i have followed this work and i get much help by your method .but i still have some questions for the balance “rdp- cookie”, can you give me some advices:
1: this balance mode “rdp- cookie” support rdp version 5.1, 6.1 or 7.0 ? in my test it can work normally only in rdp5.1.
2: this balance mode said”Note that for this to work, the frontend must ensure that an RDP cookie is already present in the request buffer” , what kind of content is stored in this RDP cookie , where and how can i get this file.
thanks
May 18th, 2011 at 4:25 pm
Steve,
1) the method supports all versions you mentioned. We’ve tested several here, you can also refer to the following Microsoft link that gives more details:
http://download.microsoft.com/download/9/5/E/95EF66AF-9026-4BB0-A41D-A4F81802D92C/MS-RDPBCGR.pdf
its a big document! but on page 410 it does mention the versions that are supported.
2) to view the ‘cookie’ , you could run a tcp dump and view it there , although the following Microsoft link gives an example:
http://msdn.microsoft.com/en-us/library/cc240842(v=prot.10).aspx
May 19th, 2011 at 5:39 am
thanks for your reply so soon, and it give me much help.
i’ll introduce my test environment for you so you can make an exact analyse
1: i follows your first configuration,but i have changed the win2k8 for win7, it means my terminal servers is win7, if this replacement can take any effect?
2 : when a client send a rdp request to haproxy, what kind of information (the client’s IP address? MAC? or the username ?) will be stored in haproxy, so it can make sure that the same client will persist the same terminal server after disconnected for a few time ?
3:if haproxy remember the IP address, this means that the same IP address will persist connecting the same terminal server?
thank again
May 19th, 2011 at 1:52 pm
Steve,
you should be able to use Win7, although you’ll need either Professional, Ultimate or Enterprise. The ‘balance rdp-cookie’ line causes haproxy to distribute incoming requests based on a hash of the username that is entered. If no username is present , then haproxy reverts to round robin. The load balancer does not actually log/store anything, when a new request comes in, the username is again hashed and for the same user this results in the same hash, and the request is then forwarded on to the same terminal server.
May 20th, 2011 at 6:30 am
thank you very much !
by your way i have done a test :
i have three terminal servers of win7(they are all allows many connections ) and they are all in the same domain sdw-king.com, six users and the username is “test1, test2, test3, test4, test5, test6″, the haproxy is 1.4.15 in centos 5.5, i try to connect the terminal servers through haproxy with the same computer of win7, i open six rdp connection for test1 to test6 , badly the six users are sent to the same terminal server.
how can it happen, if the test1…test6 have the same hash??
another: if it have some rules for length of username ,for example 9 characters limit?
May 20th, 2011 at 10:28 am
dear sir
i have found a question for your answer, you said “The load balancer does not actually log/store anything” and ” when a new request comes in, the username is again hashed and for the same user this results in the same hash, and the request is then forwarded on to the same terminal server.” ,it seems they are contradiction.
the question is if the load balancer doesn’t log anything, how can the haproxy know the same user result in the same hash, how does haproxy know “this hash” is same as the “last time hash”, and the last time hash was sent to 192.168.22.113(one of win7 terminal server for example) so this time the user should be send to 192.168.22.113 ??
may be this is a large doubt for me!
May 20th, 2011 at 11:02 am
Steve,
The great thing about a HASH method on login/server is that it is always consistent (thats why HASHes get used so much in networking). However we are working with the Open Source community to add stick table support for RDP cookies which will be more flexible in the long term i.e. store the data in a table.
May 20th, 2011 at 12:00 pm
hi malcolm:
i am reading the haproxy Configuration Manual of 1.4.15 , the content for “persist rdp-cookie” may be give us some advise.
it says “This statement enables persistence based on an RDP cookie. The RDP cookie
contains all information required to find the server in the list of known
servers. So when this option is set in the backend, the request is analysed
and if an RDP cookie is found, it is decoded. If it matches a known server
which is still UP (or if “option persist” is set), then the connection is
forwarded to this server. ”
and ” Note that this only makes sense in a TCP backend, but for this to work, the
frontend must have waited long enough to ensure that an RDP cookie is present
in the request buffer.”
but it still don’t tell us how does haproxy mach a RDP cookie to a known server ??
i really thanks for your patience!
May 20th, 2011 at 1:48 pm
Steve,
There are really two kinds of ‘cookies’, the first is the cookie that’s sent from the client in the Connection Request PDU, the second is actually called a ‘routing token’ which is used with microsoft session broker. The line ‘balance rdp-cookie’ in the haproxy config file causes haproxy to interact with the cookie in the request PDU, whilst ‘persist rdp-cookie’ causes haproxy to ustilise the routing token for interaction with session broker, both methods are detailed in the haproxy documentation. So it does depend on how you’re trying to set things up. For basic setups without session broker/directory, you only need ‘balance rdp-cookie’ in the config file. Also, for this to work you need to make sure that you enter the username and password on the initial login prompt, otherwise the cookie will not be present and therefore the username cannot be hashed. Another useful Micrsoft article that looks at the routing token is: http://go.microsoft.com/fwlink/?LinkId=90204
June 28th, 2011 at 10:29 am
hi malcolm
we are now using haproxy very well , thanks for your guide.
can you tell me how many servers does one haproxy support?
June 28th, 2011 at 12:46 pm
Steve,
I’m not entirely sure, somewhere between 200 and 40000?……
The main thing restricting you will be the CPU load of all of the health checks…
January 27th, 2012 at 5:57 pm
Excuse me can you post the whole conf file?
February 7th, 2012 at 1:37 pm
I think the short answer is ‘a lot’ , your two pinch points will be the total number of TCP sockets and the total load of the health checks but I don’t see a problem with 300 servers. Probably best to ask the HAProxy mailing list and give them very specific information about the load you are expecting….