However its time for a confession, the default settings for the Loadbalancer.org appliances in a cluster configuration up until v7.5 have been set by default  for both ease of use and certainty of a valid configuration. The default recommendation for setting up or disaster recovery on the high-availability of the cluster (Heartbeat) has been to force a full sync and therefore inflict a small amount of down time in a maintenance window.

Whilst we’ve always had documentation showing how to handle cluster maintenance and configuration with zero downtime it was definitely time for a change.

So in theory this was a simple change to our default configuration, and yet as always the development team found a few thorny little issues that needed resolving. Previously the default heartbeat configuration used autofailback=on, this was handy in that you always new the master node would be active if it was healthy. However when it comes to a disaster recovery scenario or cluster change it becomes problematic as any failback to the master during the process causes downtime. On the positive side the old method does have the advantage that it quickly shows you if you have a heartbeat configuration problem i.e. the failback to master doesn’t work.

So we changed the default to autofailback=off, and we also made sure that when you do a  full restore on a node using the XML backup we make sure that heartbeat is stopped until you are ready to join the cluster.

Once the XML file is restored you can double check all the settings, make any required changes, possibly relocate the unit physically or logically and then choose to restart the heartbeat.

In this case we have restored a master node from scratch and when the heartbeat is restarted, the  new master node will stay passive but join the cluster in a clean fashion. The slave node will keep handling the network traffic to ensure that their is now down time incurred.

All seems pretty simple, so why didn’t we do this before? Well because we did have a couple of little gotchas that need to be dealt with by new code in the background. One of them was the built in replay attack protection in HA-Linux:

heartbeat[27177]: 2008/07/02_15:27:44 ERROR: should_drop_message: attempted replay attack [lbmaster]? [gen = 18, curgen = 1207732349]

We had a manual work around for this before but now the software will transparently deal with this issue when restoring from XML (easier said than done as curgen is stored within the process). The other little gotcha only effected customers using network based heartbeat in combination with ping nodes, we fixed this part of heartbeat ages ago so that it would send both nodes live in the case of a network failure (split-brain) or a ping node failure, however when the network connectivity was restored who would go live? This has now been sorted in the case of an XML restore to give the desired behaviour of no down time.

BTW: The split-brain heartbeat problem is why we still ship a serial heartbeat cable with all of our hardware appliances.

On the subject of no downtime, yet another new feature seems to be causing some uncertainty with our users. Whenever you make a change to a layer 7 HAproxy configuration you now get the following prompt:

Just to re-assure you, the HAProxy restart (Ed. Why don’t we rename it to reload then?) is seamless and does not cause downtime! HAProxy allows all existing connections to stay bound to the existing process and only connects new connections to the new configuration and process. The reason we changed from automatically restarting the layer 7 engine is because this can cause confusion with some users connecting to a new configuration and some connecting to the old one. With the new method you can make all the changes that you need to make i.e. multiple changes before doing the smooth restart/activation of the new settings. Also if you have a lot of long timeout connection i.e. Terminal Services you can choose to a full restart of HAproxy rather than the smooth reload to force all the connections to re-establish on the new configuration.The future progression of this feature is to allow full commit/rollback style functionality to the configuration interface.

One last point to note is that this reload functionality takes effect on both nodes in the cluster, this is to ensure that if you have full layer 7 session table replication activated in your configuration the passive node stays up to date with all of the changes.

As always high-availability is only possible if you test and document and re-test your disaster recovery procedures. We hope these changes to our product make your life easier, and the testing and validation procedure much quicker.

As you can quickly see from this graph, the number of connections/s, bandwidth and object size are all closely correlated. Depending on your application and usage pattern you will get vastly different throughput results from your load balanced cluster.

Generally even our smallest appliance can fill a 1GB pipe (we have several customers easily doing 2Gb+), But we do have some guidelines for our sales guys:
For deployments using Layer 7 and expecting a very large number of connections/second, or deployments with a large number of SSL TPS – This is very CPU intensive so we generally recommend our MAX or Dell hardware.
For Layer 7 deployments with very large numbers of long connections i.e. Exchange 2010 with 5000+ users – This is very memory intensive so we generally recommend our MAX or Dell hardware or the ENTERPRISE VA.

So one of the problems that load balancer vendors have is specifying good looking numbers i.e big ones, relating to their load balancer performance. Loadbalancer.org is just as guilty as the other vendors in using best case scenarios for performance:

 Does this specification mean that you can get 60,000 HTTP requests a second AND 1.5GBps throughput? I don’t think so…….

Does this specification mean that you can get 500 SSL TPS on our least powerful appliance with a 2048 Bit key? I don’t think so……

Loadbalancer.org SSL stats are all based on 1024 Bit keys…..
One of our guys will shortly write a blog on the full test process we use + a comparison of the different cyphers and their effect on performance, he even has a $16K Thales crypto card he’s been putting through its paces for an interesting comparison of SSL Hardware/ASIC Acceleration versus generic multi-core CPUs….

It will be very interesting to see what the customers think!

Historically most of our customers have used Layer 4 load balancing, which is fast and transparent and simple… and while most customers were happy with the current connection number, as it sounded about right…what did it actually mean? Not a lot… as it transpired.

Loadbalancer.org, Barracuda Networks & Kemp Technologies all use LVS (Linux Virtual Server) as the underlying Layer 4 packet engine. This awesome Linux kernel based load balancer could not give a monkeys how many users or sessions are connected to it, it just blasts the traffic through as fast as possible….Don’t get me wrong, LVS does know exactly what it is doing, it can tell you how many TCP connections are ESTABLISHED, WAITING etc. But does that information actually help a network administrator know how many users/sessions are connected? Not really… especially in DR mode where the load balancer can’t even tell if the connection has been closed by the server so it just guesses that that each connection is open for 2 minutes…..

The funny thing is that only about 0.05% of customers at Layer 4 even noticed that the connection numbers were a bit weird/useless.

When it comes to Layer 7 you would think it gets easier, which it does because at Layer 7 you know exactly who is connected at every point in time….

However 78.6% of our customers were NOT happy that the connection count was so low!
Why was the count so low at Layer 7? Because it knew damn well that the server/client disconnects almost immediately with HTTP traffic (as it is a connectionless protocol remember?)

So after several beers (yes we are English) the development team decided that what we actually need was two things…

1) A ‘Gauge’ of how many new connections occurred over a specific time period i.e. 10 seconds…
2) A specific number of connections if those connections were held open over a long period i.e. 15 RDP (Terminal Server Connections) held over a 20 minute period….

Now anyone in their right mind (and by that I mean the support team) would know that you can’t put two different measures like that in the same graph….. But the development team seam to know better….

So now the graphs plot connections/10 second period if the total connection count is climbing…..
OR
Total Current Connections if the total connection count is NOT climbing…..

I just wrote that down and it still doesn’t make any sense!

I’m glad to say that the other load balancer vendors have exactly the same problem:

Look at these Barracuda Networks graphs:

What the heck does that mean?

I was clicking the web page repeatedly and it is saying I have approx 0.42 connections? Very useful.

OK, So what do the numbers show?

Um , nothing….. am I not connected then?

I must admit Kemp Technologies do a lot better:

They try and show connections over:  60 seconds avg. /  5 mins avg. / 30 mins avg. etc
Which almost makes a lot of sense, but is pretty ugly and hard to read isn’t it?

Now I’m sure that both Kemp & Barracuda will say , “Ah, yeah but, please use SNMP to figure out what the heck is going on…..”
And to be honest they are right, and that is what Loadbalancer.org also recommends….
BUT…. SNMP reporting is a pain to configure… so as of v7.5 all Loadbalancer,org appliances automatically and dynamically generate graphs of all connections to virtual/real servers…..

For example, the old v7.x Loadbalancer.org appliances would report the current number of connections per virtual server/ real server:

Which is great for point in time snapshot recording….
i.e. we have 110,000 connections so our exchange server farm is pretty busy…….

Now.. in v7.5,  we will get a graph showing exactly which server is handling the most traffic and also clearly highlighting any historical drops in traffic i.e. technical problems!
Just click on the little blue graph icon next to each VIP/RIP on the system overview…..

And it would be really useful if I had an actual graph here to help with the explanation…

The graph above shows a fairly healthy pair of Terminal Servers with even load balancing distribution over a day. Any failure or uneven load would be fairly easy to spot either from the graph or the load averages. You would then need to drill down for more data.

Anyway my basic point is , please can you try our new dynamic graph stuff and tell us:

a) It sucks, you suck… what do you do again?

b) Can I have my reassuring but irrelevant connection numbers back please?

c) Something more constructive…

BTW… One of the other cool things we did is move all of the server side processing of the system overview page onto the client using Javascript… No doubt this will end in tears but the intention was to allow the system overview page to handle hundreds of servers without a performance hit… This was also a reason for removing the connection counters (performance..), the graphs are only generated when you request them on a per VIP/RIP basis….

While I’m at it….

Who the heck removed the CPU load statistics!

Um, time for a quick diagram:

We now use a standard Linux system Load graph to tell you how busy the load balancer is…….
You know the thing you get when you type uptime on a Linux box?

[root@lbmaster ~]# uptime
22:19:57 up 15:04, 1 user, load average: 0.09, 0.04, 0.01

What the heck does that mean?

I have no idea…. BUT the basic gist is:

Linux System Load = Amount of processes waiting for a processor

Why is that useful? Because it tells you very quickly if things are going wrong with any part of the system… that’s why.

Most Loadbalancer.org hardware has 4+ cores … so any load reading > 4 is VERY bad………

Contact support and have a chat…. It could be Memory / Hard Drive / CPU / Quarks … who knows, but at least the graph is pretty….

Now lets get back to the conversation about numbers v graphs….

Connection number graphs are all very well and good …. they will probably tell you at a quick glance if one of your servers is having issues….
But what other information can we get? Well at Layer 4… not an awful lot… so you will need to head straight to your server error/connection logs and start analysing them.

However at Layer 7 we do have quite a bit more information:

Haproxy gives a good snapshot of current and total session/connection/error data. However straight numbers are quite hard to analyse,
so wouldn’t graphs be useful here as well?

Also at Layer 7 you can turn on full logging on the load balancer, but Loadbalancer.org strongly recommend only doing this briefly or using an external syslog server….

You can quickly get an awful lot of logs!

Where do we start?, well our first thought was to check if there were any issues with the load balancing application or its configuration. we found that everything there looked to be fine except that not every user seemed to be in the stick table and that testing the exact same environment with Windows XP or a Linux client we were unable to replicate the issue.

OK so what do we do now. we know the load balancer and its configuration seam to be working as it should as the XP client etc.  So what is windows 7 and above doing that is different and why is there not always all the users in the stick table?…

After some research we found the TechNet documentation on the x.224 Connection Request PDU  and the following key text.

<13> Section 3.2.5.3.1: Microsoft RDP 5.1, 5.2, 6.0, 6.1, 7.0, 7.1, and 8.0 clients always include the cookie field in the X.224 Connection Request PDU if a nonempty username can be retrieved for the current user and the routingToken field is not present (the IDENTIFIER used in the cookie string is the login name of the user truncated to nine characters).

Ok so that makes sense, when making a connection request the client should send a cookie made from the users username so the loadbalancer knows who the user is. We fired up Wireshark and what do you know we found that the new client does not always send this cookie. it seems it will not send it or the incorrect cookie in the following conditions:

1. Starting the client and using the cached/saved credentials, then no hash is sent.
2. Mistyping the username and then correcting, mstsc still uses the old username as the hash and not the updated username.
3. Under certain conditions if a user entered the wrong password the cookie would default to domain/user and not the expected user@domain format that was entered in the username field.  This is an issue for users with domains over 9 characters long as the cookie is then allowed to be duplicated for many users or not match the previously used cookie.

Ok so what happens with the XP client.  Well this seems to follow the documentation and explains why that worked.

how do we fix this? Well the TechNet Docs say this is how it works, yet the client does not do this…

Sounds like a bug so lets log it with Microsoft……First problem. how the hell do you log a bug? nothing stands out from their site so instead we paid  and logged a support call with them (which is still ongoing 4 months later).

So where does this all lead.  Well so far Microsoft have stated this is and i quote (including spelling mistake) from them:

“The behavior that you are experiencing is a documentation bug and we are currently in the process of publishing a Knowledge Base article for it”

Err ok, so a Microsoft Product does not follow their own documentation and they say it is a Documentation bug? So their answer is if a product does not fit the design spec, just change the spec. I have now seen the new version of the documentation and it now says…

“cookie (variable): An optional and variable-length ANSI character string terminated by a 0x0D0A two-byte sequence. This text string MUST be “Cookie: mstshash=IDENTIFIER”, where IDENTIFIER is an ANSI character string (an example cookie string is shown in section 4.1.1). The length of the entire cookie string and CR+LF sequence is included in the X.224 Connection Request Length Indicator field. This field MUST NOT be present if the routingToken field is present.”

So now its an optional item,and not always sent.  Well thats no good to anyone using RDP cookies.

They also say there is a perfectly valid workaround….. but failed to advise what it was, well after chasing for it we are told….

“This problem occurs only when the same MSTSC process is used to connect to a server with 2 different credentials. The scenario can easily workaround by starting a new MSTSC process before connecting to the server with different credentials.”

Really?  Well to us this indicates an issue with the MSTSC client where it is unable to update its own cookies without having to restart the application, and after many emails back and forth we get the very latest response.

“Based on my internal discussions, we conclude that for load balancing terminal servers, using Session Directory is the recommended approach. This whitepaper talks more about the implementation http://download.microsoft.com/download/8/6/2/8624174c-8587-4a37-8722-00139613a5bc/TS_Session_Directory.doc. In the business impact statement that you had provided, I did see that you have mentioned that you cannot use Session Directory.”

Well thats lovely for anyone with Windows 2003 Enterprise and above, what about users of 2003 Standard who do not have the Session Directory included? hell their document even says this.

“Note:

Terminal servers must be running Windows Server 2003 Enterprise Edition or Windows Server 2003 Datacenter Edition to participate in a Session Directory-enabled farm.”

Well we are still waiting for an answer to this from Microsoft but it looks like the RDP cookie maybe being dropped by Microsoft and no fix in sight… Guess we will just have to wait and see what they say next, however we would love to hear from anyone else who is experiencing the same issues as us so we can advise Microsoft of other users who are being hampered by this issue.

Update…

Well it looks like Microsoft have indeed silently dropped support for mstshash cookies for load balancing as suspected…..

As detailed in the last post we had a call open with Microsoft and have just received the following response that confirms our suspicions that they have in fact dropped support for mstshash silently in favour of their Session Directory/Broker solutions.

Mathew

Hope this email finds you well. I regret to state that, I’ve a negative news for you.

Based on the triage of code with product group, they suggest you to devise some alternate way to achieve load balancing.

According to them, the dependency you took on the optional cookie field is not recommended or supported.

The protocol documentation does not specify what is passed in that field and the value can change based on the scenario.

Unfortunately, there are no plans on updating the documentation as it does not call out what is being passed in this field and is already marked as optional.

Please let me know if you wish to have a conference call with us on this matter.

I would attempt to bring the PM on call.

Thank you for your understanding and patience. I again, is truly regretful.

Subheet Rastogi | Support Lead

Enterprise Platforms | Microsoft Corporation | Office: <number removed>

]]> http://blog.loadbalancer.org/microsoft-drops-support-for-mstshash-cookies/feed/ 4 http://blog.loadbalancer.org/exchange-2013-microsoft-finally-have-an-email-solution-designed-for-high-availability-and-load-balancing/ http://blog.loadbalancer.org/exchange-2013-microsoft-finally-have-an-email-solution-designed-for-high-availability-and-load-balancing/#comments Sat, 23 Feb 2013 21:14:42 +0000 Malcolm Turnbull http://blog.loadbalancer.org/?p=977 OK, So maybe my blog title is a little harsh. The Microsoft Exchange products have been pretty scaleable since Exchange 2007.
Exchange 2010 had some vast improvements and you could tell that the Microsoft engineers had put a lot of effort into trying to ease the painful mess that was load balancing an Exchange 2007 cluster. They even started recommending hardware load balancers and certifying vendors for Exchange 2010 compatibility.

But with Exchange 2013 they appear to have started from scratch and written it properly! They actually sat down and said “What do we need to have a scaleable Exchange email cluster?”. Funnily enough they came up with the same answer that Loadbalancer.org has been banging on about for last 10 years. In order to have a scaleable cluster, each node in the cluster must be able to handle any user session at any time; i.e. no messing around with persistence, sticky plasters, and hack jobs.

Well done Microsoft!

But please can you put the same infrastructure architects on your LYNC team?

Seriously, who puts 3 DMZs in an enterprise product? I had to put Rob (who made sure Loadbalancer.org gained Microsoft Lync certification) on suicide watch …

Better get back to discussing the point of this blog I guess …

In order to gain high-availability for an application a load balancer is fairly fundamental, because a load balancer gives you:

Resilience: A load balancer continuously monitors your application servers and, if it detects a problem with them, it moves the traffic onto another server in the cluster. This was not possible before Exchange 2013 without a brief interruption of service. Now it’s easy. Tony Redmond has some good points about how important flexible health monitoring of Exchange 2013 still is.

Maintainability: You can’t have 100% uptime on every server, eventually they will break or at least need security updates or software updates. With a load balancer you simply drain the connections, perform the maintenance, and then bring the server back online. Exchange 2013 makes software upgrades easy as discussed by Rand Morimoto.

Performance: This one is often misunderstood. Contrary to popular belief you don’t need a super fast load balancer to increase your performance – you just add more backend servers to the cluster! However this requires that the cluster can handle horizontal scaling which also was not possible before Exchange 2013.

Kemp Technologies (one of our main competitors) is going to be gutted by the changes in Exchange 2013. They made this lovely sell up tool which shows how much horse power you need to SSL terminate in front of Exchange 2010 clusters. Don’t get me wrong they have a great product; it’s just that with Exchange 2013, customers will no longer need any SSL termination on the load balancer so their cheapest product will handle the load easily.

F5 – who are very deservedly the market leader in the load balancing space – are mounting a defensive PR reaction to Exchange 2013 improvements already. BTW: This post by Ryan Korock is terrible; at least come up with some real reasons to use F5, i.e. awesome logging, monitoring, snmp interrogation, L7 performance based re-directing, re-direct on failure of connections. Come on man … L7 rocks when you have the capability of an F5!

Anyway long story short but if you are in the middle of an Exchange 2010 migration then do yourself a favour: Stop now! Whatever the initial cost to move to Exchange 2013 right now, it’s awesome!

PS: Dimitrios Kalemis agrees with me …

First from the WUI :

1. Set the external relay(Smart Host) under Edit Configuration > Physical – Advanced Configuration.

2. Enable HAProxy Logging under Edit Configuration > Layer 7 – Advanced Configuration.

Then from the CLI :

1. Install the logwatch package using yum like so :

[root@lbmaster ~]# yum --disableexcludes=all install logwatch

2. Create the following file and set your To/From email addresses : /etc/logwatch/conf/logwatch.conf

MailTo = myemailaddress@example.com
MailFrom = LBMaster@example.com

At this point you’ll have a standard Logwatch install which will send an email once per day (you may want to disable this).

*To disable the daily logwatch email execute : chmod -x /etc/cron.daily/0logwatch

3. Create a custom Layer 7 check

a. Create the following file adding the contents below : /etc/logwatch/conf/logfiles/layer7.conf

LogFile = /var/log/haproxy.log
*OnlyHost
*ApplyStdDate

b. Next create the script : /etc/logwatch/scripts/services/layer7

use strict;
my $find = "is UP|is DOWN";
my @lines = <STDIN>;
for (@lines) {
     if ($_ =~ /$find/) {
         print "$_\n";
     }
 }

c. Finally create the following file : /etc/logwatch/conf/services/layer7.conf

Title = "Layer 7 Errors"
LogFile = layer7

4. Enable the Logwatch job to run every minute with Cron

a. Edit crontab with the following command :

[root@lbmaster ~]# crontab -e

b. Add the following new line to root’s crontab :

*/01 * * * * /bin/nice -n 19 /usr/sbin/logwatch --service layer7 --range '-1 minutes for that minute'

Once this is complete you’ll now receive an email in the event of a real server failure. The way this works is that logwatch will run every minute and search the previous minutes log entries for servers that are taken down or brought up during that time.

Loadbalancer.org GmbH,
Alt Pempelfort 2,
40211 Düsseldorf,
Germany
phone – +49 (0)30 920 383 6494
email – vertrieb@loadbalancer.org

“I have always wanted to open an office in Germany but felt I wanted a German speaking support team first” states Malcolm Turnbull, Managing Director, Loadbalancer.org.

Henry Meitzer, Sales Manager, Loadbalancer.org states “Opening the Germany is a huge step forward for Loadbalancer.org, I think its a great opportunity to expand our customer base within Germany“.

To celebrate the opening of our new office we are exhibiting at the Cebit 2013, Hannover.  Why not come and meet the Loadbalancer.org team, we are in Hall 12, Stand C40. More information email sales@loadbalancer.org or visit our profile page on the Cebit Website. We have spare entry tickets if you require one then get in contact.

UPDATE – We survived, well just about. Below are some of the pictures from the show:

Our previous trade show include:

Infosecurity Europe (London) www.infosec.co.uk

This was my first custom build stand at a trade show. The stand came together smoothly but without the help of Marina and her team at Inspire Displays I would not have been able to get it looking so good. Their attention to detail amazed me.

IPexpo (London) - www.ipexpo.co.uk

We skipped the IPexpo trade show for a couple of years, but this year we came back with bang. This turned out to be the perfect year to visit, we met existing customers as well as picking up a handful of new ones.

Infosecurity (Netherlands) – www.infosecurity.nl

Our first visit to Holland, what fun we had. We have a large customer base here, it was the perfect opportunity to meet some of them.
 

This posting shows how to setup a blank virgin installation of Centos 6.3 64bit minimum installation.

This guide works on the assumption that you have a public facing IP Address of 192.168.10.50 (I know thats not a real public address) and are using an internal network address space of 10.10.10.x/24 with our two web servers on 10.10.10.10 and 10.10.10.15. So we will have two network interfaces on our LoadBalancer eth0 will be set with our real world IP of 192.168.10.50 and eth1 will be set up with 10.10.10.1.

After installing our basic Centos 6.3 64bit OS, it maybe worth running a ‘yum update‘ command first to ensure that the system is fully updated.

As this is a minimum installation you will also need to install a few other packages. These can be installed with the following command:

yum install make wget gcc pcre-static pcre-devel

I’m using the HAProxy 1.5 dev7 build for this example but at the time of writing dev12 is the latest available build and I’ll assume that the following will also work with that Development Release. However, to get all the features that we require we will need to build HAProxy from source and not from the package repository. The following steps enable us to do just that:

• wget http://haproxy.1wt.eu/download/1.5/src/devel/haproxy-1.5-dev7.tar.gz
• tar -zxf haproxy-1.5-dev7.tar.gz
• cd haproxy-1.5-dev7
• make TARGET=linux26 USE_STATIC_PCRE=1 USE_LINUX_TPROXY=1
• cp haproxy /usr/bin/haproxy
• cp examples/haproxy.cfg /etc/haproxy.cfg

The installation is now completed. However, we have only an example configuration file installed at ‘/etc/haproxy.cfg’ this is the file that will store all of the settings that we require to ensure our website is available for the maximum number of visitors. So we now need to edit this configuration file I’m going to use ‘vim’ but if you are more familiar with ‘nano’, ‘ee’ or another editor please use that.

vim /etc/haproxy.cfg

Have a quick look through the file if you wish and see the basic structure of the configuration file, we are going to create a VERY basic config to start with just to make sure that our installation is working.

global
daemon
log /dev/log local4
maxconn 40000
ulimit-n 81000

defaults
log global
contimeout 4000
clitimeout 42000
srvtimeout 43000

listen http1
bind 192.168.10.50:80
mode http
balance roundrobin
server http1_1 10.10.10.10:80 cookie http1_1 check inter 2000 rise 2 fall 3
server http1_2 10.10.10.15:80 cookie http1_2 check inter 2000 rise 2 fall 3

Save the above configuration file and then to start the HAProxy service use the following command from the command line:

/usr/sbin/haproxy -f /etc/haproxy.cfg

If everything starts correctly you should be able to browse to your real IP Address using a different compute and see you default page, as mine are just two Debian Web Server I get the following:

If you see the above image or the page for your servers. Congratulations your two web servers are now in High Availability mode. If you do not see your default page stop HAProxy with a killall haproxy command and run /usr/bin/haproxy -d -f /etc/haproxy.cfg this will restart HAProxy with debugging displayed on the console screen to stop the debug info being printed and the HAProxy Service simply press Crtl+C

Now that the basic High Availability is working lets move to Transparent mode.

So with a stopped HAProxy service open your /etc/haproxy.cfg file again with your editor of choice and in the ‘listen http1 section’ add the following

option http-server-close
option forwardfor
source 0.0.0.0 usesrc clientip

You will now need to edit your iptables rules. I have this as my ‘iptables-rules.sh’ file:

iptables -t mangle -N DIVERT
iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
iptables -t mangle -A DIVERT -j MARK –set-mark 111
iptables -t mangle -A DIVERT -j ACCEPT
ip rule add fwmark 111 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100

If you now run this file and then start your new modified HAProxy file and retest to your web server on the Real IP Address you should be able to see in the HTTP Access logs that the address that your site was visited from is not that of the LoadBalancer.

The main problems were getting the required loopback adapter on the TMG servers to function correctly

So this is how we got it to work.

Please note this is not a guide on how to loadbalance your firewalls for outbound connections where the TMG servers are acting as a reverse proxy / web filter aka. squid (allthough it would be very similar and that can be discussed in another blog)

The Network we will build will look like the following where we are loadbalancing the connections between the two TMG servers via a loadbalancer, this is then sent from the TMG Server to the WebServer via a NAT rule.

The first step is to Add the loopback adaptors to the TMG servers, this process should be quite trivial but can not be done from the TMG snap-in so here is the instructions step by step.

1. Add Loopback Adapter to the TMG server.
This needs to be done via the Device Manager – Add Legacy Hardware option

2. Follow the Wizard

3.  Select Manual

4. Select Microsoft Loopback Adaptor

5. Once installed, edit the IP address of the loopback Adaptor and remove any unneeded services

6. Assigned the VIP address to the Loopback, the subnet mask should be a /32 (this can be edited in TMG but it does not allow a /32 there so we do it here)

8. TMG does not find this interface right away so a reboot of the TMG server is needed.

9. We now need to create a network Object for the new Adaptor, to do this we do the following

Access the TMG Snap-in

10. Click on Toolbox on the right hand side, then Network objects, once their right click on Networks, and then New Network…

11. Name the new Network

12. Select Perimeter for the network type, as this will be internet facing

13.  Now we add the ip assigned, so click on Add Adaptor and add the loopback.

You should then be left with this

14. click finish

15. Apply the change

Next we need to setup the Network Rules to allow the the traffic to traverse the loopback

16. Click on Networks, then Network Rules

17. Click on Create a Network Rule

18. Add the traffic sources

19. Add a destination

20. Set the Network Relationship to NAT

21. Leave the IP selection as default

22. Thats it for the Network Rules

Now apply the settings.

Next we need to configure the Firewall Rules and Listener

23. Go to Firewall Policy

23. Click Publish Web Site, and name the new Rule

24. Allow the connections

24. Now select what you would like to publish in my case its a single IIS site running on the DC (this is a test lab, in a production setup it would be its own server)

25. Select HTTP (you can use HTTPS but in my lab its just http)

26. Enter the server that is hosting the Site, in my lab its the DC

27. Next is the path. i left this as default

28. next setup the URLs to be accepted, I set this to allow all.

29. Now we configure a Web Listner, so click on new

30. Now name your new listen, I called mine IIS – Ext and Loopback as it will listen to Ext and Loopback networks

31. Select HTTP so this will listen for HTTP traffic

32. Next Select the networks that the listner should listen on.

33. Set the Auth as required, I use No Auth in my example

34. Next SSO if you use it, I dont

35. That is all for the Listener

36. your listener box should now look like this

37. Leave the Auth Delegation as defaults.

38.  Set the users who can access this, I left it as default

39. That should be it for the rules.

40. Apply the new settings

41. now we need to setup a rule to allow External Traffic to the Loopback via the firewall, so add a new access rule

42. Allow the Rule

43. Add the correct protocols

44.  Enable Malware Scanning

45.  Select the External as source network

46. Select Loopback as the destination network

47. Allow all users access to this rule

48. Finalise the rule

49. Apply settings and you should be done

1. Update

root@localhost:~# apt-get update

2. Install dependencies

root@localhost:~# apt-get install build-essential make libpcre3 libpcre3-dev

Downloading/Building the HAProxy package :

1. Download the HAProxy Package available from http://haproxy.1wt.eu

root@localhost:~# wget http://haproxy.1wt.eu/download/1.5/src/devel/haproxy-1.5-dev11.tar.gz

2. Extract the package

root@localhost:~# tar -xvzf haproxy-1.5-dev11.tar.gz

3. Change directory and make the package

root@localhost:~# cd haproxy-1.5-dev11

root@localhost:~/haproxy-1.5-dev11# make TARGET=linux2628 ARCH=x86_64 USE_PCRE=1

4. Install the newly compiled package and confirm it is installed.

root@localhost:~/haproxy-1.5-dev11# make install
root@localhost:~/haproxy-1.5-dev11# /usr/local/sbin/haproxy -vv

Assuming you didn’t run into any errors with the previous commands you should now have HAProxy installed.

Configuring startup script :

1. Create the startup script

root@localhost:~/haproxy-1.5-dev11# nano -w /etc/init.d/haproxy

2. Paste the following into the new file and save it(with Ctrl+X)

#!/bin/sh
# /etc/init.d/haproxy

PATH=/bin:/usr/bin:/sbin:/usr/sbin

pidfile=/var/run/haproxy.pid
binpath=/usr/local/sbin/haproxy
options="-f /etc/haproxy/haproxy.cfg"

test -x $binpath || exit 0

case "$1" in
  start)
    echo -n "Starting HAproxy"
        $binpath $options
    #start-stop-daemon --start --quiet --exec $binpath -- $options
    echo "."
    ;;
  stop)
    echo -n "Stopping HAproxy"
    kill `cat $pidfile`
        #start-stop-daemon --stop --quiet --exec $binpath --pidfile $pidfile
    echo "."
    ;;
  restart)
    echo -n "Restarting HAproxy"
    #start-stop-daemon --stop --quiet --exec $binpath --pidfile $pidfile
    kill `cat $pidfile`
        sleep 1
    $binpath $options
    echo "."
    ;;
  *)
    echo "Usage: /etc/init.d/haproxy {start|stop|restart}"
    exit 1
esac

exit 0

3. Change permissions and register the startup script

root@localhost:~/haproxy-1.5-dev11# cd /etc/init.d
root@localhost:~/etc/init.d# chmod +x haproxy
root@localhost:~/etc/init.d# update-rc.d haproxy defaults

You should now be able to start and stop haproxy with “service haproxy <action>” where the action is start/stop/restart.

Creating the HAProxy configuration file :

1. Create folder structure and open the config file for editing

root@localhost:~/etc/init.d# mkdir /etc/haproxy
root@localhost:~/etc/init.d# nano -w /etc/haproxy/haproxy.cfg

2. Paste in the example configuration and adapt for your settings.

N.B. The bind lines below can be adapted to listen on a specific IP address, simply add your desired local IP: bind 192.168.72.120:135,192.168.72.120:60200,192.168.72.120:60201

global
 daemon
 stats socket /var/run/haproxy.stat mode 600 level admin
 pidfile /var/run/haproxy.pid
 maxconn 40000
 ulimit-n 81000
 defaults
 mode http
 balance roundrobin
 timeout connect 4000
 timeout client 86400000
 timeout server 86400000

frontend CAS-RPC
 bind :135,:60200,:60201
 mode tcp
 maxconn 40000
 default_backend CAS-RPC-SERVERS

frontend CAS-WEB
 bind :80,:443
 mode tcp
 maxconn 40000
 default_backend CAS-WEB-SERVERS

frontend HT-SMTP
 bind :25
 mode tcp
 maxconn 40000
 default_backend HT-SERVERS

backend CAS-RPC-SERVERS
 stick-table type ip size 10240k expire 60m
 stick on src
 option redispatch
 option abortonclose
 balance leastconn
 server EXCH01 192.168.72.222 weight 1 check port 135 inter 2000 rise 2 fall 3 on-marked-down shutdown-sessions
 server EXCH02 192.168.72.223 weight 1 check port 135 inter 2000 rise 2 fall 3 on-marked-down shutdown-sessions

backend CAS-WEB-SERVERS
 stick-table type ip size 10240k expire 60m
 stick on src
 option redispatch
 option abortonclose
 balance leastconn
 server EXCH01 192.168.72.222 weight 1 check port 443 inter 2000 rise 2 fall 3 on-marked-down shutdown-sessions
 server EXCH02 192.168.72.223 weight 1 check port 443 inter 2000 rise 2 fall 3 on-marked-down shutdown-sessions

backend HT-SERVERS
 option redispatch
 option abortonclose
 balance leastconn
 server EXCH01 192.168.72.222 weight 1 check port 25 inter 2000 rise 2 fall 3 on-marked-down shutdown-sessions
 server EXCH02 192.168.72.223 weight 1 check port 25 inter 2000 rise 2 fall 3 on-marked-down shutdown-sessions

listen stats :7777
 stats enable
 stats uri /
 option httpclose
 stats admin if TRUE
 stats auth admin:password

3. Start HAProxy with your new configuration

root@localhost:~/etc/init.d# service haproxy start

N.B. At this stage if you receive errors like below please check that something else is not listening on any of the required ports.

[ALERT] 205/123152 (2839) : Starting proxy CAS: cannot bind socket [192.168.72.120:135]

You now also have a WUI in the form of the HAProxy stats page which includes useful options such as taking a server offline etc.

http://<IP-ADDRESS>:7777/

Configuring the Exchange 2010 CAS role :

1. Either configure the ports manually or using the following Registry file(user beware)

Link to the Registry file = http://downloads.loadbalancer.org/RPC%20Ports.reg

Manual Static Port Configuration

To set a static port for the RPC Client Access Service, open the registry on each CAS and navigate to:

HKEY_LOCAL_MACHINESYSTEMCurrentControlSetservicesMSExchangeRPC

Here, you need to create a new key named ParametersSystem, and under this key create a new DWORD(32-bit) Value named TCP/IP Port as shown below. The Value for the DWORD should be the port number you want to use. Microsoft recommends you set this to a unique value between 59531 and 60554 and use the same value on all CAS. In this Blog the port used is 60200.

N.B. Make sure you use a DWORD Value for this key

To set a static port for the Address Book Service, open the registry on each CAS and navigate to:

HKEY_LOCAL_MACHINESYSTEMCurrentControlSetservicesMSExchangeAB

Here, you need to create a new key named Parameters, and under this key create a new String Value named RpcTcpPort as shown below. Microsoft recommends you set this to a unique value between 59531 and 60554 and use the same value on all CAS. In this Blog the port used is 60201.

N.B. Make sure you use a STRING Value for this key

2. Creating the DNS entry

Create a DNS record for the CAS Array, this should be the same as the load balancer’s IP address(bind address if used earlier), e.g. cas.domain.com

3. Configure the CAS array object

Use the following command from the Exchange 2010 management shell to create the object :

New-ClientAccessArray –Name “CAS-array” –FQDN “cas.domain.com” -Site “default-first-site-name”

N.B. change “default-first-site-name” to the AD site appropriate for your Client Access Servers
N.B. change “cas.domain.com” to the FQDN of the CAS array(same as the DNS entry)

If the mail database already existed before creating the array, you’ll also need to run the following command to relate the new CAS array to the database:

Set-MailboxDatabase "NameofDatabase" -RpcClientAccessServer “cas.domain.com”

To verify the configuration of the CAS array, use the following commands from the Exchange Shell :

get-ClientAccessServer

lists the available Client Access Servers

get-ClientAccessArray

lists the Client Access Array and its members

Finished

Once you’ve completed all the previous steps you can now access your CAS services via your load balancer IP, it should also be correctly load balancing connections for better performance and real server resilience. There are still many ways you could build further resilience or add more features to this solution such as HA, DAG’s and SSL Termination but this will still give you perfectly adequate load balancing of the CAS and HT roles.