Loadbalancer.org has always been about high-availability, that is the fundamental reason for our products existence. Performance has always been a nice side effect while maintainability of your application cluster is generally a key sub-set of the primary high-availability objective.
However its time for a confession, the default settings for the Loadbalancer.org appliances in a cluster configuration up until v7.5 have been set by default for both ease of use and certainty of a valid configuration. The default recommendation for setting up or maintaining the high-availability of the cluster (Heartbeat) has been to force a full sync and therefore inflict a small amount of down time in a maintenance window.
Whilst we’ve always had documentation showing how to handle cluster maintenance and configuration with zero downtime it was definitely time for a change.
Some of the most common questions we get at Loadbalancer.org are performance related. It is quite difficult to give a straight answer to these questions as the real answer is the slightly unsatisfactory, ” Um… well it depends on your application…”. The following graph showing HAProxy performance for different object sizes gives you a much better idea of the problem:
As you can quickly see from this graph, the number of connections/s, bandwidth and object size are all closely correlated. Depending on your application and usage pattern you will get vastly different throughput results from your load balanced cluster.
I must confess , at certain times it has looked like open warfare would break out between the support team and development team at Loadbalancer.org over the last few months.
One reason being that a major change in v7.5 is that all connection information is now displayed as a graph (rather than a static number). The support team was convinced that customers just wanted a number, a single number , a golden number… The development team was convinced that such a number was illogical irrelevant and meaningless……
It will be very interesting to see what the customers think!
Recently we have seen quite a few customer issues where using RDP cookies (mstshash cookies see - http://www.snakelegs.org for more details) seems to result in multiple active sessions over several RDP servers as shown below (Notice the user Rob on both TS Servers).
So we decided to investigate this and find out why……..
Where do we start?, well our first thought was to check if there were any issues with the load balancing application or its configuration. we found that everything there looked to be fine except that not every user seemed to be in the stick table and that testing the exact same environment with Windows XP or a Linux client we were unable to replicate the issue.
OK, So maybe my blog title is a little harsh. The Microsoft Exchange products have been pretty scaleable since Exchange 2007.
Exchange 2010 had some vast improvements and you could tell that the Microsoft engineers had put a lot of effort into trying to ease the painful mess that was load balancing an Exchange 2007 cluster. They even started recommending hardware load balancers and certifying vendors for Exchange 2010 compatibility.
But with Exchange 2013 they appear to have started from scratch and written it properly! They actually sat down and said “What do we need to have a scaleable Exchange email cluster?”. Funnily enough they came up with the same answer that Loadbalancer.org has been banging on about for last 10 years. In order to have a scaleable cluster, each node in the cluster must be able to handle any user session at any time; i.e. no messing around with persistence, sticky plasters, and hack jobs.
Well done Microsoft!
In this guide I show a very simple solution to get HAProxy email alerts configured using Logwatch. While the first part is aimed at users of our V7 appliance I think anyone wanting to get email alerts for HAProxy will also find this a good example.
First from the WUI :
1. Set the external relay(Smart Host) under Edit Configuration > Physical – Advanced Configuration.
Loadbalancer.org are pleased to announce the opening of an office in Germany.
Alt Pempelfort 2,
phone – +49 (0)30 920 383 6494
email – firstname.lastname@example.org
“I have always wanted to open an office in Germany but felt I wanted a German speaking support team first” states Malcolm Turnbull, Managing Director, Loadbalancer.org.
Transparent mode with HAProxy allows you to see the IP Address of the clients computer while still having a high availability service using HAProxy.
This posting shows how to setup a blank virgin installation of Centos 6.3 64bit minimum installation.
We were looking at Microsoft’s new TMG server and loadbalancing and after a search of the web found there was not really any Guide on how to loadbalance incoming web connections via two TMG servers to an internal NAT’ed web sever at layer4.
The TMG servers are effectivley acting as WAFs (Web Application Firewalls) for the incomming traffic.
They require the traffic to be transparent (so they can inspect the client source IP address), so layer 4 DR mode is a logical choice.
The main problems were getting the required loopback adapter on the TMG servers to function correctly
So this is how we got it to work.
Please note this is not a guide on how to loadbalance your firewalls for outbound connections where the TMG servers are acting as a reverse proxy / web filter aka. squid (allthough it would be very similar and that can be discussed in another blog)
The Network we will build will look like the following where we are loadbalancing the connections between the two TMG servers via a loadbalancer, this is then sent from the TMG Server to the WebServer via a NAT rule.
This Blog is for anyone wanting to load balance the Exchange 2010 CAS role using only open source software. In my example I will be starting with a simple Debian net-install and building the HAProxy package from source because I wanted the latest feature set available. I would definitely recommend using a recent 1.5-dev build if following this guide or parts of the HAProxy configuration may be incompatible.
Yes its true!
Feel free to rub your eyes, do a double take, reload the page, get a cup of coffee to wake yourself up then reload the page again.
ITS FINALLY HERE! Version 7.4 is available for your hardware Loadbalancer.org appliances!
Continue reading for the installation procedure…………
Over the last few weeks we have seen more and more users reporting that they have run a security check on the SSL certificate thats installed on their Loadbalancer appliance using the Trustworty Internet Movement web site (https://www.trustworthyinternet.org/ssl-pulse/).
The idea behind the site is basically to test as many SSL certificates on the Internet as possible and check for any vulnerabilities like having SSLv2 enabled or weak Key Cipher lists. The test takes about 2 minutes to run and will give you a report on the status of your SSL Certificate and the associated services that it uses.
From this we found that the version of Pound SSL Proxy that we were using with our v6.x appliance was not as secure as it could be. Which has lead to a new release of our hardware software to v6.19.
NB. ‘not as secure as it could be’ does not mean a security problem, the BEAST attack is really a client side attack and nothing to do with load balancers <- Anoying comment added by Editor .
We have been doing some internal testing with Collectd.
“collectd gathers statistics about the system it is running on and stores this information. Those statistics can then be used to find current performance bottlenecks (i.e. performance analysis) and predict future system load (i.e. capacity planning). Or if you just want pretty graphs of your private server and are fed up with some homegrown solution you’re at the right place, too ;).” (taken from their website)
Anyway we are quite impressed with it but during the configuration I couldn’t locate an upstart script for it so I made my own.
Any engineer dealing with PCI DSS compliance issues probably looses a little bit of the joy in life .
Now don’t get me wrong, The PCI DSS has a laudable aim and is written quite well and mostly sensibly but like the bible is open to a vast amount of interpretation…
So lets start with the fundemental issue:
Q. Is my load balancer secure?
A. If you have Firewalled port 22 (SSH) and 9080/9443 (Loadbalancer.org admin ports) then YES IT IS SECURE – job done, go home.
Microsoft print server provides a great way to share printers throughout your organisation, but when the print server service falls over, the phone quickly starts to ring. By adding a load balancer and a second print server configured with the same print queues , you’ll quickly have a load balanced and resillient printing infrastructure for your users.
We do quite a bit of work with web proxy vendors, loadbalancing multiple web filters/proxies with one of our appliances and our customers have requested a way of health checking through the proxy when they have NTLM authentication enabled. Always happy to help where we can I have created a script that will retrieve a web page via your proxy (logging in first of course) if it retrieves it successfully then the program exits with a code 0, if it fails it exits with a code 1. Simple!!
Lies, Damn Lies, and Benchmarks….
I get quite frustrated with benchmarks because they are very hard to perform properly, and even when you do them properly its very hard to get any useful data from them.
Its all very well knowing that a web server can do 4,000 connections per second, but what we really want to know is something along the lines of:
How many shoppers at my ecommerce site can one web server handle IF:
- 200 users are doing free text searches
- 100 users are in the HTTPS shopping basket
- 500 users are just browsing
- 2 hackers are trying to get in
- & 1 proxy server is spooling 10,000 connections to cache the site
Anyway after getting hassled by yet another customer for a Benchmark on our EC2 VA load balancing appliance I thought I’d take a quick crack at it:
As a follow on to my previous blog, its easier to get Apache to log client IP addresses utilizing X-Forwarded-For headers than it is using IIS. By default, the logs do not record source IP addresses for clients but this is very easy to change using the LogFormat directive in the httpd.conf file as explained below.
So, you’re using IIS and you want to track your clients by IP address in your IIS logs. Unfortunately, out of the tin, this is not directly supported. The X-Forwarded-For (XFF) HTTP header is an industry standard method to find the IP address of a client machine that is connecting to your web server via an HTTP proxy, load balancer etc. Fortunately, depending on the version of IIS being used, there are a number of ways to enable this.
OK, Before the flames start let me state the usual caveat, “GSLBs don’t ALWAYS suck, just most of the time”.
Here at Loadbalancer.org we have toyed with the idea of selling a GSLB (as most of our competitors do), it wouldn’t take long… to hack a decent PowerDNS interface onto one of our appliances…
But every time we look at how it would work, we keep coming back to the fact that it doesn’t work at all (or at least not as the customer would expect).
Let me continue this rant by describing what customers probably want and then move onto what GSLBs actually do… and then suggest some simple alternatives: