Vegas Baby!

We made a group decision after a couple of beers at the local dog racing track (but thats another story) to put a trade show stand at Interop 2010 Vegas.
For a small investment of about $10K for a stand and $5K for hotel, flights and gambling expenses (are they tax deductible?). Of course we paid in advance and it didn’t occur to us that the whole of European airspace would be shut down to flights because of a little volcano in Iceland called Eyjafjallajoekull !

Given the disaster in the skies we didn’t think we would be seeing Vegas.. Luckily at the last minute the airspace opened up and Karen, Jake, Mark and I  packed rapidly and shot up to Gatwick where we discovered that the USA visa site had not accepted our details (probably thought we were terrorists)… The hotel we stayed at was just off the strip behind Hooters and MGM called the ‘Desert Rose Resort’ and unlike the Hotel Lily it was really very nice, big appartment style rooms, nice pool, ok breakfast and free alchol from 5-7 in the evening!

Mark and Jake shot off by helicopter at 6am to see the Canyon which the really enjoyed (got a little burnt though)…. We took the first few days fairly easy and enjoyed the strip without loosing too much money. Then came the show, it was at the Mandalay convention center and was slightly smaller than the Infosecurity show in London. We had a tiny little stand next to Coyote Point and F5, all the vendors (competitors or not) were very friendly and we had some interesting chats with people.However the Gigasys stand opposite had a guy doing the same presentation very loudly every 15 minutes which was slowly driving us insane.

Jake went to his first poker tournament and came 4th, the other players wanted to split the pot but Jake refused and immediately got kicked out next round…. He is now convinced that he is a natural so will no doubt end up broke …..

On the second day the beer / booth crawl gave us the opportunity to see how much alcohol we could fit on our small stand.. and after that we could not remember an awful lot apart from winning at craps (what on earth are the rules?) and roulette luckily eventually all of our bank cards were refused so we couldn’t loose any more money…

Day 3… Ouch….best not to talk about day 3.

Will we do it again? - Lets say no for now, and then maybe change our minds later? $15K is still a lot of Google ads.

Ps. Thanks very much to Karia Kaiser of OpenText who sent us some photos….

]]> http://blog.loadbalancer.org/google-killed-the-it-trade-show%e2%80%a6-part-ii/feed/ http://blog.loadbalancer.org/load-balancing-microsoft-office-communications-server-ocs-with-haproxy/ http://blog.loadbalancer.org/load-balancing-microsoft-office-communications-server-ocs-with-haproxy/#comments Tue, 23 Mar 2010 12:02:44 +0000 rob http://blog.loadbalancer.org/?p=322 Here at Loadbalancer.org we have recently started the certification process of our product with Microsoft Office Communications Server (OCS). We already have several customers doing this with our units in Direct Routing mode but with the new Loadbalancer.org - ENTERPRISE v6.8 you can do it with the Microsoft recommended SNAT mode. So how can you do this yourself for free with the open source load balancer HAProxy? Read on……

Right, couple of points first:
1) I’m assuming that you have installed at least version 1.4.1 of HAProxy, plenty of blogs to show you how to do that around….
2) You have already installed at least a pair of Microsoft OCS servers and know roughly what you want to achieve

The following ports need to be Load Balanced:
(source http://technet.microsoft.com/en-us/library/dd572362(office.13).aspx )

Ports Required

5060 : SIP communication over TCP.
5061 : SIP communication over TLS.
135 : To move users from a pool and other remote DCOM-based operations.
443 : HTTPS traffic to the pool URLs.
444 : Communication between the focus (Office Communications Server 2007 R2 component that manages conference state) and the conferencing servers.
5065 : SIP listening requests for Application Sharing.
5069 : Monitoring Server.
5071 : SIP listening requests for Response Group Service.
5072 : SIP listening requests for Conferencing Attendant.
5073 : SIP listening requests for Conferencing Announcement Server.
5074 : SIP listening requests for Outside Voice Control.
8404 : TLS (remoting over MTLS) listening for inter-server communications for Response Group Service.


So lets jump right to the configuration file (I will explain the important bits in a minute):


# HAProxy configuration file generated by load balancer appliance
global
#uid 99
#gid 99
daemon
stats socket /var/run/haproxy.stat mode 600 level admin
log /dev/log local4
maxconn 40000
ulimit-n 81000
pidfile /var/run/haproxy.pid
defaults
log global
mode http
contimeout 4000
clitimeout 1800000
srvtimeout 1800000
balance roundrobin
listen OCS_ALL_SERVICES 10.10.2.20:5061
bind 10.10.2.20:5060,10.10.2.20:5065
bind 10.10.2.20:5071,10.10.2.20:5072
bind 10.10.2.20:5073,10.10.2.20:5074
bind 10.10.2.20:5073,10.10.2.20:5074
bind 10.10.2.20:8404,10.10.2.20:444
bind 10.10.2.20:443,10.10.2.20:135
bind 10.10.2.20:5069

mode tcp
option persist
balance leastconn
stick-table type ip size 10240k expire 30m
stick on src
server OCS_Node1 10.10.2.4 weight 1 check port 5061 inter 2000 rise 2 fall 3
server OCS_Node2 10.10.2.5 weight 1 check port 5061 inter 2000 rise 2 fall 3
server backup 127.0.0.1:9081 backup
option redispatch
option abortonclose
maxconn 40000
log global
listen stats :7777
stats enable
stats uri /
option httpclose
stats auth loadbalancer:loadbalancer


The important bits are:

The new source IP table sticky functionality (expire after 30mins inactivity):

stick-table type ip size 10240k expire 30m
stick on src


Listen to every single required port on the same front end:

listen OCS_ALL_SERVICES 10.10.2.20:5061
bind 10.10.2.20:5060,10.10.2.20:5065

Send to your backends but DON’T SPECIFY A destination port (i.e. use the original destination port)

server OCS_Node1 10.10.2.4 weight 1 check port 5061

Set a 30 minute timeout for long TCP connections (required for SIP):

clitimeout 1800000
srvtimeout 1800000

And try to balance fairly evenly even though we have 30mins persistence set:

balance leastconn


Simple isn’t it….?

Any comments welcome,
Can this be split down into more than one cluster?

The current development version of HAProxy has made an important step forward in making this possible. Thanks to work by Exceliance, it now supports RDP Cookies, offering a solution to the persistence problem.

We have been testing the latest development release of HAProxy, 1.4-dev4, on a loadbalancer.org Enterprise R16 device. The real servers were two Windows Server 2008 machines, with identical test users set up on both.

We settled upon the following HAProxy configuration (RDP Cookies):

   defaults
        clitimeout 1h
        srvtimeout 1h
   listen VIP1 192.168.0.10:3389
        mode tcp
        tcp-request inspect-delay 5s
        tcp-request content accept if RDP_COOKIE
        persist rdp-cookie
        balance rdp-cookie
        option tcpka
        option tcplog
        server Win2k8-1 192.168.0.11:3389 weight 1 check   inter 2000 rise 2 fall 3
        server Win2k8-2 192.168.0.12:3389 weight 1 check   inter 2000 rise 2 fall 3
        option redispatch

Note that this is only a fragment of the haproxy.cfg file, showing the relevant options.

The load balancer’s Virtual IP is set to 192.168.0.10, listening on port 3389 for RDP. The two real servers are on 192.168.0.11 and 192.168.0.12, in the same subnet as the Virtual IP.

The two new configuration directives are persist rdp-cookie and balance rdp-cookie. These instruct HAProxy to inspect the incoming RDP connection for a cookie; if one is found, it is used to persistently direct the connection to the correct real server. The two tcp-request lines help to ensure that HAProxy sees the cookie on the initial request.

The only other tweak needed is to increase the clitimeout and srvtimeout values to one hour. In testing, this was found to be necessary to keep idle RDP sessions established.

Testing involved making multiple connections with different usernames, from varying IP addresses, using both Windows XP Professional and Linux clients. Sessions were disconnected and reconnected, and real servers removed from the cluster and re-inserted.

We found that, once a user had established a session with a particular real server, that user consistently reconnected to the correct server if it was available. When we removed and re-inserted servers, existing sessions were unaffected. After a simulated server failure, users could start a session on the remaining server.

When a failed server was brought back on-line, users that had been connected to that server would reconnect to it again - even if they had started a new session on the other server in the meantime. This may not be what you want, and requires further testing.

With client and server time-outs set to one hour, we were able to leave idle sessions running for 16 hours without problems.

For more information on the new configuration options, see the development version of HAProxy’s Configuration Manual.

NB. For some daft reason Microsoft restricted the login cookie in RDP to 9 characters! Now as the domain is usually listed first (mydomain/myusername) the first 9 characters may always be the same and RDP cookie session persistence wont work. Two work arounds for this are either reduce the length of your domain name (ouch!) OR use the myusername@mydomain format when you log in….

So what about Microsoft Connection Broker (session directory or whatever they call it) ?

A simple one line change in your HAProxy configuration (RDP Connection Broker):

#Balance rdp-cookie ->        balance leastconn
i.e.

   defaults
        clitimeout 1h
        srvtimeout 1h
   listen VIP1 192.168.0.10:3389
        mode tcp
        tcp-request inspect-delay 5s
        tcp-request content accept if RDP_COOKIE
        persist rdp-cookie
        balance leastconn
        option tcpka
        option tcplog
        server Win2k8-1 192.168.0.11:3389 weight 1 check   inter 2000 rise 2 fall 3
        server Win2k8-2 192.168.0.12:3389 weight 1 check   inter 2000 rise 2 fall 3
        option redispatch

Note that this is only a fragment of the haproxy.cfg file, showing the relevant options.

One of our illustrious members of staff had earned far too much money by shorting banks on the stock market so he was leaving for Hong Kong and we needed to double the event up as a leaving do. We all trudged up to London in Karen’s 7 seater (she has one child!?) averaging 7 miles per gallon. Now being penny wise and pound foolish Karen had booked us into the Hotel Lily, which while very cheap was disgusting, dirty and down right dangerous. After thinking briefly about finding another Hotel , Jake had the brain wave that we could just get drunk and then it would be easy to sleep!

So several beers later James ended up with a dog (literally):

Why the owner thought he was trustworthy I don’t know… So to cut a long story short, we did sleep briefly… and then on to the show!

Yes I known we look a motley crew, but at least I managed to stay out of the photo! As expected the show was heavy going, existing customers were nice to chat to, most of the other vendors had a good sense of humour (Coyote didn’t seem to know what humour was…).  After lunch we started strugling after way too many coffees.. But soon it was all over and the free beer and wine came out!

I managed to make the show organiser want to commit hari kari when I made my prediction “That all the vendors would be gone in 3 years as Google ads were far more cost effective…”, just as the technical press has been detroyed by Google (shame…..did anyone mourn the loss?).

So more alcohol and a great Thai meal that cost more than our entire  hotel bill for 2 nights… a little more sleep…

And then the really hard day… I think a picture tells a thousand words:

I manged to make it to the stand before the show opened.. But the rest of the staff couldn’t make it in for at least another hour….unsurprisingly the day did not go well and we couldn’t even be bothered to fend off the vultures steeling USB sticks and demo CDs, so we decided to make them famous by taking photos of them!

I picked up load-balancing.org fairly quickly… but I really wanted loadbalancing.org and a domain squatter had that (sigh..) still I thought what the heck and made an enquiry and they said $2,000… The sales process was interesting as the vendor wanted to use www.Escrow.com, quite a clever little site as they held on to my money until the domain transfer was complete. It was a bit confusing as they had a PayPal button that didn’t work but it did give assurance that you were not buying from a conman.

Anyway slightly poorer now, I looked at trying to create possibly the worst looking web page ever! Concentrating on factual (ish) content and links to load balancing instead. Now the previous owner had a search result position of bottom of page 4 for a load balancing holding page with some click revenue streams, Lets see how high I can get this great site on Google as a load balancing FAQ.
I’ve tried not to make it a blatant advert like some of our competitors do, try searching for hardware load balancer and the top link is JetNexus’s little Google search term marvel….

If anyone thinks my Load Balancing site is the worst page ever… Let me know… And do you think I am morally corrupt?
Ps. Comments welcome, and I’m happy to change the content of the page if anyone has suggestions.

Like my previous blog entry I will endeavour  explain how to recover from master failure but this time with version 5.10. (NOTE: once again this will only work with a simple configuration anything more complicated than simple direct routing its best to contact support@loadbalancer.org)

Hopefully you have your backup files – getharesources.php, getloadbalancercf.php and getrealip.php.

If not via the web interface they are located under “Maintenance” - “Disaster Recovery Options”.

• Disconnect the network and the Heartbeat (Serial) cable from the master.

• Connect a Monitor and a Keyboard.

• Recover the Loadbalancer V5.10 image to a CF card or DOM depending on your systems configuration from the ISO that is available via the website. If your master has failed because the CF card or DOM has failed please contact support who will be able to issue you with a new one. (support@loadbalancer.org)

When you have a working Loadbalancer.

• Copy the configuration files and lbrecoverv510 (lbrecoverv510 can be obtained from our website http://www.loadbalancer.org/download/recoveryscripts/ right click on lbrecoverv510 and select save link as) to a USB Stick. (be sure to keep the configuration files and the recovery script in the same folder)

• Insert the USB Stick into the master.

• Mount the USB Stick if your not sure how to do this see below:

Enter the command: “fdisk -l”.

This should give you a list of Drives attached to your machine from that you should be able to work out which one it is, if you only have one drive either a CF card or DOM then its probably /dev/sda1

To mount the device enter the below command:

“mount /dev/sda1 /mnt”.

then you can “cd /mnt” and a simple “ls” should show you your files.

• To run the recovery type “php lbrecoverv510”, then you will be prompted “Are you sure you want to recover the master configuration [y/n]” press “y” which will start the recovery process. It will then copy the files across, when completed you will be prompted to reboot the machine.

• After the machine has fully rebooted reconnect the Heartbeat (Serial) cable and the network cable.

Your cluster should be restored. To confirm this on the web interface click “Reports” then “Current Connections” if you do this on both the master and the slave you should see all connections being routed through the master and none to the slave.

Some times during this process the heartbeat may need to be restarted to restore full cluster functionality, this is a simple process on the web interface go to “Edit Configuration” then “Restart Heartbeat”.

Hopefully you have handy the backup configuration file – lb_config.xml

If you don’t have a copy already and your looking at this article because your organised and planning for disaster, NOW might be a good time to go and get one.

Log into your master Loadbalancer and select “Maintenance” and click on “Disaster Recovery” then “Download XML Configuration file” and keep the file that is downloaded somewhere safe.

If your reading this and your cluster has failed and you don’t have a backup there are a few ways of recovering the file.

The simplest way is take a copy of the lb_config.xml from the slave machine and edit the following lines: (NOTE: if your setup is more complicated and uses firewall marks or SSL or your not sure what other changes to make its probably best to contact support support@loadbalancer.org)

<hostname>lbslave</hostname> to <hostname>lbmaster</hostname>

<slave></slave> to <slave>192.168.2.81</slave> (replace IP address with the address of your slave machine)

<eth0>

<ip>192.168.2.80</ip> to <eth0>

<ip>192.168.2.81</ip> so change the IP address from the address of your slave machine to the address of the master.

That should be it, click save and we can carry on with recovering the master.

• Shut-down the master if possible, if its already off you can skip this step.

• Disconnect the Heartbeat (Serial) cable and the network cable.

• Repair what ever problems you are having with the master.

• Connect a mouse, monitor and keyboard do the server.

• Restore the master from the Loadbalancer image visit -

http://blog.loadbalancer.org/how-to-recover-your-load-balancers-to-v65-via-usb-stick/ for instructions on how to restore from an image, if you need help contact support@loadbalancer.org

• During the aforementioned restore at no point reconnect the cables!

• Log onto the machine with:

User name: root

Password: loadbalancer and at the terminal stop the heartbeat service by issuing the command : “service heartbeat stop”

• Load your lb_config.xml and lbrecoverv66 (which can be found at http://www.loadbalancer.org/download/recoveryscripts/ right click on lbrecoverv66 and click save link as) onto a USB stick

• Insert the USB stick into the server

• Enter the command: “fdisk -l”

This should give you a list of Drives attached to your machine from that you should be able to work out which one it is, if you only have one drive in your server then its probably /dev/sdb1

• To mount the device enter the below command:

“mount /dev/sdb1 /mnt”

then you can “cd /mnt” and a simple “ls” should show you your files.

• Enter the command: “php lbrecoverv66” at the terminal to load the recovery script and wait for 2 mins then shutdown the machine by entering: “shutdown -h 0”

• Reconnect the Heartbeat (Serial) cable and the network cable

• Turn the machine back on

• Restart the Heartbeat on the master by logging into the web interface on your recovered master and click on “Maintenance” then “Restart Heartbeat”

Wait a few minutes and your cluster should be restored!

Recovering from Slave failure

If, instead, your Slave loadbalancer has failed, here’s how to recover it…

Again, the easiest way is if you have a backup of the lb_config.xml from the Slave. If you haven’t backed that up as well, you might like to do it now.

If the Slave has failed and you do not have a backup of its config, below is a procedure for recovering from the Master.

1. Download the Master’s config, using Maintenance > Disaster Recovery > Download XML configuration file.
2. In a text editor, make the following changes to the lb_config.xml :
   1. In the <physical> <network> section,
      1. Change <hostname> to lbslave.
      2. Remove any IP address in the <slave> tag.
      3. Change <fullsync> to off.
   2. In the <physical> <rip> section, change the IP addresses for eth0 and eth1 as necessary.
3. Save your modifications to lb_config.xml
4. If the Slave loadbalancer is still connected to the network and the serial link, disconnect it.
5. Fix the slave server.
6. With the slave still disconnected from the network and serial link, restore the Loadbalancer image, using the instructions at http://blog.loadbalancer.org/how-to-recover-your-load-balancers-to-v65-via-usb-stick/
7. When the install is complete, log in to the slave with username root and password loadbalancer.
8. At the root prompt, run service heartbeat stop to stop the heartbeat service.
9. Transfer the master’s config and recovery script, from http://www.loadbalancer.org/download/recoveryscripts/, to the new slave on a USB key.
10. Plug the USB key into the slave. Run the command fdisk -l to discover the device that the system has allocated to the USB key. It will usually be /dev/sdb1
11. Mount the USB key using mount <device> /mnt, and change to the directory with the config and recovery script.
12. Start the recovery by running php lbrecoverv66. When the prompt returns, wait a couple of minutes then shut down the slave server with shutdown -h 0
13. Reconnect the slave to the network and the serial link, and restart the server.
14. When the slave has finished booting, log on the master’s web interface and restart the heartbeat service using Maintenance > Restart Heartbeat.

Your cluster will now be restored.

Many thanks to Krisztián Ivancsó  for working on the TPROXY patch for Pound for us, we can finally do this!

First of all lets explain what we are trying to achieve. We have clients coming in from the external subnet 10.0.0.x with both HTTP and HTTPS requests to our virtual server (10.0.0.142), The HTTPS traffic is terminated by pound and sent to an HAProxy backend (10.0.0.142:81) which in turn inserts session cookies and passes the traffic to the backend servers (192.168.2.x).

The HTTP traffic hits a seperate HAProxy instance on (10.0.0.142:80) where cookies are inserted and traffic passed to the backend servers (192.168.2.x). Why a second instance? Unfortunately it is not currently possible to have TPROXY running for Pound and HAProxy using the same IP and port combination (which makes sense if you think about it).

Just a reminder - “why are we doing all of this?”, because we want to clients source IP address to be presented to the backend server even though the traffic is coming through a proxy!

So lets assume that you have already set up HAProxy in TPROXY mode for full transparency.

First of all we need to grab a recent copy of Pound and the TPROXY patch, configure, make & install etc.

wget http://www.loadbalancer.org/download/PoundSSL-Tproxy/Pound-2.4.5.tgz
tar -xvf Pound-2.4.5.tgz
cd Pound-2.4.5
wget http://www.loadbalancer.org/download/PoundSSL-Tproxy/poundtp-2.4.5.diff


patch -p1 < poundtp-2.4.5.diff
./configure
make TPROXY=1
make install


Make sure the firewall rules are set correctly for standard TPROXY (HAProxy binds to these automatically), as in the previous blog.


# Standard rules for TPROXY setup
#!/bin/bash
iptables -t mangle -N DIVERT
iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
iptables -t mangle -A DIVERT -j MARK --set-mark 111
iptables -t mangle -A DIVERT -j ACCEPT
ip rule add fwmark 111 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100


Add the rules to make sure that local Pound -> HAProxy traffic is transparent:

# Rules to match PoundSSL -> Haproxy backend
iptables -t mangle -A OUTPUT -s 10.0.0.142 -p tcp –sport 81 -j DIVERT
iptables -t mangle -A OUTPUT -d 10.0.0.142 -p tcp –dport 81 -j DIVERT

Then configure HAProxy making sure you have two instances with the same real servers, one for HTP traffic and one for the transparent terminated HTTPS traffic:


# HAProxy configuration file generated by load balancer appliance
global
#uid 99
#gid 99
daemon
stats socket /var/run/haproxy.stat mode 600
maxconn 40000
ulimit-n 81000
pidfile /var/run/haproxy.pid
defaults
mode    http
contimeout    4000
clitimeout    42000
srvtimeout    43000
balance    roundrobin
listen    VIP_Name 10.0.0.142:80
mode    http
option    forwardfor
source 0.0.0.0 usesrc clientip
cookie    SERVERID insert nocache indirect
server RIP_Name 192.168.2.98:80 weight 1 cookie RIP_Name check  inter 2000 rise 2 fall 3
server    backup 127.0.0.1:80 backup  source 0.0.0.0
option redispatch
option abortonclose
maxconn 40000
listen    SSL_Backend 10.0.0.142:81
mode    http
option    forwardfor
source 0.0.0.0 usesrc clientip
cookie    SERVERID insert nocache indirect
server RIP_Name 192.168.2.98:80 weight 1 cookie RIP_Name check  inter 2000 rise 2 fall 3
server    backup 127.0.0.1:81 backup  source 0.0.0.0
option redispatch
option abortonclose
maxconn 40000


Make sure that when you do the Pound configuration that Pound is running as root:

# Pound2 configuration file generated by load balancer appliance
#User    "nobody"
#Group    "nobody"
LogLevel    0
Client     30
TimeOut     60
ListenHTTPS
Address 10.0.0.142
Port 443
Cert "/usr/local/etc/server1.pem"
Service
BackEnd
Address 10.0.0.142
Port 81
TProxy 1
End

End

End

Obviously make sure that:

• The client is in the 10.0.0.x network.
• The load balancer has IPs in both networks.
• The backend server uses the load balancer as the default gateway.
• You have your fingers crossed :-).

I put this together fairly quickly, so please let me know if I have missed anything. With any luck it should make it into the v6.7 appliance fairly soon.. available as a manual update for now until v6.7 is ready.

Since writting this Ivan has added three new features (I’ve updated the dowload link to have this latest version).

Three new features:
- You can use patched Pound without root privileges
- It adds a new global TProxy option which disables/enables TProxy globally. (If this global option is set, then pound will preserve needed privileges to work as transparent proxy as a simple user. If it’s not set pound will work as unpatched version and will not preserve additional rights.)
- Let tproxy to set a random port for source IP when Pound connects to backend. It means original source port is not preserved in communication with backend.
Example config:


User    "ivan"
Group   "ivan"

ListenHTTP
Address 192.168.254.22
Port    81
End
Service
BackEnd
Address 192.168.254.22
Port    80
TProxy 1
End
End


Enter Amazon Web Services. You’ve probably been living under a rock if you haven’t heard of AWS - it’s the new vogue of the IP world - “Cloud Computing”. Amazon certainly aren’t the only proponents; there’s Google App Engine, Rackspace’s “Mosso“, 3tera, and probably numerous others. So what’s so great about the cloud? I guess the main advantage is flexibility. Essentially we’re talking about Virtualization, so if you want to launch a clone of your current server no one has to haul a physical server over to your rack to replicate your data. In the cloud the capacity is already there; you just have to concern yourself with creating enough demand. But scalability isn’t our main concern - we “just” want HA and decent global delivery of our website’s content, and for this we chose to leverage Amazon’s EC2, S3 and CloudFront services.

I think of these three services as the following:

• EC2 is your server - it allows you to provide dynamic content via whatever platform you prefer (e.g. LAMP). Your server can be in Europe, or the US.
• S3 is just storage. You can host your static files on here, but there is no logic layer. Give the requester what they asked for, and nothing else. It’s geographically redundant, and pretty cheap per GB.
• CloudFront - a global CDN for your S3 files, serving files via US, Europe, Hong Kong, or Japan, depending on the location of the request.

All of Amazon’s services are accessed through their API, which exists in command-line form with a Java backend, and also as a web service, via SOAP requests, or straightforward HTTP(S). Other APIs which leverage the web service have cropped up, written in PHP, Perl, Python, C++ etc. For EC2 our favourite tool is ElasticFox, a pretty robust and feature-rich Firefox plugin.

So how do they all fit together? Well we have some dynamic content on our website, such as this blog, and we need a scripting language to send emails. So clearly we need a server, rather than being able to dump all our web files on S3, and serve them through CloudFront. There are a few features that make the EC2 platform attractive for use as a web server:

Elastic IPs - These are permanent public IPs assigned to your account. If your server (an instance in AWS lingo) crashes or becomes unavailable for any reason, you can reassign your elastic IP to another instance. Or launch a new instance and assign the elastic IP to it. So in theory you should never need to repoint your DNS entries to a different IP.

Elastic Block Store (EBS) - A similar principle to the elastic IP, but for storage devices. An instance has its normal storage device, but an EBS is more redundant and can be reattached to another instance. Essentially the EBS is a SAN in the cloud, whereas normal instance storage is vulnerable to individual disk failure. However, you can’t avoid storing your essential OS files on the instance’s disk, since networking must be up before you can mount your EBS. But you can store as much other stuff as you can on it: web files, databases, subversion repositories, config files, etc. So if your instance goes bang, you can very quickly launch another one and re-attach your EBS. For an extra layer of redundancy you can take snapshots of your EBS, which are stored on S3. RightScale have an excellent article regarding the ins and outs of the EBS.

System Images - So what about the rest of your files, the essential OS files called upon during boot? If your instance dies won’t you have to waste time recreating your config? The simple answer is no. Once you have an instance just the way you like it, you take an image of it and it transfer it to S3. You can register it as public or private. There is a huge number of free public images to choose from, ranging from all flavours of Linux, Windows, Solaris, DB2. Just pick a clean image, make your changes, and register your own image. In some cases, you may want to just mount an EBS and serve your content from there.

Straightforward Security Maintenance - when you launch an instance, by default none if its ports are open to the outside world. Opening a port simply involves a one line call to the AWS API. And obviously closing one is just as easy.

So how did we fit all this around our scenario? Well, currently AWS instances can only be located in the US or Europe. So it struck us that the optimal thing to do is host as much of your static content as possible on CloudFront (images, CSS, downloads etc.). So we’d store our blog database (and a couple of others) on an EBS, attached to an instance in Europe. The instance is running Ubuntu 8.04 and also serves our Subversion repository. We set up an elastic IP and pointed the www and blog A records to it. Then we set up some cron jobs to take snapshots of the EBS. If our instance fails we can relaunch another one and it will be up and running and serving the same content within 30 seconds.

But what about the Load Balancer I hear you ask? Can’t you ensure zero downtime and well distributed traffic amongst a cluster of web servers? Actually you can. AWS does include a load balancing facility, which essentially acts as a proxy, allowing you to forward traffic on a certain port to various servers in the same availability zone. This is a very sensible thing to do, but in the case of our website, is probably overkill. Plus the static content is effectively geographically load balanced via CloudFront. Although their load balancer is a valuable addition to the service, we do think it would be a massive improvement to be able to span multiple availability zones and even regions (i.e. Europe and US). In theory this is doable with HAProxy, and this is something we are experimenting with at the moment. Another downside is that the load balancing is only available in the US region at the moment.

All in all, we are happy with the level of availability our setup provides, and (touch wood) we have had no issues as yet. But for those hosting more critical services in the cloud, we suggest taking a look at the load balancing service, and for greater flexibility, running an instance with an HAProxy setup.

NB. This will only work on 64Bit hardware. All version 6 appliances are 64Bit. If you are running an older version this may still be possible depending on the hardware you are running on.

If you are running v5 and wish to determine whether your appliance is 64Bit then enter the following command:

grep flags /proc/cpuinfo

If lm (long mode) is present in the output then they are 64Bit and you can proceed. If not then your appliance is 32Bit and you are limited to the latest v5 software.

NB. Loadbalancer.org are continuing to develop and support v5.

The v6.6 image requires a high speed 4Gb IDE DOM / Flash. If you are already running v6 then you will already have this and should be able to simply re-image your current DOM / Flash. If you are upgrading from v5 you will need to purchase a 4Gb DOM / flash card and then use the following procedure to build it from any USB stick 1Gb or greater:

NB. If you are already running v6 then you can keep your current configuration by backing up the XML file on BOTH the master and the slave. This can then be uploaded to the new v6 appliance once the following steps have been completed.

The USB stick bootable version of the ISO file is here:
http://www.loadbalancer.org/download/v6.6g.iso

You can use UNetBootIn (Windows or Linux) to transfer the ISO onto a USB stick.

Then change the server BIOS to boot from the USB (stick must be plugged in at that stage).

First boot device (Removable)

Hard disk boot priority (USB)

When it boots choose the:
Default
image

Then at the command line do:

cd /etc/recovery/
./clone-dsk.sh

option 1.

Then

option 1.

Then Yes to all.

The image transfers onto any IDE HD or IDE DOM / Flash.

You now have a fully functioning v6.6 appliance!

NB. If you need to upload a previous configuration go to Maintenance > Disaster Recovery in the web interface to upload a backup XML file.