Symphony.

Recently, we've seen the Symphony site go offline in four separate occasions. For the past several years, the site has been hosted on Amazon's AWS. For the most part, the server environment had been very stable with only a handful of server hiccups all these years. The most recent downtime however, had been the most severe with a downtime of over 24 hours. We've since moved our home to a new environment and here's the low-down.

Symphony, by virtue of the project's age, had gathered enough "Google points" to make Symphony an attractive target for spammers and their bots. As many might have noticed, spam member registrations and posts have been on a steady increase. However, the way the server had been setup on AWS, we weren't really taking advantage of the cloud infrastructure. The site was on a single medium sized node connecting to a separate dedicated DB instance. The site wasn't load balanced and no special firewall rules (to my knowledge) or spam protection policies have been set in place. Granted, only until recently, we've not had the need to change this setup.

It's been our plan for the past couple years to scrap the existing setup and start afresh when the new Symphony website is ready. In our ideal setup, we wanted to enforce the following requirements:

• The site needs to have an access control policy that serves the organisational structure ideal for our project. - There needs to be an easy way for contributing site admins to control and maintain the server.
• The site needs to accomodate for the dynamic nature of Symphony network sites.
• The site needs to have proper solutions to fight spam.

Stephen (bauhouse) and I had a few meetings in the past few months about setting up the new server environment. It was to be a development sandbox for him to work on the new site, in addition to his collaboration with other members on the docs site. The provision to set up the server kept being pushed back (for that I am sorry, Stephen). However, the recent server downtime had forcibly expedited our migration plans. So last Thursday, Brendan (Brendo) volunteered to help build the new server architecture with me.

The site is now hosted on Rackspace Cloud, with the following:

• 1 x Load Balancer
• 2 x Application Servers (512MB RAM / 20GB Storage)
• 1 x DB Server (1GB RAM / 3GB Storage)
• Running on Ubuntu 12.04 (Precise Pangolin)
• Load balanced master/slave DB setup (coming soon)

We've had a two incidents since we migrated over to Rackspace earlier this week. The first was caused by a VM level crash, which caused our DB instance to be unresponsive. Calls to reboot the instance over the control panel was ineffective and we had to contact Rackspace technical staff to issue a restart on the VM process, on their end. Our second incident was caused by an overload of the server due to spambots crawling the site (we had a string of calls to /wp-login/ on our access logs). We had yet to set up the load balancer or review our spam protection policy when spams started to flood in. Since then, we've added the application server load balancer, added policy to prevent too many simultaneous connections from a single IP and also added CloudFlare to help prescreen any naughty hawkers.

So far, our new server environment seems to be handling the site quite nicely. All of the server nodes have monitoring; this is an active process that sits on each node that makes calls back to Rackspace HQ, rather than a HTTP ping-based monitoring. We'll definitely be keeping a close eye on things and make sure everything is stable before we start introducing new server nodes for our Symphony Network sites.

In the mean time, we're calling out for more admins to help us maintain the site. Ideally, we're looking for 2 additional admins from different locations to cover the full 24-hour spectrum.