It's 2013 folks. The 1U web server blueprint for building out a medium to large scale web property is old...and busted. 1U servers simply don't give you enough throughput or cores to really scale. We all know the deal. PHP blows up. Your rails apps use all the memory. Some cron job uses all the disk I/O. Oh well, just get more 1U servers. They're cheap, right? Wrong. They're actually very expensive for how they're actually utilized. They're expensive in terms of throughput, rack space occupied, and especially power. Yep. Your new Dell R620 servers might be doing great at 75-90 watts of consumption. Or so you thought. That's a ton of power compared to the alternatives.

One alternative: virtualization on bigger servers. It's a popular approach for those who listen too much to their EMC salesmen. I guess virtualization is OK if you're a lazy sysadmin, but the applications are usually limited to multi-tenancy or test/QA setups. To get really good power efficiency, you also need to buy really beefy boxes which are usually over-built and very costly in terms of SPECint/$.

Another option is to stop running your own hardware and only pay for what you need in the "cloud." Definitely an option if you don't use much storage nor have much traffic in terms of bandwidth. Then again, you're probably not running a popular site if both of those apply.

The option we chose is to go smaller than 1U. No, not blades. Micro servers. There are a few vendors selling microservers (Dell, Supermicro, HP), but we settled on the Dell C5000 chassis running C5220 Intel blades. These things are awesome. Seriously. I cannot emphasize how great these servers are. I've posted some photos and a quick video review below. I'm not going into the detailed specs since dell.com can do that -- but here's the gist: single socket, 4 DIMMs, 4 2.5" drives or 2 3.5" drives. 20-50 watts of power. 12 sleds in 3U. Bam

We've got more than a few of these chassis running our web front ends, and I think it's safe to say that we're done with 1U web servers. You should be, too.

Since some people only put their faith in numbers, here's a plot showing average power consumption between an R610 and a C5220 on the same web tier we run.

One thing to note is that the chassis only runs on 200V and up. You're SOL if you run 110V circuits. Then again, what are you running 110V for? Here's the video.

Like us, you're probably run an "ops box" or two in PoPs to handle some things like DNS, LDAP, Puppet, distributing code, etc. One of the things have our ops boxes do is create IPSec tunnel endpoints back to the mothership. We don't push much traffic through the tunnel -- just little things like puppet files, code updates, internal DNS, etc. We're not using these tunnels for our core business, just the ancillary stuff.

We decided to deploy one of these boxes on CentOS 6 and get with the new program. We've been running CentOS 5.x for a while and have been pretty pleased with it overall. However, sometimes you gotta run the new distros to get the new things (or just to get the server to boot). So we went with CentOS 6 for one ops box and started to work through some things. Once we thought we had it all up and running, we enabled the IPSec tunnel using racoon (gotta build your own)...then all hell broke loose.

Not initially, but over the course of a day, things started to stall on the box. Puppet runs hiccuped. NFS requests timed out. Things were getting bad. Looking into the matter, I noticed that the power governor was causing some trouble. I also found this bug to correlate our findings (https://bugzilla.kernel.org/show_bug.cgi?id=42981). Basically, don't run ACPI with any kind of power governors on RHEL/CentOS 6. The problem can occur when 2 or more cores are told to idle. It doesn't take 16 or 32 like in the bug report. Hopefully this will get backported to the RHEL/CentOS kernels, but in the meantime just build your own out of the latest.

Having found and dealt with that problem, we thought we were in the clear. Ends up that it was just the tip of the iceberg. The bigger problem is that IPSec is just plain busted in these kernels. It still encrypts and authenticates everything OK, but the processing of the incoming and outgoing datagrams is painfully slow. Every time that iptables, VPN, or some other kernel networking module needs to do something, you'll see ksoftirqd run. ksoftirqd is the equivalent of services.exe on a Windows box. Lots of stuff runs through it, but it generally means "server kernel stuffs." When you see lots of these run, you've got a busy box. Once ksoftirqd hits 100%, your box is toast. It doesn't matter if you have 48 cores or 2 cores. Once you see it hit 100%, the box is a goner. Here's what I mean (8 core server):

I think you can see where we enabled IPSec on the graph. The ksoftirqd stuff is represented by the light blue area (system). We disabled the power stuff on Monday. We "fixed" the IPSec stuff on Wednesday night. What's the fix? It's really obtuse, but we think it's this:

echo 32768 > /proc/sys/net/ipv4/xfrm4_gc_thresh

You'll probably want to cat that value before you change it just in case things get worse. On our machines it was over 4 million. That seems broken for something that's supposed to be some sort of garbage collection threshold. I've seen some distros hardcode it at 262144. Some are at 32768. One guy said he got great IPSec performance setting it at as low as 100. Your mileage might vary (like it might break other things), so try experimenting with different values.

Here's a post that led me down this path: http://en.usenet.digipedia.org/thread/16263/17508/

Don't get me wrong and think that adding IPSec shouldn't create more work for your CPU's. It will. However, one can expect to send a few megabits of traffic without having the box fall over.

Or just scrap distro 6 altogether and just run RHEL/CentOS 5. It works fine there. Personally, I think we're just going to buy a bunch of SRX210's off of eBay.

Update: for those who wanted a longer term graph and didn't think the problem was fixed: