w output during caldot effect

SlashCal-dotted

closeThe world moved on since this content was published 2 years 3 months 26 days ago. Exercise diligent care when following any instructions and see opinions in the time they were written. If you must have an updated version, please ask kindly through the contact page.

The cause

Yesterday, I published a new post announcing my immediate availability. After tweeting that, I got a couple of very flattering retweets (thanks guys! Really!), which lead to the first couple of visitors to my blogpost.

Caldot tweets in reverse chronological order

Caldot tweets in reverse chronological order

All fine and dandy so far, but then something completely unexpected happened. My awesome friend & mentor Cal Evans decided that just the tweet wasn’t enough, and he send out a couple of InMails via LinkedIn:

LinkedIn InMail from Cal Evans

LinkedIn InMail from Cal Evans

The effect

This caused quite a spike in traffic. More than I realized. So within 15 minutes after that, I got alerted that my website wasn’t so responsive any more. Better said: that it had come to a screeching halt. This caught me completely off-guard.

Given this is just my blog, I am hosting it on a 512MB VPS at LiNode. While I have tuned Apache a bit to be more efficient with resources, it is not exactly designed to be a load-balanced redundant high-traffic setup. After all, it’s just my blog; I’m usually getting around a 100 visitors per month. When I switched to this current WordPress setup, I did install the WP SuperCache plugin, and did some ab-testing on it. The tests with 50 concurrent users still had acceptable response times (for a personal blog), so I left it at that. But if I could handle 50 concurrent, then what the $favoriteProfanity was going on? I logged into ssh, and was greeted with the following:

w output during caldot effect

w output during caldot effect

No wonder the box was going slow. Okay, on to emergency maintenance to get it responsive again. First up: look into memory usage. Yup, just what I thought: box swapping like mad. So: add memory. Unfortunately, this required a reboot, so in the 15 minutes total the reboot took, I pretty much lost all traffic. However, the box was very responsive again, happily serving the few visitors that were late to the party or were patient enough to try again.

The aftermath

Of course, I immediately went on an investigation to figure out why this happened. I did not have analytics available yet, so all I could go on was the local stats. It quickly became apparent that Apache had been using up all resources, even beyond swapping capabilities. So, offloading traffic was the first concern. I do have another VPS in the same DC, which could happily serve as a static cache. The WP SuperCache plugin has a feature that allows you to define a CDN and it will automatically replace all registered links to /wp-content/ to a CDN link (note: I am using the word CDN here because that is what SuperCache calls it. It doesn’t care if you have a single static cache or an actual edging network.). I installed and configured varnish on the secondary VPS, which didn’t help a lot. After finally figuring out how to write a vcl that would actually cache something *and* not error out (man, the docs are scarce there!), it was actually caching static components. This offloaded all “don’t need Apache” requests, which eased the load even more, and my main webserver was primarily back to it’s usual idling.

Today, I could see the statistics, and was blown away. For a personal blogpost: ~600 hits in an hour, most of them concurrently… that is apparently what you get for getting Cal-dotted (note: itrebal coined the term first). The total viewcount (for the post alone) at the time of writing is at 848 completely served requests. (note: the math doesn’t add up if you compare pageviews with requests served. Keep in mind that Apache also served every single static file (and I have too many of those :$), and this also includes all pageviews for my resume, homepage, contact page, etc etc etc etc).

Caldotted - annotated

Caldotted - annotated

The credits

  • Skoop, Cal, Rafael, Martin, sir Grumpy, Neal, and everybody else who retweeted: Thanks a lot! You have no idea how much that little effort means
  • Cal Evans: My webserver has a severe grudge against you now, but I am most grateful. I owe you one, and that means one Jack
  • Itrebal: thanks for helping out when I needed it, and for inventing the term ;)

And finally…

All this traffic did generate some initial leads already. But, I haven’t been able to sign anything yet, so if you are (aware of someone) looking: I’m still available.

~RW

Filed Under: blog

Tags: , , , , , , , ,

Released: on Apr 03, 2012 under a Creative Commons Attribution-NoDerivs (CC-BY-ND) licenseCC-BY-ND

Comments (1)

Trackback URL | Comments RSS Feed

  1. Cal Evans says:

    Remi!
    Sorry about hosing your blog. :)
    You don’t owe me anything; this is what communities do to help each other.
    =C=