Organization the road to SSL

A question we often get asked at Stack Exchange is why and all our other domains aren’t served over SSL.  It’s a user request we see at least a few times a month asking to ensure their security and privacy.  So why haven’t we done it yet?  I wanted to address that here, it’s not that we don’t want to do it, it’s just a lot of work and we’re getting there.

So, what’s needed to move our network to SSL? Let’s make a quick list:

  • Third party content must support SSL:
    • Ads
    • Avatars
    • Facebook
    • Google Analytics
    • Inline Images & Videos
    • MathJax
    • Quantcast
  • Our side has to support SSL:
    • Our CDN
    • Our load balancers
    • The sites themselves
    • Websockets
    • Certificates (this one gets interesting)

Ok, so that doesn’t look so hard, what’s the big deal?  Let’s look at third party content first.  Note that with all of these items, they’re totally outside our control.  All we can do is ask for them to support SSL…but luckily we work with awesome people that are they’re helping us out.

    • Ads: We’re working with Adzerk to support SSL, this had to be done on their side and it’s ready for testing now.
    • Avatars: Gravatars and Imgur can support SSL – Gravatar is ready but where our images are hosted is not yet (we’re working on this).
    • Facebook: done.
    • Google Analytics: done.
    • Inline Images: we can’t include insecure content on the page…so that means turning our images to SSL when is ready.  For other domains images are embedded from we have to turn them into links, or solve via another approach.
    • MathJax: we currently use MathJax’s CDN for that content, but they don’t currently support SSL so we may have to host this on our CDN.
    • Quantcast: done – under another domain.

Now here’s where stuff gets fun, we have to support SSL on our side.  Let’s cover the easy parts first.  Our CDN has to support SSL.  Okay, it’s not cheap but it’s a problem we can buy away.  We only use for production content so a & * combo cert should cover us.  The CDN endpoint on our side (it’s a pull model) has to support SSL as well so it’s an encrypted handshake on both legs, but that’s minor traffic and easily doable.

With websockets we’ll just try and see when all dependencies are in place.  We don’t anticipate any particular problems there, and when it works it should work for more people.  Misconfigured or old proxies tend to interfere with HTTP websocket traffic they don’t understand, but those same proxies will just forward on the encrypted HTTPS traffic.

Of course, our load balancers have to support SSL as well, so let’s take a look at how that works in our infrastructure.  Whether our SSL setup is typical or not I have no idea, but here’s how we currently do SSL:

nginx network layout

HTTPS traffic goes to nginx on the load balancer machines and terminates there.  From there a plain HTTP request is made to HAProxy which delegates the request to whichever web server set it should go to.  The response then goes back along the same path.  So you have a secure connection all the way to us, but inside our internal network it’s transitioned to a regular HTTP request.

So what’s changing?  Logically we’re not changing much, we’re just switching from nginx to HAProxy for the terminator.  The request now goes all the way to HAProxy (specifically, a process only for SSL termination) then to the HTTP front-end to continue to a web server.  This is both a management-easing change (one install and config via puppet) and hope that HAProxy handles SSL more efficiently, since CPU load on the balancers is an unknown once we go full SSL.  An HAProxy instance (without crazy) only ties to a single core, but you can have many SSL processes doing termination all feeding to the single HTTP front-end process.  With this approach, the heavy load does scale out across our 12 physical core machines well…we hope.  If it doesn’t work then we need active/active load balancers, which is another project we’re working on just in case.

Now here’s the really fun part, certificates.  Let’s take a look a sample of domains we’d have to cover:


Ok so the top level domains are easy, a SAN cert which allows many domains on a single cert – we can sanely combine up to 100 here.  So what about all of our * domains? A wildcart cert, excellent we’re knocking these out like crazy. What about meta.* Damn. Can’t do that. You can’t have a wildcard of that form – at least not one supported by most major browsers, which means effectively it’s not an option.  Let’s see where these restrictions originate.

Section 3.1 of RFC 2818 is very open/ambiguous on wildcard usage, it states:

Names may contain the wildcard character * which is considered to match any single domain name component or component fragment. E.g., * matches but not f*.com matches but not

It doesn’t really disallow meta.* or *.*  So far so good…then some jerk tried to make a certificate for *.com which obviously wasn’t good, so that was revoked and disallowed.  So what happened? Some other jerk went and tried *.*.com.  Well, that ruined it for everyone.  Thanks, jerks.

The rules were further clarified in Section 6.4.3 of RFC 6125 which says (emphasis mine):

The client SHOULD NOT attempt to match a presented identifier in which the wildcard character comprises a label other than the left-most label (e.g., do not match bar.*

This means no *.* or meta.*  Enough major browsers conform to this RFC that it’s a non-option.  So what do we do?  We thought of a few approaches.  We would prefer not to change domains for our content, so the first thought was setting up an automated operation to install new entries on a SAN cert for each new meta created.  As we worked though this option, we found several major problems:

  • We are limited to approximately 100 entries per SAN cert, so every new 100 metas means another IP allocation on our load balancer. (This appears to be an industry restriction due to abuse, rather than a technical limitation)
  • The IP usage issues would be multiplied as we move to active/active load balancers, draining our allocated IP ranges faster and putting an even shorter lifetime on this solution.
  • It delays site launches, due to waiting on certificates to be requested, generated, merge, received and installed.
  • Every site launch has to have a DNS entry for the meta, this exposes us to additional risk of DNS issues and complicates data center failover.
  • We have to build an entire system to support this, rotating through certificates a bit rube goldberg style, installing into HAProxy, writing to BIND, etc.

So what do we do?  We’re not 100% decided yet – we’re still researching and talking with people.  We may have to do the above.  The alternatives would be to shift child meta domains to be on the same level as * or under a common * for which we can get a wildcard.  If we shift domains, we have to put in redirects and update URLs in already posted content to point to the new place (to save users a redirect).  Also changing the domain to not be a child means setting a child cookie on the parent domain that is shared down is no longer an option – so the login model has to change there but still be as transparent and easy as possible.

Now let’s say we do all of that and it all works, what happens to our google rank when we start sending everyone to HTTPS, including crawlers?  We don’t know, and it’s a little scary…so we’ll test the best we can with a slow rollout.  Hopefully, it won’t matter. We have to send everything to SSL because otherwise logged-in users would get a redirect every time after clicking an http:// link from google…that needs to be an https:// from the search results.

Some of those simple items above aren’t so simple, especially changing child meta logins and building cert grabbing/installing system.  So is that why we aren’t doing it?  Of course not, we love pain – so we are actively working on SSL now that some of our third party content providers are ready.  We’re not ignoring the request for enhanced security and privacy while using our network, it’s just not as simple as many people seem to think it is at first glance – not when you’re dealing with our domain variety.  We’ll be working on it over the next 6-8 weeks.

Stack Exchange: Planning for failure, part 1

This will be the first in an series of interspersed posts about how our backup/secondary infrastructure is built and designed to work.

Stack Overflow started as the only site we had. That was over 3 years ago (August 2008) in a data center called PEAK Internet in Corvallis, Oregon.  Since them we’ve grown a lot, moved the primary network to New York, and added room to grow in the process.  A lot has changed since then in both locations, but much of the activity has stuck to the New York side of things.  The only services we’re currently running out of Oregon are chat and data explorer (so if you’re wondering why chat still runs during most outages, that’s why).

Back a few months ago we outgrew our CruiseControl.Net build system, changing over to TeamCity by JetBrains.  We did this for manageability, scalability, extensibility and because it’s just generally a better product (for our needs at least)  These build changes were pretty straightforward in the NY datacenter because we have a homogeneous web tier.  Our sysadmins insist that all the web servers be identical in configuration, and it pays off in many ways…such as when you change them all to a new build source. Once NY was converted, it was time to set our eyes on Oregon.  This was going to net us several benefits: a consistent build, a single URL (NY and OR CC.Net instances were in no way connected, the same version, etc.), and a single build system managing it all – including notifications, etc.

So what’s the problem?  Oregon is, for lack of a more precise description, a mess.  No one here forgets it’s where we started out, and the configuration there was just fine at the time, but as you grow things need to be in order.  We felt the time has come to do that organization.  Though some cleanup and naming conventions were applied when we joined OR to the new domain a few months ago, many things were all over the place.  Off the top of my head:

  • Web tier is on Windows 2k8, not 2k8 SP1
  • Web tier is not homogeneous
    • OR-WEB01 – Doesn’t exist, this became a DNS server a looooong time ago
    • OR-WEB02 – Chat,
    • OR-WEB03 – Stack Exchange Data Explorer, CC.Net primary build server
    • OR-WEB04 –
    • OR-WEB05 –, used to be a VM server (we can’t get this to uninstall, heh)
    • OR-WEB06 – Chat
  • The configuration looks nothing like NY
  • Automatic updates are a tad bit flaky
  • Missing several components compared to NY (such as physical redis, current & upcoming service boxes)

So we’re doing what any reasonable person would do.  NUKE. EVERYTHING. New hardware for the web tier and primary database server has already been ordered by our Sysadmin team (Kyle’s lead on this one) and racked by our own Geoff Dalgas.  Here’s the plan:

  • Nuke the web tier, format it all
  • Replace OR-DB01 with the new database server with plenty of space on 6x Intel 320 Series 300GB drives
  • Re-task 2 of the web tier as Linux load balancers running HAProxy (failover config)
  • Re-task the old OR-DB01 as a service box (upcoming posts on this – unused at the moment, but it has plenty of processing power and memory, so it fits)
  • Install 4 new web tier boxes as OR-WEB01 through OR-WEB04

Why all of this work just for Stack Exchange Chat & Data Explorer?  Because it’s not just for that.  Oregon is also our failover in case of catastrophic failure in New York.  We send backups of all databases there every night.  In a pinch, we want to switch DNS over and get Oregon up ASAP (probably in read-only mode though, until we’re sure NY can’t be recovered any time soon). The OR web tier will tentatively look something like this:

  • OR-WEB01 – Chat,
  • OR-WEB02 – Chat,
  • OR-WEB03 – Data Explorer
  • OR-WEB04 – Idle

Now that doesn’t look right, does it?  I said earlier that the web tier should be homogeneous and that’s true.  The above list is what’s effectively running on each server.  In reality (just like New York) they’ll have identical IIS configs, all running the same app pools.  The only difference is which ones get traffic for which sites via HAProxy.  Ones that don’t get traffic, let’s say OR-WEB04 for chat, simply won’t spin up that app pool. In addition to the above, each of the servers will be running everything else we have in New York, just not active/getting any traffic.  This includes things like every Q&A site in the network (including Stack Overflow),,,,,, etc.  All of these will be in a standby state of some sort…we’re working on the exact details.  In any case, it won’t be drastically different from the New York load balancing setup, which I’ll cover in detail in a future post.

Things will also get more interesting on the backup/restore side with the SQL 2012 move.  I’ll also do a follow-up post on our initial plans around the SQL upgrade in the coming weeks – we’re waiting on some info around new hardware in that department.

Copyright © 2015 . All rights reserved. Theme based on the Jarrah by Templates Next