Surviving the Storms

by @Nick_Craver

A typical month at Stack Overflow

  • 1,221,560,439 Page Views
  • 5,127,732,306 HTTP Hits
  • 48,286,916,506,506 Bytes Sent
  • 3,202,505,376 CDN Hits
  • 54,400,000,000,000 CDN Bytes
  • 23,788,557,797 SQL Queries
  • 185,405,583,038 Redis Ops
  • 19.5ms Average Render Time

Priciples

Decide what's important to you

What's important to us

Keeping it stupid, simple

Going faster

...and maybe staying online

Real Life

Things Will Go Wrong

Suck it up

Fix it

Make it better

Small Storms

Dealing with matters of scale

The Problem

Seemingly Random Slow HTTP Responses

Start Simple

var sw = Stopwatch.StartNew();
doThing();
sw.Stop();

MiniProfiler

Timings

Dapper

var users = Current.DB.Query<User>(@"
Select * 
  From Users 
 Where Id In @ids", new { ids = userIds });

var mdUsers = Current.DB.Query<User, UserMetadata, User>(@"
Select u.*, umd.* 
  From Users u 
       Join UserMetadata umd On u.Id = umd.UserId 
 Where u.Id IN @ids",
  (u, umd) => { u.CachedMetadata = umd; return u; }, 
  new { ids = sanitizedIds }, 
  splitOn: "UserId");

Man-made Storms

Tag Engine DDoS

Denver Meetup - October, 2014

Big Storms

What We Planned For

Colo Move - January, 2013

Prepration

Setting Goals

Getting Ready

Starting Point

Lockout

The Problem

Decisions

The Plan v1

The Plan v1 v2

Improvements: Maintenance

Improvements: Maintenance

Improvements: Monitoring

Improvements: Monitoring

What Actually Happened

New York - October, 2012

Hurricane Sandy

Respect

Q&A

We love Q&A

Links