Edge 2014: Bypass Surgery - Akamai's Heartbleed Response Case Study

Bypass Surgery and Other Tales Brian Sniffen

©2014 AKAMAI | FASTER FORWARDTM

Akamai Security Research & Architecture

•  Crypto engineering expertise •  Technical backstop •  Product review •  Akamai Architecture Group seat •  Safety engineering •  Incident management


2014: The Year of Vulnerabilities

“You people in InfoSec have become the Product Managers!”

Yes, and we can’t wait to get out of that role.

•  Heartbleed •  INRIA-Prosecco Cookies •  Shellshock

2009 2010 2011 2012 2013 2014

1 0 0 1 1 3+

2008

1


Akamai Incident Management Principles

•  Technical Incident Manager (TIM) coordinates all work •  Incident Executive communicates upwards, shields TIM •  GSS Business Incident Leads manage customer comms

•  No single-point harm can cause a Severity 1 Incident •  A hot meal and 6 hours sleep fix more problems than an all-nighter •  If the TIM becomes an SME, get a new TIM


We tell ourselves who we wish to be:

•  Akamai says thank you. •  Akamai doesn't respond to name calling,

but does respond to the useful technical content. •  Akamai presents itself as a responsible and respected member

of the Internet community. •  Akamai will use this incident to improve both its own security

and the general security of the Internet. •  Akamai can laugh at itself.


Heartbleed mail

From: Brian Sniffen Date: 7 Apr 2014 21:34:08 Subject: Sev 1: Heartbleed Will, I'm contacting you because you're the Ghost SME on call. I'm looking for evidence to refute the statement: "The Heartbleed bug can’t extract long-term customer private keys from a Ghost; we put them only in a wired, mmaped page way lower on the stack."


Heartbleed Timeline

April 1: Notice; QA begins April 4: last Akamai Deployed Systems patched April 7, 1pm: Public Notice April 7, 6pm: What did we leak? April 8, 1am: Working exploit in Akamai lab April 9–12: Hastily publish Akamai Secure Allocator April 13, 11pm: Begin cert rotations & revocations


“Don’t worry, we restored the old functionality”

April 14, 6am: “Why is this message in the old log format?”

A “Manual Change” had restored an old version.


The Akamai Secure Memory Allocator

•  1999 code •  One author, three redactors •  State machine inspired by CLOS “advice” system

Turns out it works fine

Code Secure Heap mmap’d file Long-term

Allocations Heap


Cert Revocation Progress

21 Apr

28 Apr

5 May

12 May

19 May

26 May

2 Jun

9 Jun

70% 90% 95%

# of

cer

ts

16 Jun

23 Jun

30 Jun


Learning from Heartbleed

Nobody’s paying for OpenSSL! Practice in mass, fast, patching Practice in releasing helpful patches Simplicity promotes safety.


Shellshock Timeline

Sep 23, 12pm: Notice from Florian Weimer, Debian Security Sep 23, 9pm: Manual change: replace bash wish dash;

Patches started Sep 24, 5am: WAF rule in place

SSH command= systems made safe Sep 24, 12pm: Public Notice Sep 25: “Kobrin Patch” to remove dangerous feature Sep 28: bash mostly replaced with dash on deployed network


Bash patches

Pre-release: •  Embargoed patch: 195 lines, 7 files (1/6 CVEs) •  Kobrin patch: 2 lines, 1 file (6/6 CVEs)

Post-release: •  NetBSD patch: 3 lines, 2 files (6/6 CVEs) •  Fixed patch: 164 lines, 11 files (6/6 CVEs) •  Apple patch: unpublished (exposure unclear)


SSH command= limits

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFnHfYyS5onAN72oFpaopm+/yKbRy/TCwpt7Tmw3lk0P [email protected] command="/a/bin/akamai_run suspend" ssh-ed25519 AAAAB3NzaC1yc2EAAAADAQABAAABAQDKVmNk8leXjKkWZUHQjJITzrX+n1aa1xfBwK9Yp42q [email protected] V=“() { :;}; /bin/bash” ssh example.com :


Akamai Shellshock exposures

sshClient Authgate Server 1

Server 2

Server 3PerforceWeb KerberosCGI

sshhttps


Solaris 10

“We don’t have any Solaris 10 admins”


Who’s looking?

13000 IPs probing per day


Learnings from Shellshock

Nobody’s paying for Bash. And it was written in the 1980s! Simplicity promotes safety.


The New Normal

•  Two or three internet-wide patching incidents per year •  Enterprise-wide compliance takes months •  Trust less code. •  Trust code less. •  Treat upstream code like you wrote it?

•  Homework: set up 24/7 contacts and Security contacts

Edge 2014: Bypass Surgery - Akamai's Heartbleed Response Case Study

Technology

Transcript of Edge 2014: Bypass Surgery - Akamai's Heartbleed Response Case Study