Bypass Surgery and Other Tales Brian Sniffen

©2014 AKAMAI | FASTER FORWARDTM

Akamai Security Research & Architecture

•  Crypto engineering expertise •  Technical backstop •  Product review •  Akamai Architecture Group seat •  Safety engineering •  Incident management

©2014 AKAMAI | FASTER FORWARDTM

2014: The Year of Vulnerabilities

“You people in InfoSec have become the Product Managers!”

Yes, and we can’t wait to get out of that role.

•  Heartbleed •  INRIA-Prosecco Cookies •  Shellshock

2009 2010 2011 2012 2013 2014

1 0 0 1 1 3+

2008

1

©2014 AKAMAI | FASTER FORWARDTM

Akamai Incident Management Principles

•  Technical Incident Manager (TIM) coordinates all work •  Incident Executive communicates upwards, shields TIM •  GSS Business Incident Leads manage customer comms

•  No single-point harm can cause a Severity 1 Incident •  A hot meal and 6 hours sleep fix more problems than an all-nighter •  If the TIM becomes an SME, get a new TIM

©2014 AKAMAI | FASTER FORWARDTM

We tell ourselves who we wish to be:

•  Akamai says thank you. •  Akamai doesn't respond to name calling,

but does respond to the useful technical content. •  Akamai presents itself as a responsible and respected member

of the Internet community. •  Akamai will use this incident to improve both its own security

and the general security of the Internet. •  Akamai can laugh at itself.

©2014 AKAMAI | FASTER FORWARDTM

Heartbleed mail

From: Brian Sniffen Date: 7 Apr 2014 21:34:08 Subject: Sev 1: Heartbleed Will, I'm contacting you because you're the Ghost SME on call. I'm looking for evidence to refute the statement: "The Heartbleed bug can’t extract long-term customer private keys from a Ghost; we put them only in a wired, mmaped page way lower on the stack."

©2014 AKAMAI | FASTER FORWARDTM

Heartbleed Timeline

April 1: Notice; QA begins April 4: last Akamai Deployed Systems patched April 7, 1pm: Public Notice April 7, 6pm: What did we leak? April 8, 1am: Working exploit in Akamai lab April 9–12: Hastily publish Akamai Secure Allocator April 13, 11pm: Begin cert rotations & revocations

©2014 AKAMAI | FASTER FORWARDTM

“Don’t worry, we restored the old functionality”

April 14, 6am: “Why is this message in the old log format?”

A “Manual Change” had restored an old version.

©2014 AKAMAI | FASTER FORWARDTM

The Akamai Secure Memory Allocator

•  1999 code •  One author, three redactors •  State machine inspired by CLOS “advice” system

Turns out it works fine

Code Secure Heap mmap’d file Long-term

Allocations Heap

©2014 AKAMAI | FASTER FORWARDTM

Cert Revocation Progress

21 Apr

28 Apr

5 May

12 May

19 May

26 May

2 Jun

9 Jun

70% 90% 95%

# of

cer

ts

16 Jun

23 Jun

30 Jun

©2014 AKAMAI | FASTER FORWARDTM

Learning from Heartbleed

Nobody’s paying for OpenSSL! Practice in mass, fast, patching Practice in releasing helpful patches Simplicity promotes safety.

©2014 AKAMAI | FASTER FORWARDTM

Shellshock Timeline

Sep 23, 12pm: Notice from Florian Weimer, Debian Security Sep 23, 9pm: Manual change: replace bash wish dash;

Patches started Sep 24, 5am: WAF rule in place

SSH command=  systems made safe Sep 24, 12pm: Public Notice Sep 25: “Kobrin Patch” to remove dangerous feature Sep 28: bash mostly replaced with dash on deployed network

©2014 AKAMAI | FASTER FORWARDTM

Bash patches

Pre-release: •  Embargoed patch: 195 lines, 7 files (1/6 CVEs) •  Kobrin patch: 2 lines, 1 file (6/6 CVEs)

Post-release: •  NetBSD patch: 3 lines, 2 files (6/6 CVEs) •  Fixed patch: 164 lines, 11 files (6/6 CVEs) •  Apple patch: unpublished (exposure unclear)

©2014 AKAMAI | FASTER FORWARDTM

SSH command= limits

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFnHfYyS5onAN72oFpaopm+/yKbRy/TCwpt7Tmw3lk0P bts@tereva.evenmere.org-2014-03-19 command="/a/bin/akamai_run suspend" ssh-ed25519 AAAAB3NzaC1yc2EAAAADAQABAAABAQDKVmNk8leXjKkWZUHQjJITzrX+n1aa1xfBwK9Yp42q bsniffen@akamai.com-example-2014-10-01 V=“() { :;}; /bin/bash” ssh example.com :

©2014 AKAMAI | FASTER FORWARDTM

Akamai Shellshock exposures

sshClient Authgate Server 1

Server 2

Server 3PerforceWeb KerberosCGI

sshhttps

©2014 AKAMAI | FASTER FORWARDTM

Solaris 10

“We don’t have any Solaris 10 admins”

©2014 AKAMAI | FASTER FORWARDTM

Who’s looking?

13000 IPs probing per day

©2014 AKAMAI | FASTER FORWARDTM

Learnings from Shellshock

Nobody’s paying for Bash. And it was written in the 1980s! Simplicity promotes safety.

©2014 AKAMAI | FASTER FORWARDTM

The New Normal

•  Two or three internet-wide patching incidents per year •  Enterprise-wide compliance takes months •  Trust less code. •  Trust code less. •  Treat upstream code like you wrote it?

•  Homework: set up 24/7 contacts and Security contacts