Post on 18-Jul-2015
+Kazunori Sato@kazunori_279
Developer Advocate, Cloud Platform- Developer Community Support- gcp ja night, Docker Meetup, GCPUG- Product Launch Support
Cloud Technology Innovations
2012 2013
MapReduce
Spanner/F1
2003 2006 2007 2010 2011
GFS
Omega
Colossus
Cloud Storage
Dremel
BigQuery
Bigtable
Cloud Datastore
Paxos impl.
2004
Google Compute Engine
Google App Engine
Google BigQuery Data Analytics with Google’s massively parallel query engine
IaaS with Google’s network and computing infra
PaaS with Google’s container technology
In Google, we useGoogle BigQuery
for processing Big Data
SELECT top(appId, 20) AS app, count(*) AS countFROM installlog.2013ORDER BY count DESC
Scans 100B rowsin 10 sec, without index.
select top(title), count(*) from publicdata:samples.wikipedia
To scan 1 TB in 1 sec,you need 5,000 disks in parallel
Massively Parallel Processing
Mixer 0
Mixer 1 Mixer 1
Shard Shard Shard Shard
ColumnIO on Colossus SELECT state, year
COUNT(*)GROUP BY stateWHERE year >= 1980 and year < 1990
ORDER BY count_babies DESCLIMIT 10
COUNT(*)GROUP BY state
Tree structure for fast aggregation
Google confidential | Do not distribute
Google’s Container Technology
Every Google service runs in containers
Launches 2B containers every week
Google confidential | Do not distribute
Scaling Out by Google’s Container Technology
Instantiationin 40 ms
GAE Architecture And Optmization
The Traditional Web Development
Hardware Failures
Traffic Spike
Growing Big Data
Complex Design
Complex Dev.
Complex Admin
Cost
GAE Architecture And Optmization
Google App Engine Encapsulates Them All
Hardware Failures
Traffic Spike
Growing Big Data
Simpler Design
Simpler Dev
No Admin
No Initial Funding
700 million
“App Engine enabled us to focus on developing the application. We wouldn’t have gotten here without the ease of development that App Engine gave us.”Bobby Murphy, CTO
Snapchat sends
photos and videos each day Google App Enginescaled seamlesslyduring growth to
millions of users
Small team is ableto innovate quickly
and expandglobally
Andromeda: The latest Software Defined Network (SDN)
- Intra/Inter zone throughput = 9G bps- Inter regional private network by default
Google-scale Load Balancing
region 2 region 3region 1
VMs VMs VMs
:::::::::::::::::::::::::::::::::::::::::::::::HTTP LB
DNS :)
One global IP: 11.22.33.44
● Load Balancing infra shared with Google.com
● Easily handles 1M req/s
● TCP anycast with single global IP
● Fast fail-over in seconds
Run and manage an optimized cluster
KubernetesOpen Source
Workload
API & Tools Google Container Engine (GKE)
Kubernetes ● Growing Community ● Linux containers support● Open source orchestration
create and manage
Container Engine ● クラスタをすぐに起動可能
● クラスタ環境の最適化
● Kubernetesを実行管理する
GKEMaster GKE Nodes
Compute Engine
VMs
Google Container Engine