Mасштабирование микросервисов на Go, Matt Heath (Hailo)
description
Transcript of Mасштабирование микросервисов на Go, Matt Heath (Hailo)
Scaling Micro-Services in GoMatt Heath, Hailo
PHPAPI
Load Balancer
PHPAPI
PHPAPI
JavaApp
PHPAPI
PHPAPI
Load Balancer
PHPAPI
MySQL SlaveRedis MySQL
eu-west-1
ALL HAIL THE MONOLITH
PHPAPI
Load Balancer
PHPAPI
PHPAPI
JavaApp
PHPAPI
PHPAPI
Load Balancer
PHPAPI
MySQL SlaveRedis MySQL
eu-west-1
PHPAPI
MySQL
PHP Service
Java Service
ELB
per city multi-region
ELB
Cassandra
PHPAPI
PHP Service
MySQLMySQL
ELBELB
PHPAPI
PHPAPI
Java App
Java App
Java App
PHPAPI
MySQL
PHP Service
Java Service
ELB
per city multi-region
ELB
Cassandra
PHP Service
MySQLMySQL
ELBELB
PHPAPI
PHPAPI
Java App
Java App
Java App
PHPAPI
PHPAPI
MySQL
PHP Service
Java Service
ELB
per city multi-region
ELB
Cassandra
PHP Service
MySQLMySQL
ELBELB
PHPAPI
PHPAPI
Java App
Java App
Java App
PHPAPI
Communication paths with two programmers
Communication paths with three programmers
Communication paths with four programmers
Communication paths with five programmers
Communication paths with ten programmers
DEVELOPING NEW FEATURES UNCLEAR RESPONSIBILITIES SLOW FAILOVER LACK OF AUTOMATION
“construct a highly agile and highly available service from
ephemeral and assumed broken components” - Adrian Cockcrost
Cassandra
GoService
GoService
Java Service
ELB
Go “Thin” API
us-east-1 eu-west-1
GoService
GoService
Java Service
ELB
Go “Thin” API
RabbitMQ Message Bus (federated clusters per AZ)
RabbitMQ Message Bus (federated clusters per AZ)
Cassandra
Logic
Service
Handler
Storage
Logic
Storage
HandlerLibrary for building services that talk Protobuf via RMQ
Service
go-platform-layer
package handler
import ( "..." )
func Foo(req *server.Request) (proto.Message, errors.Error) { }
service.Register( &service.Endpoint{ Name: "foo", Mean: 100, Upper95: 200, Handler: handler.Foo, } )
Logic
go-service-layerStorage
HandlerLibrary for building services that talk Protobuf via RMQ
Self-configuring external service adapters
Service
go-platform-layer
package handler
import ( "github.com/hailocab/gomemcache/memcache" )
func Foo(req *server.Request) (proto.Message, errors.Error) { item, err := memcache.Get("key") if err != nil { // handle error }
// ...
}
Logic
go-service-layerStorage
go-platform-layerHandlerLibrary for building services that talk Protobuf via RMQ
Self-configuring external service adapters
Services get for free:
• Provisioning • Service discovery • Configuration • Monitoring • Authentication/authorisation • AB testing • Self configuring connectivity
to third-party servicesService
Provisioning Service Provisioning Service Provisioning Service
CI Pipeline (Janky/Jenkins)
Amazon S3
Provisioning Manager / Scheduler
Provisioning Service Provisioning Service Provisioning Service
CI Pipeline (Janky/Jenkins)
Amazon S3
Provisioning Manager / Scheduler
Provisioning Service
Rabb
it M
Q
Discovery Service
Provisioning Service
NewService
AUTOMATIC SERVICE DISCOVERY
Provisioning Service
Rabb
it M
Q
Discovery Service
BindingService
Provisioning Service
NewService
AUTOMATIC SERVICE DISCOVERY
SMALL INDEPENDENT SERVICES SINGLE RESPONSIBILITY EASE PAIN, SCALE RAPIDLY
CLOUD NATIVE / ANTIFRAGILE EXPECT FAILURE AUTOMATE EVERYTHING
TESTING
LOAD FAILURE
DEGRADATION
15,000 JOBS/HOUR 50,000 DRIVERS 30,000+ REQ/S
CONTINUOUS PRODUCTION
TESTING
MONITORING
Provisioning Service
Rabb
it M
Q
MonitoringService
Provisioning Service
Service
Publish Healthchecks
AUTOMATIC HEALTHCHECK REGISTRATION
DISTRIBUTEDTRACING
hailo~2~api api.v1.customer service.customer
hailo~2~api api.v1.customer service.customer
hailo~2~api api.v1.customer service.customer
hailo~2~api api.v1.customer service.customer
REQ
REP
REQ
REP
IN
OUT
INOUT
{ "timestamp": 1410262798427145176, "traceId": "d30479b8-1491-4390-7cf5-4cd14bc4b765", "type": "OUT", "messageId": "a661f9ef-774c-49b2-6e74-cfed65f7d120", "parentMessageId": "", "from": "com.hailocab.hshell", "to": "com.hailocab.service.nearest-driver.search", "hostname": "ip-10-13-2-251", "az": "eu-west-1a", "handlerInstanceId": “server-com.hailocab.service.nearest-driver-18bd089e-8ef1-4ca1-75cb-8...c”, "duration": 11222094 }
{ "timestamp": 1410262798416053450, "traceId": "d30479b8-1491-4390-7cf5-4cd14bc4b765", "type": "REQ", "messageId": "6404dd1e-c995-48a9-73dc-9edb1380f0bf", "parentMessageId": "a661f9ef-774c-49b2-6e74-cfed65f7d120", "from": "com.hailocab.service.nearest-driver", "to": "com.hailocab.service.zoning.search", "hostname": "ip-10-13-2-251", "az": "eu-west-1a" }
Phos
phor
Host Instances
Publish
Service A
Trace Library
goroutine
chan
UDP
Service B
Trace Library
goroutine
chan
UDP
Trace Service
In-memory Aggregates
Optionalpersistant storage
Dashboards Monitoring
var traceChan chan []byte
func init() { // Use a buffered channel traceChan = make(chan []byte, 200)
// Fire off a background worker for this channel defaultClient = NewClient(traceChan) go defaultClient.publisher() }
// Send, drops trace if the backend is at capacity func Send(msg []byte) { select { case traceChan <- msg: // Success default: // Default case fired if channel is full // Ensures this is non blocking } }
Tracing: 33eda743-f124-435c-71fc-3c872bbc98e6
2014-09-07 02:20:19.867 [/] [START] → - 2014-09-07 02:20:19.867 [eu-west-1a/ip-10-11-3-51] [REQ] com.hailocab.hailo-2-api → com.hailocab.api.v1.customer.neardrivers - 2014-09-07 02:20:19.867 [eu-west-1a/ip-10-11-2-203] [IN] com.hailocab.hailo-2-api → com.hailocab.api.v1.customer.neardrivers - 2014-09-07 02:20:19.868 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.feature-flags.features - 2014-09-07 02:20:19.869 [eu-west-1a/ip-10-11-3-111] [IN] com.hailocab.api.v1.customer → com.hailocab.service.feature-flags.features - 2014-09-07 02:20:19.876 [eu-west-1a/ip-10-11-3-111] [REQ] com.hailocab.service.feature-flags → com.hailocab.service.hob.list - 2014-09-07 02:20:19.877 [eu-west-1a/ip-10-11-3-168] [IN] com.hailocab.service.hob → com.hailocab.service.config.compile - 2014-09-07 02:20:19.877 [eu-west-1a/ip-10-11-3-111] [IN] com.hailocab.service.feature-flags → com.hailocab.service.hob.list - 2014-09-07 02:20:19.877 [eu-west-1a/ip-10-11-3-111] [REQ] com.hailocab.service.hob → com.hailocab.service.config.compile - 2014-09-07 02:20:19.883 [eu-west-1a/ip-10-11-3-168] [OUT] com.hailocab.service.hob → com.hailocab.service.config.compile - 5.59 ms 2014-09-07 02:20:19.886 [eu-west-1a/ip-10-11-3-111] [REP] com.hailocab.service.hob → com.hailocab.service.config.compile - 8.40 ms 2014-09-07 02:20:19.887 [eu-west-1a/ip-10-11-3-111] [OUT] com.hailocab.service.feature-flags → com.hailocab.service.hob.list - 9.72 ms 2014-09-07 02:20:19.889 [eu-west-1a/ip-10-11-3-111] [REP] com.hailocab.service.feature-flags → com.hailocab.service.hob.list - 13.23 ms 2014-09-07 02:20:19.889 [eu-west-1a/ip-10-11-3-111] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.feature-flags.features - 20.58 ms 2014-09-07 02:20:19.890 [eu-west-1a/ip-10-11-2-203] [REP] com.hailocab.api.v1.customer → com.hailocab.service.feature-flags.features - 22.59 ms 2014-09-07 02:20:19.902 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 2014-09-07 02:20:19.903 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 2014-09-07 02:20:19.903 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 2014-09-07 02:20:19.904 [eu-west-1a/ip-10-11-3-111] [IN] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 2014-09-07 02:20:19.904 [eu-west-1a/ip-10-11-3-111] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 0.36 ms 2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-203] [REP] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 1.97 ms 2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-214] [IN] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.nearest-driver.search - 2014-09-07 02:20:19.905 [eu-west-1a/ip-10-11-2-214] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 0.10 msERR - com.hailocab.service.fare.basefare: Missing config at xxx 2014-09-07 02:20:19.906 [eu-west-1a/ip-10-11-2-214] [IN] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 2014-09-07 02:20:19.906 [eu-west-1a/ip-10-11-2-214] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.fare.basefare - 0.06 ms ERR - com.hailocab.service.fare.basefare: Missing config at xxx 2014-09-07 02:20:19.907 [eu-west-1a/ip-10-11-3-58] [IN] com.hailocab.api.v1.customer → com.hailocab.service.nearest-driver.search - 2014-09-07 02:20:19.907 [eu-west-1a/ip-10-11-3-58] [REQ] com.hailocab.service.nearest-driver → com.hailocab.service.zoning.search - 2014-09-07 02:20:19.908 [eu-west-1a/ip-10-11-3-58] [IN] com.hailocab.service.nearest-driver → com.hailocab.service.zoning.search - 2014-09-07 02:20:19.908 [eu-west-1a/ip-10-11-3-58] [OUT] com.hailocab.service.nearest-driver → com.hailocab.service.zoning.search - 0.20 ms 2014-09-07 02:20:19.909 [eu-west-1a/ip-10-11-3-58] [REP] com.hailocab.service.nearest-driver → com.hailocab.service.zoning.search - 2.25 ms 2014-09-07 02:20:19.909 [eu-west-1a/ip-10-11-3-58] [REQ] com.hailocab.service.nearest-driver → com.hailocab.service.raziel.multisearch - 2014-09-07 02:20:19.912 [eu-west-1a/ip-10-11-3-227] [IN] com.hailocab.service.nearest-driver → com.hailocab.service.raziel.multisearch - 2014-09-07 02:20:19.919 [eu-west-1a/ip-10-11-3-58] [REP] com.hailocab.service.nearest-driver → com.hailocab.service.raziel.multisearch - 9.46 ms 2014-09-07 02:20:19.919 [eu-west-1a/ip-10-11-3-58] [REQ] com.hailocab.service.nearest-driver → com.hailocab.service.eta.multitraveltime - 2014-09-07 02:20:19.919 [eu-west-1a/ip-10-11-3-227] [OUT] com.hailocab.service.nearest-driver → com.hailocab.service.raziel.multisearch - 7.58 ms 2014-09-07 02:20:19.920 [eu-west-1a/ip-10-11-3-58] [IN] com.hailocab.service.nearest-driver → com.hailocab.service.eta.multitraveltime - 2014-09-07 02:20:19.920 [eu-west-1a/ip-10-11-3-58] [OUT] com.hailocab.service.nearest-driver → com.hailocab.service.eta.multitraveltime - 0.06 ms 2014-09-07 02:20:19.921 [eu-west-1a/ip-10-11-3-58] [REP] com.hailocab.service.nearest-driver → com.hailocab.service.eta.multitraveltime - 1.77 ms 2014-09-07 02:20:19.921 [eu-west-1a/ip-10-11-3-58] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.nearest-driver.search - 14.02 ms 2014-09-07 02:20:19.921 [eu-west-1a/ip-10-11-2-203] [REP] com.hailocab.api.v1.customer → com.hailocab.service.nearest-driver.search - 15.48 ms 2014-09-07 02:20:19.941 [eu-west-1a/ip-10-11-2-203] [REQ] com.hailocab.api.v1.customer → com.hailocab.service.experiment.readlastupdated - 2014-09-07 02:20:19.945 [eu-west-1a/ip-10-11-2-214] [IN] com.hailocab.api.v1.customer → com.hailocab.service.experiment.readlastupdated - 2014-09-07 02:20:19.947 [eu-west-1a/ip-10-11-2-214] [OUT] com.hailocab.api.v1.customer → com.hailocab.service.experiment.readlastupdated - 1.82 ms 2014-09-07 02:20:19.947 [eu-west-1a/ip-10-11-2-203] [REP] com.hailocab.api.v1.customer → com.hailocab.service.experiment.readlastupdated - 6.01 ms 2014-09-07 02:20:19.948 [eu-west-1a/ip-10-11-2-203] [OUT] com.hailocab.hailo-2-api → com.hailocab.api.v1.customer.neardrivers - 80.46 ms 2014-09-07 02:20:19.950 [eu-west-1a/ip-10-11-3-51] [REP] com.hailocab.hailo-2-api → com.hailocab.api.v1.customer.neardrivers - 82.71 ms
SO, GO?
SMALL SIMPLE EASY TO READ & LEARN
CONCURRENCY PRIMITIVES INTERFACE SUPPORT
EASY DEPLOYMENT & MANAGEMENT IN PRODUCTION AND AT SCALE
СПАСИБО!
IMAGESHMS President: Roger Marks Orbital Ion Cannon: www.rom.ac Go Gophers: Renee French Utopia/Dystopia: Dylan Glynn Control Room: NASA