Re-arquitetando oRe-arquitetando oStack OverflowStack Overflow
ou como construímos o Stack Overflow for Teams
Roberta Arcoverde1
/whois /whois
recifenseprogramadora há 15 anosprincipal softwaredeveloper na stackoverflowco-host do hipsters.tech@rla4
2
desde 200850+ milhões de usuários únicos/mês18 milhões de perguntas27 milhões de respostastop 50 sites mais acessados do mundo
3
3k Teams criados, 50k usuários10 meses em desenvolvimentolançado em maio/2018equipe tinha originalmente 3 devs,agora são 7
melhor nome de time da história:Teams Team �
4
https://stackoverflow.com/c/demo 5
https://stackoverflow.com/c/demo 5
https://stackoverflow.com/c/demo 5
https://stackoverflow.com/c/demo 5
https://stackoverflow.com/c/demo 5
https://stackoverflow.com/c/demo 5
https://stackoverflow.com 6
https://stackoverflow.com 6
>170 sites>170 sites
7
números do dia 03/05números do dia 03/05
278.912.108 HTTP requests67.188.355 page views3.506.670.995.363 bytes (3.5 TB) enviados953.860.308 SQL queries executadas5.250.697.564 redis hits600.000 websockets ativos19ms de tempo de renderização da Question page
54.290.431 page views, ou 80% do total123ms de tempo de renderização geral
8
9 WEB SERVERS
4 SQL SERVERS
LIVE HOT STANDBY LIVE HOT STANDBY
9
Stack Exchange, Meta, TalentStack Overflow
~350 req/s
por servidor
528 Mqueries/dia
498 Mqueries/dia
~5% CPU
imagem gentilmente cedida por Marco (@sklivvz) em http://www.slideshare.net/howtoweb/marco-cecconi-stack-overflow-architecture
10
11
como?como?
spoilers: é boring12
performance performance é umaé umafeaturefeature
13
tech stacktech stackc#asp.net mvc*sql server
dapper, ef coretypescript
vanillarediselasticsearchha proxy
*migrando pra .NET Core
14
15
�♀
15
multi tenant applicationmulti tenant application
um único app pool paratodos os sitesroteado via host headers
16
17
https://nickcraver.com/blog/2016/02/03/stack-overflow-a-technical-deconstruction/ 18
Q&A pra dadosQ&A pra dadosprivados?privados?
19
(o nome original do SO for Teams era Channels)
nasce uma ideia! (sim, o screenshot é legítimo)20
times são sites que existem
dentro do Stack Overflowtratá-los como se fossem novos sites narede, porém visíveis apenas a partir do
public class Post { public int Id { get; } public string Title { get; } public int? TeamId { get; } ... } // reusar banco // criar novo código
public class Post { public int Id { get; } public string Title { get; } ... } // criar novo banco // reusar código
21
[StackRoute("help/search-inline")]public async Task<ActionResult> SearchInline(string q){ var searchSite = GetSearchSite(); var results = await searchSite.HelpPostIndex.SearchAsync(searchSite, q); var sm = new SearchModel { SearchString = q, Results = results }; return PartialView("~/Views/Help/SearchInline.cshtml", sm);}
123456789
10111213
https://stackoverflow.com/help/search-inlinehttps://askubuntu.com/help/search-inlinehttps://stackoverflow.com/c/demo/help/search-inline
22
ModeloModeloevitar forks, DRY, minimizar alterações no core do projeto
23
ModeloModelo
EscalabilidadeEscalabilidade
evitar forks, DRY, minimizar alterações no core do projeto
capacity planning, o que acontece se tivermos 1k, 10k, 100ktimes?
23
ModeloModelo
SegurançaSegurança
EscalabilidadeEscalabilidade
evitar forks, DRY, minimizar alterações no core do projeto
default private, mudança de mindset, crash na aplicação >vazamento de dados
capacity planning, o que acontece se tivermos 1k, 10k, 100ktimes?
23
��
Bases isoladas entreTeamsDados isolados dosdados públicosMínimo de alteraçõesno código (usar modeloexistente pra novossites)
��
Escalabilidade. AGdistribuídos começam adegradar rapidamentea partir de 1k bancosHardware einstrumentação paragerenciar milhares debases de dados
Plano A: um banco paraPlano A: um banco paracada Teamcada Team
24
��
EscalabilidadeDados isolados dosdados públicos
��
Sem isolamento entreTeamsReescrever boa partedas consultasConsultas não são maisas mesmas para sites vsTeams
Plano B: um banco paraPlano B: um banco paratodos os Teamstodos os Teams
25
��
Dados isolados entreTeamsDados isolados dosdados públicosEscalabilidade é...decenteBaixo custo de reescrita
��
Precisamos escreverinfra deprovisionamentodinâmico
Plano C: um schema porPlano C: um schema portime no mesmo bancotime no mesmo banco
26
27
28
basicamente: saindo de 170 para 10k+ sites
SQL Server1 banco per-site1 banco pra todos os Teams, 1 schema per-Team
Elasticsearch1 índice per-site1 índice per-team, até 5k
Provisionamentotarefa agendada cria sempre um buffer de 100schemas para futuros Teams
EscalabilidadeEscalabilidade
29
onde manter os dados dos Teams?como comunicar o site público com o Team?migrar *tudo* pra lugares seguros
notificaçõesemailsmonitoramentointernal APIwebsocketstags
SegurançaSegurança
30
31
como as redes secomo as redes secomunicam?comunicam?
32
ProxyingProxyingJá usávamos no /jobsRequisição é "clonada" e enviada para a CFZResponse é jogada direto no stream de saída800 LoCPor que não usar APIs/serviços?
custo de serializaçãomais código, menos uniformidade
33
[StackRoute("c/{slug}")][StackRoute("c/{slug}/{*pathInfo}")]public async Task<ActionResult> Proxy(string slug){ if (!Current.Settings.Channels.Enabled) { return PageNotFound(); } ... if (Current.Request.IsProxied()) { // yo dawg, I heard you like proxies so we put a proxy in your proxy // so you can channel yo inner channels... Let's not allow this return PageNotFound(); } var returnUrl = Current.Request.Url.PathAndQuery; if (!Current.SiteChannels.Contains(channelSite.Id)) { // user does not have access to this channel return RedirectToJoinPage(); } ... return await this.BlindProxy(channelSite, path);} // BlindProxy: // valida a requisição (authorization);// constrói um Request;// envia via HTTP para o Team app;// retorna o resultado// profit :D
123456789
101112131415161718192021222324252627282930313233
34
[StackRoute("c/{slug}")][StackRoute("c/{slug}/{*pathInfo}")]public async Task<ActionResult> Proxy(string slug){ if (!Current.Settings.Channels.Enabled) { return PageNotFound(); } ... if (Current.Request.IsProxied()) { // yo dawg, I heard you like proxies so we put a proxy in your proxy // so you can channel yo inner channels... Let's not allow this return PageNotFound(); } var returnUrl = Current.Request.Url.PathAndQuery; if (!Current.SiteChannels.Contains(channelSite.Id)) { // user does not have access to this channel return RedirectToJoinPage(); } ... return await this.BlindProxy(channelSite, path);} // BlindProxy: // valida a requisição (authorization);// constrói um Request;// envia via HTTP para o Team app;// retorna o resultado// profit :D
123456789
101112131415161718192021222324252627282930313233
[StackRoute("c/{slug}")][StackRoute("c/{slug}/{*pathInfo}")]
12
public async Task<ActionResult> Proxy(string slug)3{4 if (!Current.Settings.Channels.Enabled)5 {6 return PageNotFound();7 } 8 ...9 if (Current.Request.IsProxied())10 {11 // yo dawg, I heard you like proxies so we put a proxy in your proxy12 // so you can channel yo inner channels... Let's not allow this13 return PageNotFound();14 }15 16 var returnUrl = Current.Request.Url.PathAndQuery;17 if (!Current.SiteChannels.Contains(channelSite.Id))18 {19 // user does not have access to this channel20 return RedirectToJoinPage();21 }22 ...23 24 return await this.BlindProxy(channelSite, path);25}26 27// BlindProxy: 28// valida a requisição (authorization);29// constrói um Request;30// envia via HTTP para o Team app;31// retorna o resultado32// profit :D33
34
[StackRoute("c/{slug}")][StackRoute("c/{slug}/{*pathInfo}")]public async Task<ActionResult> Proxy(string slug){ if (!Current.Settings.Channels.Enabled) { return PageNotFound(); } ... if (Current.Request.IsProxied()) { // yo dawg, I heard you like proxies so we put a proxy in your proxy // so you can channel yo inner channels... Let's not allow this return PageNotFound(); } var returnUrl = Current.Request.Url.PathAndQuery; if (!Current.SiteChannels.Contains(channelSite.Id)) { // user does not have access to this channel return RedirectToJoinPage(); } ... return await this.BlindProxy(channelSite, path);} // BlindProxy: // valida a requisição (authorization);// constrói um Request;// envia via HTTP para o Team app;// retorna o resultado// profit :D
123456789
101112131415161718192021222324252627282930313233
[StackRoute("c/{slug}")][StackRoute("c/{slug}/{*pathInfo}")]
12
public async Task<ActionResult> Proxy(string slug)3{4 if (!Current.Settings.Channels.Enabled)5 {6 return PageNotFound();7 } 8 ...9 if (Current.Request.IsProxied())10 {11 // yo dawg, I heard you like proxies so we put a proxy in your proxy12 // so you can channel yo inner channels... Let's not allow this13 return PageNotFound();14 }15 16 var returnUrl = Current.Request.Url.PathAndQuery;17 if (!Current.SiteChannels.Contains(channelSite.Id))18 {19 // user does not have access to this channel20 return RedirectToJoinPage();21 }22 ...23 24 return await this.BlindProxy(channelSite, path);25}26 27// BlindProxy: 28// valida a requisição (authorization);29// constrói um Request;30// envia via HTTP para o Team app;31// retorna o resultado32// profit :D33
if (Current.Request.IsProxied()) { // yo dawg, I heard you like proxies so we put a proxy in your proxy // so you can channel yo inner channels... Let's not allow this return PageNotFound(); }
[StackRoute("c/{slug}")]1[StackRoute("c/{slug}/{*pathInfo}")]2public async Task<ActionResult> Proxy(string slug)3{4 if (!Current.Settings.Channels.Enabled)5 {6 return PageNotFound();7 } 8 ...9
101112131415
16 var returnUrl = Current.Request.Url.PathAndQuery;17 if (!Current.SiteChannels.Contains(channelSite.Id))18 {19 // user does not have access to this channel20 return RedirectToJoinPage();21 }22 ...23 24 return await this.BlindProxy(channelSite, path);25}26 27// BlindProxy: 28// valida a requisição (authorization);29// constrói um Request;30// envia via HTTP para o Team app;31// retorna o resultado32// profit :D33
34
[StackRoute("c/{slug}")][StackRoute("c/{slug}/{*pathInfo}")]public async Task<ActionResult> Proxy(string slug){ if (!Current.Settings.Channels.Enabled) { return PageNotFound(); } ... if (Current.Request.IsProxied()) { // yo dawg, I heard you like proxies so we put a proxy in your proxy // so you can channel yo inner channels... Let's not allow this return PageNotFound(); } var returnUrl = Current.Request.Url.PathAndQuery; if (!Current.SiteChannels.Contains(channelSite.Id)) { // user does not have access to this channel return RedirectToJoinPage(); } ... return await this.BlindProxy(channelSite, path);} // BlindProxy: // valida a requisição (authorization);// constrói um Request;// envia via HTTP para o Team app;// retorna o resultado// profit :D
123456789
101112131415161718192021222324252627282930313233
[StackRoute("c/{slug}")][StackRoute("c/{slug}/{*pathInfo}")]
12
public async Task<ActionResult> Proxy(string slug)3{4 if (!Current.Settings.Channels.Enabled)5 {6 return PageNotFound();7 } 8 ...9 if (Current.Request.IsProxied())10 {11 // yo dawg, I heard you like proxies so we put a proxy in your proxy12 // so you can channel yo inner channels... Let's not allow this13 return PageNotFound();14 }15 16 var returnUrl = Current.Request.Url.PathAndQuery;17 if (!Current.SiteChannels.Contains(channelSite.Id))18 {19 // user does not have access to this channel20 return RedirectToJoinPage();21 }22 ...23 24 return await this.BlindProxy(channelSite, path);25}26 27// BlindProxy: 28// valida a requisição (authorization);29// constrói um Request;30// envia via HTTP para o Team app;31// retorna o resultado32// profit :D33
if (Current.Request.IsProxied()) { // yo dawg, I heard you like proxies so we put a proxy in your proxy // so you can channel yo inner channels... Let's not allow this return PageNotFound(); }
[StackRoute("c/{slug}")]1[StackRoute("c/{slug}/{*pathInfo}")]2public async Task<ActionResult> Proxy(string slug)3{4 if (!Current.Settings.Channels.Enabled)5 {6 return PageNotFound();7 } 8 ...9
101112131415
16 var returnUrl = Current.Request.Url.PathAndQuery;17 if (!Current.SiteChannels.Contains(channelSite.Id))18 {19 // user does not have access to this channel20 return RedirectToJoinPage();21 }22 ...23 24 return await this.BlindProxy(channelSite, path);25}26 27// BlindProxy: 28// valida a requisição (authorization);29// constrói um Request;30// envia via HTTP para o Team app;31// retorna o resultado32// profit :D33
return await this.BlindProxy(channelSite, path);
[StackRoute("c/{slug}")]1[StackRoute("c/{slug}/{*pathInfo}")]2public async Task<ActionResult> Proxy(string slug)3{4 if (!Current.Settings.Channels.Enabled)5 {6 return PageNotFound();7 } 8 ...9 if (Current.Request.IsProxied())10 {11 // yo dawg, I heard you like proxies so we put a proxy in your proxy12 // so you can channel yo inner channels... Let's not allow this13 return PageNotFound();14 }15 16 var returnUrl = Current.Request.Url.PathAndQuery;17 if (!Current.SiteChannels.Contains(channelSite.Id))18 {19 // user does not have access to this channel20 return RedirectToJoinPage();21 }22 ...23 24
25}26 27// BlindProxy: 28// valida a requisição (authorization);29// constrói um Request;30// envia via HTTP para o Team app;31// retorna o resultado32// profit :D33
34
[StackRoute("c/{slug}")][StackRoute("c/{slug}/{*pathInfo}")]public async Task<ActionResult> Proxy(string slug){ if (!Current.Settings.Channels.Enabled) { return PageNotFound(); } ... if (Current.Request.IsProxied()) { // yo dawg, I heard you like proxies so we put a proxy in your proxy // so you can channel yo inner channels... Let's not allow this return PageNotFound(); } var returnUrl = Current.Request.Url.PathAndQuery; if (!Current.SiteChannels.Contains(channelSite.Id)) { // user does not have access to this channel return RedirectToJoinPage(); } ... return await this.BlindProxy(channelSite, path);} // BlindProxy: // valida a requisição (authorization);// constrói um Request;// envia via HTTP para o Team app;// retorna o resultado// profit :D
123456789
101112131415161718192021222324252627282930313233
[StackRoute("c/{slug}")][StackRoute("c/{slug}/{*pathInfo}")]
12
public async Task<ActionResult> Proxy(string slug)3{4 if (!Current.Settings.Channels.Enabled)5 {6 return PageNotFound();7 } 8 ...9 if (Current.Request.IsProxied())10 {11 // yo dawg, I heard you like proxies so we put a proxy in your proxy12 // so you can channel yo inner channels... Let's not allow this13 return PageNotFound();14 }15 16 var returnUrl = Current.Request.Url.PathAndQuery;17 if (!Current.SiteChannels.Contains(channelSite.Id))18 {19 // user does not have access to this channel20 return RedirectToJoinPage();21 }22 ...23 24 return await this.BlindProxy(channelSite, path);25}26 27// BlindProxy: 28// valida a requisição (authorization);29// constrói um Request;30// envia via HTTP para o Team app;31// retorna o resultado32// profit :D33
if (Current.Request.IsProxied()) { // yo dawg, I heard you like proxies so we put a proxy in your proxy // so you can channel yo inner channels... Let's not allow this return PageNotFound(); }
[StackRoute("c/{slug}")]1[StackRoute("c/{slug}/{*pathInfo}")]2public async Task<ActionResult> Proxy(string slug)3{4 if (!Current.Settings.Channels.Enabled)5 {6 return PageNotFound();7 } 8 ...9
101112131415
16 var returnUrl = Current.Request.Url.PathAndQuery;17 if (!Current.SiteChannels.Contains(channelSite.Id))18 {19 // user does not have access to this channel20 return RedirectToJoinPage();21 }22 ...23 24 return await this.BlindProxy(channelSite, path);25}26 27// BlindProxy: 28// valida a requisição (authorization);29// constrói um Request;30// envia via HTTP para o Team app;31// retorna o resultado32// profit :D33
return await this.BlindProxy(channelSite, path);
[StackRoute("c/{slug}")]1[StackRoute("c/{slug}/{*pathInfo}")]2public async Task<ActionResult> Proxy(string slug)3{4 if (!Current.Settings.Channels.Enabled)5 {6 return PageNotFound();7 } 8 ...9 if (Current.Request.IsProxied())10 {11 // yo dawg, I heard you like proxies so we put a proxy in your proxy12 // so you can channel yo inner channels... Let's not allow this13 return PageNotFound();14 }15 16 var returnUrl = Current.Request.Url.PathAndQuery;17 if (!Current.SiteChannels.Contains(channelSite.Id))18 {19 // user does not have access to this channel20 return RedirectToJoinPage();21 }22 ...23 24
25}26 27// BlindProxy: 28// valida a requisição (authorization);29// constrói um Request;30// envia via HTTP para o Team app;31// retorna o resultado32// profit :D33
// BlindProxy: // valida a requisição (authorization);// constrói um Request;// envia via HTTP para o Team app;// retorna o resultado// profit :D
[StackRoute("c/{slug}")]1[StackRoute("c/{slug}/{*pathInfo}")]2public async Task<ActionResult> Proxy(string slug)3{4 if (!Current.Settings.Channels.Enabled)5 {6 return PageNotFound();7 } 8 ...9 if (Current.Request.IsProxied())10 {11 // yo dawg, I heard you like proxies so we put a proxy in your proxy12 // so you can channel yo inner channels... Let's not allow this13 return PageNotFound();14 }15 16 var returnUrl = Current.Request.Url.PathAndQuery;17 if (!Current.SiteChannels.Contains(channelSite.Id))18 {19 // user does not have access to this channel20 return RedirectToJoinPage();21 }22 ...23 24 return await this.BlindProxy(channelSite, path);25}26 27
282930313233
34
// No, you can't: // - Use a CookieCollection (it'll get headers, but not pass them here) // - Set the Set-Cookie header on the response (ASP.Net strips it) // - Set an additional Set-Cookie (also stripped) // - Take the raw header and pass it (comma delimited, only the first cookie wil // - Use Headers.GetValues(string) (it screws up on commas) // - Maintain your sanity working with ASP.Net and cookie headers // Fun fact: half of the cookie BS here is supporting IIS6 and IE5. Not kidding. if (cResponse.Headers["Set-Cookie"].HasValue()) { var nvc = cResponse.Headers; var result = new List<string>(); for (var i = 0; i < nvc.Count; i++) { if (nvc.GetKey(i) == "Set-Cookie") { // Don't ask. You'll cry. var vals = nvc.GetValues(i); if (vals != null) result.AddRange(vals); } } // ... } 35
liçõesliçõesentenda seus cenários de escalabilidadequando não souber: capacity planningsegurança vai além de proteger dados de acessoexterno
36
outras palestrasoutras palestrasinstrumentação
adaptamos todos os nossos sistemas demonitoramento pra incluir Teams
proxy v2protobufgrpcstructured model
single sign-onre-arquitetando o modelo de autenticação eautorização
modelo de segurançadados (perguntas, respostas, tags)metadados (traffic logs, IPs, urls)external endpoints (ads, APIs, emails) 37
obrigada!obrigada!
rla4roberta at stackoverflow.comrla4.comhipsters.tech 38
instância privada, standalone do StackOverflowSLA, priority supportsingle sign-onon premise ou Azurereleases trimestraiscompletamente customizávelapropriado para grandes empresas$$$
39
Top Related