DataPortals

EliDart,NetworkEngineerESnetScienceEngagementLawrenceBerkeleyNationalLaboratory

GlobusWorld

Chicago,IL

April11,2017

Overview

4/15/172

• ScienceDMZandDataPortals

• ThisassumesyoualreadyhaveaScienceDMZ– Ifyoudon’thaveone,wecanchatabouthowyoumightbuildone– Ifitwouldbehelpful,Icantalktoyoursystemsandnetworkingfolks– Orcheckoutthefasterdataknowledgebase:

• http://fasterdata.es.net/science-dmz/

ScienceDMZDesignPattern(Abstract)

10GE

10GE

10GE

10GE

10G

Border Router

WAN

Science DMZSwitch/Router

Enterprise Border Router/Firewall

Site / CampusLAN

High performanceData Transfer Node

with high-speed storage

Per-service security policy control points

Clean, High-bandwidth

WAN path

Site / Campus access to Science

DMZ resources

perfSONAR

perfSONAR

perfSONAR

3 – ESnet Science Engagement (engage@es.net) - 4/15/17 ©2015,EnergySciencesNetwork

HPCCenterDataPath

©2014,EnergySciencesNetwork4 – ESnet Science Engagement (engage@es.net) - 4/15/17

Routed

Border Router

WAN

Core Switch/Router

Firewall

Offices

perfSONAR

perfSONAR

perfSONAR

Supercomputer

Parallel Filesystem

Front endswitch

Data Transfer Nodes

Front endswitch

High Latency WAN Path

Low Latency LAN Path

NextSteps– BuildingOnTheScienceDMZ

• Enhancedcyberinfrastructuresubstratenowexists– Wideareanetworks(ESnet,GEANT,Internet2,Regionals)– ScienceDMZsconnectedtothosenetworks– DTNsintheScienceDMZs

• Whatdoesthescientistsee?– Scientistseesascienceapplication

• Datatransfer• Dataportal• Dataanalysis

– ScienceapplicationsaretheuserinterfacetonetworksandDMZs

• Large-scaledata-intensivesciencerequiresthatwebuildlargerstructuresontopofthosecomponents

4/15/175

ScienceDataPortals

• Largerepositoriesofscientificdata– Climatedata– Skysurveys(astronomy,cosmology)– Manyothers– Datasearch,browsing,access

• Manyscientificdataportalsweredesigned15+yearsago– Single-web-serverdesign– Databrowse/search,dataaccess,userawarenessallinasinglesystem– Allthedatagoesthroughtheportalserver

• Inmanycasesbydesign• E.g.embargobeforepublication(enforceaccesscontrol)

4/15/176

LegacyPortalDesign

10GE

Border Router

WAN

Firewall

Enterprise

perfSONAR

perfSONAR

Filesystem(data store)

10GE

Portal Server

Browsing pathQuery pathData path

Portal server applications:· web server· search· database· authentication· data service

4/15/177

• Verydifficulttoimproveperformancewithoutarchitecturalchange– Softwarecomponentsalltangledtogether

– DifficulttoputthewholeportalinaScienceDMZbecauseofsecurity

– EvenifyoucouldputitinaDMZ,manycomponentsaren’tscalable

• Whatdoesarchitecturalchangemean?

ExampleofArchitecturalChange– CDN

• Let’slookatwhatContentDeliveryNetworksdidforwebapplications

• CDNsareawell-deployeddesignpattern(Netflix,etc)• WhatdoesaCDNdo?

– Storestaticcontentinaseparatelocationfromdynamiccontent• Complexityisn’tinthestaticcontent– it’sintheapplicationdynamics• Webapplicationsarecomplex,full-featured,andslow• Dataserviceforstaticcontentissimplebycomparison

– Separationofapplicationanddataserviceallowseachtobeoptimized

4/15/178

ClassicalWebServerModel

4/15/179

• Webbrowserfetchespagesfromwebserver– Allcontentstoredonthewebserver– Webapplicationsrunonthewebserver– Webserversendsdatatoclientbrowseroverthenetwork

• Perceivedclientperformancechangeswithnetworkconditions– Severalproblemsinthegeneralcase– Latencyincreasestimetopagerender– Packetloss+latencycauseproblemsforlargestaticobjects

HostingProvider

TransitNetwork

Residential BroadbandWEB

Long Distance / High Latency

Web Server

Browser

Solution:PlaceLargeStaticObjectsNearClient

HostingProvider

TransitNetwork

Residential BroadbandWEB

Long Distance / High Latency

CDN

DATA

Short Distance / Low Latency

Web Server

CDN Data Server

Browser

4/15/1710

• CDNprovidesstaticcontent“close”toclient

• Webserverstillmanagescomplexbehavior

• Latencygoesdown– Timetopagerendergoesdown– Staticcontentperformancegoesup

• Loadonwebservergoesdown(noneedtoservestaticcontent)

• Significantwinforwebapplicationperformance

ClientSimplySeesIncreasedPerformance

4/15/1711

• Clientdoesn’tseetheCDNasaseparatething– Webcontentisallstillviewedinabrowser

• Browserfetcheswhatthepagetellsittofetch• Differentcontentcomesfromdifferentplaces• Userdoesn’tknow/care

• CDNsprovideanarchitecturalsolutiontoaperformanceproblem– Notbrute-force– Worksmarter,notharder

The‘NetWEB

Browser

Web Server

Rich, Slow

DATA

CDN Data Server

Simple,Fast

The‘NetWEB

Browser

Web Server

ArchitecturalExaminationofDataPortals

• Commondataportalfunctions(mostportalshavethese)– Search/query/discovery– Datadownloadmethodfordataaccess– GUIforbrowsingbyhumans– APIformachineaccess– ideallyincorporatessearch/query+download

• Performancepainisprimarilyinthedatahandlingpiece– Rapidincreaseindatascaleeclipsedlegacysoftwarestackcapabilities– Portalserversoftenstuckinenterprisenetwork

• Canwe“disassemble”theportalandputthepiecesbacktogetherbetter?– UseScienceDMZasaplatformforthedatapiece– AvoidplacingcomplexsoftwareintheScienceDMZ

4/15/1712

LegacyPortalDesign

10GE

Border Router

WAN

Firewall

Enterprise

perfSONAR

perfSONAR

Filesystem(data store)

10GE

Portal Server

Browsing pathQuery pathData path

Portal server applications:· web server· search· database· authentication· data service

4/15/1713

Next-GenerationPortalLeveragesScienceDMZ

10GE10GE

10GE

10GE

Border Router

WAN

Science DMZSwitch/Router

Firewall

Enterprise

perfSONAR

perfSONAR

10GE

10GE

10GE10GE

DTN

DTN

API DTNs(data access governed

by portal)

DTN

DTN

perfSONAR

Filesystem (data store)

10GE

Portal Server

Browsing pathQuery path

Portal server applications:· web server· search· database· authentication

Data Path

Data Transfer Path

Portal Query/Browse Path

4/15/1714

PutTheDataOnDedicatedInfrastructure

• Wehaveseparatedthedatahandlingfromtheportallogic• Portalisstillitsnormalself,butenhanced

– PortalGUI,database,search,etc.allfunctionastheydidbefore– QueryreturnspointerstodataobjectsintheScienceDMZ– Portalisnowfreedfromtiestothedataservers(runitonAmazonifyouwant!)

• Datahandlingisseparate,andscalable– High-performanceDTNsintheScienceDMZ– Scaleasmuchasyouneedtowithoutmodifyingtheportalsoftware

• Outsourcedatahandlingtocomputingcenters– Computingcentersaresetupforlarge-scaledata– Letthemhandlethelarge-scaledata,andlettheportaldotheorchestrationofdataplacement

4/15/1715

ScalabilityExample– PetascaleDTNProject

10.0 Gbps

17.6 Gbps

14.8 Gbps

19.3 Gbps

17.4 Gbps 17.0 Gbps

32.4 Gbps

25.3 Gbps

18.3 Gbps

16.3 Gbps

24.1 Gbps

24.0 Gbps

DTN

DTN

DTN

DTN

alcf#dtn_miraALCF

nersc#dtnNERSC

olcf#dtn_atlasOLCF

ncsa#BlueWatersNCSA

Data set: L380Files: 19260Directories: 211Other files: 0Total bytes: 4442781786482 (4.4T bytes)Smallest file: 0 bytes (0 bytes)Largest file: 11313896248 bytes (11G bytes)Size distribution:

1 - 10 bytes: 7 files10 - 100 bytes: 1 files100 - 1K bytes: 59 files1K - 10K bytes: 3170 files10K - 100K bytes: 1560 files100K - 1M bytes: 2817 files1M - 10M bytes: 3901 files10M - 100M bytes: 3800 files100M - 1G bytes: 2295 files1G - 10G bytes: 1647 files10G - 100G bytes: 3 files

March 2017L380 Data Set

4/15/1716

LinksandLists

– ESnetfasterdataknowledgebase• http://fasterdata.es.net/

– ScienceDMZpaper• http://www.es.net/assets/pubs_presos/sc13sciDMZ-final.pdf

– ScienceDMZemaillist• Sendmailtosympa@lists.lbl.gov withsubject"subscribeesnet-sciencedmz”

– perfSONAR• http://fasterdata.es.net/performance-testing/perfsonar/• http://www.perfsonar.net

– Globus• https://www.globus.org/

17 – ESnet Science Engagement (engage@es.net) - 4/15/17 ©2015,EnergySciencesNetwork

Thanks!

EliDartdart@es.netEnergySciencesNetwork(ESnet)LawrenceBerkeleyNationalLaboratory

http://fasterdata.es.net/

http://my.es.net/

http://www.es.net/

ExtraSlides

4/15/1719

DTNClusterDetail

10GE10GE

10GE10GE

10GE

10GE

Border Router

WAN

Science DMZSwitch/Router

Firewall

Enterprise

perfSONAR

perfSONAR

10GE10GE

10GE

10GE

10GE10GE

DTN

DTN

Filesystem

HEAD

“Sealed” DTNs(Globus only, no

shell access)

ClusterHead/Login

Nodes

DTN

DTN

Cluster compute nodes

HEAD

perfSONAR

Configure as DTN Cluster

4/15/1720

DTNClusterDesign

• ConfigureallfourDTNsasasingleGlobusendpoint– Globushasdocsonhowtodothis– https://support.globus.org/entries/71011547-How-do-I-add-multiple-I-O-nodes-to-a-Globus-endpoint-

• Recentoptionsforincreasedperformance– Useadditionalparallelconnections– DistributetransfersacrossmultipleDTNs(GlobusI/ONodes)– Critical– onlydothiswhenallDTNsintheendpointmountthesamesharedfilesystem

• UsetheGlobusCLIcommandendpoint-modify – Usethe--network-useoption– Adjustsconcurrencyandparallelism– Moreinfoatglobus.org (http://dev.globus.org/cli/reference/endpoint-modify/)

4/15/1721