Beyond Earth Simulator

海洋研究開発機構

地球情報基盤センター

情報システム部

塚越 眞

September 11, 2017Makoto Tsukakoshi

Information Systems Department

Center for Earth Information Science and Technology

JAMSTEC

International Computing for the Atmospheric Sciences Symposium (iCAS2017)

Japan Agency for Marine-Earth Science and Technology

2

The main seven research and development issues during the third mid-term plan

During the third mid-term plan, we set and address the seven research and development issues with all our strength due to promote strategic and focused research and development

based on the national and social needs.

Exploring untapped

submarine resources

Detecting signals of global environmental change

Understanding seismogeniczones, and contributing to disaster mitigation

Marine Bioscience -Exploring the unknown extreme biosphere to solve the mystery of life

Ocean drilling –Getting to know the Earth from beneath the seabed

Information Science -

Predicting the Earth's

future by simulations

Construction of research base

to spawn the ocean frontier3

Typhoon Vera in 1959 by Kazuhisa Tsuboki (2015)Typhoon-Ocean Interaction Study Using the Coupled Atmosphere-Ocean Non-hydrostatic Model:With Careful Consideration of Upper Outflow Layer Clouds of Typhoon

Typhoon-marine interaction using nonstatic atmospheric wave ocean coupling model

4

5

• Earth Simulator ：- Developed by NASDA, JAERI and JAMSTEC (563.4* Oku Yen)- Operation started on March 2002 and immediately recognized as #1 supercomputer- “It will keep #1 for 2 years in peak and 5 years in effective performance” by H. Miyoshi

*inclusing facilities and building

Earth Simulator has changed its rolewhen “K computer” is completed

既存のスーパーコンピュータは、本プロジェクトで開発する計算機システムの完成後も、例えば地球シミュレータについて

は、これまでに蓄積してきたアプリケーションソフトウェアを活用して効果的・効率的に海洋地球科学分野を主とした計算を

担当するなど、それぞれが得意とする各分野を中心として、目的や性能に応じた計算に応するとの役割分担の下に、今後

も増大が予想される我が国の計算機需要に応えることとしている。

Council for Science and Technology Policy (CSTP) Report on Nov 25, 2005:

After the completion of Leading Edge High Performance Supercomputer (i.e. K-Computer), other existing supercomputers should carry their roles in each fields, according to their purpose and performance , for example, Earth Simulator, should responsible for Marine and Earth Science on the base of accumulated application software assets, and thus they should meet growing demand for HPC in our country.

Operated as national infrastructure thorugh HPCI*

＊ “Innovative High Performance Computing Infrastructure” initiative

ES

ES

K-computerHPCI Strategy - MEXT

6

March 2002Earth Simulator (1st)40TFLOPS / 10TB

1 TFLOPS

10 TFLOPS

100 TFLOPS

1 PFLOPS

10 PFLOPS

March 2009Earth Simulator (ES2)131TFLOPS / 20TB

Peak Performance ３x

* Single Program Turn Around Time Performance: 2.1x to 2.8x

Number of Programs: 4x= System Throughput 8x to 10 x

Foundation of Marine- Earth SimulationProof of Feasibility of

Simulation and Prediction

“Computenik” (Jack Donguarra,NY Times, April 20, 2002)

System for Marine-Earth Science

System for prediction and creation ofMarine-Earth Science data and information

for Society

8x to 10x Effective Performance based on Application Programs*

7

June 2015Earth Simulator1.3PFLOPS / 320TBPeak/Effective Performance 10x

3 Generation of Earth Simulator since 2002

About 5MW*

* Effective Power Consumption

About 3MW*

About 1.5MW*

Utilize Simulation, Prediction and data for SocietyChallenge new subjects in Marine-Earth Science

Flagship supercomputerfor All Science

8

JAMSTEC Information System Outline

Earth Simulator(NEC SX-ACE)

Large Scale Shared Memory Server （SGI UV2000)

Super computer systems(ICE-X, UV2000, SX-9)

10Gbps ×2JAMSTEC backbone network

Earth Simulator backbone

SINETScience Information NETwork 4

（1G～10Gbps）

40Gbps ×2Visualization Server

User terminals

Mass Storage System

User terminals

Service servers

(HPE Apollo 6000, 380 node in 2018)

Earth Simulator Outline

Work

13.5 PB

/home87.6 TB

/data

4.7 PB

4 servers

Main Computer

Computing Nodes (5120) + Internode Network

Storage

Model: NEC SX-ACETotal Performance

- Number of nodes 5,120- Peak Performance 1.31 PFLOPS- Memory Bandwidth 1.31PB/s- Memory size 320 TB

Node Performance- Number of CPU 1(4cores)- Peak Performance 256 GFLOPS

(64GFLOPS x 4cores) Double Precision

- Memory Bandwidth 256 GB/s- Memory size 64 GBMass Storage System

JAMSTEC Backbone network

Frontend Server

Large Scale Shared Memory Server（UV2000)

Pre and Post System

17PB

Earth Simulator

9

- Memory Bandwidth 1.31PB/s

温暖化 海洋/同化温暖化 地震 地震 ミクロ大気全球大気

Average (ASIS) 2.1

Average (TUNE) 2.8

Tuning

Benchmark Test showed ES’ｓ value

The same #CPU (sockets) DOUBLES the performance with ASIS code.

2.8X with further tuning.

MIROC MIROC NICAM Specfem3D RSGDX MSSG ODA

Both ES2 and SX-ACE deploy same #CPU

(Performance Ratio vs. ES2)

This means : Effective Throughput Performance is > 10x(TAT – 2 to 2.8x, #of jobs 4x)

10

Indeed, FLOPs counts increased by 12x260 EFLOPs(executed in FY2012) to3,119EFLOPs(executed in FY2016)

Earth Science（18.8%）

Engineering(22.0%)

BiologyLife Science

(15.5%)

Atomic Energy Fusion(1.0%)

Mathematics(1.7%)

Particle PhysicsAstronomy(16.4%)

Material ScienceChemistry(24.3%)

Others(0.3%)

K Computer Usage in FY2015

Equivalent to 28% of Next Earth Simulator

Earth Science90.3%

Effective Resource Comparison with K-computer

Earth Simulator in FY2016

~28%

~2/3(estimation)

11

Earth Simulator Resource Allocation/Usage（FY2016）

Earth Simulator Resource Allocation (71 projects)

[node hour]

40%

10%

30%

20%

JAMSTEC proposed project (23)

Strategic project withspecial support (6)

Fee based usage(8)

Government contract (7)

External proposed project (27)

70.0%

20.3%

5.4%3.7% 0.7%

Earth Simulator Usage by Application [node hour]

Environmental Change

Earth Internal,Earthquake, Tsunami

Math Sci & Engineering

Industry &Innovation

Life Science

Institutes using Earth Simulator

Outside Japan

Industry

Government Agency

University

12

• Stable Operation in FY2016

- Availability 99.86%,

- Utilization 89.07%

- Total 3,119EFLOPs * executed in FY2016

(7.5% of total peak performance :

24x366xall nodes)

Earth Simulator Operation Highlights

• Power Efficiency- Improved 30% from the previous ES2(NEC SX-9) - Continued effort to optimized the cooling “Total Executed FLOPs/Total Power*”

*:including facilities):

171TFLOPs/kwh in 2016- improved 9.2% from 2015

*Using 2016 Yellowstone CPU hours/24x366 (81%), 3,119EF requires 2.3% of peak performance on Cheyanne

13

(proposal) Use actual FLOPs per watt to evaluate total energy efficiency of computing centers

Efficiency should be “actually achieved Performance/Total energy consumed” Peak or Linpack Performance : Don’t represent real work loads (“Green500”) HPCG or other Benchmark suites : Better but do not fit each center’s AP

spectrum (each centers should have been optimized to its own application work loads.)

Benchmarks show only capability. Do not represent the center’s operational efforts.

Power consumption must be actual and total include cooling.

Evaluate the energy efficiency by “Executed FLOPs per watt” ! Where, the energy is measured total inclusive amount.(Example)In 2016 (JAN-DEC), Earth Simulator (Entire System including cooling facilities) has actually achieved 178,956 GFLOPS/Kwh (7.6% of theoretical peak x 100% non-stop operation)Using HPCG/peak performance ratio (5.2%) and annual GWh from the annual report 2014, efficiency of “K-Computer” is estimated 109,221 GFLOPS/Kwh

14

Goal: “Marine-Earth Informatics”

1. Produce large and accurate datadata from observation, experimentation, simulationdata of various form (structured/unstructured, images, texts)

• Building large and precise data sets by accurate observation and simulation applying the latest data handling method.

2. Create information from large and diverse data• Development of a computing method to deal with large data with

high performance• Applying ML, pattern recognition, analysis of precursor

characteristic of rare events

3. Present new application and use of information• New use case of information for government and industry• New application area

4. Develop ICT technology to handle large and complex data• Integrating Simulation and Data Analysis• Interactive real time visualization• Leading-edge High performance platforms and devices

15

Historical Simulations6,000(AGCM)+3,000(RCM) years

database for Policy Decision making for Future climate change (d4PDF)• d4PDF is a database including both historical and +4K future climate simulation results,

and provides probabilistic Information on climate change in extreme events by high-resolution large ensemble simulations with a 60km AGCM and 20km NHRCM.

• 2.6M node hours* and 2PB output on Earth Simulator in 2015 by MRI and MIROC team.*9% of entire resources in 2015

MRI-AGCM60km

MRI-NHRCM

20km

downscaling

・60 years (1950-2010)・100 members (AGCM)・50 members (NHRCM)(SST perturbations representing observational error)

+4K Future Climate Simulations

5,400 years (AGCM+RCM)

・60 years (1950-2010)・90 members (AGCM)・90 members (NHRCM)(6 ΔSST from 6 CMIP5 models x 15 SST perturbations)

Histogram of daily mean precipitation at Tokyo (60km AGCM)

(a) Present day (b) Changes in the +4K world

Daily precipitation (mm/day) Daily precipitation (mm/day)

Shiogama,H. (2016) Database for Policy Decision making for Future climate change (d4PDF)

Freq

uen

cy(%

)

Rat

io o

f fr

eq. (

+4K

/pre

sen

t)

10yr

100yr

16

•First-ever dataset covering the western North Pacific (15-65N, 117E -160W) over the last three decades (1982-2014) at eddy-resolving (about 10km) resolution.•Produced by 4-dimensional variational ocean data assimilation system, MOVE-4DVAR developed in JMA/MRI (Usui et al., 2015).

- 0.1°latitude × 0.1°longitude - vertical 54 layers（0-6300m depth）- temperature and salinity profiles above 1500m-depth, gridded sea surface temperature (SST)

and satellite altimeter-derived sea surface height (SSH) are assimilated•Provides accurate estimation of ocean environmental changes around Japan.•Available as a basic ocean reanalysis dataset for oceanography, climatology, meteorology, and fisheries.

http://synthesis.jamstec.go.jp/FORA/e/index.html

Four-dimensional Variational Ocean ReAnalysis for the Western North Pacific over 30 years (FORA-WNP30)

17

Advancement of prediction of tropical cyclone (TC) generation by deep learning

Example

（Currently) prediction two days before its generation by real-time simulation

and satellite observation data

Egg of TC

Egg of TC

not TC

Learning feature quantities of “egg” of TCs from simulation data

Applying to observation data

(Target) prediction 3 to 7 days before its generation by machine learning

Yamaguchi et al., 2016

Total >10,000,000 images + 2535 typhoon tracking data generated by NICAM simulation

Egg of TCs

TCs

Ex-TCs

Not TCs

Matsuoka et al., 7th International Workshop on Climate Informatics, Boulder, USA, Sep 2017 18

next “SC” system – complements ES /meets growing needs

Earth Simulator

[High-end Supercomputer]

Next “SC” system

[Standard Linux Cluster]Standard CPU, Considerable computing loads, Commercial APs, Community codes

Growing& Emerging

Needs

Interoperability:job, file,

accounting etc.

SIMULATIONANALYSIS

• Enhance existing functions - pre and post processing, data analysis, hosting community APs• Flexible to meet the emerging and growing needs from various scientific approach• Capable to host project servers in the future

Custom CPU, High memory bandwidth, High vector performance, Proprietary codes

Used for:•Community codes on standard Linux clusters.•Programs of integer and Logical operation,• large memory space or high i/o performance.•Bio informatics. •Earth informatics. •Big data analysis.•Real time computing for observation and ship-operation.• Industry applications.

Integrated use with Earth Simulator:(example)•run reginal model downscaling from global model on ES•run short term forecast while ES runs seasonal forecast

19

ES/SC Administration Server

• User I/F for both information & operation

• Autonomous data search & retrieve• Interface to Web Service

Meta D/B（data directory）

Data Management Layer Computing LayerUser/Service Layer

Internet

Cyber Manager

Data AnalysisData Management Server

Database Server Grand Challenge Simulation

JAMSTEC Future Platform

Mass Storage System

Backbone Network 100GbE

SINET5 Network Service

Frontend Servers

Internet

From Data, Create and Supply Information

VIsualization

20

Internet

Research Institutes

Users

Smartphones,Sensors

Images, locations

Various networks

Edge/Cloud ComputingPlatform

Edge Server

Data

Internet

Industry

Administration

JAMSTEC Future PlatformFrom Data, Create and Supply Information

Data Management Layer Computing LayerUser/Service Layer

Network Service

21

ES/SC Administration Server

Data AnalysisGrand Challenge Simulation

Backbone Network 100GbE

VIsualization

JAMSTEC Future PlatformFrom Data, Create and Supply Information

Internet

Research Institutes

Users

Smartphones,Sensors

Images, locations

Various networks

Edge/Cloud ComputingPlatform

Edge Server

Data

Internet

Industry

Administration

Data Management Layer Computing LayerUser/Service Layer

Network Service

22

Search

RemoteObservation

Backbone Network 100GbE

JAMSTEC Future PlatformFrom Data, Create and Supply Information

Internet

Research Institutes

Users

Smartphones,Sensors

Images, locations

Various networks

Edge/Cloud ComputingPlatform

Edge Server

Data

Internet

Industry

Administration

Data Management Layer Computing LayerUser/Service Layer

Network Service

Data handlingSimulation,

Analysis

SINET5

leased line

23

Search

RemoteObservation

Data handlingSimulation,

Analysis

SINET5

leased line

Backbone Network 100GbE

JAMSTEC Future PlatformFrom Data, Create and Supply Information

Research Institutes

Users

Smartphones,Sensors

Images, locations

Edge/Cloud ComputingPlatform

Edge Server

Data

Internet

Industry

Administration

Data Management Layer Computing LayerUser/Service Layer

• User I/F for both information & operation

• Autonomous data search & retrieve• Interface to Web Service

Meta D/B（data directory）

Data Management ServerDatabase Server

Mass Storage System

Frontend Servers

Internet

Various networks

Network Service

Cyber Manager

24

(1) Curationcurate>200TB deep sea image data

VisualizationLarge data handling

Various Database

Collaboration

Large Simulation

Data Analysis

JAMSTEC Future PlatformFrom Data, Create and Supply Information

(2) Data Managementprocess and manage observation data from >100 research cruises per year

(3) Integration of observation and simulationcreate new data sets by integrating observation and simulation data

(4) HPC TechnologyPFLOPS class supercomputers for large scale simulation, data analysis and machine learning

Information

25

(1) Curationcurate>200TB deep sea image data

VisualizationLarge data handling

Various Database

Collaboration

Large Simulation

JAMSTEC Future PlatformFrom Data, Create and Supply Information

(2) Data Managementprocess and manage observation data from >100 research cruises per year

(3) Integration of observation and simulationcreate new data sets by integrating observation and simulation data

(4) HPC TechnologyPFLOPS class supercomputers for large scale simulation, data analysis and machine learning

Data Analysis

Information

26

ES/SC Administration Server

• User I/F for both information & operation

• Autonomous data search & retrieve• Interface to Web Service

Meta D/B（data directory）

User/Service Layer

Internet

Cyber Manager

Data AnalysisData Management Server

Database Server Grand Challenge Simulation

JAMSTEC Future Platform

Mass Storage System

Backbone Network 100GbE

SINET5

Frontend Servers

From Data, Create and Supply Information

VIsualization

Internet

Research Institutes

Users

Smartphones,Sensors

Images, locations

Various networks

Edge/Cloud ComputingPlatform

Edge Server

Data

Internet

Industry

Administration

Network Service

可視化情報処理巨大データ圧縮技術

連携

Data handlingSimulation,

Analysis

SINET5

leased line

RemoteObservation

(1) Curation (2) Data Management

Data Management Layer Computing Layer

(3) Integration of observation and simulation

(4) HPC Technology

Information

27

Summary

• “Earth Simulator”, once flagship computer, is renewed and providing major computing resources for earth science in Japan.

• CEIST, Center for Earth Information Science and Technology, JAMSTEC, is widening the goals – focusing data and information.

• The future platform will be integrated and interoperable system consist of diversedlayers.

28

Thank you for listening !