GERSolar - Eloyecolell.github.io/assets/slides/streamprocessing.pdf · 2020. 6. 21. · CentroTIC....
Transcript of GERSolar - Eloyecolell.github.io/assets/slides/streamprocessing.pdf · 2020. 6. 21. · CentroTIC....
![Page 2: GERSolar - Eloyecolell.github.io/assets/slides/streamprocessing.pdf · 2020. 6. 21. · CentroTIC. 12 GB RAM 2.880 CUDA Cores a 745 Mhz . 2 CPUs x 6 Cores a 2,3 Ghz. 256 GB RAM. 500](https://reader034.fdocument.pub/reader034/viewer/2022052006/601a5d3e4733844b71235b87/html5/thumbnails/2.jpg)
![Page 3: GERSolar - Eloyecolell.github.io/assets/slides/streamprocessing.pdf · 2020. 6. 21. · CentroTIC. 12 GB RAM 2.880 CUDA Cores a 745 Mhz . 2 CPUs x 6 Cores a 2,3 Ghz. 256 GB RAM. 500](https://reader034.fdocument.pub/reader034/viewer/2022052006/601a5d3e4733844b71235b87/html5/thumbnails/3.jpg)
![Page 4: GERSolar - Eloyecolell.github.io/assets/slides/streamprocessing.pdf · 2020. 6. 21. · CentroTIC. 12 GB RAM 2.880 CUDA Cores a 745 Mhz . 2 CPUs x 6 Cores a 2,3 Ghz. 256 GB RAM. 500](https://reader034.fdocument.pub/reader034/viewer/2022052006/601a5d3e4733844b71235b87/html5/thumbnails/4.jpg)
![Page 5: GERSolar - Eloyecolell.github.io/assets/slides/streamprocessing.pdf · 2020. 6. 21. · CentroTIC. 12 GB RAM 2.880 CUDA Cores a 745 Mhz . 2 CPUs x 6 Cores a 2,3 Ghz. 256 GB RAM. 500](https://reader034.fdocument.pub/reader034/viewer/2022052006/601a5d3e4733844b71235b87/html5/thumbnails/5.jpg)
![Page 6: GERSolar - Eloyecolell.github.io/assets/slides/streamprocessing.pdf · 2020. 6. 21. · CentroTIC. 12 GB RAM 2.880 CUDA Cores a 745 Mhz . 2 CPUs x 6 Cores a 2,3 Ghz. 256 GB RAM. 500](https://reader034.fdocument.pub/reader034/viewer/2022052006/601a5d3e4733844b71235b87/html5/thumbnails/6.jpg)
![Page 7: GERSolar - Eloyecolell.github.io/assets/slides/streamprocessing.pdf · 2020. 6. 21. · CentroTIC. 12 GB RAM 2.880 CUDA Cores a 745 Mhz . 2 CPUs x 6 Cores a 2,3 Ghz. 256 GB RAM. 500](https://reader034.fdocument.pub/reader034/viewer/2022052006/601a5d3e4733844b71235b87/html5/thumbnails/7.jpg)
![Page 8: GERSolar - Eloyecolell.github.io/assets/slides/streamprocessing.pdf · 2020. 6. 21. · CentroTIC. 12 GB RAM 2.880 CUDA Cores a 745 Mhz . 2 CPUs x 6 Cores a 2,3 Ghz. 256 GB RAM. 500](https://reader034.fdocument.pub/reader034/viewer/2022052006/601a5d3e4733844b71235b87/html5/thumbnails/8.jpg)
utopía
predecir
la irradiancia solar
1 hora antes
para cada punto de la argentina
con refrescos de 30 min
![Page 9: GERSolar - Eloyecolell.github.io/assets/slides/streamprocessing.pdf · 2020. 6. 21. · CentroTIC. 12 GB RAM 2.880 CUDA Cores a 745 Mhz . 2 CPUs x 6 Cores a 2,3 Ghz. 256 GB RAM. 500](https://reader034.fdocument.pub/reader034/viewer/2022052006/601a5d3e4733844b71235b87/html5/thumbnails/9.jpg)
1. Descargar imágenes.
2. Obtener mediciones de tierra.
3. Estimar utilizando heliosat2 (GPU).
4. Calcular errores contra las mediciones en tierra
(CPU).
procesos
![Page 10: GERSolar - Eloyecolell.github.io/assets/slides/streamprocessing.pdf · 2020. 6. 21. · CentroTIC. 12 GB RAM 2.880 CUDA Cores a 745 Mhz . 2 CPUs x 6 Cores a 2,3 Ghz. 256 GB RAM. 500](https://reader034.fdocument.pub/reader034/viewer/2022052006/601a5d3e4733844b71235b87/html5/thumbnails/10.jpg)
1. calibración de las imágenes (paquete de 1 mes).
2. estimar la irradiancia global a nivel del suelo
para cada pixel.
heliosat2 (GPU)
![Page 11: GERSolar - Eloyecolell.github.io/assets/slides/streamprocessing.pdf · 2020. 6. 21. · CentroTIC. 12 GB RAM 2.880 CUDA Cores a 745 Mhz . 2 CPUs x 6 Cores a 2,3 Ghz. 256 GB RAM. 500](https://reader034.fdocument.pub/reader034/viewer/2022052006/601a5d3e4733844b71235b87/html5/thumbnails/11.jpg)
area geográfica
3.544 pixels
2.26
7 p
ixel
s
data
![Page 12: GERSolar - Eloyecolell.github.io/assets/slides/streamprocessing.pdf · 2020. 6. 21. · CentroTIC. 12 GB RAM 2.880 CUDA Cores a 745 Mhz . 2 CPUs x 6 Cores a 2,3 Ghz. 256 GB RAM. 500](https://reader034.fdocument.pub/reader034/viewer/2022052006/601a5d3e4733844b71235b87/html5/thumbnails/12.jpg)
area geográfica
3.544 pixels
2.26
7 p
ixel
s
data
lonlat
![Page 13: GERSolar - Eloyecolell.github.io/assets/slides/streamprocessing.pdf · 2020. 6. 21. · CentroTIC. 12 GB RAM 2.880 CUDA Cores a 745 Mhz . 2 CPUs x 6 Cores a 2,3 Ghz. 256 GB RAM. 500](https://reader034.fdocument.pub/reader034/viewer/2022052006/601a5d3e4733844b71235b87/html5/thumbnails/13.jpg)
area geográfica
3.544 pixels
2.26
7 p
ixel
s
data
lonlat
Ej: 353,23134 W/(m2 * sr)
![Page 14: GERSolar - Eloyecolell.github.io/assets/slides/streamprocessing.pdf · 2020. 6. 21. · CentroTIC. 12 GB RAM 2.880 CUDA Cores a 745 Mhz . 2 CPUs x 6 Cores a 2,3 Ghz. 256 GB RAM. 500](https://reader034.fdocument.pub/reader034/viewer/2022052006/601a5d3e4733844b71235b87/html5/thumbnails/14.jpg)
area geográfica
3.544 pixels
2.26
7 p
ixel
s
data
lonlatEj: -45,23131417º (Sur)
![Page 15: GERSolar - Eloyecolell.github.io/assets/slides/streamprocessing.pdf · 2020. 6. 21. · CentroTIC. 12 GB RAM 2.880 CUDA Cores a 745 Mhz . 2 CPUs x 6 Cores a 2,3 Ghz. 256 GB RAM. 500](https://reader034.fdocument.pub/reader034/viewer/2022052006/601a5d3e4733844b71235b87/html5/thumbnails/15.jpg)
area geográfica
3.544 pixels
2.26
7 p
ixel
s
data
lonlat Ej: -55,23131417º (Oeste)
![Page 16: GERSolar - Eloyecolell.github.io/assets/slides/streamprocessing.pdf · 2020. 6. 21. · CentroTIC. 12 GB RAM 2.880 CUDA Cores a 745 Mhz . 2 CPUs x 6 Cores a 2,3 Ghz. 256 GB RAM. 500](https://reader034.fdocument.pub/reader034/viewer/2022052006/601a5d3e4733844b71235b87/html5/thumbnails/16.jpg)
pixel
data calibrada: float 32 bits: W/m2
4 bytes/px
1 pixel
1 p
ixel
data
![Page 17: GERSolar - Eloyecolell.github.io/assets/slides/streamprocessing.pdf · 2020. 6. 21. · CentroTIC. 12 GB RAM 2.880 CUDA Cores a 745 Mhz . 2 CPUs x 6 Cores a 2,3 Ghz. 256 GB RAM. 500](https://reader034.fdocument.pub/reader034/viewer/2022052006/601a5d3e4733844b71235b87/html5/thumbnails/17.jpg)
pixel
lat: float 32 bits: grados4 bytes/px
lon: float 32 bits: grados4 bytes/px
1 pixel
1 p
ixel
1 pixel
1 p
ixel
lat lon
![Page 18: GERSolar - Eloyecolell.github.io/assets/slides/streamprocessing.pdf · 2020. 6. 21. · CentroTIC. 12 GB RAM 2.880 CUDA Cores a 745 Mhz . 2 CPUs x 6 Cores a 2,3 Ghz. 256 GB RAM. 500](https://reader034.fdocument.pub/reader034/viewer/2022052006/601a5d3e4733844b71235b87/html5/thumbnails/18.jpg)
pixel
data calibrada + lat + lon4 bytes/px * 3
12 bytes/px
1 pixel
1 p
ixel
1 pixel
1 p
ixel
1 pixel
1 p
ixel
lat londata
![Page 19: GERSolar - Eloyecolell.github.io/assets/slides/streamprocessing.pdf · 2020. 6. 21. · CentroTIC. 12 GB RAM 2.880 CUDA Cores a 745 Mhz . 2 CPUs x 6 Cores a 2,3 Ghz. 256 GB RAM. 500](https://reader034.fdocument.pub/reader034/viewer/2022052006/601a5d3e4733844b71235b87/html5/thumbnails/19.jpg)
imagen
8.034.248 px/imagen
12 bytes/px * 8.034.248 px/imagen
97 MB/imagen
3.544 pixels
2.26
7 p
ixel
s
![Page 20: GERSolar - Eloyecolell.github.io/assets/slides/streamprocessing.pdf · 2020. 6. 21. · CentroTIC. 12 GB RAM 2.880 CUDA Cores a 745 Mhz . 2 CPUs x 6 Cores a 2,3 Ghz. 256 GB RAM. 500](https://reader034.fdocument.pub/reader034/viewer/2022052006/601a5d3e4733844b71235b87/html5/thumbnails/20.jpg)
paquete
2 imagen/hora * 12 hora luz/dia * 30 dia/mes
720 imagen/mes
97 MB/imagen * 720 imagen/mes
70 GB/paquete
![Page 21: GERSolar - Eloyecolell.github.io/assets/slides/streamprocessing.pdf · 2020. 6. 21. · CentroTIC. 12 GB RAM 2.880 CUDA Cores a 745 Mhz . 2 CPUs x 6 Cores a 2,3 Ghz. 256 GB RAM. 500](https://reader034.fdocument.pub/reader034/viewer/2022052006/601a5d3e4733844b71235b87/html5/thumbnails/21.jpg)
stream de entrada
70 GB/30 min
2,3 GB/min
![Page 22: GERSolar - Eloyecolell.github.io/assets/slides/streamprocessing.pdf · 2020. 6. 21. · CentroTIC. 12 GB RAM 2.880 CUDA Cores a 745 Mhz . 2 CPUs x 6 Cores a 2,3 Ghz. 256 GB RAM. 500](https://reader034.fdocument.pub/reader034/viewer/2022052006/601a5d3e4733844b71235b87/html5/thumbnails/22.jpg)
stream de salida
8034248 px/imagen * 4 bytes/px
321 MB/30min
10,7 MB/min
![Page 23: GERSolar - Eloyecolell.github.io/assets/slides/streamprocessing.pdf · 2020. 6. 21. · CentroTIC. 12 GB RAM 2.880 CUDA Cores a 745 Mhz . 2 CPUs x 6 Cores a 2,3 Ghz. 256 GB RAM. 500](https://reader034.fdocument.pub/reader034/viewer/2022052006/601a5d3e4733844b71235b87/html5/thumbnails/23.jpg)
hardware
CentroTIC
![Page 24: GERSolar - Eloyecolell.github.io/assets/slides/streamprocessing.pdf · 2020. 6. 21. · CentroTIC. 12 GB RAM 2.880 CUDA Cores a 745 Mhz . 2 CPUs x 6 Cores a 2,3 Ghz. 256 GB RAM. 500](https://reader034.fdocument.pub/reader034/viewer/2022052006/601a5d3e4733844b71235b87/html5/thumbnails/24.jpg)
12 GB RAM2.880 CUDA Cores a 745 Mhz
![Page 25: GERSolar - Eloyecolell.github.io/assets/slides/streamprocessing.pdf · 2020. 6. 21. · CentroTIC. 12 GB RAM 2.880 CUDA Cores a 745 Mhz . 2 CPUs x 6 Cores a 2,3 Ghz. 256 GB RAM. 500](https://reader034.fdocument.pub/reader034/viewer/2022052006/601a5d3e4733844b71235b87/html5/thumbnails/25.jpg)
2 CPUs x 6 Cores a 2,3 Ghz
![Page 26: GERSolar - Eloyecolell.github.io/assets/slides/streamprocessing.pdf · 2020. 6. 21. · CentroTIC. 12 GB RAM 2.880 CUDA Cores a 745 Mhz . 2 CPUs x 6 Cores a 2,3 Ghz. 256 GB RAM. 500](https://reader034.fdocument.pub/reader034/viewer/2022052006/601a5d3e4733844b71235b87/html5/thumbnails/26.jpg)
256 GB RAM
![Page 27: GERSolar - Eloyecolell.github.io/assets/slides/streamprocessing.pdf · 2020. 6. 21. · CentroTIC. 12 GB RAM 2.880 CUDA Cores a 745 Mhz . 2 CPUs x 6 Cores a 2,3 Ghz. 256 GB RAM. 500](https://reader034.fdocument.pub/reader034/viewer/2022052006/601a5d3e4733844b71235b87/html5/thumbnails/27.jpg)
500 GB Disk
![Page 28: GERSolar - Eloyecolell.github.io/assets/slides/streamprocessing.pdf · 2020. 6. 21. · CentroTIC. 12 GB RAM 2.880 CUDA Cores a 745 Mhz . 2 CPUs x 6 Cores a 2,3 Ghz. 256 GB RAM. 500](https://reader034.fdocument.pub/reader034/viewer/2022052006/601a5d3e4733844b71235b87/html5/thumbnails/28.jpg)
x7
![Page 29: GERSolar - Eloyecolell.github.io/assets/slides/streamprocessing.pdf · 2020. 6. 21. · CentroTIC. 12 GB RAM 2.880 CUDA Cores a 745 Mhz . 2 CPUs x 6 Cores a 2,3 Ghz. 256 GB RAM. 500](https://reader034.fdocument.pub/reader034/viewer/2022052006/601a5d3e4733844b71235b87/html5/thumbnails/29.jpg)
24 TB Disk (12 TB con RAID)
![Page 30: GERSolar - Eloyecolell.github.io/assets/slides/streamprocessing.pdf · 2020. 6. 21. · CentroTIC. 12 GB RAM 2.880 CUDA Cores a 745 Mhz . 2 CPUs x 6 Cores a 2,3 Ghz. 256 GB RAM. 500](https://reader034.fdocument.pub/reader034/viewer/2022052006/601a5d3e4733844b71235b87/html5/thumbnails/30.jpg)
desafíos
estrategia de tiles o mosaicos
normalización de mediciones en tierra
distribución de carga en el Cluster
optimizaciones en GPU
generar una arquitectura de procesamiento
![Page 31: GERSolar - Eloyecolell.github.io/assets/slides/streamprocessing.pdf · 2020. 6. 21. · CentroTIC. 12 GB RAM 2.880 CUDA Cores a 745 Mhz . 2 CPUs x 6 Cores a 2,3 Ghz. 256 GB RAM. 500](https://reader034.fdocument.pub/reader034/viewer/2022052006/601a5d3e4733844b71235b87/html5/thumbnails/31.jpg)
codigo fuente
repositorios github
gersolar/netcdf
gersolar/noaaclass
ecolell/goesdb
rossant/playdoh
ecolell/solar_radiation_model
![Page 32: GERSolar - Eloyecolell.github.io/assets/slides/streamprocessing.pdf · 2020. 6. 21. · CentroTIC. 12 GB RAM 2.880 CUDA Cores a 745 Mhz . 2 CPUs x 6 Cores a 2,3 Ghz. 256 GB RAM. 500](https://reader034.fdocument.pub/reader034/viewer/2022052006/601a5d3e4733844b71235b87/html5/thumbnails/32.jpg)
¿preguntas?
![Page 33: GERSolar - Eloyecolell.github.io/assets/slides/streamprocessing.pdf · 2020. 6. 21. · CentroTIC. 12 GB RAM 2.880 CUDA Cores a 745 Mhz . 2 CPUs x 6 Cores a 2,3 Ghz. 256 GB RAM. 500](https://reader034.fdocument.pub/reader034/viewer/2022052006/601a5d3e4733844b71235b87/html5/thumbnails/33.jpg)
gracias