Post on 03-Jan-2016
1
www.lpds.sztaki.hu/pgportalpgportal@lpds.sztaki.hu
Advanced features of the P-GRADE Advanced features of the P-GRADE portalportal
Peter Kacsuk, Gergely SiposPeter Kacsuk, Gergely Sipos MTA SZTAKI
Tamás KissUniversity of Westminster
2
ContentsContents
• Solving Grid interoperability
• Supporting collaborative Grid application development
• Classification of Grid portals
4
Grid interoperability 1Grid interoperability 1
• P-GRADE portal supports job submission in 2nd generation Grids: – LCG – EGEE– GT2 – UK NGS and US OSG– NorduGrid
• P-GRADE Portal enables:– Connecting 2nd generation production Grids
5
User
P-GRADE Portal
v2.3
SZTAKI Portal Server
Connecting 2Connecting 2ndnd generation production Grids generation production Grids
LCG-2broker
Certificate server
Proxy
Workflow
Manchester
Leeds
UK NGS GT2
EGEE-VOCE LCG-2
Job
Job
Job
Job
Budapest
Poznan
Brno
Supports both direct and brokered job submission
6
Grid interoperability 2Grid interoperability 2
• P-GRADE Portal with GEMLCA supports legacy code service invocation in 2nd and 3rd generation Grids:– LCG, gLite – EGEE– GT2, GT4 – UK NGS
• Combination of jobs and legacy code services enables P-GRADE Portal to support:– Extending 2nd generation Grids with legacy code
services
7
Extending 2Extending 2ndnd generation Grids with generation Grids with legacy code serviceslegacy code services
Job Repository Job Repository (GEMLCA)(GEMLCA)
ManchesterUser
Leeds
GEMLCAP-GRADE
Portal
UoW Portal Server
Executable
NGS GT2
Poznan
Budapest
EGEE/VOCEExecutable
Executable
8
Grid interoperability 3Grid interoperability 3
• Combination of jobs and legacy code services enables P-GRADE Portal to support:– Transition from 2nd generation Grids to 3rd
generation Grids:• Extending 2nd generation production Grids with 3rd
generation volunteer Grid sites
• Connecting 2nd generation production Grids with 3rd generation volunteer Grids
– Connecting 2nd generation and 3rd generation production Grids
9
GEMLCA GEMLCA RepositoryRepository
Manchester
Westminster
User
Oxford
P-GRADE NGS
GEMLCA
Portal
UoW Portal Server
UoW site – GT4 with GEMLCA Service
Service Invocation
Executable
Executable
Executable
Extending 2Extending 2ndnd generation Grids with generation Grids with 33rdrd generation volunteer Grid sites generation volunteer Grid sites
UK NGS GT2
10
Extending the NGS with legacy code services Extending the NGS with legacy code services and GT4 Resourcesand GT4 Resources
GT2 Job submission to Rutherford
GT4 Service Invocation at UoW
GEMLCA Legacy code submitted
to Oxford
GT2 Job submission to Manchester
11
GEMLCA GEMLCA RepositoryRepository
Manchester
User
Leeds
P-GRADE NGS
GEMLCA
Portal
UoW Portal ServerNGS GT2 Grid
WestFocus GT4 Grid
UoW
Brunel
ServiceInvocation
Extending 2Extending 2ndnd generation production Grids generation production Grids with 3with 3rdrd generation volunteer Grids generation volunteer Grids
Executable
Executable
12
Connecting 2Connecting 2ndnd generation and generation and 33rdrd generation production Grids generation production Grids
Job Repository Job Repository (GEMLCA)(GEMLCA)
ManchesterUser
Leeds
GEMLCAP-GRADE
Portal
UoW Portal Server
Executable
NGS GT2
WestFocus GT4 Grid
UoW
Brunel
Poznan
Budapest
EGEE LCG/gLite
Executable
Executable
ServiceInvocation
Executable
13
Plan: Connecting major Grids by GEMLCA and the P-GRADE Portal
P-GRADE
GEMLCA
Portal
GEMLCA GEMLCA RepositoryRepository
In test phase Available
14
Regional coverage of Regional coverage of using P-GRADE Portalusing P-GRADE Portal
+ GILDAFor education & training
15
Conclusion:Conclusion:Easy-to-use, technology-neutral Easy-to-use, technology-neutral
Grid portal for e-ScientistsGrid portal for e-Scientists
• The P-GRADE Portal hides differences of Grids• Globus 2 and 4 – LCG2 – gLite Grid interoperability• Transparent switching between Grid technologies
• Graphical tools for application development, execution and monitoring
• Sequential & parallel components can be integrated into large Grid applications
• Direct or broker-based resource allocation• Your code does not have to contain grid specific calls
17
Grid usageGrid usage
Grid
Grid middleware and resources
Portal
Browser Special client
Grid portal
Executeapplication
Get results
Defineapplication
Gridapp.
18
The problemThe problem
• Grids have been designed for highly complex applications (The overhead is too big to execute small programs…)
• How can a single person define highly complex applications?
19
The solution today…The solution today…
Grid
Portal
Browser Special client
Executeapplication
Get results
Defineapplication
Gridapp.
E-mail, phone, skype, MSN, …
20
The solution today…The solution today…
Grid
Portal
Browser Special client
Executeapplication
Get results
Defineapplication
Gridapp.
E-mail, phone, skype, MSN, …
How can I integrate the How can I integrate the knowledge of many people knowledge of many people into a single and consistent into a single and consistent entity in the shortest time?entity in the shortest time?
21
Similar problems Similar problems have been already solved…have been already solved…
• Concurrent editing of Web pages: WIKI, MediaWiki
• Concurrent editing of documents: Online collaboration tool in MS Word
• Concurrent editing of tables: Online collaboration tool in MS Excel
• …
22
The solution: The solution: collaborative Grid portalscollaborative Grid portals
• Grid portals that support – concurrent engineering of applications
– concurrent management of application execution
Grid
Portal
Gridapplication
23
The CollaborativeThe Collaborative P-GRADE Portal P-GRADE Portal
Grid
P-GRADE Portal
Browser Special client Browser Special client
submitresults results
edit edit
24
The CollaborativeThe Collaborative P-GRADE Portal P-GRADE Portal
Grid
P-GRADE Portal
Browser Special client Browser Special client
submitresults results
edit edit
No No time-consuming time-consuming communicationcommunication
…………only efficient, only efficient,
workflow-workflow-oriented oriented
collaboration!collaboration!
25
Collaborative Grid Collaborative Grid workflowsworkflows
Collaborative workflow
Collaborative workflow
• Integrate knowledge – development phase• Generate and distribute results – execution phase
26
An e-Science exampleAn e-Science example
Collaborative workflow
Collaborative workflow
Topology of the graph
Input protein molecules
Experiment results
Protein folding
algorithm
27
A business exampleA business example
Collaborative workflow
Collaborative workflow
Graph topology
Input for the market
prediction application
Results of the
prediction workflow
Market prediction
model
28
Collaborative workflow Collaborative workflow developmentdevelopment
– The portal has to guarantee that the result is a consistent workflow
– Protect against data loss locking mechanisms– As real-time as possible! to increase parallelism
during development
29
Workflow development Workflow development modelmodel
• Locking workflow components (jobs) exclusively• First come first served• Only the owner can edit the component
30
Workflow development Workflow development modelmodel
• Locking workflow components (jobs) exclusively• First come first served
31
Workflow development Workflow development modelmodel
• Locking workflow components (jobs) exclusively• First come first served
32
ScreenshotScreenshot
Locked Locked for mefor me
Locked for Locked for somebody somebody
elseelse
UnlockedUnlocked
33
Portal server
Workflow editor #2
Requesting a lockRequesting a lock
Workflow editor #1
Lockrequest
Lockaccepted
34
Portal server
Synchronizing the Synchronizing the different viewsdifferent views
Workflow editor #1
Push updatedlocal view
Workflow editor #2
Pull global view
35
Result of the collaborative Result of the collaborative developmentdevelopment
Job
Job Job
JobJob
data data
datadata data
Job
data
Grid resource
Grid resourceGrid resource
Grid resource
Grid resource
Grid resource
36
Executing collaborative Executing collaborative workflowsworkflows
1. Concurrent control requestsIt can happen in single-user case too (multiple browsers are opened)
2. Different users’ proxies to be used to execute different workflow nodes
Grid 1Grid 1
Grid 2Grid 2
Grid 3Grid 3
P
1
P
3
P
2
37
ConclusionsConclusions
• Extend the single-user version of the P-GRADE Portal:– Concurrent engineering of workflow-based
grid applications– Application execution by multiple users in a
concurrent fashion– Prototype version already available
39
Two orthogonal aspects of Grid Two orthogonal aspects of Grid portals usage portals usage
• Number of accessible Grids– Single isolated Grid (SI)– Multiple isolated Grids (MI)– Multiple collaborative Grids (MC)
• Collaborative vice not collaborative usage– Multiple isolated users (MI)– Multiple collaborative users (MC)
40
Portal classificationPortal classification
Multiple Isolated users
(MIxx)
Multiple Collaborative users
(MCxx)
Single Isolated Grid(xxSI)
MISI portals (current portals)
MCSI portals
Multiple Isolated Grids(xxMI) MIMI portals MCMI portals
Multiple Collaborative Grids (xxMC) MIMC portals MCMC portals
41
LCG-2
P-GRADE-Portal
LondonRome
Athens
MISI Portal: P-GRADE portal 1.0MISI Portal: P-GRADE portal 1.0
42
Portal classificationPortal classification
Multiple Isolated users
(MIxx)
Multiple Collaborative users
(MCxx)
Single Isolated Grid(xxSI)
MISI portals (current portals)
MCSI portals
Multiple Isolated Grids(xxMI) MIMI portals MCMI portals
Multiple Collaborative Grids (xxMC) MIMC portals MCMC portals
Multi-Grid portals
43
GridLab
SEE-Grid
P-GRADE-Portal
MIMI Portal: P-GRADE portal (2.1)MIMI Portal: P-GRADE portal (2.1)
London Rome
Athens
44
Portal classificationPortal classification
Multiple Isolated users
(MIxx)
Multiple Collaborative users
(MCxx)
Single Isolated Grid(xxSI)
MISI portals (current portals)
MCSI portals
Multiple Isolated Grids(xxMI) MIMI portals MCMI portals
Multiple Collaborative Grids (xxMC) MIMC portals MCMC portals
45
GridLab
SEE-Grid
P-GRADE-Portal
London Rome
Athens
MIMC Portal: P-GRADE portal (2.1)MIMC Portal: P-GRADE portal (2.1)
46
Portal classificationPortal classification
Multiple Isolated users
(MIxx)
Multiple Collaborative users
(MCxx)
Single Isolated Grid(xxSI)
MISI portals (current portals)
MCSI portals
Multiple Isolated Grids(xxMI) MIMI portals MCMI portals
Multiple Collaborative Grids (xxMC) MIMC portals MCMC portals
47
SEE-Grid
P-GRADE-Portal
MCMC Portal: P-GRADE portal (3.0)MCMC Portal: P-GRADE portal (3.0)
GridLab
LondonRome
Athens
48
MI users WF GUI MC users WF GUI
SI GridWF
manager
MI Grids WF
manager
MC Grids WF
manager
Summary of classificationSummary of classification
.
.
.User
Grid
.
.
.
User
User
.
.. Grid
Grid
Grid
.
..
.
..
.
.
.User
.
.
.
User
User
.
..Grid
Grid
.
..
.
..
.
.
.
Grid
Grid
.
..
.
..
User
.
.
.
User
User
.
..Grid
Grid
.
..
.
..
P-GRADE 2.3P-GRADE 3.0+
49
ConclusionsConclusions
• xxMC portals: execute a workflow simultaneously in many Grids
• P-GRADE portals 2.1 - 2.3 already work as xxMC portal providing service for several production Grids:
– HunGrid– SEE-Grid– VOCE– UK NGS
www.lpds.sztaki.hu/pgportal
• MCxx portal: Enables to collaborate through workflow applications!
• Collaborative P-GRADE Portal: Prototype was demonstrated at SC05 exhibition