Introduction to glideinWMS

CERN, Dec 2012 glideinWMS Intro 1

glideinWMS for users

Introduction to glideinWMSby Igor Sfiligoi (UCSD)


Scope of this talk

This talk provides auser perspective of glideinWMS

for users with previous experiencewith Grid computing.

It does not provide much detailbut concentrates on the concepts

behind the system instead.


The problem(s)

● Users have many jobs that must be run● Each user has

multiple tasks at once

● Resources provided by O(100) Grid sites

● How do we schedule them to get the results in the shortest amount of time?● Assuming one result per task

● How do we treat all users in a fair way?● Independently of how many jobs they submit


glideinWMS approach

● Separates● Resource provisioning from

● Resource scheduling● In practice, sends out pilot jobs to the Grid

● And creates an “overlay batch system”● Pilot jobs get ownership of the Grid slots

● At least for a limited time

● Known also as the pilot approach

Never theuser jobs!

Grid Site

Grid Site

Grid Site

Grid Site

Grid Site

Overlaybatch

system


Job scheduling

● Once we have the overlay batch system,job scheduling works like in any dedicated B.S.● We own the B.S. and can set the

job scheduling policies● glideinWMS based on HTCondor

● So can do whatever HTCondor can do● Which is quite flexible

– But also nothing more...

● HTCondor-based pilots are usually called glideins

HTCondor(formerly knowns as Condor)

is a widely usedbatch system.

More detailslater on.

Thus glideinWMS stands for “glidein based Workflow Management System”


Creating the overlayi.e. resource provisioning

● glideinWMS will grow and shrink the overlay B.S. automatically● No human intervention needed

● Expansion based on user jobs in the queue● The more jobs, the faster it will try to grow● Since not all jobs can run at all sites,

different attempted growth rates for different sites● Shrinks automatically if resources unused

● Again, based on user jobs in the queue Each glidein should run

at least one user job,but will try to run many

if the Grid slot is long enough


glideinWMS in a picture

Grid siteHTCondor

CPU Handler

glideinWMS

User Job

Grid SiteGrid Site

HTCondorJob

Repository


From the user point of view

● Users see just a “regular” HTCondor system● Just a dynamic one

● However● Have to be aware of the resource provisioning logic

– No native HTCondor tools to help with this● Debugging system problems much harder

– Again, no native HTCondor tools to help with this– Most likely question a user will ask is

“Why is my job not starting?”


A few more details

● So, glideinWMS is really HTCondor++● First you have to understand how HTCondor works● If you do, 99% of the problems are solved

● HTCondor is composed of 3 logical pieces● Submit nodes – keep the job queue(s)● Execute nodes – owns and operates a resource● A central manager – glues the other two categories

together and executes policies


HTCondor in a picture

Central manager

Condor

Submit node

Condor

Execute node

Condor

Submit node

Submit node

Execute node

Execute node

Execute node

Execute node


Even more details

● The actual work quanta are● Jobs – on the submit node, typically many/node

● Slots – on the execute node, typically only a few/node

● While internally implemented differently,jobs and slots are conceptually very similar● Both describe a logical entity● Both have attributes describing it● Both have requirements

● HTCondor policy engine is really all aboutmatchmaking jobs to slots

“ClassAd” inHTCondor speak


Matchmaking

● Jobs that are not running(i.e. are “idle” in HTCondor speak)will be matched against Slots that don't yet run anything(i.e. are “Unclaimed” in HTCondor speak)

● Requirements expressions can (and usually do) reference attributes in the other ClassAd

● Both sides must evaluate to True for a match● Although we encourage all logic to reside in the

Slot Requirement in glideinWMS setups(more on this later on)


A non-technical example

Pet Ad MyType = “Pet” TargetType = “Buyer” Requirements = DogLover == True PetType = “Dog” Color = “Brown” Price = 75 Breed = "Saint Bernard" Size = "Very Large" ...

Buyer Ad MyType = “Buyer” TargetType = “Pet” Requirements = (PetType == “Dog”) && (Price <= AcctBalance) && (Size == "Large"||Size == "Very Large") AcctBalance = 1000 DogLover = True LegalName = “Curly Howard” . . .

Dog == Resource ~= SlotBuyer ~= Job


Matching order

● Most of the time, there are way more idle jobs than Unclaimed slots● So order is important

● Two policies1) Jobs from highest priority user first2) Priority-FIFO policy for jobs of the same user

● User priority based on usage● The more resources you use, the lower the priority

(with priority recovery over time)

● But some users may be marked as “more important”(priority multipliers and group quotas)


The glideinWMS part

● i.e. The layer on top of HTCondor that finds resources where to start the “Execute node” daemons● i.e. glideins

● Composed of two parts:● Glidein Factory – The abstraction layer● VO Fronend – The brain


Glidein Factory

● The splitting in two allows for theGlidein Factory to be generic● I.e can (and should) be shared between many VOs

● The G.F. is really just an abstraction layer● It insulates the VO from the provisioning details

– e.g. knowing the name of the Grid CE and relative RSL● Allows new technology to be added seamlessly

(e.g. Clouds)

● It also provides a troubleshooting service● The factory operators are supposed to

address any Grid related problems they observe

i.e. servemany VO FEs


The VO Frontend

● The name may be misleading● It is really the “matchmaker of Grid resources”

● Introduces a new quanta● Entry – logical equivalent of a “queue at a site”

Basic working block of a G.F.● The VO Frontend

1) Matches idle Jobs to Entries2) Instructs the affected G.F. to

increase or decrease the number of glideins on that Entry

Thus regulates theresource provisioning


Updated glideinWMS picture

Central manager

Condor

Submit node

Condor

Execute node

Condor

Submit node

Submit node

Execute node

Execute node

Execute node

Execute node

Grid

G.F.

G.F.VO FE

+3

+1


VO FE Matchmaking logic

● Based on Job attributes● Jobs don't have “FE-specific requirements”

● The exact matchmaking policy depends on the VO FE instance

● glideinWMS has 2 level matchmaking● Once in the FE, then in the HTCondor C.M.● Recommended to avoid explicit

“HTCondor requirements” in the Job ClassAd– The glideins should set “Slot requirements” based on the

same attributes used by the VO FE, instead

Will describe CMS policies in a different talk

Since VO FE configures the glideins


What is the user to do?

0) Learn how to use HTCondor1) Learn what the VO FE policy is2) Create the HTCondor submit file (i.e. JDL)

containing the necessary attributes3) Submit jobs4) Wait for the results to come back5) Rinse and repeat (from (2))


This is it

● Hopefully you have a high level view of the system now

● More details in separate talks


Pointers

● glideinWMS Home Pagehttp://tinyurl.com/glideinWMS

● HTCondor Home Pagehttp://research.cs.wisc.edu/htcondor/

● HTCondor [email protected]@cs.wisc.edu

● glideinWMS [email protected]

http://tinyurl.com/glideinWMS

http://research.cs.wisc.edu/htcondor/

mailto:[email protected]




Acknowledgments

● The creation of this document was sponsored by grants from the US NSF and US DOE,and by the University of California system

Introduction to glideinWMS

Technology

Transcript of Introduction to glideinWMS