Similar to the Grid Plugin for job submission and details, as of version 7.0 the Pipeline server uses another customize-able plugin for getting detailed grid usage and statistical information. You can implement your own “Grid Stat Plugin” for your grid resource manager. As shown in the diagram below, the Pipeline Grid Stat Plugin functions very similarly to the Pipeline Grid Plugin in that it is a bridge between the Pipeline server and the grid resource manager.
In Pipeline package there is a JAR file called pipeline-grid-stat-plugin.jar. The jar contains an interface called “GridStatPlugin”. Any custom grid stat plugin developed for a grid resource manager should implement this class. To implement your own plugin for your grid resource manager, you have to override the following 2 methods:
public Map<String, QueueUsageInfo> updateQueueUsageMap()
This method returns a map of Grid Queue names to their respective QueueUsageInfo objects.
QueueUsageInfo has only 3 fields:
- totalSlots: total number of slots that a queue can use (assuming no overlapping resources)
- usedSlots: number of occupied slots (by a running job) belonging to any node that the queue can submit to
- freeSlots: simply total – used
public GridDetails updateGridDetails()
This method returns a fairly complex object consisting of 3 collections. When the GridDetails object is initialized, the 3 collections are created and can be set/added to/retrieved.
- List gridHostList: A list of GridHost objects that comprise your grid
- Map<String, Set> queuedJobsMap: Map of users to a list of their queued jobs (GridJob objects) on the grid
- Map<String, Set> runningJobsMap: Map of grid hostnames to a list of the running jobs (GridJob objects) on that host
Implementing the functionality of those 2 methods requires an understanding of what a GridHost and GridJob are. Here is how we have defined these objects:
public class GridHost
The following properties of a grid host need to be set in your custom plugin:
- String name: common name of the host
- int usedSlots: number of used cores at the current moment
- int totalSlots: total number of cores on the host for running jobs
- float loadAverage: floating point value representing on average how many cores are in use
- Boolean enabled: if this is false, the host has been disabled or is not usable for whatever reason String state; //Some grid resources managers use String codes to represent host states (i.e. d=disabled, a=alarm)
- int loadPercent: used slots / total slots as a percent Set queueList; //list of queues serviced by this host
- Set<String> queueList: list of queues serviced by this host
- int reservedSlots: should be set to 0 if you are not sure what this is for
public class GridJob
The following properties of a grid job need to be set in your custom plugin:
- int id: job id given by the grid engine
- String name: common name of the job
- String user: owner of the job (user who submitted it)
- String status: some string representation of the state (i.e. r for running, q (or sometimes qw) for queued, E for error)
- String submitDate: job submission date
- String submitTime: job submission time
- int slots: number of slots that the job takes up, normally 1
- int taskId: if the job is part of an array submission, this will represent what number in the array group it is
- String queue: queue that the job was submitted from
- GridHost host: if the job is running, this should be set to the host object running the job, otherwise null
