Monitoring and managing your jobs on the cluster

The UMSL clusters use a software package called ROCKS to manage jobs sent to them. You can read more about ROCKS here.

Commands:

qstat

To see the status of all the jobs that have been sent to the cluster, type "qstat -u '*'".

To see only your jobs, type "qstat -u yourusername".

How do I interpret the qstat output? The man page for qstat (type, "man qstat") gives a detailed explanation of the the output and a number of options that you can use with the command. I provide a summary here.

Here is a typical output:

qstat output

  • job-ID - self-explanatory, a unique identifying number for your job. You can use this number to control the job (see qdel below).
  • prior - gives the priority of the job. You can give jobs differing priorities (see the man pages). You probably will not need to concern yourself with this.
  • name - the job name.
  • user - the user who submitted the job.
  • state - the state of the job - one of d(eletion), t(ransfering), r(unning), R(estarted), s(uspended), S(uspended), T(hreshold), (q)w(aiting) or h(old). See the man page for details.
  • submit/start at - the submission or start time and date of the job.
  • queue - the queue the job is assigned to (for running or suspended jobs only). There are two sets or racks (referred to as rack 0 and rack 1) of 32 dual processor computers on expedition. The computers are numbered 0 through 31. The queue "compute-0-10.q" means that your job is running on the number 10 computer of rack 0. There are two processors per computer, so you will often see two jobs running on the same computer. Knowing which computer you job is using is useful for troubleshooting. For example, if you consistently have a job that takes a relatively long time to run or fails when sent to a certain computer, it may be that that computer is having problems. You should notify the cluster manager if this occurs.
  • master - the function of the running jobs (MASTER or SLAVE - the latter for parallel jobs only).
  • ja-task-ID - the array job task id. Will be empty for non-array jobs. Don't worry about this.

qdel

To delete a queued or running job, type "qdel job-ID".

You can delete more than one job at a time. For example, "qdel 868834 868835 868839".

To delete all of your current jobs, type "qdel '*'".

To delete a range of jobs, type "qdelmult startjob-ID endjob-ID". For example, "qdelmult 868834 868850", will delete jobs from 868834 through 868850.