This file is /cdf/run1/documents/development/cdfsga_batch.note
Last update was November 27th, 2001


We are using the NQS batch system on cdfsga.  This is the preferred method 
of running long jobs on cdfsga because some scheduling is provided.
This scheduling is important to us, as we have found that the interactive
response on cdfsga starts getting somewhat ragged once the number of 
hard running jobs exceeds 29.

And so, the running of long jobs in batch keeps the interactive 
response on cdfsga "bouncy", which is an important goal for us.


(To be precise, this version of NQS is called CONNECT:Queue and
 is from Sterling Software)

 
1. The queues
=============
We have three queues:
   LPQ   intended for long jobs, for example monte carlo generation.
	 This queue has no time limit. The priority is b218.
   MPQ   intended for medium jobs, for example stripping events from tape.
	 This queue has a CPU time limit of 4 hours. The priority is b216.
   HPQ   intended for short jobs, for example linking an executable.
	 This queue has a CPU time limit of 5 minutes. The priority is b214.

The LPQ is set to allow 22 jobs to run at one time during the
daytime (7:30 AM to 5:30 PM) on weekdays, and to allow 25 jobs 
at once during nights and weekends.   


2. Job submission
=================
A typical command for submitting a batch job is
   > qsub -me -re -ro -q LPQ \
     -o /spool/username/job.log \
     -e /spool/username/job.err \
     <script name>

Note, that the argument to qsub is a script (not an executable).

You can create script file that submit your jobs. A example is given below.
#!/bin/csh -f
set jobname = $1
echo $PATH | wc -m
qsub -mb -me -re -ro -q LPQ \
     -o $USR_SCRATCH/$LOGNAME/$jobname.log \
     -e $USR_SCRATCH/$LOGNAME/$jobname.err \
     -r $jobname $jobname

This example takes as an argument the job name (must be the
name of a script file). It writes both standard output and 
standard error to the user's scratch area with the same name 
as the job (as on the VAX). It sends mail when the job starts 
and completes.


3. Using the MPQ and LPQ queues during busy times
=================================================

There are some times when cdfsga gets extremely busy,
for example, the week before a collaboration meeting.  

During these busy times, in order to be kind to your collegues, 
please arrange things so that you do not have any of
the following conditions:

   1) More than three of your jobs actively running in the MPQ at once or
   2) More than four of your jobs actively running in the LPQ at once or
   3) More than six jobs total actively running at once.

An aid to making this happen is the -a qualifier on the qsub command,
an example is:
   > qsub -a "tomorrow 2:00" -q MPQ -re -ro <script name>
which has the job wait until after 2 AM before it runs.


4. Monitoring your jobs
=======================
The batch queues may be monitored with the command
   > qstat -a 
This command can tell you the request-id of the batch job.

If one has used the qualifiers -re -ro on the qsub command,
the log file will be written to disk as the job runs.  It
can be read at any time.  The "tail" command is especially useful.


5. Killing jobs
===============
A batch job can be killed with the command
   > qdel -17 <request-id>
A good signal to use is -17, which is trapped by A_C. This will also
deallocate any tape drives that you have in your job (although you should
check to make sure that they really were deallocated).  Because tapes are not
part of the system on UNIX they do not get dismounted etc when the job dies,
unless A_C takes care of them.


6. Documentation
================
The commands
   > man qsub
   > man qstat 
   > man qdel
give important additional information about using batch queues.


7. Bug Fixes
============

Note: the bug which prevented writing to physics group disk
such as /data07 has been fixed by Sterling Software.

Also Note: The bug which prevented more than 4194303 bytes from being 
written to a file in a job has been fixed.

The bug which allowed jobs to run even after their CPU limit
expired has been fixed.


8. One typical problem
======================
   
A reminder from page 38 of the UNIX C shell field guide

   "The first character in a script must be the symbol #"

NQS really insists that this be there.


9. Another common problem
=========================

In order to be able to access the "man" pages for nqs,
one needs make sure to execute the command

   source /usr/local/etc/fermi.cshrc

at some point, either directly in the file $HOME/.cshrc 
or by hand.
  

10. A third problem
===================

If one obtains the error message:

   qsub: Command not found

or similar,

one needs to execute the command

   source /usr/local/etc/fermi.cshrc

at some point, either directly in the file $HOME/.cshrc 
or by hand.
  

This script in turn executes /usr/local/etc/local.cshrc 
which adds the nqs executables to the "path".


11. A bug
=========

When the PATH environmental variable is longer than 1024 characters,
then the qsub command will report that it has submitted a job, 
for example with a message such as:

   Request 7368.cdfsga submitted to queue: LPQ.

However, this job won't actually be submitted (!)

If you see this odd and frustrating behaviour, 
check (just before the qsub command) the length of the
PATH with the command:

   > echo $PATH | wc -m


12. Another bug
======================

When the log file for any batch job completely fills up 
a physical disk, the batch system reacts to this error by
not starting any new jobs for any users (!).

Please try to avoid filling up a physical disk with your log files, 
to avoid inconveniencing your collegues.  It is suggested that the log 
file only be written to the /spool disk, which has disk quotas.  
The disk quotas on /spool help to prevent the physical disk from 
filling up.

If you notice that the batch queue on cdfsga has stopped,
please call Fermilab Extension 2345.  Press whatever
buttons are necessary to talk with a person, and report
the problem.  Thanks !






13. A note about tapes
======================

As 8mm tapes are not perfectly reliable, it might be
best to read just one tape per job.  Reading many
tapes in a single job probably will not work.


14. A rare problem
==================

Some users have redefined their path with a command similar to:
set path = (/usr/sbin /usr/bsd /usr/bin /bin /usr/bin/X11 /home/lv0/rebcdf/bin)
in the .cshrc file.  This is not always a good idea, as this way
one cannot find certain system commands (for example, the batch system
commands).

Other users have better success by **adding** to the path, in a manner 
similar to:
set path = ( $path $HOME/bin )


15. Scripts in Batch
====================

Normally, scripts run the same way either in batch or interactively.
However, in some cases, a script may need to know whether it
is running in batch or running interactively.   One of the many ways 
to distinguish the cases is:

if ($?REMOTEHOST) then
   echo "Interactive Session"
else
   echo "Batch Session"
endif



If you obtain error messages similar to:

    stty: : Function not implemented

or

    TERM: Undefined variable.

you may need to apply this technique.



16. Beeping
===========

Some people like the terminal to beep at them when the job
completes.  One way to do this is by intercepting the mail
using techiques from the mail system.  Please see the mail manual
   "MH and xmh, E-mail for Users & Programmers", By Jerry Peek,
    Published August 1993 by O'Reilly & Associates.
    Section 11.2 for more information.

One creates the file 
    $HOME/.maildelivery 
with the lines
    from root ^ R "/usr/products/mh/v6_8_3c/lib/rcvtty"
    default - > A /var/mail/username

And one creates the file                                               
    $HOME/.forward
with the line
    |/usr/products/mh/v6_8_3c/lib/slocal


Questions or comments ?  Please send mail to 
   cdfsga-team@fnal.gov