Batch Systems

The following document describes the batch queueing system that have been evaluated at CDF for data handling for Run II data analysis. The list of batch systems include:

Network Queuing System (NQS)
Load Sharing Facility (LSF)

This document provides a high level overview of each system's pros and cons as well as a list of features. Any additional batch queuing system will be added to this document as they are evaluated. At the end of this document there is a side by side feature comparison table of each evaluated system.

Network Queuing Service (NQS)

Latest Version: 3.50.1

Features:

Client/Server architecture.
Supports SMP processor sets for parallel batch processing (for OS's that the supports them).
Supports multiple queue types.

batch
device
pipe

Supports user and group access at the queue level.
Limited support for devices.
Provides account mapping.
Provides generic load balancing.
Provides a method for dynamic "cluster-wide" scheduling.
Future support for file staging.
Persistent state data for job restarts after critical/fatal error recovery.
Allows pre- and post- command execution around a submitted batch job.

Pros:

Software is free.
Source code distributed.
Explicitly written for UNIX systems (quotas, resource allocation, logging...).
Implements basic queuing functionality (job tracking, suspending, termination etc.)

Cons:

NQS depends heavily on the underlying UNIX system to enforce resource limits and quotas.
Simplistic load balancing does not take into account of other system activity outside of NQS.
Users are able to abuse scheduling algorithm and "flood" the NQS system.

Additional Information:

www.shef.ac.uk/~nqs

[ Back up to batch system list ]

Load Sharing Facility (LSF)

Current Version: 3.0

Features:

Client/Server architecture.
Offers multiple scheduling algorithms (applied at host and/or queue level).

First Come First Serve (default)
host partition fair share
queue level fair share
pre-emptive and pre-emptable
exclusive

Provides granular configuration of queues and host machines.
Enforces effective load balancing/sharing and cpu normalization within a batch cluster.
Can run jobs in parallel (if jobs are parallel in nature).
Provides account mapping.
Allows external program access to import or export data into the system.
Provides check pointing and job migration.
Allows pre- and post- command execution around a submitted batch job.
Highly configurable.

Pros:

LSF is already used in-house (FNALU).
Customized programs and scripts have already been created to interface with LSF (fbatch, accounting, etc.).
Supports many flavors of UNIX.
Can be used in conjunction with an existing NQS cluster.
Low CPU overhead for resident LSF processes (between 3-5% of total CPU on each machine).
Designed to scale very well in a configuration with many machines.

Cons:

Installation and setup are nontrivial and can get complex.
Rather expensive commercial software (with no source code).
Does not directly support tape devices.

More Information:

www.platform.com/Products/TheLSFSuite/Batch

[ Back up to batch system list ]

Side by Side Comparison of Features


Batch Feature	NQS	LSF
Account Mapping	X	X
External Programming Hooks	-	X
Device Support	X*	-
Pre/Post Commands	X	X
Parallel Jobs	X	X
Load Balancing	X **	X
Multiple Queue Types	X	-
Multiple Sched Algorithms	-	X
Checkpointing	-	X
Job Migration	-	X
Job Tracking	X	X
Delayed Job Execution	-	X
File Staging	-	X
Queue Level Resource Management	X	X
Host Level Resource Management	-	X
Source Code Available	X	-
Client/Server	X	X

* Only does load balancing within the NQS system itself (not total system load).
** Does not have direct tape drive/utility support.

[ Back up to batch system list ]