Using Batch Processing

Using the SP

Batch processing is an extremely powerful way to do computationally-intensive work.  The main system for doing batch processing at IU (for research purposes) is the IBM RS/6000 Scalable POWERParallel System, known as the Research SP System.

The Research SP System is made up of several different nodes.  Matlab, Maple and Mathematica are available on libra. If you're doing batch processing, you can log into any node and (assuming you follow the instructions) the job will execute on the right node.

Please note that the SP is intended for use only as a batch processing system.  All processes that accumulate more than 20 minutes of CPU time in an interactive (non-batch) setting will be terminated.

Preparing to batch process

If this is your first time batch processing, we recommend you first attempt a "smaller" version of your task.  Don't try to solve your research problem on the first attempt.

Create a text file that has the exact command you which to run.  This file will be fed into your math package as if you typed it in.  In other words, each line in the text file should be the exact same thing as what you would type at the Maple, Matlab, or Mathematica prompt.  You are free to have your text file load other files or worksheets, but be sure to use the full path when specifying filenames.  You can find out the full path by typing pwd . This tells you the "present working directory".  For example, if you want to load a file in your home directory named "work.mws", you should give the filename as "/N/u/john_smith/work.mws".  The same goes for all output commands.

Remember, you won't be able to interact with the math software while it is running.  This means you need to save the results of your work in some way.  You should probably use the math package's built-in save function.  For graphical output, we recommend outputting to a file format such as VRML, GIF, or JPEG.

Also, you must test your plain text file before executing it on the SP.  We recommend testing it on Steel.  If you saved your text file as "foo.txt", you can test it using the command maple < foo.txt > foo.out (or matlab, or math).  This feeds foo.txt to Maple one line at a time, and any output that would normally go to the screen is redirected to a new file called foo.out.  Make sure that your list of commands does exactly what you think it will.

After you have perfected your math package command script, you are ready to batch proccess it.

Writing a LoadLeveler Script

The SP uses a program called LoadLeveller to do batch processing.  You wrap up all the commands neccesary to do your job in a file called a LoadLeveller Script.
A good tutorial on using LoadLeveller is  The UITS Research SP System - Submitting Batch Jobs to the LoadLeveler .
Sample scripts can be found in the /N/u/statmath/SP/scripts directory on the SP.

Previous: Using Graphics over the Network
Next: Further Reading