Running jobs with LSF

Introduction

A single laptop or desktop PC may be used to process, analyse and visualise biological datasets. However, these datasets can be quite large, making computational analysis on a single machine painfully slow and sometimes impossible. In these situations, a compute farm can be used to run the analysis more efficiently. In this tutorial, we will look at what a compute farm is and how we can use a job manager, such as LSF, to perform computational tasks more efficiently.

Learning outcomes

On completion of the tutorial, you can expect to be able to:

• view cluster structure using LSF
• view queue information using LSF
• submit and manage jobs using LSF
• submit job arrays using LSF
• submit dependent jobs using LSF
• understand priority and fairshare in LSF
• troubleshoot LSF issues

Tutorial sections

This tutorial comprises the following sections:

1. Introduction
2. Job queues
3. Submitting jobs
4. Managing jobs
5. Job arrays and dependencies
6. Priority and fairshare
7. Troubleshooting
8. LSF cheat sheet

Authors

This tutorial was written by Victoria Offord.

Running the commands from this tutorial

You can run the commands in this tutorial either directly from the Jupyter notebook (if using Jupyter), or by typing the commands in your terminal window.

Running commands on Jupyter

If you are using Jupyter, command cells (like the one below) can be run by selecting the cell and clicking Cell -> Run from the menu above or using ctrl Enter to run the command. Let's give this a try by printing our working directory using the pwd command and listing the files within it. Run the commands in the two cells below.

Running commands in the terminal

You can also follow this tutorial by typing all the commands you see into a terminal window. This is similar to the "Command Prompt" window on MS Windows systems, which allows the user to type DOS commands to manage files.

To get started, select the cell below with the mouse and then either press control and enter or choose Cell -> Run in the menu at the top of the page.

Now open a new terminal on your computer and type the command that was output by the previous cell followed by the enter key. The command will look similar to this:

cd /home/manager/pathogen-informatics-training/Notebooks/LSF/

Now you can follow the instructions in the tutorial from here.

Let’s get started!

This tutorial assumes that you are connected to a compute farm with LSF installed. To check that you have installed these correctly, you can run the following commands:

This should return the help message for bqueues, bsub and bjobs respectively.

To get started with the tutorial, head to the first section: Introduction