Kellogg Linux Cluster
System Description
The Kellogg Linux Cluster (KLC) is a group of 10 high-memory Linux servers, or "nodes," each of which has 1.5 - 2.0 TB of RAM. The latest generation of nodes (klc0304 - klc0307) each have 64 CPU cores, and the next-older generations (klc0201 - klc0303) have 52 CPU cores
Networked storage is shared among all the KLC servers. This includes a private 80 GB home directory for each user and 600 TB of network storage available for project directories, where you can share files with each other.
You connect to KLC by making a secure shell (ssh) connection to a particular node. To help you decide which node to use, the table below shows the amount of CPU and RAM currently available on each node. The links will open a FastX session in your browser that connects to that node.
Why use KLC?
The KLC servers are far more powerful than even top-of-the-line personal workstations. The high amounts of memory, storage, and CPU available there allow you to tackle much bigger computational problems and work with much larger data files than you otherwise would. In addition, because KLC is a shared system, it is well suited for enabling work that is both highly collaborative and easily reproducible.
KLC offers advantages for those who are familiar with Kellogg's high-performance Windows servers. KLC offers the same, vast library of scientific computing software that Northwestern Quest uses. As a Linux-based system, KLC also offers straightforward ways for you to schedule jobs to run at certain times and to script sequences of tasks, both of which can save you effort and help with reproducibility.
Access
KLC is intended for Kellogg researchers and their collaborators. Current tenure-line faculty and PhD students automatically have access. Please contact Kellogg Research Support to request access for others.
Job Priority
Although you can submit as many jobs as you like, each user is allowed to run processes on up to 24 CPU cores concurrently across all the KLC nodes at normal priority. When one goes beyond this limit, all their processes incur a reduction in priority. This is how we protect users from having their work slow down because somebody else is using too much of the system.
If your work needs more than 24 CPU cores at a time, please ask Kellogg Research Support to advise you on your options.
Quest Inter-Operability
KLC is actually a special part of NUIT's Quest High Performance Computing facility. Your files on KLC are also available on Quest, and this connectivity allows you to easily scale your computation up to thousands of cores on Quest, if necessary.
Working with KLC
You have two basic choices for how to connect to a remote server such as KLC.
One option is to use a secure shell (ssh) client. An ssh client will usually give you the option of working either with a point-and-click graphical interface or through a plain-text command line interface. Which ssh client you use depends on whether you are connecting from a Windows or a Mac computer.
Alternatively, you can use FastX as a simple way to get a graphical interface. With FastX, you could open Stata on KLC directly through your web browser, for example, and the experience would be much like running Stata on your personal computer.
Which One Should I Use?
Here are a few considerations to keep in mind when choosing whether to use ssh or FastX.
Graphical interfaces are slower than plain text interfaces are. If you open a program with X11, it might take a few seconds to load when the non-X11 version would load almost instantly. Like the name implies, FastX is relatively fast compared to other X11 tools. So FastX is a wise choice if you want a graphical interface.
Most ssh clients come with file transfer (sftp) capability built into them. With FastX, however, you will need to install and use a separate program (like Cyberduck) in order to move files to and from the KLC servers.
In general, we suggest becoming comfortable making ssh connections and using the command line, at least occasionally. Although point-and-click interfaces are convenient and intuitive, relying on graphical interfaces will present challenges to the future reproducibility of your work.
-
Kellogg Windows computers do not come with a pre-installed ssh client. So you will need to choose one and install it yourself. Research Support staff like to use MobaXterm, which has a very good free version. Because neither Research Support nor KIS can support the MobaXterm application, you should consider purchasing their pro version in order to access technical support. Other popular ssh clients include SecureCRT, PuTTY, and Cygwin.
The image below shows how you can set up a connection to KLC using MobaXterm.- In the Sessions menu, select SSH.
- In Remote Host, give the address of the server (klc.northwestern.edu.)
- Specify your NetID as the username.
- Under Advanced SSH settings, check the box for X11-Forwarding if you want a graphical interface; leave it unchecked for a plain text interface.
Notice in the example above, we have previously defined several user sessions. Some use X11 forwarding while others do not. Some connect to the KLC load balancer while others connect to specific nodes. Still others connect to different sessions, such as WRDS. If you work on remote servers regularly, you will find it convenient to save and label a set of commonly used sessions.
-
Getting a FastX graphical interface to a KLC server is as simple as putting your web browser to one of these server addresses.
http://klc0301.ci.northwestern.edu:3300http://klc0302.ci.northwestern.edu:3300
http://klc0303.ci.northwestern.edu:3300
http://klc0304.ci.northwestern.edu:3300
http://klc0305.ci.northwestern.edu:3300
http://klc0306.ci.northwestern.edu:3300
http://klc0307.ci.northwestern.edu:3300
http://klc0201.ci.northwestern.edu:3300
http://klc0202.ci.northwestern.edu:3300
http://klc0203.ci.northwestern.edu:3300
- After you log in with your NetID and password, click the "Launch Session" button, and then select the application you want to run. By default, the sessions you start through the FastX web client will persist indefinitely - until either you terminate the session or the server is restarted.
- If you need to find a session you launched previously, you can find it on the My Sessions tab, but you must be logged on to the same KLC node where that session was launched! Therefore, we advise you to remember the specific node you are on if you intend to run very long jobs on KLC. We also advise you to Terminate sessions (under the Actions button in My Sessions) when you are done with your session.
-
Mac computers come with a the Terminal program already installed as a standard program. To create a "plain" (not graphical) connection to a specific KLC node (for instance KLC0201), you would simply open Terminal and type:
ssh your-netid@klc0201.ci.northwestern.edu
If you prefer a graphical (X11) interface, just add the -Y option:
ssh -Y your-netid@klc0201.ci.northwestern.edu
Occasionally you might receive an error that says something like "host key verification failed." Most of the time, you can fix these problems by deleting your host key file and trying again:
rm ~/.ssh/known_hosts
Note: Depending on the version of your Mac, you might need to install the free XQuartz application before X11 forwarding will work.
- <tab> key auto-completes program and file names
- <up> key goes through your command history
- <ctrl>-<c> interrupts the current command
- ~ refers to your home directory
- * is used to match multiple files
- exit - quit the current shell (terminal)
- man [program] - show documentation for a program
Module management:
- module avail - list available modules
- module load [module] - load a module
- module list - list loaded modules
- module rm [module] - unload a module
- module load R; R
- module load stata; stata-mp
- module load sas; sas -memsize 0 -sortsize 0 -maxmemquery 0 -work /kellogg/tmp [program.sas]
- module load matlab/r2016a; matlab
File Commands:
- ls - list contents of directory
- ls -la - include hidden files and show details
- cd [dir] - change directory
- cd - change to your home directory
- pwd - print working directory
- mkdir [dir] - create a directory
- rm [file] - delete a file
- rmdir [dir] - delete an empty directory
- cp [file1] [file2] - copy file1 to file2
- cp -r [dir1] [dir2] - copy directory
- cat [file] - print the contents of a file
- less [file] - browse a file
- head [file] - print the beginning of a file
- tail [file] - print the end of a file
- touch [file] - create an empty file
- ln -s [file] [link] - create a symbolic link
- df -h - list free disk space on system
- du -sh [dir] - print size of folder's contents
- nano [file] - a simple editor
- vi [file] - a complex editor
- emacs [file] - another complex editor
Archiving:
- tar tvf [tarball] - list contents of a tarball
- tar xvf [tarball] - extract a tarball
- tar czvf [tarball.tar.gz] [file1] [file2] ... - create a tarball
- unzip [zipfile] - unzip a zipfile
- zip [zipfile] [file1] [file2] ... - create a zipfile
- ps - list your currently active processes
- top - list all processes on the system
- w - list who is logged in and what they are doing
- kill [pid] - kill a process by process id
- killall [process] - kill all processes by name
- screen - a background process manager
- bg - list bash background jobs
- fg - brings the more recent bash bg job to the foreground
- chmod go-rwx [file] - revoke permissions for group and others to read, write, or execute a file.
- chmod +x [file] - make a file executable by everyone
- You can mix and match the u,g,o,+,-,r,w,x parameters as needed.
- grep [pattern] [files] - search for a regular expression pattern in the files.
- grep -r [pattern] [dir] - search recursively in a directory
- [command] | grep [pattern] - search for a pattern in the output of a command
- find . | grep [pattern] - search for a file whose name matches a pattern
- find . -name "*.txt" - more efficiently find all files ending in ".txt"
- diff [file1] [file2] - show the difference between two files
Pipes:
- [command1] | [command2] - use output of command1 as input to command 2
- [command] > [file] - save output of command to a file
- [command] >> [file] - append output of command to a file
- [command] &> [file] - save standard output and error output to a file
Environment variables:
- env - list environment variables
- export PATH=[dir]:$PATH - add a directory to your executable search path
- which [application] - show location of a program
- source [file]- run commands in a file
- ./[script_file] - run an executable script in the current folder
- ssh [user]@[hostname] - log into another machine
- wc -l [file] - count lines in a file
- awk "{print \$3}" [file] - print the third word of every line
- sort [file] - sort the lines of a file alphabetically
- sort [file] | uniq - print unique lines in a file
- crontab -e - define jobs to be run periodically
- [command] & - run command in the background
- nohup [command] - allow command to continue running even if terminal is closed
- [command1]; [command2]; ... - run a sequence of commands
- my_var="hello world" - set a variable $my_var to the value "hello world"
- my_var=$([command]) - save a command's output in a variable
- echo ${my_var} - print the value of a variable
- wget [url] - download a file
- lynx [http_address] - browse the web
- echo "message body" | mail -s "message subject" recipient@gmail.com - send an email
- finger [username] - print information about a user
- uptime - show system's uptime
- cat /proc/cpuinfo - list CPU info
Directory Conventions
All KLC users have a home directory where you can store up to 80 GB of files. Your home directory is completely private, meaning that you cannot share files in your home directory with other users. You can point programs to your home directory with the path/home/your-netid/ or the shortcut ~.For projects where you more storage or when you want to share files with other collaborators, you will need a "project" directory. Project directories reside under
/kellogg/proj/. Ask Kellogg Research Support if you want to create a project directory or want to change the other users who share that directory with you.Note: Each KLC project directory will have a strict capacity limit, or "quota." Tenure-line faculty and PhD students are entitled to a project directory with a default quota of 2 TB. Researchers may purchase additional quota at the same price that Northwestern IT charges for Quest storage. The current price is $195 per TB for up to 5 year of access.
Viewing and Editing Files
You have many options for viewing and editing your files that are saved on KLC.Text Editors
KLC has several text editor programs installed. If you work on a Linux system with any regularity, we recommend that you become familiar working with at least one text editor. Nano is probably the most intuitive program for a new user to learn. Other popular text editors include Emacs and vi. You can open a file directly with Nano by typing the command
nano name-of-the-file...and then exit the file by typing CTRL-X.
Emacs also provides a graphical user interface, which you can use by issuing this command if your ssh session has X11 forwarding:
emacs name-of-the-file &FastX / Gnome
If you connect to KLC using FastX and the Gnome Desktop, then you can also simply double-click a file to open it with a program called Gedit. Gedit is another intuitive graphical interface for viewing and editing text files.
Edit Files Locally
Some users prefer the third option of using their personal computers to view and manipulate the files they create on KLC. Suppose you run a Matlab program on KLC to create an output file. You could download the file you created (see Transferring Files below for instructions.) This option is especially attractive if your program produces images or PDF files as its output.
Transferring Files
You have many options for transferring files between your computer and KLC. Northwestern IT maintains detailed file transfer instructions for Quest, of which KLC is part. We summarize some of the most popular options below.- ssh clients like MobaXterm have drag-and-drop sftp capability built into the product.
- Cyberduck is a free program that also offers drag-and-drop sftp capability. The same company offers Mountain Duck for a small license fee.
- For transferring large files or transferring files between universities, consider using Globus.
KLC offers the same large library of scientific computing software - or, "modules" - that you find on the campus-wide Quest system. At Kellogg, the most popular programs are R, Stata, Matlab, SAS, and Python. See the Software on Quest for a full list of titles. This page describes details for how you can launch and monitor your jobs on KLC.
Job Limits
Although you can submit as many jobs as you like, each user is allowed up to 24 CPU cores concurrently across all the KLC nodes at normal priority. When one goes beyond this limit, all their processes incur a reduction in priority. This is how we protect users from having their work slow down because somebody else is using too much of the system.
If your work needs more than 24 CPU cores at a time, please ask Kellogg Research Support to advise you on your options.
-
Often you will want to run programs in batch mode - that is, executing a program you have already written in a program such as Stata without using any graphical or interactive interfaces. Running in batch mode definitely makes your work easier when you need to run the same pieces of code repeatedly, and it definitely helps make your work more reproducible.
Suppose you have a program called "my_program" that is written in Stata, SAS, or one of the other popular languages for statistical analysis. Below you can find the recommended command line syntax for running "my_program" in batch mode:
Matlab
[~]$ matlab -nosplash -nodesktop -r my_program &
R
[~]$ Rscript my_program.R &
SAS
[~]$ sas -work /kellogg/tmp -memsize 175G my_program.sas &
Note: This examples sets the maximum working memory for SAS to 175 GB. We overwrite the default memory allocation that Quest has set because the Quest limit tends to be much lower than the memory typically available on the KLC servers. You may need to adjust the number up or down depending on the size of data you want SAS to use.
Stata
[~]$ stata -b do my_program.do &
-
In order to run licensed software on KLC or Quest, you must first "load" that software module. For example, if you want to run a Python program that uses the Anaconda release of Python version 3.6, these two commands will do that:
[user@klc01 ~]$ module load python/anaconda3.6 [user@klc01 ~]$ pythonIf you want to check with modules are available to load, you can see the full list (which is very long!) by using
module avail. You can also filter the results by adding a case-sensitive search term aftermodule avail. For example, if you wanted to see all the versions of Stata available, you could type:[user@klc01 ~]$ module avail stata stata/14-b1041 Where: D: Default Module /software/Modules/3.2.9/modulefiles stata/14 (D) stata/15 Use "module spider" to find all possible modules. Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".Notice, in this example, the default version of Stata is version 14. This is the version that you would get if you simply typed
module load stata. We recommend you to specify an explicit version whenever you load software on KLC. This is especially important when you use open-source programs like Python or R.If you then want to run Stata version 15 with a graphical interface, you would just type:
[user@klc01 ~]$ module load stata/15 [user@klc01 ~]$ xstata-se & [1] 24389Similarly, to run RStudio with version 3.3.3 of R, you would just type:
[user@klc01 ~]$ module load R/3.6.0 [user@klc01 ~]$ rstudio & [1] 25305 -
One advantage of running jobs on KLC is so that you can allow your long-running jobs to continue for several days, if necessary, to complete. It is important for you to understand when KLC will automatically keep your jobs running and when it will automatically terminate your jobs.
If you make an ssh connection and launch jobs (either batch or interactive), then those jobs will die abruptly when you log out of your session. To keep your jobs running after you log out, you can either...
- Run your jobs inside a
screensession, or - Use
nohupbefore running your batch process. For example:nohup stata -b do my_program.do &
To check on the status of your jobs after you have logged out, you must log in to the same KLC node as before. Then you can issue this command:
ps -u your-netidto see all of the processes running under your NetID on that node.
You can terminate any unwanted or orphaned processes with the following command:kill -9 the-PID-numberWhen you make a FastX connection, the behavior is the opposite. FastX will continue to run your jobs even after your close your browser window and reboot your computer. Your connections will remain active until you explicitly Terminate the FastX session. For this reason, we recommend you take care to terminate sessions after your work is complete.
One final piece of advice: Even though it is possible to run jobs on KLC for a very long time, your jobs will not always run indefinitely. Sometimes servers need to reboot for maintenance. Other times, a server could go offline because of an unplanned crash. It is always a good idea to write your programs in such a way that your work will not be completely lost if your jobs end early. Long, iterative jobs should write out intermediate output periodically and be written in a way that you could easily restart them from their last checkpoint.
- Run your jobs inside a