Overview Ares Cluster Kronos Cluster Batch Queues Support Community

Programming Environment

Compilers

Several compiler suites are available on kronos. Table 6 summarizes the installed compilers.

Note: The MPI compilers are wrapper scripts which invoke one among Portland Group, Intel or GNU compilers depending on the module chosen - see module section below for more information.

Modules

The Modules Package is installed on the kronos cluster which allows users to quickly switch between different programming environments. The Modules package sets up appropriate environment variables such as PATH, MANPATH, LD_LIBRARY_PATH etc. depending on the module chosen. Table 7 lists commonly used modules commands.

The currently available modules as listed by the module avail command is shown below:

[user@kronos ~]$ module avail

------- /nethome/apps//Modules/versions ----------------------------------------------
3.2.6
----- /nethome/apps//Modules/3.2.6/modulefiles ---------------------------------------
                        fluent                  mpich2-intel            openmpi-intel
archives/fluent/6.3.26  ftnchek                 mpich2-pgi              openmpi-pgi
archives/icc/10.1.015   gmt/gcc-32              mpich2-pgi-test         paraview
archives/ifort/10.1.015 gmt/gcc-64              netcdf/gcc-32           pgi
archives/pgi/7.1-5      gromacs-gcc             netcdf/gcc-64           pvm
archives/pgi/7.2-1      intel                   netcdf/intel-32         python-2.6.1
blast                   java                    netcdf/intel-64         schrodinger
clustalw                matlab                  netcdf/pgi-32           use.own
dot                     module-cvs              netcdf/pgi-64           xmgrace
eclipse                 module-info             null                    yasara
emboss                  modules                 openeye
fftw                    mpich2-gcc              openmpi-gcc

[user@kronos ~]$

Typically, modules are loaded as a part of the login by placing module commands in the .bash_profile/.profile (for bash/sh/ksh users) or .login for (csh/tcsh users).

Compiling Sequential Programs

Assume your Fortran 90/C source program is stored in file 'program.f/program.c'. These files are compiled with the command:

Portland Group Fortran:

  pgf90 program.f -o program

Intel Fortran:

  ifort program.f -o program

Portland Group C:

  pgcc   program.c -o program

Intel Group C:

  icc   program.c -o program

The '-o program' option instructs the compiler to name the executable program 'program'. If the '-o' option is not used, the executable program is called 'a.out' by default. If your source program is spread over multiple files, say for example that a subroutine in the file 'program.f' calls a subroutine/function that is declared in the file 'program2.f/program2.c'. Then one can compile in two steps:

Portland Group Fortran:

  pgf90 -c program2.f
  pgf90 program.f program2.o -o program

Intel Fortran:

  ifort -c program2.f
  ifort  program.f program2.o -o program

Portland Group C:

  pgcc -c program2.c
  pgcc program.c program2.o -o program

Intel C:

  icc -c program2.c
  icc program.c program2.o -o program

The '-c' option instructs the compiler to create an object file (program2.o) instead of an executable file. This object file is used at a later step of the compilation. In the last step of the compilation, all object files (or one source file with other object files) are usually linked together to create the executable file. By default, the compiler links these object files together with several libraries that are present on the system and that contain implementations of the language standard (for example Fortran intrinsics like abs, sin, mod, ...).

When your code consists of many source files, the venerable unix make utility can automate the maintenance, update, compilation, and regeneration of object and executable files. The make utility requires by default a file called 'Makefile' in which you specify your compiler options, source/object files, and rules for compilation. After you have created the Makefile, you can build your program with the command 'make' or 'gmake' (GNU make). The /nethome/examples directory contains an example Makefile for GNU make (which we recommend), that can be used to build an example Fortran program. The sample Makefile provides a template and can be edited to suit your needs Writing Makefiles is not always easy but can simplify your life considerably. Contact user support for help when needed.

Compiling OpenMP Programs

Most options for compiling sequential programs can also be used for compiling parallel OpenMP (or threaded) programs. For example:

Portland Fortran:

  pgf90 -mp program.f

Intel Fortran:

  ifort -openmp program.f

Portland C:

  pgcc -mp program.c

Intel C:

  icc -openmp program.c

Compiling MPI Programs

Two open source MPI (Message Passing Interface) libraries MPICH3/OpenMPI are installed. A module corresponding to the compiler you wish to use must be loaded in order to set up the correct environment. MPI Libararies are specific to compiler and to networking protocol. For example, to use MPICH3 library compiled with the Intel compiler over the Ethernet networking protocol, which is the only protocol available, the command would be

module load intel openmpi-intel

Once the module is loaded the following commands should be used to compile programs that use MPI code:

Fortran:

  mpif90 program.f

C:

  mpicc  program.c

The 'mpi...' commands are wrapper shell scripts that invoke the chosen compiler. Flags are passed by mpif90 script to the backend compiler, so any of the compiler options can be used on the mpi shell script. By default, the mpi.. scripts pass the proper MPI header files to the compilers. It is therefore not necessary to specify the directory (normally with the '-I' option) where the MPI header file is located. The MPI compilers also link in the MPI libraries by default (and linking with something like '-L... -lmpi' is not needed).

Frequently used Compiler Options

The different compilers accept many options e.g. for debugging, optimization for code size or performance. Use 'man pgf90' or man ifort to get an overview of the options for the Fortran compilers and 'man pgcc/pgCC' or 'man icc' for the C/C++ compilers for detailed information on the available options.

For basic performance and run-time optimization a good starting point is to compile your code with the options

Portland Group:

	-fastsse 

Intel:

	-O3

Debugging Tools

Portland Linux Cluster Debugger (PGDBG)

You can use PGDBG to debug MPI-parallel and OpenMP thread-parallel programs on kronos. PGDBG complements the powerful MPI and OpenMP parallel performance analysis profiler, PGPROF. Some features of PGDBG:

Control Threads and Processes Separately - The PGDBG cluster debugger provides the ability to separately debug and control OpenMP threads, pthreads, and MPI processes on your Linux cluster. You can Step, Next, Break, Halt, Wait or Continue on threads or processes individually or collectively as a group.

Powerful GUI Windows - PGDBG provides a comprehensive set of GUI windows at your disposal to assist you in this difficult task. Using a single window, you now have precise control over each node of your cluster or each thread of your multiple-CPU SMP node. The Main GUI window displays source code for your F77, F95, C or C++ program and it has buttons for one-touch commands such as Run, Break, Quit, Print, or Stack traceback. Buttons on the Main window also allow you to select and control individual or collective threads and processes as well as the commands to access them. Input and output is displayed in the Main window or in a separate I/O Window. A Disassembly window shows you how the currently executing high-level source code has been compiled into assembly language. The assembly is interleaved with the source code as well. Assembly language Stepping is enabled as well as breakpoint indicators. Separate Active Process and Active Thread summary windows can be spawned that give you a current status on each active thread or process. A Register window can be used to display register values in a variety of formats - integer, hex, floating point, etc. Custom Display windows can be used to follow values for important variables or memory regions in your executing program. A separate I/O Window can receive the stdout and stderr from your debug session.

Intel Debuggers

The Intel Debugger is a component of the Intel compilers. It is a full-featured symbolic source code application debugger that helps programmers locate run-time programming errors (i.e., bugs) in their code. It provides extensive debugging support for the following languages: C, C++ and Fortran (including Fortran 90). It also provides a choice of command-line or graphical user interface (GUI). The command-line interface provides DBX-like (default) or GDB (GNU Debugger)-like (optional) command syntax. The GUI supports all Intel Debugger command-line functionality. The main window includes the basic debugging and convenience features, optional windows, various pop-up menus and dialog boxes, and a command-entry prompt. Intel Debugger works with the following compilers Intel C/C++, Fortran GCC, G++ and G77 compilers.

Basic Profiling

gprof is a simple text-based utility that provides procedural-level profiling of serial and parallel codes. This helps users to identify how much time is being spent in subroutines and functions. xprofiler generates a graphical display of the performance, and provides application profiling at the source statement level. Both gprof and xprofiler are very simple to use:

• Compile the code with the -pg option, in addition to optimisation flags. If you use xprofiler, using -g in addition to the -pg option will offer profiling at source line level, however the -g will degrade the performance and is incomptible with some optimisation flags (e.g. inlining).
• Run the parallel code as usual.
• Each process will write an additional file to disk named gmon.out.pid
• Process the output with gprof or xprofiler where exec_file is the name of the compiled executable:
• gprof exec_file gmon.out.pid

Portability Tips

The following table 8 may help you in porting your code (esp. makefiles) from IBM to LINUX and/or to change compilers on the Linux cluster. Always check options carefully before using it.

---