Parallelizing your code

To parallize your code, you have several options. The simplest way is to use OpenMP, which can be used if the memory on the machine is shared by the different nodes. For more advanced parallelization and when you want to run your code on clusters like Abel, where memory is distributed, we recommend the use of MPI (Message Passing Interface). In the following, we will show how you set things up and get started.


OpenMP (Open Multiprocessing) supports multiprocessing with shared memory. You do not have to install OpenMP because it is a feature of the compiler. To make sure that your compiler can be used, check the OpenMP website,, for a list of compilers and platforms that support OpenMP programming.

To use OpenMP, you should make sure to link to it when compiling. Here to examples, one for the g++-compiler on Linux, one for Intels compiler:

g++ -o myprogram myprogram.o -fopenmp

icpc -o myprogram myprogram.o -openmp



MPI (Message Passing Interface) can be used to parallelize the code on machines with distributed memory (e.g. on clusters). While with OpenMP, much of the parallelization (e.g. distributing data to different processors) happens hidden for the user,  MPI required everything to be implemented self, giving the user more control and minimizing overhead.

To use MPI, we recommand OpenMPI or MPICH2, two open source implementations, which are supported by most platforms. In contrast to OpenMP, MPI is a library that needs to be installed.
Note: The machines in the lab have MPI already installed, the following instructions are for your home PC/laptop.

Installation on Ubuntu

If you use Ubuntu, simply run

sudo apt-get install openmpi-bin openmpi-doc libopenmpi-dev

to install OpenMPI, all necessary libraries, and the documentation for the MPI calls.


Installation not using Ubuntu

If you so not use Ubuntu, we recommand the installation of MPICH2 (of course, you are also free to use it with Ubuntu). The following steps should bring you to a working library:

  1. Download the latest version of MPICH2 from Download the source code, uncompress the folder, and change into the MPICH2 directory.
  2. Configure your installation by performing ./configure. If you don’t have root access to your machine, you need to install MPICH2 to a local directory type ./configure --prefix=/installation_directory_path.  For more information about possible configuration parameters, type ./configure --help.
  3. When configuration is done, build and install MPICH2 with make; sudo make install

Compiling and running programs with MPI

To compile programs, use mpicxx. This is a wrapper for g++, so you can provide exactly the same options as you would usually with g++.

mpicxx -o myprogram myprogram.o -larmadillo -lblas -llapack

To run the program, you use mpirun hand specify the number of nodes with the option -n.
Example for running on 4 nodes:

mpirun -n 4 ./myprogram



By Sarah Reimann
Published Jan. 29, 2013 9:26 PM - Last modified Jan. 29, 2013 9:49 PM