How to run GridMPI with PSPacer

This document describes a step-by-step installation and test procedure of GridMPI with PSPacer

($Date: 2008/07/07 05:30:16 $)

Contents

1. Prerequisite: Installation of PSPacer

You need to install and setup PSPacer on all nodes. The source codes of both iproute2 and libnl are required for the compilation. To compile PSPacer, issue the following commands: 

$ ./configure --with-iproute2-pkg=/path/iproute2-.tar.gz \
  --with-libnl-pkg=/path/libnl-1.0-pre6.tar.gz
$ make
# make install
Check the files in the installation directories. 
/lib/modules/`uname -r`/kernel/net/sched/sch_psp.ko
/usr/lib/tc/q_psp.so
/usr/local/lib/libnl.so*

Hereinafter, we assume that the network interface, which is used for the inter-cluster (external) communication, is eth0.

First of all, you should confirm that PSPacer works well by using iperf. To control the transmission rate at 500 Mbps, issue the following commands: 

# /sbin/tc qdisc add dev eth0 root handle 1: psp default 1
# /sbin/tc class add dev eth0 parent 1: classid 1:1 psp rate 500mbit
# /sbin/tc qdisc add dev eth0 parent 1:1 handle 10: pfifo
To change the transmission rate: 
# /sbin/tc class change dev eth0 parent 1: classid 1:1 psp rate 800mbit
To remove the setting: 
	# /sbin/tc qdisc del dev eth0 parent 1:1 handle 10:
	# /sbin/tc class del dev eth0 parent 1: classid 1:1
	# /sbin/tc qdisc del dev eth0 root handle 1:

For your further information, please see PSPacer project page.

2. Installation of GridMPI

No special operation is required. Do ./configure & make & make install. PSPacer support (--with-libpsp) is enabled by default on the Linux platform. 
$ mpicc -qconfig|grep psp --with-libpsp yes
Check the library collpsp.so in the $MPIROOT/lib directory. Set the following two environment variables on each host: 
_YAMPI_EXTS=$MPIROOT/lib/collpsp.so
IMPI_PSP_OPT=eth0,1:1,1250000000
IMPI_PSP_OPT is a comma-separated triplet (dev,classid,rate), where dev is the name of an interface; classid is a class ID, which equals to a classid parameter of the tc command; and rate is the inter-cluster bandwidth. rate is optional. The default value is the physical bandwidth of the interface.

3. Execution of pspd

The GridMPI library communicates with PSPacer via a pspd daemon process. pspd should be run on all compute hosts with root permission. Set the configuration of PSPacer before pspd runs: 

# /sbin/tc qdisc add dev eth0 root handle 1: psp default 1
# /sbin/tc class add dev eth0 parent 1: classid 1:1 psp mode 0
# /sbin/tc qdisc add dev eth0 parent 1:1 handle 10: pfifo

NOTE: If the libnl library, which is provided by the Linux distribution, is installed before you will install the PSPacer, set the environment variable LD_LIBRARY_PATH to use a proper (i.e., PSPacer supported) library as the following: 

# LD_LIBRARY_PATH=/usr/local/lib /usr/sbin/pspd -d

a. Changes of NAS Parallel Benchmarks

To use this feature, changing code of a target application is required.

Here is a patch for NPB version 3.2. In this case, applications set the target transmission rate by using the following code. The function nhosts_in_client returns the number of hosts in the cluster. The function psp_change_nconns regulates the transmission rate at "inter-cluster-bandwidth / (n / 2)". 

call nhosts_in_client(n)
call psp_change_nconns(n / 2)