Gamess is an electronic structure calculation package. Its installation is easy if you just want to use “sockets” communication mode. Just emerge it as you regularly do. Then use “rungms” to submit your job. The default rungms is okay to run the serial code. For the parallel computation, you still need to tune the script slightly. But since our cluster has Infiniband installed, it is better to go with the “mpi” communication mode. It took me quite some time to figure out how to install it correctly and make it run with mpiexec.hydra alone or with OpenPBS (Torque). Here is how I did it.
Software packages related:
1. gamess-20101001.3 (Dowload it beforehand from its developer’s website)
2. mvapich2-1.7rc1. (Previous versions should be okay and I installed it under /usr/local/)
3. OFED-1.5.3.2. (Userspace libraries for Infiniband. See my previous post. Only updated kernel modules installed. Userspace libraries should be the same as in OFED-1.5.3.1)
4. torque-2.4.14 (OpenPBS)
Steps
1. Update the gamess-20101001.3.ebuild with this one and manifest it.
2. Unmask the mpi user flag for gamess in /usr/portage/profiles/base/package.use.mask.
3. Add sci-chemistry/gamess mpi to /etc/portage/package.use; then emerge -av gamess.
4. Update rungms with this one;
5. Create a new script pbsgms as this one;
6. Add kernel.shmmax=XXXXX to /etc/sysctl.conf, in which XXXXX is a large enough integer for shared memory (default value 32MB is too small for DDI). Run /sbin/sysctl -w kernel.shmmax=XXXX to update the setting in-the-fly.
Added on Sept. 9, 2011. It seems that kernel.shmall=XXXXX should be modified as well. Please bear in mind that the unit for kernel.shmall is pages and kernel.shmmax is bytes. And a page is 4096 bytes in usual(use getconf PAGE_SIZE to verify).
7. Environment setting. Create a file /etc/env.d/99gamess
GMS_TARGET=mpi
GMS_SCR=/tmp/gamess
GMS_HOSTS=~/.hosts
GMS_MPI_KICK=hydra
GMS_MPI_PATH=/usr/local/bin
Then update your profile.
8. Create a hostfile, ~/.hosts
node1
node2
...
This file is only needed by invoking rungms directly.
9. Test your installation: copy a test job input file exam20.inpunder/usr/share/gamess/tests/; submit the job using pbsgms exam20 (other settings will be prompted), or using rungms exam20 00 4.
Explanations
1. Two changes were made on the ebuild file.
(a). The installation suggestions given in the documentation of Gamess is not enough. More libraries other than mpich are needed to pass over to lked, the linker program for Gamess.
(b) MPI environment constants are needed to exported to the installation program, compddi through an temporary file install.info.
2. Many changes were made for the script, rungms. I could not remember all of them. Some are as following.
(a) For parallel computation, the scratch file will be put under /tmp on each node by default.
(b) The script will be working with pbsgms.
(c) System-wide setting for Gamess can be put under /etc/env.d.
(d) A host file is needed if not using PBS. By default, it should be at ~/.hosts. If not found, running on the local host only.
3. The script pbsgms is based on sge-pbs shipped with the Gamess installation package. I have made it to work with Torque. Numerous changes were made.
Tags: gamess, gentoo, infiniband, mpi, mvapich2, pbs, torque