Coarray example from Michael Ehrlichman. Intel documentation at:
	<https://software.intel.com/sites/products/documentation/hpc/ics/ics2013/ics_tutorial/Linux_co_array_fortran.htm>

-----------------------------------------------------------------------------------------

The following environment setup worked at Cornell as of August 2014.

Necessary for Building:
export ACC_ENABLE_MPI=0

Necessary for Running:
export PATH=/nfs/opt/intel/composer_xe_2013_sp1.1.106/composer_xe_2013_sp1.1.106/mpirt.disabled/bin/intel64:$PATH
export LD_LIBRARY_PATH=/nfs/opt/intel/composer_xe_2013_sp1.1.106/composer_xe_2013_sp1.1.106/mpirt.disabled/lib/intel64/:$LD_LIBRARY_PATH

Set the number of threads at runtime using:
export FOR_COARRAY_NUM_IMAGES=2

-----------------------------------------------------------------------------------------
Note for Cornell People: 

As of August 14, 2013, the lab version of Ifort has Intel's MPI
disabled due to conflicts with the system MPI.

From Attilio:
Just to give some background on Intel's MPI implementation, it's to
add run-time support for Coarrays in Intel's ifort and not intended to
be a full MPI solution nor is it compatible with other MPI packages
(e.g. OpenMPI installed in our SL6.4 Linux image).  For more
background, please see:
  http://www.polyhedron.com/web_images//intel/productbriefs/8_CAF.pdf
and 
 http://software.intel.com/en-us/forums/topic/402426

-----------------------------------------------------------------------------------------
From: Michael Ehrlichman [mailto:mpe5@cornell.edu] 
Sent: Thursday, July 24, 2014 6:05 AM
To: David Carl Sagan; Christopher Earl Mayes
Subject: Parallelizing PTC


Hello,

PTC seems to be incorrigibly not thread safe.

I have found a really simply way to get past this and write parallel
PTC codes.  It is to use coarrays.  An example has been copied to:
/home/ehrlichm/svn_bmad_dist_cmake/for_dcs.

When a program with coarrays is run, several copies are spawned.  The
number spawned is the number of cores on the computer where the binary
was compiled, but this can be overridden with the environment variable
FOR_COARRAY_NUM_IMAGES.  The copies are independent processes, which
of course have their own memory and are not sensitive to thread
safety.  Those variables that are declared as coarrays are shared
among the copies.  The coarray variables are indexed by process id.

coarrays only work with the ifort compiler. gfortran support of coarrays is very beta.

It is not necessary to recompile the entire distribution with -coarray.

Cheers, Mike

-----------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------